# The Math Inside the Model *The math that makes a model learn, and the math that lets it compare.* Before training, GPT-4 is 1.8 trillion numbers. Random ones. Noise. After training, those same numbers can write poetry, debug code, and explain quantum mechanics. ## Two things to understand This post answers two questions, and both come down to math you can work through by hand. **How does a model learn the right numbers?** It starts with random weights, measures how wrong they are using a loss function, figures out which direction to adjust using derivatives, and steps toward better numbers through optimization. By the end of Part 1, we'll have done this ourselves — found the exact best weights for a simple model, using the same math that trains GPT-4. **How does a model compare things?** Those learned numbers encode meaning in a space where similar things land nearby — "king" and "queen" are close, "king" and "banana" aren't. Part 2 covers the dot product and cosine similarity that make this work. We'll build both from one running example: predicting Uber fares from ride data. (Builds on the [first post](/blogs/foundation-model-internals-1) — vectors, embeddings, Word2Vec — but works standalone.) --- ## Part 1: Learning — finding the right weights ### The setup You take three Uber rides: ``` 5 km → ₹120 8 km → ₹250 12 km → ₹310 ``` You want to predict the fare for a 10 km ride. Not by guessing, but by finding a pattern. Plot the three rides: <svg viewBox="0 0 500 240" style="width:100%;max-width:500px;display:block;margin:1.5em auto;"> <defs> <marker id="si-arrowX" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6" fill="#64748b"/></marker> <marker id="si-arrowY" markerWidth="6" markerHeight="8" refX="3" refY="0" orient="auto"><path d="M0,8 L3,0 L6,8" fill="#64748b"/></marker> </defs> <!-- axes --> <line x1="55" y1="200" x2="430" y2="200" stroke="#64748b" stroke-width="1" marker-end="url(#si-arrowX)"/> <line x1="55" y1="208" x2="55" y2="10" stroke="#64748b" stroke-width="1" marker-end="url(#si-arrowY)"/> <text x="445" y="205" fill="#64748b" font-size="10" font-family="IBM Plex Mono">km</text> <text x="12" y="10" fill="#64748b" font-size="10" font-family="IBM Plex Mono">fare ₹</text> <!-- x ticks --> <text x="55" y="215" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">0</text> <text x="148" y="215" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">4</text> <text x="241" y="215" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">8</text> <text x="334" y="215" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">12</text> <line x1="148" y1="198" x2="148" y2="202" stroke="#64748b" stroke-width="1"/> <line x1="241" y1="198" x2="241" y2="202" stroke="#64748b" stroke-width="1"/> <line x1="334" y1="198" x2="334" y2="202" stroke="#64748b" stroke-width="1"/> <!-- y ticks --> <text x="48" y="200" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">0</text> <text x="48" y="147" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">100</text> <text x="48" y="94" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">200</text> <text x="48" y="41" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">300</text> <line x1="53" y1="147" x2="57" y2="147" stroke="#64748b" stroke-width="1"/> <line x1="53" y1="94" x2="57" y2="94" stroke="#64748b" stroke-width="1"/> <line x1="53" y1="41" x2="57" y2="41" stroke="#64748b" stroke-width="1"/> <!-- light grid lines --> <line x1="55" y1="147" x2="420" y2="147" stroke="#334155" stroke-width="0.5" opacity="0.3"/> <line x1="55" y1="94" x2="420" y2="94" stroke="#334155" stroke-width="0.5" opacity="0.3"/> <line x1="55" y1="41" x2="420" y2="41" stroke="#334155" stroke-width="0.5" opacity="0.3"/> <!-- data points — smaller, with subtle outline --> <circle cx="171" cy="136" r="4.5" fill="#4ade80" stroke="#4ade80" stroke-width="1" opacity="0.9"/> <circle cx="241" cy="67" r="4.5" fill="#4ade80" stroke="#4ade80" stroke-width="1" opacity="0.9"/> <circle cx="334" cy="35" r="4.5" fill="#4ade80" stroke="#4ade80" stroke-width="1" opacity="0.9"/> <!-- labels — all consistently to the right --> <text x="183" y="140" fill="#4ade80" font-size="10" font-family="IBM Plex Mono">(5, ₹120)</text> <text x="253" y="71" fill="#4ade80" font-size="10" font-family="IBM Plex Mono">(8, ₹250)</text> <text x="346" y="39" fill="#4ade80" font-size="10" font-family="IBM Plex Mono">(12, ₹310)</text> <!-- 10 km question — larger, more visible --> <circle cx="288" cy="50" r="8" fill="none" stroke="#e5c07b" stroke-width="1.5" stroke-dasharray="4,3"/> <text x="288" y="54" fill="#e5c07b" font-size="11" font-family="IBM Plex Mono" font-weight="600" text-anchor="middle">?</text> <text x="310" y="54" fill="#e5c07b" font-size="10" font-family="IBM Plex Mono">10 km</text> </svg> </div> There's clearly a pattern — fare goes up with distance. The simplest version of that pattern is a straight line through these dots. **That line is your model.** Give it any distance, it returns a predicted fare. Every straight line is described by a function: $f(x) = w \cdot x + b \qquad \text{where } y = f(x)$ Here $x$ is the input (distance in km) and $y$ is the output (predicted fare). But two numbers control everything about the line's shape: $w$ (the **weight** — how steep the line is, how much the fare goes up per km) and $b$ (the **bias** — where the line starts, the base charge when distance is zero). <svg id="wb-svg" viewBox="0 0 460 220" style="width:100%;max-width:540px;display:block;margin:1.5em auto;"> <!-- axes --> <line x1="60" y1="190" x2="420" y2="190" stroke="#64748b" stroke-width="1"/> <line x1="60" y1="190" x2="60" y2="15" stroke="#64748b" stroke-width="1"/> <text x="430" y="195" fill="#64748b" font-size="10" font-family="IBM Plex Mono">x</text> <text x="50" y="12" fill="#64748b" font-size="10" font-family="IBM Plex Mono">f(x)</text> <!-- ticks x --> <text x="60" y="205" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">0</text> <text x="132" y="205" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">2</text> <text x="204" y="205" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">4</text> <text x="276" y="205" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">6</text> <text x="348" y="205" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">8</text> <line x1="132" y1="188" x2="132" y2="192" stroke="#64748b" stroke-width="1"/> <line x1="204" y1="188" x2="204" y2="192" stroke="#64748b" stroke-width="1"/> <line x1="276" y1="188" x2="276" y2="192" stroke="#64748b" stroke-width="1"/> <line x1="348" y1="188" x2="348" y2="192" stroke="#64748b" stroke-width="1"/> <!-- ticks y --> <text x="52" y="190" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">0</text> <text x="52" y="148" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">100</text> <text x="52" y="106" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">200</text> <text x="52" y="64" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="end">300</text> <line x1="58" y1="148" x2="62" y2="148" stroke="#64748b" stroke-width="1"/> <line x1="58" y1="106" x2="62" y2="106" stroke="#64748b" stroke-width="1"/> <line x1="58" y1="64" x2="62" y2="64" stroke="#64748b" stroke-width="1"/> <!-- dynamic line --> <line id="wb-line" x1="60" y1="186" x2="348" y2="102" stroke="#4ade80" stroke-width="2.5"/> <!-- bias annotation --> <circle id="wb-bdot" cx="60" cy="186" r="4" fill="#f472b6"/> <line id="wb-bdash" x1="65" y1="186" x2="105" y2="186" stroke="#f472b6" stroke-width="1" stroke-dasharray="4,3"/> <text id="wb-blabel" x="108" y="183" fill="#f472b6" font-size="10" font-family="IBM Plex Mono">b = 10</text> <text id="wb-bsub" x="108" y="195" fill="#64748b" font-size="9" font-family="IBM Plex Mono">base fare at x=0</text> <!-- weight annotation: rise over run at x=4..6 --> <circle id="wb-wdot1" cx="204" cy="144" r="3" fill="#e5c07b"/> <circle id="wb-wdot2" cx="276" cy="123" r="3" fill="#e5c07b"/> <line id="wb-whoriz" x1="204" y1="144" x2="270" y2="144" stroke="#e5c07b" stroke-width="1.5"/> <polygon id="wb-whoriz-head" points="270,141 276,144 270,147" fill="#e5c07b"/> <text id="wb-wrun" x="238" y="158" fill="#e5c07b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">+2 km</text> <line id="wb-wvert" x1="276" y1="144" x2="276" y2="126" stroke="#e5c07b" stroke-width="1.5"/> <polygon id="wb-wvert-head" points="273,126 276,120 279,126" fill="#e5c07b"/> <text id="wb-wrise" x="288" y="136" fill="#e5c07b" font-size="9" font-family="IBM Plex Mono">+₹50</text> <text id="wb-wlabel" x="310" y="148" fill="#e5c07b" font-size="10" font-family="IBM Plex Mono">w = 25</text> <text id="wb-wsub" x="310" y="160" fill="#64748b" font-size="9" font-family="IBM Plex Mono">₹25 per km</text> <!-- formula badge --> <rect x="100" y="24" width="260" height="30" rx="6" fill="#1e293b" stroke="#334155" stroke-width="1"/> <text id="wb-formula" x="230" y="45" fill="#c8d6e5" font-size="15" font-family="IBM Plex Mono" font-weight="600" text-anchor="middle">f(x) = 25x + 10</text> </svg> <div style="display:flex;gap:1.2em;justify-content:center;flex-wrap:wrap;margin:0.3em 0 0.5em;font-family:'IBM Plex Mono';font-size:12px;color:#94a3b8;"> <label style="display:flex;align-items:center;gap:0.4em;">w <span id="wb-wv" style="color:#e5c07b;min-width:2.5em;text-align:right;">25</span> <input id="wb-ws" type="range" min="0" max="50" value="25" step="1" style="width:110px;accent-color:#e5c07b;"></label> <label style="display:flex;align-items:center;gap:0.4em;">b <span id="wb-bv" style="color:#f472b6;min-width:3.5em;text-align:right;">10</span> <input id="wb-bs" type="range" min="-50" max="100" value="10" step="1" style="width:110px;accent-color:#f472b6;"></label> </div> (function(){ function mX(x){ return 60 + x*36; } function mY(y){ return 190 - y*0.42; } var el = function(id){ return document.getElementById(id); }; function update(){ var w = parseFloat(el('wb-ws').value), b = parseFloat(el('wb-bs').value); el('wb-wv').textContent = w; el('wb-bv').textContent = b; // line x=0..8 el('wb-line').setAttribute('y1', mY(b)); el('wb-line').setAttribute('y2', mY(8*w+b)); // bias annotation var yb = mY(b); el('wb-bdot').setAttribute('cy', yb); el('wb-bdash').setAttribute('y1', yb); el('wb-bdash').setAttribute('y2', yb); el('wb-blabel').setAttribute('y', yb-3); el('wb-blabel').textContent = 'b = '+b; el('wb-bsub').setAttribute('y', yb+9); // weight annotation at x=4..6 var x4=mX(4), x6=mX(6), y4=mY(4*w+b), y6=mY(6*w+b); el('wb-wdot1').setAttribute('cy', y4); el('wb-wdot2').setAttribute('cy', y6); el('wb-whoriz').setAttribute('y1', y4); el('wb-whoriz').setAttribute('y2', y4); el('wb-whoriz').setAttribute('x2', x6-6); el('wb-whoriz-head').setAttribute('points', (x6-6)+','+(y4-3)+' '+x6+','+y4+' '+(x6-6)+','+(y4+3)); el('wb-wrun').setAttribute('y', y4+14); // vertical arrow — direction follows sign of w var tipY = y6 + (w>=0 ? 6 : -6); el('wb-wvert').setAttribute('y1', y4); el('wb-wvert').setAttribute('y2', tipY); el('wb-wvert-head').setAttribute('points', (x6-3)+','+tipY+' '+x6+','+y6+' '+(x6+3)+','+tipY); var midY = (y4+y6)/2; var rise = Math.round(2*w); el('wb-wrise').setAttribute('y', midY+4); el('wb-wrise').textContent = (rise>=0?'+':'')+'₹'+rise; el('wb-wlabel').setAttribute('y', midY+16); el('wb-wlabel').textContent = 'w = '+w; el('wb-wsub').setAttribute('y', midY+28); el('wb-wsub').textContent = '₹'+w+' per km'; // formula var sign = b>=0?' + ':' − '; el('wb-formula').textContent = 'f(x) = '+w+'x'+sign+Math.abs(b); } el('wb-ws').addEventListener('input', update); el('wb-bs').addEventListener('input', update); update(); })(); </div> The ride data $x$ is given to us. $w$ and $b$ are what we need to find — and there are infinitely many possible lines, one for every combination of $(w, b)$. Which one fits the data best? To answer that, we need a way to **score** how wrong each candidate line is — a single number that's low when the line fits well and high when it doesn't. ### Measuring wrongness — the loss function For any choice of $w$ and $b$, each ride has a prediction error: what the model predicted minus what actually happened. **We square the error so that positive and negative misses don't cancel out**, and so that bigger misses get punished disproportionately — a miss of ₹100 costs $100^2 = 10{,}000$, while a miss of ₹10 costs only $10^2 = 100$, a hundred times less: $\text{error}_i = (w x_i + b - y_i)^2$ Average the squared errors across all rides and you get the **loss function** (also called **mean squared error**): $\text{Loss}(w, b) = \frac{(w x_1 + b - y_1)^2 + (w x_2 + b - y_2)^2 + (w x_3 + b - y_3)^2}{3} \qquad \text{(shorthand: } L \text{)}$ Loss takes any $(w, b)$ and returns one number: how wrong that line is. **Our objective: find the $w^\ast$ and $b^\ast$ that minimize Loss.** That's training. Use the sliders below to try different values of $w$ and $b$ — watch how the line changes and Loss goes up or down: <svg id="rg-svg" viewBox="0 0 500 340" style="width:100%;max-width:520px;display:block;margin:1.5em auto;"> <defs> <marker id="rg-arrowX" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6" fill="#64748b"/></marker> <marker id="rg-arrowY" markerWidth="6" markerHeight="8" refX="3" refY="0" orient="auto"><path d="M0,8 L3,0 L6,8" fill="#64748b"/></marker> </defs> <!-- axes --> <line x1="60" y1="280" x2="470" y2="280" stroke="#64748b" stroke-width="1" marker-end="url(#rg-arrowX)"/> <line x1="60" y1="290" x2="60" y2="15" stroke="#64748b" stroke-width="1" marker-end="url(#rg-arrowY)"/> <text x="475" y="285" fill="#64748b" font-size="11" font-family="IBM Plex Mono">km</text> <text x="15" y="15" fill="#64748b" font-size="11" font-family="IBM Plex Mono">fare ₹</text> <!-- axis ticks --> <text x="60" y="296" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">0</text> <text x="168" y="296" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">4</text> <text x="276" y="296" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">8</text> <text x="384" y="296" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">12</text> <line x1="168" y1="278" x2="168" y2="282" stroke="#64748b" stroke-width="1"/> <line x1="276" y1="278" x2="276" y2="282" stroke="#64748b" stroke-width="1"/> <line x1="384" y1="278" x2="384" y2="282" stroke="#64748b" stroke-width="1"/> <text x="55" y="280" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">0</text> <text x="55" y="214" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">100</text> <text x="55" y="148" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">200</text> <text x="55" y="82" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">300</text> <line x1="58" y1="214" x2="62" y2="214" stroke="#64748b" stroke-width="1"/> <line x1="58" y1="148" x2="62" y2="148" stroke="#64748b" stroke-width="1"/> <line x1="58" y1="82" x2="62" y2="82" stroke="#64748b" stroke-width="1"/> <!-- error lines (updated by JS) --> <line id="rg-err1" x1="0" y1="0" x2="0" y2="0" stroke="#e5c07b" stroke-width="1.5" stroke-dasharray="4,3"/> <line id="rg-err2" x1="0" y1="0" x2="0" y2="0" stroke="#e5c07b" stroke-width="1.5" stroke-dasharray="4,3"/> <line id="rg-err3" x1="0" y1="0" x2="0" y2="0" stroke="#e5c07b" stroke-width="1.5" stroke-dasharray="4,3"/> <!-- model line --> <line id="rg-line" x1="60" y1="280" x2="460" y2="280" stroke="#f472b6" stroke-width="2"/> <!-- data points (on top) --> <circle cx="195" cy="201" r="6" fill="#4ade80"/> <circle cx="276" cy="115" r="6" fill="#4ade80"/> <circle cx="384" cy="76" r="6" fill="#4ade80"/> <!-- predicted dots on line --> <circle id="rg-pd1" cx="0" cy="0" r="4" fill="#f472b6" opacity="0.7"/> <circle id="rg-pd2" cx="0" cy="0" r="4" fill="#f472b6" opacity="0.7"/> <circle id="rg-pd3" cx="0" cy="0" r="4" fill="#f472b6" opacity="0.7"/> <!-- point labels --> <text x="195" y="218" fill="#4ade80" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">(5, ₹120)</text> <text x="276" y="108" fill="#4ade80" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">(8, ₹250)</text> <text x="384" y="69" fill="#4ade80" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">(12, ₹310)</text> <!-- loss badge --> <rect x="300" y="6" width="190" height="28" rx="6" fill="#1e293b" stroke="#334155" stroke-width="1"/> <text id="rg-loss" x="395" y="25" fill="#e5c07b" font-size="13" font-family="IBM Plex Mono" font-weight="600" text-anchor="middle">loss = 0</text> </svg> <div style="display:flex;gap:1.2em;justify-content:center;flex-wrap:wrap;margin:0.3em 0 0.2em;font-family:'IBM Plex Mono';font-size:12px;color:#94a3b8;"> <label style="display:flex;align-items:center;gap:0.4em;">w <span id="rg-wv" style="color:#f472b6;min-width:2.2em;text-align:right;">25</span> <input id="rg-ws" type="range" min="0" max="50" value="25" step="0.5" style="width:110px;accent-color:#f472b6;"></label> <label style="display:flex;align-items:center;gap:0.4em;">b <span id="rg-bv" style="color:#f472b6;min-width:3em;text-align:right;">10</span> <input id="rg-bs" type="range" min="-80" max="120" value="10" step="1" style="width:110px;accent-color:#f472b6;"></label> </div> <div style="text-align:center;font-family:'IBM Plex Mono';font-size:12px;color:#94a3b8;margin:1em 0 0.3em;font-weight:600;">loss landscape — loss for each value of w</div> <canvas id="rg-bowl" width="500" height="140" style="width:100%;max-width:520px;display:block;margin:0 auto;"></canvas> <div style="text-align:center;font-family:'IBM Plex Mono';font-size:11px;color:#64748b;margin:0.3em 0 0.5em;">drag the w slider — the line above changes, and the dot below tracks your position on the loss curve</div> (function(){ var data = [[5,120],[8,250],[12,310]]; var mapX = function(km){ return 60 + (km/14)*400; }; var mapY = function(fare){ return 280 - (fare/350)*214; }; var line = document.getElementById('rg-line'); var err = [document.getElementById('rg-err1'),document.getElementById('rg-err2'),document.getElementById('rg-err3')]; var pd = [document.getElementById('rg-pd1'),document.getElementById('rg-pd2'),document.getElementById('rg-pd3')]; var lossT = document.getElementById('rg-loss'); var ws = document.getElementById('rg-ws'), bs = document.getElementById('rg-bs'); var wv = document.getElementById('rg-wv'), bv = document.getElementById('rg-bv'); var bowl = document.getElementById('rg-bowl'), bctx = bowl.getContext('2d'); function lossAt(w,b){var s=0;for(var i=0;i<3;i++){var e=w*data[i][0]+b-data[i][1];s+=e*e;}return s/3;} function drawBowl(curW,curB){ var W=500,H=140;bctx.clearRect(0,0,W,H); var pad=50,maxL=0; var pts=[]; for(var ww=0;ww<=50;ww+=0.5){ var l=lossAt(ww,curB);pts.push([ww,l]);if(l>maxL)maxL=l; } maxL=Math.min(maxL,80000); function mx(w){return pad+(w/50)*(W-pad-20);} function my(l){return H-25-(Math.min(l,maxL)/maxL)*(H-40);} // curve bctx.beginPath(); for(var k=0;k<pts.length;k++){ var px=mx(pts[k][0]),py=my(pts[k][1]); if(k===0)bctx.moveTo(px,py);else bctx.lineTo(px,py); } bctx.strokeStyle='#4ade80';bctx.lineWidth=2;bctx.stroke(); // current dot var cl=lossAt(curW,curB); bctx.beginPath();bctx.arc(mx(curW),my(cl),5,0,Math.PI*2); bctx.fillStyle='#f472b6';bctx.fill(); bctx.strokeStyle='#fff';bctx.lineWidth=1.5;bctx.stroke(); // axis labels bctx.fillStyle='#64748b';bctx.font='10px IBM Plex Mono'; bctx.fillText('w',W-15,H-10); bctx.fillText('loss',5,15); bctx.fillText('0',pad-10,H-10); bctx.fillText('50',W-35,H-10); // axis lines bctx.strokeStyle='#64748b';bctx.lineWidth=0.8; bctx.beginPath();bctx.moveTo(pad,H-25);bctx.lineTo(W-15,H-25);bctx.stroke(); bctx.beginPath();bctx.moveTo(pad,H-25);bctx.lineTo(pad,10);bctx.stroke(); } function update(){ var w = parseFloat(ws.value), b = parseFloat(bs.value); wv.textContent = w; bv.textContent = b; var y0 = w*0+b, y14 = w*14+b; line.setAttribute('x1', mapX(0)); line.setAttribute('y1', mapY(y0)); line.setAttribute('x2', mapX(14)); line.setAttribute('y2', mapY(y14)); var loss = 0; for(var i=0;i<3;i++){ var px = mapX(data[i][0]); var pred = w*data[i][0]+b; var py = mapY(pred); var ay = mapY(data[i][1]); err[i].setAttribute('x1',px); err[i].setAttribute('y1',ay); err[i].setAttribute('x2',px); err[i].setAttribute('y2',py); pd[i].setAttribute('cx',px); pd[i].setAttribute('cy',py); loss += (pred - data[i][1])*(pred - data[i][1]); } loss /= 3; lossT.textContent = 'loss = ' + Math.round(loss).toLocaleString(); drawBowl(w,b); } ws.addEventListener('input', update); bs.addEventListener('input', update); update(); })(); </div> But Loss is a curved surface — the slope changes everywhere. How do we find the bottom? Before we search for it, let's understand what the landscape of Loss looks like. --- ### The shape of the problem — one minimum Since Loss is built entirely from $(wx + b - y)^2$ terms — **quadratic**, highest power is 2 — it always forms a U-shape. With one unknown ($w$), that's a U-curve. With two unknowns ($w$ and $b$), it becomes a 3D bowl. And **a bowl has exactly one bottom**: one deepest point, one best $(w^\ast, b^\ast)$. No traps, no false valleys. **If we keep moving downhill, we're guaranteed to reach it.** (This guarantee breaks for neural networks — more at the end.) <canvas id="lb-canvas" width="500" height="340" style="width:100%;max-width:500px;display:block;margin:1.5em auto;"></canvas> <div style="text-align:center;font-family:'IBM Plex Mono';font-size:11px;color:#64748b;margin-top:0.3em;">the loss surface — one bottom, one best (w*, b*)</div> (function(){ var c=document.getElementById('lb-canvas'),ctx=c.getContext('2d'); var W=500,H=340,cx=250,cy=195; function proj(u,v,z){ return [cx + u*130 + v*45, cy - z*95 + v*50]; } function draw(){ ctx.clearRect(0,0,W,H); var N=20; // filled slices back to front for depth for(var i=N-1;i>=0;i--){ var v=(i/N-0.5)*2; ctx.beginPath(); for(var j=0;j<=N;j++){ var u=(j/N-0.5)*2; var z=u*u+v*v; var p=proj(u,v,z); if(j===0)ctx.moveTo(p[0],p[1]);else ctx.lineTo(p[0],p[1]); } // close bottom var pR=proj(1,v,0),pL=proj(-1,v,0); ctx.lineTo(pR[0],pR[1]);ctx.lineTo(pL[0],pL[1]);ctx.closePath(); var bright=Math.floor(20+15*(1-Math.abs(v))); ctx.fillStyle='rgba('+bright+','+Math.floor(bright*2.5)+','+bright+',0.25)'; ctx.fill(); ctx.strokeStyle='rgba(74,222,128,'+(0.15+0.25*(1-Math.abs(v)))+')'; ctx.lineWidth=1;ctx.stroke(); } // horizontal rings (cross-sections) for(var i=0;i<N;i++){ var u=(i/N-0.5)*2; ctx.beginPath(); for(var j=0;j<=N;j++){ var v=(j/N-0.5)*2; var z=u*u+v*v; var p=proj(u,v,z); if(j===0)ctx.moveTo(p[0],p[1]);else ctx.lineTo(p[0],p[1]); } ctx.strokeStyle='rgba(74,222,128,'+(0.08+0.12*(1-Math.abs(u)))+')'; ctx.lineWidth=0.7;ctx.stroke(); } // minimum dot var mp=proj(0,0,0); ctx.beginPath();ctx.arc(mp[0],mp[1],6,0,Math.PI*2); ctx.fillStyle='#e5c07b';ctx.fill(); ctx.strokeStyle='#1a1f2e';ctx.lineWidth=2;ctx.stroke(); // labels ctx.font='bold 13px IBM Plex Mono';ctx.fillStyle='#e5c07b'; ctx.fillText('minimum',mp[0]+12,mp[1]+5); ctx.font='bold 12px IBM Plex Mono'; // axis labels — each axis a distinct color var aw=proj(1.15,0,0),ab=proj(0,1.15,0),al=proj(0,0,1.8); ctx.fillStyle='#4ade80';ctx.fillText('w →',aw[0],aw[1]+15); ctx.fillStyle='#f472b6';ctx.fillText('b →',ab[0]+5,ab[1]+5); ctx.fillStyle='#e5c07b';ctx.fillText('loss ↑',al[0]+5,al[1]+5); // axis lines — match their label color ctx.lineWidth=1.5; ctx.setLineDash([5,4]); var o=proj(0,0,0); ctx.strokeStyle='#4ade80'; ctx.beginPath();ctx.moveTo(proj(-1.1,0,0)[0],proj(-1.1,0,0)[1]);ctx.lineTo(aw[0],aw[1]);ctx.stroke(); ctx.strokeStyle='#f472b6'; ctx.beginPath();ctx.moveTo(proj(0,-1.1,0)[0],proj(0,-1.1,0)[1]);ctx.lineTo(ab[0],ab[1]);ctx.stroke(); ctx.strokeStyle='#e5c07b'; ctx.beginPath();ctx.moveTo(o[0],o[1]);ctx.lineTo(al[0],al[1]);ctx.stroke(); ctx.setLineDash([]); } draw(); })(); </div> We know the bowl has one bottom. Now we need to find it. Here's the key insight: at the very bottom of the bowl, the surface is perfectly flat — no tilt in any direction. On the sides, it tilts. The **slope** measures that tilt. If the slope is negative, you're on a downward slope — move in that direction and Loss decreases. If it's positive, you're on an upward slope — move the other way. Keep following the slope downhill, and stop when slope = 0. **That zero is not an approximation — it is the exact mathematical condition for a minimum.** This is why finding the minimum means finding where the slope of Loss equals zero. --- ### Finding the direction — derivatives For a straight line $y = mx + c$, slope is constant. But **Loss is a curve — the slope is different at every point**. Steep at the edges of the bowl, gentle near the bottom, flat at the very bottom. We need a formula for slope at *any* point. Take $y = x^2$. Pick any $x$. Nudge $x$ by a tiny amount $h$ — $y$ changes from $x^2$ to $(x+h)^2$. Slope is $\Delta y / \Delta x$: $\text{slope} = \frac{\Delta y}{\Delta x} = \frac{(x+h)^2 - x^2}{h} = \frac{2xh + h^2}{h} = 2x + h$ $h \to 0 \quad \Rightarrow \quad \frac{dy}{dx} = 2x$ As $h$ shrinks to zero, we get a formula — $2x$ — that gives the exact slope at every point on the curve. This process is called **differentiation**. The result is the **derivative** — a formula that gives the slope at any point we ask about. **The minimum is exactly where the derivative equals zero**: the slope is flat, you've stopped going downhill. So our strategy is clear — take the derivative of the loss function, set it to zero, and solve: $\frac{\partial L}{\partial w} = 0 \quad \text{and} \quad \frac{\partial L}{\partial b} = 0$ <canvas id="gd-canvas" width="500" height="220" style="width:100%;max-width:500px;display:block;margin:1.5em auto;"></canvas> <div style="display:flex;gap:0.6em;justify-content:center;margin:0.3em 0 0.2em;"> <button id="gd-play" style="background:#1e293b;border:1px solid #334155;color:#4ade80;font-family:'IBM Plex Mono';font-size:12px;padding:5px 14px;border-radius:4px;cursor:pointer;">▶ play</button> <button id="gd-step" style="background:#1e293b;border:1px solid #334155;color:#4ade80;font-family:'IBM Plex Mono';font-size:12px;padding:5px 14px;border-radius:4px;cursor:pointer;">step</button> <button id="gd-reset" style="background:#1e293b;border:1px solid #334155;color:#f472b6;font-family:'IBM Plex Mono';font-size:12px;padding:5px 14px;border-radius:4px;cursor:pointer;">↺ reset</button> </div> <div style="text-align:center;font-family:'IBM Plex Mono';font-size:11px;color:#64748b;margin-bottom:0.5em;">gradient descent — follow the slope downhill, step by step</div> (function(){ var data=[[5,120],[8,250],[12,310]]; var c=document.getElementById('gd-canvas'),ctx=c.getContext('2d'); var W=500,H=220,pad=50,b=10; function lossAt(w){var s=0;for(var i=0;i<3;i++){var e=w*data[i][0]+b-data[i][1];s+=e*e;}return s/3;} function gradAt(w){var d=0;for(var i=0;i<3;i++){var e=w*data[i][0]+b-data[i][1];d+=2*e*data[i][0];}return d/3;} var maxL=lossAt(0); function mx(w){return pad+(w/50)*(W-pad-20);} function my(l){return H-35-(Math.min(l,maxL)/maxL)*(H-55);} var curW=5,timer=null,steps=0,trail=[]; function render(){ ctx.clearRect(0,0,W,H); // curve ctx.beginPath(); for(var w=0;w<=50;w+=0.5){var px=mx(w),py=my(lossAt(w));if(w===0)ctx.moveTo(px,py);else ctx.lineTo(px,py);} ctx.strokeStyle='#4ade80';ctx.lineWidth=2.5;ctx.stroke(); // axes ctx.strokeStyle='#64748b';ctx.lineWidth=1; ctx.beginPath();ctx.moveTo(pad,H-35);ctx.lineTo(W-10,H-35);ctx.stroke(); ctx.beginPath();ctx.moveTo(pad,H-35);ctx.lineTo(pad,8);ctx.stroke(); ctx.fillStyle='#64748b';ctx.font='11px IBM Plex Mono'; ctx.fillText('w',W-15,H-20);ctx.fillText('loss',5,18); ctx.font='9px IBM Plex Mono'; ctx.fillText('0',pad-8,H-20);ctx.fillText('50',W-pad,H-20); // minimum label var optW=27.3,optL=lossAt(optW); ctx.beginPath();ctx.arc(mx(optW),my(optL),4,0,Math.PI*2);ctx.fillStyle='#e5c07b';ctx.fill(); ctx.fillStyle='#e5c07b';ctx.font='10px IBM Plex Mono';ctx.fillText('w*',mx(optW)+8,my(optL)+4); // trail for(var i=0;i<trail.length;i++){ ctx.beginPath();ctx.arc(trail[i][0],trail[i][1],3,0,Math.PI*2); ctx.fillStyle='rgba(244,114,182,'+(0.3+0.5*i/trail.length)+')';ctx.fill(); } // tangent line at current point (shows slope) var g=gradAt(curW),cl=lossAt(curW),cx=mx(curW),cy=my(cl); var sl=-g/maxL*(H-55)/((W-pad-20)/50); ctx.beginPath();ctx.moveTo(cx-40,cy-40*sl);ctx.lineTo(cx+40,cy+40*sl); ctx.strokeStyle='#f472b6';ctx.lineWidth=1.5;ctx.setLineDash([5,4]);ctx.stroke();ctx.setLineDash([]); // current dot ctx.beginPath();ctx.arc(cx,cy,7,0,Math.PI*2); ctx.fillStyle='#f472b6';ctx.fill();ctx.strokeStyle='#fff';ctx.lineWidth=2;ctx.stroke(); // info ctx.fillStyle='#1e293b';ctx.strokeStyle='#334155';ctx.lineWidth=1; ctx.beginPath();ctx.roundRect(280,5,210,24,6);ctx.fill();ctx.stroke(); ctx.fillStyle='#e5c07b';ctx.font='bold 12px IBM Plex Mono';ctx.textAlign='center'; ctx.fillText('w='+curW.toFixed(1)+' loss='+Math.round(cl),385,22);ctx.textAlign='start'; } function step(){ trail.push([mx(curW),my(lossAt(curW))]); var g=gradAt(curW);curW-=0.005*g; curW=Math.max(0,Math.min(50,curW));steps++; render(); if((Math.abs(g)<5||steps>60)&&timer){clearInterval(timer);timer=null;document.getElementById('gd-play').textContent='▶ play';} } function reset(){if(timer){clearInterval(timer);timer=null;}curW=5;steps=0;trail=[];document.getElementById('gd-play').textContent='▶ play';render();} document.getElementById('gd-step').addEventListener('click',step); document.getElementById('gd-reset').addEventListener('click',reset); document.getElementById('gd-play').addEventListener('click',function(){ if(timer){clearInterval(timer);timer=null;this.textContent='▶ play';} else{timer=setInterval(step,150);this.textContent='⏸ pause';} }); render(); })(); </div> The animation above shows **gradient descent**: start at a random $w$, compute the slope of Loss, take a small step opposite to the slope (downhill), repeat. For our 3-ride problem we can solve $\partial L / \partial w = 0$ exactly with algebra. For neural networks with billions of parameters, gradient descent is the only practical option — you can't solve a billion-variable equation by hand. Either way, we need the slope of Loss at any point. For $y = x^2$, the derivative $2x$ tells us the slope everywhere. Drag the dot below to see this live: <svg viewBox="0 0 500 320" style="width:100%;max-width:500px;display:block;margin:1.5em auto;"> <defs> <marker id="arrowX" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6" fill="#64748b"/></marker> <marker id="arrowY" markerWidth="6" markerHeight="8" refX="3" refY="0" orient="auto"><path d="M0,8 L3,0 L6,8" fill="#64748b"/></marker> </defs> <line x1="50" y1="280" x2="460" y2="280" stroke="#64748b" stroke-width="1" marker-end="url(#arrowX)"/> <line x1="250" y1="290" x2="250" y2="20" stroke="#64748b" stroke-width="1" marker-end="url(#arrowY)"/> <text x="465" y="285" fill="#64748b" font-size="13" font-family="IBM Plex Mono">x</text> <text x="255" y="18" fill="#64748b" font-size="13" font-family="IBM Plex Mono">y</text> <path d="M70,40 Q130,100 160,150 Q200,230 250,260 Q300,230 340,150 Q370,100 430,40" fill="none" stroke="#4ade80" stroke-width="2.5"/> <circle cx="250" cy="260" r="6" fill="#e5c07b" stroke="#e5c07b" stroke-width="2"/> <text x="258" y="275" fill="#e5c07b" font-size="12" font-family="IBM Plex Mono">minimum (slope = 0)</text> <circle id="drag-point" cx="160" cy="150" r="8" fill="#f472b6" stroke="white" stroke-width="2" style="cursor:grab"/> <line id="tangent-line" x1="100" y1="105" x2="220" y2="195" stroke="#f472b6" stroke-width="1.5" stroke-dasharray="6,4"/> <text id="slope-label" x="110" y="130" fill="#f472b6" font-size="13" font-family="IBM Plex Mono" font-weight="600">slope = −3.2</text> <text x="250" y="310" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">drag the pink dot along the curve</text> </svg> (function(){ var svg = document.getElementById('parabola-graph').querySelector('svg'); var dot = document.getElementById('drag-point'); var tan = document.getElementById('tangent-line'); var lab = document.getElementById('slope-label'); var dragging = false; function curve(cx) { var t = (cx - 250) / 200; return 260 - 220 * t * t; } function slope(cx) { return -2 * (cx - 250) / 200; } function update(cx) { cx = Math.max(70, Math.min(430, cx)); var cy = curve(cx); dot.setAttribute('cx', cx); dot.setAttribute('cy', cy); var s = slope(cx), dx = 60, dy = dx * s; tan.setAttribute('x1', cx - dx); tan.setAttribute('y1', cy - dy); tan.setAttribute('x2', cx + dx); tan.setAttribute('y2', cy + dy); var sVal = (s * 5).toFixed(1); lab.textContent = 'slope = ' + (sVal >= 0 ? '+' : '') + sVal; lab.setAttribute('x', cx - 70); lab.setAttribute('y', Math.max(30, cy - 20)); } function getX(e) { var r = svg.getBoundingClientRect(); return ((e.touches ? e.touches[0].clientX : e.clientX) - r.left) / r.width * 500; } dot.addEventListener('mousedown', function(){ dragging = true; dot.style.cursor='grabbing'; }); dot.addEventListener('touchstart', function(e){ dragging = true; e.preventDefault(); }, {passive:false}); window.addEventListener('mousemove', function(e){ if(dragging) update(getX(e)); }); window.addEventListener('touchmove', function(e){ if(dragging) update(getX(e)); }, {passive:false}); window.addEventListener('mouseup', function(){ dragging = false; dot.style.cursor='grab'; }); window.addEventListener('touchend', function(){ dragging = false; }); })(); </div> These are the differentiation rules we'll need to take the derivative of our loss function: $y = c \quad \Rightarrow \quad \frac{dy}{dx} = 0$ $y = cx \quad \Rightarrow \quad \frac{dy}{dx} = c$ $y = x^2 \quad \Rightarrow \quad \frac{dy}{dx} = 2x$ $y = f(g(x)) \quad \Rightarrow \quad \frac{dy}{dx} = \frac{df}{dg} \cdot \frac{dg}{dx} \quad \text{(chain rule)}$ The last one — the **chain rule** — is the key. Our loss function isn't a simple $x^2$. It has layers: $w$ feeds into the prediction error, the error gets squared, the squares get averaged. Each layer is simple on its own. **The chain rule says: differentiate each layer separately, then multiply.** Let's apply this to our actual loss. --- ### Solving for w* and b* We have the tool (differentiation) and the rule for layered functions (chain rule). Now use them. For ride 1 (5 km, ₹120), the prediction error is $\text{err} = w \times 5 + b - 120$: $\text{loss} = \frac{\text{err}^2}{3}$ Differentiate both sides with respect to $w$: $\frac{\partial \text{loss}}{\partial w} = \frac{1}{3} \cdot \frac{\partial(\text{err}^2)}{\partial w}$ $\text{err}^2$ is a function of err, and err is a function of $w$. Chain rule: $\frac{\partial(\text{err}^2)}{\partial w} = 2 \cdot \text{err} \cdot \frac{\partial \text{err}}{\partial w}$ $\text{err} = w \times 5 + b - 120$, so $\frac{\partial \text{err}}{\partial w} = 5$: $\frac{\partial(\text{err}^2)}{\partial w} = 2 \cdot \text{err} \cdot 5$ Put it all together: $\frac{\partial \text{loss}}{\partial w} = \frac{1}{3} \cdot 2 \cdot \text{err} \cdot 5 = \frac{10 \cdot \text{err}}{3}$ <svg viewBox="0 0 500 200" style="width:100%;max-width:540px;display:block;margin:1.5em auto;"> <!-- Box 1: gap --> <rect x="10" y="30" width="130" height="56" rx="8" fill="none" stroke="#4ade80" stroke-width="1.5"/> <text x="75" y="52" fill="#4ade80" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">Layer 1</text> <text x="75" y="72" fill="#c8d6e5" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">err = w×5+b−120</text> <!-- Arrow 1→2 --> <line x1="140" y1="58" x2="175" y2="58" stroke="#64748b" stroke-width="1.5"/> <polygon points="175,53 185,58 175,63" fill="#64748b"/> <text x="162" y="48" fill="#f472b6" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">×5</text> <!-- Box 2: square --> <rect x="185" y="30" width="130" height="56" rx="8" fill="none" stroke="#e5c07b" stroke-width="1.5"/> <text x="250" y="52" fill="#e5c07b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">Layer 2</text> <text x="250" y="72" fill="#c8d6e5" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">sq = err²</text> <!-- Arrow 2→3 --> <line x1="315" y1="58" x2="350" y2="58" stroke="#64748b" stroke-width="1.5"/> <polygon points="350,53 360,58 350,63" fill="#64748b"/> <text x="338" y="48" fill="#f472b6" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">×2err</text> <!-- Box 3: average --> <rect x="360" y="30" width="130" height="56" rx="8" fill="none" stroke="#f472b6" stroke-width="1.5"/> <text x="425" y="52" fill="#f472b6" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">Layer 3</text> <text x="425" y="72" fill="#c8d6e5" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">loss = err / 3</text> <!-- slope label for layer 3 --> <text x="470" y="100" fill="#f472b6" font-size="10" font-family="IBM Plex Mono" text-anchor="end">×⅓</text> <!-- w input arrow --> <text x="4" y="16" fill="#94a3b8" font-size="11" font-family="IBM Plex Mono">nudge w</text> <line x1="40" y1="22" x2="40" y2="28" stroke="#94a3b8" stroke-width="1"/> <polygon points="36,28 40,34 44,28" fill="#94a3b8"/> <!-- total slope --> <rect x="100" y="120" width="300" height="36" rx="8" fill="#1e293b" stroke="#334155" stroke-width="1"/> <text x="250" y="135" fill="#94a3b8" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">total slope = multiply the arrows:</text> <text x="250" y="150" fill="#e5c07b" font-size="13" font-family="IBM Plex Mono" font-weight="600" text-anchor="middle">5 × 2·err × ⅓</text> <!-- connecting lines from boxes to total --> <line x1="162" y1="86" x2="162" y2="100" stroke="#64748b" stroke-width="0.8" stroke-dasharray="3,2" opacity="0.4"/> <line x1="338" y1="86" x2="338" y2="100" stroke="#64748b" stroke-width="0.8" stroke-dasharray="3,2" opacity="0.4"/> <!-- flow label --> <text x="250" y="185" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">a nudge to w ripples through all three layers</text> </svg> </div> That's the slope of Loss with respect to weight ($\partial L / \partial w$) for one ride. For bias: nudging $b$ moves err by 1 (not $x_i$), so $\frac{\partial \text{err}}{\partial b} = 1$. Apply to all three rides and set both slopes to zero: $\frac{\partial L}{\partial w} = \frac{1}{3} \sum_{i=1}^{3} 2(w x_i + b - y_i) \cdot x_i = 0$ $\frac{\partial L}{\partial b} = \frac{1}{3} \sum_{i=1}^{3} 2(w x_i + b - y_i) \cdot 1 = 0$ Expand with data $(5, 120), (8, 250), (12, 310)$: $466w + 50b = 8720$ $50w + 6b = 1360$ $\Rightarrow \quad w^\ast \approx 27.3, \quad b^\ast \approx -17.7$ Two equations, two unknowns, standard algebra. The math found the best possible line for our data: <svg viewBox="0 0 500 320" style="width:100%;max-width:520px;display:block;margin:1.5em auto;"> <defs> <marker id="of-arrowX" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6" fill="#64748b"/></marker> <marker id="of-arrowY" markerWidth="6" markerHeight="8" refX="3" refY="0" orient="auto"><path d="M0,8 L3,0 L6,8" fill="#64748b"/></marker> </defs> <!-- axes --> <line x1="60" y1="265" x2="470" y2="265" stroke="#64748b" stroke-width="1" marker-end="url(#of-arrowX)"/> <line x1="60" y1="275" x2="60" y2="15" stroke="#64748b" stroke-width="1" marker-end="url(#of-arrowY)"/> <text x="475" y="270" fill="#64748b" font-size="11" font-family="IBM Plex Mono">km</text> <text x="15" y="15" fill="#64748b" font-size="11" font-family="IBM Plex Mono">fare ₹</text> <!-- tick marks --> <text x="60" y="281" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">0</text> <text x="168" y="281" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">4</text> <text x="276" y="281" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">8</text> <text x="384" y="281" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">12</text> <line x1="168" y1="263" x2="168" y2="267" stroke="#64748b" stroke-width="1"/> <line x1="276" y1="263" x2="276" y2="267" stroke="#64748b" stroke-width="1"/> <line x1="384" y1="263" x2="384" y2="267" stroke="#64748b" stroke-width="1"/> <text x="55" y="265" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">0</text> <text x="55" y="204" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">100</text> <text x="55" y="143" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">200</text> <text x="55" y="82" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="end">300</text> <line x1="58" y1="204" x2="62" y2="204" stroke="#64748b" stroke-width="1"/> <line x1="58" y1="143" x2="62" y2="143" stroke="#64748b" stroke-width="1"/> <line x1="58" y1="82" x2="62" y2="82" stroke="#64748b" stroke-width="1"/> <!-- optimal line: fare = 27.3*km - 17.7 --> <!-- mapX(km) = 60 + (km/14)*400, mapY(fare) = 265 - (fare/350)*196 --> <!-- km=0: fare=-17.7 → y=265-(-17.7/350)*196=265+9.9=274.9 --> <!-- km=14: fare=27.3*14-17.7=364.5 → y=265-(364.5/350)*196=265-204=61 --> <line x1="60" y1="275" x2="441" y2="61" stroke="#4ade80" stroke-width="2.5"/> <!-- error lines --> <!-- ride1: km=5, fare=120. pred=27.3*5-17.7=118.8. mapX=60+5/14*400=203. mapY(120)=265-120/350*196=197.8. mapY(118.8)=265-118.8/350*196=198.5 --> <line x1="203" y1="198" x2="203" y2="199" stroke="#e5c07b" stroke-width="1.5" stroke-dasharray="4,3"/> <!-- ride2: km=8, fare=250. pred=27.3*8-17.7=200.7. mapX=60+8/14*400=289. mapY(250)=265-250/350*196=125. mapY(200.7)=265-200.7/350*196=152.6 --> <line x1="289" y1="125" x2="289" y2="153" stroke="#e5c07b" stroke-width="1.5" stroke-dasharray="4,3"/> <!-- ride3: km=12, fare=310. pred=27.3*12-17.7=309.9. mapX=60+12/14*400=403. mapY(310)=265-310/350*196=91.5. mapY(309.9)=91.6 --> <line x1="403" y1="92" x2="403" y2="92" stroke="#e5c07b" stroke-width="1.5" stroke-dasharray="4,3"/> <!-- data points --> <circle cx="203" cy="198" r="6" fill="#4ade80"/> <circle cx="289" cy="125" r="6" fill="#4ade80"/> <circle cx="403" cy="92" r="6" fill="#4ade80"/> <!-- predicted dots on line --> <circle cx="203" cy="199" r="3.5" fill="#f472b6" opacity="0.7"/> <circle cx="289" cy="153" r="3.5" fill="#f472b6" opacity="0.7"/> <circle cx="403" cy="92" r="3.5" fill="#f472b6" opacity="0.7"/> <!-- labels --> <text x="203" y="216" fill="#4ade80" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">(5, ₹120)</text> <text x="289" y="118" fill="#4ade80" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">(8, ₹250)</text> <text x="403" y="85" fill="#4ade80" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">(12, ₹310)</text> <!-- result badge --> <rect x="120" y="30" width="260" height="28" rx="6" fill="#1e293b" stroke="#334155" stroke-width="1"/> <text x="250" y="49" fill="#e5c07b" font-size="13" font-family="IBM Plex Mono" font-weight="600" text-anchor="middle">w*=27.3 b*=−17.7 loss≈1,636</text> <text x="250" y="310" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">the math found the best line — errors are tiny</text> </svg> </div> **Objective 1 — achieved.** We started with random $w$ and $b$, defined a loss function to score how wrong any line is, took its derivative to find the direction of steepest descent, set the derivative to zero, and solved. The result: $w^\ast = 27.3$, $b^\ast = -17.7$ — the line that best fits the data. That process — loss → derivative → minimize — is what "training" means. Every foundation model does exactly this. GPT-4 has 1.8 trillion parameters instead of two, and uses gradient descent instead of algebra, but the math is identical. Random numbers in, useful numbers out. --- ## Part 2: Comparing — measuring similarity Training found the right weights — the model has learned a relationship between inputs and outputs. But models don't just predict. They also **compare**. In the [first post](/blogs/foundation-model-internals-1), Word2Vec turned every word into a vector of 300 numbers — "king" becomes one list of numbers, "queen" becomes another. The whole point was that similar words should land nearby in that number space. But how does the model actually measure "nearby" between two lists of numbers? That's the second question this post answers. Before we get there, we need to extend our model from one input to many — because that's where vectors naturally appear. ### Vectors and the dot product So far our input $x$ was a single number — distance in km. Real predictions depend on many features: distance, time, surge pricing, traffic. A **vector** groups them into an ordered list: $\mathbf{x} = [x_1, x_2, x_3] = [\text{distance}, \text{ time}, \text{ surge}] = [5, 15, 1.2]$ Now each input needs its own weight: $\mathbf{w} = [w_1, w_2, w_3]$. To get a single prediction from these two lists, we use the **dot product** — multiply matching pairs and add up: $\mathbf{w} \cdot \mathbf{x} = w_1 x_1 + w_2 x_2 + w_3 x_3 \qquad \Rightarrow \qquad y = \mathbf{w} \cdot \mathbf{x} + b$ This is the same $y = wx + b$ from Part 1, just with vectors instead of single numbers. Everything we built — loss function, differentiation, chain rule, setting the derivative to zero — works identically, just with more terms to add up. A foundation model's input might have thousands of dimensions. The formula doesn't change. <svg viewBox="0 0 500 170" style="width:100%;max-width:520px;display:block;margin:1.5em auto;"> <!-- w vector --> <text x="20" y="22" fill="#f472b6" font-size="12" font-family="IBM Plex Mono" font-weight="600">w</text> <rect x="45" y="8" width="50" height="30" rx="5" fill="none" stroke="#f472b6" stroke-width="1.5"/> <text x="70" y="28" fill="#f472b6" font-size="13" font-family="IBM Plex Mono" text-anchor="middle">25</text> <rect x="45" y="44" width="50" height="30" rx="5" fill="none" stroke="#f472b6" stroke-width="1.5"/> <text x="70" y="64" fill="#f472b6" font-size="13" font-family="IBM Plex Mono" text-anchor="middle">3</text> <!-- × symbol --> <text x="112" y="28" fill="#64748b" font-size="16" font-family="IBM Plex Mono" text-anchor="middle">×</text> <text x="112" y="64" fill="#64748b" font-size="16" font-family="IBM Plex Mono" text-anchor="middle">×</text> <!-- x vector --> <text x="128" y="22" fill="#4ade80" font-size="12" font-family="IBM Plex Mono" font-weight="600">x</text> <rect x="148" y="8" width="50" height="30" rx="5" fill="none" stroke="#4ade80" stroke-width="1.5"/> <text x="173" y="28" fill="#4ade80" font-size="13" font-family="IBM Plex Mono" text-anchor="middle">5</text> <rect x="148" y="44" width="50" height="30" rx="5" fill="none" stroke="#4ade80" stroke-width="1.5"/> <text x="173" y="64" fill="#4ade80" font-size="13" font-family="IBM Plex Mono" text-anchor="middle">15</text> <!-- = products --> <text x="215" y="28" fill="#64748b" font-size="16" font-family="IBM Plex Mono" text-anchor="middle">=</text> <text x="215" y="64" fill="#64748b" font-size="16" font-family="IBM Plex Mono" text-anchor="middle">=</text> <rect x="233" y="8" width="55" height="30" rx="5" fill="none" stroke="#e5c07b" stroke-width="1.5"/> <text x="260" y="28" fill="#e5c07b" font-size="13" font-family="IBM Plex Mono" text-anchor="middle">125</text> <rect x="233" y="44" width="55" height="30" rx="5" fill="none" stroke="#e5c07b" stroke-width="1.5"/> <text x="260" y="64" fill="#e5c07b" font-size="13" font-family="IBM Plex Mono" text-anchor="middle">45</text> <!-- + arrow down --> <line x1="260" y1="78" x2="260" y2="100" stroke="#64748b" stroke-width="1.5"/> <polygon points="256,100 260,108 264,100" fill="#64748b"/> <text x="275" y="94" fill="#64748b" font-size="11" font-family="IBM Plex Mono">add</text> <!-- sum result --> <rect x="225" y="112" width="70" height="32" rx="6" fill="#1e293b" stroke="#334155" stroke-width="1"/> <text x="260" y="133" fill="#e5c07b" font-size="15" font-family="IBM Plex Mono" font-weight="700" text-anchor="middle">170</text> <!-- + b --> <text x="310" y="133" fill="#64748b" font-size="13" font-family="IBM Plex Mono">+ b</text> <!-- = fare --> <text x="358" y="133" fill="#64748b" font-size="13" font-family="IBM Plex Mono">=</text> <rect x="372" y="112" width="100" height="32" rx="6" fill="none" stroke="#4ade80" stroke-width="1.5"/> <text x="422" y="133" fill="#4ade80" font-size="14" font-family="IBM Plex Mono" font-weight="700" text-anchor="middle">fare = 180</text> <!-- labels --> <text x="70" y="88" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">₹/km, ₹/min</text> <text x="173" y="88" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">5km, 15min</text> <text x="260" y="160" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">dot product: multiply matching pairs, add them up</text> </svg> </div> --- ### The problem with raw dot products The dot product gives us a number for any pair of vectors. Can we use it directly to measure similarity? Let's try. Take two ride profiles that have the **exact same proportions** — same pattern, one just has bigger numbers: <svg viewBox="0 0 420 170" style="width:100%;max-width:440px;display:block;margin:1.5em auto;"> <!-- origin --> <circle cx="40" cy="150" r="2" fill="#64748b"/> <!-- short green: angle = atan(15/5) = 72°. Visual: dx=20, dy=-60 --> <line x1="40" y1="150" x2="60" y2="90" stroke="#4ade80" stroke-width="3"/> <circle cx="60" cy="90" r="4" fill="#4ade80"/> <text x="68" y="88" fill="#4ade80" font-size="11" font-family="IBM Plex Mono">[5, 15]</text> <!-- long pink: same angle, 3x longer visually: dx=60, dy=-180 → clamped to dx=45,dy=-135 --> <line x1="40" y1="150" x2="85" y2="15" stroke="#f472b6" stroke-width="3" stroke-dasharray="6,3"/> <circle cx="85" cy="15" r="4" fill="#f472b6"/> <text x="93" y="19" fill="#f472b6" font-size="11" font-family="IBM Plex Mono">[50, 150]</text> <!-- labels on right --> <text x="180" y="55" fill="#e5c07b" font-size="12" font-family="IBM Plex Mono">same direction</text> <text x="180" y="75" fill="#e5c07b" font-size="12" font-family="IBM Plex Mono">different magnitude</text> <text x="180" y="110" fill="#64748b" font-size="11" font-family="IBM Plex Mono">dot product = 2,500</text> <text x="180" y="130" fill="#64748b" font-size="11" font-family="IBM Plex Mono">big number = scale</text> <text x="180" y="150" fill="#64748b" font-size="11" font-family="IBM Plex Mono">not similarity</text> </svg> </div> $[50, 150]$ has the exact same proportions as $[5, 15]$ — the ratio is 1:3 in both cases. They represent the same pattern. But their dot product is $5 \times 50 + 15 \times 150 = 2{,}500$ — a huge number that tells us nothing about similarity. **It's big because the numbers are big, not because the patterns are similar.** We need to strip the scale and keep only the direction. ### Deriving cosine similarity from scratch The fix is to **normalize** — divide each vector by its length to get a unit vector: $\text{NormalizedVector} = \frac{\text{Vector}}{\text{Length(Vector)}}$ $\text{Length(Vector)} = \sqrt{v_1^2 + v_2^2 + \ldots}$ In notation: $\hat{\mathbf{v}} = \mathbf{v} / \|\mathbf{v}\|$ — the hat ($\hat{\phantom{v}}$) means "normalized". So $\hat{\mathbf{a}}$ is shorthand for NormalizedVector($\mathbf{a}$). The result always has length 1 — scale stripped, direction preserved. Let's verify with our two vectors. **Vector $[5, 15]$:** $\text{Length}([5, 15]) = \sqrt{5^2 + 15^2} = \sqrt{250} \approx 15.8$ $\text{NormalizedVector}([5, 15]) = \frac{[5, 15]}{15.8} = \left[\frac{5}{15.8}, \frac{15}{15.8}\right] \approx [0.316, 0.949]$ **Vector $[50, 150]$:** $\text{Length}([50, 150]) = \sqrt{50^2 + 150^2} = \sqrt{25000} \approx 158.1$ $\text{NormalizedVector}([50, 150]) = \frac{[50, 150]}{158.1} = \left[\frac{50}{158.1}, \frac{150}{158.1}\right] \approx [0.316, 0.949]$ After normalizing — **identical**. The 10x scale difference is gone. Only the direction remains. Both normalized vectors sit at the same point on a **unit circle** (a circle of radius 1): <svg viewBox="0 0 500 200" style="width:100%;max-width:500px;display:block;margin:1.5em auto;"> <!-- BEFORE (left side) --> <text x="80" y="15" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">before</text> <line x1="30" y1="180" x2="160" y2="180" stroke="#334155" stroke-width="0.5"/> <line x1="30" y1="180" x2="30" y2="20" stroke="#334155" stroke-width="0.5"/> <!-- short green --> <line x1="30" y1="180" x2="50" y2="120" stroke="#4ade80" stroke-width="2.5"/> <circle cx="50" cy="120" r="4" fill="#4ade80"/> <text x="58" y="118" fill="#4ade80" font-size="10" font-family="IBM Plex Mono">[5, 15]</text> <!-- long pink — same angle --> <line x1="30" y1="180" x2="130" y2="30" stroke="#f472b6" stroke-width="2.5" stroke-dasharray="5,3"/> <circle cx="130" cy="30" r="4" fill="#f472b6"/> <text x="138" y="34" fill="#f472b6" font-size="10" font-family="IBM Plex Mono">[50, 150]</text> <!-- arrow --> <text x="210" y="100" fill="#64748b" font-size="18" font-family="IBM Plex Mono">→</text> <text x="200" y="120" fill="#64748b" font-size="9" font-family="IBM Plex Mono">normalize</text> <!-- AFTER (right side) --> <text x="370" y="15" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">after</text> <circle cx="300" cy="120" r="70" fill="none" stroke="#475569" stroke-width="1" stroke-dasharray="4,3"/> <text x="375" y="65" fill="#475569" font-size="9" font-family="IBM Plex Mono">unit circle</text> <line x1="300" y1="120" x2="300" y2="120" stroke="#334155" stroke-width="0.5"/> <!-- both normalized vectors land at same point --> <line x1="300" y1="120" x2="322" y2="53" stroke="#4ade80" stroke-width="2"/> <line x1="300" y1="120" x2="322" y2="53" stroke="#f472b6" stroke-width="2" stroke-dasharray="5,3"/> <circle cx="322" cy="53" r="6" fill="#e5c07b" stroke="#fff" stroke-width="1.5"/> <text x="334" y="50" fill="#e5c07b" font-size="10" font-family="IBM Plex Mono">same point!</text> <text x="334" y="65" fill="#e5c07b" font-size="9" font-family="IBM Plex Mono">[0.316, 0.949]</text> <!-- origin dot --> <circle cx="300" cy="120" r="2" fill="#64748b"/> </svg> </div> Now we have two unit vectors pointing in the same direction. We want a single number that captures how similar they are — how closely they point the same way. The dot product does exactly this. Recall from earlier: the dot product multiplies corresponding components and sums them. When two vectors point in the same direction, their components align — big times big, small times small — and the sum is large. When they point in different directions, some terms are negative and they cancel out. So **the dot product of two unit vectors naturally gives a similarity score**: high when directions match, low when they don't. $\hat{\mathbf{a}} \cdot \hat{\mathbf{b}} = 0.316 \times 0.316 + 0.949 \times 0.949 = 0.100 + 0.901 = 1.0$ Result is $1.0$ — these two vectors point in exactly the same direction. Same pattern, perfect similarity. If they pointed in completely different directions, the dot product of the normalized vectors would shrink toward $0$. Opposite directions gives $-1$. So this gives us a clean similarity score on $[-1, 1]$: $1$ = same direction, $0$ = unrelated, $-1$ = opposite. This operation has a name: the **cosine of the angle** between the two vectors. | Angle | Relationship | Score | |---|---|---| | θ = 0° | Same direction | cos(0°) = 1 | | θ = 90° | Perpendicular | cos(90°) = 0 | | θ = 180° | Opposite | cos(180°) = −1 | Writing it out explicitly: $\text{CosineSimilarity}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\text{Length}(\mathbf{a}) \times \text{Length}(\mathbf{b})}$ In notation: $\cos(\theta) = \dfrac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \cdot \|\mathbf{b}\|}$ **That's cosine similarity — not convention, it falls directly out of normalizing vectors and taking their dot product.** <svg viewBox="0 0 460 400" style="width:100%;max-width:460px;display:block;margin:1.5em auto;"> <line x1="50" y1="350" x2="420" y2="350" stroke="#64748b" stroke-width="1"/> <line x1="50" y1="350" x2="50" y2="20" stroke="#64748b" stroke-width="1"/> <text x="418" y="344" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="end">feature 1</text> <text x="58" y="18" fill="#64748b" font-size="11" font-family="IBM Plex Mono">feature 2</text> <line id="vecA" x1="50" y1="350" x2="230" y2="60" stroke="#4ade80" stroke-width="2.5"/> <circle cx="230" cy="60" r="4" fill="#4ade80"/> <text x="236" y="58" fill="#4ade80" font-size="11" font-family="IBM Plex Mono">A [5, 15]</text> <line id="vecB" x1="50" y1="350" x2="370" y2="150" stroke="#f472b6" stroke-width="2.5"/> <circle id="vecB-dot" cx="370" cy="150" r="8" fill="#f472b6" stroke="white" stroke-width="2" style="cursor:grab"/> <text id="vecB-label" x="378" y="148" fill="#f472b6" font-size="11" font-family="IBM Plex Mono">B (drag me)</text> <path id="angle-arc" d="" fill="none" stroke="#e5c07b" stroke-width="2.5"/> <text id="theta-label" x="0" y="0" fill="#e5c07b" font-size="13" font-family="IBM Plex Mono" font-weight="600">θ</text> <rect x="110" y="5" width="240" height="32" rx="6" fill="#1e293b" stroke="#334155" stroke-width="1"/> <text id="cos-val" x="230" y="26" fill="#e5c07b" font-size="15" font-family="IBM Plex Mono" font-weight="700" text-anchor="middle">cos θ = 0.87</text> <text x="230" y="395" fill="#64748b" font-size="11" font-family="IBM Plex Mono" text-anchor="middle">drag the pink dot to change B's direction</text> </svg> (function(){ var svg = document.getElementById('cosine-graph').querySelector('svg'); var vecB = document.getElementById('vecB'); var dot = document.getElementById('vecB-dot'); var lab = document.getElementById('vecB-label'); var arc = document.getElementById('angle-arc'); var thetaLab = document.getElementById('theta-label'); var cosText = document.getElementById('cos-val'); var dragging = false; var ox = 50, oy = 350, ax = 180, ay = -290; function update(bx, by) { bx = Math.max(60, Math.min(440, bx)); by = Math.max(20, Math.min(345, by)); var bdx = bx - ox, bdy = by - oy; var magA = Math.sqrt(ax*ax + ay*ay), magB = Math.sqrt(bdx*bdx + bdy*bdy); if (magB < 10) return; var cosVal = Math.max(-1, Math.min(1, (ax*bdx + ay*bdy) / (magA * magB))); vecB.setAttribute('x2', bx); vecB.setAttribute('y2', by); dot.setAttribute('cx', bx); dot.setAttribute('cy', by); lab.setAttribute('x', bx + 6); lab.setAttribute('y', by - 4); var angA = Math.atan2(-ay, ax), angB = Math.atan2(-bdy, bdx); var r = 55, startA = Math.min(angA, angB), endA = Math.max(angA, angB); if (endA - startA > Math.PI) { var tmp = startA; startA = endA; endA = tmp + 2*Math.PI; } var sx = ox + r*Math.cos(startA), sy = oy - r*Math.sin(startA); var ex = ox + r*Math.cos(endA), ey = oy - r*Math.sin(endA); var largeArc = (endA - startA > Math.PI) ? 1 : 0; arc.setAttribute('d', 'M'+sx+','+sy+' A'+r+','+r+' 0 '+largeArc+' 1 '+ex+','+ey); var midAng = (startA + endA) / 2, tr = r + 18; thetaLab.setAttribute('x', ox + tr*Math.cos(midAng) - 5); thetaLab.setAttribute('y', oy - tr*Math.sin(midAng) + 5); cosText.textContent = 'cos θ = ' + cosVal.toFixed(2); } function getXY(e) { var r = svg.getBoundingClientRect(); var cx = e.touches?e.touches[0].clientX:e.clientX; var cy = e.touches?e.touches[0].clientY:e.clientY; return [(cx-r.left)/r.width*460,(cy-r.top)/r.height*400]; } dot.addEventListener('mousedown', function(){ dragging=true; dot.style.cursor='grabbing'; }); dot.addEventListener('touchstart', function(e){ dragging=true; e.preventDefault(); }, {passive:false}); window.addEventListener('mousemove', function(e){ if(dragging){ var p=getXY(e); update(p[0],p[1]); }}); window.addEventListener('touchmove', function(e){ if(dragging){ var p=getXY(e); update(p[0],p[1]); }}, {passive:false}); window.addEventListener('mouseup', function(){ dragging=false; dot.style.cursor='grab'; }); window.addEventListener('touchend', function(){ dragging=false; }); update(370, 150); })(); </div> Drag B around — same direction gives $1$, perpendicular gives $0$, opposite gives $-1$. **Objective 2 — achieved.** We started with a problem: the dot product is polluted by scale, so we can't use it raw to compare vectors. We fixed it by normalizing — dividing each vector by its length to strip magnitude and keep only direction. The dot product of normalized vectors is the cosine of the angle between them, giving a clean similarity score from $-1$ to $1$. This is how similarity works across all of ML — comparing word embeddings, finding similar documents, matching images, detecting fraud. The vectors can have thousands of dimensions. The formula doesn't change. --- ## What changes at scale Neural networks are the scaled-up version of everything we just built — we'll get into them properly in the next post. But the math doesn't change. Everything in this post — loss functions, derivatives, the chain rule, dot products, cosine similarity — appears in every neural network ever trained. The operations are identical. The chain rule gets applied through millions of layers instead of three. Gradient descent replaces algebra because you can't solve a billion-variable equation by hand. But the math is the same math. The one thing that breaks is the bowl. Neural networks use non-linear activations (like ReLU), which destroy the smooth quadratic shape. The loss surface becomes a bumpy landscape with many valleys and ridges. Gradient descent can get stuck in a shallow dip that isn't the global best. That's why training requires tricks: learning rate schedules, momentum, batch normalization, careful weight initialization — all strategies for navigating a landscape where "walk downhill" no longer guarantees the deepest valley. <svg viewBox="0 0 500 150" style="width:100%;max-width:500px;display:block;margin:1.5em auto;"> <!-- Left: smooth bowl (U-shape, valley) --> <text x="120" y="18" fill="#4ade80" font-size="12" font-family="IBM Plex Mono" text-anchor="middle">linear model</text> <path d="M20,30 Q70,30 90,55 Q110,85 120,115 Q130,85 150,55 Q170,30 220,30" fill="none" stroke="#4ade80" stroke-width="2.5"/> <circle cx="120" cy="117" r="4" fill="#e5c07b"/> <text x="120" y="135" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">one minimum — guaranteed</text> <!-- Arrow --> <text x="260" y="75" fill="#64748b" font-size="20" font-family="IBM Plex Mono" text-anchor="middle">→</text> <!-- Right: bumpy landscape (valleys, not peaks) --> <text x="385" y="18" fill="#f472b6" font-size="12" font-family="IBM Plex Mono" text-anchor="middle">neural network</text> <path d="M280,40 Q300,50 310,75 Q315,90 320,80 Q330,50 340,60 Q350,90 360,110 Q370,130 380,110 Q390,90 400,80 Q410,60 420,70 Q430,90 440,75 Q460,40 480,40" fill="none" stroke="#f472b6" stroke-width="2.5"/> <circle cx="370" cy="131" r="4" fill="#e5c07b"/> <circle cx="310" cy="77" r="3" fill="#64748b" opacity="0.6"/> <circle cx="430" cy="91" r="3" fill="#64748b" opacity="0.6"/> <text x="370" y="145" fill="#e5c07b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">global min</text> <text x="310" y="97" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">local</text> <text x="430" y="108" fill="#64748b" font-size="9" font-family="IBM Plex Mono" text-anchor="middle">local</text> <text x="385" y="135" fill="#64748b" font-size="10" font-family="IBM Plex Mono" text-anchor="middle">many dips — can get stuck</text> </svg> </div> That's the next thing. <div id="post-footer-insert"></div> --- *Part of the [Foundation Model Internals](/blogs) series — a software engineer learning how models work, from scratch.* *This post came together after building understanding around the math with Claude — hope it helps build yours.* --- One thing I keep thinking about while writing this: AI isn't doing magic. It is just math, executing at scale. And yet we keep treating consciousness as something separate, something beyond. Max Tegmark's [Our Mathematical Universe](https://www.amazon.com/Our-Mathematical-Universe-Ultimate-Reality/dp/0307599809) goes further — maybe the universe itself is mathematics, not just described by it. If that's true, we aren't building tools. We're doing something stranger.