# Road to Foundation Models *I've been extensively using LLMs without knowing how any of it works. Time to fix that.* "Apple released a new iPhone" and "I ate an apple for breakfast." Same word. The model I use every day — the one writing my code and explaining my bugs — somehow knows these are completely different. How? I'm a software engineer. I've been using LLMs embarrassingly often. They write my code, explain my errors, occasionally hallucinate with total confidence. TARS and Marvin came alive and I missed the memo on how. **So I decided to trace it. From the very beginning.** From the dumbest possible approach to the cleverest one. This is what I found. --- ## 2000s — Machines can't understand For decades, the entire field of Natural Language Processing was stuck on one problem: how do you make a computer understand words? Not just store them — *understand* them. Know that "fraud" and "suspicious" are related. Know that "credit" can mean a credit card or "amount credited to your account" — same word, completely different meaning. Know that "declined card international transaction" is very different from "international transaction declined card" even though the words are the same. Every few years, someone came up with a slightly better approach. Each one solved something and broke something else. That progression — from dumb to less dumb to surprisingly smart — is the story of how we got to transformers. It starts here. With counting. The machine has no idea what language is. It can't read. So the first thing anyone tried was: just count how many times each word appears. ``` "I love this movie this movie is great" ``` Becomes: `{I: 1, love: 1, this: 2, movie: 2, is: 1, great: 1}` This was called [**Bag of Words**](https://en.wikipedia.org/wiki/Bag-of-words_model). Throw the words in a bag, shake it, count what's inside. Word order is gone. Grammar is gone. The surprising thing? It worked for simple tasks. Want to flag suspicious transactions from their descriptions? Count keywords — see "refund", "failed", "error" show up a lot? Flag the batch. Spam filters worked this way for years — lots of "free", "win", "click"? Spam. Not elegant, but it got the job done. [**TF-IDF**](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) was the natural upgrade. Bag of Words treats every word equally, but some words are useless. "My" appears in every support ticket — it tells you nothing. But "refund" appearing in a specific ticket? That actually means something. TF-IDF gives higher scores to words that are distinctive to a document and lower scores to words that show up everywhere. So now you can sort things. A ticket with high scores for "refund" and "failed" goes to one team. A ticket with high scores for "KYC" and "documents" goes to another. Search engines used the same idea — rank pages by how well their distinctive words match your query. People used this for over a decade. Simple, fast, good enough. But both approaches hit the same wall. "Customer was not refunded" and "Customer was refunded" — nearly the same word counts. Completely different situations. The system doesn't know "not" changes everything because it threw away word order. And "UPI" and "NEFT" are as unrelated to the machine as "UPI" and "broccoli" — no sense that they're both payment methods. Just frequencies in a table. We needed something fundamentally different. --- ## 2013 — Words become numbers that mean something Then Google published a paper called [Word2Vec](https://arxiv.org/abs/1301.3781). For 40 years, words were just counts. Word2Vec made them coordinates. Word2Vec does something clever. It gives every word an array of numbers — called a <span class="term">vector</span> (just an array, don't overthink the word) — random at first, meaningless. ``` paid = [0.12, -0.34, 0.78, 0.56, ...] ↑ ↑ ↑ ↑ just numbers. 300 of them. ``` These vectors, plus the rules for adjusting them — that's a <span class="term">model</span>. A model is just a file full of numbers and the math that connects them. Nothing more. Word2Vec is a model. BERT and GPT — which we'll get to — are also models, just much bigger ones. Then the model plays a game: given one word, guess the words around it. "she **paid** the bill online" — given "paid", what words are nearby? It gets this wrong. A lot. So it adjusts the numbers slightly to get better. Then tries again. And again. Billions of times. The thing is — the guessing isn't the point. It's the side effect. Each time the model adjusts the numbers to predict better, those numbers start capturing something about what the word *means*. "Paid" and "bill" keep appearing near each other, so their numbers drift closer. The prediction game is just an excuse to learn meaning. How the adjustment works — gradient descent — is its own topic. For now: it works and I'm moving on before I get stuck here. I'll come back to it separately in detail. After billions of rounds, the numbers stop being random. "Cat" and "dog" end up with almost identical numbers — because they both keep appearing near "pet", "food", "vet." The model never learned what a cat is. It just noticed cats and dogs show up in the same places. "Democracy"? Completely different numbers. Different universe. When vectors capture meaning like this, they get a fancier name: <span class="term">embeddings</span>. I thought that was the cool part. Then I kept reading. ```mermaid graph LR A["MAN"] -->|"+ royalty"| B["KING"] C["WOMAN"] -->|"+ royalty"| D["QUEEN"] A -->|"+ gender"| C B -->|"+ gender"| D ``` King - Man + Woman = Queen. That's not a metaphor. Take the 300 numbers for "King." Subtract the 300 numbers for "Man." Add the 300 numbers for "Woman." The result lands right next to "Queen." The model learned that royalty is a direction. Gender is a direction. You can navigate between them with arithmetic. Nobody programmed this. `Bangalore - India + USA = San Francisco` — it learned geography. Nobody gave it a map. **A model that was just trying to guess nearby words accidentally learned the structure of human concepts. This is the moment, for me, where "interesting CS paper" became "something real is happening here."** Word2Vec was the domino that started everything. It proved that neural networks could learn meaning from raw text, that you could pre-train on unlabeled data and reuse what it learned, and that language could be treated as geometry — words as points in space, relationships as directions. This is where I got stuck for a bit: **who decided what each dimension means?** The cat vector has 300 numbers. What does dimension 1 represent? Height? Furriness? Animal-ness? Turns out — nobody decided. The model figured it out during training. No human assigned meanings. **They just... emerged.** I still don't fully get how this happens. I need to see the math to believe it. Future post. But here's how I'm thinking about it for now. A fingerprint has patterns. Nobody designed what each line means. But two prints from the same finger match, and two prints from different fingers don't. The pattern is the identity. Word2Vec vectors are fingerprints for words. Similar words, similar fingerprints. And here's the part I find beautiful. Google [trained Word2Vec on 100 billion words](https://code.google.com/archive/p/word2vec/). That produced a big file: ``` cat: [0.12, -0.34, 0.78, 0.56, ...] (300 numbers) dog: [0.14, -0.31, 0.75, 0.59, ...] (300 numbers) democracy: [-0.45, 0.82, -0.12, 0.03, ...] (300 numbers) ... 3 million words ``` That file is a collection of **pre-trained embeddings** — meaning someone already did the expensive training. Anyone could download it and use it. Building a search engine? A chatbot? A recommendation system? Load these pre-trained vectors and your model already knows that "refund" and "return" are related, that "Bangalore" and "Mumbai" are both Indian cities — before seeing a single line of your data. Someone else spends the compute. You get the knowledge for free. A year later, in 2014, Stanford published [GloVe](https://nlp.stanford.edu/projects/glove/). Different approach — instead of reading text sentence by sentence like Word2Vec, GloVe counts how often every word appears near every other word across the entire corpus, then learns vectors from those counts. Different route, same destination: words as vectors. But both Word2Vec and GloVe had the same problem. Remember the puzzle from the opening? ```mermaid graph TD A["APPLE RELEASED IPHONE"] --> C["SAME VECTOR"] B["I ATE AN APPLE"] --> C ``` Same word. One vector. Apple-the-company and apple-the-fruit get identical numbers. Static vectors are <span class="term">blind</span>. They don't see the sentence around them. Every word that changes meaning based on context hits this wall. --- ## 2015-17 — RNNs and LSTMs try to add memory The idea: process words one at a time, carry something forward. A normal network takes input, gives output, forgets everything. ``` word → [math] → output ``` [Recurrent Neural Networks (RNNs)](https://en.wikipedia.org/wiki/Recurrent_neural_network) feed their output back as input for the next step. That's what "recurrent" means — it talks to its past self. ``` word1 → [math] → H1 word2 + H1 → [math] → H2 word3 + H2 → [math] → H3 ``` No understanding. Just numbers combining with numbers at every step. But trained on billions of sentences, the carried-forward numbers turn out to be useful. Google used this for Translate. Siri and Alexa for speech. Gmail's Smart Reply — those "Sounds good!" suggestions — was built on this. But the numbers fade. By step 50, word 1's influence has passed through 49 multiplications — still in there, just very faint. Long sentence? The beginning is nearly gone by the end. Like a game of telephone. [LSTMs](https://www.bioinf.jku.at/publications/older/2604.pdf) (Long Short-Term Memory) had actually existed since 1997 — invented long before this NLP boom. Applied on top of RNNs, they added learned gates that decide what to keep and what to throw away at every step, instead of blindly passing numbers forward. Better at remembering context than plain RNNs. **Google Translate, Siri, Alexa all switched to LSTMs during this period.** But fading wasn't eliminated — long sequences still lost the beginning. The problem got smaller, not solved. And from what I learnt reading about it, there was a second problem baked in from the start: **everything is sequential.** Word 2 waits for word 1. Word 3 waits for word 2. You can't process them in parallel. This made training slow — and apparently, fixing that is a big part of what made transformers so revolutionary when they arrived. More on that later. --- ## Early 2018 — ELMo duct-tapes it [Allen AI](https://allenai.org/allennlp/software/elmo) published ELMo (Embeddings from Language Models), and it had a clever idea: run TWO LSTMs. One reads left-to-right. One reads right-to-left. Combine their outputs. ```mermaid graph LR A["SHE PAID THE BILL ONLINE"] --> B["→ FORWARD LSTM →"] A --> C["← BACKWARD LSTM ←"] B --> D["COMBINE"] C --> D ``` Now every word gets a vector shaped by context from both directions. "Apple" near "iPhone" gets a tech-company vector. "Apple" near "ate" and "breakfast" gets a fruit vector. Same word, different vectors. That hadn't happened before. The results were dramatic — based on what I've read, every NLP task ELMo was tested on improved, some by 6-20%. Question answering, sentiment analysis, detecting names and places in text. And apparently, Google later used exactly this kind of context-aware approach to make Search better at understanding queries where the same word means different things. But. The forward LSTM and the backward LSTM never talk to each other while processing. They read the sentence independently and their outputs are stapled together after the fact. It's like two detectives investigating the same crime — one reads the evidence front-to-back, the other back-to-front — and they only compare notes after they're both done. Good enough to prove context matters. Not good enough for complete contextual understanding. --- ## June 2017 — "Attention Is All You Need" So let me recap where we are. Bag of Words — no meaning. Word2Vec — meaning, but frozen. RNNs — context, but it fades. ELMo — bidirectional context, but two models that never talk. Every step forward, something still breaks. Then a team at Google published a paper. The title was ["Attention Is All You Need."](https://arxiv.org/abs/1706.03762) The idea: what if every word could look at every other word? Not through a chain. Not through two separate passes. Directly. All at once. In an RNN, information travels like this: ```mermaid graph LR A["SHE"] -->|H1| B["PAID"] B -->|H2| C["THE"] C -->|H3| D["BILL"] D -->|H4| E["ONLINE"] ``` "online" only knows about "she" through a chain of 4 steps. By then, "she" is faded. In a transformer: ```mermaid graph TD A["SHE"] <--> B["PAID"] A <--> C["THE"] A <--> D["BILL"] A <--> E["ONLINE"] B <--> C B <--> D B <--> E C <--> D C <--> E D <--> E ``` Every word sees every word. No chain. No fading. Direct. They called this mechanism <span class="term">self-attention</span>. And it's parallel. All of it happens at the same time. RNNs are like one chef cooking each dish one after another — dish 2 waits for dish 1. The transformer is a kitchen where every chef can see what every other chef is cooking, all at the same time. Everyone works in parallel, and everyone knows the full picture. And it solved everything. Sequential processing? Gone. Fading context? Gone. Two models stitched together? Gone. Slow training? Gone. And it didn't stop at language. The same architecture worked for images, protein folding, code generation, music. **One design that works everywhere.** That's why people call this the biggest leap in AI history. One layer of self-attention isn't enough though. You can **stack** them. Each layer, every word looks at every other word — same as before. But in the second layer, each word's vector has already been shaped by the first layer. So when "apple" looks at "released" in layer 2, "released" already carries some "iPhone" context from layer 1. The connections get richer with every layer. Think of it like rounds of a group discussion. First round: everyone shares their initial take. Second round: people react to what others said. By round 12, the group understanding is far deeper than any single person's first impression. That's stacking. The original transformer had two halves. An <span class="term">encoder</span> — the detective. Reads input, investigates, builds understanding. And a <span class="term">decoder</span> — the author. Takes that understanding and generates output. Together, designed for translation: the encoder reads a French sentence and understands the meaning, the decoder writes it in English. But researchers realized you could pull them apart. OpenAI took the decoder half and built GPT — a model that generates text. Google took the encoder half and built BERT — a model that understands text. Same transformer, split into two different tools for two different purposes. --- ## June 2018 — OpenAI GPT takes the decoder OpenAI published [GPT](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) (Generative Pre-trained Transformer). They took just the decoder half and trained it on a simple task: predict the next word. ```mermaid graph LR A["SHE"] --> B["PAID"] --> C["THE"] --> D["???"] ``` Given "she paid the" → predict "bill". Because it's a decoder, it can only look left — at words that came before. It's blindfolded to the right. This makes it great at generating text (write one word, then the next, then the next) but limited at understanding. When processing "apple" in "Apple released a new iPhone", GPT has only seen "Apple" — it hasn't seen "iPhone" yet. It's making decisions with half the information. But OpenAI's real contribution wasn't just using the decoder. They showed that you could pre-train a transformer on a huge pile of unlabeled text, and the model would learn enough about language that you could then fine-tune it for any specific task with very little data. That idea — pre-train big, fine-tune small — became the blueprint for everything that followed, including BERT. But that left-to-right limitation bugged people. If you want to *understand* a sentence, you need to see the whole thing. Not just what came before. --- ## October 2018 — BERT puts it all together Google published [BERT](https://arxiv.org/abs/1810.04805): Bidirectional Encoder Representations from Transformers. BERT takes just the encoder half of the transformer. No decoder. Pure understanding. Why encoder-only? Because for most tasks — classifying emails, detecting fraud, understanding search queries — you don't need to *generate* text. You need to *understand* it. ### The architecture Stack 12 encoder layers on top of each other. Each layer runs self-attention — every word looks at every other word, then passes its updated vector to the next layer. But here's the clever part. BERT adds a fake first word to every input called the **[CLS] token** (CLS = classification). It's not a real word. It has no meaning of its own. It's just there to *listen*. Imagine a meeting with 10 people. Everyone is talking, sharing ideas, debating. One person sits in the corner with a notepad. Doesn't speak. Just listens. By the end of the meeting, their notes capture the gist of the entire discussion. [CLS] is that person. Through 12 layers of self-attention — 12 rounds of every word looking at every other word — [CLS] is listening to all of it. After 12 rounds, its vector — 768 numbers — has been shaped by every word in the sentence. That's the embedding. One vector that represents the whole input. ### Training: Masked Language Model Here's where BERT got clever. But first — what does it mean to "train" a model? Normally, you need **labels**. A label is a human-provided answer. Someone reads an email and tags it: "spam." Someone reviews a transaction and tags it: "fraud." You give the model thousands of these input-answer pairs and it learns the pattern. This is expensive — someone has to manually tag every single example. BERT skipped all of that. But there's a problem with bidirectional models. If every word can see every other word, then when you ask "what word goes here?" the model can just... look at the answer. It's cheating. It can indirectly see itself through the layers. The fix? **Randomly hide words — about one in seven — and make the model guess what's missing.** Now it can't cheat. The hidden word IS the label. It comes from the text itself. No human needed. ``` Input: "she [MASK] the bill online" Model: "bill... online... probably a payment... 'paid'?" Answer: "paid" ✓ ``` ``` Input: "Apple [MASK] a new iPhone yesterday" Model: "Apple... iPhone... product launch... 'released'?" Answer: "released" ✓ ``` To guess "paid," the model has to understand that bills get paid. To guess "released," it needs to know Apple is a tech company that launches products. The blanks force it to actually understand what's going on. Do this billions of times across all of Wikipedia and a pile of books. The model learns grammar, meaning, context, common sense — all as a side effect of guessing hidden words. This is called <span class="term">self-supervised learning</span>. The data labels itself. ### Using it: the classifier So now you have a model that understands language. How do you actually use it? BERT gives you the [CLS] embedding — 768 numbers that encode the meaning of the input. But those 768 numbers don't tell you "fraud" or "legit." They tell you what the input *means*. You need a decision-maker. A judge. That's the <span class="term">classifier</span>. And it's almost comically simple. Take the 768 numbers. Multiply by a set of weights. Get a score. ```mermaid graph LR A["768 NUMBERS"] --> B["CLASSIFIER"] B --> C["FRAUD?"] B --> D["LEGIT?"] ``` Same idea works for any task — just change the output labels: ```mermaid graph LR A["768 NUMBERS"] --> B["CLASSIFIER"] B --> C["REFUND"] B --> D["KYC"] B --> E["SETTLEMENT"] ``` The classifier gives each option a score. Higher score = more likely. ``` refund: 4.2 KYC: 1.1 settlement: 7.8 ``` But what does 7.8 mean? Is that confident? These raw scores (called **logits**) could be any number. Hard to judge on their own. **Softmax** fixes this — it converts raw scores into percentages that add up to 100%: ``` refund: 4.2 → 3% KYC: 1.1 → 1% settlement: 7.8 → 96% ``` Now you know. The model is 96% sure this is a settlement issue. If it was 40% refund, 35% KYC, 25% settlement — you'd know the model is guessing. So the full picture: ```mermaid graph BT subgraph "ENCODER - THE DETECTIVE" A["[CLS] SHE PAID THE BILL"] --> L1["LAYER 1"] L1 --> L2["LAYER 2"] L2 --> L3["..."] L3 --> L12["LAYER 12"] end L12 --> C["[CLS] = 768 NUMBERS"] subgraph "CLASSIFIER - THE JUDGE" C --> D["SOFTMAX"] D --> E["FRAUD 96%"] end ``` BERT reads the input and produces 768 numbers. The classifier takes those numbers and scores each option. Softmax turns scores into confidence. Detective does the investigation, judge makes the call. ### Why this was a big deal Before BERT, if you wanted to build a fraud detector, you trained a model from scratch on fraud data. Wanted to auto-route support tickets? Train a separate model from scratch. Every task started from zero. BERT changed this. Train one massive model on understanding language. Download it. Stick a tiny classifier on top. Fine-tune on your specific task. Want to detect fraud? Download BERT, add a classifier, train on transaction data. Want to route support tickets? Same BERT, different classifier, train on ticket data. The understanding is shared. Only the decision-maker changes. One model that understands language. Download it. Plug in different classifiers for different jobs. The understanding is reusable. Only the decision-maker changes. --- ## The road — zoomed out | Year | What happened | What it solved | What still broke | |---|---|---|---| | 2000s | Bag of Words, TF-IDF | Turn text into numbers | No meaning, no order | | 2013 | Word2Vec | Words become meaningful vectors | One vector per word — no context | | 2014 | GloVe | Better training for same idea | Same flaw — static embeddings | | 2015 | RNNs / LSTMs | Added sequential memory | Memory fades, can't parallelize | | 2018 | ELMo | Bidirectional context | Two separate models stitched together | | 2017 | Transformer | Self-attention, parallel processing | Full architecture — complex | | 2018 | GPT | Decoder-only, pre-train + fine-tune | Left-to-right only, can't see full context | | 2018 | BERT | Encoder-only, masked training, reusable | (Next post's problem) | Each step solved the previous step's problem. Each step hit a new wall. Until the transformer solved the walls that mattered most: context and speed. And the same idea works beyond text. Images. Audio. Transactions. Code. Anywhere you have structured data and want meaningful representations, the transformer architecture shows up. --- ## What's next This post covered the road — how we got from counting words to transformers, and why BERT's architecture became the foundation for everything that followed. Next: **self-attention deep dive.** The actual mechanism. How does "apple" *look at* "iPhone" and decide to become a tech-company vector? What are queries, keys, and values? How does the math actually work? That's the part I'm most curious — and most confused — about. See you there. --- *This is part of the [Foundation Model Internals](/blogs) series — a software engineer learning how models work, from scratch.* *Like the AI models in this post, this blog itself builds on the understanding that already exists. Jay Alammar's [The Illustrated BERT](https://jalammar.github.io/illustrated-bert/) was my starting point. I read it, had a ton of questions, discussed them with Claude to build a deeper understanding of this evolution, and turned that discussion into this blog — simplified for non-ML folks like me beginning the journey of understanding ~~how the universe works~~ how AI works.*