LLMs Understand Language via Machine Learning
You don't need to understand the math. You just need the mental model.
It's Just Next-Word Prediction
Here's the whole thing: a large language model reads a sequence of words and predicts what comes next. That's it. Everything else—the conversations, the code, the poetry—emerges from doing that one thing really, really well.
Next-Word Prediction
Hover over each word to see how much it influences the prediction. The lines show attention—thicker means more influence.
↑ attention weight (how much each token influences the prediction)
The model doesn't "know" what a cat is. But it's seen "cat sat on the" followed by "mat" enough times that "cat" pulls hard toward that prediction. Every token votes on what comes next.
When you type "the cat sat on the", the model has learned from reading billions of sentences that "mat" or "floor" or "couch" are likely next words. Not because anyone told it what a cat is—but because it saw the pattern enough times.
Machine Learning = Learning From Examples
Traditional software is rules someone wrote: if the user types X, do Y. Machine learning is different. You show the computer millions of examples and it figures out the rules on its own. Nobody hand-wrote a dictionary of grammar rules for ChatGPT—it picked up the patterns of language by seeing enough of it, the same way a baby does by hearing the family talk.
Traditional Software
Human writes rules ↓ Computer follows rules
Machine Learning
Human shows examples ↓ Computer learns rules
An LLM is Just Two Files
Andrej Karpathy (one of the founders of OpenAI) put it best: an LLM is literally two files.
An LLM is Two Files
~140 GB of floating point numbers
1.2003, -0.0042, 0.5517,
-0.7782, 0.1193, 0.3344,
... 70 billion more
~500 lines of code
compute attention
apply feed-forward
predict next token
... that's basically it
Via Andrej Karpathy. The magic is in the parameters—the code is almost trivially simple.
The parameters file is what the model learned. Every relationship between every word it's ever seen, compressed into billions of numbers. Llama 2 70B has 70 billion parameters—each one a decimal number that encodes some tiny fragment of how language works.
The run file is the code that reads those parameters and does the next-word prediction. It's about 500 lines of code. That's it. The magic is in the parameters, not the code.
Training = Compressing the Internet
To create those parameters, researchers take a massive chunk of the internet—roughly 10 terabytes of text—and feed it through the model. The model reads every sentence and tries to predict the next word. When it's wrong, it adjusts its parameters slightly. Do this trillions of times and you get something that has internalized the patterns of human language.
Training = Compression
Like a JPEG of the internet. Lossy, but the structure is preserved.
Karpathy calls this "lossy compression." You're taking 10TB of text and squeezing it down to ~140GB of parameters. Like a JPEG of the internet—you lose some detail, but the structure is preserved.
How Information Flows
If you want to go deeper: the architecture these models use is called a Transformer. The key insight (from
@repligate
) is that there are two highways for information:
Each word gets refined as it passes through layers of the network. Like revising a draft—each pass makes it better.
Each word looks back at all previous words to gather context. "The" means something different after "cat" vs after "president."
The number of possible paths information can take through these two highways explodes combinatorially. This is part of why these models are so capable—and so hard to fully understand.
For the interactive version of this, see
How LLMs Actually Work
.