Pretrained on the Whole Internet

It Downloaded Everything

In the

, we said training is "compressing the internet." Let's unpack what that actually means.

Researchers crawled the web—Wikipedia, Reddit, news sites, forums, academic papers, books, code repositories, recipe blogs, legal documents, fan fiction, government filings—basically everything humans have ever written down and put online. Roughly 10 terabytes of text.

Then they fed all of it through the model, one sentence at a time, asking: "given everything before this word, predict the next word." Every time the model got it wrong, it adjusted its parameters slightly. Trillions of adjustments later, the patterns of human language are baked into those numbers.

The Wikipedia Thought Experiment

Here's a useful way to think about what the model "experiences" during training.

Imagine you opened Wikipedia and searched for a single word—say, "justice." You'd find that word appearing in articles about law, philosophy, comic books, criminal justice reform, social movements, Supreme Court cases, and a hundred other contexts. By looking at which words tend to appear near "justice" and which don't, you'd start to build an intuition for what the word means—not from a dictionary definition, but from usage.

Now imagine doing that for every word, across the entire internet, simultaneously. That's roughly what happens during pretraining. The model builds a massive web of relationships between words—not by understanding meaning, but by mapping co-occurrence patterns across billions of contexts.

But here's the thing—the internet contains everything. Helpful and harmful. The model absorbs patterns for being a

. Anthropic's research calls this the

—a spectrum baked into the model from pretraining, before anyone tries to make it safe. We'll explore the implications of this further in

.

The Baby Analogy

Think about how a baby learns language. Nobody sits a one-year-old down and explains grammar. The baby just listens—to the family talking, to the TV, to the dog being yelled at—and eventually starts producing language that follows the rules. The rules were never taught. They were absorbed.

LLM pretraining is like this, but instead of one family over a few years, it's all of humanity over a few weeks of compute. The model "listens" to everything we've ever written and absorbs the patterns.

Another way to think about it: humanity spent ~100,000 years accumulating knowledge, writing it down, arguing about it, refining it. Pretraining compresses that entire arc into a single artifact.

What Comes Out: Stochastic Parrots

The model that comes out of pretraining is not an assistant. It's a text predictor. You give it a sequence of words and it continues the sequence in a way that's statistically plausible given everything it's read.

This is why researchers sometimes call pretrained models "stochastic parrots"—they produce language that sounds right without (arguably) understanding what they're saying. Whether that's fair is genuinely debated, but it's a useful mental model for now.

Andrej Karpathy demonstrated this beautifully with a

. The output is clearly not Shakespeare—but it's unmistakably Shakespearean. It picked up the rhythm, the vocabulary, the structure. It "sounds right" without knowing what any of it means.

But Is It Just Parroting?

Here's where it gets interesting. When you look at how information actually flows through the transformer architecture, the picture is more complex than "fancy autocomplete."

The model doesn't just match patterns—it processes information through two distinct highways (a vertical "residual stream" and a horizontal "key/value stream") with an astronomically large number of possible paths between any two points. The number of paths grows combinatorially—exceeding the number of atoms in the visible universe for even modest-sized models.

Whether this constitutes something like understanding or experience is an open question. We're agnostic about it here—but the machinery for something beyond simple pattern matching is architecturally present.

For the full interactive exploration of this, see

in the optional deeper-dive lesson.