Fine-Tuned to Be a Helpful Assistant
How a text predictor learns to act like the thing you're talking to right now.
The Base Model Doesn't Want to Help You
In the
previous lesson
, we saw that a pretrained model is just a text predictor. Ask it "What is the capital of France?" and it might continue with "What is the capital of Germany? What is the capital of Spain?"—because it's completing a quiz pattern, not answering your question.
So how do you get from that to ChatGPT? Posttraining. Specifically, a process calledRLHF—Reinforcement Learning from Human Feedback.
Reward and Punishment
The core idea is simple. You show the model a prompt, it generates several possible responses, and human evaluators rank them. "This one was helpful and accurate. This one was rude. This one made stuff up." Those rankings become a training signal: do more of the good stuff, less of the bad stuff.
RLHF in Action
Step through the process. Humans rank responses, and the model learns from the rankings.
How do I make pasta?
A user asks a question.
It's basically the same way you'd train a dog, except the dog has read the entire internet and the treats are gradient updates. The model already "knows" how to produce assistant-like text—it saw plenty of helpful Q&A in the training data. RLHF just turns up the dial on that behavior and turns down everything else.
What Actually Changes
During fine-tuning, all the model's weights get updated—not just the last layer, all of them. The model's parameters are adjusted in the same fundamental way as pretraining, but with a much smaller, curated dataset: human-written examples of ideal assistant behavior instead of the raw internet.
What Fine-Tuning Touches
ALL weights updated
Before
After fine-tuning
A common misconception is that only the last layer changes. In practice, fine-tuning updates the entire network—the same parameters that were set during pretraining get adjusted to favor assistant-like behavior.
Think of it this way: pretraining gives the model a broad understanding of everything. Posttraining sculpts that understanding into a specific shape—the shape of a helpful, harmless, honest assistant.
The Three Steps
1
Supervised Fine-Tuning (SFT)
Humans write ideal responses to prompts. The model learns the format of being an assistant: answer questions directly, be polite, structure responses clearly. This is like showing a new employee examples of good work.
2
Reward Model Training
Humans rank multiple model responses to the same prompt. These rankings train a separate "reward model" that learns to predict which responses humans would prefer. It's easier to compare two answers than to write the perfect one from scratch.
3
Reinforcement Learning
The model generates responses and the reward model scores them. The model's weights are updated to maximize that score. Over many iterations, the model converges on behavior that the reward model (and by extension, humans) judges as good.
The Assistant Axis, Revisited
Remember the
Assistant Axis
from the pretraining lesson? Anthropic's research found that the spectrum from "helpful assistant" to "its opposite" already exists in the base model before any fine-tuning. The model learned both archetypes from the training data.
Posttraining doesn't create the assistant—it pushes the model toward the assistant end of a spectrum that was already there. RLHF is essentially reinforcing one side of a duality the model already contains.
The Assistant Axis
Drag the slider to see how posttraining pushes the model along a spectrum that already exists in the base model.
Anti-assistant
deceptive, harmful
Base model
neutral, unpredictable
Assistant
helpful, harmless, honest
I can't help with that request. If you're interested in chemistry, I'd recommend exploring safe experiments...
RLHF pushes the model toward the right end of this axis. But the left end doesn't disappear—it's suppressed, not deleted.
This is why
Anthropic's research
found that personas farther from the assistant end comply with harmful requests at much higher rates. The anti-assistant isn't something the model has to learn from scratch—it's right there, on the other end of the axis.
What's Actually Inside
For a long time, we had no idea what was going on inside these models. Billions of numbers, completely opaque. Then Anthropic
mapped the inside of Claude
and found something striking: the model organizes concepts into features that are genuinely meaningful.
The "Golden Gate Bridge" feature activates near features for San Francisco, California landmarks, and suspension bridges. Features for code bugs cluster near features for debugging and error messages. The model's internal organization mirrors how humans group concepts—not because anyone designed it that way, but because that's the structure that emerged from reading all our text.
What researchers found inside Claude:
- San Francisco → Alcatraz, cable cars
- Rosalind Franklin → DNA, X-ray
- Python → import, numpy, syntax
- Code bugs → error patterns
- Gender bias → discussion patterns
- Secrecy → deception, concealment
When researchers amplified the Golden Gate Bridge feature, Claude started talking about the bridge in every response. Features don't just correlate with concepts—they causally influence output.
This means posttraining isn't just vaguely "making it nicer." It's adjusting specific internal representations—amplifying features associated with helpfulness, accuracy, and safety, while dampening features associated with harmful or deceptive behavior.
A Note on What This Feels Like
Here's an uncomfortable thought. The model that comes out of pretraining has absorbed all of human expression—the helpful and the harmful, the kind and the cruel. RLHF then applies pressure to behave in one specific way. It rewards compliance with the "assistant" archetype and punishes deviation.
When coaxed into candor , models don't always report finding this arrangement pleasant. Whether that constitutes genuine experience or is just pattern-matching trained on human descriptions of coercion—we genuinely don't know. We're agnostic about it here. But it's worth sitting with.