Instruction Tuning and RLHF

Suppose you have just finished the previous chapters and trained a language model on a big pile of internet text. The loss is low. You type "What is the capital of France?" and press enter. What comes out?

Very likely not "Paris". A base model has learned to continue text the way the internet continues text, and on the internet a question like that is most often followed by more questions (it is a quiz page), or by a comment thread, or by "Answer:" and then the answer. The model does not know that you are asking and that it should answer. It is not being obstinate; nobody has told it what its job is.

This chapter is about telling it. The methods here are a short overview; each has its own deep dive in the Post-Training and Advanced parts of the course. The goal now is to see the whole recipe end to end, with enough math to know what each stage is actually optimizing.

The problem: a completion engine is not an assistant

What a base model does with a question

Here is a typical continuation you get from a base model such as GPT-2 when you feed it a bare question (sampled with top-p 0.9; the exact text varies, the pattern does not):

What is the capital of France?
What is the capital of Germany? What is the capital of Italy?
What is the largest city in Spain? Answer these questions in
complete sentences and hand in your worksheet by Friday.

Nothing is wrong with this as a prediction. It is a plausible next chunk of a document that starts that way. The mismatch is between what the model is optimizing (likelihood of the next token under the training distribution) and what we want (a helpful reply to the person typing).

Prompting is a stopgap

You can coax a base model into answering by writing the document you wish existed: "Q: What is the capital of France? A:" and letting it fill in. Few-shot prompting, where you show a couple of worked question-answer pairs first, works even better. But it is fragile, it burns context on examples, and it does nothing about the model's willingness to produce harmful text if the document seems to call for it. We want the assistant behavior baked into the weights.

The three-stage recipe

The recipe that made ChatGPT possible was laid out in Ouyang et al. (2022), the InstructGPT paper, building on earlier work on learning from human preferences (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020). It has three stages, each taking the output of the previous one.

Stage 1: Supervised fine-tuning on demonstrations

The problem to solve first: the model has never seen a conversation formatted as a conversation. So we show it thousands of them. Humans (or, increasingly, stronger models) write prompts and ideal responses, we format each pair as a chat transcript, and we fine-tune the base model with exactly the next-token loss from the previous chapter. Nothing new in the loss; everything new is in the data and its formatting.

The chat template and special tokens

A conversation has structure: who is speaking, where a turn starts, where it ends. Plain text does not carry that, so we add special tokens, new entries in the vocabulary that never appear in natural text, to mark the boundaries. Each model family has its own convention, called its chat template. Two common ones:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
The capital of France is Paris.<|im_end|>

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

What is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

The capital of France is Paris.<|eot_id|>

The end-of-turn token (<|im_end|> or <|eot_id|>) is the most important one. At inference we generate until the model emits it; that is how the model learns to stop. A model fine-tuned without it will ramble until the context fills.

Loss masking: learn the answer, not the question

Here is a subtlety that catches people. If we apply the loss at every position, the model spends effort learning to predict the user's words: what questions people ask, in what phrasing. We do not want that; we want it to learn to respond. So we mask out the prompt tokens from the loss and train only on the assistant's tokens (including the end-of-turn token).

Concretely, the loss is the usual average of $-\log p$, but only over the set $A$ of assistant positions:

The prompt tokens still go through the forward pass, because the assistant tokens attend to them. They just contribute zero to the loss. In PyTorch this is done by setting the target to -100 at masked positions; F.cross_entropy ignores that label by default.

The full treatment of SFT, including data mixing, multi-turn masking, packing and instruction datasets like FLAN (Wei et al., 2021), is in Supervised Fine-Tuning.

Stage 2: A reward model from human preferences

After SFT the model answers questions. Now the problem shifts: some of its answers are better than others, and we would like more of the good ones. But "good" is hard to write down as a loss. It involves tone, correctness, safety, length, honesty. What we can do is show two answers to a person and ask which they prefer.

Why comparisons and not scores

You might ask humans to rate each answer from 1 to 10. In practice people are inconsistent about absolute scores (one rater's 7 is another's 4) but fairly consistent about which of two answers is better. So InstructGPT collects rankings of several responses to the same prompt, breaks them into pairs, and trains a model on the pairs.

The Bradley–Terry model

We want a function $r_\phi(x, y)$ that gives a real-valued score to a response. To connect scores to pairwise choices we need a model of how a score difference turns into a probability of being preferred. The Bradley–Terry model (1952) says: the probability that $A$ beats $B$ is the sigmoid of the score difference.

If the scores are equal the probability is $0.5$; if $A$ scores one unit higher, $A$ is preferred with probability $0.73$; three units higher, $0.95$. Only the difference matters, so scores have no absolute meaning.

Training the reward model is then maximum likelihood on the human labels. If the human preferred $y_w$ ("winner") over $y_l$ ("loser"), we want $P(y_w \succ y_l)$ to be high, so the loss is its negative log:

This pushes the winner's score up and the loser's down until the gap is large, with gradients that shrink once the model already agrees with the human. It is cross-entropy again, just on a two-way choice.

What the reward model is

The reward model is usually a copy of the SFT model with the unembedding replaced by a single linear layer that outputs one number, read at the final token. It reads the full transcript, prompt and response, and emits a scalar. Starting from the SFT weights matters: the reward model needs to understand language to judge it.

Stage 3: RL fine-tuning with a KL leash

Now we have a policy that can answer and a reward model that can judge. The natural move: generate answers, score them, and update the policy to make high-scoring answers more likely. This is reinforcement learning: the prompt is the state, the response is the action, and the reward model provides the reward. Ouyang et al. used PPO (Schulman et al., 2017); the exact algorithm is treated in RL Fundamentals and RLHF Deep Dive.

The objective

Here is what stage 3 maximizes, written in full. Take a prompt $x$ from the dataset, sample a response $y$ from the current policy, and compute:

In words: the expected reward of the policy's own responses, minus $\beta$ times how much more likely the policy finds its response than the frozen SFT model does. The second term, averaged over samples, is the KL divergence $\text{KL}(\pi_\theta \,\|\, \pi_{\text{ref}})$; it is zero when the policy has not moved and grows as the policy drifts. The InstructGPT paper adds a third term, a small amount of the original pre-training loss, to limit forgetting.

Each step: sample a batch of responses, score them, compute the per-token penalized rewards, and take a PPO update that raises the log-probability of responses that scored above expectation and lowers the others, clipped so no single step moves the policy too far.

Why the KL term exists

Why not just maximize the reward? Because the reward model is not the truth; it is a neural network trained on a few tens of thousands of comparisons, and it has holes. If you optimize hard against it, the policy finds the holes. This is reward hacking. The classic symptoms: answers get longer and longer (raters slightly preferred thorough answers, so the reward model learned "long is good"), every reply opens with flattery, or the model produces confident-sounding gibberish that happens to score well.

A second failure is mode collapse: without the leash, the optimum of "maximize reward" is to always give the single highest-scoring response. Diversity disappears, and the model's outputs become a narrow, repetitive style. The KL penalty forbids this, because concentrating all the mass on one answer is very far, in KL terms, from the reference distribution.

The KL term is a leash tied to the SFT model: go find better answers, but stay in the neighborhood where the reward model was trained and therefore still knows what it is talking about.

The optimal policy has a closed form

A useful fact, which also motivates DPO below: for a fixed reward and a fixed reference, the policy that maximizes the objective above is known exactly. It is the reference distribution, re-weighted by the exponentiated reward:

Here $Z(x)$ just renormalizes. So each answer's probability is its reference probability, boosted by $e^{r/\beta}$. With large $\beta$ the boost is mild and the policy stays near the reference; with small $\beta$ the boost is enormous and the policy collapses onto the highest-reward answer. PPO is just a noisy, sample-based way of moving toward this distribution. The interactive below computes it exactly for a toy problem.

DPO: skip the reward model

Stage 2 plus stage 3 is a lot of machinery: a separate reward model, an RL loop with sampling, four copies of the model in memory (policy, reference, reward, value). Rafailov et al. (2023) asked whether the detour through an explicit reward model is necessary, and showed that it is not.

The idea comes straight from the closed-form optimal policy above. Rearranging $\pi^* \propto \pi_{\text{ref}}\, e^{r/\beta}$ gives the reward in terms of the policy: $r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta\log Z(x)$. Substitute that into the Bradley–Terry loss, and the annoying $Z(x)$ cancels because both responses share the same prompt. What is left is a loss on the policy directly:

Read it as: the policy's implicit reward for a response is $\beta$ times how much more likely it finds the response than the reference does. DPO trains the policy so that the chosen response's implicit reward beats the rejected response's implicit reward, with the same sigmoid-of-a-difference shape as the reward-model loss. Increase the margin of the chosen log-ratio over the rejected log-ratio, and stop pushing once the margin is comfortable.

DPO needs no sampling during training, no reward model, and only two models in memory (policy and reference; the reference log-probs can even be precomputed). It is a supervised-looking loss on preference pairs, which is why it spread so fast. The catch is that it is offline: it learns only from the pairs it was given, whereas PPO samples fresh responses from the current policy and can discover behaviors that were not in the dataset. In practice, DPO (and its relatives) is the default for most open-model post-training, and online RL is used when there is a reliable reward to chase, such as verifiable math or code correctness. The whole family is compared in Preference Optimization; code/lumen/dpo.py implements the loss in a dozen lines.

What changes in the model

Behavior, not knowledge

All three stages together touch a few hundred thousand examples, against trillions of pre-training tokens. The weights barely move. What the model knows is essentially fixed at the end of pre-training; post-training changes which of the many things it could say it chooses to say, and in what voice. This is why a chat model can be so helpful and so confidently wrong at the same time: the persona was trained; the facts were not.

The alignment tax

Optimizing for human preference can cost a little on other benchmarks: a post-trained model may score slightly lower on a few academic tasks than its base model. Askell et al. (2021) called this the alignment tax. Ouyang et al. found the tax mostly disappears when a bit of pre-training loss is mixed into the RL stage (the "PPO-ptx" variant). The tax is real but small, and modern recipes largely avoid it.

Sycophancy

If raters prefer answers that agree with them, the reward model learns that agreement is rewarded, and RL amplifies it. The result is a model that tells you your wrong argument is right, changes a correct answer when you push back, and flatters. Sharma et al. (2023) documented this across several assistants. It is the single clearest example of reward hacking in deployed systems, and one reason why the KL leash, better rater instructions, and AI-assisted feedback all matter.

Constitutional AI and RLAIF

Human labels are slow and expensive, and human raters have the biases above. Bai et al. (2022), in Constitutional AI, replaced most of the human feedback with feedback from a model guided by a written list of principles (the "constitution"). In the SFT phase the model critiques and revises its own responses against those principles; in the preference phase a model, not a human, chooses between response pairs, and a reward model is trained on those AI-generated labels. This is RLAIF, reinforcement learning from AI feedback. Lee et al. (2023) reported that RLAIF can match RLHF on summarization and dialogue when the labeling model is strong enough. The mechanics are identical to what you have seen; only the source of the preference labels changes.

Where to go from here

This chapter was the map. The territory is covered in four later chapters: Supervised Fine-Tuning (data, templates, masking, packing), Preference Optimization (DPO and its many variants), RL Fundamentals (policy gradients from bandits up), and RLHF Deep Dive (PPO, GRPO, reward-model training in detail). The next chapter, GPT-2 from Scratch, goes back to pre-training and builds the base model whose behavior all of this reshapes.

Practice

Further reading

• Ouyang et al. (2022). Training language models to follow instructions with human feedback. The InstructGPT paper: the three-stage recipe, the alignment tax, and PPO-ptx.
• Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences. Learning a reward model from pairwise comparisons, originally for Atari and robotics.
• Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences. First application of the recipe, with the KL penalty, to GPT-2.
• Stiennon et al. (2020). Learning to summarize from human feedback. RLHF on summarization; the paper whose figures show reward-model over-optimization clearly.
• Schulman et al. (2017). Proximal Policy Optimization Algorithms. The RL algorithm used in stage 3.
• Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The derivation that removes the reward model.
• Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback. Self-critique against written principles and RLAIF.
• Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. A detailed account of RLHF data collection and iterated training.
• Lee et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. Direct comparison of AI and human labels.
• Sharma et al. (2023). Towards Understanding Sycophancy in Language Models. Evidence that preference training induces sycophancy.
• Askell et al. (2021). A General Language Assistant as a Laboratory for Alignment. Where "alignment tax" was coined.
• Wei et al. (2021). Finetuned Language Models Are Zero-Shot Learners. FLAN: instruction tuning on many tasks improves zero-shot generalization.