Post-Training: Overview

Here is a strange fact. After spending millions of dollars and trillions of tokens on pre-training, you get a model that, when asked "What is the capital of France?", might reply with "What is the capital of Germany? What is the capital of Spain?" It is not being difficult. It is doing exactly what it was trained to do: continue text.

A base model is a text continuer. It has read a huge slice of the internet and learned to predict what comes next. Quiz questions on the internet usually come in lists, so it continues the list. Nobody asked it to be helpful, and nothing in its training ever rewarded being helpful.

Users do not want a text continuer. They want an assistant: something that reads a question as a request, answers it, stops, follows instructions, admits when it does not know, declines the genuinely dangerous stuff, and can call a calculator when it needs one. The gap between "predicts the next token" and "behaves like an assistant" is what post-training closes.

This chapter is the map. The next four chapters are the territory: supervised fine-tuning, preference optimization, tools and safety, and a case study of a fully open recipe.

The problem: a base model completes, it does not answer

Let us be concrete about what goes wrong. Feed a base model a prompt and it produces the most likely continuation, which is not the most useful one.

What "assistant" actually means

When people say they want an assistant, they are bundling several separate things together. It helps to unbundle them, because different stages of post-training target different items on the list.

• Format. Read the prompt as a request. Reply. Stop. Use the roles of a conversation (user, assistant, system, tool).
• Helpfulness. Actually do the thing asked, at the right length and level, in the right tone.
• Honesty. Say "I do not know" instead of inventing a citation. Calibrate confidence.
• Harmlessness. Decline requests that would cause serious harm, without refusing everything that sounds edgy.
• Reasoning. Think through multi-step problems rather than pattern-matching to the surface.
• Interfaces. Call tools with the right syntax, read their results, and continue.

Format is cheap to teach. Taste is harder. Reasoning is the expensive frontier. That ordering is, not coincidentally, the order of the pipeline.

The pipeline at a glance

Every modern post-training recipe is some arrangement of the same six or seven stages. Here they are as one picture.

What each stage changes, and what it costs

The picture above hides the most important number in this chapter. Let us get it out in the open before we go stage by stage.

A quick way to estimate training compute is that each token costs about six floating-point operations per parameter (forward pass plus backward pass). So if a model has $N$ parameters and sees $D$ tokens, the cost is roughly:

That is it: parameters times tokens times six. It ignores attention's quadratic term and all the engineering, but it is within a factor of two for every transformer you will meet.

Keep that ratio in mind as we walk through the stages. Post-training is where most of the behaviour comes from, and almost none of the cost.

Supervised fine-tuning: teach the format

The problem SFT solves is the one we opened with: the model does not know that a request was made. The fix is embarrassingly direct. Show it thousands of conversations where a user asks and an assistant answers, and train it with the same next-token loss as pre-training, but only on the assistant's tokens.

What changes: the model learns to reply, to stop, to use markdown, to say "I cannot help with that" in the right places, and to emit tool-call syntax. What does not change much: what it knows. SFT is a thin coat of paint on a large building. The next chapter goes deep on this, including why training on too many things the model does not know can make it hallucinate more.

Preference optimization: teach taste

After SFT the model answers, but it answers the way its demonstration writers did, which is one particular good answer among many. Users have opinions about which answers are better. Capturing those opinions as demonstrations is expensive (someone has to write the ideal answer); capturing them as comparisons is cheap (someone glances at two answers and clicks).

So the second stage collects pairs $(x, y_w, y_l)$: a prompt, the answer people preferred, the one they did not. Then one of two families of methods pushes the model toward $y_w$: train a reward model and optimise against it with RL (the RLHF path, Ouyang et al. 2022), or skip the reward model and optimise the pairs directly (the DPO path, Rafailov et al. 2023). Either way the model is kept on a leash to a frozen reference copy so it does not drift into gibberish that happens to score well. This is the flagship chapter of this part.

RL with verifiable rewards: teach reasoning where you can check it

Preferences capture taste but they are noisy and expensive at scale. For some tasks you do not need a human at all: a math problem has an answer key, a coding problem has unit tests. When a verifier exists, you can sample many attempts per prompt, score each one automatically, and reinforce the successful ones.

This is the stage that DeepSeek-R1 (DeepSeek-AI, 2025) made famous, using the GRPO algorithm (Shao et al. 2024). Reported effects include much longer chains of thought and large gains on math and code. It is dashed in Figure 1 because it only applies where a checker can be written, and because it is the most compute-hungry post-training stage: the cost is dominated by generating those many samples, not by the gradient steps.

Safety and tool tuning: teach boundaries and interfaces

Two things remain that neither generic demonstrations nor generic preferences cover well. The model must decline a small set of genuinely dangerous requests without becoming a scold, and it must speak the structured dialect of tools: emit a JSON call, wait, read the result, continue. In practice these are rarely separate training runs; the data is mixed into the SFT and preference stages. Chapter 4 covers both.

Evaluation: decide what ships

Here is the catch that makes post-training feel different from pre-training. During pre-training, held-out loss is an excellent progress bar. During post-training it is almost useless: a model can have lower loss on demonstration data and be a worse assistant, because the loss rewards imitating one specific answer, not being helpful.

So the field built other instruments. LLM-as-judge benchmarks such as MT-Bench (Zheng et al. 2023) and AlpacaEval (Li et al. 2023; the length-controlled version by Dubois et al. 2024) ask a strong model to compare answers. Human evaluation, including the crowd-sourced Chatbot Arena (Chiang et al. 2024), is the gold standard and the most expensive. Capability benchmarks (GSM8K, MATH, IFEval, HumanEval) check that specific skills did not regress. A typical release decision looks at all three.

All sizes are rough ranges spanning published open recipes (Ouyang et al. 2022; Touvron et al. 2023; Lambert et al. 2024); individual labs vary by an order of magnitude in either direction.

The four kinds of data

Different stages need different data, and the difference is mostly about how much a human has to do per example. Cheapest to most expensive per item, then reversed for how much each item teaches:

Prompts are the cheapest and the most reusable. A good prompt set (diverse, realistic, decontaminated against your test sets) can be paired with model samples over and over. Real user traffic, once you have it, is the best prompt source there is.

Demonstrations are a prompt plus an ideal answer. Writing a good answer takes minutes of skilled human time, which is why labs distill them from a stronger model when they can, and why quality beats quantity (Zhou et al. 2023, LIMA).

Preference pairs are a prompt plus two candidate answers plus a verdict. The candidates come from the model itself, so the only human work is the verdict, which can also be delegated to an LLM judge (with known biases toward longer and first-listed answers).

Verifiable tasks are a prompt plus a program that checks answers. The upfront cost is writing the checker; after that every sample is labelled for free. This is the data type that scales best, and the one that exists for the fewest tasks.

Chat templates and roles

Before any of this training can happen we need to answer a mechanical question: a transformer takes one flat sequence of tokens. A conversation has multiple speakers. How do you flatten a conversation into a string so that the model can tell who said what?

The answer is a chat template: a fixed way of wrapping each message in role markers. Different model families use different markers, but the structure is always the same. Here is a common style, with special tokens spelled out:

<|im_start|>system
You are a concise assistant. Use tools when arithmetic is needed.<|im_end|>
<|im_start|>user
What is 17% of 3,482?<|im_end|>
<|im_start|>assistant
<tool_call>{"name": "calculator", "arguments": {"expression": "0.17 * 3482"}}</tool_call><|im_end|>
<|im_start|>tool
591.94<|im_end|>
<|im_start|>assistant
17% of 3,482 is 591.94.<|im_end|>

Four roles appear. system sets standing instructions (persona, rules, available tools). user is the human turn. assistant is what the model produces. tool is a message injected by the runtime with a tool's result; the model never writes it, it only reads it.

Two details matter enormously and are easy to get wrong.

First, the markers such as <|im_start|> are usually single special tokens added to the vocabulary, not the text characters. If you train with one template and run inference with a different one, the model sees tokens it has never seen in that position and behaves like a base model again.

Second, at inference time the prompt ends right after <|im_start|>assistant followed by a newline. The model's job is to continue from there and to emit <|im_end|> when it is done. Learning to emit that stop token is, quite literally, how the model learns to stop talking.

Six ideas you will meet again

The next chapters lean on a handful of concepts repeatedly. Here is each one in a paragraph, so that when it shows up in the middle of a derivation it feels familiar.

Loss masking

Question: if we train on whole conversations, will the model learn to write user messages too? Yes, and that wastes capacity and teaches the wrong thing. The fix is to zero out the loss on every token the model did not author. In PyTorch this is the label value -100, which cross_entropy ignores. Same forward pass, fewer positions counted.

Reward models

Question: preferences are comparisons, but RL needs a number per response. Where does the number come from? Train a copy of the model with a scalar head to predict which of two answers a human preferred. That network is the reward model $r_\phi(x, y)$. It is a learned, imperfect stand-in for human judgement, and "imperfect" is the source of half the trouble in the flagship chapter.

KL to the reference

Question: if we optimise the model hard against a reward, what stops it from finding weird outputs the reward model loves but humans hate? Nothing, unless we add a leash. The leash is a penalty on how far the policy has drifted from a frozen copy, measured by KL divergence. In words: for each token, how much more probable does the new model make its own choices than the old model did, averaged over the new model's choices:

It is zero when the two models agree and grows as the policy moves away. Adding $-\beta\,\mathrm{KL}$ to the reward means "score well, but do not wander".

On-policy versus off-policy data

Question: does it matter whether the responses in your preference pairs were written by this model or by some other model? It matters a lot. On-policy data is sampled from the model you are currently training, so the feedback is about mistakes it actually makes. Off-policy data (pairs from a different model, or from an earlier checkpoint) is cheaper to reuse but describes someone else's mistakes. RLHF with PPO is on-policy by construction; plain DPO on a downloaded dataset is off-policy; most strong recipes now regenerate pairs from the current model each round (Lambert et al. 2024).

Rejection sampling

Question: is there a simpler way to use a reward model than running RL? Yes. Sample $N$ answers per prompt, keep the highest scoring one, and run ordinary SFT on the keepers. This is rejection sampling fine-tuning, used heavily in Llama 2 (Touvron et al. 2023) and DeepSeek-R1. It is a gentle, stable first step toward RL, and it makes a good bridge between chapters 2 and 3.

Iterative rounds

Question: once the model improves, is the old preference data still relevant? Less and less, because the new model makes different mistakes. So labs run rounds: train, sample from the new model, collect fresh comparisons, train again. Llama 2 reported five such rounds. Post-training is a loop.

The alignment tax and the capability–safety tension

A fair question at this point: if we keep pushing the model toward what humans prefer, do we lose anything?

Sometimes, yes. Fine-tuning on a narrow distribution of chatty demonstrations can degrade performance on benchmarks the base model was good at; InstructGPT (Ouyang et al. 2022) observed such regressions and mitigated them by mixing pre-training gradients back in during RL. This cost is called the alignment tax (the term is from Askell et al. 2021). Later recipes reduced it mostly through better data mixtures: keep math, code and knowledge-heavy prompts in the post-training set, and the model keeps those skills.

A related tension sits between helpfulness and harmlessness. A model trained hard on refusals becomes cautious in ways that annoy everyone ("I cannot help with killing a Python process"). A model trained only on helpfulness will help with anything. Bai et al. (2022) framed this explicitly as a trade-off and showed that better data and better reward modelling move the whole frontier outward rather than merely sliding along it; over-refusal test sets such as XSTest (Röttger et al. 2023) exist precisely to measure the other side. The honest summary is that the tension is real, it is mostly a data-curation problem rather than an algorithm problem, and it is measured, not assumed.

Five recipes side by side

Post-training has only existed at scale since 2022, and the recipe has changed every year. Comparing five landmark releases shows both the constant skeleton and what moved. Numbers are approximate and taken from the cited reports.

Three trends stand out. Human labels are being replaced by model labels and verifiers wherever quality allows. Off-policy datasets are being replaced by on-policy sampling. And the RL stage, once a fragile add-on, has become the main event for reasoning. The rest of this part explains the machinery behind each.

Practice

Further reading

• Ouyang et al. (2022). Training language models to follow instructions with human feedback. The InstructGPT paper; the three-stage recipe everyone copied.
• Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. Section 3 is the most detailed public account of an iterative RLHF pipeline.
• Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment. How far you can get with no human labels and DPO.
• Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. The fully open recipe we study in the case-study chapter.
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. RL with verifiable rewards as the main event.
• Rafailov et al. (2023). Direct Preference Optimization. The paper that removed the reward model from the loop.
• Askell et al. (2021). A General Language Assistant as a Laboratory for Alignment. Source of the "alignment tax" framing.
• Bai et al. (2022). Training a Helpful and Harmless Assistant with RLHF. The helpfulness–harmlessness trade-off, measured.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. How judged evaluation works and where it is biased.
• Dubois et al. (2024). Length-Controlled AlpacaEval. Why judge benchmarks need a length correction.
• Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Crowd-sourced pairwise human evaluation at scale.