Case Study: Tülu 3

A guided reading of the first fully open post-training recipe that matched closed instruct models. By the end you will know exactly what data went into each stage, which choices actually moved the numbers, and how to shrink the recipe to a single GPU.

For two years after ChatGPT, the awkward truth of open post-training was this: anyone could download a strong base model, but nobody outside the big labs could turn it into a strong assistant. The data, the hyperparameters, and the evaluation setups that made the difference were the parts that stayed private.

Tülu 3 (Lambert et al., 2024, arXiv 2411.15124) is the paper that closed most of that gap. The team at Ai2 took Llama 3.1 base models, published every prompt, every training script, and every evaluation, and ended up with models that beat the official Llama 3.1 Instruct releases on their own suite.

This chapter is not a summary of the abstract. It is a walk through the recipe, stage by stage, with the reported numbers where we could verify them and honest flags where we could not. Everything in this chapter marked as reported comes from the paper; where we say roughly, the exact figure depends on the paper version or was not clearly stated.

Read this first

This chapter assumes the previous four. You should know what SFT is (Supervised Fine-Tuning), how DPO works (Preference Optimization), and what a verifier is (Tools and Safety Tuning). If you want to see the same "read the paper carefully" treatment for pre-training, see Case Study: Llama 3.

What problem was the paper solving?

Post-training is where a text continuer becomes an assistant, and by 2024 everyone knew the ingredients: instruction data, preference data, a bit of reinforcement learning. The problem was that open recipes were built from scraps. Tülu 2 (Ivison et al., 2023, arXiv 2311.10702) and Zephyr used public datasets like UltraFeedback (Cui et al., 2023, arXiv 2310.01377), and they lagged Llama-2-Chat and later Llama 3 Instruct by a wide margin on hard skills like math and precise instruction following.

The paper states its goal plainly: a fully open, reproducible recipe on top of Llama 3.1 that reaches the quality of the closed post-training pipelines. "Fully open" means the models, the data, the mixing code, the training code, and the evaluation framework are all released.

The three things that made it work, in the authors' own framing, were careful data curation with decontamination, on-policy preference data, and a new final stage they call RLVR, reinforcement learning with verifiable rewards.

Where it came from

Lambert et al. (2024), "Tülu 3: Pushing Frontiers in Open Language Model Post-Training". First posted November 2024, revised through 2025 with 405B results. It mattered because it was the first open recipe that a small team could run end to end and get an instruct model competitive with the official one.

The recipe at a glance

Before the details, hold the whole pipeline in your head. It is the same four-box picture from the overview chapter, but every box now has a number attached.

Llama 3.1 base 8B / 70B SFT ~939k prompts DPO ~300k pairs RLVR math + IFEval learn format + skills learn taste (on-policy) learn what a checker can verify text continuer 8B average on the dev suite, as reported: 60.1 → 64.7 → 65.1
Figure 1. The Tülu 3 pipeline with rough data sizes. Look at where the average score jumps: the big move is DPO, and RLVR adds a small, targeted lift on the tasks it can verify.
Symbols
$\pi_\theta$ = policy being trained
$\pi_{ref}$ = frozen reference (the SFT model)
$y_w, y_l$ = chosen / rejected response
$v(x,y)$ = verifier, 1 if correct else 0
STEP 1
SFT
Next-token loss on roughly 939k curated (prompt, response) pairs, response tokens only.
STEP 2
DPO
Length-normalized DPO on preference pairs where one side was sampled from the SFT model itself.
STEP 3
RLVR
PPO on math and instruction-following prompts; reward only when $v(x,y)=1$.
STEP 4
Evaluate
A development suite for iteration, plus an unseen suite the team never tuned on.
InteractiveRecipe stepper: what each stage changedstep through the stages

Watch the reported Tülu 3 8B scores at each stage. Bars are the four evals the paper tracks most closely; the note lists the data that went in.

Two things to notice in the stepper. First, DPO is the big lever on the average: it lifts GSM8K by eight points and MATH by ten. Second, RLVR adds only a few tenths to the average but a clear three points on GSM8K, which is exactly the kind of skill a verifier can check. That is the whole thesis of RLVR in one glance.

Data curation: prompts first

Here is the question the paper asks before anything else: what should the model practice on? A post-training set is a collection of prompts before it is a collection of answers. If the prompts do not cover the skills you care about, no amount of clever optimization will produce them.

Where the prompts came from

Tülu 3 organizes prompts by core skill: knowledge recall, reasoning, math, coding, precise instruction following, general chat, and safety. For each skill the team gathered public sources and, where public data was thin, generated synthetic prompts.

The public sources include real user conversations from WildChat (Zhao et al., 2024, arXiv 2405.01470), the older FLAN v2 task collection, OpenAssistant, and code and math sets like Evol CodeAlpaca, NuminaMath-TIR and OpenMathInstruct 2. As reported, the WildChat subset alone is about 100k conversations.

Persona-driven synthetic data

The synthetic part is the interesting bit. If you ask one model to write ten thousand math problems, you get ten thousand very similar math problems. The fix the paper borrows from Persona Hub (Ge et al., 2024, arXiv 2406.20094) is to condition each generation on a persona: "a marine biologist who tracks whale migration", "a bakery owner planning a budget". Roughly 250k personas seed prompts for math, code and precise instruction following, and the diversity of personas pulls diversity into the prompts.

Precise instruction following deserves its own mention. This is the skill measured by IFEval (Zhou et al., 2023, arXiv 2311.07911): "answer in exactly three bullet points", "do not use the letter e". The team wrote synthetic prompts with such constraints, with verifiers attached. Keep that phrase in mind, because those verifiers come back in RLVR.

Decontamination: the boring step that matters most

Now the problem nobody likes to talk about. Public instruction datasets are scraped from everywhere, and "everywhere" includes the test sets you will later evaluate on. If your SFT mix contains GSM8K test questions, your GSM8K score is a lie.

The paper's method, as reported, is 8-gram matching: for each test instance, check what fraction of its tokens appear inside 8-gram matches with a single training instance. If more than half of the test tokens match, that training instance is flagged. Any dataset with more than about 2% of a benchmark contaminated was either dropped or filtered at the instance level, and the team found real contamination in several popular sources, including Evol CodeAlpaca, NuminaMath-TIR and a WildChat safety subset.

eval test sete.g. GSM8K, 1.3k items candidate SFT sete.g. 60k rows 8-gram overlap>50% of test tokens hit? >2% contaminateddrop set or filter rows cleankeep in the mix
Figure 2. The decontamination gate, as described in the paper. Every candidate dataset is compared against every evaluation before it is allowed into the mix. Look at the threshold: it is per benchmark, not per dataset overall.
Common confusion

Decontamination is not about exact duplicates. A paraphrased test question with the same numbers still leaks the answer pattern, and 8-gram matching catches many of those while exact-match hashing catches almost none. Also note the direction: you filter the training data against the test sets, never the other way round.

The SFT mixture and how it was chosen

With clean prompts in hand, the next question is how to combine them. Mix too much math and chat quality drops; mix too much chat and math stalls. The paper's answer is unglamorous: run ablations, one component at a time, and read the evaluation suite.

The final mix

The final SFT set is roughly 939k prompts, as reported. The largest single block is the persona-driven synthetic data (on the order of a few hundred thousand instances, mostly math), followed by WildChat, Evol CodeAlpaca, FLAN v2 and NuminaMath-TIR. Responses for synthetic prompts were written by a strong model, and the team regenerated some low-quality responses from older datasets so that formatting was consistent across the mix.

Training details, as reported: full fine-tuning of Llama 3.1, response-only loss masking as in the SFT chapter, a couple of epochs, a learning rate in the low $10^{-6}$ to $10^{-5}$ range, and long sequence lengths to fit multi-turn conversations. The exact hyperparameters live in the paper's appendix and changed between versions, so we will not quote them as fact.

Reading the ablations

The paper removes one component from the mix at a time and reports the full evaluation. The pattern is what you would hope for: removing math data hurts GSM8K and MATH the most, removing code data hurts HumanEval, removing safety data hurts the safety average, and removing WildChat, the real-user chat data, lowers the average more than you would expect for its size. As reported, dropping WildChat moved the 8B SFT average from 60.1 to 58.9.

InteractiveAblation explorer (illustrative)pick a component to remove

Pick a data component to remove from the SFT mix and see which evals drop. The shape follows the paper's ablation tables; the exact bar heights are illustrative, except the WildChat average which is as reported.

The lesson is not the specific numbers. It is that every component in the final mix earned its place through a measured drop when removed, and the team published those tables. When you build your own mix, that is the discipline to copy.

Preference data and length-normalized DPO

SFT shows the model one good answer per prompt. It cannot say "this answer is better than that one". For taste, tone, and the difference between a correct answer and a correct, well-organized answer, you need preferences. The question is where the pairs come from.

On-policy pairs

Older open recipes reused UltraFeedback, a fixed set of pairs generated by other models. The catch is that those responses look nothing like what your SFT model writes, so the preference signal is off-distribution. It teaches the model about mistakes it was never going to make.

Tülu 3 instead samples one or both responses from its own SFT checkpoint, mixes in responses from a pool of other open and closed models, and asks an LLM judge (GPT-4o, as reported, which came out slightly ahead of alternatives in their comparison) to pick the better one along several axes: helpfulness, instruction following, honesty, truthfulness. The winner becomes $y_w$, the loser $y_l$.

prompt xunused in SFT Tülu 3 SFTon-policy sample other modelsoff-policy sample LLM judgeGPT-4o, 4 axes (y_w, y_l)pair
Figure 3. How a Tülu 3 preference pair is made. The point to look at is the top branch: at least one response usually comes from the model being trained, so the pair is on-policy. The paper reports that this beats purely off-policy pairs and that using prompts not already in the SFT mix helps further.

Two more findings, as reported. Using new prompts for preference data (not the ones already in the SFT mix) beat reusing SFT prompts. And scaling the number of unique prompts kept improving downstream DPO results, which is a strong hint that diversity, not volume of pairs per prompt, is the thing to buy.

Length-normalized DPO

Plain DPO (Rafailov et al., 2023, arXiv 2305.18290) has a known bias: the implicit reward is a sum over tokens, so longer responses can rack up a bigger margin just by being longer. Tülu 3 uses a length-normalized variant throughout, which divides each log-probability by the number of tokens before taking the difference. In symbols, with $|y|$ the token count of a response, the normalized log-ratio for one response is

$$r_{norm}(x,y) = \frac{\beta}{|y|}\left[\log \pi_\theta(y\mid x) - \log \pi_{ref}(y\mid x)\right]$$

and the loss is the usual $-\log\sigma\big(r_{norm}(x,y_w) - r_{norm}(x,y_l)\big)$. What just happened: every response is scored by its average per-token log-ratio, so a 400-token answer and a 40-token answer compete on equal footing. This is the same idea SimPO (Meng et al., 2024, arXiv 2405.14734) uses, except Tülu 3 keeps the reference model.

Worked example

Suppose the chosen response has 4 tokens with total log-ratio $\log\pi_\theta - \log\pi_{ref} = 0.8$, and the rejected response has 10 tokens with total log-ratio $1.2$. Plain DPO with $\beta = 0.1$ sees a margin of $0.1(0.8 - 1.2) = -0.04$: the rejected response wins, purely because it is longer. Length-normalized DPO sees $0.1(0.8/4 - 1.2/10) = 0.1(0.2 - 0.12) = 0.008$: the chosen response wins. Same model, same pair, opposite gradient sign.

Training details, as reported: the DPO stage starts from the SFT checkpoint as both policy and reference, runs for roughly one epoch, and uses a small learning rate with $\beta$ in the usual $0.1$ region. The paper preferred DPO over PPO with a learned reward model at this stage mainly for simplicity and speed of experimentation, and it says so.

RLVR: reinforcement learning with verifiable rewards

Now the new part. Preferences from an LLM judge are noisy and, for math, often wrong. But for some prompts you do not need a judge at all. A GSM8K problem has a numeric answer. An IFEval prompt has a constraint you can check with a regular expression. Why train on a judge's opinion when you could train on the truth?

RLVR replaces the learned reward model with a verifier function. The reward for a sampled response is a constant if the verifier passes and zero otherwise:

$$R(x, y) = \begin{cases} \alpha & \text{if } v(x,y) = 1 \\ 0 & \text{otherwise} \end{cases}$$

Here $v$ is the checker (extract the final number and compare; run the constraint check) and $\alpha$ is a fixed positive constant (the paper uses a value around 10, as reported). That reward is plugged into a standard PPO loop (Schulman et al., 2017, arXiv 1707.06347) with a KL penalty to the DPO checkpoint, exactly the machinery of the RLHF deep dive, except the reward model is a Python function.

Intuition

RLVR is rejection sampling with gradient updates. The model samples several answers, the verifier says which ones were right, and the policy shifts probability toward the right ones. Because the verifier cannot be fooled by confident tone or length, the usual reward-hacking failure modes of a learned reward model mostly disappear.

InteractiveThe RLVR loop, one step at a timestep through the loop

Follow one prompt through a single RLVR iteration. The active stage is highlighted and the note shows what data is flowing.

What RLVR was run on

As reported, three sources: the GSM8K training set (Cobbe et al., 2021, arXiv 2110.14168), the MATH training set (Hendrycks et al., 2021, arXiv 2103.03874), and synthetic prompts with IFEval-style verifiable constraints. Together this is on the order of tens of thousands of prompts, tiny next to the SFT mix. The verifier for math is answer matching; for constraints it is the same checker code that IFEval uses.

The paper reports two implementation choices that helped. Initializing the PPO value model from a general reward model, rather than from the policy, gave better results. And the training loop was asynchronous: a vLLM server generates samples while the learner does gradient steps, which keeps the GPUs busy.

The effect on the 8B model, as reported and visible in the stepper above, is GSM8K 84.3 → 87.6, MATH 42.0 → 43.7, IFEval 81.1 → 82.4. Everything else is roughly flat, which is the right outcome: RLVR only touches what it can verify.

Why this matters beyond Tülu

RLVR is the same recipe, scaled up, that DeepSeek-R1 (2025, arXiv 2501.12948) used to produce long chains of reasoning: verifiable rewards on math and code, no learned reward model, an RL loop on top. The preference optimization chapter covers GRPO, the algorithm R1 used in place of PPO. Tülu 3 was an early, small-scale demonstration that the idea works on an ordinary instruct model.

Evaluation: a development suite and an unseen suite

You cannot iterate on a recipe without a fast, reliable evaluation, and you cannot trust a recipe that was tuned to its own evaluation. The paper handles this with two suites and an open evaluation framework.

The development suite is what the team looked at while choosing mixtures: MMLU, PopQA, TruthfulQA, BigBench-Hard, DROP, MATH, GSM8K, HumanEval and HumanEval+, IFEval, AlpacaEval 2, and a safety average over six tasks. The averages you saw in the stepper are averages over this suite.

The unseen suite was held out until the end: AGIEval English, MMLU-Pro, GPQA, DeepMind Mathematics, BigCodeBench, an out-of-distribution IFEval variant the team built, and a new reference-guided chat evaluation called HREF. The point of reporting both is honesty: if a recipe improves the dev suite and not the unseen one, you have overfit the recipe, not the model.

Every evaluation decision (prompt format, few-shot count, answer extraction) is recorded in their released framework so that others can reproduce the exact number. That is less glamorous than a new algorithm, and probably worth more.

What moved the needle, and what did not

Papers usually only tell you what worked. Tülu 3 has a section literally titled "Insights from the Unfruitful". Here is the honest tally.

Moved the needle

  • Curated, decontaminated SFT mix. The 8B SFT model already beat Tülu 2's full pipeline, and each component earned its place in an ablation.
  • On-policy preference pairs with fresh prompts. The DPO stage gave the largest average gain of the three stages.
  • Length-normalized DPO. Reported as a consistent improvement over the plain objective in their setup.
  • RLVR on verifiable tasks. Small but reliable gains exactly where the verifier lives, with no regression elsewhere.
  • Scaling to 70B. The same recipe transferred, as reported, with the 70B model reaching an average of roughly 76.2 versus roughly 72.6 after SFT.

Did not move the needle

  • Online DPO (regenerating pairs from the current policy during training) did not reliably beat the offline on-policy setup in their experiments.
  • Rejection-sampling fine-tuning (sample many, keep the best by a judge, SFT on it) was tried and reported as not paying off relative to the effort.
  • Safety data turned out to be close to orthogonal to the other skills: including it did not hurt them, and removing it only hurt the safety average. This is a "did not move the other needles" result, and a useful one.
Common confusion

"RLVR barely changed the average, so it barely matters." Look at the per-task numbers, not the average. A three-point GSM8K gain from a few thousand prompts and a fraction of the DPO compute is a very good deal, and the average is diluted by ten other benchmarks the verifier never touched.

Lessons for a learner replicating at small scale

You are not going to fine-tune a 70B model on a laptop, and you do not need to. The recipe shrinks well if you keep the priorities straight.

  1. Start with a clean, small SFT mix. A few thousand high-quality, format-consistent examples across the skills you care about beat a hundred thousand scraped ones. Reuse the released Tülu 3 SFT mixture, subsampled, rather than assembling your own from scratch.
  2. Decontaminate before you evaluate anything. Implement the 8-gram check against every test set you plan to report. It is fifty lines of Python and it protects every number you will ever publish.
  3. Build preference pairs from your own SFT model. Sample two responses per prompt from your checkpoint, judge them with the strongest model you can afford, and train length-normalized DPO. On-policy pairs from a small model beat off-policy pairs from a big one for teaching your model.
  4. Add RLVR only where you have a verifier. GSM8K-style arithmetic and format constraints are enough. Do not bolt a learned reward model onto a small project; the verifier is the cheap, honest reward.
  5. Keep a held-out suite you never look at. Even one benchmark you refuse to tune on tells you whether you are improving the model or the number.

The companion code gives you the pieces: code/lumen/train.py for the SFT loop with response masking, code/lumen/dpo.py for the DPO loss (add the length normalization yourself, see Exercise 2), and code/lumen/rl.py for the GRPO-style advantage that you can use in place of PPO for a small RLVR run.

Practice

Exercise 1 — build the decontamination gate

Write a function contaminated(train_text, test_text, n=8) that returns the fraction of test tokens covered by n-grams that also appear in the training text. Apply it to a small instruction dataset against 200 GSM8K test problems and report the fraction of test items with more than 50% coverage. Then paraphrase one test problem by hand and check whether your gate still catches it.

Solution sketch

Tokenize both texts (whitespace is fine for the exercise). Build a set of all 8-grams from the training text. Slide an 8-gram window over the test text; mark all 8 positions covered whenever the window is in the set. Coverage is covered positions divided by test length. For the paraphrase, you will find that changing a few words breaks most 8-grams, which is why the paper's threshold is "more than 50% of tokens" rather than "any match": partial overlap still gets flagged when the numbers and structure survive.

Exercise 2 — length-normalized DPO

Starting from the DPO loss in code/lumen/dpo.py, add a flag length_normalize that divides each sequence log-probability by its response token count before forming the margin. Construct a synthetic pair where the rejected response is three times longer than the chosen one and has a higher total log-ratio. Show that plain DPO produces a negative margin and the normalized version a positive one, as in the worked example above.

Solution sketch

Compute per-token log-probs under policy and reference with the label mask from the SFT chapter, sum over response tokens to get $\log\pi(y\mid x)$, and keep the mask sum as $|y|$. The normalized log-ratio is (logp_pi - logp_ref) / mask.sum(-1). The loss is unchanged otherwise: -F.logsigmoid(beta * (r_w - r_l)). For the synthetic pair, set the chosen total log-ratio to 0.8 over 4 tokens and the rejected to 1.2 over 12 tokens; plain margin is negative, normalized margin is $0.2 - 0.1 = 0.1$ times $\beta$.

Exercise 3 — a toy RLVR loop

Take a small instruction-tuned model, 100 GSM8K training problems, and a verifier that extracts the last integer in the output. For each prompt, sample 4 answers, assign reward 1 or 0, compute group-normalized advantages with the helper in code/lumen/rl.py, and do a policy-gradient step with a KL penalty to the starting model. Track the fraction of correct samples per batch over 20 iterations.

Solution sketch

The loop is exactly the stepper above. The main engineering trap is the verifier: strip commas and units before parsing the number, and treat any unparseable output as reward 0. Expect the correct fraction to rise from something like 0.3 to 0.5 on the training prompts within a few dozen updates on a small model, and expect it to fall back if you remove the KL penalty, because the policy will collapse onto short outputs that happen to end in a common number.

Check yourself
Which stage of the Tülu 3 8B recipe produced the largest increase in the development-suite average, as reported?
The reported 8B averages are roughly 60.1 after SFT, 64.7 after DPO and 65.1 after RLVR. DPO is the biggest jump on the average; RLVR gives a targeted lift on GSM8K, MATH and IFEval.
What is the reward signal in RLVR?
RLVR replaces the learned reward model with a checker: answer matching for math, constraint checks for instruction following. Because the reward is binary and computed by code, it cannot be gamed by tone or length.
Why does Tülu 3 sample at least one response in each preference pair from its own SFT checkpoint?
Off-policy pairs from other models teach the policy about errors it was never going to make. On-policy pairs put the gradient where the model's real failure modes are, and the paper reports this beats purely off-policy data.

Key takeaways

  • Tülu 3 is a fully open SFT → length-normalized DPO → RLVR recipe on Llama 3.1 that matched the official instruct models, with all data and code released.
  • The SFT mix (roughly 939k prompts) was chosen by one-at-a-time ablations, and every dataset was decontaminated against the evaluation suite with 8-gram matching.
  • Preference pairs are on-policy (sampled from the SFT model) and use fresh prompts; this, not more pairs per prompt, is what improved DPO.
  • RLVR uses a verifier instead of a reward model and gives reliable gains exactly on the tasks it can check, with no regression elsewhere.
  • Online DPO and rejection-sampling fine-tuning were tried and reported as not worth it in this setup; safety data was found to be nearly orthogonal to other skills.
  • At small scale: clean SFT mix, decontaminate, on-policy DPO, RLVR only with a verifier, and one held-out benchmark you never tune on.

Further reading