Preference Optimization

Supervised fine-tuning has a quiet flaw. It shows the model one good answer per prompt and says "be like this". But for almost any prompt there are many good answers and many more bad ones, and what we really want the model to learn is the direction from bad to good. A single demonstration does not contain that direction. A comparison does.

That observation is the whole chapter. If you can get a human (or a model) to look at two answers and say which is better, you have a signal that is cheaper to collect than writing demonstrations, captures taste that is hard to write down, and can be turned into a training objective in several ways. This chapter is about those ways.

The problem: one demonstration is not a direction

Imagine you want the model to write better explanations of recursion. You could pay an expert to write one excellent explanation per prompt and SFT on it. Three things go wrong.

First, cost. A good demonstration takes an expert several minutes. A comparison of two existing answers takes a non-expert thirty seconds.

Second, the model is not the expert. If the demonstrations are far above what the model can currently produce, SFT teaches it to imitate surface features of expert answers (confident tone, technical vocabulary) without the substance. We saw in the SFT chapter that this is one route to hallucination.

Third, taste does not fit in one example. "Not too long", "answer the question that was asked", "do not hedge everything" are properties you recognise when you see two answers side by side, and cannot fully specify by writing one.

Why comparisons are cheaper than demonstrations

There is a deeper reason comparisons work, from decision theory: humans are much more consistent at ranking than at scoring. Ask ten people to rate an essay from 1 to 10 and you get a spread of four points. Ask them which of two essays is better and they mostly agree. So preference data has lower noise per unit of annotator time, which is why every recipe since InstructGPT (Ouyang et al. 2022) collects it.

Collecting preferences

Before optimising anything we need the pairs. Where do they come from?

Formats: pairwise, ranking, Likert

Pairwise is the workhorse: show prompt $x$ with responses $y_A$ and $y_B$, ask which is better (optionally with a "tie" or "both bad" option). The record is $(x, y_w, y_l)$, winner and loser.

Rankings show $K$ responses and ask for a full order. InstructGPT had labellers rank 4 to 9 responses per prompt, which yields $\binom{K}{2}$ pairs from one annotation session; the pairs are correlated, so they are treated as one batch during training.

Likert or absolute scores (1 to 5) are used when you want unpaired data, for example to train a model such as KTO that learns from "good" and "bad" labels separately. Scores are noisier, but any two scored answers can be turned into a pair after the fact.

Annotator guidelines

The pairs encode whatever the annotators believed "better" means, so the guideline document is the specification of the model. Typical guidelines rank criteria: correctness first, then following the instruction, then safety, then style. Llama 2 (Touvron et al. 2023) additionally asked annotators to write the prompt themselves and to mark confidence in their choice, and kept a separate stream of safety-focused pairs. Inter-annotator agreement on such tasks is reported around 70 to 80 percent; the disagreement is a floor on how well any reward model can do.

LLM-as-judge and its biases

Human labels are slow. Since about 2023, most open recipes use a strong model as the judge, prompted with a rubric (Zheng et al. 2023; the UltraFeedback dataset of Cui et al. 2023 is a well-known example). This scales, but the judge has documented biases that leak straight into your preference data:

• Length bias. Judges prefer longer answers even when the extra length adds nothing. Models trained on such pairs get verbose. Length-controlled AlpacaEval (Dubois et al. 2024) exists because of this.
• Position bias. Judges prefer whichever answer is shown first (or, for some models, second). The standard fix is to judge both orders and keep only consistent verdicts.
• Self-preference. A judge rates answers in its own style higher. Using the same model as judge and generator is especially risky.
• Surface over substance. Confident tone, headings and bullet points earn points independently of correctness.

Reward models

We have pairs. RL algorithms want a number per response: "how good is this?" So the first thing InstructGPT did, and the thing every PPO-based pipeline still does, is train a reward model (RM) that turns a prompt and a response into a scalar.

The Bradley–Terry model

The question is how to connect a scalar score to a pairwise verdict. The Bradley–Terry model (1952) gives the standard answer: assume each response has a hidden strength $r$, and the probability that $y_w$ beats $y_l$ is the sigmoid of the difference in strengths.

What just happened: the model only ever sees differences of rewards, so the absolute scale is free (adding a constant to every reward changes nothing) and a gap of 0 means a coin flip. This is the same model behind chess Elo ratings.

The reward-model loss

Given a dataset of pairs, we want the RM's parameters $\phi$ to make the observed verdicts likely. Maximum likelihood turns that into minimising the negative log of the Bradley–Terry probability:

In words: push the winner's score above the loser's, with a penalty that shrinks as the gap grows. It is logistic regression where the "feature" is the difference of two network outputs.

Training details that matter

Architecture: take the SFT model, remove the vocabulary head, and add a linear layer that reads the final hidden state at the last token and emits one number. Everything is initialised from the SFT weights, so the RM starts out already understanding language and the chat format.

Training: one epoch, often less, with a small learning rate (about $10^{-5}$ or lower). RMs overfit fast because a pair is a weak label; a second epoch usually hurts held-out accuracy. Both responses of a pair go through the network in the same batch so their scores share the same parameter state.

Evaluation: accuracy on held-out pairs. Reported numbers are typically 65 to 75 percent, which sounds low until you remember that humans agree with each other at roughly the same rate. Some recipes add a small penalty on the absolute magnitude of rewards to keep the scale from drifting, and Llama 2 added a margin term so that pairs annotators were confident about get a larger gap.

Reward hacking and over-optimisation

Now the catch. The RM is a proxy for human judgement, trained on a few hundred thousand pairs. The policy, during RL, will explore regions of response space that no annotator ever saw. In those regions the RM extrapolates, and it extrapolates badly in ways the policy will find and exploit. This is reward hacking: the proxy score keeps rising while the true quality falls.

Gao et al. (2022) measured this carefully with a synthetic setup: a large "gold" RM stands in for the human, smaller proxy RMs are trained on its labels, and the policy is optimised against the proxy. The gold score rises, peaks, then declines as the policy moves further from its starting point, while the proxy score rises monotonically. The distance is measured as $d = \sqrt{\mathrm{KL}(\pi \,\|\, \pi_{ref})}$, and they fitted the gold reward of RL-optimised policies with:

Reading it: reward grows linearly at first (the $\alpha d$ term), then the $\log d$ term wins and pulls it down. Larger proxy RMs have larger $\alpha$ and smaller $\beta$, so they can be pushed further before breaking. More RM training data helps the same way. This is why every practical recipe stops RL early, keeps $\mathrm{KL}$ small, or retrains the RM on fresh on-policy pairs.

RLHF with PPO, in one section

With a reward model in hand, the original recipe optimises the policy with reinforcement learning. The objective says: earn reward, but pay a price proportional to how far you drift from the SFT model.

What it does: the expectation is over the policy's own samples (on-policy by construction), the reward comes from the RM, and $\beta$ sets the price of drift. In practice the KL is estimated per token as $\log \pi_\theta(y_t) - \log \pi_{ref}(y_t)$ and subtracted from the reward at each step.

PPO (Schulman et al. 2017) is the algorithm of choice because of Step 3: the clipped surrogate. In words, it computes how much more likely the new policy makes each sampled token than the policy that generated it, multiplies by the advantage, and refuses to reward ratio changes beyond a small window (typically ±20 percent). This keeps updates conservative without a hard constraint.

The costs are real: four networks live in memory (policy, reference, reward, value), generation dominates wall-clock time, and there are a dozen hyperparameters that interact. The RLHF deep dive walks through every one of them with code. Here we only need to know what PPO is doing, so we can see what DPO removes.

Direct Preference Optimization

Here is the question that led to DPO (Rafailov et al. 2023). The RM is trained on pairs. The policy is trained against the RM. Both steps are just fitting the same pairs. Could we skip the middle and fit the policy to the pairs directly?

The answer is yes, and the derivation is short enough to do on a napkin. Follow along; every step is one line of algebra.

Step 1: the objective has a closed-form optimum

Start with the same KL-regularised objective PPO optimises, for a single prompt $x$ and any reward function $r$:

This is a maximisation over a probability distribution $\pi(\cdot \mid x)$, and it has a known solution (it is the same calculation that gives the Boltzmann distribution in physics). The optimal policy reweights the reference by the exponentiated reward:

What this says: the best policy is the reference, with each response boosted by $e^{r/\beta}$ and then renormalised. High reward, big boost. Small $\beta$, bigger boosts. $Z(x)$ is the normaliser, a sum over every possible response, which is why we cannot use this formula directly: $Z$ is intractable.

Step 2: solve for the reward instead

Here is the trick. Instead of solving for the policy given a reward, rearrange the same equation to express the reward in terms of the policy. Take logs and multiply by $\beta$:

Now any policy $\pi_\theta$ can be read as the optimal policy for some reward, namely $\beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$ plus a per-prompt constant. That quantity is called the implicit reward. It is high for responses the policy has made more likely than the reference did, and low for ones it has suppressed.

Step 3: the partition function cancels

Plug this reward into Bradley–Terry. The preference probability depends on the difference of two rewards for the same prompt, and $\beta \log Z(x)$ is the same for both responses, so it cancels:

That cancellation is the entire reason DPO works. The intractable sum over all responses disappears because we only ever compare two responses to the same prompt.

Step 4: the DPO loss

The rest is maximum likelihood exactly as for the reward model. Define the log-ratio of each response, $\Delta = \log \pi_\theta(y \mid x) - \log \pi_{ref}(y \mid x)$, and minimise:

In words: increase the log-probability of the chosen response relative to the reference, decrease the rejected one relative to the reference, and care about their gap through a sigmoid so that pairs already ordered correctly stop contributing. No reward model, no sampling, no value network. It is a classification loss on log-probabilities you can compute with two forward passes.

A fully worked numeric example

What β means

In the objective, $\beta$ is the price of KL. In the loss, it is the temperature that converts a log-ratio into a reward. Both readings agree: small $\beta$ means drift is cheap, so a given margin counts for little, so the optimiser must move the log-ratios a lot to satisfy the loss, so the policy ends far from the reference. Large $\beta$ means a small log-ratio movement already satisfies the loss, and the policy stays close. Typical values are 0.01 to 0.5, with 0.1 the common default.

The gradient, in words

Differentiate the loss and something familiar appears:

Read it left to right. The bracket at the end says: push up the chosen response's log-probability and push down the rejected one's, exactly like SFT on the winner combined with "anti-SFT" on the loser. The weight in front is a sigmoid of the negative margin: it is near 1 when the implicit reward currently ranks the pair backwards, and near 0 when the pair is already well separated. So DPO spends its gradient on the pairs it gets wrong and leaves the rest alone. That weighting is what stops it from being plain SFT on winners, and it is the same "weight by how wrong you are" we saw in the reward-model gradient.

Practical issues with DPO

DPO is simple to run and that simplicity hides three failure modes you will meet in the first week.

Both log-probabilities fall. Look at the gradient again: it only constrains the difference $\Delta_w - \Delta_l$. The easiest way to grow that difference is often to push $\log \pi(y_l)$ down a lot while $\log \pi(y_w)$ also drifts down a little. In practice you will frequently see the chosen response's log-probability decrease during training. The model is not becoming more likely to produce the good answer; it is becoming much less likely to produce the bad one, and the freed probability mass goes somewhere else, which may be worse than either.

Likelihood displacement. Razin et al. (2024) named and analysed where that mass goes: when chosen and rejected responses are similar (share many tokens, differ in a phrase), pushing one down drags the other down too, and probability shifts to responses that look like neither. In the worst case a model trained to prefer refusals over compliance ends up complying more. The practical fixes are to filter out pairs that are too similar, add a small SFT term on the chosen response (as RPO does), or use on-policy pairs where chosen and rejected are naturally different.

Length bias. DPO sums log-probabilities over tokens. A longer response has a more negative log-probability, but also more tokens through which $\Delta$ can grow. Combined with judges that prefer long answers, DPO models get verbose. Length-normalising the log-ratio (dividing by token count) is a common fix, used in SimPO and in Tülu 3's "length-normalised DPO" (Lambert et al. 2024).

Off-policy, on-policy, and iterative DPO

The derivation assumed nothing about where the pairs came from. In practice it matters a great deal.

Off-policy DPO trains on pairs whose responses were generated by other models (a downloaded preference dataset). Cheap, and it is how Zephyr (Tunstall et al. 2023) was trained. The weakness: the responses may be far from anything $\pi_\theta$ would say, so the implicit reward is being fitted on examples the policy never produces and the lesson transfers imperfectly. Several careful comparisons (Xu et al. 2024; Tang et al. 2024) found that PPO's on-policy sampling is a large part of why it beats off-policy DPO when it does.

On-policy DPO generates the responses from the current SFT model, has a judge label them, and then runs DPO. Now the pairs describe the model's own mistakes. Tülu 3 found on-policy pairs to be the single most important ingredient in its preference stage.

Iterative (online) DPO repeats that loop: sample from the current policy, judge, DPO, update the reference to the new policy or keep it fixed, sample again. Self-Rewarding Language Models (Yuan et al. 2024) went as far as using the policy itself as the judge. Each round is a step toward what PPO does continuously, at a fraction of the engineering.

The DPO family

DPO's simplicity invited variants. Each addresses one of the issues above. Read the table as a menu, not a ranking; in most reported comparisons the differences are smaller than the effect of data quality.

GRPO and RL with verifiable rewards

Everything so far used preferences: human or judge opinions. For some tasks there is a better signal. A math problem has an answer key; a coding problem has unit tests; an instruction such as "reply in exactly three sentences" can be checked by a regex. These are verifiable rewards: a program returns 1 if the response is correct and 0 otherwise. No annotator, no reward model, no reward hacking of the usual kind (the checker cannot be flattered, though it can be gamed if it is sloppy).

The catch is that a binary reward on a whole response is a sparse, high-variance signal, and PPO's value model struggles to estimate baselines for it. Group Relative Policy Optimization (GRPO; Shao et al. 2024, DeepSeekMath) answers this with a simple idea: sample a group of $G$ responses for the same prompt and use the group's own statistics as the baseline.

What this does: each sample's advantage is how much better it did than its siblings, in units of the group's spread. If 2 of 8 samples are correct, those two get a large positive advantage and the six failures get a moderate negative one. If all 8 are correct or all 8 are wrong, every advantage is zero and the prompt teaches nothing this round, which is exactly right: there is no contrast to learn from.

The advantage is then applied to every token of the response through a PPO-style clipped update, with a KL term to the reference. No value network, so memory drops to three models (policy, reference, and nothing else if the reward is a program). This is the algorithm behind DeepSeek-R1 (DeepSeek-AI 2025), where the reported effect of running it long enough on math and code was the spontaneous emergence of long, self-checking chains of thought.

Why verifiable RL fits math and code

Three reasons, all about the reward. It is exact: no annotator noise, no judge bias, no reward-model extrapolation. It is cheap: once you have the answer key, every sample is scored for free, so you can afford 8 or 64 samples per prompt. And it is hard to hack: a unit test does not care about tone, length, or confidence. The limitation is the mirror image: for open-ended writing, advice, or dialogue there is no checker, and preference methods remain the only option. The RLHF deep dive covers GRPO's full update rule and its training dynamics; code/lumen/rl.py has a small implementation of the advantage computation.

Choosing a method

You have a base or SFT model and a budget. Which stage do you run? The honest answer depends on what data you can get and what you can verify.

Most strong open recipes as of 2025 run all three in sequence: SFT for format, DPO for taste, verifiable RL for reasoning, each on data the previous stage's model generated. The Tülu 3 case study shows what that looks like end to end.

Implementation: DPO in PyTorch

The whole method fits in two functions. The first computes a sequence log-probability with the prompt masked out, exactly as in SFT loss masking. The second is the loss. This is the core of code/lumen/dpo.py.

import torch
import torch.nn.functional as F

def sequence_logprob(model, input_ids, attention_mask, labels):
    """Sum of log p(token) over positions where labels != -100 (the response)."""
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits
    logits = logits[:, :-1, :].float()          # predict token t+1 from position t
    targets = labels[:, 1:]
    mask = (targets != -100)
    logp = torch.log_softmax(logits, dim=-1)
    tok_logp = logp.gather(-1, targets.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (tok_logp * mask).sum(-1)            # shape: (batch,)

def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):
    """pi_*: policy log-probs, ref_*: reference log-probs (precomputed). All (batch,)."""
    delta_w = pi_w - ref_w
    delta_l = pi_l - ref_l
    margin = beta * (delta_w - delta_l)
    loss = -F.logsigmoid(margin).mean()
    with torch.no_grad():
        reward_acc = (margin > 0).float().mean()   # fraction of pairs ranked correctly
        chosen_reward = (beta * delta_w).mean()      # watch this: it often goes negative
    return loss, reward_acc, chosen_reward

# training step (reference log-probs were computed once and stored with the batch)
pi_w = sequence_logprob(policy, batch["w_ids"], batch["w_mask"], batch["w_labels"])
pi_l = sequence_logprob(policy, batch["l_ids"], batch["l_mask"], batch["l_labels"])
loss, acc, r_w = dpo_loss(pi_w, pi_l, batch["ref_w"], batch["ref_l"], beta=0.1)
loss.backward()

Three things to notice in the output. The loss starts at $\ln 2$, as promised. Reward accuracy (the fraction of pairs whose margin is positive) climbs and is the most useful training-time metric. And the chosen reward, $\beta\Delta_w$, goes slightly negative: the chosen responses got less likely than under the reference even as the model learned to prefer them. That is the "both fall" behaviour in the wild.

For GRPO, the advantage computation from the interactive is four lines and lives in code/lumen/rl.py:

def group_advantages(rewards, eps=1e-4):
    """rewards: (num_prompts, G). Returns z-scored advantages within each group."""
    mean = rewards.mean(dim=1, keepdim=True)
    std = rewards.std(dim=1, keepdim=True, unbiased=False)
    return (rewards - mean) / (std + eps)

Practice

Further reading

• Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The derivation in full, plus the connection to the implicit reward.
• Ouyang et al. (2022). Training language models to follow instructions with human feedback. Reward model training details and the PPO recipe.
• Gao, Schulman and Hilton (2022). Scaling Laws for Reward Model Overoptimization. The proxy-versus-gold experiments behind the over-optimisation interactive.
• Schulman et al. (2017). Proximal Policy Optimization Algorithms. The clipped surrogate objective.
• Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Introduces GRPO.
• DeepSeek-AI (2025). DeepSeek-R1. Verifiable-reward RL at scale and the emergence of long reasoning chains.
• Razin et al. (2024). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. Why both log-probabilities can fall and where the mass goes.
• Xu et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. A careful comparison focusing on on-policy data.
• Tang et al. (2024). Understanding the performance gap between online and offline alignment algorithms. Evidence that on-policy sampling, not the loss, explains most of the gap.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Documents position, verbosity and self-enhancement biases in LLM judges.
• Yuan et al. (2024). Self-Rewarding Language Models. Iterative DPO where the policy is also the judge.
• Lambert et al. (2024). Tülu 3. On-policy preference data and length-normalised DPO in a fully open recipe.