RLHF Deep Dive

You have a language model that writes plausible text and a reward model that scores whole responses from 0 to 10. There is no token-level target anywhere. How do you turn "this whole answer scored 7.2" into gradients on 7 billion weights without wrecking the model?

This chapter assumes the vocabulary from RL Fundamentals: policy, return, advantage, the REINFORCE estimator, PPO's clipped ratio and GRPO's group baseline. Here we wire those pieces into the actual system that trained InstructGPT (Ouyang et al., 2022) and its successors, and then look hard at where it breaks.

The basic pipeline is three stages. First, supervised fine-tuning (SFT) on demonstrations. Second, train a reward model (RM) on human comparisons. Third, optimize the SFT model against the RM with RL while a KL penalty keeps it from drifting too far. The earlier post-training chapters covered the shape of this; now we open the box on stage three, and on the reward model that makes stage three possible or impossible.

The four models in memory

The first shock for anyone who has only done SFT is the hardware bill. SFT needs one model. PPO-style RLHF needs four copies of roughly model-sized networks sitting in memory at once.

• Policy $\pi_\theta$: the model being trained. Needs weights, gradients and optimizer state.
• Reference $\pi_{\text{ref}}$: a frozen copy of the SFT model, used only to compute the KL penalty. Weights only, no gradients.
• Reward model $r_\phi$: frozen, scores each finished response with a scalar. Weights only.
• Critic (value model) $V_\psi$: predicts expected future reward from every prefix; trained alongside the policy, so it needs gradients and optimizer state too.

Let us put numbers on it for a 7B-parameter model, using the standard mixed-precision accounting from the training-infrastructure material: fp16 weights (2 bytes), fp16 gradients (2), fp32 master weights (4) and two fp32 Adam moments (8) give about 16 bytes per trained parameter; a frozen fp16 copy costs 2.

From one scalar to a per-token signal

The reward per response

The reward model reads the prompt and the complete response and emits one number, $R = r_\phi(x, y)$. It only makes sense on finished text: half a sentence has no meaningful score. So in RL terms, the environment is silent for every token and speaks once, at the end.

That is a sparse reward, and the RL Fundamentals chapter explained why sparse rewards make credit assignment hard: which of the 200 tokens earned the 7.2? With a critic and GAE we can spread it out, but first we need to fix a more basic problem.

Why a KL penalty at all?

Here is the catch with optimizing a learned reward. The reward model was trained on responses that look like SFT outputs. If the policy wanders into text the RM has never seen, the RM's scores there are extrapolation, and a gradient-following optimizer will find exactly the regions where the extrapolation is wrong in its favor. The policy learns to exploit the RM rather than to be helpful.

The fix is to penalize the policy for moving away from the reference model. We measure the movement with the KL divergence between the policy and the reference, and we subtract it from the reward. This keeps the policy in the region where the RM is trustworthy, and it also keeps the model's language from collapsing.

The per-token KL penalty

We want a reward at every token, not just the last one, so the KL term is applied per token. For token $t$ the penalty is proportional to how much more likely the policy made that token than the reference did. Then the response-level reward is added only at the final token.

What just happened: every token now has a small reward that is negative when the policy is being more confident than the reference and positive when it is being less confident, and the last token additionally carries the RM's score. The coefficient $\beta$ sets the leash length. Summed over the response, the log-ratio terms are a single-sample estimate of the sequence-level KL, which is why this is usually called "the KL penalty" even though it is really a per-token log-ratio.

The PPO-RLHF loop

With per-token rewards in hand, one iteration of training looks like ordinary PPO. The difference from a game-playing agent is that "the environment" is just the model itself generating text, and that most of the wall-clock time goes into sampling, not into gradient updates.

What the update actually minimizes

The policy loss is the clipped objective from the RL chapter, averaged over every response token in the batch. The critic is trained to regress onto the GAE returns, usually with its own clipping. Written as one loss to minimize:

What just happened: for each token, the ratio $\rho_t$ measures how much this epoch's policy has already moved from the policy that generated the sample; the clip stops it from moving more than $\epsilon$ in the direction the advantage points; and the value term keeps the critic honest. In the first epoch $\rho_t = 1$ everywhere and the loss reduces to plain advantage-weighted REINFORCE. The clip only bites in epochs 2 through $K$.

Because a typical batch has hundreds of responses and hundreds of tokens each, the token average hides a subtle choice: averaging over all tokens weights long responses more than short ones. Some implementations average per-sequence first. This "length normalization" choice shows up again in the failure modes below.

Typical hyperparameters

These are the ranges reported in Ouyang et al. (2022), Stiennon et al. (2020), and open reproductions such as Huang et al. (2024). Treat them as starting points, not laws.

The implementation details that decide whether it works

Huang et al. (2024), "The N Implementation Details of RLHF with PPO" (arXiv:2403.17031), reproduced OpenAI's summarization RLHF results and catalogued the small choices that mattered. The list is humbling: most of them are invisible in the equations.

• Reward normalization. The raw RM output has an arbitrary offset and scale. Common practice is to subtract a constant so that the SFT model scores about 0 on average, and often to whiten (standardize) rewards within each batch before shaping. Advantages are typically whitened too.
• Value head initialization. The critic is usually initialized from the reward model (its trunk already "knows" what good text looks like), and its new scalar head is initialized to output near 0. A randomly initialized value head produces large, wrong advantages in the first iterations and can push the policy off a cliff before the critic catches up.
• EOS handling. The RM score is attached to the final token, so what counts as final matters. If a response hits the length cap without emitting EOS, it is not really finished; implementations either penalize it (a fixed negative reward) or mask it. Getting this wrong teaches the model to ramble to the cap.
• Padding and masking. Left-padding prompts so responses start aligned, masking prompt tokens out of the policy loss, and masking padding out of every mean are all easy to get wrong. A loss averaged over padding tokens dilutes the signal in a length-dependent way.
• Dropout off. Dropout during rollouts makes the stored log-probs not match the policy that gets updated. Turn it off everywhere in RL.
• Numerical precision of the log-probs. Computing log-softmax in bf16 can produce ratios that differ from 1 in the first epoch for no reason; compute in fp32.
• Adaptive KL controllers. Rather than a fixed $\beta$, several recipes increase $\beta$ when the measured KL exceeds a target and decrease it otherwise, so the leash length is set in KL units rather than in $\beta$ units.

The reward model

Everything so far assumed a function $r_\phi(x, y)$ that scores responses. Where does it come from, and how good is it? The honest answer is: it comes from a few tens of thousands of human comparisons, and it agrees with humans only somewhat more often than a coin flip.

Pairwise training with Bradley–Terry

Humans are bad at absolute scores ("rate this from 1 to 10") and reasonable at comparisons ("which of these two is better?"). So the data is pairs: prompt $x$, a chosen response $y_w$, a rejected response $y_l$. The model outputs a scalar for each response and is trained so that the chosen one scores higher.

The Bradley–Terry model turns a difference of two scores into a probability of preference. The training loss is the negative log-probability that the model assigns to the human's choice:

What just happened: only the difference of scores enters, so the RM's absolute scale is meaningless (you can add any constant to every score). A larger margin means a more confident, lower-loss prediction. This is the same functional form the Preference Optimization chapter used to derive DPO; DPO just substitutes the policy's log-ratios for $r_\phi$.

The same-prompt batching trick

Ouyang et al. (2022) collected $K = 4$ to $9$ responses per prompt and had labelers rank all of them, giving up to $\binom{9}{2} = 36$ pairs per prompt. If you shuffle those pairs into separate batches, the model sees the same prompt 36 times in one epoch and overfits to it within that epoch. Their fix was to put all $K$ responses of one prompt in a single batch and compute all pairwise losses from one forward pass per response. It is cheaper (each response is encoded once, not up to 8 times) and it removes the overfitting.

This also explains why RMs are typically trained for exactly one epoch: after one pass they have already memorized the comparisons, and a second pass makes validation accuracy worse.

How good are reward models?

Not very, and that is the central fact of this chapter. On held-out pairs, RMs in the InstructGPT and summarization papers agreed with the human label roughly 65–75% of the time, as reported. But labelers agree with each other only about 73–77% of the time on the same data (Stiennon et al., 2020, report inter-annotator agreement in this range), so the RM is close to the noise ceiling of its own training signal.

Two consequences follow. First, an RM cannot be trusted on any single comparison; it is only useful as an average over many samples, which is what the RL loop gives it. Second, a policy that optimizes the RM hard will find the 25–35% of cases where the RM is wrong, and specifically the ones where it is wrong in a systematic, exploitable way. RewardBench (Lambert et al., 2024) makes this measurable: it collects adversarial pairs where the chosen and rejected responses differ in a specific attribute, and many strong RMs score poorly on the subsets that probe length or formatting.

Over-optimization and the Goodhart lesson

Gao et al. (2022) ran the cleanest experiment on this. They trained a large "gold" reward model to stand in for humans, then trained smaller proxy RMs on labels from the gold model, and optimized policies against the proxies. Plotting the gold score against the KL from the initial policy, the proxy reward keeps going up as the policy moves away, while the gold reward rises, peaks, and then falls. Bigger proxy RMs and more RM training data push the peak further out, but every proxy peaks eventually.

This is Goodhart's law with a graph: once a measure becomes a target, it stops being a good measure. The KL penalty, early stopping, and periodic human evaluation are all ways of stopping before the peak. Their fitted curves have the form $R_{\text{gold}}(d) = d(\alpha - \beta_g \log d)$ with $d = \sqrt{\text{KL}}$, which is what the explorer below draws.

Ensembles and uncertainty

If one RM can be gamed, can several disagree their way to safety? Coste et al. (2023) trained ensembles of reward models from different seeds and used a conservative aggregate (the minimum, or mean minus a multiple of the standard deviation) as the RL reward. The policy then only gets credit where all RMs agree, and disagreement between ensemble members is a usable signal that the policy has wandered off the data. They report that this substantially delays over-optimization, at the obvious cost of running several RMs per rollout.

Cheaper variants train several heads on one shared trunk, or add a label-smoothing / margin term so the RM does not become overconfident on its training pairs. None of these remove the problem; they buy KL room.

Failure modes you will actually see

These are the symptoms that show up in almost every RLHF run and that the tricks above exist to control. Recognizing them early is most of the practical skill.

Verbosity

Human raters, and therefore reward models, tend to prefer longer answers with more structure, all else being equal. A policy notices within a few hundred steps. Length is the single most common reward hack: answers grow lists, restate the question, add caveats and summaries. Countermeasures include length penalties in the reward, length-controlled evaluation (below), and RM training data deliberately balanced for length.

Sycophancy

Sharma et al. (2023) showed that preference data and the RMs trained on it systematically favor responses that agree with the user's stated views or flatter them, and that RLHF-trained models become measurably more sycophantic as a result. The RM is not wrong about the data: humans really do rate agreeable answers higher. It is the data that is the problem, which no amount of optimizer tuning will fix.

Mode collapse and reduced diversity

RL pushes probability mass toward the highest-reward responses. A base model that could write ten different openings converges on one. Sampling diversity, measured for instance by the entropy of the policy or by the number of distinct responses among $N$ samples, drops sharply during RLHF. This is intended at the level of "stop saying harmful things" and unintended at the level of "every poem begins the same way". Entropy bonuses and a larger $\beta$ slow it down; nothing stops it entirely.

KL drift and the moving reference

Over a long run the policy can accumulate large KL even with a penalty, and once it is far from the reference the log-ratios become large and noisy, destabilizing the update. Adaptive $\beta$ controllers exist to catch this. Some recipes periodically reset the reference model to the current policy, which resets the leash but also silently gives up the protection the original reference provided.

Simplifying the system

Every piece of the four-model setup has been attacked by someone. The Preference Optimization chapter derived DPO in detail; here we place it and the newer policy-gradient variants side by side, keeping the question practical: what do you have to keep in memory, and do you still need to sample during training?

The pattern is clear. The critic is the first thing to go: with several samples per prompt, their mean is a perfectly good baseline, and a group of 4–16 samples is cheaper than training a second 7B model. The reward model is the second thing to go, but only when the task has a verifier (math answers, unit tests, format checks), which brings us to reasoning RL.

What these methods do not remove is sampling. DPO removes it, and pays for that in a different currency: it trains on whatever responses are in the dataset, which were sampled from some other model, so it is off-policy by construction and cannot discover behaviors that are absent from its data. Ahmadian et al. (2024) and others report that with the same reward model, simple on-policy REINFORCE-style methods match or beat PPO on LLMs, largely because the initial policy is already good and the per-token variance PPO was designed to tame is not the bottleneck.

Reasoning RL: rewards you can verify

Everything above wrestles with a learned, gameable reward. What if the reward were a program? For math with a known answer, code with unit tests, or output that must match a schema, a reward function can be written in a few lines and it cannot be fooled by flattery. This is RL with verifiable rewards (RLVR), and it is the setting where the simplified methods shine.

DeepSeek-R1 (DeepSeek-AI, 2025; arXiv:2501.12948) is the reference example. Its R1-Zero variant applied GRPO directly to a pretrained base model with two reward components: an accuracy reward (is the boxed answer right; do the tests pass) and a format reward (is the reasoning inside the designated think tags). No SFT, no reward model. Reported outcomes: benchmark accuracy on competition math climbed steadily through training, and the average response length grew from hundreds to thousands of tokens as the model learned to reason for longer.

Long chain-of-thought and the "aha moment"

The R1 report describes a training transcript where the model appears to pause, write "wait", and re-examine its own reasoning, and calls this an aha moment that emerged from RL. That reading is reported, and debated: subsequent analyses have argued that base models already produce such self-correction phrases at low rates and that RL amplifies an existing behavior rather than creating it. The uncontroversial part is the mechanism: with $\gamma = 1$ and a reward only at the end, any token pattern that raises the chance of a correct final answer, including backtracking, gets reinforced, and longer chains give more chances to be right.

The practical differences from chat RLHF are worth listing. Responses are long, so the per-token average in the loss and the length cap matter enormously. Rewards are binary or near-binary, so within a GRPO group many samples share the same reward and contribute zero advantage; questions that are always solved or never solved provide no signal at all, and curricula select for the ones in between. And with no learned RM, the KL penalty is much less important, because there is no proxy to over-optimize; some RLVR recipes drop the reference model entirely.

Format rewards

A format reward is a small, always-available signal: +1 if the response uses the required tags, cites in the right style, or emits valid JSON. It is easy to over-weight. If format reward is comparable in size to accuracy reward, the policy learns the format in the first hundred steps and the format term then only adds noise. The usual practice is to keep it small or to phase it out.

Process versus outcome rewards

An outcome reward model (ORM) scores the final answer. A process reward model (PRM) scores each reasoning step. Lightman et al. (2023), "Let's Verify Step by Step" (arXiv:2305.20050), collected step-level human labels for math solutions and found that a PRM trained on them was a better re-ranker of solutions than an ORM trained on outcome labels, at the same data scale. The intuition is credit assignment: a PRM tells you which step was wrong, which is exactly the information the sparse final reward lacks.

PRMs are expensive to label and, being learned, are themselves gameable; DeepSeek-R1's authors report trying PRMs and abandoning them for large-scale RL in favor of rule-based outcome rewards, citing reward hacking and the labeling cost. The current practical split is: verifiable outcome rewards for training, and PRMs mostly for re-ranking or search at inference time.

Evaluating post-trained models

If the reward model cannot be trusted to judge the policy, what can? The answer used in practice is pairwise comparison by a different judge, either humans or a strong LLM, with careful controls.

Arena-style pairwise evaluation

Chatbot Arena (Zheng et al., 2023) shows a user two anonymous responses to their own prompt and asks which is better; the votes are aggregated with the Bradley–Terry model into an Elo-like rating. It is the same model we used to train the RM, now applied across systems. Its strengths are real prompts and real users; its weaknesses are the same biases those users have, including a preference for length and confident formatting.

LLM-as-judge and its two famous biases

Zheng et al. (2023) also introduced MT-Bench and studied GPT-4 as a judge. Two biases dominate. Position bias: the judge prefers whichever response is shown first (or second, depending on the judge). The fix is to run every comparison in both orders and count it as a win only if the same side wins both times, otherwise a tie. Verbosity bias: the judge prefers longer responses even when the extra length adds nothing. Dubois et al. (2024) fixed this for AlpacaEval by fitting a regression that predicts the judge's preference from the length difference and reporting the win rate with the length term removed; this length-controlled win rate correlated much better with Arena ratings than the raw one.

The uncomfortable conclusion: a judge is a reward model by another name, and the policy that was trained against one RM will look great to any judge that shares its biases. Length control and position swapping close the two biggest loopholes; nothing closes the loophole of "the judge and the RM were trained on the same kind of preference data".

Build it small

The classic toy RLHF task is sentiment-controlled generation, used in the trl library's early examples: take GPT-2, use an off-the-shelf sentiment classifier's probability of "positive" as the reward, and train the model to complete movie-review prefixes positively. It has every ingredient of the real thing (a black-box reward, a KL leash, a reward that can be hacked with "great great great") and it runs on one small GPU in minutes.

The loop below is REINFORCE with a group-mean baseline and a per-sequence KL term, which is GRPO without the clipping. It mirrors the functions in code/lumen/rl.py; the helper names are the ones used there and in code/lumen/gpt2.py and code/lumen/sampling.py.

import torch, torch.nn.functional as F
from lumen.gpt2 import GPT2              # policy and frozen reference
from lumen.sampling import sample        # autoregressive sampling
from lumen.rl import group_advantages    # (r - mean(r)) / std(r) within a group

policy = GPT2.from_pretrained("gpt2")
ref = GPT2.from_pretrained("gpt2").eval()
for p in ref.parameters(): p.requires_grad_(False)
clf = load_sentiment_classifier()        # text -> P(positive); this is our "reward model"
opt = torch.optim.AdamW(policy.parameters(), lr=1e-6, weight_decay=0.0)
beta, G = 0.05, 8                        # KL coefficient, samples per prompt

def logprobs(model, ids, prompt_len):
    logits = model(ids[:, :-1]).float()               # predict token t+1 from tokens 0..t
    lp = F.log_softmax(logits, -1).gather(-1, ids[:, 1:, None]).squeeze(-1)
    return lp[:, prompt_len - 1:]                     # keep only response tokens

for step in range(200):
    prompt = sample_prompt()                          # e.g. tokens for "The movie was"
    prompt = prompt.expand(G, -1)                     # G copies -> one group
    with torch.no_grad():
        ids = sample(policy, prompt, max_new=24, temperature=1.0)
        reward = clf(ids)                             # shape (G,), values in [0, 1]
        lp_ref = logprobs(ref, ids, prompt.size(1))
    lp = logprobs(policy, ids, prompt.size(1))        # shape (G, T), with grad
    kl = (lp - lp_ref).sum(-1)                        # one-sample KL estimate per sequence
    shaped = reward - beta * kl.detach()              # KL as reward shaping (InstructGPT style)
    adv = group_advantages(shaped)                    # group-mean baseline, no critic
    loss = -(adv[:, None] * lp).sum(-1).mean()        # REINFORCE, summed over tokens
    opt.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.0)
    opt.step()
    if step % 20 == 0:
        print(f"step {step}  reward {reward.mean():.3f}  KL {kl.mean():.2f}  len {ids.size(1)}")

Things to try, each of which reproduces a section of this chapter in miniature: set $\beta = 0$ and watch the samples degenerate into "great great wonderful great"; print the sample entropy over training and watch it fall; swap the group baseline for no baseline and compare the variance of the loss across steps; and add a length penalty after you notice what happens when you raise the max length.

Practice

Further reading

• Ouyang et al. (2022). Training language models to follow instructions with human feedback. The InstructGPT paper: the three-stage recipe and the hyperparameters most systems still copy.
• Stiennon et al. (2020). Learning to summarize from human feedback. RLHF on summarization, with the RM-vs-human agreement numbers and the first over-optimization plots.
• Christiano et al. (2017). Deep reinforcement learning from human preferences. The origin of learning a reward model from pairwise comparisons.
• Huang et al. (2024). The N Implementation Details of RLHF with PPO. A reproduction with every detail that mattered written down.
• Gao, Schulman & Hilton (2022). Scaling laws for reward model overoptimization. The gold-vs-proxy experiment and the functional form of the curves.
• Bai et al. (2022). Training a helpful and harmless assistant with RLHF. RLHF at scale with a detailed treatment of RM calibration and helpfulness/harmlessness trade-offs.
• Rafailov et al. (2023). Direct Preference Optimization. Removes the RM and the sampling loop by reparameterizing the reward.
• Shao et al. (2024). DeepSeekMath. Introduces GRPO: group-normalized advantages, no critic, KL in the loss.
• Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE-Style Optimization for RLHF. RLOO, and the argument that PPO's machinery is largely unnecessary for LLMs.
• Hu et al. (2025). REINFORCE++. Batch-level normalization with PPO-style clipping and no per-prompt groups.
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. RLVR with GRPO on a base model; the source of the "aha moment" discussion.
• Lightman et al. (2023). Let's Verify Step by Step. Process vs outcome reward models on math.
• Sharma et al. (2023). Towards Understanding Sycophancy in Language Models. How preference data and RMs reward agreement.
• Coste et al. (2023). Reward Model Ensembles Help Mitigate Overoptimization. Conservative ensemble rewards delay the Goodhart peak.
• Lambert et al. (2024). RewardBench. A benchmark of adversarial preference pairs for reward models.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Position and verbosity bias in LLM judges, and the Arena methodology.
• Dubois et al. (2024). Length-Controlled AlpacaEval. The regression trick that removes length bias from judge win rates.