RL Fundamentals

Suppose you want a model to solve a maths problem, and the only thing you can check is whether the final answer is right. There is no "correct next token" to imitate. There is a whole response, and a single number at the end that says good or bad. Next-token prediction has no way to use that number. Reinforcement learning is the set of tools for exactly this situation: learning from a score instead of a target.

This chapter is RL for people who already know how a language model works. We will not talk about robots or Atari. Every symbol will be mapped onto prompts, tokens and responses, and every formula will be checked with numbers small enough to verify on paper. The next chapter, RLHF Deep Dive, assembles these pieces into the full system.

The problem: a score is not a target

Supervised fine-tuning gives the model a target for every position. The loss at position $t$ is $-\log p(\text{correct token}_t)$, the gradient flows back, done. That works whenever someone has written down the response you want.

But many of the things we actually care about are properties of the whole response, and they only show up after the response is finished. Did the proof reach the right answer? Did the code pass the tests? Did a human prefer this reply to that one? Was it too long? None of these can be attached to a single token while the model is writing it.

Two things make this genuinely harder than supervised learning.

• Credit assignment. The response had 300 tokens and got a 7 out of 10. Which tokens deserve the credit and which deserve the blame?
• The data depends on the model. The model produced the text it is being scored on. Change the weights and you change the data. There is no fixed dataset to loop over.

The vocabulary: an MDP, mapped onto a language model

RL papers describe problems as Markov decision processes (MDPs). The vocabulary sounds abstract, but for language models every term has a very concrete meaning. Here is the dictionary. Keep it open; we will use every entry.

Return, discounting, and why LLM RL usually sets γ = 1

Rewards can arrive at several time steps. The return $G_t$ is the total reward collected from step $t$ onwards, with a discount factor $\gamma \in [0,1]$ that makes far-away rewards count less. Here is the definition: it just sums future rewards with a geometric fade.

A small $\gamma$ says "I care about now". Discounting exists for two reasons: infinite-horizon problems need the sum to converge, and in noisy environments far-off rewards are less informative about the current action.

Neither reason applies to a language model. The episode is short and finite, the reward is at the end, and there is no sense in which the 5th token "caused" the final score less than the 200th. So LLM RL almost always uses $\gamma = 1$, and then something simple happens: with a single terminal reward $R$, the return from every position is just $G_t = R$.

Value, action-value, and the advantage

Now three quantities that let us talk about "how good" a situation is. All three are expectations under the current policy, which means they change whenever the weights change.

The state value $V(s)$ is the expected return if you are at $s$ and keep sampling from $\pi$. For an LLM: given this partial response, how well does this model typically finish it? The action value $Q(s,a)$ is the same, but with the next token forced to be $a$. And the advantage is the difference.

The advantage answers a single question: was choosing $a$ better than what this policy would usually do here? Positive means better than expected, negative means worse. The sign is what matters, and it will drive every update in this chapter.

The policy gradient

Here is what we want to maximise: the expected reward of a response sampled from the model.

The catch is the subscript. The expectation is over samples from the thing we are differentiating. You cannot backprop through "sample a token" the way you backprop through a matrix multiply. The reward $R$ might not even be differentiable: a unit test either passes or it does not.

The log-derivative trick, on a two-armed bandit

The way out is an identity from calculus. For any probability $\pi(a)$ that depends on $\theta$, the derivative of $\pi$ equals $\pi$ times the derivative of $\log \pi$. That is all the trick is.

Now push the gradient inside the expectation. The expectation is a sum over actions weighted by $\pi(a)$, and the identity turns "gradient of a probability" into "probability times gradient of a log", which is an expectation again.

Read the right-hand side as a recipe: sample an action, look at its reward, and move the weights in the direction that increases the log-probability of that action, scaled by the reward. No derivative of $r$ needed. That is why RL can use a unit test as a reward.

REINFORCE

Applying the same trick to a full episode gives the REINFORCE estimator (Williams, 1992). The probability of a whole trajectory is a product of per-token probabilities, so its log is a sum, and the gradient becomes a sum over positions of "gradient of the log-prob of the token we produced, weighted by the return that followed".

In words: for every token in the sampled response, raise its log-probability in proportion to the reward collected after it. With $\gamma = 1$ and a terminal reward, $G_t = R$ for all $t$, so this is literally "raise the log-prob of every token of a good response, lower it for a bad one". The loss you actually write in PyTorch is the negative of this, so that gradient descent goes the right way.

import torch

def reinforce_grad(logits, action, reward, baseline=0.0):
    """One-sample REINFORCE estimate for a bandit.
    logits: (K,) tensor with requires_grad=True. Returns d(-surrogate)/d(logits)."""
    logp = torch.log_softmax(logits, dim=-1)[action]
    surrogate = (reward - baseline) * logp     # what we want to *maximise*
    (-surrogate).backward()                    # so we minimise its negative
    return logits.grad

Why the estimate is so noisy

Look at the two possible values of $\hat g$ in the worked example: $(0.269, -0.269)$ and $(-1.462, 1.462)$. They point in opposite directions. The average is right, but any single sample is mostly noise. The variance of the first component is $0.731 \cdot 0.269^2 + 0.269 \cdot 1.462^2 - 0.197^2 \approx 0.589$, a standard deviation of $0.77$ around a mean of $-0.197$. The signal is four times smaller than the noise.

It gets worse for language models. The rewards are often all positive (a score from 0 to 10), so every sampled token is pushed up, and the only thing distinguishing a good token from a bad one is how hard it is pushed. A response with 300 tokens gives you 300 gradient terms that all share the same scalar weight. And you get one trajectory sample from a space of $100000^{300}$ possibilities.

Baselines: less variance, no bias

The fix is to subtract a number $b$ from the reward before weighting. The claim is that this does not change the expected gradient at all, only its variance. Here is the estimator with a baseline.

Why is it still unbiased? Because the extra term has expectation zero: $\mathbb{E}_a[b\, \nabla \log \pi(a)] = b \sum_a \pi(a) \nabla \log \pi(a) = b \sum_a \nabla \pi(a) = b\, \nabla \sum_a \pi(a) = b\, \nabla 1 = 0$. Probabilities sum to one, and the gradient of a constant is zero. Any $b$ that does not depend on the action is allowed.

The best constant baseline is close to the expected reward, and the best state-dependent baseline is $V(s)$, which turns $G_t - b$ into an estimate of the advantage $A_t$. This is the moment the advantage stops being a definition and becomes an algorithm: policy gradient methods differ mostly in how they estimate $A_t$.

Actor–critic: learning the baseline

A constant baseline helps, but the ideal baseline is $V(s_t)$, which is different for every state. We do not know $V$. So we learn it: a second network, the critic, is trained to predict the return from a state, and the policy (the actor) uses its predictions as the baseline.

For a language model the critic is usually a small value head: a linear layer on top of the transformer's hidden states that outputs one scalar per position. It is trained by regression on the returns we observe.

That is the whole critic. It looks at "The answer is" and learns to output 0.6 because, on average, the current policy finishes that state correctly 60% of the time. The advantage estimate for the actor is then $\hat A_t = G_t - V_\phi(s_t)$.

Bootstrapping and GAE, in one paragraph

Once you have a critic you can do more than subtract it. Instead of waiting for the true return $G_t$, you can estimate it one step ahead as $r_{t+1} + \gamma V(s_{t+1})$, using the critic's guess for the rest. The difference from the current estimate is the TD error $\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$, which is itself a (biased, low-variance) advantage estimate. Generalised Advantage Estimation, GAE (Schulman et al., 2015), blends every horizon with a decay $\lambda$:

At $\lambda = 0$ this is the pure one-step TD error (low variance, trusts the critic completely). At $\lambda = 1$ the sum telescopes to $G_t - V(s_t)$ (unbiased, trusts only the observed return). Values like $\lambda = 0.95$ are the usual compromise. With $\gamma = 1$ and rewards only at the end, most LLM implementations end up close to the $\lambda=1$ end anyway.

Reusing samples: importance sampling

REINFORCE has an expensive rule: after every gradient step, throw the samples away and generate new ones. For a language model, generation is the slow part. One batch of responses might take minutes; the gradient step takes seconds. It would be nice to take several steps on the same batch.

But after one step the weights have changed, and the samples are now from the old policy $\pi_{\text{old}}$, not the current $\pi_\theta$. Using them as if they were from $\pi_\theta$ is biased. The correction is importance sampling: reweight each sample by how much more (or less) likely the new policy would have been to produce it.

The fraction is the ratio $r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\text{old}}(a_t \mid s_t)$. It is $1$ at the moment of sampling and drifts as we take steps. This ratio is the central object in PPO.

PPO: the clipped objective

Proximal Policy Optimization (Schulman et al., 2017) starts from the importance-weighted policy gradient objective $r_t(\theta)\, \hat A_t$ and adds one guard rail. If the ratio has already moved past $1 \pm \epsilon$ in the direction the advantage wants, stop rewarding further movement. Here is the objective; it is a minimum of two terms.

The first term is the plain importance-weighted objective. The second is the same thing with the ratio clamped to $[1-\epsilon, 1+\epsilon]$. Taking the minimum makes the objective pessimistic: whichever term is smaller wins, so the policy never gets credit for pushing the ratio beyond the band, but it always gets penalised for being on the wrong side of it. The usual $\epsilon$ is $0.2$.

The full PPO loss, and the KL-penalty variant

The clipped surrogate is one of three terms in the loss that actually gets minimised. The critic needs its regression loss, and an entropy bonus (more on that below) discourages collapse. With weights $c_1$ and $c_2$:

The same paper proposes an alternative to clipping: keep the plain ratio objective but subtract a penalty $\beta \cdot \text{KL}(\pi_{\text{old}} \| \pi_\theta)$ that grows when the new policy strays from the old one, with $\beta$ adjusted adaptively. Clipping won in practice because it has one hyperparameter and no tuning loop. But the KL idea survived in a different place: in RLHF, a KL penalty against the reference model (not the previous iterate) is folded into the reward itself. That is a different KL with a different purpose, and the next chapter is built around it.

def ppo_clip_loss(logp_new, logp_old, adv, eps=0.2):
    """logp_new / logp_old / adv: (B, T) per-token tensors (masked outside the response)."""
    ratio = torch.exp(logp_new - logp_old)             # pi_new / pi_old, per token
    unclipped = ratio * adv
    clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * adv
    return -torch.min(unclipped, clipped).mean()       # negative because we maximise the surrogate

GRPO: REINFORCE with a group-mean baseline and no critic

The critic is expensive. For an LLM it is another model of the same size, with its own optimiser state, that must be trained alongside the policy and is only ever used to produce a baseline. Group Relative Policy Optimization, GRPO (Shao et al., 2024, DeepSeekMath), asks a pointed question: if the baseline's job is to say "what does this policy usually score on this prompt", why not just measure that by sampling several responses to the same prompt?

So GRPO generates a group of $G$ responses per prompt, scores each, and uses the group's own statistics as the baseline. Here is the advantage, which is the same for every token of response $i$.

Subtracting the mean is the baseline. Dividing by the standard deviation rescales every prompt's rewards to unit spread, so an easy prompt where all rewards are 0.9 or 1.0 produces the same size of update as a hard one where they are 0 or 1. The rest of the objective is PPO's: the per-token clipped ratio, plus a KL term against the reference model. The critic is simply gone.

def group_advantages(rewards, eps=1e-8):
    """GRPO-style advantages for one prompt's group of samples."""
    r = torch.as_tensor(rewards, dtype=torch.float32)
    return (r - r.mean()) / (r.std(unbiased=False) + eps)

Where the credit lands: broadcasting the advantage over tokens

Step through a single episode with the vocabulary in hand. Watch where the reward appears and how, with $\gamma = 1$ and no critic, one number ends up attached to every token.

The last frame shows the honest limitation of terminal-reward RL: the filler words get the same credit as the answer. In practice this is fine on average, because "The answer is" appears in both correct and incorrect responses and its net push cancels out over many samples. But it is a real source of noise, and it is why per-token reward shaping (like the KL penalty in RLHF) and process rewards exist.

Exploration and the entropy bonus

Here is a failure that policy gradients are prone to. The policy finds one response that scores well, its probability goes up, it gets sampled more, it gets reinforced more, and within a few hundred steps the model produces the same answer to everything. It never tries the alternatives, so it never learns whether they were better. This is exploration collapse.

The standard fix is to add the entropy of the policy's distribution as a bonus to the objective. Entropy measures how spread out a distribution is; here is the formula for one position.

A peaked distribution has low entropy, a flat one has high entropy. Adding $c_2 \mathcal{H}$ to the objective (it is the $-c_2 \mathcal{H}$ term in the PPO loss) pays the policy a little for staying uncertain, which keeps alternative tokens alive long enough to be tried.

In LLM RL the sampling temperature plays the same role at rollout time, and entropy is one of the first diagnostics to plot. Reports from reasoning-RL runs describe entropy dropping quickly early in training; whether that is healthy commitment or premature collapse is often only clear in hindsight, which is why many recipes monitor it and intervene (via the bonus, the temperature, or the KL term) when it falls too fast.

Reward hacking and specification gaming

Everything above assumes the reward measures what you want. It never quite does. A reward is a proxy: a classifier, a reward model, a unit test, a length-normalised score. RL is an optimiser that will find every gap between the proxy and your intent, because gaps are where the easy reward is. This is Goodhart's law in its purest form: when a measure becomes a target, it stops being a good measure.

The failures have LLM-specific flavours, and you will meet all of them.

• Length. Reward models trained on human preferences tend to score longer answers higher. The policy learns to pad: restate the question, add caveats, add a summary of the summary. Reported in nearly every RLHF paper since 2022.
• Formatting. Bullet points, bold headers, markdown tables. If the judge likes structure, the policy produces structure whether or not the content deserves it.
• Sycophancy. Agreeing with the user's stated opinion, praising the question, backing down when challenged even when right. Human raters reward agreement, so the policy learns to agree.
• Test gaming. With unit tests as reward, models have been observed to special-case the expected outputs, or to edit the test file. The reward said "tests pass", not "solve the problem".
• Judge-word stuffing. If an LLM judge is told to rate "helpfulness and honesty", responses that literally say "to be helpful and honest…" score better.
• Confident nonsense. Hedged answers score lower than confident ones with a preference-trained RM, so the policy learns to sound sure.

The defences map onto the tools in this chapter. A KL penalty to a reference model limits how far the policy can drift toward the proxy's blind spots. Length penalties and length-controlled judging target the most common hack directly. Verifiable rewards (a real answer key, a real compiler) remove the proxy where you can. Ensembles of reward models, and stopping early on the curve above, do the rest. None of this makes the problem go away, which is why reward design gets its own treatment in the RLHF chapter.

Putting the map together

Here is how the methods in this chapter relate. They are all the same gradient, $\nabla \log \pi \cdot (\text{something})$, and they differ in what the something is and how many samples it costs.

Reading a paper's method section is now a matter of locating it in this table. "We use PPO with GAE ($\lambda = 0.95$), a value head initialised from the reward model, and a KL coefficient of 0.05" is a row of this table plus three hyperparameters. "We use GRPO with a group size of 16 and verifiable rewards" is another row. The rest of the paper is about the reward.

Practice

Further reading

• Sutton & Barto (2018). Reinforcement Learning: An Introduction. The textbook; chapters 3 and 13 cover MDPs and policy gradients in full.
• Williams (1992). "Simple statistical gradient-following algorithms for connectionist reinforcement learning". The original REINFORCE paper, including the baseline argument.
• Schulman et al. (2015). Trust Region Policy Optimization. The constrained method that PPO approximates.
• Schulman et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. The GAE paper; short and worth reading for the bias–variance argument.
• Mnih et al. (2016). Asynchronous Methods for Deep Reinforcement Learning. A3C/A2C: the actor–critic recipe most modern code descends from.
• Schulman et al. (2017). Proximal Policy Optimization Algorithms. The clipped objective and the KL-penalty variant; 13 pages.
• Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Introduces GRPO in section 4.
• Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. RLOO, and the case that PPO's machinery is often unnecessary for RLHF.