Supervised Fine-Tuning

You have a base model. You type "What is the capital of France?" and it replies "What is the capital of Germany? What is the capital of Spain?" It is not broken. It is doing exactly what it was trained to do: continue text that looks like the text it saw. Nobody ever showed it that a question should be followed by an answer.

Supervised fine-tuning (SFT) is the fix, and it is almost embarrassingly simple. You write down a few thousand conversations that look the way you want conversations to look, and you keep training the model with the same next-token loss it already knows. The only new trick is a mask that says "only learn to produce the assistant's part."

That simplicity is deceptive. The choices around SFT, which examples, how many, how formatted, how many epochs, how you evaluate, decide most of what the model feels like to use. This chapter goes through each of them, with tiny numbers you can check by hand and code you can run.

The problem: a base model does not know it is being asked

Recall from the chapter on learning to predict what pre-training optimizes. The model sees a window of tokens and predicts the next one, over trillions of tokens of web text. The result is a superb text continuer. It has read every kind of question-and-answer page, so it "knows" the answer to most simple questions. But a question on the web is followed by all sorts of things: another question, an advertisement, a forum reply that is wrong, or the correct answer. The model has no reason to prefer the last option.

What we want is a model that treats the input as a request and the output as a response to that request. That is a different distribution over text, and the base model has to be moved toward it. The cheapest way to move a model toward a distribution is to show it samples from that distribution and run the loss you already have. That is SFT.

The overview chapter placed SFT as the first stage of post-training. Everything after it, preference optimization, RL, tool use, starts from the SFT model, so mistakes here propagate. Let us get it right.

SFT is next-token prediction with a mask

Suppose we have a dataset of pairs $(x, y)$ where $x$ is a prompt (possibly a whole conversation so far) and $y$ is the response we want. We concatenate them into one token sequence, feed it through the model, and ask: how surprised is the model by each token of $y$, given everything before it?

Here is the SFT loss. It is the average negative log-probability of the response tokens, each conditioned on the prompt and on the earlier response tokens.

What happened: that is precisely the pre-training loss, except the sum runs only over the response positions. The prompt tokens are still in the context, the model still attends to them, but we do not ask the model to predict them. We do not care whether the model could have guessed the user's question; we care whether it can produce the answer.

Why mask the prompt at all?

You could skip the mask and train on the whole sequence. Some early recipes did. Two things go wrong. First, prompts are often long (a document to summarize, a code file) and responses short, so most of the gradient would be spent learning to write prompts, which is not the skill we want. Second, prompts are typically written by humans or copied from the web, and their style is not the assistant's style. Training on them pulls the model back toward "continue arbitrary text."

In practice, masking the prompt is the default. A few papers report small gains from unmasked training when responses are very short and prompts carry useful knowledge, so treat it as a knob, not a law.

A worked example with ten tokens

Take one training example and pretend our tokenizer splits on spaces and uses three special tokens. The sequence is ten tokens long:

Positions 1 to 6 are the prompt, including the <|assistant|> tag that tells the model it is now its turn. Positions 7 to 10 are the response. The label at a masked position is the number −100, which PyTorch's cross-entropy treats as "skip me."

Now remember the one-position shift. The model at position $t$ predicts token $t+1$. So the response token "It" at position 7 is predicted by the hidden state at position 6, the <|assistant|> tag. The four loss terms come from positions 6, 7, 8 and 9, predicting tokens 7, 8, 9 and 10. Position 10 predicts nothing; the sequence is over.

Where demonstrations come from

The loss is trivial. The dataset is where the work is. There are three families of sources, and modern recipes mix all three.

Human-written demonstrations

The original approach. InstructGPT (Ouyang et al., 2022) hired contractors to write roughly 13,000 prompt-and-response demonstrations, following detailed guidelines about being helpful, honest and harmless. Humans produce the highest-quality data and the most expensive; a careful multi-paragraph answer can take an expert twenty minutes, and you need experts across many domains.

Humans are also inconsistent. Two annotators given the same prompt will format lists differently, hedge differently, and disagree on how long the answer should be. Good SFT datasets come with a style guide and a review pass, not just a writing pass.

Distillation from a stronger model

If a stronger assistant already exists, you can ask it to write the responses. Alpaca (Taori et al., 2023) fine-tuned a 7B Llama on roughly 52,000 examples generated by an OpenAI model, and the result was a surprisingly capable assistant for a few hundred dollars of API calls. Zephyr (Tunstall et al., 2023) did the same at higher quality with the UltraChat dataset, calling the step "distilled SFT."

Distillation is cheap and consistent, and it inherits the teacher's style, including its verbosity, its favorite phrases and its mistakes. It is covered more generally in the distillation chapter; here the point is that the teacher's outputs are just another source of $(x, y)$ pairs.

Self-Instruct and Evol-Instruct: generating the prompts too

Responses are only half the problem. Where do the prompts come from? Self-Instruct (Wang et al., 2022) started from 175 hand-written seed tasks and asked a model to generate new instructions in the same spirit, then generate responses, then filtered near-duplicates. The loop bootstrapped tens of thousands of instructions from almost nothing.

Evol-Instruct (Xu et al., 2023, the WizardLM paper) adds a twist: take an existing instruction and ask the model to rewrite it to be harder, by adding constraints, requiring more reasoning steps, or generalizing to a broader topic. Iterating the rewrite produces a curriculum of increasingly demanding prompts, which matters because easy prompts teach the model very little.

Quality over quantity

How many examples do you need? The honest answer is "far fewer than you think, if they are good." LIMA (Zhou et al., 2023) fine-tuned a 65B Llama on just 1,000 carefully curated examples, with no RLHF, and reported that its responses were judged equivalent to or better than GPT-4's in roughly 43% of pairwise comparisons. The authors called this the superficial alignment hypothesis: almost all of the model's knowledge comes from pre-training, and SFT mostly teaches which sub-distribution of formats and styles to use.

The hypothesis is a slight overstatement, as later work on reasoning showed, but the practical lesson has held up: a small, clean, diverse SFT set beats a large, noisy one. Noisy examples do not average out. The model learns the noise, because the loss rewards it for reproducing every token of every example, including the wrong ones.

Diversity

The second lesson from LIMA is that diversity of prompts matters more than the number of prompts. A thousand examples covering coding, writing, math, advice, summarization and role-play teach a general "how to respond" prior. Ten thousand examples of the same task teach that task and quietly erode everything else. When in doubt, add a new kind of prompt before adding more of an existing kind.

Formatting consistency

The third lesson is about style. If half your examples use Markdown headers and half do not, the model will flip a coin at inference time. If some responses open with "Sure! Here is..." and others do not, you get an assistant that sometimes does. Consistent formatting across the set is what produces a model that feels deliberate. This is also why distilled data trains so cleanly: one teacher, one style.

Multi-turn conversations, system prompts and packing

Real assistants hold conversations, not single exchanges. A multi-turn example is a list of messages with roles, and the masking rule generalizes naturally: train on every assistant turn, mask every user and system turn. One conversation with three assistant turns is one sequence with three trained spans.

System prompts

A system prompt is a message at the start of the conversation that sets the assistant's persona, constraints, or tools. During SFT it is treated exactly like a user message: present in the context, masked from the loss. The important thing is to include system prompts in training data, and to vary them. A model that never saw a system prompt during SFT will not follow one at inference time, and a model that only ever saw the same one will ignore the others. The tools and safety chapter comes back to this under the name instruction hierarchy.

Packing several examples into one sequence

Here is a systems problem that becomes a correctness problem. SFT examples vary wildly in length, from a 40-token exchange to a 4,000-token code review. If you pad every example to the maximum, most of your batch is padding and you waste most of your compute. The standard fix is packing: concatenate several examples end to end until you fill the context window, then train on the whole thing as one sequence.

The catch is attention. With plain causal attention, tokens of the second conversation can attend to the first conversation, which is unrelated garbage as far as they are concerned. The clean fix is a block-diagonal attention mask so each conversation sees only itself, plus a reset of positional indices at each boundary. Many libraries expose this as a flag. Without it, packing still mostly works, because the model learns to ignore text before an end-of-sequence token, but it is a source of subtle quality loss and of prompts leaking into each other.

Hyperparameters that matter

SFT is forgiving compared to pre-training, but a few settings matter a lot. The pattern is: small learning rate, few epochs, and the same tricks from code/lumen/train.py (warmup, cosine decay, gradient clipping, mixed precision) that you already have.

As reported examples: Llama-2-Chat's SFT used a learning rate of $2\times10^{-5}$, cosine decay, batch size 64 and two epochs; Tülu 3's 8B model used $5\times10^{-6}$ for two epochs. Recipes vary, but they all live in this neighborhood.

LoRA or full fine-tuning?

Everything above assumes you update every weight. If you cannot afford that, LoRA (Hu et al., 2021) trains small low-rank adapters and typically wants a learning rate about ten times higher (around $1\times10^{-4}$ to $2\times10^{-4}$). For style and format, LoRA SFT is usually as good as full SFT. For teaching a lot of new material it lags. The trade-offs are the subject of the LoRA chapter; nothing else in this chapter changes.

What SFT teaches, and what it cannot fix

It helps to be precise about what a few thousand examples can do to a model that has seen trillions of tokens.

What SFT teaches well. Format: answer in Markdown, use numbered steps, put code in fences. Style: concise or chatty, formal or friendly, how much to hedge. Turn structure: stop when the answer is done, emit the end-of-turn token. Tool syntax: how to write a function call in the expected JSON shape. Refusal patterns: what to decline and how. All of these are surface-level regularities, and surface-level regularities are exactly what a few thousand examples can pin down.

What SFT cannot fix. Knowledge the base model does not have. Reasoning ability the base model does not have. If the base model cannot do three-digit multiplication, showing it a thousand worked multiplications will make it look like it multiplies, which is worse than looking like it cannot. Deep capability comes from pre-training scale and, more recently, from RL with verifiable rewards, which we get to in the preference optimization chapter.

Hallucination from SFT

This leads to the sharpest warning in the chapter. Suppose a demonstration asks "Who won the 1987 regional chess championship of some small town?" and the human-written response names the winner. The base model has no idea who that is. What does the gradient do? It pushes the model to produce a confident name after a question it cannot answer. Multiply by thousands of examples and you have trained the model that the correct behaviour when you do not know is to make something up.

John Schulman made this argument in a 2023 talk on RL and truthfulness, and Gekhman et al. (2024) measured it. They split fine-tuning examples into those whose facts the model already knew and those it did not. The "unknown" examples were learned much more slowly, and as the model finally fit them, its tendency to hallucinate on other questions rose. Examples the model could not have known were actively harmful.

Overfitting: signs and evaluation

SFT overfits easily because the dataset is tiny relative to the model. You will see it as: training loss well below the loss on held-out examples; the model reproducing sentences from the training set verbatim; a collapse in diversity, with every response opening the same way; and sometimes a drop on general benchmarks as the model forgets pre-training abilities it does not need for the SFT set.

Held-out loss is not enough

Here is a counter-intuitive finding from the InstructGPT paper. The authors trained their SFT model for 16 epochs and observed that validation loss started rising after the first epoch, the textbook sign of overfitting. Yet human raters kept preferring the models trained longer. The loss was measuring "does the model put probability on this exact demonstration," and there are many good answers to any prompt. A model that has moved toward its own phrasing gets a worse held-out loss and better ratings.

So the loss is a sanity check, not a target. The evaluation that matters is judged win-rate: sample responses from your model and a reference model on a fixed set of held-out prompts, and have humans or a strong LLM judge say which is better. Track win-rate across checkpoints and pick the best one. Combine it with a small set of capability benchmarks to catch forgetting. The overview chapter listed the standard suites; the next chapter discusses the judge's biases, which you must correct for.

Rejection-sampling fine-tuning: a bridge to RL

The hallucination problem and the "many good answers" problem share a fix: let the model write its own demonstrations. Sample $N$ responses for each prompt, score them with something you trust, keep the best, and run SFT on what you kept. This is rejection-sampling fine-tuning, and it is the simplest possible form of learning from a reward.

What can the scorer be? A reward model trained on human preferences, as in Llama-2-Chat, which used it as a stage before PPO. A verifier: a unit test for code, an answer checker for math, a format validator. Or a stronger model acting as a judge. When the scorer is a verifier, the method is called self-training with correctness filtering, the idea behind STaR (Zelikman et al., 2022). RAFT (Dong et al., 2023) and ReST (Gulcehre et al., 2023) formalize the loop and iterate it.

Why is this a bridge to RL? Because it is RL with a very crude policy update. Sampling from the model and reinforcing high-reward samples is what policy-gradient methods do; rejection sampling just uses a hard top-$k$ selection instead of a weighted gradient, and it never pushes probability down on bad samples. It gets you a surprising fraction of the benefit with none of the machinery. The full machinery, with a KL penalty and clipped updates, is in the RLHF deep dive.

Two practical notes. First, the top-$k$ set is biased toward whatever the scorer likes, including its blind spots; if the reward model likes long answers, your next model will be verbose. Second, because it never penalizes anything, rejection sampling cannot remove a behaviour that appears in every sample. For that you need a method that pushes down as well as up, which is the topic of the next chapter.

Implementation walkthrough

Let us build the whole thing. The pieces are: render a conversation with a chat template, tokenize it with a label mask, batch with padding, and run a short training loop. The loop reuses the schedule and clipping from code/lumen/train.py; the tokenizer can be the one from code/lumen/tokenizer.py extended with a few special tokens, or any Hugging Face tokenizer.

Step 1: the chat template

A chat template is a deterministic function from a list of messages to a string. Ours uses three special tokens, which must be added to the tokenizer's vocabulary so they encode as single ids and can never be produced by ordinary text.

ROLE_TOKENS = {"system": "<|system|>", "user": "<|user|>", "assistant": "<|assistant|>"}
END = "<|end|>"

def render(messages):
    """messages: list of {"role": ..., "content": ...}. Returns the training string."""
    out = []
    for m in messages:
        out.append(f"{ROLE_TOKENS[m['role']]}\n{m['content']}{END}\n")
    return "".join(out)

def render_prompt(messages):
    """Same, but ends with an open assistant tag: this is what you feed at inference."""
    return render(messages) + ROLE_TOKENS["assistant"] + "\n"

msgs = [{"role": "user", "content": "What is 2+2?"},
        {"role": "assistant", "content": "It is 4."}]
print(render(msgs))

Two rules that save you pain. The training string and the inference prompt must be produced by the same function, or the model will see a format at inference it never saw in training. And the end-of-turn token must be part of the trained span, or the model will never learn to stop.

Step 2: tokenize and build the label mask

We tokenize each message separately so we know exactly which ids belong to which role, then concatenate. Assistant content gets copied into labels; everything else becomes −100. The role header of the assistant turn is masked too, since the model is given it rather than asked to produce it.

IGNORE = -100

def build_example(messages, tok, max_len=2048):
    """Returns input_ids and labels for one (possibly multi-turn) conversation.
    Every assistant turn is trained; system and user turns are masked."""
    ids, labels = [], []
    for m in messages:
        header = tok.encode(f"{ROLE_TOKENS[m['role']]}\n", add_special_tokens=False)
        body = tok.encode(f"{m['content']}{END}\n", add_special_tokens=False)
        ids += header + body
        if m["role"] == "assistant":
            labels += [IGNORE] * len(header) + body      # train on the answer and its END token
        else:
            labels += [IGNORE] * (len(header) + len(body))
    return {"input_ids": ids[:max_len], "labels": labels[:max_len]}

ex = build_example(msgs, tok)
print(sum(l != IGNORE for l in ex["labels"]), "of", len(ex["labels"]), "positions trained")

The count you see depends on the tokenizer; the point is that it is the response tokens plus the end token, and nothing else. A quick unit test worth writing: decode the ids where labels != -100 and check that you get back exactly the assistant text.

Step 3: batching with padding

Sequences in a batch have different lengths. We right-pad the ids with the pad token, pad the labels with −100 so padding is never scored, and build an attention mask so padding is never attended to. If you use packing instead, this is where you would concatenate examples and build the block-diagonal mask from Figure 3.

import torch

def collate(batch, pad_id):
    L = max(len(b["input_ids"]) for b in batch)
    ids = torch.full((len(batch), L), pad_id, dtype=torch.long)
    labels = torch.full((len(batch), L), IGNORE, dtype=torch.long)
    attn = torch.zeros((len(batch), L), dtype=torch.long)
    for i, b in enumerate(batch):
        n = len(b["input_ids"])
        ids[i, :n] = torch.tensor(b["input_ids"])
        labels[i, :n] = torch.tensor(b["labels"])
        attn[i, :n] = 1
    return ids, labels, attn

Step 4: the loss, with the shift made explicit

Most libraries shift labels for you inside the model. Doing it by hand once makes the worked example above concrete: logits at position $t$ are compared to the label at position $t+1$, and ignore_index drops every masked term from both the sum and the denominator.

import torch.nn.functional as F

def sft_loss(model, ids, labels, attn):
    logits = model(ids, attention_mask=attn).logits          # (B, L, V)
    shift_logits = logits[:, :-1, :]                          # position t predicts ...
    shift_labels = labels[:, 1:]                              # ... token t+1
    return F.cross_entropy(shift_logits.reshape(-1, shift_logits.size(-1)),
                           shift_labels.reshape(-1), ignore_index=IGNORE)

Because ignore_index averages over non-masked positions only, the loss is per trained token. That matches the formula, and it means a batch full of long prompts and short answers is not diluted toward zero.

Step 5: the training loop

The loop is the pre-training loop with a smaller learning rate and a shorter schedule. Gradient accumulation gets you a large effective batch on one GPU; clipping at 1.0 and bf16 autocast are the defaults from code/lumen/train.py.

import math, torch
from torch.utils.data import DataLoader

def lr_lambda(step, warmup, total):
    if step < warmup:
        return step / max(1, warmup)
    progress = (step - warmup) / max(1, total - warmup)
    return 0.5 * (1 + math.cos(math.pi * min(1.0, progress)))    # cosine to zero

def train_sft(model, examples, tok, epochs=2, lr=2e-5, micro_bs=4, accum=16, device="cuda"):
    loader = DataLoader(examples, batch_size=micro_bs, shuffle=True,
                        collate_fn=lambda b: collate(b, tok.pad_token_id))
    total = epochs * len(loader) // accum
    opt = torch.optim.AdamW(model.parameters(), lr=lr, betas=(0.9, 0.95), weight_decay=0.0)
    sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: lr_lambda(s, int(0.03 * total), total))
    model.train(); step = 0
    for epoch in range(epochs):
        for i, (ids, labels, attn) in enumerate(loader):
            ids, labels, attn = ids.to(device), labels.to(device), attn.to(device)
            with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
                loss = sft_loss(model, ids, labels, attn) / accum
            loss.backward()
            if (i + 1) % accum == 0:
                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
                opt.step(); sched.step(); opt.zero_grad(set_to_none=True); step += 1
                if step % 20 == 0:
                    print(f"epoch {epoch} step {step}/{total} loss {loss.item() * accum:.3f}")
    return model

Those numbers are typical of a small model on a few thousand examples: the loss drops fast in the first few hundred steps as the model learns the template, then slowly as it learns the content. If your loss starts near zero, your labels are probably all −100 except the padding. If it never falls below the base model's loss on ordinary text, your labels are probably not shifted.

Step 6: sanity checks before you trust it

• Decode the trained positions. For a few examples, print the tokens where labels != -100. You should see exactly the assistant text plus the end token.
• Generate on a held-out prompt. Render it with render_prompt, sample with code/lumen/sampling.py, and confirm the model stops at <|end|>. If it runs on and starts writing a user turn, your end token is masked.
• Compare checkpoints by win-rate. Sample from the epoch-1 and epoch-2 checkpoints on 100 held-out prompts and judge them pairwise. Keep the winner, not the one with the lower loss.

Practice

Further reading

• Ouyang et al. (2022). Training language models to follow instructions with human feedback. The InstructGPT paper; Section 3 and the SFT epochs observation.
• Zhou et al. (2023). LIMA: Less Is More for Alignment. The 1,000-example result and the superficial alignment hypothesis.
• Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions. Bootstrapping instructions from a seed set.
• Xu et al. (2023). WizardLM: Empowering Large Language Models to Follow Complex Instructions. Evol-Instruct.
• Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment. Distilled SFT followed by distilled DPO.
• Gekhman et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?. The measured version of the hallucination-from-SFT argument.
• Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. SFT data choices and rejection-sampling fine-tuning before PPO.
• Zelikman et al. (2022). STaR: Bootstrapping Reasoning With Reasoning. Self-training with a correctness filter.
• Dong et al. (2023). RAFT: Reward rAnked FineTuning and Gulcehre et al. (2023). Reinforced Self-Training (ReST). Two formalizations of the rejection-sampling loop.
• Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. A fully open SFT mixture with ablations; the subject of the case study chapter.