Distillation

You have a 70B model that answers well and costs a fortune per token. You have an 8B model that is cheap and mediocre. Fine-tuning the 8B model on the same data does not close the gap; the big model learned things from a trillion tokens that no fine-tuning set will contain. Is there a way to move what the big model knows into the small one directly?

Yes, and the idea is almost embarrassingly simple: instead of training the small model on the right answers, train it on the big model's opinions about all the answers. That is knowledge distillation. This chapter covers the classic version, the variants that matter for language models, and the practical recipe.

The Problem: Big Models Are Expensive to Serve

Inference cost scales roughly with the number of active parameters per token. A dense 70B model needs about $2 \times 70 \times 10^9 = 140$ GFLOPs per generated token, plus a KV cache read that is 9× larger than an 8B model's. On the same hardware, the 8B model serves roughly 8–10× more tokens per second per GPU. If you are serving a billion requests a day, that ratio is the difference between a product and a bankruptcy.

So the question of "how good can a small model be?" is not academic. And the answer depends heavily on how you train it. Pretraining an 8B model from scratch to match a 70B model is not possible: at equal data, the bigger model wins (that is what scaling laws say). But matching a specific behavior of the big model, on a specific distribution of inputs, is a much easier target.

Can a Small Model Learn From a Big One?

Think about what a training example gives a model. A hard label says "the next token is cat." One bit of information about a 100k-way decision, more or less. But the big model, when it looks at the same prefix, produces a full distribution: cat 60%, dog 25%, kitten 8%, car 0.001%. That distribution encodes that dog is a plausible alternative here and car is not, which is a statement about the structure of the problem that a single hard label can never make.

Hinton, Vinyals and Dean (2015) called this dark knowledge: the information hidden in the small probabilities of the wrong answers. Training a student to reproduce the teacher's whole distribution transfers that structure, and it turns out to be a much richer signal per example than the label.

Dark Knowledge: What Soft Targets Carry

There is a snag. A well-trained teacher is confident. Its distribution is often 99% on one token and dust on the rest, and "dust" carries almost no gradient. The dark knowledge is there, but it is buried at the $10^{-4}$ level where the student cannot feel it.

Hinton's fix: raise the temperature. Divide the logits by $T \gt 1$ before the softmax, for both teacher and student, so the small probabilities inflate and the ranking among the wrong answers becomes visible:

What just happened: dividing every logit by $T$ shrinks the gaps between them, so the softmax output is flatter. $T=1$ is the ordinary distribution; $T\to\infty$ is uniform. Somewhere in between, the second and third choices become large enough to learn from without the top choice disappearing.

The Temperature-Softened KL Loss

Now the loss. We want the student's softened distribution $q(T)$ to match the teacher's softened distribution $p(T)$. The natural measure of mismatch between two distributions is the KL divergence, so the distillation loss on one position is:

What just happened: for each token $i$, we weight the log-ratio of teacher to student probability by the teacher's probability. Tokens the teacher thinks are plausible count a lot; tokens it rules out barely matter. Minimizing this over the student's parameters pushes $q(T)$ toward $p(T)$. Since $\sum p\log p$ does not depend on the student, this is the same as minimizing the soft cross-entropy $-\sum_i p_i(T)\log q_i(T)$.

Why the $T^2$?

Differentiate the soft cross-entropy with respect to the student's logit $v_i$. The result (standard softmax calculus) is $\partial\mathcal L/\partial v_i = \frac{1}{T}\big(q_i(T) - p_i(T)\big)$. At high temperature both distributions are nearly uniform, and Hinton showed that $q_i(T) - p_i(T) \approx (v_i - z_i)/(NT)$ where $N$ is the vocabulary size. So the gradient scales like $1/T^2$. If you ever change $T$, the soft loss's gradient would shrink by $T^2$ relative to any other loss you are adding, which makes tuning a nightmare. Multiplying by $T^2$ cancels that, so the relative weight of soft and hard losses stays roughly constant as you sweep $T$.

Combining with the hard label

In practice you usually have the real next token too. Hinton's recipe uses both, with a mixing weight $\alpha$:

The hard-label term uses the student at $T=1$ (the real distribution it will be used with). The soft term uses $T\gt1$. Typical values: $T\in[2,5]$, $\alpha\in[0.1, 0.5]$, with the soft term doing most of the work. The hard term acts as a safety rail: if the teacher is confidently wrong somewhere, the label pulls the student back.

Distillation for Language Models

The classifier picture has one output per example. A language model has one output per token, and generates sequences, and that opens two quite different ways to distill.

Sequence-level distillation: the teacher as a data generator

The simplest thing that works: prompt the teacher, collect its responses, fine-tune the student on them with plain cross-entropy. Kim and Rush (2016) introduced this as sequence-level knowledge distillation for machine translation; today it is just called "training on synthetic data," and it is how most "distilled" open models were actually made.

Why does it work when it throws away the soft targets? Because the teacher's choice of sequence is itself a strong signal. Its outputs are cleaner, more consistent and easier to model than human-written data, so the student learns them faster. And it sidesteps every practical obstacle: no logits to store, no shared tokenizer required, the teacher can even be an API you do not control.

The downside is exactly the hard-label problem: per token, the student sees one sample from the teacher's distribution, not the distribution. It learns the teacher's mode, and it takes many samples to learn the teacher's spread.

Logit-level distillation: matching every token's distribution

The full Hinton recipe applied per position: run teacher and student on the same prefix, and at each position minimize $\text{KL}(p_t \| q_t)$ over the vocabulary. Per token you get a $|\mathcal V|$-dimensional target instead of a single index, which is orders of magnitude more information. Gemma 2 (2024) reports using this at pretraining scale for its smaller models, distilling from a larger Gemma on trillions of tokens, and attributes much of the small models' quality to it.

Two costs. First, compute: every training token needs a teacher forward pass, which for a 27B teacher is bigger than the student's training step. (You can precompute and store top-$k$ logits, at a storage cost.) Second, the teacher and student must share a vocabulary, since the KL is over the same set of tokens. We come back to that.

Which prefixes? On-policy distillation

There is a subtle question hiding in "run both on the same prefix": whose prefixes? If you use prefixes from a fixed dataset (or from the teacher's own samples), the student learns what to do in situations the teacher would be in. But at inference the student is in situations the student got itself into, including its own mistakes, and it has never been taught how to recover from those. This is the exposure-bias problem, and it is the same one that motivates RL over SFT in preference optimization.

Agarwal et al. (2023), "Generalized Knowledge Distillation" (GKD), address it by sampling prefixes from the student and having the teacher label them: for each student-generated sequence, compute the teacher's distribution at every position and minimize a divergence toward it. This is on-policy distillation. It is like RL with the teacher's per-token log-probabilities as a dense reward, and it fixes the train/test mismatch. GKD also lets you choose the divergence, which brings us to the most important design decision.

Forward vs reverse KL: covering vs seeking

KL is not symmetric. $\text{KL}(p\|q)$ and $\text{KL}(q\|p)$ are different numbers and, when the student is too small to match the teacher exactly, they produce different students.

Forward KL, $\text{KL}(p\|q) = \sum_i p_i\log(p_i/q_i)$, is what we wrote above. It is weighted by the teacher's $p$. Wherever the teacher has mass, $q$ had better have mass too, or $\log(p/q)$ explodes. So the student is forced to cover every mode of the teacher, and if it cannot fit all of them it spreads out and sits in between. This is mode-covering (or mean-seeking). Note that minimizing it equals maximum likelihood on the teacher's samples, which is why plain SFT on synthetic data has the same character.

Reverse KL, $\text{KL}(q\|p) = \sum_i q_i\log(q_i/p_i)$, is weighted by the student's $q$. The student is punished wherever it puts mass that the teacher does not have. So the safest thing for a small student is to pick one mode of the teacher and sit on it precisely, ignoring the others. This is mode-seeking.

For a generative model, mode-seeking is usually what you want. A student that averages two good answers produces a bad answer; a student that commits to one of them produces a good answer. Gu et al. (2023), "MiniLLM", make exactly this argument and distill with reverse KL, optimized with a policy-gradient-style estimator since you must sample from the student to estimate it. GKD reports that reverse KL and a symmetric variant (generalized Jensen–Shannon) generally beat forward KL for on-policy distillation of generation tasks, while forward KL is fine when the student is large enough to cover the teacher.

The vocabulary mismatch problem

Logit-level distillation assumes the teacher's and student's softmax outputs are over the same tokens. Distilling Llama into Qwen, or any model into one with a different tokenizer, breaks that: the two models tokenize the same text into different pieces, so there is no position-by-position correspondence and no shared index to compare probabilities at.

Fixes come in three flavors. (1) Give up on logits and use sequence-level distillation, which only needs text. This is the common answer. (2) Align the tokenizations: find spans where both tokenizers agree on boundaries (which happens often, at word boundaries) and distill only at those positions, or map both vocabularies to a shared space (e.g. bytes). (3) Use a loss that does not need aligned indices: Boizard et al. (2024) propose the Universal Logit Distillation loss, which compares the sorted probability vectors of teacher and student via an optimal-transport-style distance, sidestepping token identity entirely. Later work (dual-space distillation, approximate-likelihood matching) refines the alignment approach. None is as clean as sharing a tokenizer, which is why model families (Gemma, Llama, Qwen) distill within the family.

Intermediate-Layer and Feature Distillation

Everything so far matches outputs. You can also match insides. TinyBERT (Jiao et al. 2019) trains the student to reproduce the teacher's hidden states and attention maps layer by layer (with a learned linear map when the widths differ), on top of the output loss; DistilBERT (Sanh et al. 2019) initializes the student from every other layer of the teacher and adds a cosine loss on hidden states. The intuition is that the intermediate representations are a much denser teaching signal than the final distribution. This works well when the student is a shrunk copy of the teacher's architecture (same depth pattern, narrower or shallower), and it is the basis of "prune then distill" pipelines like Minitron (Muralidharan et al. 2024), where the student literally starts as a pruned teacher. It is rarely used across architecture families, where there is no natural layer correspondence.

Distilling Reasoning

The most visible use of distillation in 2025 was transferring chain-of-thought reasoning. The DeepSeek-R1 report (2025) describes taking roughly 800k reasoning traces generated by R1 (long, step-by-step solutions to math, code and logic problems, filtered for correctness) and fine-tuning off-the-shelf Qwen and Llama models from 1.5B to 70B on them with plain SFT. As reported, the resulting "R1-Distill" models substantially outperformed the same base models trained with RL directly, and the 32B distilled model was competitive with much larger models on math and code benchmarks.

Notice what kind of distillation this is: sequence-level, with hard labels, and with no logits at all. The teacher's contribution is the traces themselves. The lesson people drew is that the expensive part of teaching a model to reason is discovering good reasoning behavior (which R1 did with large-scale RL), and once discovered, that behavior is cheap to copy into smaller models by imitation. Whether the copies reason "as well" or mostly reproduce the surface form is still debated; they are clearly worse at generalizing to new domains than the teacher, but far better than their untouched base models.

Distillation vs Pruning vs Quantization

Three different tools get lumped together as "model compression." They are not substitutes; they are often combined. See the quantization chapter for the details of the third.

A Practical Recipe

Here is what a working distillation run looks like in 2026, in the order you would decide things.

1. Pick the mode. Different tokenizer, or teacher behind an API: sequence-level (generate, filter, SFT). Same family and you can afford teacher forward passes: logit-level, ideally with some on-policy student samples mixed in.
2. Prompts matter more than anything. The student only learns the teacher's behavior on the inputs you show it. Cover the deployment distribution, and generate several samples per prompt when doing sequence-level.
3. Filter. For tasks with checkable answers, keep only correct teacher outputs (rejection sampling). This is most of what made the R1 distillation data good.
4. Loss. Logit-level: forward KL at $T = 1$–$2$ on teacher-prefix data, or reverse/JSD on student-sampled data. Full-vocabulary KL is better than top-$k$ but top-$k$ (e.g. $k=128$) with a renormalization loses little and is 100× cheaper to store.
5. Hyperparameters are those of SFT: learning rate around $10^{-5}$ for full fine-tuning (higher with LoRA, see the LoRA chapter), a few epochs over the synthetic set, cosine decay, and an eval on held-out prompts scored by the teacher or a judge.

import torch, torch.nn.functional as F

def kd_loss(student_logits, teacher_logits, targets=None, T=2.0, alpha=0.0, reverse=False):
    """student_logits, teacher_logits: (B, L, V). targets: (B, L) hard labels or None.
    Returns alpha * CE(hard) + (1 - alpha) * T^2 * KL(soft)."""
    log_q = F.log_softmax(student_logits / T, dim=-1)
    log_p = F.log_softmax(teacher_logits / T, dim=-1)
    if reverse:   # KL(q || p): weighted by the student
        kl = (log_q.exp() * (log_q - log_p)).sum(-1)
    else:         # KL(p || q): weighted by the teacher
        kl = (log_p.exp() * (log_p - log_q)).sum(-1)
    soft = (T * T) * kl.mean()
    if targets is None or alpha == 0.0:
        return soft
    hard = F.cross_entropy(student_logits.flatten(0, 1), targets.flatten(), ignore_index=-100)
    return alpha * hard + (1 - alpha) * soft

The companion code/lumen/distill.py has this loss, a top-$k$ variant that renormalizes over the teacher's top tokens, and a small sequence-level generator that samples a teacher and writes an SFT dataset for code/lumen/train.py.

A Note on Ethics and Terms of Service

Distilling from a model you run yourself is uncontroversial and is how every major lab builds its small models; distilling from a third party's API is a different matter: several providers' terms of service prohibit using outputs to train competing models, the legal status of such training is unsettled and varies by jurisdiction, and the research community is still debating whether the practice is a healthy form of knowledge diffusion or a free ride on someone else's compute, so read the terms and treat "distilled from X" as a claim with consequences.

Practice

Further reading

• Hinton, Vinyals & Dean (2015). Distilling the Knowledge in a Neural Network. Short, readable, the origin of temperature and dark knowledge.
• Kim & Rush (2016). Sequence-Level Knowledge Distillation. Why training on teacher samples works for sequence models.
• Gu et al. (2023). MiniLLM: Knowledge Distillation of Large Language Models. The reverse-KL argument for generative students.
• Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD). Student-sampled prefixes and a choice of divergences.
• Boizard et al. (2024). Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs. Logit distillation across vocabularies.
• Jiao et al. (2019). TinyBERT: Distilling BERT for Natural Language Understanding. Layer-wise feature distillation.
• Muralidharan et al. (2024). Compact Language Models via Pruning and Knowledge Distillation (Minitron). Prune, then distill, at LLM scale.
• Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. Logit distillation at pretraining scale, as reported.
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Section on distilled models.