Evaluation During Pretraining

Your training loss is going down. Great. But loss is one number, averaged over trillions of tokens of everything from recipes to Rust. It tells you the model is getting better at predicting text on average. It does not tell you whether it can do arithmetic, recall facts, follow a few-shot pattern, or write a function that passes its tests. Two models with the same loss can differ a lot on those. So what is the model actually good at, and how would you know?

This chapter is about answering that question honestly. There are three layers. The first is the loss itself and how to make it comparable. The second is benchmarks: what they measure and, more importantly, the mechanics of how a base model is scored on them. The third is statistics: how much of a benchmark difference is real.

Held-out loss and perplexity

The cheapest evaluation is the one you already have: the next-token loss on a held-out set the model never trained on. Compute it on a few million tokens, sampled from the same mix as training, and you get an unbiased estimate of how well the model predicts unseen text.

Perplexity is the same number in a different unit. If the average loss per token is $\mathcal{L}$ nats, then

and the interpretation is "the model is as uncertain, on average, as if it were choosing uniformly among PPL options". A loss of 2.3 nats is a perplexity of about 10: at each step, the model is roughly as confused as a ten-way coin flip. Lower is better, and the scale is exponential, so a drop from 3.0 to 2.9 nats is a 10% drop in perplexity.

Pitfalls: tokenizer, context, and domain

Here is the catch that trips people up. Perplexity is per token, and tokens are whatever your tokenizer says they are. A tokenizer with a bigger vocabulary chops text into fewer, longer tokens. Each of those tokens is harder to predict, so the per-token loss is higher, even if the model is exactly as good at predicting the underlying text. Two models with different tokenizers cannot be compared on per-token perplexity at all.

Two more pitfalls. Perplexity depends on how much context each token gets: evaluating with 512-token windows gives worse numbers than evaluating with 4k windows, because early tokens in each window have little to condition on. And it depends heavily on the domain of the held-out text: perplexity on Wikipedia and perplexity on GitHub are different quantities. Always state the tokenizer, the context length, and the evaluation corpus.

Bits per byte: the tokenizer-independent measure

The fix for the tokenizer problem is to charge the model per byte of raw text rather than per token. Total information is total information; how you chunked it should not matter. Convert the loss from nats to bits (divide by $\ln 2$), then multiply by how many tokens there are per byte:

What just happened: the model's total surprise on the document is the loss per token times the number of tokens. Dividing that total by the number of bytes gives surprise per byte, which no longer cares about the tokenizer. Bits per byte was popularised for LLM evaluation by the Pile paper (Gao et al. 2020) for exactly this reason.

The companion code/lumen/eval.py computes held-out loss with a sliding window and reports nats per token, perplexity, and bits per byte, so that you can compare runs that use different tokenizers from the very first chapter.

Benchmarks: what they measure and how a base model is scored

Loss says how well the model predicts text. Benchmarks say whether it can do particular things. Each standard benchmark is a dataset plus a scoring rule, and the scoring rule matters as much as the data. There are two families of scoring for a base model, and you need both in your head before reading any leaderboard.

Log-likelihood scoring. For multiple-choice tasks, do not ask the model to answer. Instead, for each candidate answer, compute the log-probability the model assigns to that answer text given the question, and pick the candidate with the highest score. No generation, no parsing, works for a raw base model that has never been taught to answer questions.

Generation scoring. For tasks with a free-form answer (a number, a program), sample a completion from the model and check it: exact match against the reference, or run the tests. This needs the model to stop at the right place and format its answer, which base models are bad at unless you show them examples.

That is where few-shot prompting comes in. Prepend $k$ solved examples in the same format, and the base model continues the pattern. Five-shot MMLU means five example questions with answers precede the real one. The examples are fixed and drawn from a separate development split, so they are not the test questions.

MMLU

Massive Multitask Language Understanding (Hendrycks et al. 2020): about 14,000 four-choice questions across 57 subjects, from elementary mathematics to professional law. It measures broad world knowledge and reading. The standard base-model protocol is 5-shot with log-likelihood scoring of the answer letter: the prompt ends with "Answer:" and the model's log-probabilities for " A", " B", " C", " D" are compared. Random is 25%.

HellaSwag

Zellers et al. (2019): commonsense sentence completion. Given the start of a short scenario, choose the most plausible of four endings, where the wrong endings were adversarially generated to fool models. About 10,000 validation examples. Scored by log-likelihood of each full ending, usually length-normalized (more on that below). Random is 25%; it climbs early and smoothly in training, which makes it a useful signal for small models.

ARC

The AI2 Reasoning Challenge (Clark et al. 2018): grade-school science questions, split into an Easy set and a Challenge set of questions that simple retrieval methods got wrong. Four (occasionally three or five) options, scored by log-likelihood of the answer text, typically length-normalized, often 25-shot. ARC-Challenge is the one people report; it is the benchmark Llama 3 used as the target for its downstream scaling predictions.

GSM8K

Grade School Math 8K (Cobbe et al. 2021): about 8,500 word problems requiring two to eight arithmetic steps, with 1,319 in the test split. This is generation scoring: the model writes a chain-of-thought solution and the final number is extracted and compared exactly. The standard base-model protocol is 8-shot with worked solutions in the prompt. Base models score near zero for a long time in training and then climb, which makes it a late-moving and noisy signal.

HumanEval

Chen et al. (2021): 164 hand-written Python programming problems, each a function signature plus docstring, scored by running hidden unit tests on the generated function body. The metric is pass@k: the probability that at least one of $k$ samples passes. With $n$ samples of which $c$ pass, the unbiased estimator is $1 - \binom{n-c}{k}/\binom{n}{k}$. Zero-shot, generation, and with only 164 problems, very noisy.

BBH

BIG-Bench Hard (Suzgun et al. 2022): 23 tasks from BIG-Bench on which earlier models scored below the human average, covering logic, multi-step arithmetic, and tricky language. Scored by generation with 3-shot chain-of-thought prompts and exact match of the final answer. It is a reasoning-flavoured benchmark that base models do poorly on until they are large.

Scoring mechanics: multiple choice via log-likelihoods

Let us do the log-likelihood scoring properly, because a subtle choice inside it changes results by several points.

For each option $o_i$ with tokens $o_{i,1},\dots,o_{i,n_i}$, the model gives a log-probability of the whole option given the prompt:

Pick $\arg\max_i s_i$. This is the "acc" metric in the lm-evaluation-harness. The problem: $s_i$ is a sum of negative numbers, so a longer option is penalised simply for having more tokens. If the correct answer is "the water evaporates into the atmosphere" and a wrong one is "it rains", the wrong one has a head start of several nats.

Length normalization divides by the length, giving an average per-token (or per-byte) log-likelihood:

This is "acc_norm". The harness normalizes by the option's byte length rather than token count, to keep it tokenizer-independent, but the idea is the same. Neither choice is "right": raw likelihood is correct if the options are the same length; normalized is fairer when they differ. The important thing is to know which one a reported number used, because they can differ by five points or more on HellaSwag and ARC.

Contamination and decontamination

Now for the uncomfortable problem. The training set is a large fraction of the internet. The benchmarks are on the internet. If the test questions, or paraphrases of them, are in the training data, the score measures memory, not ability.

The standard defence is n-gram overlap decontamination. For each benchmark example, check whether any n-gram of it (GPT-3, Brown et al. 2020, used 13-grams; Llama 3 reports an 8-gram analysis) appears in the training corpus. Either remove the matching training documents before training, or flag the contaminated test examples and report scores on the clean subset. Llama 3 reports both the contamination rate per benchmark and the estimated performance gain from contamination, which is the honest way to present it.

N-gram matching misses paraphrases, translations, and questions that appear with reformatted whitespace, so it is a floor on contamination, not a measurement of it. It also cannot catch the subtler effect of benchmark-style synthetic data (see the advanced objectives chapter) that was never a literal copy of any test item.

Benchmarks saturate, and the move to harder sets

A benchmark stops being useful when the best models reach its ceiling, either because they are genuinely that good or because the annotation errors in the test set cap the achievable score. HellaSwag and ARC-Easy are near that point for frontier models; MMLU is close. The field responds by moving to harder sets: GPQA (Rein et al. 2023), graduate-level science questions written so that experts score around 65% and skilled non-experts with web access around 34%; MATH (Hendrycks et al. 2021), 12,500 competition problems; MMLU-Pro with ten options and harder questions. The treadmill will continue. For a small model you train yourself, saturation is not your problem; the opposite one is, and we come to it next.

Evaluation as a curve during training

A single evaluation at the end tells you where you landed. Evaluating every few thousand steps tells you the shape of the journey, and the shape is where the information is. But not all metrics are equally useful along the way.

Which metrics move early, and which are noise

Loss and bits per byte move smoothly from step one. Log-likelihood benchmarks with large test sets (HellaSwag, ARC-Easy, PIQA, LAMBADA) start climbing above chance early and rise steadily, so they are good progress signals even for models under a billion parameters. MMLU stays at chance (25%) for a long stretch and then lifts; for small models it is a flat line and tells you nothing. Generation benchmarks with exact match (GSM8K, HumanEval) sit near zero for most of a small run, and when they do move, they jump around from checkpoint to checkpoint because a formatting quirk can swing dozens of examples.

Practical rule: for tracking during training, watch loss, BPB, and two or three large log-likelihood benchmarks. Run the expensive generation benchmarks at milestones, and report them with error bars.

"Emergence" as a choice of metric

You will read that some abilities "emerge" suddenly at scale: flat at small sizes, then a sharp jump (Wei et al. 2022). Schaeffer et al. (2023) argued that much of this is an artefact of the metric. Exact match on a multi-digit arithmetic problem is all-or-nothing: the model gets zero credit until every digit is right, so a model whose per-digit accuracy improves smoothly shows a sudden jump in exact match once the product of per-digit accuracies crosses a threshold. Measure the same models with a continuous metric, such as token-level edit distance or the log-likelihood of the right answer, and the curve is smooth.

Small-scale proxies and scaling ladders

Before spending the compute on a large run, you want to know what it will do. The tool is a scaling ladder: train a sequence of small models with the same recipe, measure them, fit a trend, and extrapolate. The scaling-laws chapter does this for loss. Doing it for benchmarks is harder, because benchmark accuracy is bounded, noisy, and often flat at small scale.

Grattafiori et al. (2024) describe a two-step approach for Llama 3 (Section 3.2.1). First, fit how the negative log-likelihood of the correct answer on a benchmark decreases with training compute, using models from about 40M to 16B parameters trained with between $6\times10^{18}$ and $10^{22}$ FLOPs. That quantity is continuous and moves smoothly even when accuracy does not. Second, fit a sigmoid from that log-likelihood to accuracy, using the small models and older Llama 2 models as data points. Chain the two and you get a predicted accuracy for the 405B model on ARC-Challenge before training it; they report the prediction was close to the final result. The key move is the one from the previous section: predict a smooth quantity, then map it to the jumpy one.

Downstream versus upstream: does low loss transfer?

Loss is the "upstream" quantity; benchmarks are "downstream". Usually a lower loss means better benchmarks, and the relationship is tight enough that people use loss as the primary target. Gadre et al. (2024) showed that average downstream error across a suite of tasks is well predicted by held-out loss across a wide range of model sizes and over-training ratios.

But not always, and the exceptions are instructive. Two models trained on different data mixes can have the same held-out loss on a shared evaluation set and very different GSM8K scores, because one saw far more math. Loss on a broad mix averages over domains, and the benchmark asks about one. A model can also have lower loss and worse benchmark scores if the loss gain came from memorizing boilerplate. And a few tasks show inverse scaling, where larger models with lower loss do worse, typically because they imitate a common but wrong pattern more faithfully (McKenzie et al. 2023). Loss is the compass; benchmarks are the map; use both.

Practical evaluation

The lm-evaluation-harness

You do not have to implement any of the above yourself. EleutherAI's lm-evaluation-harness (Gao et al. 2021 onward) implements hundreds of tasks with fixed prompts, few-shot examples, and scoring rules, and runs them against any model that can return log-likelihoods or generations. It is the reference implementation behind most published base-model numbers and the Open LLM Leaderboard. Biderman et al. (2024), "Lessons from the Trenches", is the maintainers' account of why reproducible evaluation is hard: prompt formatting, tokenization of few-shot separators, and normalization choices all move scores.

Seeds and variance

Several sources of randomness sit inside an evaluation number. Which few-shot examples were chosen and in what order. The sampling temperature for generation tasks. The training seed of the model itself: two runs with identical hyperparameters and different seeds can differ by a point or two on MMLU-scale benchmarks and far more on HumanEval. When you compare two recipes, ideally compare across seeds, and at minimum fix everything but the recipe.

Confidence intervals: the binomial formula

Even with everything fixed, a benchmark score is an estimate of a "true" accuracy from a finite sample of $n$ questions. Each question is a coin flip with probability $p$ of success. The standard error of the observed accuracy $\hat p$ is:

That is the normal approximation to a binomial proportion. It says the uncertainty shrinks with the square root of the number of questions, so four times the questions halves the interval. For small $n$ or accuracies near 0 or 1, the Wilson interval is more accurate, but the formula above is what you should be able to do in your head.

Reporting

A useful evaluation report states, for every number: the benchmark and split, the number of examples, the shots and where they came from, the scoring rule and normalization, the decoding settings for generation tasks, the harness version, and a confidence interval or a standard error. If you compared two models, say whether the difference exceeds the interval. That is more words than a table cell, but a table of bare percentages without them is closer to marketing than measurement.

Practice

Further reading

• Gao et al. (2020). The Pile. Popularised bits per byte for cross-tokenizer comparison.
• Hendrycks et al. (2020). Measuring Massive Multitask Language Understanding (MMLU).
• Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?
• Clark et al. (2018). Think you have Solved Question Answering? Try ARC.
• Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K).
• Chen et al. (2021). Evaluating Large Language Models Trained on Code (HumanEval, pass@k).
• Suzgun et al. (2022). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (BBH).
• Rein et al. (2023). GPQA; Hendrycks et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset.
• Wei et al. (2022). Emergent Abilities of Large Language Models; Schaeffer et al. (2023). Are Emergent Abilities of Large Language Models a Mirage?
• Gadre et al. (2024). Language Models Scale Reliably with Over-Training and on Downstream Tasks; McKenzie et al. (2023). Inverse Scaling: When Bigger Isn't Better.
• Grattafiori et al. (2024). The Llama 3 Herd of Models. Section 3.2.1 for the two-step downstream prediction; Section 5 for evaluation and contamination analysis.
• Brown et al. (2020). Language Models are Few-Shot Learners. The 13-gram decontamination appendix.
• Biderman et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models; the lm-evaluation-harness repository.