Pre-Training: Overview
By the end of this chapter you will be able to sketch the whole pre-training pipeline on a whiteboard, estimate what a training run costs from three numbers, and know exactly which later chapter answers each of the big design questions.
Here is the problem. In GPT-2 from scratch you built a transformer and loaded someone else's weights into it. Those weights knew English, Python, a bit of chemistry, and how to finish a limerick. Your freshly initialized model knows nothing: every logit is noise. How do you get from random numbers to a model that knows things?
The answer is almost embarrassingly simple to state and enormously expensive to do. You show the model trillions of tokens of text and ask it, over and over, to predict the next one. That is pre-training. Everything else in this part of the course, seven more chapters, is about doing that one simple thing well at a scale where every mistake costs millions of dollars.
This chapter is the map. We will look at the recipe from ten thousand feet, trace the lifecycle of a training run from raw data to a released checkpoint, separate what pre-training buys you from what it does not, and build a cost calculator so the numbers in later chapters feel real. Each key decision gets a pointer to the chapter that goes deep on it.
The recipe at a glance
Strip away the engineering and a modern pre-training run has four ingredients.
- A giant corpus. Trillions of tokens of text, mostly scraped from the web, plus code, books, papers and reference material. Llama 3 reportedly trained on about 15 trillion tokens (Grattafiori et al., 2024).
- A decoder-only transformer. The architecture you built in the Architecture part, with a handful of modern refinements (RMSNorm, RoPE, SwiGLU, grouped-query attention) that we cover in the next chapter.
- Next-token prediction. One loss function: the cross-entropy of the model's prediction for token $t+1$ given tokens $1 \dots t$. Every position in every sequence is a training example.
- Weeks on thousands of GPUs. The compute budget is the thing that makes pre-training different from every other kind of machine learning you have done. Llama 3 405B reportedly used about 16,000 H100s for months.
The surprising part is that this is enough. No labels, no task-specific heads, no curriculum of hand-designed puzzles. Predicting the next token, done at scale, forces the model to learn grammar, facts, some reasoning, and the ability to pick up a new task from a few examples in its context.
Why does "predict the next word" teach so much? Because the next word depends on everything. To predict the last word of "The capital of France is", you need geography. To predict the next token of a Python function, you need to track variable names and indentation. To predict a mystery novel's final chapter, you need to have followed the plot. Any regularity in text that helps prediction is a regularity the model is rewarded for learning.
What the loss actually is
Let us write down the single quantity the whole enterprise minimizes, so the symbols in later chapters have a home. The model, with parameters $\theta$, assigns a probability to each next token. We want that probability to be high for the token that actually came next, averaged over every position in the corpus.
$$\mathcal{L}(\theta) = -\frac{1}{D}\sum_{i=1}^{D} \log p_\theta\!\left(x_i \mid x_{<i}\right)$$Here $D$ is the number of tokens in the training set and $x_{<i}$ means "all the tokens before position $i$". Each term is the negative log-probability the model gave to the correct next token. Lower is better. A loss of $2.0$ nats means that, on average, the model gave the true next token a probability of $e^{-2} \approx 13.5\%$. Every plot of "training loss" you will see in this course is this number.
Three tokens, a tiny vocabulary. The model sees "the cat" and must predict the third token. Suppose it puts probability $0.5$ on "sat", $0.3$ on "ran", $0.2$ on "is", and the true token is "sat". The loss at this position is $-\log 0.5 = 0.693$. If the true token had been "is", the loss would be $-\log 0.2 = 1.609$. Pre-training is just this, summed over a few trillion positions and pushed downhill by gradient descent.
Symbols
$N$ = parameters$D$ = training tokens
$C$ = compute in FLOPs
$\theta$ = the weights
$\mathcal{L}$ = next-token loss
Collect
Crawl, clean, dedupe and filter a corpus of $D$ tokens. Tokenize it once and shard it to disk.Configure
Choose $N$ (width, depth, vocabulary) and the optimizer, learning-rate schedule and batch size.Train
Stream batches, compute $\mathcal{L}$, backpropagate, update. Repeat for $D / \text{batch}$ steps. Save checkpoints.Evaluate & release
Track loss and benchmarks throughout; anneal on high-quality data; publish the final checkpoint.The lifecycle of a training run
A pre-training run is not one program; it is a pipeline of stages, each of which is its own engineering discipline. Here is the whole thing on a whiteboard. Read it left to right.
Click through the stages below to see what each one does, what its failure modes are, and where it is covered.
Each stage lights up with a summary of what happens there and what goes wrong when it is done badly.
What pre-training gives you
It is worth being precise about what falls out of next-token prediction, because the list is both longer and shorter than people expect.
Knowledge
A base model has read more text than any human ever will. It knows dates, chemical formulas, the plot of every public-domain novel, and how a thousand programming libraries are used. This knowledge is stored in the weights, mostly in the MLP layers, and it is retrieved by pattern completion. Ask a base model to complete "The Eiffel Tower is located in" and it will say "Paris" not because it was taught a fact, but because that continuation minimized the loss.
In-context learning
This was the surprise of GPT-3 (Brown et al., 2020). Put a few examples of a task in the prompt, then a new input, and the model performs the task without any weight update. Nobody trained for this. It emerges because text on the internet is full of patterns like lists, tables, and Q&A pairs, and predicting the next item in a pattern requires inferring the pattern.
Reasoning-ish behaviors
Base models can do multi-step arithmetic, follow chains of logic, and write code that runs, especially if you prompt them to "think step by step". The honest framing is that pre-training gives a model the raw material for reasoning: it has seen millions of worked solutions. How much of that is genuine reasoning versus sophisticated pattern matching is a live research debate, and we take it up in the scaling laws chapter under "emergent abilities".
What pre-training does not give you
A base model is a text completer, not an assistant. This matters enough that a whole part of the course, Post-Training, exists to fix it.
- Instruction following. Ask a base model "Write me a poem about autumn" and a common completion is "Write me a poem about winter. Write me a poem about spring." It has learned that such requests appear in lists. It has not learned to answer them. Fixing this is supervised fine-tuning.
- Safety and refusals. The web contains instructions for everything. A base model will happily complete any of it. The behaviors people associate with chat models, declining harmful requests, hedging on uncertain claims, come from preference optimization.
- Knowing when to stop. A base model does not know your prompt was a question. It keeps generating until it hits an end-of-text token or the length limit, often drifting into a new "document".
- Calibrated self-knowledge. A base model has no notion of "I" and no reliable sense of what it does not know.
People say "GPT-4 was trained on the internet" as if that explains its chat behavior. It does not. Pre-training on the internet gives you a model that continues the internet. The helpful, polite, question-answering persona is layered on afterwards with orders of magnitude less compute. Pre-training builds capability; post-training shapes behavior.
The key decisions and where they are covered
Every pre-training run is defined by a handful of decisions. Here they are, with the question each one answers and the chapter that goes deep.
| Decision | The question | Chapter |
|---|---|---|
| Objective and architecture | Causal LM or something else? Which block design (norms, activation, attention variant, vocabulary)? | pre-02 |
| Scale and optimization | Given a compute budget, how big a model and how many tokens? Which optimizer, learning rate, schedule, batch size? | pre-03 |
| Data | Where do trillions of tokens come from, how are they cleaned, and in what proportions are sources mixed? | pre-04 |
| Systems | How do you split one model across thousands of GPUs and keep them all busy? What about failures? | pre-05 |
| Extra objectives | Beyond next-token: multi-token prediction, fill-in-the-middle, long-context stages, and other additions. | pre-06 |
| Evaluation | How do you know the run is going well before it finishes? Which numbers to watch, which to distrust? | pre-07 |
| Putting it together | A full walk through one real, documented run. | pre-08 (Case Study: Llama 3) |
A cost primer: how many FLOPs does training take?
Before any of those decisions can be made you need one number: how much compute the run will consume. Happily there is a rule of thumb so simple you can do it in your head, and it is accurate to within tens of percent for every dense transformer ever trained.
Deriving 6ND
Think about a single weight $w$ sitting inside a matrix multiplication somewhere in the network, and a single token flowing through. What work does that weight cause?
In the forward pass, the weight gets multiplied by one input activation and the product gets added into an output. One multiply, one add: 2 FLOPs.
In the backward pass, two things need computing. First, the gradient with respect to the input activation (so the gradient can keep flowing to earlier layers): one multiply-add against the weight, 2 FLOPs. Second, the gradient with respect to the weight itself (so we can update it): one multiply-add of the input activation with the output gradient, another 2 FLOPs. Backward costs twice the forward: 4 FLOPs.
Total: 6 FLOPs per weight per token. A model with $N$ weights processing $D$ tokens therefore does about:
$$C \approx 6\,N\,D \quad \text{FLOPs}$$That is the whole formula. It ignores the attention score computation ($QK^\top$ and the softmax-weighted sum over $V$), which is not a "per parameter" cost but scales with context length. For typical context lengths of a few thousand tokens and models with billions of parameters, that term is a small correction (Kaplan et al., 2020, treat it as a few percent). It also ignores the final softmax over the vocabulary, which is already counted because the output projection is a weight matrix.
$N = 7\times 10^{9}$, $D = 2\times 10^{12}$. Then $C \approx 6 \times 7\times 10^{9} \times 2\times 10^{12} = 8.4 \times 10^{22}$ FLOPs. Written out, that is 84 sextillion floating-point operations. Keep this number in mind; we are about to turn it into GPU-days and dollars.
GPU throughput and MFU
A FLOP count is useless until you know how fast you can execute FLOPs. An NVIDIA H100 SXM has a peak dense bf16 throughput of roughly $10^{15}$ FLOP/s (the datasheet figure is about 989 TFLOP/s without sparsity). An A100 is about $3.1\times 10^{14}$. Newer chips are higher; treat any specific number here as the order of magnitude, not gospel.
But no training run hits peak. The GPU spends time waiting on memory, on communication with other GPUs, and on small kernels that cannot saturate the chip. The fraction of peak you actually achieve, measured as useful model FLOPs divided by peak FLOPs, is called Model FLOPs Utilization (MFU), a term from the PaLM paper (Chowdhery et al., 2022).
$$\text{MFU} = \frac{6ND / T}{\text{peak FLOP/s} \times \text{GPUs}}$$where $T$ is the wall-clock training time in seconds. Read it as: the FLOPs the model needed divided by the FLOPs the hardware could have delivered in that time. Good large-scale runs achieve 35 to 45 percent. Llama 3 reported 38 to 43 percent on 16k H100s (Grattafiori et al., 2024). Reaching 50 percent is excellent; anything below 30 means something is broken.
We need $8.4\times 10^{22}$ FLOPs. At 40% MFU on an H100 delivering $10^{15}$ FLOP/s peak, each GPU gives us $0.4 \times 10^{15} = 4\times 10^{14}$ useful FLOP/s.
Seconds of GPU time: $8.4\times 10^{22} / 4\times 10^{14} = 2.1\times 10^{8}$ GPU-seconds.
Divide by 3600: about $58{,}000$ GPU-hours. Divide by 24: about $2{,}400$ GPU-days.
On 1,024 GPUs that is roughly 2.4 days of wall-clock time. On 256 GPUs, about 9.5 days. At a rental price of \$2 per H100-hour, the bill is around \$117,000.
Sanity check against reality: Llama 2 7B was trained on 2T tokens and reportedly used 184,320 A100-hours (Touvron et al., 2023). Plugging in: $8.4\times 10^{22} / (184{,}320 \times 3600) \approx 1.27\times 10^{14}$ FLOP/s per GPU, which is 41% of an A100's $3.1\times 10^{14}$ peak. The formula and the reported numbers agree.
Two traps. First, "FLOPs" as a count of operations versus "FLOP/s" as a rate; keep the "per second" straight or your estimates will be off by many orders of magnitude. Second, NVIDIA's marketing numbers often quote sparse throughput, which is double the dense figure and does not apply to normal training. Use the dense bf16 number.
The cost calculator
Now build intuition by moving the sliders. Start with the 7B on 2T example, then ask: what does it cost to double the model? To double the tokens? To go from 20% MFU to 40%? The last one is why systems engineers are paid well.
Watch how wall-clock time and dollars change. The bars show days of training for several cluster sizes; the highlighted bar is your current GPU count.
A few things you should have noticed. Cost is linear in $N$ and linear in $D$, so the product $ND$ is the thing that matters, and the question "should I train a bigger model or on more data?" is exactly the question scaling laws answer. And MFU is a pure multiplier on cost: a team that lifts MFU from 30% to 45% has cut a \$10M run to \$6.7M without changing the model at all.
A short history of notable runs
The numbers below are drawn from the respective papers and are approximate. Some of them (GPT-2's token count in particular) are estimates because the original reports gave dataset size in gigabytes rather than tokens. Treat every entry as "roughly".
| Model | Year | Params (approx.) | Tokens (approx.) | Tokens / param | Note |
|---|---|---|---|---|---|
| GPT-2 | 2019 | 1.5B | ~10B (40 GB WebText; token count estimated) | ~7 | Showed zero-shot transfer from web text (Radford et al., 2019). |
| GPT-3 | 2020 | 175B | 300B | ~2 | In-context learning at scale (Brown et al., 2020). |
| Gopher | 2021 | 280B | 300B | ~1 | DeepMind; later shown to be under-trained (Rae et al., 2021). |
| Chinchilla | 2022 | 70B | 1.4T | ~20 | Same compute as Gopher, better results: the "20 tokens per parameter" rule (Hoffmann et al., 2022). |
| Llama 1 | 2023 | 7B to 65B | 1.0T to 1.4T | ~20 to 140 | Small models trained far past Chinchilla-optimal for cheap inference (Touvron et al., 2023a). |
| Llama 2 | 2023 | 7B to 70B | 2T | ~30 to 290 | Longer context, GQA on the 70B (Touvron et al., 2023b). |
| Llama 3 | 2024 | 8B, 70B, 405B | ~15T | ~40 to 1,900 | 128k vocabulary, 8B trained on ~1,900 tokens per parameter (Grattafiori et al., 2024). |
Read the tokens-per-parameter column top to bottom and you see the field's arc in a single number. GPT-3 and Gopher spent their compute on parameters. Chinchilla showed that was a mistake and that tokens and parameters should grow together. Then Llama went the other way on purpose: small models drowned in data, because a model you will serve a billion times should be cheap to run, even if that means training it "inefficiently". The scaling laws chapter makes this trade-off quantitative.
The idea that unsupervised pre-training on text, followed by task-specific fine-tuning, beats training from scratch was established by Radford et al. (2018) with GPT-1 and, in parallel, Devlin et al. (2018) with BERT. GPT-2 (2019) removed the fine-tuning step and showed that a large enough model prompted with plain text could do many tasks zero-shot. GPT-3 (2020) scaled that by 100x and the modern era began.
Why this is hard
If the recipe is so simple, why does it take a hundred-person team and a paper the length of a novel? Because every quantity is at the edge of what is possible.
Data at 15T tokens means you have processed most of the usable public web and are now fighting over the last few trillion tokens of quality text. Filtering decisions that move benchmark scores by a point are worth millions.
Compute at $10^{25}$ FLOPs means a single wasted week is a seven-figure line item, and a bug discovered on day 40 of a 60-day run is an existential question: restart, or patch and pray?
Systems at 16,000 GPUs means a hardware failure every few hours, on average. The run must checkpoint fast, restart fast, and never stall.
Hyperparameters cannot be swept at full scale. You get one shot. Everything must be predicted from small-scale experiments, which is why scaling laws and hyperparameter transfer (μP) are not academic curiosities but the tools that make a single-shot run survivable.
The next seven chapters take these one at a time.
Practice
Llama 3 405B was reportedly trained on about 15.6T tokens. Using $C = 6ND$, compute the total FLOPs. Then, assuming 16,000 H100s at $10^{15}$ FLOP/s peak and 40% MFU, estimate the wall-clock days. Compare with the ~54-day figure reported in the paper for the main run and explain any gap.
Solution sketch
$C = 6 \times 4.05\times 10^{11} \times 1.56\times 10^{13} \approx 3.8\times 10^{25}$ FLOPs. Useful throughput: $16{,}000 \times 10^{15} \times 0.4 = 6.4\times 10^{18}$ FLOP/s. Time: $3.8\times 10^{25} / 6.4\times 10^{18} \approx 5.9\times 10^{6}$ s $\approx 69$ days. The reported ~54 days is in the same ballpark; differences come from the exact MFU (reported 38 to 43%), the exact peak figure used, the fact that not every stage ran on the same cluster size, and the attention FLOPs we ignored. Being within 30% from a back-of-the-envelope formula is the point.
Using code/lumen/gpt2.py and code/lumen/train.py, train the smallest GPT-2 configuration on a few hundred steps of the tiny corpus in code/lumen/data.py. Count parameters $N$ (exclude the embedding matrix if you want to match how papers report non-embedding parameters), measure tokens per second, and compute your MFU against your GPU's dense bf16 peak (or fp32 on CPU). Then double the batch size and measure again.
Solution sketch
$N$ is sum(p.numel() for p in model.parameters()). Tokens per second is batch × sequence length × steps / elapsed seconds. MFU is $6 N \times \text{tokens/s} / \text{peak}$. On a laptop you will see single-digit percentages: small matrices cannot saturate the hardware. Doubling batch size usually improves MFU because each kernel does more work per launch. This is the first lesson of the systems chapter: utilization is about keeping the matrix multiplications big.
Load pretrained GPT-2 with code/lumen/gpt2.py and code/lumen/sampling.py. Prompt it with "Q: What is the capital of France?\nA:" and sample ten continuations at temperature 0.8. Then prompt it with "Write a haiku about autumn." and sample ten more. Classify each output: did the model answer, or did it continue the "document"?
Solution sketch
The Q/A format usually works, because Q/A pairs are a common pattern on the web and "A:" strongly cues an answer. The bare instruction usually fails: expect more instructions, a list, or a web-page-like continuation. This is the "what pre-training does not give you" section made concrete. In the Post-Training part you will fix it.
Key takeaways
- Pre-training is next-token prediction on trillions of tokens with a decoder-only transformer, run for weeks on thousands of GPUs.
- It produces knowledge, in-context learning and raw reasoning ability; it does not produce instruction following or safety. Those come from post-training.
- Training compute is about $6ND$ FLOPs: 2 for the forward pass, 4 for the backward, per parameter per token.
- MFU (useful FLOPs over peak) is the efficiency knob; 40% is good, and it multiplies cost directly.
- A 7B model on 2T tokens is about $8.4\times 10^{22}$ FLOPs, roughly 58k H100-hours at 40% MFU, a couple of days on a thousand GPUs.
- The field moved from "spend compute on parameters" (GPT-3) to "balance them" (Chinchilla) to "over-train small models for cheap inference" (Llama).
Further reading
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners. The GPT-2 report: web text plus scale gives zero-shot ability.
- Brown et al. (2020). Language Models are Few-Shot Learners. GPT-3 and the discovery of in-context learning at scale.
- Kaplan et al. (2020). Scaling Laws for Neural Language Models. Where the $C \approx 6ND$ accounting is laid out.
- Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. Chinchilla.
- Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways. Introduces the MFU metric (Appendix B).
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. Reports the GPU-hours used in the sanity check above.
- Grattafiori et al. (2024). The Llama 3 Herd of Models. The most detailed public account of a frontier pre-training run; the subject of the case study chapter.