Case Study: Llama 3

Most frontier models are described in a blog post. "The Llama 3 Herd of Models" (Grattafiori et al. 2024) is a 92-page paper that says what was decided and, unusually, why. That makes it the best available textbook example of the whole pretraining pipeline, and the reason this part of the course ends with it.

We will read it as a sequence of decisions, one per course topic: architecture, data, scaling laws, infrastructure, training stages, post-training, evaluation. For each: what did they do, what did they say the reason was, and what would you do at your scale? Every specific number below is as reported in the paper, with the section it comes from; where the paper gives an approximation, so do we.

The model family and the central bet

The paper describes three dense decoder-only transformers of 8B, 70B, and 405B parameters, each pre-trained and then post-trained into an instruct model, with 128k-token context and support for eight languages. The 405B model is the flagship; the 8B and 70B are, in their words, "the best models at their scale" at release, and they were trained on the same data and recipe so that lessons transfer across sizes.

The authors frame their approach around three levers: data, scale, and managing complexity. On the third one they are explicit: they chose a standard dense architecture with minor changes over a mixture-of-experts model in order to maximize training stability, and a relatively simple post-training procedure over more complex reinforcement-learning algorithms for the same reason. "Managing complexity" is a theme worth carrying through the whole chapter, because it is the one that most transfers to a small run.

The configuration table

Table 3 of the paper (Section 3.2) gives the shapes. Reported values:

Notice three regularities. The head dimension is 128 in every model (dimension divided by heads). The FFN dimension is 3.5× the model dimension throughout, the usual SwiGLU choice that keeps parameter count near the 4× of a classic MLP with its three matrices. And the peak learning rate falls as the model grows, roughly as the inverse square root of the width, which is the pattern the scaling-laws chapter predicts.

Architecture: what they chose and what they refused

Section 3.2 says the architecture is essentially Llama 2's, and lists the changes. Grouped-query attention with 8 key/value heads on all three sizes, for inference speed and a smaller KV cache (the attention chapter). A 128k-token vocabulary, combining 100k tokens from the tiktoken tokenizer with 28k extra tokens for non-English languages; they report this improved compression from 3.17 to 3.94 characters per token on English, so the same compute reads more text. The RoPE base frequency raised to 500,000 to support long context (positional encoding chapter). And an attention mask that stops tokens attending across document boundaries within a packed sequence, which they report had limited effect at 8k but mattered for the long-context stage.

Everything else is the standard recipe of the objectives-and-architecture chapter: pre-norm with RMSNorm, SwiGLU MLPs, no biases, dense attention over every layer.

What they explicitly did not do

No mixture of experts. The paper says they opted for a standard dense transformer rather than MoE "to maximize training stability". At the time, several competitors had gone sparse; Meta chose the option with fewer failure modes when training for months on 16,000 GPUs. No exotic attention variants, no novel optimizers, no learned positional schemes. The architectural bet was to spend the novelty budget on data and scale and keep the model boring.

The data pipeline

Section 3.1 is where the effort went. The headline: roughly 15 trillion multilingual tokens, versus 1.8 trillion for Llama 2, with a knowledge cutoff at the end of 2023. The data engineering chapter described the general toolkit; here is how it was applied.

Cleaning and de-duplication

The web pipeline reported in Section 3.1.1 starts with filtering out domains known to contain personal data or adult content, then extracts text with a custom HTML parser tuned to keep math and code, which they say beat generic extractors. De-duplication runs at three levels: URL-level (keep the newest crawl of each page), document-level (MinHash near-duplicate detection across the corpus), and line-level, removing lines that appeared more than six times in buckets of 30 million documents, which strips navigation menus, cookie banners, and boilerplate. They also apply heuristics: duplicated-n-gram ratios to catch logging and error text, "dirty word" counting, and a token-distribution divergence check against the training-corpus distribution to drop outliers.

Quality classifiers

On top of the heuristics, model-based filters. A fastText classifier trained to recognise text that Wikipedia references. A more expensive RoBERTa-based classifier trained on quality labels produced by Llama 2, prompted to judge whether a document met quality requirements. Separate pipelines for code and for reasoning-heavy pages, with classifiers trained to find "math deduction, reasoning in STEM areas, and code interleaved with natural language". The lesson for the small-scale learner: quality classification is cheap relative to training, and the classifiers can be bootstrapped from a previous-generation model.

The mix

Section 3.1.3 reports the final data mix as roughly 50% general-knowledge tokens, 25% mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens. Two things stand out. Math and reasoning at a quarter of the mix is high compared to earlier open models. And the mix was not chosen by intuition: they report knowledge classification of the web data (to downsample over-represented categories like entertainment) and scaling-law experiments in which small models were trained on candidate mixes to predict the large model's benchmark performance on each.

Annealing on high-quality data

Section 3.1.3 (annealing data) reports that upsampling small amounts of high-quality code and mathematical data while annealing the learning rate at the end of training boosted the 8B model's GSM8K and MATH scores substantially, with negligible effect on the 405B model. They also turned this into a tool: to judge whether a new data source is valuable, anneal a partially trained 8B model on it for 40 billion tokens with 30% of the mix from the candidate, and see what moves. That is a cheap experiment any lab, or student, can copy.

Scaling laws: choosing 405B and 15.6T

Section 3.2.1 is a compact worked example of the scaling-laws chapter. Given a compute budget of about $3.8\times10^{25}$ FLOPs, how big should the model be and how many tokens should it see?

They report training models from 40M to 16B parameters over compute budgets from $6\times10^{18}$ to $10^{22}$ FLOPs, each with a range of token counts, and fitting the compute-optimal token count as a power law in compute: $A^\star(C) = A\,C^{\alpha}$ with reported fit $(\alpha, A) = (0.53, 0.29)$. Extrapolated to $3.8\times10^{25}$ FLOPs, this gave a compute-optimal model of about 402B parameters on 16.55T tokens. They chose 405B and about 15.6T tokens, and note the flatness of the optimum: modest changes in size cost little, so they leaned toward a slightly larger model for robustness to the extrapolation.

Predicting the benchmarks in advance

The same section describes the two-step downstream prediction discussed in the evaluation chapter: fit the negative log-likelihood of the correct answer on a benchmark against training FLOPs using the small models, then fit a sigmoid from that log-likelihood to accuracy using the small models plus Llama 2 models, and chain them to predict the 405B model's ARC-Challenge accuracy. They report the prediction was accurate. This is the practice that lets a lab commit tens of millions of dollars with some confidence about the outcome.

Compute and infrastructure

Section 3.3 is the concrete version of the systems chapter. Reported facts: the 405B model was trained on up to 16,384 H100 GPUs (80 GB HBM3, 700 W), in Meta's production clusters, with servers of eight GPUs connected by NVLink; the cluster network was RoCE Ethernet at 400 Gbps per GPU; and a distributed storage system offering 240 PB with sustained throughput of 2 TB/s (7 TB/s peak) absorbed checkpoints, which are trillions of bytes per checkpoint for the 405B model, so that checkpoint writes did not stall training.

4-D parallelism

Section 3.3.2 describes tensor, context, pipeline, and data parallelism stacked in that order from innermost to outermost. For 8k context on 16,384 GPUs the reported degrees are TP 8, CP 1, PP 16, DP 128; for the 128k stage they report TP 8, CP 16, PP 16, DP 8. Data parallelism used FSDP with weights sharded but not resharded after forward, to avoid an extra all-gather during backward. They describe several pipeline-schedule fixes (flexible micro-batch counts per stage, balancing the first and last stages that carry the embedding and output layers, and asynchronous send/receive) that together they report improved 405B throughput by around 10%.

Utilization and reliability

Reported BF16 MFU across their configurations: 38 to 43%, with per-GPU throughput of 380 to 430 TFLOP/s. And the numbers on failures, from Section 3.3.4: over a 54-day snapshot, 466 job interruptions, 419 of them unexpected; about 78% of the unexpected ones attributed to hardware, with GPU faults the largest category at 58.7%; effective training time over 90%; only three interruptions needing significant manual work. They also describe silent data corruption, thermal throttling that varied throughput by 1 to 2% over the day, and power swings of tens of megawatts when thousands of GPUs idled and resumed simultaneously.

The staged training recipe

Section 3.4 describes three pretraining stages followed by post-training. The reported numbers, and what changes at each stage, are worth having as a timeline.

Initial pre-training

Two details from Section 3.4.1 that matter beyond the numbers. First, the batch-size ramp: starting at 4M tokens and 4k context, then 8M tokens and 8k context, then 16M tokens, each step taken after the model had stabilized. They report very few loss spikes and no need for interventions to correct divergence. Second, they changed the data mix during training based on what evaluations showed, which tells you that the mix is not sacred and that evaluation during training (the evaluation chapter) is an input to decisions, not just a report card.

Long context and annealing

Both stages are described in the advanced objectives chapter; the timeline above carries the reported numbers. The design principle they share is "do the expensive or delicate thing late": long context costs quadratic attention and heavy communication, so it is trained last on a small fraction of tokens; the highest-quality data has most effect when the learning rate is small, so it is concentrated in the final 40M tokens.

Post-training in summary

Section 4 describes turning the base models into Llama 3 Instruct. The reported approach is iterative: six rounds, each of reward modeling on human preference annotations, supervised fine-tuning on a mix of human-written and synthetic examples (much of it generated by the previous round's model and filtered by the reward model, i.e. rejection sampling), and direct preference optimization on preference pairs. They report choosing DPO over on-policy RL such as PPO because it needed less compute and was easier to keep stable, and they averaged models obtained with different data or hyperparameters at each stage. Specific capabilities such as coding, multilinguality, math, long context, tool use, factuality, and steerability each got dedicated data work. The SFT chapter and the preference optimization chapter take these apart.

Evaluation highlights, with caveats

Section 5 reports the pre-trained and post-trained models on the standard benchmarks. A few reported post-trained numbers to anchor the scale: MMLU (5-shot) of about 69 for 8B, 84 for 70B, and 87 for 405B; GSM8K in the mid-80s for 8B and mid-90s for the larger two; HumanEval in the 70s for 8B and around 89 for 405B; GPQA around 51 for 405B. The paper positions the 405B model as comparable to the leading closed models of mid-2024 across these tasks.

The honest caveats, most of which the paper itself raises. The numbers are from Meta's own harness and prompts, so they are not directly comparable with other papers' numbers for the same benchmarks (Section 5.1 discusses this and reports contamination analysis with per-benchmark rates and estimated gains). Several benchmarks are near saturation for the 405B model, so differences at the top are within noise. Human evaluations, which they report as pairwise comparisons against other models, depend heavily on the prompt distribution. And "reported" is the operative word throughout: these are the authors' measurements of their own models.

Lessons for a learner training a small model

You will not train on 16,000 GPUs. So which of these decisions transfer to a 100M-parameter model on one card? More than you would think, because the authors' own smaller experiments are what justified most of the choices.

What transfers down

Data quality over data quantity. The three-level de-duplication and the classifier-based filtering are the highest-leverage part of the recipe, and they are cheap. For a small model with a small corpus, a day spent de-duplicating and filtering beats a day spent on architecture.

Annealing on the best data. The 8B model gained substantially from annealing on high-quality math and code; the 405B did not. The effect is strongest at small scale, which is exactly where you are. Hold back your best data for the end, decay the learning rate to zero over it, and average the last few checkpoints.

A stable, simple architecture. Pre-norm RMSNorm, SwiGLU, RoPE, GQA, no biases, no tricks. It is what code/lumen/gpt2.py and code/lumen/block.py already give you, and it is what a frontier lab chose when stability mattered most.

A careful learning rate. Warmup, a peak scaled down as the model grows, cosine decay to a small floor, and a batch-size ramp early. Llama 3 reports almost no loss spikes at 405B with this recipe. code/lumen/train.py implements the schedule.

Predict before you spend. Run a ladder of tiny models, fit the loss trend, and use annealing experiments to test data sources, before committing your compute to the final run.

What does not transfer

The parallelism strategy, the checkpointing infrastructure, and the failure-handling automation are solutions to problems you do not have at one GPU. Long-context extension to 128k is a large-model concern. And absolute benchmark targets are meaningless at small scale: a 100M model will score at chance on MMLU no matter how well it is trained, so evaluate with loss, bits per byte, and easy log-likelihood benchmarks as the evaluation chapter advises.

Practice

Further reading

• Grattafiori et al. (2024). The Llama 3 Herd of Models. The paper this chapter reads; Sections 3 and 4 are the relevant ones for this course.
• Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. The previous generation, useful for seeing what changed.
• Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models. Where the architecture recipe (pre-norm, SwiGLU, RoPE) was assembled.
• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla). The scaling-law methodology Llama 3 adapts.
• Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.
• Rafailov et al. (2023). Direct Preference Optimization. The post-training algorithm they chose for stability.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. A contemporaneous report that made the opposite architectural bet (MoE, fp8), good to read side by side.