Training Data Engineering

Two teams train the same 7B architecture with the same optimizer on the same number of tokens. One model is noticeably better at everything. The difference is the data. Scaling laws from the last chapter tell you how many tokens to use, but they say nothing about which tokens, and by 2024 it had become clear that quality and mix matter as much as scale.

The problem is that the raw material is the public web, and the public web is mostly garbage: boilerplate, navigation menus, SEO spam, machine-translated junk, the same press release copied ten thousand times, and, scattered among it, a few percent of text worth learning from. Data engineering is the discipline of finding that few percent at petabyte scale without accidentally throwing away the good parts.

Where the data comes from

Every large training set is a mix of a handful of source types. Their character differs a lot, and so does how much of each exists.

• Web crawl. Common Crawl is a nonprofit that has crawled the web since 2008 and publishes snapshots of raw HTML; a recent snapshot is a few billion pages and on the order of 100 TB of compressed WARC files. Nearly all web data in LLMs starts here. It is the largest source by far, and the dirtiest.
• Code. Public repositories from GitHub, deduplicated and license-filtered (The Stack, Kocetkov et al., 2022; The Stack v2 at about 900B tokens). Code teaches precise syntax, long-range structure and, apparently, reasoning.
• Books. Project Gutenberg (public domain), plus, in older datasets, shadow-library collections whose use has been litigated. Long, coherent, edited prose.
• Papers. arXiv, PubMed Central, Semantic Scholar's peS2o. Dense technical text and math.
• Reference. Wikipedia and its siblings. Small (English Wikipedia is only about 4B tokens) but very high quality, so it is often repeated several times.
• Math. Math-heavy web pages (OpenWebMath, Paster et al., 2023, about 15B tokens), textbooks, competition problems.
• Synthetic. Text generated by another model: textbook-style explanations, rewritten web pages, instruction data. Small in most pre-training mixes, growing fast.

Proportions in known open datasets

The table gives rough composition by tokens for four influential open datasets. The numbers are approximate; each paper reports them slightly differently (by bytes, by documents, or by effective tokens after upsampling), so treat them as the shape of the mix, not a spec.

Two things to notice. Even in the curated datasets, web text is the majority, because nothing else exists in the needed quantity. And the high-quality sources (Wikipedia, books, papers) are tiny relative to a 15T-token budget, which is why the mixing section below is mostly about how many times you can repeat them.

The pipeline

Here is the whole thing, from crawl to shards. Each box is a stage that a real team owns, tunes, and ablates. We walk through them in order.

Text extraction

A crawled page is HTML: navigation bars, cookie banners, ads, footers, and somewhere in the middle, the article. Extraction pulls out the main content. Tools like trafilatura and resiliparse do this with heuristics about tag structure and text density. Common Crawl also ships a pre-extracted "WET" text format, but it keeps far more boilerplate; RefinedWeb and FineWeb both found that re-extracting from raw HTML with trafilatura produced measurably better models, and it is now standard. This one stage discards most of the bytes: HTML markup and boilerplate are the bulk of a page.

Language identification

A fastText classifier (Joulin et al., 2016) assigns each document a language and a confidence. English-only datasets keep documents scoring above about 0.65 for English; multilingual ones keep a set of languages with per-language thresholds. This is where roughly half of the remaining documents go, because roughly half the web is not in English.

URL and blocklist filtering

Before reading any content, drop domains on known adult, spam, malware and piracy lists (RefinedWeb used a 4.6M-domain blocklist plus URL-word scoring). Also honor opt-outs: domains that ask not to be crawled for AI training, and sites that have requested removal. This stage is cheap and removes a few percent, but a poorly built blocklist causes harm: Dodge et al. (2021) showed that C4's word-based filter disproportionately removed text about and by minority groups.

Quality filters: heuristics

The problem now is that the remaining documents are in English and not obviously spam, but many are still junk: lists of product codes, pages that are 90% bullet points, auto-generated text. Heuristic filters encode what junk looks like. The Gopher rules (Rae et al., 2021) are the canonical set, and every later pipeline uses some variant:

• Keep documents with 50 to 100,000 words, mean word length 3 to 10 characters.
• Drop if the symbol-to-word ratio (for # or ...) exceeds 0.1.
• Drop if more than 90% of lines start with a bullet, or more than 30% end with an ellipsis.
• Require that at least 80% of words contain an alphabetic character.
• Require at least two of the stop words the, be, to, of, and, that, have, with.
• Drop documents with excessive repetition: duplicate lines, paragraphs, or frequent n-grams above a threshold.

C4 (Raffel et al., 2020) adds line-level rules: keep only lines ending in terminal punctuation with at least five words, drop pages containing "lorem ipsum" or a curly brace (code) or any word from a bad-word list. FineWeb's ablations found that a few extra rules of this kind (fraction of lines ending in punctuation, fraction of short lines) each gave a small but real gain.

Quality filters: model-based

Heuristics can tell junk from prose, but not a great explanation from a mediocre one. The next step is to train a classifier for "quality" and keep the high scorers. GPT-3 did this with a linear classifier trained to distinguish WebText from raw crawl. FineWeb-Edu (Penedo et al., 2024) did it more directly: ask Llama 3 70B to rate 460k web pages on a 0 to 5 scale of "educational value for grade-school to college level", train a small embedding-based classifier on those labels, and keep pages scoring 3 or higher. That kept about 1.3T of 15T tokens and produced large gains on knowledge and reasoning benchmarks (MMLU, ARC) at the same token budget. Llama 3 reports the same approach with its own classifiers.

Deduplication

The web copies itself. Press releases, syndicated news, license boilerplate, product descriptions and Wikipedia mirrors appear thousands of times each. Lee et al. (2021) found that in C4, a corpus that was already filtered, over 3% of the tokens sat in near-duplicate documents, and that a single 61-word sentence appeared over 60,000 times.

Why it matters

Three reasons, in increasing order of seriousness. First, wasted compute: training on the same document twice teaches less than training on two different ones. Second, skewed distribution: a passage that appears 60,000 times is being upweighted 60,000x relative to what you intended. Third, memorization: Lee et al. showed that deduplication cut the rate at which models emitted training text verbatim by about 10x, and Carlini et al. (2022) showed that memorization of a sequence grows roughly log-linearly with the number of times it appeared. Memorized text is a privacy problem (Kandpal et al., 2022), a copyright problem, and a sign the model is doing lookup instead of learning. Lee et al. also found that deduplicated models reached the same accuracy in fewer steps.

Exact deduplication

The easy part: hash every document (or every line, or every 50-token span) and drop repeats. Line-level exact dedup catches boilerplate ("Click here to subscribe") that appears inside otherwise-unique pages. Exact matching misses the common case, though: the same article with a different byline, date, or footer.

Near-duplicate detection with MinHash

The problem: to find near-duplicates you want to compare every pair of documents, and with a billion documents that is $10^{18}$ comparisons. MinHash (Broder, 1997) is the trick that makes it linear. It has three steps.

Shingles. Represent each document as the set of its $n$-grams (for text, typically 5-word shingles; in the toy below we use 3-word shingles). Two documents that share most of their sentences share most of their shingles, regardless of small edits.

Jaccard similarity. The natural similarity between two sets is the size of their intersection over the size of their union.

A Jaccard of 1 means identical shingle sets; 0 means nothing shared. Pipelines usually call documents near-duplicates above about 0.7 to 0.8.

MinHash. Computing $J$ exactly for every pair is still quadratic. But here is the trick: pick a random hash function $h$, and for each document record only the minimum hash value over its shingles. The probability that two documents have the same minimum equals their Jaccard similarity, exactly:

Why: the minimum over $A \cup B$ lands on some shingle; the two minima agree exactly when that shingle is in both sets, and by symmetry of a random hash each shingle of the union is equally likely to be the minimum, so the chance is $|A \cap B| / |A \cup B|$. Use $k$ independent hash functions and you get a $k$-number signature per document; the fraction of positions where two signatures agree is an unbiased estimate of $J$ with standard error about $\sqrt{J(1-J)/k}$. Then, instead of comparing all pairs, the signatures are chopped into bands and hashed into buckets (locality-sensitive hashing); only documents that collide in some bucket are compared. FineWeb uses 5-word shingles and 112 hash functions in 14 bands of 8, which flags pairs above roughly 0.75 similarity.

PII, toxicity and decontamination

PII and toxicity

Regular expressions catch the easy personal information: email addresses, phone numbers, IP addresses, which are replaced with placeholder tokens (Dolma does this). Toxicity is scored by a classifier and documents above a threshold are dropped or downweighted. Both stages are small in volume and large in consequence, and both involve trade-offs: a too-aggressive toxicity filter removes discussions about harmful topics along with harmful content, which then hurts the model's ability to recognize and refuse them later.

Decontamination against benchmarks

The problem: benchmark test sets are on the web. If MMLU questions are in the training data, the evaluation score is meaningless. Decontamination searches the corpus for overlap with every benchmark you plan to report and removes the matching documents. GPT-3 used 13-gram overlap; Llama and most current pipelines use 8- or 10-gram token overlap with a threshold on the fraction of the benchmark example covered. It is imperfect (paraphrases slip through, and it is only as good as your list of benchmarks), which is one reason the evaluation chapter spends time on held-out and freshly written evals.

From text to batches

Tokenization

The clean corpus is tokenized once, up front, with the BPE tokenizer from the tokenization chapter, and stored as arrays of integer ids. This is the point at which "how many tokens do we have" becomes a definite number, and it depends on the tokenizer: the same text is 15% fewer tokens with Llama 3's 128k vocabulary than with Llama 2's 32k.

Mixing and sampling weights

Now the sources come together. You have, say, 12T tokens of filtered web, 900B of code, 200B of math, 100B of books, 4B of Wikipedia, and a budget of 15T tokens. You assign each source a sampling weight and the loader draws batches accordingly. The weight determines two things: what fraction of the model's training is spent on that source, and how many times that source gets repeated.

Llama 3's reported mix is roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code, and 8% multilingual, arrived at through scaling-law experiments on small models with different mixes. Upweighting code and math well past their natural share of the web is now standard, because both improve reasoning benchmarks out of proportion to their volume.

Epochs and repetition: how many times can you reuse data?

The problem behind the mixing sliders: high-quality data is scarce, and budgets have outrun it. Can you just repeat it? Muennighoff et al. (2023) trained models with fixed compute on corpora of varying size, forcing repetition, and fit a scaling law for the effective value of repeated tokens. The finding: up to about 4 epochs, repeated data is nearly as good as fresh data; the loss is barely distinguishable. Beyond that, returns fall off quickly, and by around 40 epochs additional repeats are worth almost nothing. So repeating Wikipedia four times is fine and common; repeating it forty times just overfits.

They also found that when data is the constraint, it is better to train a somewhat smaller model for more epochs than the Chinchilla rule would suggest, and that mixing in code helps fill the gap without much penalty on natural-language evaluations.

Curriculum and the annealing phase

Data order matters too. Most of a run uses a fixed mix, but two adjustments have become standard. First, long documents are held back for the long-context stage at the end (see pre-02). Second, and more important, the final stretch of training, during the learning-rate decay, is run on an upsampled high-quality mix: more code, more math, more curated sources, and sometimes instruction-style data. This is called annealing. Llama 3 reports that annealing on a small amount of high-quality math data during the last 40M tokens of the 8B run improved GSM8K and MATH substantially, and they used the same trick as a cheap way to evaluate candidate data sources (Blakeney et al., 2024, study this directly). MiniCPM (Hu et al., 2024) mixes high-quality and SFT data into the decay phase of its WSD schedule for the same reason. The intuition: the model's final weights are determined disproportionately by what it sees while the learning rate is small, so that is where the best data should go.

Packing sequences and document masking

The trainer wants fixed-length sequences (8,192 tokens, say). Documents are not that length. The standard solution is to concatenate all documents with an end-of-text token between them and chop the stream into fixed windows. A window then usually contains the tail of one document, several whole ones, and the head of another.

The catch: with a plain causal mask, tokens of document 3 attend to tokens of documents 1 and 2 in the same window, which are unrelated. The model learns to ignore them, but it is wasted attention and can cause subtle problems. The fix is a document mask: in addition to the causal mask, block attention across document boundaries, so each token attends only within its own document. Llama 3 uses this and reports it matters little in standard pre-training but is important in the long-context stage, where a single 128k window can hold dozens of documents. Efficient attention kernels support it as a "variable-length" mode that also skips the blocked-out computation.

Sharding

Finally the token stream is written as many fixed-size shards (memory-mapped binary files of a few hundred MB to a few GB each) with an index, so that thousands of data-loader workers can each read their assigned slice without coordination, and so that a restart from a checkpoint can resume at exactly the right token. Shuffling happens at the document level before packing and at the shard level during loading; the exact order is recorded so the run is reproducible. The systems chapter picks up from here.

The filter funnel

Put the numbers together and the pipeline is a funnel. The retention percentages below are illustrative, modeled on what RefinedWeb and FineWeb report for English web data; real values depend on the crawl, the language, and how aggressive each stage is set. The point is the shape: two stages (extraction and language ID) throw away most of the bytes, a model-based quality filter throws away most of the rest, and what remains is around 1% of what you started with.

A snapshot of 100 TB of crawl ends up as a few hundred billion tokens of training text. Since Common Crawl has published around a hundred snapshots (with heavy overlap between them, which dedup removes), that is how a 15T-token web corpus is assembled.

Synthetic data

If quality data is scarce, why not generate it? The Phi series (Gunasekar et al., 2023, "Textbooks Are All You Need") trained small models largely on model-written textbook-style text and exercises and got strong results on coding benchmarks, and Cosmopedia and similar datasets have made the recipe open. Synthetic data has clear advantages: you control the topic distribution, the format, and the quality; you can target gaps (rare languages, step-by-step math); and it does not run out.

The risks are just as clear. Synthetic text carries the generating model's biases, errors and stylistic tics, and a model trained on it inherits them. Trained recursively, model on model on model, the distribution narrows: Shumailov et al. (2024) call this model collapse, showing that the tails of the distribution (rare facts, unusual phrasings) disappear first, and each generation forgets a little more. The current consensus is that synthetic data is a supplement, valuable in targeted amounts and in the annealing phase, and that it must be mixed with, not substituted for, real data, with the generating model kept out of the loop of its own evaluation.

Licensing and ethics

A short section, because the issues are real and unsettled, and this course is not the place to settle them.

On one side: web-scale training uses text that was published to be read, and the models learn from it in a way that does not reproduce most of it; most jurisdictions have not ruled definitively, and several (Japan, the EU with an opt-out) permit text and data mining in some form. Common Crawl respects robots.txt, and pipelines increasingly honor opt-out signals.

On the other side: authors and publishers did not consent to this use, the models can and sometimes do reproduce training text verbatim, and the commercial value created is enormous and flows to the model builders. Several lawsuits (the New York Times against OpenAI and Microsoft, filed in December 2023, among others) are working through exactly these questions. The Pile's Books3 component, drawn from a shadow library, was removed after takedown requests, and datasets built since have been more careful about provenance: Dolma documents the source and license of every component, and the Stack respects repository licenses and offers opt-out.

Practically: know where every byte came from, keep provenance metadata through the whole pipeline, honor opt-outs, deduplicate (which reduces verbatim reproduction), and be honest in the model card. What you can legally do and what you should do are different questions, and the answer to the second one is a judgment your team has to own.

Companion code

code/lumen/data.py contains a small but complete version of the bottom of this pipeline: a tiny corpus, a loader that tokenizes it, packs documents into fixed-length sequences with end-of-text separators, builds the document mask, and shards the result. It also has a minimal MinHash implementation you can run on the toy corpus. The packing logic is the part people most often get wrong, so here it is.

import torch

def pack(docs, seq_len, eos_id):
    """docs: list of lists of token ids. Returns (tokens, doc_ids) of shape [n_seq, seq_len]."""
    stream, ids = [], []
    for d, doc in enumerate(docs):
        stream.extend(doc + [eos_id])
        ids.extend([d] * (len(doc) + 1))
    n = len(stream) // seq_len                      # drop the ragged tail
    tokens = torch.tensor(stream[: n * seq_len]).view(n, seq_len)
    doc_ids = torch.tensor(ids[: n * seq_len]).view(n, seq_len)
    return tokens, doc_ids

def document_mask(doc_ids):
    """[n_seq, L] -> [n_seq, L, L] bool: True where query i may attend key j."""
    L = doc_ids.size(1)
    causal = torch.tril(torch.ones(L, L, dtype=torch.bool))
    same_doc = doc_ids[:, :, None] == doc_ids[:, None, :]
    return causal[None] & same_doc

toks, dids = pack([[5, 6, 7], [8, 9], [10, 11, 12]], seq_len=8, eos_id=0)
print(toks, document_mask(dids)[0].int(), sep="\n")

Three block-diagonal triangles, one per document. The second document's block is rows 4 to 6; the eighth token is the start of document 3 and can see only itself.

Practice

Further reading

• Penedo et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. The most transparent account of a web pipeline, with ablations per stage; read this first.
• Penedo et al. (2023). The RefinedWeb Dataset for Falcon LLM. Filtered and deduplicated web alone can match curated mixes.
• Soldaini et al. (2024). Dolma: an Open Corpus of Three Trillion Tokens. Open toolkit and full provenance.
• Gao et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. The first widely used curated mix.
• Rae et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. The Gopher quality rules are in Appendix A.
• Lee et al. (2021). Deduplicating Training Data Makes Language Models Better. Why dedup matters, with numbers.
• Carlini et al. (2022). Quantifying Memorization Across Neural Language Models. Memorization grows with duplication and model size.
• Muennighoff et al. (2023). Scaling Data-Constrained Language Models. The four-epoch result.
• Blakeney et al. (2024). Does your data spark joy? Performance gains from domain upsampling at the end of training. Annealing as both a training trick and an evaluation method.
• Grattafiori et al. (2024). The Llama 3 Herd of Models. Section 3 covers the data mix, annealing and document masking.
• Dodge et al. (2021). Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. What blocklist filtering actually removed from C4.
• Gunasekar et al. (2023). Textbooks Are All You Need. Synthetic textbook data for small models.
• Shumailov et al. (2024). The Curse of Recursion: Training on Generated Data Makes Models Forget. Model collapse.