Introduction

Type a half-finished sentence into a modern language model and it finishes it. Ask it a question and it answers in fluent paragraphs. How can a program do that? There is no grammar table inside, no dictionary of facts, no list of rules. What is actually running?

Here is the honest answer, and it is smaller than most people expect. A large language model is a single mathematical function. It takes a sequence of tokens (pieces of text) and outputs one probability distribution: what comes next? That is the whole job.

Everything else, the essays, the code, the conversations, comes from calling that function over and over. Predict the next token, pick one, glue it on, predict again. This loop is called autoregression, and it is the first idea to get into your bones. The rest of this course is about what happens inside the function.

The one function that does everything

Let's make the claim precise. Write the tokens seen so far as $x_1, x_2, \dots, x_t$. The model is a function $f_\theta$ with parameters $\theta$ (millions or billions of numbers that were learned from data). Here is what it computes:

Read it as: "the probability of each possible next token, given the tokens so far, is whatever $f_\theta$ outputs." The output is a vector with one entry per token in the vocabulary, all entries non-negative and summing to one. Nothing about the sentence's meaning is stored anywhere explicitly. Meaning is whatever the parameters need to encode in order to be good at this guessing game.

Why is this one question enough? Because the probability of a whole sequence factorises into a product of next-token probabilities. This is just the chain rule of probability, nothing deep:

So a model that can answer "what comes next?" perfectly is, in principle, a model that assigns the right probability to every possible text. Being a good next-token predictor is the same thing as being a good model of language. That is why this humble objective scales to such surprising behaviour.

Autoregression: the loop that writes

The function on its own produces nothing readable. It is the loop around it that writes text. Here is the loop in full, and it really is this short:

Two things in that loop matter more than they look. First, the model's own output becomes its next input. That is why it can go off the rails: one unlucky token early on changes everything after it. Second, the choice in Step 2 is where "creativity" lives. Always picking the most likely token gives dull, repetitive text; sampling with a bit of randomness gives variety. The Learning to Predict chapter and code/lumen/sampling.py cover temperature, top-k and top-p in detail.

Try the loop yourself. The model below is deliberately tiny: fifteen tokens and a hand-written table of "how often does token B follow token A" (a bigram table, with a couple of three-token overrides). It is not a neural network. But the loop around it is exactly the loop around GPT-4, and watching it run makes autoregression concrete.

The whole pipeline on one whiteboard

Now let's open the function. What happens between "a string of text goes in" and "a probability for every token comes out"? There are seven stages. Each one gets its own chapter in this part of the course; here we only need the shape.

Stage 1: text → tokens

Neural networks eat numbers, not letters. A tokenizer chops the input string into pieces from a fixed vocabulary (GPT-2's has 50,257 entries) and replaces each piece with its integer id. "the cat sat" might become [464, 3797, 3332]. The pieces are learned by a compression-like algorithm called byte-pair encoding, and the choices it makes leak into model behaviour in odd ways. All of that is the Tokenization chapter.

Stage 2: tokens → embeddings (+ position)

An integer id is not something you can do calculus with. So the model keeps a big table with one row of $d_{model}$ numbers per vocabulary entry (768 numbers for GPT-2 small) and looks up the row for each token. These learned rows are embeddings. The model also needs to know where each token sits, because "dog bites man" and "man bites dog" contain the same tokens. Position information is added to the embedding. The Embedding Layer and Positional Encoding chapters cover both.

Stage 3: N transformer blocks

Now every token is a vector of $d_{model}$ numbers. The stack of transformer blocks refines those vectors, one block at a time, so that by the end the vector at position $t$ contains everything needed to predict token $t+1$. Every block has two halves. Attention lets each token look at the others and pull in relevant information ("I am 'sat', and there is a 'cat' two places back, so the subject is an animal"). The MLP then processes each token vector on its own, which is where a lot of factual knowledge seems to live. Both halves add their result to the vector rather than replacing it, so early information survives to the end. That is the residual connection, and it is the reason deep stacks train at all. Attention is the Attention chapter; the MLP, normalisation and residual stream are the Layers of Understanding chapter.

Stage 4: unembedding

After the last block, the vector at the final position is multiplied by a $V \times d_{model}$ matrix to give one raw score per vocabulary entry. These scores are called logits. Often this matrix is literally the same table used for embedding in stage 2, read in the other direction; that trick is called weight tying, and the Embedding Layer chapter explains when it helps.

Stage 5: softmax over the vocabulary

Logits can be any real numbers. To turn them into probabilities we exponentiate each one and divide by the total, so the result is positive and sums to one. This is the softmax function, and it appears again inside attention, so it is worth knowing well:

Here $z_i$ is the logit for token $i$ and $p_i$ is its probability. Bigger logits get exponentially more mass, so softmax is a "soft" version of picking the maximum, hence the name. With logits $[2, 1, 0]$: $e^2 \approx 7.39$, $e^1 \approx 2.72$, $e^0 = 1$, total $11.11$, probabilities $[0.67, 0.24, 0.09]$. Check that they sum to one.

Stage 6: sample

Pick a token from the distribution. Greedy decoding always takes the top one. Sampling draws at random in proportion to the probabilities, often after sharpening or flattening them with a temperature. The Learning to Predict chapter and code/lumen/sampling.py go through temperature, top-k and top-p.

Stage 7: append and repeat

The chosen token joins the input and the whole thing runs again. Naively this recomputes everything from scratch each step; in practice the model caches the intermediate results for earlier tokens (the "KV cache"), which is what makes generation fast enough to use. That is the Inference chapter and code/lumen/kv_cache.py.

A short history: how we got here

Predicting the next word is an old idea. What changed is the machinery that does the predicting. Knowing the lineage helps, because each step was a response to a concrete failure of the previous one.

n-grams: counting

The simplest next-token predictor is a table. Count how often each word follows each pair of previous words in a big corpus, and predict by looking up the counts. These n-gram models powered speech recognition and machine translation for decades. Their problem is sparsity: with a 50,000-word vocabulary there are $50000^3 \approx 10^{14}$ possible trigrams, and almost all of them never appear in any corpus. The table has no idea that "the cat sat" and "the dog sat" are related, because it has never seen either. Our toy interactive above is an n-gram model, which is why it can only ever say things we wrote into its table.

Neural language models: similarity

Bengio et al. (2003) proposed replacing the table with a neural network that first maps each word to a learned vector (an embedding), then predicts from the vectors. Now "cat" and "dog" can end up with similar vectors, so evidence about one transfers to the other. This is the ancestor of stage 2 in our pipeline. It fixed sparsity but still looked at a fixed window of a few previous words.

RNNs and LSTMs: memory, one step at a time

Recurrent neural networks read the sequence one token at a time, carrying a hidden state vector forward as a summary of everything read so far. In principle the summary can hold arbitrarily long context. In practice two things broke. First, gradients flowing back through many steps either vanish or explode, so the network forgets; the LSTM (Hochreiter & Schmidhuber, 1997) added gates to fix this and worked much better, but a fixed-size state still had to squeeze a whole paragraph into a few hundred numbers. Second, and decisively, the computation is sequential: token 500 cannot be processed until token 499 is done. GPUs are good at doing a million things at once and bad at doing one thing a million times in a row, so RNNs could not use the hardware that was becoming available.

Transformers: attention, in parallel

Vaswani et al. (2017) threw away recurrence. Instead of a summary state, every token gets to look directly at every earlier token and pull in what it needs, through a mechanism called attention (which had earlier been bolted onto RNNs for translation by Bahdanau et al., 2014). Because each position's computation depends only on the input, not on a previous position's result, all positions are computed in one big matrix multiplication, which is exactly what GPUs are built for. Long-range dependencies are one hop away instead of hundreds. The paper's title, "Attention Is All You Need", was a claim that the recurrent part was unnecessary, and it turned out to be right.

GPT-1, GPT-2, GPT-3: the same recipe, bigger

Radford et al. (2018) at OpenAI took the decoder, trained it as a plain next-token predictor on books, and showed that this "pre-training" gave a model that could be fine-tuned to many tasks. GPT-2 (2019) scaled the same recipe to 1.5 billion parameters and web text, and found that the model could do tasks nobody trained it for, just by being prompted. GPT-3 (Brown et al., 2020) scaled to 175 billion and made that "in-context learning" the main event. The architecture barely changed between these three; the discovery was that performance keeps improving predictably with more parameters, more data and more compute, a relationship studied as scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) that the Scaling Laws chapter covers.

Instruction-tuned assistants

A raw next-token predictor does not answer questions; it continues text. Ask it "What is the capital of France?" and it may well continue with "What is the capital of Germany?" because that is what a list of quiz questions looks like. Ouyang et al. (2022) showed how to take such a model and teach it to be helpful by fine-tuning on examples of good responses and then on human preferences (InstructGPT, the direct ancestor of ChatGPT). That post-training stage is the subject of the Instruction Tuning and RLHF chapter and the whole Post-Training part of the course.

What "parameters" are and where they live

You keep hearing "124 million parameters" or "70 billion parameters". What is being counted? A parameter is one number inside the model that was set by training rather than by the programmer. Almost all of them live in matrices: the embedding table, and the weight matrices inside attention and the MLP of every block. When you download a model, you are downloading these numbers.

The best way to understand parameter counts is to do one by hand. Let's take GPT-2 small, the model you will build in the GPT-2 from Scratch chapter. Its shape is: vocabulary $V = 50257$, width $d = 768$, context length $L = 1024$, $N = 12$ blocks, MLP hidden width $4d = 3072$.

Three things to notice. The MLPs hold nearly half the parameters, more than attention, even though attention gets all the press. The embedding table is a third of this small model, but it does not grow with depth, so in bigger models it becomes a small fraction. And the number of attention heads does not appear anywhere: splitting $d = 768$ into 12 heads of 64 or 6 heads of 128 uses the same matrices, just sliced differently.

Explore this yourself. The sliders below let you change every dimension and watch where the parameters go. Try the presets: GPT-2 small, medium, large and XL share one recipe and differ only in $d$, $N$ and head count.

Training versus inference

The same function is used in two very different ways, and confusing them causes real misunderstandings. Training is how the parameters get their values. Inference is using the model with its parameters frozen. Here is the split.

The three phases of a modern LLM's life

A model you can chat with has been through three distinct stages, each with its own data, objective and cost profile. This course is organised around them.

Pre-training

Take a randomly initialised transformer and train it on as much text as you can get, with the plain next-token objective. This is where the model learns grammar, facts, code, and (implicitly) a great deal about how the world works, because predicting text well requires it. It is by far the most expensive phase: GPT-3 was trained on roughly 300 billion tokens; Llama 3 models on around 15 trillion (Dubey et al., 2024). The result, often called a base model, is a superb autocomplete and a poor assistant. The Pre-Training part of the course, starting with its overview, is all about this phase.

Post-training

Take the base model and shape its behaviour. Supervised fine-tuning on example conversations teaches the format of an assistant. Preference optimisation (RLHF, DPO and relatives) teaches it to prefer responses that people rate highly. Further stages can teach tool use, refusals, and reasoning. This uses a tiny fraction of the pre-training compute but has an outsized effect on how the model feels to use. Instruction Tuning and RLHF is the introduction; the Post-Training part goes deep.

Deployment

Freeze the weights and serve them. The problems now are speed, memory and cost: how to run a 70-billion-parameter model fast enough to feel interactive, for thousands of users at once. This drives a separate body of techniques (KV caching, batching, quantisation, speculative decoding) that the Inference part covers.

What this part of the course will build

Over the next eight chapters we open each box in Figure 1 and then assemble them into a working GPT-2 that loads OpenAI's released weights and generates text. Here is the route, so you know where every piece fits:

Each chapter has companion code in code/lumen/. The pieces compose: tokenizer.py feeds embeddings.py, which feeds attention.py and block.py, which gpt2.py stacks. By the end you will have written every line of a language model yourself.

Practice

import random

def generate(model, tokens, n, greedy=False):
    tokens = list(tokens)
    for _ in range(n):
        probs = model(tokens)                       # STEP 1: predict
        if greedy:
            nxt = max(range(len(probs)), key=probs.__getitem__)
        else:
            nxt = random.choices(range(len(probs)), weights=probs)[0]   # STEP 2: choose
        tokens.append(nxt)                          # STEP 3: append
    return tokens                                   # STEP 4 is the for-loop

Further reading

• Vaswani et al. (2017). Attention Is All You Need. The transformer paper; short and readable once you finish this part of the course.
• Radford et al. (2019). Language Models are Unsupervised Multitask Learners. The GPT-2 report, and the model you will rebuild.
• Brown et al. (2020). Language Models are Few-Shot Learners. GPT-3 and in-context learning.
• Bengio et al. (2003). A Neural Probabilistic Language Model. Where learned word embeddings for language modelling began.
• Ouyang et al. (2022). Training language models to follow instructions with human feedback. InstructGPT: how a predictor becomes an assistant.
• Kaplan et al. (2020). Scaling Laws for Neural Language Models. Why "just make it bigger" was a rational strategy.
• Karpathy. nanoGPT. A minimal GPT-2 implementation; a good thing to read alongside code/lumen/gpt2.py.