How to Use This Course & the One-Year Roadmap

By the end of this page you will know exactly what a language model is, why one person can build one in a year of evenings, and what you will ship each month to get there.

You type a question into a chat box and a paragraph comes back that reads like a person wrote it. Somewhere behind that box is a program. What is it, and could you write it yourself?

The honest answer is: yes, and the core of it is shorter than most web apps. The hard part is not the code. It is that the ideas arrive in an unfamiliar order and in notation nobody explained. This course fixes the order and explains the notation.

What an LLM actually is

Strip away the chat interface, the safety layers and the marketing, and a large language model is a function. Text goes in, text comes out. More precisely: a list of tokens (chunks of text roughly the size of a short word) goes in, and a probability for every possible next token comes out. That is the whole job. The model looks at "The cat sat on the" and says: "mat" 41%, "floor" 12%, "sofa" 9%, and so on for every token it knows. Pick one, append it, feed the longer text back in, repeat. A paragraph is just this loop run a few hundred times.

The "large" part is the size of the function: billions of adjustable numbers (the parameters, or weights) arranged in a specific structure called a transformer. The "learning" part is how those numbers were chosen: the model was shown trillions of tokens of real text and each time it guessed the next token badly, the numbers were nudged so it would guess slightly better. Nobody wrote rules for grammar or facts. They fell out of predicting the next token over and over, because predicting well requires knowing grammar and facts.

Figure 1. The whole interface of a language model. Text goes in as tokens, one probability per possible next token comes out. Everything else in this course is about what happens inside the yellow box and how its knobs get set.

The one thing it does: predict the next token

What is the simplest possible "language model"? Forget transformers. Suppose we only look at the last word and keep a table of what usually follows it. That is called a bigram model, and it is a real (if terrible) language model: it takes text in and gives a next-word distribution out.

Play with one. The table below was made up by hand over twelve words. Click a word to see what the model thinks comes next, then let it write a sentence by sampling from those bars. Notice two things: it is genuinely random (sample twice, get different sentences) and it has no memory beyond one word, so it wanders. A transformer is this same loop with a far smarter table.

InteractiveWhat does an LLM do? A 12-word bigram modelclick a word, then "Sample next"

The bars are the model's belief about the next word given the current one. "Sample next" rolls a die weighted by those bars and appends the result.

Common confusion

"The model picks the most likely word" is only true if you tell it to (that is "greedy" decoding, the third button above). Real chat models usually sample, which is why the same prompt gives different answers. Try greedy repeatedly in the toy: it loops forever, "the cat sat on the cat sat on the…". The Learning to Predict chapter covers temperature, top-k and top-p, which are the knobs between those two extremes.

Why this is learnable in a year

Isn't this a field where labs spend hundreds of millions of dollars? Yes, on compute. But the ideas are few, and the code that expresses them is small. A complete GPT-2, the model that started the modern era in 2019, is about 300 lines of PyTorch. The training loop is 40 more. You will write both.

Here is the full list of ideas you need. Everything else in the field is a variation on one of them.

Tokenization: chop text into a fixed vocabulary of chunks.
Embeddings: turn each chunk into a list of numbers a program can do arithmetic on.
Attention: let each position look at the others and pull in what is relevant.
The MLP and the residual stream: a small per-position computation, stacked in layers.
Next-token loss and gradient descent: measure how wrong the guess was, nudge every knob to be less wrong.
Post-training: fine-tune the predictor into an assistant with examples and preferences.
Inference tricks: cache, batch and shrink so it runs fast and cheap.

Seven ideas. Each one is a chapter or two here, each has a tiny-numbers example you can check by hand, and each ends with something you build. A year at 4 to 6 hours a week is roughly 250 hours. That is plenty.

Where it came from

The transformer architecture that every modern LLM uses was introduced by Vaswani et al. (2017), "Attention Is All You Need", a machine-translation paper. GPT-2 (Radford et al., 2019) showed that the same architecture, trained only to predict the next token on web text, learned to do many tasks it was never explicitly taught.

How the course is organized

Which order should you learn things in? The one the model computes in, then the one it was trained in. Six parts, in the sidebar:

Figure 2. The six parts in order. Architecture is the forward pass; pre-training and post-training are how the weights were chosen; advanced and inference are what practitioners actually spend their time on once the basics work.

Part	Question it answers	You will build
Foundations	What math and what tool do I need?	Nothing yet. Set up Python, run the tests.
Architecture	What does the model compute, step by step?	Tokenizer, attention, a full GPT-2 that loads OpenAI's weights.
Pre-training	How do the weights get their values?	A 10M-parameter model trained from scratch on a small corpus.
Post-training	How does a predictor become an assistant?	A chat model via supervised fine-tuning and DPO.
Advanced	What do the frontier labs do differently?	LoRA, a tiny mixture-of-experts, linear attention, a GRPO toy.
Inference	How do you serve it fast and cheap?	A KV-cached, quantized model behind a small server.

How to read a chapter

Every chapter has the same shape, on purpose. Once you know it, you can skim on a re-read and know where things are.

Symbols

$x$ = whatever goes in
$\theta$ = the model's knobs
$L$ = the loss (how wrong we are)

STEP 1

Prose

The problem first. What breaks without this idea? Then the idea, in short paragraphs.

STEP 2

Figure

A picture of the mechanism. The caption says what to look at.

STEP 3

Interactive

Drag the parameter yourself. This is where it clicks.

STEP 4

Code

Runnable PyTorch that matches the math line for line.

STEP 5

Quiz

Three to five questions. Wrong answers explain why.

STEP 6

Exercise

Something to build, pointing at code/lumen/.

Two conventions to know. Every formula is sandwiched: a plain sentence before it saying what it will do, the formula, then a sentence saying what happened, and a worked example with tiny numbers you can check on paper. And every named algorithm gets a strip like the one above: a Symbols card, then numbered steps.

Callouts are color-coded. Intuition is the mental picture. The math, slowly is a derivation you can skip on a first pass. Common confusion is what people get wrong. Worked example is numbers. Read the warnings even if you skip the math.

What to do with the interactives

Do not just look at them. Each one has a parameter, and the point is to predict what will happen when you move it, then move it and see if you were right. Being wrong there is the cheapest possible way to find a gap in your understanding. The "Open large" button in each panel's header makes it fullscreen.

The study rhythm

How much time does this take? The course is about 28 hours of reading if you only read. Nobody should only read. Budget roughly three times that for the interactives, exercises and code, and then some project weeks for the monthly deliverables. That works out to 4 to 6 hours a week for a year, and the planner below lets you check the arithmetic for your own schedule.

The rhythm that works: two sessions a week. One is reading a chapter with the interactives open. The other is building the exercise from that chapter, with the chapter closed. If you cannot build it with the chapter closed, you did not learn it yet, and that is useful to know on Tuesday rather than in month eight.

The one-year roadmap

What does "done" look like each month? Something you can run. Each row below ends with a deliverable you can show someone. If the month ends and the thing does not run, stay on it; the schedule bends, the deliverables do not.

Month	Chapters	Ship this
1	Foundations; Architecture: Introduction, Tokenization	A BPE tokenizer that round-trips any text: `decode(encode(s)) == s`. (`code/lumen/tokenizer.py`)
2	Embeddings, Positional Encoding, Attention	An attention layer with tests: causal mask verified, matches a reference implementation. (`code/lumen/attention.py`)
3	Layers of Understanding, Learning to Predict, Instruction Tuning intro, GPT-2 from Scratch	Your own GPT-2 that loads OpenAI's released weights and generates coherent text. (`code/lumen/gpt2.py`)
4–5	Pre-training: Overview, Objectives, Scaling Laws, Data Engineering	A ~10M-parameter model trained from scratch on a small corpus, with a loss curve you can explain. (`code/lumen/train.py`, `code/lumen/data.py`)
6	Infrastructure, Advanced Objectives, Evaluation, Llama 3 case study	A mixed-precision training run and a perplexity evaluation. (`code/lumen/eval.py`)
7–8	Post-training: all five chapters	A chat model: SFT on instruction data, then DPO on preference pairs. (`code/lumen/dpo.py`)
9–10	Linear Attention, Distillation, LoRA, Mixture of Experts	A LoRA fine-tune, a tiny MoE layer, and a linear-attention layer, each with a test. (`code/lumen/lora.py`, `code/lumen/moe.py`, `code/lumen/linear_attention.py`)
11	Optimizers, RL Fundamentals, RLHF Deep Dive	A reward model and a GRPO toy that improves a policy on a verifiable task. (`code/lumen/rl.py`)
12	Inference & Serving, Quantization, Capstone	Your model, quantized and KV-cached, behind a small server, plus a write-up of what you built. (`code/lumen/kv_cache.py`, `code/lumen/quantize.py`)

InteractiveOne-year plannerdrag hours per week

Reading time comes from the chapter estimates in the sidebar (about 28 hours total). Practice is assumed to be three times reading, plus a fixed budget for each "ship this" project. Drag to see when each part lands.

The hardware you need

Do you need a GPU? Not for the first three months. Tokenizers, attention layers and even loading GPT-2 small (124M parameters) to generate text all run fine on a laptop CPU or an Apple-silicon Mac. The interactives in this course run in your browser.

From month 4 you will train a model, and training is where the arithmetic adds up. A single consumer GPU (anything with 8 GB or more of memory) is enough for every project in this course, including the 10M-parameter pre-training run and the LoRA fine-tunes. If you do not own one, a free or cheap hosted notebook such as Google Colab works; the code is written so that a training run fits in a single session. You will never need more than one GPU here. The chapters on infrastructure explain what changes when you have a thousand.

A note on Macs

PyTorch supports Apple GPUs through the "MPS" backend. It works for everything in this course, but it is slower than a discrete NVIDIA card and a few operations fall back to the CPU. The PyTorch chapter shows how to pick the device automatically.

Setting up Python and PyTorch

What do you need installed? Python 3.9 or newer, PyTorch 1.12 or newer, and a handful of small packages. The companion code deliberately avoids anything exotic: no torch.compile, no custom CUDA kernels, no framework on top of PyTorch.

# from the project root
python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r code/requirements.txt
python -c "import torch; print(torch.__version__)"

2.3.1

If the last line prints a version number, you are set. To check which accelerator PyTorch can see:

import torch

if torch.cuda.is_available():
    device = "cuda"
elif getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
    device = "mps"
else:
    device = "cpu"
print("using", device)

using cpu

"cpu" is the right answer for months 1 to 3. Do not spend a weekend fighting driver installs before you need them.

Using the companion code

Every chapter references files under code/lumen/. They are small, readable, and written to match the math in the chapters line for line rather than to be fast. Each has tests under code/tests/. The single most useful habit in this course is to run those tests before and after you touch anything.

pip install -r code/requirements.txt
python -m pytest code/tests            # everything
python -m pytest code/tests -k attention   # just one topic

The exercises at the end of each chapter usually ask you to re-implement one of these files yourself, from the chapter alone, and then make the existing tests pass against your version. That is the "build it with the chapter closed" rule made mechanical.

File	What is in it	Chapter
`code/lumen/tokenizer.py`	BPE train, encode, decode	Tokenization
`code/lumen/attention.py`	Scaled dot-product, causal mask, multi-head, GQA	Attention
`code/lumen/block.py`	LayerNorm, RMSNorm, MLP, SwiGLU, TransformerBlock	Layers of Understanding
`code/lumen/gpt2.py`	Full GPT-2, loads pretrained weights	GPT-2 from Scratch
`code/lumen/train.py`	Training loop, LR schedule, grad clipping, mixed precision	Scaling Laws and Optimization
`code/lumen/dpo.py`	Direct preference optimization	Preference Optimization
`code/lumen/lora.py`, `moe.py`, `linear_attention.py`	LoRA, mixture of experts, linear attention	Advanced part
`code/lumen/kv_cache.py`, `quantize.py`	KV cache, int8 quantization	Inference

How to not quit

Most people who start a year-long course stop in month two. Not because it got hard, but because the feedback loop got long. Three habits shorten it.

Spaced repetition

Re-derive last week's thing before starting this week's thing. Not re-read: re-derive, on paper, from memory. It takes ten minutes. Attention's formula, the shape of a weight matrix, why we divide by $\sqrt{d_k}$. The quizzes at the end of each chapter are built for exactly this; come back to them a week later and answer them cold. If you use a flashcard app, the Key Takeaways box at the end of each chapter is written to be pasted straight into it.

Build before you read on

Every chapter ends with an exercise that produces a running program. Do not start the next chapter until this one's program runs. The temptation is to read ahead because reading is pleasant and building is friction. But the reading only sticks to something you built. A month of reading with no code is a month you will have to redo.

Keep a lab notebook

A plain text file. Each session, three lines: what you tried, what happened, what you think is going on. When your loss curve goes flat in month five, the notebook is how you find out that you changed the learning rate in month four. It is also, at the end, the raw material for the capstone write-up, and the thing you can show someone who asks what you have been doing all year.

Intuition

Learning this is like learning an instrument, not like reading a novel. Ten minutes of practice a day beats three hours on Sunday, and you cannot skip the scales. The interactives are the scales.

A map of the whole model

Where are you going? Here is the entire forward pass of a GPT-style model on one whiteboard. You will build every box in this figure by the end of month 3. Come back to it whenever you feel lost in the middle of a chapter and find the box you are in.

Figure 3. The whole forward pass. Token ids become vectors of width $d$, pass through $N$ identical blocks (attention, then a small MLP, each with a residual connection), and are projected back to one score per vocabulary word. Softmax turns the scores for the last position into the next-token probabilities from Figure 1. Note the shape never changes inside the stack: $(L, d)$ in, $(L, d)$ out, which is what makes the blocks stackable.

Three things to hold onto from this picture. The model is a stack of identical blocks, so once you understand one block you understand the model. The shape $(L, d)$ is preserved through the stack; every block reads the previous one's output and writes the same shape. And the only place the model "speaks" is the unembedding at the end, which turns a $d$-dimensional vector into a score per vocabulary entry.

Practice

Exercise 1 — set up and run the tests

Create a virtual environment, install code/requirements.txt, and run python -m pytest code/tests. Write the output of the run and the device PyTorch picked into the first entry of your lab notebook. If any test fails on your machine, note which and keep going; the chapter that owns that file will explain what it checks.

Solution sketch

The commands are in the "Setting up" section above. A failing test on day one is usually a version mismatch; check python -c "import torch; print(torch.__version__)" shows 1.12 or newer. The notebook entry should have three lines: what you ran, what it printed, what you think it means.

Exercise 2 — a bigram model in Python

Reproduce the bigram toy above in about 20 lines of Python: a dictionary mapping a word to a dictionary of next-word probabilities, a sample_next(word) function using random.choices, and a loop that grows a sentence from "the". Then improve it: count bigrams from any text file you have on disk instead of hand-writing the table. This is, sincerely, a trained language model.

Solution sketch

Counting: split the text on whitespace, iterate over adjacent pairs (w, nxt), and increment counts[w][nxt]. Normalize each inner dictionary so it sums to one. Sampling: random.choices(list(p.keys()), weights=list(p.values()))[0]. You will see the same behaviors as the toy: locally plausible, globally aimless. Everything after this in the course is about giving the model more context than one word.

Exercise 3 — plan your year

Use the planner to find the hours per week that finishes in twelve months. Put the "ship this" deliverables from the roadmap table into your calendar as end-of-month events, with the file names. Decide now which two evenings are reading and building.

Check yourself

At each step, what does a language model actually output?

It outputs a full distribution over the vocabulary. Choosing one token from it (greedy or sampling) is a separate decoding step, and generating a sentence is that step repeated.

Which happens first when a chat model is made?

Pre-training produces the base predictor; post-training (SFT, preference optimization) shapes it into an assistant; inference tricks like quantization come last. The course follows that order.

When in the roadmap do you first need a GPU or hosted notebook?

Tokenizers, attention layers and loading GPT-2 to generate text all run on a laptop CPU. Training from scratch in months 4 and 5 is the first place the compute bill arrives.

Key takeaways

An LLM is a function from tokens to a probability distribution over the next token; text generation is that function in a loop.
The whole field rests on about seven ideas; the code for a complete GPT-2 is a few hundred lines, and you will write it.
The course runs in the order the model computes (architecture) then the order it was trained (pre-training, post-training), then what practitioners do (advanced, inference).
Each month ends in something that runs. The deliverables are fixed; the calendar bends.
A laptop is enough until month 4; one consumer GPU or a Colab session is enough after.
Re-derive last week's idea, build before reading on, keep a lab notebook.