Positional Encoding

Here is a fact that surprises everyone the first time. Take the attention mechanism from the next chapter, feed it the tokens of "dog bites man", then feed it "man bites dog". Without positional information, it produces exactly the same output for "bites" in both sentences. Not similar. Identical. The most important architecture in machine learning cannot, by itself, tell who bit whom.

That is the problem this chapter solves. It is a problem of our own making: we chose attention because it treats every position the same way (which is what makes it parallel and what lets it look far back). The price is that it treats every position the same way. We now have to put order back in, and the interesting part is that there are several good answers, each with its own trade-offs.

The problem: attention is a bag of vectors

Let's see the blindness with numbers, using a stripped-down attention. Each token has a vector; for a chosen token (the query) we score every token by a dot product, turn the scores into weights with softmax, and output the weighted sum of the vectors. Nothing in that recipe mentions where a token sits. The output is a function of the set of vectors, and sets have no order.

So the fix must live in the vectors. If the vector for "dog" is different when "dog" is at position 0 than when it is at position 2, attention can tell the sentences apart. Every method in this chapter is a way of making token vectors position-dependent, and they differ in how position gets in and what the model can then do with it.

Option 1: learned absolute positions (GPT-2)

The simplest answer: keep a second embedding table, indexed by position instead of token, and add its rows to the token embeddings. GPT-2 does exactly this. Its wpe matrix has shape $1024 \times 768$, one learned row per position up to the maximum context length. The input to the first block at position $t$ for token $i$ is

Here $E_i$ is the token's row from the previous chapter and $P_t$ is row $t$ of the position table. The two are simply added; the network learns to keep them separable enough to use both. Training treats $P$ like any other parameter, so the model discovers whatever representation of position helps it predict.

class LearnedPositions(nn.Module):
    def __init__(self, max_len, d_model):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(max_len, d_model) * 0.01)   # GPT-2 uses std 0.01 here

    def forward(self, x):                 # x: (batch, seq, d) token embeddings
        seq = x.shape[1]
        return x + self.weight[:seq]      # broadcast over the batch

It works well, and it is what the GPT-2 you will build uses. Its limits are just as simple. There is no row 1025, so the model physically cannot process a longer input. Nothing ties row 7 to row 8: the model has to learn from data that they are neighbours, and positions that appear rarely in training (the far end of long documents) are learned worse. And the representation is absolute: to figure out that two tokens are three apart, the network has to subtract two learned vectors it was never told were numbers.

Option 2: sinusoidal encodings (the original transformer)

Vaswani et al. (2017) instead designed the position vectors with a fixed formula, no learning. The goal was a vector for every position, of any length, in which relative distances are easy to read off. Their choice: fill the $d$ dimensions with sines and cosines of the position at a range of frequencies. For position $pos$ and dimension pair $i$ (so $2i$ and $2i{+}1$ are two adjacent coordinates):

Read it as: each pair of coordinates is a point on a circle, rotating as position increases; the first pair rotates fast (one radian per position), the last pair rotates extremely slowly (one radian every ten thousand positions), with the frequencies spaced geometrically in between. Write $\omega_i = 10000^{-2i/d}$ for the frequency of pair $i$. Then pair $i$ at position $pos$ is $[\sin(\omega_i\, pos), \cos(\omega_i\, pos)]$.

The binary-counter picture

Why a spread of frequencies? Think about how you write the number 13 in binary: 1101. The lowest bit flips every step, the next every two steps, the next every four. Fast bits resolve fine position, slow bits resolve coarse position, and together a handful of bits pin down a large range. Sinusoidal encoding is the smooth version: instead of bits that flip, coordinates that rotate, at rates spaced by a constant factor. Smoothness matters because the network needs gradients, and a bit that flips has none.

Why a relative shift is a linear transform

The property Vaswani et al. actually cared about: for any fixed offset $k$, the encoding of position $pos + k$ is a linear function of the encoding of position $pos$, and that function does not depend on $pos$. In other words there is a matrix $M_k$ with $PE(pos + k) = M_k \, PE(pos)$ for every $pos$. Here is why, for one pair. We want to express $[\sin(\omega(pos+k)), \cos(\omega(pos+k))]$ in terms of $[\sin(\omega\, pos), \cos(\omega\, pos)]$. The angle-addition formulas do it:

The matrix on the right is a rotation by angle $\omega k$. It contains $k$ but not $pos$. So "move $k$ positions forward" is the same rotation wherever you start, and a network that learns that rotation (which a linear layer can) has learned "look $k$ tokens back" for every position at once. That is the whole reason to use sinusoids rather than, say, the raw integer.

Explore the full encoding below. Rows are positions, columns are dimensions; you can see the fast columns on the left striping rapidly and the slow columns on the right barely changing. Switch to the curve view to watch individual pairs rotate.

Why absolute positions extrapolate poorly

The sinusoidal formula produces a vector for position 5,000 even if training stopped at 512. So why not just run the model longer? Because the formula is only half the story. The network learned, from data, how to use those vectors, and it only ever saw positions 0–511. The rotation trick means "look 3 back" generalises across positions, but the network also learned absolute patterns (attend to the first token; the slow dimensions never exceeded certain values), and at position 5,000 the slow dimensions take values it has never seen. Empirically, perplexity degrades quickly past the training length for both learned and sinusoidal encodings (Press et al., 2021 show this clearly in their Figure 1). Position information that the network only meets in comparisons between tokens, never as an absolute value, turns out to generalise better. That is the motivation for everything that follows.

Option 3: relative positions

If what matters is how far apart two tokens are, why not give attention that directly? Instead of adding a position vector to each token and hoping the network subtracts them, modify the attention score between positions $m$ and $n$ so it depends on $m - n$. Three influential versions:

Shaw et al. (2018): learned relative embeddings

Shaw, Uszkoreit & Vaswani (2018) add a learned vector $a_{m-n}$ to the key (and optionally the value) when query $m$ attends to key $n$, with the offset clipped to a window (say $[-16, 16]$) so that the table is small and every distance beyond the window shares a vector. The score becomes $q_m \cdot (k_n + a_{m-n})$. It works and it generalises to any length (offsets beyond the window just clip), but it costs an extra term in every score and the clipping throws away long-range distance information.

T5: bucketed scalar biases

The T5 model (Raffel et al., 2020) simplifies further. Instead of a vector per offset, it adds a learned scalar bias $b_{\text{bucket}(m-n)}$ to the attention score, shared across layers, with a separate bias per head. The offsets are grouped into buckets: exact for small distances, logarithmically coarser for large ones (so offsets 1, 2, 3 each get their own bias, but 64–127 share one). Cheap, effective, and it says explicitly what many heads want: "prefer nearby tokens" or "look exactly one back".

ALiBi: a fixed linear penalty

Press, Smith & Lewis (2021), "Attention with Linear Biases", remove the learning entirely. Each head $h$ subtracts $\lambda_h \cdot |m - n|$ from every score, with $\lambda_h$ a fixed slope per head (a geometric sequence such as $1/2, 1/4, \dots$). No position embeddings at all. Far tokens are penalised in proportion to distance, and different heads see different effective ranges. Its selling point is length extrapolation: trained on 1,024 tokens, models tested well at several times that length. Its cost is that the bias is the same regardless of content, so "attend to the token 100 back if it is a quote mark" is harder to express.

RoPE: rotary position embedding

Now the method almost every model since 2022 uses (Llama, Mistral, Qwen, Gemma, DeepSeek, and most others): Rotary Position Embedding, from Su et al. (2021). It sits between the absolute and relative worlds and gets the good properties of both. The idea in one sentence: rotate each query and key vector by an angle proportional to its position, so that the dot product between them depends only on the difference of their positions.

The two-dimensional derivation

Start with a single pair, so $q$ and $k$ are 2-d vectors. A rotation by angle $\alpha$ is the matrix

RoPE defines the position-aware query and key as $\tilde q_m = R(m\theta)\, q$ and $\tilde k_n = R(n\theta)\, k$. Now take their dot product. A dot product $a \cdot b$ can be written $a^\top b$, and the transpose of a rotation is the rotation the other way ($R(\alpha)^\top = R(-\alpha)$), so:

Look at what remains: the original $q$, the original $k$, and a single rotation by $(n - m)\theta$. The absolute positions $m$ and $n$ have vanished; only their difference survives. Shift both tokens ten positions to the right and the score is unchanged. That is the whole trick, and the derivation is three lines.

There is also a nice geometric reading. Rotating both vectors by the same angle does not change the angle between them, so the dot product (which is $\|q\|\|k\|\cos$ of the angle between them) only changes by how much more one was rotated than the other. If both are at the same position, they are rotated identically and the score is the plain $q \cdot k$.

The demo below lets you move $q$ and $k$ to any positions and watch the dot product. The proof of the property is in the note: shifting both positions by the same amount leaves the score unchanged.

Frequency bands: many clocks at once

A single pair with a single $\theta$ has a problem: rotations wrap around. With $\theta = 30°$, positions 0 and 12 look identical (both at $0°$). So RoPE uses all $d/2$ pairs, each with its own rate, exactly like the sinusoidal frequencies:

Here $d$ is the per-head dimension (typically 64 or 128), not $d_{model}$, because RoPE is applied inside each attention head. Pair 0 rotates one radian per position; the last pair rotates $10000^{-(d-2)/d}$ radians per position, which for $d = 128$ is about $1.2 \times 10^{-4}$: one full turn every fifty thousand positions or so. The high-frequency pairs resolve "the token right before me"; the low-frequency pairs resolve "roughly which part of the document". Any position within a huge range gets a unique combination of angles.

The clock picture makes this concrete. Each pair is a clock hand; advancing one position turns every hand by its own rate. Step through positions below and watch the fast hands spin while the slow ones creep.

How modern models apply RoPE inside attention

RoPE is not added to the token embeddings at the bottom of the network. It is applied inside every attention layer, to the query and key vectors after their linear projections and before the dot product, separately per head. Values are left alone, because values carry content that should not depend on where it came from. The rotation angles are precomputed once for every position and cached as cosine and sine tables.

def rope_cache(seq_len, d_head, base=10000.0):
    i = torch.arange(0, d_head, 2).float()            # 0, 2, 4, ... (one per pair)
    theta = base ** (-i / d_head)                     # (d_head/2,)  rotation rate per pair
    pos = torch.arange(seq_len).float()
    ang = pos[:, None] * theta[None, :]               # (seq, d_head/2)  angle = position x rate
    return ang.cos(), ang.sin()

def apply_rope(x, cos, sin):                          # x: (batch, heads, seq, d_head)
    x1, x2 = x[..., 0::2], x[..., 1::2]               # coordinates (2i) and (2i+1) of each pair
    y1 = x1 * cos - x2 * sin                          # 2-d rotation, applied to every pair at once
    y2 = x1 * sin + x2 * cos
    return torch.stack((y1, y2), dim=-1).flatten(-2)  # interleave back to (…, d_head)

# inside attention, per head:
#   q = apply_rope(q, cos, sin); k = apply_rope(k, cos, sin)
#   scores = q @ k.transpose(-2, -1) / sqrt(d_head)   # values v are NOT rotated

Two practical notes. First, the Llama reference code pairs coordinate $j$ with coordinate $j + d/2$ ("rotate half") instead of adjacent coordinates; the two conventions are equivalent up to a fixed permutation of dimensions, but weights trained with one cannot be loaded with the other without permuting, which has bitten many people converting checkpoints. Second, the base matters: Llama 1 and 2 used 10,000; Llama 3 raised it to 500,000 (Dubey et al., 2024), which slows every clock so that positions well beyond 8,000 remain distinguishable and the slow pairs do not wrap within the training context.

Extending the context after training

A model trained on 4,096 tokens meets position 20,000 at deployment. The fast pairs are fine (they wrapped around thousands of times during training already), but the slow pairs reach angles they never reached in training, and attention behaves erratically. Since about 2023 a family of tricks lets a RoPE model be extended, often with a little fine-tuning, by changing how positions are mapped to angles. The picture: rather than letting positions run past the trained range, squeeze or reshape them so that the slow clocks stay within familiar angles.

Position interpolation

Chen et al. (2023) observed that neural networks interpolate far better than they extrapolate, and proposed simply scaling every position by $L_{train}/L_{new}$ before computing the angles: to run at 8,192 with a 4,096-trained model, use position $m/2$ in place of $m$. Every clock now turns at half speed, so at token 8,192 the angles are exactly what they were at token 4,096 during training. A short fine-tune (about a thousand steps) recovers most of the quality. The cost is resolution: neighbouring tokens are now only half a step apart on every clock, which blurs fine-grained local order.

NTK-aware scaling

A community proposal by bloc97 (2023, published as a Reddit post rather than a paper) fixes that blurring by noting that the fast pairs never needed rescaling; only the slow pairs reach untrained angles. Instead of dividing positions, it raises the base (the 10,000), which slows the low-frequency pairs a lot and the high-frequency pairs hardly at all. This spreads the "interpolation" across the frequency spectrum and works reasonably well even with no fine-tuning; it is essentially what Llama 3's larger base does from the start.

YaRN

Peng et al. (2023), "Yet another RoPE extensioN", combine the two ideas explicitly: pairs whose wavelength is much shorter than the training length are left untouched, pairs whose wavelength is longer than the training length are interpolated, and pairs in between get a smooth blend. They add a small temperature adjustment to the attention logits to compensate for the change in score distributions at long range. YaRN needs far fewer fine-tuning tokens than plain interpolation and is the basis of many long-context models; the same paper reports extending to 128k tokens.

Summary table

Practice

The companion file code/lumen/positional.py implements sinusoidal(seq_len, d), LearnedPositions, rope_cache and apply_rope. The attention module in code/lumen/attention.py accepts an optional RoPE cache.

Further reading

• Vaswani et al. (2017). Attention Is All You Need. Section 3.5 for the sinusoidal encoding and its motivation.
• Shaw, Uszkoreit & Vaswani (2018). Self-Attention with Relative Position Representations. The first relative-position attention.
• Raffel et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. T5, including the bucketed relative bias.
• Press, Smith & Lewis (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ALiBi, and a clear demonstration of how badly absolute encodings extrapolate.
• Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. The RoPE paper; section 3 has the derivation done here.
• Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation. Position interpolation.
• Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models. Also summarises NTK-aware scaling.
• Kazemnejad et al. (2023). The Impact of Positional Encoding on Length Generalization in Transformers. Compares encodings, including none at all.
• Dubey et al. (2024). The Llama 3 Herd of Models. A production model's RoPE settings (base 500,000) and context extension recipe.