Linear Attention: Breaking the Quadratic Barrier

Attention is the reason transformers work, and also the reason they are expensive. Every token looks at every earlier token, so the cost of a layer grows with the square of the sequence length. At a few thousand tokens nobody cares. At a few hundred thousand, the attention scores alone would not fit on the GPU.

This chapter is about the family of ideas that tries to keep what attention does well (content-based lookup over the whole context) while making the cost grow linearly with length. The story has a clean spine: replace the softmax with something that factorizes, notice that the result is a recurrent network with a matrix-shaped memory, discover that the memory is too small, and then spend five years inventing better rules for writing into it.

If you have read the attention chapter and the inference chapter (KV cache), you have everything you need.

Breaking the Quadratic Barrier

Let's start by being precise about what is expensive. Standard attention for one head with $L$ tokens and head width $d$ computes an $L \times L$ score matrix $QK^\top$, softmaxes each row, and multiplies by $V$.

Count the arithmetic. $QK^\top$ is an $(L\times d)$ by $(d \times L)$ product: $2L^2 d$ floating-point operations. Multiplying the $L\times L$ weights by $V$ is another $2L^2 d$. So a head costs about $4L^2d$ FLOPs, and it has to materialize (or at least stream through) $L^2$ scores.

Plug in a modern long-context setting: $L = 131{,}072$ (128k tokens) and $d = 128$.

• $L^2 = 1.7 \times 10^{10}$ scores per head. In 16-bit that is about 34 GB per head if you wrote them all down. (FlashAttention avoids writing them down, but it still has to compute them.)
• $4L^2 d \approx 8.8 \times 10^{12}$ FLOPs per head per layer. With 32 heads and 32 layers, that is roughly $9 \times 10^{15}$ FLOPs for attention alone in a single forward pass: about a second of an H100 running flat out, for one sequence.
• At inference time the KV cache holds $2 \times L \times d$ numbers per head per layer. For a 32-layer, 32-head model in 16-bit that is about 2 GB per 128k-token conversation, and every generated token must read all of it.

Here is the catch: the "useful" part of attention, the projections $W_Q, W_K, W_V, W_O$ and the MLP, all grow linearly with $L$. Only the score matrix is quadratic. So the question is very specific: can we get content-based lookup without ever forming an $L \times L$ matrix?

The Kernel Insight

Look at what one output row of attention actually is. For query $i$, ignoring the $\sqrt{d}$ scale:

A weighted average of the values, where the weight of value $j$ is a similarity between $q_i$ and $k_j$. The specific similarity is $\exp(q^\top k)$, but nothing about "weighted average by similarity" requires the exponential.

Katharopoulos et al. (2020) made the observation that unlocks everything. Suppose the similarity could be written as a dot product of feature maps: $\text{sim}(q,k) = \phi(q)^\top \phi(k)$ for some function $\phi$ that maps a $d$-vector to an $m$-vector. Then the numerator becomes:

What just happened: the query was pulled outside the sum. The thing inside the parentheses no longer depends on $i$. It is one $m\times d$ matrix, computed once, shared by every query. The same trick works for the denominator, which becomes $\phi(q_i)^\top \sum_j \phi(k_j)$, a dot product with one $m$-vector.

In matrix form, with $\Phi_Q$ and $\Phi_K$ the $L\times m$ matrices of feature-mapped queries and keys:

This is just associativity of matrix multiplication. But the two sides have wildly different costs. The left side forms an $L\times L$ matrix: $O(L^2 m)$. The right side forms an $m\times d$ matrix: $O(L m d)$. With $m \approx d$, the cost fell from $O(L^2 d)$ to $O(L d^2)$. Linear in $L$.

Restoring Factorization

Why can't we just do this with softmax? Because $\exp(q^\top k)$ is not a dot product of a function of $q$ and a function of $k$. Well, technically it is, but only with an infinite-dimensional $\phi$ (that is the kernel-machine view: $\exp(q^\top k)$ is a valid kernel with an infinite feature expansion). Infinite $m$ is not helpful when the whole point was to make $m$ small.

So linear attention restores the factorization by choosing a finite $\phi$. The original paper used $\phi(x) = \text{elu}(x) + 1$, elementwise, which keeps everything positive so the "weights" $\phi(q)^\top\phi(k)$ are non-negative and the denominator is safe. Other choices that show up in the literature:

• Identity $\phi(x)=x$, often with an L2 normalization on $q$ and $k$. Used by RetNet, GLA, DeltaNet and Mamba-2. Weights can be negative, so these models drop the denominator entirely and rely on normalization layers instead.
• Random features approximating the exponential (Performer, Choromanski et al. 2020): $\phi$ is a random projection followed by $\exp$, chosen so that $\mathbb{E}[\phi(q)^\top\phi(k)] = \exp(q^\top k)$. Unbiased but high variance.
• Learned or polynomial maps (e.g. Based, Hedgehog) that try to imitate the "spikiness" of softmax with a small $m$.

Whatever the $\phi$, the resulting layer is called linear attention, and the important thing is not the choice of $\phi$ but what the associativity trick reveals about the computation.

The recurrent view: attention is an RNN

Go back to the causal case, where token $t$ may only look at tokens $1..t$. The memory matrix for query $t$ is $\sum_{j\le t}\phi(k_j)v_j^\top$. Call it $S_t$. Then obviously:

Read it slowly. $S_t$ is an $m\times d$ matrix (we will say $d\times d$ from here on, assuming $m=d$). Each new token adds a rank-1 outer product "key times value" to it. Each output reads the memory by multiplying with the query. There is no sum over the past any more: the past has been compressed into one matrix. That makes the layer a recurrent neural network with a matrix-valued hidden state, and it makes decoding cost $O(d^2)$ per token regardless of how long the context is.

import torch

def linear_attention_recurrent(q, k, v, phi=lambda x: torch.nn.functional.elu(x) + 1):
    """q, k, v: (L, d). Returns outputs (L, d) one token at a time."""
    L, d = q.shape
    S = torch.zeros(d, d)          # the memory matrix
    z = torch.zeros(d)             # the normalizer
    outs = []
    for t in range(L):
        kt, qt = phi(k[t]), phi(q[t])
        S = S + torch.outer(kt, v[t])      # write: rank-1 update
        z = z + kt
        outs.append((qt @ S) / (qt @ z + 1e-6))   # read
    return torch.stack(outs)

def linear_attention_parallel(q, k, v, phi=lambda x: torch.nn.functional.elu(x) + 1):
    """Same numbers, computed with a masked L×L product (fine for training at short L)."""
    Q, K = phi(q), phi(k)
    A = torch.tril(Q @ K.T)                       # causal mask on the (unnormalized) weights
    return (A @ v) / (A.sum(-1, keepdim=True) + 1e-6)

The companion file code/lumen/linear_attention.py contains both forms plus an assertion that they agree to float precision. Run it; the agreement is the whole point.

The Compression Bottleneck

If linear attention is $L/d$ times cheaper and mathematically almost the same thing, why is every frontier model still using softmax? Because "almost" hides a real loss, and it is easy to say exactly what was lost.

Softmax attention's KV cache stores $L$ keys and $L$ values: $2Ld$ numbers that grow with the context. Linear attention stores $d^2$ numbers, full stop. Once $L$ exceeds $d$, the memory is provably holding less information than was put in. Something has to be blurred.

What gets blurred is recall. Consider the "associative recall" task: the context contains pairs like a→7, b→3, c→9, … and at the end the model is asked "what did b map to?" Softmax attention solves this trivially: the query for b matches the key for b almost exactly, gets nearly all the weight, and copies the value. Linear attention must retrieve $\phi(q_b)^\top S$, and $S$ is the sum of all the pairs. Unless the feature-mapped keys are exactly orthogonal, every other pair leaks into the answer. With hundreds of pairs and $d = 128$, they cannot all be orthogonal. The answer becomes a mush.

This shows up in the benchmarks. Plain linear attention has noticeably worse perplexity than softmax at equal parameters, and the gap is concentrated in exactly the tokens where in-context lookup matters: copying names, numbers, code identifiers, and rare tokens that appeared earlier in the document. On the synthetic "needle in a haystack" style tests, plain linear attention is not even close.

Not All Memories Are Equally Important

If the memory is going to be too small, the sensible thing is to stop treating every token as equally worth remembering. Look at the plain update again: $S_t = S_{t-1} + \phi(k_t)v_t^\top$. Every token that was ever written is still in there at full strength. A name mentioned 100k tokens ago has the same weight as one mentioned in the last sentence.

Two different fixes come from two different observations about language:

1. Recency matters. Most of the time, what you need is nearby. So let the memory fade: multiply $S_{t-1}$ by a number slightly less than one before adding the new token. This is decay.
2. Some tokens matter and some don't. A period at the end of a sentence is not worth remembering; a proper noun is. So let the model decide, token by token, how much to keep. This is gating.

And a third fix, which we will come to later, says that when the same key comes up again the old value should be replaced, not added to. That is the delta rule. Almost every modern linear-attention or state-space model is some combination of these three moves.

From Decay to Gating

Let's make the two knobs precise before looking at the papers. Write the general update as

where $G_t$ is a matrix of numbers in $[0,1]$ multiplied elementwise ($\odot$) with the old memory. What is $G_t$?

• $G_t = 1$ everywhere: plain linear attention. Nothing is ever forgotten.
• $G_t = \gamma$, a fixed scalar in $(0,1)$: decay. Every entry fades by the same factor per step. This is RetNet.
• $G_t = \alpha_t \mathbf{1}^\top$ with $\alpha_t\in[0,1]^d$ computed from the input: gating, row-wise (per key dimension). This is GLA.
• $G_t = \alpha_t$ a data-dependent scalar: the gate in Mamba-2 and Gated DeltaNet.

The difference between decay and gating is the difference between "memories fade at a fixed rate" and "the model chooses what to forget." Decay is free (no parameters, and it makes the math especially clean). Gating costs a small projection per token but lets the model hold a fact for thousands of steps when it wants to and dump irrelevant tokens immediately.

Retention

Take the decay case first, because its algebra is the cleanest and everything else is built on it. Sun et al. (2023) called this mechanism retention. The recurrent form, with identity feature map and no normalizer:

Unroll it and look at what each output actually is. $S_t = \sum_{j\le t}\gamma^{t-j}k_jv_j^\top$, so

What just happened: this is attention with weights $q_t^\top k_j$, multiplied by a positional factor $\gamma^{t-j}$ that shrinks with distance. It is a built-in relative position bias, and it costs nothing. There is no softmax, so nothing normalizes the weights; RetNet instead applies a GroupNorm to the outputs of each head.

RetNet

RetNet is the architecture built around retention. Three details matter beyond the formula.

Multi-scale decay. Each head gets its own $\gamma$, fixed (not learned), spaced so that some heads remember only the last few tokens and others remember hundreds. Concretely the paper sets $\gamma_h = 1 - 2^{-5-h}$ for head $h$, giving decays from about $0.97$ to $0.999$. A head with $\gamma = 0.97$ has a half-life of roughly $23$ tokens; with $\gamma = 0.999$, about $700$.

Rotary-style position encoding. Retention is combined with a complex rotation of $q$ and $k$ (the xPos / RoPE family from the positional encoding chapter), so that $q_t^\top k_j$ itself also depends on relative position, not just the decay.

Three computation modes. This is the part that made RetNet influential: the paper showed that the same layer can be run as a parallel matrix product for training, a recurrence for decoding, and a hybrid "chunkwise recurrent" form for long sequences. Let's look at those next, since every model after RetNet uses the same trick.

Three Views of the Same Computation

Here is the problem the three forms solve. The recurrent form is sequential: to get $S_t$ you need $S_{t-1}$. That is fine for decoding one token at a time, but disastrous for training, where you have all $L$ tokens at once and a GPU that wants big matrix multiplies, not $L$ tiny dependent steps. The parallel form gives you the matrix multiplies but builds the $L\times L$ matrix we were trying to avoid.

The chunkwise form gets the best of both. Split the sequence into chunks of $C$ tokens (typically $C = 64$ or $128$). Inside a chunk, run the parallel form: a $C\times C$ score block, which is small. Between chunks, pass the memory matrix $S$ exactly as in the recurrent form. For a query in chunk $c$, its output has two parts:

where $t_c$ is the start of the chunk and $S_{c-1}$ is the memory after all previous chunks. At the end of the chunk, the state is updated in one shot: $S_c = \gamma^{C} S_{c-1} + \sum_{j \in c}\gamma^{t_{c}+C-j}\,k_j v_j^\top$, which is a single $(d\times C)(C\times d)$ product.

Count the work. Intra-chunk: $L/C$ blocks of $C\times C \times d$, i.e. $O(LCd)$. Inter-chunk: $L/C$ state updates of $O(Cd^2)$, i.e. $O(Ld^2)$. There are only $L/C$ sequential steps instead of $L$. With $C=64$ and $d=128$ this is a handful of well-shaped matmuls per chunk and it maps beautifully onto GPU tensor cores. This is the algorithm inside the flash-linear-attention kernels that essentially all of these models train with.

Learning How to Update Memory

Decay and gating control how fast old memories fade. They do nothing about the other problem: collisions. If the context says "the capital is Paris" and later "the capital is Lyon" (a correction), plain and decayed linear attention both store capital → Paris + Lyon. The query for "capital" retrieves a blend. Decay makes the blend lean toward Lyon, but it never removes Paris, and it fades Lyon too.

What you want is: when a key that is already in memory is written again, erase the old value at that key first, then write the new one. That is an update rule, and there is a classic one.

Think of the memory as a linear map that should satisfy $S^\top k_j \approx v_j$ for every stored pair, or in our row convention, $k_j^\top S \approx v_j^\top$. When pair $(k_t, v_t)$ arrives, the current memory predicts $\hat v_t^\top = k_t^\top S_{t-1}$. The prediction error is $v_t - \hat v_t$. Move $S$ to reduce that error, by one gradient step of size $\beta_t$ on $\tfrac12\|k_t^\top S - v_t^\top\|^2$:

Collect the terms and you get the delta rule (Widrow & Hoff, 1960, rediscovered many times):

Read the two pieces. The second term writes the new value at key $k_t$, scaled by $\beta_t$. The first term multiplies the old memory by $(I - \beta_t k_t k_t^\top)$: with a unit-norm key and $\beta_t = 1$, that is a projection that removes everything stored along the direction $k_t$, and leaves everything orthogonal to $k_t$ untouched. Erase, then write. With $\beta_t = 0$ nothing changes; with $\beta_t = \tfrac12$ you get halfway.

DeltaNet

Schlag, Irie and Schmidhuber (2021) noticed that linear attention's $S_t = S_{t-1} + k_tv_t^\top$ is precisely the Hebbian update of a "fast weight" network from the 1990s, and that the delta rule was the obvious upgrade. Their DeltaNet uses the update above with a learned, input-dependent $\beta_t = \sigma(w_\beta^\top x_t) \in (0,1)$ and L2-normalized keys (so that $\beta_t=1$ means a full overwrite). They showed dramatically better associative recall than plain linear attention on synthetic tasks.

The paper did not become widely used for three years, for a very practical reason: the update is not a simple decay. $(I - \beta_tk_tk_t^\top)$ is a different matrix at every step, so the unrolled form $S_t = \sum_j \big(\prod_{i=j+1}^{t}(I-\beta_ik_ik_i^\top)\big)k_jv_j^\top$ involves products of $d\times d$ matrices, and nobody had a chunkwise algorithm for it. Training was sequential and slow.

Yang et al. (2024), "Parallelizing Linear Transformers with the Delta Rule over Sequence Length", fixed this. The key observation: the product $\prod (I - \beta_ik_ik_i^\top)$ over a chunk is $I$ minus a low-rank matrix (this is the WY representation from Householder QR), so it can be written as $I - \sum_i w_i k_i^\top$ for some vectors $w_i$ computable with a small triangular solve inside each chunk. That reduces the whole thing to matmuls of chunk size, and DeltaNet became trainable at the same speed as GLA.

Gated Linear Attention

Now the gating branch. Yang et al. (2023), "Gated Linear Attention Transformers with Hardware-Efficient Training", asked: what is the most general forgetting rule that still has a fast chunkwise algorithm? Their answer is a per-row, data-dependent gate:

What just happened: instead of one scalar $\gamma$ for the whole memory, each of the $d$ rows of $S$ (each key dimension) gets its own forget factor, and that factor is computed from the current token. $\tau$ is a temperature (they use 16) that biases the gates toward 1 so the model does not forget too eagerly early in training. Because $\alpha_t\mathbf{1}^\top$ multiplies row $i$ of $S$ by $\alpha_{t,i}$, the unrolled form has weights $\prod_{i=j+1}^{t}\alpha_i$ (a cumulative product), and the chunkwise trick still works with one extra bookkeeping step: multiply queries and keys by ratios of cumulative gates.

The effect is exactly what the "not all memories are equal" argument asked for. On a token the model deems unimportant, it can set $\alpha_t \approx 1$ (keep everything). On a section break, it can set $\alpha_t$ small (flush). And because the gate is a vector, it can flush some key dimensions and keep others.

Gated DeltaNet: both knobs at once

Gating flushes; the delta rule overwrites. They fix different failures, so the natural next step is to combine them. Yang, Kautz and Hatamizadeh (2024), "Gated Delta Networks: Improving Mamba2 with Delta Rule", do exactly that with a scalar gate $\alpha_t\in(0,1)$:

What just happened: first decay the whole memory by $\alpha_t$ (fast, global erasure when the model wants to start fresh), then apply the delta rule (targeted overwrite of one key). The paper shows the combination beats both Mamba-2 (gate only) and DeltaNet (delta only) on language modeling, in-context recall and length extrapolation, and that the chunkwise WY trick extends to the gated case with almost no extra cost. Gated DeltaNet is the layer that Qwen3-Next and several other 2025 hybrid models reportedly adopted for their linear layers.

Kimi Linear and Kimi Delta Attention

Moonshot AI's Kimi Linear technical report (October 2025) pushes the gated delta rule one step further. Its core layer, Kimi Delta Attention (KDA), is described in the report as Gated DeltaNet with a finer-grained gate: instead of one scalar $\alpha_t$ per head, the decay is a per-channel vector (a diagonal matrix $\text{Diag}(\alpha_t)$ applied to the memory), similar in spirit to GLA's row-wise gate but combined with the delta-rule erase. In the report's framing, this gives the model finer control over which parts of its finite state to keep and which to overwrite, and they present a chunkwise algorithm built on a specialized diagonal-plus-low-rank (DPLR) transition structure that is cheaper than the general DPLR case.

The other reported design decisions are just as instructive as the layer itself:

• Hybrid by construction. Kimi Linear interleaves KDA layers with full-attention layers (Multi-Head Latent Attention, MLA) at a reported ratio of 3:1. Three linear layers, then one global attention layer, repeated. The full-attention layers are what give the model exact recall; the KDA layers are what make the KV cache small.
• No positional encoding in the attention layers. The report says the global MLA layers use no RoPE; position information comes from the recurrent KDA layers, which are inherently order-aware.
• Reported results. A 48B-total / 3B-active MoE model trained on 1.4T tokens is reported to match or beat a full-attention baseline of the same size at short context, long context and in RL scaling, while reducing KV-cache memory by up to 75% and reaching up to about 6× the decoding throughput at 1M-token context.

Treat those numbers as the authors' report rather than independently verified fact. The design pattern, though, is now the consensus one: a delta-rule linear layer with gating, in a hybrid with a minority of full-attention layers.

Beyond Memory Matrices

Everything so far started from attention and removed the softmax. There is a second lineage that started from a completely different place, control theory, and arrived at the same recurrence. Understanding it explains where Mamba came from and why Mamba-2 was such a surprise.

Before that, a word on what actually ships. Pure linear models exist and are competitive at small scale, but in production the pattern is hybrid: mostly linear (or SSM) layers plus a few softmax-attention layers, often with the softmax layers restricted to a sliding window. The linear layers carry the bulk of the "language" work cheaply; the attention layers provide exact, unbounded recall for the tokens where it matters. Jamba (AI21, 2024) mixed Mamba and attention at 7:1; Gated DeltaNet's paper reports hybrids with sliding-window attention; Kimi Linear uses 3:1; NVIDIA's Nemotron-H family uses Mamba-2 with a minority of attention layers. The ratio is an empirical knob, but "a few global attention layers fix most of the recall gap" is the robust finding.

Why State-Space Models Became Interesting

A state-space model (SSM) is the control theorist's description of a system with memory. There is an input signal $u(t)$, a hidden state $x(t)\in\mathbb{R}^N$, and an output $y(t)$, related by linear differential equations:

In words: the state drifts according to $A$ (an $N\times N$ matrix, which for a stable system shrinks the state), gets pushed by the input through $B$ (an $N\times1$ vector), and the output is a linear readout $C$ (a $1\times N$ row). This is a scalar-input, scalar-output system; for a $d$-channel input you run $d$ independent copies.

Text is not continuous, so we discretize with a step size $\Delta$. The standard zero-order-hold (ZOH) rule, which assumes the input is constant between samples, gives:

What just happened: the differential equation turned into a linear recurrence. Compare it with linear attention's $S_t = \gamma S_{t-1} + k_tv_t^\top$. Same shape: a state, multiplied by a decay-like matrix, plus a term from the current input. The difference is that in an SSM the decay $\bar A$ is a fixed matrix, and the state is a vector of size $N$ per channel rather than a $d\times d$ matrix.

Because $\bar A, \bar B, C$ do not depend on $t$, you can unroll the recurrence into a convolution: $y_t = \sum_{j\le t} C\bar A^{\,t-j}\bar B\,u_j$, i.e. $y = u * \bar K$ with kernel $\bar K = (C\bar B,\ C\bar A\bar B,\ C\bar A^2\bar B,\ \dots)$. A convolution of length $L$ can be done with FFTs in $O(L\log L)$. That is the SSM's version of the "parallel form."

S4 in a paragraph

Gu, Goel and Ré (2021) made this practical with S4 (Structured State Spaces). Two problems had to be solved. First, a random $A$ forgets everything almost immediately or blows up; S4 initializes $A$ with the "HiPPO" matrix, which is derived so that the state $x$ holds an optimal polynomial approximation of the recent input history, giving a principled long memory. Second, computing the kernel $\bar K$ naively requires powers of an $N\times N$ matrix for every lag; S4 restricts $A$ to a diagonal-plus-low-rank structure so that $\bar K$ can be computed in $\tilde O(N + L)$ via a clever Cauchy-kernel trick. The result was the first model to solve the Long Range Arena's hardest tasks (Path-X, 16k-length pixel sequences), and it started a wave of SSM work. Its successors (DSS, S4D, S5) simplified $A$ to purely diagonal with little loss.

The Importance of Selectivity

S4 was excellent on audio and long signals and disappointing on language. Why? Here is the problem, from Gu and Dao's paper. Consider selective copying: the input is a stream of mostly noise tokens with a few "content" tokens scattered at random positions, and the task is to output the content tokens in order, ignoring the noise.

A time-invariant SSM cannot do this well. Its $\bar A$, $\bar B$, $C$ are the same for every token, so it treats every input identically: each token is written into the state with the same $\bar B$ and decays at the same rate $\bar A$. The model has no way to say "this token is noise, skip it" or "this token is content, remember it." It is a fixed linear filter, and a fixed filter cannot depend on what the tokens are.

Attention has no such problem: the query-key match is exactly a content-dependent decision about which past tokens to read. The whole reason attention beat RNNs on language is that language needs content-dependent selection. So to make an SSM work on language, the parameters must become functions of the input.

Mamba

Gu and Dao (2023) made the SSM parameters input-dependent and called the result a selective SSM, or S6; the model built around it is Mamba. Three things change:

with $A$ still a fixed (diagonal, negative) matrix, and the discretization $\bar A_t = \exp(\Delta_t A)$, $\bar B_t \approx \Delta_t B_t$ now computed per token. The recurrence becomes $x_t = \bar A_t x_{t-1} + \bar B_t u_t$, $y_t = C_t x_t$.

The step size $\Delta_t$ is the interesting one. Think of it as "how much time passes at this token." A large $\Delta_t$ makes $\bar A_t = \exp(\Delta_t A)$ close to zero (the old state is forgotten) and $\bar B_t$ large (the new input is written strongly): reset and focus on this token. A small $\Delta_t$ makes $\bar A_t \approx I$ and $\bar B_t \approx 0$: ignore this token, keep the state. That is precisely a gate, and it is exactly what selective copying needs. $B_t$ and $C_t$ being input-dependent means the model can also choose where in the state to write and what to read, which is the key/query role.

The price: with $\bar A_t$ varying per token, the recurrence is no longer a convolution, so the FFT trick is gone. Mamba's second contribution is the hardware-aware scan. The recurrence $x_t = \bar A_tx_{t-1} + \bar B_tu_t$ is an associative operation (a "scan"), so it can be parallelized across $L$ in $O(\log L)$ depth. Mamba implements this scan in a fused GPU kernel that keeps the $N=16$ state per channel in fast SRAM, never materializes the full $L\times d\times N$ tensor of states in HBM, and recomputes it in the backward pass. That kernel is why Mamba trains fast despite being a "real" recurrence.

Mamba-2

By early 2024 there were two communities with two vocabularies (memory matrices and feature maps on one side; states, discretization and scans on the other) building models that looked suspiciously alike. Dao and Gu (2024), "Transformers are SSMs", proved they are the same thing. The result is called state-space duality (SSD).

Here is the argument in its simplest form. Restrict Mamba's $A$ to a scalar times the identity per head: $\bar A_t = a_t I$ for a data-dependent scalar $a_t\in(0,1)$. Stack the $N$-dimensional states of the $P$ channels in a head into a matrix $H_t\in\mathbb{R}^{N\times P}$. Then the selective SSM recurrence reads

Now rename: $B_t \to k_t$, $C_t\to q_t$, $u_t\to v_t$, $H_t\to S_t$, $a_t \to \alpha_t$. You get $S_t = \alpha_tS_{t-1} + k_tv_t^\top$, $o_t = q_t^\top S_t$. That is gated linear attention with a scalar gate. Unrolled, it is a masked attention $y = (L \odot QK^\top)V$ where the mask $L_{ij} = \prod_{k=j+1}^{i}a_k$ is a cumulative product of decays, a "1-semiseparable" matrix. Mamba's selective scan and GLA's chunkwise algorithm are two ways to multiply by such a matrix.

Mamba-2 is what you get when you take that seriously: a scalar gate, a much larger state ($N = 64$ to $256$ instead of 16, now affordable because the chunkwise algorithm uses tensor cores), multi-head structure borrowed from attention, and a simpler block. It trains 2–8× faster than Mamba-1's scan and slightly improves quality.

import torch

def gated_delta_rule(q, k, v, alpha, beta):
    """Reference recurrent form of Gated DeltaNet.
    q, k, v: (L, d); alpha, beta: (L,) in (0,1). Keys are L2-normalized inside.
    S_t = alpha_t * (I - beta_t k k^T) S_{t-1} + beta_t k v^T ;  o_t = q^T S_t
    """
    L, d = q.shape
    k = torch.nn.functional.normalize(k, dim=-1)
    S = torch.zeros(d, d)
    outs = []
    for t in range(L):
        kt, vt = k[t], v[t]
        pred = kt @ S                                 # what the memory believes about k_t
        S = alpha[t] * S + beta[t] * torch.outer(kt, vt - alpha[t] * pred)   # decay, then erase-and-write
        outs.append(q[t] @ S)
    return torch.stack(outs)

(Check that the update line equals $\alpha_t(I-\beta_tk_tk_t^\top)S_{t-1} + \beta_tk_tv_t^\top$ by expanding it; that is Exercise 2.) The chunkwise version with the WY trick is in code/lumen/linear_attention.py alongside a test that it matches this loop.

Where the Field Stands Today

Let's be honest about the scoreboard as of 2026, because the marketing around these models is not.

The quality gap is real but narrow, and it lives in recall. At equal parameters and training tokens, the best pure linear models (Gated DeltaNet, Mamba-2, KDA-style layers) match transformers on perplexity and on most reasoning-style benchmarks up to the few-billion scale where they have been compared. On tasks that require retrieving a specific earlier token out of a long context (needle-in-a-haystack, multi-key associative recall, long-document QA, and, importantly, in-context learning with many examples), they are worse, and the gap grows with the number of things to remember. That is the compression bottleneck, and no update rule removes it; a $d\times d$ (or $N\times P$) state simply cannot store an unbounded number of exact key-value pairs.

Hybrids are the answer that shipped. Because a handful of full-attention layers restore exact recall, and because those layers only cost quadratic time in a quarter or an eighth of the network, hybrids get most of the speed with almost none of the quality loss. The open questions are the ratio, whether the attention layers should be global or sliding-window, and how to interleave them, and different labs have landed on different answers.

The economics are about inference, not training. Training cost at 4–8k context is dominated by the MLPs and projections; linear layers do not change it much. The win is at inference with long contexts: a linear layer's state is a few hundred KB per layer regardless of context, versus a KV cache that grows by tens of KB per token. That means far more concurrent sequences per GPU, no cache eviction, and per-token latency that does not grow with conversation length. For agentic workloads with 100k+ token contexts, this is the difference between viable and not.

Honest open questions.

• How much of the recall gap is intrinsic to finite state versus fixable with bigger states, better feature maps, or smarter write rules? Results keep improving; nobody has shown a ceiling.
• Do linear layers scale as well as attention past the tens of billions of parameters and tens of trillions of tokens? Public evidence is thinner than for transformers.
• Length generalization: recurrent models often extrapolate to longer contexts better than RoPE transformers, but "the state does not blow up" is not the same as "the model uses the extra context well."
• Interpretability and steering tools built for attention (attention maps, KV-cache editing) do not transfer directly to a compressed matrix state.

The safe summary: softmax attention is still the default for frontier models, but the linear family has become a serious component of the toolkit, and the most likely future is not "linear replaces attention" but "attention becomes the expensive minority layer it deserves to be."

Practice

Further reading

• Katharopoulos et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. The associativity trick and the recurrent view, in six pages.
• Schlag, Irie & Schmidhuber (2021). Linear Transformers Are Secretly Fast Weight Programmers. The delta rule for linear attention, and the capacity argument.
• Sun et al. (2023). Retentive Network: A Successor to Transformer for Large Language Models. Decay, and the parallel / recurrent / chunkwise trio.
• Yang et al. (2023). Gated Linear Attention Transformers with Hardware-Efficient Training. Data-dependent gates and the chunkwise algorithm that made them fast.
• Yang et al. (2024). Parallelizing Linear Transformers with the Delta Rule over Sequence Length. The WY trick that made DeltaNet trainable.
• Yang, Kautz & Hatamizadeh (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. Gating plus delta rule; the current workhorse linear layer.
• Moonshot AI (2025). Kimi Linear: An Expressive, Efficient Attention Architecture. Kimi Delta Attention and a 3:1 hybrid at scale, as reported by the authors.
• Gu, Goel & Ré (2021). Efficiently Modeling Long Sequences with Structured State Spaces. S4: where the SSM lineage starts.
• Gu & Dao (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Selectivity and the hardware-aware scan.
• Dao & Gu (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. The unification; long but worth it.
• Choromanski et al. (2020). Rethinking Attention with Performers. Random-feature approximations of the softmax kernel.