Quantization

A 7-billion-parameter model in fp16 is 14 GB of weights. Every single generated token requires reading all 14 GB from GPU memory, because decode runs the whole network for one token. At 1 TB/s of memory bandwidth that read alone takes 14 ms, so you can never get more than about 70 tokens per second, no matter how fast the arithmetic units are.

Here is the catch: the arithmetic units are mostly idle during that read. Decode is memory-bound (the inference chapter works this out in detail). So the question becomes: what if each weight took fewer bytes? Halve the bytes and you halve the read time, and the model fits on a smaller GPU as a bonus.

That is quantization. Store each number with fewer bits, decode it back to a float just before you use it, and eat a small error. The whole chapter is about how to make that error small, where it is not small, and when the speed-up is real.

Why fewer bits means faster

Think about what a GPU does during one decode step. It streams every weight matrix through the compute units exactly once and does a matrix-vector product with it. For a matrix of shape $d_{out} \times d_{in}$, that is $2\,d_{out} d_{in}$ floating-point operations against $2\,d_{out} d_{in}$ bytes in fp16. One operation per byte.

An H100 can do roughly 1,000 trillion fp16 operations per second but read only about 3.35 trillion bytes per second. At one operation per byte, the arithmetic finishes 300 times sooner than the memory read. The chip is waiting on memory.

So the time per decode step is basically bytes of weights divided by bandwidth. Store the weights in 4 bits instead of 16 and the bytes drop by 4×, and so does the step time, as long as the unpacking is cheap. That is the entire speed argument, and it is why the same tricks do not speed up prefill, where the arithmetic dominates. We come back to that in the speed section.

Number formats: what a bit buys you

Before we throw bits away we should know what each bit is doing. Every format you will meet is one of two kinds: floating point, which splits its bits into a sign, an exponent, and a mantissa, or integer, which just counts.

Floating point in one picture

A floating-point number is $(-1)^{\text{sign}} \times 1.\text{mantissa} \times 2^{\text{exponent} - \text{bias}}$. The exponent bits decide the range (how big and how small you can go), the mantissa bits decide the precision (how finely you can tell two nearby numbers apart). Every format is a decision about how to split a fixed budget of bits between those two jobs.

fp16 versus bf16

Both are 16 bits. fp16 spends 5 bits on exponent and 10 on mantissa, so it is precise (about 3 decimal digits) but tops out at 65,504 and underflows below about $6 \times 10^{-5}$. Gradients and attention logits overflow that easily, which is why fp16 training needs loss scaling.

bf16 (brain float) simply chops the bottom 16 bits off an fp32. It keeps the 8-bit exponent, so it never overflows where fp32 would not, and it pays with a 7-bit mantissa: only about 2 decimal digits. For training that trade is worth it, because neural nets care far more about not overflowing than about the third digit. Almost every modern model is trained and shipped in bf16.

fp8: two flavours for two jobs

Micikevicius et al. (2022) proposed two 8-bit floats, and both are now in hardware. e4m3 has 4 exponent bits and 3 mantissa bits, a maximum of 448, and eight representable values between 1 and 2. e5m2 trades a mantissa bit for an exponent bit: maximum 57,344, but only four values between 1 and 2. The convention is e4m3 for weights and activations (they need precision), e5m2 for gradients (they need range). Hopper and later GPUs have fp8 tensor cores, so fp8 is one of the few formats where both weights and arithmetic get cheaper.

Integers and NF4

int8 and int4 have no exponent. An int8 holds $-128$ to $127$, an int4 holds $-8$ to $7$, and every step is the same size. To represent real weights you multiply by a scale factor stored alongside, which is the whole subject of the next section.

Since the levels are evenly spaced, integers waste levels where weights are rare. Weights are roughly bell-shaped: most sit near zero, few sit near the maximum. NF4 (NormalFloat-4, from the QLoRA paper by Dettmers et al. 2023) fixes this by placing its 16 levels at quantiles of a normal distribution instead of evenly, so each level is used about equally often. It costs a table lookup to decode and buys a bit of accuracy at 4 bits.

Uniform affine quantization

Here is the problem in its simplest form. You have a tensor of real numbers, say weights between $-1.2$ and $2.5$. You want to store each one as an 8-bit integer, 0 to 255. How do you map one onto the other and back?

The obvious answer is a straight line: stretch the real interval onto the integer interval, round each value to the nearest integer, and remember the line so you can undo it. "Affine" just means "a straight line with an offset". Two numbers describe the line.

The scale $s$ is how much real-valued distance one integer step covers. With $b$ bits there are $2^b - 1$ steps between the smallest and largest integer:

So $s$ is the resolution of the quantized grid. Everything finer than $s/2$ is lost.

The zero-point $z$ is the integer that represents the real number 0. We need it because the real interval is usually not centred on zero, and we want real 0 to be exactly representable (padding and ReLU outputs are exactly zero, and a fuzzy zero would be a systematic bias):

Now quantizing a number means: divide by the scale, shift by the zero-point, round, and clip to the integer range in case rounding pushed us over the edge:

And dequantizing is the line run backwards, minus the rounding, which cannot be undone:

The difference $\hat{x} - x$ is the quantization error. Because we rounded to the nearest level, it is never larger than $s/2$ in magnitude for any value inside the range. That single fact drives the whole chapter: the error is proportional to the scale, and the scale is proportional to the range of the tensor.

Worked example: six numbers to int8

Symmetric versus asymmetric

The version above is asymmetric: the range is $[x_{\min}, x_{\max}]$ wherever it happens to be, and the zero-point shifts the grid so it fits. It uses every level.

Symmetric quantization forces the range to be $[-\alpha, \alpha]$ with $\alpha = \max|x|$, and sets $z = 0$. The scale becomes $s = \alpha / (2^{b-1} - 1)$, so for int8, $s = \alpha/127$. Half the levels go to negatives, half to positives.

Why give up levels? Because in the matmul, $\hat{W}x = s\,(q - z)\,x = s\,q x - s\,z\,x$, and that second term is extra work in the inner loop. With $z = 0$ it vanishes and the integer kernel is simpler and faster. Weights are roughly symmetric around zero anyway, so symmetric quantization is the default for weights. Activations after a GELU or a ReLU are lopsided, so asymmetric is the default for activations.

Per-tensor, per-channel, per-group

Now the question that matters most in practice. The scale is set by the largest value in the range. What if one weight in a matrix of ten million is a hundred times bigger than the rest?

Then that one weight sets the scale for all ten million. The step size becomes enormous, and every ordinary weight rounds to zero or to the first level. One outlier has erased the whole matrix. This is not hypothetical: transformer weight matrices and especially activations have exactly these outliers.

The fix is to not share one scale across so many numbers. There is a spectrum:

• Per-tensor: one $s$ for the whole matrix. Cheapest to store (2 bytes for millions of weights), most fragile.
• Per-channel: one $s$ per output row (or per input column). A $4096 \times 4096$ matrix gets 4096 scales, which is nothing. An outlier now only wrecks its own row. This is the standard for int8.
• Per-group: one $s$ per block of $g$ consecutive weights within a row, typically $g = 128$. A row of 4096 weights gets 32 scales. An outlier now only wrecks the 127 weights sitting next to it. This is the standard for int4 and int3, and it is what "g128" means in a model name like "Q4_K" or "4bit-g128".

The storage cost of grouping is small. A 16-bit scale per 128 four-bit weights adds $16/128 = 0.125$ bits per weight, so "int4 g128" is really 4.125 bits per weight (4.25 if you also keep a zero-point, or use a shared 8-bit scale of scales as in QLoRA's "double quantization").

Weights are easy, activations are hard

Everything so far quantized the weights, which sit still on disk and can be studied at leisure. A weight-only int4 model still streams fp16 activations through fp16 matmuls. To make the arithmetic cheaper you need to quantize the activations too, so both matmul inputs are int8. That is called W8A8, and it is where things get hard.

Outlier channels

Activations are computed on the fly, so their scale must be found on the fly (dynamic quantization) or fixed from a calibration set (static quantization). Either way there is a nastier problem. Dettmers et al. (2022) showed that once a transformer passes roughly 6.7B parameters, a handful of hidden dimensions, the same few in every layer and for every token, carry values up to 100× larger than the rest. They call them emergent outlier features, and they are not noise: zero them out and the model's perplexity collapses.

Per-tensor int8 on such a tensor is the outlier example above at full scale. Per-channel does not help either, because a "channel" of an activation is a hidden dimension and the outlier lives in exactly one of them. Worse, per-channel scaling on the activation side does not commute with the matmul: you cannot factor a per-input-column scale out of $XW$ the way you can factor a per-output-row scale out of $W$. So the standard fix for weights is unavailable for activations.

LLM.int8(): pull the outliers out

The LLM.int8() answer is a mixed-precision decomposition. Find the columns of $X$ whose magnitude exceeds a threshold (they use 6.0). Do those few columns, and the matching rows of $W$, in fp16. Do the other 99.9% of columns in int8 with per-row scales on $X$ and per-column scales on $W$ ("vector-wise" quantization). Add the two partial products. The result matches fp16 perplexity up to 175B parameters with zero accuracy loss, at the cost of a slower kernel because of the split.

SmoothQuant: move the difficulty into the weights

Xiao et al. (2022) noticed something cleaner. The outliers live in specific input channels of $X$, and a per-input-channel scale can be absorbed, just on the other side. If $X$ is tokens $\times$ channels and $W$ is channels $\times$ outputs, then for any positive diagonal matrix $S$:

Divide column $j$ of $X$ by $s_j$ and multiply row $j$ of $W$ by the same $s_j$, and the product is unchanged. The multiplication into $W$ happens once, offline. The division of $X$ folds into the previous LayerNorm's weights, so at run time nothing extra is computed. But now the outlier channel of $X$ has been shrunk by $s_j$, and $W$'s row $j$ has grown by $s_j$.

The trick is that weights are easy to quantize (per-channel scales work fine for them) so they can absorb some of the outlier without much harm. The knob is how much to migrate. SmoothQuant sets, per channel,

with $\alpha = 0.5$ by default. At $\alpha = 1$ all the difficulty moves into the weights (every activation channel gets a max of 1). At $\alpha = 0$ none of it moves. At $0.5$ the two sides end up with equal maximum magnitudes per channel, which is the geometric mean of their old ranges.

Post-training quantization methods

Everything above assumed we round each weight to its nearest level and move on. That is called round-to-nearest (RTN), and with per-group scales it is surprisingly hard to beat at 8 bits. At 4 bits and below the gap opens up, and two families of methods close it. Both quantize a trained model without retraining it, using a few hundred calibration examples, in minutes to hours on one GPU.

GPTQ: compensate as you go

Here is the observation. Rounding weight $w_1$ introduces an error. Nothing says the other weights in the same row have to ignore that. If $w_1$ rounded up a bit, its neighbours could round slightly down to keep the layer's output close to the original. What matters is not each weight but $\hat{W}X \approx WX$ on real inputs.

Frantar et al. (2022) make this precise. For one row $w$ of a weight matrix, minimize $\|wX - \hat{w}X\|^2$ over the calibration inputs $X$. Expanding it, the error depends on the weights only through $H = 2XX^\top$, the Hessian of that quadratic. GPTQ quantizes the row one column at a time: quantize $w_j$, compute the error $e_j = (w_j - \hat{w}_j)/[H^{-1}]_{jj}$, then update every not-yet-quantized weight by $-e_j\,[H^{-1}]_{j,:}$, which is the least-squares-optimal correction. Move to column $j+1$ and repeat.

The clever engineering is doing this for all rows at once (they share $H$), processing columns in blocks of 128 so the updates hit memory efficiently, and precomputing the Cholesky factor of $H^{-1}$ so nothing goes numerically bad. The upshot is that a 175B model quantizes to 4 bits in about four GPU-hours, and at 3 and 4 bits GPTQ roughly halves the perplexity gap that RTN leaves.

AWQ: protect the weights that matter

Lin et al. (2023) start from a different observation: not all weights are equally important. Roughly 1% of the weight channels, the ones that multiply large activations, account for most of the damage when quantized. Keep those 1% in fp16 and the perplexity gap nearly closes. But mixed precision is awkward in a kernel.

So AWQ protects them arithmetically instead. Scale the important input channel of $W$ up by $s > 1$ before quantizing and scale the matching activation down by $s$ (the SmoothQuant identity, run the other way). A bigger weight has a proportionally smaller relative rounding error, so scaling it up shrinks the error it contributes. The scale per channel is set from activation statistics, $s_j = \max|X_j|^{\alpha}$, with $\alpha$ searched on a grid to minimize output error. AWQ needs no backprop and no Hessian, is fast to run, and holds up well on instruction-tuned and multimodal models where GPTQ can overfit its calibration set.

Quantization-aware training

Post-training methods take the model as given. What if you could train the model to be quantized? That is quantization-aware training (QAT): keep fp32 "latent" weights, but in the forward pass quantize them and dequantize them, so the network computes with $\hat{w} = s \cdot \operatorname{round}(w/s)$ and learns to live with the grid.

There is an obvious problem. The gradient of $\operatorname{round}$ is zero everywhere (flat steps) and does not exist at the jumps. Backpropagate honestly and the latent weights never receive any gradient.

The straight-through estimator

The straight-through estimator (STE, Bengio et al. 2013) is a lie that works: in the backward pass, pretend that $\operatorname{round}$ is the identity. That is, use $\partial\hat{w}/\partial w = 1$ instead of $0$:

The gradient computed at the quantized weight is applied to the latent fp32 weight. The latent weight drifts smoothly; the quantized weight it produces jumps to a new level only when the latent one crosses a rounding boundary.

QAT costs a training run, so for LLMs it is used mostly for the aggressive end: 2-bit and ternary weights, fp8 training, and models designed to ship in int4 from day one. For fine-tuning, the everyday form is quantization-aware fine-tuning: QLoRA freezes an NF4-quantized base, trains LoRA adapters in bf16, and dequantizes the base on the fly in every forward pass. The base never learns to be quantized, but the adapters learn to correct for it.

Quantizing the KV cache

Weights are not the only thing being streamed during decode. The KV cache grows with every token, and for long contexts and large batches it can exceed the weights (the inference chapter has the numbers: Llama-2 7B at 4k context needs 2 GB per sequence). Every decode step reads all of it. So the same argument applies: fewer bits per cached key and value means less traffic and more sequences per GPU.

The cache is activation data, so it inherits the activation problem, with a twist that Liu et al. (2024, KIVI) made explicit. Keys have outlier channels (the same hidden dimensions across all tokens), so keys should be quantized per channel. Values do not, but they are consumed by a softmax-weighted sum across tokens, so values should be quantized per token. With that asymmetry, int8 KV cache is essentially free, int4 costs very little, and KIVI reports 2-bit with modest loss. Production engines expose fp8 and int8 KV cache as a one-line flag; int4 KV is common for long-context serving.

What actually happens in practice

The honest summary, for models in the 7B to 70B range, with per-group scales and a modern method:

• int8 weights are free. Per-channel RTN int8 is indistinguishable from fp16 on every benchmark anyone has run. There is no reason not to.
• int4 weights on a good method cost very little. A tenth or two of a perplexity point at g128 with GPTQ or AWQ, and a few tenths with plain RTN. Downstream task accuracy moves within noise.
• int3 is where it starts to hurt. Half a point to a point of perplexity, more on small models.
• Small models suffer more. A 125M model has fewer weights to spread the error over, and each one carries more information. int4 on a 125M model can add 3 to 10 perplexity points; the same recipe on 70B adds a tenth.
• W8A8 needs SmoothQuant or a decomposition; naive activation int8 breaks above 6.7B.

All numbers are approximate, assembled from figures reported in the GPTQ and AWQ papers and from community reports; they vary with calibration data, sequence length, and evaluation code. Treat them as a shape, not a table to cite. Run your own perplexity evaluation with code/lumen/eval.py before and after.

Speed: when quantization helps and when it does not

Decode is memory-bound, so bits are time

Come back to the opening. Decode at batch size 1 reads every weight per token and does almost no arithmetic per byte. Halving the bytes halves the time, up to the point where unpacking the integers becomes the bottleneck. Measured decode speed-ups for int4 weight-only on a single GPU are typically 2 to 3.5× over fp16 (the AWQ and GPTQ papers report roughly 3× on A100-class hardware), not the theoretical 4×, because of the dequantization overhead and the fact that some parts of the step (attention over the cache, the LM head, kernel launches) do not shrink.

Prefill is compute-bound, so bits are not time

Prefill processes the whole prompt in one pass: thousands of tokens against every weight. The weights are read once and used thousands of times, so the arithmetic dominates and the weight bytes are a rounding error. Weight-only int4 prefill is not faster than fp16 prefill; it is often slightly slower, because the kernel has to dequantize on top of the same fp16 matmul. The same goes for large-batch decode, which is prefill in disguise: once the batch is big enough that arithmetic dominates, the memory saving stops buying speed.

To speed up compute-bound work you must make the arithmetic itself cheaper, which means both operands in a format the tensor cores natively multiply: int8×int8 (W8A8, on Ampere and later) or fp8×fp8 (Hopper and later). That is why activation quantization, for all its difficulty, is the only thing that helps prefill throughput on big serving fleets.

Dequantization kernels

The unpacking is where the engineering lives. A weight-only int4 kernel loads packed nibbles, applies the group's scale (and zero-point), converts to fp16 in registers, and feeds the tensor cores. Done naively that conversion can cost more than the saved memory traffic. Good kernels (the "Marlin" kernel family, ExLlama, the llama.cpp CUDA and Metal kernels) interleave the weight layout on disk so the unpack is a few bit-shifts, and reach close to the bandwidth-bound ceiling at small batch. When you see two "int4" checkpoints with very different speeds, the difference is almost always the kernel, not the numbers.

The code

The companion file code/lumen/quantize.py implements everything in this chapter that fits in a hundred lines: affine and symmetric quantization, per-channel and per-group scales, an int8 linear layer that dequantizes on the fly, and a straight-through rounding function for QAT. The core is small enough to read here.

import torch

def quantize_affine(x, bits=8, symmetric=False):
    """Uniform quantization of a whole tensor. Returns (q, scale, zero_point)."""
    if symmetric:
        qmax = 2 ** (bits - 1) - 1
        scale = x.abs().amax().clamp(min=1e-8) / qmax
        zp = torch.zeros((), dtype=x.dtype)
        q = torch.clamp(torch.round(x / scale), -qmax, qmax)
    else:
        qmax = 2 ** bits - 1
        lo, hi = x.amin(), x.amax()
        scale = ((hi - lo) / qmax).clamp(min=1e-8)
        zp = torch.round(-lo / scale)
        q = torch.clamp(torch.round(x / scale) + zp, 0, qmax)
    return q, scale, zp

def dequantize(q, scale, zp):
    return scale * (q - zp)

def quantize_groupwise(w, bits=4, group=128):
    """Symmetric per-group quantization along the input dimension. w: [out, in]."""
    out, inn = w.shape
    assert inn % group == 0
    qmax = 2 ** (bits - 1) - 1
    wg = w.reshape(out, inn // group, group)
    scale = wg.abs().amax(dim=-1, keepdim=True).clamp(min=1e-8) / qmax
    q = torch.clamp(torch.round(wg / scale), -qmax, qmax)
    return q.reshape(out, inn).to(torch.int8), scale.squeeze(-1)   # scales: [out, in/group]

def dequantize_groupwise(q, scale, group=128):
    out, inn = q.shape
    return (q.reshape(out, inn // group, group).float() * scale.unsqueeze(-1)).reshape(out, inn)

class QuantLinear(torch.nn.Module):
    """Weight-only int8 linear: per-output-channel symmetric scales, dequantized in forward."""
    def __init__(self, linear: torch.nn.Linear):
        super().__init__()
        w = linear.weight.detach()
        scale = w.abs().amax(dim=1, keepdim=True).clamp(min=1e-8) / 127
        self.register_buffer("q", torch.round(w / scale).clamp(-127, 127).to(torch.int8))
        self.register_buffer("scale", scale)
        self.bias = linear.bias

    def forward(self, x):
        w = (self.q.float() * self.scale).to(x.dtype)     # dequantize on the fly
        return torch.nn.functional.linear(x, w, self.bias)

class RoundSTE(torch.autograd.Function):
    """round() in forward, identity in backward: the straight-through estimator."""
    @staticmethod
    def forward(ctx, x):
        return torch.round(x)
    @staticmethod
    def backward(ctx, g):
        return g

def fake_quant(w, bits=4):
    """QAT forward: quantize-dequantize with gradients flowing straight through."""
    qmax = 2 ** (bits - 1) - 1
    scale = w.abs().amax().clamp(min=1e-8) / qmax
    return RoundSTE.apply(w / scale).clamp(-qmax, qmax) * scale

Two things to notice. QuantLinear stores half the bytes of the original layer and produces the same output up to the rounding error, but on a GPU it is not faster: it dequantizes to a full fp16 matrix and calls the same matmul. Making it fast means writing a fused kernel, which is what the serving engines do for you. And fake_quant is the entire mechanism of QAT: wrap the weight in it during the forward pass and let autograd do the rest.

Practice

Further reading

• Dettmers, Lewis, Belkada, Zettlemoyer (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. The discovery of emergent outlier features and the mixed-precision decomposition.
• Xiao, Lin, Seznec, Wu, Demouth, Han (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. The migration identity and the α knob.
• Frantar, Ashkboos, Hoefler, Alistarh (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. Hessian-based error compensation; read Section 3 for the algorithm.
• Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. Salient channels, per-channel scaling search, and a fast kernel.
• Dettmers, Pagnoni, Holtzman, Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NF4, double quantization, paged optimizers.
• Micikevicius et al. (2022). FP8 Formats for Deep Learning. Why e4m3 and e5m2, and what they were measured to do in training.
• Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. Why keys go per channel and values per token.
• Bengio, Léonard, Courville (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. The straight-through estimator, in its original setting.