LoRA and Parameter-Efficient Fine-Tuning

Fine-tuning a model means updating its weights. Updating weights with AdamW means storing, per parameter, the weight, its gradient, and two optimizer moments: about 16 bytes in mixed precision. For a 70B model that is 1.1 TB of GPU memory before you have loaded a single training example. For a 7B model it is 112 GB, more than any single consumer GPU has. And at the end you get a second 140 GB copy of the model that differs from the original by a little.

Here is the question that leads to LoRA: does fine-tuning really need to touch every one of those billions of numbers, or is the change the model needs to learn much smaller than the model itself?

How Intrinsic Dimensionality Gave Birth To LoRA

Li et al. (2018) asked a strange question about neural networks: if you are only allowed to train in a random $d$-dimensional subspace of the full parameter space, how big does $d$ need to be before you reach (say) 90% of full-training accuracy? Concretely they set $\theta = \theta_0 + P\,\theta_d$ with $P$ a fixed random matrix from $\mathbb{R}^d$ into the full parameter space and trained only the $d$ numbers in $\theta_d$. They called the smallest working $d$ the intrinsic dimension of the task, and found it was often hundreds or thousands, for networks with hundreds of thousands of parameters.

Aghajanyan, Zettlemoyer and Gupta (2020) ran the same experiment on pretrained language models being fine-tuned, and the numbers were striking: RoBERTa-large (355M parameters) could reach 90% of full fine-tuning accuracy on a sentence-pair task by training in a random subspace of only about 200 dimensions. Larger pretrained models had lower intrinsic dimension, not higher. Pretraining had put the model somewhere from which the task-specific adjustment was tiny.

That is the observation behind every parameter-efficient method: the update $\Delta W$ that fine-tuning learns is low-dimensional. Random subspaces are a clumsy way to exploit it (a random projection into a 355M-dimensional space is expensive to even store). Hu et al. asked whether there was a structured, cheap subspace that worked at least as well.

LoRA: Low Rank Adaptation

Take one weight matrix in the model, say a $d\times k$ projection $W_0$ in an attention layer. Full fine-tuning learns a new matrix $W_0 + \Delta W$ with $\Delta W$ also $d\times k$. Hu et al. (2021) constrain $\Delta W$ to have low rank $r$ by writing it as a product of two thin matrices:

What just happened: $W_0$ is frozen. Only $A$ and $B$ are trained. $A$ first squeezes the $k$-dimensional input down to $r$ numbers, then $B$ expands those $r$ numbers back up to $d$. The product $BA$ is a full $d\times k$ matrix, but it has rank at most $r$, so it can only represent $\Delta W$'s that "live in" $r$ directions. Given intrinsic dimensionality, that is exactly the kind of $\Delta W$ we expect.

Why low rank is a reasonable bet

A rank-$r$ matrix is one whose columns all live in an $r$-dimensional subspace. Equivalently (via the singular value decomposition) it is a sum of $r$ outer products $\sigma_i u_i v_i^\top$. The claim LoRA makes is that the fine-tuning update, whatever it is, is well approximated by its top few singular components. Hu et al. checked this directly: they trained full-rank updates on GPT-3 and found that the top singular directions of the learned $\Delta W$ captured most of it, and that LoRAs of rank 1 to 64 on the attention projections gave nearly indistinguishable downstream accuracy.

The demo below makes the "top few components capture most of it" idea concrete on a small matrix. A matrix with real structure is well approximated by a rank far below its size; pure noise is not. Fine-tuning updates are, empirically, closer to the first kind.

import math, torch, torch.nn as nn

class LoRALinear(nn.Module):
    """Wraps a frozen nn.Linear with a trainable low-rank update: h = W0 x + (alpha/r) B A x."""
    def __init__(self, base: nn.Linear, r: int = 8, alpha: float = 16.0, dropout: float = 0.05):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad_(False)                     # freeze W0 (and bias)
        d, k = base.out_features, base.in_features
        self.A = nn.Parameter(torch.empty(r, k))
        self.B = nn.Parameter(torch.zeros(d, r))        # B = 0  =>  BA = 0 at start
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.scale * (self.drop(x) @ self.A.T @ self.B.T)

    @torch.no_grad()
    def merge(self) -> nn.Linear:
        """Fold the adapter into the weight; returns a plain Linear with zero extra inference cost."""
        self.base.weight += self.scale * (self.B @ self.A)
        return self.base

code/lumen/lora.py has this class plus apply_lora(model, targets=[...], r, alpha) that walks a model and replaces the named linear layers, and merge_all to undo it.

Scaling and Initialization

Three details in that code are not decoration.

$B$ starts at zero. With $B=0$, $BA = 0$, so at step 0 the model is exactly the pretrained model. Training starts from a known-good point and moves away only as fast as the gradient asks. If both $A$ and $B$ were random, the initial $BA$ would be random noise injected into every adapted layer, and you would spend the first part of training undoing the damage. (Why not $A = 0$ instead? Then the gradient with respect to $B$ would be $\partial\mathcal L/\partial h\cdot (Ax)^\top = 0$ and the gradient with respect to $A$ would be $B^\top\partial\mathcal L/\partial h\, x^\top$, also zero: nothing would ever move. One of them must be nonzero, and the convention is $A$ random, $B$ zero.)

The $\alpha/r$ scale. The update is $\frac{\alpha}{r}BA$, not $BA$. The idea from the paper: if you double $r$, the product $BA$ is a sum of twice as many rank-1 terms and its typical magnitude grows; dividing by $r$ compensates, so you can change the rank without retuning the learning rate. $\alpha$ is then just a constant you set once (commonly $\alpha = 16$ or $\alpha = 2r$). Kalajdzievski (2023) argued that the correct compensation is $\alpha/\sqrt{r}$, not $\alpha/r$, if you want the update's scale to be truly rank-independent (the sum of $r$ random terms grows like $\sqrt r$, not $r$); this rsLoRA scaling lets higher ranks actually learn more instead of being throttled, and libraries offer it as an option.

Dropout on the adapter input (5–10%) is the only regularization most LoRA runs use; the low rank itself is already a strong regularizer.

And the learning rate: because the adapters are small and start at zero, LoRA wants a learning rate roughly $10\times$ higher than full fine-tuning of the same model; $1$–$3\times10^{-4}$ is the usual range where full fine-tuning would use $1$–$2\times10^{-5}$. Thinking Machines' "LoRA Without Regret" post (discussed below) reports this $10\times$ ratio holding remarkably consistently across models and tasks.

Merging and Serving

Here is the part that made LoRA the default rather than one PEFT method among many. Adapter methods that insert extra layers (the older "adapter" modules of Houlsby et al. 2019, or prefix tuning) make inference slower forever, because the extra computation is on the path. LoRA's update is a matrix added to a matrix, so once training is done you can compute

once, store $W'$ in place of $W_0$, and delete $A$ and $B$. The fine-tuned model has exactly the same architecture, size and latency as the base model. Zero inference overhead. And the merge is reversible: $W_0 = W' - \frac{\alpha}{r}BA$.

Serving many adapters from one base

The flip side of merging is not merging. Because the base weights are untouched, a single copy of the base model in GPU memory can serve hundreds of different fine-tunes: keep each customer's $(A, B)$ pair in memory (a few tens of MB each), and for each request compute $W_0x$ once for the batch and $B_iA_ix$ for that request's adapter $i$. S-LoRA (Sheng et al. 2023) and Punica (Chen et al. 2023) built the batched kernels that make this efficient, and it is now a standard feature of vLLM and similar servers. This is why "fine-tuning as a service" is economically possible: the marginal cost of hosting one more fine-tune is an adapter, not a model.

Which Layers to Adapt?

Hu et al. originally adapted only the attention projections, and among those found that $W_Q$ and $W_V$ together at rank 4 beat any single matrix at rank 8 for the same parameter budget, with all four attention matrices at rank 2 about as good. The MLP layers were left frozen, mostly to keep the study simple.

Practice moved on. Several findings, reported consistently across QLoRA (Dettmers et al. 2023), the Thinking Machines post and many practitioners' sweeps: adapting all linear layers, including the MLP (and MoE expert) matrices, matters more than rank. A rank-8 LoRA on every linear layer generally beats a rank-64 LoRA on attention only. The intuition: the MLP layers hold most of the parameters and most of the "knowledge"; leaving them frozen caps what the fine-tune can change. For MoE models, adapt the experts (each expert is a small MLP) rather than only the shared attention.

Rank. For instruction tuning and most task adaptation, $r\in[8, 64]$; results are flat across that range once all layers are targeted. Increase rank when the dataset is large (hundreds of millions of tokens or more) or the task needs the model to absorb a lot of new material; the "capacity" of a LoRA is roughly its parameter count, and a rank that is too small simply stops learning before the data runs out.

QLoRA

LoRA removes the optimizer-state and gradient memory, but the frozen base model still has to sit in GPU memory in 16-bit: 14 GB for 7B, 130 GB for 65B. Dettmers et al. (2023) asked: since the base is frozen anyway, why store it at 16 bits? QLoRA keeps the frozen base in 4-bit, dequantizes each weight matrix to bf16 on the fly when it is needed for a matmul, and trains bf16 LoRA adapters on top. The result: fine-tuning a 65B model on a single 48 GB GPU, with quality that the paper reports matches 16-bit LoRA. Three ingredients:

NF4: a 4-bit type shaped like the weights

With 4 bits you have 16 possible values per weight. The question is which 16. Ordinary int4 spaces them evenly, but neural network weights are approximately Gaussian: dense near zero, sparse in the tails, so evenly spaced levels waste half their codes on values that almost never occur. NormalFloat4 places the 16 levels at the quantiles of a standard normal: chosen so each of the 16 bins contains equal probability mass under $\mathcal N(0,1)$. Levels are dense near zero (where the weights are) and sparse in the tails. To use it, each block of 64 weights is divided by its absolute maximum (so it lies in $[-1,1]$), and each scaled value is rounded to the nearest NF4 level. The paper reports NF4 beating int4 and 4-bit float at equal size.

Double quantization

Every block of 64 weights stores its absmax as a 32-bit float: $32/64 = 0.5$ extra bits per weight, which is a 12% overhead on a 4-bit format. Double quantization quantizes those absmax constants themselves to 8 bits, in blocks of 256, with a second-level scale. The overhead falls to about $0.127$ bits per weight, saving roughly $0.37$ bits per parameter, about 3 GB on a 65B model.

Paged optimizers

Even with a 4-bit base, memory spikes during long sequences (activations, gradient checkpoints) can exceed the GPU. QLoRA uses NVIDIA unified memory to page the optimizer states to CPU RAM when the GPU runs out and back when there is room, the same way an OS pages memory to disk. It costs some speed on the steps where it triggers and prevents an out-of-memory crash on the rest.

When LoRA Fails

LoRA is not free. The constraint that makes it cheap, $\text{rank}(\Delta W)\le r$, is a real constraint, and there are tasks where the update the model needs does not fit.

Biderman et al. (2024), "LoRA Learns Less and Forgets Less", ran the cleanest comparison: Llama-2-7B, full fine-tuning versus LoRA, on two domains (code and math) in two regimes (instruction fine-tuning with tens of millions of tokens; continued pretraining with billions). Their findings, as reported:

• LoRA learns less. In continued pretraining on code and math, LoRA fell substantially short of full fine-tuning in target-domain performance, and the gap grew with data. In instruction fine-tuning the gap was smaller but present.
• LoRA forgets less. Full fine-tuning degraded the model's original abilities (measured on general benchmarks) much more than LoRA did. LoRA acts as a strong regularizer toward the base model, stronger than weight decay or dropout.
• Full fine-tuning's updates are high-rank. They measured the rank of $\Delta W$ learned by full fine-tuning and found it 10–100× higher than the ranks people use for LoRA, and increasing over training. That is the direct reason LoRA learns less: the target update does not fit in the subspace.
• Learning vs forgetting is a trade-off curve, and LoRA sits at a different point on it than full fine-tuning, not strictly below it.

So the failure mode is predictable: large domain shift, or lots of new knowledge. Teaching a model a new programming language, a new natural language, or a large body of facts from billions of tokens is a high-rank update. Teaching it a format, a style, a persona, a tool-calling convention, or a preference (which is most post-training) is low-rank, and LoRA is fine.

The RL angle

There is one regime where LoRA does not seem to lose at all: reinforcement learning. Thinking Machines' "LoRA Without Regret" (Schulman et al., 2025), a blog post with extensive experiments rather than a peer-reviewed paper, reports that for policy-gradient RL fine-tuning (the GRPO-style setups in the RL chapters), LoRA matched full fine-tuning even at very low rank, down to rank 1 in their runs. Their explanation is an information argument: a policy-gradient update extracts on the order of one bit of information per episode (was this trajectory better or worse than expected?), so even a long RL run absorbs far less information than a LoRA can store; the update is low-rank-ish because the signal is low-bandwidth. For supervised fine-tuning, by contrast, every token is a label and a large dataset can exceed the adapter's capacity, which is exactly the Biderman finding from the other direction.

Treat this as a well-supported empirical report from one lab rather than a theorem, but it matches what practitioners see: RL post-training with LoRA is now common, and it makes RL on large models feasible on modest hardware because the frozen base can be shared between the policy and the reference model.

LoRA Variants

Dozens of papers modify LoRA. Most fall into three buckets depending on what they are trying to fix: get more quality per parameter, start from a better initialization, or spend a given budget more wisely. Here is each with the one idea that defines it.

Performance variants: DoRA, MiSS

DoRA (Weight-Decomposed Low-Rank Adaptation; Liu et al. 2024) starts from an observation about full fine-tuning: it tends to change the direction of weight columns and their magnitude somewhat independently, while LoRA couples them. DoRA decomposes each weight matrix into a per-column magnitude vector $m$ and a direction matrix, applies LoRA to the direction only, and trains $m$ separately:

The norm is taken column by column, so the LoRA update can rotate each column freely and $m$ sets its length. The extra cost is $k$ parameters per matrix (the magnitudes), and the paper reports consistent gains over LoRA at the same rank, especially at low ranks. It merges just like LoRA.

MiSS (Matrix Shard Sharing; Kang et al. 2024, the successor to their earlier "Bone" method) drops the $BA$ product entirely. It splits the weight matrix into equal-width shards and adds one shared trainable matrix, the size of a single shard and initialized to zero, to every shard. The update is therefore a block-repeated matrix rather than a low-rank one, with a parameter count comparable to LoRA. The authors report better performance-per-parameter than LoRA and faster training, and the method is available in Hugging Face PEFT; it is newer and less independently validated than DoRA.

Initialization variants: PiSSA, LoRA+

PiSSA (Principal Singular values and Singular vectors Adaptation; Meng et al. 2024) keeps LoRA's shape but changes the starting point. Instead of $A$ random and $B = 0$, take the SVD of the pretrained $W_0 = U\Sigma V^\top$, initialize $B = U_{:r}\Sigma_{:r}^{1/2}$ and $A = \Sigma_{:r}^{1/2}V_{:r}^\top$ from the top-$r$ singular components, and freeze the residual $W_0 - BA$ as the base. The forward pass at step 0 is still exactly $W_0$, but now the trainable part is the most important directions of the weight rather than a random subspace, and the authors report faster convergence and better final loss than LoRA, including in the QLoRA setting.

LoRA+ (Hayou, Ghosh & Yu 2024) is a training fix, not an architecture: they show through a width-scaling analysis that giving $A$ and $B$ the same learning rate is suboptimal, because the two matrices see gradients of very different scale ($B$ starts at zero and is multiplied by the small activations $Ax$). The fix is a single extra hyperparameter: $\eta_B = \lambda\,\eta_A$ with $\lambda \approx 16$. Reported gains are modest (a few percent) but the change is free.

Budget variants: AdaLoRA, VeRA, IA³

AdaLoRA (Zhang et al. 2023) observes that a fixed rank for every matrix in every layer is wasteful: some need more, some less. It parameterizes each update as $P\Lambda Q$ in SVD-like form (with an orthogonality penalty on $P$ and $Q$), scores each singular value's importance from gradients during training, and prunes the least important ones to meet a global parameter budget, so rank is allocated across the model rather than fixed.

VeRA (Vector-based Random Matrix Adaptation; Kopiczko, Blankevoort & Asano 2023) pushes the intrinsic-dimension idea to its limit. $A$ and $B$ are random, frozen, and shared across all layers; the only trainable parameters are two small vectors per layer that rescale the rows of the random matrices: $\Delta W = \Lambda_b B \Lambda_d A$. The adapter for a 7B model shrinks from tens of MB to well under a megabyte, at some quality cost on harder tasks. Useful when you need to store or ship thousands of adapters.

IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations; Liu et al. 2022) does not modify weights at all; it learns three vectors per layer that rescale activations: the keys, the values, and the hidden activations of the MLP, elementwise. That is roughly $3d$ parameters per layer, about $10\times$ fewer than a rank-8 LoRA, and it is also mergeable (a rescaling of $K$ is a rescaling of $W_K$'s rows). It was designed for few-shot learning with very small budgets and is less expressive than LoRA when you have data.

What Actually Works In Practice

The low-regret regime

Put the Biderman result and the Thinking Machines result together and a clear picture emerges. There is a regime, which the Thinking Machines post calls low-regret, where LoRA configured properly (all layers, adequate rank, $10\times$ learning rate, moderate batch size) matches full fine-tuning in both sample efficiency and final quality. As reported, that regime covers:

• supervised fine-tuning on small-to-medium datasets (up to roughly the adapter's capacity in tokens, which for rank 32–128 on a 7B model is in the hundreds of millions of tokens or more);
• essentially all RL fine-tuning, at any rank;
• style, format, persona, tool-use and safety tuning, which are low-information updates.

Outside it, full fine-tuning wins: continued pretraining on billions of tokens of a new domain, and any task where you need to install a lot of new facts. The good news is that most post-training is inside the regime, which is why LoRA is the default and not a compromise.

PEFT methods for reinforcement learning

Three practical reasons LoRA and RL go together. First, the capacity argument above: RL's per-episode information is small, so a small adapter suffices. Second, memory: RLHF/GRPO training keeps a policy, a reference policy and often a reward model resident; with LoRA the policy and the reference share the same frozen base, and the reference is just "the base with adapters disabled," which halves the biggest memory line. Third, stability: the implicit regularization toward the base model reduces the reward-hacking drift that full-parameter RL is prone to, and makes the KL penalty easier to keep in range. The reported caveat is batch size: LoRA is less tolerant of very large batches than full fine-tuning, so keep effective batches moderate.

What to avoid

• Attention-only targets with a high rank. The rank is not the bottleneck; the frozen MLPs are. Adapt everything at a modest rank instead.
• Full fine-tuning learning rates. $10^{-5}$ with LoRA barely moves; you will conclude LoRA "doesn't work." Start at $10^{-4}$.
• Huge batches to speed things up. LoRA's loss degrades with batch size more than full fine-tuning's, as reported, and the rank does not fix it.
• Expecting new knowledge. If the goal is "make it know our 10 GB of internal docs," LoRA on the docs will underperform full fine-tuning, and both will underperform retrieval. Use the tool that fits.
• Merging into a quantized base. $W' = W_0 + BA$ requires a full-precision $W_0$; merging into an NF4 base re-quantizes the sum and loses accuracy. Merge into the 16-bit weights, then quantize.
• Sweeping $\alpha$, $r$ and the learning rate independently. See the confusion box above: fix $\alpha$'s convention, then sweep the learning rate, then (maybe) $r$.

The Bottom Line

LoRA works because fine-tuning updates are low-dimensional, and it won because it is mergeable: the fine-tuned model costs nothing extra to run. Configure it with all linear layers as targets, a rank of 8–64, $\alpha$ fixed by convention, $B=0$, and a learning rate about ten times what full fine-tuning would use; do that and, for most post-training, it matches full fine-tuning at a fraction of the memory. QLoRA extends the same recipe to models that would not otherwise fit on your GPU. Where it falls short is where the update is genuinely high-rank: absorbing lots of new knowledge from lots of new tokens. The variants are mostly small refinements; DoRA and PiSSA are the ones worth trying first, LoRA+ is a free tweak, and VeRA and IA³ are for when the adapter's size, not its quality, is the constraint.

Practice

Further reading

• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. The original; short and clear, with the rank and target ablations.
• Aghajanyan, Zettlemoyer & Gupta (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. The motivation.
• Li et al. (2018). Measuring the Intrinsic Dimension of Objective Landscapes. Where the random-subspace idea started.
• Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NF4, double quantization, paged optimizers.
• Biderman et al. (2024). LoRA Learns Less and Forgets Less. The cleanest LoRA vs full fine-tuning comparison.
• Schulman et al. / Thinking Machines (2025). LoRA Without Regret. The low-regret regime, the 10× learning rate, and the RL result, as reported.
• Kalajdzievski (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (rsLoRA).
• Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation.
• Meng, Wang & Zhang (2024). PiSSA: Principal Singular Values and Singular Vectors Adaptation.
• Hayou, Ghosh & Yu (2024). LoRA+: Efficient Low Rank Adaptation of Large Models.
• Zhang et al. (2023). AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning.
• Kopiczko, Blankevoort & Asano (2023). VeRA: Vector-based Random Matrix Adaptation.
• Liu et al. (2022). Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (IA³).
• Kang et al. (2024). MiSS: Revisiting the Trade-off in LoRA with an Efficient Shard-Sharing Structure.
• Sheng et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. Multi-adapter serving.