How to Regularize your LoRA Training Better

Six ways to add dropout-style noise to LoRA fine-tuning, compared across three datasets, two base models and five seeds. One of them, Drop-LoRA-Path, beats standard LoRA dropout and weight decay in every comparison.

Introduction

A year ago, while running supervised classification fine-tuning for ExPLoRA (ICML 2025), I encountered an odd result. I was training with LoRA (low-rank adaptation) and wanted to use regularization to avoid overfitting over multi-epoch fine-tuning. I tried two regularization settings - one, default LoRA dropout that was suggested in the original paper, and two, “drop-path” (i.e. stochastic depth) suggested for full-backbone fine-tuning of vision transformers (ViTs) like MAE. There wasn’t a clear reason to try the latter, as it was not previously a recommended setting. But empirically, it worked. Concretely, across different trained backbones and datasets, the “drop-path” regularization variant achieved lower validation loss and higher validation accuracy. As the focus of the paper was elsewhere, I didn’t think too much of this, and so I just left a note in the appendix:

We note that the original LoRA configuration outperforms other PEFT techniques when paired with the drop-path regularization technique

But this has remained a small itch that I’ve wanted to scratch. And so a year later, now armed with my own DGX Spark, I thought I’d test this appendix note’s generalizability to language models. It strikes me as weird - drop-path is essentially zeroing out the entire result of one layer’s contribution to the model’s residual path. Isn’t it too aggressive for low-rank fine-tuning? Why does it work better than vanilla dropout and why isn’t this a standard regularization tool for LoRA?

So this exploration is what came of pulling that thread. In this blog, we’ll aim to answer a few questions -

  1. Does the pattern I saw in vision transfer to language modeling?
  2. Does this “drop-path” regularization beat weight decay and LoRA dropout?
  3. Why does it work? Can we find basic mechanistic explanations?

To answer these questions, we’ll pretrain 286M parameter language models, define a ‘regularization ladder’ of six variants that change where and how the noise is applied, and sweep them across three downstream datasets. The variant we’ll call “drop-LoRA-path” consistently wins across all configurations. If you LoRA fine-tune, this is worth a try!

TL;DR

If your LoRA fine-tune overfits, replace lora_dropout with DLP at a drop rate of 0.3, the same in every layer. It costs nothing at inference!

Code: github.com/samar-khanna/lora-regularization

Background

Residual block, LoRA inside the branch, dropout and drop-path
Figure 1. (a) A transformer block adds the output of a branch \(f\), an attention or MLP sublayer, to the residual stream \(x\). (b) Inside \(f\), LoRA adds a trainable low-rank path \(BA\) next to each frozen linear map \(W\). For simplicity, \(f\) is drawn as a single linear map, so \(h = f(x)\). (c) Dropout zeroes individual elements of \(f(x)\). (d) Drop-path zeroes all of \(f(x)\) for an example. In (c) and (d), each row is one token, the rows are grouped by example, and white cells are zeroed.

Low-rank adaptation (LoRA)

Fine-tuning every weight of a large model is expensive, so LoRA freezes each pretrained weight matrix \(W \in \mathbb{R}^{d \times k}\) and learns a low-rank update on top of it (Figure 1b). For an input \(x\), the layer computes

\[h = W x + \frac{\alpha}{r}\, B A x,\]

where \(A \in \mathbb{R}^{r \times k}\) and \(B \in \mathbb{R}^{d \times r}\) are the only trainable parameters, structured as low-rank matrices with rank \(r \ll \min(d, k)\), and \(\alpha\) is a fixed scale. \(A\) is initialized randomly and \(B\) to zero, so the adapter starts as a no-op and fine-tuning begins exactly at the pretrained model.

Why regularize LoRA?

A low-rank update is itself a form of regularization, as it limits the degrees of freedom with which the model can move. However, it does not prevent overfitting. Fine-tuning data for a specific task or domain is often small, consisting of a few thousand to tens of thousands of high-quality examples, and getting the best validation performance usually takes several epochs. In that regime LoRA overfits. In our runs, validation loss increases after about 2000 steps (~4 epochs) while training loss keeps falling (Figure 4). The usual defenses are weight decay and dropout.

Dropout and drop-path

A transformer is a stack of residual blocks. Each block adds the output of a branch \(f\), an attention or MLP sublayer, to the residual stream (Figure 1a):

\[x' = x + f(x), \qquad x,\, f(x) \in \mathbb{R}^{L \times d},\]

where each example \(x\) consists of \(L\) tokens’ hidden states, each of dimension \(d\). The linear maps that LoRA adapts live inside these branches and act on each token separately (Figure 1b). Dropout and drop-path both regularize this update by zeroing parts of \(f(x)\) at random during training. They differ in which parts they zero.

Dropout zeroes each element of \(f(x)\) independently (Figure 1c):

\[x' = x + \frac{m \odot f(x)}{1-p}, \qquad m \in \{0,1\}^{L \times d}, \quad m_{ij} \sim \mathrm{Bernoulli}(1-p),\]

where \(\odot\) is element-wise multiplication. A new mask is drawn for every example.

Drop-path, also called stochastic depth, zeroes all of \(f(x)\) at once (Figure 1d):

\[x' = x + \frac{m}{1-p}\, f(x), \qquad m \in \{0,1\}, \quad m \sim \mathrm{Bernoulli}(1-p),\]

where the single mask value \(m\) is drawn per example and shared by all of its tokens. With probability \(p\) the block is skipped for that example. Drop-path is standard when training and fine-tuning ViTs. In both cases, dividing by \(1-p\) keeps \(\mathbb{E}[x'] = x + f(x)\).

Each training step therefore trains a random sub-network: dropout removes individual units, and drop-path removes whole blocks. The noise prevents parts of the network from relying too much on each other, forcing these parts to learn updates that are useful on their own. Under dropout, a unit cannot rely on any specific other unit being present. Under drop-path, a block cannot rely on the blocks before it having run. This can prevent co-adaptation in deep networks, as it helps each block individually contribute. At inference the full network runs, and it behaves roughly like an average over all of these sub-networks.

Figure 2 shows these sub-networks on a toy problem.

Figure 2. The difference between each regularizer on a toy problem. A 6-block residual MLP is fit to 30 noisy samples of a sine wave, starting from the same weights in every column. Thin lines are sub-networks with 30% of units removed (top row, what dropout trains) or 30% of blocks removed (middle row, what drop-path trains). Without regularization, and even with weight decay, the network's parts co-adapt and its sub-networks get steadily worse (bottom row). Dropout keeps the first kind accurate, and drop-path keeps both.

The LoRA regularization ladder

Rather than only compare standard LoRA dropout and drop-path, for this post I wanted to examine the regularization choices on the residual path more closely. Standard LoRA dropout and drop-path differ in two ways: where the noise goes, and whether it is applied per element or per example. Dropout acts on the adapter’s input, one element at a time, along the feature dimension of a token’s hidden state. Drop-path acts on the whole branch, one example at a time, as a binary choice determining whether the branch’s output is kept or dropped, usually for all of that example’s tokens. Varying both gives a ladder of six regularizers (Figure 3). Each rung changes one thing from the rung below: L1–L3 move element-wise dropout from the adapter’s input to its output, L3–L5 keep the noise on the adapter’s output and coarsen the mask from one value per element to one per token to one per sequence, and L6 extends drop-path from the adapter to the whole branch.Coarser masks appear only on \(BAx\) because a mask value shared by all of a token's features is a scalar for that token, so it passes through \(A\) and \(B\): \(BA(mx) = m\,BAx\). A per-token or per-sequence mask on \(x\) or \(Ax\) is therefore the same regularizer as L4 or L5. Only element-wise masks depend on where they are applied.

The LoRA regularization ladder drawn on the residual block of Figure 1
Figure 3. The LoRA regularization ladder, drawn on the residual block of Figure 1. The rungs run from the adapter's input (L1) out to the whole branch \(f\) (L6). L3, L4 and L5 all mask \(BAx\) and differ only in the shape of the mask.

To make this explicit, write \(m \sim \mathrm{Bernoulli}(1-p)\) for a random mask (i.e. entries are 1 with probability \(1-p\) and 0 otherwise), broadcast over any dimension its shape lacks. Leaving out the \(\alpha/r\) scale and the division by \(1-p\), each rung on the ladder computes:

Rung Name Computation during training Mask \(m\) Noise
L0 none \(f(x) = Wx + BAx\) — none
L1 dropout\((x)\) \(f(x) = Wx + BA\,(m \odot x)\) \(\{0,1\}^{L \times k}\) per element
L2 dropout\((Ax)\) \(f(x) = Wx + B\,(m \odot Ax)\) \(\{0,1\}^{L \times r}\) per element
L3 dropout\((BAx)\) \(f(x) = Wx + m \odot BAx\) \(\{0,1\}^{L \times d}\) per element
L4 token DLP \(f(x) = Wx + m \odot BAx\) \(\{0,1\}^{L}\) per token
L5 sequence DLP \(f(x) = Wx + m \odot BAx\) \(\{0,1\}\) per sequence
L6 drop-path\((Wx+BAx)\) \(x' = x + m \odot f(x)\) \(\{0,1\}\) per sequence

All six act only during training. At inference the adapter runs in full and can be merged into \(W\) as usual.

Example DLP implementation

DLP (implemented below as L5) takes a few lines, where L4 and L5 differ only in the mask’s shape, (batch, seq, 1) versus (batch, 1, 1):

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, rank=32, alpha=32, p=0.3):
        super().__init__()
        self.base = base.requires_grad_(False)                # frozen W
        bound = 1 / base.in_features ** 0.5
        self.A = nn.Parameter(torch.empty(rank, base.in_features).uniform_(-bound, bound))
        self.B = nn.Parameter(torch.zeros(base.out_features, rank))
        self.scale, self.p = alpha / rank, p

    def forward(self, x):                                     # x: (batch, seq, in_features)
        delta = (x @ self.A.T) @ self.B.T                     # BAx
        if self.training and self.p > 0:                      # Drop-LoRA-Path:
            keep = torch.rand(x.shape[0], 1, 1, device=x.device) >= self.p   # one coin per sequence
            delta = delta * keep / (1 - self.p)
        return self.base(x) + self.scale * delta

In summary, the regularization ladder allows us to vary the granularity and placement of activation noise along the residual path. Compared with standard LoRA dropout, DLP changes two things: which tensor is masked (BAx instead of x) and the shape of the mask (one value per example instead of one per element).

Setup

Base models. I pretrained two 12-layer Llama-style models with nanochat. Each has 286M parameters, of which 110M are non-embedding parameters. Both were trained on 2.2B tokens of FineWeb-Edu, about 20 tokens per non-embedding parameter. They differ in one setting: the drop-path base was pretrained with drop-path at a rate rising linearly with depth to 0.1, and the standard base was not. Most open LLMs are pretrained without drop-path, so the standard base is the main setting. The drop-path base tests a prediction in the results, and matches a common ViT setting. Pretraining curves are in Appendix A.

LoRA. Rank 32 and \(\alpha = 32\) on all six linear maps of every block (query, key, value, attention output and both MLP projections). AdamW with learning rate \(3 \times 10^{-3}\), effective batch size 16, and 4,500 steps on a cosine schedule. The loss covers response tokens only. These settings came from a 96-run sweep without regularization over adapter placement, rank, learning rate and batch size, guided by Thinking Machines’ LoRA Without Regret (Appendix B).

Datasets. Three supervised fine-tuning tasks, each small enough that 4,500 steps covers it several times:

Dataset Task Train / val examples Epochs in
4,500 steps
OpenCodeReasoning,
HackerEarth subset		 competitive programming with reasoning and solutions, split 90/10 by problem ~11.2K / ~1.2K ~6.4
GSM8K grade-school math word problems with worked solutions 7,473 / 1,319 ~9.6
Glaive function
calling v2		 first-turn function-calling conversations with responses of at least 100 tokens, 10K subsample 9,000 / 1,000 8.0

The datasets cover domains that may not be strongly represented in the pretraining data. For example, Glaive’s JSON function-call format barely appears in FineWeb-Edu, so the model has to learn something new rather than recall it.

Protocol. Each ladder rung was swept over \(p \in \{0.05, 0.1, 0.2, 0.3\}\), with the same rate in every transformer layer except for L6, which uses the linear depth schedule (for parity with ViTs). AdamW’s decoupled weight decay on \(A\) and \(B\) (no dropout) was swept over \(\{0.01, 0.05, 0.1, 0.2, 0.3, 0.4\}\). I ran every configuration with 5 seeds on 3 datasets and both base models, totaling 930 fine-tuning runs.

The metric used is the best average validation loss (cross-entropy on response tokens, in nats)Cross-entropy is the average of \(-\ln p\) over response tokens, where \(p\) is the probability the model assigns to the correct next token. Using the natural logarithm gives the loss in nats; log base 2 would give bits (1 nat ≈ 1.44 bits). Perplexity is \(e\) raised to the loss, so a loss of 0.75 nats is a perplexity of 2.1. achieved during training, as with early stopping. For each (dataset, base model, seed) tuple, each method uses its best regularization rate \(p\) (or weight decay). Methods are then compared in pairs that share the dataset, base model and seed, and therefore the adapter initialization, data order and validation split. That gives 15 pairs per base model (3 datasets \(\times\) 5 seeds), tested with a one-sided Wilcoxon signed-rank test.The signed-rank test compares paired values without assuming that their differences are normally distributed, which is hard to justify for 15 pairs drawn from three datasets with different loss scales. It is one-sided because the claim is directional: DLP reaches a lower validation loss than the method it is compared with. A two-sided test doubles every p-value, to at most \(1.2 \times 10^{-4}\), changing no conclusion.

Results

LoRA overfits without regularization

Training and validation loss during fine-tuning for each regularizer
Figure 4. Training loss (top) and validation loss (bottom) during fine-tuning on the standard base (mean of 5 seeds). Standard LoRA dropout and DLP use \(p = 0.3\), their best rate for every dataset and seed, and weight decay uses its best value, usually 0.4. Training loss is measured with regularization noise on, so dropout and DLP are higher partly by construction.

Figure 4 motivates the problem we are trying to solve. Without regularization, LoRA overfits: validation loss traces a signature U-curve around four epochs in (1,850 to 2,750 steps, depending on the dataset). Even as training loss keeps falling, validation loss climbs steeply with more passes over the dataset. The best weight decay value partly delays overfitting, but does not fully mitigate it on any of the three datasets. On the other hand, both dropout-style regularizers keep validation loss falling longer. Standard LoRA dropout reaches its best around step 3,400, and DLP around step 4,000.

Drop-LoRA-Path (DLP) is the best rung on the regularization ladder

Each method's validation loss minus DLP's on the standard base, with one dot per seed
Figure 5. Each method's validation loss minus sequence DLP's on the standard base, per dataset. Every method uses its best rate \(p\) (or weight decay) for each dataset and seed. Bars are means over 5 seeds, and dots are single seeds, each compared with DLP on the same seed. The purple line at zero is DLP; above it, DLP is better. Every dot is above zero, so DLP is better in all 15 pairs against every method (one-sided Wilcoxon \(p = 3 \times 10^{-5}\), the smallest possible with 15 pairs). No regularization is off the scale: DLP's validation loss is 0.06–0.08 nats (7–12%) lower.

Given the significance of regularization in preventing overfitting, Figure 5 compares the rungs of the LoRA regularization ladder, across 3 datasets and 5 seeds. Sequence DLP (L5) has the lowest validation loss, on every dataset (Figure 5). It beats standard LoRA dropout in all 15 pairs on each base, by 0.009 nats on average (1.4%, about 1% in perplexity). Token DLP (L4) is next, which reinforces the regularizing benefits of drop-path on the LoRA residual path. Of the other rungs, dropout\((Ax)\) (L2) comes closest. Weight decay trails DLP by 0.03 nats (5.5%) on average, even with its best value picked separately for every seed.

Paired comparisons with DLP on the standard base
Compared with DLP DLP better in Mean gap (nats)
No regularization 15/15 0.075
L1 dropout\((x)\), standard 15/15 0.009
L2 dropout\((Ax)\) 15/15 0.006
L3 dropout\((BAx)\) 15/15 0.012
L4 token DLP 15/15 0.005
L6 drop-path\((Wx+BAx)\) 15/15 0.021
Weight decay 15/15 0.032

Pairs are dataset × seed. Every comparison has \(p = 3 \times 10^{-5}\) (one-sided Wilcoxon signed-rank), the smallest possible value with 15 pairs, so correcting for multiple comparisons would not change any conclusion. Confidence intervals, absolute losses and the drop-path base are in Appendix C.

For scale, DLP lowers validation loss relative to no regularization by 0.060, 0.083 and 0.082 nats on OpenCodeReasoning, GSM8K and Glaive (7–12%, 6–8% in perplexity).

DLP’s lead grows with the drop rate

Change in validation loss relative to no regularization at each drop rate, for every rung on the standard base
Figure 6. Change in validation loss relative to no regularization at each drop rate, for every rung on the standard base (mean of 5 seeds, ±1 standard error of the paired difference). Lower is better. The dashed line is weight decay at its best value.

At \(p \le 0.1\) the rungs are close, and from \(p = 0.2\) on DLP is ahead on every dataset (Figure 6). On the standard base, DLP beats standard dropout in 12, 13, 15 and 15 of the 15 pairs at the four rates. Averaged over the four rates, it beats standard dropout across all 15 pairs, so the result doesn’t depend on a single well-chosen rate.

Weight decay at its best value is worth about as much as standard dropout at \(p = 0.2\) on GSM8K, between \(p = 0.2\) and \(0.3\) on Glaive, and only \(p = 0.05\) on OpenCodeReasoning.

For an additional ablation that tests the impact of changing the drop-path rate per layer, you can review Appendix D. The result confirms the default choice of keeping the same rate in every layer, which works best.

Full drop-path (L6) needs a base model pretrained with it

L6's and standard dropout's validation loss minus sequence DLP's at each drop rate, on both base models
Figure 7. Validation loss minus L5 seq. DLP's at each drop rate, for L6 and, for comparison, standard dropout (L1), on the standard base (solid) and the drop-path base (dashed) (mean of 5 seeds, ±1 standard error of the paired difference). The dotted purple line at zero is DLP; above it, DLP is better.

So far, I haven’t directly reproduced my original finding in ExPLoRA, where L6 outperformed standard dropout. Here, on the standard base it is the weakest rung in Figure 5. My hypothesis is that pretraining explains this discrepancy. Many ViT backbones, such as DINOv2, are pretrained with drop-path, so their residual streams are used to losing whole branches. Our standard base never saw drop-path. With LoRA, the original pretrained weights are frozen, and yet we drop its branches via L6. This breaks computations it did not learn to make robust, and the adapters spend capacity compensating.

To test this hypothesis, I trained the drop-path base. If it is right, L6 should catch up on this base. It partly does. At its best rate, its gap to DLP shrinks from 0.021 to 0.009 nats (2.9% to 1.3%)The paired shrinkage is 0.012 nats, 95% CI [0.006, 0.018], two-sided Wilcoxon \(p = 0.0003\)., and Figure 7 shows this at higher rates (\(p \ge 0.2\)), most visibly on Glaive.

However, the drop-path base does not seem to be a better starting point overall. Its best result per dataset and seed is 0.006 nats (0.7%) worse than the standard base’s on average. I use it only as a probe. For LoRA regularization specifically, L6 on either base is fine at low rates, where it roughly ties DLP at \(p \le 0.1\). In summary, DLP remains the preferred regularization method, where its wins are agnostic to the base’s pretraining.

Why does Drop-LoRA-Path work?

The intuition we can build on is this: when DLP drops an adapter, that layer computes exactly what the base model would for the whole example. So every training step runs a network that is partly the base model. The noise is also correlated across every token and feature of an adapter’s output. This has three testable effects: the fine-tuned model stays close to the base model, each adapter has to learn an update that works without the others, and training penalizes interactions between the entries of each adapter’s output. Unless noted, the measurements in this section use the adapters at the end of training on all three datasets, on the standard base, with every dropout-style rung at \(p = 0.3\).

DLP forgets the least

Of all the methods, DLP changes the base model’s behavior on general text the least (Figure 8).

KL divergence from the base model on pretraining text after fine-tuning on each dataset, for every method
Figure 8. \(\mathrm{KL}(p_\theta \,\Vert\, p_{\theta'})\) per token on held-out pretraining text, where \(\theta\) are the base model's weights and \(\theta'\) the fine-tuned model's, after training on each dataset (standard base; dropout-style rungs at \(p = 0.3\), weight decay 0.4). Bars are means of 5 seeds and dots are single seeds. Lower means the model has changed less.

Fine-tuning costs a model some of its general ability, otherwise known as “forgetting”. Denoting the base model’s weights as \(\theta\) and the fine-tuned model as \(\theta'\), we compute \(\mathrm{KL}(p_\theta \,\Vert\, p_{\theta'})\) on held-out pretraining text. Averaged over the three datasets, DLP’s model moved 1.6 nats per token, 26% less than standard dropout’s 2.1 and 62% less than the unregularized model’s 4.2.L6 drops whole blocks, frozen weights included, so its sub-networks are never the base model, and it forgets more than DLP on every dataset. Intuitively, this tracks: DLP reuses the base model’s exact computation per layer more frequently than other rungs during training, which limits the fine-tuned model’s drift. Mixout regularizes full fine-tuning on the same principle, by randomly resetting individual weights to their pretrained values.

DLP’s forgetting levels off

Here we plot \(\mathrm{KL}(p_\theta \,\Vert\, p_{\theta'})\) as before against next-token prediction accuracy on each dataset’s validation split, over the course of training. Standard dropout keeps forgetting as it learns (higher KL), while DLP’s forgetting levels off (Figure 9).

KL divergence from the base model against next-token accuracy over training, for standard dropout and DLP
Figure 9. \(\mathrm{KL}(p_\theta \,\Vert\, p_{\theta'})\) on held-out pretraining text against next-token accuracy on each dataset's validation set, measured every 250 training steps (standard base, \(p = 0.3\); mean of 5 seeds). Training runs from left to right, and the large circles mark each method's best validation loss.

For example, on GSM8K below about 78% accuracy the two curves are close. After that, standard dropout’s KL keeps rising with each point of accuracy, while DLP’s stays near 1.7. OpenCodeReasoning shows the same pattern, with DLP’s KL leveling off near 2.0. On Glaive the two curves stay close, and both rise in the last stretch of training.

DLP prevents the adapters from co-adapting

To measure co-adaptation, for each method we drop all but one transformer block’s 6 trained adapters and measure the resulting model’s validation loss minus the base model’s per dataset. Without regularization, the adapters in the last block are the weakest on their own across datasets. Under DLP, every block’s adapters work on their own (Figure 10).

Validation loss with only one block's adapters kept, relative to the base model, for each block
Figure 10. Validation loss when only one block's six adapters are kept and all others are removed, minus the base model's validation loss. Below zero, that block's adapters improve on the base model on their own. Standard base, end of training (mean of 5 seeds, ±1 standard error).

On GSM8K and Glaive the last two blocks are harmful in the full model too: removing either one’s adapters lowers the unregularized model’s validation loss by 0.10–0.23 nats (7–15%). Both dropout rungs remove this, and DLP’s blocks are the most useful alone: on average a single DLP block lowers validation loss by 0.18 nats (8.7%), against 0.10 (4.8%) for standard dropout.

Appendix E shows the results of swapping blocks of adapters between two separately trained runs of the same method, differing in seed. DLP is more robust to these swaps than standard dropout or no regularization.

DLP penalizes interactions within each adapter’s output

Staying close to the base model doesn’t explain the whole ladder, though. L3 drifts less from the base than standard dropout on OpenCodeReasoning and GSM8K (Figure 8), yet its validation loss is worse. L3, L4 and L5 put noise on the same tensor, \(BAx\), and differ only in the shape of the mask, yet DLP beats L3 in all 15 pairs (Figure 5). To understand why, we can expand the loss to second order to show what the mask shape changes: DLP’s mask penalizes interactions between entries of the adapter update \(BAx\), and L3’s mask cannot.

Recall, without noise, a block computes

\[x' = x + Wx + BAx\]

where we assume \(f\) is a single linear map as in Figure 1. L3, L4 and L5 multiply only the adapter’s term by the mask, so during training the block computes

\[\tilde{x}' = x + Wx + \frac{m}{1-p} \odot BAx,\]

where, as in the ladder, \(m \in \{0, 1\}^{L\times d}\) for L3, \(m \in \{0, 1\}^L\) for L4, and \(m \in \{0,1 \}\) for L5. Taking the difference of the noisy and noise-free outputs, the noise shifts the block’s output by

\[\delta = \tilde{x}' - x' = \epsilon \odot BAx, \qquad \epsilon = \frac{m}{1-p} - 1 .\]

Each entry of \(\epsilon\) is \(-1\) with probability \(p\), when the update is dropped, and \(\frac{p}{1-p}\) otherwise, so it has mean \(\mathbb{E}_m[\epsilon_i] = 0\) and variance \(\text{Var}_m[\epsilon_i] = \sigma^2 = \frac{p}{1-p}\).

Let \(\ell\) be one example’s loss as a function of the block’s output \(x'\), with gradient \(\nabla \ell\) and Hessian \(H\) at \(x'\), treating \(x', BAx \in \mathbb{R}^{Ld}\) (flattened vectors of \(Ld\) entries). Expanding the loss averaged over masks to second order gives

\[\begin{aligned} \mathbb{E}_m\big[\ell(\tilde{x}')\big] & = \mathbb{E}_m\big[\ell(x' + \delta)\big] \\ &\approx \ell(x') + \nabla\ell^\top\, \mathbb{E}_m[\delta] + \tfrac{1}{2}\, \mathbb{E}_m\big[\delta^\top H \delta\big] \\ &= \ell(x') + \underbrace{\tfrac{1}{2} \textstyle\sum_{i,j} H_{ij}\, (BAx)_i\, (BAx)_j\, \mathbb{E}_m[\epsilon_i\, \epsilon_j]}_{\Omega(x)}, \end{aligned}\]

since \(\mathbb{E}_m[\delta] = 0\). Training with the noise therefore minimizes the clean loss plus the last term \(\Omega(x)\), a penalty. The mask influences \(\Omega\) only through \(\mathbb{E}_m[\epsilon_i \epsilon_j]\). Note that when entries \(i\) and \(j\) share a mask value, their \(\epsilon\) values are also equal, so \(\mathbb{E}[\epsilon_i \epsilon_j] = \mathbb{E}[\epsilon_i^2] = \sigma^2\). When the mask values for \(i\neq j\) are independent, we have \(\mathbb{E}[\epsilon_i \epsilon_j] = \mathbb{E}[\epsilon_i]\mathbb{E}[\epsilon_j] = 0\), visualized in Figure 11.

Which pairs of entries of BAx share a mask value under L3, L4 and L5
Figure 11. Pairs of entries \((i, j)\) of \(BAx\) that share a mask value under (a) L3, (b) L4 and (c) L5, for 3 tokens with 3 features each. Shaded pairs have \(\mathbb{E}_m[\epsilon_i \epsilon_j] = \sigma^2\) and enter the penalty \(\Omega\); blank pairs have 0.

With \(BAx\) flattened token by token, token \(t\) holds entries \((t-1)d+1\) to \(td\), and the penalty for each rung is

\[\begin{aligned} \Omega_{\text{L3}}(x) &= \frac{\sigma^2}{2} \sum_{i=1}^{Ld} H_{ii}\, (BAx)_i^2, \\ \Omega_{\text{L4}}(x) &= \frac{\sigma^2}{2} \sum_{t=1}^{L}\, \sum_{i,j=(t-1)d+1}^{td} H_{ij}\, (BAx)_i\, (BAx)_j, \\ \Omega_{\text{L5}}(x) &= \frac{\sigma^2}{2} \sum_{i,j=1}^{Ld} H_{ij}\, (BAx)_i\, (BAx)_j. \end{aligned} \tag{1}\]

\(\Omega_{\text{L3}}\) counts each entry on its own, \(\Omega_{\text{L4}}\) adds every pair of entries in the same token, and \(\Omega_{\text{L5}}\) adds every pair of entries in the sequence.

For two entries \(i \neq j\), the pair term is their interaction. To second order,

\[H_{ij}\,(BAx)_i\,(BAx)_j \approx \Delta\ell_{i,j} - \Delta\ell_i - \Delta\ell_j ,\]

where \(\Delta\ell = \ell(x'+\delta) - \ell(x')\), \(\Delta\ell_i\) is the rise in loss from dropping entry \(i\) of \(BAx\) alone, and \(\Delta\ell_{i,j}\) the rise from dropping entries \(i\) and \(j\) together.Recall the second-order expansion of the loss around \(x'\) for a shift \(\delta\) of the block's output:\[\ell(x' + \delta) \approx \ell(x') + \nabla\ell^\top \delta + \tfrac{1}{2}\, \delta^\top H \delta .\]The rise in loss from the shift is therefore \(\Delta\ell = \ell(x' + \delta) - \ell(x') \approx \nabla\ell^\top \delta + \tfrac{1}{2}\, \delta^\top H \delta\). Dropping entry \(i\) of \(BAx\), and nothing else, removes the adapter's contribution to that one entry of the block's output:\[x' + \delta = x + Wx + BAx - (BAx)_i\, e_i , \quad \text{so} \quad \delta = -(BAx)_i\, e_i ,\]where \(e_i\) is the \(i\)-th unit vector. Substituting this \(\delta\),\[\begin{aligned} \nabla\ell^\top \delta &= -(BAx)_i\, \nabla\ell^\top e_i = -(\nabla\ell)_i\, (BAx)_i, \\ \delta^\top H \delta &= (BAx)_i^2\, e_i^\top H e_i = H_{ii}\, (BAx)_i^2, \\ \Delta\ell_i &\approx -(\nabla\ell)_i\, (BAx)_i + \tfrac{1}{2} H_{ii}\, (BAx)_i^2 , \end{aligned}\]and likewise for \(\Delta\ell_j\). Dropping entries \(i\) and \(j\) removes both contributions, so \(\delta = -(BAx)_i\, e_i - (BAx)_j\, e_j\), and since \(H\) is symmetric,\[\begin{aligned} \Delta\ell_{i,j} &\approx -(\nabla\ell)_i (BAx)_i + \tfrac{1}{2} H_{ii} (BAx)_i^2 \\ &\quad\, - (\nabla\ell)_j (BAx)_j + \tfrac{1}{2} H_{jj} (BAx)_j^2 \\ &\quad\, + H_{ij} (BAx)_i (BAx)_j . \end{aligned}\]The first two lines are \(\Delta\ell_i\) and \(\Delta\ell_j\), so what remains is the pair term. The pair term is positive when dropping the two entries together costs more than the sum of the two separate drops. For example, if both entries raise the logit of the correct next token, the pair term is positive by Jensen’s inequality, since cross-entropy is convex in that token’s logit.

Each penalty can be measured on a trained model. We can switch on a rung’s noise at \(p=0.3\) in every adapter, as during training, and record how much the training loss rises averaged over 64 training examples and fresh masks drawn for each. By the expansion above, this is \(\Omega\) averaged over those example sequences. The masks of the 72 adapters across the transformer are independent, so this rise is the sum of their penalties. We can thus compute \(\Omega_{\text{L3}}\), \(\Omega_{\text{L4}}\) and \(\Omega_{\text{L5}}\), and \(\Omega_{\text{L5}} - \Omega_{\text{L3}}\) leaves only the non-diagonal pair terms. Figure 12 compares all four with each rung’s validation loss.

Each rung's validation loss minus DLP's against L3's, L4's and L5's penalties and the pair terms
Figure 12. Each point is one rung on one dataset. The y axis is that rung's validation loss minus DLP's (the metric of Figure 5, at \(p = 0.3\)). The x axis is a penalty of its trained adapters, measured as the rise in training loss when a rung's noise is applied at \(p = 0.3\): (a) L3's per-entry penalty \(\Omega_{\text{L3}}\), (b) token DLP's \(\Omega_{\text{L4}}\), (c) sequence DLP's \(\Omega_{\text{L5}}\), and (d) the non-diagonal pair terms alone, \(\Omega_{\text{L5}} - \Omega_{\text{L3}}\). Dashed lines are least-squares fits across all 15 points, labelled with their Pearson correlation. Standard base, end of training; mean of 5 seeds, ±1 standard error.

There is a clear correlation between the pair terms a rung leaves in its adapters and its validation loss.DLP's three points sit at zero on the y axis by construction, and DLP trains against \(\Omega_{\text{L5}}\) at the rate used to measure it. Without them, the correlation over the other 12 points is \(r = 0.76\) for the pair terms and \(-0.02\) for the per-entry penalty. The per-entry penalty (i.e. diagonal terms) fails to predict validation loss. In Figure 12, the penalties of token and sequence DLP, which contain the pair terms, can better predict validation loss than L3 dropout’s per-entry penalty (i.e. Figures 12b, 12c vs. 12a). L3 dropout indeed does have the lowest per-entry penalty since it trains against that, but it also has the worst validation loss. DLP is the only rung whose penalty includes every pair term, and the pair terms measured on its adapters are about zero (Figure 12d).

Appendix F tracks the pair terms over training: DLP removes them as it trains, while standard dropout’s stay near their early level. Appendix G demonstrates that Gaussian noise with the same shape, which doesn’t drop adapter outputs, performs the same as DLP.

Discussion

When should you use drop-LoRA-path?

My biggest takeaway is that LoRA fine-tuning can be heavily under-regularized. As we’ve seen, validation loss can improve a lot with heavier regularization (of any type) when training for multiple epochs. Simply raising the regularization rate from 0.05 to 0.3 increases the gain in validation loss over no regularization from 1.7% to 8.5% for standard dropout, and from 2.3% to 9.7% for DLP. My second takeaway is that DLP was the best regularizer in this post, and it beat standard dropout on every dataset, base model and seed, with no cost to inference. So if your LoRA fine-tuning on a small dataset overfits within your training budget, replace lora_dropout with DLP at \(p=0.3\), with the same rate in every layer. However, if your fine-tuning budget is smaller and covers one or two epochs, you can likely use weight decay, standard dropout, or DLP at a lower rate.

Stochastic depth on adapters has precedent in vision. Adapters Strike Back found that stochastic depth on bottleneck adapters beats dropout inside them on a frozen ViT (+0.7 vs. +0.1 points on VTAB), and that stochastic depth on the frozen backbone matters even more. That is also the pattern I concurrently ran into in ExPLoRA. DropPath has also been applied to a prompt adapter for text-to-speech. Stable Diffusion LoRA tools (sd-scripts, LyCORIS) offer a module_dropout option that skips whole LoRA modules. sd-scripts draws one coin per batch rather than per example and does not rescale.

For LoRA on language models, prior work on dropout varies where in the transformer to drop (HiddenKey), masks the adapter’s parameters, or drops rank dimensions (DropLoRA, our L2). AdapterDrop removes adapters from lower layers to save compute. CoTo switches off all of a layer’s adapters at random, once per training step, with a probability that falls from 1 to 0 over the first three quarters of training. It reports better single-task accuracy, adapter merging and pruning than other LoRA variants. DLP instead drops each adapter independently for each sequence, at a fixed rate for the whole run, to regularize against overfitting. ALLoRA argues that dropout is unreliable in short LoRA runs; our runs are long enough to overfit, which is where dropout and DLP help (Figure 4). I have not found a controlled comparison of per-example drop-path on the LoRA branch against standard dropout and weight decay, a LoRA regularization ladder, or a study of how the base model’s pretraining changes the answer.

Don’t Drop Dropout argues for bringing layer dropout back into LLM pretraining. If that catches on, bases like our drop-path base will become common, and full drop-path during fine-tuning becomes a reasonable choice.

Limitations

Open questions

Appendix

A. Pretraining

Pretraining curves for the two base models
Pretraining loss curves for the standard and drop-path base models
Figure A1. Pretraining curves. Left: training loss (50-step moving average). Right: validation bits per byte.

Both models ran 4,200 steps of 524,288 tokens (2.2B tokens). The drop-path base’s training loss is higher because drop-path is active during training. Its validation loss, measured without drop-path, ends close to the standard base’s: 0.881 versus 0.875 bits per byte. On nanochat’s CORE evaluation the drop-path base scores 0.153 and the standard base 0.147.

B. LoRA hyperparameters

LoRA hyperparameter sweep
LoRA hyperparameter sweep: best validation loss against learning rate for each placement, batch size and rank
Figure B1. LoRA sweep on OpenCodeReasoning with the standard base, without regularization (3,000 steps, one seed): best validation loss against learning rate, one line per rank, for LoRA on the query and value maps only (top) or on all linear maps (bottom), at each batch size (columns). At learning rate 1e-2, LoRA on all linear maps does much worse, with best validation losses of 0.80–1.61 nats, so most of those points lie above the plot. The star marks the configuration used in the post.

The sweep covered adapter placement (query and value maps only, or all six linear maps) × rank {4, 8, 16, 32} × learning rate {3e-4, 1e-3, 3e-3, 1e-2} × effective batch size {16, 32, 64}, 96 runs in total. LoRA on all linear maps beats query and value only at every rank, by a wide margin, as LoRA Without Regret also found. Higher rank helps at every stable learning rate, and batch 16 beats larger batches in every case but one: query and value only at rank 4, where batch 64 is slightly better. The best run, rank 32 on all linear maps at learning rate 3e-3 and batch 16, is the configuration used throughout.

C. Per-dataset results

Absolute validation loss and confidence intervals

Standard base

  OpenCodeReasoning GSM8K Glaive
L0 none 0.505 ± 0.028 0.828 ± 0.003 1.114 ± 0.008
L1 dropout\((x)\), standard 0.454 ± 0.027 0.758 ± 0.003 1.037 ± 0.004
L2 dropout\((Ax)\) 0.450 ± 0.026 0.753 ± 0.002 1.036 ± 0.003
L3 dropout\((BAx)\) 0.457 ± 0.026 0.761 ± 0.002 1.041 ± 0.005
L4 drop-path\((BAx)\), token DLP 0.450 ± 0.025 0.752 ± 0.004 1.036 ± 0.003
L5 drop-path\((BAx)\), sequence DLP 0.445 ± 0.025 0.746 ± 0.002 1.031 ± 0.004
L6 drop-path\((Wx+BAx)\) 0.459 ± 0.027 0.764 ± 0.002 1.063 ± 0.003
Weight decay 0.492 ± 0.026 0.778 ± 0.004 1.048 ± 0.003

Drop-path base

  OpenCodeReasoning GSM8K Glaive
L0 none 0.507 ± 0.029 0.830 ± 0.003 1.124 ± 0.008
L1 dropout\((x)\), standard 0.456 ± 0.026 0.765 ± 0.003 1.048 ± 0.003
L2 dropout\((Ax)\) 0.455 ± 0.026 0.759 ± 0.001 1.044 ± 0.002
L3 dropout\((BAx)\) 0.461 ± 0.027 0.769 ± 0.001 1.051 ± 0.003
L4 drop-path\((BAx)\), token DLP 0.452 ± 0.025 0.758 ± 0.002 1.047 ± 0.003
L5 drop-path\((BAx)\), sequence DLP 0.447 ± 0.027 0.751 ± 0.001 1.041 ± 0.004
L6 drop-path\((Wx+BAx)\) 0.453 ± 0.026 0.767 ± 0.003 1.046 ± 0.003
Weight decay 0.495 ± 0.027 0.782 ± 0.002 1.056 ± 0.006

Mean ± standard deviation over 5 seeds, each method at its best grid value. The standard deviations are large on OpenCodeReasoning because each seed holds out a different validation split. The paired gaps below remove that variation.

Compared with DLP Standard base: mean gap [95% CI] Drop-path base: mean gap [95% CI]
No regularization 0.0748 [0.0678, 0.0818] 0.0741 [0.0675, 0.0807]
L1 dropout\((x)\), standard 0.0088 [0.0066, 0.0111] 0.0101 [0.0080, 0.0123]
L2 dropout\((Ax)\) 0.0055 [0.0040, 0.0071] 0.0063 [0.0048, 0.0078]
L3 dropout\((BAx)\) 0.0123 [0.0105, 0.0140] 0.0139 [0.0115, 0.0162]
L4 drop-path\((BAx)\), token DLP 0.0053 [0.0037, 0.0068] 0.0063 [0.0050, 0.0075]
L6 drop-path\((Wx+BAx)\) 0.0212 [0.0165, 0.0259] 0.0090 [0.0060, 0.0119]
Weight decay 0.0317 [0.0246, 0.0389] 0.0314 [0.0235, 0.0393]

Gaps are the method’s validation loss minus DLP’s in nats, averaged over 15 dataset-seed pairs, with t-based 95% confidence intervals.

D. Drop rate across layers

Uniform, ascending and descending DLP rate schedules

All DLP results in the main text use the same rate in every layer, while ordinary drop-path ramps the rate up with depth. To test whether the schedule matters for DLP, I reran it on the standard base (3 seeds, \(p \in \{0.1, 0.2, 0.3\}\)) with the rate rising linearly from 0 in the first layer to \(p\) in the last (ascending), or falling from \(p\) to 0 (descending). Each schedule gets its best peak rate.

Schedule Validation loss minus uniform’s (nats) Uniform better in
Ascending, 0 → \(p\) +0.017 9/9 pairs
Descending, \(p\) → 0 +0.061 9/9 pairs

Part of uniform’s advantage is that it applies twice the average noise at the same peak rate. Ascending and descending apply the same average noise, yet ascending beats descending in all 9 pairs, by 0.044 nats (6.6%). Heavy noise in the early layers hurts. Uniform is a good default.

E. Swapping adapters between training runs

Hybrids built from two training runs of the same method

This test of co-adaptation swaps adapters between training runs. We train the same method twice with different seeds, then build a hybrid from the first run’s adapters in blocks 0–5 and the second run’s in blocks 6–11. If each block’s update is individually useful, the hybrid works about as well as the two originals; if the blocks are co-dependent, the hybrid would have worse validation loss (Figure E1).

Validation loss of hybrids built from two training runs of the same method, minus the originals', for each method
Figure E1. Validation loss of a hybrid built from two training runs of the same method (blocks 0–5 from one seed, blocks 6–11 from another), minus the mean of the two originals' validation loss. Bars are means over 5 seed pairs and dots are single pairs; lower means each block's adapters depend less on their own run's other blocks. No regularization is off the scale: 0.063 on OpenCodeReasoning, 0.095 on GSM8K and 0.062 on Glaive. OpenCodeReasoning holds out different problems for each seed, so each of its pairs is scored on the 9–16 problems that both runs held out. Standard base, end of training.

On all three datasets, the hybrid’s validation loss is 0.06–0.09 nats (4–11%) worse than the originals’ without regularization, 0.017–0.018 (2–4%) worse with standard dropout, and only 0.011–0.012 (1–3%) worse with DLP.

F. Pair terms over training

Pair terms at each checkpoint for standard dropout and DLP

The pair terms in Figure 12 were measured at the end of training. Figure F1 tracks them through training, on the checkpointed reruns of standard dropout and DLP used for Figure 9.

Pair terms of the adapters during training for standard dropout and DLP
Figure F1. Pair terms \(\Omega_{\text{L5}} - \Omega_{\text{L3}}\) of the adapters during training, measured on training batches as in Figure 12, for standard dropout (L1) and DLP (L5) at \(p = 0.3\) on the standard base. Mean of 5 seeds, ±1 standard error. Dotted lines mark each method's lowest validation loss; on GSM8K both fall at step 3,250.

Standard dropout’s pair terms are there from the first checkpoint and stay near that level through training, including after its validation loss turns up. DLP’s start lower and fall to about zero by step 2,500. The gap in Figure 12 comes from DLP removing the pair terms as it trains, not from standard dropout accumulating them as it overfits.

G. Gaussian noise with DLP’s shape

DLP’s noise shape without dropping the adapter

DLP’s mask does two things: it removes an adapter’s whole output, so the layer computes what the base model would, and it shares one noise value across every token and feature of that output. To separate the two, I replaced DLP’s keep/drop factor \(m/(1-p)\) with a Gaussian factor \(g \sim \mathcal{N}\big(1, \tfrac{p}{1-p}\big)\), drawn the same way: once per sequence for each adapter. The noise \(g - 1\) has the same mean and variance as DLP’s \(\epsilon\), so the expansion in the main text gives the same penalty \(\Omega_{\text{L5}}\), but the adapter is never removed (about 6% of draws are negative and flip the adapter’s update for that sequence). This is Gaussian dropout with DLP’s mask shape. I trained it on the standard base at \(p = 0.3\), with 5 seeds on each dataset and every other setting as in the sweep.

Compared with Gaussian DLP Mean gap (nats) Gaussian DLP better in
L1 dropout\((x)\), standard 0.0098 15/15
L2 dropout\((Ax)\) 0.0065 14/15
L3 dropout\((BAx)\) 0.0133 15/15
L4 drop-path\((BAx)\), token DLP 0.0062 15/15
L5 drop-path\((BAx)\), sequence DLP 0.0010 9/15

Gaps are each method’s validation loss minus Gaussian DLP’s, averaged over 15 dataset-seed pairs, with every method at \(p = 0.3\).

Gaussian DLP matches DLP (two-sided Wilcoxon \(p = 0.19\)) and beats L3 in all 15 pairs, although L3 puts noise of the same variance on each entry of \(BAx\). So DLP’s advantage over L3 comes more from sharing noise across the adapter’s output than from removing the adapter. This is what the pair terms in the penalty capture.

Citation

Please cite this work as:

Khanna, Samar. “How to Regularize your LoRA Training Better.” samar-khanna.github.io, November 2026. https://samar-khanna.github.io/blog/2026/lora-regularization/

Or use the BibTeX citation:

@misc{khanna2026lorareg,
  author = {Khanna, Samar},
  title  = {How to Regularize your {LoRA} Training Better},
  year   = {2026},
  month  = nov,
  url    = {https://samar-khanna.github.io/blog/2026/lora-regularization/},
  note   = {Blog post}
}