On the Effectiveness of Parameter-Efficient Fine-Tuning

In plain words

Fine-tuning a large pre-trained language model on one task usually means storing a full new copy of the model for that task. Parameter-efficient methods instead change only a small set of weights and keep the rest frozen, so each task needs little extra storage. Many of these methods have worked surprisingly well, and they also seem more stable from run to run than full fine-tuning. Nobody had a clear explanation for why.

This paper argues that most of these methods are doing the same thing: they pick a small subset of the weights to update, which makes them sparse fine-tuned models. Keeping most weights fixed acts like a constraint on training, which can make results steadier and generalize better. The open question is which weights to pick. The authors propose a way to choose them by looking at gradients from a few steps at the start, before the chosen weights are trained. The method is called SAM, and it is tested on a set of language understanding benchmarks.

TL;DR

Parameter-efficient fine-tuning methods (random, rule-based and projection-based) are all sparse fine-tuned models, and the paper gives stability and generalization bounds that improve as the number of tuned weights shrinks. The open problem is choosing which weights to tune. SAM chooses them from squared gradients over a few burn-in batches at the pre-trained weights §4. The reported averages beat the baselines (author-reported). The paper reports no compute saving; the benefit it names is storage app. A.5.

Problem

Full fine-tuning gives a new full-size model per task §1. Parameter-efficient methods store only the changed parameters, and they are reported to be more stable and sometimes better than full tuning §1. The reason was not known §1. Choosing the tunable subset is NP-hard, because the feasible set of masks is non-convex §2.1, eq. 1 §2.2. The paper also argues that the two existing data-aware strategies have weaknesses: random and rule-based choices ignore the task data, and projection-based choices are unstable §1 §2.2 fig. 2.

Method

Unified view. Let \theta^0 be the pre-trained weights, m=\dim(\theta), and M a diagonal binary mask. A p-sparse fine-tuned model has \theta = \theta^0 + M\Delta\theta with \|M\|_0 = \lfloor mp \rfloor [§2.1, def. 1]. Selecting the mask and the values is written as the problem below §2.1, eq. 1:

\min_{\Delta\theta, M} \mathcal{L}(\theta^0 + M\Delta\theta) \quad \text{s.t.} \quad \|M\|_0 = \lfloor mp \rfloor,\; M_{ij} = 0\ \forall i \ne j,\; M_{ii} \in \{0,1\} eq. 1

Existing methods as sparse fine-tuned models §2.2:

1. Random approaches: Random picks entries at random with a given ratio; Mixout resets a fraction of tuned weights to their pre-trained values §2.2.1.
2. Rule-based approaches: BitFit tunes only bias terms; MagPruning tunes the largest-magnitude weights; Adapter and LoRA add new parameters. For Adapter and LoRA the paper uses an equivalent model with the same structure as the pre-trained one: Adapter's matrix A starts at zero, and LoRA's matrix B starts at zero, so both give the same outputs as the pre-trained model §2.2.2 fig. 1.
3. Projection-based approaches: DiffPruning relaxes the mask to a continuous variable, then projects it back onto the set of valid masks; ChildPruning trains the full model, then projects a mask §2.2.3. The paper calls the problem with these methods "projection discontinuity": the set of valid masks is non-convex, so a tiny change in the scores can flip the projection to a very different mask §2.2.3 fig. 2.

Regularization view. With the mask M held fixed, the constrained problem can be written as an unconstrained regularized problem L_R, and optimizing it bounds the original loss from above §3.1, eq. 4:

L_R = \min_\theta \mathcal{L}(\theta) + \|(I - M)(\theta - \theta^0)\|^2 \le \bar{L} eq. 4

Stability. The paper uses pointwise hypothesis stability: removing one training sample should change the loss on that sample only a little [§3.2, eq. 5]. Under a Lipschitz loss and a local second-order argument around the minimizer, the paper bounds the expected change for the regularized problem §3.2, eq. 6 app. A.2. The bound depends on the sparsity p through the term (1-p):

\mathbb{E}\big[|\ell(\mathcal{A}(S^i), z_i) - \ell(\mathcal{A}(S), z_i)|\big] \le \frac{2\rho^2}{(\Lambda_{min} + 2(1-p))\, n} eq. 6

Generalization. Combining the stability bound with a standard result gives a high-probability bound on the gap between expected and empirical risk [§3.3, eq. 7]. The paper notes that as the mask gets sparser, the second term falls, but the empirical error can rise when there are too few tunable parameters, so the generalization error may first fall and then rise as sparsity decreases §3.3. The paper says it tests this conjecture experimentally §3.3 fig. 5.

SAM (Second-order Approximation Method) §4:

• Take a second-order Taylor expansion of the loss around \theta^0, with a diagonal Hessian H = \mathrm{diag}\{h_1, \dots, h_m\} assumed positive semidefinite §4.
• Solve the resulting problem exactly for the mask. The optimal mask keeps the \lfloor mp \rfloor entries with the largest |g_i^2 / h_i|, where g = \nabla\mathcal{L}(\theta^0) §4, eq. 9.
• Because h_i is costly to compute, replace H with D = |\lambda_{max}| I, where \lambda_{max} is the largest eigenvalue of H. This gives an upper bound on the approximation, and the ranking becomes g_i^2 alone [§4, eq. 10].
• Estimate g by accumulating gradients over a few burn-in steps at \theta^0. Keep the selected entries fixed for the rest of training §4 §5.1.

\hat{M}_{ii} = \mathbb{1}\Big(\sum_{j=1}^m \mathbb{1}\big(|g_i^2/h_i| > |g_j^2/h_j|\big) \ge m - \lfloor mp \rfloor\Big) eq. 9

\min_{\Delta\theta} \mathcal{L}(\theta^0) + g^{\mathrm T} M\Delta\theta + \tfrac{1}{2}(M\Delta\theta)^{\mathrm T} D\, M\Delta\theta \quad \text{s.t. the mask constraints of eq. 1} eq. 10

The selection step, as the section describes it, in our own pseudocode:

# SAM mask selection  [§4] [eq. 10] [§5.1]
# Input:  pre-trained weights theta0 (m values), sparsity p, burn-in steps T
# Output: binary mask with floor(m * p) ones

def select_sam_mask(theta0, batches, p, burn_in_steps):
    # Accumulate the loss gradient at theta0 over the burn-in batches  [§5.1]
    grad_sum = zeros_like(theta0)                      # (m,)
    for batch in batches[:burn_in_steps]:
        grad_sum = grad_sum + gradient(theta0, batch)  # (m,)
    # Score each entry by its squared gradient; the curvature term is
    # dropped by the bound D = |lambda_max| * I  [§4] [eq. 10]
    scores = grad_sum ** 2                             # (m,)
    budget = floor(size(theta0) * p)
    # Keep the highest-scoring entries; all others stay fixed  [§4] [eq. 9]
    mask = zeros_like(theta0)                          # (m,)
    mask[top_k_indices(scores, budget)] = 1
    return mask

Training then updates only the selected entries; the paper states that the selected set is not changed during optimization §4. The paper does not state the optimizer or the learning rate. The repository implements this by zeroing gradients of unselected encoder entries runner.py.

Hyperparameters

Results

Author-reported. Setup: RoBERTa-base, seven GLUE and SuperGLUE tasks (CoLA, STS-B, MRPC, RTE, CB, COPA, WSC) evaluated on the original dev sets, 10 seeds, mean ± std §5.1 tab. 1.

Main table tab. 1:

Data-perturbation stability tab. 2: the same setup, but each run uses a training set with a random 10% removed; ten such sets §5.2.

Training time, hours tab. 3, app. A.5. The paper says masking does not save compute, because the mask is applied to gradients rather than to a smaller computation app. A.5 app. A.8:

Table 3 omits MixOut and MagPruning. The paper also says the sparse models need less storage, because only the changed parameters need saving app. A.5. No storage numbers are given.

Tunable parameter ratio, % tab. 4, app. A.6: FullTuning 100; ChildPruning 0.5; Adapter 0.72; LoRA 0.91; Bitfit 0.09; DiffPruning 0.5; Random 0.5; MixOut 0.5; MagPruning 0.5; SAM 0.5.

Significance tab. 5, tab. 6, app. A.7: t-tests compare SAM's per-seed scores with each baseline's. Tables 5 and 6 give t-statistic / p-value. The paper colours cells green when SAM is significantly better (t > 0, p < 0.05) and red when a baseline is significantly better. Tables 5 and 6 are in the appendix app. A.7.

Sparsity sweep §5.2 fig. 5: SAM was run at tunable ratios {0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2}. The paper reports that the mean and std on most tasks fall as the ratio falls, until a threshold below which performance drops sharply and training error rises §5.2 fig. 5.

Stability vs performance §5.2 fig. 4: rank correlation between stability rank and performance rank, Spearman \rho = 0.4356, p = 0.000014 §5.2.

Projection instability §5.2 fig. 3: DiffPruning on CB, mask updated every 600 steps; training loss returns to near its initial value after each mask change §5.2 fig. 3.

Paper vs code

The paper's abstract footnote links the repository, which is how we take it as the official code. The repository, at commit ea1a73b40dfe9aaa740bf1979caf0b87357d20f1, contains only the SAM training code (src/) and one training script for CoLA (scripts/train.sh). Baseline code and the per-task scripts are not in the repository. Its README names the paper and describes SAM README.md. It has no LICENSE file, so the code is cited for comparison only and nothing is quoted from it.

• Scoring rule matches §4 ranking, not the eq. 9 form. Burn-in gradients are summed per entry runner.py, then the default path ranks by the absolute value of that sum runner.py. Ranking by |\sum g| gives the same order as ranking by (\sum g)^2, so this matches the g_i^2 ranking of eq. 10. It does not divide by h_i as in eq. 9, and the Hessian-based branches are not used by the CoLA script runner.py.
• Budget is per tensor, not global. The paper states one global constraint, \|M\|_0 = \lfloor mp \rfloor §2.1, eq. 1. The code computes k = \lfloor p \cdot \text{numel} \rfloor separately for each parameter tensor runner.py. The total is the same, but the split across layers is set by the per-tensor count, not by one global top-k.
• Only encoder tensors are masked. The mask is applied only to parameter names starting with encoder runner.py. Other parameters, including task heads, are not masked. The paper's 0.5% refers to the whole model's parameter count §5.1 app. A.6 (not further specified). The head's size is not stated.
• No optimizer step during burn-in. The burn-in branch zeroes gradients and returns before the optimizer step runner.py, so the model stays at \theta^0 until the mask is built. The step that builds the mask (burn-in step 600 for CoLA) is trained with the masked gradient runner.py runner.py.
• Early stopping. The paper states tolerance 40 only §5.1. The CoLA script sets the patience to 40 evaluations, evaluates every 500 steps, and does not stop before 20000 steps train.sh main_runscript.py.
• Learning rate and batch size are not stated in the paper. The CoLA script uses learning rate 1e-4 and batch size 16 train.sh. Those values come from one task's script and should not be assumed for the others.
• Optimizer and clipping are not stated in the paper. Code defaults are AdamW with eps 1e-8, gradient clipping at 1.0, and weight decay 0.01 on non-bias, non-LayerNorm weights runscript.py runscript.py model_setup.py model_setup.py.
• Burn-in values per task (600, 1000, 700, 2000) are in the paper §5.1. The repository has only the CoLA value (600) and cannot confirm the others.
• Baselines were re-implemented in jiant §5.1. The repository has no code for them, so their hyperparameters can only be checked against the paper.

No discrepancies were checked for the baseline methods, because their code is not in the repository.

Pitfalls

• Three scoring rules appear in the paper. The Theorem form divides by the Hessian diagonal h_i §4, eq. 9. The final rule drops h_i §4, eq. 10. The code uses the summed-gradient magnitude runner.py. Only the last two are fully implementable from the paper. The paper never estimates h_i or \lambda_{max}.
• Burn-in gradient definition is unstated. The paper says \nabla\mathcal{L}(\theta^0) is estimated by accumulating gradients over a few steps §5.1, without saying whether the sum or the mean is used. Ranking by squared gradients is invariant to a shared scale factor, so this matters only if the scores are compared across runs.
• Burn-in length is a tuned hyperparameter. It is selected on the split used for early stopping §5.1. The test numbers come from the original dev set. The paper does not say how sensitive results are to the burn-in choice.
• Gradient masking does not freeze weights under weight decay. The code zeroes gradients of unselected encoder entries runner.py. The optimizer applies weight decay to all non-bias, non-LayerNorm parameters model_setup.py. With AdamW's decoupled decay, the unselected weights still shrink each step, although they receive no gradient update. This is inferred from the code and was not run.
• Stability proof assumes a random mask. The proof takes an expectation over the mask with \mathbb{E}(1-M_{ii})^2 = 1-p app. A.2, which holds for a mask chosen at random with probability p. SAM chooses its mask deterministically. The bound is therefore for the regularized random-mask model, and it is not a direct proof for SAM.
• Stability bound needs assumptions. It assumes the solution on the reduced dataset stays close to the full one, and it holds for large n §3.2 app. A.2. The paper's own sweep shows instability below a threshold sparsity §5.2, which the bound does not predict.
• Index and expectation typos. The empirical risk in the generalization theorem drops the index: it reads \ell(\mathcal{A}(S), z) instead of \ell(\mathcal{A}(S), z_i) §3.3, eq. 7. The stability proof averages over i \sim U(m) app. A.2, while the definition uses U(n) §3.2.
• Random beats full tuning only on one table. Section 5.2 says "even the Random model outperforms the FullTuning model" §5.2. The averages in Table 1 are 74.55 for Random and 74.78 for FullTuning tab. 1, so that is not true on the average. In Table 2, Random is 74.83 vs 74.81 tab. 2. The gap is well within the standard deviations, which are 2.4 to 2.8 for these two rows.
• Significance colour. Table 6 colours Adapter's RTE cell green at p = 0.05 (2.14 / 0.05) tab. 6. The caption's threshold is p < 0.05, and two decimals cannot settle it.
• Unmatched budgets. Adapter, LoRA and BitFit use different tunable ratios (0.72%, 0.91%, 0.09%) from the 0.5% used for the others app. A.6 tab. 4. The paper says the Adapter and BitFit budgets are fixed by the method.
• Compute is not reduced. SAM's average training time is 2.02 hours against 1.67 for full tuning tab. 3. The paper says masking does not save computation app. A.5 app. A.8.
• Single model family and task set. All results use RoBERTa-base on seven classification tasks §5.1. The paper does not test other models, sizes or generation tasks.

Lineage

• Builds on: the sparse and parameter-efficient methods it reanalyses, each named in the paper with its citation key: Adapter (houlsby2019parameter), Mixout (lee2019mixout), MagPruning (han2015deep), DiffPruning (guo2021parameter), ChildPruning (xu2021raise), BitFit (arXiv:2106.10199, cited as zaken2021bitfit) and LoRA (arXiv:2106.09685, cited as hu2021lora) §2.2 [bib].
• Compared against: Random, Mixout, BitFit, MagPruning, Adapter, LoRA, DiffPruning and ChildPruning, all re-implemented in jiant §5.1.
• Superseded by: not stated. No later work is named in this card's sources.