---
kind: concept
slug: sparse-fine-tuning
title: Sparse fine-tuning
summary: >-
  Adapt a pretrained model by updating only a fixed subset of its existing weights and freezing the rest. The subset can be chosen randomly, by rule, by projection or by gradient scores; SAM ranks entries by squared gradients from burn-in steps and reports the best average on seven RoBERTa-base tasks at 0.5% tunable weights (author-reported).
tags: [fine-tuning, sparsity, stability, generalization, nlp]
papers: ["2211.15583"]
related: [parameter-efficient-fine-tuning, transfer-learning]
status: draft
updated_at: 2026-10-19
---

## Overview

Sparse fine-tuning keeps most of a pretrained model fixed and updates only a
chosen set of entries. The fixed mask $M$ is the defining object: the tuned
model is $\theta = \theta^0 + M\Delta\theta$, with $\|M\|_0 = \lfloor mp \rfloor$
for a budget of fraction $p$ of the $m$ weights [arxiv:2211.15583 §2.1, def. 1].
The invariant all variants share is that every untouched entry keeps its
pretrained value. Storage per task then covers only the changed entries
[arxiv:2211.15583 app. A.5].

The paper's central claim is that several parameter-efficient methods are
sparse fine-tuned models in this sense, and that a fixed mask constrains
training in a way that bounds stability and generalization
[arxiv:2211.15583 §2.2] [arxiv:2211.15583 §3.1]. The open choice is which
entries to tune. Where the paper gives numbers, it reports them for one
backbone and one set of seven classification tasks.

## Variants

### Random and rule-based masks

Random masks pick entries without looking at the data. Mixout resets a
fraction of the tuned weights to their pretrained values
[arxiv:2211.15583 §2.2.1]. Rule-based masks select by a fixed rule: BitFit
tunes only bias terms, and MagPruning tunes the largest-magnitude weights
[arxiv:2211.15583 §2.2.2]. Adapter and LoRA add new parameters; the paper
treats them as equivalent sparse models of the same shape as the pretrained
network, with one matrix starting at zero [arxiv:2211.15583 §2.2.2] [arxiv:2211.15583 fig. 1].
Random masks reach 74.55 average in the paper's table, against 74.78 for full
fine-tuning, so the data-blind choice does not beat full tuning there
[arxiv:2211.15583 tab. 1]. Card: [Effectiveness of Parameter-Efficient Fine-Tuning](/papers/2211.15583).

### Projection-based masks

DiffPruning relaxes the mask to a continuous variable and projects it back to
valid masks. ChildPruning trains the full model first and then projects a mask
[arxiv:2211.15583 §2.2.3]. The paper calls the failure mode "projection
discontinuity": a small change in scores can flip the projection to a very
different mask [arxiv:2211.15583 §2.2.3] [arxiv:2211.15583 fig. 2]. In the
paper's runs, DiffPruning changes its mask every 600 steps and the training
loss returns to near its starting value after each change [arxiv:2211.15583 §5.2] [arxiv:2211.15583 fig. 3].
DiffPruning has the lowest average of the masked methods in the main table
(72.65) [arxiv:2211.15583 tab. 1].

### Gradient-scored masks (SAM)

SAM, the paper's own method, chooses the mask once, before training. It runs
a second-order expansion of the loss around the pretrained weights, drops the
curvature term by bounding the Hessian with a multiple of the identity, and
keeps the entries with the largest squared gradients [arxiv:2211.15583 §4]
[arxiv:2211.15583 eq. 10]. The gradients come from a burn-in phase at the
pretrained weights, whose length is set per task, between 600 and 2000 steps
[arxiv:2211.15583 §5.1]. The mask is then fixed for the rest of training
[arxiv:2211.15583 §4]. On the main table at 0.5% tunable weights, SAM averages
77.12 ± 1.51 against 74.78 ± 2.60 for full fine-tuning and 76.03 ± 2.35 for
Adapter [arxiv:2211.15583 tab. 1]. Under 10% training-set removal, SAM keeps
the lead with 76.40 average [arxiv:2211.15583 tab. 2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Many tasks, storage per task is the binding constraint | Any sparse method; the paper's tables do not give storage sizes | Only changed entries need saving [arxiv:2211.15583 app. A.5] |
| Training time is the binding constraint | Do not expect masking to help | The mask is applied to gradients, not to a smaller computation. SAM averages 2.02 h against 1.67 h for full tuning [arxiv:2211.15583 tab. 3] [arxiv:2211.15583 app. A.8] |
| Architecture must stay unchanged and no new modules are allowed | A mask on existing weights (Random, SAM, MagPruning) | Adapter and LoRA add parameters [arxiv:2211.15583 §2.2.2] |
| Only the model and a loss are available, with no task data for a pre-pass | Random or rule-based masks | Random and rule-based choices ignore the task data [arxiv:2211.15583 §1] |
| A few burn-in gradient steps are affordable and accuracy matters most | SAM | Best reported average on the paper's seven tasks [arxiv:2211.15583 tab. 1] |
| Sensitivity to training-data changes matters | Avoid projection-based masks | DiffPruning's mask instability is documented [arxiv:2211.15583 §5.2] [arxiv:2211.15583 fig. 3] |
| Tunable fraction far below 0.5% | Check the sweep first | Performance falls sharply below a threshold; the paper's sweep runs from 0.02% to 20% [arxiv:2211.15583 §5.2] [arxiv:2211.15583 fig. 5] |

All accuracy figures come from one backbone, RoBERTa-base, and seven
classification tasks [arxiv:2211.15583 §5.1]. The paper reports no
generation tasks and no other model sizes, so the ranking is not established
beyond that setting.

## Implementation sketch

The common core is one mask, computed before training, and a training loop
that moves only the masked entries. Variants differ only in how the mask is
chosen.

```pseudocode
# Common core of sparse fine-tuning  [arxiv:2211.15583 §2.1 def. 1]
# Input:  pretrained weights theta0 (m values), budget fraction p
# Output: fine-tuned weights theta = theta0 + mask * delta

def sparse_finetune(theta0, select_mask, data, p, train):
    # Choose the fixed mask: random, rule-based, projected or gradient-scored
    # [arxiv:2211.15583 §2.2] [arxiv:2211.15583 §4]
    mask = select_mask(theta0, data, floor(size(theta0) * p))   # (m,)
    # Entries outside the mask stay at their pretrained values  [arxiv:2211.15583 §4]
    delta = zeros_like(theta0)                                  # (m,)
    # Train only masked entries; the mask does not change during training
    # [arxiv:2211.15583 §4]
    return train(theta0, delta, grad_mask=mask, data=data)
```

For SAM, `select_mask` accumulates the loss gradient at `theta0` over the
burn-in steps, squares it, and keeps the top-`k` entries
[arxiv:2211.15583 §4] [arxiv:2211.15583 eq. 10]. The paper does not state the
optimizer, learning rate, batch size or weight-decay handling
[arxiv:2211.15583 §5.1]. Those choices change the results and are left to the
implementer. Whether the burn-in gradient is a sum or a mean is also not stated
[arxiv:2211.15583 §5.1].

## Contradictions

Within the paper, not between papers:

- **Random versus full tuning.** Section 5.2 says even the Random model beats
  full fine-tuning [arxiv:2211.15583 §5.2]. Table 1 gives 74.55 against 74.78
  on the average [arxiv:2211.15583 tab. 1], and the standard deviations are
  2.4 to 2.8, so the gap is within noise.
- **Stability bound versus the sweep.** The stability bound falls smoothly as
  the tunable fraction shrinks [arxiv:2211.15583 eq. 6]. The paper's sweep
  shows a sharp drop below a threshold [arxiv:2211.15583 §5.2]
  [arxiv:2211.15583 fig. 5]. The paper's proof assumes a random mask, while
  SAM's mask is chosen deterministically [arxiv:2211.15583 app. A.2]. So the
  bound does not directly cover SAM.

No second paper on this concept has a card, so no cross-paper disagreement is
recorded here.

## Papers

- [Effectiveness of Parameter-Efficient Fine-Tuning](/papers/2211.15583): argues that random, rule-based and projection-based parameter-efficient methods are sparse fine-tuned models, bounds their stability and generalization, and proposes SAM, a gradient-scored mask.