Sparse fine-tuning

Overview

Sparse fine-tuning keeps most of a pretrained model fixed and updates only a chosen set of entries. The fixed mask M is the defining object: the tuned model is \theta = \theta^0 + M\Delta\theta, with \|M\|_0 = \lfloor mp \rfloor for a budget of fraction p of the m weights 2211.15583 §2.1, def. 1. The invariant all variants share is that every untouched entry keeps its pretrained value. Storage per task then covers only the changed entries 2211.15583 app. A.5.

The paper's central claim is that several parameter-efficient methods are sparse fine-tuned models in this sense, and that a fixed mask constrains training in a way that bounds stability and generalization 2211.15583 §2.2 2211.15583 §3.1. The open choice is which entries to tune. Where the paper gives numbers, it reports them for one backbone and one set of seven classification tasks.

Variants

Random and rule-based masks

Random masks pick entries without looking at the data. Mixout resets a fraction of the tuned weights to their pretrained values 2211.15583 §2.2.1. Rule-based masks select by a fixed rule: BitFit tunes only bias terms, and MagPruning tunes the largest-magnitude weights 2211.15583 §2.2.2. Adapter and LoRA add new parameters; the paper treats them as equivalent sparse models of the same shape as the pretrained network, with one matrix starting at zero 2211.15583 §2.2.2 2211.15583 fig. 1. Random masks reach 74.55 average in the paper's table, against 74.78 for full fine-tuning, so the data-blind choice does not beat full tuning there 2211.15583 tab. 1. Card: Effectiveness of Parameter-Efficient Fine-Tuning.

Projection-based masks

DiffPruning relaxes the mask to a continuous variable and projects it back to valid masks. ChildPruning trains the full model first and then projects a mask 2211.15583 §2.2.3. The paper calls the failure mode "projection discontinuity": a small change in scores can flip the projection to a very different mask 2211.15583 §2.2.3 2211.15583 fig. 2. In the paper's runs, DiffPruning changes its mask every 600 steps and the training loss returns to near its starting value after each change 2211.15583 §5.2 2211.15583 fig. 3. DiffPruning has the lowest average of the masked methods in the main table (72.65) 2211.15583 tab. 1.

Gradient-scored masks (SAM)

SAM, the paper's own method, chooses the mask once, before training. It runs a second-order expansion of the loss around the pretrained weights, drops the curvature term by bounding the Hessian with a multiple of the identity, and keeps the entries with the largest squared gradients 2211.15583 §4 2211.15583 eq. 10. The gradients come from a burn-in phase at the pretrained weights, whose length is set per task, between 600 and 2000 steps 2211.15583 §5.1. The mask is then fixed for the rest of training 2211.15583 §4. On the main table at 0.5% tunable weights, SAM averages 77.12 ± 1.51 against 74.78 ± 2.60 for full fine-tuning and 76.03 ± 2.35 for Adapter 2211.15583 tab. 1. Under 10% training-set removal, SAM keeps the lead with 76.40 average 2211.15583 tab. 2.

When to use which

All accuracy figures come from one backbone, RoBERTa-base, and seven classification tasks 2211.15583 §5.1. The paper reports no generation tasks and no other model sizes, so the ranking is not established beyond that setting.

Implementation sketch

The common core is one mask, computed before training, and a training loop that moves only the masked entries. Variants differ only in how the mask is chosen.

# Common core of sparse fine-tuning  [arxiv:2211.15583 §2.1 def. 1]
# Input:  pretrained weights theta0 (m values), budget fraction p
# Output: fine-tuned weights theta = theta0 + mask * delta

def sparse_finetune(theta0, select_mask, data, p, train):
    # Choose the fixed mask: random, rule-based, projected or gradient-scored
    # [arxiv:2211.15583 §2.2] [arxiv:2211.15583 §4]
    mask = select_mask(theta0, data, floor(size(theta0) * p))   # (m,)
    # Entries outside the mask stay at their pretrained values  [arxiv:2211.15583 §4]
    delta = zeros_like(theta0)                                  # (m,)
    # Train only masked entries; the mask does not change during training
    # [arxiv:2211.15583 §4]
    return train(theta0, delta, grad_mask=mask, data=data)

For SAM, select_mask accumulates the loss gradient at theta0 over the burn-in steps, squares it, and keeps the top-k entries 2211.15583 §4 2211.15583 eq. 10. The paper does not state the optimizer, learning rate, batch size or weight-decay handling 2211.15583 §5.1. Those choices change the results and are left to the implementer. Whether the burn-in gradient is a sum or a mean is also not stated 2211.15583 §5.1.

Contradictions

Within the paper, not between papers:

• Random versus full tuning. Section 5.2 says even the Random model beats full fine-tuning 2211.15583 §5.2. Table 1 gives 74.55 against 74.78 on the average 2211.15583 tab. 1, and the standard deviations are 2.4 to 2.8, so the gap is within noise.
• Stability bound versus the sweep. The stability bound falls smoothly as the tunable fraction shrinks 2211.15583 eq. 6. The paper's sweep shows a sharp drop below a threshold 2211.15583 §5.2 2211.15583 fig. 5. The paper's proof assumes a random mask, while SAM's mask is chosen deterministically 2211.15583 app. A.2. So the bound does not directly cover SAM.

No second paper on this concept has a card, so no cross-paper disagreement is recorded here.

Papers

• Effectiveness of Parameter-Efficient Fine-Tuning: argues that random, rule-based and projection-based parameter-efficient methods are sparse fine-tuned models, bounds their stability and generalization, and proposes SAM, a gradient-scored mask.