Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

In plain words

Fine-tuning a very large language model means updating all of its weights, which needs a lot of memory and time. Parameter-efficient fine-tuning (PEFT) is a family of tricks that train only a small add-on or a small slice of the model, so the rest stays fixed. Many such tricks have been published, but each paper tests them on its own models and tasks, so it is hard to tell which one actually works best.

This paper does two things. It sorts the published tricks into a few families and explains how they differ. Then it runs them all under the same conditions, on models of three sizes, and measures both how accurate they are and how much memory and time they use. The main message is that the simple, older tricks hold up well, and that several methods that looked better in their own papers do not beat them once the setup is fixed. The paper also points out how inconsistently parameter counts are reported, which makes the numbers hard to compare across papers.

TL;DR

A survey with one large experiment. The authors compare 14 PEFT variants and full fine-tuning on T5 at three sizes, under a small and fixed training budget, and report accuracy, GPU memory and throughput. Houlsby adapters and LoRA are the most reliable choices in their runs, and LayerNorm tuning is a surprisingly strong, cheap baseline. Use this card to check a PEFT claim against a common protocol, or to reproduce the comparison. Keep the pitfalls below in mind: several numbers in the text do not match the tables.

Problem

Full fine-tuning updates every weight, so training and storing a task model become expensive as the backbone grows. PEFT updates a small set of parameters instead §1. The authors argue that papers rarely compare PEFT methods head to head: they use different models, datasets and budgets, report different parameter counts, and seldom report memory or speed §1 §11.

Assumptions of the comparison: the backbone is an encoder-decoder T5 model used in text-to-text form; the training budget is short (three epochs, or at least 100 update steps, on the language understanding tasks) and the tuning per method is limited §11.1.

Method

The paper contributes three things. The taxonomy and method descriptions cover additive, selective, reparametrization-based and hybrid methods §3 to §10. The "Challenges and guidelines" section lists reporting problems §12. The experiment is a fixed protocol that the card below describes in full §11.1.

Protocol for the experiment §11.1:

1. Backbones: T5 at Large (0.7B), 3B and 11B parameters §11.1.
2. Methods: full fine-tuning as the baseline and 14 PEFT variants: Houlsby adapters, Pfeiffer adapters, scaled parallel adapters, (IA)3, prefix tuning, prompt tuning, LayerNorm tuning, LoRA on the query and value projections, LoRA on all linear layers, KronA, MAM adapters, Compacter, Compacter++ and UniPELT §11.1. The method sections are cited in §5 to §10.
3. Datasets: BoolQ, RTE and COPA from SuperGLUE, and CNN/DailyMail summarization §11.1. The paper counts four datasets in its setup, although the introduction says five (see Pitfalls) §1.
4. Input and output lengths: 512 input tokens everywhere; 8 output tokens for the classification tasks and 128 for summarization §11.1.
5. Batch: total batch 32, reached with gradient accumulation where the memory does not fit §11.1 tab. 3.
6. Training length: classification runs use three epochs or at least 100 update steps; CNN/DailyMail trains for one epoch of about 9,000 steps and had not plateaued §11.1.
7. Learning rate: a sweep over three values, keeping the best run §11.1. The exact values differ between text, appendix and scripts (see Hyperparameters).
8. Seeds: the best setting is repeated on two more seeds for the spread §11.1.
9. Metrics: accuracy on SuperGLUE and ROUGE-L on CNN/DailyMail. GPU memory is the maximum allocated during training. Throughput is tokens per second in training and inference. Reparametrization methods are not merged into the weights at inference §11.1.
10. Aggregation: the score is the plain mean over the four datasets, and the spread is an aggregate of the per-dataset variances §11.2 tab. 4.

# One (T5 size, method) cell on the classification tasks  [§11.1]
# Input:  model_size in {large, 3b, 11b}; method; datasets (RTE, COPA, BoolQ)
# Output: per-dataset accuracy for the best learning rate, plus spread

def run_cell(model_size, method, datasets):
    # Sweep three learning rates with seed 0  [§11.1] [app. B]
    best_lr = None
    best_score = None
    for lr in LEARNING_RATES:
        score = train_and_eval(model_size, method, datasets, lr, seed=0)
        if best_score is None or score > best_score:
            best_lr = lr
            best_score = score
    # Repeat the best setting on two more seeds to estimate the spread  [§11.1]
    # The paper does not say which split the sweep selected on.
    runs = [train_and_eval(model_size, method, datasets, best_lr, seed=s)
            for s in [1, 42]]
    return best_score, runs

Overview of the three adaptation families that are compared, as the paper describes them §5 §6 §9:

• Additive: small new modules added to the network. Adapters insert small bottleneck layers after sub-layers; soft prompts add trainable vectors to the input or to each attention layer §5 §6.
• Selective: a subset of existing weights is trained. LayerNorm tuning is one example used in the comparison §8.
• Reparametrization: the update to a weight matrix is written in a cheaper form. LoRA uses a low-rank product; KronA and Compacter use Kronecker products §9 §10.
• Hybrid: several of the above combined; MAM and UniPELT are the two compared here §10.

Hyperparameters

Results

Author-reported. Setup: T5 Large, 3B and 11B; RTE, COPA, BoolQ and CNN/DailyMail; best learning rate per cell; memory and throughput measured on A100 GPUs §11.1. The 95% intervals are two standard deviations tab. 4. Scores are the mean over four datasets, where CNN/DailyMail contributes ROUGE-L §11.2.

Downstream averages tab. 4. The paper bolds means above full fine-tuning and underlines values within its interval; that markup is not reproduced here:

Source: tab. 4 and §11.2.

Efficiency tab. 5. Trainable parameters in millions and as a percent of the backbone. RAM is peak GPU memory in GB. Throughput is a percent of full fine-tuning (100), so higher is faster. The paper's text quotes throughput in thousands of tokens per second, but the table gives only the percentages.

Source: tab. 5 and §11.3. Throughput intervals reported as ±6 (Large), ±7 (3B) and ±3 (11B).

Headline findings, as the authors state them:

• Houlsby adapters and LoRA are the only methods that consistently reach full fine-tuning with little tuning §11.2.
• Most methods that claimed to beat LoRA or adapters did not in this setup. Parallel adapters, Compacter and KronA were the exceptions, on par with the best methods in several cases, especially at 11B §11.2.
• Prompt tuning was far behind and seed-sensitive at the sizes tested §11.2.
• MAM and UniPELT were hard to train; extra runs gave only marginal gains §11.2.
• Pfeiffer adapters trail Houlsby at every size, and the gap grows with scale §11.2.
• LayerNorm tuning is competitive with full fine-tuning at Large and 11B, and the authors recommend it as a baseline §11.2 §11.3.
• Every PEFT method cut training memory. The largest cut is at 11B §11.3.
• Adding parameters slows inference unless the update is merged into the weights. UniPELT is the slowest at inference at all three sizes §11.3 tab. 5.

Paper vs code

Code: official repository at 6683c4cec35648198d7ea31848d32306045924e7, owned by the first author and linked from the paper §1 §11.1. Licence Apache-2.0 [LICENSE]. Method code is in scripts/finetuning_seq2seq.py and peft_comparison/mappings.py; sweep configs are in configs/ and bash_scripts/. Most adapter methods run through the Adapters library and the hf_ methods through PEFT finetuning_seq2seq.py.

• Learning-rate grid. Text: 1e-3, 1e-4, 1e-5 §11.1. Appendix: 1e-3, 1e-4, 5e-5 app. B. Scripts: 1e-3, 1e-4, 5e-5 01_grid_serarch_lora.sh 02_grid_serarch.sh. The sweep config uses five values, 2e-3 to 5e-5 sweep_grid.yaml. The paper does not say which values produced each table cell.
• Seeds. Text: five runs per dataset, then two more seeds after the sweep §11.1. Raw results: seeds 0, 1 and 42 for the classification tasks app. B. Code default is seed 0 finetuning_seq2seq.py.
• CNN/DailyMail grid script. bash_scripts/02_grid_serarch.sh runs Pfeiffer, Houlsby and scaled parallel adapters on CNN/DailyMail with three learning rates, weight decay 0 and 0.1, seeds 0, 1 and 42, total batch 64, one epoch and 3 beams 02_grid_serarch.sh 02_grid_serarch.sh. The paper uses learning rate 1e-3 only, one seed, total batch 32 and one epoch §11.1.
• LoRA grid script. bash_scripts/01_grid_serarch_lora.sh uses per-device and total batch 32 and three epochs 01_grid_serarch_lora.sh, which matches the paper's batch and epoch settings §11.1.
• LoRA rank. The paper does not state a rank §11.1. The code default for the query and value projections is rank 8 with scale 8 mappings.py. The all-linear target list names k, q, v, o, wi, wo mappings.py. Gated feed-forward layers (wi_0, wi_1) would not match; I did not check which T5 checkpoint was used.
• Optimizer and schedule. The paper states neither §11.1. The code uses AdamW finetuning_seq2seq.py, linear decay and a 6% warmup finetuning_seq2seq.py finetuning_seq2seq.py. Default weight decay is 0, but the sweep sets 0.1 sweep_grid.yaml.
• Decoding. The paper says accuracy and ROUGE-L, with no decoding rule §11.1. The code uses beam search with 5 beams in the sweep config sweep_grid.yaml and 3 beams in the CNN script 02_grid_serarch.sh.
• Gradient clipping. Not stated in the paper. I found none in the training script.
• Matches. Input length 512 finetuning_seq2seq.py; the classification minimum of 100 steps (default at finetuning_seq2seq.py, enforced at finetuning_seq2seq.py); total batch 32 as a default finetuning_seq2seq.py.

Pitfalls

• Dataset count. The introduction says five datasets §1. The setup lists four §11.1, and the averages match four: recomputing the full fine-tuning Large mean from the raw runs over RTE, COPA, BoolQ and CNN/DailyMail gives 67.22, the printed value app. B tab. 4.
• Mixed metrics in one average. The averages add accuracy and ROUGE-L with equal weight. CNN/DailyMail scores near 27 in ROUGE-L, so the average is not a standard metric tab. 4.
• Seed counts. The text says five runs and then two more seeds §11.1. The raw table has seeds 0, 1 and 42 for the classification tasks app. B. The CNN/DailyMail seed counts vary by cell: at 3B and 11B most methods have one seed, while at Large most have three app. B. The spreads are therefore not comparable across methods.
• Repeated CNN values. At 11B, seed 0 on CNN/DailyMail shows 27.83 for IA3, KronA, prefix tuning, MAM and UniPELT app. B. Five different methods having the same value suggests a transcription error. Full fine-tuning at 3B and 11B also shows 27.32 app. B. I could not check either against the source. Many 11B CNN cells are blank for seeds 1 and 42 app. B.
• Sweep selection. The table reports the best of the sweep §11.1. The paper does not say whether that choice used a separate validation split or the test split, so the reported scores may be optimistic.
• Pfeiffer gap. The text says the Houlsby–Pfeiffer difference is up to 15 points and grows with scale §11.2. The table gives a gap of about 25 points at 11B (76.16 vs 50.72) tab. 4.
• Inference slowdown range. The text gives a 33–55% slowdown for Large §11.3. In tab. 5 the additive methods range from about 13% (LoRA, query and value) to about 55% (UniPELT); Houlsby is about 30%.
• Training speed claims. The text says MAM is 20% slower and Compacter 5% slower than full fine-tuning at Large §11.3. The table gives MAM 106.1 (faster) and Compacter 81.3 (about 19% slower) tab. 5.
• KronA parameters. The text says KronA and Compacter each train fewer than 0.5 million parameters at 11B §11.3. The table gives KronA 0.77 million tab. 5.
• Memory saving. The text gives 71.5 GB saved by Compacter++ at 11B §11.3. The table gives 104.9 minus 28.5, i.e. 76.4 GB tab. 5. Full fine-tuning at 11B is the sum over two 80GB GPUs §11.1, so the saving depends on that accounting.
• Throughput units. The text quotes thousands of tokens per second §11.3; the table reports percent of full fine-tuning only tab. 5. Absolute speeds cannot be recovered from the paper. Throughput at 11B for UniPELT, MAM and prefix tuning was measured on 80GB cards, while the others used 40GB §11.1.
• Method count. The introduction says 30 methods §1; the text says 28 in the scale table §11; the table has 22 rows tab. 2.
• "Consistently best". The Houlsby Large interval (67.34 ±9.58) is wide enough to overlap most methods, so "consistently" rests mainly on the 3B and 11B results tab. 4.
• Prompt tuning. The text says it "never outperformed the constant prediction baseline" §11.2. No constant-prediction baseline appears in the tables, so this could not be checked.
• Learning-rate sensitivity. The claim that hybrid methods are hard to train rests on three learning rates. The extra runs the authors describe for MAM and UniPELT are not reported §11.2.
• Short budget. The authors say their setup favours fast learners, and that the CNN/DailyMail runs had not converged §11.1. Rankings may change with longer training.
• Parameter counts for adapters. Adapter sizes come from library defaults §11.1. The Houlsby count is identical at all three sizes tab. 5, which the paper does not explain.
• Cost of the study. The paper reports about 700 experiments in total §11.1. Full fine-tuning at 11B needed two 80GB GPUs with model parallelism §11.1, so reproducing the full table needs comparable hardware.

Lineage

• Builds on: T5 as the backbone and text-to-text format §11.1; the methods compared are described in §5 to §10.
• Compared against: full fine-tuning as the baseline, and the 14 PEFT variants in §11.1. The authors contrast their study with delta-tuning, which compared four PEFT methods with a narrower focus §11.1.
• Related to: modular deep learning surveys, which the authors cite as covering multi-task inference §1 §12.
• Superseded by: not known from this wiki.