LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

In plain words

Large language models are costly to adapt to one job, because changing every weight needs a lot of memory and compute. A common shortcut keeps the model frozen and trains a few small extra pieces instead. This paper gathers the main versions of that shortcut into one toolkit, LLM-Adapters, and tries them on three open models of 6 to 13 billion parameters.

The authors compare four kinds of add-on: learned prompt vectors (prefix-tuning), small bottleneck modules placed in series or beside the model's sublayers, and LoRA, which learns a low-rank change to existing weight matrices. They fine-tune on two new datasets, one of math word problems and one of commonsense questions. The math answers' step-by-step explanations were written by ChatGPT. They also test where each add-on should go and how large it should be.

The headline is that a 13-billion-parameter model with a small add-on can match or beat ChatGPT on commonsense questions, and beat GPT-3.5 on some simple math sets. The math comparison is weaker than it looks, because some of those test sets overlap the training data (see Pitfalls). All numbers are the authors' own.

TL;DR

LLM-Adapters is a toolkit plus an empirical study of parameter-efficient fine-tuning (PEFT) on LLaMA-7B and 13B, BLOOMz-7B and GPT-J-6B §3.3. The authors train four adapter families on about 10K math and 170K commonsense samples, then compare placement and size §3.2 §4.1. Author-reported: LLaMA-13B with a parallel adapter averages 81.5 over eight commonsense tasks, against 77.0 for ChatGPT tab. 4. The math comparison with GPT-3.5 uses benchmarks that the code repository says overlap the training data README.md.

Problem

• Full fine-tuning of LLMs is computationally and storage heavy §1.
• PEFT was studied mainly on BERT-scale models. It was unclear which module, placed where, with which hyperparameters, works best for LLMs on a given task §1.
• The authors claim PEFT can match full fine-tuning while training few parameters, and that it reduces catastrophic forgetting §1.
• The study covers three open base models only (BLOOM, GPT-J, LLaMA) §1.

Method

The paper is an empirical study with a software toolkit, not a new architecture. It describes four PEFT families §2 tab. 1 fig. 1: prompt-based learning, reparametrization (LoRA), series adapters and parallel adapters. Four are tested: Prefix-Tuning, Series Adapter, LoRA and Parallel Adapter §3.3.

1. Prefix-tuning (prompt-based). Learnable key and value vectors of length L are prepended inside each attention layer. The base model stays frozen §2 eq. 1. The paper sweeps 10, 20, 30 and 40 virtual tokens and uses 10 §4.1 app. A.4.
2. LoRA (reparametrization). The weight update is a product of two low-rank matrices, added to the frozen weight §2 eq. 2. In the chosen setting it is applied to both attention and MLP layers at rank 32 §4.1.
3. Series adapter. A bottleneck module (down-projection to r, non-linearity f, up-projection back to d) is inserted sequentially inside a sublayer §2 eq. 3. Best placement: after the MLP layers, bottleneck 256 §4.1.
4. Parallel adapter. The same bottleneck shape, but it reads the sublayer's input, and its output is added to the sublayer's output §2 eq. 4. Best placement: in parallel with the MLP layers, bottleneck 256 §4.1.
5. Math data, Math10K. About 10K step-by-step samples built from the training sets of GSM8K, MAWPS, MAWPS-single and 1000 AQuA examples. ChatGPT generated the rationales with zero-shot chain-of-thought prompts. Samples with wrong final answers were removed §3.2 app. A.1.
6. Commonsense data, Commonsense170K. About 170K samples from the training sets of BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c and OBQA, each formatted with a per-dataset template §3.2 app. A.2.
7. Training. One model per task family (math or commonsense), evaluated on every dataset in that family §3.3. Batch size 16, learning rate 3e-4 (prefix-tuning 3e-2), three epochs §3.3.
8. Placement and configuration study. Run on LLaMA-7B with Math10K, scored on the six math test sets §4.1 app. A.3 app. A.4.
9. Evaluation. Accuracy on six math and eight commonsense test sets, with GPT-3.5, GPT-3, PaLM and ChatGPT baselines §3.1 §4.2 §4.3.

The adapter forms below are our paraphrase of the paper's equations, with the choices the released code makes. The paper does not give them as an algorithm.

# LoRA on one frozen linear map  [eq. 2]
# [code: peft/src/peft/tuners/lora.py#L308]
# Input:  hidden: (B, T, d) layer input
# Output: (B, T, d_out) layer output

def lora_linear(hidden, frozen_weight, lora_down, lora_up, alpha, rank):
    # Shapes assumed: lora_down (d, r), lora_up (r, d_out)
    # The paper prints both matrices as r x d  [§2]
    # Frozen path  [eq. 2]
    frozen_out = hidden @ frozen_weight
    # Low-rank update scaled by alpha / rank  [code: lora.py#L308]
    scale = alpha / rank
    return frozen_out + scale * (hidden @ lora_down @ lora_up)

# Series adapter placed on the input of a frozen linear map  [eq. 3]
# Code form for LLaMA up_proj and gate_proj
# [code: peft/src/peft/tuners/bottleneck.py#L355-L366]
# Input:  hidden: (B, T, d) layer input
# Output: (B, T, d_out) layer output

def series_adapter_on_input(hidden, frozen_layer, down, up, scale):
    # Bottleneck: down-project, non-linearity, up-project  [eq. 3]
    # tanh is the code default  [code: finetune.py#L53]
    update = up(tanh(down(hidden)))
    # Residual adds the update to the input  [code: bottleneck.py#L364]
    hidden = hidden + scale * update
    # The frozen layer then reads the updated input  [code: bottleneck.py#L366]
    return frozen_layer(hidden)

# Series adapter placed on the output of a frozen linear map  [eq. 3]
# Code form for LLaMA down_proj, BLOOMz dense_4h_to_h, GPT-J fc_out
# [code: peft/src/peft/tuners/bottleneck.py#L367-L378]
# Input:  hidden: (B, T, d) layer input
# Output: (B, T, d) layer output

def series_adapter_on_output(hidden, frozen_layer, down, up, scale):
    # Frozen layer first  [code: bottleneck.py#L369]
    frozen_out = frozen_layer(hidden)
    # Bottleneck reads the frozen output  [eq. 3]
    update = up(tanh(down(frozen_out)))
    # Residual on the frozen output  [eq. 3]
    return frozen_out + scale * update

# Parallel adapter beside a frozen linear map  [eq. 4]
# [code: peft/src/peft/tuners/bottleneck.py#L379-L389]
# Input:  hidden: (B, T, d) layer input
# Output: (B, T, d_out) layer output

def parallel_adapter(hidden, frozen_layer, down, up, scale):
    # Frozen path  [eq. 4]
    frozen_out = frozen_layer(hidden)
    # Bottleneck reads the same input as the frozen layer  [eq. 4]
    update = up(tanh(down(hidden)))
    # Add the update to the frozen output  [eq. 4]
    return frozen_out + scale * update

# Fine-tuning loop of the released script  [§3.3]  [code: finetune.py#L106]
# Input:  base: frozen LLM; samples: prompt and answer pairs
# Output: trained adapter weights only

def finetune(base, adapter, samples, learning_rate):
    # Effective batch 16 = micro batch 4 x 4 accumulation steps  [§3.3]
    accumulation_steps = 16 // 4  # [code: finetune.py#L106]
    # Optimizer and warmup are not in the paper  [code: finetune.py#L280-L285]
    optimizer = adamw(trainable_params(adapter), learning_rate)
    for epoch in range(3):  # [§3.3]
        for micro_batch in batches(samples, size=4):
            # Loss also covers prompt tokens  [code: finetune.py#L62]
            loss = causal_lm_loss(base, adapter, micro_batch)
            scaled_loss = loss / accumulation_steps
            scaled_loss.backward()
            if step_done(accumulation_steps):
                optimizer.step()
                optimizer.zero_grad()
    return adapter

Key equations:

H_{o} = \text{Attn}(H_{i}W_{Q},\ [P_{K};H_{i}W_{K}],\ [P_{V};H_{i}W_{V}]) eq. 1

H_{o} = H_{i}W_{0} + H_{i}\Delta W = H_{i}W_{0} + H_{i}BA eq. 2

H_{o} \leftarrow H_{o} + f(H_{o}W_{down})\,W_{up} eq. 3

H_{o} \leftarrow H_{o} + f(H_{i}W_{down})\,W_{up} eq. 4

Figures, described only (no images are published):

• fig. 1 shows four panels: (a) Prefix-Tuning, (b) LoRA, (c) Series Adapter, (d) Parallel Adapter.
• fig. 2 plots average math accuracy (LLaMA-7B) for each adapter placement: attention, MLP or both §4.1. Its values are in app. A.3.
• fig. 3 is a 2 x 2 grid, one panel per method, plotting average math accuracy against the method's main setting: virtual tokens, bottleneck size or LoRA rank §4.1. Its values are in app. A.4.

Hyperparameters

Sources are the paper, the released script (finetune.py, defaults), and the repository's example commands. Where a value is only in the code, the row says so.

Results

Author-reported. Setup: Math10K for math and Commonsense170K for commonsense; LLaMA-7B and 13B, BLOOMz-7B and GPT-J-6B §3.3. Math baseline: GPT-3.5 (text-davinci-003) with zero-shot CoT, numbers taken from another paper §4.2 tab. 3. Commonsense baselines: GPT-3 175B and PaLM 540B (numbers from the LLaMA paper) and ChatGPT (gpt-3.5-turbo, zero-shot CoT) §4.3 tab. 4.

Headline numbers:

• Commonsense, average over 8 tasks: LLaMA-13B Parallel 81.5, LLaMA-13B LoRA 80.5, LLaMA-13B Series 79.5, ChatGPT 77.0 tab. 4.
• Math, average over 6 sets: GPT-3.5 70.4; LLaMA-13B LoRA 65.4; LLaMA-13B Parallel 64.4; LLaMA-7B LoRA 61.9 tab. 3. The paper states 65.4 as "approximately 92.8%" of 70.4; the ratio is 92.9% (see Pitfalls).
• Math sets where LLaMA-13B LoRA is above GPT-3.5 (as the paper reports): MultiArith 94.8 vs 83.8; AddSub 87.3 vs 85.3; SingleEq 89.8 vs 88.1 tab. 3 §4.2. See Paper vs code for the overlap issue.

Math, six datasets (Math10K training; author-reported) tab. 3:

Commonsense, eight datasets (Commonsense170K training; author-reported) tab. 4:

Placement, LLaMA-7B, math, Math10K (author-reported) app. A.3:

Configuration sweep, LLaMA-7B, math, Math10K (author-reported) app. A.4:

The paper's observations, in its own terms: LoRA reaches strong accuracy with far fewer trainable parameters §4.2; the commonsense gains come from training on the matching training sets, which the paper says are included in Commonsense170K §4.3; and the math gap to GPT-3.5 remains on GSM8K and SVAMP §4.2.

Paper vs code

Code: https://github.com/AGI-Edgerunners/LLM-Adapters at commit 816657208af4db747803f87ba40a4c71383fed7a. The root LICENSE is Apache-2.0 LICENSE. The peft/ subfolder is also Apache-2.0 and contains some code copied from Microsoft's MIT-licensed LoRA and adapter sources bottleneck.py lora.py. The paper names no code. The README cites this paper and lists eight of its nine authors README.md, so we treat this repository as the official one. The owner is the AGI-Edgerunners organisation; the card does not verify that it is an author lab.

• Math benchmark overlap (largest issue). The paper trains on Math10K, which includes MAWPS §3.2, then reports MultiArith, AddSub and SingleEq as evaluation sets §4.2. The repository says MAWPS consists of AddSub, MultiArith, SingleOp, SingleEq, SimulEq-S and SimulEq-L, so those three sets overlap the training data. It says the results table was updated to use the MAWPS test set instead README.md. The updated table reports LLaMA-13B LoRA at 83.6 on MAWPS against 87.4 for GPT-3.5, and an average of 51.1 against 63.2 over GSM8K, AQuA, MAWPS and SVAMP README.md. So the paper's claim that LLaMA-13B LoRA beats GPT-3.5 on those three sets is not in the updated table. The README says "the findings in the paper are consistent", but the numbers it shows are not the paper's README.md.
• Prefix-tuning batch and epochs. Paper: batch 16 and three epochs for all methods §3.3. Command: batch 8 and five epochs, learning rate 3e-2 math_running_commands.
• LoRA targets. Paper: attention and MLP layers §4.1. Command: q_proj, k_proj, v_proj, up_proj, down_proj. It omits o_proj (the attention output projection) and gate_proj math_running_commands.
• Series placement, LLaMA. Paper: series adapter after the MLP layers §4.1. The LLaMA series command targets up_proj and gate_proj, and its directory is named "attn" math_running_commands. In bottleneck.py, LLaMA's up_proj and gate_proj get the input-side form (adapter on the layer input, residual on the input) bottleneck.py bottleneck.py. The output-side form, which matches the paper's "after the MLP", is used only for down_proj. By contrast, the BLOOMz series command targets dense_4h_to_h math_running_commands and the GPT-J series command targets fc_out math_running_commands. Both are output-side. Which configuration produced the paper's LLaMA series numbers cannot be determined from the repository.
• 8-bit base. The paper does not mention quantisation. Some commands pass --load_8bit (LLaMA series, BLOOMz LoRA, BLOOMz parallel) math_running_commands math_running_commands math_running_commands. Results from 8-bit bases may differ from fp16 bases, and the paper does not say which it used.
• Parallel adapter. The paper's equation 4 matches the code's parallel branch bottleneck.py. Agrees.
• Series equation residual. Equation 3 adds the update to H_o, the sublayer output. The code's input-side form adds it to the sublayer input bottleneck.py. The paper's notation does not say which placement equation 3 covers.
• Adapter initialisation. Not stated in the paper. The default "bert" init sets both adapter matrices to normal(0, 0.02), so the up-projection is not zero at start bottleneck.py bottleneck.py. LoRA's B matrix starts at zero lora.py.
• LoRA alpha. Not stated in the paper. The code default is 16 at rank 8 finetune.py; the commands use alpha 64 at rank 32 math_running_commands, a scale of 2.0 either way.
• Prefix projection. Not stated in the paper. The code default is no projection, so the prefix vectors are learned directly prefix_tuning.py.
• Code-only defaults the paper does not mention. Warmup of 100 steps finetune.py; fp16 training finetune.py; AdamW finetune.py; loss on prompt tokens unless train_on_inputs is off finetune.py finetune.py; default batch size 128 if not passed finetune.py.
• Commonsense training. The paper states the hyperparameters apply to all fine-tuning §3.3. The repository has no commonsense training command; the README gives only the math commands README.md.
• Data. The repository ships math_10k.json (about 9.9K records by a grep count of the instruction key) and commonsense_170k.json (about 170K) math_10k.json commonsense_170k.json. It also ships math_7k, math_14k and math_50k variants that the paper does not use README.md. The data is under ODC-By DATA_LICENSE. We do not reproduce the data.

Pitfalls

• Test-set overlap in math. Some math test sets overlap the training data (see Paper vs code). Do not use the paper's MultiArith, AddSub or SingleEq results as evidence that PEFT beats GPT-3.5 on those tasks. The repository's updated MAWPS numbers show a GPT-3.5 lead.
• Selection on test data. The placement and configuration choices were made on the same six math test sets reported in the main results §4.1 app. A.3 app. A.4. The paper describes no validation split. The reported 61.9 for LLaMA-7B LoRA is therefore an optimistic estimate.
• Unmatched baselines. The math baseline is text-davinci-003 with zero-shot CoT, and its numbers come from another paper §4.2. The commonsense ChatGPT row uses gpt-3.5-turbo tab. 4. The GPT-3 and PaLM rows are copied from the LLaMA paper §4.3. The adapters are trained on the training splits of the same commonsense datasets, while the baselines are zero- or few-shot. The comparison is not like-for-like in data access.
• Single runs. No seeds, run counts or standard deviations are reported. Differences of one or two points, such as the Parallel vs LoRA gaps at LLaMA-13B, cannot be read as significant.
• Points, not percent. The paper calls the commonsense gap "4.5%" §4.3. It is 81.5 − 77.0 = 4.5 points. It also writes that 65.4 is "approximately 92.8%" of 70.4 §4.2; the ratio is 92.9%.
• LoRA equation shapes. Equation 2 is printed with B and A both as r × d §2. For H_i B A to be a d × d update, B must be d × r. The card's pseudocode assumes the standard shapes.
• Series placement ambiguity. The paper's series adapter is after the MLP layers. The LLaMA series command is input-side (see Paper vs code). Treat the paper's LLaMA series numbers as not reproducible from this repository alone.
• Hidden costs. The paper says adapters use few extra parameters, but gives no counts, memory or time. The README resource table is commented out, so do not rely on it README.md.
• Decoding is not in the paper. Evaluation uses beam search with four beams and sampling settings that the paper does not state evaluate.py. Answer extraction is done by code (extract_answer_number, extract_answer_letter) evaluate.py. Reproduce the paper's tables only with the same extraction.
• ChatGPT-generated rationales. The Math10K answers are ChatGPT output §3.2. The paper does not discuss terms of use for that output, and the card does not assess them.
• Data licence. The repository's data is under ODC-By DATA_LICENSE. The original benchmark licences (GSM8K, AQuA, MAWPS and others) are not checked here.
• Appendix and figure numbers are derived from LaTeX source order, not from the compiled PDF. The appendix labels (A.1 to A.5) follow the \appendix structure. If in doubt, cite the section.

Lineage

• Builds on: LLaMA, cited as [llama] and arXiv:2302.13971 (card) §1 §3.3.
• Builds on: Prefix-Tuning [prefix], Series adapters [adapters], Parallel adapters [parallel_adapter] and LoRA [lora], as surveyed in §2. None of these has a card here, and the source gives no arXiv ids for them.
• Builds on: the Alpaca-style prompt format used in the released code finetune.py. This is not described in the paper.
• Tested models: BLOOM [bloom] and GPT-J [gpt-j] §3.3. No cards.
• Compared against: GPT-3.5 (text-davinci-003) §4.2; gpt-3.5-turbo, GPT-3 175B, PaLM 540B §4.3.
• Superseded by: not stated in this wiki's content.