Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

In plain words

Large language models can learn a new task in two ways. One is to paste a few solved examples into the prompt every time you ask a question. That is easy, but every question then pays for reading all those examples again, and the answers are sensitive to small changes in how the examples are written. The other is to spend a little training on those examples so the model keeps what it learned. Training a whole model is expensive, so the cheaper option is to train only a small add-on.

This paper argues that the small add-on wins on both counts. It adds a tiny set of learned numbers that multiply the model's internal signals, so the model changes its behaviour without the base weights moving. Along with a training recipe that needs no tuning per task, this gives better accuracy than prompting on new tasks, and the extra cost is paid once, not on every question.

TL;DR

T-Few is a fixed recipe for few-shot classification: the T0 model, a parameter-efficient method called (IA)3 that learns a handful of scaling vectors, and two extra loss terms that help the model rank answer choices. The same settings are used for every task. The authors report that it beats few-shot in-context learning with GPT-3 175B at much lower inference cost, and that it is the first method to beat the human baseline on the RAFT benchmark. Relevant if you need a cheap, adaptable few-shot classifier and can run an 11B encoder-decoder.

Problem

Few-shot in-context learning (ICL) puts the labelled examples in the prompt, so every prediction processes all of them again. That costs compute, memory and storage, and it usually trails fine-tuning in accuracy. Prompt wording and example order also shift results more than run-to-run noise does §1 §2.1.

Parameter-efficient fine-tuning (PEFT) updates only a small set of parameters, which cuts training and storage cost, but PEFT methods had seldom been compared under very little labelled data. The paper asks whether PEFT, given a fixed recipe, beats ICL in that regime §1.

Assumptions: the target tasks are classification or multiple-choice, and the few-shot sets have 20 to 70 labelled examples per task §3.1. All experiments are on classification tasks §5.

Method

The recipe has three parts §3.5:

1. Backbone. T0, an encoder-decoder model fine-tuned on a multitask mixture of prompted datasets. The design experiments use T0-3B; the final results use T0 with 11B parameters §3.1 §4.
2. Adaptation. (IA)3 adds learned vectors that rescale activations. The base weights stay frozen, and the vectors start at ones so the initial model equals T0 §3.3.
3. Objective. The sum of a standard language-modelling loss, an unlikelihood loss on wrong answers eq. 1, and a length-normalized softmax loss over answer choices eq. 2 §3.2.

The vectors are pre-trained on the T0 multitask mixture before the few-shot fine-tuning §3.4. Prompts come from the P3 templates, with one template drawn at random for each example at each step §3.1.

Losses

The language-modelling loss is the average per-token cross-entropy of the correct answer, as usual §3.2.

The unlikelihood loss pushes down the probability of every token in every wrong answer eq. 1. The paper normalizes it by the total number of wrong-answer tokens eq. 1. For a wrong answer \hat{y}^{(n)} of length T^{(n)}:

L_{\mathrm{UL}} = -\frac{\sum_{n=1}^{N}\sum_{t=1}^{T^{(n)}} \log\left(1 - p(\hat{y}^{(n)}_t \mid \mathbf{x}, \hat{y}^{(n)}_{<t})\right)}{\sum_{n=1}^{N} T^{(n)}} eq. 1

The length-normalized loss scores each answer by its average per-token log-probability \beta(\mathbf{x}, \mathbf{y}), then applies a softmax cross-entropy over all answer choices so the correct one wins eq. 2 §3.2. It is meant to mirror length-normalized rank classification at evaluation §3.2.

L_{\mathrm{LN}} = -\log \frac{\exp(\beta(\mathbf{x}, \mathbf{y}))}{\exp(\beta(\mathbf{x}, \mathbf{y})) + \sum_{n=1}^{N} \exp(\beta(\mathbf{x}, \hat{\mathbf{y}}^{(n)}))} eq. 2

The three losses are summed with no weights, so the recipe adds no new hyperparameters to tune §3.2.

# One T-Few training step  [§3.2] [§3.5] [eq. 1] [eq. 2]
# Input:  input_text, choices (list of answer strings), gold index
# Output: scalar loss; gradients reach only the (IA)3 vectors

def tfew_loss(input_text, choices, gold):
    # Fill a randomly drawn P3 template for this example  [§3.1]
    prompt = apply_random_p3_template(input_text, choices)
    # Per-token log-probabilities of each answer given the prompt  [§3.2]
    token_logprobs = [answer_logprobs(prompt, c) for c in choices]
    # Average log-probability per answer, i.e. beta  [§3.2] [eq. 2]
    beta = [mean(lp) for lp in token_logprobs]
    # Standard LM loss on the gold answer  [§3.2]
    lm_loss = -mean(token_logprobs[gold])
    # Softmax cross-entropy over the betas, gold as the target  [eq. 2]
    ln_loss = cross_entropy(beta, gold)
    # Push down every token of every wrong answer, normalized by their
    # total token count  [eq. 1]
    ul_loss = unlikelihood(token_logprobs, gold)
    # Sum with unit weights  [§3.2]
    return lm_loss + ul_loss + ln_loss

(IA)3

(IA)3 is an activation-scaling method. It leaves the frozen weights alone and multiplies selected intermediate activations by learned vectors §3.3. The paper scales:

• the keys and values of self-attention and of encoder-decoder attention, in every layer block §3.3;
• the intermediate activation of each position-wise feed-forward network, after its nonlinearity §3.3.

The equations for attention and the feed-forward network are not numbered in the paper; they are given in §3.3. Each block has its own vectors. Per block, the paper gives the count as d_k + d_v + d_{ff} for an encoder block and 2d_k + 2d_v + d_{ff} for a decoder block §3.3.

Because each scaling sits directly after a matrix multiply, the scaling can be folded into that matrix for single-task deployment, which then adds no inference cost §3.3. The per-example vector is what makes mixed-task batches possible: each example in a batch can use a different vector at no extra model cost §3.3.

# (IA)3 on one linear layer: scale its output by a learned vector  [§3.3]
# Input:  x (B, T, d_in); frozen weight, bias; learned vector scale (d_out,)
# Output: y (B, T, d_out)

def ia3_linear(x, weight, bias, scale):
    # Frozen projection; the base weights are never updated  [§3.3]
    hidden = linear(x, weight, bias)                   # (B, T, d_out)
    # Element-wise rescale; initialised to ones so the model is unchanged
    # before training  [§3.3]
    return hidden * scale                              # broadcast over B, T

Pre-training (IA)3

The (IA)3 vectors are first trained for 100,000 steps on the same multitask mixture used to train T0, with batch size 16, before few-shot fine-tuning on each task §3.4. The paper's reported gain from this step is covered in the Results section.

Training and evaluation

Each few-shot task is trained for 1,000 steps with batch size 8, Adafactor, and a linear learning-rate decay after a 60-step warmup §3.5. The per-task settings are the same for every task §3.5.

For each task the paper draws five few-shot subsets, with different seeds, and reports the median across them and across all P3 templates §3.1. Evaluation uses rank classification: the answer with the highest (length-normalized) score is the prediction §3.1 §3.2.

Hyperparameters

Baseline hyperparameters for the PEFT comparison are listed per method in app. D. For example, LoRA uses rank 4 with 1,000 steps at learning rate 3e-3, prompt tuning uses 1,000 steps at 3e-1, and full fine-tuning uses 300 steps at 3e-4 app. D.

Results

Author-reported. Setup: nine held-out T0 datasets (COPA, H-SWAG, StoryCloze, Winogrande, WSC, WiC, RTE, CB, and ANLI, which is reported as three rounds R1-R3), few-shot training sets built by the authors, median accuracy over five subsets and all P3 templates §3.1 §4.1. GPT-3 numbers are copied from its paper, not re-run app. F.

Source: tab. 1 and §4.2. The paper reports T-Few at about 1,000 times fewer inference FLOPs than GPT-3 175B ICL tab. 1. Fine-tuning T-Few costs about as much as GPT-3 175B ICL on 20 examples §4.2.

RAFT (11 real-world tasks, 50 training examples each, no validation set, hidden test labels), top five methods tab. 2 §4.3:

T-Few uses the standard RAFT prompts with the same recipe and no per-task tuning §4.3. Per-task results are in app. H.

Per-dataset accuracy for the held-out T0 tasks, from app. F and app. G (IQR not shown; see the appendix):

Ablations on the held-out tasks, median accuracy §4.4 app. G:

Other author-reported findings:

• Pre-training the (IA)3 vectors raises fine-tuned accuracy from 64.6 to 65.8 on the design set §3.4 app. E.
• The paper states that (IA)3 is the only PEFT method that beats full fine-tuning §3.3 fig. 2. The averages below, as printed in app. D, show LoRA ahead of full fine-tuning in the no-loss and unlikelihood-only settings; see Pitfalls.

Source: app. D. The averaging rule is not stated. SAID rows are blank in the appendix, and the setting with both losses has no printed average.

fig. 1 is the method diagram: the three (IA)3 vectors and the two loss terms. fig. 2 plots PEFT accuracy against parameter count. fig. 3 plots few-shot method accuracy against inference cost.

Paper vs code

Code: official repository at 114deced63ae722a368e06cb08ea956b20c393a6, linked from the abstract footnote and under the authors' lab organisation (MIT).

• Unlikelihood epsilon. Paper eq. 1 has \log(1 - p). Code computes -\log(1 - p + 0.01) EncoderDecoder.py. The 0.01 is not in the paper.
• Unlikelihood token set. Code applies the loss to all non-padding tokens of every answer except the gold one EncoderDecoder.py. This matches eq. 1 except for the epsilon above.
• Length normalization. Code divides each answer's summed negative log-likelihood by its non-padding token count, to the power length_norm, which the dataset configs set to 1 EncoderDecoder.py copa.json. Whether the count includes the end-of-sequence token is not verified here.
• Loss weights. The dataset configs set mc_loss=1 and unlikely_loss=1, which means the sum with unit weights the paper describes copa.json. The RAFT Banking-77 run sets both unlikely_loss=0 and mc_loss=0 raft_experiments.sh. The paper says only that unlikelihood was turned off for Banking-77 app. H. The length-normalized loss is also off for that task in the code, and the paper does not say so.
• Which FFN activation is scaled. Paper: the feed-forward intermediate activation after the nonlinearity §3.3. Code: the output of the linear layers named wi_1 inside the feed-forward module ia3.json lora.py. In a gated feed-forward layer, wi_1 is the branch that is not passed through the nonlinearity. I did not verify the layout of T0's feed-forward layers, because the Hugging Face model code is not in this repository, so the two may or may not be the same tensor.
• Which vector is trained. Code creates input and output scaling vectors, but only the output-side vector (multi_lora_b) matches the trainable pattern; the input-side vector stays at ones lora.py ia3.json get_optimizer.py.
• Pre-training loss. The pre-training config sets no mc_loss, unlikely_loss or length_norm, so it trains only the language-modelling loss pretrain.json. The paper does not state the pre-training loss.
• Pre-training learning rate. Code: 3e-4 pretrain.json. Paper: not stated §3.4.
• Pre-training batch. The script sets batch_size=4 on four GPUs with distributed data parallel pretrain_11b.sh. This gives 16 only if batch_size is per process; I did not check the data loader.
• Gradient clipping. Code clips the gradient norm at 1 by default Config.py pl_train.py. The paper does not mention it.
• RAFT validation split. The code's default is a cross-validation split that holds 10 of the 50 training examples out Config.py dataset_readers.py. The RAFT training runs set this to False raft_experiments.sh. The Banking-77 evaluation sets it to True raft_experiments.sh. The paper does not describe this.
• Matches. Adafactor, 1,000 steps, learning rate 3e-3, 60-step warmup, batch 8 via gradient accumulation, and the five few-shot seeds all match the paper ia3.json few-shot-11b.sh.

Pitfalls

• Parameter count does not match storage. The appendix lists 540K parameters for (IA)3 app. D. The paper gives 4.2 MB for the same vectors stored as single-precision floats tab. 1 §4.2. At 4 bytes per parameter, 4.2 MB is about 1.05M parameters. The paper does not reconcile the two. Measure the actual checkpoint before you budget storage.
• Loss attribution in the text. Section 3.2 says that adding the length-normalized loss raises full fine-tuning from 60.7% to 62.71%. In the appendix, 62.71 is the average for the unlikelihood-only setting, and the length-normalized-only setting averages 63.25 §3.2 app. D. Use the tables, not the sentence.
• "Only (IA)3 beats full fine-tuning" does not hold in the averages. The no-loss averages put LoRA at 61.01 against 60.66 for full fine-tuning, and the unlikelihood-only averages put it at 62.96 against 62.71 app. D. Only the (IA)3 is above full fine-tuning in all three printed settings (61.72 vs 60.66, 64.06 vs 62.71, 64.05 vs 63.25) app. D. The averaging rule is not stated, so the gaps are small.
• Ablation arithmetic. Section 4.4 says that removing both pre-training and the two losses reduces accuracy by 2.5%. The ablation table gives 72.4 vs 69.7, a difference of 2.7 §4.4 app. G.
• Unlikelihood epsilon. The code's +0.01 inside the log changes the gradient for low-probability tokens and is not in the paper [code: src/models/EncoderDecoder.py#L91]. Implement it deliberately or leave it out.
• Every choice is processed. Rank evaluation and the unlikelihood loss both need every answer choice scored §4.2. The paper's cost estimate uses a median of 103 tokens for the input plus all choices §4.2. Banking-77, with 77 classes, caused memory problems for unlikelihood training, so the paper turned that loss off for it app. H.
• Memory. Training needs activation caches and Adafactor state. The paper reports training on a single 80 GB A100 §4.2.
• Baselines are not re-run. GPT-3 numbers come from its paper, on the GPT-3 authors' few-shot subsets. This paper's subsets are its own, so the comparison is between different few-shot samples §3.1 app. F.
• Prompt tuning is unstable. The authors saw validation accuracy fluctuate widely during prompt-tuning training and suspect optimization trouble §3.3.
• Cost figures are estimates. FLOPs use per-token constants from the scaling literature, not measured latency, and the paper says real cost depends on hardware §4.2.
• Scope. All experiments are classification and multiple choice. The paper says generative tasks are future work §5.

Lineage

• Builds on: T0 (Sanh et al., 2021) as the backbone and the P3 prompts §3.1; T5 as the base architecture §3.1; the unlikelihood losses cited in §3.2; the length normalization of GPT-3 §3.2.
• Compared against: few-shot in-context learning with GPT-3 at 6.7B, 13B and 175B parameters, and with T5+LM §4.1; PEFT methods BitFit, Adapter, Compacter and Compacter++, prompt tuning, FishMask, Intrinsic SAID, prefix tuning and LoRA §3.3; PET and SetFit on RAFT tab. 2.
• Related to: pre-training prompt embeddings for prompt tuning (Gu et al., 2021; Vu et al., 2021), which motivates the pre-training step §3.4.
• Superseded by: not known from this wiki.