---
kind: concept
slug: parameter-efficient-fine-tuning
title: Parameter-efficient fine-tuning
summary: >-
  Adapt a pretrained model by changing a small part of it and keeping the rest frozen, so each task stores megabytes, not a full copy. Variants train scaling vectors, residual adapters, sparse masks, tensor factors, unlearning layers, random subspaces, paired or per-layer prompts, or control adapters; Tip-Adapter sets its few-shot CLIP cache without gradient training.
tags: [fine-tuning, parameter-efficient, few-shot-learning, personalization, diffusion, adapter, video, vision-language, sparse, tensor-decomposition, tracking, robotics, unlearning, intrinsic-dimension, prompt-learning, controllable-generation, agents, image-captioning, camera-control, nlu, sequence-labeling]
papers: ["2205.05638", "2212.04488", "2111.03930", "2206.13559", "2211.15583", "2212.03145", "2103.05247", "2312.10611", "2312.16217", "2310.20150", "2304.01097", "2402.18013", "2012.13255", "2210.03117", "2302.08453", "2506.02153", "2111.01243", "2111.09734", "2512.07237", "2304.01933", "2110.07602", "2307.06435", "2205.09904"]
related: [low-rank-adaptation, in-context-learning, few-shot-learning, text-to-image-personalization, model-merging, diffusion-models, soft-prompt-tuning, transfer-learning, vision-transformer, zero-shot-learning, vision-language-pretraining, machine-unlearning, multimodal-language-model, visual-object-tracking, language-model-pretraining, self-attention, instruction-tuning, dialogue-generation, intrinsic-dimension, text-to-image-generation, llm-agent, image-captioning, prompt-engineering, camera-pose-encoding, positional-encoding, sequence-labeling, extractive-question-answering]
status: draft
updated_at: 2026-10-24
---

## Overview

Parameter-efficient fine-tuning (PEFT) adapts a pretrained model by changing a
small part of it. Most weights stay frozen. The trained part is either new
parameters or a chosen subset of existing ones, so the difference stored per
task or concept is small. The training-free cache is an edge case: it sets its
small part from the labelled examples without gradient steps
[arxiv:2111.03930 §3.2].

Twenty-three cards cover different settings. The T-Few paper adapts a text model to
classification tasks that have 20 to 70 labelled examples each
[arxiv:2205.05638 §3.1]. The Tip-Adapter paper adapts CLIP to image classes
with 1 to 16 labelled examples per class [arxiv:2111.03930 §4.1]. The
ST-Adapter paper adapts an image transformer to video action recognition
[arxiv:2206.13559 §1]. The Custom Diffusion paper adapts an image model to a
single new concept from a few photos [arxiv:2212.04488 §3.1]. Further cards
cover a sparse-mask method on GLUE-style text tasks [arxiv:2211.15583 §5.1],
a tensor-factorised adapter for vision transformers [arxiv:2212.03145 §4.1], a
frozen GPT-2 applied to non-text data [arxiv:2103.05247 §2.2], a two-stream
RGB-thermal tracker [arxiv:2312.10611 §4.1.2], a robot-manipulation model built
on LLaMA-Adapter [arxiv:2312.16217 §3.1.1], an unlearning layer for T5
[arxiv:2310.20150 §3.1], and a medical dialogue model tuned with LoRA
[arxiv:2304.01097 §3.3]. A survey of dialogue systems describes these methods
but runs no experiments [arxiv:2402.18013 §2.4.2]. Four more cover a random
subspace of the weights [arxiv:2012.13255 §3], learned prompts in both halves of
CLIP [arxiv:2210.03117 §3.2], a control adapter for a diffusion model
[arxiv:2302.08453 §3.3], and a position paper that proposes LoRA for small
models in agents, with no experiments [arxiv:2506.02153 §3.1]. Three more add a learned prefix or a parallel
adapter in a new setting: a CLIP embedding mapped to prefix vectors for a
captioner [arxiv:2111.09734 §3.1], a ray-based camera adapter on a video
diffusion transformer [arxiv:2512.07237 §3.3], and a survey of NLP methods that
describes adapters and prefix tuning but runs no experiments [arxiv:2111.01243
§2.4.4] [arxiv:2111.01243 §3.2]. A study of LLM-Adapters compares prefix
tuning, series and parallel adapters, and LoRA on LLaMA, BLOOMz and GPT-J
models of 6 to 13 billion parameters [arxiv:2304.01933 §3.3]. P-tuning v2 adds trainable prefix vectors to every layer of a frozen language model, for language understanding and sequence labeling [arxiv:2110.07602 §3.2]. A survey of large language models lists parameter-efficient fine-tuning among its efficiency methods, with no experiments of its own [arxiv:2307.06435 §3.6]. A survey of deep transfer learning for image
classification names parameter-efficient tuning as one group of parameter-based
methods, covering prompts, adapters and low-rank residuals [arxiv:2205.09904
§6.1]. Its one experiment reports Big Transfer fine-tuning settings, not a
parameter-efficient method [arxiv:2205.09904 §3.4] [arxiv:2205.09904 tab. 3].
The papers use different
backbones, tasks and metrics, so their numbers are not comparable across
variants unless a section says otherwise.

The problem behind these is storage, training cost and overfitting. Storing a
full copy of the model per task is expensive: the ST-Adapter authors make this
point for video [arxiv:2206.13559 §1], and the Custom Diffusion authors report
about 3GB per concept for the full-model baseline against 75MB for their method
[arxiv:2212.04488 §4.1]. Gradient training of few-shot adapters is slow and
memory-hungry, according to the Tip-Adapter authors [arxiv:2111.03930 §1].
Fine-tuning every weight on very little data also overfits, and in the image
case it causes language drift: after training on a rare object name, related
words lose their old meaning [arxiv:2212.04488 §3.1]. The sparse-mask paper
argues that many of these methods, including adapters and LoRA, are one family:
each trains a fixed sparse subset of the weights [arxiv:2211.15583 §2.2].
The intrinsic-dimension paper asks the question from the other side: why a
few trainable values can match full fine-tuning at all [arxiv:2012.13255 §1].

The image method rests on one measurement. When the whole model is fine-tuned
on one concept, the largest relative weight changes fall in cross-attention,
which holds about 5% of the parameters. Text reaches the image network only
through the key and value projections of those layers [arxiv:2212.04488 §3.1].

For the low-rank adapter family used in language models, see
[Low-rank adaptation](/concepts/low-rank-adaptation).

## Variants

### Learned activation scaling, (IA)3 (few-shot text classification)

Freeze the whole backbone and learn one vector per module. Each vector
multiplies the keys and values of the attention layers and the intermediate
activation of each feed-forward block, in every block. The vectors start at
ones, so the model is unchanged before training [arxiv:2205.05638 §3.3]. The
paper's trainable count is 540K parameters [arxiv:2205.05638 app. D]; its
storage figure is 4.2 MB [arxiv:2205.05638 tab. 1]. Because each vector sits
after a matrix multiply, it can be folded into that matrix for a single task,
which then adds no inference cost. Keeping the vectors separate lets one batch
mix tasks [arxiv:2205.05638 §3.3]. Card:
[Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](/papers/2205.05638).

The recipe starts from T0 11B. The vectors are first pre-trained for 100,000
steps on T0's multitask mixture, then fine-tuned per task for 1,000 steps at
batch size 8 with learning rate 3e-3 [arxiv:2205.05638 §3.4] [arxiv:2205.05638
§3.5]. The authors report 72.4% median accuracy over nine held-out tasks,
against 66.9% for zero-shot T0 and 66.6% for few-shot GPT-3 175B in-context
learning [arxiv:2205.05638 tab. 1]. Without the pre-training step, accuracy is
70.8% [arxiv:2205.05638 §4.4]. Results on the RAFT benchmark are in the card.

### Trained subset plus a new token (single concept, image generation)

Freeze the model except the key and value projections of every cross-attention
layer and the embedding of one new placeholder token. Train on the target
images mixed about 1:1 with roughly 200 real images retrieved for similar
captions. The paper reports 250 steps at batch 8 and a learning rate of
$8\times10^{-5}$ [arxiv:2212.04488 §3.2] [arxiv:2212.04488 app. E]. Card:
[Multi-Concept Customization of Text-to-Image Diffusion](/papers/2212.04488).

Reported on the single-concept benchmark: text alignment 0.795, image
alignment 0.775, KID 20.96 [arxiv:2212.04488 tab. 1].

### Joint training of several concepts

Give each concept its own placeholder token and train the same projections plus
all tokens on the union of the concept datasets, with a batch drawn per
concept [arxiv:2212.04488 §3.2]. The paper uses 500 steps for two concepts
[arxiv:2212.04488 §3.2]. Joint training has the best multi-concept scores in
the paper: text 0.801, image 0.706 [arxiv:2212.04488 tab. 1].

### Closed-form merge of separately trained concepts

Fine-tune each concept on its own, then combine the projection matrices in one
least-squares solve per layer and per key or value matrix. The merged matrix
reproduces each concept's own target words while staying close to the original
matrix on regularization captions [arxiv:2212.04488 §3.2] [arxiv:2212.04488
eq. 4] [arxiv:2212.04488 eq. 5]. The authors report about 2 s for the merge
[arxiv:2212.04488 §3.2]. Multi-concept scores are slightly below joint
training: text 0.800 and image 0.695 for the optimization variant, text 0.797
and image 0.700 for sequential training [arxiv:2212.04488 tab. 1].

### Low-rank compression of the stored update

The difference between fine-tuned and original projections has quickly
decaying singular values, so it can be stored as a truncated SVD. Keeping the
top 60% of the singular-value mass gives about five times smaller storage with
similar metrics; keeping only the top 1% lowers image alignment [arxiv:2212.04488
app. C]. Forcing a low-rank update during training gave worse results than
compressing afterwards [arxiv:2212.04488 app. C].

### Training-free key-value cache (few-shot CLIP classification)

Store the CLIP image features of the labelled examples as keys, and their
one-hot labels as values. For a new image, the affinity to each key is an
exponential of its cosine similarity, and the weighted label vote, scaled by a
factor alpha, is added to CLIP's zero-shot logits. Building the cache takes one
pass over the examples and no gradient steps [arxiv:2111.03930 §3.2] [eq. 3]
[eq. 4]. With the weights set this way, the adapter form equals the cache
lookup [eq. 7] [eq. 8]; the paper lists three differences from a learned
adapter: the weights are set rather than learned, the hidden width is NK, and
the activation is bounded in [0, 1] [arxiv:2111.03930 §3.2]. Card:
[Tip-Adapter: Training-Free CLIP-Adapter for Better Vision-Language Modeling](/papers/2111.03930).

On ImageNet with a ResNet-50 at 16 shots, the paper reports 62.03% against
60.33% for zero-shot CLIP and 63.59% for CLIP-Adapter [arxiv:2111.03930 tab. 2].
The ablation uses beta 5.5 and alpha 1.0 on that setting [arxiv:2111.03930 tab.
4]. Above 16 shots, the paper averages random groups of examples into 16
prototypes per class, and reports 63.15% at 128 shots [arxiv:2111.03930 §4.4]
[tab. 4]. The paper reports only fine-tuning time, not cache-building time
[arxiv:2111.03930 tab. 3].

### Cache with key fine-tuning (Tip-Adapter-F)

Start from the cache and unfreeze only the keys. Train for 20 epochs with
cross-entropy on the few-shot set; the values, the text classifier and both
CLIP encoders stay fixed [arxiv:2111.03930 §3.2] [§4.1]. This reaches 65.51%
at 16 shots on ImageNet [arxiv:2111.03930 tab. 2]. Training only the values
gives 60.90, and training keys and values together collapses
[arxiv:2111.03930 tab. 5]. The paper reports 5 minutes on one RTX 3090, against
50 minutes for CLIP-Adapter and 14 hours 40 minutes for CoOp
[arxiv:2111.03930 tab. 3]. Card: the same
[Tip-Adapter card](/papers/2111.03930), which covers the training-free variant
as well.

### Residual bottleneck adapter per block (video recognition)

Insert a bottleneck into each block of a frozen image transformer. It
projects the patch tokens down to width 384, applies a depth-wise 3D convolution
with kernel 3×1×1 that mixes each channel over three neighbouring frames, applies
an activation the paper does not name, and projects back up before a residual
addition [arxiv:2206.13559 §3.2] [eq. 2] [tab. 7] [tab. 8]. Only the adapters
and a linear head train. The default is one adapter per block on Kinetics-400
[arxiv:2206.13559 tab. 8]. On Kinetics-400 with a CLIP ViT-B/16 and 8 frames,
the paper reports 82.0% top-1 against 81.7% for full fine-tuning, with 7.20M
trained parameters against 121.57M [arxiv:2206.13559 tab. 1]. Training takes
23 hours against 40 hours on 8 V100 GPUs [arxiv:2206.13559 tab. 6]. Card:
[ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](/papers/2206.13559).

The kernel shape is tuned: 3×3×3 gives a similar Kinetics-400 score but a
lower Something-Something-v2 score [arxiv:2206.13559 tab. 7]. Adapters in
deeper blocks give larger gains [arxiv:2206.13559 §4.3] [tab. 5].

### Sparse mask chosen from burn-in gradients (SAM)

Keep the pretrained weights as they are and pick a fixed set of entries to
update. The budget is 0.5% of the model's parameters [arxiv:2211.15583 §5.1].
Choose the entries with the largest squared gradients. The gradients come from
a few hundred to two thousand burn-in steps at the pretrained weights
[arxiv:2211.15583 §4] [§5.1]. After that the
mask is fixed, and only the selected entries change [arxiv:2211.15583 §4]. The
paper shows that the random, rule-based and projection-based methods it
reviews can all be written in this form, and it proposes the burn-in selection
as an alternative [arxiv:2211.15583 §2.2] [§4]. Card:
[On the Effectiveness of Parameter-Efficient Fine-Tuning](/papers/2211.15583).

On seven GLUE and SuperGLUE tasks with RoBERTa-base, averaged over 10 seeds,
the paper reports 77.12 for this method against 74.78 for full tuning, 76.03 for
an adapter and 75.11 for LoRA [arxiv:2211.15583 tab. 1]. When each run drops a
random 10% of the training set, the averages are 76.40 and 74.81
[arxiv:2211.15583 tab. 2]. The mask does not shrink the computation: the
paper's average training time is 2.02 hours against 1.67 for full tuning
[arxiv:2211.15583 tab. 3]. Only the storage of the changed entries is lower,
and no storage numbers are given [arxiv:2211.15583 app. A.5].

The paper also sweeps the budget. The mean and spread of scores fall as the
budget shrinks, until a threshold below which performance drops sharply and
training error rises [arxiv:2211.15583 §5.2] [fig. 5]. A smaller budget is
therefore not safe to assume without a check.

### Shared tensor factorisation of the weight change (FacT)

Stack the attention and feed-forward matrices of all blocks into one tensor,
then train a low-rank factorisation of the change to that tensor. Two formats
are proposed: Tensor-Train (FacT-TT) and Tucker (FacT-TK)
[arxiv:2212.03145 §3.2.2] [§3.2.3]. The matrices share two factors across all
layers; each matrix has its own small core in TT, or its own row of a
layer-factor matrix in Tucker. The factor that maps back to the weight width
starts at zero, so training starts from the pretrained model. The factors are
folded into the weights before inference, so inference cost does not change
[arxiv:2212.03145 §3.2]. The paper says the matrix-batch case of its format is
LoRA applied to each matrix separately [arxiv:2212.03145 §3.2.1]. Card:
[FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](/papers/2212.03145).

On VTAB-1K, with 19 tasks and a ViT-B/16 pretrained on ImageNet-21K, the
Tucker variant at its searched ranks averages 75.6% with 0.069M backbone
parameters. NOAH averages 75.5% with 0.361M, and full tuning averages 68.9%
with 85.8M [arxiv:2212.03145 tab. 1]. A TT variant at rank 4 averages 73.5%
with 0.008M parameters [arxiv:2212.03145 tab. 1]. On a Swin-B backbone, the
TT variant at rank 16 averages 77.4% with 0.135M parameters, against 75.0% for
full tuning [arxiv:2212.03145 tab. 2]. Hierarchical models need one factor set
per stage [arxiv:2212.03145 §4.3].

### Trainable norms and input/output layers around a frozen transformer (cross-modal)

Keep the self-attention and feed-forward blocks of a language-pretrained
transformer frozen. Train a new linear input layer that maps the new data into
the model width, a new linear output layer, the layer-norm parameters and the
positional embeddings [arxiv:2103.05247 §2.2]. The paper tests this on bit
strings, pixel patches, nested arithmetic expressions and amino-acid sequences
[arxiv:2103.05247 §2.1]. Card:
[Pretrained Transformers as Universal Computation Engines](/papers/2103.05247).

The layer-norm parameters carry most of the gain. On CIFAR-10, training them
alone gives 54% and training only the output layer gives 25%
[arxiv:2103.05247 tab. 15]. Freezing them drops CIFAR-10 from 68.2% to 61.5%
[arxiv:2103.05247 tab. 16]. Language pretraining helps the frozen model: with
random initialisation, bit memory reaches 75.8% and CIFAR-10 reaches 61.7%,
against 100% and 68.2% with language pretraining [arxiv:2103.05247 tab. 2].
Training only the output layer on cached features gives 32.8% on ListOps and
24.7% on CIFAR-10 LRA, with much faster epochs [arxiv:2103.05247 tab. 10].

### Bi-directional adapter in a frozen two-stream tracker (RGB and thermal)

Start from a frozen ViT tracker (OSTrack) that runs one stream per modality,
with the two streams sharing layer weights. Add one adapter in the attention stage and one in the MLP stage of each layer.
Each adapter reads the other stream and adds a prompt to this one, so neither
modality is fixed as the main one [arxiv:2312.10611 §3.2]. Only the adapters
train; the encoder, the head and the layer norms are frozen
[arxiv:2312.10611 §3.2]. The paper reports 0.32M trained parameters
[arxiv:2312.10611 §1]. Card:
[Bi-directional Adapter for Multi-modal Tracking](/papers/2312.10611).

On LasHeR the paper reports 70.2 precision and 56.3 success, and on RGBT234
86.8 and 64.1 [arxiv:2312.10611 tab. 1]. A one-direction RGB-only adapter
reaches 69.0 and 55.4 on LasHeR [arxiv:2312.10611 tab. 2]. Adapters in one
layer only give 61.4 and 49.3 on LasHeR, and adapters in all 12 layers give the
headline numbers [arxiv:2312.10611 tab. 3]. Training takes about 8 hours on
four GPUs for LasHeR [arxiv:2312.10611 §4.1.2].

### Adapter in a multimodal LLM for robot manipulation (ManipLLM)

Build on LLaMA-Adapter: a CLIP visual encoder, a projection into the language
model, and adapters injected into the language model with the LoRA method
[arxiv:2312.16217 §3.1.1]. The model predicts a contact pixel and gripper
directions from an image and a task prompt. Its training is on simulated
articulated objects, with a separate test-time update of a visual adapter
[arxiv:2312.16217 §3.3]. On 10 unseen simulated categories, the paper reports
0.54 initial-movement success against 0.41 for the best baseline
[arxiv:2312.16217 tab. 1]. Card:
[ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](/papers/2312.16217).

### Unlearning layers on a frozen language model

Insert small unlearning layers after the feed-forward networks of a frozen T5
model, and train only those layers [arxiv:2310.20150 §3.1] [fig. 1]. The
paper tunes 0.5% of the parameters [arxiv:2310.20150 §4.4]. The objective is a
selective teacher-student KL term: it pulls the student toward the teacher on
retained data and pushes it away on forgotten data. A task loss on retained
data and a masked-language-model term on forgotten data are added
[arxiv:2310.20150 §3.1] [eq. 2] [eq. 5]. The
adapter architecture is not stated [arxiv:2310.20150 §4.4]. Card:
[Unlearn What You Want to Forget: Efficient Unlearning for LLMs](/papers/2310.20150).

On IMDB with T5-base and 0.5% of the training data forgotten, the paper reports
a forget-set accuracy of 65.7 against 92.5 for re-training, with test accuracy
93.0 against 92.8 [arxiv:2310.20150 tab. 2]. The update takes 1200 seconds
against 6685 for re-training [arxiv:2310.20150 tab. 2]. The paper reads a higher
masked-language-model loss on forgotten data as the information being harder
to extract [arxiv:2310.20150 §4.2]. Merging the layers across deletion requests
uses a closed-form step [arxiv:2310.20150 eq. 7].

### LoRA on a medical dialogue model

Freeze ChatGLM-6B and add LoRA to the query and value projections, with rank 8
and alpha 16. Train one epoch with the Lion optimizer at learning rate 2e-5, no
warmup, no weight decay and batch size 4 [arxiv:2304.01097 §3.3]. The method
itself is LoRA; see [Low-rank adaptation](/concepts/low-rank-adaptation) for
the general case. A survey of dialogue systems describes LoRA and adapters in
the same terms and reports no experiments [arxiv:2402.18013 §2.4.2]. Card:
[DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task](/papers/2304.01097).

The paper gives no quantitative benchmark. Its evidence is a few answers
marked by a doctor [arxiv:2304.01097 tab. 2] [tab. 3] [tab. 4]. The authors say
the work is not suitable for commercial or clinical use [arxiv:2304.01097 §6].
Their cost estimate is 18.75 USD for 100K question-answer pairs, based on an
assumed throughput [arxiv:2304.01097 §5].

### Series and parallel adapters, prefix tuning and LoRA on LLMs (LLM-Adapters)

Compare four families on one frozen base model: learned prefix vectors, a
series bottleneck adapter, a parallel bottleneck adapter, and LoRA. A series
adapter can sit on the input or the output of a linear map; the paper places it
after the MLP layers [arxiv:2304.01933 §2] [§4.1]. A parallel adapter runs
beside a frozen linear map and adds its output to that map's output
[arxiv:2304.01933 §2]. The shared recipe is a batch of 16 and three epochs,
with learning rate 3e-4 for the adapters and LoRA and 3e-2 for prefix tuning.
The bottleneck width is 256 and the prefix has 10 virtual tokens
[arxiv:2304.01933 §3.3] [§4.1]. LoRA rank 32 is the setting used in the main
table, chosen from a sweep up to 32 [arxiv:2304.01933 app. A.4]. Training uses
about 10K math samples and 170K commonsense samples [arxiv:2304.01933 §3.3].
Card: [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of
Large Language Models](/papers/2304.01933).

On eight commonsense tasks, LLaMA-13B with a parallel adapter averages 81.5,
against 80.5 with LoRA, 79.5 with a series adapter and 77.0 for ChatGPT
[arxiv:2304.01933 tab. 4]. On LLaMA-13B, prefix tuning is the weakest of the
four, at 68.4 [arxiv:2304.01933 tab. 4]. On six math sets, LLaMA-13B LoRA
averages 65.4 against 70.4 for GPT-3.5 [arxiv:2304.01933 tab. 3]. Three math
sets are reported as wins over GPT-3.5, but they overlap the training data (see
Contradictions). The paper's placement study on LLaMA-7B math puts the parallel
adapter on the MLP layers at 61.7, against 59.3 on the attention layers and 59.8
on both [arxiv:2304.01933 app. A.3]. Those sweeps were run on the same test sets
as the main table, so the chosen settings are optimistic [arxiv:2304.01933 app.
A.4]. Each setting is a single run with no spread reported [arxiv:2304.01933
tab. 4].

### Random subspace of the weights (intrinsic dimension)

Freeze all pretrained weights and train only d numbers. A fixed random
projection, the Fastfood transform, maps those numbers into the full weight
space, and training starts from zero, so the model starts at its pretrained
weights [arxiv:2012.13255 §3] [§4.1] [eq. 2]. The paper's measure is the
smallest d that reaches 90% of full fine-tuning, called d90. Its SAID variant
adds one learned scale per layer [arxiv:2012.13255 §3] [eq. 3]. For
RoBERTa-Large on MRPC, d90 is 207 with SAID and 322 with DID; on QQP it is 774
for both [arxiv:2012.13255 tab. 1]. Card:
[Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](/papers/2012.13255).

The authors use d90 as a measure of task complexity, not as a recipe for
deployment [arxiv:2012.13255 §1] [§5]. It is an upper bound [§4.2], the
learning rates are not stated [§4.1] [§5.1], and a dense projection would need
an estimated 1.42 TB for RoBERTa-Large at d = 1000, which is why the Fastfood
form is used [§3]. A stored model is the vector plus the seed that rebuilds the
projection [§5]. See also [intrinsic dimension](/concepts/intrinsic-dimension).

### Prompt tokens in both encoders of CLIP (MaPLe)

Add learned prompt tokens to the text branch and to the image branch of a
frozen CLIP ViT-B/16. Each vision prompt is a learned linear function of the
text prompt in the same layer. Prompts are inserted in the first nine blocks,
and only the prompt parts train [arxiv:2210.03117 §3.2] [§3.2.3] [§4.1]. The
paper reports about 3.55M trainable parameters, 2.85% of CLIP
[arxiv:2210.03117 tab. 6]. Training uses 16 shots per class [§4.1].

Averaged over 11 datasets, base-to-novel harmonic mean is 78.55 against 75.83
for Co-CoOp, and novel-class accuracy is 75.14 against 71.69
[arxiv:2210.03117 tab. 3]. A variant with one coupling layer shared across
layers has 0.41M parameters and a harmonic mean of 78.11 [tab. 6]. Card:
[MaPLe: Multi-modal Prompt Learning](/papers/2210.03117). Training runs 5
epochs against 10 for Co-CoOp [§4.6]. At batch size 1 MaPLe is slower than
Co-CoOp (60.2 against 64.6 frames per second), and at batch size 4 it is faster
(239.0 against 114.7) [arxiv:2210.03117 tab. 6].

### Deep prefix vectors in every layer (P-tuning v2)

Keep a pretrained transformer frozen and give each layer its own trainable
prefix, read as extra keys and values. A linear head on the first token makes
the prediction [arxiv:2110.07602 §3.2] [§3.3] [fig. 2]. Input-only prompt
tuning, the baseline the paper compares against, adds vectors at the input
alone [arxiv:2110.07602 §2]. The paper reports that input-only prompts fall
well behind fine-tuning on models below about 10B parameters and on hard
sequence labeling [arxiv:2110.07602 §1] [§3.1]. An optional two-layer MLP can
reparameterise the prefix; the paper says it helps on some datasets and not
on others [arxiv:2110.07602 §3.3]. Card:
[P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](/papers/2110.07602).

Prompt length depends on the task. The paper suggests fewer than 20 prefix
positions for simple classification and around 100 for hard sequence labeling
[arxiv:2110.07602 §3.3] [fig. 4]. With the same number of prompted layers,
prompts in the deeper layers beat prompts in the shallow layers on RTE and
BoolQ with BERT-large [arxiv:2110.07602 §4.3] [fig. 3]. Each prefix position
adds attention work in every layer, and the paper does not discuss this cost.

On SuperGLUE dev, the paper reports RTE at 89.5 for P-tuning v2 against 86.6
for full fine-tuning with RoBERTa-large, and 88.8 against 88.7 on BoolQ with a
10B GLM [arxiv:2110.07602 tab. 2]. On RoBERTa-large, BoolQ is lower, at 84.8
against 86.9 [arxiv:2110.07602 tab. 2]. On CoNLL05 WSJ with RoBERTa-large, it
reaches 89.2 F1 against 76.8 for input-only prompt tuning and 90.2 for full
fine-tuning [arxiv:2110.07602 tab. 3]. These are dev-set numbers. The paper
does not say how the reported dev number was chosen, and the setting is fully
supervised [arxiv:2110.07602 §4].

### Control-map adapters beside a frozen diffusion model (T2I-Adapter)

Unlike the variants above, the base model's weights are not changed or
subset-tuned. A separate small convolutional adapter is trained for each
control map, such as sketch, segmentation, keypose, depth or colour, while
Stable Diffusion stays frozen. The adapter's features are added to the UNet
encoder features at four scales [arxiv:2302.08453 §3.3] [eq. 3] [eq. 4]. Several
adapters are combined by a weighted sum with no retraining [eq. 5]; the paper
sets the weights by hand [§3.3] [§5]. The base adapter is about 77M parameters,
and the 18M small adapter is used for colour [§1] [§4.4.2]. Training uses
learning rate 1e-5, batch size 8 and 10 epochs [§4.1]. On COCO validation,
text plus segmentation gives FID 16.78 and text plus sketch gives 17.36, against
24.68 for Stable Diffusion with text only [arxiv:2302.08453 tab. 1]. The adapter
runs once per sample at inference [§1]. Card:
[T2I-Adapter](/papers/2302.08453).

### Mapped CLIP prefix for a language model (image captioning)

Map a frozen CLIP image embedding to k vectors, each the size of a word
embedding, and place them in front of the caption tokens of GPT-2. The loss is
cross-entropy over the caption tokens only, conditioned on the prefix
[arxiv:2111.09734 §3.1] [eq. 5]. The paper gives two mappers. An MLP with one
hidden layer is used when GPT-2 is also fine-tuned [arxiv:2111.09734 §3.3]. A
transformer mapper, with 8 layers and learned constant inputs, is used when
GPT-2 is frozen, so only the mapper trains [arxiv:2111.09734 §3.2] [§3.3]
[§4]. Fine-tuning CLIP was tried and did not help [arxiv:2111.09734 §3.2]. The
paper's implementation details give a prefix length of 10 [arxiv:2111.09734
§4]. Card: [ClipCap: CLIP Prefix for Image Captioning](/papers/2111.09734).

On Conceptual Captions the MLP variant reaches CIDEr 87.26 against 77.57 for
VLP, and the transformer variant reaches 71.82 [arxiv:2111.09734 tab. 1 (A)].
On nocaps, the transformer variant has an overall CIDEr of 65.83 against 63.8
for Oscar, which uses object tags [arxiv:2111.09734 tab. 1 (B)]. On COCO,
Oscar stays best at 124.12 against 113.08 for the transformer variant
[arxiv:2111.09734 tab. 1 (C)]. The paper's own runs take 6 to 80 GPU hours on
one GTX 1080, depending on the dataset [arxiv:2111.09734 tab. 1].

### Parallel zero-initialised camera adapter (video diffusion transformer)

Keep a video diffusion transformer frozen and add a spatial attention adapter
beside each self-attention block. Each token carries a ray-based frame, in
which lens distortion enters the attention, and an optional latitude-and-up
map sets pitch and roll [arxiv:2512.07237 §3.2]. The adapter's output
projection starts at zero, so the pretrained model is unchanged at the first
step [arxiv:2512.07237 §3.3]. The paper's trainable size is 35.6M parameters
on Wan2.1 T2V 1.3B, with the base weights frozen [arxiv:2512.07237 tab. 1]
[app. D]. For the camera encodings it is compared with, see
[camera pose encoding](/concepts/camera-pose-encoding). Card:
[Unified Camera Positional Encoding for Controlled Video Generation](/papers/2512.07237).

On the paper's synthetic set with absolute orientation, the rotation error is
4.12 degrees, against 9.23 for ReCamMaster with 354M trainable parameters and
17.92 for Wan CameraCtrl with 1.5B [arxiv:2512.07237 tab. 1]. On a zero-shot
RealEstate10K test with 100 clips, the paper reports 0.56 degrees against 0.62
for AC3D, which was trained on that dataset [arxiv:2512.07237 tab. 2]. One clip
takes 184 s on a single A800 [arxiv:2512.07237 app. D].

### Full fine-tuning (comparison, not parameter-efficient)

In the text paper, full fine-tuning of the 3B model averages 60.66 across the
held-out tasks without the extra losses, against 61.72 for (IA)3
[arxiv:2205.05638 app. D]. The image paper's full-weight baseline reaches the
best KID of its single-concept runs, 19.27, but has lower image alignment,
0.748, than the subset method [arxiv:2212.04488 tab. 1]. It takes about 20
minutes on 4 A100s against about 6 minutes on 2 A100s for the subset method
[arxiv:2212.04488 §4.1]. The sparse-mask paper's full-tuning baseline reaches
74.78 on GLUE-style tasks, against 77.12 for its method
[arxiv:2211.15583 tab. 1].

A survey describes a staged alternative as common practice: keep the pretrained
layers frozen while a new head trains, then unfreeze them gradually with a
learning rate between 1e-3 and 1e-5. The order of unfreezing and the number of
stages are not stated [arxiv:2111.01243 §2.4.2]. Card:
[Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](/papers/2111.01243).

### Comparison of PEFT methods on the text tasks

Averages as printed in the T-Few appendix, for the setting with neither extra
loss. The averaging rule is not stated [arxiv:2205.05638 app. D].

| Method | Trainable parameters | Avg. accuracy |
|---|---|---|
| Full fine-tuning | 3B | 60.66 |
| Adapter | 12.9M | 59.27 |
| LoRA | 9.1M | 61.01 |
| Prefix tuning | 576K | 54.14 |
| Prompt tuning (100 prompts) | 409K | 50.23 |
| (IA)3 | 540K | 61.72 |

### Comparison of PEFT methods on video

Kinetics-400, CLIP ViT-B/16, 8 frames, with the backbone frozen unless noted
[arxiv:2206.13559 tab. 1].

| Method | Trainable parameters | Top-1 |
|---|---|---|
| Full fine-tuning (SA+TA, as labelled in the table) | 121.57M | 81.7 |
| ST-Adapter | 7.20M | 82.0 |
| Adapter | 6.77M | 81.6 |
| Prompt tuning | 1.18M | 79.3 |
| Linear probe | 0.31M | 76.6 |

## When to use which

| Situation | Use | Why |
|---|---|---|
| Classification or multiple-choice task with 20 to 70 labelled examples, and an 11B encoder-decoder can be served | Learned activation scaling, (IA)3, on T0 | 72.4% against 66.6% for GPT-3 175B in-context learning [arxiv:2205.05638 tab. 1]; the paper reports about 1,000 times fewer inference FLOPs [arxiv:2205.05638 tab. 1] |
| Each query must not re-read the examples | Any PEFT method over in-context learning | Inference cost is paid once, not per question [arxiv:2205.05638 §4.2]; fine-tuning T-Few costs about as much as GPT-3 175B in-context learning on 20 examples [arxiv:2205.05638 §4.2] |
| No labels, or no training run possible | Zero-shot or in-context prompting | Zero-shot T0 reaches 66.9% with no training [arxiv:2205.05638 tab. 1], 5.5 points below T-Few |
| Generative output rather than classification | Not covered by the text card | The paper's experiments are all classification and multiple choice [arxiv:2205.05638 §5] |
| Few-shot CLIP image classification, no training budget | Training-free key-value cache | 62.03% against 60.33% for zero-shot CLIP at 16 shots, but below CLIP-Adapter's 63.59% [arxiv:2111.03930 tab. 2]; no cache-building time is reported [arxiv:2111.03930 tab. 3] |
| Few-shot CLIP image classification, about 20 epochs on the few-shot set are affordable | Cache with key fine-tuning (Tip-Adapter-F) | 65.51% at 16 shots, 5 min on one RTX 3090 [arxiv:2111.03930 tab. 2] [tab. 3]; the test-split choices in the released code affect the numbers [arxiv:2111.03930 §4.1] |
| Video recognition from an image-pretrained transformer, with per-task storage or training time as the constraint | Residual adapter per block (ST-Adapter) | Kinetics-400 82.0% against 81.7% for full fine-tuning, with 7.20M trained parameters [arxiv:2206.13559 tab. 1]; 23 hours against 40 hours of training [arxiv:2206.13559 tab. 6] |
| Video, where a cheaper tuning method might be enough | Measure the temporal part against a generic adapter, prompt tuning and a linear probe on the same backbone | On Kinetics-400, ST-Adapter is 0.4 points above a generic adapter with 6.77M parameters and 2.7 points above prompt tuning [arxiv:2206.13559 tab. 1] |
| Video with an ImageNet-21K backbone | Check against full fine-tuning before choosing | 76.6% against 78.0% on Kinetics-400, but 62.8% against 59.5% on Something-Something-v2 [arxiv:2206.13559 tab. 1] |
| One new visual concept, a few dozen images, a few GPU minutes | Trained subset plus new token | About 6 min on 2 A100s, 75MB per concept [arxiv:2212.04488 §4.1]; text 0.795, image 0.775, KID 20.96 [arxiv:2212.04488 tab. 1] |
| Concept must keep the meaning of its category word | Keep regularization images, use real ones | Without them (with half the iterations) KID rises to 32.64; generated images give 34.70 instead of 20.96 [arxiv:2212.04488 tab. 3] |
| Several concepts in one image, all data available up front | Joint training | Best multi-concept scores [arxiv:2212.04488 tab. 1]; all datasets must be present for the run [arxiv:2212.04488 §3.2] |
| Concepts already trained separately, composition needed in seconds | Closed-form merge | About 2 s for the merge [arxiv:2212.04488 §3.2]; similar categories still fail to compose [arxiv:2212.04488 §4.4] |
| Storage per concept is the binding limit | Truncated SVD of the stored update | About five times smaller at the top-60% setting [arxiv:2212.04488 app. C] |
| Best KID matters more than training cost or image alignment | Full fine-tuning | KID 19.27 against 20.96 [arxiv:2212.04488 tab. 1]; about 20 min on 4 A100s [arxiv:2212.04488 §4.1] |
| Only a new word is needed, lowest training cost | Token embedding only (Textual Inversion, as a baseline) | Text 0.670, image 0.827, KID 22.27, a weaker text fit [arxiv:2212.04488 tab. 1] |
| Text task, a fixed sparse subset is acceptable, and the goal is accuracy rather than training time | Sparse mask from burn-in gradients (SAM) | 77.12 against 74.78 for full tuning at 0.5% of parameters [arxiv:2211.15583 tab. 1]; no compute saving, so training time is not reduced [arxiv:2211.15583 tab. 3] |
| Many repeated blocks of one shape, and storage per task is the constraint | Shared tensor factorisation (FacT-TK) | 0.069M backbone parameters for 75.6% on VTAB-1K, against 75.5% for NOAH at 0.361M [arxiv:2212.03145 tab. 1]; the paper's evidence is single runs without variance [arxiv:2212.03145 tab. 1] |
| Hierarchical vision backbone, such as Swin | Tensor factorisation per stage | 77.4% at 0.135M against 75.0% for full tuning [arxiv:2212.03145 tab. 2]; the parameter count grows with the number of stages [arxiv:2212.03145 §4.3] |
| New non-text data, a text-pretrained transformer, and an option to skip training attention and feed-forward | Train the norms and input/output layers only | On CIFAR-10 the norms alone give 54%, against 68.2% for the full trainable set at 12 layers [arxiv:2103.05247 tab. 15] [tab. 8]; the paper reports the gap to fully trained models for some tasks [arxiv:2103.05247 tab. 1] |
| Two input streams that must help each other, such as RGB and thermal video | Bi-directional adapter in a frozen tracker | 70.2 precision and 56.3 success on LasHeR with 0.32M trained parameters [arxiv:2312.10611 tab. 1] [§1]; the authors report results on two benchmarks only [arxiv:2312.10611 tab. 1] |
| Robot manipulation from a multimodal LLM, where simulated test categories are an acceptable proxy | Adapter on LLaMA-Adapter, as in ManipLLM | 0.54 against 0.41 on 10 unseen simulated categories [arxiv:2312.16217 tab. 1]; the paper's main comparison is in simulation; the trained set in the public code differs from the paper (see Contradictions) |
| Remove a forgotten subset after training, with a time budget | Unlearning layers on a frozen T5 | Forget accuracy 65.7 against 92.5 for re-training, 1200 s against 6685 s on IMDB [arxiv:2310.20150 tab. 2]; the paper's results are for T5 only [arxiv:2310.20150 §4.4] |
| Chat model for a narrow domain on one data-centre GPU, with no benchmark needed | LoRA on the base model (DoctorGLM setting) | Trained on one A100 for about 8 hours [arxiv:2304.01097 §3.3]; evidence is a few doctor-marked answers [arxiv:2304.01097 tab. 2], so it is not a quality claim |
| Commonsense or math task on a 7B to 13B base, choosing between adapter types | Parallel adapter on the MLP layers, with LoRA as the close alternative | LLaMA-13B commonsense average 81.5 for parallel against 80.5 for LoRA [arxiv:2304.01933 tab. 4]; the gap is within what single runs can show, and the paper gives no cost figures |
| Question is how many trainable values a task needs, not a deployable fine-tune | Random subspace of the weights (intrinsic dimension) | 207 values reach 90% of full fine-tuning for RoBERTa-Large on MRPC with SAID [arxiv:2012.13255 tab. 1]; d90 is an upper bound [§4.2], so it is a measure rather than a guaranteed budget |
| Few-shot CLIP with base-to-novel classes, where text-only prompt learning falls short, and batch inference is the serving mode | Prompts in both encoders (MaPLe) | Harmonic mean 78.55 against 75.83 for Co-CoOp [arxiv:2210.03117 tab. 3]; 3.55M trainable parameters [tab. 6]; at batch size 1 it is slower than Co-CoOp [tab. 6] |
| Fully supervised language understanding or tagging with a frozen backbone, where a 10B-scale model is available or input-only prompts have failed | Deep prefix vectors in every layer (P-tuning v2) | 89.2 against 76.8 for input-only prompt tuning on CoNLL05 WSJ with RoBERTa-large [arxiv:2110.07602 tab. 3]; 0.1% to 3% trainable parameters [arxiv:2110.07602 §1]; the paper's scope is fully supervised, so few-shot use is not covered [arxiv:2110.07602 §4] |
| Several control signals for one frozen text-to-image model, composed without retraining each time | Control-map adapters (T2I-Adapter) | FID 16.78 against 24.68 for text-only Stable Diffusion [arxiv:2302.08453 tab. 1]; composition weights are set by hand [§3.3], so a check is needed per combination |
| Narrow, repetitive agent calls where a small model is wanted, and the evidence needed is a plan rather than a measured result | LoRA or QLoRA on a small language model, as the position paper proposes | The paper runs no experiments [arxiv:2506.02153 §3.1, app. B] and gives no LoRA settings [§6]; it suggests 10k–100k examples per task as a rule of thumb [§6, S2] |
| Image captioning from a frozen CLIP encoder and a language model, with no object detector and a small trained part | Transformer mapper with GPT-2 frozen (ClipCap) | Overall nocaps CIDEr 65.83 against 63.8 for Oscar, with 43M parameters in the table [arxiv:2111.09734 tab. 1 (B)]; Oscar stays ahead on COCO, 124.12 against 113.08 [tab. 1 (C)] |
| Captioning on Conceptual Captions, where a larger trainable part is affordable | MLP mapper with GPT-2 fine-tuned (ClipCap) | CIDEr 87.26 against 77.57 for VLP, with 156M parameters [arxiv:2111.09734 tab. 1 (A)] |
| Camera control on a frozen video diffusion transformer, where lens shape or absolute pitch and roll matter | Parallel ray-encoded adapter (UCPE) | Rotation error 4.12 degrees against 9.23 for ReCamMaster, with 35.6M against 354M trainable parameters [arxiv:2512.07237 tab. 1]; the main evaluation is on a synthetic set [§4.1] |

## Implementation sketch

Eight shapes recur across the cards: scale a frozen layer's output by a learned
vector, add a residual bottleneck beside a frozen block, look up a cache of
labelled features without training, train a chosen subset of existing weights,
share a low-rank factor across all layers, train only the norms and the input
and output layers, map a frozen encoder's output to prefix vectors for a
frozen language model, and prepend trainable key and value prefixes to every
layer of a frozen backbone. The sketches below cover the first five.

The (IA)3 layer, in outline. The vector starts at ones, and the frozen weight
is never updated [arxiv:2205.05638 §3.3].

```pseudocode
# (IA)3 on one linear layer: scale its output by a learned vector  [§3.3]
# Input:  x (B, T, d_in); frozen weight and bias; learned scale (d_out,)
# Output: y (B, T, d_out)

def ia3_linear(x, weight, bias, scale):
    # Frozen projection; its weights are never updated  [§3.3]
    hidden = linear(x, weight, bias)                   # (B, T, d_out)
    # Element-wise rescale; ones at init, so the model is unchanged  [§3.3]
    return hidden * scale                              # broadcast over B, T
```

The residual adapter on one block's patch tokens. The paper writes an
activation but does not name it, so it is a parameter here. The class token
is left out of the sketch.

```pseudocode
# Residual adapter with a temporal depth-wise conv  [eq. 2] [§3.2]
# Input:  patch tokens x (B, T, N, d); N patches per frame
# Output: adapted tokens, same shape

def st_adapter(x, w_down, w_up, activation):
    # Down-project to the bottleneck width, default 384  [§3.2] [tab. 8]
    h = x @ w_down                             # (B, T, N, r)
    # Depth-wise conv over frames only; each channel sees 3 frames  [tab. 7]
    h = depthwise_conv_time(h, kernel=3)       # (B, T, N, r)
    # Activation f; the paper leaves it unnamed  [eq. 2]
    h = activation(h)
    # Up-project and add back residually  [eq. 2]
    return x + h @ w_up                        # (B, T, N, d)
```

The Tip-Adapter cache, training-free. The paper's equation for the
zero-shot term has no scale; see the Contradictions section before matching
reported numbers.

```pseudocode
# Training-free key-value cache for few-shot CLIP  [§3.2] [eq. 3-6]
# Input:  K labelled images per class, CLIP encoders, text weights (C, N)
# Output: class logits (N,) for one test image

def build_cache(train_images, train_labels):
    # Keys: normalised CLIP image features of the few-shot set  [eq. 3]
    keys = stack([normalise(clip_image(x)) for x in train_images])
    # Values: one-hot labels of the same examples  [eq. 4]
    values = one_hot(train_labels, num_classes)
    return keys, values

def cache_logits(image, keys, values, text_weights, alpha, beta):
    # Normalised test feature is the query  [§3.2]
    query = normalise(clip_image(image))
    # Affinity decreases with cosine distance to each key  [eq. 5]
    affinity = exp(-beta * (1 - keys @ query))
    # Zero-shot logits plus the alpha-scaled label vote  [eq. 6]
    return query @ text_weights + alpha * (affinity @ values)
```

The sparse mask from burn-in gradients. The mask is built once, before the
trained run, and it is fixed afterwards [arxiv:2211.15583 §4].

```pseudocode
# Sparse mask from burn-in gradients  [§4] [eq. 10] [§5.1]
# Input:  pretrained weights theta0 (m values), budget p, burn-in batches
# Output: binary mask with floor(m * p) ones

def sam_mask(theta0, batches, p, burn_in):
    # Sum the loss gradient at theta0 over the burn-in batches  [§5.1]
    grad_sum = zeros_like(theta0)                 # (m,)
    for batch in batches[:burn_in]:
        grad_sum = grad_sum + gradient(theta0, batch)
    # Rank entries by squared gradient; the curvature term is dropped  [eq. 10]
    scores = grad_sum ** 2                        # (m,)
    mask = zeros_like(theta0)                     # (m,)
    mask[top_k_indices(scores, floor(size(theta0) * p))] = 1
    # Training updates only the entries where mask is 1  [§4]
    return mask
```

The shared-factor increment for one stacked matrix, in the Tensor-Train form.
The second factor is zero at the start, so the increment is zero at the start.

```pseudocode
# FacT-TT increment for one stacked weight matrix  [§3.2.2] [eq. 8] [eq. 9]
# Input:  tokens (B, N, d); shared U, V (d, r); per-matrix core (r, r)
# Output: increment added to the frozen matrix output (B, N, d)

def fact_tt_increment(tokens, U, V, core, scale):
    # Project into the shared rank-r space  [eq. 9]
    hidden = tokens @ U                           # (B, N, r)
    # This matrix's own r x r core  [eq. 8]
    hidden = hidden @ core                        # (B, N, r)
    # Back to width d; V is zero at init  [§3.2.2]
    return scale * (hidden @ V.T)                 # (B, N, d)
```

## Contradictions

- **Within the T-Few paper, the claim about full fine-tuning.** The text says
  (IA)3 is the only PEFT method that beats full fine-tuning [arxiv:2205.05638
  §3.3]. The appendix averages show LoRA slightly ahead of full fine-tuning in
  two of three settings: 61.01 against 60.66 with no extra losses, and 62.96
  against 62.71 with the unlikelihood loss only [arxiv:2205.05638 app. D]. The
  gaps are small, and the averaging rule is not stated.
- **Within the T-Few paper, the loss attribution.** The method section gives
  62.71% as the result of adding the length-normalized loss to full
  fine-tuning. In the appendix, 62.71 is the unlikelihood-only average, and the
  length-normalized-only average is 63.25 [arxiv:2205.05638 §3.2]
  [arxiv:2205.05638 app. D].
- **Within the T-Few paper, parameter count against storage.** The appendix
  lists 540K trainable parameters for (IA)3, but the paper gives 4.2 MB, which
  is about 1.05M single-precision parameters. The paper does not reconcile the
  two [arxiv:2205.05638 app. D] [arxiv:2205.05638 tab. 1]. Measure the
  checkpoint before budgeting storage.
- **Within the Custom Diffusion paper, paper against released code.** The
  stated settings and the released code differ on training steps, the
  retrieval filter and the size range of augmentation. The differences are
  listed on [the card](/papers/2212.04488) and should be checked before a
  re-implementation.
- **Within the Tip-Adapter paper, "comparable" against its own table.** The
  authors describe the training-free variant as comparable to CLIP-Adapter and
  say it falls behind on some datasets as shots increase [arxiv:2111.03930
  §4.3]. On ImageNet its table has it below CLIP-Adapter at every shot count
  [arxiv:2111.03930 tab. 2].
- **Within the Tip-Adapter paper, logit scale.** Eq. 6 has no scale on the
  zero-shot term [arxiv:2111.03930 eq. 6]. The card reports that the alpha
  values in the paper's tables match a scale of 100 used in the released code
  [arxiv:2111.03930 tab. 4]. Reproduce the numbers with the scale, or the
  numbers will not match. See the [card](/papers/2111.03930).
- **Within the Tip-Adapter paper, model selection.** The paper does not state
  how the fine-tuning epoch, alpha or beta were chosen [arxiv:2111.03930
  §4.1]. The released code chooses some of these on the test split, so test
  numbers from that code are optimistic. The card lists the details.
- **Within the ST-Adapter paper, activation.** Eq. 2 includes an activation f,
  but the paper does not name it [arxiv:2206.13559 eq. 2]. The released code
  applies none, which makes the adapter linear apart from its residual form
  (see the [card](/papers/2206.13559)).
- **Within the ST-Adapter paper, placement.** The text puts one adapter before
  multi-head self-attention [arxiv:2206.13559 §3.3]. The released Kinetics-400
  model inserts it before the MLP instead. The card gives the code references.
- **Within the ST-Adapter paper, parameter ratio.** The abstract's figure of
  about 20 times fewer updated parameters does not match Table 1, which gives
  121.57M against 7.20M, about 17 times [arxiv:2206.13559 tab. 1].
- **Within the sparse-mask paper, random against full tuning.** The text says
  even the random mask outperforms full tuning [arxiv:2211.15583 §5.2]. In its
  own Table 1 the averages are 74.55 for random and 74.78 for full tuning
  [arxiv:2211.15583 tab. 1]. The gap is well within the standard deviations.
- **Within the sparse-mask paper, the budget.** The paper states one global
  count of masked entries [arxiv:2211.15583 §2.1]. The released code applies
  the count to each parameter tensor and masks only the encoder parameters.
  The card records the code details, which are not in the paper. The paper's
  stability proof also assumes a random mask, while SAM chooses one
  deterministically [arxiv:2211.15583 app. A.2].
- **Within the sparse-mask paper, the reported training cost.** The paper
  presents its approach as a parameter-efficient one, but its own timing table
  puts it above full tuning [arxiv:2211.15583 tab. 3].
- **Across PEFT papers, compute saving.** The sparse-mask paper says masking
  does not reduce compute [arxiv:2211.15583 app. A.5]. The ST-Adapter paper
  reports 23 hours against 40 for full tuning [arxiv:2206.13559 tab. 6]. The
  Custom Diffusion paper reports about 6 minutes against about 20 minutes
  [arxiv:2212.04488 §4.1]. These are different backbones, hardware and tasks,
  so a PEFT method is not reliably cheaper to train. Measure it for the setup
  in use.
- **Across PEFT papers, beating full tuning.** The sparse-mask paper reports
  77.12 against 74.78 for full tuning on GLUE-style tasks [arxiv:2211.15583
  tab. 1]. The FacT paper reports 75.6 against 68.9 on VTAB-1K, with full
  tuning measured in its own setup [arxiv:2212.03145 tab. 1]. The Custom
  Diffusion paper finds full tuning has the best KID [arxiv:2212.04488 tab. 1].
  Each full-tuning baseline is set up differently, so these results do not
  settle which approach wins.
- **Within the FacT paper, the reporting protocol.** The paper says the final
  results come from training on all samples and testing, and it does not
  describe any selection rule [arxiv:2212.03145 app. A]. The card records that
  the released code keeps the best of several test evaluations as the reported
  accuracy [arxiv:2212.03145 app. A]. The gap between FacT-TK at its searched
  ranks and NOAH is 0.1 points with single runs, so the claim is parity only
  [arxiv:2212.03145 tab. 1].
- **Within the universal-computation paper, the evaluation split.** The paper
  calls its evaluation set a held-out test set [arxiv:2103.05247 app. C]. The
  card notes that the released code evaluates on validation splits for ListOps
  and remote homology [arxiv:2103.05247 app. C]. Results may not be on the
  split the paper's wording suggests.
- **Within the universal-computation paper, parameter arithmetic.** The
  LayerNorm count in the text does not match its own arithmetic, and neither
  does the positional count [arxiv:2103.05247 §2.2]. The card gives the
  corrected values. Totals are the same order of magnitude.
- **Within the P-tuning v2 paper, "comparable" against its own tables.** The
  text says P-tuning v2 is comparable to fine-tuning at every scale tested
  [arxiv:2110.07602 §4.1]. The tables show it below full fine-tuning in some
  rows, for example WiC with RoBERTa-large at 73.4 against 75.6, and WSC with
  GLM-xxlarge at 93.3 against 95.2 [arxiv:2110.07602 tab. 2]. The [card](/papers/2110.07602)
  also finds that the released README's numbers differ from Table 2 and 3 in
  several rows, and that some per-task settings in the LaTeX source disagree
  with the released scripts. Check which set a reproduction should match.
- **Within the BAT paper, adapters per block.** The equations use one adapter
  symbol for both the attention and the MLP stage, but the released code
  creates two adapter modules per block. The paper's 0.32M trainable parameters match two per
  block; one per block would give about 0.16M [arxiv:2312.10611 §1]
  [arxiv:2312.10611 §3.2].
- **Within the MaPLe paper, learning rate.** Section 4.1 gives 0.0035 for
  base-to-novel training, while appendix A gives 0.0025 for MaPLe
  [arxiv:2210.03117 §4.1] [app. A]. The card follows the released configuration,
  which agrees with 0.0035.
- **Within the intrinsic-dimension paper, text against table.** The text gives
  "roughly 200" for RoBERTa-Large on MRPC and 800 for QQP, while Table 1 gives 207
  and 774 [arxiv:2012.13255 §4.2] [tab. 1]. The text says SAID improves
  consistently, but the two methods tie on QQP for RoBERTa-Large [tab. 1].
- **Within the T2I-Adapter paper, paper against released code.** The paper
  trains on 512 px inputs with COCO and LAION data, with Stable Diffusion 1.4
  [arxiv:2302.08453 §4.1]. The released main branch targets SDXL at 1024 px, with
  other training data and a different injection layout [arxiv:2302.08453 §3.3].
  The [card](/papers/2302.08453) lists the differences.
- **Across few-shot CLIP papers, different protocols.** Tip-Adapter reports
  ImageNet accuracy at 16 shots [arxiv:2111.03930 tab. 2]. MaPLe reports the
  harmonic mean of base and novel classes over 11 datasets [arxiv:2210.03117
  tab. 3]. The numbers are not comparable across these two papers.
- **Within the ManipLLM paper, the trained set.** The paper says the injected
  adapters in the visual encoder and in the language model are fine-tuned
  [arxiv:2312.16217 §3.1.1]. The card reports that the public code leaves the
  visual side frozen and trains only the language-model norm, bias and LoRA
  weights. The card did not run the code. Check which set a reproduction uses.
- **Within the DoctorGLM paper, trainable parameters.** The paper states about
  7 million trainable parameters [arxiv:2304.01097 §3.3]. The released config
  with rank 8 on query and value gives about 3.7 million by the card's
  arithmetic, which the card did not verify against the released weights.
  Verify before budgeting memory.
- **Within the DoctorGLM paper, the training length.** The paper says one epoch
  and batch size 4 [arxiv:2304.01097 §3.3]. The abstract gives 13 hours on one
  A100 and §3.3 gives 8 hours [arxiv:2304.01097 §3.3]. Treat the timing as
  author-reported.
- **Within the Tip-Adapter paper, cost of the training-free variant.** The
  paper reports tuning time for the key-tuned variant and for CLIP-Adapter, but
  no cache-building time [arxiv:2111.03930 tab. 3]. The cost of the
  training-free variant is therefore not measured against the others.
- **Within the ClipCap paper, prefix length.** The implementation details give
  a prefix length of 10 for both variants [arxiv:2111.09734 §4]. The released
  README runs the transformer variant with 40, and the paper's ablation goes up
  to 80 [arxiv:2111.09734 §4]. The paper does not say which length produced its
  table. The [card](/papers/2111.09734) lists the code details.
- **Within the ClipCap paper, language model size.** The paper says GPT-2
  "large" [arxiv:2111.09734 §3.1]. The released code loads the smaller `gpt2`
  checkpoint. The card's arithmetic matches the reported 156M to the smaller
  model, which is the card's own calculation, not the paper's [card](/papers/2111.09734).
- **Within the UCPE paper, base model size.** The paper calls Wan2.1 T2V 1.3B
  "7.3 billion parameters" [arxiv:2512.07237 app. D] and says the adapter is
  under 1% of the model [§1]. Against the 1.3B base, 35.6M is about 2.7%
  [arxiv:2512.07237 tab. 1]. Use the 1.3B figure for memory budgets.
- **Within the UCPE paper, Lat-Up channel order.** The text lists latitude
  before the up components [arxiv:2512.07237 §3.2]. The released code
  concatenates up, then latitude [card](/papers/2512.07237). Weights trained
  with one order do not load correctly with the other.
- **Within the LLM-Adapters paper, math test overlap.** The paper trains on
  Math10K, which includes MAWPS, and reports MultiArith, AddSub and SingleEq as
  test sets [arxiv:2304.01933 §3.2] [§4.2]. The released repository's README
  says those three sets come from MAWPS and overlap the training data. Its
  updated table puts GPT-3.5 ahead on MAWPS, 87.4 against 83.6 for LLaMA-13B
  LoRA, so the paper's claim of a win over GPT-3.5 on those sets is not in the
  updated table (see the [card](/papers/2304.01933)).
- **Within the LLM-Adapters paper, series placement.** The paper places the
  series adapter after the MLP layers [arxiv:2304.01933 §4.1]. The released
  repository adds the LLaMA series adapter on the input side of the up and gate
  projections, not after the MLP. Which placement produced the paper's LLaMA
  series numbers cannot be settled from the repository (see the
  [card](/papers/2304.01933)).

## Papers

- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](/papers/2205.05638): learned scaling vectors on a frozen T0 11B, with two extra losses; beats few-shot GPT-3 175B on held-out tasks at much lower inference cost (author-reported). Its code is MIT-licensed.
- [Tip-Adapter: Training-Free CLIP-Adapter for Better Vision-Language Modeling](/papers/2111.03930): sets an adapter's weights from a cache of few-shot CLIP features with no gradient training, with an optional key fine-tune. The released code has no licence, so reimplement from the paper.
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](/papers/2206.13559): a depth-wise temporal residual adapter in each block of a frozen image ViT; about 7.2M trained parameters for ViT-B/16 on Kinetics-400 (author-reported). Its code is MIT-licensed.
- [Multi-Concept Customization of Text-to-Image Diffusion](/papers/2212.04488): trains only the cross-attention key and value projections plus a placeholder token, with a closed-form merge for several concepts. Its code is under a non-commercial licence, so reimplement from the paper for commercial use.
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](/papers/2211.15583): argues that many PEFT methods are sparse fine-tuning, and proposes SAM, a mask from burn-in gradients; 77.12 average on seven RoBERTa-base tasks (author-reported). The repository has no LICENSE file, so nothing may be quoted from it.
- [FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](/papers/2212.03145): shares a Tensor-Train or Tucker factorisation of the weight change across all layers of a ViT; 0.069M parameters at parity with NOAH on VTAB-1K (author-reported). Its code is MIT-licensed.
- [Pretrained Transformers as Universal Computation Engines](/papers/2103.05247): a frozen GPT-2 with trained input and output layers, positional embeddings and norms, applied to bit, image, ListOps and protein tasks (author-reported). Its code is MIT-licensed.
- [Bi-directional Adapter for Multi-modal Tracking](/papers/2312.10611): two-stream adapters in a frozen OSTrack ViT for RGB-thermal tracking, 0.32M trained parameters (author-reported).
- [ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](/papers/2312.16217): LLaMA-Adapter fine-tuned on simulated articulated objects for contact-point prediction (author-reported). The released code has no licence.
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](/papers/2310.20150): unlearning layers on a frozen T5 with a selective teacher-student KL objective, using 0.5% of parameters (author-reported). The pinned repository has no unlearning code.
- [DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task](/papers/2304.01097): LoRA on ChatGLM-6B for Chinese medical dialogue, with a doctor-marked example evaluation rather than a benchmark (author-reported). The repository has no LICENSE file.
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](/papers/2402.18013): a survey that describes adapters, LoRA and other adaptation methods for dialogue models; it reports no experiments of its own.
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](/papers/2012.13255): trains d numbers in a fixed random Fastfood projection; RoBERTa-Large reaches 90% of MRPC full fine-tuning with about 200 values (author-reported). No official code.
- [MaPLe: Multi-modal Prompt Learning](/papers/2210.03117): learned prompts in both CLIP encoders, linked by a learned coupling; 78.55 base-to-novel harmonic mean over 11 datasets (author-reported). Its code is MIT-licensed.
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](/papers/2302.08453): small adapters that read a control map and steer a frozen Stable Diffusion UNet, composable by weighted sum (author-reported). The code repository at the recorded commit has no LICENSE file.
- [Small Language Models are the Future of Agentic AI](/papers/2506.02153): a position paper that argues small models should handle most repetitive agent calls and gives a five-step conversion procedure; it has no experiments of its own.
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](/papers/2111.01243): a survey of pre-train-then-fine-tune, prompting and generation methods for NLP, including adapters and prefix tuning; it runs no experiments of its own. No official code.
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](/papers/2304.01933): compares prefix tuning, series and parallel adapters and LoRA on LLaMA, BLOOMz and GPT-J; the parallel adapter tops the LLaMA-13B commonsense average, and its math wins over GPT-3.5 rest on overlapping test sets (author-reported). Its code is Apache-2.0 licensed.
- [ClipCap: CLIP Prefix for Image Captioning](/papers/2111.09734): maps a frozen CLIP embedding to GPT-2 prefix vectors, with an MLP mapper and GPT-2 tuned, or a transformer mapper and GPT-2 frozen (author-reported). Its code is MIT-licensed.
- [Unified Camera Positional Encoding for Controlled Video Generation](/papers/2512.07237): a ray-based encoding and latitude-up map, injected through a zero-initialised parallel attention adapter on Wan2.1 T2V 1.3B (author-reported). Its code is MIT-licensed.
- [P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](/papers/2110.07602): trainable prefix vectors at every layer of a frozen model, plus a linear head, comparable to fine-tuning on SuperGLUE dev from BERT-large to a 10B GLM with 0.1% to 3% trainable parameters (author-reported). Its code is Apache-2.0 licensed.
- [Deep transfer learning for image classification: a survey](/papers/2205.09904): a survey that lists parameter-efficient tuning among its parameter-based methods; its one experiment covers Big Transfer fine-tuning settings, with no parameter-efficient method tested.
- [A Comprehensive Overview of Large Language Models](/papers/2307.06435): a survey of LLMs of 10B parameters or more that lists parameter-efficient fine-tuning among efficiency methods; it runs no experiments, and its numbers are copied from cited papers.