Parameter-efficient fine-tuning

Overview

Parameter-efficient fine-tuning (PEFT) adapts a pretrained model by changing a small part of it. Most weights stay frozen. The trained part is either new parameters or a chosen subset of existing ones, so the difference stored per task or concept is small. The training-free cache is an edge case: it sets its small part from the labelled examples without gradient steps 2111.03930 §3.2.

Twenty-three cards cover different settings. The T-Few paper adapts a text model to classification tasks that have 20 to 70 labelled examples each 2205.05638 §3.1. The Tip-Adapter paper adapts CLIP to image classes with 1 to 16 labelled examples per class 2111.03930 §4.1. The ST-Adapter paper adapts an image transformer to video action recognition 2206.13559 §1. The Custom Diffusion paper adapts an image model to a single new concept from a few photos 2212.04488 §3.1. Further cards cover a sparse-mask method on GLUE-style text tasks 2211.15583 §5.1, a tensor-factorised adapter for vision transformers 2212.03145 §4.1, a frozen GPT-2 applied to non-text data 2103.05247 §2.2, a two-stream RGB-thermal tracker 2312.10611 §4.1.2, a robot-manipulation model built on LLaMA-Adapter 2312.16217 §3.1.1, an unlearning layer for T5 2310.20150 §3.1, and a medical dialogue model tuned with LoRA 2304.01097 §3.3. A survey of dialogue systems describes these methods but runs no experiments 2402.18013 §2.4.2. Four more cover a random subspace of the weights 2012.13255 §3, learned prompts in both halves of CLIP 2210.03117 §3.2, a control adapter for a diffusion model 2302.08453 §3.3, and a position paper that proposes LoRA for small models in agents, with no experiments 2506.02153 §3.1. Three more add a learned prefix or a parallel adapter in a new setting: a CLIP embedding mapped to prefix vectors for a captioner 2111.09734 §3.1, a ray-based camera adapter on a video diffusion transformer 2512.07237 §3.3, and a survey of NLP methods that describes adapters and prefix tuning but runs no experiments [arxiv:2111.01243 §2.4.4] 2111.01243 §3.2. A study of LLM-Adapters compares prefix tuning, series and parallel adapters, and LoRA on LLaMA, BLOOMz and GPT-J models of 6 to 13 billion parameters 2304.01933 §3.3. P-tuning v2 adds trainable prefix vectors to every layer of a frozen language model, for language understanding and sequence labeling 2110.07602 §3.2. A survey of large language models lists parameter-efficient fine-tuning among its efficiency methods, with no experiments of its own 2307.06435 §3.6. A survey of deep transfer learning for image classification names parameter-efficient tuning as one group of parameter-based methods, covering prompts, adapters and low-rank residuals [arxiv:2205.09904 §6.1]. Its one experiment reports Big Transfer fine-tuning settings, not a parameter-efficient method 2205.09904 §3.4 2205.09904 tab. 3. The papers use different backbones, tasks and metrics, so their numbers are not comparable across variants unless a section says otherwise.

The problem behind these is storage, training cost and overfitting. Storing a full copy of the model per task is expensive: the ST-Adapter authors make this point for video 2206.13559 §1, and the Custom Diffusion authors report about 3GB per concept for the full-model baseline against 75MB for their method 2212.04488 §4.1. Gradient training of few-shot adapters is slow and memory-hungry, according to the Tip-Adapter authors 2111.03930 §1. Fine-tuning every weight on very little data also overfits, and in the image case it causes language drift: after training on a rare object name, related words lose their old meaning 2212.04488 §3.1. The sparse-mask paper argues that many of these methods, including adapters and LoRA, are one family: each trains a fixed sparse subset of the weights 2211.15583 §2.2. The intrinsic-dimension paper asks the question from the other side: why a few trainable values can match full fine-tuning at all 2012.13255 §1.

The image method rests on one measurement. When the whole model is fine-tuned on one concept, the largest relative weight changes fall in cross-attention, which holds about 5% of the parameters. Text reaches the image network only through the key and value projections of those layers 2212.04488 §3.1.

For the low-rank adapter family used in language models, see Low-rank adaptation.

Variants

Learned activation scaling, (IA)3 (few-shot text classification)

Freeze the whole backbone and learn one vector per module. Each vector multiplies the keys and values of the attention layers and the intermediate activation of each feed-forward block, in every block. The vectors start at ones, so the model is unchanged before training 2205.05638 §3.3. The paper's trainable count is 540K parameters 2205.05638 app. D; its storage figure is 4.2 MB 2205.05638 tab. 1. Because each vector sits after a matrix multiply, it can be folded into that matrix for a single task, which then adds no inference cost. Keeping the vectors separate lets one batch mix tasks 2205.05638 §3.3. Card: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning.

The recipe starts from T0 11B. The vectors are first pre-trained for 100,000 steps on T0's multitask mixture, then fine-tuned per task for 1,000 steps at batch size 8 with learning rate 3e-3 2205.05638 §3.4 [arxiv:2205.05638 §3.5]. The authors report 72.4% median accuracy over nine held-out tasks, against 66.9% for zero-shot T0 and 66.6% for few-shot GPT-3 175B in-context learning 2205.05638 tab. 1. Without the pre-training step, accuracy is 70.8% 2205.05638 §4.4. Results on the RAFT benchmark are in the card.

Trained subset plus a new token (single concept, image generation)

Freeze the model except the key and value projections of every cross-attention layer and the embedding of one new placeholder token. Train on the target images mixed about 1:1 with roughly 200 real images retrieved for similar captions. The paper reports 250 steps at batch 8 and a learning rate of 8\times10^{-5} 2212.04488 §3.2 2212.04488 app. E. Card: Multi-Concept Customization of Text-to-Image Diffusion.

Reported on the single-concept benchmark: text alignment 0.795, image alignment 0.775, KID 20.96 2212.04488 tab. 1.

Joint training of several concepts

Give each concept its own placeholder token and train the same projections plus all tokens on the union of the concept datasets, with a batch drawn per concept 2212.04488 §3.2. The paper uses 500 steps for two concepts 2212.04488 §3.2. Joint training has the best multi-concept scores in the paper: text 0.801, image 0.706 2212.04488 tab. 1.

Closed-form merge of separately trained concepts

Fine-tune each concept on its own, then combine the projection matrices in one least-squares solve per layer and per key or value matrix. The merged matrix reproduces each concept's own target words while staying close to the original matrix on regularization captions 2212.04488 §3.2 [arxiv:2212.04488 eq. 4] 2212.04488 eq. 5. The authors report about 2 s for the merge 2212.04488 §3.2. Multi-concept scores are slightly below joint training: text 0.800 and image 0.695 for the optimization variant, text 0.797 and image 0.700 for sequential training 2212.04488 tab. 1.

Low-rank compression of the stored update

The difference between fine-tuned and original projections has quickly decaying singular values, so it can be stored as a truncated SVD. Keeping the top 60% of the singular-value mass gives about five times smaller storage with similar metrics; keeping only the top 1% lowers image alignment [arxiv:2212.04488 app. C]. Forcing a low-rank update during training gave worse results than compressing afterwards 2212.04488 app. C.

Training-free key-value cache (few-shot CLIP classification)

Store the CLIP image features of the labelled examples as keys, and their one-hot labels as values. For a new image, the affinity to each key is an exponential of its cosine similarity, and the weighted label vote, scaled by a factor alpha, is added to CLIP's zero-shot logits. Building the cache takes one pass over the examples and no gradient steps 2111.03930 §3.2 eq. 3 eq. 4. With the weights set this way, the adapter form equals the cache lookup eq. 7 eq. 8; the paper lists three differences from a learned adapter: the weights are set rather than learned, the hidden width is NK, and the activation is bounded in [0, 1] 2111.03930 §3.2. Card: Tip-Adapter: Training-Free CLIP-Adapter for Better Vision-Language Modeling.

On ImageNet with a ResNet-50 at 16 shots, the paper reports 62.03% against 60.33% for zero-shot CLIP and 63.59% for CLIP-Adapter 2111.03930 tab. 2. The ablation uses beta 5.5 and alpha 1.0 on that setting [arxiv:2111.03930 tab. 4]. Above 16 shots, the paper averages random groups of examples into 16 prototypes per class, and reports 63.15% at 128 shots 2111.03930 §4.4 tab. 4. The paper reports only fine-tuning time, not cache-building time 2111.03930 tab. 3.

Cache with key fine-tuning (Tip-Adapter-F)

Start from the cache and unfreeze only the keys. Train for 20 epochs with cross-entropy on the few-shot set; the values, the text classifier and both CLIP encoders stay fixed 2111.03930 §3.2 §4.1. This reaches 65.51% at 16 shots on ImageNet 2111.03930 tab. 2. Training only the values gives 60.90, and training keys and values together collapses 2111.03930 tab. 5. The paper reports 5 minutes on one RTX 3090, against 50 minutes for CLIP-Adapter and 14 hours 40 minutes for CoOp 2111.03930 tab. 3. Card: the same Tip-Adapter card, which covers the training-free variant as well.

Residual bottleneck adapter per block (video recognition)

Insert a bottleneck into each block of a frozen image transformer. It projects the patch tokens down to width 384, applies a depth-wise 3D convolution with kernel 3×1×1 that mixes each channel over three neighbouring frames, applies an activation the paper does not name, and projects back up before a residual addition 2206.13559 §3.2 eq. 2 tab. 7 tab. 8. Only the adapters and a linear head train. The default is one adapter per block on Kinetics-400 2206.13559 tab. 8. On Kinetics-400 with a CLIP ViT-B/16 and 8 frames, the paper reports 82.0% top-1 against 81.7% for full fine-tuning, with 7.20M trained parameters against 121.57M 2206.13559 tab. 1. Training takes 23 hours against 40 hours on 8 V100 GPUs 2206.13559 tab. 6. Card: ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning.

The kernel shape is tuned: 3×3×3 gives a similar Kinetics-400 score but a lower Something-Something-v2 score 2206.13559 tab. 7. Adapters in deeper blocks give larger gains 2206.13559 §4.3 tab. 5.

Sparse mask chosen from burn-in gradients (SAM)

Keep the pretrained weights as they are and pick a fixed set of entries to update. The budget is 0.5% of the model's parameters 2211.15583 §5.1. Choose the entries with the largest squared gradients. The gradients come from a few hundred to two thousand burn-in steps at the pretrained weights 2211.15583 §4 §5.1. After that the mask is fixed, and only the selected entries change 2211.15583 §4. The paper shows that the random, rule-based and projection-based methods it reviews can all be written in this form, and it proposes the burn-in selection as an alternative 2211.15583 §2.2 §4. Card: On the Effectiveness of Parameter-Efficient Fine-Tuning.

On seven GLUE and SuperGLUE tasks with RoBERTa-base, averaged over 10 seeds, the paper reports 77.12 for this method against 74.78 for full tuning, 76.03 for an adapter and 75.11 for LoRA 2211.15583 tab. 1. When each run drops a random 10% of the training set, the averages are 76.40 and 74.81 2211.15583 tab. 2. The mask does not shrink the computation: the paper's average training time is 2.02 hours against 1.67 for full tuning 2211.15583 tab. 3. Only the storage of the changed entries is lower, and no storage numbers are given 2211.15583 app. A.5.

The paper also sweeps the budget. The mean and spread of scores fall as the budget shrinks, until a threshold below which performance drops sharply and training error rises 2211.15583 §5.2 fig. 5. A smaller budget is therefore not safe to assume without a check.

Shared tensor factorisation of the weight change (FacT)

Stack the attention and feed-forward matrices of all blocks into one tensor, then train a low-rank factorisation of the change to that tensor. Two formats are proposed: Tensor-Train (FacT-TT) and Tucker (FacT-TK) 2212.03145 §3.2.2 §3.2.3. The matrices share two factors across all layers; each matrix has its own small core in TT, or its own row of a layer-factor matrix in Tucker. The factor that maps back to the weight width starts at zero, so training starts from the pretrained model. The factors are folded into the weights before inference, so inference cost does not change 2212.03145 §3.2. The paper says the matrix-batch case of its format is LoRA applied to each matrix separately 2212.03145 §3.2.1. Card: FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer.

On VTAB-1K, with 19 tasks and a ViT-B/16 pretrained on ImageNet-21K, the Tucker variant at its searched ranks averages 75.6% with 0.069M backbone parameters. NOAH averages 75.5% with 0.361M, and full tuning averages 68.9% with 85.8M 2212.03145 tab. 1. A TT variant at rank 4 averages 73.5% with 0.008M parameters 2212.03145 tab. 1. On a Swin-B backbone, the TT variant at rank 16 averages 77.4% with 0.135M parameters, against 75.0% for full tuning 2212.03145 tab. 2. Hierarchical models need one factor set per stage 2212.03145 §4.3.

Trainable norms and input/output layers around a frozen transformer (cross-modal)

Keep the self-attention and feed-forward blocks of a language-pretrained transformer frozen. Train a new linear input layer that maps the new data into the model width, a new linear output layer, the layer-norm parameters and the positional embeddings 2103.05247 §2.2. The paper tests this on bit strings, pixel patches, nested arithmetic expressions and amino-acid sequences 2103.05247 §2.1. Card: Pretrained Transformers as Universal Computation Engines.

The layer-norm parameters carry most of the gain. On CIFAR-10, training them alone gives 54% and training only the output layer gives 25% 2103.05247 tab. 15. Freezing them drops CIFAR-10 from 68.2% to 61.5% 2103.05247 tab. 16. Language pretraining helps the frozen model: with random initialisation, bit memory reaches 75.8% and CIFAR-10 reaches 61.7%, against 100% and 68.2% with language pretraining 2103.05247 tab. 2. Training only the output layer on cached features gives 32.8% on ListOps and 24.7% on CIFAR-10 LRA, with much faster epochs 2103.05247 tab. 10.

Bi-directional adapter in a frozen two-stream tracker (RGB and thermal)

Start from a frozen ViT tracker (OSTrack) that runs one stream per modality, with the two streams sharing layer weights. Add one adapter in the attention stage and one in the MLP stage of each layer. Each adapter reads the other stream and adds a prompt to this one, so neither modality is fixed as the main one 2312.10611 §3.2. Only the adapters train; the encoder, the head and the layer norms are frozen 2312.10611 §3.2. The paper reports 0.32M trained parameters 2312.10611 §1. Card: Bi-directional Adapter for Multi-modal Tracking.

On LasHeR the paper reports 70.2 precision and 56.3 success, and on RGBT234 86.8 and 64.1 2312.10611 tab. 1. A one-direction RGB-only adapter reaches 69.0 and 55.4 on LasHeR 2312.10611 tab. 2. Adapters in one layer only give 61.4 and 49.3 on LasHeR, and adapters in all 12 layers give the headline numbers 2312.10611 tab. 3. Training takes about 8 hours on four GPUs for LasHeR 2312.10611 §4.1.2.

Adapter in a multimodal LLM for robot manipulation (ManipLLM)

Build on LLaMA-Adapter: a CLIP visual encoder, a projection into the language model, and adapters injected into the language model with the LoRA method 2312.16217 §3.1.1. The model predicts a contact pixel and gripper directions from an image and a task prompt. Its training is on simulated articulated objects, with a separate test-time update of a visual adapter 2312.16217 §3.3. On 10 unseen simulated categories, the paper reports 0.54 initial-movement success against 0.41 for the best baseline 2312.16217 tab. 1. Card: ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation.

Unlearning layers on a frozen language model

Insert small unlearning layers after the feed-forward networks of a frozen T5 model, and train only those layers 2310.20150 §3.1 fig. 1. The paper tunes 0.5% of the parameters 2310.20150 §4.4. The objective is a selective teacher-student KL term: it pulls the student toward the teacher on retained data and pushes it away on forgotten data. A task loss on retained data and a masked-language-model term on forgotten data are added 2310.20150 §3.1 eq. 2 eq. 5. The adapter architecture is not stated 2310.20150 §4.4. Card: Unlearn What You Want to Forget: Efficient Unlearning for LLMs.

On IMDB with T5-base and 0.5% of the training data forgotten, the paper reports a forget-set accuracy of 65.7 against 92.5 for re-training, with test accuracy 93.0 against 92.8 2310.20150 tab. 2. The update takes 1200 seconds against 6685 for re-training 2310.20150 tab. 2. The paper reads a higher masked-language-model loss on forgotten data as the information being harder to extract 2310.20150 §4.2. Merging the layers across deletion requests uses a closed-form step 2310.20150 eq. 7.

LoRA on a medical dialogue model

Freeze ChatGLM-6B and add LoRA to the query and value projections, with rank 8 and alpha 16. Train one epoch with the Lion optimizer at learning rate 2e-5, no warmup, no weight decay and batch size 4 2304.01097 §3.3. The method itself is LoRA; see Low-rank adaptation for the general case. A survey of dialogue systems describes LoRA and adapters in the same terms and reports no experiments 2402.18013 §2.4.2. Card: DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task.

The paper gives no quantitative benchmark. Its evidence is a few answers marked by a doctor 2304.01097 tab. 2 tab. 3 tab. 4. The authors say the work is not suitable for commercial or clinical use 2304.01097 §6. Their cost estimate is 18.75 USD for 100K question-answer pairs, based on an assumed throughput 2304.01097 §5.

Series and parallel adapters, prefix tuning and LoRA on LLMs (LLM-Adapters)

Compare four families on one frozen base model: learned prefix vectors, a series bottleneck adapter, a parallel bottleneck adapter, and LoRA. A series adapter can sit on the input or the output of a linear map; the paper places it after the MLP layers 2304.01933 §2 §4.1. A parallel adapter runs beside a frozen linear map and adds its output to that map's output 2304.01933 §2. The shared recipe is a batch of 16 and three epochs, with learning rate 3e-4 for the adapters and LoRA and 3e-2 for prefix tuning. The bottleneck width is 256 and the prefix has 10 virtual tokens 2304.01933 §3.3 §4.1. LoRA rank 32 is the setting used in the main table, chosen from a sweep up to 32 2304.01933 app. A.4. Training uses about 10K math samples and 170K commonsense samples 2304.01933 §3.3. Card: LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models.

On eight commonsense tasks, LLaMA-13B with a parallel adapter averages 81.5, against 80.5 with LoRA, 79.5 with a series adapter and 77.0 for ChatGPT 2304.01933 tab. 4. On LLaMA-13B, prefix tuning is the weakest of the four, at 68.4 2304.01933 tab. 4. On six math sets, LLaMA-13B LoRA averages 65.4 against 70.4 for GPT-3.5 2304.01933 tab. 3. Three math sets are reported as wins over GPT-3.5, but they overlap the training data (see Contradictions). The paper's placement study on LLaMA-7B math puts the parallel adapter on the MLP layers at 61.7, against 59.3 on the attention layers and 59.8 on both 2304.01933 app. A.3. Those sweeps were run on the same test sets as the main table, so the chosen settings are optimistic [arxiv:2304.01933 app. A.4]. Each setting is a single run with no spread reported [arxiv:2304.01933 tab. 4].

Random subspace of the weights (intrinsic dimension)

Freeze all pretrained weights and train only d numbers. A fixed random projection, the Fastfood transform, maps those numbers into the full weight space, and training starts from zero, so the model starts at its pretrained weights 2012.13255 §3 §4.1 eq. 2. The paper's measure is the smallest d that reaches 90% of full fine-tuning, called d90. Its SAID variant adds one learned scale per layer 2012.13255 §3 eq. 3. For RoBERTa-Large on MRPC, d90 is 207 with SAID and 322 with DID; on QQP it is 774 for both 2012.13255 tab. 1. Card: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.

The authors use d90 as a measure of task complexity, not as a recipe for deployment 2012.13255 §1 §5. It is an upper bound §4.2, the learning rates are not stated §4.1 §5.1, and a dense projection would need an estimated 1.42 TB for RoBERTa-Large at d = 1000, which is why the Fastfood form is used §3. A stored model is the vector plus the seed that rebuilds the projection §5. See also intrinsic dimension.

Prompt tokens in both encoders of CLIP (MaPLe)

Add learned prompt tokens to the text branch and to the image branch of a frozen CLIP ViT-B/16. Each vision prompt is a learned linear function of the text prompt in the same layer. Prompts are inserted in the first nine blocks, and only the prompt parts train 2210.03117 §3.2 §3.2.3 §4.1. The paper reports about 3.55M trainable parameters, 2.85% of CLIP 2210.03117 tab. 6. Training uses 16 shots per class §4.1.

Averaged over 11 datasets, base-to-novel harmonic mean is 78.55 against 75.83 for Co-CoOp, and novel-class accuracy is 75.14 against 71.69 2210.03117 tab. 3. A variant with one coupling layer shared across layers has 0.41M parameters and a harmonic mean of 78.11 tab. 6. Card: MaPLe: Multi-modal Prompt Learning. Training runs 5 epochs against 10 for Co-CoOp §4.6. At batch size 1 MaPLe is slower than Co-CoOp (60.2 against 64.6 frames per second), and at batch size 4 it is faster (239.0 against 114.7) 2210.03117 tab. 6.

Deep prefix vectors in every layer (P-tuning v2)

Keep a pretrained transformer frozen and give each layer its own trainable prefix, read as extra keys and values. A linear head on the first token makes the prediction 2110.07602 §3.2 §3.3 fig. 2. Input-only prompt tuning, the baseline the paper compares against, adds vectors at the input alone 2110.07602 §2. The paper reports that input-only prompts fall well behind fine-tuning on models below about 10B parameters and on hard sequence labeling 2110.07602 §1 §3.1. An optional two-layer MLP can reparameterise the prefix; the paper says it helps on some datasets and not on others 2110.07602 §3.3. Card: P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks.

Prompt length depends on the task. The paper suggests fewer than 20 prefix positions for simple classification and around 100 for hard sequence labeling 2110.07602 §3.3 fig. 4. With the same number of prompted layers, prompts in the deeper layers beat prompts in the shallow layers on RTE and BoolQ with BERT-large 2110.07602 §4.3 fig. 3. Each prefix position adds attention work in every layer, and the paper does not discuss this cost.

On SuperGLUE dev, the paper reports RTE at 89.5 for P-tuning v2 against 86.6 for full fine-tuning with RoBERTa-large, and 88.8 against 88.7 on BoolQ with a 10B GLM 2110.07602 tab. 2. On RoBERTa-large, BoolQ is lower, at 84.8 against 86.9 2110.07602 tab. 2. On CoNLL05 WSJ with RoBERTa-large, it reaches 89.2 F1 against 76.8 for input-only prompt tuning and 90.2 for full fine-tuning 2110.07602 tab. 3. These are dev-set numbers. The paper does not say how the reported dev number was chosen, and the setting is fully supervised 2110.07602 §4.

Control-map adapters beside a frozen diffusion model (T2I-Adapter)

Unlike the variants above, the base model's weights are not changed or subset-tuned. A separate small convolutional adapter is trained for each control map, such as sketch, segmentation, keypose, depth or colour, while Stable Diffusion stays frozen. The adapter's features are added to the UNet encoder features at four scales 2302.08453 §3.3 eq. 3 eq. 4. Several adapters are combined by a weighted sum with no retraining eq. 5; the paper sets the weights by hand §3.3 §5. The base adapter is about 77M parameters, and the 18M small adapter is used for colour §1 §4.4.2. Training uses learning rate 1e-5, batch size 8 and 10 epochs §4.1. On COCO validation, text plus segmentation gives FID 16.78 and text plus sketch gives 17.36, against 24.68 for Stable Diffusion with text only 2302.08453 tab. 1. The adapter runs once per sample at inference §1. Card: T2I-Adapter.

Mapped CLIP prefix for a language model (image captioning)

Map a frozen CLIP image embedding to k vectors, each the size of a word embedding, and place them in front of the caption tokens of GPT-2. The loss is cross-entropy over the caption tokens only, conditioned on the prefix 2111.09734 §3.1 eq. 5. The paper gives two mappers. An MLP with one hidden layer is used when GPT-2 is also fine-tuned 2111.09734 §3.3. A transformer mapper, with 8 layers and learned constant inputs, is used when GPT-2 is frozen, so only the mapper trains 2111.09734 §3.2 §3.3 §4. Fine-tuning CLIP was tried and did not help 2111.09734 §3.2. The paper's implementation details give a prefix length of 10 [arxiv:2111.09734 §4]. Card: ClipCap: CLIP Prefix for Image Captioning.

On Conceptual Captions the MLP variant reaches CIDEr 87.26 against 77.57 for VLP, and the transformer variant reaches 71.82 2111.09734 tab. 1 (A). On nocaps, the transformer variant has an overall CIDEr of 65.83 against 63.8 for Oscar, which uses object tags 2111.09734 tab. 1 (B). On COCO, Oscar stays best at 124.12 against 113.08 for the transformer variant 2111.09734 tab. 1 (C). The paper's own runs take 6 to 80 GPU hours on one GTX 1080, depending on the dataset 2111.09734 tab. 1.

Parallel zero-initialised camera adapter (video diffusion transformer)

Keep a video diffusion transformer frozen and add a spatial attention adapter beside each self-attention block. Each token carries a ray-based frame, in which lens distortion enters the attention, and an optional latitude-and-up map sets pitch and roll 2512.07237 §3.2. The adapter's output projection starts at zero, so the pretrained model is unchanged at the first step 2512.07237 §3.3. The paper's trainable size is 35.6M parameters on Wan2.1 T2V 1.3B, with the base weights frozen 2512.07237 tab. 1 app. D. For the camera encodings it is compared with, see camera pose encoding. Card: Unified Camera Positional Encoding for Controlled Video Generation.

On the paper's synthetic set with absolute orientation, the rotation error is 4.12 degrees, against 9.23 for ReCamMaster with 354M trainable parameters and 17.92 for Wan CameraCtrl with 1.5B 2512.07237 tab. 1. On a zero-shot RealEstate10K test with 100 clips, the paper reports 0.56 degrees against 0.62 for AC3D, which was trained on that dataset 2512.07237 tab. 2. One clip takes 184 s on a single A800 2512.07237 app. D.

Full fine-tuning (comparison, not parameter-efficient)

In the text paper, full fine-tuning of the 3B model averages 60.66 across the held-out tasks without the extra losses, against 61.72 for (IA)3 2205.05638 app. D. The image paper's full-weight baseline reaches the best KID of its single-concept runs, 19.27, but has lower image alignment, 0.748, than the subset method 2212.04488 tab. 1. It takes about 20 minutes on 4 A100s against about 6 minutes on 2 A100s for the subset method 2212.04488 §4.1. The sparse-mask paper's full-tuning baseline reaches 74.78 on GLUE-style tasks, against 77.12 for its method 2211.15583 tab. 1.

A survey describes a staged alternative as common practice: keep the pretrained layers frozen while a new head trains, then unfreeze them gradually with a learning rate between 1e-3 and 1e-5. The order of unfreezing and the number of stages are not stated 2111.01243 §2.4.2. Card: Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey.

Comparison of PEFT methods on the text tasks

Averages as printed in the T-Few appendix, for the setting with neither extra loss. The averaging rule is not stated 2205.05638 app. D.

Comparison of PEFT methods on video

Kinetics-400, CLIP ViT-B/16, 8 frames, with the backbone frozen unless noted 2206.13559 tab. 1.

When to use which

Implementation sketch

Eight shapes recur across the cards: scale a frozen layer's output by a learned vector, add a residual bottleneck beside a frozen block, look up a cache of labelled features without training, train a chosen subset of existing weights, share a low-rank factor across all layers, train only the norms and the input and output layers, map a frozen encoder's output to prefix vectors for a frozen language model, and prepend trainable key and value prefixes to every layer of a frozen backbone. The sketches below cover the first five.

The (IA)3 layer, in outline. The vector starts at ones, and the frozen weight is never updated 2205.05638 §3.3.

# (IA)3 on one linear layer: scale its output by a learned vector  [§3.3]
# Input:  x (B, T, d_in); frozen weight and bias; learned scale (d_out,)
# Output: y (B, T, d_out)

def ia3_linear(x, weight, bias, scale):
    # Frozen projection; its weights are never updated  [§3.3]
    hidden = linear(x, weight, bias)                   # (B, T, d_out)
    # Element-wise rescale; ones at init, so the model is unchanged  [§3.3]
    return hidden * scale                              # broadcast over B, T

The residual adapter on one block's patch tokens. The paper writes an activation but does not name it, so it is a parameter here. The class token is left out of the sketch.

# Residual adapter with a temporal depth-wise conv  [eq. 2] [§3.2]
# Input:  patch tokens x (B, T, N, d); N patches per frame
# Output: adapted tokens, same shape

def st_adapter(x, w_down, w_up, activation):
    # Down-project to the bottleneck width, default 384  [§3.2] [tab. 8]
    h = x @ w_down                             # (B, T, N, r)
    # Depth-wise conv over frames only; each channel sees 3 frames  [tab. 7]
    h = depthwise_conv_time(h, kernel=3)       # (B, T, N, r)
    # Activation f; the paper leaves it unnamed  [eq. 2]
    h = activation(h)
    # Up-project and add back residually  [eq. 2]
    return x + h @ w_up                        # (B, T, N, d)

The Tip-Adapter cache, training-free. The paper's equation for the zero-shot term has no scale; see the Contradictions section before matching reported numbers.

# Training-free key-value cache for few-shot CLIP  [§3.2] [eq. 3-6]
# Input:  K labelled images per class, CLIP encoders, text weights (C, N)
# Output: class logits (N,) for one test image

def build_cache(train_images, train_labels):
    # Keys: normalised CLIP image features of the few-shot set  [eq. 3]
    keys = stack([normalise(clip_image(x)) for x in train_images])
    # Values: one-hot labels of the same examples  [eq. 4]
    values = one_hot(train_labels, num_classes)
    return keys, values

def cache_logits(image, keys, values, text_weights, alpha, beta):
    # Normalised test feature is the query  [§3.2]
    query = normalise(clip_image(image))
    # Affinity decreases with cosine distance to each key  [eq. 5]
    affinity = exp(-beta * (1 - keys @ query))
    # Zero-shot logits plus the alpha-scaled label vote  [eq. 6]
    return query @ text_weights + alpha * (affinity @ values)

The sparse mask from burn-in gradients. The mask is built once, before the trained run, and it is fixed afterwards 2211.15583 §4.

# Sparse mask from burn-in gradients  [§4] [eq. 10] [§5.1]
# Input:  pretrained weights theta0 (m values), budget p, burn-in batches
# Output: binary mask with floor(m * p) ones

def sam_mask(theta0, batches, p, burn_in):
    # Sum the loss gradient at theta0 over the burn-in batches  [§5.1]
    grad_sum = zeros_like(theta0)                 # (m,)
    for batch in batches[:burn_in]:
        grad_sum = grad_sum + gradient(theta0, batch)
    # Rank entries by squared gradient; the curvature term is dropped  [eq. 10]
    scores = grad_sum ** 2                        # (m,)
    mask = zeros_like(theta0)                     # (m,)
    mask[top_k_indices(scores, floor(size(theta0) * p))] = 1
    # Training updates only the entries where mask is 1  [§4]
    return mask

The shared-factor increment for one stacked matrix, in the Tensor-Train form. The second factor is zero at the start, so the increment is zero at the start.

# FacT-TT increment for one stacked weight matrix  [§3.2.2] [eq. 8] [eq. 9]
# Input:  tokens (B, N, d); shared U, V (d, r); per-matrix core (r, r)
# Output: increment added to the frozen matrix output (B, N, d)

def fact_tt_increment(tokens, U, V, core, scale):
    # Project into the shared rank-r space  [eq. 9]
    hidden = tokens @ U                           # (B, N, r)
    # This matrix's own r x r core  [eq. 8]
    hidden = hidden @ core                        # (B, N, r)
    # Back to width d; V is zero at init  [§3.2.2]
    return scale * (hidden @ V.T)                 # (B, N, d)

Contradictions

• Within the T-Few paper, the claim about full fine-tuning. The text says (IA)3 is the only PEFT method that beats full fine-tuning [arxiv:2205.05638 §3.3]. The appendix averages show LoRA slightly ahead of full fine-tuning in two of three settings: 61.01 against 60.66 with no extra losses, and 62.96 against 62.71 with the unlikelihood loss only 2205.05638 app. D. The gaps are small, and the averaging rule is not stated.
• Within the T-Few paper, the loss attribution. The method section gives 62.71% as the result of adding the length-normalized loss to full fine-tuning. In the appendix, 62.71 is the unlikelihood-only average, and the length-normalized-only average is 63.25 2205.05638 §3.2 2205.05638 app. D.
• Within the T-Few paper, parameter count against storage. The appendix lists 540K trainable parameters for (IA)3, but the paper gives 4.2 MB, which is about 1.05M single-precision parameters. The paper does not reconcile the two 2205.05638 app. D 2205.05638 tab. 1. Measure the checkpoint before budgeting storage.
• Within the Custom Diffusion paper, paper against released code. The stated settings and the released code differ on training steps, the retrieval filter and the size range of augmentation. The differences are listed on the card and should be checked before a re-implementation.
• Within the Tip-Adapter paper, "comparable" against its own table. The authors describe the training-free variant as comparable to CLIP-Adapter and say it falls behind on some datasets as shots increase [arxiv:2111.03930 §4.3]. On ImageNet its table has it below CLIP-Adapter at every shot count 2111.03930 tab. 2.
• Within the Tip-Adapter paper, logit scale. Eq. 6 has no scale on the zero-shot term 2111.03930 eq. 6. The card reports that the alpha values in the paper's tables match a scale of 100 used in the released code 2111.03930 tab. 4. Reproduce the numbers with the scale, or the numbers will not match. See the card.
• Within the Tip-Adapter paper, model selection. The paper does not state how the fine-tuning epoch, alpha or beta were chosen [arxiv:2111.03930 §4.1]. The released code chooses some of these on the test split, so test numbers from that code are optimistic. The card lists the details.
• Within the ST-Adapter paper, activation. Eq. 2 includes an activation f, but the paper does not name it 2206.13559 eq. 2. The released code applies none, which makes the adapter linear apart from its residual form (see the card).
• Within the ST-Adapter paper, placement. The text puts one adapter before multi-head self-attention 2206.13559 §3.3. The released Kinetics-400 model inserts it before the MLP instead. The card gives the code references.
• Within the ST-Adapter paper, parameter ratio. The abstract's figure of about 20 times fewer updated parameters does not match Table 1, which gives 121.57M against 7.20M, about 17 times 2206.13559 tab. 1.
• Within the sparse-mask paper, random against full tuning. The text says even the random mask outperforms full tuning 2211.15583 §5.2. In its own Table 1 the averages are 74.55 for random and 74.78 for full tuning 2211.15583 tab. 1. The gap is well within the standard deviations.
• Within the sparse-mask paper, the budget. The paper states one global count of masked entries 2211.15583 §2.1. The released code applies the count to each parameter tensor and masks only the encoder parameters. The card records the code details, which are not in the paper. The paper's stability proof also assumes a random mask, while SAM chooses one deterministically 2211.15583 app. A.2.
• Within the sparse-mask paper, the reported training cost. The paper presents its approach as a parameter-efficient one, but its own timing table puts it above full tuning 2211.15583 tab. 3.
• Across PEFT papers, compute saving. The sparse-mask paper says masking does not reduce compute 2211.15583 app. A.5. The ST-Adapter paper reports 23 hours against 40 for full tuning 2206.13559 tab. 6. The Custom Diffusion paper reports about 6 minutes against about 20 minutes 2212.04488 §4.1. These are different backbones, hardware and tasks, so a PEFT method is not reliably cheaper to train. Measure it for the setup in use.
• Across PEFT papers, beating full tuning. The sparse-mask paper reports 77.12 against 74.78 for full tuning on GLUE-style tasks [arxiv:2211.15583 tab. 1]. The FacT paper reports 75.6 against 68.9 on VTAB-1K, with full tuning measured in its own setup 2212.03145 tab. 1. The Custom Diffusion paper finds full tuning has the best KID 2212.04488 tab. 1. Each full-tuning baseline is set up differently, so these results do not settle which approach wins.
• Within the FacT paper, the reporting protocol. The paper says the final results come from training on all samples and testing, and it does not describe any selection rule 2212.03145 app. A. The card records that the released code keeps the best of several test evaluations as the reported accuracy 2212.03145 app. A. The gap between FacT-TK at its searched ranks and NOAH is 0.1 points with single runs, so the claim is parity only 2212.03145 tab. 1.
• Within the universal-computation paper, the evaluation split. The paper calls its evaluation set a held-out test set 2103.05247 app. C. The card notes that the released code evaluates on validation splits for ListOps and remote homology 2103.05247 app. C. Results may not be on the split the paper's wording suggests.
• Within the universal-computation paper, parameter arithmetic. The LayerNorm count in the text does not match its own arithmetic, and neither does the positional count 2103.05247 §2.2. The card gives the corrected values. Totals are the same order of magnitude.
• Within the P-tuning v2 paper, "comparable" against its own tables. The text says P-tuning v2 is comparable to fine-tuning at every scale tested 2110.07602 §4.1. The tables show it below full fine-tuning in some rows, for example WiC with RoBERTa-large at 73.4 against 75.6, and WSC with GLM-xxlarge at 93.3 against 95.2 2110.07602 tab. 2. The card also finds that the released README's numbers differ from Table 2 and 3 in several rows, and that some per-task settings in the LaTeX source disagree with the released scripts. Check which set a reproduction should match.
• Within the BAT paper, adapters per block. The equations use one adapter symbol for both the attention and the MLP stage, but the released code creates two adapter modules per block. The paper's 0.32M trainable parameters match two per block; one per block would give about 0.16M 2312.10611 §1 2312.10611 §3.2.
• Within the MaPLe paper, learning rate. Section 4.1 gives 0.0035 for base-to-novel training, while appendix A gives 0.0025 for MaPLe 2210.03117 §4.1 app. A. The card follows the released configuration, which agrees with 0.0035.
• Within the intrinsic-dimension paper, text against table. The text gives "roughly 200" for RoBERTa-Large on MRPC and 800 for QQP, while Table 1 gives 207 and 774 2012.13255 §4.2 tab. 1. The text says SAID improves consistently, but the two methods tie on QQP for RoBERTa-Large tab. 1.
• Within the T2I-Adapter paper, paper against released code. The paper trains on 512 px inputs with COCO and LAION data, with Stable Diffusion 1.4 2302.08453 §4.1. The released main branch targets SDXL at 1024 px, with other training data and a different injection layout 2302.08453 §3.3. The card lists the differences.
• Across few-shot CLIP papers, different protocols. Tip-Adapter reports ImageNet accuracy at 16 shots 2111.03930 tab. 2. MaPLe reports the harmonic mean of base and novel classes over 11 datasets [arxiv:2210.03117 tab. 3]. The numbers are not comparable across these two papers.
• Within the ManipLLM paper, the trained set. The paper says the injected adapters in the visual encoder and in the language model are fine-tuned 2312.16217 §3.1.1. The card reports that the public code leaves the visual side frozen and trains only the language-model norm, bias and LoRA weights. The card did not run the code. Check which set a reproduction uses.
• Within the DoctorGLM paper, trainable parameters. The paper states about 7 million trainable parameters 2304.01097 §3.3. The released config with rank 8 on query and value gives about 3.7 million by the card's arithmetic, which the card did not verify against the released weights. Verify before budgeting memory.
• Within the DoctorGLM paper, the training length. The paper says one epoch and batch size 4 2304.01097 §3.3. The abstract gives 13 hours on one A100 and §3.3 gives 8 hours 2304.01097 §3.3. Treat the timing as author-reported.
• Within the Tip-Adapter paper, cost of the training-free variant. The paper reports tuning time for the key-tuned variant and for CLIP-Adapter, but no cache-building time 2111.03930 tab. 3. The cost of the training-free variant is therefore not measured against the others.
• Within the ClipCap paper, prefix length. The implementation details give a prefix length of 10 for both variants 2111.09734 §4. The released README runs the transformer variant with 40, and the paper's ablation goes up to 80 2111.09734 §4. The paper does not say which length produced its table. The card lists the code details.
• Within the ClipCap paper, language model size. The paper says GPT-2 "large" 2111.09734 §3.1. The released code loads the smaller gpt2 checkpoint. The card's arithmetic matches the reported 156M to the smaller model, which is the card's own calculation, not the paper's card.
• Within the UCPE paper, base model size. The paper calls Wan2.1 T2V 1.3B "7.3 billion parameters" 2512.07237 app. D and says the adapter is under 1% of the model §1. Against the 1.3B base, 35.6M is about 2.7% 2512.07237 tab. 1. Use the 1.3B figure for memory budgets.
• Within the UCPE paper, Lat-Up channel order. The text lists latitude before the up components 2512.07237 §3.2. The released code concatenates up, then latitude card. Weights trained with one order do not load correctly with the other.
• Within the LLM-Adapters paper, math test overlap. The paper trains on Math10K, which includes MAWPS, and reports MultiArith, AddSub and SingleEq as test sets 2304.01933 §3.2 §4.2. The released repository's README says those three sets come from MAWPS and overlap the training data. Its updated table puts GPT-3.5 ahead on MAWPS, 87.4 against 83.6 for LLaMA-13B LoRA, so the paper's claim of a win over GPT-3.5 on those sets is not in the updated table (see the card).
• Within the LLM-Adapters paper, series placement. The paper places the series adapter after the MLP layers 2304.01933 §4.1. The released repository adds the LLaMA series adapter on the input side of the up and gate projections, not after the MLP. Which placement produced the paper's LLaMA series numbers cannot be settled from the repository (see the card).

Papers

• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning: learned scaling vectors on a frozen T0 11B, with two extra losses; beats few-shot GPT-3 175B on held-out tasks at much lower inference cost (author-reported). Its code is MIT-licensed.
• Tip-Adapter: Training-Free CLIP-Adapter for Better Vision-Language Modeling: sets an adapter's weights from a cache of few-shot CLIP features with no gradient training, with an optional key fine-tune. The released code has no licence, so reimplement from the paper.
• ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning: a depth-wise temporal residual adapter in each block of a frozen image ViT; about 7.2M trained parameters for ViT-B/16 on Kinetics-400 (author-reported). Its code is MIT-licensed.
• Multi-Concept Customization of Text-to-Image Diffusion: trains only the cross-attention key and value projections plus a placeholder token, with a closed-form merge for several concepts. Its code is under a non-commercial licence, so reimplement from the paper for commercial use.
• On the Effectiveness of Parameter-Efficient Fine-Tuning: argues that many PEFT methods are sparse fine-tuning, and proposes SAM, a mask from burn-in gradients; 77.12 average on seven RoBERTa-base tasks (author-reported). The repository has no LICENSE file, so nothing may be quoted from it.
• FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer: shares a Tensor-Train or Tucker factorisation of the weight change across all layers of a ViT; 0.069M parameters at parity with NOAH on VTAB-1K (author-reported). Its code is MIT-licensed.
• Pretrained Transformers as Universal Computation Engines: a frozen GPT-2 with trained input and output layers, positional embeddings and norms, applied to bit, image, ListOps and protein tasks (author-reported). Its code is MIT-licensed.
• Bi-directional Adapter for Multi-modal Tracking: two-stream adapters in a frozen OSTrack ViT for RGB-thermal tracking, 0.32M trained parameters (author-reported).
• ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation: LLaMA-Adapter fine-tuned on simulated articulated objects for contact-point prediction (author-reported). The released code has no licence.
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs: unlearning layers on a frozen T5 with a selective teacher-student KL objective, using 0.5% of parameters (author-reported). The pinned repository has no unlearning code.
• DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task: LoRA on ChatGLM-6B for Chinese medical dialogue, with a doctor-marked example evaluation rather than a benchmark (author-reported). The repository has no LICENSE file.
• A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems: a survey that describes adapters, LoRA and other adaptation methods for dialogue models; it reports no experiments of its own.
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning: trains d numbers in a fixed random Fastfood projection; RoBERTa-Large reaches 90% of MRPC full fine-tuning with about 200 values (author-reported). No official code.
• MaPLe: Multi-modal Prompt Learning: learned prompts in both CLIP encoders, linked by a learned coupling; 78.55 base-to-novel harmonic mean over 11 datasets (author-reported). Its code is MIT-licensed.
• T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models: small adapters that read a control map and steer a frozen Stable Diffusion UNet, composable by weighted sum (author-reported). The code repository at the recorded commit has no LICENSE file.
• Small Language Models are the Future of Agentic AI: a position paper that argues small models should handle most repetitive agent calls and gives a five-step conversion procedure; it has no experiments of its own.
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey: a survey of pre-train-then-fine-tune, prompting and generation methods for NLP, including adapters and prefix tuning; it runs no experiments of its own. No official code.
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models: compares prefix tuning, series and parallel adapters and LoRA on LLaMA, BLOOMz and GPT-J; the parallel adapter tops the LLaMA-13B commonsense average, and its math wins over GPT-3.5 rest on overlapping test sets (author-reported). Its code is Apache-2.0 licensed.
• ClipCap: CLIP Prefix for Image Captioning: maps a frozen CLIP embedding to GPT-2 prefix vectors, with an MLP mapper and GPT-2 tuned, or a transformer mapper and GPT-2 frozen (author-reported). Its code is MIT-licensed.
• Unified Camera Positional Encoding for Controlled Video Generation: a ray-based encoding and latitude-up map, injected through a zero-initialised parallel attention adapter on Wan2.1 T2V 1.3B (author-reported). Its code is MIT-licensed.
• P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks: trainable prefix vectors at every layer of a frozen model, plus a linear head, comparable to fine-tuning on SuperGLUE dev from BERT-large to a 10B GLM with 0.1% to 3% trainable parameters (author-reported). Its code is Apache-2.0 licensed.
• Deep transfer learning for image classification: a survey: a survey that lists parameter-efficient tuning among its parameter-based methods; its one experiment covers Big Transfer fine-tuning settings, with no parameter-efficient method tested.
• A Comprehensive Overview of Large Language Models: a survey of LLMs of 10B parameters or more that lists parameter-efficient fine-tuning among efficiency methods; it runs no experiments, and its numbers are copied from cited papers.