--- kind: concept slug: soft-prompt-tuning title: Soft prompt tuning summary: >- Replace hand-written prompt tokens with trainable continuous vectors, usually while the model stays frozen, and fit them by gradient descent. Variants differ in where the vectors go, what they are fitted on, and whether several prompts are mixed. Reported gains are author-reported, on different tasks, and not comparable. tags: [prompt-learning, continuous-prompts, knowledge-probing, few-shot-learning, test-time-adaptation, instruction-tuning, controllable-generation] papers: ["2103.10385", "2104.06599", "2107.13586", "2108.04106", "2109.04332", "2110.07602", "2201.05337", "2203.17274", "2205.14865", "2210.03117", "2212.13138", "2303.17569", "2304.01933", "2307.06948", "2308.06038", "2310.14735", "2312.03594"] related: [prompt-engineering, masked-language-modeling, few-shot-learning, in-context-learning, noisy-channel-model, test-time-training, parameter-efficient-fine-tuning, low-rank-adaptation, gradient-projection, diffusion-inpainting, vision-language-pretraining, instruction-tuning, chain-of-thought-prompting, self-consistency, visual-prompting, language-model-pretraining, sequence-labeling, extractive-question-answering, catastrophic-forgetting, consistency-regularization, controllable-text-generation] status: draft updated_at: 2026-11-01 --- ## Overview A pretrained model can be steered with a prompt, a short text that is part of its input. Soft prompt tuning keeps the model frozen and learns the prompt instead. The prompt is a sequence of continuous vectors that need not match any vocabulary entry, and gradient descent fits them to a training signal [arxiv:2104.06599 §3.1] [arxiv:2108.04106 §4.2.3]. The idea appears in seventeen papers here, for several jobs. For masked language models, the vectors are fitted to labelled (subject, object) pairs for a relation [arxiv:2104.06599 §3.1]. P-Tuning keeps a hand-written template and places trainable slots in it, linked by a small encoder [arxiv:2103.10385 §2.2] [§2.3]. For a frozen GPT-2 used as a classifier, they are prepended to the input and fitted to labelled examples [arxiv:2108.04106 §4.2.3]. For CLIP, the learned context is fitted per unlabelled test image [arxiv:2308.06038 §3.1], or used as a pair of judging prompts when training an image enhancer without paired data [arxiv:2303.17569 §3.1]. For a frozen CLIP or vision model, a single learned image frame added to every input can play the same role [arxiv:2203.17274 §3.2.1]. MaPLe learns prompts in both CLIP branches [arxiv:2210.03117 §3.2]. Med-PaLM puts one learned prefix in front of an ordinary written prompt for a 540B-parameter model [arxiv:2212.13138 §3.3.3]. For a diffusion inpainting model, learned task tokens let one model handle several jobs [arxiv:2312.03594 §3.2]. For a frozen T5 model, PPT first trains the vectors on unlabelled text with cloze tasks, then tunes them on each task [arxiv:2109.04332 §3.1]. P-Tuning v2 places trainable prefix vectors at every layer of a frozen model and adds a task head, and reports results on sequence labeling and extractive QA [arxiv:2110.07602 §3.2]. Each setting has its own training signal and its own reported numbers, so compare results only within a setting. A 2021 survey calls such vectors continuous prompts and lists fixed-LM prompt tuning, where the model is frozen, among five training strategies [arxiv:2107.13586 §4.3.2] [§7.2]. This page uses "soft prompt tuning" for the same idea. A variant that also tunes the model is the survey's prompt+LM tuning [arxiv:2107.13586 §7.2]; P-Tuning does this on SuperGLUE [arxiv:2103.10385 §3.2]. A 2023 review of prompt engineering describes CoOp, CoCoOp and MaPLe in the same terms and reports no new results of its own [arxiv:2310.14735 §4.3] [§4.4] [§4.5]. A 2022 survey of controllable text generation files prefix-tuning and P-tuning under prompt-based fine-tuning, next to adapters, reinforcement learning and instruction tuning. It describes the prefix as a small continuous block that is optimised while the model stays frozen [arxiv:2201.05337 §3.2]. ## Variants ### Input-layer token tuning Only the token vectors are trained. Each vector starts at the embedding of the corresponding hard-prompt token and is then tuned freely, so it can leave the set of real words [arxiv:2104.06599 §3.1]. This is the same as perturbing layer 0 only [arxiv:2104.06599 §3.2]. For a six-token prompt the search space is 6d, with d the vector size. In the paper's ablation on T-REx with BERT-large, tuning token vectors alone reaches 50.7 P@1 from a 39.4 start [arxiv:2104.06599 tab. 4]. ### All-layer perturbation Every layer receives a trainable additive perturbation to its token states, initialised to zero [arxiv:2104.06599 §3.2]. For a six-token prompt over L layers this gives 6d(L+1) parameters. On T-REx with BERT-large, tuning all layers reaches 51.6 P@1 and tuning layer 0 only reaches 51.0 [arxiv:2104.06599 tab. 5]. The main text describes the all-layer form as an add-on to the basic method, and the headline table follows the all-layer numbers [arxiv:2104.06599 §4.7] [arxiv:2104.06599 tab. 1]. See the contradictions section. ### Mixture of prompts with EM A relation can have several prompts. The predicted distribution over the answer is a weighted sum of the language model's probabilities under each prompt [arxiv:2104.06599 §3.3 / eq. 1]. The training objective is the negative log-likelihood of the correct answer under this mixture [arxiv:2104.06599 §3.5 / eq. 2]. The paper optimises it with EM: the E step computes a posterior over prompts for each pair and the M step does gradient steps on the prompt parameters [arxiv:2104.06599 §3.5]. Mixture weights alone add little in the ablation (40.0 P@1 against 39.4), so the gain comes mostly from the token vectors [arxiv:2104.06599 tab. 4]. The mixture is the setting behind the "Soft (min.)" rows in the main tables [arxiv:2104.06599 tab. 1] [arxiv:2104.06599 tab. 2]. ### Random instead of informed initialisation The paper also starts prompts from random vectors. It draws them from a Gaussian fitted to the language model's embeddings [arxiv:2104.06599 §4.4]. Random starts come out close to informed starts on T-REx: 51.9 against 51.6 with BERT-large [arxiv:2104.06599 tab. 1]. On ConceptNet the paper reports random initialisation as the best soft variant, at 14.5 P@1 with BERT-large against 0.1 for LAMA [arxiv:2104.06599 tab. 3]. ### Pre-trained prompts before task tuning (PPT) The prompt is first trained on unlabelled text with the model frozen, using self-supervised cloze tasks, and only then tuned on the labelled task [arxiv:2109.04332 §3.1]. The paper's three pre-training pairs are a sentence pair with three labels, a multiple-choice choice among six sentences, and a single-sentence sentiment-style task with pseudo-labels from a RoBERTa labeller [arxiv:2109.04332 §3.2] [arxiv:2109.04332 eq. 3-5]. The authors motivate it by the high variance of prompt tuning from random starts with 32 examples per task. On SST-2 with T5-XXL, vanilla prompt tuning scores 70.5 ± 15.5 against 93.5 ± 0.3 for PPT and 91.4 ± 0.8 for full fine-tuning [arxiv:2109.04332 tab. 4]. The gain depends on starting downstream tuning from the pre-trained prompt rather than from a random one [arxiv:2109.04332 §3.1] [arxiv:2109.04332 §4.2]. The unified variant converts every classification task to multiple choice, so one pre-trained prompt serves all tasks [arxiv:2109.04332 §3.3]. Pre-training is done once and its prompts are reused across tasks [arxiv:2109.04332 app. E]. ### Data-dependent mixture weights (not recommended) Section 3.4 describes weights that depend on the input through a learned temperature. The authors report little effect, because training pushed the temperature high enough to ignore the input [arxiv:2104.06599 §3.4] [arxiv:2104.06599 §4.7]. The released config switches this path off [code: config.yaml#L28]. ### Channel prompt tuning for few-shot classification A frozen GPT-2 Large is given 20 learned vectors in front of the input, and only those vectors are trained [arxiv:2108.04106 §4.2.3] [arxiv:2108.04106 tab. 4]. The score is the channel direction, P(input given label), rather than the direct P(label given input) [arxiv:2108.04106 §3]. On 11 text classification datasets with K=16 examples, the paper reports 61.7 average and 53.0 worst case for channel prompt tuning, against 48.4 and 29.5 for direct prompt tuning [arxiv:2108.04106 tab. 4] [arxiv:2108.04106 §6.2]. The vectors are a single prefix at the input, not a per-layer set, so this variant has the input-layer shape above with a different score. ### Test-time context tuning (CLIP) DiffTPT tunes four learned context tokens of CLIP's text prompt, once per unlabelled test image, and resets them for the next image [arxiv:2308.06038 §3.1] [arxiv:2308.06038 §4.1]. The loss is the entropy of the averaged prediction over selected augmented views [arxiv:2308.06038 eq. 4] [eq. 7]. The views are the original image, 64 crop and flip views, and 63 Stable Diffusion variations. The paper keeps the 80% of generated views most similar to the original, then the 30% lowest-entropy views [arxiv:2308.06038 §3.4] [arxiv:2308.06038 §4.1]. On the ImageNet variants, the ViT-B/16 average is 62.28 against 61.59 for the earlier TPT method, and the RN50 average is 48.71 against 46.65 [arxiv:2308.06038 tab. 1]. The paper's headline gain is discussed under Contradictions. ### Additive image frame on a frozen model (visual prompting) The learned vectors are not text tokens here. The learned object is one image-sized tensor that is added to every input, while the model stays frozen [arxiv:2203.17274 §3.2.1]. Only the frame is trained, so the method needs access to the input and not to the model's weights or features. The default template is a padding frame of width 30 on 224×224 inputs, with parameters only in the border [arxiv:2203.17274 §3.2.2] [§6.2]. Training uses SGD at learning rate 40 for 1,000 epochs with cosine decay [arxiv:2203.17274 §3.4]. On CLIP ViT-B/32 the 12-dataset average rises from 54.2 with the text prompt alone to 78.2 with the frame, against 83.1 for a linear probe and 86.9 for full fine-tuning [arxiv:2203.17274 tab. 1]. The gain is uneven. The frame beats the linear probe on EuroSAT, SVHN and CLEVR and trails it on the other nine datasets [arxiv:2203.17274 tab. 1]. On the three WILDS sets with CLIP, the average is 58.1 against 62.6 for the linear probe, while Camelyon17 is 89.8 against 84.9 [arxiv:2203.17274 tab. 2]. On vision-only backbones (Instagram ResNeXt, BiT-M and RN50) the 12-dataset averages are 32.4, 32.6 and 25.7, against 71.7, 72.6 and 72.5 for linear probes [arxiv:2203.17274 tab. 4]. In the template ablation on EuroSAT with frozen CLIP, fixed-location templates beat random-location ones, and padding with p = 30 is best overall [arxiv:2203.17274 §6.2] [fig. 6]. ### Learned task tokens (inpainting) PowerPaint trains one Stable Diffusion inpainting model with three learned prompts: one for context filling, one for text-guided object insertion and one for shape-guided insertion [arxiv:2312.03594 §3.2]. The context prompt starts from a random token array and is learned by minimising the diffusion loss [arxiv:2312.03594 §3.2] [eq. 4]. The number of vectors per prompt is not stated in the paper. Replacing the learned prompts with rare tokens that have no learned meaning makes every reported column worse, including Local-FID 13.03 against 11.61 and Context FID 3.11 against 2.59 [arxiv:2312.03594 tab. 6]. For object removal, the object prompt is used as the negative prompt in classifier-free guidance [arxiv:2312.03594 §3.2] [eq. 5]. ### Learned prompt pair as a judge (image enhancement) CLIP-LIT learns a positive and a negative prompt, each a block of 16 learnable token embeddings of width 512 in CLIP's text encoder [arxiv:2303.17569 §3.1]. The prompts are fitted with a binary loss to separate backlit and well-lit images, then refined with a ranking loss while the enhancer is trained against them [arxiv:2303.17569 §3.1] [arxiv:2303.17569 §3.2]. The learned prompts are not used to produce text. They score the enhancer's output. On the BAID test set, the paper reports PSNR 21.579 against 19.740 for Zero-DCE, the best released baseline in its table [arxiv:2303.17569 tab. 1]. ### Encoder-linked slots in a hand-written template (P-Tuning) P-Tuning keeps the written words of a template and adds continuous slots before, between and after the input and the label [arxiv:2103.10385 §2.2]. The slot vectors go through a small prompt encoder, an LSTM or an MLP, so that the slots depend on each other [arxiv:2103.10385 §2.3]. With a frozen model on LAMA, the paper reports LAMA-34k P@1 of 48.3 for BERT-base, against 43.3 for AutoPrompt on the same model size [arxiv:2103.10385 tab. 3]. On few-shot SuperGLUE with ALBERT-xxlarge-v2, the four-run average is 71.81 against 70.74 for PET-FT. The gap is about one point, inside the reported standard deviations on several tasks [arxiv:2103.10385 tab. 5] [§3.3]. On LAMA-P17, the spread across six manual prompts falls from 10.1 to 0.46. That is one relation, so it does not show the same for others [arxiv:2103.10385 tab. 6]. ### Multimodal prompts in both encoders (MaPLe) MaPLe adds learned tokens to CLIP's text branch and to its image branch. Each vision prompt is a linear function of the text prompt in the same layer [arxiv:2210.03117 §3.2.3]. Prompts go into the first nine blocks for the base-to-novel setting [arxiv:2210.03117 §4.1]. With 16 shots per class, the average harmonic mean over base and novel classes is 78.55 across 11 datasets, against 75.83 for Co-CoOp [arxiv:2210.03117 tab. 3]. Novel-class accuracy is 75.14, against 63.22 for text-only CoOp and 71.69 for Co-CoOp [arxiv:2210.03117 tab. 3]. In the paper's ablation, independent vision and language prompts give a harmonic mean of 77.90, so the coupling adds 0.65 points [arxiv:2210.03117 tab. 1]. The trainable parameters total 3.55M, which is 2.85% of CLIP [arxiv:2210.03117 tab. 6]. ### Regularised deep prompts with epoch averaging (PromptSRC) PromptSRC adds deep vision and text prompts to a frozen CLIP, as MaPLe does [arxiv:2307.06948 §3.1]. In the base-to-novel and few-shot settings, the prompts go into the first nine transformer layers, with four tokens per branch [arxiv:2307.06948 §4.1]. The change is in the loss. Besides cross-entropy, it adds an L1 distance from the frozen image and text features, and a KL term that pulls the prompted class distribution toward the frozen one [arxiv:2307.06948 §3.2.1] [eq. 2] [eq. 3]. The text target for each class is the mean over 60 prompt templates and is used only in training [arxiv:2307.06948 §3.2.3]. The feature weights are 10 for images and 25 for text [arxiv:2307.06948 §4.1]. After training, prompts from several epochs are averaged with Gaussian weights and the average is used for inference [arxiv:2307.06948 §3.2.2]. The author-reported base-to-novel harmonic mean over 11 datasets is 79.97, against 78.55 for MaPLe and 71.70 for CLIP [arxiv:2307.06948 tab. 1]. Novel-class accuracy is 76.10, against 75.14 for MaPLe [arxiv:2307.06948 tab. 1]. In the ablation, the agreement loss alone takes the IVLP baseline from 77.51 to 79.55, which accounts for most of the gain. Epoch averaging and the template ensemble add 0.15 and 0.27 [arxiv:2307.06948 tab. 2]. The paper reports 46K tunable parameters, against 3.55M for MaPLe [arxiv:2307.06948 §4.3]. The released code differs from the text in several places, listed on the card [Self-regulating Prompts](/papers/2307.06948). ### Gradient-projected context tuning (ProGrad) ProGrad learns CoOp-style context vectors for a frozen CLIP, with one change. The cross-entropy gradient is projected so that it never opposes the gradient of a term that keeps predictions near zero-shot CLIP [arxiv:2205.14865 §3.2] [eq. 4]. With λ = 0 the method reduces to plain context tuning [arxiv:2205.14865 §3.2]. The motivation is that such tuning can fall below zero-shot CLIP with one or two shots [arxiv:2205.14865 §1] [fig. 1]. With ResNet-50 on 11 datasets, the author-reported average is 62.61% at 1 shot against 59.46% for CoOp, and 74.28% against 73.49% at 16 shots [arxiv:2205.14865 tab. 1]. The projection can only protect what the zero-shot prompt already gets right, because many of the method's failures are also misclassified by zero-shot CLIP [arxiv:2205.14865 §4] [fig. 5]. ### Per-layer prefix keys and values (prefix-tuning) Prefix-tuning leaves the input embeddings alone. It prepends learned key and value vectors inside each attention layer, and the base model stays frozen [arxiv:2304.01933 §2] [eq. 1]. LLM-Adapters tests it as one of four adapter families, next to series and parallel adapters and LoRA [arxiv:2304.01933 §3.3]. The paper uses 10 virtual tokens, the value it chose from a sweep over 10 to 40 [arxiv:2304.01933 §4.1] [app. A.4]. In that comparison prefix-tuning has the lowest math average on all four base models. On LLaMA-13B it reaches 46.5 over six math sets, against 65.4 for LoRA [arxiv:2304.01933 tab. 3]. On the eight commonsense tasks it reaches 68.4, against 80.5 for LoRA [arxiv:2304.01933 tab. 4]. On LLaMA-7B the sweep is not monotonic: 42.0 at 10 tokens, 38.0 at 20, 33.1 at 30 and 38.1 at 40 [arxiv:2304.01933 app. A.4]. All numbers are the authors' own. Three of the six math sets overlap the training data, according to the repository [code: README.md#L48], so the math averages are weaker evidence than the commonsense ones. ### Per-layer prompts with a task head (P-Tuning v2) P-Tuning v2 gives every layer of a frozen backbone its own trainable prefix vectors, read as keys and values, and adds a trainable head. It follows deep prompt tuning [arxiv:2110.07602 §3.2]. The paper contrasts this with input-layer prompting, where vectors are trained only at the input. That baseline performs poorly on hard sequence labeling and extractive QA [arxiv:2110.07602 §1] [arxiv:2110.07602 §2]. The head is a linear layer on the first-token representation for classification [arxiv:2110.07602 §3.3]. For sequence labeling it is a per-token linear classifier [arxiv:2110.07602 app. A]. The reparameterising MLP is optional, and the paper says its value depends on the task [arxiv:2110.07602 §3.3]. Prefix length depends on the task. Simple classification usually prefers fewer than 20 positions, and hard sequence labeling around 100 [arxiv:2110.07602 §3.3] [fig. 4]. Adding the same number of prompts to deeper layers beats adding them to shallower ones, on RTE and BoolQ with BERT-large [arxiv:2110.07602 §4.3] [fig. 3]. On SuperGLUE dev, the paper reports results comparable to fine-tuning from BERT-large to a 10B model, with 0.1% to 3% of parameters trained [arxiv:2110.07602 §1] [arxiv:2110.07602 tab. 2]. On RTE, the same method beats full fine-tuning with BERT-large (78.3 against 70.4) and RoBERTa-large (89.5 against 86.6) [arxiv:2110.07602 tab. 2]. On sequence labeling the gap to fine-tuning is small with RoBERTa-large: CoNLL03 is 92.8 against 92.6 [arxiv:2110.07602 tab. 3]. The results are fully supervised, not few-shot [arxiv:2110.07602 §4]. The paper's dev numbers are discussed under Contradictions. ### Soft prefix before a hand-written prompt (instruction prompt tuning) Med-PaLM uses one shared soft prefix of 100 vectors, 1.84M parameters in all, placed before a written instruction, few-shot exemplars and the question [arxiv:2212.13138 §3.3.3] [app. A.1]. Only the prefix is trained, on 40 clinician-written exemplars [arxiv:2212.13138 §3.3.4]. The checkpoint is chosen by clinician ranking rather than by a validation metric [arxiv:2212.13138 app. A.1]. On long-form answers, the paper reports 92.9% of Med-PaLM answers aligned with scientific consensus against 61.9% for Flan-PaLM [arxiv:2212.13138 §4.5]. The human-evaluation table disagrees with the text on this axis (see Contradictions). The rated set is 140 questions with one rater per answer, so the numbers are thin [arxiv:2212.13138 §4.5]. Med-PaLM is worse than Flan-PaLM on inappropriate or incorrect content, 18.7% against 16.1% [arxiv:2212.13138 app. A.7]. ## When to use which | Situation | Use | Why | |---|---|---| | Labelled (subject, object) pairs for the relation and white-box access to the model | Soft prompt tuning | Reported T-REx P@1 with BERT-large rises from 39.4 (LPAQA) to 51.6 [arxiv:2104.06599 tab. 1] | | No labelled pairs for the relation | Not covered by this paper | Training and early stopping both use a labelled dev set [arxiv:2104.06599 §4.3] [arxiv:2104.06599 §4.5] | | Need a prompt a person can read, audit or paste elsewhere | Discrete prompts (hand-written, mined or searched) | Learned vectors are not interpretable; nearest-word projections form no phrases [arxiv:2104.06599 §5] [arxiv:2104.06599 app. C] | | No good hand-written prompt to start from | Random initialisation | Close to informed starts on T-REx and best on ConceptNet [arxiv:2104.06599 tab. 1] [arxiv:2104.06599 tab. 3] | | Tight parameter or memory budget per relation | Input-layer tuning | 6d parameters against 6d(L+1); layer 0 costs about 0.6 P@1 in the BERT-large example [arxiv:2104.06599 §3.2] [arxiv:2104.06599 tab. 5] | | Best reported accuracy on the paper's benchmarks | All-layer perturbation in a mixture | Headline numbers use this setting [arxiv:2104.06599 tab. 1] [arxiv:2104.06599 tab. 5] | | Need results from BART | Not reproducible from the release | The paper reports BART, but the released code raises an error for it [arxiv:2104.06599 tab. 5] [code: soft_prompts/lm/construct_lm.py#L8-L9] | | Model family other than BERT or RoBERTa | Measure it yourself | Reported results cover BERT and RoBERTa only [arxiv:2104.06599 tab. 1] [arxiv:2104.06599 tab. 2] [arxiv:2104.06599 tab. 3] | | A hand-written pattern exists and must stay in the input, with the frozen model's knowledge probed through it | P-Tuning | Adds learned slots around the written words; LAMA-34k 48.3 against 43.3 for AutoPrompt at BERT-base [arxiv:2103.10385 tab. 3] | | Few labelled examples for a frozen CLIP, and test classes unseen in training | MaPLe | Novel-class accuracy 75.14 against 71.69 for Co-CoOp and 63.22 for text-only CoOp [arxiv:2210.03117 tab. 3]; adds 3.55M parameters [arxiv:2210.03117 tab. 6] | | Frozen CLIP trained on labelled base classes, where accuracy on unseen classes must not fall and the parameter budget is small | PromptSRC | Author-reported base-to-novel HM 79.97 against 78.55 for MaPLe [arxiv:2307.06948 tab. 1]; 46K parameters against 3.55M [arxiv:2307.06948 §4.3] | | A large frozen LLM, about 40 expert exemplars, and no budget to train the full model | Instruction prompt tuning | Trains 1.84M parameters on 40 exemplars [arxiv:2212.13138 §3.3.4] [app. A.1]; long answers still fall short of experts on several axes [arxiv:2212.13138 §4.5] | | Learning the vocabulary of prompt-based learning before choosing a method | Read the survey | No numbers of its own; defines the design axes and five training strategies [arxiv:2107.13586 §2.3] [§7.2] | | About 32 labelled examples per task on a large frozen T5, and unlabelled text available for a pre-training run | Pre-trained prompt tuning (PPT) | Author-reported SST-2 at 93.5 against 70.5 for prompt tuning from random start with the same examples [arxiv:2109.04332 tab. 4]; pre-training runs up to 200,000 steps [arxiv:2109.04332 app. C] | | Many small classification tasks sharing one frozen model | Unified PPT | One pre-trained prompt for all tasks; reported TNews at 50.6 against 41.2 for prompt tuning with 8 examples per label [arxiv:2109.04332 tab. 5] | | Full labelled training set | Any method on this page | Differences are small: SST-2 scores are 96.1 (FT), 96.8 (PT) and 96.9 (PPT) [arxiv:2109.04332 tab. 6] | | Few labelled examples (K up to 16) for a frozen LM classifier, with imbalanced or unseen labels | Channel prompt tuning | Channel scoring beats direct prompt tuning on average and worst case at K=16 [arxiv:2108.04106 tab. 4] [arxiv:2108.04106 §6.2] | | Many examples, or tasks like TREC and Subj | Direct head tuning | The paper reports direct head tuning ahead on these [arxiv:2108.04106 §6.2] [arxiv:2108.04106 §7] | | Frozen CLIP, no labels, one test image at a time, and a diffusion model is available | Test-time context tuning | Tunes four context tokens per image without labels [arxiv:2308.06038 §4.1]; needs the view generation of [arxiv:2308.06038 §3.3] | | One generative model that must add objects and also remove them | Learned task tokens | The paper trains one model with separate learned prompts and reports they beat unlearnable identifiers [arxiv:2312.03594 §3.2] [arxiv:2312.03594 tab. 6] | | Training a model with only unpaired image sets and a frozen CLIP | Learned prompt pair as a judge | The prompt pair scores the output without paired ground truth [arxiv:2303.17569 §3.1] [arxiv:2303.17569 §4] | | Frozen CLIP, labelled images for the task, and access only to the input | Additive image frame (visual prompting) | Author-reported 12-dataset average 78.2 against 54.2 for the text prompt, with one frame per dataset [arxiv:2203.17274 tab. 1] [arxiv:2203.17274 §3.2.1]; it still trails the linear probe on most datasets [arxiv:2203.17274 tab. 1] | | Frozen vision-only classifier (Instagram, BiT-M or RN50) and a labelled task | Linear probe rather than an image frame | The frame reaches 25.7 to 32.6 on the 12-dataset average, against 72.5 to 72.6 for linear probes on those backbones [arxiv:2203.17274 tab. 4]; the paper's results favour the probe here | | Frozen CLIP with few labelled examples, where the zero-shot prompt is a fair teacher | Gradient-projected context tuning (ProGrad) | Author-reported 1-shot average 62.61 against 59.46 for CoOp [arxiv:2205.14865 tab. 1]; gains depend on the zero-shot prompt being good [arxiv:2205.14865 §4] [fig. 5] | | An LLM where a per-layer prefix is one candidate among adapters | Prefer LoRA or an adapter over prefix-tuning | Prefix-tuning has the lowest math average on all four base models [arxiv:2304.01933 tab. 3] and the lowest commonsense average on BLOOMz and both LLaMA sizes [arxiv:2304.01933 tab. 4] | | Frozen encoder for sequence labeling or extractive QA, with labelled data and a task head allowed | P-Tuning v2 | Author-reported CoNLL03 with RoBERTa-large is 92.8, against 86.1 for input-layer prompt tuning [arxiv:2110.07602 tab. 3] | | Frozen model, many tasks, and a small stored file per task instead of a model copy | P-Tuning v2 | Trains 0.1% to 3% of parameters [arxiv:2110.07602 §1]; the released scripts train one dataset per run [code: run_script/run_rte_roberta.sh#L5-L9] | | Few labelled examples, or the task must stay in written words | Not covered by P-Tuning v2 | Experiments are fully supervised [arxiv:2110.07602 §4]; P-Tuning v1 keeps the written template [arxiv:2103.10385 §2.2] | Compute: the paper reports that one prompt trains in minutes even for BERT-large, and mixture training grows roughly linearly with the number of prompts [arxiv:2104.06599 §4.5]. MaPLe reports 5 training epochs where Co-CoOp needs 10 [arxiv:2210.03117 §4.6]. ## Implementation sketch The steps below follow the paper's method; they are not the released code. Each step is cited to the section that defines it. ```pseudocode # Build per-layer soft prompt parameters [arxiv:2104.06599 §3.1] [§3.2] # Input: hard prompt tokens; LM depth L; vector size d # Output: token vectors (trainable) and zero perturbations per layer def make_soft_prompt(hard_tokens, num_layers): # Start each token vector at its embedding [§3.1] vectors = embed(hard_tokens) # (T, d) # One zero-initialised perturbation per layer, 0..L [§3.2] deltas = zeros(num_layers + 1, len(hard_tokens), d) # (L+1, T, d) return vectors, deltas ``` ```pseudocode # Mixture prediction for one candidate answer [arxiv:2104.06599 §3.3] [eq. 1] # Input: prompts with mixture weights; input x; candidate answer y def mixture_probability(prompts, weights, x, y): # Weighted sum of the LM probability of y under each prompt [eq. 1] return sum(w * lm_probability(p, x, y) for p, w in zip(prompts, weights)) ``` Notes for implementers, each from the paper: - Optimiser: Adam with default settings. The learning rate is not stated [arxiv:2104.06599 §4.5]. - Gradient batch size 64, at most 16 epochs, early stopping with patience 4, and keep the checkpoint with the best dev P@1 [arxiv:2104.06599 §4.5]. - Answers are single tokens, so predictions are the most probable answer under the mixture [arxiv:2104.06599 §4.2] [arxiv:2104.06599 §4.6]. - The paper states no regulariser on perturbation size beyond early stopping, and the released config applies no penalty [arxiv:2104.06599 §3.2] [code: config.yaml#L24]. - The release differs from the text in batch size, layers tuned, and the weight update. The card's "Paper vs code" section lists each difference: [Learning How to Ask](/papers/2104.06599). - Channel prompt tuning uses 20 vectors and trains with the length-normalised channel loss [arxiv:2108.04106 §4] [arxiv:2108.04106 §4.2.3]. The released code drops the GPT-2 BOS token to make room for the vectors, which the paper does not mention [code: util.py#L210-L227]. - P-Tuning slots are passed through an encoder before they replace the slot embeddings [arxiv:2103.10385 §2.3]. The LAMA code uses a two-layer bidirectional LSTM and an MLP [code: LAMA/p_tuning/prompt_encoder.py#L26-L40]. The paper's manual prompt is not inserted by default in that code [code: LAMA/p_tuning/modeling.py#L74-L105], so add it yourself and say so. - Initialisation matters in low-data settings, according to prefix-tuning results the survey cites, and starting from a discrete prompt can help [arxiv:2107.13586 §4.3.2]. - MaPLe's text says sharing deep prompts across layers is better, but its equations and code keep one prompt tensor and one coupling layer per layer [arxiv:2210.03117 §3.2.2] [code: trainers/maple.py#L111-L117]. - Med-PaLM's prefix is uniform over [-0.5, 0.5], trained with AdamW at learning rate 0.003, weight decay 0.00001 and batch size 32 for 200 steps [arxiv:2212.13138 app. A.1]. The loss and the sampling scheme are not stated. - ProGrad's λ is 0.8 at 16 shots in the paper, and 1.0 everywhere in the released scripts [arxiv:2205.14865 §4] [code: ProGrad.public/scripts/prograd.sh#L15]. The released trainer builds its teacher from a single template, so ProGrad++ cannot be rerun from that release [code: ProGrad.public/trainers/prograd.py#L220-L249]. - ProGrad's optimiser is SGD with initial learning rate 0.002, a cosine schedule and batch size 32 [arxiv:2205.14865 app. A2]. The paper does not state momentum or weight decay [arxiv:2205.14865 app. A2]. - Prefix-tuning uses learning rate 3e-2, against 3e-4 for the other families, with batch size 16 for three epochs [arxiv:2304.01933 §3.3]. The released prefix command uses batch size 8 and five epochs [code: math_running_commands#L5]. The code defaults to 30 virtual tokens, not the paper's 10 [code: finetune.py#L60] [arxiv:2304.01933 §4.1]. - P-Tuning v2 tunes the learning rate and prefix length per task. The released scripts range from 1e-4 to 6e-2 for the rate and from 8 to 224 positions for the prefix [code: run_script/run_wic_bert.sh#L6] [code: run_script/run_conll05_roberta.sh#L8]. The optimiser, weight decay and warmup are not stated in the paper, and no script sets them [code: run_script/run_rte_roberta.sh#L5-L9]. The code uses an exponential learning-rate schedule for the sequence-labeling and QA runs [code: training/trainer_exp.py#L42-L45]. - P-Tuning v2 has no reference code for the multi-task MPT-2 rows in the paper [arxiv:2110.07602 tab. 3]. Train one dataset per run. - The image-frame code is a CIFAR-100-only release, so it cannot rerun the 12-dataset table [arxiv:2203.17274 tab. 6] [code: main_clip.py#L144-L148]. Its CLIP script fixes the text template to "This is a photo of a {}" for every run, while the paper uses per-dataset templates [code: main_clip.py#L141] [arxiv:2203.17274 tab. 6]. - The code validates on the CIFAR-100 test split after every epoch and keeps the best checkpoint [code: main_clip.py#L147-L148] [code: main_clip.py#L202-L213]. Any number taken from that checkpoint was selected on test data, and the paper does not say which checkpoint it reports. - The vision script maps class i to output i, not the semantic mapping that the paper's dog-and-cat test relies on [code: main_vision.py#L181] [arxiv:2203.17274 §6.3]. Use a semantic mapping if you adapt it. - The frame's initialisation is not stated in the paper. The code draws standard-normal values in the normalised input space [code: models/prompters.py#L13-L16]. ## Contradictions - **Layers tuned in the headline result.** §4.7 presents tuning layer 0 as the basic method and all layers as an add-on [arxiv:2104.06599 §4.7]. Table 1 matches the all-layer column of Table 5 [arxiv:2104.06599 tab. 1] [arxiv:2104.06599 tab. 5]. - **Same method, different numbers.** Table 4 reports 51.0 P@1 with both token vectors and perturbations adjusted, and 50.7 with token vectors alone [arxiv:2104.06599 tab. 4]. Table 5 reports 51.0 for layer 0 only and 51.6 for all layers [arxiv:2104.06599 tab. 5], and Table 1 uses 51.6 [arxiv:2104.06599 tab. 1]. The tables do not state that these rows share a configuration, so they are not reconciled here. - **Batch size.** The paper states 64 for gradient training. The released config uses 32 [arxiv:2104.06599 §4.5] [code: config.yaml#L5]. - **Headline gain of test-time context tuning.** The paper states an average gain of 5.13% over TPT [arxiv:2308.06038 §4.2]. The averages printed in its tables give smaller gains, from 0.7 to 2.9 points on the averages the tables print [arxiv:2308.06038 tab. 1] [arxiv:2308.06038 tab. 2]. The paper does not say how 5.13% is computed, so this page uses the table values. - **Printed average in DiffTPT.** The RN50 average in tab. 1 is printed as 42.26, while its five values average to 44.26. The OOD average does match [arxiv:2308.06038 tab. 1]. Recompute averages before comparing. - **Headline LAMA gain of P-Tuning.** The text reports 43.3 to 50.6 [arxiv:2103.10385 §3.1]. That pairs AutoPrompt on BERT-base with P-Tuning on BERT-large. The same-size comparison is 43.3 against 48.3 [arxiv:2103.10385 tab. 3]. - **GPT results, text against table.** The paper says P-Tuning "consistently achieves the best performance" on the GPT-2 models [arxiv:2103.10385 §3.2]. Table 4 shows PET-FT higher on several GPT-2 columns [arxiv:2103.10385 tab. 4]. - **MaPLe learning rate.** Section 4.1 and the code give 0.0035 for base-to-novel. Appendix A gives 0.0025 [arxiv:2210.03117 §4.1] [arxiv:2210.03117 app. A]. The code is the reference used here. - **Med-PaLM aligned share.** The text gives 92.9% aligned for Med-PaLM [arxiv:2212.13138 §4.5]. The appendix table gives 92.6% in the row labelled "Aligned", and the same values appear in the row labelled "No consensus" [arxiv:2212.13138 app. A.7]. Check the source table before citing either. - **P-Tuning v2 text against table.** The paper says P-Tuning v2 is comparable to fine-tuning at every scale tested [arxiv:2110.07602 §4.1]. The table shows some cases where fine-tuning is higher, such as WiC with RoBERTa-large (75.6 against 73.4) and WSC with GLM-xxlarge (95.2 against 93.3) [arxiv:2110.07602 tab. 2]. - **P-Tuning v2 released numbers.** The README's RoBERTa-large row gives RTE 86.6 and COPA 92.0, against 89.5 and 93.0 in table 2 [code: README.md#L85] [arxiv:2110.07602 tab. 2]. The README gives 85.3 for CoNLL12 with BERT-large, where table 3 gives 83.2 [code: README.md#L77] [arxiv:2110.07602 tab. 3]. The README says reproduction may need tuning for the local environment [code: README.md#L32-L34]. - **P-Tuning v2 head.** The paper describes a linear head on the first-token position [arxiv:2110.07602 §3.3]. The released classifier reads the backbone's pooler output, which is frozen with the rest of the backbone [code: model/sequence_classification.py#L404-L407]. - **P-Tuning v2 hidden hyperparameters.** The LaTeX source holds a per-task table that disagrees with the run scripts. CoNLL03 with RoBERTa-large has prefix length 112 in the hidden table and 11 in the script [code: run_script/run_conll03_roberta.sh#L7]. SQuAD 1.1 with RoBERTa-large has batch size 16 in the hidden table and 8 in the script [code: run_script/run_squad_roberta.sh#L5]. The paper does not say which values produced its numbers. - **P-Tuning v2 dev numbers.** SuperGLUE and SQuAD numbers are dev-set results. The base trainer keeps the best dev score over epochs [code: training/trainer_base.py#L52-L54], and the paper does not say how its reported number was chosen [arxiv:2110.07602 tab. 2]. - **PromptSRC prompt depth.** Section 3.1 says the prompts go into every transformer block. Section 4.1 says the first nine layers for base-to-novel and few-shot, and the first three for cross-dataset and domain generalisation [arxiv:2307.06948 §3.1] [arxiv:2307.06948 §4.1]. - **PromptSRC ablation text against table.** Section 4.2 says the agreement loss raises novel accuracy by 3.95 points. Table 2 gives 75.38 minus 71.79, which is 3.59 [arxiv:2307.06948 §4.2] [arxiv:2307.06948 tab. 2]. - **PPT claims against its own table.** The paper says PPT beats fine-tuning on all Chinese datasets and most English ones [arxiv:2109.04332 §4.2]. In table 4, PPT with T5-XXL scores 66.43 on BoolQ and 71.2 F1 on CB, against 80.8 and 86.5 for fine-tuned T5-XXL [arxiv:2109.04332 tab. 4]. The summary above uses the table values. - **Pre-training cost.** The paper's pre-training uses about 10 GB of text per format and up to 200,000 steps, but no wall-clock time is stated [arxiv:2109.04332 app. C]. The PPT rows in tables 4 and 5 use prompts pre-trained this way [arxiv:2109.04332 tab. 4] [arxiv:2109.04332 tab. 5]. ## Papers - [Exploring Visual Prompts for Adapting Large-Scale Models](/papers/2203.17274): learns one additive image frame for a frozen CLIP or vision model and adds it to every test input. - [GPT Understands, Too](/papers/2103.10385): P-Tuning, trainable prompt vectors with an LSTM or MLP encoder inside a hand-written template. - [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](/papers/2104.06599): introduces soft prompts with per-layer perturbations and an EM-trained mixture for cloze-style knowledge probing. - [Survey of controllable generation with pre-trained transformers](/papers/2201.05337): a survey that places prefix-tuning and P-tuning under prompt-based fine-tuning for controllable generation; it reports no experiments of its own. - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](/papers/2107.13586): a survey defining prompt-based learning, continuous prompts and the training strategies used here. - [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106): scores labels by P(input given label) and tunes 20 prompt vectors on a frozen GPT-2 Large for few-shot classification. - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](/papers/2109.04332): pre-trains soft prompts on unlabelled text with three cloze tasks, then tunes them per task on a frozen T5 with 32 examples. - [Prompt-aligned Gradient for Prompt Tuning](/papers/2205.14865): ProGrad, which removes from the CLIP prompt-tuning gradient the part that opposes a zero-shot KL term. - [MaPLe: Multi-modal Prompt Learning](/papers/2210.03117): learns coupled prompts in both the text and vision branches of a frozen CLIP. - [Large Language Models Encode Clinical Knowledge](/papers/2212.13138): Med-PaLM, a 100-token learned prefix on Flan-PaLM 540B, with clinician evaluation of long answers. - [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](/papers/2110.07602): trainable prefix vectors at every layer of a frozen model with a task head, evaluated on SuperGLUE, sequence labeling and extractive QA. - [Iterative Prompt Learning for Unsupervised Backlit Image Enhancement](/papers/2303.17569): learns a positive and negative CLIP prompt to judge enhanced images, without paired training data. - [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](/papers/2304.01933): compares prefix-tuning with series and parallel adapters and LoRA on four open LLMs. - [Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning](/papers/2308.06038): tunes four CLIP context tokens per test image using augmented and diffusion-generated views. - [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](/papers/2312.03594): trains one Stable Diffusion inpainting model with three learned task prompts. - [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](/papers/2307.06948): PromptSRC, deep CLIP prompts with feature and logit agreement losses and epoch-level prompt averaging. - [Unleashing the potential of prompt engineering for large language models](/papers/2310.14735): a 2023 survey of prompt engineering, including CoOp, CoCoOp and MaPLe; it reports no new results.