Soft prompt tuning

Overview

A pretrained model can be steered with a prompt, a short text that is part of its input. Soft prompt tuning keeps the model frozen and learns the prompt instead. The prompt is a sequence of continuous vectors that need not match any vocabulary entry, and gradient descent fits them to a training signal 2104.06599 §3.1 2108.04106 §4.2.3.

The idea appears in seventeen papers here, for several jobs. For masked language models, the vectors are fitted to labelled (subject, object) pairs for a relation 2104.06599 §3.1. P-Tuning keeps a hand-written template and places trainable slots in it, linked by a small encoder 2103.10385 §2.2 §2.3. For a frozen GPT-2 used as a classifier, they are prepended to the input and fitted to labelled examples 2108.04106 §4.2.3. For CLIP, the learned context is fitted per unlabelled test image 2308.06038 §3.1, or used as a pair of judging prompts when training an image enhancer without paired data 2303.17569 §3.1. For a frozen CLIP or vision model, a single learned image frame added to every input can play the same role 2203.17274 §3.2.1. MaPLe learns prompts in both CLIP branches 2210.03117 §3.2. Med-PaLM puts one learned prefix in front of an ordinary written prompt for a 540B-parameter model 2212.13138 §3.3.3. For a diffusion inpainting model, learned task tokens let one model handle several jobs 2312.03594 §3.2. For a frozen T5 model, PPT first trains the vectors on unlabelled text with cloze tasks, then tunes them on each task 2109.04332 §3.1. P-Tuning v2 places trainable prefix vectors at every layer of a frozen model and adds a task head, and reports results on sequence labeling and extractive QA 2110.07602 §3.2. Each setting has its own training signal and its own reported numbers, so compare results only within a setting.

A 2021 survey calls such vectors continuous prompts and lists fixed-LM prompt tuning, where the model is frozen, among five training strategies 2107.13586 §4.3.2 §7.2. This page uses "soft prompt tuning" for the same idea. A variant that also tunes the model is the survey's prompt+LM tuning 2107.13586 §7.2; P-Tuning does this on SuperGLUE 2103.10385 §3.2. A 2023 review of prompt engineering describes CoOp, CoCoOp and MaPLe in the same terms and reports no new results of its own 2310.14735 §4.3 §4.4 §4.5. A 2022 survey of controllable text generation files prefix-tuning and P-tuning under prompt-based fine-tuning, next to adapters, reinforcement learning and instruction tuning. It describes the prefix as a small continuous block that is optimised while the model stays frozen 2201.05337 §3.2.

Variants

Input-layer token tuning

Only the token vectors are trained. Each vector starts at the embedding of the corresponding hard-prompt token and is then tuned freely, so it can leave the set of real words 2104.06599 §3.1. This is the same as perturbing layer 0 only 2104.06599 §3.2. For a six-token prompt the search space is 6d, with d the vector size. In the paper's ablation on T-REx with BERT-large, tuning token vectors alone reaches 50.7 P@1 from a 39.4 start 2104.06599 tab. 4.

All-layer perturbation

Every layer receives a trainable additive perturbation to its token states, initialised to zero 2104.06599 §3.2. For a six-token prompt over L layers this gives 6d(L+1) parameters. On T-REx with BERT-large, tuning all layers reaches 51.6 P@1 and tuning layer 0 only reaches 51.0 2104.06599 tab. 5. The main text describes the all-layer form as an add-on to the basic method, and the headline table follows the all-layer numbers 2104.06599 §4.7 2104.06599 tab. 1. See the contradictions section.

Mixture of prompts with EM

A relation can have several prompts. The predicted distribution over the answer is a weighted sum of the language model's probabilities under each prompt 2104.06599 §3.3 / eq. 1. The training objective is the negative log-likelihood of the correct answer under this mixture [arxiv:2104.06599 §3.5 / eq. 2]. The paper optimises it with EM: the E step computes a posterior over prompts for each pair and the M step does gradient steps on the prompt parameters 2104.06599 §3.5. Mixture weights alone add little in the ablation (40.0 P@1 against 39.4), so the gain comes mostly from the token vectors 2104.06599 tab. 4. The mixture is the setting behind the "Soft (min.)" rows in the main tables 2104.06599 tab. 1 2104.06599 tab. 2.

Random instead of informed initialisation

The paper also starts prompts from random vectors. It draws them from a Gaussian fitted to the language model's embeddings 2104.06599 §4.4. Random starts come out close to informed starts on T-REx: 51.9 against 51.6 with BERT-large 2104.06599 tab. 1. On ConceptNet the paper reports random initialisation as the best soft variant, at 14.5 P@1 with BERT-large against 0.1 for LAMA 2104.06599 tab. 3.

Pre-trained prompts before task tuning (PPT)

The prompt is first trained on unlabelled text with the model frozen, using self-supervised cloze tasks, and only then tuned on the labelled task 2109.04332 §3.1. The paper's three pre-training pairs are a sentence pair with three labels, a multiple-choice choice among six sentences, and a single-sentence sentiment-style task with pseudo-labels from a RoBERTa labeller 2109.04332 §3.2 2109.04332 eq. 3-5. The authors motivate it by the high variance of prompt tuning from random starts with 32 examples per task. On SST-2 with T5-XXL, vanilla prompt tuning scores 70.5 ± 15.5 against 93.5 ± 0.3 for PPT and 91.4 ± 0.8 for full fine-tuning 2109.04332 tab. 4. The gain depends on starting downstream tuning from the pre-trained prompt rather than from a random one 2109.04332 §3.1 2109.04332 §4.2. The unified variant converts every classification task to multiple choice, so one pre-trained prompt serves all tasks 2109.04332 §3.3. Pre-training is done once and its prompts are reused across tasks 2109.04332 app. E.

Data-dependent mixture weights (not recommended)

Section 3.4 describes weights that depend on the input through a learned temperature. The authors report little effect, because training pushed the temperature high enough to ignore the input 2104.06599 §3.4 2104.06599 §4.7. The released config switches this path off config.yaml.

Channel prompt tuning for few-shot classification

A frozen GPT-2 Large is given 20 learned vectors in front of the input, and only those vectors are trained 2108.04106 §4.2.3 2108.04106 tab. 4. The score is the channel direction, P(input given label), rather than the direct P(label given input) 2108.04106 §3. On 11 text classification datasets with K=16 examples, the paper reports 61.7 average and 53.0 worst case for channel prompt tuning, against 48.4 and 29.5 for direct prompt tuning 2108.04106 tab. 4 2108.04106 §6.2. The vectors are a single prefix at the input, not a per-layer set, so this variant has the input-layer shape above with a different score.

Test-time context tuning (CLIP)

DiffTPT tunes four learned context tokens of CLIP's text prompt, once per unlabelled test image, and resets them for the next image 2308.06038 §3.1 2308.06038 §4.1. The loss is the entropy of the averaged prediction over selected augmented views 2308.06038 eq. 4 eq. 7. The views are the original image, 64 crop and flip views, and 63 Stable Diffusion variations. The paper keeps the 80% of generated views most similar to the original, then the 30% lowest-entropy views 2308.06038 §3.4 2308.06038 §4.1. On the ImageNet variants, the ViT-B/16 average is 62.28 against 61.59 for the earlier TPT method, and the RN50 average is 48.71 against 46.65 2308.06038 tab. 1. The paper's headline gain is discussed under Contradictions.

Additive image frame on a frozen model (visual prompting)

The learned vectors are not text tokens here. The learned object is one image-sized tensor that is added to every input, while the model stays frozen 2203.17274 §3.2.1. Only the frame is trained, so the method needs access to the input and not to the model's weights or features. The default template is a padding frame of width 30 on 224×224 inputs, with parameters only in the border 2203.17274 §3.2.2 §6.2. Training uses SGD at learning rate 40 for 1,000 epochs with cosine decay 2203.17274 §3.4.

On CLIP ViT-B/32 the 12-dataset average rises from 54.2 with the text prompt alone to 78.2 with the frame, against 83.1 for a linear probe and 86.9 for full fine-tuning 2203.17274 tab. 1. The gain is uneven. The frame beats the linear probe on EuroSAT, SVHN and CLEVR and trails it on the other nine datasets 2203.17274 tab. 1. On the three WILDS sets with CLIP, the average is 58.1 against 62.6 for the linear probe, while Camelyon17 is 89.8 against 84.9 2203.17274 tab. 2. On vision-only backbones (Instagram ResNeXt, BiT-M and RN50) the 12-dataset averages are 32.4, 32.6 and 25.7, against 71.7, 72.6 and 72.5 for linear probes 2203.17274 tab. 4. In the template ablation on EuroSAT with frozen CLIP, fixed-location templates beat random-location ones, and padding with p = 30 is best overall 2203.17274 §6.2 fig. 6.

Learned task tokens (inpainting)

PowerPaint trains one Stable Diffusion inpainting model with three learned prompts: one for context filling, one for text-guided object insertion and one for shape-guided insertion 2312.03594 §3.2. The context prompt starts from a random token array and is learned by minimising the diffusion loss 2312.03594 §3.2 eq. 4. The number of vectors per prompt is not stated in the paper. Replacing the learned prompts with rare tokens that have no learned meaning makes every reported column worse, including Local-FID 13.03 against 11.61 and Context FID 3.11 against 2.59 2312.03594 tab. 6. For object removal, the object prompt is used as the negative prompt in classifier-free guidance 2312.03594 §3.2 eq. 5.

Learned prompt pair as a judge (image enhancement)

CLIP-LIT learns a positive and a negative prompt, each a block of 16 learnable token embeddings of width 512 in CLIP's text encoder 2303.17569 §3.1. The prompts are fitted with a binary loss to separate backlit and well-lit images, then refined with a ranking loss while the enhancer is trained against them 2303.17569 §3.1 2303.17569 §3.2. The learned prompts are not used to produce text. They score the enhancer's output. On the BAID test set, the paper reports PSNR 21.579 against 19.740 for Zero-DCE, the best released baseline in its table 2303.17569 tab. 1.

Encoder-linked slots in a hand-written template (P-Tuning)

P-Tuning keeps the written words of a template and adds continuous slots before, between and after the input and the label 2103.10385 §2.2. The slot vectors go through a small prompt encoder, an LSTM or an MLP, so that the slots depend on each other 2103.10385 §2.3. With a frozen model on LAMA, the paper reports LAMA-34k P@1 of 48.3 for BERT-base, against 43.3 for AutoPrompt on the same model size 2103.10385 tab. 3. On few-shot SuperGLUE with ALBERT-xxlarge-v2, the four-run average is 71.81 against 70.74 for PET-FT. The gap is about one point, inside the reported standard deviations on several tasks 2103.10385 tab. 5 §3.3. On LAMA-P17, the spread across six manual prompts falls from 10.1 to 0.46. That is one relation, so it does not show the same for others 2103.10385 tab. 6.

Multimodal prompts in both encoders (MaPLe)

MaPLe adds learned tokens to CLIP's text branch and to its image branch. Each vision prompt is a linear function of the text prompt in the same layer 2210.03117 §3.2.3. Prompts go into the first nine blocks for the base-to-novel setting 2210.03117 §4.1. With 16 shots per class, the average harmonic mean over base and novel classes is 78.55 across 11 datasets, against 75.83 for Co-CoOp 2210.03117 tab. 3. Novel-class accuracy is 75.14, against 63.22 for text-only CoOp and 71.69 for Co-CoOp [arxiv:2210.03117 tab. 3]. In the paper's ablation, independent vision and language prompts give a harmonic mean of 77.90, so the coupling adds 0.65 points 2210.03117 tab. 1. The trainable parameters total 3.55M, which is 2.85% of CLIP 2210.03117 tab. 6.

Regularised deep prompts with epoch averaging (PromptSRC)

PromptSRC adds deep vision and text prompts to a frozen CLIP, as MaPLe does 2307.06948 §3.1. In the base-to-novel and few-shot settings, the prompts go into the first nine transformer layers, with four tokens per branch 2307.06948 §4.1. The change is in the loss. Besides cross-entropy, it adds an L1 distance from the frozen image and text features, and a KL term that pulls the prompted class distribution toward the frozen one 2307.06948 §3.2.1 eq. 2 eq. 3. The text target for each class is the mean over 60 prompt templates and is used only in training 2307.06948 §3.2.3. The feature weights are 10 for images and 25 for text 2307.06948 §4.1. After training, prompts from several epochs are averaged with Gaussian weights and the average is used for inference 2307.06948 §3.2.2.

The author-reported base-to-novel harmonic mean over 11 datasets is 79.97, against 78.55 for MaPLe and 71.70 for CLIP 2307.06948 tab. 1. Novel-class accuracy is 76.10, against 75.14 for MaPLe 2307.06948 tab. 1. In the ablation, the agreement loss alone takes the IVLP baseline from 77.51 to 79.55, which accounts for most of the gain. Epoch averaging and the template ensemble add 0.15 and 0.27 2307.06948 tab. 2. The paper reports 46K tunable parameters, against 3.55M for MaPLe 2307.06948 §4.3. The released code differs from the text in several places, listed on the card Self-regulating Prompts.

Gradient-projected context tuning (ProGrad)

ProGrad learns CoOp-style context vectors for a frozen CLIP, with one change. The cross-entropy gradient is projected so that it never opposes the gradient of a term that keeps predictions near zero-shot CLIP 2205.14865 §3.2 eq. 4. With λ = 0 the method reduces to plain context tuning 2205.14865 §3.2. The motivation is that such tuning can fall below zero-shot CLIP with one or two shots 2205.14865 §1 fig. 1. With ResNet-50 on 11 datasets, the author-reported average is 62.61% at 1 shot against 59.46% for CoOp, and 74.28% against 73.49% at 16 shots 2205.14865 tab. 1. The projection can only protect what the zero-shot prompt already gets right, because many of the method's failures are also misclassified by zero-shot CLIP 2205.14865 §4 fig. 5.

Per-layer prefix keys and values (prefix-tuning)

Prefix-tuning leaves the input embeddings alone. It prepends learned key and value vectors inside each attention layer, and the base model stays frozen 2304.01933 §2 eq. 1. LLM-Adapters tests it as one of four adapter families, next to series and parallel adapters and LoRA 2304.01933 §3.3. The paper uses 10 virtual tokens, the value it chose from a sweep over 10 to 40 2304.01933 §4.1 app. A.4. In that comparison prefix-tuning has the lowest math average on all four base models. On LLaMA-13B it reaches 46.5 over six math sets, against 65.4 for LoRA 2304.01933 tab. 3. On the eight commonsense tasks it reaches 68.4, against 80.5 for LoRA 2304.01933 tab. 4. On LLaMA-7B the sweep is not monotonic: 42.0 at 10 tokens, 38.0 at 20, 33.1 at 30 and 38.1 at 40 2304.01933 app. A.4. All numbers are the authors' own. Three of the six math sets overlap the training data, according to the repository README.md, so the math averages are weaker evidence than the commonsense ones.

Per-layer prompts with a task head (P-Tuning v2)

P-Tuning v2 gives every layer of a frozen backbone its own trainable prefix vectors, read as keys and values, and adds a trainable head. It follows deep prompt tuning 2110.07602 §3.2. The paper contrasts this with input-layer prompting, where vectors are trained only at the input. That baseline performs poorly on hard sequence labeling and extractive QA 2110.07602 §1 2110.07602 §2. The head is a linear layer on the first-token representation for classification 2110.07602 §3.3. For sequence labeling it is a per-token linear classifier 2110.07602 app. A. The reparameterising MLP is optional, and the paper says its value depends on the task 2110.07602 §3.3.

Prefix length depends on the task. Simple classification usually prefers fewer than 20 positions, and hard sequence labeling around 100 2110.07602 §3.3 fig. 4. Adding the same number of prompts to deeper layers beats adding them to shallower ones, on RTE and BoolQ with BERT-large 2110.07602 §4.3 fig. 3.

On SuperGLUE dev, the paper reports results comparable to fine-tuning from BERT-large to a 10B model, with 0.1% to 3% of parameters trained 2110.07602 §1 2110.07602 tab. 2. On RTE, the same method beats full fine-tuning with BERT-large (78.3 against 70.4) and RoBERTa-large (89.5 against 86.6) 2110.07602 tab. 2. On sequence labeling the gap to fine-tuning is small with RoBERTa-large: CoNLL03 is 92.8 against 92.6 2110.07602 tab. 3. The results are fully supervised, not few-shot 2110.07602 §4. The paper's dev numbers are discussed under Contradictions.

Soft prefix before a hand-written prompt (instruction prompt tuning)

Med-PaLM uses one shared soft prefix of 100 vectors, 1.84M parameters in all, placed before a written instruction, few-shot exemplars and the question 2212.13138 §3.3.3 app. A.1. Only the prefix is trained, on 40 clinician-written exemplars 2212.13138 §3.3.4. The checkpoint is chosen by clinician ranking rather than by a validation metric 2212.13138 app. A.1. On long-form answers, the paper reports 92.9% of Med-PaLM answers aligned with scientific consensus against 61.9% for Flan-PaLM 2212.13138 §4.5. The human-evaluation table disagrees with the text on this axis (see Contradictions). The rated set is 140 questions with one rater per answer, so the numbers are thin 2212.13138 §4.5. Med-PaLM is worse than Flan-PaLM on inappropriate or incorrect content, 18.7% against 16.1% 2212.13138 app. A.7.

When to use which

Compute: the paper reports that one prompt trains in minutes even for BERT-large, and mixture training grows roughly linearly with the number of prompts 2104.06599 §4.5. MaPLe reports 5 training epochs where Co-CoOp needs 10 2210.03117 §4.6.

Implementation sketch

The steps below follow the paper's method; they are not the released code. Each step is cited to the section that defines it.

# Build per-layer soft prompt parameters  [arxiv:2104.06599 §3.1] [§3.2]
# Input:  hard prompt tokens; LM depth L; vector size d
# Output: token vectors (trainable) and zero perturbations per layer

def make_soft_prompt(hard_tokens, num_layers):
    # Start each token vector at its embedding  [§3.1]
    vectors = embed(hard_tokens)                          # (T, d)
    # One zero-initialised perturbation per layer, 0..L  [§3.2]
    deltas = zeros(num_layers + 1, len(hard_tokens), d)   # (L+1, T, d)
    return vectors, deltas

# Mixture prediction for one candidate answer  [arxiv:2104.06599 §3.3] [eq. 1]
# Input:  prompts with mixture weights; input x; candidate answer y

def mixture_probability(prompts, weights, x, y):
    # Weighted sum of the LM probability of y under each prompt  [eq. 1]
    return sum(w * lm_probability(p, x, y)
               for p, w in zip(prompts, weights))

Notes for implementers, each from the paper:

• Optimiser: Adam with default settings. The learning rate is not stated 2104.06599 §4.5.
• Gradient batch size 64, at most 16 epochs, early stopping with patience 4, and keep the checkpoint with the best dev P@1 2104.06599 §4.5.
• Answers are single tokens, so predictions are the most probable answer under the mixture 2104.06599 §4.2 2104.06599 §4.6.
• The paper states no regulariser on perturbation size beyond early stopping, and the released config applies no penalty 2104.06599 §3.2 config.yaml.
• The release differs from the text in batch size, layers tuned, and the weight update. The card's "Paper vs code" section lists each difference: Learning How to Ask.
• Channel prompt tuning uses 20 vectors and trains with the length-normalised channel loss 2108.04106 §4 2108.04106 §4.2.3. The released code drops the GPT-2 BOS token to make room for the vectors, which the paper does not mention util.py.
• P-Tuning slots are passed through an encoder before they replace the slot embeddings 2103.10385 §2.3. The LAMA code uses a two-layer bidirectional LSTM and an MLP prompt_encoder.py. The paper's manual prompt is not inserted by default in that code modeling.py, so add it yourself and say so.
• Initialisation matters in low-data settings, according to prefix-tuning results the survey cites, and starting from a discrete prompt can help 2107.13586 §4.3.2.
• MaPLe's text says sharing deep prompts across layers is better, but its equations and code keep one prompt tensor and one coupling layer per layer 2210.03117 §3.2.2 maple.py.
• Med-PaLM's prefix is uniform over [-0.5, 0.5], trained with AdamW at learning rate 0.003, weight decay 0.00001 and batch size 32 for 200 steps 2212.13138 app. A.1. The loss and the sampling scheme are not stated.
• ProGrad's λ is 0.8 at 16 shots in the paper, and 1.0 everywhere in the released scripts 2205.14865 §4 prograd.sh. The released trainer builds its teacher from a single template, so ProGrad++ cannot be rerun from that release prograd.py.
• ProGrad's optimiser is SGD with initial learning rate 0.002, a cosine schedule and batch size 32 2205.14865 app. A2. The paper does not state momentum or weight decay 2205.14865 app. A2.
• Prefix-tuning uses learning rate 3e-2, against 3e-4 for the other families, with batch size 16 for three epochs 2304.01933 §3.3. The released prefix command uses batch size 8 and five epochs math_running_commands. The code defaults to 30 virtual tokens, not the paper's 10 finetune.py 2304.01933 §4.1.
• P-Tuning v2 tunes the learning rate and prefix length per task. The released scripts range from 1e-4 to 6e-2 for the rate and from 8 to 224 positions for the prefix run_wic_bert.sh run_conll05_roberta.sh. The optimiser, weight decay and warmup are not stated in the paper, and no script sets them run_rte_roberta.sh. The code uses an exponential learning-rate schedule for the sequence-labeling and QA runs trainer_exp.py.
• P-Tuning v2 has no reference code for the multi-task MPT-2 rows in the paper 2110.07602 tab. 3. Train one dataset per run.
• The image-frame code is a CIFAR-100-only release, so it cannot rerun the 12-dataset table 2203.17274 tab. 6 main_clip.py. Its CLIP script fixes the text template to "This is a photo of a {}" for every run, while the paper uses per-dataset templates main_clip.py 2203.17274 tab. 6.
• The code validates on the CIFAR-100 test split after every epoch and keeps the best checkpoint main_clip.py main_clip.py. Any number taken from that checkpoint was selected on test data, and the paper does not say which checkpoint it reports.
• The vision script maps class i to output i, not the semantic mapping that the paper's dog-and-cat test relies on main_vision.py 2203.17274 §6.3. Use a semantic mapping if you adapt it.
• The frame's initialisation is not stated in the paper. The code draws standard-normal values in the normalised input space prompters.py.

Contradictions

• Layers tuned in the headline result. §4.7 presents tuning layer 0 as the basic method and all layers as an add-on 2104.06599 §4.7. Table 1 matches the all-layer column of Table 5 2104.06599 tab. 1 2104.06599 tab. 5.

• Same method, different numbers. Table 4 reports 51.0 P@1 with both token vectors and perturbations adjusted, and 50.7 with token vectors alone 2104.06599 tab. 4. Table 5 reports 51.0 for layer 0 only and 51.6 for all layers 2104.06599 tab. 5, and Table 1 uses 51.6 2104.06599 tab. 1. The tables do not state that these rows share a configuration, so they are not reconciled here.

• Batch size. The paper states 64 for gradient training. The released config uses 32 2104.06599 §4.5 config.yaml.

• Headline gain of test-time context tuning. The paper states an average gain of 5.13% over TPT 2308.06038 §4.2. The averages printed in its tables give smaller gains, from 0.7 to 2.9 points on the averages the tables print 2308.06038 tab. 1 2308.06038 tab. 2. The paper does not say how 5.13% is computed, so this page uses the table values.

• Printed average in DiffTPT. The RN50 average in tab. 1 is printed as 42.26, while its five values average to 44.26. The OOD average does match 2308.06038 tab. 1. Recompute averages before comparing.

• Headline LAMA gain of P-Tuning. The text reports 43.3 to 50.6 2103.10385 §3.1. That pairs AutoPrompt on BERT-base with P-Tuning on BERT-large. The same-size comparison is 43.3 against 48.3 2103.10385 tab. 3.

• GPT results, text against table. The paper says P-Tuning "consistently achieves the best performance" on the GPT-2 models 2103.10385 §3.2. Table 4 shows PET-FT higher on several GPT-2 columns 2103.10385 tab. 4.

• MaPLe learning rate. Section 4.1 and the code give 0.0035 for base-to-novel. Appendix A gives 0.0025 2210.03117 §4.1 2210.03117 app. A. The code is the reference used here.

• Med-PaLM aligned share. The text gives 92.9% aligned for Med-PaLM 2212.13138 §4.5. The appendix table gives 92.6% in the row labelled "Aligned", and the same values appear in the row labelled "No consensus" 2212.13138 app. A.7. Check the source table before citing either.

• P-Tuning v2 text against table. The paper says P-Tuning v2 is comparable to fine-tuning at every scale tested 2110.07602 §4.1. The table shows some cases where fine-tuning is higher, such as WiC with RoBERTa-large (75.6 against 73.4) and WSC with GLM-xxlarge (95.2 against 93.3) 2110.07602 tab. 2.

• P-Tuning v2 released numbers. The README's RoBERTa-large row gives RTE 86.6 and COPA 92.0, against 89.5 and 93.0 in table 2 README.md 2110.07602 tab. 2. The README gives 85.3 for CoNLL12 with BERT-large, where table 3 gives 83.2 README.md 2110.07602 tab. 3. The README says reproduction may need tuning for the local environment README.md.

• P-Tuning v2 head. The paper describes a linear head on the first-token position 2110.07602 §3.3. The released classifier reads the backbone's pooler output, which is frozen with the rest of the backbone sequence_classification.py.

• P-Tuning v2 hidden hyperparameters. The LaTeX source holds a per-task table that disagrees with the run scripts. CoNLL03 with RoBERTa-large has prefix length 112 in the hidden table and 11 in the script run_conll03_roberta.sh. SQuAD 1.1 with RoBERTa-large has batch size 16 in the hidden table and 8 in the script run_squad_roberta.sh. The paper does not say which values produced its numbers.

• P-Tuning v2 dev numbers. SuperGLUE and SQuAD numbers are dev-set results. The base trainer keeps the best dev score over epochs trainer_base.py, and the paper does not say how its reported number was chosen 2110.07602 tab. 2.

• PromptSRC prompt depth. Section 3.1 says the prompts go into every transformer block. Section 4.1 says the first nine layers for base-to-novel and few-shot, and the first three for cross-dataset and domain generalisation 2307.06948 §3.1 2307.06948 §4.1.

• PromptSRC ablation text against table. Section 4.2 says the agreement loss raises novel accuracy by 3.95 points. Table 2 gives 75.38 minus 71.79, which is 3.59 2307.06948 §4.2 2307.06948 tab. 2.

• PPT claims against its own table. The paper says PPT beats fine-tuning on all Chinese datasets and most English ones 2109.04332 §4.2. In table 4, PPT with T5-XXL scores 66.43 on BoolQ and 71.2 F1 on CB, against 80.8 and 86.5 for fine-tuned T5-XXL 2109.04332 tab. 4. The summary above uses the table values.

• Pre-training cost. The paper's pre-training uses about 10 GB of text per format and up to 200,000 steps, but no wall-clock time is stated 2109.04332 app. C. The PPT rows in tables 4 and 5 use prompts pre-trained this way 2109.04332 tab. 4 2109.04332 tab. 5.

Papers

• Exploring Visual Prompts for Adapting Large-Scale Models: learns one additive image frame for a frozen CLIP or vision model and adds it to every test input.
• GPT Understands, Too: P-Tuning, trainable prompt vectors with an LSTM or MLP encoder inside a hand-written template.
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts: introduces soft prompts with per-layer perturbations and an EM-trained mixture for cloze-style knowledge probing.
• Survey of controllable generation with pre-trained transformers: a survey that places prefix-tuning and P-tuning under prompt-based fine-tuning for controllable generation; it reports no experiments of its own.
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing: a survey defining prompt-based learning, continuous prompts and the training strategies used here.
• Noisy Channel Language Model Prompting for Few-Shot Text Classification: scores labels by P(input given label) and tunes 20 prompt vectors on a frozen GPT-2 Large for few-shot classification.
• PPT: Pre-trained Prompt Tuning for Few-shot Learning: pre-trains soft prompts on unlabelled text with three cloze tasks, then tunes them per task on a frozen T5 with 32 examples.
• Prompt-aligned Gradient for Prompt Tuning: ProGrad, which removes from the CLIP prompt-tuning gradient the part that opposes a zero-shot KL term.
• MaPLe: Multi-modal Prompt Learning: learns coupled prompts in both the text and vision branches of a frozen CLIP.
• Large Language Models Encode Clinical Knowledge: Med-PaLM, a 100-token learned prefix on Flan-PaLM 540B, with clinician evaluation of long answers.
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks: trainable prefix vectors at every layer of a frozen model with a task head, evaluated on SuperGLUE, sequence labeling and extractive QA.
• Iterative Prompt Learning for Unsupervised Backlit Image Enhancement: learns a positive and negative CLIP prompt to judge enhanced images, without paired training data.
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models: compares prefix-tuning with series and parallel adapters and LoRA on four open LLMs.
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning: tunes four CLIP context tokens per test image using augmented and diffusion-generated views.
• A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting: trains one Stable Diffusion inpainting model with three learned task prompts.
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting: PromptSRC, deep CLIP prompts with feature and logit agreement losses and epoch-level prompt averaging.
• Unleashing the potential of prompt engineering for large language models: a 2023 survey of prompt engineering, including CoOp, CoCoOp and MaPLe; it reports no new results.