P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

In plain words

Fine-tuning a language model for a task means updating all of its weights. That needs a lot of memory, and each task needs its own full copy of the model. Prompt tuning freezes the model and learns only a few "soft" vectors that are fed in next to the text. It is cheap, but earlier work found it only worked well on very large models and on simple classification tasks.

P-tuning v2 also keeps the model frozen, but gives every layer its own learned vectors. Each layer reads them as extra context, so the vectors can steer what the model computes at every depth. A small classification head on top produces the answer. Only the vectors and the head are trained, so each task stores a small file instead of a full model.

The authors report that this works across model sizes and across tasks, including tagging words in a sentence and finding answer spans in a text. The gains are their own measurements on their own benchmarks.

TL;DR

• P-tuning v2 is deep prompt tuning adapted to language understanding: trainable prefix vectors at every layer of a frozen backbone, plus a trainable head §3.2 §3.3.
• Author-reported: comparable to full fine-tuning on SuperGLUE dev, SQuAD, and NER/SRL/CoNLL tasks, from BERT-large to GLM-xxlarge (10B), with 0.1%-3% trainable parameters §1 tab. 2 tab. 3.
• The paper body fixes almost no hyperparameters. The released run scripts do, and some disagree with a commented-out appendix table in the LaTeX source.
• SuperGLUE and SQuAD numbers are dev-set results, and the released code reports the best epoch on dev. The split for the NER and SRL numbers is not stated. Read the gaps with that in mind.

Problem

• Fine-tuning must store gradients and optimizer state for every parameter, and keeping one model copy per task is inconvenient at inference §1.
• Prompt tuning, which trains only continuous inputs, matches fine-tuning mainly on models of about 10B parameters and on simple classification tasks §1 §2. On 100M-1B models it performs much worse §3.1.
• It also performs poorly on hard sequence labeling, such as extractive QA and NER §1 §3.1 §4.2.
• Scope of the experiments: fully supervised, not few-shot, and every method except fine-tuning keeps the backbone frozen §4.

Method

Input-layer prompt tuning, as in Lester et al. and P-tuning, prepends trainable embeddings to the input embeddings only. The trainable part sits at the input §2 fig. 2. P-tuning v2 instead prepends trainable prefix vectors at every transformer layer, as keys and values, following deep prompt tuning §3.2 fig. 2.

1. Freeze the whole pretrained backbone. Only the prompt table and the head are trained §4. The released code sets requires_grad = False on the backbone sequence_classification.py.
2. Create a prompt table with one row per prefix position. Each row holds a key and a value vector for each layer §3.2 prefix_encoder.py. The code applies dropout to the prompt values sequence_classification.py.
3. Optional reparameterization: a two-layer MLP (hidden size 512 by default) can encode the prompt rows. The paper says its usefulness depends on the task §3.3. The code uses a direct embedding table unless --prefix_projection is set arguments.py.
4. Classification (sequence level): a randomly initialised linear head on the first-token representation, as in BERT §3.3. In the code, that representation is the backbone's pooler output sequence_classification.py.
5. Sequence labeling: a linear classifier per token app. A. The NER code labels only the first sub-token of each word dataset.py.
6. Extractive QA: a start and an end label per token, pick the most confident span, and call the question unanswerable below a threshold app. A. The code uses a fixed 384-token window with stride 128 dataset.py dataset.py.
7. Multi-task (optional): several datasets of one task family share the prompt table, each with its own linear classifier §3.3 §4.

Prompt length matters per task: simple classification tasks usually prefer fewer than 20 prefix positions, and hard sequence labeling tasks prefer around 100 §3.3 fig. 4. In the RoBERTa-large ablation, the best length and the choice of reparameterization differ by dataset app. B fig. 4.

Prompt depth matters too. With the same number of prompted layers, adding prompts to the deeper layers beat adding them to the shallow ones on RTE and BoolQ with BERT-large §4.3 fig. 3.

# P-tuning v2 forward pass, sequence classification
# Input:  input_ids (B, T), attention_mask (B, T)
# Output: logits (B, num_labels)
# The paper has no algorithm box; this is read from the released code.

def forward(input_ids, attention_mask):
    # One trainable row per prefix position  [§3.2]
    prefix_rows = prompt_table(arange(L))  # (L, 2 * layers * d)
    # Dropout on the prompt values
    # [code: model/sequence_classification.py#L367]
    prefix_rows = dropout(prefix_rows)
    # One key prefix and one value prefix per layer  [§3.2]
    past = split_per_layer(prefix_rows, batch_size=B)
    # Prefix positions are always attended (mask = 1)
    # [code: model/sequence_classification.py#L389]
    attention_mask = concat(ones(B, L), attention_mask)
    # Frozen backbone reads the prefix as past keys and values  [§3.2]
    outputs = frozen_backbone(input_ids, attention_mask, past)
    # Pooled first token from the frozen pooler  [§3.3]
    # [code: model/sequence_classification.py#L404]
    pooled = outputs.pooled
    # Trainable linear head after dropout  [§3.3]
    # [code: model/sequence_classification.py#L406-L407]
    logits = classifier(dropout(pooled))
    return logits

Equations: the paper has no numbered equations. Its input-sequence formulas in §2 are unnumbered and cite as §2. The per-layer scheme is described in words §3.2.

Hyperparameters

The paper body gives only the prompt length guidance above §3.3 and the 0.1%-3% parameter ratio §1. The table below is taken from the released run scripts. Every script sets these values on lines 5-9 run_rte_roberta.sh. Each row cites its own script. Unless noted, the common settings are: --prefix on, --seed 11 (checked in the RTE and SQuAD scripts), --hidden_dropout_prob as "Dropout", max_seq_length 128 by default arguments.py, and per-epoch evaluation.

Other values in the code, not in the paper body:

Results

Author-reported, on the dev sets of each benchmark tab. 2 tab. 3. Setup: frozen backbones for PT and PT-2, full fine-tuning for FT, fully supervised §4. PT is the input-layer prompt tuning baseline (Lester et al. and P-tuning). MPT-2 is multi-task P-tuning v2 tab. 3. Sizes: BERT-large 335M, RoBERTa-large 355M, DeBERTa-xlarge 750M, GLM-xlarge 2B, GLM-xxlarge 10B tab. 2 tab. 3.

SuperGLUE dev, FT / PT / PT-2 tab. 2. The metric for each row is not named in the caption, except MultiRC F1a and ReCoRD F1 as printed in the column headers:

Sequence labeling and QA, FT / PT / PT-2 / MPT-2 tab. 3. NER and SRL are micro-F1. The caption does not say which split the sequence-labeling rows use, only that the SQuAD rows are dev.

In the SQuAD rows, each cell lists FT, PT, PT-2 and MPT-2 in turn, separated by " · ". The CoNLL12 PT value for BERT-large is typeset as "64.5." in the source. It is read here as 64.5.

Ablation, RoBERTa-large, accuracy (metric not named in the caption) tab. 4:

Headline reading, with the paper's own framing: on RTE, PT-2 beats full fine-tuning for BERT-large (78.3 vs 70.4) and RoBERTa-large (89.5 vs 86.6) tab. 2. On SQuAD 2.0 with DeBERTa-xlarge, PT-2 matches fine-tuning (88.4 / 91.1 vs 88.3 / 91.1) tab. 3. The paper says PT-2 is comparable to fine-tuning at every scale it tested §4.1. The table does not support "always": for example, WiC with RoBERTa-large is 73.4 vs 75.6, and WSC with GLM-xxlarge is 93.3 vs 95.2 tab. 2.

Figures, in words only. fig. 1 shows the average dev score on RTE, BoolQ and CB for PT-2 against fine-tuning and prompt tuning across model scales. fig. 2 compares input-layer prompting with P-tuning v2: orange blocks are trainable prompts and blue blocks are embeddings from the frozen model. fig. 3 shows, for BERT-large on RTE and BoolQ, that adding the same number of prompts to deeper layers beats adding them to shallower layers. fig. 4 shows the prompt-length and reparameterization curves for RoBERTa-large on RTE, CoNLL04, BoolQ and CoNLL12; the paper says the best choice depends on the dataset app. B.

Paper vs code

• Reparameterization default. The paper says the MLP helps on some datasets and not others §3.3. The code's default is off arguments.py. Only the BERT-large WiC script turns it on run_wic_bert.sh.
• Hyperparameters in the LaTeX source but not in the PDF. appendix.tex holds a full per-task table inside \hide{...}, which the macro discards [raw/src/2110.07602/appendix.tex]. Some entries disagree with the scripts:
  • CoNLL03 RoBERTa-large: prefix length 112 in the hidden table; psl=11 in the script run_conll03_roberta.sh.
  • OntoNotes 5.0 RoBERTa-large: learning rate 7e-2 and 30 epochs in the hidden table; lr=7e-3 and epoch=60 in the script run_ontonotes_roberta.sh run_ontonotes_roberta.sh.
  • SQuAD 1.1 RoBERTa-large: batch size 16 in the hidden table; bs=8 in the script run_squad_roberta.sh.
  • WSC RoBERTa-large: prefix length 20 and 100 epochs in the hidden table; psl=8 and epoch=10 in the script run_wsc_roberta.sh.
  • Which of each pair produced the paper's numbers is not stated.
• Best-epoch dev reporting. The base trainer keeps the best dev score over epochs and records its epoch trainer_base.py trainer_base.py. The released README lists a "Best Epoch" row beside its reproduced numbers README.md README.md. The paper does not say how the reported dev number was chosen.
• Released numbers differ from the paper. The README's RoBERTa-large row gives RTE 86.6 and COPA 92.0 README.md. Table 2 gives PT-2 RTE 89.5 and COPA 93.0 tab. 2. The README's BERT-large row gives CoNLL12 85.3 and COPA 77.0 README.md. Table 3 gives 83.2 for CoNLL12 and table 2 gives 73.0 for COPA. The README says reproduction may need tuning for the local environment README.md.
• Head. The paper says a linear head on the [CLS] position §3.3. The code reads the backbone's pooler output, which is a dense layer with tanh over the first token, and that pooler is frozen sequence_classification.py sequence_classification.py.
• Multi-task. The paper describes MPT-2 §3.3 §4 tab. 3. No multi-task training code or script was found in the repository. Every run script trains one dataset.
• Models without configs. The repository has no run script for DeBERTa, GLM, CB, MultiRC, ReCoRD, or BERT-large on SQuAD and CoNLL03. GLM appears only in the README. The DeBERTa classes exist in the code utils.py, but no script calls them.
• Prompt-tuning baseline. The --prompt path covers only sequence classification and multiple choice utils.py. Token classification and QA have no baseline path, so the PT rows of table 3 cannot be reproduced from this code. No --prompt script exists.
• Sub-token labels. The paper says nothing about sub-tokens. The NER code labels only the first sub-token of each word and sets the rest to -100 dataset.py. A comment in the code mentions a label_all_tokens flag, but the flag is not used.
• SRL verb. The paper says the target verb token is appended to the sentence app. A. The code appends it with no word id, so the loss ignores it dataset.py.
• Unanswerable questions. The paper gives a threshold but no value app. A. The code defaults the null-score threshold to 0.0 and no script overrides it arguments.py.

Pitfalls

• Dev set, best epoch. The SuperGLUE and SQuAD numbers are dev numbers, and the code keeps the best epoch on dev. A test score from the same recipe would likely be lower. The paper does not say how many runs or seeds each cell averages. The scripts I read fix seed 11.
• Fully supervised only. The paper's own scope excludes few-shot settings §4. The README also says its SuperGLUE comparison with P-tuning v1 uses a different backbone regime, with the backbone tuned jointly README.md.
• Frozen is not eval mode. The code sets requires_grad = False on the backbone sequence_classification.py. HF Trainer normally switches the whole model to training mode, which would keep the backbone's dropout active during training. This is standard Trainer behaviour; I did not check it in this repository.
• Frozen pooler. The pooler belongs to the frozen backbone, so the classifier sits on a fixed nonlinear projection of the first token sequence_classification.py.
• Long prefixes are not free. The prefix is concatenated to the keys and values of every layer, so each prefix position adds attention work at every layer. CoNLL05 RoBERTa-large uses 224 prefix positions run_conll05_roberta.sh. The paper does not discuss this cost.
• Learning-rate sensitivity. Learning rates in the scripts range from 1e-4 to 6e-2 across tasks run_wic_bert.sh run_conll04_roberta.sh. A single rate does not transfer. The README says the best hyperparameters can be sensitive to the environment and recommends a search README.md.
• Optimizer details are missing. The optimizer, weight decay, and warmup are not stated in the paper, and no script sets them. Only the NER/SRL/QA exponential schedule is explicit in the code.
• Parameter ratio. The 0.1%-3% figure is computed by the paper against transformer parameters §4. The code prints a trainable count, but I did not reproduce the ratio.
• Typo in the source table. CoNLL12 PT for BERT-large is typeset "64.5." tab. 3 source.

Lineage

• Builds on: deep prompt tuning, as in prefix-tuning (Li and Liang, arXiv:2101.00190) §1 §3.2. The paper also cites Qin and Eisner's soft prompts tab. 1.
• Compared against: Lester et al., "The power of scale for parameter-efficient prompt tuning", arXiv:2104.08691 §1 tab. 2. P-tuning (GPT Understands, Too) §1. Full fine-tuning tab. 2.
• Superseded by: not stated in the paper, and no card in this wiki covers a successor.