GPT Understands, Too

In plain words

A language model can be asked a question by writing a sentence pattern around the input and letting the model fill in a blank, for example "The capital of Britain is [MASK]." Small wording changes to such a pattern can change the answer quality a lot, and finding a good pattern by hand is slow and fiddly.

P-Tuning keeps the written words of the pattern and adds a few learnable vectors at chosen spots. These vectors are not real words, so training can adjust them directly with gradient descent. A small network links them so that they depend on each other. The method works with a frozen model, where only these vectors are trained, and with a model that is also fine-tuned. The authors report better scores on fact recall and on language-understanding tasks, and less spread between different hand-written patterns.

TL;DR

P-Tuning inserts trainable continuous vectors into a hand-written prompt, and a small LSTM or MLP encoder ties them together §2.2 §2.3. The language model stays frozen for LAMA knowledge probing and is tuned for SuperGLUE §3. The authors report gains over discrete prompt search on LAMA and over PET on few-shot SuperGLUE, and a smaller spread across manual prompts tab. 3 tab. 5 §3.4. Before implementing, read Paper vs code: the released LAMA code does not insert the hand-written relation prompt that the paper describes.

Problem

Hand-written discrete prompts are unstable. On LAMA-TREx P17 with BERT-base-cased, four manual prompts reach precision@1 of 31.3, 19.8, 31.4 and 51.1 without P-Tuning, so a one-word change can move the score by about 20 points §2.1 tab. 1. Automatic discrete search (AutoPrompt, LPAQA, mining, or prompts generated by another model) finds better prompts, but the prompts stay real words and the instability remains §1 §2.1. The paper asks whether trainable continuous prompts can stabilise and improve adaptation, with a frozen model (knowledge probing) and with a tuned model (SuperGLUE) §1 §3.

Method

Template and input §2.2. Let M be a pretrained language model with hidden size h and vocabulary V. A template T places discrete tokens [D_i] and continuous prompt slots [P_i] around the input x and the label y. Slots are placed before, between and after x and y. The paper writes the continuous template as {[P_0:i], x, [P_(i+1):j], y, [P_(j+1):k]} (unnumbered display equation in §2.2). The discrete tokens and the input tokens use the usual embedding table e. Each slot P_i is mapped by a function f to a vector h_i, so the model sees {h_0..h_i, e(x), h_(i+1)..h_j, e(y), h_(j+1)..h_k}.

Prompt encoder §2.3. f is a lightweight network. The paper tries an LSTM, an MLP and an identity map (called EMB in the few-shot ablation) §3.3. The stated reason is that f lets the slots depend on each other, instead of being independent free vectors §2.3.

Combination with discrete prompts §2.2. The paper concatenates continuous slots with discrete prompt tokens, and reports that this performs better. All its experiments use the concatenation.

Frozen or tuned model §2.2 §3. The same mechanism works with a frozen model, where only the slot vectors and f are trained, and with a fine-tuned model, where the model weights are trained together with them. LAMA uses frozen models. SuperGLUE uses tuned models, both fully supervised and few-shot.

# P-Tuning loss for one labelled example  [§2.2] [§2.3]
# Input:  text_ids, label_ids, template (layout of slots), model M, encoder f
# Output: scalar loss

def ptuning_loss(text_ids, label_ids, template, model, encoder):
    # Lay out discrete tokens and continuous slots in the template  [§2.2]
    ids, slot_positions, label_positions = fill_template(
        template,
        text_ids,
        label_ids,
    )
    # One trainable vector per slot, the input to the encoder  [§2.3]
    slot_inputs = prompt_table  # shape (k, h), learned by backprop
    # The encoder links the slot vectors to each other  [§2.3]
    slot_vectors = encoder(slot_inputs)  # shape (k, h)
    # Embed every token with the model's own embedding layer  [§2.2]
    inputs = model.embedding(ids)  # shape (T, h)
    # Put the encoder output in place of the slot embeddings  [§2.2]
    inputs[slot_positions] = slot_vectors
    # Run M on the mixed sequence  [§2.2]
    logits = model(inputs_embeds=inputs)
    # Score the label tokens at their positions  [§2.2]
    return cross_entropy(logits[label_positions], label_ids)

Notes on the sketch. The paper gives the mapping but not the exact loss bookkeeping for each model family. How the label is scored (masked position for BERT-style models, last position for GPT-style models) follows the code, see Paper vs code.

Training, LAMA (frozen model) §3.1. Only the prompt encoder is trained, with Adam at learning rate 1e-5. The number of slots per position is selected on the development sets. The paper chooses one configuration for most relations.

Training, SuperGLUE fully supervised (tuned model) §3.2. The model and the slots are trained jointly with AdamW and a linearly decayed learning rate. The grid covers learning rate, batch size and warm-up ratio. Early stopping is used. Model selection uses the development sets.

Training, few-shot SuperGLUE (tuned model, ALBERT-xxlarge-v2) §3.3. The setup follows PET and the FewNLU evaluation procedure. The paper uses random data splits and selects models only on a small labelled set. It searches the learning rate, the maximum number of training steps and the evaluation frequency §3.3. Each result is averaged over four runs with different splits tab. 5. The paper inserts different numbers of continuous tokens at different positions in PET's manual patterns, then picks the best candidate with the FewNLU validation procedure §3.3.

Figures (described from captions only; no image is used on this page). fig. 1 is the introduction's summary plot, with the caption reporting the average development score over seven SuperGLUE datasets with P-Tuning. fig. 2 is the method illustration for the prompt "The capital of Britain is [MASK]". Panel (a) shows a discrete prompt generator that receives only discrete rewards. Panel (b) shows continuous prompt embeddings and an encoder that are optimised differentiably §2.2.

Hyperparameters

Values from the paper unless marked as code. "not stated" means the paper and the code do not give it.

Results

Author-reported. Setups are given per table.

Knowledge probing, LAMA, frozen LM, precision@1 tab. 3.

LAMA-34k (left in the paper; the paper says it covers all BERT vocabularies):

LAMA-29k (right in the paper; manual prompt vs P-Tuning per model):

The text reports 43.3 to 50.6 on LAMA-34k and 45.2 to 64.2 on LAMA-29k §3.1. These compare different backbone sizes, see Pitfalls.

Fully supervised SuperGLUE, development sets, tuned LM tab. 4. Columns follow the paper. The Avg column averages the seven tasks, with CB and MultiRC each averaged over their two metrics. The paper's footnote says the BERT-large CLS-FT row is taken from the SuperGLUE paper.

Few-shot SuperGLUE, ALBERT-xxlarge-v2, mean over four runs tab. 5. The Avg column matches the paper's seven-task averaging.

The ± values are the standard deviations the table reports.

Robustness to the choice of pattern tab. 6. Few-shot BoolQ with ALBERT over six manual patterns (P#0 to P#5), and LAMA-P17 with BERT-base-cased over the same six manual prompts.

Few-shot: number and position of continuous tokens on CB, ALBERT tab. 7. Seg = whether the inserted tokens split a sentence. Pos = edge or middle of the input. [P] = continuous token; [M] = mask token.

Few-shot: prompt encoder type, ALBERT tab. 8, mean ± std.

Few-shot: comparison with auto-searched discrete prompts, RoBERTa-large tab. 9. The paper evaluates the top-3 LM-BFF prompts under its own protocol, and adds P-Tuning on top of the same discrete prompts.

Headline claims, as the paper states them: P-Tuning improves LAMA-34k from 43.3 to 50.6 with the best BERT models §3.1; on few-shot SuperGLUE it averages more than one point above PET and more than 13 points above prompt tuning §3.3; and on LAMA it cuts the spread across manual prompts from 10.1 to 0.46 §3.4.

Paper vs code

Code: the official repository THUDM/P-tuning at commit f4225d4d8003868b31ed659dfe9732a351ad42d9 (MIT). It contains two parts, LAMA and few-shot SuperGLUE README.md. It has no code for fully supervised SuperGLUE, which the paper reports.

1. Hand-written prompt missing in the LAMA code. The paper says P-Tuning concatenates continuous slots with the relation's manual prompt, with the template (3, sub, org_prompt, 3, obj, 3) §3.1. In the P-tuning branch of get_query, the query contains only slot tokens, the subject tokens and the [MASK] or '.' placeholder. The relation template from relations.jsonl is used only when use_original_template is set [code: LAMA/p_tuning/modeling.py#L74-L105]. The default is False [code: LAMA/cli.py#L52]. Implement the paper's version only if you add the template yourself, and say so.
2. Slot order in the BERT branch. The paper's layout puts the subject before the object §3.1. The code's BERT branch places [MASK] (which receives the target label) before the subject tokens, then three slots, then the subject, then three slots or '.', then [SEP] modeling.py. The code comments are not clear on this. Treat the code as the reference for the BERT query layout, and check it against the paper's figure of the template.
3. GPT branch. The template is forced to (t0, t1, 0) [code: LAMA/p_tuning/modeling.py#L43-L44]. The query is slots, subject, slots, and the label is predicted at the last position [code: LAMA/p_tuning/modeling.py#L97-L104] modeling.py. This matches the paper apart from the missing manual prompt (item 1).
4. LAMA prompt encoder. The paper describes a lightweight network and tries LSTM, MLP and identity §2.3. The LAMA code always uses a 2-layer bidirectional LSTM followed by an MLP [code: LAMA/p_tuning/prompt_encoder.py#L26-L40]. The identity option is not in the LAMA code.
5. LAMA optimisation details not in the paper. Weight decay 5e-4 [code: LAMA/cli.py#L47]. Exponential decay with factor 0.98 per epoch [code: LAMA/cli.py#L46] cli.py cli.py. At most 100 epochs cli.py. Early stopping after 20 epochs without a better dev Hit@1, with the test score recorded at the best dev epoch [code: LAMA/cli.py#L189-L196].
6. Frozen LM runs in half precision. When the LM is frozen it is converted with .half() models.py. The forward pass casts inputs to half too modeling.py. The paper does not mention precision.
7. Few-shot learning rates. The paper searches one learning rate in {1e-5, 2e-5} §3.3. The code uses --learning_rate only for the prompt encoder and prompt embeddings wrapper.py [code: PT-Fewshot/cli.py#L117]. The LM optimiser uses a fixed 1e-5 [code: PT-Fewshot/pet/wrapper.py#L312].
8. Few-shot prompt count is the pattern id. The number of continuous tokens is set from pattern_id wrapper.py. Pattern choice and token count are therefore one parameter in code. The paper varies them separately §3.3 tab. 7.
9. Few-shot CB pattern in code. The CB pattern used by the released script (--pattern_ids 1) is text_a [SEP] text_b ? [P] answer: [MASK], where the continuous slot is the position of the word "the" [code: PT-Fewshot/data_utils/task_pvps.py#L325-L329] [code: PT-Fewshot/scripts/cb_pt_few_shot.sh#L16]. This is not one of the rows of tab. 7. The paper does not state the exact patterns used for its main few-shot results, only that PET's manual patterns were extended with continuous tokens §3.3.
10. EMB option missing. The paper's ablation includes the identity encoder (EMB) tab. 8. The few-shot CLI accepts only lstm and mlp [code: PT-Fewshot/cli.py#L142].
11. Few-shot LSTM size. The few-shot encoder uses an embedding and LSTM of size 128 and a two-layer MLP wrapper.py cli.py. The paper does not state these sizes.
12. Evaluation frequency. The paper searches {0.02, 0.04} without a unit §3.3. The code evaluates every 20 steps by default [code: PT-Fewshot/cli.py#L143]. The CB script does not override it.
13. Few-shot defaults not in the paper. Weight decay 0.1 [code: PT-Fewshot/cli.py#L119], linear warm-up with 0 steps by default [code: PT-Fewshot/cli.py#L125] wrapper.py, and a training batch of 2 per GPU in the CB script [code: PT-Fewshot/scripts/cb_pt_few_shot.sh#L12] cli.py.

No discrepancies were checked for the fully supervised SuperGLUE setting, because the repository has no code for it.

Pitfalls

• The LAMA code omits the manual prompt (Paper vs code, item 1). A faithful reimplementation of the paper's numbers needs the template with the relation prompt. The paper does not give the exact manual prompts in the text; they come from LAMA and are listed in tab. 1 only for P17.
• Compare like with like. The headline "43.3 to 50.6" §3.1 compares AutoPrompt with BERT-base against P-Tuning with BERT-large. The same-size comparison is 43.3 vs 48.3 tab. 3. LAMA-34k and LAMA-29k numbers are not comparable with each other.
• Text overstates GPT results. The paper says P-Tuning "consistently achieves the best performance on all tasks" for GPT2-Base and GPT2-Medium §3.2. tab. 4 shows PET-FT higher on WSC and COPA for GPT2-Base (66.3 vs 63.5, 78.0 vs 76.0), and on WiC, MultiRC EM and MultiRC F1a for GPT2-Medium (70.4 vs 69.4, 32.1 vs 29.3, 74.4 vs 74.2).
• Few-shot gains are small and noisy. The average gain over PET is 1.07 points tab. 5, inside the reported standard deviations on several tasks. RTE is lower than PET (63.27 vs 72.83) tab. 5. The text says model selection uses only a small labelled set §3.3. The repository ships the 32-example dev sets used for this (FewGLUE_32dev) README.md; check that your split matches them before comparing numbers.
• Appendix not in the compiled paper. The few-shot hyperparameter table (K = 4 runs, split ratio 1:1, one continuous token, the encoder choices) sits in an appendix file that is commented out in main.tex. The paper's compiled PDF may not contain it. The values that appear in the main text §3.3 are the reliable ones.
• Numbers from source order. The paper was read from LaTeX source. The section numbers (§1 Introduction, §2 Method, §3 Experiments, §4 Related work, §5 Conclusions) and table and figure numbers follow the order of their definitions in the source. Check them against the PDF before citing externally. The method's display equations are unnumbered, so the card cites them by section.
• Development set results, not test. The fully supervised numbers are on the SuperGLUE development sets tab. 4. The few-shot numbers come from the FewNLU protocol §3.3. Neither is a leaderboard score.
• Stability metric. The spread across patterns is a standard deviation over patterns tab. 6. The LAMA figure of 0.46 is for one relation (P17) tab. 6. It does not cover the other relations.
• MegatronLM 11B is frozen and loaded from merged model shards. The repository provides a merge function in megatron_11b/megatron_wrapper.py; the 11B weights themselves come from FairSeq README.md. Expect a large memory requirement; the code's loader warns of about 28 GB of GPU memory in fp16 models.py.
• Results depend on the frozen-LM setup. The LAMA results need the frozen LM (fp16 in code). Tuning the LM changes the setting, and the paper reports the LAMA setup only with frozen LMs §3.
• Data split. The LAMA data comes from AutoPrompt and is an external download README.md. The shared vocabulary option (vocab_strategy) changes the candidate set cli.py.

Lineage

• Builds on: manual discrete prompting, as used by PET, which the paper follows for few-shot patterns and for its SuperGLUE comparison §3.2 §3.3. Concept slug: prompt-engineering.
• Compared against: AutoPrompt and LPAQA on LAMA tab. 3 §3.1; LM-BFF auto-searched prompts tab. 9; Prompt Tuning tab. 5; standard classification fine-tuning tab. 4.
• Concept slugs: soft-prompt-tuning (the method), few-shot-learning (the few-shot setting).