PPT: Pre-trained Prompt Tuning for Few-shot Learning

In plain words

A large language model can be steered with a short run of learned vectors put in front of the input, while the model itself stays frozen. This is cheap to store, one small set per task. The catch is that with only a few labelled examples, these vectors are hard to learn when they start from random values, and results swing a lot from one random seed to the next.

The idea of PPT is to give those vectors a better starting point before the task begins. The vectors are first trained on plenty of unlabelled text, using small made-up exercises that look like the real tasks. One exercise asks whether two sentences belong together, another asks which of several sentences comes next, and a third asks for a sentiment-style word. After that, the vectors are tuned on the real task as usual. The authors report that this approach can match or beat fully retraining the model, even with few examples, and that the results are more stable across seeds.

TL;DR

PPT is prompt tuning with pre-trained prompts. A frozen T5-style encoder-decoder gets a 100-vector soft prompt on its input side §3.1 [code: configs/prompt/ppt.json#L5]. The prompt is first trained on unlabelled text with three cloze tasks, one per task shape: sentence pairs, multiple choice and single sentences §3.2. On 32-example tasks with the 11B models, the authors report PPT reaching or exceeding full fine-tuning on most English and all Chinese datasets they tested, with much lower variance across seeds §4.2 tab. 4. Read Paper vs code first: the repository's base model name, pre-training lengths and batch arithmetic differ from the appendix.

Problem

Prompt tuning (PT) freezes the whole model and trains only soft prompt tokens. For a 100-token prompt on a model with width 4096 that is about 410K trainable parameters, against 11B for full fine-tuning (FT) §4.1 fig. 2. With plenty of data, PT is comparable to FT §1 fig. 2. With 32 training examples per task, it is much worse: on SST-2 with T5-XXL, FT reaches 91.4 ± 0.8 accuracy while PT with random prompt initialisation reaches 70.5 ± 15.5 tab. 1. The large standard deviation means some seeds perform close to chance §4.2.

The paper's pilot experiments tested three levers, none of which closes the gap. Hard, human-written prompts combined with soft prompts help PT but stay below FT §2 tab. 1. Changing the verbalizer (the word that stands for each label) changes results a lot §2 tab. 1. Initialising the soft prompt from embeddings of real words, which earlier work found helpful on smaller models, does little or harm at 11B on SST-2 and BoolQ §2 tab. 2.

Assumptions: 32 training and 32 validation examples per task, sampled from the original training data, with the original validation sets used as test sets because the test sets are not public §2 app. A.

Method

All tasks are written as cloze questions. A pattern f inserts the input into a template with a mask token, and a verbalizer v maps each label to a word §3.1 eq. 1:

\arg\max_{\theta} \sum_{x} \log p(y \mid x;\theta) = \arg\max_{\theta} \sum_{x} \log p\big(\langle X \rangle = v(y) \mid f(x);\theta\big) \qquad [\text{eq. 1}]

Prompt tuning concatenates a soft prompt P in front of the pattern output and trains only P, with the model parameters fixed §3.1 eq. 2:

\arg\max_{P} \sum_{x} \log p\big(\langle X \rangle = v(y) \mid [P; f(x)];\, P\big) \qquad [\text{eq. 2}]

PPT groups the downstream tasks into m groups. Each group gets one pre-training pattern-verbalizer pair. For each group, the prompt P_i is pre-trained on unlabelled data with the model frozen. Each downstream task in the group then starts prompt tuning from P_i §3.1 §3.2. The paper's three pre-training pairs are:

• Sentence pair (tasks such as NLI or similarity). This extends BERT's next sentence prediction to three labels. Label 2 means the two sentences are adjacent, 1 means they come from the same document but are not adjacent, and 0 means different documents. The verbalizer words are no, maybe and yes §3.2.1 eq. 3.

f^{\text{pre}}(x) = \text{"}s_1 \langle X \rangle . s_2\text{"}, \quad v^{\text{pre}}(\mathcal{Y}) = [\text{no}, \text{maybe}, \text{yes}] \qquad [\text{eq. 3}]

• Multiple choice (tasks with a query and candidate answers). Given a query sentence and six candidates, one adjacent sentence, one non-adjacent sentence from the same document and four from other documents, the model picks the adjacent one. Labels are the letters A to F §3.2.2 eq. 4.

f^{\text{pre}}(x) = \text{"}s_q\text{? A.}s_1 \cdots \text{F.}s_6\text{. Answer is }\langle X \rangle\text{."}, \quad v^{\text{pre}}(\mathcal{Y}) = [\text{A}, \dots, \text{F}] \qquad [\text{eq. 4}]

• Single sentence (sentiment-style tasks). Pseudo-labels come from a RoBERTa-base model fine-tuned on a five-class sentiment dataset that is not one of the few-shot evaluation sets. Only confident predictions are kept §3.2.3 §4.1 app. C. The verbalizer words are terrible, bad, maybe, good and great eq. 5.

f^{\text{pre}}(x) = \text{"}s.\langle X \rangle\text{."}, \quad v^{\text{pre}}(\mathcal{Y}) = [\text{terrible}, \text{bad}, \text{maybe}, \text{good}, \text{great}] \qquad [\text{eq. 5}]

Unified PPT §3.3 converts every classification task into the multiple-choice form, so one pre-trained prompt serves all tasks. Pre-training samples have 2 to 16 options, and the caps on query and option lengths shrink as the option count grows §3.3 app. C tab. 9. The released code maps each downstream task to its prompt file: SST-2 and SST-5 to the single-sentence prompt, BoolQ, CB and RTE to the sentence-pair prompt, RACE to the multiple-choice prompt, and all tasks in the unified variant to the unified prompt pt_pretrain.sh pt_pretrain.sh [code: scripts/rte/few-shot/pt_pretrain.sh#L20] pt_pretrain.sh.

Pre-training of one prompt, as the released code does it:

# Pre-train the prompt of one task group  [§3.1] [app. C]
# Input:  frozen PLM; unlabelled corpus; pattern f and verbalizer v
# Output: prompt matrix P, shape (100, d_model)

def pretrain_prompt(plm, corpus, pattern, verbalizer):
    # Fill every prompt slot with the <mask> token's embedding; the
    # paper does not state this choice  [code: pretrain.py#L435-L437]
    prompt = repeat(embedding("<mask>"), 100)           # (100, d_model)
    # Train for up to 200,000 steps  [app. C]
    for step in range(200000):
        # Sample one unlabelled example and build its cloze input  [§3.2]
        text, label = corpus.sample(pattern, verbalizer)
        # Only the prompt gets gradients; the PLM is frozen  [§3.1]
        # [code: model/enc_dec_modeling.py#L148-L158]
        logits = plm(concat(prompt, text))
        loss = cross_entropy(logits, label)
        # Learning rate 0.1 with inverse-square-root decay  [app. C]
        # [code: learning_rates.py#L41-L42]
        prompt = optimizer_step(prompt, loss, lr=noam_lr(step))
    return prompt

Downstream prompt tuning, starting from the pre-trained prompt:

# Downstream prompt tuning from a pre-trained prompt  [§3.1] [app. C]
# Input:  frozen PLM; pre-trained prompt P_i; train set; dev set
# Output: the prompt with the best dev accuracy

def tune_prompt(plm, prompt, train_set, dev_set):
    # Load the pre-trained prompt instead of initialising it  [§3.1]
    # [code: utils.py#L325-L328] [code: configs/prompt/ppt.json#L2]
    best_prompt = prompt
    best_acc = -1.0
    step = 0
    # 50 epochs with batches of 16  [app. C]
    for epoch in range(50):
        for batch in batches(train_set, size=16):
            # Predict the verbalizer word at the mask; only prompt trains
            logits = plm(concat(prompt, batch.text))
            loss = cross_entropy(logits, batch.label)
            # Learning rate searched in {5e-3, 1e-2, 2e-2, 5e-2}  [app. C]
            # Script default 0.01  [code: scripts/sst-2/few-shot/pt.sh#L14]
            prompt = optimizer_step(prompt, loss, lr=0.01)
            step += 1
            # Evaluate every 6 steps and keep the best dev model  [app. C]
            if step % 6 == 0:
                acc = accuracy(plm, prompt, dev_set)
                if acc > best_acc:
                    best_acc = acc
                    best_prompt = prompt
    return best_prompt

Hyperparameters

Results

Author-reported throughout. Each cell is mean ± standard deviation over 5 seeds, with 32 training and 32 dev examples per dataset app. C tab. 4. The paper's source marks the best cells in bold and the best prompt-tuning cells underlined; these marks are omitted here.

Setup: English tasks use T5 sizes from small to XXL for FT and T5-XXL (11B) for PT. Chinese tasks use CPM-2 (11B) for PT and mT5 sizes for FT §4.1. CB is scored by F1; all other columns are accuracy tab. 4.

English (T5-XXL, 11B for PT; tab. 4)

Chinese (CPM-2, 11B for PT; tab. 4)

More than five labels, single-text tasks (8 examples per label; tab. 5)

Full training data (3 seeds, validation set; tab. 6)

Headline numbers, author-reported: on SST-2 with 32 examples, PPT (T5-XXL) scores 93.5 against 91.4 for FT and 70.5 for vanilla PT tab. 4. On SST-5, PPT scores 50.2 against 40.6 for FT tab. 4. On the Chinese ChnSent set, PPT (CPM-2) scores 90.1 against 86.1 for FT on CPM-2 tab. 4. The paper says PPT beats FT on all Chinese datasets and most English ones, and that PPT variance is low on most datasets §4.2.

Cost, from the appendix: a single optimisation step on SST-2 takes 4,416 ms and 259 GB of GPU memory for FT, against 794 ms and 72 GB for PT. Pre-training is done once and its prompts are reused across tasks app. E tab. 10.

Paper vs code

Official repository: thu-coai/PPT, linked from the abstract, MIT licence (Copyright 2021 Yuxian Gu) LICENSE. Commit read: dad6f2da91bc354999ba6ab8571b484417e3d0a1.

• Base model name. The paper says T5-XXL and cites the original T5 §1 §4.1. The README says the base model is "T5.1.1" from the T5 v1.1 XXL checkpoint README.md. Treat these as possibly different checkpoints until someone confirms which one produced the numbers.
• Pre-training encoder length. The appendix says 512 app. C. The scripts use 512 for NSP pretrain_nsp.sh, 368 for single-sentence pretrain_cls.sh, 1024 for NSS pretrain_nss.sh and 768 for unified NSS pretrain_nss_uni.sh.
• Pre-training batch. The appendix says 256 app. C. The scripts set per-rank batch 16 with gradient accumulation 4 (NSS, unified NSS) or 64 with accumulation 1 (NSP, CLS) pretrain_nss_uni.sh. The script sets model-parallel size 4 pretrain_nss_uni.sh and NUM_NODES=2 pretrain_nss_uni.sh. Its launch line passes no --nnodes pretrain_nss_uni.sh. By arithmetic (derived, not run), two nodes of eight GPUs give four data-parallel ranks, so 16 × 4 × 4 = 256 and 64 × 1 × 4 = 256, matching the paper. One node gives 128. The paper does not state the node count.
• Downstream batch. The paper says 16 app. C. The scripts set per-rank 8 pt.sh on one node with model-parallel size 4 pt.sh, so two data-parallel ranks give 16 (derived, not run).
• Pre-training prompt initialisation. The paper does not say how the 100 pre-training slots are initialised. The released config sets init_scratch to true with empty init_tokens pretrain.json pretrain.json. The code pads the list with the <mask> token id, so every slot starts from the <mask> embedding [code: pretrain.py#L435-L437] transformer_enc_dec.py. This assumes encode("") returns nothing, which I did not check.
• Downstream initialisation. The downstream PPT config sets init_scratch to false ppt.json, and the checkpoint loader overwrites the encoder prompt with the saved pre-trained prompt [code: utils.py#L325-L328]. The paper's "use P_i as the initialisation" §3.1 is consistent with this.
• Sentence-pair label mix. The paper describes the three labels but not how often each occurs. The code draws each label with probability one third [code: tools/preprocess_pretrain_data_nsp.py#L92-L115]. The verbalizer tokens are given as ids 150, 2087 and 4273. The code does not name the words, and I did not verify these ids against the tokenizer.
• Length filters. The appendix says to drop sentences under 5 tokens and pairs whose length ratio exceeds 100 app. C. The code drops pairs whose ratio is below 0.01 or above 100, a two-sided version of the same rule [code: tools/preprocess_pretrain_data_nsp.py#L117-L118].
• Unified sample lengths. The paper text gives query truncation at 389 tokens and options at 86 app. C. The table gives 400 down to 150 for the query and 50 down to 20 for options by option count tab. 9. The code's length table matches the table preprocess_pretrain_data_nss_uni.py. Use the table.
• Downstream unified template. The pre-training template in equation 4 is not the one the downstream unified SST-2 dataset uses. That one adds the words "The correct one is" and separator token 58 [code: data_utils/downstream/SST2Datasets.py#L78]. The paper says downstream templates are "simple" §3.2.1.
• Single-sentence pseudo-label thresholds match the appendix app. C [code: tools/infer_roberta.py#L220-L227].
• Evaluation. Prediction is the argmax over the whole vocabulary at the label position, and a prediction counts as correct only if it equals the gold label token train.py train.py. The paper does not say whether predictions are restricted to the verbalizer words.
• Test logging. When --do-eval-while-valid is set, test accuracy is computed at every evaluation step and written to the same log line as dev accuracy [code: train.py#L143-L150]. Checkpoints are kept by dev accuracy train.py. Report the dev-selected number, not the best logged test number.
• Released data and prompts are on a Google Drive link in the README, not in the repository README.md README.md [code: README.md#L88]. I did not download them, so I have not checked that they match the reported numbers.
• Dependencies. The code needs the CUDA 10.2 toolkit README.md, Apex, and DeepSpeed v0.3.9 with two patched files from the repository [code: README.md#L30].

Pitfalls

• Initialisation is the central change. Random prompts give the high variance in tab. 1 and tab. 4. The gain depends on loading the pre-trained prompt downstream rather than reinitialising it utils.py.
• Pre-training gives three task-group prompts plus one unified prompt. Do not mix them. The mapping from task to prompt file is in the Method section.
• Pre-training cost. Pre-training runs for up to 200,000 steps on about 10 GB of text per format, with 4-way model parallelism and fp16 app. C [code: scripts/pretrain/pretrain_nss_uni.sh#L24]. Wall-clock time is not stated in the paper.
• Learning rates differ by a factor of ten. Pre-training uses LR 0.1 with inverse-square-root decay; downstream tuning searches 5e-3 to 5e-2 with a constant schedule app. C. The paper says PT needs a much larger LR than FT app. C.
• Few-shot splits are not reproducible from the repo. The 32 + 32 samples are in the Google Drive download, and the sampling code is not in the repository. Variance across seeds is large, so compare means with the reported standard deviations.
• Test is the original validation set. The paper uses it as the test set because the true test sets are not public §2 app. A. Do not compare with numbers from other papers that used a different held-out split.
• Verbalizer and pattern matter. On SST-2 with PT and the same pattern, the verbalizer dog/cat gives 60.0 and good/bad gives 87.6 tab. 1. Use the verbalizer and pattern from the paper or the repository, not your own.
• Unified PPT and the MC prompt depend on option counts. The count is sampled uniformly from 2 to 16 during pre-training app. C [code: tools/preprocess_pretrain_data_nss_uni.py#L73]. A downstream task with more than 16 options is outside the paper's tested range.
• Single-sentence pre-training depends on the RoBERTa labeller. Its confidence thresholds set the label balance app. C. The paper notes that this method does not transfer easily to new single-text domains and label counts, which is why unified PPT exists §3.2.3.
• Decoder prompt is empty. The decoder prompt length is 0 [code: configs/prompt/ppt.json#L10]. Only the encoder side gets soft tokens.

Lineage

• Builds on: prompt tuning, which the paper cites as Lester et al. (bibliography key prompt_tuning) §1; prompt-oriented fine-tuning and patterns with verbalizers, cited as PET §1 §3.1; BERT's next sentence prediction §3.2.1; and the unified question-answering format of MultiQA and UnifiedQA §3.3.
• Compared against: full fine-tuning at several sizes; vanilla PT; hybrid PT with hard prompts, close to P-Tuning §2 §4.2 (/papers/2103.10385); and LM adaptation, the continued pre-training step from the PT work §4.2.
• Related but not compared: meta-tuning with hard prompts over a collection of QA datasets §3.3.
• Superseded by: not stated in the paper, and not known from this wiki.