Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

In plain words

Classic text models learn one task at a time from labelled examples. This survey looks at a newer way of using language models that were first trained only to predict text. Instead of training a new output layer for each task, you rewrite the task as a sentence with a blank in it, such as "I love this movie. Overall, it was a ___ movie." The model fills the blank, and the filled word is mapped back to the task's label. The rewritten sentence is the prompt, and writing it well is most of the work.

The survey gives this idea a shared vocabulary and a map of the field. It covers how to design prompts, how to choose the words that fill the blank, how to combine several prompts, and how to tune the model or the prompt when some labelled data exists. It also lists applications and open problems. It runs no new experiments.

TL;DR

• Survey and formalism, not a new method: it reformulates supervised NLP as filling a template slot in a pre-trained LM's input §1 §2.2.
• Three prediction steps: prompt addition, answer search, answer mapping §2.2.
• The organising taxonomy has prompt engineering (shape and manual/automated search), answer engineering, multi-prompt learning and training strategies §2.3.
• Use it for vocabulary, a method map and pointers; no reported numbers of its own §11.

Problem

Fully supervised NLP needs a labelled dataset for each task, and such datasets are often too small §1 §2.1. Pre-train-then-fine-tune reduced this need but still required task-specific objectives and parameters §1. The survey frames prompt-based learning as a response to both: the pre-trained LM is used as-is, and the task is adapted through the input text §1 §2.2. The survey states the cost of this approach: it needs prompt engineering, finding a prompt that makes the LM solve the task §1. The paper gives no quantitative threshold for when the approach is preferable.

Method

The survey defines the paradigm in three parts. Figures and notation are given in the paper; the wording below is ours.

Notation §2.2 (Table of terms in §2.2):

• Input x, output y, and a pre-trained LM P(·; θ) §2.1 §2.2.
• A prompting function f_prompt(x) builds the prompt x' from x. It normally applies a template with an input slot [X] and an answer slot [Z], then fills [X] with x §2.2.1.
• A filled prompt f_fill(x', z) puts a candidate answer z into [Z]. An answered prompt uses a true answer z* §2.2.
• Answer z comes from a set Z of permissible answers, mapped to the output y §2.2.2 §2.2.3.

Basic prompting predicts ŷ in three steps §2.2:

1. Prompt addition §2.2.1: build x' = f_prompt(x) from a template. The slot [Z] can sit in the middle of the text (cloze prompt) or at the end (prefix prompt). The number of [X] and [Z] slots can vary §2.2.1.

2. Answer search §2.2.2 eq. 1: take the answer that scores best under the LM, using argmax or sampling.

   \hat{z} = \underset{z \in \mathcal{Z}}{\text{search}}\; P(f_{\text{fill}}(x', z); \theta) eq. 1

3. Answer mapping §2.2.3: map ẑ to ŷ. This is the identity for generation tasks, and a label map for classification (e.g. several sentiment words mapping to one class).

# Basic prompting: predict y-hat from x  [§2.2]
# Inputs: x (input text), theta (LM parameters), Z (permissible answers)
# Output: y-hat (the predicted output)

def predict(x, prompting_function, lm, Z, answer_to_output):
    # 1. Prompt addition: fill [X] with x, keep [Z] open  [§2.2.1]
    prompt = prompting_function(x)
    # 2. Answer search: best-scoring z under P(.; theta)  [§2.2.2] [eq. 1]
    answer = argmax(Z, lambda z: lm.score(fill(prompt, z)))
    # 3. Answer mapping: turn the answer into the output  [§2.2.3]
    return answer_to_output(answer)

Design axes the survey uses §2.3:

• Choice of pre-trained model: left-to-right, masked, prefix and encoder-decoder LMs §3 (family overview in the typology figure §2.3).
• Prompt engineering §4: prompt shape (cloze or prefix) §4.1; manual template design §4.2; automated template learning, split into discrete §4.3.1 and continuous prompts §4.3.2.
• Answer engineering §5: answer shape (token, span, sentence) and answer space design, manual or automated, discrete or continuous §5.
• Multi-prompt learning §6: prompt ensembling, augmentation, composition and decomposition §6.
• Training strategies §7: whether the LM and the prompt parameters are tuned §7.2.

Continuous prompts §4.3.2: the template may use vectors in the embedding space rather than natural-language words, and its parameters are trained apart from the LM. The survey's example is prefix tuning, which prepends trainable vectors and keeps the LM frozen. Its objective is the log-likelihood of y given x and the prefix eq. 2:

\max_{\phi} \log P(\bm{y} \mid \bm{x}; \theta; \phi) = \max_{\phi} \sum_{y_i} \log P(y_i \mid h_{<i}; \theta; \phi) eq. 2

Here φ denotes the trainable prefix M_φ. For time step i, h_<i is copied from M_φ when i is inside the prefix and computed by the LM otherwise §4.3.2.

Training strategies §7.1 §7.2: the survey lists five strategies, distinguished by three questions: whether the LM is tuned, whether extra prompt parameters exist, and whether those parameters are tuned. The five are promptless fine-tuning, tuning-free prompting, fixed-LM prompt tuning, fixed-prompt LM tuning, and prompt+LM tuning §7.2. The survey's table of these strategies lists their examples §7.2.

Hyperparameters

This is a survey with no experiments of its own, so the paper states no training hyperparameters for its own work.

Results

Author-reported. The paper runs no new experiments. Its quantitative content is a meta-analysis of the surveyed works, shown as figures; the numbers are not given in the text §11. Setup: the counts are based on the works in the two large comparison tables of the paper §11 (figure caption).

Paper vs code

No official code for the method, because the paper is a survey. The paper links a companion website, http://pretrain.nlpedia.ai/, which has a paperlist, a timeline and slides. The project page also links the GitHub repository github.com/pfliu-nlp/NLPedia-Pretrain. Its owner is the account of author Pengfei Liu, but GitHub reports no license, and it holds the website's resources rather than implementation code. We therefore do not record it in code, code_commit or code_license, and we use nothing from it. Searches: the paper text and abstract, the linked project page, and a GitHub repository search for "pretrain prompt predict survey" (no results). No discrepancies between a paper and code are checked here.

Pitfalls

• No numbers of its own. Every number in the paper comes from other works, or from its meta-analysis figures. Do not treat any number here as a result of this paper. Its counts are over the works it lists, at the time of writing (2021, v1) §1 §11.
• Zero-shot is not clean. Many prompt-engineering methods use annotated samples to build or validate prompts, even though the downstream model is not trained on them. The survey notes this is arguably not true zero-shot learning §7.1.
• Continuous prompts are sensitive to initialisation in low-data settings, according to the prefix-tuning results the survey cites §4.3.2. Initialising from a discrete prompt can give a better start §4.3.2.
• Answer search can be argmax or sampling §2.2.2. The paper does not say which to use for a given task. For classification the answer set Z is usually small; for generation it can be the whole vocabulary §2.2.2.
• Slot count is flexible §2.2.1. The number of [X] and [Z] slots can change with the task, and no rule for choosing them is given.
• Notation vs usage. The paper uses "template" for the text with empty slots and "prompt" for the text with [X] filled. Read "filled prompt" and "answered prompt" in the sense of §2.2 §2.2.
• Equation and figure numbers here follow the arXiv v1 TeX order (equation numbers 1 and 2 are counted in order). Table and figure numbers were not checked against the compiled PDF, so the card cites sections for them.
• Dated field. The survey was written in 2021 and describes the state of the field then §1. Later methods are not covered.

Lineage

• Builds on: pre-trained LMs such as GPT, BERT, BART and T5 §3 §2.3.
• Compared against: pre-train-then-fine-tune and fully supervised learning §1 §2.1.
• Superseded by: not stated in this card.