The Prompt Report: A Systematic Survey of Prompt Engineering Techniques

In plain words

Prompting is how people instruct a generative AI model: you write text, and the model continues from it. Many people have invented ways to word these prompts, and the names and groupings differ from paper to paper. This survey gathered the papers on text prompting, filtered them with a mix of human and automated labelling, and arranged the techniques into a family tree of six groups.

The authors also ran one small test: a few of these techniques on a standard multiple-choice exam, using a single model. Treat that test as a sketch, not as a verdict on which technique to use. The card's pitfalls section explains why.

TL;DR

This is a survey, not a method. Its useful outputs are a shared vocabulary, a taxonomy of 58 text-based techniques grouped into six categories, and a pipeline for finding such papers. The one experiment is a sample of 2,800 MMLU questions run on gpt-3.5-turbo. The paper reports that Few-Shot CoT performed best there, but the accuracy values appear only in a bar chart. Use this card to find and name techniques, then read the original papers before implementing any.

Problem

Prompt engineering has many named techniques, but the names and categories vary across papers, and only some techniques are widely known among practitioners § Introduction (Sections Overview). The survey aims to give a shared vocabulary and a map of text-based techniques, so readers can see how methods relate and choose what to try § Introduction (Sections Overview). It does not establish that the techniques transfer across models or tasks: the authors state that their stance is observational and make no validity claims § Conclusions.

Method

Figures are cited by section, because figure numbers in the compiled chapter layout were not verified against the PDF.

Scope and definitions

• A prompt is an input that guides a generative model's output. A prompt template is a function with variables that becomes a prompt when filled § Introduction (What is a Prompt?, Prompt Template).
• Scope is limited to prefix prompts, not cloze prompts; to hard (discrete) prompts, not soft (continuous) ones; to techniques that involve no gradient-based training or fine-tuning; and to task-agnostic techniques § Introduction (Scope of Study).

Review pipeline

The review draws on arXiv, Semantic Scholar and the ACL Anthology, queried with 44 keywords about prompting and prompt engineering § Meta-Analysis (Systematic Review Process). Humans labelled a sample of 1,661 arXiv articles against four rules. The rules include papers only if they propose a new prompting technique, cover hard prefix prompts, and do not rely on backpropagated training; the fourth rule concerns masked frames for non-text modalities § Meta-Analysis (The Pipeline). An LLM then classified the remaining articles from their title and abstract. The combined human and LLM labels gave 1,565 papers from 4,247 unique records § Meta-Analysis (The Pipeline).

# Machine-assisted screening  [§ Meta-Analysis] (The Pipeline)
# Input:  keywords (44 search phrases), sources (arXiv, Semantic Scholar, ACL)
# Output: included papers (1,565 in the paper's run)

def screen_papers(keywords, sources, human_labels):
    # 1. Query each source with each keyword; merge duplicate titles
    #    [code: src/prompt_systematic_review/collect_papers.py#L100]
    records = dedupe_by_title(search(s, k) for s in sources for k in keywords)

    # 2. Drop titles on the authors' blacklist
    #    [code: src/prompt_systematic_review/collect_papers.py#L103-L105]
    records = [r for r in records if r.title not in blacklist]

    # 3. Keep only papers whose PDF text contains "prompt" (code only; the
    #    paper's pipeline text does not describe this step)
    #    [code: src/prompt_systematic_review/collect_papers.py#L128-L134]
    records = [r for r in records if contains_prompt(r.pdf_text)]

    # 4. Human labels on a sample: 1,661 articles, 300 double-annotated
    #    [§ Meta-Analysis] (The Pipeline)
    included = [r for r in human_labels if r.human_included]

    # 5. LLM rating of the unlabelled rest, from title and abstract
    #    [code: src/prompt_systematic_review/automated_review.py#L31-L45]
    for r in records_not_in(human_labels):
        rating = llm_rate(r.title, r.abstract)
        # 6. Keep the three top rating levels (exact string match)
        #    [code: src/prompt_systematic_review/collect_papers.py#L198-L202]
        if rating in ["highly relevant", "somewhat relevant", "neutral"]:
            included.append(r)

    return included

Taxonomy

The taxonomy places 58 text-based techniques into six top-level groups. Each technique sits in the single group the authors judged most relevant § Meta-Analysis (Text-Based Techniques). The table lists the names shown in the taxonomy figure. Subgroup headings such as "Exemplar Generation" are not techniques and are omitted.

Other parts of the survey cover multilingual and multimodal prompting, agents, evaluation, security and safety. The abstract counts 40 techniques for other modalities. This card does not summarise those chapters.

Benchmark protocol (MMLU)

The technique benchmark uses a sample of MMLU, with gpt-3.5-turbo as the model § Benchmarking (Technique Benchmarking). The paper's protocol is as follows.

# MMLU technique benchmark, as the paper describes it
#   [§ Benchmarking] (Technique Benchmarking)
# Input:  2,800 MMLU questions (20% of each category), gpt-3.5-turbo

def run_benchmark(questions, technique):
    # 1. Zero-shot baseline: 3 instruction phrasings x 2 question formats
    #    = 6 runs  [§ Benchmarking] (Comparing Prompting Techniques)
    # 2. Zero-shot CoT: try 3 thought inducers, keep the best one
    #    [§ Benchmarking] (Comparing Prompting Techniques)
    # 3. Few-shot and few-shot CoT: exemplars written by one author,
    #    3 phrasings x 2 formats
    #    [§ Benchmarking] (Comparing Prompting Techniques)
    # 4. Self-Consistency: best phrasing, 3 samples at temperature 0.5,
    #    majority vote  [§ Benchmarking] (Self-Consistency)
    # 5. All other runs use temperature 0  [§ Benchmarking] (Self-Consistency)
    # 6. Score by pattern-matching the answer letter A-D
    #    [§ Benchmarking] (Evaluating Responses)
    return accuracy(questions, technique)

Hyperparameters

Results

Author-reported. Setup: a systematic review of prefix, hard, task-agnostic prompting papers; the MMLU test uses gpt-3.5-turbo on a 2,800-question sample.

Paper vs code

The official repository is trigaten/The_Prompt_Report (MIT, copyright Sander Schulhoff), linked from the paper's footnotes at commit cd0ff19. Searched: the paper's footnote links and the repository it names. The MMLU protocol and the screening pipeline are in the repository; the human labels and the Self-Consistency runs are not.

• Keywords agree: the paper lists 44 keywords § Meta-Analysis (Systematic Review Process), and the code's list also has 44 entries keywords.py.
• Screening model agrees: the paper's footnote names gpt-4-1106-preview, and the code calls the same model collect_papers.py. The code sets no temperature, so the API default applies. The paper does not state one.
• Rating scale mismatch: the screening prompt asks for the label "neutrally relevant" automated_review.py, but the keep-list tests for the exact string "neutral" collect_papers.py. A paper labelled "neutrally relevant" would be dropped. The paper does not say which levels it keeps.
• Extra filters in code: the code drops papers without a downloaded PDF, and papers whose PDF text lacks "prompt" collect_papers.py, collect_papers.py. The paper's pipeline text does not describe these steps.
• MMLU sampling agrees: 20% per category, rounded up with a minimum of 1 benchmarking.py.
• Self-Consistency differs: the paper uses temperature 0.5 and 3 samples with a majority vote § Benchmarking (Self-Consistency). The repository's benchmark defaults to temperature 0.0 benchmarking.py and contains no majority-vote aggregation. The 0.5 setting was not found.
• Answer extraction: the code matches several phrases and falls back to a single parenthesised letter benchmarking.py. The paper describes the rule only loosely § Benchmarking (Evaluating Responses).

Pitfalls

• Scope is narrow on purpose. Soft prompts, cloze prompts, gradient-trained methods and task-specific techniques are outside the review § Introduction (Scope of Study). Their absence is not a finding about their quality.
• Recall of the screen. Search used 44 keywords across three sources, and the paper does not claim the set is exhaustive. The LLM screen had 75% recall on its 100-item validation set, so about one relevant item in four in that sample was missed by that step § Meta-Analysis (The Pipeline).
• Reported statistics are unusual. The text gives both Krippendorff's alpha and Cohen's kappa as 81%. Treat both figures as stated, not as a check.
• Citation counts are a proxy. Usage is inferred from how often other papers in the dataset cite a technique § Meta-Analysis (Prompting Technique Usage). Citation is not evidence of use or of effect.
• The benchmark is thin. One model, one 2,800-question sample, exemplars written by one author, and answers scored by pattern matching. One MMLU category was excluded because the model refused to answer those questions § Benchmarking (Technique Benchmarking). The per-technique accuracies are not in the text.
• Counts differ within the paper. The abstract and taxonomy give 58 text-based techniques, but the conclusion says the paper discusses "over 200" prompting techniques § Conclusions. Quote the 58 figure for the taxonomy only.
• Best-practice advice is thin. The abstract promises guidance for current chat models. The sections read for this card contain only general recommendations, in the conclusion § Conclusions. No model-specific advice was found.
• Exemplar order can matter a lot. The paper cites prior work reporting that exemplar order alone moved accuracy from below 50% to above 90% on some tasks § Meta-Analysis (In-Context Learning (ICL)). Test order when implementing any few-shot technique.
• Technique descriptions are summaries. Each entry is a short paragraph. Details such as thresholds, iteration counts and prompt wording are in the original papers, which this card does not summarise.
• Gradient-based optimisers are out of scope for the taxonomy. The prompt-engineering section mentions AutoPrompt, RLPrompt and DP2O, which use backpropagation or reinforcement learning, and treats them separately § Meta-Analysis (Prompt Engineering).

Lineage

• Builds on: the PRISMA reporting guideline for the review process § Meta-Analysis (The Pipeline).
• Compared against: not stated. The related-work section names other systematic reviews of prompt engineering, one of them for software engineering § Related Work.
• Superseded by: not known from content already in this wiki.