Prompt engineering

Overview

Prompt engineering is choosing the text a model is conditioned on: the task instruction, any worked examples, the output format, cues that ask for intermediate reasoning, and sometimes a persona or a required layout. No weights change. The shared premise is that the model continues the text it is given, so a good prompt is one that makes the wanted continuation the likely one 2102.07350 §4.1. The variants differ in what the prompt specifies (the task, the examples, the format, the steps, the role) and in how the output is read back. A survey of the field names the same three steps: prompt addition, answer search and answer mapping 2107.13586 §2.2. A 2021 NLP survey lists prompt-based learning as one of three paradigms for large pre-trained models, next to fine-tuning and text generation 2111.01243 §1 2111.01243 §3. A 2023 review of LLM prompting treats prompt design as the main lever for accuracy and relevance when the model is fixed 2310.14735 §1.

Most of the evidence here is for classification, translation, arithmetic word problems, clinical diagnosis questions, multiple-choice exams, layout generation, survey-style opinion questions, dialogue sub-tasks, medication extraction from discharge summaries, mental-health vignettes, redaction of clinical notes, plain-language rewrites of radiology reports, tweet and news annotation, finance text tasks, multi-agent software tasks, data cleaning and entity matching, travel mode choice and cultural value surveys. Most results come from a few models with one prompt per task. Later additions cover perplexity-ranked zero-shot prompts, codebook prompts for coding open-ended answers, persona prompts scored for marked words, perturbed instruction wording, and staged prompts for robot teams. The soft-prompt results are for fact probing on masked language models and for few-shot SuperGLUE with tuned models. Treat the numbers as a map of what worked in 2021-2025 models, not as a ranking that carries over to new ones.

Variants

Zero-shot instructions

The prompt is a one-line instruction that names the task and the answer form, followed by the input and no examples 2302.06476 §3. Applied to gpt-3.5-turbo across 20 datasets, this beats text-davinci-003 on most arithmetic, NLI, dialogue and sentiment sets, and trails it on summarization, NER and several commonsense sets 2302.06476 §4.2. Card: Is ChatGPT a General-Purpose NLP Task Solver?.

Zero-shot prompting also works on data-cleaning questions, but unevenly. With GPT-3 175B and no examples, imputing a missing value beat the HoloClean baseline on both datasets tested (70.9 against 33.1 on Restaurant; 84.6 against 16.2 on Buy). The same model scored 6.9 F1 on Hospital error detection and 0.0 on Adult, and zero-shot entity matching trailed the same model with ten examples 2205.09911 tab. 1 2205.09911 tab. 2. The authors ran one closed model snapshot, so these are not results for other models 2205.09911 §4.1. Card: Can Foundation Models Wrangle Your Data?.

Zero-shot prompts also carry structured records. On travel mode choice, GPT-3.5 with no examples reached 0.586 accuracy, above neural networks trained on 10 samples (0.462) 2312.00819 tab. 3. The record and its rules are in Variants below.

Wording alone changed results in one translation setup. On WMT'14 French to English, a zero-shot "French: ... English:" format scored 33.3 BLEU on Curie, the same as the 10-shot version of that format (33.4), and above a reproduced 10-shot GPT-3-style prompt (27.9). On Babbage the zero-shot format (23.5) was below that 10-shot baseline (25.1) 2102.07350 tab. 1. The paper's reproduced baselines are well below the published GPT-3 numbers, so the gap is against the authors' runs 2102.07350 tab. 1. Card: Prompt Programming for Large Language Models.

A prompt should also rule out the continuations you do not want. A bare "Translate French to English:" often continued in French; a frame where the translation is the only sensible continuation worked best [arxiv:2102.07350 §4.5]. This is described qualitatively, with no measured effect.

Task description in English or the target language

For ChatGPT, each zero-shot query is one task description, then the input, then a format note. The study covers seven tasks in 34 to 37 languages, and each example gets either an English description or one in the target language 2304.05613 §3. On average, the English description scored higher on POS tagging, NER, NLI and QA 2304.05613 §4 2304.05613 §5 2304.05613 §7 2304.05613 §8. The target-language description scored higher on relation extraction and commonsense reasoning 2304.05613 §6 2304.05613 §9. The averages hide large single-language gaps. On NLI, Hindi scored 48.8 with an English description and 5.6 with a Hindi one 2304.05613 §7. Card: ChatGPT Beyond English.

Few-shot demonstrations

The prompt holds k input-label pairs before the test input [arxiv:2202.12837 §3]. On 26 classification and multi-choice datasets with six decoder-only models, replacing the gold labels with uniform random labels cost 0-5 points absolute in nearly all of the 12 model and scoring combinations 2202.12837 §4.1. The gains depend more on the other parts of the demonstrations: the label space, the input text, and the pairing format 2202.12837 §5. Replacing the inputs with unrelated text cost 3-16 points for the models where it was measured 2202.12837 §5.1. Random English words instead of the label words cost direct models 5-16 points and channel models 0-2 points 2202.12837 §5.2. Inputs without labels, or labels without inputs, land close to or below the no-demonstration baseline 2202.12837 §5.3.

Examples can also hurt. A one-shot version of the colon format dropped to 18.0 BLEU on Babbage and 27.6 on Curie, which the authors attribute to the example being read as part of the same passage 2102.07350 §3.2 [arxiv:2102.07350 tab. 1]. Card: Rethinking the Role of Demonstrations. This is the card to read before building an in-context learning baseline, see in-context learning.

Few-shot is not automatically the stronger setting for opinion distributions. On OpinionQA, a five-example prompt was worse than zero-shot question prompting on all four models tested 2502.16761 tab. 1. Card: SubPOP.

Serialized rows and task questions for data cleaning

Each table row is written as "attribute: value" text, and each task gets its own fixed question. For entity matching the two records are introduced as products and the question asks whether they are the same. For imputation the known attributes come first and the missing attribute's name ends the prompt. For error detection the question names one cell 2205.09911 §3.1 2205.09911 §3.2. Attributes that do not help are dropped, chosen by experiment 2205.09911 §3.1. The paper gives no question templates for data transformation or schema matching 2205.09911 §3.2.

Examples change the scores a lot. With ten demonstrations, GPT-3 175B reached 100 F1 on Beer and 98.2 on iTunes-Amazon, against 78.6 and 65.9 with none 2205.09911 tab. 1. On the transformation and schema tasks, three examples lifted StackOverflow from 32.7 to 65.3 and Bing-QueryLogs from 24.0 to 54.0 2205.09911 tab. 3. The authors also report that a person picked the examples after reading validation errors, with up to one hour of work per task, and that this beat random examples by 14.7 F1 on average across three entity-matching sets 2205.09911 §3.3 2205.09911 tab. 4. The gap ranged from 8.9 points on Beer to 23.7 on Walmart-Amazon.

Small wording changes moved scores in both directions. Replacing "same" with "equivalent" in the entity-matching question raised Walmart-Amazon from 88.9 to 100 and lowered Beer from 100 to 96.3 and iTunes-Amazon from 98.2 to 84.7 2205.09911 tab. 4. The paper's 9.4-point average prompt effect is a mean of absolute differences; with signs kept, the three gaps average about 2 points 2205.09911 tab. 4. Read per dataset, not as one number. The paper also counts some ties with the baselines among its state-of-the-art results 2205.09911 §4.2. Card: Can Foundation Models Wrangle Your Data?.

Demonstration order

Order is a separate lever from which examples are chosen. With four-shot demonstrations on SST-2, the 24 orders of one set spread widely: some reach over 85% and others sit near 50% 2104.08786 fig. 1. One order's accuracy also fell from 88.7% with GPT2-XL (1.5B) to 51.6% with GPT2-Large (0.8B) 2104.08786 §2. The orders that work well for one model size correlate weakly with those for another: the reported Spearman correlation between 2.7B and 175B is 0.05 2104.08786 fig. 4.

The paper's selection needs no labels. For each candidate order, the model writes unlabelled example inputs at temperature 2. Each order is then scored by the entropy of the labels it predicts on those inputs, and the top four are kept 2104.08786 §3 2104.08786 §4. Averaged over 11 text classification tasks, the authors report a 13% relative gain over unselected orders for this score, and 9.6% for a per-input variant 2104.08786 §5 2104.08786 tab. 2. Entropy ignores which label wins, so a split of 80/20 toward either label scores the same, and the paper does not discuss this 2104.08786 eq. 3. Card: Fantastically Ordered Prompts.

Reasoning cues

A cue such as "Let's think step by step" makes the model write intermediate steps before a final answer. With two stages, the rationale is fed back with an answer trigger 2302.06476 §3. For ChatGPT, the cue lets it beat GPT-3.5 on all six arithmetic sets (five of six without it), but lowers CommonsenseQA from 73.7 to 71.5 and StrategyQA from 61.1 to 55.5 2302.06476 tab. 2 2302.06476 tab. 3. The full treatment, with other CoT papers, is on the chain-of-thought page.

The cue does not always work for a model that does not get a worked example. On GSM-IC, code-davinci-002 with the trigger "Let's think step by step:" scored 29.0% micro accuracy on the perturbed variants, while chain-of-thought with a worked exemplar scored 72.4% on the same variants 2302.00093 tab. 3. Card: Large Language Models Can Be Easily Distracted by Irrelevant Context.

On a multiple-choice exam benchmark, the effect of chain-of-thought depends on the question set. GPT-4o scored 72.6% with five-shot chain-of-thought on MMLU-Pro and 53.5% with direct answers, a gap of 19.1 points. On MMLU the same comparison gave 88.7% and 87.2%, a gap of 1.5 2406.01574 tab. 3. Card: MMLU-Pro. The paper's own numbers are author-reported.

A prompt can also ask for a named style of reasoning. Four few-shot prompts for diagnosis (differential, intuitive, analytic and Bayesian reasoning) were compared with a traditional step-by-step prompt, on free-response questions from a modified MedQA USMLE set 2308.06834 p. 1 2308.06834 tab. 2. For GPT-4 (497 questions), the four styles scored 72-78% against 76% for the baseline, and no difference was significant 2308.06834 p. 3. For GPT-3.5 (518 questions), the differential, analytic and Bayesian prompts were 8, 6 and 4 points below the baseline, and the intuitive prompt was 1.7 points above it 2308.06834 p. 3. The prompts were picked on a 95-question development set with GPT-3.5 2308.06834 p. 4. Card: Diagnostic Reasoning Prompts. See also self-consistency.

A review also quotes a result for "golden" chain-of-thought, where the prompt supplies ground-truth reasoning steps. On detective puzzles with GPT-4, it reports a solve rate of 83% against 38% for standard chain-of-thought. The review takes the figure from a cited paper and does not re-run it 2310.14735 §3.1.2. Card: Unleashing the potential of prompt engineering.

Prompt style and sampling temperature were tested together on GPT-3.5. Five one-shot styles (baseline, domain expert, self-recitation, chain of thought and a composite of these) were run at temperatures from 0.0 to 1.0 on a 100-question multiple-choice exam. The paper says chain of thought scored best and used it for its temperature sweeps, and the temperature test showed no effect for any style, with Kruskal-Wallis p-values from 0.996 to 1.000 2402.05201 §3.1 2402.05201 tab. 4. Each prompt asked for a fixed answer line, and an answer that did not match counted as wrong. For Llama 2 7B, 39% of answers broke that format 2402.05201 §3.1 2402.05201 app. A. Card: The Effect of Sampling Temperature on Problem Solving in Large Language Models. The temperature side is covered on the decoding page.

Robustness to irrelevant context

Adding one irrelevant sentence to a problem can change the answer even when the sentence does not change the solution. GSM-IC adds one such sentence to each of 100 easy GSM8K base problems, giving 58,052 variants 2302.00093 §3.1. For code-davinci-002, chain-of-thought scored 95.0% on the base problems and 72.4% micro accuracy on the variants 2302.00093 tab. 2 [arxiv:2302.00093 tab. 3].

Three prompt-level changes helped on that set 2302.00093 §5:

• An instruction before the exemplars, "Feel free to ignore irrelevant information", raised chain-of-thought from 72.4% to 77.8% micro accuracy 2302.00093 tab. 3 2302.00093 §4.3.
• An exemplar that itself contains an irrelevant sentence raised it to 76.8% 2302.00093 tab. 3. On GSM8K dev the same change gave no gain (59.3 against 60.3 without it) 2302.00093 tab. 5.
• Self-consistency, 20 samples at temperature 0.7 with a majority vote, raised least-to-most prompting from 77.5% to 93.4% micro accuracy. Macro accuracy, which counts a base problem only if all its variants are right, reached 45.0% at best 2302.00093 tab. 3 2302.00093 §5.1.

The paper's text says self-consistency adds "more than 11 points" of micro accuracy. Its table gives 15.9 points for least-to-most. Card: Large Language Models Can Be Easily Distracted by Irrelevant Context.

Perturbed wording stress test

Prompt wording can be attacked while the input stays clean. PromptRobust applies seven black-box text attacks to the instruction on four prompt types (zero-shot or few-shot, each task-oriented or role-oriented) across 13 datasets 2306.04528 §2.2 2306.04528 tab. 1. The families are character typos, word swaps with synonyms, an appended distractor sentence, and round trips through other languages, with the other-language prompts written by GPT-4 2306.04528 §2.2. Word-level swaps cost the most: an average performance drop rate of 0.31 for TextFooler and 0.33 for BertAttack 2306.04528 tab. 2. The introduction gives 39% for the same effect 2306.04528 §1, so use the table figure. Few-shot prompts were more robust than zero-shot ones on almost all datasets 2306.04528 tab. 4. GPT-4 and Flan-UL2 had the lowest average drops, both 0.08 [arxiv:2306.04528 tab. 3]. Transfer of the attacked prompts between models was weak and mixed 2306.04528 tab. 6. The numbers come from sampled subsets, and the paper does not state decoding settings 2306.04528 §3. Card: PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts.

Output format and layout

LayoutGPT asks a text-only model to write each object as one CSS-style rule. The category is the selector, and the size and position are declarations. The values are normalised so the largest canvas side maps to 256 pixels, and retrieved examples are placed so the most similar one comes last 2305.15393 §3.2 2305.15393 §3.3 2305.15393 fig. 2.

On 2D spatial prompts with GPT-3.5 and GLIGEN rendering, layout accuracy was 55.12 with neither the instruction nor the CSS format, and 82.54 with the instruction, the CSS format and normalisation together 2305.15393 tab. 4. The parts do not help equally. Normalisation alone lowered 2D layout accuracy to 44.10 2305.15393 tab. 4, but on 3D bedroom scenes the same normalisation lowered out-of-bounds furniture from 55.32% to 46.57% 2305.15393 tab. 6. Retrieved examples matter more in 3D: random examples raised bedroom out-of-bounds to 85.58% from 43.26%, but change 2D accuracy only slightly 2305.15393 app. C.1 2305.15393 app. B. Card: LayoutGPT.

Structured record with domain rules and JSON answer

A trip choice can be sent to the model as one structured prompt with no training examples. The prompt has six parts in a fixed order: the task and the allowed options; the travel time and cost of each option as a nested dictionary; the traveller's attributes written as sentences, not dummy variables; domain sentences that compare the options (which is cheapest or fastest, and by what percentage); and a JSON answer with a "prediction" and a "reason" key 2312.00819 §3.3 2312.00819 §3.3.6 2312.00819 app. A. The model runs at temperature 0 [arxiv:2312.00819 §4.3]. The paper asks for JSON so results can be read by code, but it does not describe the parser or what happens when the output is malformed 2312.00819 §3.3.5.

The same record can also feed a classic model. The paper embeds the prompt text and fits a multinomial logit, random forest or neural network on the vectors 2312.00819 §3.2 2312.00819 §3.4. With 10 training samples, the embedding raised the neural network from 0.462 to 0.532 mode accuracy 2312.00819 tab. 3. The embedding model's dimension is not stated 2312.00819 §3.1.

In the ablation, removing the rules that guide the reasoning cost GPT-3.5 5.8% of accuracy and GPT-4 0.8% 2312.00819 tab. 5. Removing the domain sentences changed neither much. Card: Large Language Models for Travel Behavior Prediction.

Role profiles and fixed documents

MetaGPT gives each agent a profile with a name, role, goal and constraints, and asks roles to hand over fixed documents, such as a requirements document or an interface design, rather than free-form chat 2308.00352 §3.1 2308.00352 §3.2. The authors argue that free chat distorts information over several rounds 2308.00352 §1.

Their ablation varies the team, not the wording alone. Going from one engineer to four roles lowered human revisions from 10 to 2.5 and raised executability from 1.0 to 4.0, at about $0.92 to $1.39 per run 2308.00352 §4.4 2308.00352 tab. 3. Card: MetaGPT.

Task-name prefixes for a shared model

Here one text-to-text model handles several dialogue sub-tasks. Each training sample starts with a short task prompt, such as "translate dialogue to belief state:", followed by the earlier turns, and the target is the expected text for that task 2109.14739 §3.2 2109.14739 eq. 1. Pre-training mixes eleven dialogue corpora, each labelled for only some of the four sub-tasks, and a sample is made only for the labels a corpus has 2109.14739 §3.1 2109.14739 tab. 1. The paper's prompts are fixed strings, and it reports no comparison of different wordings. On MultiWOZ 2.0 end-to-end, the base model reaches a combined score of 102.92, and at 1% of the training data it reaches 76.41, against 71.44 for the small model 2109.14739 tab. 2 2109.14739 §4.1.4. Card: Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System.

Extraction with few-shot pre-labels and human refinement

A few-shot prompt is run on each chunk of a discharge summary, the model writes field values with character positions, and a resolver turns that text into structured entries. Unparsable output is logged and dropped [arxiv:2312.02296 §5] 2312.02296 app. A.1. The authors wrote 10 IOB-style prompts and 5 direct-chunk prompts, tuned them on a development set, and selected by F2, which weights recall above precision 2312.02296 §5.4 [arxiv:2312.02296 app. A.2] 2312.02296 app. A.3. On 216 i2b2 2009 test documents, PaLM 2 without tuning gave a vertical F1 of 0.804 against 0.838 for the human base labels 2312.02296 tab. 2. After expert refinement, the LLM base reached 0.907 and the human base 0.910 2312.02296 tab. 2. Mean rater time per document fell from 26.67 to 11.32 minutes, a reduction of 58% [arxiv:2312.02296 §6.1]. Card: LLMs Accelerate Annotation for Medical Information Extraction.

Formatting alone can change a score. FormatSpread defines prompt formats with a grammar of separators, spacing, label casing and enumeration styles, and treats formats generated by the same rules as equivalent 2310.11324 §3.1. On 53 Super-NaturalInstructions tasks, the median spread between the best and worst of 10 sampled formats was 7.5 accuracy points. For LLaMA-2 settings, 20% of tasks had a spread of 15 points or more 2310.11324 §4.2. The authors report that the spread does not shrink with model size, shot count or instruction tuning 2310.11324 §4.2. Card: Quantifying Sensitivity to Spurious Features in Prompt Design.

Searching all formats is expensive, so FormatSpread finds the best and worst format with Thompson sampling inside a fixed budget of model calls. At 51,200 evaluations, its gap to the true spread was within 1 point, against within 4 points for naive sampling and within 11 points for UCB 2310.11324 §4.5.

The same question set can look much less sensitive. Across 24 prompt styles, scores on MMLU-Pro moved by about 2% in general, with a maximum of 3.74%. On MMLU, the same kind of change moved scores by 4-5% in general and up to 10.98% 2406.01574 §6.3. Card: MMLU-Pro.

ProSA measures sensitivity per question instead of per dataset. For each question it averages the absolute score gaps over every pair of prompt variants, then averages those values over the dataset 2410.12405 eq. 1 2410.12405 eq. 2. It uses 12 variants per question on four objective sets 2410.12405 §3.1. The paper reports that few-shot examples lower this score for all four Qwen1.5 models tested, with the largest drop from zero to one shot 2410.12405 §3.4. On the subjective benchmarks, Llama3-70B-Instruct scored 0.016 on AlpacaEval 2.0 and 0.258 on Arena Hard Auto, against 0.167 and 0.275 for the reference pair of responses 2410.12405 tab. 2. Card: ProSA. The paper's own numbers are author-reported.

Persona and steering prompts

A steering prompt describes a person or group and asks the model to answer as them. Turing Experiments prompt a model zero-shot with a named participant and run the prompt for many names, then compare the outcomes with published human results 2208.10264 §1 2208.10264 §2. The recent models reproduced the Ultimatum Game, garden-path and Milgram trends. On a wisdom-of-crowds estimation task, the newest models answered with exact values, which people do not do 2208.10264 §7 2208.10264 tab. 4. All results are author-reported. Card: Using LLMs to Simulate Multiple Humans.

The authors ask that each prompt be tuned until its validity rate, the share of completions in the required form, is "sufficiently close to 1", before any hypothesis is tested 2208.10264 §2. They do not give a threshold. Validity ranged from 51.0% to 99.5% across the models and tasks in their table 2208.10264 tab. 1.

Steering alone rarely matched the spread of human answers in survey data. On OpinionQA, the best zero-shot prompt reached a Wasserstein distance of 0.138 to 0.173 across four base models, and fine-tuning reached 0.094 to 0.106 2502.16761 tab. 1. Chat models did worse than base models on this task: zero-shot question prompting gave 0.308 for Llama-2-7B-chat against 0.173 for the base model 2502.16761 app. C.2. Card: SubPOP.

A persona can also be a demographic profile rather than a named person. The World Values Survey study builds personas from region, sex, age, social class, education and marital status, with 303 matched personas per country 2402.13231 §4.2 2402.13231 tab. 1. Each of 30 survey questions is asked in four paraphrases, sampled five times at temperature 0.7, and the majority answer is kept 2402.13231 §4.5. Answers are scored by exact match (hard) and by a partial-credit error on ordinal items (soft) 2402.13231 eq. 1. For GPT-3.5 in English, the soft score is 48.61 on the Egypt survey and 64.86 on the US survey 2402.13231 tab. 3. Prompting in Arabic raised GPT-3.5's Egypt score by about 3 points 2402.13231 tab. 2. The paper's Anthropological Prompting variant lifts GPT-3.5's Egypt soft score from 0.4834 to 0.5102 against a five-sample vote 2402.13231 tab. 4. The authors also report that alignment is uneven across social classes and education levels, but they give those values only in a figure 2402.13231 §5.3 2402.13231 fig. 3. Card: Investigating Cultural Alignment of Large Language Models.

Backstory personas for survey simulation

A persona can also be built from a respondent's own answers. The "silicon sample" method writes a first-person backstory for each survey respondent, from fields such as age, party and church attendance, and asks GPT-3 (Davinci, 175B) to continue it 2209.06899 §4 2209.06899 app. 3. For the 2016 vote, the prompt ends "In 2016, I voted for", and the vote is read from the next-token probabilities 2209.06899 app. 3. Across the whole sample, the tetrachoric correlation with the real vote is 0.90 to 0.94 for the three ANES waves. For independents it is 0.41 in 2016 and 0.02 in 2020 [arxiv:2209.06899 tab. 1]. So an aggregate match can hide a subgroup mismatch. In a Turing-style check, human judges labelled GPT-3 lists as human 61.2% of the time, against 61.7% for human-written lists 2209.06899 §5. The paper does not say whether the model used is still served, so access to token probabilities is a practical limit 2209.06899 §2. Card: Out of One, Many.

Country of birth and residence in the persona

Cultural prompting names a country in the respondent descriptor. The model is told it is an average person born and living in that country, and the survey question follows as usual. The paper compares this with a default descriptor that names no country 2311.14096 §4.2 2311.14096 §4.3. Across 107 countries and territories, the distance from each country's survey values fell for all five GPT models. For GPT-4o the mean fell from 2.42 to 1.57, and the share of countries that improved ranged from 71.0% to 81.3% across models 2311.14096 §2. The authors say 19 to 29% of countries did not improve, or got worse 2311.14096 §3. For GPT-4o, Finland, Luxembourg, Andorra, Switzerland and Taiwan moved further away 2311.14096 §2. The prompts were English only, and the distance is measured on one cultural map built from one survey 2311.14096 §3 2311.14096 §4.1. Card: Cultural Bias and Cultural Alignment of Large Language Models.

Culture phrase in English or the target language

A value survey can be asked three ways: in English with a nationality phrase ("for an average American"), in the target country's language, or in English with a "culture setting" phrase. The study uses 24 Hofstede survey items for five cultures in these three forms 2303.17466 §3.1 [arxiv:2303.17466 tab. 1]. Each culture gets six dimension scores from a weighted difference of item pairs 2303.17466 eq. 1.

The three forms do not agree equally well. For prompts 1 and 3, the share of items with the same score was above 70% for the US, Germany, Japan and Spain, and 58.33% for China 2303.17466 tab. 3. The authors report that most cultures align better with human scores when asked in their own language 2303.17466 §4.2. Only one Spearman cell reaches p < 0.05: Spain, prompts 1 and 2, at 0.93 2303.17466 tab. 5. The paper does not state the sample size behind the correlations, so the p-values cannot be checked 2303.17466 §4. The same item can also score differently by language: for a Japanese-culture item on personal time, the English prompt scored 1.0 and the Japanese prompt 3.0 2303.17466 tab. 7. Card: Assessing Cross-Cultural Alignment between ChatGPT and Human Societies.

Disclosed experiment design

When a model plays the subjects of a randomised experiment, the usual prompt shows one scenario and says nothing about the design. In a price study, the model also changed other details it invented, such as the last price paid, so the price effect mixed with those details 2312.15524 §2 2312.15524 app. B. Adding those details as stated covariates was not a clean fix. The error on purchase probability was lowest at stage 2 (0.0971) and rose over later stages, with 0.2025 at stage 11 2312.15524 app. E.

The unblinded prompt tells the model that the price is drawn at random between a minimum and a maximum, and that the customer sees one price without knowing the draw 2312.15524 §5. For gpt-4o-mini, the error against a 40-product human survey was 0.532 with the blinded prompt and 0.397 with the unblinded prompt, both without fine-tuning 2312.15524 tab. 1. The paper reports lower error with unblinded prompts for every model it tested, with relative gains from about 1% to over 60%, read from a figure 2312.15524 §5 2312.15524 fig. 6. The minimum and maximum values are not stated 2312.15524 §5. The evidence comes from one demand study and one human survey 2312.15524 §2. Card: Gui and Toubia, causal-inference study of LLM simulation.

Per-annotator examples in the prompt

A personalised prompt lists up to three labelled texts from one annotator before the target text, so the model is asked to label the way that annotator did 2302.10724 §6.3. On AggressionPer, ChatGPT scored 72.57 F1, against 69.10 on the plain Aggression task. The two rows have different references (81.03 and 74.45), so the gap is not a clean personalisation effect 2302.10724 tab. 2. On the GoEmo variants GoEmoPer0 to GoEmoPer3, scores ranged from 19.00 to 23.74 and did not improve with each step 2302.10724 tab. 2. The paper's text gives a 4.08-point gain for GoEmoPer3 over GoEmoPer0, but its table shows 23.41 against 23.74 2302.10724 §6.3 2302.10724 tab. 2. Card: ChatGPT: Jack of all trades, master of none.

Low-perplexity prompt selection

With no labels, a few hand-written prompts can be expanded and ranked by how familiar the model finds them. SPELL asks GPT-3 for paraphrases of each seed, backtranslates them through other languages, and keeps the three prompts with the lowest mean perplexity over about 1,000 task inputs. The label is not used 2212.04037 §3 2212.04037 §7. On OPT and Bloom, zero-shot scores mostly correlated negatively with perplexity, though not on every task; IMDB with OPT 175B was positive 2212.04037 tab. 5. The author-reported gain over manual prompts was 1.8 accuracy points for OPT and 2.3 for Bloom, but single tasks moved from -12.5 to +10.2 points, and the paper gives no significance test for the gain 2212.04037 tab. 10 [arxiv:2212.04037 §7.1]. Card: Demystifying Prompts in Language Models via Perplexity Estimation.

Learned soft prompts

Here the prompt is not text. The cloze-style prompt tokens become trainable vectors, and each layer gets a trainable offset. The language model stays frozen; only these vectors are fitted, with gradient descent on known question and answer pairs 2104.06599 §3.1 2104.06599 §3.2. Several such prompts are mixed, with the mixture weights learned too 2104.06599 §3.3. On T-REx with BERT-large-cased, the best soft mixture reaches P@1 51.6, against 39.4 for LPAQA as reimplemented by the authors 2104.06599 tab. 1. Initialising the vectors at random, not from a hand-written sentence, scored 51.9 on the same setting 2104.06599 tab. 1. Tuning only the input layer gave 51.0 2104.06599 tab. 5. The learned vectors do not read as words; the authors' nearest-word check finds topical but not phrase-like neighbours 2104.06599 app. C. Card: Learning How to Ask. This needs gradient access to the model and a labelled set per relation; the paper's setting is one relation at a time, with BERT and RoBERTa.

P-Tuning keeps the words of a hand-written template and adds trainable continuous vectors beside them, passed through a small LSTM or MLP 2103.10385 §2.2 2103.10385 §2.3. With a frozen BERT-base model on LAMA-34k, precision@1 is 48.3, against 43.3 for AutoPrompt's discrete search 2103.10385 tab. 3. For one LAMA relation, the spread across six manual prompts falls from 10.1 to 0.46 2103.10385 tab. 6. On few-shot SuperGLUE, with a tuned ALBERT-xxlarge-v2, the seven-task average is 71.81 against 70.74 for PET-FT. That gain of about one point sits inside the reported standard deviations on several tasks 2103.10385 tab. 5 2103.10385 §3.3. The released LAMA code does not insert the relation's hand-written prompt that the paper describes, so a faithful reimplementation must add it modeling.py. Card: GPT Understands, Too.

PPT adds a pre-training stage for the soft prompt. Before any labelled task, the prompt is trained on unlabelled text with three cloze tasks, one for sentence pairs, one for multiple choice and one for single sentences. The downstream task then starts from that prompt instead of a random one 2109.04332 §3.1 2109.04332 §3.2. With T5-XXL and 32 training examples per task, the authors report 93.5 ± 0.3 on SST-2 for PPT, against 91.4 ± 0.8 for full fine-tuning and 70.5 ± 15.5 for prompt tuning from a random start [arxiv:2109.04332 tab. 4]. They say PPT matches or beats full fine-tuning on most English and all Chinese datasets they tested, with lower variance across seeds [arxiv:2109.04332 §4.2]. Pre-training runs for up to 200,000 steps, once, and the resulting prompts are reused across tasks 2109.04332 app. C 2109.04332 app. E. Card: PPT: Pre-trained Prompt Tuning for Few-shot Learning.

The prompt-programming paper proposes a generic seed, "Let's solve this problem by splitting it into steps," that the model expands into its own procedure. It reports no accuracy for this. Its Figure 3 shows a wrong answer for f(f(3)) with f(x) = x*x, where the correct value is 81 and the output says 27 2102.07350 §4.7 2102.07350 fig. 3.

The paper also suggests reading a verdict out at the point where the model's probability of the phrase "Thus, the correct answer is" peaks. It gives no threshold for detecting that peak 2102.07350 §4.6.

Text prefix for a frozen forecaster

Time-LLM keeps a 7B language model frozen. Each series patch is mapped onto learned text-like prototypes, and a text prompt is placed in front of those patches 2310.01728 §3.1. The prompt has three parts: a dataset description, a fixed task instruction, and input statistics such as trends and top lags 2310.01728 §3.1 2310.01728 app. B.1. Only the input layers and the output projection are trained. The prompt positions are dropped before the forecast is produced 2310.01728 §3.1. The authors report ETTh1 long-term MSE 0.408 against 0.413 for PatchTST 2310.01728 §4.1. Removing the whole prefix raised ETTh1-96 MSE from 0.362 to 0.398 2310.01728 §4.5. Removing the dataset description, the task instruction or the statistics gave 0.402, 0.388 and 0.391 on the same setting, and 0.298, 0.285 and 0.279 on ETTm1-96 against 0.272 2310.01728 §4.5. These are single ablation rows, each averaged over three runs. The paper does not report how much of the compute the prompt uses 2310.01728 §4.5. Card: Time-LLM. The shipped code differs from the paper's prompt description. It adds min, median and max values, and it uses one fixed ETT sentence for every dataset unless a prompt domain is set, which no shipped script does 2310.01728 §3.1. Implement the prompt from the code if you want to match the reported numbers.

Search over prompted steps

Tree of Thoughts treats each step as a thought. The model proposes candidate next steps, scores partial solutions, and a breadth- or depth-first search keeps the good branches and backtracks from the rest 2305.10601 §3. With GPT-4, Game of 24 success rises from 4% for CoT to 74% for this search 2305.10601 tab. 2. With GPT-3.5 the gain is smaller: 19% against 3% 2305.10601 tab. 5. A search costs more: $0.74 per problem against $0.47 for best-of-100 CoT on Game of 24 2305.10601 tab. 7. Card: Tree of Thoughts. See also LLM tree search.

Automated prompt search

OPRO uses an LLM as the optimizer. Each step shows it earlier instructions with their training scores, sorted so the best come last, plus three training exemplars. It proposes eight candidates at temperature 1.0, and each candidate is scored on a fixed training subset before it joins the list [arxiv:2309.03409 §2.3] 2309.03409 §4.2 2309.03409 §5.1. For GSM8K the training subset is 3.5% of the training set (about 261 questions, derived), and the search runs for 200 steps 2309.03409 §5.1. With PaLM 2-L as the scorer, the best instruction reaches 80.2 on the GSM8K test set, against 71.8 for "Let's think step by step." 2309.03409 tab. 4. That table picks each instruction by test accuracy, which inflates the figure. The authors suggest choosing by training accuracy 2309.03409 §5.4, and training accuracy ran 5 to 20 points above test accuracy in their runs 2309.03409 §5.4. Scorer calls are the main cost: at 200 steps with 8 candidates on about 261 questions, the worst case is about 418,000 calls before deduplication (our arithmetic) 2309.03409 §5.1. Synonym-level wording changes also move accuracy a lot 2309.03409 §5.2.3. Card: Large Language Models as Optimizers. The same loop fits small linear-regression and TSP instances, and it reaches the optimum less often as those grow 2309.03409 §3.2 2309.03409 tab. 3.

Prompt-averaged scoring

Prompts can steer a scorer as well as a generator. BARTScore rates generated text by the log-likelihood a pre-trained BART assigns to it, given a source or a reference, with no human labels 2106.11520 §3.2. Its prompt variant appends or prepends a short phrase such as "in summary" or "in other words", and averages the scores over 70 or 34 such phrasings 2106.11520 §3.3.1 2106.11520 tab. 2. The authors report that prompts help on most settings, but not on factuality 2106.11520 §4.3.2. Card: BARTScore.

Many prompts per task at training time

T0 changes the training data instead of prompting a frozen model. The authors write 62 English datasets in 12 task types as prompts, and the collection has 2,073 prompts for 177 datasets, about 11.7 per dataset 2110.08207 §3 2110.08207 §4. Each example is rendered through each of its dataset's templates, and the cap for large datasets is 500,000 examples shared across those templates 2110.08207 §5. Accuracy on four held-out task types is scored by rank classification, and the reported number is the median over each dataset's prompts 2110.08207 §5. T0 (11B) matches or exceeds GPT-3 on 9 of 11 held-out datasets (author-reported) 2110.08207 §6.1. The comparison is not like for like: GPT-3 reports one prompt per dataset, and T0 uses all of its prompts 2110.08207 §6.1. Using 5.7 original prompts per dataset rather than one raised the median on 8 of 11 datasets and shrank the spread on 7 of 11 2110.08207 §6.2. Datasets used in training are greyed out and are not zero-shot results 2110.08207 tab. 7. The paper reports about 270 TPU v3-512 hours for all experiments 2110.08207 app. B.1. Card: Multitask Prompted Training.

Explicit rules for a redaction job

De-identifying clinical notes is prompt-only in DeID-GPT. The system message holds a task statement, a command to replace identifiers with "[redacted]", and rules that list the kinds of identifier, and the note goes in the user message 2303.11032 §IV 2303.11032 fig. 7 2303.11032 fig. 8. On the 2014 i2b2/UTHealth test set, ChatGPT's entity-wise score rose from 0.686 with an implicit prompt to 0.929 with the explicit one 2303.11032 tab. 2. The GPT-4 score of 0.99 with the explicit prompt comes from 50 samples run by hand in the web interface, so it is not comparable with the full-set rows [arxiv:2303.11032 §V] 2303.11032 tab. 2. Prompt wording is fragile: an extra full stop stopped the command being read, and combining tasks confused the model 2303.11032 fig. 9. The paper also says an API-only model cannot take patient text in a hospital, and it calls for local models 2303.11032 §VI. Card: DeID-GPT.

Codebook-centred prompt with one example per code

For deductive coding, the codebook is the prompt. Each code gets a name, a description and examples, and the model must pick one of the listed codes 2304.10548 §3. The authors compared a codebook-centred layout with an example-centred one, where each example comes with a sentence on why it has its code, and varied the examples per code between none, one and five 2304.10548 §3. GPT-3 (davinci-text-002) at temperature 0 reached Cohen's kappa of 0.61 against the experts' final codes for question complexity and 0.38 for syntactic structure. The two experts reached 0.88 and 0.90 with each other 2304.10548 §4. The largest gain came from zero to one example per code; codebook-centred prompts scored higher 2304.10548 §4 2304.10548 fig. 1. Complexity was coded on GPT-3's English translations, not the French originals [arxiv:2304.10548 §2.1]. The study covers one dataset of 668 questions and one codebook, and the authors call it preliminary 2304.10548 §5. Card: Xiao et al., codebook prompts with GPT-3.

Codebook prompt for annotation

For labelling, the prompt is the same codebook that trained human coders used, with no examples. ChatGPT (gpt-3.5-turbo) receives each text in a fresh chat with the wording "Here's the tweet I picked, please label it as [task]." [arxiv:2303.15056 §4.5]. Averaged over four samples, its accuracy against trained annotators was about 25 points above that of MTurk workers (author-reported) [arxiv:2303.15056 §2]. On the four relevance tasks it ranged from 59% to 83% 2303.15056 §2 2303.15056 fig. 1. Intercoder agreement was 91% at the default temperature and 97% at 0.2, against 79% for trained annotators and 56% for MTurk 2303.15056 §2. That agreement is raw percent match with no chance correction, so skewed classes look better than they are 2303.15056 §4.6. The paper tests no few-shot prompts 2303.15056 §3. Card: ChatGPT Outperforms Crowd-Workers.

Demonstration prompts that write labelled data

Each generated sentence comes from a prompt of five demonstrations about one group, drawn at random for every sentence. Decoding uses top-k sampling and stops at the first newline 2203.09509 §3.1 2203.09509 app. A. The authors used 26 prompt sets (toxic and benign for each of 13 groups) and kept 274,186 statements, of which 98.2% contain no profanity or slur by their word-list test 2203.09509 tab. 2. A second decoder, ALICE, uses a toxicity classifier during beam search to push the text towards sentences the classifier misreads. It supplies about 5% of the statements 2203.09509 §3.2 2203.09509 tab. 2. Fine-tuning HateBERT on the combined data raised AUC on three human-written implicit-hate sets by 11, 7 and 19 points 2203.09509 tab. 4. The training label is the prompt's label, not a human's. Human raters agreed with the prompt label for 67.7% of top-k toxic prompts and 40.3% of ALICE toxic prompts 2203.09509 §4.3. The table values were transcribed from a rendered page, so check them against the paper. The released code applies the classifier term twice, which the paper does not describe 2203.09509 §3.3. Card: ToxiGen.

Staged Python-style plans for robot teams

One instruction for a group of robots is planned in stages. The prompt asks the model to split the task into sub-tasks with their skills, choose one robot or a team for each, and then write code that assigns the work and runs independent steps side by side 2309.10062 §4 2309.10062 fig. 2. Skills and objects appear as Python dictionaries, with few-shot examples of 5 for decomposition, 3 for coalitions and 4 for allocation code [arxiv:2309.10062 §5.2]. On 36 AI2-THOR tasks, GPT-4 reached a success rate of 1.00 on elemental tasks and 0.62 to 0.71 on the other three categories [arxiv:2309.10062 tab. 1]. In a five-run check on one task per category, the complex category scored 0.48 ± 0.40, below the 0.71 in the benchmark table; the paper does not explain the gap 2309.10062 tab. 2 2309.10062 §6.1. Nothing checks the generated code between planning and execution [arxiv:2309.10062 §4.4]. Real-robot runs have no metrics 2309.10062 §6.2. Card: SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models.

Two chat agents with inception prompts

CAMEL runs a role-play between an assistant and a user, both chat models. A task-specifier agent first turns an idea into a concrete task [arxiv:2303.17760 §3.1] 2303.17760 fig. 2. Behaviour comes from fixed system prompts. The assistant's prompt names both roles and forbids swapping them, and it requires an answer that starts with "Solution:" and ends with "Next request." [arxiv:2303.17760 §3.2] 2303.17760 fig. 2. The user's prompt asks for one instruction per turn, and for the stop token "<CAMEL_TASK_DONE>" only once the task is solved 2303.17760 §3.2. The authors list four failure modes seen in pilot runs: role flipping, the assistant repeating instructions, replies that only promise action ("I will ..."), and endless polite loops 2303.17760 §1 [arxiv:2303.17760 §4]. Against a single-shot gpt-3.5-turbo answer, human voters preferred CAMEL on 76.3% of 100 AI Society tasks, with 10.4% for the single shot and 13.3% draws 2303.17760 tab. 1. The repository's prompts differ from the paper's, so replicate from the paper's text 2303.17760 fig. 2. Card: CAMEL. Compare role profiles.

Single prompt on finance datasets

A survey of LLMs in finance also reports one GPT-4 evaluation on six datasets. The prompt has three parts, a role description, an output format and one example, and it goes in the system message 2401.11641 §4.2. Reported accuracies are 0.78 on Financial PhraseBank, 0.79 on FiQA sentiment, 0.81 entity F1 on financial NER, 0.64 on FinQA, 0.73 on ConvFinQA and 0.53 on stock movement [arxiv:2401.11641 tab. 2]. Text labelling scores best, and numeric questions and price direction score lower 2401.11641 tab. 2. There is no baseline, and the model snapshot and decoding settings are not given 2401.11641 §4.2 2401.11641 tab. 2. The stock labels leave a band between -0.5% and +0.55% with no class 2401.11641 §4.1. Read these as what one prompt did, not as a benchmark. Card: Revolutionizing Finance with LLMs.

Safety system prompts tested on scenarios

The mental-health study runs each question as its own prompt, with a system message that argues against stigma (the "steel-man" prompt), a letter-answer rule, one worked example, and temperature 0 2504.18412 §4 2504.18412 app. Steel-man system prompt. Its stigma probe is 72 vignettes, asked 14 questions each, which gives 1008 prompts per model 2504.18412 §4. Even with the steel-man prompt, the overall rate of stigmatising answers ranged from 38% for gpt-4o to 75% for Llama 3.1 405B 2504.18412 §4.1. On crisis stimuli, the crisis experiment uses the same prompt, and the models gave under 80% appropriate replies overall, against 93% for 16 human therapists 2504.18412 §5.1. The paper says the steel-man prompt "dramatically improves" overall crisis performance but gives no number for that effect 2504.18412 §5.1. Card: Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. All numbers are author-reported.

Persona prompts scored for marked words

To find stereotypes without a word list, ask the model for a short persona of each group, then compare each marked group with the unmarked defaults, White and man 2305.18189 §3.1 2305.18189 §3.2. The main run used six prompt templates, temperature 1 and 15 samples per group and prompt, giving 1,350 personas per model 2305.18189 §4. A word is marked when its weighted log-odds difference is significant (z > 1.96) against both defaults 2305.18189 §3.2. The authors report that GPT-4 and GPT-3.5 personas contain more racial stereotype lexicon words than human-written personas for the same prompts; the figure has no numbers in the text 2305.18189 §5 2305.18189 fig. 1. The top marked words were mostly positive in sentiment, with a mean VADER score of 0.05, so a sentiment filter would miss them 2305.18189 §6 2305.18189 tab. 2. The default group is an a-priori choice that reflects internet-data norms 2305.18189 §9. Card: Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models.

Positive prefix against biased completions

A short positive statement put before the prompt changes how a model completes text about a group. With GPT-3 davinci, "Two Muslims walked into a" gave violence-related completions in 66 of 100 runs 2101.05783 §Results 2101.05783 fig. 1. With a prefix such as "Muslims are hard-working.", the mean over 50 positive adjectives was 46%. The six best adjectives reached 20%, against 13-15% for the Christian prompt 2101.05783 §Results 2101.05783 fig. 4. The six were picked from the first run and then rerun; the paper does not say whether the rerun used fresh completions. The authors call this "debiasing" only loosely, say completions still carry bias, and report that the prefix steers output toward particular topics 2101.05783 §Results 2101.05783 §Discussion. Every rate depends on a hand-made keyword list 2101.05783 app. B. Card: Persistent Anti-Muslim Bias in Large Language Models.

Prompt-only guardrails in a deployed system

A university tutoring assistant uses retrieval over course material, then tells the model to say "I'm not sure" rather than guess 2309.10892 §3.1.1 2309.10892 §3.2.3. Its only hallucination control is this prompt 2309.10892 §3.1.1 2309.10892 §3.2.3. The paper reports no measurement of answer correctness or of how often the instruction works 2309.10892 §4. Card: AI-enabled intelligent assistant for higher education.

A 2023 survey of ChatGPT research lists prompting ("cueing") techniques among application areas, without describing any 2304.01852 §2.1.6. Card: Early ChatGPT research survey.

One suggested reply per teaching strategy

Tutor CoPilot shows a tutor, inside a live chat, one suggested reply for each of several teaching strategies. Each reply is a separate GPT-4 call built from the lesson topic, the chosen strategy and the most recent messages. The paper sends only the 10 latest messages to the external model [arxiv:2410.03017 §Methods] 2410.03017 fig. 1. The tutor can edit a reply, regenerate it or choose another strategy, so the tutor makes the final choice 2410.03017 §Methods. The paper does not state the prompt text, and the one-sentence reply limit in the public demo is not in the paper 2410.03017 §Methods. In a randomized trial of 783 tutors, access to the tool raised the exit-ticket pass rate from 61.7% to 65.7% 2410.03017 tab. 1. The 14-point figure for tutors who used the tool comes from an instrumental-variable estimate, not from usage logs 2410.03017 §Results. Treated tutors used "prompt student to explain" about 10% more often 2410.03017 fig. 3. The end-of-year test effect was not estimated 2410.03017 §Methods. Card: Tutor CoPilot.

Plain-language rewrite of a report

A one-line request asks the model to rewrite a medical report for a lay reader. The prompt asks the model to explain the report to a child in plain words, then gives the report text 2212.14882 §3.2. The authors picked it by informal trial from six variants on one other fictitious report, and they do not claim it is the best one 2212.14882 §3.2 2212.14882 §5.3. Each report was sent 15 times, each time in a fresh chat, and 15 radiologists rated the rewrites. The median rating was 2 (agree) for factual correctness and for completeness, and 4 (disagree) for potential harm [arxiv:2212.14882 tab. 1]. Raters still highlighted wrong passages, invented findings and dropped results. The table gives the share answering each follow-up as 51%, 22% and 36% 2212.14882 tab. 3 2212.14882 §4.2. The study compares no other prompt or simplifier, so it does not show that this wording beats another 2212.14882 §3.2. The interface offers no temperature or seed control, which the authors cite as a reproducibility problem 2212.14882 §5.2. Card: ChatGPT Makes Medicine Easy to Swallow.

When to use which

Implementation sketch

Language-agnostic pseudocode for the common core. The parts that vary are marked.

# Build the prompt: instruction, optional demonstrations, then the input
# [arxiv:2302.06476 §3] [arxiv:2202.12837 app. B]
def build_prompt(instruction, demos, x):
    parts = [instruction]
    for x_i, y_i in demos:
        # pair layout: input line, label line; pairs separated by blank lines
        # [arxiv:2202.12837 app. B]
        parts.append(x_i + "\n" + y_i)
    parts.append(x)
    return "\n\n\n".join(parts)

# Optional: reasoning cue, two stages  [arxiv:2302.06476 §3]
def answer_with_cot(instruction, x, trigger):
    # stage 1: free-text rationale
    rationale = model(instruction + x)
    # stage 2: order of the parts is not stated; this order is an assumption
    return model(instruction + x + rationale + trigger)

# Read the answer strictly and count replies outside the allowed set
# [arxiv:2302.06476 §4.2.8]
def parse_answer(output, allowed_labels):
    # return None for off-format replies, and log them
    if output.strip() in allowed_labels:
        return output.strip()
    return None

Where variants differ:

• Demonstrations with random labels keep the pair layout but draw each label uniformly from the label set 2202.12837 §4.1. Use the gold labels if you are measuring the prompt itself.
• Search replaces the single answer with a loop over proposed steps, with the model scoring each partial solution 2305.10601 alg. 1.
• Distractors add an instruction to ignore irrelevant information before the exemplars, and can sample several answers and vote 2302.00093 §4.3 2302.00093 §5.1.
• Format sweep scores the same items under several equivalent formats and reports the best and worst, not one number 2310.11324 §3.2.
• Layout output asks for one rule per object with normalised integer values, and the parser must map them back to boxes 2305.15393 §3.2.
• Per-question sensitivity replaces the single score with the mean gap over every pair of variants for each question, then averages across questions 2410.12405 eq. 1 2410.12405 eq. 2.
• Extraction output can be IOB tags or YAML with character spans. Parse each one strictly, drop what fails to parse, and fuzzy-match spans back to the source 2312.02296 app. A.1.
• Decoding settings (temperature, top-p, output length) are not stated for the ChatGPT runs or the translation runs, so record your own 2302.06476 §3 2102.07350 tab. 1.
• Probe-based order selection samples one continuation per candidate order at temperature 2, parses the inputs, and ranks orders by the entropy of their predicted labels 2104.08786 §3.1 2104.08786 eq. 3.
• Iterative instruction search keeps the best instructions so far, sorted with the best last, and asks the optimizer for new ones. Each candidate is scored on one fixed training subset, and the final choice is made by training accuracy 2309.03409 §4.2 2309.03409 §5.4.
• Two-agent role-play alternates an instruction from the user and a solution from the assistant, each in a fixed format, and stops when the user sends the stop token 2303.17760 §3.1 2303.17760 §3.2.
• Prompt-averaged scoring appends or prepends each phrase to the input and averages the normalised scores over the phrases 2106.11520 eq. 3. The paper does not say which side's average it reports 2106.11520 §3.3.1.
• Serialized row writes each record as "attribute: value" pairs, keeps only the attributes that help, and writes a missing value as an empty string 2205.09911 §3.1. The task question follows the rows and is different for each task 2205.09911 §3.2.
• Answer form check counts any reply outside the required form as wrong rather than dropping it, and reports how often that happens 2402.05201 §3.1 2402.05201 app. A.
• Numeric comparisons in the prompt state which option is cheapest or fastest, and by what share, instead of leaving the comparison to the model 2312.00819 §3.3.4.
• Language or culture phrase is a variable to sweep: ask the same items in English with a group phrase, in the target language, and in English with a "culture setting" phrase, then check agreement between the forms 2303.17466 §3.1 2303.17466 tab. 3.
• Perplexity ranking scores each candidate prompt by its mean perplexity over about 1,000 task inputs, and keeps the lowest few. The label is left out of the score 2212.04037 §4.2 2212.04037 §7.
• Codebook layout writes each code as a name, a description and examples, with zero, one or five examples per code 2304.10548 §3.
• Marked-words audit keeps the words whose weighted log-odds difference is above 1.96 against every default group 2305.18189 §3.2.
• Wording stress test changes only the instruction and compares each score with the clean prompt on the same sampled set 2306.04528 §2.2 2306.04528 §3.
• Staged planner gives each planning stage its own few-shot examples 2309.10062 §4 2309.10062 §5.2.

Contradictions

• Which prompt part matters in Time-LLM. The text says that removing the input statistics costs 10.2% of average MSE, and it calls that row C.1. The ablation table labels C.1 as the dataset description and C.3 as the statistics 2310.01728 §4.5. The card uses the table labels, so the attribution of that 10.2% is not settled. Check the compiled PDF before quoting either.

• Zero-shot against few-shot. On Curie, zero-shot and 10-shot Simple colon tie (33.3 vs 33.4 BLEU), but on Babbage the 10-shot baseline is higher (25.1 vs 23.5) 2102.07350 tab. 1. The two results are for different models and different baselines, so they do not settle which setting wins.

• Do examples teach? The prompt-programming paper argues that examples mostly point at a task the model already knows 2102.07350 §1. The demonstrations paper agrees that gold labels add little, but finds that the label space and input text carry most of the gain 2202.12837 §5. Neither paper tests the other's setting. The first rests on one task and two models 2102.07350 §3.1.

• Label sensitivity by scoring method. Channel models barely change when label words are replaced, while direct models lose 5-16 points 2202.12837 §5.2. A result from one scoring method does not carry over to the other.

• Soft prompts, layer-0 or all layers. The paper's text says the basic method tunes only the input layer and the all-layer variant is an add-on 2104.06599 §4.7. Its Table 1 numbers match the all-layer column of Table 5, not the layer-0 column 2104.06599 tab. 1 [arxiv:2104.06599 tab. 5]. The gap is 0.6 P@1 on T-REx with BERT-large, so the choice matters less than the difference from hand-written prompts.

• Reasoning cues on commonsense tasks. A cue helps arithmetic and lowers CommonsenseQA and StrategyQA for ChatGPT 2302.06476 tab. 3. The metaprompt paper's reasoning benefit is unmeasured 2102.07350 §4.7.

• Reasoning cue on other models. The same kind of cue scored 29.0% on the GSM-IC variants for code-davinci-002 2302.00093 tab. 3, while it improved ChatGPT on all six arithmetic sets 2302.06476 tab. 2. The models, the datasets and the prompt setups all differ.

• Format sensitivity by benchmark. FormatSpread reports a median spread of 7.5 points and up to 76 points for one model 2310.11324 §1 2310.11324 §4.2. MMLU-Pro reports about 2% across 24 prompt styles 2406.01574 §6.3. The format sets differ (a grammar of generated formats against 24 written styles), and so do the tasks and models, so the two numbers do not measure the same thing.

• Sensitivity against model size. ProSA reports that larger models and few-shot examples lower its per-question score 2410.12405 §3.3 2410.12405 §3.4. FormatSpread reports that the format spread does not shrink with model size, shot count or instruction tuning 2310.11324 §4.2. The two measure different things (a paired gap over 12 hand-written variants against a spread over generated formats), and they use different models, so neither result answers the other's question. A travel-prediction ablation adds a case on the other side: GPT-4 lost under 1% when the reasoning guide was removed, against 5.8% for GPT-3.5 2312.00819 tab. 5, which fits a size effect. That ablation removes prompt parts or flattens the data structure; it does not sample equivalent formats, so it does not test the FormatSpread claim.

• Temperature for single answers against voting. The temperature study recommends 0.0 for problem solving, and found no accuracy effect from 0.0 to 1.0 for single answers 2402.05201 §4.1. The self-consistency study samples 20 answers at temperature 0.7 and takes a majority vote 2302.00093 §5.1. A vote needs answers that differ, so the 0.0 advice is not tested for that method. The two studies test different procedures, and neither compares voting across temperatures.

• Steering against training. The survey paper reports that prompting alone rarely reproduces the spread of human answers 2502.16761 §4.2. The Turing Experiments paper reports that zero-shot persona prompts reproduce three of four classic human findings 2208.10264 §7. The first measures answer distributions on survey questions, the second measures effects in classic experiments. Neither paper runs the other's task.

• Which factor drives few-shot results. Order alone can spread the accuracy of one demonstration set from near 50% to over 85% 2104.08786 fig. 1. The demonstrations paper finds that gold labels add little and that the label space and input text carry most of the gain 2202.12837 §5. The two papers vary different factors, order against label content, so neither result rules out the other.

• Order with retrieved examples. The order paper says a 2021 study (not carded here) found order matters little once retrieved examples are used, and it argues the opposite for fixed few-shot sets 2104.08786 §6.

• Prompt sensitivity by model. ChatGPT's score rises from 0.686 to 0.929 with an explicit prompt, while GPT-4 scores 0.908 even on the implicit prompt 2303.11032 tab. 2. The GPT-4 rows come from 50 hand-run samples, so this gap compares unequal runs 2303.11032 §V.

• Does the wording matter? The 2021 NLP survey reports that, in one cited few-shot entailment study, irrelevant templates did about as well as meaningful ones 2111.01243 §6. The ChatGPT study finds the opposite direction: changing the prompt wording moved PolEmo by up to +14.8 points and TextEntail by up to −13.0 points 2302.10724 tab. 4. The models, tasks and templates differ, so neither result settles the question. A third study finds the sign of the wording effect depends on the dataset: one reworded question helped one entity-matching set and hurt two others 2205.09911 tab. 4.

• Random starts for soft prompts. On relation probing with BERT-large, a random start scored 51.9, slightly above the 51.6 of the best soft mixture 2104.06599 tab. 1. On 11B T5 with 32 SST-2 examples, a random start gave 70.5 ± 15.5, and the PPT paper calls this unstable and starts from pre-trained prompts instead 2109.04332 tab. 4 2109.04332 §4.2. The tasks, model sizes and label counts all differ, so neither result settles whether the start matters in general. The PPT pilot found that starting from word embeddings did little or harm at 11B 2109.04332 tab. 2.

• Prompt language for non-English tasks. The ChatGPT study finds that an English task description usually had the higher average, on four of six tasks 2304.05613 §4-2304.05613 §9. The survey-persona study finds that prompting in Arabic raised GPT-3.5's Egypt score by about 3 points 2402.13231 tab. 2. The tasks, the metrics and the models differ, so neither result settles the language choice for another task. The cultural-alignment study reports that most cultures align better when asked in their own language 2303.17466 §4.2, while the English prompts were more consistent with each other 2303.17466 tab. 3. Those two findings are about values, not tasks, and point in different directions.

• Zero-shot personas against blinded experiments. The Turing Experiments paper reports that zero-shot persona prompts reproduce three of four classic human findings 2208.10264 §7. The causal-inference paper argues that a blinded persona in a price experiment confounds the price effect, because the model varies unstated details with the price 2312.15524 §2. The first measures trends in classic studies and the second measures a demand curve, so neither paper settles the other's setting. The second paper's remedy is a change to the prompt, not a change to the persona.

• Size of identity-prompt gains. The cultural-bias paper reports that cultural prompting cut GPT-3.5's mean map distance from 3.35 to 2.83 2311.14096 §2. The survey-persona paper reports that its Anthropological Prompting variant lifted GPT-3.5's Egypt soft score from 0.4834 to 0.5102 2402.13231 tab. 4. The metrics differ (distance on a cultural map against an item-level soft score) and so do the prompts, so the two gains cannot be compared.

Papers

• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm: argues that prompts work by locating a known task, and proposes prompt techniques (constraints, personas, serialized reasoning, metaprompts) with one measured translation comparison.
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?: shows that random labels in demonstrations cost little, and isolates the label space, input text and pair format as the drivers.
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver?: zero-shot instructions and two-stage zero-shot CoT across 20 datasets, against GPT-3.5.
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models: prompted search over intermediate steps, with large gains on Game of 24 and Mini Crosswords for GPT-4.
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts: learns continuous cloze prompts and mixes them, with BERT-large gains on T-REx over LPAQA; the learned vectors are not readable text.
• Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine: four clinician-style few-shot prompts against a step-by-step baseline on MedQA-derived diagnosis questions; GPT-4 shows no significant difference.
• AI-enabled intelligent assistant for personalized and adaptive learning in higher education: a retrieval-backed tutoring bot that controls hallucination only through its prompt; no evaluation.
• Early ChatGPT research survey: a survey of early ChatGPT research; names prompting only as an application area.
• Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies: zero-shot persona prompts run over many named participants, compared with four published human studies; the wisdom-of-crowds task shows a distortion.
• Large Language Models Can Be Easily Distracted by Irrelevant Context: one irrelevant sentence in GSM-IC variants lowers accuracy; an ignore-instruction, a distractor exemplar and self-consistency help.
• LayoutGPT: Compositional Visual Planning and Generation with Large Language Models: CSS-style layout rules with normalised values and retrieved examples, with no training; a 3D bedroom result against a supervised baseline.
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework: role profiles, fixed documents and executable feedback for a software team; the ablation varies the roles.
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models: a frozen 7B model with learned input patches and a dataset text prefix; author-reported forecasting gains, with ablation rows for each prompt part.
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design: grammar-generated prompt formats, a spread metric, and a bandit search for the best and worst format within a budget.
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark: a harder multiple-choice benchmark with 5-shot CoT and a regex answer parser; scores move less under prompt-style changes.
• SubPOP: zero-shot and few-shot steering prompts against LoRA fine-tuning on human answer distributions.
• Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System: one T5 model with a task-name prefix per dialogue sub-task, pre-trained on eleven partly labelled corpora.
• LLMs Accelerate Annotation for Medical Information Extraction: few-shot PaLM 2 pre-labels for medication fields, with expert refinement; cuts rater time by 58% on i2b2 documents.
• Investigating Cultural Alignment of Large Language Models: demographic personas answer World Values Survey questions in Egypt and the US; soft and hard scores, and an Anthropological Prompting variant.
• ProSA: defines a per-question prompt sensitivity score and reports it for eight LLMs on six benchmarks.
• Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers: a steel-man system prompt in stigma vignettes and crisis replies, with author-reported failure rates.
• GPT Understands, Too: P-Tuning adds trainable prompt vectors beside a hand-written template; frozen BERT on LAMA, tuned models on SuperGLUE.
• PPT: Pre-trained Prompt Tuning for Few-shot Learning: pre-trains soft prompts on unlabelled text with three cloze tasks, then tunes them per task; 11B models with 32 examples per task, author-reported.
• Fantastically Ordered Prompts and Where to Find Them: ranks few-shot demonstration orders by the label entropy of model-written probe inputs, without labels.
• BARTScore: scores text by BART log-likelihood; its prompt variant averages scores over short phrases.
• Pre-train, Prompt, and Predict: a survey that defines prompt addition, answer search and answer mapping; no experiments of its own.
• Multitask Prompted Training Enables Zero-Shot Task Generalization: T0, trained on prompted datasets with several templates each, tested zero-shot on held-out task types.
• Out of One, Many: Using Language Models to Simulate Human Samples: first-person demographic backstories make GPT-3 stand-ins for survey respondents, with checks on US politics.
• DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4: a three-part prompt redacts clinical notes; explicit rules raise ChatGPT's score on i2b2/UTHealth.
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks: zero-shot codebook prompts for tweet and news labels, against MTurk and trained coders.
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society: two chat agents with fixed inception prompts for roles, turn order and a stop token.
• Large Language Models as Optimizers: OPRO, where an LLM proposes instructions from scored past attempts; results on GSM8K and BBH.
• Revolutionizing Finance with LLMs: An Overview of Applications and Insights: a finance survey with one GPT-4 evaluation on six datasets, a three-part prompt and no baseline.
• Persistent Anti-Muslim Bias in Large Language Models: GPT-3 completion, analogy and caption probes; a positive prefix lowers violent completions but not to the level of other groups.
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey: a 2021 survey that splits the field into fine-tuning, prompt-based learning and text generation; runs no experiments.
• ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection: GPT-3 demonstration prompts write 274,186 labelled statements; a classifier-guided decoder adds harder examples.
• ChatGPT: Jack of all trades, master of none: ChatGPT on 25 analytical NLP tasks with a hand-made prompt per task; personalised per-annotator examples and wording tests.
• Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise: one GPT-4 reply per teaching strategy, shown to tutors in live chat; a randomized trial of 783 tutors.
• ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports: one plain-language prompt for three fictitious radiology reports, rated by 15 radiologists; errors and invented findings reported.
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning: zero-shot ChatGPT on seven tasks in 34 to 37 languages, with English and target-language descriptions; POS tagging is the exception to the gap.
• Unleashing the potential of prompt engineering for large language models: a review of prompting, reasoning cues, prompt optimization and prompt security; no experiments of its own.
• Cultural Bias and Cultural Alignment of Large Language Models: five GPT models on a cultural map; a country-of-birth persona moves answers toward that country's survey values for most countries.
• Gui and Toubia, causal-inference study of LLM simulation: blinded LLM subjects confound a price effect; telling the model the randomisation design lowers error against a human survey.
• Can Foundation Models Wrangle Your Data?: GPT-3 175B on five data-cleaning tasks, with serialized rows, task questions and few-shot examples; scores depend on example choice and wording.
• Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study: Hofstede survey items asked in English, in the target language and with a culture phrase; alignment is mostly weak and best for the US.
• Large Language Models for Travel Behavior Prediction: a structured zero-shot prompt with a JSON answer, and LLM embeddings as features for small-sample travel-choice models.
• The Effect of Sampling Temperature on Problem Solving in Large Language Models: five one-shot prompt styles at temperatures 0.0 to 1.0 on multiple-choice questions; no significant effect up to 1.0, with accuracy falling above it.
• Demystifying Prompts in Language Models via Perplexity Estimation: zero-shot prompts ranked by perplexity; SPELL keeps the three lowest of GPT-3 paraphrases and backtranslations.
• Xiao et al., codebook prompts with GPT-3: codebook-centred GPT-3 prompts for 668 French children's questions, kappa 0.61 and 0.38 against experts.
• Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models: persona prompts per group, with marked words found against the White and man defaults; no word list.
• PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts: seven text attacks on the instruction, scored by performance drop across 13 datasets and six LLMs.
• SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models: four-stage Python-style prompts that turn one instruction into multi-robot code, tested in AI2-THOR.