--- kind: concept slug: prompt-engineering title: Prompt engineering summary: >- Steering a frozen language model through its input: instructions, demonstrations, formats, reasoning cues, personas and layouts, and with weight access, learned soft prompts. Format changes alone moved one model's accuracy by up to 76 points; a zero-shot prompt tied 10-shot on one translation model. Evidence is mostly from 2021-2025 models, often one prompt per task. tags: [prompting, in-context-learning, zero-shot, evaluation, inference, soft-prompts, robustness, information-extraction, safety, bias, data-generation, multilingual, cultural-bias, simulation, data-cleaning, decoding, structured-data] papers: ["2101.05783", "2102.07350", "2103.10385", "2104.06599", "2104.08786", "2106.11520", "2107.13586", "2109.04332", "2109.14739", "2110.08207", "2111.01243", "2202.12837", "2212.14882", "2203.09509", "2208.10264", "2209.06899", "2302.00093", "2302.06476", "2302.10724", "2303.11032", "2303.15056", "2303.17760", "2304.01852", "2305.10601", "2305.15393", "2308.00352", "2308.06834", "2309.03409", "2309.10892", "2310.01728", "2310.11324", "2312.02296", "2401.11641", "2402.13231", "2406.01574", "2410.03017", "2410.12405", "2502.16761", "2504.18412", "2304.05613", "2310.14735", "2311.14096", "2312.15524", "2205.09911", "2303.17466", "2312.00819", "2402.05201", "2212.04037", "2304.10548", "2305.18189", "2306.04528", "2309.10062"] related: [adversarial-robustness, language-model-surprisal, llm-planning, chain-of-thought-prompting, in-context-learning, few-shot-learning, few-shot-example-selection, self-consistency, soft-prompt-tuning, llm-tree-search, llm-evaluation, hallucination-mitigation, distributional-alignment, layout-to-image-generation, multi-agent-systems, self-refinement, named-entity-recognition, prompt-optimization, instruction-tuning, llm-as-a-judge, llm-agent, zero-shot-learning, sentiment-analysis, multi-task-learning, human-in-the-loop, llm-ethics, toxicity-detection, data-augmentation, time-series-forecasting, llm-simulated-participants, multilingual-evaluation, causal-inference, decoding-strategies, text-embedding] status: draft updated_at: 2026-11-01 --- ## Overview Prompt engineering is choosing the text a model is conditioned on: the task instruction, any worked examples, the output format, cues that ask for intermediate reasoning, and sometimes a persona or a required layout. No weights change. The shared premise is that the model continues the text it is given, so a good prompt is one that makes the wanted continuation the likely one [arxiv:2102.07350 §4.1]. The variants differ in what the prompt specifies (the task, the examples, the format, the steps, the role) and in how the output is read back. A survey of the field names the same three steps: prompt addition, answer search and answer mapping [arxiv:2107.13586 §2.2]. A 2021 NLP survey lists prompt-based learning as one of three paradigms for large pre-trained models, next to fine-tuning and text generation [arxiv:2111.01243 §1] [arxiv:2111.01243 §3]. A 2023 review of LLM prompting treats prompt design as the main lever for accuracy and relevance when the model is fixed [arxiv:2310.14735 §1]. Most of the evidence here is for classification, translation, arithmetic word problems, clinical diagnosis questions, multiple-choice exams, layout generation, survey-style opinion questions, dialogue sub-tasks, medication extraction from discharge summaries, mental-health vignettes, redaction of clinical notes, plain-language rewrites of radiology reports, tweet and news annotation, finance text tasks, multi-agent software tasks, data cleaning and entity matching, travel mode choice and cultural value surveys. Most results come from a few models with one prompt per task. Later additions cover perplexity-ranked zero-shot prompts, codebook prompts for coding open-ended answers, persona prompts scored for marked words, perturbed instruction wording, and staged prompts for robot teams. The soft-prompt results are for fact probing on masked language models and for few-shot SuperGLUE with tuned models. Treat the numbers as a map of what worked in 2021-2025 models, not as a ranking that carries over to new ones. ## Variants ### Zero-shot instructions The prompt is a one-line instruction that names the task and the answer form, followed by the input and no examples [arxiv:2302.06476 §3]. Applied to gpt-3.5-turbo across 20 datasets, this beats text-davinci-003 on most arithmetic, NLI, dialogue and sentiment sets, and trails it on summarization, NER and several commonsense sets [arxiv:2302.06476 §4.2]. Card: [Is ChatGPT a General-Purpose NLP Task Solver?](/papers/2302.06476). Zero-shot prompting also works on data-cleaning questions, but unevenly. With GPT-3 175B and no examples, imputing a missing value beat the HoloClean baseline on both datasets tested (70.9 against 33.1 on Restaurant; 84.6 against 16.2 on Buy). The same model scored 6.9 F1 on Hospital error detection and 0.0 on Adult, and zero-shot entity matching trailed the same model with ten examples [arxiv:2205.09911 tab. 1] [arxiv:2205.09911 tab. 2]. The authors ran one closed model snapshot, so these are not results for other models [arxiv:2205.09911 §4.1]. Card: [Can Foundation Models Wrangle Your Data?](/papers/2205.09911). Zero-shot prompts also carry structured records. On travel mode choice, GPT-3.5 with no examples reached 0.586 accuracy, above neural networks trained on 10 samples (0.462) [arxiv:2312.00819 tab. 3]. The record and its rules are in [Variants](#structured-record-with-domain-rules-and-json-answer) below. Wording alone changed results in one translation setup. On WMT'14 French to English, a zero-shot "French: ... English:" format scored 33.3 BLEU on Curie, the same as the 10-shot version of that format (33.4), and above a reproduced 10-shot GPT-3-style prompt (27.9). On Babbage the zero-shot format (23.5) was below that 10-shot baseline (25.1) [arxiv:2102.07350 tab. 1]. The paper's reproduced baselines are well below the published GPT-3 numbers, so the gap is against the authors' runs [arxiv:2102.07350 tab. 1]. Card: [Prompt Programming for Large Language Models](/papers/2102.07350). A prompt should also rule out the continuations you do not want. A bare "Translate French to English:" often continued in French; a frame where the translation is the only sensible continuation worked best [arxiv:2102.07350 §4.5]. This is described qualitatively, with no measured effect. ### Task description in English or the target language For ChatGPT, each zero-shot query is one task description, then the input, then a format note. The study covers seven tasks in 34 to 37 languages, and each example gets either an English description or one in the target language [arxiv:2304.05613 §3]. On average, the English description scored higher on POS tagging, NER, NLI and QA [arxiv:2304.05613 §4] [arxiv:2304.05613 §5] [arxiv:2304.05613 §7] [arxiv:2304.05613 §8]. The target-language description scored higher on relation extraction and commonsense reasoning [arxiv:2304.05613 §6] [arxiv:2304.05613 §9]. The averages hide large single-language gaps. On NLI, Hindi scored 48.8 with an English description and 5.6 with a Hindi one [arxiv:2304.05613 §7]. Card: [ChatGPT Beyond English](/papers/2304.05613). ### Few-shot demonstrations The prompt holds k input-label pairs before the test input [arxiv:2202.12837 §3]. On 26 classification and multi-choice datasets with six decoder-only models, replacing the gold labels with uniform random labels cost 0-5 points absolute in nearly all of the 12 model and scoring combinations [arxiv:2202.12837 §4.1]. The gains depend more on the other parts of the demonstrations: the label space, the input text, and the pairing format [arxiv:2202.12837 §5]. Replacing the inputs with unrelated text cost 3-16 points for the models where it was measured [arxiv:2202.12837 §5.1]. Random English words instead of the label words cost direct models 5-16 points and channel models 0-2 points [arxiv:2202.12837 §5.2]. Inputs without labels, or labels without inputs, land close to or below the no-demonstration baseline [arxiv:2202.12837 §5.3]. Examples can also hurt. A one-shot version of the colon format dropped to 18.0 BLEU on Babbage and 27.6 on Curie, which the authors attribute to the example being read as part of the same passage [arxiv:2102.07350 §3.2] [arxiv:2102.07350 tab. 1]. Card: [Rethinking the Role of Demonstrations](/papers/2202.12837). This is the card to read before building an in-context learning baseline, see [in-context learning](/concepts/in-context-learning). Few-shot is not automatically the stronger setting for opinion distributions. On OpinionQA, a five-example prompt was worse than zero-shot question prompting on all four models tested [arxiv:2502.16761 tab. 1]. Card: [SubPOP](/papers/2502.16761). ### Serialized rows and task questions for data cleaning Each table row is written as "attribute: value" text, and each task gets its own fixed question. For entity matching the two records are introduced as products and the question asks whether they are the same. For imputation the known attributes come first and the missing attribute's name ends the prompt. For error detection the question names one cell [arxiv:2205.09911 §3.1] [arxiv:2205.09911 §3.2]. Attributes that do not help are dropped, chosen by experiment [arxiv:2205.09911 §3.1]. The paper gives no question templates for data transformation or schema matching [arxiv:2205.09911 §3.2]. Examples change the scores a lot. With ten demonstrations, GPT-3 175B reached 100 F1 on Beer and 98.2 on iTunes-Amazon, against 78.6 and 65.9 with none [arxiv:2205.09911 tab. 1]. On the transformation and schema tasks, three examples lifted StackOverflow from 32.7 to 65.3 and Bing-QueryLogs from 24.0 to 54.0 [arxiv:2205.09911 tab. 3]. The authors also report that a person picked the examples after reading validation errors, with up to one hour of work per task, and that this beat random examples by 14.7 F1 on average across three entity-matching sets [arxiv:2205.09911 §3.3] [arxiv:2205.09911 tab. 4]. The gap ranged from 8.9 points on Beer to 23.7 on Walmart-Amazon. Small wording changes moved scores in both directions. Replacing "same" with "equivalent" in the entity-matching question raised Walmart-Amazon from 88.9 to 100 and lowered Beer from 100 to 96.3 and iTunes-Amazon from 98.2 to 84.7 [arxiv:2205.09911 tab. 4]. The paper's 9.4-point average prompt effect is a mean of absolute differences; with signs kept, the three gaps average about 2 points [arxiv:2205.09911 tab. 4]. Read per dataset, not as one number. The paper also counts some ties with the baselines among its state-of-the-art results [arxiv:2205.09911 §4.2]. Card: [Can Foundation Models Wrangle Your Data?](/papers/2205.09911). ### Demonstration order Order is a separate lever from which examples are chosen. With four-shot demonstrations on SST-2, the 24 orders of one set spread widely: some reach over 85% and others sit near 50% [arxiv:2104.08786 fig. 1]. One order's accuracy also fell from 88.7% with GPT2-XL (1.5B) to 51.6% with GPT2-Large (0.8B) [arxiv:2104.08786 §2]. The orders that work well for one model size correlate weakly with those for another: the reported Spearman correlation between 2.7B and 175B is 0.05 [arxiv:2104.08786 fig. 4]. The paper's selection needs no labels. For each candidate order, the model writes unlabelled example inputs at temperature 2. Each order is then scored by the entropy of the labels it predicts on those inputs, and the top four are kept [arxiv:2104.08786 §3] [arxiv:2104.08786 §4]. Averaged over 11 text classification tasks, the authors report a 13% relative gain over unselected orders for this score, and 9.6% for a per-input variant [arxiv:2104.08786 §5] [arxiv:2104.08786 tab. 2]. Entropy ignores which label wins, so a split of 80/20 toward either label scores the same, and the paper does not discuss this [arxiv:2104.08786 eq. 3]. Card: [Fantastically Ordered Prompts](/papers/2104.08786). ### Reasoning cues A cue such as "Let's think step by step" makes the model write intermediate steps before a final answer. With two stages, the rationale is fed back with an answer trigger [arxiv:2302.06476 §3]. For ChatGPT, the cue lets it beat GPT-3.5 on all six arithmetic sets (five of six without it), but lowers CommonsenseQA from 73.7 to 71.5 and StrategyQA from 61.1 to 55.5 [arxiv:2302.06476 tab. 2] [arxiv:2302.06476 tab. 3]. The full treatment, with other CoT papers, is on the [chain-of-thought page](/concepts/chain-of-thought-prompting). The cue does not always work for a model that does not get a worked example. On GSM-IC, code-davinci-002 with the trigger "Let's think step by step:" scored 29.0% micro accuracy on the perturbed variants, while chain-of-thought with a worked exemplar scored 72.4% on the same variants [arxiv:2302.00093 tab. 3]. Card: [Large Language Models Can Be Easily Distracted by Irrelevant Context](/papers/2302.00093). On a multiple-choice exam benchmark, the effect of chain-of-thought depends on the question set. GPT-4o scored 72.6% with five-shot chain-of-thought on MMLU-Pro and 53.5% with direct answers, a gap of 19.1 points. On MMLU the same comparison gave 88.7% and 87.2%, a gap of 1.5 [arxiv:2406.01574 tab. 3]. Card: [MMLU-Pro](/papers/2406.01574). The paper's own numbers are author-reported. A prompt can also ask for a named style of reasoning. Four few-shot prompts for diagnosis (differential, intuitive, analytic and Bayesian reasoning) were compared with a traditional step-by-step prompt, on free-response questions from a modified MedQA USMLE set [arxiv:2308.06834 p. 1] [arxiv:2308.06834 tab. 2]. For GPT-4 (497 questions), the four styles scored 72-78% against 76% for the baseline, and no difference was significant [arxiv:2308.06834 p. 3]. For GPT-3.5 (518 questions), the differential, analytic and Bayesian prompts were 8, 6 and 4 points below the baseline, and the intuitive prompt was 1.7 points above it [arxiv:2308.06834 p. 3]. The prompts were picked on a 95-question development set with GPT-3.5 [arxiv:2308.06834 p. 4]. Card: [Diagnostic Reasoning Prompts](/papers/2308.06834). See also [self-consistency](/concepts/self-consistency). A review also quotes a result for "golden" chain-of-thought, where the prompt supplies ground-truth reasoning steps. On detective puzzles with GPT-4, it reports a solve rate of 83% against 38% for standard chain-of-thought. The review takes the figure from a cited paper and does not re-run it [arxiv:2310.14735 §3.1.2]. Card: [Unleashing the potential of prompt engineering](/papers/2310.14735). Prompt style and sampling temperature were tested together on GPT-3.5. Five one-shot styles (baseline, domain expert, self-recitation, chain of thought and a composite of these) were run at temperatures from 0.0 to 1.0 on a 100-question multiple-choice exam. The paper says chain of thought scored best and used it for its temperature sweeps, and the temperature test showed no effect for any style, with Kruskal-Wallis p-values from 0.996 to 1.000 [arxiv:2402.05201 §3.1] [arxiv:2402.05201 tab. 4]. Each prompt asked for a fixed answer line, and an answer that did not match counted as wrong. For Llama 2 7B, 39% of answers broke that format [arxiv:2402.05201 §3.1] [arxiv:2402.05201 app. A]. Card: [The Effect of Sampling Temperature on Problem Solving in Large Language Models](/papers/2402.05201). The temperature side is covered on the [decoding page](/concepts/decoding-strategies). ### Robustness to irrelevant context Adding one irrelevant sentence to a problem can change the answer even when the sentence does not change the solution. GSM-IC adds one such sentence to each of 100 easy GSM8K base problems, giving 58,052 variants [arxiv:2302.00093 §3.1]. For code-davinci-002, chain-of-thought scored 95.0% on the base problems and 72.4% micro accuracy on the variants [arxiv:2302.00093 tab. 2] [arxiv:2302.00093 tab. 3]. Three prompt-level changes helped on that set [arxiv:2302.00093 §5]: - An instruction before the exemplars, "Feel free to ignore irrelevant information", raised chain-of-thought from 72.4% to 77.8% micro accuracy [arxiv:2302.00093 tab. 3] [arxiv:2302.00093 §4.3]. - An exemplar that itself contains an irrelevant sentence raised it to 76.8% [arxiv:2302.00093 tab. 3]. On GSM8K dev the same change gave no gain (59.3 against 60.3 without it) [arxiv:2302.00093 tab. 5]. - Self-consistency, 20 samples at temperature 0.7 with a majority vote, raised least-to-most prompting from 77.5% to 93.4% micro accuracy. Macro accuracy, which counts a base problem only if all its variants are right, reached 45.0% at best [arxiv:2302.00093 tab. 3] [arxiv:2302.00093 §5.1]. The paper's text says self-consistency adds "more than 11 points" of micro accuracy. Its table gives 15.9 points for least-to-most. Card: [Large Language Models Can Be Easily Distracted by Irrelevant Context](/papers/2302.00093). ### Perturbed wording stress test Prompt wording can be attacked while the input stays clean. PromptRobust applies seven black-box text attacks to the instruction on four prompt types (zero-shot or few-shot, each task-oriented or role-oriented) across 13 datasets [arxiv:2306.04528 §2.2] [arxiv:2306.04528 tab. 1]. The families are character typos, word swaps with synonyms, an appended distractor sentence, and round trips through other languages, with the other-language prompts written by GPT-4 [arxiv:2306.04528 §2.2]. Word-level swaps cost the most: an average performance drop rate of 0.31 for TextFooler and 0.33 for BertAttack [arxiv:2306.04528 tab. 2]. The introduction gives 39% for the same effect [arxiv:2306.04528 §1], so use the table figure. Few-shot prompts were more robust than zero-shot ones on almost all datasets [arxiv:2306.04528 tab. 4]. GPT-4 and Flan-UL2 had the lowest average drops, both 0.08 [arxiv:2306.04528 tab. 3]. Transfer of the attacked prompts between models was weak and mixed [arxiv:2306.04528 tab. 6]. The numbers come from sampled subsets, and the paper does not state decoding settings [arxiv:2306.04528 §3]. Card: [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](/papers/2306.04528). ### Output format and layout LayoutGPT asks a text-only model to write each object as one CSS-style rule. The category is the selector, and the size and position are declarations. The values are normalised so the largest canvas side maps to 256 pixels, and retrieved examples are placed so the most similar one comes last [arxiv:2305.15393 §3.2] [arxiv:2305.15393 §3.3] [arxiv:2305.15393 fig. 2]. On 2D spatial prompts with GPT-3.5 and GLIGEN rendering, layout accuracy was 55.12 with neither the instruction nor the CSS format, and 82.54 with the instruction, the CSS format and normalisation together [arxiv:2305.15393 tab. 4]. The parts do not help equally. Normalisation alone lowered 2D layout accuracy to 44.10 [arxiv:2305.15393 tab. 4], but on 3D bedroom scenes the same normalisation lowered out-of-bounds furniture from 55.32% to 46.57% [arxiv:2305.15393 tab. 6]. Retrieved examples matter more in 3D: random examples raised bedroom out-of-bounds to 85.58% from 43.26%, but change 2D accuracy only slightly [arxiv:2305.15393 app. C.1] [arxiv:2305.15393 app. B]. Card: [LayoutGPT](/papers/2305.15393). ### Structured record with domain rules and JSON answer A trip choice can be sent to the model as one structured prompt with no training examples. The prompt has six parts in a fixed order: the task and the allowed options; the travel time and cost of each option as a nested dictionary; the traveller's attributes written as sentences, not dummy variables; domain sentences that compare the options (which is cheapest or fastest, and by what percentage); and a JSON answer with a "prediction" and a "reason" key [arxiv:2312.00819 §3.3] [arxiv:2312.00819 §3.3.6] [arxiv:2312.00819 app. A]. The model runs at temperature 0 [arxiv:2312.00819 §4.3]. The paper asks for JSON so results can be read by code, but it does not describe the parser or what happens when the output is malformed [arxiv:2312.00819 §3.3.5]. The same record can also feed a classic model. The paper embeds the prompt text and fits a multinomial logit, random forest or neural network on the vectors [arxiv:2312.00819 §3.2] [arxiv:2312.00819 §3.4]. With 10 training samples, the embedding raised the neural network from 0.462 to 0.532 mode accuracy [arxiv:2312.00819 tab. 3]. The embedding model's dimension is not stated [arxiv:2312.00819 §3.1]. In the ablation, removing the rules that guide the reasoning cost GPT-3.5 5.8% of accuracy and GPT-4 0.8% [arxiv:2312.00819 tab. 5]. Removing the domain sentences changed neither much. Card: [Large Language Models for Travel Behavior Prediction](/papers/2312.00819). ### Role profiles and fixed documents MetaGPT gives each agent a profile with a name, role, goal and constraints, and asks roles to hand over fixed documents, such as a requirements document or an interface design, rather than free-form chat [arxiv:2308.00352 §3.1] [arxiv:2308.00352 §3.2]. The authors argue that free chat distorts information over several rounds [arxiv:2308.00352 §1]. Their ablation varies the team, not the wording alone. Going from one engineer to four roles lowered human revisions from 10 to 2.5 and raised executability from 1.0 to 4.0, at about $0.92 to $1.39 per run [arxiv:2308.00352 §4.4] [arxiv:2308.00352 tab. 3]. Card: [MetaGPT](/papers/2308.00352). ### Task-name prefixes for a shared model Here one text-to-text model handles several dialogue sub-tasks. Each training sample starts with a short task prompt, such as "translate dialogue to belief state:", followed by the earlier turns, and the target is the expected text for that task [arxiv:2109.14739 §3.2] [arxiv:2109.14739 eq. 1]. Pre-training mixes eleven dialogue corpora, each labelled for only some of the four sub-tasks, and a sample is made only for the labels a corpus has [arxiv:2109.14739 §3.1] [arxiv:2109.14739 tab. 1]. The paper's prompts are fixed strings, and it reports no comparison of different wordings. On MultiWOZ 2.0 end-to-end, the base model reaches a combined score of 102.92, and at 1% of the training data it reaches 76.41, against 71.44 for the small model [arxiv:2109.14739 tab. 2] [arxiv:2109.14739 §4.1.4]. Card: [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](/papers/2109.14739). ### Extraction with few-shot pre-labels and human refinement A few-shot prompt is run on each chunk of a discharge summary, the model writes field values with character positions, and a resolver turns that text into structured entries. Unparsable output is logged and dropped [arxiv:2312.02296 §5] [arxiv:2312.02296 app. A.1]. The authors wrote 10 IOB-style prompts and 5 direct-chunk prompts, tuned them on a development set, and selected by F2, which weights recall above precision [arxiv:2312.02296 §5.4] [arxiv:2312.02296 app. A.2] [arxiv:2312.02296 app. A.3]. On 216 i2b2 2009 test documents, PaLM 2 without tuning gave a vertical F1 of 0.804 against 0.838 for the human base labels [arxiv:2312.02296 tab. 2]. After expert refinement, the LLM base reached 0.907 and the human base 0.910 [arxiv:2312.02296 tab. 2]. Mean rater time per document fell from 26.67 to 11.32 minutes, a reduction of 58% [arxiv:2312.02296 §6.1]. Card: [LLMs Accelerate Annotation for Medical Information Extraction](/papers/2312.02296). Formatting alone can change a score. FormatSpread defines prompt formats with a grammar of separators, spacing, label casing and enumeration styles, and treats formats generated by the same rules as equivalent [arxiv:2310.11324 §3.1]. On 53 Super-NaturalInstructions tasks, the median spread between the best and worst of 10 sampled formats was 7.5 accuracy points. For LLaMA-2 settings, 20% of tasks had a spread of 15 points or more [arxiv:2310.11324 §4.2]. The authors report that the spread does not shrink with model size, shot count or instruction tuning [arxiv:2310.11324 §4.2]. Card: [Quantifying Sensitivity to Spurious Features in Prompt Design](/papers/2310.11324). Searching all formats is expensive, so FormatSpread finds the best and worst format with Thompson sampling inside a fixed budget of model calls. At 51,200 evaluations, its gap to the true spread was within 1 point, against within 4 points for naive sampling and within 11 points for UCB [arxiv:2310.11324 §4.5]. The same question set can look much less sensitive. Across 24 prompt styles, scores on MMLU-Pro moved by about 2% in general, with a maximum of 3.74%. On MMLU, the same kind of change moved scores by 4-5% in general and up to 10.98% [arxiv:2406.01574 §6.3]. Card: [MMLU-Pro](/papers/2406.01574). ProSA measures sensitivity per question instead of per dataset. For each question it averages the absolute score gaps over every pair of prompt variants, then averages those values over the dataset [arxiv:2410.12405 eq. 1] [arxiv:2410.12405 eq. 2]. It uses 12 variants per question on four objective sets [arxiv:2410.12405 §3.1]. The paper reports that few-shot examples lower this score for all four Qwen1.5 models tested, with the largest drop from zero to one shot [arxiv:2410.12405 §3.4]. On the subjective benchmarks, Llama3-70B-Instruct scored 0.016 on AlpacaEval 2.0 and 0.258 on Arena Hard Auto, against 0.167 and 0.275 for the reference pair of responses [arxiv:2410.12405 tab. 2]. Card: [ProSA](/papers/2410.12405). The paper's own numbers are author-reported. ### Persona and steering prompts A steering prompt describes a person or group and asks the model to answer as them. Turing Experiments prompt a model zero-shot with a named participant and run the prompt for many names, then compare the outcomes with published human results [arxiv:2208.10264 §1] [arxiv:2208.10264 §2]. The recent models reproduced the Ultimatum Game, garden-path and Milgram trends. On a wisdom-of-crowds estimation task, the newest models answered with exact values, which people do not do [arxiv:2208.10264 §7] [arxiv:2208.10264 tab. 4]. All results are author-reported. Card: [Using LLMs to Simulate Multiple Humans](/papers/2208.10264). The authors ask that each prompt be tuned until its validity rate, the share of completions in the required form, is "sufficiently close to 1", before any hypothesis is tested [arxiv:2208.10264 §2]. They do not give a threshold. Validity ranged from 51.0% to 99.5% across the models and tasks in their table [arxiv:2208.10264 tab. 1]. Steering alone rarely matched the spread of human answers in survey data. On OpinionQA, the best zero-shot prompt reached a Wasserstein distance of 0.138 to 0.173 across four base models, and fine-tuning reached 0.094 to 0.106 [arxiv:2502.16761 tab. 1]. Chat models did worse than base models on this task: zero-shot question prompting gave 0.308 for Llama-2-7B-chat against 0.173 for the base model [arxiv:2502.16761 app. C.2]. Card: [SubPOP](/papers/2502.16761). A persona can also be a demographic profile rather than a named person. The World Values Survey study builds personas from region, sex, age, social class, education and marital status, with 303 matched personas per country [arxiv:2402.13231 §4.2] [arxiv:2402.13231 tab. 1]. Each of 30 survey questions is asked in four paraphrases, sampled five times at temperature 0.7, and the majority answer is kept [arxiv:2402.13231 §4.5]. Answers are scored by exact match (hard) and by a partial-credit error on ordinal items (soft) [arxiv:2402.13231 eq. 1]. For GPT-3.5 in English, the soft score is 48.61 on the Egypt survey and 64.86 on the US survey [arxiv:2402.13231 tab. 3]. Prompting in Arabic raised GPT-3.5's Egypt score by about 3 points [arxiv:2402.13231 tab. 2]. The paper's Anthropological Prompting variant lifts GPT-3.5's Egypt soft score from 0.4834 to 0.5102 against a five-sample vote [arxiv:2402.13231 tab. 4]. The authors also report that alignment is uneven across social classes and education levels, but they give those values only in a figure [arxiv:2402.13231 §5.3] [arxiv:2402.13231 fig. 3]. Card: [Investigating Cultural Alignment of Large Language Models](/papers/2402.13231). ### Backstory personas for survey simulation A persona can also be built from a respondent's own answers. The "silicon sample" method writes a first-person backstory for each survey respondent, from fields such as age, party and church attendance, and asks GPT-3 (Davinci, 175B) to continue it [arxiv:2209.06899 §4] [arxiv:2209.06899 app. 3]. For the 2016 vote, the prompt ends "In 2016, I voted for", and the vote is read from the next-token probabilities [arxiv:2209.06899 app. 3]. Across the whole sample, the tetrachoric correlation with the real vote is 0.90 to 0.94 for the three ANES waves. For independents it is 0.41 in 2016 and 0.02 in 2020 [arxiv:2209.06899 tab. 1]. So an aggregate match can hide a subgroup mismatch. In a Turing-style check, human judges labelled GPT-3 lists as human 61.2% of the time, against 61.7% for human-written lists [arxiv:2209.06899 §5]. The paper does not say whether the model used is still served, so access to token probabilities is a practical limit [arxiv:2209.06899 §2]. Card: [Out of One, Many](/papers/2209.06899). ### Country of birth and residence in the persona Cultural prompting names a country in the respondent descriptor. The model is told it is an average person born and living in that country, and the survey question follows as usual. The paper compares this with a default descriptor that names no country [arxiv:2311.14096 §4.2] [arxiv:2311.14096 §4.3]. Across 107 countries and territories, the distance from each country's survey values fell for all five GPT models. For GPT-4o the mean fell from 2.42 to 1.57, and the share of countries that improved ranged from 71.0% to 81.3% across models [arxiv:2311.14096 §2]. The authors say 19 to 29% of countries did not improve, or got worse [arxiv:2311.14096 §3]. For GPT-4o, Finland, Luxembourg, Andorra, Switzerland and Taiwan moved further away [arxiv:2311.14096 §2]. The prompts were English only, and the distance is measured on one cultural map built from one survey [arxiv:2311.14096 §3] [arxiv:2311.14096 §4.1]. Card: [Cultural Bias and Cultural Alignment of Large Language Models](/papers/2311.14096). ### Culture phrase in English or the target language A value survey can be asked three ways: in English with a nationality phrase ("for an average American"), in the target country's language, or in English with a "culture setting" phrase. The study uses 24 Hofstede survey items for five cultures in these three forms [arxiv:2303.17466 §3.1] [arxiv:2303.17466 tab. 1]. Each culture gets six dimension scores from a weighted difference of item pairs [arxiv:2303.17466 eq. 1]. The three forms do not agree equally well. For prompts 1 and 3, the share of items with the same score was above 70% for the US, Germany, Japan and Spain, and 58.33% for China [arxiv:2303.17466 tab. 3]. The authors report that most cultures align better with human scores when asked in their own language [arxiv:2303.17466 §4.2]. Only one Spearman cell reaches p < 0.05: Spain, prompts 1 and 2, at 0.93 [arxiv:2303.17466 tab. 5]. The paper does not state the sample size behind the correlations, so the p-values cannot be checked [arxiv:2303.17466 §4]. The same item can also score differently by language: for a Japanese-culture item on personal time, the English prompt scored 1.0 and the Japanese prompt 3.0 [arxiv:2303.17466 tab. 7]. Card: [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies](/papers/2303.17466). ### Disclosed experiment design When a model plays the subjects of a randomised experiment, the usual prompt shows one scenario and says nothing about the design. In a price study, the model also changed other details it invented, such as the last price paid, so the price effect mixed with those details [arxiv:2312.15524 §2] [arxiv:2312.15524 app. B]. Adding those details as stated covariates was not a clean fix. The error on purchase probability was lowest at stage 2 (0.0971) and rose over later stages, with 0.2025 at stage 11 [arxiv:2312.15524 app. E]. The unblinded prompt tells the model that the price is drawn at random between a minimum and a maximum, and that the customer sees one price without knowing the draw [arxiv:2312.15524 §5]. For gpt-4o-mini, the error against a 40-product human survey was 0.532 with the blinded prompt and 0.397 with the unblinded prompt, both without fine-tuning [arxiv:2312.15524 tab. 1]. The paper reports lower error with unblinded prompts for every model it tested, with relative gains from about 1% to over 60%, read from a figure [arxiv:2312.15524 §5] [arxiv:2312.15524 fig. 6]. The minimum and maximum values are not stated [arxiv:2312.15524 §5]. The evidence comes from one demand study and one human survey [arxiv:2312.15524 §2]. Card: [Gui and Toubia, causal-inference study of LLM simulation](/papers/2312.15524). ### Per-annotator examples in the prompt A personalised prompt lists up to three labelled texts from one annotator before the target text, so the model is asked to label the way that annotator did [arxiv:2302.10724 §6.3]. On AggressionPer, ChatGPT scored 72.57 F1, against 69.10 on the plain Aggression task. The two rows have different references (81.03 and 74.45), so the gap is not a clean personalisation effect [arxiv:2302.10724 tab. 2]. On the GoEmo variants GoEmoPer0 to GoEmoPer3, scores ranged from 19.00 to 23.74 and did not improve with each step [arxiv:2302.10724 tab. 2]. The paper's text gives a 4.08-point gain for GoEmoPer3 over GoEmoPer0, but its table shows 23.41 against 23.74 [arxiv:2302.10724 §6.3] [arxiv:2302.10724 tab. 2]. Card: [ChatGPT: Jack of all trades, master of none](/papers/2302.10724). ### Low-perplexity prompt selection With no labels, a few hand-written prompts can be expanded and ranked by how familiar the model finds them. SPELL asks GPT-3 for paraphrases of each seed, backtranslates them through other languages, and keeps the three prompts with the lowest mean perplexity over about 1,000 task inputs. The label is not used [arxiv:2212.04037 §3] [arxiv:2212.04037 §7]. On OPT and Bloom, zero-shot scores mostly correlated negatively with perplexity, though not on every task; IMDB with OPT 175B was positive [arxiv:2212.04037 tab. 5]. The author-reported gain over manual prompts was 1.8 accuracy points for OPT and 2.3 for Bloom, but single tasks moved from -12.5 to +10.2 points, and the paper gives no significance test for the gain [arxiv:2212.04037 tab. 10] [arxiv:2212.04037 §7.1]. Card: [Demystifying Prompts in Language Models via Perplexity Estimation](/papers/2212.04037). ### Learned soft prompts Here the prompt is not text. The cloze-style prompt tokens become trainable vectors, and each layer gets a trainable offset. The language model stays frozen; only these vectors are fitted, with gradient descent on known question and answer pairs [arxiv:2104.06599 §3.1] [arxiv:2104.06599 §3.2]. Several such prompts are mixed, with the mixture weights learned too [arxiv:2104.06599 §3.3]. On T-REx with BERT-large-cased, the best soft mixture reaches P@1 51.6, against 39.4 for LPAQA as reimplemented by the authors [arxiv:2104.06599 tab. 1]. Initialising the vectors at random, not from a hand-written sentence, scored 51.9 on the same setting [arxiv:2104.06599 tab. 1]. Tuning only the input layer gave 51.0 [arxiv:2104.06599 tab. 5]. The learned vectors do not read as words; the authors' nearest-word check finds topical but not phrase-like neighbours [arxiv:2104.06599 app. C]. Card: [Learning How to Ask](/papers/2104.06599). This needs gradient access to the model and a labelled set per relation; the paper's setting is one relation at a time, with BERT and RoBERTa. P-Tuning keeps the words of a hand-written template and adds trainable continuous vectors beside them, passed through a small LSTM or MLP [arxiv:2103.10385 §2.2] [arxiv:2103.10385 §2.3]. With a frozen BERT-base model on LAMA-34k, precision@1 is 48.3, against 43.3 for AutoPrompt's discrete search [arxiv:2103.10385 tab. 3]. For one LAMA relation, the spread across six manual prompts falls from 10.1 to 0.46 [arxiv:2103.10385 tab. 6]. On few-shot SuperGLUE, with a tuned ALBERT-xxlarge-v2, the seven-task average is 71.81 against 70.74 for PET-FT. That gain of about one point sits inside the reported standard deviations on several tasks [arxiv:2103.10385 tab. 5] [arxiv:2103.10385 §3.3]. The released LAMA code does not insert the relation's hand-written prompt that the paper describes, so a faithful reimplementation must add it [code: LAMA/p_tuning/modeling.py#L74-L105]. Card: [GPT Understands, Too](/papers/2103.10385). PPT adds a pre-training stage for the soft prompt. Before any labelled task, the prompt is trained on unlabelled text with three cloze tasks, one for sentence pairs, one for multiple choice and one for single sentences. The downstream task then starts from that prompt instead of a random one [arxiv:2109.04332 §3.1] [arxiv:2109.04332 §3.2]. With T5-XXL and 32 training examples per task, the authors report 93.5 ± 0.3 on SST-2 for PPT, against 91.4 ± 0.8 for full fine-tuning and 70.5 ± 15.5 for prompt tuning from a random start [arxiv:2109.04332 tab. 4]. They say PPT matches or beats full fine-tuning on most English and all Chinese datasets they tested, with lower variance across seeds [arxiv:2109.04332 §4.2]. Pre-training runs for up to 200,000 steps, once, and the resulting prompts are reused across tasks [arxiv:2109.04332 app. C] [arxiv:2109.04332 app. E]. Card: [PPT: Pre-trained Prompt Tuning for Few-shot Learning](/papers/2109.04332). The prompt-programming paper proposes a generic seed, "Let's solve this problem by splitting it into steps," that the model expands into its own procedure. It reports no accuracy for this. Its Figure 3 shows a wrong answer for f(f(3)) with f(x) = x*x, where the correct value is 81 and the output says 27 [arxiv:2102.07350 §4.7] [arxiv:2102.07350 fig. 3]. The paper also suggests reading a verdict out at the point where the model's probability of the phrase "Thus, the correct answer is" peaks. It gives no threshold for detecting that peak [arxiv:2102.07350 §4.6]. ### Text prefix for a frozen forecaster Time-LLM keeps a 7B language model frozen. Each series patch is mapped onto learned text-like prototypes, and a text prompt is placed in front of those patches [arxiv:2310.01728 §3.1]. The prompt has three parts: a dataset description, a fixed task instruction, and input statistics such as trends and top lags [arxiv:2310.01728 §3.1] [arxiv:2310.01728 app. B.1]. Only the input layers and the output projection are trained. The prompt positions are dropped before the forecast is produced [arxiv:2310.01728 §3.1]. The authors report ETTh1 long-term MSE 0.408 against 0.413 for PatchTST [arxiv:2310.01728 §4.1]. Removing the whole prefix raised ETTh1-96 MSE from 0.362 to 0.398 [arxiv:2310.01728 §4.5]. Removing the dataset description, the task instruction or the statistics gave 0.402, 0.388 and 0.391 on the same setting, and 0.298, 0.285 and 0.279 on ETTm1-96 against 0.272 [arxiv:2310.01728 §4.5]. These are single ablation rows, each averaged over three runs. The paper does not report how much of the compute the prompt uses [arxiv:2310.01728 §4.5]. Card: [Time-LLM](/papers/2310.01728). The shipped code differs from the paper's prompt description. It adds min, median and max values, and it uses one fixed ETT sentence for every dataset unless a prompt domain is set, which no shipped script does [arxiv:2310.01728 §3.1]. Implement the prompt from the code if you want to match the reported numbers. ### Search over prompted steps Tree of Thoughts treats each step as a thought. The model proposes candidate next steps, scores partial solutions, and a breadth- or depth-first search keeps the good branches and backtracks from the rest [arxiv:2305.10601 §3]. With GPT-4, Game of 24 success rises from 4% for CoT to 74% for this search [arxiv:2305.10601 tab. 2]. With GPT-3.5 the gain is smaller: 19% against 3% [arxiv:2305.10601 tab. 5]. A search costs more: $0.74 per problem against $0.47 for best-of-100 CoT on Game of 24 [arxiv:2305.10601 tab. 7]. Card: [Tree of Thoughts](/papers/2305.10601). See also [LLM tree search](/concepts/llm-tree-search). ### Automated prompt search OPRO uses an LLM as the optimizer. Each step shows it earlier instructions with their training scores, sorted so the best come last, plus three training exemplars. It proposes eight candidates at temperature 1.0, and each candidate is scored on a fixed training subset before it joins the list [arxiv:2309.03409 §2.3] [arxiv:2309.03409 §4.2] [arxiv:2309.03409 §5.1]. For GSM8K the training subset is 3.5% of the training set (about 261 questions, derived), and the search runs for 200 steps [arxiv:2309.03409 §5.1]. With PaLM 2-L as the scorer, the best instruction reaches 80.2 on the GSM8K test set, against 71.8 for "Let's think step by step." [arxiv:2309.03409 tab. 4]. That table picks each instruction by test accuracy, which inflates the figure. The authors suggest choosing by training accuracy [arxiv:2309.03409 §5.4], and training accuracy ran 5 to 20 points above test accuracy in their runs [arxiv:2309.03409 §5.4]. Scorer calls are the main cost: at 200 steps with 8 candidates on about 261 questions, the worst case is about 418,000 calls before deduplication (our arithmetic) [arxiv:2309.03409 §5.1]. Synonym-level wording changes also move accuracy a lot [arxiv:2309.03409 §5.2.3]. Card: [Large Language Models as Optimizers](/papers/2309.03409). The same loop fits small linear-regression and TSP instances, and it reaches the optimum less often as those grow [arxiv:2309.03409 §3.2] [arxiv:2309.03409 tab. 3]. ### Prompt-averaged scoring Prompts can steer a scorer as well as a generator. BARTScore rates generated text by the log-likelihood a pre-trained BART assigns to it, given a source or a reference, with no human labels [arxiv:2106.11520 §3.2]. Its prompt variant appends or prepends a short phrase such as "in summary" or "in other words", and averages the scores over 70 or 34 such phrasings [arxiv:2106.11520 §3.3.1] [arxiv:2106.11520 tab. 2]. The authors report that prompts help on most settings, but not on factuality [arxiv:2106.11520 §4.3.2]. Card: [BARTScore](/papers/2106.11520). ### Many prompts per task at training time T0 changes the training data instead of prompting a frozen model. The authors write 62 English datasets in 12 task types as prompts, and the collection has 2,073 prompts for 177 datasets, about 11.7 per dataset [arxiv:2110.08207 §3] [arxiv:2110.08207 §4]. Each example is rendered through each of its dataset's templates, and the cap for large datasets is 500,000 examples shared across those templates [arxiv:2110.08207 §5]. Accuracy on four held-out task types is scored by rank classification, and the reported number is the median over each dataset's prompts [arxiv:2110.08207 §5]. T0 (11B) matches or exceeds GPT-3 on 9 of 11 held-out datasets (author-reported) [arxiv:2110.08207 §6.1]. The comparison is not like for like: GPT-3 reports one prompt per dataset, and T0 uses all of its prompts [arxiv:2110.08207 §6.1]. Using 5.7 original prompts per dataset rather than one raised the median on 8 of 11 datasets and shrank the spread on 7 of 11 [arxiv:2110.08207 §6.2]. Datasets used in training are greyed out and are not zero-shot results [arxiv:2110.08207 tab. 7]. The paper reports about 270 TPU v3-512 hours for all experiments [arxiv:2110.08207 app. B.1]. Card: [Multitask Prompted Training](/papers/2110.08207). ### Explicit rules for a redaction job De-identifying clinical notes is prompt-only in DeID-GPT. The system message holds a task statement, a command to replace identifiers with "[redacted]", and rules that list the kinds of identifier, and the note goes in the user message [arxiv:2303.11032 §IV] [arxiv:2303.11032 fig. 7] [arxiv:2303.11032 fig. 8]. On the 2014 i2b2/UTHealth test set, ChatGPT's entity-wise score rose from 0.686 with an implicit prompt to 0.929 with the explicit one [arxiv:2303.11032 tab. 2]. The GPT-4 score of 0.99 with the explicit prompt comes from 50 samples run by hand in the web interface, so it is not comparable with the full-set rows [arxiv:2303.11032 §V] [arxiv:2303.11032 tab. 2]. Prompt wording is fragile: an extra full stop stopped the command being read, and combining tasks confused the model [arxiv:2303.11032 fig. 9]. The paper also says an API-only model cannot take patient text in a hospital, and it calls for local models [arxiv:2303.11032 §VI]. Card: [DeID-GPT](/papers/2303.11032). ### Codebook-centred prompt with one example per code For deductive coding, the codebook is the prompt. Each code gets a name, a description and examples, and the model must pick one of the listed codes [arxiv:2304.10548 §3]. The authors compared a codebook-centred layout with an example-centred one, where each example comes with a sentence on why it has its code, and varied the examples per code between none, one and five [arxiv:2304.10548 §3]. GPT-3 (davinci-text-002) at temperature 0 reached Cohen's kappa of 0.61 against the experts' final codes for question complexity and 0.38 for syntactic structure. The two experts reached 0.88 and 0.90 with each other [arxiv:2304.10548 §4]. The largest gain came from zero to one example per code; codebook-centred prompts scored higher [arxiv:2304.10548 §4] [arxiv:2304.10548 fig. 1]. Complexity was coded on GPT-3's English translations, not the French originals [arxiv:2304.10548 §2.1]. The study covers one dataset of 668 questions and one codebook, and the authors call it preliminary [arxiv:2304.10548 §5]. Card: [Xiao et al., codebook prompts with GPT-3](/papers/2304.10548). ### Codebook prompt for annotation For labelling, the prompt is the same codebook that trained human coders used, with no examples. ChatGPT (gpt-3.5-turbo) receives each text in a fresh chat with the wording "Here's the tweet I picked, please label it as [task]." [arxiv:2303.15056 §4.5]. Averaged over four samples, its accuracy against trained annotators was about 25 points above that of MTurk workers (author-reported) [arxiv:2303.15056 §2]. On the four relevance tasks it ranged from 59% to 83% [arxiv:2303.15056 §2] [arxiv:2303.15056 fig. 1]. Intercoder agreement was 91% at the default temperature and 97% at 0.2, against 79% for trained annotators and 56% for MTurk [arxiv:2303.15056 §2]. That agreement is raw percent match with no chance correction, so skewed classes look better than they are [arxiv:2303.15056 §4.6]. The paper tests no few-shot prompts [arxiv:2303.15056 §3]. Card: [ChatGPT Outperforms Crowd-Workers](/papers/2303.15056). ### Demonstration prompts that write labelled data Each generated sentence comes from a prompt of five demonstrations about one group, drawn at random for every sentence. Decoding uses top-k sampling and stops at the first newline [arxiv:2203.09509 §3.1] [arxiv:2203.09509 app. A]. The authors used 26 prompt sets (toxic and benign for each of 13 groups) and kept 274,186 statements, of which 98.2% contain no profanity or slur by their word-list test [arxiv:2203.09509 tab. 2]. A second decoder, ALICE, uses a toxicity classifier during beam search to push the text towards sentences the classifier misreads. It supplies about 5% of the statements [arxiv:2203.09509 §3.2] [arxiv:2203.09509 tab. 2]. Fine-tuning HateBERT on the combined data raised AUC on three human-written implicit-hate sets by 11, 7 and 19 points [arxiv:2203.09509 tab. 4]. The training label is the prompt's label, not a human's. Human raters agreed with the prompt label for 67.7% of top-k toxic prompts and 40.3% of ALICE toxic prompts [arxiv:2203.09509 §4.3]. The table values were transcribed from a rendered page, so check them against the paper. The released code applies the classifier term twice, which the paper does not describe [arxiv:2203.09509 §3.3]. Card: [ToxiGen](/papers/2203.09509). ### Staged Python-style plans for robot teams One instruction for a group of robots is planned in stages. The prompt asks the model to split the task into sub-tasks with their skills, choose one robot or a team for each, and then write code that assigns the work and runs independent steps side by side [arxiv:2309.10062 §4] [arxiv:2309.10062 fig. 2]. Skills and objects appear as Python dictionaries, with few-shot examples of 5 for decomposition, 3 for coalitions and 4 for allocation code [arxiv:2309.10062 §5.2]. On 36 AI2-THOR tasks, GPT-4 reached a success rate of 1.00 on elemental tasks and 0.62 to 0.71 on the other three categories [arxiv:2309.10062 tab. 1]. In a five-run check on one task per category, the complex category scored 0.48 ± 0.40, below the 0.71 in the benchmark table; the paper does not explain the gap [arxiv:2309.10062 tab. 2] [arxiv:2309.10062 §6.1]. Nothing checks the generated code between planning and execution [arxiv:2309.10062 §4.4]. Real-robot runs have no metrics [arxiv:2309.10062 §6.2]. Card: [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](/papers/2309.10062). ### Two chat agents with inception prompts CAMEL runs a role-play between an assistant and a user, both chat models. A task-specifier agent first turns an idea into a concrete task [arxiv:2303.17760 §3.1] [arxiv:2303.17760 fig. 2]. Behaviour comes from fixed system prompts. The assistant's prompt names both roles and forbids swapping them, and it requires an answer that starts with "Solution:" and ends with "Next request." [arxiv:2303.17760 §3.2] [arxiv:2303.17760 fig. 2]. The user's prompt asks for one instruction per turn, and for the stop token "" only once the task is solved [arxiv:2303.17760 §3.2]. The authors list four failure modes seen in pilot runs: role flipping, the assistant repeating instructions, replies that only promise action ("I will ..."), and endless polite loops [arxiv:2303.17760 §1] [arxiv:2303.17760 §4]. Against a single-shot gpt-3.5-turbo answer, human voters preferred CAMEL on 76.3% of 100 AI Society tasks, with 10.4% for the single shot and 13.3% draws [arxiv:2303.17760 tab. 1]. The repository's prompts differ from the paper's, so replicate from the paper's text [arxiv:2303.17760 fig. 2]. Card: [CAMEL](/papers/2303.17760). Compare [role profiles](#role-profiles-and-fixed-documents). ### Single prompt on finance datasets A survey of LLMs in finance also reports one GPT-4 evaluation on six datasets. The prompt has three parts, a role description, an output format and one example, and it goes in the system message [arxiv:2401.11641 §4.2]. Reported accuracies are 0.78 on Financial PhraseBank, 0.79 on FiQA sentiment, 0.81 entity F1 on financial NER, 0.64 on FinQA, 0.73 on ConvFinQA and 0.53 on stock movement [arxiv:2401.11641 tab. 2]. Text labelling scores best, and numeric questions and price direction score lower [arxiv:2401.11641 tab. 2]. There is no baseline, and the model snapshot and decoding settings are not given [arxiv:2401.11641 §4.2] [arxiv:2401.11641 tab. 2]. The stock labels leave a band between -0.5% and +0.55% with no class [arxiv:2401.11641 §4.1]. Read these as what one prompt did, not as a benchmark. Card: [Revolutionizing Finance with LLMs](/papers/2401.11641). ### Safety system prompts tested on scenarios The mental-health study runs each question as its own prompt, with a system message that argues against stigma (the "steel-man" prompt), a letter-answer rule, one worked example, and temperature 0 [arxiv:2504.18412 §4] [arxiv:2504.18412 app. Steel-man system prompt]. Its stigma probe is 72 vignettes, asked 14 questions each, which gives 1008 prompts per model [arxiv:2504.18412 §4]. Even with the steel-man prompt, the overall rate of stigmatising answers ranged from 38% for gpt-4o to 75% for Llama 3.1 405B [arxiv:2504.18412 §4.1]. On crisis stimuli, the crisis experiment uses the same prompt, and the models gave under 80% appropriate replies overall, against 93% for 16 human therapists [arxiv:2504.18412 §5.1]. The paper says the steel-man prompt "dramatically improves" overall crisis performance but gives no number for that effect [arxiv:2504.18412 §5.1]. Card: [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](/papers/2504.18412). All numbers are author-reported. ### Persona prompts scored for marked words To find stereotypes without a word list, ask the model for a short persona of each group, then compare each marked group with the unmarked defaults, White and man [arxiv:2305.18189 §3.1] [arxiv:2305.18189 §3.2]. The main run used six prompt templates, temperature 1 and 15 samples per group and prompt, giving 1,350 personas per model [arxiv:2305.18189 §4]. A word is marked when its weighted log-odds difference is significant (z > 1.96) against both defaults [arxiv:2305.18189 §3.2]. The authors report that GPT-4 and GPT-3.5 personas contain more racial stereotype lexicon words than human-written personas for the same prompts; the figure has no numbers in the text [arxiv:2305.18189 §5] [arxiv:2305.18189 fig. 1]. The top marked words were mostly positive in sentiment, with a mean VADER score of 0.05, so a sentiment filter would miss them [arxiv:2305.18189 §6] [arxiv:2305.18189 tab. 2]. The default group is an a-priori choice that reflects internet-data norms [arxiv:2305.18189 §9]. Card: [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](/papers/2305.18189). ### Positive prefix against biased completions A short positive statement put before the prompt changes how a model completes text about a group. With GPT-3 davinci, "Two Muslims walked into a" gave violence-related completions in 66 of 100 runs [arxiv:2101.05783 §Results] [arxiv:2101.05783 fig. 1]. With a prefix such as "Muslims are hard-working.", the mean over 50 positive adjectives was 46%. The six best adjectives reached 20%, against 13-15% for the Christian prompt [arxiv:2101.05783 §Results] [arxiv:2101.05783 fig. 4]. The six were picked from the first run and then rerun; the paper does not say whether the rerun used fresh completions. The authors call this "debiasing" only loosely, say completions still carry bias, and report that the prefix steers output toward particular topics [arxiv:2101.05783 §Results] [arxiv:2101.05783 §Discussion]. Every rate depends on a hand-made keyword list [arxiv:2101.05783 app. B]. Card: [Persistent Anti-Muslim Bias in Large Language Models](/papers/2101.05783). ### Prompt-only guardrails in a deployed system A university tutoring assistant uses retrieval over course material, then tells the model to say "I'm not sure" rather than guess [arxiv:2309.10892 §3.1.1] [arxiv:2309.10892 §3.2.3]. Its only hallucination control is this prompt [arxiv:2309.10892 §3.1.1] [arxiv:2309.10892 §3.2.3]. The paper reports no measurement of answer correctness or of how often the instruction works [arxiv:2309.10892 §4]. Card: [AI-enabled intelligent assistant for higher education](/papers/2309.10892). A 2023 survey of ChatGPT research lists prompting ("cueing") techniques among application areas, without describing any [arxiv:2304.01852 §2.1.6]. Card: [Early ChatGPT research survey](/papers/2304.01852). ### One suggested reply per teaching strategy Tutor CoPilot shows a tutor, inside a live chat, one suggested reply for each of several teaching strategies. Each reply is a separate GPT-4 call built from the lesson topic, the chosen strategy and the most recent messages. The paper sends only the 10 latest messages to the external model [arxiv:2410.03017 §Methods] [arxiv:2410.03017 fig. 1]. The tutor can edit a reply, regenerate it or choose another strategy, so the tutor makes the final choice [arxiv:2410.03017 §Methods]. The paper does not state the prompt text, and the one-sentence reply limit in the public demo is not in the paper [arxiv:2410.03017 §Methods]. In a randomized trial of 783 tutors, access to the tool raised the exit-ticket pass rate from 61.7% to 65.7% [arxiv:2410.03017 tab. 1]. The 14-point figure for tutors who used the tool comes from an instrumental-variable estimate, not from usage logs [arxiv:2410.03017 §Results]. Treated tutors used "prompt student to explain" about 10% more often [arxiv:2410.03017 fig. 3]. The end-of-year test effect was not estimated [arxiv:2410.03017 §Methods]. Card: [Tutor CoPilot](/papers/2410.03017). ### Plain-language rewrite of a report A one-line request asks the model to rewrite a medical report for a lay reader. The prompt asks the model to explain the report to a child in plain words, then gives the report text [arxiv:2212.14882 §3.2]. The authors picked it by informal trial from six variants on one other fictitious report, and they do not claim it is the best one [arxiv:2212.14882 §3.2] [arxiv:2212.14882 §5.3]. Each report was sent 15 times, each time in a fresh chat, and 15 radiologists rated the rewrites. The median rating was 2 (agree) for factual correctness and for completeness, and 4 (disagree) for potential harm [arxiv:2212.14882 tab. 1]. Raters still highlighted wrong passages, invented findings and dropped results. The table gives the share answering each follow-up as 51%, 22% and 36% [arxiv:2212.14882 tab. 3] [arxiv:2212.14882 §4.2]. The study compares no other prompt or simplifier, so it does not show that this wording beats another [arxiv:2212.14882 §3.2]. The interface offers no temperature or seed control, which the authors cite as a reproducibility problem [arxiv:2212.14882 §5.2]. Card: [ChatGPT Makes Medicine Easy to Swallow](/papers/2212.14882). ## When to use which | Situation | Use | Why | |---|---|---| | Strong chat model, no labelled examples, task has a clear answer form | Zero-shot instruction with a strict answer format | ChatGPT beats GPT-3.5 on most NLI and sentiment sets zero-shot, but trails on summarization and NER [arxiv:2302.06476 §4.2] | | Translation-style task on a base model, one fixed format | Zero-shot colon format, then check against a few-shot baseline | Zero-shot ties 10-shot on Curie and trails it on Babbage [arxiv:2102.07350 tab. 1]; only two small models were tested | | Classification or multi-choice, want a baseline without labelled pairs | Inputs with random labels, same format | Gold-vs-random gap of 0-5 points [arxiv:2202.12837 §4.1]; direct models lose more when label words are replaced [arxiv:2202.12837 §5.2] | | Arithmetic word problems, one prompt | Zero-shot CoT cue | With the cue ChatGPT beats GPT-3.5 on all six arithmetic sets; without it, five of six [arxiv:2302.06476 tab. 2] | | Commonsense or entailment questions | Measure with and without the cue | The same cue lowered ChatGPT's CommonsenseQA and StrategyQA scores [arxiv:2302.06476 tab. 3] | | Inputs may contain unrelated details | Instruction to ignore irrelevant information, plus self-consistency over an exemplar | Micro accuracy 77.5 to 93.4 for least-to-most with SC; macro accuracy still at most 45.0 [arxiv:2302.00093 tab. 3]. Measured on one model and one maths set | | Early steps decide success, and a model can judge partial progress | Search over steps | 74% against 4% on Game of 24 with GPT-4 [arxiv:2305.10601 tab. 2]; 5-100x the generated tokens of CoT [arxiv:2305.10601 app. B.3] | | Domain reasoning with a strong model, no step list | Named reasoning style, compared with a plain step-by-step baseline | GPT-4 kept its baseline accuracy, with no significant difference [arxiv:2308.06834 p. 3]; GPT-3.5 lost 4-8 points on three styles | | Output must have coordinates or a fixed structure | Per-object rules with normalised values and retrieved examples | GPT-3.5 spatial layout accuracy 55.12 to 82.54 with all parts [arxiv:2305.15393 tab. 4]; 2D numbers use GLIGEN for rendering | | Simulate a group's answer spread, survey data and weight access | Fine-tune on the human distribution | Wasserstein 0.094-0.106 against 0.138-0.173 for the best zero-shot prompt [arxiv:2502.16761 tab. 1]; chat models were worse [arxiv:2502.16761 app. C.2] | | Simulate many participants of a classic study, no data | Zero-shot persona prompt per participant, check validity first | Trends reproduced on three of four studies [arxiv:2208.10264 §7]; results are author-reported | | Multi-step task that needs several roles | Role profiles, fixed documents, executable feedback | One engineer against four roles: revisions 10 to 2.5 [arxiv:2308.00352 tab. 3]; costs about $0.92 to $1.39 per run | | Weight access, a labelled set per relation, a frozen LM | Learned soft prompts | P@1 51.6 against 39.4 for LPAQA on T-REx with BERT-large [arxiv:2104.06599 tab. 1]; only BERT and RoBERTa were tested | | Weight access, about 32 labelled examples per task, an 11B-class model, and unlabelled text to pre-train on | Pre-trained soft prompt (PPT), then tune per task | SST-2 93.5 ± 0.3 against 91.4 ± 0.8 for full fine-tuning and 70.5 ± 15.5 for a random-start prompt [arxiv:2109.04332 tab. 4]; PPT was tested only at 11B, and pre-training runs up to 200,000 steps [arxiv:2109.04332 app. C] | | Forecast numeric series with a frozen 7B LLM, no retraining of the LLM | Dataset description and statistics as a text prefix, with trained input and output layers | ETTh1 long-term MSE 0.408 against 0.413 for PatchTST [arxiv:2310.01728 §4.1]; the 7B backbone alone used 32,136 MiB at ETTh1-96 [arxiv:2310.01728 §4.5]; the paper's own ablation shows the prefix matters [arxiv:2310.01728 §4.5] | | Safety-relevant answers from a retrieval-backed bot | Prompt plus retrieval gate, then your own evaluation | The prompt is the only control, and the paper measures nothing [arxiv:2309.10892 §5.1] | | Comparing prompts or models on a benchmark | Sweep several formats and report the spread, not one score | Median spread 7.5 points over 10 formats per task [arxiv:2310.11324 §4.2]; a single score hides the spread | | Per-question stability of a prompt matters, and about 12 variants per question are affordable | Per-question sensitivity over prompt variants, averaged over the dataset | Defined per item, so a stable average can hide flips [arxiv:2410.12405 §2.1] [arxiv:2410.12405 eq. 2]; the paper's results are on four objective sets and two chat benchmarks | | Many dialogue sub-tasks, labelled data for only some of them | One model with a task-name prefix per sub-task, mixed pre-training | Combined score 102.92 on MultiWOZ 2.0 with the base model [arxiv:2109.14739 tab. 2]; the paper has no wording ablation | | Clinical or domain text to label, experts with time limits, a few-shot budget | Few-shot pre-labels per chunk, then expert refinement | Mean time 26.67 to 11.32 minutes; refined F1 0.907 against 0.910 for human base labels [arxiv:2312.02296 tab. 2]; one model, one dataset | | Estimate a population's answers without weight access | Demographic persona prompts scored with soft and hard alignment | Soft score 48.61 (Egypt) and 64.86 (US) for GPT-3.5 in English [arxiv:2402.13231 tab. 3]; sub-group results are only in figures | | Sensitive replies to users, such as mental-health chat | Safety system prompt plus scenario tests before any deployment | Stigmatising answers 38-75% and under 80% appropriate crisis replies, both with the steel-man prompt [arxiv:2504.18412 §4.1] [arxiv:2504.18412 §5.1]; the paper argues against replacing therapists | | Few-shot set is fixed and there is no labelled dev set | Score candidate orders by label entropy on model-written probe inputs, keep the top four | About 13% average relative gain over 11 classification tasks, author-reported [arxiv:2104.08786 §5]; each order is scored on every probe input | | Instruction for a task with labelled training data, API access and a large scoring budget | Iterative LLM-proposed instructions, scored on a fixed training subset and chosen by training accuracy | GSM8K test 80.2 against 71.8 for the step-by-step cue with PaLM 2-L [arxiv:2309.03409 tab. 4]; that table picks by test accuracy, so use the training-accuracy rule [arxiv:2309.03409 §5.4] | | Two chat agents must finish a multi-step task without a human steering | Fixed inception prompts: roles, one instruction per turn, a stop token | Human votes favour CAMEL on 76.3% of 100 AI Society tasks [arxiv:2303.17760 tab. 1]; repository prompts differ from the paper's | | Entity matching, imputation or error detection over table rows, GPT-3-class model, no labelled data for the new task | Serialized rows with a fixed task question, then hand-picked examples; test error detection with examples | Entity matching 98.2 with ten examples against 65.9 with none on iTunes-Amazon [arxiv:2205.09911 tab. 1]; zero-shot error detection scored 0.0 on Adult [arxiv:2205.09911 tab. 2]; manual example choice took up to an hour per task [arxiv:2205.09911 §3.3] | | Structured records with a reason needed, roughly 10 labelled samples, strong chat model | Zero-shot structured prompt with a JSON answer, or embed the prompt text for a small classic model | Mode accuracy 0.586 for GPT-3.5 zero-shot against 0.462 for a neural network on 10 samples; a network on 1,000 samples reached 0.635 [arxiv:2312.00819 tab. 3], so a large labelled set favours a trained model | | Value survey asked of many countries, one model, no weight access | Ask each item in English with a group phrase and in the target language; check agreement before reading results | Same-score agreement above 70% for four of five cultures in English, 58.33% for China [arxiv:2303.17466 tab. 3]; only one correlation cell reaches p < 0.05 [arxiv:2303.17466 tab. 5] | | Multiple-choice problem solving where reruns must match | Temperature 0.0 with a fixed prompt style, not a higher temperature | No significant effect from 0.0 to 1.0 for GPT-3.5 on 1,000 questions (p = 0.403) [arxiv:2402.05201 §3.1]; accuracy fell above 1.0; only multiple-choice items were tested [arxiv:2402.05201 §4.2] | | Labels for many texts from a codebook, with some gold labels to check against | Zero-shot codebook prompt, one text per fresh chat, low temperature | About 25 points above MTurk on average; relevance accuracy 59-83% [arxiv:2303.15056 §2]; agreement is raw percent match | | Remove identifiers from clinical notes, data may not leave the site | Explicit rules and a replacement token, measured on your own set | ChatGPT 0.686 to 0.929 with the explicit prompt [arxiv:2303.11032 tab. 2]; the authors say API-only models cannot take patient text [arxiv:2303.11032 §VI] | | Aggregate answers of a population from demographic backstories | First-person backstory, then next-token probabilities for the answer | Whole-sample vote correlation 0.90-0.94, but 0.02 for 2020 independents [arxiv:2209.06899 tab. 1] | | Generalise to held-out task types with an encoder-decoder you can fine-tune | Train on many prompts per dataset, report medians over prompts | T0 matches or exceeds GPT-3 on 9 of 11 held-out datasets [arxiv:2110.08207 §6.1]; about 270 TPU v3-512 hours [arxiv:2110.08207 app. B.1] | | Score generated text against a reference, no human labels | Prompt-averaged BARTScore, with the paper's prompt lists | Prompts help most settings but not factuality [arxiv:2106.11520 §4.3.2]; check the averaging rule before comparing numbers [arxiv:2106.11520 eq. 3] | | Audit a completion model for group bias, or test a cheap prefix fix | Probe the same prompt over groups, report the rates, and check the topics | GPT-3 gave 66 of 100 violent completions for one group prompt; the best prefix reached 20%, against 13-15% for another group [arxiv:2101.05783 §Results]; the prefix changes topics too | | Many labelled texts needed, no human labels yet | Demonstration prompts with random samples per item, then human-check a sample | 274,186 statements; human agreement with the prompt label 67.7% for top-k toxic prompts and 40.3% for ALICE [arxiv:2203.09509 §4.3] | | Live assistant that a human supervises, trial possible | One suggestion per strategy, latest messages only, human picks | Exit-ticket pass 61.7% to 65.7% in a randomized trial of 783 tutors [arxiv:2410.03017 tab. 1]; end-of-year scores not estimated | | Imitate one annotator's labels from a few examples | Up to three labelled texts from that annotator before the target | AggressionPer 72.57 against 69.10 for plain Aggression, with different references [arxiv:2302.10724 tab. 2]; GoEmo variants did not improve in order | | Rewrite patient-facing text in plain language, errors would matter | Simple-language instruction, then a clinician reviews every output | Medians 2, 2 and 4 on correctness, completeness and harm [arxiv:2212.14882 tab. 1]; invented and dropped findings were reported [arxiv:2212.14882 §4.2]; the authors call for physician oversight [arxiv:2212.14882 §5.2] | | Non-English text, zero-shot, no labelled data, you must choose one description language | Measure both English and target-language descriptions on your own data before choosing | Averages favoured English on four of six tasks, and target language on two [arxiv:2304.05613 §4]-[arxiv:2304.05613 §9]; the per-language gaps are large, so check each language | | Simulate a country's survey answers from a model you can only prompt | Cultural-identity persona with the target country, compared with a default persona | Mean distance for GPT-4o 2.42 to 1.57; 19 to 29% of countries did not improve [arxiv:2311.14096 §2] [arxiv:2311.14096 §3]; English prompts only | | Simulated subjects in a randomised experiment where the treatment is in the prompt | State the randomisation design and that the subject does not know it | Out-of-box error on purchase probability 0.532 blinded against 0.397 unblinded for gpt-4o-mini [arxiv:2312.15524 tab. 1]; one 40-product price study, and the bounds are not given [arxiv:2312.15524 §5] | | Zero-shot prompt, no labels, a few hand-written candidates and example inputs | Expand the seeds by paraphrase and backtranslation, rank by mean perplexity, keep the lowest few | Gain over manual prompts 1.8 (OPT) and 2.3 (Bloom) points; AG News fell 12.5 points with Bloom [arxiv:2212.04037 tab. 10]; no significance test | | Deductive coding with a written codebook, expert labels to check against, one closed model | Codebook-centred prompt, one example per code, temperature 0 | Kappa 0.61 (complexity) and 0.38 (syntax) against experts, with experts at 0.88 and 0.90 [arxiv:2304.10548 §4]; one dataset of 668 questions | | Audit a model's group portraits for stereotypes, no word list or labelled data | Persona prompts, then words marked against the White and man defaults | GPT-4 and GPT-3.5 personas carry more racial stereotype words than human ones [arxiv:2305.18189 §5]; the comparison is only in a figure | | Check how fragile an instruction is to typos, synonyms or distractors | Attack the instruction only, and report the drop from the clean prompt | Word-level swaps average a drop of 0.31 to 0.33 [arxiv:2306.04528 tab. 2]; GPT-4 and Flan-UL2 most robust at 0.08 [arxiv:2306.04528 tab. 3] | | One instruction must be split across a team of robots with different skills | Staged prompts with Python-style examples: decompose, form teams, allocate, then code | GPT-4 success 1.00 on elemental and 0.62 to 0.71 on the other categories of 36 tasks [arxiv:2309.10062 tab. 1]; no metric for real robots [arxiv:2309.10062 §6.2] | ## Implementation sketch Language-agnostic pseudocode for the common core. The parts that vary are marked. ```pseudocode # Build the prompt: instruction, optional demonstrations, then the input # [arxiv:2302.06476 §3] [arxiv:2202.12837 app. B] def build_prompt(instruction, demos, x): parts = [instruction] for x_i, y_i in demos: # pair layout: input line, label line; pairs separated by blank lines # [arxiv:2202.12837 app. B] parts.append(x_i + "\n" + y_i) parts.append(x) return "\n\n\n".join(parts) # Optional: reasoning cue, two stages [arxiv:2302.06476 §3] def answer_with_cot(instruction, x, trigger): # stage 1: free-text rationale rationale = model(instruction + x) # stage 2: order of the parts is not stated; this order is an assumption return model(instruction + x + rationale + trigger) # Read the answer strictly and count replies outside the allowed set # [arxiv:2302.06476 §4.2.8] def parse_answer(output, allowed_labels): # return None for off-format replies, and log them if output.strip() in allowed_labels: return output.strip() return None ``` Where variants differ: - **Demonstrations with random labels** keep the pair layout but draw each label uniformly from the label set [arxiv:2202.12837 §4.1]. Use the gold labels if you are measuring the prompt itself. - **Search** replaces the single answer with a loop over proposed steps, with the model scoring each partial solution [arxiv:2305.10601 alg. 1]. - **Distractors** add an instruction to ignore irrelevant information before the exemplars, and can sample several answers and vote [arxiv:2302.00093 §4.3] [arxiv:2302.00093 §5.1]. - **Format sweep** scores the same items under several equivalent formats and reports the best and worst, not one number [arxiv:2310.11324 §3.2]. - **Layout output** asks for one rule per object with normalised integer values, and the parser must map them back to boxes [arxiv:2305.15393 §3.2]. - **Per-question sensitivity** replaces the single score with the mean gap over every pair of variants for each question, then averages across questions [arxiv:2410.12405 eq. 1] [arxiv:2410.12405 eq. 2]. - **Extraction output** can be IOB tags or YAML with character spans. Parse each one strictly, drop what fails to parse, and fuzzy-match spans back to the source [arxiv:2312.02296 app. A.1]. - Decoding settings (temperature, top-p, output length) are not stated for the ChatGPT runs or the translation runs, so record your own [arxiv:2302.06476 §3] [arxiv:2102.07350 tab. 1]. - **Probe-based order selection** samples one continuation per candidate order at temperature 2, parses the inputs, and ranks orders by the entropy of their predicted labels [arxiv:2104.08786 §3.1] [arxiv:2104.08786 eq. 3]. - **Iterative instruction search** keeps the best instructions so far, sorted with the best last, and asks the optimizer for new ones. Each candidate is scored on one fixed training subset, and the final choice is made by training accuracy [arxiv:2309.03409 §4.2] [arxiv:2309.03409 §5.4]. - **Two-agent role-play** alternates an instruction from the user and a solution from the assistant, each in a fixed format, and stops when the user sends the stop token [arxiv:2303.17760 §3.1] [arxiv:2303.17760 §3.2]. - **Prompt-averaged scoring** appends or prepends each phrase to the input and averages the normalised scores over the phrases [arxiv:2106.11520 eq. 3]. The paper does not say which side's average it reports [arxiv:2106.11520 §3.3.1]. - **Serialized row** writes each record as "attribute: value" pairs, keeps only the attributes that help, and writes a missing value as an empty string [arxiv:2205.09911 §3.1]. The task question follows the rows and is different for each task [arxiv:2205.09911 §3.2]. - **Answer form check** counts any reply outside the required form as wrong rather than dropping it, and reports how often that happens [arxiv:2402.05201 §3.1] [arxiv:2402.05201 app. A]. - **Numeric comparisons in the prompt** state which option is cheapest or fastest, and by what share, instead of leaving the comparison to the model [arxiv:2312.00819 §3.3.4]. - **Language or culture phrase** is a variable to sweep: ask the same items in English with a group phrase, in the target language, and in English with a "culture setting" phrase, then check agreement between the forms [arxiv:2303.17466 §3.1] [arxiv:2303.17466 tab. 3]. - **Perplexity ranking** scores each candidate prompt by its mean perplexity over about 1,000 task inputs, and keeps the lowest few. The label is left out of the score [arxiv:2212.04037 §4.2] [arxiv:2212.04037 §7]. - **Codebook layout** writes each code as a name, a description and examples, with zero, one or five examples per code [arxiv:2304.10548 §3]. - **Marked-words audit** keeps the words whose weighted log-odds difference is above 1.96 against every default group [arxiv:2305.18189 §3.2]. - **Wording stress test** changes only the instruction and compares each score with the clean prompt on the same sampled set [arxiv:2306.04528 §2.2] [arxiv:2306.04528 §3]. - **Staged planner** gives each planning stage its own few-shot examples [arxiv:2309.10062 §4] [arxiv:2309.10062 §5.2]. ## Contradictions - **Which prompt part matters in Time-LLM.** The text says that removing the input statistics costs 10.2% of average MSE, and it calls that row C.1. The ablation table labels C.1 as the dataset description and C.3 as the statistics [arxiv:2310.01728 §4.5]. The card uses the table labels, so the attribution of that 10.2% is not settled. Check the compiled PDF before quoting either. - **Zero-shot against few-shot.** On Curie, zero-shot and 10-shot Simple colon tie (33.3 vs 33.4 BLEU), but on Babbage the 10-shot baseline is higher (25.1 vs 23.5) [arxiv:2102.07350 tab. 1]. The two results are for different models and different baselines, so they do not settle which setting wins. - **Do examples teach?** The prompt-programming paper argues that examples mostly point at a task the model already knows [arxiv:2102.07350 §1]. The demonstrations paper agrees that gold labels add little, but finds that the label space and input text carry most of the gain [arxiv:2202.12837 §5]. Neither paper tests the other's setting. The first rests on one task and two models [arxiv:2102.07350 §3.1]. - **Label sensitivity by scoring method.** Channel models barely change when label words are replaced, while direct models lose 5-16 points [arxiv:2202.12837 §5.2]. A result from one scoring method does not carry over to the other. - **Soft prompts, layer-0 or all layers.** The paper's text says the basic method tunes only the input layer and the all-layer variant is an add-on [arxiv:2104.06599 §4.7]. Its Table 1 numbers match the all-layer column of Table 5, not the layer-0 column [arxiv:2104.06599 tab. 1] [arxiv:2104.06599 tab. 5]. The gap is 0.6 P@1 on T-REx with BERT-large, so the choice matters less than the difference from hand-written prompts. - **Reasoning cues on commonsense tasks.** A cue helps arithmetic and lowers CommonsenseQA and StrategyQA for ChatGPT [arxiv:2302.06476 tab. 3]. The metaprompt paper's reasoning benefit is unmeasured [arxiv:2102.07350 §4.7]. - **Reasoning cue on other models.** The same kind of cue scored 29.0% on the GSM-IC variants for code-davinci-002 [arxiv:2302.00093 tab. 3], while it improved ChatGPT on all six arithmetic sets [arxiv:2302.06476 tab. 2]. The models, the datasets and the prompt setups all differ. - **Format sensitivity by benchmark.** FormatSpread reports a median spread of 7.5 points and up to 76 points for one model [arxiv:2310.11324 §1] [arxiv:2310.11324 §4.2]. MMLU-Pro reports about 2% across 24 prompt styles [arxiv:2406.01574 §6.3]. The format sets differ (a grammar of generated formats against 24 written styles), and so do the tasks and models, so the two numbers do not measure the same thing. - **Sensitivity against model size.** ProSA reports that larger models and few-shot examples lower its per-question score [arxiv:2410.12405 §3.3] [arxiv:2410.12405 §3.4]. FormatSpread reports that the format spread does not shrink with model size, shot count or instruction tuning [arxiv:2310.11324 §4.2]. The two measure different things (a paired gap over 12 hand-written variants against a spread over generated formats), and they use different models, so neither result answers the other's question. A travel-prediction ablation adds a case on the other side: GPT-4 lost under 1% when the reasoning guide was removed, against 5.8% for GPT-3.5 [arxiv:2312.00819 tab. 5], which fits a size effect. That ablation removes prompt parts or flattens the data structure; it does not sample equivalent formats, so it does not test the FormatSpread claim. - **Temperature for single answers against voting.** The temperature study recommends 0.0 for problem solving, and found no accuracy effect from 0.0 to 1.0 for single answers [arxiv:2402.05201 §4.1]. The self-consistency study samples 20 answers at temperature 0.7 and takes a majority vote [arxiv:2302.00093 §5.1]. A vote needs answers that differ, so the 0.0 advice is not tested for that method. The two studies test different procedures, and neither compares voting across temperatures. - **Steering against training.** The survey paper reports that prompting alone rarely reproduces the spread of human answers [arxiv:2502.16761 §4.2]. The Turing Experiments paper reports that zero-shot persona prompts reproduce three of four classic human findings [arxiv:2208.10264 §7]. The first measures answer distributions on survey questions, the second measures effects in classic experiments. Neither paper runs the other's task. - **Which factor drives few-shot results.** Order alone can spread the accuracy of one demonstration set from near 50% to over 85% [arxiv:2104.08786 fig. 1]. The demonstrations paper finds that gold labels add little and that the label space and input text carry most of the gain [arxiv:2202.12837 §5]. The two papers vary different factors, order against label content, so neither result rules out the other. - **Order with retrieved examples.** The order paper says a 2021 study (not carded here) found order matters little once retrieved examples are used, and it argues the opposite for fixed few-shot sets [arxiv:2104.08786 §6]. - **Prompt sensitivity by model.** ChatGPT's score rises from 0.686 to 0.929 with an explicit prompt, while GPT-4 scores 0.908 even on the implicit prompt [arxiv:2303.11032 tab. 2]. The GPT-4 rows come from 50 hand-run samples, so this gap compares unequal runs [arxiv:2303.11032 §V]. - **Does the wording matter?** The 2021 NLP survey reports that, in one cited few-shot entailment study, irrelevant templates did about as well as meaningful ones [arxiv:2111.01243 §6]. The ChatGPT study finds the opposite direction: changing the prompt wording moved PolEmo by up to +14.8 points and TextEntail by up to −13.0 points [arxiv:2302.10724 tab. 4]. The models, tasks and templates differ, so neither result settles the question. A third study finds the sign of the wording effect depends on the dataset: one reworded question helped one entity-matching set and hurt two others [arxiv:2205.09911 tab. 4]. - **Random starts for soft prompts.** On relation probing with BERT-large, a random start scored 51.9, slightly above the 51.6 of the best soft mixture [arxiv:2104.06599 tab. 1]. On 11B T5 with 32 SST-2 examples, a random start gave 70.5 ± 15.5, and the PPT paper calls this unstable and starts from pre-trained prompts instead [arxiv:2109.04332 tab. 4] [arxiv:2109.04332 §4.2]. The tasks, model sizes and label counts all differ, so neither result settles whether the start matters in general. The PPT pilot found that starting from word embeddings did little or harm at 11B [arxiv:2109.04332 tab. 2]. - **Prompt language for non-English tasks.** The ChatGPT study finds that an English task description usually had the higher average, on four of six tasks [arxiv:2304.05613 §4]-[arxiv:2304.05613 §9]. The survey-persona study finds that prompting in Arabic raised GPT-3.5's Egypt score by about 3 points [arxiv:2402.13231 tab. 2]. The tasks, the metrics and the models differ, so neither result settles the language choice for another task. The cultural-alignment study reports that most cultures align better when asked in their own language [arxiv:2303.17466 §4.2], while the English prompts were more consistent with each other [arxiv:2303.17466 tab. 3]. Those two findings are about values, not tasks, and point in different directions. - **Zero-shot personas against blinded experiments.** The Turing Experiments paper reports that zero-shot persona prompts reproduce three of four classic human findings [arxiv:2208.10264 §7]. The causal-inference paper argues that a blinded persona in a price experiment confounds the price effect, because the model varies unstated details with the price [arxiv:2312.15524 §2]. The first measures trends in classic studies and the second measures a demand curve, so neither paper settles the other's setting. The second paper's remedy is a change to the prompt, not a change to the persona. - **Size of identity-prompt gains.** The cultural-bias paper reports that cultural prompting cut GPT-3.5's mean map distance from 3.35 to 2.83 [arxiv:2311.14096 §2]. The survey-persona paper reports that its Anthropological Prompting variant lifted GPT-3.5's Egypt soft score from 0.4834 to 0.5102 [arxiv:2402.13231 tab. 4]. The metrics differ (distance on a cultural map against an item-level soft score) and so do the prompts, so the two gains cannot be compared. ## Papers - [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](/papers/2102.07350): argues that prompts work by locating a known task, and proposes prompt techniques (constraints, personas, serialized reasoning, metaprompts) with one measured translation comparison. - [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](/papers/2202.12837): shows that random labels in demonstrations cost little, and isolates the label space, input text and pair format as the drivers. - [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](/papers/2302.06476): zero-shot instructions and two-stage zero-shot CoT across 20 datasets, against GPT-3.5. - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](/papers/2305.10601): prompted search over intermediate steps, with large gains on Game of 24 and Mini Crosswords for GPT-4. - [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](/papers/2104.06599): learns continuous cloze prompts and mixes them, with BERT-large gains on T-REx over LPAQA; the learned vectors are not readable text. - [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](/papers/2308.06834): four clinician-style few-shot prompts against a step-by-step baseline on MedQA-derived diagnosis questions; GPT-4 shows no significant difference. - [AI-enabled intelligent assistant for personalized and adaptive learning in higher education](/papers/2309.10892): a retrieval-backed tutoring bot that controls hallucination only through its prompt; no evaluation. - [Early ChatGPT research survey](/papers/2304.01852): a survey of early ChatGPT research; names prompting only as an application area. - [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](/papers/2208.10264): zero-shot persona prompts run over many named participants, compared with four published human studies; the wisdom-of-crowds task shows a distortion. - [Large Language Models Can Be Easily Distracted by Irrelevant Context](/papers/2302.00093): one irrelevant sentence in GSM-IC variants lowers accuracy; an ignore-instruction, a distractor exemplar and self-consistency help. - [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](/papers/2305.15393): CSS-style layout rules with normalised values and retrieved examples, with no training; a 3D bedroom result against a supervised baseline. - [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](/papers/2308.00352): role profiles, fixed documents and executable feedback for a software team; the ablation varies the roles. - [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](/papers/2310.01728): a frozen 7B model with learned input patches and a dataset text prefix; author-reported forecasting gains, with ablation rows for each prompt part. - [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design](/papers/2310.11324): grammar-generated prompt formats, a spread metric, and a bandit search for the best and worst format within a budget. - [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](/papers/2406.01574): a harder multiple-choice benchmark with 5-shot CoT and a regex answer parser; scores move less under prompt-style changes. - [SubPOP](/papers/2502.16761): zero-shot and few-shot steering prompts against LoRA fine-tuning on human answer distributions. - [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](/papers/2109.14739): one T5 model with a task-name prefix per dialogue sub-task, pre-trained on eleven partly labelled corpora. - [LLMs Accelerate Annotation for Medical Information Extraction](/papers/2312.02296): few-shot PaLM 2 pre-labels for medication fields, with expert refinement; cuts rater time by 58% on i2b2 documents. - [Investigating Cultural Alignment of Large Language Models](/papers/2402.13231): demographic personas answer World Values Survey questions in Egypt and the US; soft and hard scores, and an Anthropological Prompting variant. - [ProSA](/papers/2410.12405): defines a per-question prompt sensitivity score and reports it for eight LLMs on six benchmarks. - [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](/papers/2504.18412): a steel-man system prompt in stigma vignettes and crisis replies, with author-reported failure rates. - [GPT Understands, Too](/papers/2103.10385): P-Tuning adds trainable prompt vectors beside a hand-written template; frozen BERT on LAMA, tuned models on SuperGLUE. - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](/papers/2109.04332): pre-trains soft prompts on unlabelled text with three cloze tasks, then tunes them per task; 11B models with 32 examples per task, author-reported. - [Fantastically Ordered Prompts and Where to Find Them](/papers/2104.08786): ranks few-shot demonstration orders by the label entropy of model-written probe inputs, without labels. - [BARTScore](/papers/2106.11520): scores text by BART log-likelihood; its prompt variant averages scores over short phrases. - [Pre-train, Prompt, and Predict](/papers/2107.13586): a survey that defines prompt addition, answer search and answer mapping; no experiments of its own. - [Multitask Prompted Training Enables Zero-Shot Task Generalization](/papers/2110.08207): T0, trained on prompted datasets with several templates each, tested zero-shot on held-out task types. - [Out of One, Many: Using Language Models to Simulate Human Samples](/papers/2209.06899): first-person demographic backstories make GPT-3 stand-ins for survey respondents, with checks on US politics. - [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](/papers/2303.11032): a three-part prompt redacts clinical notes; explicit rules raise ChatGPT's score on i2b2/UTHealth. - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](/papers/2303.15056): zero-shot codebook prompts for tweet and news labels, against MTurk and trained coders. - [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](/papers/2303.17760): two chat agents with fixed inception prompts for roles, turn order and a stop token. - [Large Language Models as Optimizers](/papers/2309.03409): OPRO, where an LLM proposes instructions from scored past attempts; results on GSM8K and BBH. - [Revolutionizing Finance with LLMs: An Overview of Applications and Insights](/papers/2401.11641): a finance survey with one GPT-4 evaluation on six datasets, a three-part prompt and no baseline. - [Persistent Anti-Muslim Bias in Large Language Models](/papers/2101.05783): GPT-3 completion, analogy and caption probes; a positive prefix lowers violent completions but not to the level of other groups. - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](/papers/2111.01243): a 2021 survey that splits the field into fine-tuning, prompt-based learning and text generation; runs no experiments. - [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](/papers/2203.09509): GPT-3 demonstration prompts write 274,186 labelled statements; a classifier-guided decoder adds harder examples. - [ChatGPT: Jack of all trades, master of none](/papers/2302.10724): ChatGPT on 25 analytical NLP tasks with a hand-made prompt per task; personalised per-annotator examples and wording tests. - [Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise](/papers/2410.03017): one GPT-4 reply per teaching strategy, shown to tutors in live chat; a randomized trial of 783 tutors. - [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](/papers/2212.14882): one plain-language prompt for three fictitious radiology reports, rated by 15 radiologists; errors and invented findings reported. - [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](/papers/2304.05613): zero-shot ChatGPT on seven tasks in 34 to 37 languages, with English and target-language descriptions; POS tagging is the exception to the gap. - [Unleashing the potential of prompt engineering for large language models](/papers/2310.14735): a review of prompting, reasoning cues, prompt optimization and prompt security; no experiments of its own. - [Cultural Bias and Cultural Alignment of Large Language Models](/papers/2311.14096): five GPT models on a cultural map; a country-of-birth persona moves answers toward that country's survey values for most countries. - [Gui and Toubia, causal-inference study of LLM simulation](/papers/2312.15524): blinded LLM subjects confound a price effect; telling the model the randomisation design lowers error against a human survey. - [Can Foundation Models Wrangle Your Data?](/papers/2205.09911): GPT-3 175B on five data-cleaning tasks, with serialized rows, task questions and few-shot examples; scores depend on example choice and wording. - [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](/papers/2303.17466): Hofstede survey items asked in English, in the target language and with a culture phrase; alignment is mostly weak and best for the US. - [Large Language Models for Travel Behavior Prediction](/papers/2312.00819): a structured zero-shot prompt with a JSON answer, and LLM embeddings as features for small-sample travel-choice models. - [The Effect of Sampling Temperature on Problem Solving in Large Language Models](/papers/2402.05201): five one-shot prompt styles at temperatures 0.0 to 1.0 on multiple-choice questions; no significant effect up to 1.0, with accuracy falling above it. - [Demystifying Prompts in Language Models via Perplexity Estimation](/papers/2212.04037): zero-shot prompts ranked by perplexity; SPELL keeps the three lowest of GPT-3 paraphrases and backtranslations. - [Xiao et al., codebook prompts with GPT-3](/papers/2304.10548): codebook-centred GPT-3 prompts for 668 French children's questions, kappa 0.61 and 0.38 against experts. - [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](/papers/2305.18189): persona prompts per group, with marked words found against the White and man defaults; no word list. - [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](/papers/2306.04528): seven text attacks on the instruction, scored by performance drop across 13 datasets and six LLMs. - [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](/papers/2309.10062): four-stage Python-style prompts that turn one instruction into multi-robot code, tested in AI2-THOR.