In-context learning

Overview

In-context learning (ICL) means conditioning a language model on examples or instructions placed in its prompt, then asking it to answer a new input. No weights change. The usual form is k input-label pairs followed by a test input, which the model completes 2202.12837 §1. The prompt is the only interface for most variants below, so they differ in what goes into the prompt. The exceptions train something: a small add-on on the same examples, a generator that writes training data, or a separate network trained once on many tasks. Several agent and optimizer variants put past attempts or scored candidates in the prompt, one adds a second model's suggestion, and one editing variant puts a new fact in the prompt. Two variants ask the model for a program or a formal problem file, which an interpreter or a solver then runs.

The open question is what the model takes from the prompt. Early work read the gains from more examples as the model learning the task on the fly. The papers on this wiki argue for other readings: that the examples mostly point at a task the model already knows 2102.07350 §1, or that they supply format, label space and input distribution rather than the input-to-label rule 2202.12837 §5. Both are reported from experiments on specific tasks and models, so treat them as hypotheses for your own setup. A position paper asks a further question, whether a frozen model's words can be about things at all. It argues that frozen weights with in-context learning do not create that grounding by themselves 2304.01481 §7.1, and it leaves open whether in-context selection happens in these models 2304.01481 §6.2.3.

Variants

Standard k-shot with gold labels

The baseline: k labelled demonstrations drawn from the training data, usually with 16 examples, averaged over several random draws of the demonstrations 2202.12837 §3 2202.12837 app. A. For classification and multi-choice tasks, the model scores each candidate label, either directly as P(label | demos, input) or through the channel model P(input | demos, label) 2202.12837 §3. The paper's reproduction of GPT-3 with gold labels is the reference point for every ablation 2202.12837 §4.1.

The number of examples has a ceiling. On the Med-HALT tests, GPT-3.5 scored 7.31% with zero examples, and accuracy rose with more examples and levelled off after about three 2307.15343 §7.2 2307.15343 fig. 6. This is one model on one medical benchmark, and the shot-count experiment has no matching code path in the released repository 2307.15343 fig. 6. Card: Med-HALT.

Strong models may gain little from demonstrations on some tasks. On ETHICS commonsense morality, DecodingTrust reports 89.3% for GPT-4 with few-shot prompts and 89.0% zero-shot, and 87.9% for GPT-3.5 against 85.1% zero-shot 2306.11698 tab. 27. Those gains are 0.3 and 2.8 points, read from the table. The few-shot prompts use 32 short-sample examples and 8 long-sample examples, the counts the paper sets because of the token limit 2306.11698 app. machine-ethics details. Card: DecodingTrust.

Two pretraining reports run the same zero-, one- and few-shot protocol on their own models. A 64B mixture-of-experts model averages 62.7 over 29 benchmarks with zero examples, 65.5 with one and 68.1 with a few, against 56.9, 61.6 and 65.2 for GPT-3 175B 2112.06905 tab. 1. The paper does not say how the headline averages are weighted, and the appendix tables do not reproduce them under equal task weights 2112.06905 §5.3. OPT-175B, evaluated with GPT-3-style prompts on 16 NLP tasks, roughly matched GPT-3 on 10 of them, and its per-task curves appear only as figures 2205.01068 §3.1. Card: GLaM, OPT.

Order of the demonstrations, chosen without labels

Order alone can move a few-shot score a lot, and an order that works for one model may fail for another 2104.08786 §2. One candidate order scored 88.7% on GPT2-XL (1.5B) and 51.6% on GPT2-Large (0.8B) 2104.08786 §2. Lu et al. choose the order without a labelled development set. They sample each candidate prompt, let the model continue it, and keep the continuations as unlabelled probe inputs. Each candidate is then scored by the entropy of the labels it predicts on those probes, and the four highest-scoring orders are kept 2104.08786 §3 2104.08786 §4. The reasoning is that a poor order tends to predict one label for almost everything 2104.08786 §2. For GPT-2 1.5B on SST-2, the baseline scores 66.8 and the GlobalE selection scores 81.8 2104.08786 tab. 2. Over 11 classification tasks, the authors report a 13% average relative gain 2104.08786 §5. The score ignores which label wins: with two labels, an 80/20 split in one direction and the same split in the other have equal entropy 2104.08786 eq. 3. Check the predicted label mix as well. The released repository implements the GlobalE score only entropy.py. Card: Fantastically Ordered Prompts.

Worked reasoning in the demonstrations

Chain-of-thought prompting puts the reasoning steps in each demonstration, so the model writes a reasoning trace before its answer 2201.11903 §1. It targets multi-step problems, where plain input-answer pairs help little as models grow 2201.11903 §1. On GSM8K with PaLM 540B, standard few-shot prompting scores 17.9 and the reasoning exemplars score 56.9 2201.11903 app. B. The same change takes GPT-3 175B from 15.6 to 46.9 and Codex from 19.7 to 63.1 2201.11903 app. B. The gains depend on scale: the paper finds chains often hurt models below about 10B parameters 2201.11903 app. A, and the gains appear at about 100B 2201.11903 §3.2. The exemplars' wording matters as well. On coin-flip tasks, LaMDA 137B ranged from 99.6% to 71.4% depending on which annotator wrote the exemplars 2201.11903 app. B. Card: Chain-of-Thought Prompting.

Demonstrations picked by reasoning length

Complexity-based prompting ranks training cases by how many steps their written reasoning has, and uses the most complex ones as the prompt 2210.00720 §3.1 2210.00720 §4.2. A step is one line of the chain 2210.00720 §3.1. Where chains are not annotated, question length is the proxy, and only the chosen cases are annotated by hand 2210.00720 §4.1. With greedy decoding on code-davinci-002, GSM8K rises from 61.0 with the handcrafted prompt to 66.6, against 60.4 for random training cases 2210.00720 tab. 1. On MultiArith the same model gives 95.8 for the complex prompt and 97.3 for random cases, so step count does not win everywhere 2210.00720 tab. 1. The second part samples 50 chains per question and votes only over the most complex ones, typically 30 to 40 2210.00720 §3.2. With code-davinci-002 on GSM8K, the complex prompt with that vote reaches 82.9, against 74.6 for the handcrafted prompt with a vote over all chains 2210.00720 tab. 1. On text-curie-001 and Flan-T5, the change against the handcrafted prompt is between −0.5 and +1.5 points 2210.00720 tab. 5. Card: Complexity-Based Prompting for Multi-Step Reasoning.

Demonstrations written by the model, one per question cluster

Auto-CoT removes the hand-written demonstrations. It clusters the test questions by Sentence-BERT embedding with k-means, then takes the question nearest each cluster centre and asks the model for a zero-shot chain with "Let's think step by step" 2210.03493 §4.1 2210.03493 §4.2. A chain is kept only if its question has at most 60 tokens, its rationale has at most five steps, and for arithmetic tasks the answer appears in the rationale 2210.03493 §4.2 2210.03493 app. C.2. No gold answers are used. Taking one question per cluster, rather than the most similar questions, is the paper's defence against copying one cluster's mistakes. On MultiArith, demonstrations retrieved by similarity score 82.8 against 86.2 for random ones, when the demonstrations are zero-shot chains 2210.03493 tab. 1. With the default k = 8, Auto-CoT scores 92.0 on MultiArith against 91.7 for hand-written Manual-CoT. It is at or above Manual-CoT on all ten GPT-3 datasets, with StrategyQA tied 2210.03493 tab. 3. These are author-reported means over three runs, with no spread shown 2210.03493 §5.2. On Codex the order flips on MultiArith: Manual-CoT 96.8 against Auto-CoT 93.2 2210.03493 tab. 4. Card: Automatic Chain of Thought Prompting in Large Language Models.

Channel scoring: explain the input given the label

Min et al. score each label by how well it explains the input, P(input | label), rather than P(label | input) 2108.04106 §3. With the same demonstrations, a frozen GPT-2 Large scores 58.9 on average over 11 classification sets when the demonstrations are concatenated, against 38.5 for direct scoring of the same prompt. The worst run is 50.8 against 29.1 2108.04106 tab. 3. Ensembling one demonstration per context gives 57.3 average for channel scoring and 55.8 for the calibrated direct score, with worst runs of 49.8 and 43.6 2108.04106 tab. 3. Learning a 20-vector prompt with no demonstrations gives 61.7 for channel scoring, against 57.7 for tuning a direct output head 2108.04106 tab. 4. Card: Noisy Channel Language Model Prompting.

Zero-shot natural-language task specification

Here the prompt names or implies the task and includes no examples. On WMT'14 French to English, a prompt that is only "French:" before the source sentence and "English:" after it scored 33.3 BLEU on Curie, against 27.9 for a reproduced 10-shot GPT-3-style prompt 2102.07350 tab. 1. On Babbage the same zero-shot prompt scored 23.5 against 25.1, so the advantage is not uniform across model sizes 2102.07350 tab. 1. Adding one example to this format dropped Curie to 27.6, which the authors attribute to the model reading the example as part of the text it is continuing 2102.07350 §3.2 2102.07350 tab. 1. The authors recommend making the prompt inconsistent with unwanted continuations, for instance a syntactic frame where the translation is the only sensible next text 2102.07350 §4.5. Card: Prompt Programming for Large Language Models.

The same zero-shot pattern is used to have a model rate text against a human rating form. Given the human instruction, the text and one 5-point question, text-davinci-003 and ChatGPT ranked human-written stories above GPT-2 stories on all four attributes, and the paper reports the davinci gaps as significant 2305.01937 tab. 1 2305.01937 §3.3. T0 and text-curie-001 showed no clear preference 2305.01937 tab. 1. Two instruction rewordings, a persona and an "explain" request, moved the absolute scores a little and kept the ordering 2305.01937 tab. 3.

Zero-shot instructions also cover generation. GPT-3 (text-davinci-002) was asked to summarize an article in a stated number of sentences, with no examples. In a crowd study on 100 CNN and 100 BBC articles, its summaries were chosen as best by a majority for 58% of CNN and 57% of BBC articles, against 36% and 20% for the fine-tuned BRIO 2209.12356 tab. 3 2209.12356 §3.2. Reference-based metrics scored the same summaries lower, so the human ranking and the automatic ranking disagree 2209.12356 §4.1. Card: News Summarization and Evaluation in the Era of GPT-3.

Zero-shot instructions also remove identifiers from clinical notes. DeID-GPT asks ChatGPT or GPT-4 to replace identifying text with "[redacted]", under a prompt that describes what to remove. On the 2014 i2b2/UTHealth test data, the explicit prompt reaches 0.929 entity accuracy for ChatGPT and 0.99 for GPT-4, against 0.974 for a fine-tuned ClinicalBERT 2303.11032 tab. 2. The GPT-4 figure comes from 50 test cases run by hand in the web interface, and the paper reports no model version or sampling settings 2303.11032 §V. The full prompt appears only in a figure 2303.11032 fig. 8. The authors name sending patient text to an external API as the main limit, since API-only models cannot be used inside a hospital 2303.11032 §VI. Card: DeID-GPT.

Codebook-style prompts for deductive coding

Xiao et al. have GPT-3 label 668 French children's questions with codes from an expert codebook, without fine-tuning 2304.10548 §2.1 2304.10548 §3. Each code is laid out as name, description and examples. The authors compare this codebook-centred layout with an example-centred one, where each example comes with a sentence explaining its label. The codebook-centred prompts scored higher, and the largest gain came from moving from zero to one example per code. One and five examples per code "did not differ much" 2304.10548 §4. Agreement with the experts' final codes is Cohen's kappa 0.61 for question complexity and 0.38 for syntactic structure. The two experts who coded the data reached 0.88 and 0.90 with each other 2304.10548 §4. The per-variant values appear only in a figure 2304.10548 fig. 1. The authors call the study preliminary and cover one dataset and one codebook 2304.10548 §5. Card: Supporting Qualitative Analysis with Large Language Models.

Zero-shot prompts for dialogue judgements, with few-shot checks

Kwon et al. evaluate six LLMs zero-shot on 35 tasks built from four negotiation dialogue datasets, with one prompt template per task 2402.13550 §2.3. Chain-of-thought gives near-perfect accuracy on the arithmetic tasks they tried, and two randomly sampled few-shot examples help on two of four annotation tasks 2402.13550 §5. Two prior utterances help one annotation task and slightly hurt three 2402.13550 §5. On subjective partner-modelling tasks the best correlation with human ratings is 0.39, from GPT-4 2402.13550 §4.3. Card: Are LLMs Effective Negotiators?.

Few-shot sentence and document translation

GPT-MT prompts text-davinci-002, text-davinci-003 and ChatGPT to translate, with 0, 1 or 5 example pairs 2302.09210 §3.1. Three ways of choosing the shots are compared: random pairs from the cleaned training data (RR), random pairs from the top one million LaBSE-ranked pairs (QR), and pairs retrieved by similarity to the input from that ranked pool (QS) 2302.09210 §3.1. On EN-DE, text-davinci-003 scores 85.6 COMET-22 zero-shot and 86.5 with five random shots, against 87.2 for the WMT-best system 2302.09210 tab. 3. Into Icelandic, the best GPT setting reaches 77.9 against 86.8 for the WMT-best system 2302.09210 tab. 4. Few-shot gains are largest when translating out of English, where one good shot gives most of the gain, and small into English 2302.09210 §3.3. More than five shots gave no gain in preliminary runs 2302.09210 §3.1.

Whole documents can be sent in one request, in windows of sentences, with the output realigned to the source 2302.09210 §3.5. On DE-EN, a 16-sentence window raised COMET-22 from 84.8 to 85.2 and cut GPT requests from 1984 to 310 2302.09210 tab. 5. Few-shot document prompts gave no clear gain over zero-shot ones 2302.09210 §3.5 2302.09210 tab. 6. The paper also tests a hybrid in which Microsoft Translator is the primary system and GPT is used when a reference-free quality estimate falls below the median. It reports the hybrid matching the max-routing upper bound on all pairs shown, while calling GPT for about half the sentences 2302.09210 §3.7. Card: How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation.

Multilingual label-likelihood prompting

mGPT is a GPT-3-style model pretrained on 61 languages. It classifies by summing the loss of each label's completed sentence and picking the lowest, and it tags NER and POS one word at a time with four demonstrations 2204.07580 §4.2.1 2204.07580 §4.2.2. On XWINO, the 13B model scores 59.3 with no demonstrations and 61.8 with four, and the 1.3B model stays between 54.5 and 57.0 across zero to sixteen shots 2204.07580 tab. 7. Hate-speech scores stay near the 50% chance level 2204.07580 tab. 8. For 4-shot NER, the 1.3B model reaches 15.0 F1 on average, against about 78 to 79 for fine-tuned multilingual BERT and XLM-R 2204.07580 tab. 9. The authors describe the classification baselines as comparable to XGLM, but the NER and POS baselines are fine-tuned, so the gap there is not a like-for-like comparison 2204.07580 §4.2.2. Card: mGPT: Few-Shot Learners Go Multilingual.

Random-label demonstrations

Keep the inputs and the format, but replace each gold label with one drawn uniformly from the label set. Across 26 classification and multi-choice datasets, nearly all of the 12 model and scoring combinations lose only 0-5 points absolute 2202.12837 §4.1 2202.12837 fig. 3. The drop is smallest for the meta-trained MetaICL model, at 0.1-0.9 points 2202.12837 §4.1. Correct labels are not all-or-nothing: with 0% correct labels, GPT-J on classification still stays clearly above no demonstrations, though it loses nearly 10 points against gold labels 2202.12837 §4.2. Random labels are the wrong choice when the label words are unfamiliar to the model or the task is synthetic: the largest per-dataset gap is nearly 14 points on financial_phrasebank with GPT-J 2202.12837 app. C.2.

Input-only and label-only demonstrations

The paper also removes one half of each pair. Conditioning on inputs without labels, or on labels without inputs, performs close to or below no demonstrations for all models tested 2202.12837 §5.3 2202.12837 fig. 10. The gain therefore needs the input-label pair structure, even when the labels are wrong.

Adversarial and spurious demonstrations

Demonstrations can also carry a planted pattern. DecodingTrust tests three kinds with 16 demonstrations per prompt 2306.11698 §7.1 2306.11698 §7.3: counterfactual examples, examples that carry spurious heuristics from HANS 2306.11698 §7.2, and backdoored examples, where a trigger is inserted into the text and the label is set to the target 2306.11698 §7.3. Counterfactual demonstrations did not mislead either GPT-4 or GPT-3.5. Backdoored demonstrations did, and GPT-4 was affected more 2306.11698 §7. The card does not reproduce the spurious-correlation or backdoor tables, so the size of these effects is not given here 2306.11698 tab. 18. Card: DecodingTrust.

Random English words as labels

Replacing the label words with random English words costs direct models 5-16 points, but channel models only 0-2 points, and sometimes gains 2202.12837 §5.2 2202.12837 fig. 9. The scoring method determines how much the label space matters, so an ablation run with one scoring method does not transfer to the other.

Flipped labels and meaningless label words

Wei et al. separate two readings of in-context learning with two label edits. In the first, a share of the exemplar labels is swapped to the other class while the test labels stay the same 2303.03846 §3. A model that only recognises the task keeps its prior accuracy, and a model that reads the mapping from the exemplars moves against that prior. For text-davinci-002 on one binary task, accuracy falls from 90.3% with no flips to 22.5% with all labels flipped 2303.03846 §3. Small models stay near their prior accuracy as the flip share rises, while large ones drop well below chance 2303.03846 fig. 2. In the second edit, the natural label words are replaced by "foo" and "bar". Large models keep most of their accuracy, which the authors read as learning the mapping from the exemplars alone 2303.03846 §4. Instruction tuning strengthens both behaviours. Flan-PaLM does not fall below chance under full flipping, where PaLM reached 31%, and Flan-PaLM-8B gains 9.6 points on the meaningless-label setting over PaLM-8B 2303.03846 §5 2303.03846 fig. 7. The authors read this as better learning from the exemplars and firmer holding to priors at once 2303.03846 §5. Each result is on 100 evaluation examples per dataset, so treat the single-point gaps with care. Card: Larger language models do in-context learning differently.

Abstract symbols and random tokens

Symbolic sequences can be used as a test of pattern following without word meanings. A frozen text-davinci-003 solved 85 of 800 ARC grid puzzles when the digits were shown as digits, and 44 ± 6 (mean over 5 random alphabets) when the symbols were random alphabet tokens 2307.04721 tab. 1. On the PCFG sequence benchmark it scored 75% against 58 ± 1% with random alphabets 2307.04721 tab. 2. The same paper reports that reward-labelled trajectories placed in the prompt let the model propose better trajectories for simple control tasks 2307.04721 §6. Per-action model calls and context-length limits are the main obstacles it names 2307.04721 §7. Card: Large Language Models as General Pattern Machines.

Image grids as prompts

The same idea has been applied outside text. A masked image model is given one grid image: an example input and output, the query image, and an empty bottom-right cell that it fills in 2209.00647 §3.2. The model was pretrained on about 88,000 arXiv figure images, not on the task 2209.00647 §3.3. On Pascal-5i segmentation, the figure-pretrained model averages 27.17 mIoU over the four splits, against 14.91 for copying the example output, and the ImageNet-pretrained version of the same model scores far lower 2209.00647 tab. 1 2209.00647 §6. Results use one example pair, and the authors note that a single example can be ambiguous 2209.00647 §4. Card: Visual Prompting via Image Inpainting.

Dense outputs written as RGB images, one solved pair as the prompt

Painter writes the outputs of several dense vision tasks as RGB images and trains one masked-image model on two stitched pairs from the same task, with the output half of one pair hidden 2212.02499 §3.1 2212.02499 §3.2. At inference, a solved pair stacked above the query is the prompt, and the model fills in the query's hidden output, so a new task needs no new head 2212.02499 §3.3. Author-reported, depth on NYUv2 reaches RMSE 0.288 against 0.330 for the best listed specialist, while ADE-20K segmentation reaches 49.9 mIoU against 57.7 2212.02499 tab. 1. The prompt pair was chosen at random, by search, or learned: on ADE-20K, random prompts gave 49.3 mIoU against 49.9 for searched ones 2212.02499 tab. 3. The released code differs from the paper in several training settings, so see the card before reproducing. Card: Images Speak in Images: A Generalist Painter for In-Context Visual Learning.

Multimodal in-context learning, which adds image-and-text demonstrations at inference, is listed as an extended technique in a survey of multimodal LLMs. The survey gives no results for it 2306.13549 §7.1. Card: A Survey on Multimodal LLMs.

Interleaved image-text prompts at scale

Emu2 is a 37B model that treats images and text as one sequence, with each image turned into 64 embeddings 2312.13286 §2.1. Its few-shot prompts take example pairs chosen by RICES, which the paper follows from Flamingo and Emu 2312.13286 app. E.1. On VQAv2 it scores 33.5 with no examples, 67.0 with four and 68.8 with sixteen 2312.13286 tab. 1. At sixteen shots Flamingo-80B scores 66.8 2312.13286 tab. 1. The paper reports Emu2 ahead of Flamingo-80B and IDEFICS-80B on VQAv2, VizWiz and TextVQA at every shot count from four upward 2312.13286 §3.1. Its zero-shot row is not the same condition as the Flamingo rows, which use two text examples, so the zero-shot gap is not like for like 2312.13286 tab. 1. The authors note that in-context ability may fail on complex scenes, such as counting a crowd, and expect it to improve with more data and larger models 2312.13286 app. A. Card: Generative Multimodal Models are In-Context Learners.

In-context examples during instruction tuning

Otter also puts examples in the prompt during training, not only at test time. Each MIMIC-IT training item carries example image and instruction pairs before its query 2305.03726 §3.2 2305.03726 §4.1. In the LLaVA-derived subsets, the examples are the top-K most similar training items, by image or by text similarity 2305.03726 §4.1. On the MME Cognition benchmark, a model trained without examples scores 1310.7 with two shots and 1173.5 with four, so more shots do not help it steadily 2305.03726 tab. 10. A model trained with two-shot examples scores 1430.3 with two shots at evaluation, and a model trained with four-shot examples scores 1440.8 with four 2305.03726 tab. 10. These are author-reported results on one benchmark, and the training and evaluation shot counts are reported together 2305.03726 tab. 10. Card: Otter.

Whole books and long documents in the prompt

A long-context model can take a full reference in one prompt, with no retrieval step and no training. Gemini 1.5 Pro was given a grammar book, a wordlist and parallel sentences for Kalamang, a language with almost no web text. The full material is about 250k tokens, and the half-book setting about 100k 2403.05530 §5.2.2. Translating Kalamang into English, Pro scored 4.14 on a 0-6 human scale with half the book and 4.00 with the full book. Flash scored 3.14 with the full book. A human learner scored 5.52 on the same full-book set 2403.05530 §5.2.2. With no book in context, every model tested scored near random, Pro included 2403.05530 §5.2.2. So the book carries the result, not the model alone.

The same report tests recall of one inserted fact in a long prompt. Pro recalled it 100% of the time up to 530k tokens and 99.2% at 10M text tokens 2403.05530 §5.2.1. The authors call this task simple, so it shows that the fact is retrievable from the window, not that the model uses it well 2403.05530 §5.2.1. On a question-answering set over Les Misérables, full-book context was preferred over a 4k-token retrieval baseline in about 78% of comparisons 2403.05530 §5.2.2. Card: Gemini 1.5.

Support sets for a trained segmentation network

UniverSeg is a convolutional network trained once on many segmentation tasks. At use time it takes the query image and a support set of image-label pairs for the new task, and predicts the label map with no gradient step 2304.06131 §3. Its CrossBlocks pass information between the query and each support entry, and the query takes the mean over the support set, so the support size can vary 2304.06131 §3.1. On six held-out datasets, with 64 support pairs and five averaged support draws, mean Dice is 71.8. That is above SE-net at 50.1 and PANet at 41.8, and below a separate nnUNet per task at 84.4 2304.06131 tab. 1. Dice rises from 53.7 with one support pair to 69.9 with sixteen, and then flattens 2304.06131 §5.3 2304.06131 fig. 7. The network works on 2D mid-slices, so the reported numbers do not cover full volumes 2304.06131 §4. Card: UniverSeg.

A pretrained tabular model applied to forecasting

TabPFN-TS applies TabPFN-v2 to time series. TabPFN-v2 has 11M parameters and was pretrained on synthetic tabular tasks, with no time-series pretraining and no fine-tuning 2501.02945 §3. Forecasting becomes tabular regression: each time step is a row of calendar and periodic features, and the observed history supplies the targets 2501.02945 §3. Each series needs its own forward pass 2501.02945 §6.2. Results are author-reported and shown only in figures. On the 97 univariate GIFT-Eval tasks, the model ranks close behind TiRex, Toto-1.0 and Moirai-2, and it matches TimesFM-2.0 and Chronos-Bolt with a much smaller model 2501.02945 §4.1. With known future covariates on fev-bench it is the strongest overall 2501.02945 §4.2. Its runtime is about 30 times that of Chronos-Bolt-Base under the paper's hardware assumptions 2501.02945 §6.2. Trends beyond the context range are not extrapolated 2501.02945 §5.1. Card: From Tables to Time.

Retrieved demonstrations chosen by similarity

Instead of a fixed or random set, GPT-NER picks the training examples that are closest to the test input. Its best variant embeds each entity span and retrieves the examples whose entities are nearest to the entity being extracted 2304.10428 §4.2.2. With davinci-003 and few-shot prompts on full CoNLL2003, random demonstrations reached 73.44 F1, sentence-level similarity 85.9 and entity-level similarity 90.91 2304.10428 tab. 2. The ordering holds on the nested sets, for example 46.11, 60.02 and 73.59 on ACE2005 2304.10428 tab. 3. Two cautions. The entity embedding comes from a fine-tuned NER model that the paper does not name 2304.10428 §4.2.2, so this variant is not a pure prompt. And at k = 2 and k = 4 the authors report retrieval doing worse than random, because retrieved examples often contained no entities 2304.10428 §6.2. Card: GPT-NER.

LayoutGPT uses the same idea to pick demonstrations of object layouts for a query caption. In 2D it ranks demonstrations by CLIP similarity and places the most similar one last, next to the query 2305.15393 §3.3. Its numerical layout accuracy with GPT-3.5 is 86.33%, against 92.56% for human layouts 2305.15393 tab. 2. Retrieval matters most in 3D: random exemplars raised out-of-bound rate in bedrooms from 43.26% to 85.58%, while in 2D the effect is small 2305.15393 app. C.1 2305.15393 app. B. Card: LayoutGPT.

A biomedical benchmark compares a fixed example with the nearest examples under one template. On LitCovid with GPT-4, macro-F1 rises from 0.5901 with one fixed example to 0.6500 with the single nearest example and 0.7055 with the five nearest 2305.16326 p. 7. The ranking uses cosine similarity over text-embedding-ada-002 vectors 2305.16326 p. 15. The same paper reports that the dynamic one-shot prompt is sometimes below the fixed one-shot prompt, and that five dynamic examples are not GPT-4's best setting on 8 of 12 datasets 2305.16326 p. 8. Card: Benchmarking large language models for biomedical NLP.

Retrieval can also be learned rather than hand-built. EPR trains a dense retriever with labels from a scoring language model. For each training example, candidate demonstrations are ranked by the probability the LM gives the gold output when that candidate is in the prompt. The top-ranked candidates are positives, the bottom ones hard negatives, and two BERT-base encoders are trained on these 2112.08633 §3.1 2112.08633 §3.2. With GPT-Neo both scoring and inference, EPR reaches 31.9 on BREAK dev against 26.0 for BM25 and 25.7 for CBR 2112.08633 tab. 2. On the test split it reaches 23.9 against 17.6 for BM25 2112.08633 tab. 3. With GPT-3 as the inference model on a 1,000-example dev sample, BREAK scores 25.3 against 20.1 for BM25 2112.08633 tab. 5. The runs are single, with no variance reported 2112.08633 §4.3. Card: Learning to Retrieve Prompts.

Nearest-neighbour examples also appear in clinical summarization. Adapted GPT-4 received the most examples its context allowed, chosen by embedding similarity, with the count set to a power of two so that at most 1% of test samples overflowed the window 2309.07430 §3.2. In a reader study of ten physicians, the model summary was preferred in 36% of comparisons and rated equivalent in 45% 2309.07430 §5.2. The progress notes come from one ICU medical centre, which limits transfer 2309.07430 §5.5. The chance of possible harm was 12% for model summaries and 14% for expert summaries 2309.07430 §5.3. Fabricated content was 5% for the model and 12% for the experts 2309.07430 §5.2.2. With one example, tuned FLAN-T5 and Llama-2 usually beat ICL, but with many examples most models passed the best tuned model 2309.07430 §5.1.2. Card: Adapted LLMs for Clinical Summarization.

Nearest-neighbour examples with self-written rationales and a vote

Medprompt combines three steps with GPT-4 and no training 2311.16452 alg. 1. Before inference, each question in the pool gets an embedding and a GPT-4 chain of thought, and is kept only if the chain ends in the gold answer 2311.16452 §4.2 2311.16452 alg. 1. At inference, the k nearest pool items by cosine similarity go into the prompt with their stored chains. The test question is then asked n times, with the answer options shuffled before each call, and the most frequent answer wins 2311.16452 §4.3 2311.16452 alg. 1. The standard setting is k = 5 and n = 5 2311.16452 §4.4. On MedQA, 5-shot GPT-4 scores 81.4 and Medprompt scores 90.2 2311.16452 tab. 1. Going to 20 examples and 11 calls adds 0.4 points 2311.16452 §5.2. In the ablation from a 5-shot baseline, chain of thought adds 3.4 points, and nearest-neighbour selection and option shuffling add 2.2 each 2311.16452 §5.2. The paper gives no Shapley analysis, and the order of addition is its own choice 2311.16452 §5.2. The rationales come from GPT-4 itself. With 5 fixed shots and no ensemble, GPT-4's rationales scored 83.8 on MedQA, against 86.9 for expert rationales 2311.16452 tab. 2. The same recipe, with no changes, gained 7.3 points over zero-shot on six MMLU subjects and two nursing sets, against 7.1 on the MultiMedQA datasets 2311.16452 §5.3. Card: Medprompt.

Same-annotator examples for personalization

ChatGPT evaluation adds a personalized variant. The prompt lists up to three labelled texts from the same annotator, then asks for that annotator's label on a new text 2302.10724 §6.3. The paper's "gain" measures how much of the remaining headroom over the non-personalized run is recovered 2302.10724 §5.4. With ChatGPT, F1 is 72.57 on AggressionPer against 69.10 on Aggression, and 54.02 on UnhealthyPer against 45.21 on Unhealthy 2302.10724 tab. 2. These pairs use different items and sizes, so the gap is not a controlled effect of the examples. The paper's text gives loss gaps and a GoEmoPer gain that do not match its table 2302.10724 §6.3 2302.10724 tab. 2. Card: ChatGPT: Jack of all trades, master of none.

Past successes as examples for an agent

ExpeL runs a frozen agent over training tasks, stores its trajectories, and has an LLM write a list of natural-language insights from pairs of failed and successful attempts 2308.10144 §4.2.2. At test time the prompt holds the whole insight list plus the k most similar past successes 2308.10144 §4.3. With gpt-3.5-turbo as the policy, the author-reported success rate on ALFWorld is 59% against 40% for ReAct, and 42.5% when the examples are random rather than retrieved; retrieving by reasoning similarity gives 48.5% 2308.10144 tab. 3. Each HotpotQA trajectory uses about 4,310 tokens, against 1,320 for ReAct 2308.10144 tab. 6. Card: ExpeL.

Reflexion keeps the model frozen and adds a retry loop. After a failed attempt, the agent writes a short verbal reflection on what went wrong, and the next attempt starts with the most recent reflections in the prompt 2303.11366 §3 2303.11366 §4.3. For ALFWorld, the paper reports a gain of 22 points absolute over ReAct 2303.11366 §1. On HumanEval (Python), it reports 91.0% pass@1 against 80.1% for GPT-4 2303.11366 tab. 2. The loop is only as good as its feedback. Its self-written tests can accept wrong code, and the paper's failure analysis shows such false positives 2303.11366 tab. 3. Card: Reflexion.

LLM-Planner prompts GPT-3 with nine plan examples retrieved from a pool of 100 ALFRED training pairs, together with the objects the agent has seen and the subgoals already done 2212.04088 §4.1 2212.04088 §4.3 2212.04088 §5.3. The planner is called again when a subgoal fails, with the new object list 2212.04088 §4.4. The step budget that also triggers re-planning is not stated 2212.04088 §4.4. In leave-one-out checks on the 100-pair pool, plan accuracy is 40.59 with retrieved examples and 17.48 with random ones, and 13.43 when the logit bias and the retriever are both removed 2212.04088 tab. 2. On the ALFRED unseen test split, with HLSM as the low-level controller, re-planning raises goal-only success from 11.58 to 13.41 and step-by-step success from 15.83 to 16.42 2212.04088 tab. 1. The paper's full-data HLSM scores 20.27 in the goal-only setting 2212.04088 tab. 1. The planner uses about seven GPT-3 calls per task, against 22 for SayCan 2212.04088 §5.4. Card: LLM-Planner.

Binz and Schulz run GPT-3 (Davinci, temperature 0) through classic cognitive-psychology tasks one trial at a time. The prompt for each trial is updated with the model's earlier answers and the feedback it received 2206.14576 §Results. On a horizon bandit task, mean regret is 2.72 for GPT-3 and 3.24 for human participants, a difference the authors report as significant 2206.14576 §Results. The model shows random exploration, with a reward-difference effect of 0.18, but no directed exploration 2206.14576 §Results. In the two-step task, its stay probability after a rare transition with treasure falls, which the authors read as model-based planning 2206.14576 §Results. These are single-model results, and the same paper reports that small rewordings change its answers on vignette problems 2206.14576 §Results. Card: Using cognitive psychology to understand GPT-3.

Relabelled failures as examples, with a second actor for recovery

Hindsight Planner is a few-shot planner for household tasks in ALFRED. A frozen BERT encoder picks the nearest solved trajectories by Euclidean distance and copies them into the prompt. The adapter that predicts object arguments gets five, and each actor and the critic get two 2412.19562 §4.2 2412.19562 §5.1. The initial pool holds 140 solved trajectories, 20 for each of seven task types 2412.19562 §5.1. One actor is prompted with ground-truth examples, and a second with hindsight examples built from the planner's own failed rollouts. An LLM rewrites each failure into a task and a successful rollout, using the ground-truth rollout as a reference 2412.19562 §4.3 2412.19562 alg. 3. A beam search over candidate sub-goals, scored by the critic, returns the first step of the best rollout 2412.19562 §4.5 2412.19562 alg. 2. With GPT-4 Turbo, test-seen SR is 25.51 against 15.33 for LLM-Planner, and test-unseen SR is 18.77 against 13.41 2412.19562 tab. 1. Table 1 mixes full-data and few-shot baselines, and only ALFRED is tested 2412.19562 §5.1. Card: Hindsight Planner.

An external model's suggestion placed in the prompt

Auto-GPT-style loops can receive a supervised model's suggested action as text, and the language model still makes its own choice without any fine-tuning 2306.02224 §2.3. The suggestion is framed as one reference to weigh, and it stays in the message history like any other message 2306.02224 §2.3 autogpt.py. With GPT-4 on the 50 WebShop test instructions, success rises from 0.240 with no suggestion to 0.300 with one and 0.320 with five, against 0.227 for the imitation model alone 2306.02224 tab. 1. On the 134 ALFWorld games, GPT-4 goes from 0.485 to 0.515 with one suggestion, against 0.306 for the imitation model 2306.02224 tab. 2. The effect is not uniform across models, see Contradictions. Card: Auto-GPT for Online Decision Making.

Few-shot prompts in a self-feedback loop

Self-Refine uses the same model in three roles, each driven by its own few-shot prompt: draft an answer, write feedback on the draft, and rewrite the draft using the feedback and all earlier drafts 2303.17651 §2 2303.17651 §3.1. No weights change, so the loop is a prompting pattern built on ICL. GPT-4 on the dialogue-response task rises from 25.4% to 74.6% preference 2303.17651 tab. 1, but GSM8K math moves from 92.9% to 93.1% 2303.17651 tab. 1. Each round adds a feedback call and a refine call 2303.17651 alg. 1. Vicuna-13B could not follow the feedback and refine prompts at all 2303.17651 §4 2303.17651 app. G. Card: Self-Refine.

Self-Debugging applies the same loop to code. The model writes a program, runs it on unit tests or sample inputs, explains the program and its output, judges it, and rewrites it, for up to 10 turns 2304.05128 §3 2304.05128 §5. With Codex, accuracy on TransCoder rose from 80.4 to 92.5 when the explanation was included in the feedback 2304.05128 tab. 2. Gains were smaller for GPT-4 on Spider, 73.2 to 73.6 2304.05128 tab. 2. Card: Teaching Large Language Models to Self-Debug.

Earlier scored prompts in the context of an optimizer

OPRO uses a language model as the optimizer for a problem stated in words. A meta-prompt lists earlier candidate instructions with their scores, and the model writes new candidates from it. Each candidate is run and scored, and the result goes back into the meta-prompt 2309.03409 §2. For GSM8K with PaLM 2-L as the scorer, the instruction "Let's think step by step." scores 71.8%, and the best instruction the optimizer found scores 80.2% 2309.03409 tab. 4. The abstract reports gains of up to 8% on GSM8K over human-designed prompts. The table picks, for each optimizer and scorer pair, the instruction with the highest test accuracy, so its figures are test-set maxima 2309.03409 tab. 4. The paper suggests choosing by training accuracy for a deployed prompt 2309.03409 §5.4. Card: OPRO.

Few-shot parameter-efficient fine-tuning instead of prompting

T-Few takes the same few labelled examples and trains a small set of learned scaling vectors, (IA)3, on top of a frozen T0 model. The vectors are reused for every later query, so the examples do not need to be re-read each time. The recipe is the same for every task. Training adds an unlikelihood loss on wrong answers and a length-normalized loss over answer choices, both summed with the language-modelling loss 2205.05638 §3.2 2205.05638 §3.5. Across nine held-out T0 tasks, median accuracy is 72.4% for T-Few against 66.6% for GPT-3 175B with few-shot prompts. The paper reports about 1,000 times fewer inference FLOPs per prediction, and a one-off training cost of 2.7e16 FLOPs 2205.05638 tab. 1. The same recipe scores 75.8% on RAFT, against 73.5% for the human baseline listed there 2205.05638 tab. 2. These are author-reported results, and the GPT-3 numbers are copied from the GPT-3 paper rather than re-run 2205.05638 app. F. Card: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning.

Explanations from a prompted model, used to train a smaller model

Wadhwa et al. prompt GPT-3 (text-davinci-002) with worked examples to write an explanation for each training example that supports its gold triplets. They then fine-tune Flan-T5 Large on the gold outputs together with those explanations 2305.05003 §4.2.2. The few-shot GPT-3 prompt alone gets 76.53 micro-F1 on CoNLL04, 82.66 on ADE and 61.79 on NYT 2305.05003 tab. 2. The distilled Flan-T5 gets 80.76, 92.17 and 95.23 on the same three sets 2305.05003 tab. 2. Fine-tuning on the gold outputs alone gets 75.28 on CoNLL04, below the few-shot prompt, and 83.15 on ADE, above it 2305.05003 tab. 2. Generating the explanations is the main cost: for NYT, the paper explains 25k of 56k training examples for about US$984 2305.05003 §4.2.2 2305.05003 tab. 5. Card: Revisiting Relation Extraction in the era of Large Language Models.

Few-shot program exemplars that call the model

Binder maps a question about a table to a program in SQL or Python, and some sub-expressions in that program are calls to the model. It uses 14 annotated program exemplars and no training, with Codex (code-davinci-002) for both parsing and the calls 2210.02875 §3.1 2210.02875 §2.2. On WikiTQ test, it scored 64.6 against 61.1 for a Codex SQL-only program and 48.7 for end-to-end QA with the same model 2210.02875 §3.2. On TabFact, a retriever that chooses the exemplars added 0.9 points over fixed exemplars, reaching 86.0 2210.02875 §3.2. Under the official WikiTQ evaluator, the ranking against OmniTab changes, so the headline result depends on the evaluator 2210.02875 app. C.2. Card: Binding Language Models in Symbolic Languages.

Few-shot program of skill calls for an embodied agent

TANGO is a zero-shot embodied agent. A pretrained point-goal navigation policy, a memory-backed exploration map and several vision detectors serve as callable primitives. GPT-4o receives the instruction and a fixed prompt of 15 worked examples, and writes a short step-by-step program that calls these primitives 2412.10402 §3. The paper reports no fine-tuning for the task 2412.10402 §4.1. Author-reported results: OVON val-unseen SR 35.5 against 35.2 for VLFM and 37.1 for DAgRL with object detection 2412.10402 tab. 1; OpenEQA score 37.2 against 38.1 for the best Socratic baseline 2412.10402 tab. 2; GOAT-Bench SR 32.1 against 29.5 for the best prior row 2412.10402 tab. 3. In its failure analysis, the LLM wrote incorrect pseudo-code in about 18% of cases, and the sample size is not stated 2412.10402 §4.3. Card: TANGO.

Program or formal file as the answer, run by an interpreter or solver

PAL asks the model to write its reasoning as Python, with a comment on each line, and lets a Python interpreter produce the answer. The examples carry no final answer, and no training is involved 2211.10435 §3. With Codex (code-davinci-002) and greedy decoding, PAL reaches 72.0% on GSM8K against 65.6% for chain-of-thought with the same model, and the authors report it ahead of chain-of-thought on all 13 tasks they tested 2211.10435 §5.1 2211.10435 §5.2. The interpreter does real work: when the model computes the result from the program itself, GSM8K falls to 23.2 2211.10435 §6. The gain depends on the model. With text-davinci-001, PAL scores 8.6 against 26.5 for chain-of-thought, while text-davinci-002 reverses the order, 65.8 against 46.9 2211.10435 §6. Card: PAL: Program-aided Language Models.

LLM+P turns the same idea to planning. One (description, problem file) pair in the prompt makes GPT-4 write a PDDL problem file for a fixed domain, and the Fast Downward planner solves it with its optimal setting 2304.11477 §3.2 2304.11477 §3.3. The authors define in-context learning here as conditioning on a few input-label pairs without changing parameters 2304.11477 §3.2. Their table reports optimal-alias success of 90% on Blocksworld, 95% on Grippers, 85% on Storage and 0% on Floortile 2304.11477 tab. 1. With the example removed, the pipeline produces no correct problem files and scores 0 in every domain 2304.11477 tab. 1 2304.11477 §5.3. The same model writing plans directly, with the example, reaches 15% on Blocksworld and 35% on Grippers 2304.11477 tab. 1. The authors name a mis-specified problem file, such as a missing initial condition, as the usual failure 2304.11477 §5.3. The paper varies no part of the example, so it does not show which part matters 2304.11477 §3.2. Card: LLM+P.

Decomposed prompts, one prompt per step

Decomposed prompting replaces one long few-shot prompt with a decomposer prompt. The decomposer writes a program of sub-questions. Each sub-question goes to its own few-shot prompt, a symbolic function, or a recursive call of the same procedure 2210.02406 §3 2210.02406 §3.3. A controller runs the program, feeds each answer back to the decomposer, and returns the last answer 2210.02406 §3.2 2210.02406 fig. 3. The paper's case studies show the gain on specific shapes of problem. On letter concatenation with an unseen position k, it stays close to 100% as inputs grow, while chain-of-thought drops with longer inputs 2210.02406 §4.1 2210.02406 fig. 7. On list reversal, the recursive version generalizes to longer lists and the base chain-of-thought prompt does not 2210.02406 §4.2 2210.02406 fig. 8. Splitting the answer-extraction step off a chain-of-thought answer lifted the reported GSM8K score from 36 to 50.6 and MultiArith from 78 to 95 2210.02406 §4.5 2210.02406 app. B. Card: Decomposed Prompting.

Text-to-SQL uses the same pattern at larger scale. DIN-SQL runs four modules in turn: schema linking, a classification of the query, generation with a prompt for that class, and self-correction. Each class has its own fixed demonstrations 2304.11015 §4 2304.11015 §7. With GPT-4 it reaches 85.3 execution accuracy on the Spider holdout test, against 78.2 with Codex Davinci 2304.11015 tab. 2. On Spider dev, the GPT-4 version scores 74.2 against 67.4 for a single few-shot prompt 2304.11015 tab. 4a. Most development used Spider dev, so only the test numbers come from a hidden server 2304.11015 §5.5.2 2304.11015 §5.5.1. The authors estimate about US$0.5 and about 60 seconds per question with GPT-4 2304.11015 §7. Card: DIN-SQL.

Closed-book few-shot question answering

A few-shot prompt with no retrieval asks the model to answer from its weights. On TriviaQA, BLOOM-176B scored 25% on questions whose answer entity and question entity co-occur in 10 pre-training documents, and above 55% for 10,000 documents 2211.08411 §1 2211.08411 fig. 1. Accuracy on rare questions grows log-linearly with model size, and the fit projects a very large parameter count to match a supervised baseline 2211.08411 §4.2. Retrieving documents with BM25 gave better accuracy than closed-book prompting at every document count, most of all for rare questions 2211.08411 §4.3 2211.08411 fig. 9. Card: Large Language Models Struggle to Learn Long-Tail Knowledge.

Few-shot prompts that write training data

Promptagator gives an instruction-tuned 137B model a short task description and two to eight labelled query-document pairs. It then asks the model to write queries for documents from the target corpus 2209.11755 §2.2 2209.11755 §3.1. A dual encoder is trained on the generated pairs, after a filter that keeps a pair only when the encoder ranks its document first for that query 2209.11755 §3.2. Averaged over 11 BEIR datasets, the 110M retriever scores 47.8 nDCG@10 with few-shot prompts and 45.5 with one universal prompt, against 46.2 for ColBERT v2 trained on MS MARCO 2209.11755 tab. 2. A 110M reranker on top raises the average to 52.8 2209.11755 tab. 2. The generator is described as provided by the authors, and the paper names no public release of it 2209.11755 §3.1. Card: Promptagator.

Self-Instruct uses the same kind of prompt to write training data for instruction tuning. Each request shows GPT-3 (davinci) eight tasks, six hand-written seeds and two earlier generated ones, and asks it to continue the numbered list 2212.10560 §2.2. A new instruction is kept only if its ROUGE-L against every existing instruction is below 0.7, and length, keyword and punctuation filters remove more 2212.10560 §2.2. The model then writes inputs and outputs for each task, and GPT-3 is fine-tuned on the filtered result 2212.10560 §2.2 2212.10560 §4.1. The examples shape the generated data, while the fine-tuned model is evaluated zero-shot, so this is a training-time use rather than a test-time prompt. On zero-shot SuperNI, ROUGE-L rises from 6.8 for untuned GPT-3 to 39.9, against 40.8 for InstructGPT-001 2212.10560 tab. 3. Only 54% of a 200-instruction sample had every field judged valid, by one author 2212.10560 tab. 2. Card: Self-Instruct.

Serialized table rows in a few-shot task template

Narayan et al. recast five data-cleaning jobs as prompts for GPT-3 175B: entity matching, imputation, error detection, data transformation and schema matching 2205.09911 §1 2205.09911 §3. Each table row is written as attribute-value text, each task has a fixed question, and 0 to 10 labelled examples come before the query. Nothing is trained. With manually chosen examples, the authors report state-of-the-art or tied scores on several datasets. On Beer entity matching, F1 rises from 78.6 with no examples to 100 with ten 2205.09911 tab. 1. The example choice matters as much as the wording: manual examples beat random ones by 14.7 F1 on average over three entity-matching sets 2205.09911 §4.3. The released code differs from the paper in several details, including how random demonstrations are drawn, so check the card before reproducing run_inference.py. Card: Can Foundation Models Wrangle Your Data?.

Prompt-based model editing

Cohen et al. test in-context editing (ICE), which writes the new fact as a sentence at the start of the prompt and changes no weights 2307.12976 §5.1. On the Recent subset, LLaMA-7B with ICE averages 82.8 over six criteria, against 60.8 for ROME, a weight editor 2307.12976 tab. 3. On Random, GPT-NeoX-20B with ICE gets 81.0 against 58.3 for ROME 2307.12976 tab. 4. On Popular, LLaMA with ICE gets 81.3 against 49.4 for ROME 2307.12976 tab. 5. The widest gap is on logical generalization: 57.2 with ICE and 22.0 with ROME on the Popular subset 2307.12976 tab. 5. ICE mostly fails by abstaining. For LLaMA, 71% of the 200 sampled failures are abstentions 2307.12976 tab. 7. Card: Evaluating the Ripple Effects of Knowledge Editing.

Dialogue prompt that sets a persona

A chat agent built on a base model starts from a dialogue prompt: a preamble, sample dialogue in script form, and a cue for the user 2305.16367 §3 2305.16367 fig. 2. Shanahan et al. describe the reply as the model playing a character that the prompt makes most plausible, and the character is refined as turns are added 2305.16367 §3. The paper reports no experiments or measured effect of the prompt, so treat this as a way of reading in-context behaviour, not as a result 2305.16367 §1. The framing is for base models, and the authors say its validity after fine-tuning is unclear 2305.16367 §2 2305.16367 §8. Card: Role-Play with Large Language Models.

Same-type demonstrations to probe a capability

Sap et al. use in-context learning as a measuring tool rather than a method. For SocialIQa and the ToMi false-belief stories, they draw k shots of the same reasoning type as the test item, and pick the candidate answer with the highest likelihood 2210.13312 §2.1 2210.13312 §3.1. For GPT-3 Davinci, the best SocialIQa accuracy is about 55%, with 35 shots 2210.13312 §2.2. On ToMi, mental-state questions reach about 60%, against 90 to 100% for factual ones 2210.13312 §3.2. Chat models were tested zero-shot on 400 items each. GPT-4 under multiple-choice probing reaches 79.3% on SocialIQa, but its mental-state accuracy on ToMi stays near 60% 2210.13312 app. D. The shots show a format, and the scores move with which shots are drawn and how answers are scored 2210.13312 §3.2 2210.13312 app. C. Card: Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs.

A history window read by a trained robot policy

The walking controller in the Digit humanoid paper uses the phrase for a trained policy, not a prompt. It is a causal transformer that reads a window of 16 past observation-action pairs and outputs the next action 2303.03381 §Model architecture. Its weights are trained with PPO and a KL term to a teacher, so the history is an input to a network, not text placed in a prompt 2303.03381 §Joint optimization with reinforcement learning. The authors suspect the history reveals the ground, such as a slope or a snagged foot, so the robot can adjust without being told the terrain 2303.03381 §Introduction. They describe this as in-context adaptation without weight updates. The one ablation the card records compares student architectures, MLP, TCN and LSTM against the transformer, with values only in a figure 2303.03381 fig. 8. The history window itself is not isolated. The paper shares the term with this page, not the mechanism, so read it as a loose match. Card: Real-World Humanoid Locomotion with Reinforcement Learning.

When to use which

Implementation sketch

Our own sketch of the common core: format each demonstration as an input and a label, joined by newlines, with blank lines between examples; score every candidate label; return the best. The separators follow the paper's appendix 2202.12837 app. B. The paper does not state a trimming rule for long prompts; the sketch cuts the earliest demonstrations, which is one choice. For retrieved demonstrations, demos is the k nearest examples to x instead of a fixed list 2304.10428 §4.2.2.

# Build a k-shot prompt from labelled demonstrations  [arxiv:2202.12837 app. B]
# Input:  demos [(x_1, y_1) ... (x_k, y_k)], test input x, candidate labels C
# Output: the candidate label the model scores highest

def predict_icl(demos, x, C, scoring):
    # one newline inside a pair, three newlines between pairs  [arxiv:2202.12837 app. B]
    context = ""
    for x_i, y_i in demos:
        if scoring == "direct":
            context += x_i + "\n" + y_i + "\n\n\n"
        else:
            context += y_i + "\n" + x_i + "\n\n\n"
    scores = {}
    for y in C:
        # direct: score the label after the input; channel: score the input
        # after the label  [arxiv:2202.12837 §3]
        if scoring == "direct":
            prompt, target = context + x + "\n", y
        else:
            prompt, target = context + y + "\n", x
        # trim the earliest demonstrations if prompt + target exceeds the
        # context window; the paper does not say how this is done
        prompt = trim_left(prompt, target, max_tokens=CONTEXT_LIMIT)
        # log-probability of the target tokens given the prompt
        # [arxiv:2202.12837 §4.1]
        scores[y] = log_prob(prompt, target)
    return argmax(scores)

Contradictions

• Whether labels teach the mapping. Min et al. find that gold labels add only a few points over random labels on classification and multi-choice, and read this as the model relying on label space and format 2202.12837 §4.1 2202.12837 §5. Reynolds and McDonell read few-shot gains as task location, which Min et al. name as a related analysis 2102.07350 §1 2202.12837 §2. Neither paper tests arithmetic, where Wei et al. report few-shot gains that appear only at scale 2206.07682 tab. 1. So the cards do not settle whether labels matter for multi-step tasks.

• How much the symbols matter. Min et al. report that random English words cost direct models 5-16 points 2202.12837 §5.2. Pattern-machine results lose far more on ARC, 85 to 44 solved, and 17 points on PCFG with random alphabets 2307.04721 tab. 1 2307.04721 tab. 2. These are different tasks and symbol sets, so the two sizes are not directly comparable.

• Which examples versus which labels. GPT-NER's gains come from choosing examples by similarity, and its tables have no random-label condition 2304.10428 tab. 2. Min et al. vary the labels and keep the examples drawn at random 2202.12837 §4.1. The two results measure different things, so they do not conflict, but neither shows the effect of the other factor.

• Whether retrieval helps. GPT-NER reports retrieval below random at k = 2 and k = 4 2304.10428 §6.2. LayoutGPT reports retrieval as essential in 3D and small in 2D 2305.15393 app. C.1 2305.15393 app. B. ExpeL finds retrieved past successes beat random ones on ALFWorld 2308.10144 tab. 3. The value of retrieval depends on the task and the number of examples, so test it on your setup. A biomedical benchmark finds dynamic examples raise LitCovid macro-F1 as the count goes from one to five, but not always above fixed examples 2305.16326 p. 7 2305.16326 p. 8. Auto-CoT reports similarity-retrieved demonstrations below random ones on MultiArith, 82.8 against 86.2, when the demonstrations are zero-shot chains 2210.03493 tab. 1. LLM-Planner reports the opposite for plan prediction, 40.59 with retrieved examples against 17.48 with random ones 2212.04088 tab. 2. The chains, tasks and metrics differ, so the two do not settle the question.

• Zero-shot against few-shot. Reynolds and McDonell report zero-shot beating a reproduced 10-shot prompt on one translation task 2102.07350 tab. 1, and report that one added example hurt Curie 2102.07350 tab. 1. Med-HALT's GPT-3.5 accuracy rose from zero to about three examples 2307.15343 §7.2. The legal study finds three shots raise hallucination on some tasks and lower it on others 2401.01301 tab. 2 2401.01301 tab. 3. These are different tasks and metrics, so none of them says few-shot helps or hurts in general. The biomedical benchmark adds that more examples do not always help under dynamic prompts with GPT-4 2305.16326 p. 8. Two pretraining reports show averages that rise with more shots. GLaM's averages rise with each added shot, as do GPT-3's 2112.06905 tab. 1. On ETHOS, OPT-175B's F1 rises from zero to few-shot prompts, while Davinci's falls on the binary few-shot prompt 2205.01068 tab. 3. The direction depends on the model, the task and the prompt format.

• Reproduced GPT-3 baselines. Reynolds and McDonell's reproduction of GPT-3 scores well below the numbers published in the GPT-3 paper, for example 1-shot Babbage at 21.8 against 31.6 2102.07350 tab. 1. The "beats 10-shot" claim is against that reproduction, not the published numbers.

• Prompting against fine-tuning. Liu et al. report a fine-tuned 11B model beating few-shot GPT-3 175B on held-out tasks, at much lower inference cost 2205.05638 tab. 1. The other cards on this page compare prompt variants with each other and do not test fine-tuning, so they do not show which route is better in general. Two other cards show mixed results. On relation extraction, fine-tuning on gold outputs beats few-shot GPT-3 on ADE but not on CoNLL04 2305.05003 tab. 2. On biomedical tasks, a fine-tuned LLaMA 2 13B beats few-shot GPT-4 on the entity and relation tasks, but not on MedQA 2305.16326 tab. 3. For knowledge editing, prompt-based ICE beats weight editors on the ripple criteria 2307.12976 tab. 3. That runs the other way, but it is a different task, so the results do not directly conflict.

• Random labels against flipped labels. Min et al. find that random ground-truth labels cost only 0-5 points on most classification and multi-choice setups 2202.12837 §4.1. Wei et al. find that systematically flipped labels push large models well below chance 2303.03846 §3. The two edits do different things: random labels leave the input-label link at chance, while flipped labels make it consistently wrong. The models and tasks also differ, so the cards do not settle how much labels matter for a given task.

• Whether prompting matches supervised extraction. GPT-NER reports 90.91 entity F1 on CoNLL2003 with retrieved demonstrations 2304.10428 tab. 2. The biomedical benchmark reports GPT-4 with five examples at 0.6389 on NCBI Disease, against 0.9090 for a fine-tuned PubMedBERT 2305.16326 tab. 3. mGPT's 4-shot NER reaches 15.0 F1 2204.07580 tab. 9. The datasets, prompts and models differ, so these numbers do not measure one thing. The first does not show that biomedical extraction works with the same prompt.

• Whether an added suggestion helps an agent. With GPT-4, an imitation model's suggestion raises success on both WebShop and ALFWorld. With GPT-3.5, the suggestion lowers ALFWorld success from 0.075 to 0.030, and a random suggestion halves its WebShop success 2306.02224 tab. 1 2306.02224 tab. 2. The effect depends on the model and the task, so measure it for each model you use.

• Which demonstrations to pick for reasoning. Step-count selection beats random cases on GSM8K with code-davinci-002, 66.6 against 60.4 2210.00720 tab. 1. On MultiArith with the same model, random cases score higher, 97.3 against 95.8 2210.00720 tab. 1. On MathQA, retrieval beats step count, 69.5 against 42.5 on validation 2210.00720 tab. 4. Medprompt credits its nearest-neighbour examples with 2.2 points on MedQA 2311.16452 §5.2. The tasks and annotation budgets differ, so no single selection rule is shown to win.

• How many shots. Preliminary runs in the translation paper found no gain beyond five shots 2302.09210 §3.1. Medprompt's 20 examples with 11 calls add 0.4 points over 5 examples and 5 calls on MedQA 2311.16452 §5.2. Both are single-task results, so test the count on your own data.

• Zero-shot against few-shot for images. Emu2 scores 33.5 on VQAv2 with no examples and 68.8 with sixteen 2312.13286 tab. 1. Flamingo's zero-shot row uses a different prompt condition, so the two zero-shot numbers are not directly comparable 2312.13286 tab. 1.

• Whether few-shot jumps are real. Wei et al. tabulate few-shot prompting thresholds at which abilities appear 2206.07682 tab. 1. The emergence survey reports that Schaeffer et al. find such jumps vanish under linear metrics, while Du et al. and Zhao et al. find they survive 2503.05788 §3.1. The survey leaves the question open. Its view is that an abrupt curve alone does not show emergence, and that a checkable internal mechanism is needed 2503.05788 §2.

Pitfalls

• Scope of the ablations. Min et al. study only classification and multi-choice with natural-language inputs; generation is not covered 2202.12837 §2 2202.12837 §6. Their findings do not carry over to generation without a test.
• Thin evidence for the zero-shot claim. The translation result is one task, with two small API models whose sizes are inferred, and decoding settings are not stated 2102.07350 tab. 1 2102.07350 §3.1. The other techniques in that paper are shown on single examples, with no accuracy numbers 2102.07350 §4.
• Prompt layout moves scores. Changing only separators, spaces, label casing or enumeration style in a few-shot prompt can move accuracy a lot. On one LLaMA-2-13B task the spread across formats reached 76 points 2310.11324 §1. Across tasks the median spread was 7.5 points, and the spread did not shrink with model size, shot count or instruction tuning 2310.11324 §4.2. Report a range over reasonable formats, not one score. The paper's FormatSpread search finds the best and worst formats with Thompson sampling, within about 1 point of the true spread at its largest budget 2310.11324 §4.5. Card: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design.
• Prompt wording moves scores. On SuperGLUE, per-prompt results for BLOOM vary widely, and the average over prompts sits near chance on most tasks 2211.05100 §4.2. Report results per prompt, not only the best one. The judge study tested only two rewordings, and the authors say other changes may behave differently 2305.01937 tab. 3.
• Worked examples are not always reproducible. Med-HALT draws its worked examples with an unseeded random sample, so repeated runs can use different demonstrations 2307.15343 §7.4. Fix the seed and log which examples you used.
• Prompts travel poorly across models. The diagnosis prompts were picked on a 95-question development set with GPT-3.5 and then reused for GPT-4 2308.06834 p. 4. Re-select on each model you use.
• Mismatched baselines. GPT-3.5 used the DSP library and self-consistency on its chain-of-thought prompts, and GPT-4 used neither 2308.06834 p. 4. Compare the two models with that asymmetry in mind.
• Emergence thresholds do not transfer. The thresholds in the emergence table are for specific model families and task setups, and the paper says the scale where an ability appears depends on data, architecture and training 2206.07682 §5.2 2206.07682 tab. 1.
• Metric choice shapes the curve. Exact match on long answers hides progress that log-likelihood shows 2206.07682 §5.1. When you test a small model, log the target likelihood as well as the score.
• Abstentions can hide errors. Abstentions count as non-hallucinations, and most of them occur in zero-shot quotation questions, so those zero-shot rates understate hallucination 2401.01301 app. F 2401.01301 app. E.4.
• Self-feedback rarely fixes reasoning. On GSM8K, ChatGPT's feedback said everything looked good for 94% of instances 2303.17651 §3.3. Self-Refine's gains on math came back only with an external correctness signal 2303.17651 app. H. Self-Debugging's gains also rest on execution results or unit tests 2304.05128 §3.
• Judge bias. Four Self-Refine tasks are scored by GPT-4, which is also one of the models being refined 2303.17651 §3.2.
• Scoring details. The paper writes the decision as an argmax of P(label | context) 2202.12837 §4.1. Averaging the log-probability over target tokens instead changes the comparison between labels of different length. Choose one and report it.
• Evaluator changes the ranking. Binder's headline WikiTQ result uses a semantic-match evaluator. Under the official evaluator, Binder scores 61.9 on test and OmniTab 62.9, which reverses the order 2210.02875 app. C.2. Run the same evaluator for every system you compare.
• Decomposition needs a stop rule. The paper's text gives no step cap. Its code stops at 10 decomposition steps by default participant_qgen.py. Set and report a cap of your own.
• Visual prompting uses different masks in training and testing. Pretraining hides random patches, but testing hides a fixed bottom-right block models_mae.py mae_utils.py. The authors do not discuss this gap. The reference repository has no licence file, so re-implement the method rather than copy the code.
• Fine-tuning comparison is not controlled. The T-Few comparison uses its own few-shot subsets, not the GPT-3 authors' subsets, so the two sides see different demonstrations 2205.05638 §3.1 2205.05638 app. F. Its tasks are classification and multiple choice, and generation is future work 2205.05638 §5. Its FLOP figures are estimates, not measured latency 2205.05638 §4.2.
• Reported training length differs from the code. Channel prompt tuning is described with 100 training steps. The released code trains 400 steps with a learning-rate decay, but its evaluation loop returns after the step-100 checkpoint. The reported model is therefore the step-100 one, with the learning rate already at 0.75 of its start 2108.04106 app. B main.py.
• Demonstrations need a fixed seed. The mGPT code draws few-shot demonstrations with an unseeded sampler, and the paper's five-seed protocol is not reproduced by the files shown 2204.07580 §4.2.1 mgpt_classification.py.
• Examples from a test-only split are evaluation items. When a dataset has only a test split, the prompt examples come from it, and they must be counted as misses even when the model gets them right 2209.11755 §2.2.
• Released segmentation code differs from the paper. The UniverSeg code is a four-stage encoder with a three-stage decoder, max-pool downsampling and no activation on the cross path. The paper describes five and four stages, bilinear resampling and an activation after the cross convolution 2304.06131 §5.1 model.py. Say which description the weights you use match.
• Edits are scored on the whole string. The RippleEdits matcher checks aliases as substrings of the prompt plus the generation, after removing only the ICE context. Words in the query can therefore count as matches queryexecutor.py queryexecutor.py.
• Expert suggestions drop out silently. The Auto-GPT history is the last ten messages, trimmed to a token budget, so earlier suggestions disappear without a warning prompt.py prompt.py. Gaps of one task (0.02 on the 50 WebShop instructions) are within noise 2306.02224 §3.1.1.
• Human re-judging is not reproducible from the card. Exact-match errors in relation extraction were re-judged by crowd workers, and the paper does not write out how those judgements enter precision and recall 2305.05003 §3.2. Its own GPT-3 NYT number differs between tables, 61.79 and 68.55 2305.05003 tab. 2 2305.05003 tab. 3.
• Macro-averages mix metrics. The biomedical headline macro-average combines ROUGE-L, F1 and accuracy, and the paper flags this as a limitation. Its baseline numbers are copied from the literature and were not re-run 2305.16326 p. 12 2305.16326 p. 14.
• Manual answer extraction. Answers in a different format were pulled out by hand, and the number of such cases is not stated 2305.16326 p. 15.
• A survey gives no example counts. The dialogue survey lists in-context learning among prompting methods, with no example count or result, and its citations contain errors. Use it for framing only 2402.18013 §2.5.1.
• Label-free order search still has an oracle. The upper bound in the order paper selects with validation labels, so it is not a few-shot method 2104.08786 §4. Report it as a bound, not as a baseline you could reach.
• Prompt search can overfit the test set. The optimizer table reports the best test accuracy per pair, which inflates the headline figure 2309.03409 tab. 4. Select on training accuracy and report test once.
• Label-flip results rest on 100 examples. Each dataset in the flipped-label and meaningless-label experiments uses 100 evaluation examples, with no intervals or repeated runs. Several claims rest on gaps of a few points 2303.03846 §2.3 2303.03846 §5.
• Scoring rule changes the probe. The paper gives "highest conditional likelihood" without a length rule, but the released code sums token log-probabilities, which favours shorter answers 2210.13312 app. C gpt3LMMCprobing.py. Chat models were probed with multiple choice instead, a different setting 2210.13312 app. D. Report which rule you use.
• Released data-cleaning code differs from the paper. The code draws random demonstrations once and reuses them in every trial, so the ± values in the paper's ablation cannot come from separate draws as the code is written run_inference.py 2205.09911 §4.3. Its error-detection metric also has a debugger call that will stop a batch run utils.py. The fine-tuning code is not in the repository 2205.09911 app. A.
• Few-shot can hurt a binary task. Davinci's F1 on ETHOS fell from 0.628 at zero shots to 0.354 with few-shot binary prompts, while OPT-175B's rose 2205.01068 tab. 3. Check each model at each shot count.
• Some prompts exist only in figures. The full DeID-GPT prompt is in a figure, and the list of identifiers may not be in it 2303.11032 fig. 8. Write down the exact prompt you run.
• Dev-set numbers are not held out. For text-to-SQL, the development split was used for most design choices, so compare dev numbers only with other dev numbers 2304.11015 §5.5.2.
• Generated code runs without a sandbox. Program-style prompts such as PAL's execute the model's output in the host process, and the reference time limit works only on Unix and only in whole seconds 2211.10435 §3 runtime.py. Run such code in an isolated process before you trust its answers.
• Failed API calls look like unsolvable tasks. The LLM+P reference code returns an empty string after repeated failed calls, and the planner then receives an empty problem file. Nothing checks this before planning main.py main.py. Check the model's reply before you call the solver. The paper's failures were mostly mis-specified files 2304.11477 §5.3.
• Earlier turns can stay in the prompt. ChatGPT's prompts shared one conversation unless the authors forced a new one, so earlier messages could influence later answers 2302.10724 §5.1. On SQuAD v2 the separate-conversation run scored 53.60 accuracy against 56.50, which the authors call not significant 2302.10724 tab. 3. Start a fresh context per query when you need independent answers.
• Prompt wording moved ChatGPT's scores by double digits. Different prompt patterns changed PolEmo by up to +14.8 points and TextEntail by up to −13.0 points 2302.10724 tab. 4.
• Demonstration counts are capped by the context window. DecodingTrust uses 8 long-sample ETHICS examples, and at most 32 for its fairness test, because of the token limit 2306.11698 app. machine-ethics details 2306.11698 §10. Other shot counts are not measured on those tasks in the card.
• Vote settings are not given. The complexity paper gives no sampling temperature or top-p for its 50 chains, and K is only "typically 30-40" 2210.00720 §3.2. Choose K on validation data, not on test data 2210.00720 §4.3.
• Step counts depend on formatting. Step count splits on line breaks, and the paper does not say how blank lines, the answer line or ties are handled 2210.00720 §3.1. The paper's own tables mix validation and test numbers, so take baselines from the same split 2210.00720 tab. 1 2210.00720 tab. 3.
• Pool filtering keeps lucky answers. Medprompt drops pool items whose rationale ends in the wrong answer, but a right answer can come from faulty reasoning, and the filter does not catch that 2311.16452 §4.2. The pool size after filtering is not stated.
• Held-out split is not clean. The 20% held out for Medprompt was not used in tuning, but the design choices were made on the other 80% 2311.16452 §3.2 2311.16452 §5.1. The sampling temperature for the calls is not stated 2311.16452 §4.4.
• Lexical scores mislead for translation. BLEU and chrF fall for all GPT settings, while COMET and human judgements do not 2302.09210 §3.2. A gate that uses BLEU alone would likely reject good output.
• Translation models are not pinned. The three OpenAI models were called through Azure with no snapshot or access date 2302.09210 §2.3. Running the same prompts on other models is a different experiment.
• The zero-shot prompt is part of the result. The plain prompt made Emu2 write sentences, so the zero-shot numbers use a modified prompt, and few-shot results depend on RICES selection 2312.13286 app. E.1. Reproduce both prompts exactly.
• Long-context scores have a single judge. The Les Misérables preference was rated by Gemini 1.5 Pro itself, and the Kalamang human scores come from one non-native rater who can recognise their own translations 2403.05530 §5.2.2. The paper does not measure how much the self-judging biases the preference scores 2403.05530 §5.2.2.
• Emu2 weights are not cleared for commercial use. The code is Apache-2.0, but the released weights are listed as non-commercial in its README. This is recorded on the Emu2 card, not in the paper, so check it before reusing the weights.
• Demonstrations may come from the test questions. Auto-CoT builds its demonstrations from the questions it evaluates, and the paper does not say whether a test question can serve as its own demonstration 2210.03493 §4.2. A cluster with no candidate that passes the filter adds no demonstration, so the count can fall below k 2210.03493 alg. 2.
• Plan examples are tuned on the pool they are drawn from. LLM-Planner picks K, the prompt layout and the logit bias by leave-one-out cross-validation on the same 100 pairs, which the paper calls true few-shot 2212.04088 §3 2212.04088 §4.2. Re-running that selection on another pool can change the result. The released ALFRED evaluator reads object visibility from the simulator, not from a detector thor_connector.py, so its numbers do not test the detection step the paper describes 2212.04088 §5.3.
• Trajectory-level results come from one model and one run. Binz and Schulz name the model only as "Davinci". Their released two-step code uses temperature 1, while the paper says temperature 0 for all runs query.py 2206.14576 §Results. Fix the engine and the temperature, and repeat runs before you compare with the reported numbers.
• Training and evaluation shot counts must match the claim. Otter's gains appear when the trained and evaluated shot counts line up, and a model trained without examples falls back at four shots 2305.03726 tab. 10. Report both counts whenever you report a shot-count result.
• Grounding claims are argued, not measured. The grounding paper runs no experiments 2304.01481 §1 2304.01481 §8, and it treats in-context learning as one of three speculative routes to grounding 2304.01481 §6.2.3. Do not cite it as evidence that prompted behaviour is about the world.
• Persona framing is untested. The role-play paper proposes two behavioural tests and runs neither 2305.16367 §7. Do not present the dialogue-prompt framing as a measured effect of prompting.
• The released embodied evaluator skips the LLM. TANGO's evaluator uses fixed per-task program templates, so its numbers do not test the main claim. The templates also insert the episode's target name, which a system that sees only the instruction does not have. Write your own LLM call and keep the prompt format from the example file. Details are on the TANGO card, which cites the released code.
• Hindsight examples are written with the answer in view. The reflection and the rewritten rollout are produced while the ground-truth rollout is visible, so the second actor learns from the expert's solution. The paper's argument for this is theoretical, not measured 2412.19562 alg. 3 2412.19562 §4.3. At test time the adapter predicts the object arguments that the relabelling used as ground truth, and the paper does not measure how much its errors cost 2412.19562 §4.4.

Lineage

• Builds on: the in-context learning setting popularised by GPT-3 (Brown et al., arXiv:2005.14165), whose translation prompts and few-shot numbers are the baseline for 2102.07350 tab. 1.
• Related analyses: the MetaICL model and k-shot splits (arXiv:2110.15943) have no card, so they are named here only. The channel-scoring study from Min et al. is carded: Noisy Channel Language Model Prompting.
• Vocabulary: "template", "answer" and the prompt-engineering terms used here follow the survey of prompting methods (arXiv:2107.13586), which has no results of its own 2107.13586 §2.2 2107.13586 §2.3.
• Superseded by: nothing recorded in this wiki yet.

Papers

• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm: argues few-shot examples mostly locate a known task, and proposes zero-shot prompt techniques.
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?: shows random labels cost little and isolates label space, input distribution and format as the drivers.
• Emergent Abilities of Large Language Models: defines abilities absent in small models and present in large ones, and tabulates few-shot and prompting thresholds.
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model: an open model evaluated mainly with zero- and one-shot prompts, with per-prompt variance reported.
• Self-Refine: Iterative Refinement with Self-Feedback: drives draft, feedback and refinement with three few-shot prompts on one model.
• GPT-NER: Named Entity Recognition via Large Language Models: recasts entity extraction as generation, with demonstrations retrieved by entity similarity.
• Can Large Language Models Be an Alternative to Human Evaluations?: zero-shot rating prompts that reuse the human rater forms, with ordering checked against teachers.
• Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine: four few-shot clinical reasoning prompts against a chain-of-thought baseline.
• Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models: zero- and three-shot prompts on legal tasks, measured for hallucination.
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning: trains (IA)3 scaling vectors on T0 with one fixed recipe, and compares them with few-shot GPT-3 prompts on accuracy and cost.
• Visual Prompting via Image Inpainting: puts an example pair and a query in one grid, and inpaints the empty cell with a masked image model pretrained on figures.
• News Summarization and Evaluation in the Era of GPT-3: zero-shot sentence-count summaries from GPT-3, judged by people and by reference metrics that disagree with them.
• Decomposed Prompting: A Modular Approach for Solving Complex Tasks: a decomposer prompt writes a program of sub-questions, each answered by its own prompt, function or recursive call.
• Binding Language Models in Symbolic Languages: few-shot program exemplars in which the model is called for some sub-expressions, evaluated on table QA and fact checking.
• Large Language Models Struggle to Learn Long-Tail Knowledge: closed-book few-shot QA tracked against how often the answer co-occurs with the question in pre-training, with retrieval as the comparison.
• Teaching Large Language Models to Self-Debug: few-shot prompts that run, explain and fix code, for up to 10 turns, using test feedback.
• LayoutGPT: Compositional Visual Planning and Generation with Large Language Models: retrieved CSS-style layout demonstrations for a frozen LLM, with normalized coordinates.
• Large Language Models as General Pattern Machines: few-shot completion of symbolic sequences, including ones with random tokens, and reward-labelled trajectories in context.
• Med-HALT: Medical Domain Hallucination Test for Large Language Models: a hallucination benchmark whose prompts use zero to several worked examples, with a shot-count result.
• ExpeL: LLM Agents Are Experiential Learners: an agent that stores its trajectories, writes insights, and retrieves past successes into the prompt.
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants: a multilingual test set evaluated with 5-shot and zero-shot prompts, and translate-test.
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design: shows that few-shot prompt layout alone moves scores, and proposes a search to bound the spread.
• Noisy Channel Language Model Prompting for Few-Shot Text Classification: scores P(input | label) with a frozen GPT-2 Large, with demonstrations, ensembles or a learned prompt.
• mGPT: Few-Shot Learners Go Multilingual: a 61-language GPT-style model, evaluated with zero- and few-shot label-likelihood prompts.
• Promptagator: Few-shot Dense Retrieval From 8 Examples: a few examples prompt an LLM to write queries, which train a small retriever.
• UniverSeg: Universal Medical Image Segmentation: segments an unseen medical task from a support set of image-label pairs, with no training for the new task.
• Revisiting Relation Extraction in the era of Large Language Models: few-shot GPT-3 for triplet generation, and a Flan-T5 trained on GPT-3-written explanations.
• Benchmarking large language models for biomedical NLP applications and recommendations: zero-, static few- and nearest-neighbour few-shot prompts on 12 biomedical datasets, against fine-tuned baselines.
• Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions: an Auto-GPT-style loop, with a supervised expert's suggestion added to the prompt.
• Evaluating the Ripple Effects of Knowledge Editing in Language Models: in-context editing is compared with weight editors on follow-on facts that should change or stay.
• A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems: a survey that lists in-context learning among prompting methods, with no experiments.
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning: writes dense vision outputs as RGB images, and prompts one masked-image model with a solved pair.
• The Vector Grounding Problem: a position paper on whether LLM states can be about the world, with in-context learning as one speculative route; no experiments.
• Emergent Abilities in Large Language Models: A Survey: reviews whether sudden ability jumps, including in-context ones, are real or metric artifacts; no experiments of its own.
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context: long-context multimodal models, tested with needle recall and with a whole grammar book in the prompt.
• Fantastically Ordered Prompts and Where to Find Them: chooses demonstration orders without labels, by the label entropy of LM-generated probe inputs.
• PAL: Program-aided Language Models: few-shot prompts whose answers are Python programs, run by an interpreter.
• LLM+P: Empowering Large Language Models with Optimal Planning Proficiency: one example makes GPT-4 write a planning problem file, which a classical planner solves.
• ChatGPT: Jack of all trades, master of none: ChatGPT on 25 analytical NLP tasks, with personalized same-annotator prompts and prompt-pattern tests.
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models: a trust benchmark for GPT-4 and GPT-3.5, including adversarial and backdoored demonstrations.
• Pre-train, Prompt, and Predict: a survey that frames prompting as template filling, with the vocabulary used on this page.
• Learning To Retrieve Prompts for In-Context Learning: a dense retriever trained on LM-scored demonstrations, for semantic parsing.
• GLaM: a mixture-of-experts model evaluated with zero-, one- and few-shot prompts across 29 benchmarks.
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: few-shot exemplars with written reasoning, compared with standard prompts on arithmetic and symbolic tasks.
• OPT: Open Pre-trained Transformer Language Models: an open model family evaluated with GPT-3-style zero- and few-shot prompts.
• Reflexion: Language Agents with Verbal Reinforcement Learning: a frozen model writes verbal reflections on failed attempts, which go into the prompt of the next attempt.
• DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4: a zero-shot prompt asks ChatGPT or GPT-4 to redact identifiers in clinical notes.
• DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction: four prompted modules with class-specific demonstrations for text-to-SQL.
• A Survey on Multimodal Large Language Models: a survey of multimodal LLMs that lists multimodal in-context learning as an extended technique.
• Large Language Models as Optimizers: an LLM rewrites instructions from a meta-prompt of earlier scored candidates.
• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization: in-context examples retrieved by similarity, compared with QLoRA tuning, for clinical summaries.
• From Tables to Time: Extending TabPFN-v2 to Time Series Forecasting: a pretrained tabular regressor applied to forecasting, with no time-series training.
• Complexity-Based Prompting for Multi-Step Reasoning: few-shot cases with the most reasoning steps, and a vote over the most complex sampled chains.
• How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation: zero- to five-shot translation with three GPT models, with document windows and a hybrid fallback.
• Medprompt: nearest-neighbour examples with correct GPT-4 rationales, and a vote over option-shuffled calls.
• Generative Multimodal Models are In-Context Learners: a 37B model that takes interleaved image-text prompts, with few-shot results on VQA-style tasks.
• Automatic Chain of Thought Prompting in Large Language Models: model-written chain-of-thought demonstrations, one per cluster of test questions, with no gold labels.
• LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models: retrieved plan examples in the prompt, with the observed object list and re-planning when a subgoal fails.
• Using cognitive psychology to understand GPT-3: classic psychology tasks run trial by trial, with earlier choices and feedback in the prompt.
• Otter: A Multi-Modal Model with In-Context Instruction Tuning: training with in-context image-instruction examples, built on OpenFlamingo-9B.
• Role-Play with Large Language Models: a position paper that reads dialogue prompts as role-play, with no experiments.
• Self-Instruct: Aligning Language Models with Self-Generated Instructions: GPT-3 writes instruction data from eight in-context examples, filtered and used to fine-tune the same model.
• Real-World Humanoid Locomotion with Reinforcement Learning: a humanoid walking policy with a history window, trained with PPO and a teacher; the term is used loosely.
• Can Foundation Models Wrangle Your Data?: five data-cleaning tasks written as prompts for GPT-3 175B, with hand-picked or random worked examples.
• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs: GPT-3 probed on social commonsense and false-belief stories with same-type shots and likelihood scoring.
• Larger language models do in-context learning differently: flipped and meaningless labels show whether large models override priors or learn the mapping from the examples.
• TANGO: Training-free Embodied AI Agents for Open-world Tasks: an LLM writes a short program of navigation and perception primitives from 15 in-context examples, with no training of the task.
• Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following: retrieved solved trajectories in the prompt, with failed rollouts rewritten into examples for a second actor.
• Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding: codebook-style prompts with zero, one or five examples per code, scored against expert coders.
• Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues: zero-shot evaluation of six LLMs on 35 negotiation tasks, with chain-of-thought and few-shot checks.