Few-shot example selection

Overview

A few-shot prompt puts k cases before a test input, and the model completes the answer 2210.00720 §3.1. The cases are drawn from a pool, and which ones are drawn changes accuracy. The options differ in what they need: a hand-picked or random set needs nothing more, complexity-based selection needs annotated reasoning chains, retrieval needs a labelled pool that is searched for every query 2210.00720 tab. 4 2112.08633 §3.2, and cluster-based selection needs no labels at all but relies on the model's own zero-shot chains 2210.03493 §4.1.

Most of the pipeline is shared. Each candidate gets a score, the top k are kept or a fixed set is used, and the cases are placed in the prompt. Retrieval systems put the most similar case nearest the test input 2112.08633 §3.2. The papers below compare a few options each, on their own tasks and models, so the rankings do not transfer directly to a new task.

Variants

Random or fixed examples

Draw k cases from the pool, or reuse one set for every query. This is the baseline in most of the cited comparisons. On GSM8K with code-davinci-002 and greedy decoding, random chain-of-thought cases scored 60.4 against 61.0 for the handcrafted prompt 2210.00720 tab. 1. On MathQA with text-davinci-002 and a vote over sampled chains, random cases scored 48.5 against 43.8 for the handcrafted prompt 2210.00720 tab. 1. A random set is therefore not a floor. On MultiArith with Codex and greedy decoding, random cases scored 97.3 against 95.8 for the complex set 2210.00720 tab. 1.

A hand-picked set is a fixed set chosen by a person. In a data-cleaning study, the authors wrote the k = 10 examples after reviewing errors on a 10% validation split, with at most one hour of tuning per task 2205.09911 §3.3. Against k = 10 random labelled rows, the hand-picked set scored higher on three entity-matching datasets: Beer 100 against 91.1, iTunes-Amazon 98.2 against 86.6, and Walmart-Amazon 88.9 against 65.2, a mean gap of 14.7 F1 points 2205.09911 tab. 4 §4.3. Attribute selection stayed on in both arms. The random rows report three seeds, and the source card notes a code-level caveat about how those draws are made, so the spread should be read with care (card). Card: Can Foundation Models Wrangle Your Data?.

Complexity-based selection

Rank annotated training cases by the number of reasoning steps, counted as lines of the chain, and keep the top 8 2210.00720 §3.1 §4.1. Where chains are not annotated, the paper ranks by question length and annotates only the chosen cases by hand 2210.00720 §4.1. On the GSM8K validation split the complex set scored 58.5, against 52.5 for random, 52.0 for centroid, and 56.0 for retrieval 2210.00720 tab. 4. The order reverses on MathQA, where retrieval scored 69.5 and complexity 42.5 2210.00720 tab. 4. The score also depends on the step separator: with a period instead of a line break, the GSM8K complex prompt scored 54.5 rather than 58.5 2210.00720 tab. 7.

The paper pairs this with a vote over only the most complex sampled chains. That vote adds little on top of the complex prompt: 71.5 to 72.6 with text-davinci-002 and 82.6 to 82.9 with code-davinci-002 on GSM8K 2210.00720 tab. 1. See self-consistency for the vote itself.

Cluster, then one zero-shot chain per cluster

This variant needs no labelled cases. The test questions are embedded and clustered with k-means. In each cluster, questions are sorted by distance to the centre, and the first one whose zero-shot chain passes a length and answer filter becomes a demonstration 2210.03493 §4.1 §4.2. The chain is generated with the trigger "Let's think step by step". The default is k = 8, with smaller values for some datasets 2210.03493 §5.1. Card: Automatic Chain of Thought Prompting in Large Language Models.

On ten reasoning datasets with GPT-3 text-davinci-002, the paper reports that this method scored at or above the hand-written Manual-CoT prompt on every dataset, with StrategyQA tied 2210.03493 tab. 3. On MultiArith the figures are 92.0 against 91.7, and on GSM8K 47.9 against 46.9 tab. 3. A hand-written Few-Shot prompt without rationales was higher on CSQA (79.5 against 74.4) and StrategyQA (65.9 against 65.4) tab. 3. With Codex, the result is mixed: 62.8 against 59.4 on GSM8K, but 93.2 against 96.8 on MultiArith 2210.03493 tab. 4.

The filter is a set of heuristics. Its caps are at most 60 tokens for a question and at most 5 reasoning steps for a chain, and for arithmetic tasks the answer must appear in the rationale 2210.03493 §4.2 app. C.2. The demonstrations are built from the same questions the method is evaluated on, and the paper does not say whether a question can be its own demonstration 2210.03493 §4. Each candidate costs one zero-shot call, and the total number of extra calls is not reported 2210.03493 §4.2.

Nearest-neighbour retrieval per test input

Embed the pool, then for each test input take the nearest cases and place them in the prompt. The cited papers differ in what they embed.

GPT-NER retrieves by sentence embedding or by entity embedding. With davinci-003 on the full CoNLL2003 set, F1 was 73.44 for random demonstrations, 85.9 for sentence-level neighbours and 90.91 for entity-level neighbours [arxiv:2304.10428 tab. 2]. The entity variant embeds spans with a fine-tuned NER model that the paper does not name 2304.10428 §4.2.2, so it is not a pure prompt method. Its other finding goes the other way at small k: at k = 2 and k = 4, retrieved demonstrations did worse than random, because they often contained no entities 2304.10428 §6.2. Card: GPT-NER.

On LitCovid with GPT-4, macro-F1 was 0.5901 for a static one-shot prompt, 0.6500 with the single nearest case and 0.7055 with the five nearest [arxiv:2305.16326 p. 7]. Neighbours are ranked by cosine similarity over text-embedding-ada-002 vectors 2305.16326 p. 15. The same paper reports that dynamic one-shot is sometimes below static one-shot, and that five dynamic examples are not GPT-4's best setting on 8 of 12 datasets 2305.16326 p. 8. Card: Benchmarking large language models for biomedical NLP.

LLM-Planner applies the same idea to household planning. It retrieves the 9 pool examples whose instruction embeddings are nearest to the test instruction, from a pool of 100 ALFRED training pairs, using a frozen BERT-base encoder 2212.04088 §4.3 §5.3. In leave-one-out validation on that pool, high-level plan accuracy was 40.59 with retrieval and 17.48 with random examples 2212.04088 tab. 2. The same pool was used to tune K and the prompt layout, which the paper calls true few-shot 2212.04088 §3. Card: LLM-Planner.

Hindsight Planner keeps this retrieval step on ALFRED. A frozen BERT encodes the task description, and the nearest examples are copied into the prompt by Euclidean distance: 5 for its adapter that predicts hidden arguments, and 2 for each actor and for the critic 2412.19562 §4.2 §5.1. The starting pool holds 140 solved trajectories, 20 per task type 2412.19562 §5.1. The pool also grows during collection. Each rollout adds a rewritten actor sample and a critic sample, both written with the ground-truth rollout in view 2412.19562 alg. 3. Removing that relabelling lowers Valid Unseen SR from 19.00 to 16.32 2412.19562 tab. 4. The paper does not ablate retrieval alone, so this table does not show how much the nearest-neighbour step adds. Card: Hindsight Planner.

ExpeL applies the same idea to an agent. The retrieved items are past successful trajectories, not labelled input-output pairs. With gpt-3.5-turbo on ALFWorld, retrieval by reasoning similarity scored 48.5 and random in-context examples 42.5 2308.10144 tab. 3. The full ExpeL method, which also adds learned insights to the prompt, scored 59.0 2308.10144 tab. 3. Retrieval costs tokens: on HotpotQA, a trajectory used about 4,310 tokens with ExpeL and 1,320 with ReAct 2308.10144 tab. 6. Card: ExpeL.

Learned retriever

Train a small dense retriever on labels that a language model provides. For each training case, BM25 proposes candidates by their outputs. A scoring LM, GPT-Neo 2.7B, rates each candidate by how likely it makes the gold output. The top-rated candidates become positives and a low-rated one becomes a hard negative. Two BERT-base encoders are then trained on these labels 2112.08633 §3.1 2112.08633 §3.2. On BREAK dev, the learned retriever scored 31.9 against 26.0 for BM25 and 25.7 for CBR, and a BM25 oracle scored 32.3 [arxiv:2112.08633 tab. 2]. On the test split it scored 23.9, against 18.4 for CBR and 17.6 for BM25 2112.08633 tab. 3. The gains are on three semantic-parsing datasets, and transfer to other tasks is not tested 2112.08633 §4. Card: Learning To Retrieve Prompts for In-Context Learning.

When to use which

Implementation sketch

The selectors share one interface. Only the scoring step changes. The retrieval branch puts the most similar case last, nearest the test input.

# Pick k few-shot cases  [arxiv:2210.00720 §3.1] [arxiv:2112.08633 §3.2]
#                        [arxiv:2210.03493 §4.1]
# Input:  pool of cases (input, output, optional chain), test input, k, method
# Output: list of cases to place in the prompt

def steps(case):
    # One step per line of the chain  [arxiv:2210.00720 §3.1]
    return len(case.chain.split("\n"))


def select_examples(pool, test_input, k, method):
    if method == "random":
        # Draw k cases without replacement  [arxiv:2210.00720 §4.2]
        return random.sample(pool, k)
    if method == "complexity":
        # Most steps first; needs annotated chains  [arxiv:2210.00720 §3.1]
        ranked = sorted(pool, key=steps, reverse=True)
        return ranked[:k]
    if method == "cluster":
        # Cluster the questions; each cluster lists nearest-centre first
        # [arxiv:2210.03493 §4.1]
        demos = []
        for members in kmeans_clusters(pool, k):
            for case in members:
                # Zero-shot chain; keep the first that passes the filter
                # [arxiv:2210.03493 §4.2]
                chain = llm(case.input + "\nA: " + TRIGGER)
                if passes_filter(case.input, chain):
                    demos.append((case.input, chain))
                    break
        return demos
    if method == "retrieve":
        # Rank by similarity to the test input  [arxiv:2305.16326 p. 15]
        query = embed(test_input)
        scores = [cosine(query, embed(c.input)) for c in pool]
        top = argsort_desc(scores)[:k]
        # Most similar last, nearest the test input  [arxiv:2112.08633 §3.2]
        return [pool[i] for i in top][::-1]

Contradictions

• Retrieval with labelled cases against retrieval with model-written chains. With gold answers, retrieval wins on MathQA validation, 69.5 against 42.5 for complexity 2210.00720 tab. 4, and the learned retriever beats BM25 and CBR on BREAK test 2112.08633 tab. 3. With chains the model wrote itself, similarity retrieval scored 82.8 on MultiArith against 86.2 for random questions, and 46.9% of the zero-shot errors stayed wrong under retrieval against 25.8% under random 2210.03493 §3 §3.1. The settings differ in where the chains come from, and no cited paper tests the other's condition.
• Retrieval against random at small k. GPT-NER reports that retrieval beats random on CoNLL2003 with its main setting 2304.10428 tab. 2, but scores below random at k = 2 and k = 4 2304.10428 §6.2. The paper gives no other explanation than the missing entities in the retrieved examples.
• More retrieved examples. On LitCovid, five dynamic examples beat one, and one dynamic example beats the static one-shot prompt 2305.16326 p. 7. Across 12 datasets, the same paper finds dynamic one-shot sometimes below static one-shot, and five dynamic examples not the best setting on 8 of 12 2305.16326 p. 8.
• Hand-picked against random examples. On three entity-matching datasets, hand-picked examples beat random ones by 14.7 F1 points on average, with GPT-3 175B 2205.09911 §4.3 tab. 4. On GSM8K with code-davinci-002, random chain-of-thought cases scored 60.4 against 61.0 for the handcrafted prompt 2210.00720 tab. 1. The tasks, models and selection effort differ, so neither result settles whether hand-picking helps in general.
• Complexity against random. On GSM8K, complex prompts beat random ones under both greedy and vote decoding with both models tested 2210.00720 tab. 1. On MultiArith with greedy Codex, random prompts scored higher than the complex set, 97.3 against 95.8 2210.00720 tab. 1.

Papers

• Complexity-Based Prompting for Multi-Step Reasoning: picks the cases with the most reasoning steps and votes over the most complex chains.
• Automatic Chain of Thought Prompting in Large Language Models: clusters unlabelled test questions and keeps one filtered zero-shot chain per cluster as a demonstration.
• LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models: retrieves the nearest pool examples for household-task plans.
• GPT-NER: Named Entity Recognition via Large Language Models: retrieves demonstrations by sentence or entity embedding for NER.
• Benchmarking large language models for biomedical natural language processing applications and recommendations: retrieves nearest-neighbour examples by embedding, with static baselines.
• ExpeL: LLM Agents Are Experiential Learners: retrieves past successful agent trajectories by similarity.
• Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following: retrieves nearest pool examples for each planner module, from a pool that also holds relabelled failed rollouts.
• Learning To Retrieve Prompts for In-Context Learning: trains a dense retriever on labels from a scoring LM.
• Can Foundation Models Wrangle Your Data?: hand-picked demonstrations for data-cleaning prompts, compared with random ones.