---
kind: concept
slug: few-shot-example-selection
title: Few-shot example selection
summary: >-
  Choosing which cases go into a few-shot prompt: a random or fixed set, the cases with the most reasoning steps, nearest-neighbour retrieval, a learned retriever, or cluster-based picks from unlabelled questions. Retrieval often scores higher but needs a labelled pool and a search per query. Gains are author-measured, on few tasks and models.
tags: [few-shot, in-context-learning, prompting, retrieval, example-selection, reasoning, clustering]
papers: ["2112.08633", "2205.09911", "2210.00720", "2210.03493", "2212.04088", "2304.10428", "2305.16326", "2308.10144", "2412.19562"]
related: [in-context-learning, few-shot-learning, chain-of-thought-prompting, self-consistency, dense-retrieval, retrieval-augmented-generation, named-entity-recognition, prompt-engineering, llm-agent, llm-planning]
status: draft
updated_at: 2026-11-01
---

## Overview

A few-shot prompt puts k cases before a test input, and the model completes the
answer [arxiv:2210.00720 §3.1]. The cases are drawn from a pool, and which ones
are drawn changes accuracy. The options differ in what they need: a hand-picked
or random set needs nothing more, complexity-based selection needs annotated
reasoning chains, retrieval needs a labelled pool that is searched for every
query [arxiv:2210.00720 tab. 4] [arxiv:2112.08633 §3.2], and cluster-based
selection needs no labels at all but relies on the model's own zero-shot chains
[arxiv:2210.03493 §4.1].

Most of the pipeline is shared. Each candidate gets a score, the top k are kept
or a fixed set is used, and the cases are placed in the prompt. Retrieval
systems put the most similar case nearest the test input [arxiv:2112.08633 §3.2].
The papers below compare a few options each, on their own tasks and models, so
the rankings do not transfer directly to a new task.

## Variants

### Random or fixed examples

Draw k cases from the pool, or reuse one set for every query. This is the
baseline in most of the cited comparisons. On GSM8K with code-davinci-002 and
greedy decoding, random chain-of-thought cases scored 60.4 against 61.0 for the
handcrafted prompt [arxiv:2210.00720 tab. 1]. On MathQA with text-davinci-002
and a vote over sampled chains, random cases scored 48.5 against 43.8 for the
handcrafted prompt [arxiv:2210.00720 tab. 1]. A random set is therefore not a
floor. On MultiArith with Codex and greedy decoding, random cases scored 97.3
against 95.8 for the complex set [arxiv:2210.00720 tab. 1].

A hand-picked set is a fixed set chosen by a person. In a data-cleaning study,
the authors wrote the k = 10 examples after reviewing errors on a 10% validation
split, with at most one hour of tuning per task [arxiv:2205.09911 §3.3]. Against
k = 10 random labelled rows, the hand-picked set scored higher on three
entity-matching datasets: Beer 100 against 91.1, iTunes-Amazon 98.2 against 86.6,
and Walmart-Amazon 88.9 against 65.2, a mean gap of 14.7 F1 points
[arxiv:2205.09911 tab. 4] [§4.3]. Attribute selection stayed on in both arms. The
random rows report three seeds, and the source card notes a code-level caveat
about how those draws are made, so the spread should be read with care
([card](/papers/2205.09911)). Card:
[Can Foundation Models Wrangle Your Data?](/papers/2205.09911).

### Complexity-based selection

Rank annotated training cases by the number of reasoning steps, counted as lines
of the chain, and keep the top 8 [arxiv:2210.00720 §3.1] [§4.1]. Where chains
are not annotated, the paper ranks by question length and annotates only the
chosen cases by hand [arxiv:2210.00720 §4.1]. On the GSM8K validation split the
complex set scored 58.5, against 52.5 for random, 52.0 for centroid, and 56.0
for retrieval [arxiv:2210.00720 tab. 4]. The order reverses on MathQA, where
retrieval scored 69.5 and complexity 42.5 [arxiv:2210.00720 tab. 4]. The score
also depends on the step separator: with a period instead of a line break, the
GSM8K complex prompt scored 54.5 rather than 58.5 [arxiv:2210.00720 tab. 7].

The paper pairs this with a vote over only the most complex sampled chains. That
vote adds little on top of the complex prompt: 71.5 to 72.6 with text-davinci-002
and 82.6 to 82.9 with code-davinci-002 on GSM8K [arxiv:2210.00720 tab. 1]. See
[self-consistency](/concepts/self-consistency) for the vote itself.

### Cluster, then one zero-shot chain per cluster

This variant needs no labelled cases. The test questions are embedded and
clustered with k-means. In each cluster, questions are sorted by distance to the
centre, and the first one whose zero-shot chain passes a length and answer
filter becomes a demonstration [arxiv:2210.03493 §4.1] [§4.2]. The chain is
generated with the trigger "Let's think step by step". The default is k = 8,
with smaller values for some datasets [arxiv:2210.03493 §5.1]. Card:
[Automatic Chain of Thought Prompting in Large Language Models](/papers/2210.03493).

On ten reasoning datasets with GPT-3 text-davinci-002, the paper reports that
this method scored at or above the hand-written Manual-CoT prompt on every
dataset, with StrategyQA tied [arxiv:2210.03493 tab. 3]. On MultiArith the
figures are 92.0 against 91.7, and on GSM8K 47.9 against 46.9 [tab. 3]. A
hand-written Few-Shot prompt without rationales was higher on CSQA (79.5 against
74.4) and StrategyQA (65.9 against 65.4) [tab. 3]. With Codex, the result is
mixed: 62.8 against 59.4 on GSM8K, but 93.2 against 96.8 on MultiArith
[arxiv:2210.03493 tab. 4].

The filter is a set of heuristics. Its caps are at most 60 tokens for a question
and at most 5 reasoning steps for a chain, and for arithmetic tasks the answer
must appear in the rationale [arxiv:2210.03493 §4.2] [app. C.2]. The
demonstrations are built from the same questions the method is evaluated on, and
the paper does not say whether a question can be its own demonstration
[arxiv:2210.03493 §4]. Each candidate costs one zero-shot call, and the total
number of extra calls is not reported [arxiv:2210.03493 §4.2].

### Nearest-neighbour retrieval per test input

Embed the pool, then for each test input take the nearest cases and place them
in the prompt. The cited papers differ in what they embed.

GPT-NER retrieves by sentence embedding or by entity embedding. With davinci-003
on the full CoNLL2003 set, F1 was 73.44 for random demonstrations, 85.9 for
sentence-level neighbours and 90.91 for entity-level neighbours [arxiv:2304.10428
tab. 2]. The entity variant embeds spans with a fine-tuned NER model that the
paper does not name [arxiv:2304.10428 §4.2.2], so it is not a pure prompt method.
Its other finding goes the other way at small k: at k = 2 and k = 4, retrieved
demonstrations did worse than random, because they often contained no entities
[arxiv:2304.10428 §6.2]. Card: [GPT-NER](/papers/2304.10428).

On LitCovid with GPT-4, macro-F1 was 0.5901 for a static one-shot prompt, 0.6500
with the single nearest case and 0.7055 with the five nearest [arxiv:2305.16326
p. 7]. Neighbours are ranked by cosine similarity over text-embedding-ada-002
vectors [arxiv:2305.16326 p. 15]. The same paper reports that dynamic one-shot is
sometimes below static one-shot, and that five dynamic examples are not GPT-4's
best setting on 8 of 12 datasets [arxiv:2305.16326 p. 8]. Card:
[Benchmarking large language models for biomedical NLP](/papers/2305.16326).

LLM-Planner applies the same idea to household planning. It retrieves the 9
pool examples whose instruction embeddings are nearest to the test instruction,
from a pool of 100 ALFRED training pairs, using a frozen BERT-base encoder
[arxiv:2212.04088 §4.3] [§5.3]. In leave-one-out validation on that pool,
high-level plan accuracy was 40.59 with retrieval and 17.48 with random
examples [arxiv:2212.04088 tab. 2]. The same pool was used to tune K and the
prompt layout, which the paper calls true few-shot [arxiv:2212.04088 §3]. Card:
[LLM-Planner](/papers/2212.04088).

Hindsight Planner keeps this retrieval step on ALFRED. A frozen BERT encodes
the task description, and the nearest examples are copied into the prompt by
Euclidean distance: 5 for its adapter that predicts hidden arguments, and 2 for
each actor and for the critic [arxiv:2412.19562 §4.2] [§5.1]. The starting pool
holds 140 solved trajectories, 20 per task type [arxiv:2412.19562 §5.1]. The
pool also grows during collection. Each rollout adds a rewritten actor sample
and a critic sample, both written with the ground-truth rollout in view
[arxiv:2412.19562 alg. 3]. Removing that relabelling lowers Valid Unseen SR from
19.00 to 16.32 [arxiv:2412.19562 tab. 4]. The paper does not ablate retrieval
alone, so this table does not show how much the nearest-neighbour step adds.
Card: [Hindsight Planner](/papers/2412.19562).

ExpeL applies the same idea to an agent. The retrieved items are past successful
trajectories, not labelled input-output pairs. With gpt-3.5-turbo on ALFWorld,
retrieval by reasoning similarity scored 48.5 and random in-context examples
42.5 [arxiv:2308.10144 tab. 3]. The full ExpeL method, which also adds learned
insights to the prompt, scored 59.0 [arxiv:2308.10144 tab. 3]. Retrieval costs
tokens: on HotpotQA, a trajectory used about 4,310 tokens with ExpeL and 1,320
with ReAct [arxiv:2308.10144 tab. 6]. Card: [ExpeL](/papers/2308.10144).

### Learned retriever

Train a small dense retriever on labels that a language model provides. For each
training case, BM25 proposes candidates by their outputs. A scoring LM, GPT-Neo
2.7B, rates each candidate by how likely it makes the gold output. The top-rated
candidates become positives and a low-rated one becomes a hard negative. Two
BERT-base encoders are then trained on these labels [arxiv:2112.08633 §3.1]
[arxiv:2112.08633 §3.2]. On BREAK dev, the learned retriever scored 31.9 against
26.0 for BM25 and 25.7 for CBR, and a BM25 oracle scored 32.3 [arxiv:2112.08633
tab. 2]. On the test split it scored 23.9, against 18.4 for CBR and 17.6 for BM25
[arxiv:2112.08633 tab. 3]. The gains are on three semantic-parsing datasets, and
transfer to other tasks is not tested [arxiv:2112.08633 §4]. Card:
[Learning To Retrieve Prompts for In-Context Learning](/papers/2112.08633).

## When to use which

| Situation | Use | Why |
|---|---|---|
| No labelled pool, or none worth searching | Random or fixed set | Random matched the handcrafted prompt on GSM8K with greedy Codex, 60.4 against 61.0 [arxiv:2210.00720 tab. 1] |
| Data-cleaning or matching task, a validation split, and a person who can review errors for up to an hour per task | Hand-picked fixed set, checked against random | Manual beat random by 14.7 F1 points on average over three entity-matching sets [arxiv:2205.09911 §4.3] [tab. 4]. Tuning is human work [arxiv:2205.09911 §3.3] |
| No labels, a reasoning task, and no hand-written cases wanted | Cluster, then one zero-shot chain per cluster | GSM8K 47.9 against 46.9 for Manual-CoT on GPT-3 [arxiv:2210.03493 tab. 3]. Codex MultiArith is lower, 93.2 against 96.8 [arxiv:2210.03493 tab. 4] |
| Reasoning task, no annotated chains, budget for about 8 hand-annotated cases | Complexity by question length | On GSM8K, the longer-question prompt scored 52.5 against 49.0 for the shortest [arxiv:2210.00720 tab. 6] |
| Reasoning task with annotated chains, and a 175B-class model | Complexity by step count | GSM8K validation 58.5 against 52.5 for random [arxiv:2210.00720 tab. 4]. Gains are mostly from the exemplars, not the vote [arxiv:2210.00720 tab. 1] |
| Model well below 175B | Keep the cheap baseline and measure | Complex prompts changed accuracy by −0.5 to +1.5 points against handcrafted on three smaller models [arxiv:2210.00720 tab. 5] |
| Labelled set of at least 10,000 cases, and a search per query is affordable | Nearest-neighbour retrieval | On MathQA validation, retrieval scored 69.5 against 42.5 for complexity [arxiv:2210.00720 tab. 4] |
| Small labelled pool (about 100) for an agent with a fixed action set | Nearest-neighbour retrieval, checked against random | Leave-one-out accuracy 40.59 against 17.48 for random examples [arxiv:2212.04088 tab. 2]. Measured on the pool that tuned it |
| Agent with a few hundred solved trajectories that must recover from wrong steps | Nearest-neighbour retrieval over the pool, with relabelled failures added to it | Valid Unseen SR 19.00 with the relabelled samples, 16.32 without [arxiv:2412.19562 tab. 4]. One benchmark and one LLM tested, and retrieval not ablated alone |
| Extraction task where similar inputs share entities | Entity-level retrieval, checked against random | 90.91 against 73.44 F1 on CoNLL2003 [arxiv:2304.10428 tab. 2]. The embedding needs a fine-tuned NER model [arxiv:2304.10428 §4.2.2] |
| Small k (2 or 4) with sparse matches | Random, or retrieval with a check | Retrieval scored below random at k = 2 and 4 in GPT-NER [arxiv:2304.10428 §6.2] |
| Many queries on one task, a scoring LM, and a training budget for a small encoder | Learned retriever | On BREAK test, 23.9 against 18.4 for the best baseline [arxiv:2112.08633 tab. 3]. Only three datasets tested [arxiv:2112.08633 §4] |
| Agent with past runs and no labels | Retrieve past successes by similarity | 48.5 against 42.5 for random examples on ALFWorld [arxiv:2308.10144 tab. 3]. Adds prompt tokens [arxiv:2308.10144 tab. 6] |

## Implementation sketch

The selectors share one interface. Only the scoring step changes. The retrieval
branch puts the most similar case last, nearest the test input.

```pseudocode
# Pick k few-shot cases  [arxiv:2210.00720 §3.1] [arxiv:2112.08633 §3.2]
#                        [arxiv:2210.03493 §4.1]
# Input:  pool of cases (input, output, optional chain), test input, k, method
# Output: list of cases to place in the prompt

def steps(case):
    # One step per line of the chain  [arxiv:2210.00720 §3.1]
    return len(case.chain.split("\n"))


def select_examples(pool, test_input, k, method):
    if method == "random":
        # Draw k cases without replacement  [arxiv:2210.00720 §4.2]
        return random.sample(pool, k)
    if method == "complexity":
        # Most steps first; needs annotated chains  [arxiv:2210.00720 §3.1]
        ranked = sorted(pool, key=steps, reverse=True)
        return ranked[:k]
    if method == "cluster":
        # Cluster the questions; each cluster lists nearest-centre first
        # [arxiv:2210.03493 §4.1]
        demos = []
        for members in kmeans_clusters(pool, k):
            for case in members:
                # Zero-shot chain; keep the first that passes the filter
                # [arxiv:2210.03493 §4.2]
                chain = llm(case.input + "\nA: " + TRIGGER)
                if passes_filter(case.input, chain):
                    demos.append((case.input, chain))
                    break
        return demos
    if method == "retrieve":
        # Rank by similarity to the test input  [arxiv:2305.16326 p. 15]
        query = embed(test_input)
        scores = [cosine(query, embed(c.input)) for c in pool]
        top = argsort_desc(scores)[:k]
        # Most similar last, nearest the test input  [arxiv:2112.08633 §3.2]
        return [pool[i] for i in top][::-1]
```

## Contradictions

- **Retrieval with labelled cases against retrieval with model-written chains.**
  With gold answers, retrieval wins on MathQA validation, 69.5 against 42.5
  for complexity [arxiv:2210.00720 tab. 4], and the learned retriever beats BM25
  and CBR on BREAK test [arxiv:2112.08633 tab. 3]. With chains the model wrote
  itself, similarity retrieval scored 82.8 on MultiArith against 86.2 for random
  questions, and 46.9% of the zero-shot errors stayed wrong under retrieval
  against 25.8% under random [arxiv:2210.03493 §3] [§3.1]. The settings differ in
  where the chains come from, and no cited paper tests the other's condition.
- **Retrieval against random at small k.** GPT-NER reports that retrieval beats
  random on CoNLL2003 with its main setting [arxiv:2304.10428 tab. 2], but scores
  below random at k = 2 and k = 4 [arxiv:2304.10428 §6.2]. The paper gives no
  other explanation than the missing entities in the retrieved examples.
- **More retrieved examples.** On LitCovid, five dynamic examples beat one, and
  one dynamic example beats the static one-shot prompt [arxiv:2305.16326 p. 7].
  Across 12 datasets, the same paper finds dynamic one-shot sometimes below static
  one-shot, and five dynamic examples not the best setting on 8 of 12
  [arxiv:2305.16326 p. 8].
- **Hand-picked against random examples.** On three entity-matching datasets,
  hand-picked examples beat random ones by 14.7 F1 points on average, with GPT-3
  175B [arxiv:2205.09911 §4.3] [tab. 4]. On GSM8K with code-davinci-002, random
  chain-of-thought cases scored 60.4 against 61.0 for the handcrafted prompt
  [arxiv:2210.00720 tab. 1]. The tasks, models and selection effort differ, so
  neither result settles whether hand-picking helps in general.
- **Complexity against random.** On GSM8K, complex prompts beat random ones under
  both greedy and vote decoding with both models tested [arxiv:2210.00720 tab. 1].
  On MultiArith with greedy Codex, random prompts scored higher than the complex
  set, 97.3 against 95.8 [arxiv:2210.00720 tab. 1].

## Papers

- [Complexity-Based Prompting for Multi-Step Reasoning](/papers/2210.00720): picks
  the cases with the most reasoning steps and votes over the most complex chains.
- [Automatic Chain of Thought Prompting in Large Language Models](/papers/2210.03493):
  clusters unlabelled test questions and keeps one filtered zero-shot chain per
  cluster as a demonstration.
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](/papers/2212.04088):
  retrieves the nearest pool examples for household-task plans.
- [GPT-NER: Named Entity Recognition via Large Language Models](/papers/2304.10428):
  retrieves demonstrations by sentence or entity embedding for NER.
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](/papers/2305.16326):
  retrieves nearest-neighbour examples by embedding, with static baselines.
- [ExpeL: LLM Agents Are Experiential Learners](/papers/2308.10144): retrieves
  past successful agent trajectories by similarity.
- [Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following](/papers/2412.19562):
  retrieves nearest pool examples for each planner module, from a pool that
  also holds relabelled failed rollouts.
- [Learning To Retrieve Prompts for In-Context Learning](/papers/2112.08633):
  trains a dense retriever on labels from a scoring LM.
- [Can Foundation Models Wrangle Your Data?](/papers/2205.09911): hand-picked
  demonstrations for data-cleaning prompts, compared with random ones.