Promptagator: Few-shot Dense Retrieval From 8 Examples

In plain words

Search systems that find documents by meaning usually need many example searches with the right answers before they work well. Most such systems are trained on a few big public question-answer collections, so they are good at those kinds of questions and weaker on everything else. A search for counter arguments, or for entities named in a sentence, is a different job with a different idea of "relevant".

Promptagator's idea is to skip training on those big collections. The team writes a short description of the search job and shows a large language model a handful of real examples: a document, and a query that found it. The model then looks at new documents from the same collection and writes queries that would plausibly find each one. Those invented query-document pairs become the training data for a small search model. A second filter drops invented queries that the trained search model itself cannot route back to their own document.

The result is a small, fast search model built for one task from only a few examples, plus an optional small reranker that re-sorts the top candidates. The authors report that this beats models trained on large public search data on the benchmark they used.

TL;DR

• Few-shot dense retrieval: each target task gets a short description and up to eight labelled query-document examples §2.2.
• A FLAN 137B model writes queries for documents in the target corpus. The output is the training data, not the model; no question-answering data is used §3.1 §7.
• A round-trip filter keeps only queries whose source document a retriever trained on the same synthetic data ranks in its top K §3.2.
• The retriever is a 110M T5-based dual encoder. The reranker is a 110M cross-encoder. Both are trained only on the generated data §3.3 §4.1.
• Author-reported: 47.8 nDCG@10 average over 11 BEIR datasets without reranking, 52.8 with it tab. 2. No official code was found.

Problem

Dense retrievers that encode queries and documents separately are cheap to serve, but they need relevance-labelled training data §1. The usual source is MS MARCO or Natural Questions, which fits factoid questions and transfers poorly to tasks with other search intents or query styles, such as argument retrieval or entity lookup §1 §2.1. Expressive models with token-level interaction transfer better but cost more at inference §1. The paper asks whether a few examples (eight or fewer) are enough for a task-specific retriever §2.2.

Method

Task definition. A retrieval task is a triple of a corpus, a query distribution and a search intent. The same query-document pair can be relevant under one intent and not under another §2.1. The few-shot setting uses the BEIR benchmark: for each dataset, 2 to 8 random in-domain relevant pairs are the only supervision §2.2.

Step 1, prompt-based query generation §3.1. The instruction prompt puts task-specific document and query descriptions around the k examples and the new document, and asks the LLM to continue with a query:

\big(e_{doc}(d_1), e_{query}(q_1), \ldots, e_{doc}(d_k), e_{query}(q_k), e_{doc}(d)\big) §3.1 (unnumbered in the paper)

The output is accepted only if it contains the query description; otherwise it is dropped §3.1. Each dataset has its own template (see tab. 4). The paper uses sampling with a temperature of 0.7 and generates 8 queries per document, for up to one million documents sampled from each corpus §3.1 §4.1. Long examples are truncated by hand, and the number of examples is reduced if the prompt exceeds the model's input limit §3.1.

Step 2, round-trip consistency filtering §3.2. The paper trains an initial dual encoder on the generated pairs, then keeps a pair (q, d) only if d is among the top K passages the same encoder returns for q. The authors report K = 1 as best and use it for all BEIR datasets; K was tuned with MS MARCO as the validation set §4.1.

Step 3, dual-encoder training §3.3 §4.1. The encoder is a T5 v1.1 base encoder (110M parameters) with mean pooling over the top layer and a 768-dimensional output, following GTR §4.1. It is first pretrained on C4 with independent cropping from Contriever: two random crops of one document form a positive pair, with cross-entropy over in-batch random negatives §3.3. It is then fine-tuned on the generated pairs with the same loss, filtered by step 2, and trained further on the filtered data §3.3.

Step 4, reranker training §3.3 §4.1. The reranker is a cross-attention encoder initialised from a T5-base encoder checkpoint (110M). It is trained with cross-entropy over 31 negatives sampled from the top 200 passages the dual encoder returns, so the training distribution matches inference, where the top 200 are reranked §3.3 §4.1.

Zero-shot variant §3.4. The same pipeline runs with one universal prompt that needs no examples: the document followed by the instruction to read the passage and generate a query §3.4.

Inference §4.1. Retrieve with the dual encoder; rerank the top 200 with the cross-encoder for the Promptagator++ results.

Design choices for the paper's comparisons tab. 1: the dual encoder uses no hard-negative mining and no distillation, and no token-level matching.

# Few-shot Promptagator pipeline  [§3.1-§3.3]  settings in [§4.1]
# Input:  corpus docs; k <= 8 examples (q_i, d_i); instruction LLM
# Output: dual encoder E and reranker R, both trained on generated data

# 1. Sample up to 1M documents from the target corpus  [§3.1] [§4.1]
sampled_docs = sample_up_to(corpus, 1_000_000)

# 2. Generate queries per document with the LLM  [§3.1]
#    8 queries per document, sampling temperature 0.7  [§4.1]
synthetic_pairs = []
for doc in sampled_docs:
    prompt = build_prompt(examples, doc)  # task template  [tab. 4]
    for query in llm_sample(prompt, count=8, temperature=0.7):
        # Drop outputs that lack the query description  [§3.1]
        if has_query_description(query):
            synthetic_pairs.append((query, doc))

# 3. Pretrain E: C4 with Contriever-style cropping  [§3.3] [§4.1]
#    T5 v1.1 base encoder, mean pool, 768-d  [§4.1]
E = init_t5_base_v1_1_encoder()
E = pretrain_random_crops(E, c4_corpus)  # in-batch softmax  [§3.3]

# 4. Fine-tune E on the generated pairs  [§3.3]
E = finetune(E, synthetic_pairs)  # batch and steps by corpus size  [§4.1]

# 5. Round-trip filter with E, keep the pair only if the doc is top K  [§3.2]
#    K = 1 for all BEIR datasets, tuned on MS MARCO  [§4.1]
filtered_pairs = []
for query, doc in synthetic_pairs:
    top_docs = retrieve_top_k(E, query, corpus, k=1)
    if doc in top_docs:
        filtered_pairs.append((query, doc))

# 6. Continue fine-tuning E on the filtered pairs  [§3.3]
E = finetune(E, filtered_pairs)

# 7. Train R on 31 negatives sampled from E's top 200  [§3.3] [§4.1]
#    Whether R sees filtered or unfiltered pairs is not stated  [§3.3]
R = train_cross_encoder(filtered_pairs, negatives_from=E, top=200)

# 8. Inference: retrieve with E, rerank the top 200 with R  [§4.1]
candidates = retrieve_top_k(E, query, corpus, k=200)
ranked = sort_by_score(R, query, candidates)

Hyperparameters

Results

Author-reported. Setup: 11 public BEIR datasets, nDCG@10, 110M dual encoder and 110M reranker, with baselines trained on MS MARCO, a 3B reranker (monoT5) and the zero-shot variant §4.1 §4.2. The table is copied from the paper tab. 2; the average column is the paper's.

* Climate-FEVER was run with the FEVER prompt; that value is not in the average tab. 2 (see Pitfalls).

Headline comparisons, as the paper states them §4.2:

• The zero-shot variant already compares favourably with retrievers trained on on the order of 100K MS MARCO examples; few-shot adds over 2 points of average nDCG@10 §4.2.
• The few-shot dual encoder is above GenQ, GPL, ColBERT v2 and SPLADE v2 on the average tab. 2 §4.2.
• The reranker adds about 5 points over the few-shot retriever (52.8 vs 47.8) and averages above monoT5 (3B) at 51.1. monoT5 (3B) is still higher on covid, nfc, hotpot, dbp, climate, scifact and fiqa tab. 2.
• The largest gains over baselines are on touché and arguana, both argument-retrieval tasks, with the authors arguing these differ most from factoid QA data §4.2.

Ablations §4.3 fig. 2:

• Round-trip filtering improves 8 of 11 datasets and adds about 2.5 points on average; NFCorpus and SciFact get worse, and the authors suspect overfitting on their small query sets §4.3 fig. 2(a).
• With MS MARCO as target and the same dual-encoder recipe, the 8-shot generator matches a large part of the supervised data; the paper reports it "catches up" with about 50K labelled examples §4.3 fig. 2(b).
• Compared with a T5 generator trained on MS MARCO (GenQ) and one trained on NQ, the few-shot LLM generator wins by large margins under the same recipe §4.3 fig. 2(c).
• Replacing the FLAN model with one trained without NQ and Quora costs about 1.5 points of average (48.5 to 47.0, with the FEVER prompt for Climate-FEVER) tab. 3.

Paper vs code

No official code.

The paper names no repository and has no project-page link; its TeX source contains no URL. It promises that prompts and few-shot examples "will be released" §1 §2.2, but no release was found. Search, with no result for an official repository: GitHub repository search for promptagator, promptagator org:google-research, few-shot dense retrieval promptagator 8 examples, and prompt-based query generation retriever. No third-party implementation was used.

Consequence: every training detail in the Hyperparameters table comes from the paper text alone, and the gaps listed there (learning rate, pretraining steps, epochs) must be filled by the implementer from an external source. Nothing in this card is checked against code.

Pitfalls

• Few-shot examples are part of the evaluation. On BEIR datasets that have only a test split, the examples come from the test set and must count as misses even when the model retrieves them §2.2. Forgetting this inflates scores.
• Climate-FEVER. BEIR marks "not enough info" annotations as relevant, which the authors say makes some pairs unrelated to the query. Prompts built from these examples hurt generation. The main table uses 16.8 for few-shot; the FEVER prompt gives 24.0 but is excluded from the average tab. 2 §4.3. Table 3's average (48.5 FLAN original) uses the 24.0 value, so it is not comparable with the main-table average of 47.8 tab. 3.
• Abstract vs table. The abstract says the few-shot retriever beats ColBERT v2 "by more than 1.2 nDCG on average". The table averages are 47.8 vs 46.2, a gap of 1.6; a gap of 1.2 matches SPLADE v2 (46.6). Use the table values tab. 2.
• MS MARCO is used to tune K. The filter threshold K is chosen on MS MARCO, which the paper otherwise excludes from evaluation to keep the few-shot premise intact §4.1. The effect on the results is not measured. Choose K on a held-out set in your own setting, or treat K = 1 as a tuned value.
• Filter retrieval scope is implicit. The text says to keep the pair when d is among the top-K passages returned by the retriever; it does not say whether that search covers the whole corpus or a candidate set. Assume the whole corpus and check the effect on small datasets.
• Filter hurts small datasets. The paper reports losses on NFCorpus and SciFact, the smallest query sets, and suspects overfitting; there is no rule for when to skip the filter §4.3.
• Which data trains the reranker is not stated. The text says the reranker is trained on "the same synthetic data" §3.3; it does not say whether that is before or after filtering. Pick one and report it.
• Batch and step groups are inconsistent. Dual-encoder batch size is 128 under 50k and 6k otherwise, but steps are 5k only above 500k and 1k otherwise, so corpora of 50k–500k get 1k steps. Read the group boundaries literally §4.1.
• Two-stage fine-tuning is under-specified. Training runs "for a set number of epochs", then filtering, then more fine-tuning on the filtered data §3.3. Whether the encoder is reset, the number of epochs, and the learning rates are not stated.
• Batch-size sensitivity on small corpora. The authors warn that with a large in-batch softmax, several relevant documents can land in one batch and hurt training, which is why small corpora use 128 §4.1. Keep this in mind when reusing the recipe on a new small corpus.
• Prompt sensitivity is open. The authors list prompt sensitivity and generated-data efficiency as unresolved §6. The few-shot prompt is built by hand per dataset tab. 4; no automatic procedure is given.
• Generator availability. The 137B FLAN checkpoint is described as "provided by the authors" §3.1. This card does not establish whether an equivalent public checkpoint exists; the paper's own ablation suggests the generator matters more than the training recipe §4.3.
• Cost. The paper's compute section reports 29.23M generated queries times two prompts, 58.46M in total, about 610M words §7. The generator is the expensive part; the distillation view is the authors' own framing §7.
• Zero-shot prompt is a single template. The zero-shot variant uses one universal prompt for all datasets §3.4. Few-shot prompts are task-specific.
• Sample size and seeds. Few-shot examples are sampled randomly §2.2; the seeds and the number of examples per dataset actually used are not stated.

Lineage

• Builds on: GTR, the dual-encoder recipe and hyperparameters §4.1 (bib key DBLP:journals/corr/abs-2112-07899); T5 v1.1 encoder §4.1; Contriever cropping for pretraining §3.3; FLAN as the query generator §3.1.
• Related query-generation work the paper compares against: GenQ and GPL, both trained on MS MARCO-generated queries §4.2 tab. 1.
• Related reranking work with LLMs: UPR (T0), InPars (GPT-3), monoT5 §3.5 tab. 1.
• Related consistency filtering in question generation, which the paper extends to retrieval: alberti et al. (2019) and PAQ §3.2.
• Superseded by: not known from content in this wiki.