---
kind: concept
slug: synthetic-query-generation
title: Synthetic query generation
summary: >-
  Trains a retriever or embedding model on text an LLM writes, instead of labelled search data. Promptagator writes one query per document and keeps round-trip-consistent pairs for a small dual encoder. E5-Mistral has an LLM invent task-specific examples in 93 languages for a 7B model. Results are author-reported and cover different benchmarks.
tags: [retrieval, data-generation, few-shot, query-generation, text-embedding, synthetic-data]
papers: ["2209.11755", "2401.00368"]
related: [dense-retrieval, text-embedding, contrastive-learning, few-shot-learning, in-context-learning, data-augmentation, knowledge-distillation, zero-shot-learning]
status: draft
updated_at: 2026-10-31
---

## Overview

A dense retriever or text embedding model needs training pairs that tie a text
to a related one, and the usual public sources fit some search intents better
than others [arxiv:2209.11755 §1] [arxiv:2209.11755 §2.1]. Synthetic query
generation replaces those pairs with text a language model writes. Promptagator
asks the model for a query that would find each document in the target corpus
[arxiv:2209.11755 §3.1]. E5-Mistral asks for whole task definitions with matching
examples, across many task types and languages [arxiv:2401.00368 §3.1]. In both
cases the generated data trains a smaller model, and the generator is not part
of the model that is deployed [arxiv:2209.11755 §7] [arxiv:2401.00368 §3.2].

The two papers filter differently. Promptagator uses a round-trip filter, which
drops generated queries that the trained retriever cannot route back to their
source document, and a generator prompt that describes the target search job
[arxiv:2209.11755 §3.1] [arxiv:2209.11755 §3.2]. E5-Mistral keeps only outputs
that parse as JSON and removes exact duplicates. Its diversity comes from
brainstormed task definitions and sampled placeholders [arxiv:2401.00368 §3.1].
This page has no head-to-head number for the two filtering choices.

## Variants

### Few-shot prompting with round-trip filtering

The generator is a 137B FLAN checkpoint. It sees a task description and up to
eight labelled query-document examples before each new document. The paper does
not mention a public release of that checkpoint [arxiv:2209.11755 §3.1]
[arxiv:2209.11755 §7]. It samples several queries per document, keeps those
that contain the query description, trains a 110M dual encoder on them, and then
keeps only pairs whose document is the top-1 result for the query
[arxiv:2209.11755 §3.1] [arxiv:2209.11755 §3.2] [arxiv:2209.11755 §4.1].
Reported average nDCG@10 over 11 BEIR datasets: 47.8, without reranking
[arxiv:2209.11755 tab. 2]. Card: [Promptagator: Few-shot Dense Retrieval From 8 Examples](/papers/2209.11755).

### Zero-shot prompting

The same pipeline runs with one universal prompt for every dataset. The prompt
holds the document and an instruction to write a query for it, and no examples
[arxiv:2209.11755 §3.4]. Reported average nDCG@10: 45.5 [arxiv:2209.11755 tab. 2].
The paper reports that this already compares favourably with retrievers trained
on about 100K MS MARCO examples [arxiv:2209.11755 §4.2].

### Adding a reranker

A cross-encoder with a 110M T5-base backbone is trained on the same generated data.
Its negatives are sampled from the top 200 passages of the dual encoder, which
matches the depth at which it reranks at inference [arxiv:2209.11755 §3.3]
[arxiv:2209.11755 §4.1]. Reported average nDCG@10 with reranking: 52.8 for the
few-shot retriever, against 47.8 without it [arxiv:2209.11755 tab. 2].

### Task brainstorming for a decoder embedding model

The generator writes whole tasks rather than one query per document. For each
subgroup of asymmetric tasks, it first brainstorms about 20 one-sentence task
definitions, then writes one JSON example per definition. Placeholders such as
query length, difficulty and document length are sampled on each call, and so is
the language, weighted toward high-resource languages [arxiv:2401.00368 §3.1]
[arxiv:2401.00368 app. C] [arxiv:2401.00368 tab. 8]. Symmetric tasks such as
semantic similarity skip the brainstorming step [arxiv:2401.00368 §3.1].
The generators are GPT-4 and GPT-3.5-Turbo at temperature 1.0, which yields
about 500k examples in 93 languages [arxiv:2401.00368 §4.1] [arxiv:2401.00368 app. C].

The training model is Mistral-7B with LoRA rank 16, one epoch, and an InfoNCE
loss at temperature 0.02. Each query carries its task definition as a prefix,
and documents carry none [arxiv:2401.00368 §3.2] [arxiv:2401.00368 §4.2]
[arxiv:2401.00368 app. A]. Removing the instruction costs 4.2 points of MTEB
average [arxiv:2401.00368 §5.3] [arxiv:2401.00368 tab. 5]. Reported MTEB average
over 56 English datasets: 63.1 from synthetic data alone, 64.5 with MS MARCO
added, and 66.6 with the full labelled mix [arxiv:2401.00368 tab. 1]. Card:
[Improving Text Embeddings with Large Language Models](/papers/2401.00368).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Up to eight labelled pairs from the target task | Few-shot variant | Highest reported retriever average, 47.8 vs 45.5 zero-shot, both over 11 BEIR datasets [arxiv:2209.11755 tab. 2] |
| No labelled pairs at all | Zero-shot variant | Still reported at 45.5, above GenQ at 40.1 [arxiv:2209.11755 tab. 2] |
| Latency budget allows a second stage over the top 200 | Add the reranker | Reported 52.8 average, above monoT5 (3B) at 51.1 on average, though monoT5 (3B) scores higher on seven of eleven sets [arxiv:2209.11755 tab. 2] |
| Corpus with few documents or queries | Run the filter, then check it on held-out data | The paper reports losses on NFCorpus and SciFact and suspects overfitting [arxiv:2209.11755 §4.3] |
| No access to a strong instruction-tuned generator | Expect a lower ceiling | The paper's ablation finds the generator matters more than the training recipe [arxiv:2209.11755 §4.3] |
| Search intent unlike factoid QA, such as argument retrieval | Few-shot variant | The largest gains over baselines are on touché and arguana [arxiv:2209.11755 §4.2] |
| One embedding model for many task types and languages, with no labelled data | Task brainstorming | Synthetic-only MTEB average 63.1, against 62.3 for E5-large-v2 [arxiv:2401.00368 tab. 1]. Needs a 7B decoder: about 18 hours on 32 V100 GPUs for training [arxiv:2401.00368 app. A] |
| Low-resource languages are the target | Expect a gap | Even the full-data model scores 68.4 on Swahili against 74.9 for mE5-large [arxiv:2401.00368 tab. 2]. About 1k synthetic examples per low-resource language on average [arxiv:2401.00368 §4.1] |
| Generator outputs are to be reused commercially | Check the provider terms first | The paper does not discuss the terms under which the GPT outputs were used [arxiv:2401.00368 §4.1] |

## Implementation sketch

The common core is the same for every variant: generate examples, train an
encoder on them, and optionally filter. The variants differ in the prompt
(few-shot examples, a universal instruction, or a brainstormed task definition)
and in whether the reranker stage is added. Settings are the papers', not tuned
values. The Promptagator settings are eight queries per document at temperature
0.7, up to one million documents per corpus, and filter depth K = 1
[arxiv:2209.11755 §3.1] [arxiv:2209.11755 §4.1].

```pseudocode
# Synthetic query generation, few-shot variant  [arxiv:2209.11755 §3]
# Input:  corpus docs, k <= 8 labelled (query, doc) examples, LLM generator
# Output: dual encoder E and optional reranker R

# 1. Sample up to 1M documents from the target corpus  [§4.1]
docs = sample_up_to(corpus, 1_000_000)

# 2. Prompt the generator once per document, keep 8 samples  [§3.1] [§4.1]
pairs = []
for d in docs:
    prompt = task_description + examples + d
    for q in llm_sample(prompt, n=8, temperature=0.7):
        # Drop outputs without the query description  [§3.1]
        if q_is_valid(q):
            pairs.append((q, d))

# 3. Initialise E from a pretrained encoder, then train on pairs  [§3.3]
E = train_dual_encoder(pairs)

# 4. Round-trip filter: keep (q, d) only if d is E's top-1 for q  [§3.2] [§4.1]
kept = [(q, d) for (q, d) in pairs if top1(E, q, corpus) == d]

# 5. Continue training E on the kept pairs  [§3.3]
E = train_dual_encoder(E, kept)

# 6. Optional: train R on negatives from E's top 200  [§3.3] [§4.1]
#    Whether R sees filtered or unfiltered pairs is not stated  [§3.3]
R = train_cross_encoder(kept, negatives_from=E, depth=200)
```

The task-brainstorming variant replaces steps 1 and 2 with a generator that
writes tasks first, and it drops the round-trip filter. Its filter is JSON
parsing and exact-duplicate removal [arxiv:2401.00368 §3.1]:

```pseudocode
# Task-brainstorm generation, one asymmetric subgroup  [arxiv:2401.00368 §3.1]
# Output: labelled examples, unique by exact string

# 1. Ask for about 20 one-sentence task definitions  [app. C]
tasks = llm(brainstorm_prompt(subgroup))

examples = []
for task in tasks:
    # 2. Sample placeholders and a language for this call  [§3.1] [app. C]
    values = sample_placeholders(subgroup)
    lang = sample_language()
    # 3. One JSON example per task at temperature 1.0  [§4.1] [app. C]
    raw = llm(example_prompt(subgroup, task, values, lang), temp=1.0)
    ex = parse_json(raw)
    # Discard anything that is not valid JSON  [§3.1]
    if ex is not None:
        examples.append(ex)

# 4. Remove exact string duplicates  [§3.1]
examples = unique_by_exact_string(examples)
```

Not stated in the Promptagator paper: learning rates, pretraining steps, the
number of epochs before filtering, and the random seeds for the few-shot examples
[arxiv:2209.11755 §2.2] [arxiv:2209.11755 §3.3]. For the task-brainstorming
variant, not stated: the optimizer, LoRA alpha and dropout, and how one hard
negative is chosen from the top 100 mined candidates [arxiv:2401.00368 app. A].
No official code for the Promptagator pipeline was found [arxiv:2209.11755 §4.1].
The E5 repository holds evaluation scripts only, so the training details of the
task-brainstorming variant are paper-only [arxiv:2401.00368 app. A].

## Contradictions

Inside Promptagator, the abstract's margin over ColBERT v2 (more than 1.2 nDCG@10)
does not match the table, where the averages are 47.8 and 46.2
[arxiv:2209.11755 tab. 2]. The table values are used on this page. The Climate-FEVER
score differs by prompt (16.8 with the main prompt, 24.0 with the FEVER prompt,
which is left out of the average) [arxiv:2209.11755 tab. 2] [arxiv:2209.11755 §4.3].

Across the two papers, the averages are not on the same benchmark. Promptagator's
47.8, 45.5 and 52.8 are averages over 11 BEIR datasets [arxiv:2209.11755 tab. 2].
E5-Mistral's BEIR average of 56.9 is over 15 retrieval datasets
[arxiv:2401.00368 tab. 3], and its MTEB figures of 63.1 and 66.6 are over 56
English datasets [arxiv:2401.00368 tab. 1]. None of these numbers can be ranked
against one another.

## Papers

- [Promptagator: Few-shot Dense Retrieval From 8 Examples](/papers/2209.11755): prompted LLM query generation from up to eight examples, round-trip filtering, and a 110M dual encoder and reranker trained on the generated pairs.
- [Improving Text Embeddings with Large Language Models](/papers/2401.00368): an LLM brainstorms tasks and writes JSON examples in 93 languages, and a Mistral-7B embedding model is trained on them with a contrastive loss.