Synthetic query generation

Overview

A dense retriever or text embedding model needs training pairs that tie a text to a related one, and the usual public sources fit some search intents better than others 2209.11755 §1 2209.11755 §2.1. Synthetic query generation replaces those pairs with text a language model writes. Promptagator asks the model for a query that would find each document in the target corpus 2209.11755 §3.1. E5-Mistral asks for whole task definitions with matching examples, across many task types and languages 2401.00368 §3.1. In both cases the generated data trains a smaller model, and the generator is not part of the model that is deployed 2209.11755 §7 2401.00368 §3.2.

The two papers filter differently. Promptagator uses a round-trip filter, which drops generated queries that the trained retriever cannot route back to their source document, and a generator prompt that describes the target search job 2209.11755 §3.1 2209.11755 §3.2. E5-Mistral keeps only outputs that parse as JSON and removes exact duplicates. Its diversity comes from brainstormed task definitions and sampled placeholders 2401.00368 §3.1. This page has no head-to-head number for the two filtering choices.

Variants

Few-shot prompting with round-trip filtering

The generator is a 137B FLAN checkpoint. It sees a task description and up to eight labelled query-document examples before each new document. The paper does not mention a public release of that checkpoint 2209.11755 §3.1 2209.11755 §7. It samples several queries per document, keeps those that contain the query description, trains a 110M dual encoder on them, and then keeps only pairs whose document is the top-1 result for the query 2209.11755 §3.1 2209.11755 §3.2 2209.11755 §4.1. Reported average nDCG@10 over 11 BEIR datasets: 47.8, without reranking 2209.11755 tab. 2. Card: Promptagator: Few-shot Dense Retrieval From 8 Examples.

Zero-shot prompting

The same pipeline runs with one universal prompt for every dataset. The prompt holds the document and an instruction to write a query for it, and no examples 2209.11755 §3.4. Reported average nDCG@10: 45.5 2209.11755 tab. 2. The paper reports that this already compares favourably with retrievers trained on about 100K MS MARCO examples 2209.11755 §4.2.

Adding a reranker

A cross-encoder with a 110M T5-base backbone is trained on the same generated data. Its negatives are sampled from the top 200 passages of the dual encoder, which matches the depth at which it reranks at inference 2209.11755 §3.3 2209.11755 §4.1. Reported average nDCG@10 with reranking: 52.8 for the few-shot retriever, against 47.8 without it 2209.11755 tab. 2.

Task brainstorming for a decoder embedding model

The generator writes whole tasks rather than one query per document. For each subgroup of asymmetric tasks, it first brainstorms about 20 one-sentence task definitions, then writes one JSON example per definition. Placeholders such as query length, difficulty and document length are sampled on each call, and so is the language, weighted toward high-resource languages 2401.00368 §3.1 2401.00368 app. C 2401.00368 tab. 8. Symmetric tasks such as semantic similarity skip the brainstorming step 2401.00368 §3.1. The generators are GPT-4 and GPT-3.5-Turbo at temperature 1.0, which yields about 500k examples in 93 languages 2401.00368 §4.1 2401.00368 app. C.

The training model is Mistral-7B with LoRA rank 16, one epoch, and an InfoNCE loss at temperature 0.02. Each query carries its task definition as a prefix, and documents carry none 2401.00368 §3.2 2401.00368 §4.2 2401.00368 app. A. Removing the instruction costs 4.2 points of MTEB average 2401.00368 §5.3 2401.00368 tab. 5. Reported MTEB average over 56 English datasets: 63.1 from synthetic data alone, 64.5 with MS MARCO added, and 66.6 with the full labelled mix 2401.00368 tab. 1. Card: Improving Text Embeddings with Large Language Models.

When to use which

Implementation sketch

The common core is the same for every variant: generate examples, train an encoder on them, and optionally filter. The variants differ in the prompt (few-shot examples, a universal instruction, or a brainstormed task definition) and in whether the reranker stage is added. Settings are the papers', not tuned values. The Promptagator settings are eight queries per document at temperature 0.7, up to one million documents per corpus, and filter depth K = 1 2209.11755 §3.1 2209.11755 §4.1.

# Synthetic query generation, few-shot variant  [arxiv:2209.11755 §3]
# Input:  corpus docs, k <= 8 labelled (query, doc) examples, LLM generator
# Output: dual encoder E and optional reranker R

# 1. Sample up to 1M documents from the target corpus  [§4.1]
docs = sample_up_to(corpus, 1_000_000)

# 2. Prompt the generator once per document, keep 8 samples  [§3.1] [§4.1]
pairs = []
for d in docs:
    prompt = task_description + examples + d
    for q in llm_sample(prompt, n=8, temperature=0.7):
        # Drop outputs without the query description  [§3.1]
        if q_is_valid(q):
            pairs.append((q, d))

# 3. Initialise E from a pretrained encoder, then train on pairs  [§3.3]
E = train_dual_encoder(pairs)

# 4. Round-trip filter: keep (q, d) only if d is E's top-1 for q  [§3.2] [§4.1]
kept = [(q, d) for (q, d) in pairs if top1(E, q, corpus) == d]

# 5. Continue training E on the kept pairs  [§3.3]
E = train_dual_encoder(E, kept)

# 6. Optional: train R on negatives from E's top 200  [§3.3] [§4.1]
#    Whether R sees filtered or unfiltered pairs is not stated  [§3.3]
R = train_cross_encoder(kept, negatives_from=E, depth=200)

The task-brainstorming variant replaces steps 1 and 2 with a generator that writes tasks first, and it drops the round-trip filter. Its filter is JSON parsing and exact-duplicate removal 2401.00368 §3.1:

# Task-brainstorm generation, one asymmetric subgroup  [arxiv:2401.00368 §3.1]
# Output: labelled examples, unique by exact string

# 1. Ask for about 20 one-sentence task definitions  [app. C]
tasks = llm(brainstorm_prompt(subgroup))

examples = []
for task in tasks:
    # 2. Sample placeholders and a language for this call  [§3.1] [app. C]
    values = sample_placeholders(subgroup)
    lang = sample_language()
    # 3. One JSON example per task at temperature 1.0  [§4.1] [app. C]
    raw = llm(example_prompt(subgroup, task, values, lang), temp=1.0)
    ex = parse_json(raw)
    # Discard anything that is not valid JSON  [§3.1]
    if ex is not None:
        examples.append(ex)

# 4. Remove exact string duplicates  [§3.1]
examples = unique_by_exact_string(examples)

Not stated in the Promptagator paper: learning rates, pretraining steps, the number of epochs before filtering, and the random seeds for the few-shot examples 2209.11755 §2.2 2209.11755 §3.3. For the task-brainstorming variant, not stated: the optimizer, LoRA alpha and dropout, and how one hard negative is chosen from the top 100 mined candidates 2401.00368 app. A. No official code for the Promptagator pipeline was found 2209.11755 §4.1. The E5 repository holds evaluation scripts only, so the training details of the task-brainstorming variant are paper-only 2401.00368 app. A.

Contradictions

Inside Promptagator, the abstract's margin over ColBERT v2 (more than 1.2 nDCG@10) does not match the table, where the averages are 47.8 and 46.2 2209.11755 tab. 2. The table values are used on this page. The Climate-FEVER score differs by prompt (16.8 with the main prompt, 24.0 with the FEVER prompt, which is left out of the average) 2209.11755 tab. 2 2209.11755 §4.3.

Across the two papers, the averages are not on the same benchmark. Promptagator's 47.8, 45.5 and 52.8 are averages over 11 BEIR datasets 2209.11755 tab. 2. E5-Mistral's BEIR average of 56.9 is over 15 retrieval datasets 2401.00368 tab. 3, and its MTEB figures of 63.1 and 66.6 are over 56 English datasets 2401.00368 tab. 1. None of these numbers can be ranked against one another.

Papers

• Promptagator: Few-shot Dense Retrieval From 8 Examples: prompted LLM query generation from up to eight examples, round-trip filtering, and a 110M dual encoder and reranker trained on the generated pairs.
• Improving Text Embeddings with Large Language Models: an LLM brainstorms tasks and writes JSON examples in 93 languages, and a Mistral-7B embedding model is trained on them with a contrastive loss.