Dense retrieval

Overview

A dense retriever encodes each query and each document separately into a fixed-size vector and returns the documents whose vectors score highest against the query. Because document vectors can be computed offline, serving is cheap compared with models that compare query and document token by token 2209.11755 §1. The cost moves to training: the encoder needs relevance-labelled pairs, and the usual public sources (MS MARCO, Natural Questions) suit factoid questions better than other search intents such as argument retrieval or entity lookup 2209.11755 §1 2209.11755 §2.1.

The shared mechanics: one encoder per side (or one shared encoder), a similarity score, and a contrastive loss that pulls each query toward its positive document and away from the other documents in the batch 2312.15503 §3.2 2312.15503 eq. 5.

Dense retrieval also sits inside larger systems. A retrieval-augmented generator can read retrieved text in four ways: paste it into the prompt, mix the model's output distribution with the passages' distributions, feed passage features into the model's internals, or store each passage as adapter weights 2407.13193 §4. The survey says that only prompt pasting and logits mixing work with black-box APIs 2407.13193 §4.5. It runs no experiments, so its comparison is qualitative 2407.13193 tab. 2.

Variants

Labelled bi-encoder (the reference point)

Train the encoder directly on labelled query-passage pairs such as MS MARCO. The cards use this family as baselines: the GenQ and GPL rows in 2209.11755 tab. 2 are trained on MS MARCO-based data, and the Llama2Vec paper's main comparison, RepLLaMA, is fine-tuned on MS MARCO 2312.15503 §4.2. Use this when in-domain labelled pairs exist. The Promptagator ablation reports that its generated data catches up with about 50K labelled examples 2209.11755 §4.3, so the threshold where labels stop mattering is an empirical question for your corpus.

Labels do not always help. The RAG-end2end paper reports that supervised DPR fine-tuning with BM25 hard negatives did not improve news or conversation retrieval, and attributes this to answers sitting in passages longer than the 100 words DPR was trained on 2210.02627 §4.5. Check your passage lengths against the training setup before assuming labels will lift retrieval.

Few-shot synthetic queries (Promptagator)

Give an instruction LLM a short task description and up to eight labelled query-document examples. Then ask it to write queries for documents sampled from the target corpus. The generated pairs train the encoder; a round-trip filter drops pairs whose document the trained encoder does not rank in its top K 2209.11755 §3.1 2209.11755 §3.2. The paper generates 8 queries per document at temperature 0.7, for up to one million documents per corpus 2209.11755 §4.1. The encoder is a 110M T5 v1.1 base dual encoder, and a 110M cross-encoder can rerank the top 200 2209.11755 §4.1. Card: Promptagator: Few-shot Dense Retrieval From 8 Examples.

Reported results, author-reported, 11 BEIR datasets: the few-shot encoder averages 47.8 nDCG@10 and the zero-shot version 45.5 2209.11755 tab. 2. With the reranker the average is 52.8 2209.11755 tab. 2. The paper reports that round-trip filtering helps 8 of 11 datasets and hurts NFCorpus and SciFact, which it attributes to overfitting on their small query sets 2209.11755 §4.3.

Costs and limits the cards state:

• The generator is a 137B FLAN model that the authors describe as "provided by the authors"; the card finds no public release 2209.11755 §3.1. The ablation suggests the generator matters more than the training recipe 2209.11755 §4.3.
• Generation is the expensive part: 29.23M generated queries across two prompts, about 58.46M in total 2209.11755 §7.
• The filter threshold K = 1 was tuned on MS MARCO, which the paper otherwise holds out 2209.11755 §4.1.
• On BEIR datasets with only a test split, the few-shot examples come from the test set and must count as misses in the evaluation 2209.11755 §2.2.

Unsupervised LLM adaptation, then labelled fine-tuning (Llama2Vec)

Start from a decoder-only LLM (LLaMA-2-7B) and adapt it on Wikipedia text without search labels. The input is the passage followed by two prompt blocks: one asks the model to restate the passage (EBAE), the other to predict the next passage (EBAR). Each block's embedding is read from a special token, and a custom attention mask stops the two blocks from seeing each other, so one pass serves both 2312.15503 §3.2 2312.15503 fig. 2. The adaptation run uses 10,000 steps at batch size 256 and learning rate 1e-5 2312.15503 §4.1. Search fine-tuning then uses contrastive learning with LoRA on MS MARCO 2312.15503 §4.1. Card: Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval.

Reported results, author-reported: MS MARCO passage MRR@10 43.1 against 41.2 for RepLLaMA 2312.15503 tab. 1; zero-shot BEIR average NDCG@10 56.4 against 53.9 for RepLLaMA over 14 datasets 2312.15503 tab. 3.

Points to check before you implement:

• "Unsupervised" covers only the adaptation step. The search fine-tuning still uses labelled MS MARCO data 2312.15503 §4.1.
• The model is 7B parameters, against 110M for the Promptagator encoder 2312.15503 tab. 1 2209.11755 §4.1. The cards give no serving latency or memory figures for either model.
• The released code reads the embedding differently from the paper. Use the card's "Paper vs code" section before reproducing the numbers 2312.15503 §3.2.

Synthetic multi-task data and a 7B decoder (E5-Mistral)

Ask a strong proprietary LLM to invent embedding tasks, not just queries for one corpus. For each asymmetric subgroup the model first brainstorms about 20 one-sentence task definitions, then writes one JSON example per task, in 93 languages 2401.00368 §3.1 2401.00368 app. C. Invalid JSON and exact duplicates are the only filters 2401.00368 §3.1. The paper reports about 500k examples, generated with GPT-4 and GPT-3.5-Turbo at temperature 1.0 2401.00368 §4.1.

The encoder is Mistral-7B with LoRA rank 16 on all linear layers, trained for one epoch with InfoNCE at temperature 0.02 2401.00368 §4.2 2401.00368 eq. 3. Queries carry a task instruction and documents do not. The embedding is the last-layer vector at the [EOS] position 2401.00368 §3.2. The full-data model adds 13 public labelled datasets, about 1.8M examples in total 2401.00368 §4.2 2401.00368 app. A. Card: Improving Text Embeddings with Large Language Models.

Reported results, author-reported. On MTEB's 56 English datasets, the average is 66.6 with full data, against 64.2 for BGE-large-en-v1.5, and 63.1 with synthetic data alone 2401.00368 tab. 1. On BEIR's 15 retrieval datasets the average nDCG@10 is 56.9 2401.00368 tab. 3. On MIRACL dev across 16 languages it scores 62.9, against 66.5 for mE5-large 2401.00368 tab. 6. On a personalized passkey task at 32k tokens, a RoPE base of 10^5 gives over 90% top-1 accuracy, while naively enlarging the window is worse 2401.00368 §5.2 2401.00368 fig. 5.

Points to check before you implement:

• The paper says the implementation uses the HuggingFace PEFT library and releases evaluation scripts 2401.00368 app. A. The training recipe has no public code; see the card's "Paper vs code" section.
• The optimizer, LoRA alpha and dropout, the loss reduction, and how one hard negative is picked from the top 100 are not stated 2401.00368 app. A.
• Not every generated example is a query-document pair. Long-short examples carry a misleading label and symmetric ones carry graded scores, and the paper does not say how these become training pairs 2401.00368 tab. 9 2401.00368 tab. 12.
• The text and Table 8 disagree on the query length buckets 2401.00368 §3.1 2401.00368 tab. 8. Follow the table if you reproduce the templates.
• Low-resource languages are weaker. On MIRACL, Swahili scores 68.4 against 74.9 for mE5-large 2401.00368 tab. 2, and the paper generated only about 1k synthetic examples per low-resource language on average 2401.00368 §4.1.
• Several training sets are also MTEB evaluation sets, and the paper reports no result with them removed 2401.00368 app. B.
• The paper does not discuss the terms under which the generator's outputs were used 2401.00368 §4.1. Check the provider terms before you reproduce the data pipeline.

Conversation pairs and a small quantized encoder (ConveRT)

This variant targets reply selection in dialogue, not ad hoc search. A shared six-block Transformer encodes the conversation context and the candidate reply, and each side has its own output network. The model is pretrained on Reddit reply pairs with an in-batch softmax 1911.03688 §2.1 1911.03688 §3. Embeddings are stored at 8 bits and the other network weights at 16 bits; the paper's table gives 116 MB before quantization and 59 MB after 1911.03688 tab. 2. Card: ConveRT: Efficient and Accurate Conversational Representations from Transformers.

Reported results, author-reported, R100@1 (the correct reply ranked first among 100 candidates). On Reddit without fine-tuning, the single-context model scores 68.2 and the version that reads up to ten earlier messages scores 71.8 1911.03688 tab. 3 1911.03688 §2.1. On DSTC7-Ubuntu the single-context model scores 38.2 and the multi-context model 71.2 1911.03688 tab. 5.

Points to check before you implement:

• The released model is no longer available. The paper says fine-tuned models are released 1911.03688 §3, but the authors' repository states that they were removed from public availability. Plan to pretrain on your own conversation pairs; the card's "Paper vs code" section has the details.
• The single-context model drops from 68.2 on Reddit to 38.2 on DSTC7, and the paper does not discuss the gap 1911.03688 tab. 3 1911.03688 tab. 5. Do not assume the single-context model transfers.
• The paper gives two hardware descriptions that disagree 1911.03688 §3 1911.03688 §4. Its reported pretraining cost is about 18 hours and about $85 on Google Cloud 1911.03688 §3; treat both as unverified.

Multilingual, long-document, three-head encoder (M3-Embedding)

One XLM-RoBERTa-based encoder, extended to 8,192 tokens, produces three scores from one pass: a dense vector from the [CLS] state, learned sparse token weights, and per-token vectors for late interaction 2402.03216 §3.2. A weighted sum of the three scores re-ranks candidates 2402.03216 eq. 1. Training runs RetroMAE pre-training, unsupervised contrastive learning on about 1.2 billion pairs, then fine-tuning in which the integrated score teaches all three heads 2402.03216 §3.3 2402.03216 tab. 8. Card: M3-Embedding.

Reported results, author-reported, on the paper's own splits. MIRACL dev (18 languages, nDCG@10): dense alone 69.2, all three heads 71.5 2402.03216 tab. 1. MKQA (25 languages, Recall@100): dense 75.1, all three 75.5 2402.03216 tab. 2. MLDR test (documents up to 8,192 tokens, nDCG@10): dense 52.5, sparse alone 62.2, all three 65.0 2402.03216 tab. 3.

Points to check before you implement:

• Which head wins depends on the task. On MLDR the sparse head is about 10 points above dense 2402.03216 §4.3. On MKQA it is weak, because a query and its passage share few words across languages 2402.03216 §4.2.
• Hybrid weights are per benchmark: (1, 0.3, 1) on MIRACL and (0.15, 0.5, 0.35) on MLDR 2402.03216 §4.1 2402.03216 §4.3. Treat them as starting points.
• Multi-vector scoring keeps one vector per token. The paper uses it only to re-rank the dense top 200 on MIRACL, and does not discuss storage 2402.03216 §4.1.
• Without long-document training, the dense head scores 41.2 on MLDR. An inference-only trick that inserts [CLS] tokens every 256 tokens raises it to 45.0 2402.03216 tab. 3 2402.03216 app. B.2.
• The paper gives no fine-tuning learning rate or maximum length 2402.03216 app. B.1. The InfoNCE temperature is also not stated 2402.03216 eq. 2.

Reasoning text before embedding (Think-Then-Embed)

This variant targets multimodal search, where the query is an image or video plus a text instruction. A multimodal LLM first writes a short reasoning text about the query or target, an embedding-centric reasoning trace (ECR). The embedder then encodes the input together with that text 2510.05014 §4. The authors argue that some instructions need multi-step reasoning that one forward pass cannot do while compressing the input into a vector 2510.05014 §1. The paper does not measure this claim directly 2510.05014 §1. Card: Think Then Embed: Generative Context Improves Multimodal Embedding.

Three forms share the ECR step:

• Teacher reasoner (TTE_t). A 72B multimodal model writes the trace for every training item, and a Qwen2-VL 2B or 7B embedder is trained on the inputs with those traces 2510.05014 §4.2 2510.05014 §5.1. For contrastive training the teacher is asked to rephrase the answer, and only about half the items keep the ground truth 2510.05014 app. A.2 2510.05014 tab. 7.
• Finetuned small reasoner (TTE_s). First, the embedder's own backbone writes the traces zero-shot. Second, a separate reasoner starts from the same backbone and is finetuned on teacher traces with token-level negative log-likelihood 2510.05014 §4.3. The reasoner is therefore the same size as the backbone, not a smaller model.
• Unified model (TTE_u). The backbone is finetuned to reason on the trace set, then frozen. Only an embedding head, a multi-layer self-attention module initialised from the backbone's last layers, is trained on its hidden states 2510.05014 §4.4 2510.05014 §7.3. Training the contrastive and generation objectives jointly in one backbone lowered scores in the authors' runs 2510.05014 app. B 2510.05014 tab. 8.

Reported results, author-reported. On MMEB-V2 (78 tasks, the "All" average), TTE_t-7B scores 71.5 and TTE_s-7B scores 68.6. The VLM2Vec-V2-7B row, which the authors reproduced themselves, scores 61.2 2510.05014 tab. 2 2510.05014 app. D. On MMEB-V1 (36 tasks, "Overall"), TTE_t-7B scores 78.5 and TTE_t-2B scores 74.5; the strongest rows in the same table are QQMM at 72.5 (7B) and B3 at 68.1 (2B) 2510.05014 tab. 1. Zero-shot reasoning from the 7B backbone reaches 68.3 on MMEB-V1, and finetuning the reasoner adds 5.0 points 2510.05014 tab. 3.

Points to check before you implement:

• The teacher's name differs across the paper. The method section gives Qwen2.5-72B, the introduction gives Qwen2.5 VL 72B, and a prompt figure in the appendix names another model 2510.05014 §4.2 2510.05014 §1 2510.05014 app. C. The card cannot say which model wrote the released traces.
• The teacher runs once per indexed item, but a user query is new at search time, so a reasoner call sits on the query path 2510.05014 §4.2. The paper gives no latency or memory numbers.
• The headline gain for the small finetuned reasoner is not tied to a table row. The closest comparison is 68.6 against the reproduced 61.2 on MMEB-V2 2510.05014 tab. 2. The paper's "surpassing proprietary models" claim rests on TTE_t-7B at 71.5 against 71.3 for seed-1.6-embedding, a margin of 0.2 points; leaderboard figures are as of submission 2510.05014 tab. 2 2510.05014 §6.1.
• The 7B teacher scores 78.1 in the CoT ablation and 78.5 in the main table 2510.05014 tab. 4 2510.05014 tab. 1. The paper does not explain the difference.
• Where the reasoning text joins the embedder input is not stated. The pooling position, any tags around the trace and the input template are not given 2510.05014 §3 2510.05014 §4.1. The teacher prompts appear only in figures 2510.05014 fig. 7 2510.05014 fig. 8 2510.05014 fig. 9.
• The contrastive settings are given: global batch 8192, learning rate 2e-4 for the backbone and 5e-4 for the head, temperature 0.02, LoRA rank 16 and alpha 64 2510.05014 §5.1. The optimizer, schedule, image resolution, maximum length and reasoner decoding settings are not stated 2510.05014 §5.1.

Retrieval during pretraining (RETRO)

Here the lookup is part of the language model. The input is cut into 64-token chunks. For each chunk, the model retrieves neighbours by nearest-neighbour search over frozen BERT keys, from a database of up to about 2 trillion tokens. The neighbours enter through chunked cross-attention in some decoder layers 2112.04426 §2.2 2112.04426 §2.3 2112.04426 §2.4. Neighbours for training data are computed in advance, and neighbours from the same document are removed so no future text leaks into the past 2112.04426 §2.3. Card: Improving language models by retrieving from trillions of tokens.

Reported results, author-reported. A 7.5B RETRO model beats the 7B baseline on C4 and Wikitext103 in bits per byte and perplexity 2112.04426 app. F.1. It beats Jurassic-1 and Gopher on a majority of Pile test subsets 2112.04426 §4.1. On Natural Questions with DPR passages it scores 45.5 exact match, against 30.4 for the closed-book 7B baseline 2112.04426 tab. 5.

RETRO-fitting adds the retrieval layers to a pretrained model. The base model is frozen and only the new cross-attention and neighbour-encoder weights train, using about 6 million sequences 2112.04426 §4.2 2112.04426 app. C.3. With retrieval switched off, the output is unchanged 2112.04426 §4.2.

Points to check before you implement:

• The index is the cost. The paper gives about 215 GB for a Wikipedia index and about 93 TB for MassiveText 2112.04426 §4.1.
• Part of the gain comes from leaked training text. Evaluate on chunks below the overlap threshold to compare models fairly 2112.04426 §2.6 2112.04426 §4.4.
• The paper does not describe how documents are cut into the neighbour and continuation pieces, nor the BERT size or the index settings 2112.04426 §2.3. Each changes retrieval quality.
• No official code was found 2112.04426 §2.5. Implement from the paper and the card's Paper vs code section.

Example retrieval for in-context learning (EPR)

This variant retrieves few-shot examples for a prompt, not documents for an answer. Two BERT-base encoders score training examples against an input. One reads the input, the other reads example input-output pairs, and similarity is an inner product 2112.08633 §3.2. Labels come from a scoring LM: for each training example, BM25 candidates are ranked by how likely the LM gives the gold output with that candidate in the prompt. The top 5 become positives and the bottom 5 hard negatives 2112.08633 §3.1 2112.08633 §4.2. Card: Learning To Retrieve Prompts for In-Context Learning.

Reported results, author-reported, development sets, GPT-Neo as scorer and inference LM. Against the best baseline, EPR scores 31.9 against 26.0 (BM25) on BREAK, 64.2 against 57.0 (CBR) on MTOP, and 54.3 against 51.4 (CBR) on SMCalFlow 2112.08633 tab. 2. With other inference LMs (GPT-3, Codex, GPT-J) on 1,000-example subsets, EPR is highest in every row of the proxy table 2112.08633 tab. 5.

Points to check before you implement:

• Labelling is the cost. Each training example needs one scoring pass per candidate. At L = 50 candidates on BREAK's 44K training examples, that is about 2.2 million scoring passes by our arithmetic 2112.08633 §4.1 2112.08633 §4.2.
• Training candidates use the gold output, which exists only at training time, while test-time queries use the input alone 2112.08633 §3.1 2112.08633 §3.2.
• Results are single runs with no variance reported 2112.08633 §4.3. Treat gaps of a few points as uncertain.
• The paper says EPR may inherit biases from the large LMs that supply its signals 2112.08633 app. A. The repository's licence is non-commercial in part; see the card's Paper vs code section before copying any code.

Joint retriever-generator training (RAG-end2end)

Here the retriever is not frozen. The question encoder and the passage encoder both receive gradients from the generator's loss, and the generator is a BART model 2210.02627 §3.1 2210.02627 §3.3. Because the passage vectors change, the index must be rebuilt. A background job re-encodes the passages on separate GPUs and re-indexes them, and the training loop swaps in each new index when it is ready 2210.02627 §3.3 2210.02627 fig. 1. The average refresh interval is about 750 updates on the paper's 250,000-passage base 2210.02627 §3.3. An auxiliary task asks the generator to rebuild a domain sentence from retrieved passages, marked with a control token 2210.02627 §3.4 2210.02627 eq. 3. Card: Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering.

Reported results, author-reported, exact match on three domain QA sets. The end-to-end model without the auxiliary task beats original RAG trained on the same QA pairs: 8.08 against 2.95 on COVID-19, 8.39 against 7.26 on News and 24.25 against 12.09 on Conversations 2210.02627 tab. 1. With the auxiliary task it reaches 8.32, 14.08 and 25.95, against 3.66, 8.62 and 14.21 for original RAG with the task 2210.02627 tab. 1. Each configuration is a single run 2210.02627 §4.2.

Points to check before you implement:

• Refresh cost grows with the passage store. The paper reports several GPU hours to re-encode and several CPU hours to re-index on knowledge bases of millions of passages 2210.02627 §3.3. Its own domain bases hold 85,000 to 250,000 passages 2210.02627 §4.1.
• Re-indexing must wait until every passage vector is saved, or it fails 2210.02627 §3.3. Training gradients can be stale because the index lags the encoder; the paper gives no measure of how much lag is tolerable.
• The loss weight between the QA and reconstruction signals, the order in which the two kinds of example are mixed, and the number of retrieved passages k are not stated 2210.02627 §3.1 2210.02627 §3.4.
• The COVID-19 training and validation pairs are synthetic, and only its test set is human-labelled 2210.02627 §4.1.
• The footnote's code link points to an example folder that the repository's main branch no longer contains 2210.02627 §1. See the card's Paper vs code section before reproducing the setup.

Corpus pollution

A dense retriever indexes whatever text the corpus holds, so LLM-written passages added to the corpus reach the retriever like any other passage. Pan et al. test this with GPT-3.5, which writes one fake passage per question and adds it to the clean corpus, in four settings: Benign, Gen, Imi and Hack 2305.13661 §3.2 2305.13661 §4.3. Card: On the Risk of Misinformation Pollution with Large Language Models. The paper's threat assumes generated text reaches the corpus the QA system reads 2305.13661 §1 2305.13661 §4.3.

Author-reported, on NQ-1500 and CovidNews:

• Under the Gen and Hack settings, the dense retriever (DPR, with FiD reader) lost less exact match than BM25. Gen cost DPR about 14% on NQ-1500 and about 34% on CovidNews, against about 20% and 54% for BM25; Hack cost DPR 39% and 50%, against 65% and 69% for BM25 2305.13661 tab. 2.
• The dense retriever is not immune. Under Hack, a fake passage reaches the top 10 for 48.9% of NQ-1500 questions with DPR, against 96.8% with BM25. Under Imi the two are close, 67.1% against 72.5% 2305.13661 tab. 7.
• A detector trained on in-domain fakes separates them well, with AUROC 99.7 (Gen), 91.4 (Imi) and 99.8 (Hack). Trained on out-of-domain GPT-3 Wikipedia completions, it falls to 64.8, 50.7 and 52.6 2305.13661 tab. 3.
• The paper's cost estimate is $0.01 to $0.04 per 200-word fake document, using text-davinci-003 prices of the time 2305.13661 §6. Recheck current prices before quoting it.

The three defenses act on the detector or the reader, after retrieval. The paper reports them as only partly effective 2305.13661 §5 2305.13661 tab. 4.

When to use which

Implementation sketch

The serving core is the same in every variant. The training loss is shown once, in the form used for the Llama2Vec fine-tuning step.

# Serving: encode the corpus once, then one query vector per request
# [arxiv:2209.11755 §1] [arxiv:2312.15503 §3.2]
# Input:  corpus docs, a query; output: top-k doc ids
index = [encode_doc(d) for d in corpus]       # offline, stored as a matrix
# A query is encoded on its own and scored by inner product
# [arxiv:2312.15503 eq. 5]
# E5-Mistral prepends a task instruction to queries only [arxiv:2401.00368 eq. 1]
q = encode_query(query)                        # (d,)
scores = index @ q                             # (num_docs,)
top_k = argsort_desc(scores)[:k]
# Optional: a cross-encoder reranks the top 200 candidates
# [arxiv:2209.11755 §4.1]

The M3 variant adds sparse and multi-vector scores to the dense score and combines them as a weighted sum 2402.03216 eq. 1. RETRO replaces the per-query lookup with a per-chunk lookup inside the model 2112.04426 §2.4.

# Contrastive training with in-batch negatives  [arxiv:2312.15503 eq. 5]
# Input:  B (query, positive) pairs; temperature
# Output: scalar loss

def contrastive_loss(q_vecs, d_vecs, temperature):
    # Temperature is not stated in the paper  [arxiv:2312.15503 §3.2]
    scores = q_vecs @ d_vecs.T / temperature   # (B, B)
    # Row i's positive is column i; the other columns are negatives
    return cross_entropy(scores, targets=arange(B)).mean()

Where the variants differ is in producing the pairs. The Promptagator pipeline generates them with an LLM and filters them with the trained encoder 2209.11755 §3.1 2209.11755 §3.2. The Llama2Vec pipeline first adapts the LLM on Wikipedia and then uses labelled pairs 2312.15503 §4.1. The ConveRT pipeline pretrains on reply pairs from Reddit threads 1911.03688 §3. The M3 pipeline trains on mined and labelled pairs in three stages 2402.03216 §3.3. EPR labels its pairs with a scoring LM 2112.08633 §3.1. The E5-Mistral pipeline asks an LLM for task definitions and matching examples, then trains a 7B LLM with the same InfoNCE-style loss 2401.00368 §3.1 2401.00368 §3.2. Think-Then-Embed inserts a generated reasoning text between the input and the pooled embedding, and the pooling position is not stated 2510.05014 §4 2510.05014 §3. RAG-end2end trains the passage encoder through the generator loss instead of freezing it, and swaps in a rebuilt index when its background job finishes 2210.02627 §3.3.

Contradictions

The cards use different BM25 baselines for the same BEIR datasets. Promptagator reports ArguAna 31.5, Touché 36.7 and FEVER 75.3 2209.11755 tab. 2. Llama2Vec reports 39.7, 44.2 and 65.1 for the same datasets 2312.15503 tab. 3. The cards do not say why. The averages also cover different sets: 11 datasets for Promptagator, 14 for Llama2Vec, which adds NQ, Quora and CQADupStack, and 15 for E5-Mistral 2401.00368 tab. 3. So the 56.4 of Llama2Vec and the 56.9 of E5-Mistral are not on the same datasets. Compare numbers within one table, not across papers.

The survey says RETRO uses 25 times fewer parameters than GPT-3, quoting the RETRO paper 2407.13193 §4.3. The RETRO card lists the 7.5B model's size 2112.04426 tab. 2 but not GPT-3's, so the ratio is not checked here.

Papers

• Promptagator: Few-shot Dense Retrieval From 8 Examples: few-shot synthetic query generation with a round-trip filter, plus an optional reranker.
• ConveRT: Efficient and Accurate Conversational Representations from Transformers: a small quantized dual encoder for reply selection, pretrained on Reddit reply pairs.
• On the Risk of Misinformation Pollution with Large Language Models: LLM-written passages added to QA corpora, and their effect on DPR and BM25 retrieval.
• Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering: trains the DPR encoders and the passage index with a BART generator, plus a sentence-reconstruction auxiliary task.
• Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval: unsupervised Wikipedia adaptation of a 7B LLM, then LoRA fine-tuning on MS MARCO.
• Improving Text Embeddings with Large Language Models: a 7B decoder trained on LLM-generated multilingual task data, with instructions on queries only.
• M3-Embedding: one encoder with dense, sparse and multi-vector heads, for many languages and long documents.
• Improving language models by retrieving from trillions of tokens: retrieval of frozen-key neighbours inside a pretrained-from-scratch language model, plus RETRO-fitting for existing models.
• Learning To Retrieve Prompts for In-Context Learning: a dense retriever trained on LM-scored labels to pick few-shot examples.
• Retrieval-Augmented Generation for Natural Language Processing: A Survey: a survey of how retrieved text reaches the generator, with qualitative trade-offs and no experiments.
• Think Then Embed: Generative Context Improves Multimodal Embedding: a multimodal LLM writes a reasoning text before the embedding, from a 72B teacher or a finetuned reasoner.