--- kind: concept slug: dense-retrieval title: Dense retrieval summary: >- Ranks documents by one query vector against precomputed document vectors, a nearest-neighbour lookup. Variants differ in what they train on or generate first: labelled search data, LLM-written queries or tasks, unlabelled LLM adaptation, conversation pairs, multilingual data, a generator trained jointly, LM-scored examples or reasoning text. Pick by labels, generation budget and serving size. tags: [retrieval, embeddings, contrastive-learning, data-poisoning] papers: ["1911.03688", "2112.04426", "2112.08633", "2209.11755", "2210.02627", "2305.13661", "2312.15503", "2401.00368", "2402.03216", "2407.13193", "2510.05014"] related: [text-embedding, contrastive-learning, few-shot-learning, in-context-learning, low-rank-adaptation, nearest-neighbor-search, retrieval-augmented-generation, language-model-pretraining, knowledge-distillation, hallucination-mitigation, data-poisoning, ai-generated-text-detection, quantization-aware-training, multimodal-language-model, chain-of-thought-prompting, synthetic-query-generation, domain-adaptation] status: draft updated_at: 2026-11-01 --- ## Overview A dense retriever encodes each query and each document separately into a fixed-size vector and returns the documents whose vectors score highest against the query. Because document vectors can be computed offline, serving is cheap compared with models that compare query and document token by token [arxiv:2209.11755 §1]. The cost moves to training: the encoder needs relevance-labelled pairs, and the usual public sources (MS MARCO, Natural Questions) suit factoid questions better than other search intents such as argument retrieval or entity lookup [arxiv:2209.11755 §1] [arxiv:2209.11755 §2.1]. The shared mechanics: one encoder per side (or one shared encoder), a similarity score, and a contrastive loss that pulls each query toward its positive document and away from the other documents in the batch [arxiv:2312.15503 §3.2] [arxiv:2312.15503 eq. 5]. Dense retrieval also sits inside larger systems. A retrieval-augmented generator can read retrieved text in four ways: paste it into the prompt, mix the model's output distribution with the passages' distributions, feed passage features into the model's internals, or store each passage as adapter weights [arxiv:2407.13193 §4]. The survey says that only prompt pasting and logits mixing work with black-box APIs [arxiv:2407.13193 §4.5]. It runs no experiments, so its comparison is qualitative [arxiv:2407.13193 tab. 2]. ## Variants ### Labelled bi-encoder (the reference point) Train the encoder directly on labelled query-passage pairs such as MS MARCO. The cards use this family as baselines: the GenQ and GPL rows in [arxiv:2209.11755 tab. 2] are trained on MS MARCO-based data, and the Llama2Vec paper's main comparison, RepLLaMA, is fine-tuned on MS MARCO [arxiv:2312.15503 §4.2]. Use this when in-domain labelled pairs exist. The Promptagator ablation reports that its generated data catches up with about 50K labelled examples [arxiv:2209.11755 §4.3], so the threshold where labels stop mattering is an empirical question for your corpus. Labels do not always help. The RAG-end2end paper reports that supervised DPR fine-tuning with BM25 hard negatives did not improve news or conversation retrieval, and attributes this to answers sitting in passages longer than the 100 words DPR was trained on [arxiv:2210.02627 §4.5]. Check your passage lengths against the training setup before assuming labels will lift retrieval. ### Few-shot synthetic queries (Promptagator) Give an instruction LLM a short task description and up to eight labelled query-document examples. Then ask it to write queries for documents sampled from the target corpus. The generated pairs train the encoder; a round-trip filter drops pairs whose document the trained encoder does not rank in its top K [arxiv:2209.11755 §3.1] [arxiv:2209.11755 §3.2]. The paper generates 8 queries per document at temperature 0.7, for up to one million documents per corpus [arxiv:2209.11755 §4.1]. The encoder is a 110M T5 v1.1 base dual encoder, and a 110M cross-encoder can rerank the top 200 [arxiv:2209.11755 §4.1]. Card: [Promptagator: Few-shot Dense Retrieval From 8 Examples](/papers/2209.11755). Reported results, author-reported, 11 BEIR datasets: the few-shot encoder averages 47.8 nDCG@10 and the zero-shot version 45.5 [arxiv:2209.11755 tab. 2]. With the reranker the average is 52.8 [arxiv:2209.11755 tab. 2]. The paper reports that round-trip filtering helps 8 of 11 datasets and hurts NFCorpus and SciFact, which it attributes to overfitting on their small query sets [arxiv:2209.11755 §4.3]. Costs and limits the cards state: - The generator is a 137B FLAN model that the authors describe as "provided by the authors"; the card finds no public release [arxiv:2209.11755 §3.1]. The ablation suggests the generator matters more than the training recipe [arxiv:2209.11755 §4.3]. - Generation is the expensive part: 29.23M generated queries across two prompts, about 58.46M in total [arxiv:2209.11755 §7]. - The filter threshold K = 1 was tuned on MS MARCO, which the paper otherwise holds out [arxiv:2209.11755 §4.1]. - On BEIR datasets with only a test split, the few-shot examples come from the test set and must count as misses in the evaluation [arxiv:2209.11755 §2.2]. ### Unsupervised LLM adaptation, then labelled fine-tuning (Llama2Vec) Start from a decoder-only LLM (LLaMA-2-7B) and adapt it on Wikipedia text without search labels. The input is the passage followed by two prompt blocks: one asks the model to restate the passage (EBAE), the other to predict the next passage (EBAR). Each block's embedding is read from a special token, and a custom attention mask stops the two blocks from seeing each other, so one pass serves both [arxiv:2312.15503 §3.2] [arxiv:2312.15503 fig. 2]. The adaptation run uses 10,000 steps at batch size 256 and learning rate 1e-5 [arxiv:2312.15503 §4.1]. Search fine-tuning then uses contrastive learning with LoRA on MS MARCO [arxiv:2312.15503 §4.1]. Card: [Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval](/papers/2312.15503). Reported results, author-reported: MS MARCO passage MRR@10 43.1 against 41.2 for RepLLaMA [arxiv:2312.15503 tab. 1]; zero-shot BEIR average NDCG@10 56.4 against 53.9 for RepLLaMA over 14 datasets [arxiv:2312.15503 tab. 3]. Points to check before you implement: - "Unsupervised" covers only the adaptation step. The search fine-tuning still uses labelled MS MARCO data [arxiv:2312.15503 §4.1]. - The model is 7B parameters, against 110M for the Promptagator encoder [arxiv:2312.15503 tab. 1] [arxiv:2209.11755 §4.1]. The cards give no serving latency or memory figures for either model. - The released code reads the embedding differently from the paper. Use the card's "Paper vs code" section before reproducing the numbers [arxiv:2312.15503 §3.2]. ### Synthetic multi-task data and a 7B decoder (E5-Mistral) Ask a strong proprietary LLM to invent embedding tasks, not just queries for one corpus. For each asymmetric subgroup the model first brainstorms about 20 one-sentence task definitions, then writes one JSON example per task, in 93 languages [arxiv:2401.00368 §3.1] [arxiv:2401.00368 app. C]. Invalid JSON and exact duplicates are the only filters [arxiv:2401.00368 §3.1]. The paper reports about 500k examples, generated with GPT-4 and GPT-3.5-Turbo at temperature 1.0 [arxiv:2401.00368 §4.1]. The encoder is Mistral-7B with LoRA rank 16 on all linear layers, trained for one epoch with InfoNCE at temperature 0.02 [arxiv:2401.00368 §4.2] [arxiv:2401.00368 eq. 3]. Queries carry a task instruction and documents do not. The embedding is the last-layer vector at the [EOS] position [arxiv:2401.00368 §3.2]. The full-data model adds 13 public labelled datasets, about 1.8M examples in total [arxiv:2401.00368 §4.2] [arxiv:2401.00368 app. A]. Card: [Improving Text Embeddings with Large Language Models](/papers/2401.00368). Reported results, author-reported. On MTEB's 56 English datasets, the average is 66.6 with full data, against 64.2 for BGE-large-en-v1.5, and 63.1 with synthetic data alone [arxiv:2401.00368 tab. 1]. On BEIR's 15 retrieval datasets the average nDCG@10 is 56.9 [arxiv:2401.00368 tab. 3]. On MIRACL dev across 16 languages it scores 62.9, against 66.5 for mE5-large [arxiv:2401.00368 tab. 6]. On a personalized passkey task at 32k tokens, a RoPE base of 10^5 gives over 90% top-1 accuracy, while naively enlarging the window is worse [arxiv:2401.00368 §5.2] [arxiv:2401.00368 fig. 5]. Points to check before you implement: - The paper says the implementation uses the HuggingFace PEFT library and releases evaluation scripts [arxiv:2401.00368 app. A]. The training recipe has no public code; see the card's "Paper vs code" section. - The optimizer, LoRA alpha and dropout, the loss reduction, and how one hard negative is picked from the top 100 are not stated [arxiv:2401.00368 app. A]. - Not every generated example is a query-document pair. Long-short examples carry a misleading label and symmetric ones carry graded scores, and the paper does not say how these become training pairs [arxiv:2401.00368 tab. 9] [arxiv:2401.00368 tab. 12]. - The text and Table 8 disagree on the query length buckets [arxiv:2401.00368 §3.1] [arxiv:2401.00368 tab. 8]. Follow the table if you reproduce the templates. - Low-resource languages are weaker. On MIRACL, Swahili scores 68.4 against 74.9 for mE5-large [arxiv:2401.00368 tab. 2], and the paper generated only about 1k synthetic examples per low-resource language on average [arxiv:2401.00368 §4.1]. - Several training sets are also MTEB evaluation sets, and the paper reports no result with them removed [arxiv:2401.00368 app. B]. - The paper does not discuss the terms under which the generator's outputs were used [arxiv:2401.00368 §4.1]. Check the provider terms before you reproduce the data pipeline. ### Conversation pairs and a small quantized encoder (ConveRT) This variant targets reply selection in dialogue, not ad hoc search. A shared six-block Transformer encodes the conversation context and the candidate reply, and each side has its own output network. The model is pretrained on Reddit reply pairs with an in-batch softmax [arxiv:1911.03688 §2.1] [arxiv:1911.03688 §3]. Embeddings are stored at 8 bits and the other network weights at 16 bits; the paper's table gives 116 MB before quantization and 59 MB after [arxiv:1911.03688 tab. 2]. Card: [ConveRT: Efficient and Accurate Conversational Representations from Transformers](/papers/1911.03688). Reported results, author-reported, R100@1 (the correct reply ranked first among 100 candidates). On Reddit without fine-tuning, the single-context model scores 68.2 and the version that reads up to ten earlier messages scores 71.8 [arxiv:1911.03688 tab. 3] [arxiv:1911.03688 §2.1]. On DSTC7-Ubuntu the single-context model scores 38.2 and the multi-context model 71.2 [arxiv:1911.03688 tab. 5]. Points to check before you implement: - The released model is no longer available. The paper says fine-tuned models are released [arxiv:1911.03688 §3], but the authors' repository states that they were removed from public availability. Plan to pretrain on your own conversation pairs; the card's "Paper vs code" section has the details. - The single-context model drops from 68.2 on Reddit to 38.2 on DSTC7, and the paper does not discuss the gap [arxiv:1911.03688 tab. 3] [arxiv:1911.03688 tab. 5]. Do not assume the single-context model transfers. - The paper gives two hardware descriptions that disagree [arxiv:1911.03688 §3] [arxiv:1911.03688 §4]. Its reported pretraining cost is about 18 hours and about $85 on Google Cloud [arxiv:1911.03688 §3]; treat both as unverified. ### Multilingual, long-document, three-head encoder (M3-Embedding) One XLM-RoBERTa-based encoder, extended to 8,192 tokens, produces three scores from one pass: a dense vector from the [CLS] state, learned sparse token weights, and per-token vectors for late interaction [arxiv:2402.03216 §3.2]. A weighted sum of the three scores re-ranks candidates [arxiv:2402.03216 eq. 1]. Training runs RetroMAE pre-training, unsupervised contrastive learning on about 1.2 billion pairs, then fine-tuning in which the integrated score teaches all three heads [arxiv:2402.03216 §3.3] [arxiv:2402.03216 tab. 8]. Card: [M3-Embedding](/papers/2402.03216). Reported results, author-reported, on the paper's own splits. MIRACL dev (18 languages, nDCG@10): dense alone 69.2, all three heads 71.5 [arxiv:2402.03216 tab. 1]. MKQA (25 languages, Recall@100): dense 75.1, all three 75.5 [arxiv:2402.03216 tab. 2]. MLDR test (documents up to 8,192 tokens, nDCG@10): dense 52.5, sparse alone 62.2, all three 65.0 [arxiv:2402.03216 tab. 3]. Points to check before you implement: - Which head wins depends on the task. On MLDR the sparse head is about 10 points above dense [arxiv:2402.03216 §4.3]. On MKQA it is weak, because a query and its passage share few words across languages [arxiv:2402.03216 §4.2]. - Hybrid weights are per benchmark: (1, 0.3, 1) on MIRACL and (0.15, 0.5, 0.35) on MLDR [arxiv:2402.03216 §4.1] [arxiv:2402.03216 §4.3]. Treat them as starting points. - Multi-vector scoring keeps one vector per token. The paper uses it only to re-rank the dense top 200 on MIRACL, and does not discuss storage [arxiv:2402.03216 §4.1]. - Without long-document training, the dense head scores 41.2 on MLDR. An inference-only trick that inserts [CLS] tokens every 256 tokens raises it to 45.0 [arxiv:2402.03216 tab. 3] [arxiv:2402.03216 app. B.2]. - The paper gives no fine-tuning learning rate or maximum length [arxiv:2402.03216 app. B.1]. The InfoNCE temperature is also not stated [arxiv:2402.03216 eq. 2]. ### Reasoning text before embedding (Think-Then-Embed) This variant targets multimodal search, where the query is an image or video plus a text instruction. A multimodal LLM first writes a short reasoning text about the query or target, an embedding-centric reasoning trace (ECR). The embedder then encodes the input together with that text [arxiv:2510.05014 §4]. The authors argue that some instructions need multi-step reasoning that one forward pass cannot do while compressing the input into a vector [arxiv:2510.05014 §1]. The paper does not measure this claim directly [arxiv:2510.05014 §1]. Card: [Think Then Embed: Generative Context Improves Multimodal Embedding](/papers/2510.05014). Three forms share the ECR step: - Teacher reasoner (TTE_t). A 72B multimodal model writes the trace for every training item, and a Qwen2-VL 2B or 7B embedder is trained on the inputs with those traces [arxiv:2510.05014 §4.2] [arxiv:2510.05014 §5.1]. For contrastive training the teacher is asked to rephrase the answer, and only about half the items keep the ground truth [arxiv:2510.05014 app. A.2] [arxiv:2510.05014 tab. 7]. - Finetuned small reasoner (TTE_s). First, the embedder's own backbone writes the traces zero-shot. Second, a separate reasoner starts from the same backbone and is finetuned on teacher traces with token-level negative log-likelihood [arxiv:2510.05014 §4.3]. The reasoner is therefore the same size as the backbone, not a smaller model. - Unified model (TTE_u). The backbone is finetuned to reason on the trace set, then frozen. Only an embedding head, a multi-layer self-attention module initialised from the backbone's last layers, is trained on its hidden states [arxiv:2510.05014 §4.4] [arxiv:2510.05014 §7.3]. Training the contrastive and generation objectives jointly in one backbone lowered scores in the authors' runs [arxiv:2510.05014 app. B] [arxiv:2510.05014 tab. 8]. Reported results, author-reported. On MMEB-V2 (78 tasks, the "All" average), TTE_t-7B scores 71.5 and TTE_s-7B scores 68.6. The VLM2Vec-V2-7B row, which the authors reproduced themselves, scores 61.2 [arxiv:2510.05014 tab. 2] [arxiv:2510.05014 app. D]. On MMEB-V1 (36 tasks, "Overall"), TTE_t-7B scores 78.5 and TTE_t-2B scores 74.5; the strongest rows in the same table are QQMM at 72.5 (7B) and B3 at 68.1 (2B) [arxiv:2510.05014 tab. 1]. Zero-shot reasoning from the 7B backbone reaches 68.3 on MMEB-V1, and finetuning the reasoner adds 5.0 points [arxiv:2510.05014 tab. 3]. Points to check before you implement: - The teacher's name differs across the paper. The method section gives Qwen2.5-72B, the introduction gives Qwen2.5 VL 72B, and a prompt figure in the appendix names another model [arxiv:2510.05014 §4.2] [arxiv:2510.05014 §1] [arxiv:2510.05014 app. C]. The card cannot say which model wrote the released traces. - The teacher runs once per indexed item, but a user query is new at search time, so a reasoner call sits on the query path [arxiv:2510.05014 §4.2]. The paper gives no latency or memory numbers. - The headline gain for the small finetuned reasoner is not tied to a table row. The closest comparison is 68.6 against the reproduced 61.2 on MMEB-V2 [arxiv:2510.05014 tab. 2]. The paper's "surpassing proprietary models" claim rests on TTE_t-7B at 71.5 against 71.3 for seed-1.6-embedding, a margin of 0.2 points; leaderboard figures are as of submission [arxiv:2510.05014 tab. 2] [arxiv:2510.05014 §6.1]. - The 7B teacher scores 78.1 in the CoT ablation and 78.5 in the main table [arxiv:2510.05014 tab. 4] [arxiv:2510.05014 tab. 1]. The paper does not explain the difference. - Where the reasoning text joins the embedder input is not stated. The pooling position, any tags around the trace and the input template are not given [arxiv:2510.05014 §3] [arxiv:2510.05014 §4.1]. The teacher prompts appear only in figures [arxiv:2510.05014 fig. 7] [arxiv:2510.05014 fig. 8] [arxiv:2510.05014 fig. 9]. - The contrastive settings are given: global batch 8192, learning rate 2e-4 for the backbone and 5e-4 for the head, temperature 0.02, LoRA rank 16 and alpha 64 [arxiv:2510.05014 §5.1]. The optimizer, schedule, image resolution, maximum length and reasoner decoding settings are not stated [arxiv:2510.05014 §5.1]. ### Retrieval during pretraining (RETRO) Here the lookup is part of the language model. The input is cut into 64-token chunks. For each chunk, the model retrieves neighbours by nearest-neighbour search over frozen BERT keys, from a database of up to about 2 trillion tokens. The neighbours enter through chunked cross-attention in some decoder layers [arxiv:2112.04426 §2.2] [arxiv:2112.04426 §2.3] [arxiv:2112.04426 §2.4]. Neighbours for training data are computed in advance, and neighbours from the same document are removed so no future text leaks into the past [arxiv:2112.04426 §2.3]. Card: [Improving language models by retrieving from trillions of tokens](/papers/2112.04426). Reported results, author-reported. A 7.5B RETRO model beats the 7B baseline on C4 and Wikitext103 in bits per byte and perplexity [arxiv:2112.04426 app. F.1]. It beats Jurassic-1 and Gopher on a majority of Pile test subsets [arxiv:2112.04426 §4.1]. On Natural Questions with DPR passages it scores 45.5 exact match, against 30.4 for the closed-book 7B baseline [arxiv:2112.04426 tab. 5]. RETRO-fitting adds the retrieval layers to a pretrained model. The base model is frozen and only the new cross-attention and neighbour-encoder weights train, using about 6 million sequences [arxiv:2112.04426 §4.2] [arxiv:2112.04426 app. C.3]. With retrieval switched off, the output is unchanged [arxiv:2112.04426 §4.2]. Points to check before you implement: - The index is the cost. The paper gives about 215 GB for a Wikipedia index and about 93 TB for MassiveText [arxiv:2112.04426 §4.1]. - Part of the gain comes from leaked training text. Evaluate on chunks below the overlap threshold to compare models fairly [arxiv:2112.04426 §2.6] [arxiv:2112.04426 §4.4]. - The paper does not describe how documents are cut into the neighbour and continuation pieces, nor the BERT size or the index settings [arxiv:2112.04426 §2.3]. Each changes retrieval quality. - No official code was found [arxiv:2112.04426 §2.5]. Implement from the paper and the card's Paper vs code section. ### Example retrieval for in-context learning (EPR) This variant retrieves few-shot examples for a prompt, not documents for an answer. Two BERT-base encoders score training examples against an input. One reads the input, the other reads example input-output pairs, and similarity is an inner product [arxiv:2112.08633 §3.2]. Labels come from a scoring LM: for each training example, BM25 candidates are ranked by how likely the LM gives the gold output with that candidate in the prompt. The top 5 become positives and the bottom 5 hard negatives [arxiv:2112.08633 §3.1] [arxiv:2112.08633 §4.2]. Card: [Learning To Retrieve Prompts for In-Context Learning](/papers/2112.08633). Reported results, author-reported, development sets, GPT-Neo as scorer and inference LM. Against the best baseline, EPR scores 31.9 against 26.0 (BM25) on BREAK, 64.2 against 57.0 (CBR) on MTOP, and 54.3 against 51.4 (CBR) on SMCalFlow [arxiv:2112.08633 tab. 2]. With other inference LMs (GPT-3, Codex, GPT-J) on 1,000-example subsets, EPR is highest in every row of the proxy table [arxiv:2112.08633 tab. 5]. Points to check before you implement: - Labelling is the cost. Each training example needs one scoring pass per candidate. At L = 50 candidates on BREAK's 44K training examples, that is about 2.2 million scoring passes by our arithmetic [arxiv:2112.08633 §4.1] [arxiv:2112.08633 §4.2]. - Training candidates use the gold output, which exists only at training time, while test-time queries use the input alone [arxiv:2112.08633 §3.1] [arxiv:2112.08633 §3.2]. - Results are single runs with no variance reported [arxiv:2112.08633 §4.3]. Treat gaps of a few points as uncertain. - The paper says EPR may inherit biases from the large LMs that supply its signals [arxiv:2112.08633 app. A]. The repository's licence is non-commercial in part; see the card's Paper vs code section before copying any code. ### Joint retriever-generator training (RAG-end2end) Here the retriever is not frozen. The question encoder and the passage encoder both receive gradients from the generator's loss, and the generator is a BART model [arxiv:2210.02627 §3.1] [arxiv:2210.02627 §3.3]. Because the passage vectors change, the index must be rebuilt. A background job re-encodes the passages on separate GPUs and re-indexes them, and the training loop swaps in each new index when it is ready [arxiv:2210.02627 §3.3] [arxiv:2210.02627 fig. 1]. The average refresh interval is about 750 updates on the paper's 250,000-passage base [arxiv:2210.02627 §3.3]. An auxiliary task asks the generator to rebuild a domain sentence from retrieved passages, marked with a control token [arxiv:2210.02627 §3.4] [arxiv:2210.02627 eq. 3]. Card: [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](/papers/2210.02627). Reported results, author-reported, exact match on three domain QA sets. The end-to-end model without the auxiliary task beats original RAG trained on the same QA pairs: 8.08 against 2.95 on COVID-19, 8.39 against 7.26 on News and 24.25 against 12.09 on Conversations [arxiv:2210.02627 tab. 1]. With the auxiliary task it reaches 8.32, 14.08 and 25.95, against 3.66, 8.62 and 14.21 for original RAG with the task [arxiv:2210.02627 tab. 1]. Each configuration is a single run [arxiv:2210.02627 §4.2]. Points to check before you implement: - Refresh cost grows with the passage store. The paper reports several GPU hours to re-encode and several CPU hours to re-index on knowledge bases of millions of passages [arxiv:2210.02627 §3.3]. Its own domain bases hold 85,000 to 250,000 passages [arxiv:2210.02627 §4.1]. - Re-indexing must wait until every passage vector is saved, or it fails [arxiv:2210.02627 §3.3]. Training gradients can be stale because the index lags the encoder; the paper gives no measure of how much lag is tolerable. - The loss weight between the QA and reconstruction signals, the order in which the two kinds of example are mixed, and the number of retrieved passages k are not stated [arxiv:2210.02627 §3.1] [arxiv:2210.02627 §3.4]. - The COVID-19 training and validation pairs are synthetic, and only its test set is human-labelled [arxiv:2210.02627 §4.1]. - The footnote's code link points to an example folder that the repository's main branch no longer contains [arxiv:2210.02627 §1]. See the card's Paper vs code section before reproducing the setup. ## Corpus pollution A dense retriever indexes whatever text the corpus holds, so LLM-written passages added to the corpus reach the retriever like any other passage. Pan et al. test this with GPT-3.5, which writes one fake passage per question and adds it to the clean corpus, in four settings: Benign, Gen, Imi and Hack [arxiv:2305.13661 §3.2] [arxiv:2305.13661 §4.3]. Card: [On the Risk of Misinformation Pollution with Large Language Models](/papers/2305.13661). The paper's threat assumes generated text reaches the corpus the QA system reads [arxiv:2305.13661 §1] [arxiv:2305.13661 §4.3]. Author-reported, on NQ-1500 and CovidNews: - Under the Gen and Hack settings, the dense retriever (DPR, with FiD reader) lost less exact match than BM25. Gen cost DPR about 14% on NQ-1500 and about 34% on CovidNews, against about 20% and 54% for BM25; Hack cost DPR 39% and 50%, against 65% and 69% for BM25 [arxiv:2305.13661 tab. 2]. - The dense retriever is not immune. Under Hack, a fake passage reaches the top 10 for 48.9% of NQ-1500 questions with DPR, against 96.8% with BM25. Under Imi the two are close, 67.1% against 72.5% [arxiv:2305.13661 tab. 7]. - A detector trained on in-domain fakes separates them well, with AUROC 99.7 (Gen), 91.4 (Imi) and 99.8 (Hack). Trained on out-of-domain GPT-3 Wikipedia completions, it falls to 64.8, 50.7 and 52.6 [arxiv:2305.13661 tab. 3]. - The paper's cost estimate is $0.01 to $0.04 per 200-word fake document, using text-davinci-003 prices of the time [arxiv:2305.13661 §6]. Recheck current prices before quoting it. The three defenses act on the detector or the reader, after retrieval. The paper reports them as only partly effective [arxiv:2305.13661 §5] [arxiv:2305.13661 tab. 4]. ## When to use which | Situation | Use | Why | |---|---|---| | Roughly 50K or more in-domain labelled pairs | Labelled bi-encoder | In the ablation, generated data is reported to catch up with about 50K labelled examples [arxiv:2209.11755 §4.3] | | A new corpus, a few examples, LLM API access, budget for large-scale generation | Few-shot synthetic queries | Few-shot results beat the zero-shot version by about 2 points average nDCG@10 [arxiv:2209.11755 tab. 2]; generation is costly [arxiv:2209.11755 §7] | | Small serving budget and a small encoder | Few-shot synthetic queries with a 110M encoder | Both the encoder and the reranker are 110M [arxiv:2209.11755 §4.1] | | No in-domain labels, budget for LLM-generated data and a 7B encoder, and a need for many languages | Synthetic multi-task data with a 7B decoder (E5-Mistral) | Author-reported MTEB 63.1 with synthetic data alone against 66.6 with full data [arxiv:2401.00368 tab. 1]; weaker on low-resource languages [arxiv:2401.00368 tab. 2] | | Highest reported quality, can serve a 7B model and run adaptation | Unsupervised LLM adaptation | The adapted 7B model has the highest reported number on the MS MARCO passage and BEIR averages [arxiv:2312.15503 tab. 1] [arxiv:2312.15503 tab. 3] | | Reranking budget available | Add a cross-encoder on the top 200 | Promptagator's reranker adds about 5 points average over its encoder [arxiv:2209.11755 tab. 2] | | Dialogue reply selection, small encoder, and conversation pairs you can pretrain on | Conversation-pair dual encoder | Reported 68.2 R100@1 on Reddit with a small model [arxiv:1911.03688 tab. 3]; the released model is no longer public, so budget for pretraining [arxiv:1911.03688 §3] | | Texts in many languages, or documents up to 8,192 tokens, with one model | Three-head encoder (M3-Embedding) | Author-reported MIRACL 71.5 and MLDR 65.0 with all heads [arxiv:2402.03216 tab. 1] [arxiv:2402.03216 tab. 3]; multi-vector storage is the cost [arxiv:2402.03216 §4.1] | | Pretraining a language model with a very large text store, and budget for a multi-terabyte index | Retrieval during pretraining (RETRO) | Author-reported gains at 7.5B [arxiv:2112.04426 app. F.1]; the index is about 93 TB for MassiveText [arxiv:2112.04426 §4.1] | | Adding retrieval to an existing pretrained model, base weights frozen | RETRO-fitting | Only the new layers train, about 6 million sequences [arxiv:2112.04426 §4.2] [arxiv:2112.04426 app. C.3] | | A RAG system whose retriever must adapt to a new domain, a trainable generator, and budget for re-encoding and re-indexing during training | Joint retriever-generator training (RAG-end2end) | Author-reported EM on COVID-19 of 8.32 against 3.66 for original RAG with the same auxiliary task [arxiv:2210.02627 tab. 1]; the refresh cost is GPU and CPU hours [arxiv:2210.02627 §3.3] | | Few-shot prompt selection for a fixed task, with labelled input-output pairs and a scoring LM | Example retrieval (EPR) | 31.9 against 26.0 on BREAK, with the labelling pass as the cost [arxiv:2112.08633 tab. 2] [arxiv:2112.08633 §4.2] | | Model behind an API with no weight access, documents change often | Prompt pasting (text concatenation) | The survey says it fits fast rollout and changing knowledge, and works with black-box APIs [arxiv:2407.13193 tab. 2] [arxiv:2407.13193 §4.5] | | Corpus that can receive LLM-written text, such as open web or user submissions | Treat as a poisoning risk and test retrieval under pollution | Fake passages reached the top 10 for 48.9% (DPR) to 96.8% (BM25) of NQ-1500 questions under Hack [arxiv:2305.13661 tab. 7]; the defenses are partial [arxiv:2305.13661 §5] | | Multimodal search where instructions need multi-step reading, and a 72B teacher can label the training items | Teacher reasoner (TTE_t) | Author-reported MMEB-V2 "All" 71.5 against 61.2 for the reproduced VLM2Vec-V2-7B [arxiv:2510.05014 tab. 2]; the search-time reasoner call is not measured [arxiv:2510.05014 §4.2] | | Multimodal search with no 72B teacher, but a reasoner from the same backbone | Finetuned small reasoner (TTE_s) | Its first stage uses the backbone's own zero-shot traces: 68.3 against 65.8 for the 7B VLM2Vec-V1 baseline on MMEB-V1 [arxiv:2510.05014 tab. 3] [arxiv:2510.05014 tab. 1]; the SFT step still needs teacher traces [arxiv:2510.05014 §4.3] | | Multimodal search that must run one model for both reasoning and embedding | Unified model (TTE_u) | 68.8 on MMEB-V1 at 2B, against 68.7 for TTE_s [arxiv:2510.05014 tab. 1]; joint training of both objectives lowered scores in the authors' runs [arxiv:2510.05014 tab. 8] | | No labels, no LLM access and no conversation pairs | Not covered by these cards | Each variant needs labelled pairs, an LLM, or conversation pairs; M3's unsupervised stage still uses 1.2 billion mined pairs [arxiv:2402.03216 tab. 8] | ## Implementation sketch The serving core is the same in every variant. The training loss is shown once, in the form used for the Llama2Vec fine-tuning step. ```pseudocode # Serving: encode the corpus once, then one query vector per request # [arxiv:2209.11755 §1] [arxiv:2312.15503 §3.2] # Input: corpus docs, a query; output: top-k doc ids index = [encode_doc(d) for d in corpus] # offline, stored as a matrix # A query is encoded on its own and scored by inner product # [arxiv:2312.15503 eq. 5] # E5-Mistral prepends a task instruction to queries only [arxiv:2401.00368 eq. 1] q = encode_query(query) # (d,) scores = index @ q # (num_docs,) top_k = argsort_desc(scores)[:k] # Optional: a cross-encoder reranks the top 200 candidates # [arxiv:2209.11755 §4.1] ``` The M3 variant adds sparse and multi-vector scores to the dense score and combines them as a weighted sum [arxiv:2402.03216 eq. 1]. RETRO replaces the per-query lookup with a per-chunk lookup inside the model [arxiv:2112.04426 §2.4]. ```pseudocode # Contrastive training with in-batch negatives [arxiv:2312.15503 eq. 5] # Input: B (query, positive) pairs; temperature # Output: scalar loss def contrastive_loss(q_vecs, d_vecs, temperature): # Temperature is not stated in the paper [arxiv:2312.15503 §3.2] scores = q_vecs @ d_vecs.T / temperature # (B, B) # Row i's positive is column i; the other columns are negatives return cross_entropy(scores, targets=arange(B)).mean() ``` Where the variants differ is in producing the pairs. The Promptagator pipeline generates them with an LLM and filters them with the trained encoder [arxiv:2209.11755 §3.1] [arxiv:2209.11755 §3.2]. The Llama2Vec pipeline first adapts the LLM on Wikipedia and then uses labelled pairs [arxiv:2312.15503 §4.1]. The ConveRT pipeline pretrains on reply pairs from Reddit threads [arxiv:1911.03688 §3]. The M3 pipeline trains on mined and labelled pairs in three stages [arxiv:2402.03216 §3.3]. EPR labels its pairs with a scoring LM [arxiv:2112.08633 §3.1]. The E5-Mistral pipeline asks an LLM for task definitions and matching examples, then trains a 7B LLM with the same InfoNCE-style loss [arxiv:2401.00368 §3.1] [arxiv:2401.00368 §3.2]. Think-Then-Embed inserts a generated reasoning text between the input and the pooled embedding, and the pooling position is not stated [arxiv:2510.05014 §4] [arxiv:2510.05014 §3]. RAG-end2end trains the passage encoder through the generator loss instead of freezing it, and swaps in a rebuilt index when its background job finishes [arxiv:2210.02627 §3.3]. ## Contradictions The cards use different BM25 baselines for the same BEIR datasets. Promptagator reports ArguAna 31.5, Touché 36.7 and FEVER 75.3 [arxiv:2209.11755 tab. 2]. Llama2Vec reports 39.7, 44.2 and 65.1 for the same datasets [arxiv:2312.15503 tab. 3]. The cards do not say why. The averages also cover different sets: 11 datasets for Promptagator, 14 for Llama2Vec, which adds NQ, Quora and CQADupStack, and 15 for E5-Mistral [arxiv:2401.00368 tab. 3]. So the 56.4 of Llama2Vec and the 56.9 of E5-Mistral are not on the same datasets. Compare numbers within one table, not across papers. The survey says RETRO uses 25 times fewer parameters than GPT-3, quoting the RETRO paper [arxiv:2407.13193 §4.3]. The RETRO card lists the 7.5B model's size [arxiv:2112.04426 tab. 2] but not GPT-3's, so the ratio is not checked here. ## Papers - [Promptagator: Few-shot Dense Retrieval From 8 Examples](/papers/2209.11755): few-shot synthetic query generation with a round-trip filter, plus an optional reranker. - [ConveRT: Efficient and Accurate Conversational Representations from Transformers](/papers/1911.03688): a small quantized dual encoder for reply selection, pretrained on Reddit reply pairs. - [On the Risk of Misinformation Pollution with Large Language Models](/papers/2305.13661): LLM-written passages added to QA corpora, and their effect on DPR and BM25 retrieval. - [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](/papers/2210.02627): trains the DPR encoders and the passage index with a BART generator, plus a sentence-reconstruction auxiliary task. - [Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval](/papers/2312.15503): unsupervised Wikipedia adaptation of a 7B LLM, then LoRA fine-tuning on MS MARCO. - [Improving Text Embeddings with Large Language Models](/papers/2401.00368): a 7B decoder trained on LLM-generated multilingual task data, with instructions on queries only. - [M3-Embedding](/papers/2402.03216): one encoder with dense, sparse and multi-vector heads, for many languages and long documents. - [Improving language models by retrieving from trillions of tokens](/papers/2112.04426): retrieval of frozen-key neighbours inside a pretrained-from-scratch language model, plus RETRO-fitting for existing models. - [Learning To Retrieve Prompts for In-Context Learning](/papers/2112.08633): a dense retriever trained on LM-scored labels to pick few-shot examples. - [Retrieval-Augmented Generation for Natural Language Processing: A Survey](/papers/2407.13193): a survey of how retrieved text reaches the generator, with qualitative trade-offs and no experiments. - [Think Then Embed: Generative Context Improves Multimodal Embedding](/papers/2510.05014): a multimodal LLM writes a reasoning text before the embedding, from a 72B teacher or a finetuned reasoner.