Retrieval-augmented generation

Overview

Retrieval-augmented generation (RAG) answers a question in two steps: it first finds material relevant to the question in a store, then gives that material to a language model as context. The answer can then draw on material the model was not trained on, such as a course's own files 2309.10892 §3.2. The survey treats this as one way of supplying evidence to a model, next to other input augmentation 2202.03629 §5.1.3. A 2025 review of AI agents lists RAG as one of ten mitigation mechanisms for agent systems, next to tool use and memory architectures 2505.10468 §6.1. A 2024 survey of hallucination mitigation sorts retrieval by when it happens: before, during or after generation, or end to end 2401.01313 §2.1.1.

A survey of retrieval for NLP groups the ways retrieved text reaches the model into four routes: text pasted into the prompt, a blend of output distributions, passage features fed into the model's internals, and per-document add-on weights 2407.13193 §4. Its comparison of the four is qualitative 2407.13193 tab. 2.

One card asks a different question: should new facts reach a model through retrieval, or through more training on the same documents? Its tests use 7B models, Wikipedia text, four MMLU subjects and 910 current-events questions. Retrieval beat training on the original text in nearly every setting. Training on several paraphrases of each chunk improved on the base model for all three models, though it stayed below retrieval 2312.05934 tab. 1 tab. 2. Training on the original text gave small or mixed gains, and for one model it lowered the current-events score tab. 2. The results are author-reported, and the number of passages retrieved in the main tables is not stated tab. 1 §5.

The cards describe several ways to find the material. A course chatbot embeds every chunk and ranks chunks by cosine similarity [arxiv:2309.10892 §3.2.2, §3.2.3]. A medical chat model has the language model write search terms and ranks sections by how many terms they contain 2303.14070 p. 4. A chip-design assistant fine-tunes its retriever on synthetic query-passage pairs 2311.00176 §2.4. A dialogue agent runs web searches and shows the excerpts to raters 2209.14375 §2.4. A medical Gemini model searches only in rounds where its own sampled answers disagree 2404.18416 §2.1. An agent retrieves its own past successful attempts as examples 2308.10144 §4.2.1. A protein model mixes a retrieved sequence alignment into its own predictions [arxiv:2205.13760 §4.2]. A visual question system names the pictured entity before it reads Wikipedia 2302.11713 §4.2. A drug-repositioning agent places retrieval inside a tree search over agent actions 2507.07426 §2.3. Newer cards add retrieval inside pretraining 2112.04426 §2.4, one passage prepended every few tokens to a frozen model 2302.00083 §3.2, retrieval steps interleaved with reasoning 2212.10509 §3.1, and a fact-checking pipeline that ranks web pages by publisher tier 2403.11169 §Approach. A model trained to emit reflection tokens decides for itself when to retrieve and grades its own passages and sentences 2310.11511 §3.1. A tool-calling model gets the documentation of the API it should call 2305.15334 §3.2, and a graph index answers questions about a whole collection from community summaries [arxiv:2404.16130 §3.1.6]. Another card trains the retriever together with the generator and rebuilds the stored passage vectors during training 2210.02627 §3.3. The shared design question is how much to trust the retrieved text. The answer is only as good as what retrieval returns, and errors can pass from one stage to the next 2303.09136 §6.1. The cut-off in the course chatbot, which decides what counts as "close enough", is not justified 2309.10892 §3.1.1.

The evidence is thin and varies in kind. The course chatbot reports no retrieval accuracy, answer correctness, latency or user study [arxiv:2309.10892 §4]. The medical chat model reports one BERTScore comparison with ChatGPT, author-reported, and does not say whether its scored model used retrieval 2303.14070 tab. 1 p. 7. The chip-design assistant reports a gain with and without RAG on an internal chatbot test [arxiv:2311.00176 app. Engineering Assistant Chatbot Human Evaluation]. The dialogue agent's gains come from a full pipeline that includes reinforcement learning, so they do not isolate search 2209.14375 §3.2. The survey lists RAG among LLM mitigations without measuring it 2202.03629 §13.3. The protein model's overall gain over EVE is 0.003 Spearman, and the tables give no uncertainty estimates 2205.13760 §6.2. The visual system's pipeline numbers come from a split the main table does not name 2302.11713 §5.1. The newer cards are author-reported too: RETRO and In-Context RALM report perplexity and exact match on their own setups, and IRCoT reports answer F1 on four multi-hop QA sets 2112.04426 app. F.1 [arxiv:2302.00083 tab. 1] 2212.10509 tab. 4. Self-RAG reports 7B and 13B models against ChatGPT and Ret-ChatGPT, and its ablations appear only in a figure 2310.11511 tab. 2 fig. 3. Gorilla's results are author-reported on its own API benchmark, with no seeds or error bars 2305.15334 §4. GraphRAG's wins are judged by an LLM. Its claim-based check agrees with the judge on 78% of non-tied comprehensiveness pairs 2404.16130 §5.2. Med-Gemini's search gains are author-reported on the authors' own setup. Its search engine is not named, and its results per query and retrieval date are not stated 2404.18416 §2.1. RAG-end2end reports one run per setting, with no seeds or variance 2210.02627 §4.2.

Variants

Chunk, embed, top-k by cosine, with a similarity cut-off

This is the pipeline of VirtualTA, a Canvas-integrated course assistant.

• Sources. Course files from the learning platform, and transcripts of lecture recordings made with Whisper 2309.10892 §3.2. Resources are ranked, with primary ones such as lectures above secondary ones such as external readings. The ranking mechanism is not specified [arxiv:2309.10892 §3.2.1].
• Chunking. About 800 characters per chunk, keeping words and sentences whole. Overlap between chunks is not stated. Structured platform records such as announcements and assignments are embedded without chunking 2309.10892 §3.2.2.
• Embedding. text-embedding-ada-002, 1,536 dimensions [arxiv:2309.10892 §3.1.1].
• Retrieval. Every chunk is scored by cosine similarity against the query embedding. The ten best are kept, then only those above 0.75 [arxiv:2309.10892 §3.1.1, §3.2.3]. The order is top ten first, then the filter, so at most ten chunks reach the prompt and possibly none 2309.10892 §3.2.3.
• Generation. A fine-tuned classifier first routes the query to a feature. The model is then prompted with the retrieved context and an instruction to say "I'm not sure" rather than guess. The paper does not measure how well this works 2309.10892 §3.1, §3.2.3.
• Disclosure. Answers show the source file and a percentage. How that percentage is computed is not stated 2309.10892 §4.1.3, fig. 5. When nothing passes the cut-off, the paper says the answer was generated without course material 2309.10892 §4.1.3.

Domain-tuned dense retriever, top-k passages

This is the retriever of ChipNeMo, an adapted LLaMA2 for chip design. Its base is e5-small-unsupervised, fine-tuned with contrastive learning on 3,000 generated samples 2311.00176 §2.4. A sample is a passage, a query that a language model writes for it, and hard negatives: top retrieved passages that the same model judges do not answer the query, topped up with random passages if too few remain [arxiv:2311.00176 app. Sampling Procedure]. At answer time the top 8 passages are fetched from a store of about 67K passages, each about 512 characters 2311.00176 §3.6. On top-8 hit accuracy the tuned retriever beats the untuned e5-small by about 2x and a Sentence Transformer by about 30%. The per-category values appear only in a figure 2311.00176 §3.6. Learning rate, epochs and batch size for the retriever are not stated 2311.00176 §2.4.

Retriever and generator trained together, with a background index refresh

This is RAG-end2end from Improving the Domain Adaptation of RAG Models, aimed at domain text. The question encoder, the passage encoder and a BART generator are fine-tuned together through the generator's loss. The gradient reaches the passage encoder through the passage-selection probability 2210.02627 §3.3 eq. 2. A changed passage encoder makes the stored vectors stale, so a re-encoding job runs on other GPUs while training goes on, and a re-indexing job builds a new index once every vector is saved. The training loop loads each new index when it is ready 2210.02627 §3.3 fig. 1. On a store of millions of passages the paper reports several GPU hours for re-encoding and several CPU hours for re-indexing §3.3. A refresh starts about every 750 updates on average, measured on a 250,000-passage base §3.3. Gradients can be stale because the index lags the training, and the paper gives no measure of how much lag is tolerable §3.3.

The paper adds a side task. A sentence from the domain corpus is rebuilt from passages retrieved from a store that does not contain it. A control token marks these inputs 2210.02627 §3.4 eq. 3. The loss weight between this task and the QA task is not stated §3.4.

Against the original design, which keeps the passage encoder frozen §3.3, exact match on QA-only training is 8.08 against 2.95 on COVID-19, 8.39 against 7.26 on news and 24.25 against 12.09 on conversations 2210.02627 tab. 1. With the reconstruction task added, the figures are 8.32 against 3.66, 14.08 against 8.62 and 25.95 against 14.21 tab. 1. These are author-reported single runs. On open-domain SQuAD, the in-domain check, exact match is 40.02 against 28.12 tab. 5. The learning rate, batch size, optimiser and warmup are not stated §4.2.

Model-written keywords, term-count ranking

This is the autonomous retrieval of ChatDoctor, a LLaMA-7B model fine-tuned on patient-doctor dialogue. The same model drives every step:

• Keywords. The model writes comma-separated search terms for the question 2303.14070 fig. 4 p. 4.
• Lookup and ranking. The terms are matched against a knowledge base: live Wikipedia pages, or a disease table built from MedlinePlus [arxiv:2303.14070 p. 3] p. 4. Sections are ranked by the number of term hits. The paper calls this a term-matching system, not embedding search 2303.14070 p. 4-5. Section size is not stated in the paper.
• Reading. The model reads the top N sections one at a time and keeps what helps the question 2303.14070 fig. 5 p. 4. The paper uses N = 5 2303.14070 p. 4.
• Answer. The model answers from all the kept notes in one prompt 2303.14070 fig. 6 p. 5.

The paper's qualitative examples show this retrieval answering recent terms such as Mpox, which the baseline did not recognise. The examples were selected by the authors 2303.14070 fig. 7 p. 5-7.

Web search with evidence excerpts and reranking

This is the pipeline of Sparrow, a dialogue agent built on Chinchilla 70B. The model writes a search query, Google Search returns results, and the page is cut to an excerpt of up to 500 characters [arxiv:2209.14375 §2.4] app. C. The excerpt joins the dialogue as a search-result turn and is shown to raters as quoted evidence 2209.14375 §2.4. Per turn the agent samples eight candidates, four without evidence and four with evidence from two queries of two results each, and keeps the highest-scoring one [arxiv:2209.14375 §2.6].

On evidence-backed factual replies, the full pipeline scores 0.78 for "supported and plausible", against 0.61 for the base model that always searches 2209.14375 §3.2. The same model choosing when to search scores 0.70 2209.14375 §3.2.

Web search only when sampled answers disagree

This is the text pipeline of Med-Gemini, a set of Gemini models tuned for medicine. The model samples several reasoning paths and measures the Shannon entropy of the answer choices across them. If the entropy is at or below a threshold, it returns the majority answer. Otherwise it writes three queries aimed at the disagreement, and the results join the prompt for the next round 2404.18416 §2.1. The threshold and the log base are not stated. The search runs on the three text benchmarks only [arxiv:2404.18416 §2.1]. The MedQA run uses four rounds, and the NEJM CPC and GeneTuring runs use one 2404.18416 §3.1.

The same search step produces training data. Each training question gets one chain without search results and one with them, and chains with a wrong final answer are dropped before fine-tuning 2404.18416 §2.1 fig. 2.

On MedQA, accuracy rises from 87.2% to 91.1% over the search rounds, and the paper credits self-training with a further 3.2 points 2404.18416 §4.1 fig. 4. Search adds 4.0 points to top-10 accuracy on NEJM CPC. In the per-specialty top-10 table, search helps internal medicine and pediatrics but not neurology or psychiatry 2404.18416 §4.1 tab. C4. These are author-reported on the authors' own setup.

Evidence chains, a utility check and a feedback loop around a black-box LLM

This is the wrapper of LLM-Augmenter, which keeps a ChatGPT-style model fixed. A Knowledge Consolidator searches for evidence, links entities to descriptions and keeps a short list of evidence chains 2302.12813 §2.3.1. The prompt carries the query, history, evidence and any feedback §2.3.2. A Utility module scores each candidate and writes feedback, which goes into the next prompt if the score is too low §2.4. In the main experiments the policy is a fixed rule that always consolidates first, and the loop repeats until a candidate passes §3.2 §1. A learned policy appears only in a learning curve §3.3 fig. 3.

The dialog results are author-reported KF1 on the paper's own splits. On news chat with BM25 and feedback, KF1 is 36.41 against 26.71 for ChatGPT alone 2302.12813 tab. 1. On customer service it is 37.41 against 31.33 2302.12813 tab. 2. Feedback adds about 3 points over no feedback on that split 2302.12813 tab. 4. On OTT-QA the F1 goes from 0.59 for ChatGPT to 2.38 with DPR evidence, 8.08 with the CORE consolidator, and 11.80 with feedback added 2302.12813 tab. 5. Rows labelled gold use human-annotated knowledge and are an oracle §3.2.

Past successes as in-context examples

This is the experience recall of ExpeL, an agent that keeps its weights fixed. Task descriptions are embedded with all-mpnet-base-v2, and the top-k successful trajectories closest to the new task go into the prompt 2308.10144 §4.2.1. The retrieved units are whole trajectories, not passages. A list of natural-language insights, distilled from successes and failures on training tasks, goes into the same prompt 2308.10144 §4.3.

The paper reports the two parts separately 2308.10144 §5.2.1. On HotpotQA, insights alone reach 36% and retrieved trajectories alone 31%, against 28% for ReAct. On ALFWorld the order flips: insights alone 50%, retrieved trajectories alone 55%, against 40% for ReAct. The full method reaches 39% and 59% 2308.10144 tab. 3. Each HotpotQA trajectory uses about three times the tokens of ReAct, 4,310 against 1,320 2308.10144 tab. 6.

Stored edits routed to a second model

This is the memory of SERAC, a model editor that leaves the base model's weights unchanged. Each edit is stored as a text descriptor. A scope classifier scores the input against the stored edits. If the best score is at least 0.5, a counterfactual model answers from that edit and the input. Otherwise the base model answers 2206.06520 §3.1 eq. 1. The comparison is with retrieve-and-prompt (RP), which keeps the scope classifier but lets the base model answer 2206.06520 §5.1. SERAC has the higher edit success on all four benchmarks. RP had the lower drawdown on FC, 0.010 against 0.051 2206.06520 tab. 3. These numbers are author-reported on the paper's own benchmarks.

Live page retrieval inside a multi-agent workflow

This is the knowledge branch of SurgRAW, a zero-shot workflow for questions about robotic-surgery frames. For cognitive-inference questions, the workflow retrieves MedlinePlus text for the question and gives it to a chain-of-thought agent 2503.10265 §3.3 §3.4. In the authors' ablation on GPT-4o, retrieval raises the cognitive-inference average from 20.11 to 41.32 without chain-of-thought, and from 58.97 to 70.39 with it [arxiv:2503.10265 tab. 4]. The effect differs by task. With chain-of-thought, action prediction rises from 49.66 to 70.48 and patient detail from 78.16 to 96.21, but surgical outcome moves only from 41.51 to 42.87 2503.10265 tab. 4.

Retrieved alignment mixed into a generative model's scores

This is the inference-time retrieval of Tranception, a protein sequence transformer. Retrieval runs once per wild-type protein. A multiple sequence alignment is built and turned into a per-position amino-acid profile 2205.13760 §4.1 app. B.5. At scoring time the model's next-residue log-probabilities are mixed with that profile, with weight 0.6, chosen on a validation subset 2205.13760 §4.2 eq. 3 app. B.5. Positions the alignment does not cover keep the model's own term. A mutant is scored as its log-likelihood ratio against the wild type, averaged over the two reading directions 2205.13760 §3.4 eq. 2. Building the alignment is part of the method: the paper uses a Jackhmmer search with a bit-score threshold and an evolutionary-coupling selection 2205.13760 app. C.

Reported results are author-reported Spearman on the ProteinGym substitution benchmark. With retrieval the score is 0.451. Without it, the same model gets 0.406 2205.13760 §6.2. The gap to EVE over all assays is 0.003. On shallow alignments the paper reports 0.453 against EVE's 0.408. On deep alignments EVE leads, 0.507 against 0.488 2205.13760 §6.2.

Name the entity from the image, then read a Wikipedia page

This is the With-KB pipeline of InfoSeek, a visual question-answering benchmark. A fine-tuned CLIP model scores a 100K-entry Wikipedia knowledge base against the image and question, and keeps the top five entities 2302.11713 §4.2. A Fusion-in-Decoder reader then takes the top 20 passages for each entity, 100 in all 2302.11713 §4.2 app. B.

The paper's comparison is by overall score. On the human split the pipeline reaches 18.2 against 10.8 for the best end-to-end model. On the Wikidata split the end-to-end model is ahead, 22.1 against 19.3 2302.11713 tab. 5. Recognition is the limit: with the gold entity, the same reader reaches 45.6 on the human split 2302.11713 tab. 5. The pipeline beats the end-to-end model on the most popular entities and loses on the least popular 2302.11713 §5.2. The released evaluation code has no CLIP entity recognizer or FiD reader, so the pipeline cannot be re-run from it.

Search as one step in a tree of agent actions

This is the retrieval action of DrugMCTS, a drug repositioning framework. A retrieval agent takes the top ten molecules by Tanimoto similarity and the top ten by ChemBERTa cosine, merges them, and collects the proteins known to bind any of them 2507.07426 §2.3 eq. 2 eq. 3. The retrieval scores are deterministic, so the pool is the same on every run 2507.07426 §2.3. Later actions summarise the query, filter the candidates with the model, and pull pocket text and PubMed abstracts for the reference proteins 2507.07426 §2.3 alg. 1. Monte Carlo tree search decides which sequence of actions to try, and a leaf is scored by how often the model repeats its choice and how often it answers yes when asked whether the molecule interacts with the target 2507.07426 §2.2 §2.4.

The paper's headline recall on DrugBank is 0.4466 with the ground-truth count used as selection, against 0.1552 for GPT-4o-mini with no retrieval 2507.07426 tab. 1. That protocol uses the ground-truth count, so compare it with care 2507.07426 §3.4. The released code's default retrieval is Tanimoto only. The paper's two-metric union is an option [code: src/drugmcts/config.py#L35].

Retrieval built into the architecture, trained with the model

This is the design of RETRO. The input is cut into 64-token chunks. For each chunk the model looks up neighbours in a database of up to about 2 trillion tokens, with keys from a frozen BERT encoder [arxiv:2112.04426 §2.2] §2.3. A small encoder turns each neighbour into features, and chunked cross-attention in some decoder layers lets later chunks read them [arxiv:2112.04426 §2.4] alg. 1. Training uses two neighbours per chunk 2112.04426 §4.1. Training neighbours are looked up ahead of time. At sampling time the lookup runs after each chunk 2112.04426 §2.3 §2.4.

On the 7.5B model, C4 bits per byte falls from 0.78 to 0.66 with retrieval on. The same model with retrieval switched off scores 0.78 [arxiv:2112.04426 app. F.1]. On Natural Questions with DPR passages it reaches 45.5 exact match, below FiD's 51.4 2112.04426 tab. 5. In a 247M ablation, removing retrieval raises C4 bits per byte from 0.822 to 0.987 2112.04426 app. D. A pretrained model can get the lookup later by training only the new cross-attention and encoder weights, under 10% of the weights for the 7B model 2112.04426 §4.2. The paper says the store can change without retraining, but does not test it 2112.04426 §3.2. The MassiveText index is about 93 TB 2112.04426 §4.1.

Frozen model, one retrieved passage prepended every few tokens

This is In-Context Retrieval-Augmented Language Models. The model is unchanged. The last 32 tokens of the text form a BM25 query, the top passage goes before the prefix, and the query runs again every 4 tokens 2302.00083 §3.1 §3.2 §5.2 §5.3. Passages are 100-word chunks, truncated to 256 tokens 2302.00083 §4.3. Two rerankers can choose among the top 16 BM25 hits. One scores each passage by the LM's likelihood of the recent tokens. The other is a trained RoBERTa classifier 2302.00083 §6.1 §6.2. The released code covers only the first, with an oracle option README.md.

With BM25, a 345M GPT-2 reaches 21.5 word perplexity on WikiText-103, against 26.3 without retrieval. The 1.5B model without retrieval reaches 20.0 2302.00083 tab. 1. With the trained reranker the 345M model reaches 19.7 tab. 1. On Natural Questions, LLaMA-13B rises from 12.0 to 31.0 exact match with two DPR passages 2302.00083 tab. 4. These are perplexity results, and the paper does not measure generated text quality [arxiv:2302.00083 tab. 1].

Interleaved retrieval and reasoning

This is IRCoT. The model writes one reasoning sentence, and that sentence becomes the next BM25 query. The paragraphs it retrieves join the prompt before the next sentence. The loop ends when the reasoning contains "answer is" or at a step limit. The first retrieval uses the question alone, and the collected paragraphs are capped at 15 2212.10509 §3.1. No training is needed. The prompt holds annotated demonstrations, 15 questions per set, drawn from 20 written 2212.10509 §4.1.

With GPT-3 (code-davinci-002), answer F1 is 60.7 on HotpotQA. One-step retrieval gives 53.6 and no retrieval 47.5. On 2WikiMultihopQA it is 68.0 against 54.8. On MuSiQue it is 36.5 against 29.4. On IIRC it is 49.9 against 49.8 2212.10509 tab. 4. With Flan-T5-XXL (11B), HotpotQA reaches 59.1 against 49.7 tab. 4. In a manual check of 40 questions per set, the number of chains with at least one false fact fell by 50% on HotpotQA [arxiv:2212.10509 §5]. Each retrieval step needs an extra model call, and the prompt grows with each collected paragraph 2212.10509 §Limitations.

Model-chosen retrieval with graded passages and sentences

This is Self-RAG, a Llama 2 model trained to emit reflection tokens. A retrieve token chooses, for each segment, between no retrieval, retrieval, and continuing with evidence already retrieved 2310.11511 tab. 1 §3.1. If retrieval is chosen, the model fetches K passages and writes one continuation per passage. Each continuation is graded with the relevance, support and usefulness tokens, and the highest-scoring candidates are kept in a beam 2310.11511 §3.3 eq. 3 eq. 4 alg. 1. The score adds the LM probability to a weighted sum of the grades. The weights and the retrieval threshold δ are inference-time settings. The paper's threshold is 0.2 for most tasks and 0 for ALCE §4.3. Raising δ means less retrieval 2310.11511 fig. 3c.

The critic is trained on GPT-4 labels, and the generator is trained on the labelled text with retrieved passage text excluded from the loss 2310.11511 §3.2.1 §3.2.2. Results are author-reported. The 13B model scores 55.8 on PopQA, against 50.8 for Ret-ChatGPT, and 70.3 citation precision on ASQA, against 65.1 for Ret-ChatGPT. ChatGPT stays ahead on TriviaQA, 74.3 against 69.3 2310.11511 tab. 2. In the 7B ablation, the PopQA score is 45.5 with the full model, 42.6 without the critic, and 41.8 with only the top passage 2310.11511 fig. 3. The paper reports no wall-clock cost, and the decoding cost scales with K and B §3.3.

Web search ranked by publisher credibility, with quoted evidence

This is MUSE, a fact-checking pipeline that writes replies to social-media posts. An LLM writes three search queries. Results are grouped into publisher tiers, derived from factuality and bias ratings, and searched from the most credible tier down 2403.11169 §Approach §Experiments [code: model/main.py#L792-L793]. Only pages published before the post are kept. Pages that pass a relevance threshold go to an LLM, which quotes refuting and context passages. The reply links each fact 2403.11169 §Approach. The thresholds and the stop rule appear only in the code main.py [code: model/main.py#L831].

Expert raters scored replies 8.1 out of 10. The figures were 6.3 for high-helpfulness Community Notes and 5.9 for GPT-4 without search [arxiv:2403.11169 fig. 2]. The authors report that retrieval adds a relative 25% to quality over the version without it 2403.11169 §Results. Their cost estimate is about 0.2 USD and two minutes per post 2403.11169 §Discussion.

Retrieved API documentation in the prompt

This is the documentation variant of Gorilla, a LLaMA-7B model that writes API calls. It is instruction-tuned on pairs of a request and the correct call, built from model-hub records 2305.15334 §3.1 §3.2. In retrieval mode a retriever returns the single most relevant API document, which is appended to the request. Oracle retrieval, which returns the correct document, is only an upper bound 2305.15334 §4. Training with the documentation in the user turn is a separate choice, and the paper trains both versions 2305.15334 §3.2.

On its own APIBench, zero-shot Gorilla reaches 59.13 overall accuracy on TorchHub, 71.68 on HuggingFace and 83.79 on TensorFlow Hub. Zero-shot GPT-4 reaches 38.70, 19.80 and 18.20 2305.15334 tab. 1. The model trained with the oracle document scores 0 overall without a retriever on every hub. The model trained without documentation drops to 37.63, 11.28 and 34.30 with BM25 at test time. With the GPT-based index it is slightly higher on TorchHub, 60.21, and lower on the other two 2305.15334 tab. 2.

Corpus-wide knowledge graph with community summaries

This is GraphRAG, which indexes a whole corpus once and then answers questions about all of it. An LLM extracts entities, relations and claims into a graph. Hierarchical Leiden clustering groups the graph, and an LLM writes a summary for each community at each level 2404.16130 §3.1.2 §3.1.4 §3.1.5. At query time the summaries are packed into windows of about 8k tokens. Each window gives a partial answer with a helpfulness score, and the partial answers are merged into one reply 2404.16130 §3.1.6.

On two corpora of about one million tokens each, the authors judged the answers against vector RAG. Comprehensiveness win rates were 72 to 83% on Podcast and 72 to 80% on News. Diversity win rates were 75 to 82% on Podcast and 62 to 71% on News 2404.16130 §5.1. The judging used 125 questions per corpus §4.1. Empowerment is mixed, and directness favours vector RAG, as the control predicts §5.1. Root-level summaries need over 97% fewer context tokens than map-reduce over the source text §5.1 tab. 2. Indexing took 281 minutes on the Podcast corpus §4.1.3.

Analyses and proposals that name retrieval

• Rare facts. In closed-book QA, accuracy rises with the number of pre-training documents that mention both the question's entity and the answer 2211.08411 §3.1. The authors fit a log-linear trend in model size on rare questions, with R² = 0.98, and read it as needing very large models to match humans 2211.08411 §4.2. A BM25 setting that adds the top three Wikipedia paragraphs beats closed-book GPT-Neo at every count, most of all for rare questions. The paper gives no numbers for this in the text 2211.08411 §4.3.
• Corpus pollution. The retrieved text is only as trustworthy as the store. The authors add one GPT-3.5 fake passage per question to the corpus of open-domain QA systems 2305.13661 §4.3. Under the three adversarial settings, exact match falls 14% to 54% for DPR-based setups and 20% to 87% for BM25-based ones 2305.13661 §4.4 tab. 2. The most damaging setting, Hack, asks for the false answer to be rephrased ten ways, aimed at the machine reader rather than people 2305.13661 §3.2. Defenses are preliminary. A detector trained on in-domain text reaches 91.4 to 99.8 AUROC, but an out-of-domain detector stays between 50.7 and 64.8 2305.13661 tab. 3. Reader voting raises exact match by 11% to 23% relative to the baseline 2305.13661 tab. 4. Card: Misinformation Pollution with Large Language Models.
• Authority. The legal survey proposes authority-grounded retrieval. A source must be checked for hierarchy, binding force, jurisdiction and valid time, and for whether it supports the claim. Conflicts stay visible 2303.09136 §3.2. The survey labels these requirements as design proposals, not a validated standard 2303.09136 §1.2. Card: A Short Survey of Viewing Large Language Models in Legal Aspect.
• Named only. The hallucination survey lists retrieval-augmented knowledge-grounded dialogue 2202.03629 §8.2.4 and RAG for LLMs 2202.03629 §13.3 among its mitigation methods. Card: Survey of Hallucination in Natural Language Generation. The agent review lists RAG among its ten mitigation mechanisms and gives it no pipeline detail 2505.10468 §6.1. Card: AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges. Neither gives enough detail to build a variant, so no pipeline is attributed to them.
• Surveys that name RAG. The LLM survey lists retrieval among its ways of augmenting a model, with no pipeline detail 2402.06196 §4.3. Card: Large Language Models: A Survey.
• Agent memory. The memory survey describes retrieved textual memory: each entry is embedded when written, scored against the current query when read, and the top K are used 2404.13501 §5.2.1. It leaves K and the similarity function open, and gives Generative Agents' cosine score with recency and importance as one example 2404.13501 §5.2.1. It argues that textual memory is cheaper to write and parametric memory cheaper to read, without measuring either 2404.13501 §5.2.3. Card: Agent-memory survey. The survey runs no experiments.
• Taxonomy and small models. The RAG survey sorts retrieval fusions and gives pseudocode for each, but runs no experiments of its own [arxiv:2407.13193 §4] alg. 3. Card: Retrieval-Augmented Generation for Natural Language Processing: A Survey. The small-model survey lists RAG as one way a small model can enhance an LLM 2409.06857 §3.1.2. Card: Small-model survey.
• Multi-agent chat. AutoGen's retrieval chat app reports F1 23.40% and recall 62.60% without interactive retrieval. The interactive-retrieval result is only in a figure 2308.08155 app. A2 fig. 4b. Card: AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation.

When to use which

The table uses the constraints each claim depends on. Rows that rest on the cards cite them. Rows marked as our inference or recommendation go beyond them.

Implementation sketch

A minimal version of the VirtualTA pipeline, then the keyword variant. Stages and thresholds are from the cards; the functions transcribe, chunk, embed, generate and hits are placeholders.

# Index  [arxiv:2309.10892 §3.2.2]
def index(documents):
    store = []
    for doc in documents:
        # Recordings become text first; the ASR upload limit is 25MB  [§5.1]
        text = transcribe(doc) if doc.is_recording else doc.text
        # About 800 characters per chunk, sentences kept whole  [§3.2.2]
        for piece in chunk(text, target_chars=800):
            # 1,536-dimensional embedding  [§3.1.1]
            store.append((piece, embed(piece), doc.source))
    return store

# Answer  [arxiv:2309.10892 §3.1.1, §3.2.3]
def answer(query, store):
    q = embed(query)
    # Score every chunk by cosine similarity and keep the ten best  [§3.1.1]
    best = top_k(store, score=lambda c: cosine(q, c.vector), k=10)
    # Keep chunks above 0.75; the value is not justified in the paper  [§3.1.1]
    context = [c for c in best if c.score > 0.75]
    if not context:
        # The paper labels the answer as generated without course material  [§4.1.3]
        return generate(query, context=[]), []
    # Prompt with the context and an "I'm not sure" instruction  [§3.2.3]
    return generate(query, context), [c.source for c in context]

# Keyword variant  [arxiv:2303.14070 fig. 4-6] [p. 4-5]
def answer_by_keywords(query, entries, n=5):
    # The model writes the search terms  [fig. 4]
    raw = generate(keyword_prompt(query)).split(",")
    # Drop empty terms: an empty term matches every entry
    terms = [t.strip() for t in raw if t.strip()]
    # Rank entries by term hits and keep the top n  [p. 4-5]
    ranked = sorted(entries, key=lambda e: hits(e.text, terms), reverse=True)
    # The model reads each kept entry for what helps  [fig. 5] [p. 4]
    notes = [generate(select_prompt(query, e.text)) for e in ranked[:n]]
    # One final prompt over all notes  [fig. 6] [p. 5]
    return generate(final_prompt(query, notes))

Where variants differ: chunking and embedding change index, and a domain-tuned retriever changes only embed 2311.00176 §2.4. A hybrid or re-ranking step would replace the scoring line in answer. The keyword variant replaces both halves: index keeps whole entries, and answer_by_keywords ranks by term hits. The web-search variant replaces index with a live query 2209.14375 §2.4, and the past-success variant returns whole trajectories instead of passages 2308.10144 §4.2.1. The LLM-Augmenter wrapper keeps generate unchanged and adds a utility check that returns feedback for the next prompt 2302.12813 §2.4. Self-RAG moves the retrieve decision into the decoding loop, generating one continuation per retrieved passage and keeping the best B [arxiv:2310.11511 alg. 1]. RAG-end2end leaves answer as it is and moves the index refresh into a background job 2210.02627 §3.3. RETRO moves retrieval into the forward pass, through chunk-wise cross-attention 2112.04426 §2.4. In-Context RALM keeps the model and changes only its input 2302.00083 §3.1. IRCoT loops over reasoning steps, with a retrieval call at each step 2212.10509 §3.1. The protein variant has no context prompt at all: it mixes a retrieved profile into the model's per-position scores 2205.13760 §4.2. The paper's source priority 2309.10892 §3.2.1 has no mechanism to copy, so it is left out. The API-documentation variant is the same answer step with top-1 retrieval and the document appended to the request [arxiv:2305.15334 §3.2] §4. The graph variant replaces the top-k step with a map-reduce over community summaries 2404.16130 §3.1.6. The Med-Gemini loop adds an early exit: answer_with_search returns the majority answer once the entropy is at or below a threshold, and otherwise adds search results to the next round 2404.18416 §2.1.

Contradictions

No two cards report incompatible numbers for the same pipeline. Self-RAG compares its own retrieval with top-1 and no-retrieval variants in an ablation 2310.11511 fig. 3, and no other card compares retrieval designs directly. Seven points still pull in different directions, and none is resolved here.

• Relevance and authority. The legal survey says retrieving relevant material does not establish that it is authoritative 2303.09136 §3.2. The course chatbot, the medical chat model and the chip-design assistant rank passages by similarity or term hits 2309.10892 §3.2.3 [arxiv:2303.14070 p. 4] 2311.00176 §3.6. None reports checking source status or jurisdiction, so these papers neither confirm nor refute the survey's point.
• Search on every query or on some. On Sparrow's supported-and-plausible metric, always searching scored 0.61 and choosing when to search scored 0.70 2209.14375 §3.2. The course chatbot sends every query through retrieval 2309.10892 §3.2.3. SurgRAW retrieves only for cognitive-inference questions 2503.10265 §3.3. Med-Gemini searches only in rounds where its sampled answers disagree, a third rule [arxiv:2404.18416 §2.1]. None of the cited comparisons tests that gate against the other two. The metrics and tasks differ, so this is a difference in design, not a tested conflict.
• Insights or retrieved trajectories. Within ExpeL, insights alone beat retrieved trajectories on HotpotQA, 36% against 31%. On ALFWorld the order flips, 50% against 55% 2308.10144 §5.2.1.
• Effect sizes are not comparable. ChipNeMo's gain is on a 7-point rating 2311.00176 app. Engineering Assistant Chatbot Human Evaluation, Sparrow's on a support rate 2209.14375 §3.2, ExpeL's on success rate 2308.10144 tab. 3 and SurgRAW's on accuracy 2503.10265 tab. 4. Read each within its own paper.
• Always-on retrieval. Self-RAG argues that retrieving for every input can lower output quality, because passages may be off-topic 2310.11511 §1. Its own table shows the fixed-retrieval Llama 2 7B scoring 38.2 on PopQA, against 14.7 without retrieval [arxiv:2310.11511 tab. 2]. In-Context RALM and IRCoT also report gains from retrieving often, at every few tokens or every reasoning sentence 2302.00083 tab. 1 2212.10509 tab. 4. LLM-Augmenter's main experiments also consolidate evidence on every query, and its learned policy is tested only in a learning curve 2302.12813 §3.2 fig. 3. The Self-RAG claim is a motivation, not a test, and the setups differ, so none of these settles the question.
• Retrieval frequency. RETRO looks up neighbours once per 64-token chunk 2112.04426 §2.2. In-Context RALM finds a stride of 4 tokens best among the values it tried, and says that tying the stride to the query length hurts 2302.00083 §5.2 §3.2. The models, objectives and metrics differ, so neither result tests the other.
• Retrieval can lower accuracy. Most cards report gains from retrieval. Gorilla reports that BM25 at test time lowers overall accuracy for the model trained without documentation on every hub 2305.15334 tab. 2. The paper says an imperfect test-time retriever can lower accuracy §4. The task is API choice rather than question answering, so this is a difference in setting, not a tested conflict.

Within IRCoT, the paper sets a limit of 8 reasoning steps, while the released code defaults to 10 2212.10509 §3.1 [code: commaqa/inference/ircot.py#L727].

Within the ChatDoctor paper, the text says the model reads five sections per question while the released scripts read four 2303.14070 p. 4 csv_reader.py. Both values are stated here; the paper's five is the one the paper describes.

Papers

• Artificial Intelligence-Enabled Intelligent Assistant for Personalized and Adaptive Learning in Higher Education: VirtualTA, a course chatbot with chunking, cosine top-k and a 0.75 cut-off; no quantitative evaluation.
• ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge: a medical chat model with keyword-driven retrieval over Wikipedia or a disease table; one author-reported BERTScore comparison with ChatGPT.
• ChipNeMo: Domain-Adapted LLMs for Chip Design: a domain-tuned e5-small retriever with top-8 passages; author-reported human ratings and EDA script results on internal benchmarks.
• Improving alignment of dialogue agents via targeted human judgements: web-search evidence excerpts, eight-candidate reranking and RL; author-reported, with search entangled with RL.
• ExpeL: LLM Agents Are Experiential Learners: retrieved past successes and distilled insights in the prompt of a frozen agent; author-reported on three text benchmarks.
• Memory-Based Model Editing at Scale: SERAC, a stored-edit cache with a scope classifier and a counterfactual model; author-reported on its own benchmarks.
• SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis: MedlinePlus retrieval for cognitive-inference questions inside a multi-agent workflow; author-reported ablation.
• Large Language Models Struggle to Learn Long-Tail Knowledge: closed-book accuracy follows document counts; retrieval helps rare questions most; no numbers for retrieval in the text.
• A Short Survey of Viewing Large Language Models in Legal Aspect: an authority-grounded framework for retrieval in legal AI; a proposal, not a validated standard.
• Survey of Hallucination in Natural Language Generation: names RAG for LLMs and retrieval-augmented dialogue as mitigations; gives no pipeline detail.
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges: a narrative review that lists RAG among ten mitigation mechanisms for agent systems; no experiments and no pipeline detail.
• Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval: a protein language model that mixes a retrieved alignment profile into its scores; author-reported ProteinGym results, with a small overall margin over EVE.
• Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?: InfoSeek, a visual QA benchmark, and a CLIP-then-FiD pipeline over Wikipedia; author-reported, on splits the main table does not name.
• On the Risk of Misinformation Pollution with Large Language Models: LLM-written fake passages injected into open-domain QA corpora, and three preliminary defenses; author-reported on two QA sets.
• DrugMCTS: a drug repurposing framework combining multi-agent, RAG and Monte Carlo Tree Search: retrieval as one action in a Monte Carlo tree search over agent steps; author-reported, with a ground-truth-count protocol.
• Retrieval-Augmented Generation for Natural Language Processing: A Survey: a taxonomy of retrieval fusions with pseudocode and a qualitative comparison; no experiments of its own.
• In-Context Retrieval-Augmented Language Models: one BM25 passage prepended to a frozen LM every few tokens; perplexity results on five corpora, author-reported.
• Improving language models by retrieving from trillions of tokens: RETRO, chunked cross-attention over a trillion-token store, with a retrofitting option; author-reported.
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions: IRCoT, retrieval and reasoning sentences interleaved; author-reported multi-hop QA gains with GPT-3 and Flan-T5.
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation: a multi-agent framework with a retrieval chat app; its retrieval result is only in a figure.
• Customized large language models can outperform Community Notes in correcting misinformation: MUSE, credibility-ranked web search with evidence quotes for fact-check replies; author-reported expert and reader studies.
• Small-model survey: a survey that lists RAG among ways small models help LLMs; no numbers of its own.
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection: a Llama 2 model trained to emit reflection tokens that decide when to retrieve and grade passages; author-reported, with a GPT-4-labelled critic.
• Large Language Models: A Survey: a broad LLM survey that names retrieval as an augmentation method; no experiments.
• Memory mechanisms in LLM-based agents: a survey: a survey of agent memory, including top-K retrieved textual memory; no experiments.
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback: LLM-Augmenter, evidence consolidation and a utility-driven feedback loop around a black-box LLM; author-reported, with no code released.
• Gorilla: Large Language Model Connected with Massive APIs: an API-calling model with optional retrieved documentation in the prompt; author-reported on its own APIBench.
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization: GraphRAG, a graph index with community summaries answered map-reduce style; author-reported, judged by an LLM on two corpora.
• Capabilities of Gemini Models in Medicine: Med-Gemini, text models that search the web only when sampled answers disagree, plus multimodal models; author-reported, with models not released.
• Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering: RAG-end2end, joint training of the retriever with the generator and a background index refresh; author-reported on three domain QA sets.
• Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs: fine-tuning against RAG for adding facts to 7B models, on MMLU subsets and 910 current-events questions; author-reported, RAG ahead in nearly all settings.
• Loss Functions and Metrics in Deep Learning: a survey whose RAG section defines context precision and claim-level faithfulness; no experiments.
• A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models: a taxonomy of 32 techniques, with retrieval placed before, during or after generation or end to end; no experiments.