Retrieval-Augmented Generation for Natural Language Processing: A Survey

In plain words

Language models answer from what they memorised during training. That memory goes stale, can be wrong, and says little about a specialist field. Retrieval- augmented generation (RAG) reduces this by looking up relevant text in a document store when a question arrives and giving that text to the model along with the question.

This survey explains how such systems are built. It splits a RAG system into three parts: a search step that finds the passages, a way of combining those passages with the model, and the model itself. For the combining step it names four routes: paste the text into the prompt, mix the model's word predictions with those of the passages, feed passage features into the model's internals, or store each passage as small add-on weights. It then covers how such systems are trained and tested, and what goes wrong when they are deployed.

The paper runs no experiments of its own. Read it as a map of design choices and their trade-offs, not as a benchmark.

TL;DR

• A RAG system is a retriever, a retrieval fusion and a generator; the survey describes each part and the pipeline that joins them §2 eq. 1 eq. 2.
• Four fusion types are defined: query-based, logits-based, latent and parametric. Each has a pseudocode algorithm and a cost analysis §4 alg. 3 to alg. 6.
• The survey reports no measurements of its own. Its one quantitative claim about a method, RETRO, is quoted from that paper §4.3.
• The paper footnotes a tutorial repository (Apache-2.0) with minimal code for four of the fusions. It differs from the paper's pseudocode in several places; see ## Paper vs code.

Problem

Large language models store knowledge in their parameters, which the survey names as the source of several failures: hallucination, slow knowledge update and weak domain expertise. Updating the parameters means retraining or fine-tuning, which is costly §1. The survey's premise is that an external knowledge base (KB), read at inference time, can address these limits and gives answers that can be traced to sources §1.

The method assumes that the relevant knowledge can be stored as text chunks, or as key-value pairs, that an encoder can embed. It also assumes that the generator can use retrieved text, which the survey treats as a property of the architecture and training §5.1.

Method

This is a survey, so the "method" is a decomposition of RAG, not a new algorithm. The authors give a general pipeline, then the components of each part.

Pipeline §2

1. Retrieve the top-k chunks most relevant to the query: Retriever(q, D) -> Z_q eq. 1 §2.
2. Fuse the query with the retrieved chunks: Retrieval Fusion(q, Z_q) eq. 2 §2.
3. Generate the answer: Generator(...) -> answer eq. 2 §2.

The retriever has three components: an encoder that embeds inputs, an approximate nearest-neighbour (ANN) index, and a vector store of key-value pairs §2. The survey calls the main trade-off "retrieval efficiency against retrieval quality" §2.

Building the retriever §3.1 alg. 1

• Chunking §3.1.1. Three types: fixed length, semantic (sentence boundaries, for example from NLTK or spaCy) and content-based (sections, function blocks). The survey says there is "no golden rule" for chunk size §3.1.1.
• Encoding §3.1.2. Sparse methods (one-hot, bag of words, TF-IDF, BM25) and dense methods (BERT variants, siamese encoders such as DPR, SimCSE and Contriever, LLM-based embedders). Hybrid encodings are also mentioned §3.1.2.
• Index §3.1.3. The index supports ANN search. It covers similarity metrics (cosine, Euclidean, Manhattan), dimension reduction (PCA, locality-sensitive hashing, product quantization), and ANN structures: IVFPQ, HNSW and tree-based indexes such as Annoy §3.1.3.
• Key-value store §3.1.4. Keys are embeddings and values are the content. The survey gives a QA example, question embedding as the key and question-answer pair as the value §3.1.4.

# Build the retriever  [alg. 1] [§3.1]
def build_retriever(corpus, encoder):
    keys = []
    values = []
    for document in corpus:
        # Split each document into chunks  [alg. 1 l.3]
        chunks = chunk(document)
        for j in range(len(chunks)):
            # Key: the embedding of this chunk  [alg. 1 l.5] [alg. 1 l.6]
            keys.append(encoder(chunks[j]))
            # Value: this chunk and the next one  [alg. 1 l.6]
            values.append(join(chunks[j:j + 2]))
    # Index over the keys, and a key-value store  [alg. 1 l.8-l.9]
    index = build_index(keys)
    store = key_value_store(keys, values)
    return index, store

Our reading of chunks[j:j + 2]: the paper writes c^j + c^(j+1) and does not say what the last chunk gets. Slicing keeps it alone alg. 1 l.6. The paper also says keys and values can be persisted to disk §3.1.4 alg. 1 l.7.

Querying the retriever §3.2 alg. 2

# Query the retriever  [alg. 2] [§3.2]
def query_retriever(query, encoder, index, store, k):
    # Embed the query with the same encoder as the chunks  [§3.2.1]
    embedding = encoder(query)
    # Top-k nearest neighbours by ANN search  [alg. 2 l.2] [§3.2.1]
    ids = index.search(embedding, k)
    # Fetch the stored values for those keys  [alg. 2 l.3]
    values = store.fetch(ids)
    # Reranking and similar steps are left open  [alg. 2 l.4] [§3.2.2]
    return post_process(values)

Retrieval fusions §4

All four fusions take the top-k retrieved items and feed them to the generator. They differ in where the retrieved information enters the model.

• Query-based §4.1 alg. 3. Text concatenation puts the retrieved text in front of the query; the survey notes that a length limit can cut the query off §4.1. Feature concatenation encodes the query and each retrieved text and feeds the joined features to a decoder, as in FiD §4.1.
• Logits-based §4.2 alg. 4. The generator runs on the query and on each retrieved text, and the output distributions are combined. Ensemble fusion uses a fixed weight lambda; calibration fusion sets the weight from the outputs, as in robust kNN-MT §4.2.
• Latent §4.3 alg. 5. Attention-based fusion adds a cross-attention module that reads encoded retrievals inside each block, as in RETRO. Weighted-addition fusion adds a weighted sum of retrieval vectors to the hidden states, as in EAE and ReFusion §4.3.
• Parametric §4.4 alg. 6. Each document is stored as a LoRA-style module offline. At query time the top-k modules are merged into one, added to the generator's weights for that query, and removed afterwards. The survey lists PRAG, DyPRAG and Poly-PRAG as examples §4.4.
• Hybrid §4.6. Query-based text concatenation is often the base, with another fusion added. Examples: DVD, and PRAG with text concatenation §4.6.

# Logits-based fusion, as written in the paper  [alg. 4] [§4.2]
# query_out and neighbour_out are output distributions from the generator
def logits_fusion(query, neighbours, generator, lam):
    # Output for the query alone  [alg. 4]
    query_out = generator(query)
    # Output for each retrieved text  [alg. 4]
    neighbour_out = [generator(v) for v in neighbours]
    # Interpolate; lam weights the neighbours here  [alg. 4]
    return lam * sum(neighbour_out) + (1 - lam) * query_out

The survey's comparison of the fusions is in tab. 2. Its complexity notes are: text concatenation costs O((Lq + sum Li)^2) for a Transformer §4.1; logits-based fusion runs the model k+1 times §4.2; weighted addition costs O(k*d) extra §4.3; parametric fusion's memory is O(N_w * l * d * r) §4.4.

Generators §5

Three generator families: API-only LLMs (GPT-4, Gemini), which usually limit RAG to prompt-time fusion; open-weight decoder-only LLMs, which allow adapters and fine-tuning; and retrieval-aware models (RAG, FiD, REALM, Atlas, RETRO, kNN-LM) §5.2. The survey gives selection guidelines for API-only, citation-critical, many-passage, latency-critical and frequently-updated settings §5.4.

Tasks §6

Language modelling is written as the chain-rule factorisation of a token sequence eq. 3 §6.1. The other tasks covered are machine translation (§6.2), summarisation (§6.3), question answering (§6.4), information extraction (§6.5), text classification (§6.6) and dialogue (§6.7).

Evaluation §7 tab. 3

Metrics are grouped in four focus areas: retrieval quality (precision@k, recall@k, NDCG and similar), generation performance (faithfulness, relevance, correctness), end-to-end robustness (noise, negative rejection, information integration, counterfactual robustness) and system efficiency (latency, cost, throughput, memory) tab. 3. Methods are human evaluation, automatic metrics, LLM-as-a-judge and end-user evaluation §7.3. The survey lists benchmark gaps: Wikipedia-centric data, few diagnostic benchmarks, little efficiency reporting, few multi-modal and temporal tests §7.4.

Training and knowledge-base updates §8

Without a KB update §8.1, three options: train the retriever encoder and rebuild the index (REPLUG minimises a KL divergence between retrieval and LM scores) §8.1.1; train the generator or the fusion modules, with LoRA as the parameter-efficient option §8.1.2; or train both jointly §8.1.3. With a KB update §8.2, the KB changes first (new values, a new corpus, or trainable values), then the retriever and generator are trained as before. The survey adds that such training is "not always necessary" given in-context learning §8.2. fig. 3 shows these strategies.

Applications and deployment §9

Industrial use from an agent-centric view §9.1; frameworks such as LangChain, LlamaIndex, Haystack, Semantic Kernel and AutoGen §9.2; production considerations §9.3.

Discussion §10

Security and privacy §10.1; retrieval quality §10.2; efficiency §10.3; reliability, including repetitive "looping" output §10.4; RAG against long-context LLMs §10.5; training §10.6; cross-modality retrieval §10.7; and GraphRAG §10.8.

Hyperparameters

The paper states few numeric settings. Values from the tutorial code are listed separately and marked as code.

Results

The survey reports no experiments. Its numerical statements are quoted from cited work.

Author-reported and qualitative only: the fusion ratings in tab. 2 (High, Moderate, Low) and the benchmark list in tab. 3. No numbers are given for FiD, kNN-LM or the parametric methods in the text.

Table 2 (qualitative comparison of retrieval fusions, reproduced from tab. 2):

Paper vs code

The paper's footnote §1 links github.com/luffy06/RAG-Tutorials (commit 6afa14d1ae733298771914f5df65a2a81d478170, Apache-2.0). The paper does not name that account's owner as an author. We use it because the paper links it as its tutorial code. The repo is "tutorial-style" by its own README and covers BM25 and FAISS retrieval with query, logits, latent and parametric fusion, but not the full survey. We did not check whether it is the authors' own account beyond that link.

Differences between the paper's pseudocode and the code:

• Logits weights and side. The paper's algorithm weights the neighbour sum with lambda and the query with 1 - lambda alg. 4. The code weights each neighbour by a softmax over its retrieval score [code: src/fusion/logits_fusion.py#L135-L140], takes only the top token of each neighbour distribution logits_fusion.py, renormalises logits_fusion.py, and gives lambda to the base model: blend_lambda * base + (1 - blend_lambda) * neighbours [code: src/fusion/logits_fusion.py#L187]. The default is 0.5 main.py.
• Latent injection point. The paper adds a weighted retrieval vector to the hidden state after attention alg. 5. The code builds a softmax-weighted average of retrieval embeddings [code: src/fusion/latent_fusion.py#L200-L213], passes it through a trainable linear projector, and adds the result to each QKV projection's output [code: src/fusion/latent_fusion.py#L28-L52].
• Parametric merge. The paper leaves the merge f unspecified alg. 6 l.2. The code weights each document's adapter by a softmax over retrieval scores and averages LoRA matrices A and B separately [code: src/fusion/parametric_fusion.py#L79-L93] [code: src/fusion/parametric_fusion.py#L274-L283].
• Stored values. The paper stores c^j + c^(j+1) as the value alg. 1 l.6. The code stores the single chunk text and returns it [code: src/retriever/retriever_faiss.py#L94].
• Index. The paper stresses IVFPQ and HNSW §3.1.3. The code defaults to an exact flat inner-product index retriever_faiss.py retriever_faiss.py. Other FAISS index strings go through index_factory retriever_faiss.py.
• Chunk units. The paper gives chunk sizes in tokens §3.1.1. The code's fixed-length chunker counts characters, with defaults 500 and 50 [code: src/retriever/chunking_util.py#L17-L26]. The sentence chunker defaults to 5 sentences with overlap 1 chunking_util.py.

Pitfalls

• No measurements. The fusion ratings in tab. 2 are qualitative, and the survey does not benchmark the fusions against each other §4.5.
• Algorithm 4 is not normalised as written. The neighbour term sums k distributions with weight lambda, so the total mass is k*lambda + (1 - lambda) and is 1 only when k = 1 alg. 4. The code normalises instead (see above).
• Algorithm 1 leaves the last chunk unspecified. It has no c^(j+1) [alg. 1 l.6]. The slicing above is our choice, not the paper's.
• Undefined symbols. w_j and the 1/k factor in weighted addition alg. 5 and the merge f in parametric fusion alg. 6 l.2 are not defined. PostProcess in alg. 2 l.4 is also left open.
• Averaging LoRA factors. Averaging matrices A and B separately is not the same as averaging the products B·A. The paper does not discuss this alg. 6.
• Input truncation. Text concatenation can exceed the model's length limit and cut the question §4.1. The survey says prompt design is the key step.
• Cost. Text concatenation makes the input long and inflates the KV cache §4.5. Logits fusion needs k+1 forward passes §4.2. Parametric fusion needs one adapter per document or a routed bank §4.4.
• Access. Latent and parametric fusion need white-box access and fine-tuning or pre-training §4.5. Only text concatenation and logits fusion work with black-box APIs §4.5.
• Chunk size. No value fits all tasks §3.1.1. The 128- and 512-token examples are not rules.
• Evaluation. LLM-as-a-judge can inherit judge biases and costs extra inference §7.3. Many benchmarks are Wikipedia-based §7.4.

Lineage

• Surveys compared: six earlier RAG surveys in tab. 1 §1.
• Methods surveyed: RAG, REALM, RETRO, FiD, kNN-LM, kNN-MT, EAE, ReFusion, PRAG, DyPRAG and Poly-PRAG are discussed in §4 §4.
• Superseded by: not stated.