Evaluating Retrieval Quality in Retrieval-Augmented Generation

In plain words

A retrieval-augmented system first searches for useful documents and then lets a language model write an answer from them. The usual way to check the search part is to compare it with human relevance labels, but those labels often say little about whether the final answer gets better. Checking the whole pipeline is the most direct test, but it is slow and needs a lot of memory, because every document is read at once.

The idea here is to ask the language model itself. Each retrieved document is handed to the model on its own, the answer is checked against the known correct answer, and the result is recorded as that document's score. A list of documents then gets a single number from standard ranking measures, applied to those scores. Because each document is processed alone, the work is cheaper and needs much less memory than handling the full list together.

TL;DR

• eRAG scores a retrieval list by running the RAG system's LLM once per retrieved document, with that document as its only context §2 eq. 1.
• Each per-document answer is scored against the gold output by the task's downstream metric, and those scores serve as graded relevance labels §2.
• Author-reported: its Kendall tau with downstream RAG performance beats the KILT provenance and LLM-annotation baselines on five datasets tab. 1.
• Author-reported: lower runtime and up to 50x less GPU memory than end-to-end evaluation tab. 2 §3.2.
• The released repository is only the scoring function; the experiments are not in it eval.py.

Problem

End-to-end RAG evaluation compares the final output with reference outputs. It does not show which retrieved document helped, it is costly in time and GPU memory, and it gives only list-level feedback, which is a problem for interleaving and document-level optimisation §1. Relevance labels from humans or from an LLM judge describe what a person or judge finds relevant, and the paper reports that such labels correlate only weakly with downstream RAG performance §1 tab. 1. Concatenating all k documents into one long input makes a vanilla transformer's cost grow quadratically with input length, which the paper uses as the basis of its cost comparison §2.

The method assumes that every evaluated query has a gold downstream output and that the downstream metric returns a value in [0, 1] §2 eval.py.

Method

For a retrieval model R that returns a ranked list R_k of k documents for a query q, the RAG LLM M produces one answer per document, M(q, {d}), where the document is the only input besides the query §2. The answer is scored against the expected output y by the downstream metric E_M, which gives the label G_q[d] for that document §2 eq. 1. The scores over the list are then passed to a set-based or ranking metric to give one retrieval score for the list §2. The paper names precision, recall, MAP, MRR, NDCG and hit rate as options. With non-binary labels, precision averages the labels and hit rate takes their maximum §2.

The paper states the cost as O(l k^2 d^2) for end-to-end evaluation and O(l k d^2) for eRAG, where l is the output length and d the average document length §2. These are asymptotic statements about a vanilla transformer; the paper's own timing and memory numbers are in tab. 2 §3.2.

The procedure follows the code's loop over ranks, which calls the LLM once per rank position with one document per query eval.py.

# Input:  queries; ranked_docs[q] = the k documents for q, best first
#         gold[q] = expected outputs for query q
# Output: one eRAG score for the retrieval list  [§2] [eq. 1]

def erag_score(
    queries, ranked_docs, gold, llm, downstream_metric, rank_metric
):
    # Build one document-level label per query and rank  [§2] [eq. 1]
    labels = {q: [] for q in queries}
    for rank in range(k):
        # Run the RAG LLM on one document at a time  [§2]
        answers = {
            q: llm(q, [ranked_docs[q][rank]]) for q in queries
        }
        # Score each single-document answer against the gold output  [§2]
        for q in queries:
            score = downstream_metric(answers[q], gold[q])  # in [0, 1]
            labels[q].append(score)
    # Apply a set-based or ranking metric to each query's labels  [§2]
    per_query = [rank_metric(labels[q]) for q in queries]
    # Average over queries; the paper does not state this step.
    # [code: erag/eval.py#L113-L117]
    return mean(per_query)

Document-level relevance, as defined in the paper:

\mathcal{G}_q[d] = \mathcal{E}_{\mathcal{M}}(\mathcal{M}(q, \{d\}), y) \quad : \quad \forall d \in \mathbf{R}_k eq. 1

Retrieval set-up used in the experiments: BM25 (via Pyserini) and Contriever (Faiss flat index) retrievers over the KILT Wikipedia dump, split into 100-word passages, each passage prefixed with its article title §3.1.

Hyperparameters

Results

Author-reported. Setup: T5-small with FiD, 50 retrieved documents, KILT validation sets (NQ, TriviaQA, HotpotQA, FEVER, WoW) §3.1 tab. 1. The table reports Kendall tau for BM25 retrieval. The "best non-eRAG" column is the highest tau among the baselines in tab. 1 for that dataset, which the card computed from the table; the paper does not present this comparison.

The paper states the tau gain as 0.168 to 0.494 §1. The gap on FEVER with BM25 in tab. 1 is larger than that upper bound; see Paper vs code.

Efficiency, T5-small FiD consuming 50 documents tab. 2:

The paper reports an average runtime speed-up of 2.468x, ranging from 1.232x (WoW) to 3.252x (FEVER) §3.2. These values match the runtimes in tab. 2.

Other findings, in words (figures are not reproduced here):

• For every retriever, the paper's eRAG variant has the highest correlation among the compared approaches §3.2 tab. 1.
• Correlation falls as the number of retrieved documents grows, for every method. The eRAG advantage holds at each list size tested on NQ, FEVER and WoW §3.2 fig. 1.
• Correlation with T5-small and T5-base differs by dataset, with no significant difference reported §3.2 fig. 2.
• With 10 BM25 documents, correlation is similar for IPA and FiD readers, and is consistently higher for FiD §3.2 fig. 3.

Paper vs code

The official repository is https://github.com/alirezasalemi7/eRAG, linked from the paper's introduction §1. Its owner is the first author, and it is MIT licensed LICENSE. It contains only an evaluation package with erag.eval and a batching helper eval.py utils.py.

• Scope. The repository has no retrieval code, no FiD or T5 training code, no Mistral annotation code, no KILT data loader, no correlation script and no timing or memory script. None of the paper's tables or figures can be regenerated from it eval.py README.md.
• Downstream metrics. The paper uses KILT exact match, accuracy and F1 §3.1. The repository does not implement them; the caller supplies downstream_metric. The README example uses ROUGE-L (maximum over references) instead README.md.
• Metric availability. The paper's definitions of non-binary labels match the code: precision is a mean and hit rate is a maximum §2 eval.py. But for non-binary labels the code accepts only success and P, and raises for other metrics eval.py eval.py. The paper's WoW column reports only P and hit rate for that reason tab. 1 caption.
• Binary path. If every label is 0 or 1, the code uses pytrec_eval's standard metrics, including MAP, MRR, NDCG and recall eval.py.
• Aggregation. The paper does not say how the per-query scores are combined into one number. The code takes an unweighted mean over queries eval.py.
• Batching. The code calls the generator once per rank position with one document per query, batched across queries eval.py. This matches the per-document procedure of §2.
• Range check. The code raises only when a label is above 1 or below 0 eval.py. It does not reject NaN, which fails both range tests silently and turns the list into a non-binary case.
• Cut-off handling. On the non-binary path, a cut-off larger than the list length (for example P_10 on a 5-document list) indexes a missing key and raises KeyError eval.py. The paper does not discuss this.

The core formula matches: the label is the downstream metric of the RAG LLM's answer to one document, and the list score is a ranking metric over those labels §2 eval.py.

Pitfalls

• Needs gold outputs. eRAG labels are computed against the expected output for each query. It cannot score retrieval on queries without gold outputs, and the paper's experiments use the KILT validation sets §3.1.
• Scores depend on the RAG LLM. The labels reflect what this LLM does with a single document, so the score measures the retriever together with a specific reader and downstream metric. The paper's own argument is that this is the intended target §2. Changing the reader, the prompt or the metric can change the ranking of retrievers.
• Effect of list size. The paper reports that correlation falls as k grows §3.2 fig. 1. Choose k with the target use in mind; the paper uses 50 for its main table tab. 1 caption.
• Efficiency numbers depend on batch settings. The runtime comparison uses the largest batch that fits for each method §3.2, and the memory comparison is reported at two configurations. Speed-ups at one batch size do not transfer to another.
• Memory ratios in the text are off at the ends. Computing ratios from tab. 2: query-level memory gains are about 7.4x to 16.2x, and the text gives 7 to 15 §3.2. Document-level gains are about 31x to 50x, and the text gives 30 to 48 §3.2. The abstract's "up to 50 times" matches NQ at document-level.
• Tau range is not reproduced from the table. The text gives a tau gain of 0.168 to 0.494 §1. Comparing each dataset's eRAG tau with the best baseline in tab. 1 gives 0.168 for NQ but 0.542 for FEVER with BM25. The paper does not say which comparison produced its range, so treat the range as unverified.
• Labels are binary or graded depending on the metric. A downstream metric that returns only 0 and 1 takes the binary path. One that returns a value such as 0.5 on any query forces the restricted non-binary path, so the available retrieval metrics change with the metric choice eval.py.
• Repeated LLM calls. The method calls the RAG LLM k times per query. The paper argues this is cheaper than one long input §2, but the call count still scales with k and with the number of queries.
• Validation split. The paper evaluates on public validation sets because test labels are unavailable §3.1. Results on other splits or datasets are not reported.

Lineage

• Builds on: the LLM-as-relevance-labeller idea the paper cites as inspiration §2, and the Fusion-in-Decoder reader §3.1.
• Compared against: KILT provenance labels, answer-containment labels, and LLM-based relevance annotation with Mistral-7B-Instruct-v0.2 §3.1 tab. 1.
• Superseded by: not stated in the content of this wiki.