---
kind: concept
slug: retrieval-evaluation
title: Retrieval evaluation
summary: >-
  Scores a retriever by how much its documents help the downstream output, not by human relevance labels. eRAG runs the RAG reader once per retrieved document and feeds the scores to ranking metrics. Author-reported: higher correlation with end-to-end RAG results than the baselines, and far less GPU memory. Trade-off: needs gold outputs and ties scores to one reader and metric.
tags: [retrieval, evaluation, retrieval-augmented-generation]
papers: ["2404.13781"]
related: [retrieval-augmented-generation, dense-retrieval, passage-reranking, llm-as-a-judge, llm-evaluation]
status: draft
updated_at: 2026-11-06
---

## Overview

A retrieval system is usually judged by how well its ranked list matches
relevance labels. Those labels describe what a person or a judge finds
relevant, which need not be what a downstream language model can use. The
direct test is end-to-end: run the whole pipeline and compare the final
outputs. That is slow and memory-hungry, and it says little about which
retrieved document helped [arxiv:2404.13781 §1].

The shared idea of this concept is to score each retrieved document by its
effect on the downstream output, then turn those scores into a list-level
number with a standard ranking metric. The variants differ in how they get the
per-document scores and in the metric applied to them. Only one card so far
covers this concept.

## Variants

### Downstream-answer labels (eRAG)

Each retrieved document is given to the RAG system's own language model alone,
together with the query. The answer is scored against the gold output by the
downstream metric, and that score is the document's relevance label. The
labels for a ranked list then go to a set-based or ranking metric such as
precision, recall, MAP, MRR, NDCG or hit rate [arxiv:2404.13781 §2]. With
non-binary labels the paper names precision (a mean) and hit rate (a maximum)
as the usable options [arxiv:2404.13781 §2]. Card:
[Evaluating Retrieval Quality in Retrieval-Augmented Generation](/papers/2404.13781).

Its reported correlation with downstream RAG performance is higher than that of
the KILT provenance and LLM-annotation labels, on five KILT datasets with BM25
retrieval [arxiv:2404.13781 §3.2] [arxiv:2404.13781 tab. 1]. Its reported
cost is lower than end-to-end evaluation, with up to about 50 times less GPU
memory at document level [arxiv:2404.13781 §3.2] [arxiv:2404.13781 tab. 2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Queries have gold outputs and you want to rank retrievers for a fixed RAG system | Downstream-answer labels | The paper's own target: the labels reflect what this reader does with each document [arxiv:2404.13781 §2] |
| No gold outputs for the queries | Not applicable as described | The method scores answers against the expected output, and the paper's experiments use KILT validation sets [arxiv:2404.13781 §2] [arxiv:2404.13781 §3.1] |
| Full pipeline is affordable and you need the final answer quality | End-to-end evaluation | The paper's baseline; it is the slow, memory-heavy reference that the variant approximates [arxiv:2404.13781 §1] |
| Reader or metric may change later | Re-run the check | Labels depend on the reader and the metric, so the ranking of retrievers can change with either [arxiv:2404.13781 §2] |
| Long retrieval lists | Choose k deliberately | Correlation falls as k grows for every method reported [arxiv:2404.13781 §3.2] |

Notes on the table:

- The paper's main correlation table uses 50 documents per query [arxiv:2404.13781 tab. 1]. Correlation falls as the list grows, for every method compared, so a longer list is not a free gain in signal [arxiv:2404.13781 §3.2] [arxiv:2404.13781 fig. 1].
- The efficiency gains depend on the batch size used. The paper compares each method at its largest feasible batch, so the gains do not carry over to other batch settings [arxiv:2404.13781 §3.2].
- Per-document calls scale with the number of retrieved documents, so the call count is k per query, even though each call is short [arxiv:2404.13781 §2].

## Implementation sketch

The common core is one reader call per rank position, a per-document score
against the gold output, and a list-level metric over those scores.
Pseudocode in our own words; the paper gives the same procedure in its
method section [arxiv:2404.13781 §2].

```pseudocode
# Input:  queries; ranked_docs[q] = the k documents for q, best first
#         gold[q] = expected output for q
# Output: one retrieval score for the list  [arxiv:2404.13781 §2]

def retrieval_score(queries, ranked_docs, gold, reader, metric, rank_metric):
    # One label list per query, filled rank by rank  [arxiv:2404.13781 §2]
    labels = {q: [] for q in queries}
    for rank in range(k):
        # Reader sees the query and one document only  [arxiv:2404.13781 §2]
        for q in queries:
            answer = reader(q, [ranked_docs[q][rank]])
            # Downstream metric in [0, 1] gives the graded label  [arxiv:2404.13781 §2]
            labels[q].append(metric(answer, gold[q]))
    # Ranking metric per query, then combine  [arxiv:2404.13781 §2]
    per_query = [rank_metric(labels[q]) for q in queries]
    return mean(per_query)  # combination step not stated in the paper
```

The paper does not state how per-query values are combined into one number.
The released scoring code takes an unweighted mean; see the card's "Paper vs
code" section for details [arxiv:2404.13781 §1].

## Contradictions

- **Tau gain range.** The text gives the gain in Kendall tau over the best
  baseline as 0.168 to 0.494 [arxiv:2404.13781 §1]. Taking each dataset's
  eRAG tau against the best baseline in the BM25 column of the table gives
  0.168 for NQ and 0.542 for FEVER, with values between them for the other
  datasets [arxiv:2404.13781 tab. 1]. The paper does not say which comparison
  its range comes from, so the upper bound cannot be checked against the table.
- **Memory ratios.** The text gives the query-level memory gain as 7 to 15 times
  and the document-level gain as 30 to 48 times [arxiv:2404.13781 §3.2]. The
  memory values in the efficiency table give about 7.4 to 16.2 and 31 to 50
  [arxiv:2404.13781 tab. 2]. The abstract's "up to 50 times" matches the
  table's largest document-level ratio [arxiv:2404.13781 §1].

## Papers

- [Evaluating Retrieval Quality in Retrieval-Augmented Generation](/papers/2404.13781): proposes scoring each retrieved document by the downstream metric of the RAG reader's answer to it alone, and reports better correlation with end-to-end results at lower cost.