Retrieval evaluation

Overview

A retrieval system is usually judged by how well its ranked list matches relevance labels. Those labels describe what a person or a judge finds relevant, which need not be what a downstream language model can use. The direct test is end-to-end: run the whole pipeline and compare the final outputs. That is slow and memory-hungry, and it says little about which retrieved document helped 2404.13781 §1.

The shared idea of this concept is to score each retrieved document by its effect on the downstream output, then turn those scores into a list-level number with a standard ranking metric. The variants differ in how they get the per-document scores and in the metric applied to them. Only one card so far covers this concept.

Variants

Downstream-answer labels (eRAG)

Each retrieved document is given to the RAG system's own language model alone, together with the query. The answer is scored against the gold output by the downstream metric, and that score is the document's relevance label. The labels for a ranked list then go to a set-based or ranking metric such as precision, recall, MAP, MRR, NDCG or hit rate 2404.13781 §2. With non-binary labels the paper names precision (a mean) and hit rate (a maximum) as the usable options 2404.13781 §2. Card: Evaluating Retrieval Quality in Retrieval-Augmented Generation.

Its reported correlation with downstream RAG performance is higher than that of the KILT provenance and LLM-annotation labels, on five KILT datasets with BM25 retrieval 2404.13781 §3.2 2404.13781 tab. 1. Its reported cost is lower than end-to-end evaluation, with up to about 50 times less GPU memory at document level 2404.13781 §3.2 2404.13781 tab. 2.

When to use which

Notes on the table:

• The paper's main correlation table uses 50 documents per query 2404.13781 tab. 1. Correlation falls as the list grows, for every method compared, so a longer list is not a free gain in signal 2404.13781 §3.2 2404.13781 fig. 1.
• The efficiency gains depend on the batch size used. The paper compares each method at its largest feasible batch, so the gains do not carry over to other batch settings 2404.13781 §3.2.
• Per-document calls scale with the number of retrieved documents, so the call count is k per query, even though each call is short 2404.13781 §2.

Implementation sketch

The common core is one reader call per rank position, a per-document score against the gold output, and a list-level metric over those scores. Pseudocode in our own words; the paper gives the same procedure in its method section 2404.13781 §2.

# Input:  queries; ranked_docs[q] = the k documents for q, best first
#         gold[q] = expected output for q
# Output: one retrieval score for the list  [arxiv:2404.13781 §2]

def retrieval_score(queries, ranked_docs, gold, reader, metric, rank_metric):
    # One label list per query, filled rank by rank  [arxiv:2404.13781 §2]
    labels = {q: [] for q in queries}
    for rank in range(k):
        # Reader sees the query and one document only  [arxiv:2404.13781 §2]
        for q in queries:
            answer = reader(q, [ranked_docs[q][rank]])
            # Downstream metric in [0, 1] gives the graded label  [arxiv:2404.13781 §2]
            labels[q].append(metric(answer, gold[q]))
    # Ranking metric per query, then combine  [arxiv:2404.13781 §2]
    per_query = [rank_metric(labels[q]) for q in queries]
    return mean(per_query)  # combination step not stated in the paper

The paper does not state how per-query values are combined into one number. The released scoring code takes an unweighted mean; see the card's "Paper vs code" section for details 2404.13781 §1.

Contradictions

• Tau gain range. The text gives the gain in Kendall tau over the best baseline as 0.168 to 0.494 2404.13781 §1. Taking each dataset's eRAG tau against the best baseline in the BM25 column of the table gives 0.168 for NQ and 0.542 for FEVER, with values between them for the other datasets 2404.13781 tab. 1. The paper does not say which comparison its range comes from, so the upper bound cannot be checked against the table.
• Memory ratios. The text gives the query-level memory gain as 7 to 15 times and the document-level gain as 30 to 48 times 2404.13781 §3.2. The memory values in the efficiency table give about 7.4 to 16.2 and 31 to 50 2404.13781 tab. 2. The abstract's "up to 50 times" matches the table's largest document-level ratio 2404.13781 §1.

Papers

• Evaluating Retrieval Quality in Retrieval-Augmented Generation: proposes scoring each retrieved document by the downstream metric of the RAG reader's answer to it alone, and reports better correlation with end-to-end results at lower cost.