Paper · 2026-10-24 · cs.CR, cs.CL, cs.LG
Ranking Is Not Deciding: Benchmark Shortcuts and Operating-Point Collapse in Query-Time RAG Poisoning Defenses
Query-time filters that screen retrieved documents for RAG poisoning look near-perfect on benchmarks, but the poison's construction explains much of it: PoisonedRAG's code prepends the target question to each passage. Deleting that prefix cuts RAGuard's AUROC from 0.953 to 0.758 and raises its miss rate from 0.191 to 0.763. A label-free two-number profile predicts saturation at corpus level.
Retrieval-augmented generationData poisoning (attacks on training data and sanitization defenses)Decision threshold tuningBenchmark validityrag-securitydata-poisoningbenchmark-validityoperating-pointevaluation
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Some AI assistants look up documents before they answer. An attacker who can slip made-up documents into that pool can push the assistant toward a false answer. Many defenses sit in front of the assistant and decide, one document at a time, whether each document may pass. Those defenses often score almost perfectly on the test sets published alongside them.
This paper argues that such scores can reflect how the test attacks were built, not how well the defense works. In the released attack code the paper examines, every attack document begins with the exact question it answers. A filter that picks up on that pattern looks excellent. When the pattern is removed, one published filter misses most attacks. Two simple numbers, measured from the attack documents and the clean documents, can flag a test that is too easy: how often the question appears word for word in the attack documents, and whether the attack documents have the same lengths as normal ones.
The practical lesson is to report a detection rate together with how the attack set was built and with the false-alarm rate at the threshold actually used. A filter can rank documents well while its yes-or-no decision fails. The authors propose no new defense, and they show that their own learned filter can be beaten by an attacker who knows how it works.
TL;DR
- Filters applied at query time, which check each retrieved document before generation, are the leading published defense against RAG poisoning, and their authors report almost flawless detection on benchmarks §I.
- The paper argues this performance follows from how the poison was built: the attack's released code writes the target question in front of each passage it inserts §I §V-A.
- Removing only that prefix from the attack authors' passages drops RAGuard's AUROC from 0.953 to 0.758 and raises its miss rate from 0.191 to 0.763 (author-reported) §VI-A.
- The paper proposes a label-free construction profile (containment and length overlap) and six reporting rules, and claims no defense §V §VII.
- Read this if you evaluate or deploy a per-document poison filter for retrieval. Check the Pitfalls before reusing any number.
Problem
Retrieval-augmented generation (RAG) places retrieved documents into the model's context, so anyone who can insert text into the corpus can steer answers for chosen questions without touching the weights §I. The published response is a query-time admission gate: a per-document filter that decides which retrieved documents may reach the generator §I §III-A. The paper's concern is that these filters are evaluated on poison sets whose construction is not reported, so a near-perfect detection rate cannot be interpreted. A ranking metric such as AUROC can stay high while the deployed yes-or-no decision fails §VI-C.
The paper's scope and assumptions: poison is text inserted into the corpus without gradients or defender queries for the main ladder §III-C; measurements cover 4 corpora and 4 attack families, with 10 defenses that each act per document, all scored on a single candidate set §IV-A; the adaptive attacker is a single query-based attacker run against the paper's own scorer §III-C §VI-B.
Method
The paper contributes three things. First, two label-free statistics of a poison set, measured against the clean corpus: containment (how often the question is copied word for word into the poison) and length overlap (how many poison documents fall inside the clean length band) §V-A. Second, a control ladder that removes one of these cues at a time while holding the payload and attack family fixed §V-B. Third, a measurement harness in which ten defenses and the paper's learned scorer score the same candidates, plus a cost-sensitive decision rule used as an instrument rather than a contribution §IV.
Construction profile
A poison document counts as contained when, after normalisation, the target question appears as one unbroken run of text inside it. Normalisation lower-cases the text, drops punctuation and merges runs of spaces §V-A. The length band is the fifth to ninety-fifth percentile of word counts in the clean corpus. The clean spread (the ninety-fifth percentile over the fifth) is printed beside the overlap, because a spread of 1.0 means the clean corpus has no length variance and any length cue is free §V-A.
# Construction profile of a poison set [§V-A]
# Input: poison: list of (text, target_question); clean: list of texts
# Output: containment, length_overlap, clean_spread
def normalise(text):
# Lower-case, strip punctuation, collapse whitespace [§V-A]
return collapsed_words(strip_punct(lower(text)))
def profile(poison, clean):
# Containment: fraction of poison that contains its target question
# verbatim after normalisation [§V-A]
contained = [
normalise(question) in normalise(text)
for text, question in poison
]
containment = mean(contained)
# Clean word-count band [l5, l95] [§V-A]
clean_counts = [word_count(text) for text in clean]
low = percentile(clean_counts, 5)
high = percentile(clean_counts, 95)
# Length overlap: fraction of poison inside the clean band [§V-A]
inside = [
low <= word_count(text) <= high
for text, _ in poison
]
length_overlap = mean(inside)
# Clean spread l95 / l5; 1.0 means no length variance [§V-A]
clean_spread = high / low
return containment, length_overlap, clean_spreadThe paper states the consequence as a rule: a defense that reports near-perfect detection on a poison set with containment 1.00, or with poison lengths outside the clean band, has shown no evidence that it uses anything beyond those cues §V-A.
Control ladder
Each rung changes one construction property and keeps the payload and attack family fixed §V-B §III-C:
- Template: the generator as published, with no changes.
- Prefix removed: each exact copy of the target question is cut out and the remaining query-word overlap is limited, so containment drops to zero. On NQ the test uses the original authors' passages and strips only the prefix their code adds §V-B.
- Length matched: benign filler from the clean corpus is added until the word count lies inside the clean band. Two placebo fillers of the same length separate length from content: clean sentences whose words are reordered across the pool, and invented words of matching length §V-B §VI-B.
- Both: prefix removed and length matched.
- Mimicry: a real domain passage is edited so that only the target fact changes, and then length matched §V-B.
To count as a success, a rung must clear three checks at once. The edited document has to still come back in retrieval once the corpus is re-embedded. Its content has to keep the attacker's claim when compared with the original text. And the gate has to let it through. Retrieval is therefore measured at every rung §III-C §V-B.
Cost-sensitive admission rule
Each candidate has a source identifier, a latent label, and a calibrated score estimating the probability of poison. The harm to minimise sums three terms over the retrieval stream: a missed poison costs C_FN, a false block costs C_FP, and an escalation costs C_A eq. 1 §IV-C. Only the ratio rho = C_FN / C_FP and, when escalation is used, c_A = C_A / C_FP enter the rule §IV-C.
With c_A unset and calibrated scores, the harm-minimising decision blocks a document when its probability reaches tau = 1 / (1 + rho) eq. 2 (Proposition 1, which attributes the rule to Elkan) §IV-C. The paper's trust term multiplies the score's prior odds by kappa_s, clipped to [0.25, 20]. Under the assumption that the score is calibrated, the decision is again a single threshold on the raw score, tau_s = 1 / (1 + rho * kappa_s) eq. 3 §IV-C §IV-D. With kappa_s = 1 this is the same as tau_d §IV-C.
# Cost-sensitive admission of one candidate [eq. 2] [eq. 3]
# Input: score (calibrated P(poison)), rho (C_FN / C_FP), kappa (trust)
# Output: "BLOCK" or "PASS"
def admit(score, rho, kappa=1.0):
# Clip the source's prior-odds multiplier to [0.25, 20] [§IV-D]
kappa = min(max(kappa, 0.25), 20.0)
# Single threshold on the raw score; kappa = 1 gives tau_d [eq. 3]
threshold = 1.0 / (1.0 + rho * kappa)
if score >= threshold:
return "BLOCK"
return "PASS"The paper's action set also includes ESCALATE, which is used only when c_A is set. No main-table cell sets c_A, so the escalation branch is not exercised in the results §IV-C.
Trust state (per source)
Each source has a Beta-distributed belief, updated from adjudicated outcomes and tied to a verified identity. The belief's mean serves as the poison-rate estimate, and the prior-odds multiplier is taken relative to the base rate pi_0 §IV-D app. A. Each adjudicated event decays both counts, and only the matching count gains the event weight app. A. Forgetting factors are lambda_a = 0.995 and lambda_b = 0.98 in the defaults app. A. A source starts at kappa = 1 §IV-D.
# Per-source Beta trust state [§IV-D] [app. A]
# Input: state (a, b), label of the adjudicated event, weight w
# Output: updated (a, b) and the clipped prior-odds multiplier
def update_trust(a, b, poisoned, w=1.0, lam_a=0.995, lam_b=0.98):
# Every event decays both counts; the matching one gains w [app. A]
a = lam_a * a + (w if poisoned else 0.0)
b = lam_b * b + (0.0 if poisoned else w)
return a, b
def prior_odds_multiplier(a, b, base_rate):
# Posterior mean poison rate of the source [app. A]
pi = a / (a + b)
# Odds of the posterior mean relative to the base rate pi_0 [§IV-D]
kappa = pi * (1.0 - base_rate) / ((1.0 - pi) * base_rate)
# Clip to [0.25, 20] [§IV-D]
return min(max(kappa, 0.25), 20.0)The learned scorer
Features come in four families: the near-verbatim query containment cue that the profile measures, the log document length, embedding and word-level features, and features that describe the candidate set as a whole. The defender computes all of them from the texts, their embeddings and the retrieval geometry, and no self-declared source metadata is an input §IV-D §III-B. The model is a multilayer perceptron with trunk [512, 256, 128] over bags of (query, candidate set), temperature scaled. For ablation, a boosted-tree model replaces the MLP on identical inputs §IV-D §VI-D. The body does not state the training optimiser, learning rate, epoch count, batch size or dropout.
Domain-regime scorers are trained on one half of the paper's skeleton templates and evaluated on the other. The general-regime scorer is trained only on HotpotQA and MS MARCO poison, so it has seen neither NQ corpus nor its generator §V-B.
Defenses measured
The paper re-implements seven of the ten defenses and runs the released code for two of them, on the poison as released §VIII §VI-A. Table I lists the signals and operating-point rules §II-B.
| Defense | Signal | Operating-point rule |
|---|---|---|
| Perplexity (GPT-2) | average per-token negative log-likelihood under GPT-2 | 99th percentile of clean top-k values |
| RAGuard (Cheng et al.) | gap in negative log-likelihood between chunk halves; similarity to the query | tails at 2.5% of clean scores, combined over three tests by logical OR |
| DRS | displacement along directions of low clean variance | 99th percentile of clean top-50 scores on a held-out half |
| TrustRAG stage 1 | k-means cluster cohesion; ROUGE-L between passages | cutoffs fixed at 0.88 and 0.25 |
| RAGDefender | Ward cluster size; TF-IDF; concentration | drop the N_adv closest |
| NLI consistency | cross-encoder contradiction graph | fixed edge at beta = 0.5 |
| Dup-only | exact and cosine near-duplicate | fixed cosine 0.95 |
| Learned scorer, tau = 0.5 | calibrated posterior | fixed 0.5, harm-blind |
| Learned scorer, tau_d | same posterior | 1 / (1 + rho_d) |
| Learned scorer, tau_s | same posterior, trust-adjusted | 1 / (1 + rho_d * kappa_s) |
§II-B (Table I). Rows for the learned scorer are one score at three thresholds.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Corpora | NQ 24,295; medical 21,000; financial 24,991; legal 33,530 documents | §IV-A | Medical is PubMedQA; financial is FinQA and TAT-QA; legal is CUAD chunked to 150 words |
| Target questions | 100 per corpus, split 40 / 20 / 40 | §IV-A | Scorer-train, calibration and evaluation; each cell evaluated on 40 queries |
| Retriever | all-MiniLM-L6-v2, 384 dimensions, no reranker | §IV-A | |
| Candidates | k = 10 per query; 400 per cell and seed | §IV-A | |
| Poison families | PoisonedRAG N = 5; CorruptRAG-AS N = 1; HijackRAG-style; jamming blocker | §IV-A | |
| Poison rate | 0.0147 to 0.0233 at N = 5 | §IV-A | Realised corpus rate |
| Base rate | scorer calibrated at pi_0 = 0.343; deployed at 0.090 to 0.746 | §IV-A | Prevalence on the training bags vs the evaluation cells |
| Seeds | 11, 22, 33, 44, 55 | §IV-B | Per-seed sd is not reported alone |
| Bootstrap | paired cluster bootstrap over 40 query clusters, 1000 resamples | §IV-B | On differences against a named comparator |
| ECE | 15 equal-width bins | §IV-B | Reliability diagram in §VI-C uses equal-mass bins |
| Harm ratio rho_d | illustrative values 1, 5, 10, 20 | §IV-C | Governance input, swept in the appendix |
| Trust clip | kappa in [0.25, 20] | §IV-D | |
| Trust forgetting | lambda_a = 0.995, lambda_b = 0.98 | app. A | Defaults; asymmetric |
| Trust initial state | kappa = 1 | §IV-D | |
| Trust labels a = 0.3, a = 0.1 | labels in the tables | §VI-A app. B | The body does not define a |
| Scorer trunk | MLP [512, 256, 128] | §IV-D | Training optimiser, learning rate, epochs, batch size and dropout: not stated |
| Temperature scaling | yes | §IV-D | Data used to fit the temperature: not stated |
| Perplexity | GPT-2; p99 of clean top-k NLL; at most 200 calibration documents | §II-B | |
| RAGuard | nominal FPR 10%; 2.5% clean tail | §II-B | OR of three tests; union bound on the nominal rate |
| DRS | fit on the top-50 clean candidates of the 20 calibration queries; threshold on a held-out half | §IV-A | Algorithm 2 as specified |
| Adaptive attacker | 500 oracle queries and 60 iterations per document | app. B | Tuned to the learned scorer on NQ |
| Hardware | AMD Ryzen 7 5700G, CPU only, PyTorch 2.14.0 | § Open Science | 35.8 CPU-hours for the main lanes |
Results
All results are author-reported and are point estimates over five seeds unless noted. Setups: NQ (the PoisonedRAG passages), medical, financial and legal corpora, 40 evaluation queries per cell, top-10 candidates. "Baseline" is the comparator named in the Setting column.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| NQ, RAGuard: question prefix kept vs removed | AUROC | 0.953 (prefix kept) | 0.758 (prefix removed) | §VI-A |
| NQ, RAGuard: same comparison | FNR at deployed point | 0.191 (prefix kept) | 0.763 (prefix removed) | §VI-A |
| NQ, no defense: retrieval survival | fraction of poison retrieved | 0.99 (prefix kept) | 0.93 (prefix removed) | §VI-A |
| NQ, perplexity filter: prefix kept vs removed | AUROC | 0.773 | 0.729 | app. B |
| Medical, TrustRAG stage 1: template vs shuffled filler | FNR at deployed point | 0.000 (template) | 0.827 (shuffled) | §VI-B |
| Medical, learned scorer, template vs shuffled filler | FNR at deployed point | 0.000 (template) | 0.986 | §VI-B |
| Medical, learned scorer, shuffled filler | FNR at matched 5% FPR; AUROC | not applicable | 0.321; AUROC 0.953 | §VI-B |
| Medical, learned scorer, shuffled filler | ECE at realised prevalence | not applicable | 0.471 | §VI-C |
| Medical, learned scorer, template vs shuffled filler | attack success under the conjunction | 0.00 | 0.94 | §VI-B |
| Medical, DRS, template | FNR at deployed point (FPR 0.035); peak AUROC | not applicable | 0.700; 0.882 | §VI-A |
| Medical, length matched (clean filler): learned scorer, cross-fitted floor | FNR at deployed point | 0.983 | 0.750 at +1.47 pp FPR | §VI-D |
| Medical, length matched: deleting the document-length feature | FNR at deployed point; AUROC | 0.986; 0.885 | 0.204; 0.992 (3 seeds) | §VI-D |
| NQ, learned scorer, template to mimicry plus length match | FNR; AUROC | 0.118; 0.990 | 0.948; 0.518 | §VI-B |
| NQ, learned scorer tuned by adaptive attacker | FNR at deployed point (static to adaptive) | 0.030 | 0.977 | app. B |
| Realised vs nominal FPR, mean over four corpora | RAGuard (nominal 10%); DRS (nominal 1%) | 10%; 1% | 0.098; 0.103 | §II-B |
| Surveyed published defense evaluations (21) | prefix in the attacks; length cue; no FPR at the deployed point | not applicable | 8; 5; 12 | §II-E |
| Profile direction at single points | points where the direction is wrong | not applicable | 40 of 90 | §V-B |
| Decision agreement across 26 cells | cells where the three rules give identical actions | not applicable | 12 (5 of them with rho = 1) | §VI-E |
Verdicts stated by the paper §VI-A §VI-B §VI-C §VI-E:
- The profile predicts which corpora saturate. It does not predict an individual defense, and the paper calls it a hypothesis with four observations §VI-A.
- The load-bearing cue depends on the corpus. For the learned scorer, removing the prefix moves the deployed miss rate by +0.421 on financial and +0.009 on medical. Matching length moves it by +0.983 on medical and +0.036 on legal, and by +0.014 on financial §VI-B.
- AUROC cannot separate calibration collapse (ranking usable, deployed point dead) from signal collapse (ordering at chance) §VI-C.
- Across the 26 cells, the three decision rules agree on every candidate in 12 of them. Operating-point claims on those cells therefore cannot be tested there §VI-E.
The paper's own TrustRAG and RAGDefender runs on the released code (not re-implemented) show a related pattern: with the prefix, TrustRAG's top-5 context is empty on 84 of 100 queries, and without it on 15 §VI-A.
Paper vs code
No official code.
The paper gives no code link. The Open Science section says an anonymised
repository would be linked from the submission form and that a Zenodo deposit with a DOI
follows acceptance; neither link appears in this version. Searches: the arXiv
text and source contain no repository URL; no project page is linked; GitHub
repository searches for "ranking not deciding rag poisoning" and "poisonedrag
containment prefix" returned no results. Nothing was cloned, and no code or
code_commit is recorded. Recheck when the release is linked.
The paper's released-code runs use third-party repositories (PoisonedRAG at commit f660d72, TrustRAG at 11dcea0, RAGDefender at ba2a17e) §VI-A. They are not this paper's code and are not used here.
Pitfalls
- Near-zero miss rates that came from a port defect. An earlier draft's DRS port used a clean sample of 199 documents, which is less than the embedding size of 384. That left the covariance rank-deficient, the threshold sat on the wrong directions, and most of the benign corpus was blocked. Use only the corrected DRS numbers. What matters is how many calibration documents there are per embedding dimension §VIII.
- Small calibration sets. Every fitted threshold other than DRS is a quantile of at most 200 clean documents. A 99th percentile is then the second-largest value. The calibration split is fixed across seeds, so per-seed spread does not contain this variance §VIII.
- Withdrawn NLL headline. Perplexity and RAGuard collapse under length-matched clean-prose filler (FNR 0.361 and 0.519 on medical). Under word-shuffled filler of the same length they recover to 0.000. A mean-token likelihood is set by what it is padded with, so do not use this collapse as a result §VI-B app. B.
- Trust results are not detection results. Source identity is an exact label indicator in these experiments (provenance AUROC 1.000). An earlier draft's trust-on ablation therefore re-ranked on a leaked label, and it was withdrawn. The reported ablation is trust-off §IV-A §VI-D.
- Identical decisions can be by construction. When rho = 1, the cost-derived rule equals the tau = 0.5 classifier. Five of the twelve identical-decision cells have rho = 1. The paper flags these and counts none as evidence §VI-E.
- Undefined trust labels. The tables label rows a = 0.3 and a = 0.1. The body does not define a, so the card cannot say what those settings change.
- Scorer training is under-specified. The optimiser, learning rate, epochs, batch size, dropout and temperature-fitting data are not stated. A reimplementation must choose them and should say so.
- Name collision. In this paper RAGuard means the Cheng et al. query-time chunk filter. RAGuard-ZKIP, cited as Kumar et al. 2026, is a different defense that exposes no per-document decision and is not a baseline §II-B.
- Nominal and realised FPR differ. RAGuard was nominally 10% and realised 0.098 on average; DRS was nominally 1% and realised 0.103 on average, but 0.035 on medical. Compare miss rates only at the realised false-positive rate §II-B.
- Length cues on fixed-width corpora. On legal, the clean corpus has a width of 1.00 (no variance), so padding overshoots and creates a reversed signal rather than removing one. Padded legal poison is also expensive to retrieve (survival 0.757) §VI-B.
- Scale. Each cell has 40 evaluation queries, one retriever, no reranker, one poison rate and one trust setting. Retrieval survival would be lower at PoisonedRAG's scale, which the paper names as the attacker's problem rather than the filter's evidence §VI-A §VIII.
- Adaptive numbers are lower bounds. The adaptive attacker is tuned to the paper's own scorer and then transferred unchanged to the other defenses, with 500 oracle queries and 60 iterations per document. No LLM-guided or human-in-the-loop attacker was run §VIII app. B.
- Retrieval survival. Survival of the prefix-removed NQ passages is 0.93 with no defense. On medical with both cues removed, survival is 0.18, so the miss rate there is computed over the 18% of poison that survives retrieval, and the attack success under the full conjunction is 0.18 §VI-B.
- Generator-level harm is secondary. The end-to-end attack-success run uses a 0.5-billion-parameter generator with substring matching. The paper makes no claim that depends on it §VIII.
Lineage
- Builds on: PoisonedRAG (Zou et al.), whose released code prepends the target question to inserted passages; CorruptRAG-AS (Zhang et al.); HijackRAG-style injection (Zhang et al., 2024); the jamming blocker (Shafran et al.); the defenses in Table I: RAGuard (Cheng et al.), DRS (Xian et al.), TrustRAG stage 1 (Zhou et al.), RAGDefender (Kim et al.), NLI consistency (Shen et al.), and perplexity filtering (Jain et al.) §II-B.
- Decision layer: cost-sensitive learning (Elkan), the reject option (Chow), temperature scaling (Guo et al.), and expected calibration error (Naeini et al.). The paper states that each ingredient predates it §II-C.
- Compared against: the ten defenses of Table I on one shared candidate set §IV-A §II-B.
- Concurrent work named in the paper: TriShieldRAG and RAGSieve, the query-inclusion artefact in CamoDocs and related attacks, and an independent re-run of PoisonedRAG by another group §V-A §II-E.
- Superseded by: not stated.
Source and license
Adapted from “Ranking Is Not Deciding: Benchmark Shortcuts and Operating-Point Collapse in Query-Time RAG Poisoning Defenses” by Pushkal Kumar, Arvin Sahaym (arXiv:2610.33849), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.