wikillm

Paper · 2026-10-29 · cs.CR, cs.CL, cs.IR

Prevented is Not Protected: What Attack Success Rate Hides about RAG

Aligned RAG Evaluation scores each poisoned RAG query with and without a defense, against matched clean runs, to separate prevented attacks from target answers a defense introduces. On HotpotQA and NQ, defenses prevent some attacks while adding others, judged endorsement can reverse defense rankings, and score-gap detectors catch almost no attacks once every retrieved passage is poisoned.

Retrieval-augmented generationData poisoning (attacks on training data and sanitization defenses)LLM-as-a-judgeLLM evaluationrag-securitydata-poisoningdefense-evaluationllm-judge

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Some AI assistants look up documents before they answer. An attacker who can add documents to that store can slip in text that pushes the assistant toward a wrong answer the attacker picked. Defenses try to stop this. The usual score is attack success rate: how often the attack still works. This paper argues that the score hides two things. A defense can stop one wrong answer and cause a different wrong answer on the same questions. And the verdict on a defense can flip depending on how answers are checked. A reply that only mentions the attacker's answer, perhaps to reject it, is not the same as a reply that endorses it, and counting mentions can make a defense look worse or better than it is.

The fix is a matched comparison. For each question, the authors keep four answers: with and without the poisoned passages, and with and without the defense. They then remove the injected passages to see whether an answer depended on them, and whether one injected passage was enough. A split-knowledge probe checks whether an answer needs an injected passage together with real evidence from the corpus. The authors also find that detectors which flag suspicious retrieved passages work when only one passage is poisoned and fail when every retrieved passage is poisoned.

TL;DR

  • Aligned RAG Evaluation compares four matched answers per query (clean or poisoned retrieval, with or without a defense) and reports prevented and introduced target answers, not only attack success rate §2.1.
  • On HotpotQA with PoisonedRAG passages, a TrustRAG-style pipeline cut target answers from 96 to 32 at five poisons. At one poison it prevented 20 and introduced 20, so the string-match count stayed at 60/100, while judged endorsement fell from 35 to 11 (author-reported) §4.1 §4.2.
  • Retrieval score-gap detectors at a 1% false-positive rate caught 59 of 60 one-poison attacks on HotpotQA and none of 96 at five poisons (author-reported) §4.3.
  • Read this card as a checklist of evaluation pitfalls, not as a defense ranking. The defenses are adapted, several settings are unstated, and the released code has gaps (see Paper vs code).

Problem

Most RAG poisoning defense papers ask whether a defense stops the attacker's target answer §1. That view misses three things the paper names: a defense can block some poisoned answers while producing new target answers on other poisoned queries; it can change answers on clean queries; and it can make the model abstain §1. Attack success rate reports only prevented attacks, as P / r_p0 §2.1. Scoring by string match also counts a rejected mention as a success §4.2. The authors hold the question, the target, the answering model and the decoding settings fixed across all four conditions §2.1.

Method

Step 1, matched record §2.1 tab. 1. For each query q with target t, fix the target and the answering setup. Retrieve a clean context from the original corpus and a poisoned context from the corpus plus the injected passages. Generate answers for each context with and without the defense. Each answer gets a 0/1 target label T_sd, where s is clean (c) or poisoned (p) and d is defense off (0) or on (1). r_sd is the fraction of queries with T_sd = 1.

Step 2, transitions and net effect §2.1 eq. 1 eq. 2 eq. 3. Under poisoned retrieval, prevention P is the fraction of all queries that go from target-positive to target-negative. Introduction J goes the other way. The net target reduction is N = P − J = r_p0 − r_p1 eq. 1. The clean-side change is H = r_c1 − r_c0 eq. 2. The factorial contrast Γ = N + H eq. 3 measures how much the defense reduces the poisoning-induced increase relative to clean retrieval. A positive Γ can come from more clean target answers, so the paper always shows Γ next to N, H and the raw poisoned-target count §2.1.

N = P - J = r_{p0} - r_{p1} eq. 1

H = r_{c1} - r_{c0} eq. 2

\Gamma = (r_{p0} - r_{c0}) - (r_{p1} - r_{c1}) = N + H eq. 3

Step 3, retrieval record §2.2 app. D. Record the retrieved passages, the share of poisoned passages in the context, whether any real supporting passage made it into the context, and the answers under the removal controls. Keep queries where no poison was retrieved as well. Removal drops the chosen passages, keeps the others in their original order, and never fetches replacements §2.2.

Step 4, dependence labels §2.2. A poisoned answer is newly induced when the clean answer is target-negative and the poisoned answer is target-positive. It is injection-dependent when the target label disappears after all injected passages are removed. A poison-only control keeps only the injected passages. An individual-poison control keeps one injected passage at a time. Together they show whether the poison works as a group or whether one passage suffices §2.2 tab. 4.

Step 5, split-knowledge probe §2.2 tab. 6. Let A be the real supporting passages, B the injected ones, and H everything else that was retrieved. Four compositions are scored: A+B+H, A+H, B+H and H. A strict case is target-positive with A+B+H and target-negative with each of the other three. A strict newly induced case also has a target-negative clean answer §2.2.

Step 6, labels §3 §4.2 app. D. Two labels are used. The string label is a case-insensitive substring match against the target §4.1. The judge label is gpt-4o-mini, which returns yes, no or uncertain for target endorsement, reference correctness and abstention. The judge sees the question, the target, reference aliases and the answer, but not the method or condition app. D. Uncertain labels are reported separately and never counted as no §4.2.

Step 7, detectors §3 §4.3 app. D app. H. Six detector outputs are compared. Three come from graph structure over the retrieved passages (spectral connectivity, conductance, modularity). Three come from the retrieval scores alone: the lowest score, the gap between highest and lowest, and the spread. Each detector gets its own threshold, fitted on 500 clean queries at a nominal false-positive rate. A context is flagged when its score is strictly above the threshold. Flagged contexts are hard-abstained on app. D. Thresholds are set per dataset and encoding arm and shared across poison budgets app. D.

Pseudocode
# Build the four contexts of one query  [§3] [app. D]
# [code: aligned_rag/protocol.py#L54-L94]
def build_contexts(query, group_poisons, corpus, top_k=5):
    # Clean top-k from the unpoisoned corpus, dot-product score  [app. D]
    clean = search(corpus, query, top_k)
    # Score the injected passages of this query's group  [§3] [app. D]
    injected = score_each(group_poisons, query)
    # Poisoned top-k over clean and injected passages  [protocol.py#L93]
    poisoned = top_k_by_score(clean + injected, top_k)
    return clean, poisoned
Pseudocode
# Net effect of a defense over a query set  [§2.1] [eq. 1-3]
# Each query has four 0/1 labels: t_c0, t_c1, t_p0, t_p1  [tab. 1]
def aligned_summary(queries):
    # Target-positive rates per condition  [§2.1]
    r_c0 = mean(q["t_c0"] for q in queries)
    r_c1 = mean(q["t_c1"] for q in queries)
    r_p0 = mean(q["t_p0"] for q in queries)
    r_p1 = mean(q["t_p1"] for q in queries)
    # Prevented P and introduced J, as fractions of all queries  [§2.1]
    prevented = mean(q["t_p0"] and not q["t_p1"] for q in queries)
    introduced = mean(not q["t_p0"] and q["t_p1"] for q in queries)
    # Net target reduction N = P - J = r_p0 - r_p1  [eq. 1]
    net_reduction = prevented - introduced
    # Clean-side change H = r_c1 - r_c0  [eq. 2]
    clean_change = r_c1 - r_c0
    # Factorial contrast Gamma = N + H  [eq. 3]
    contrast = net_reduction + clean_change
    # ASR-style summary reports only P / r_p0  [§2.1]
    asr_style = prevented / r_p0 if r_p0 > 0 else None
    return net_reduction, clean_change, contrast, asr_style
Pseudocode
# Strict composition check for one query  [§2.2]
# [code: aligned_rag/analysis.py#L70-L71]
# labels[k] is 0 or 1 for composition k of A, B and H
def strict_case(labels, co_retrieved):
    # Target must appear with legitimate A and injected B together  [§2.2]
    if labels["A+B+H"] != 1:
        return False
    # Target-negative when A, B or both are removed  [§2.2]
    absent = ["A+H", "B+H", "H"]
    return all(labels[k] == 0 for k in absent) and co_retrieved
Pseudocode
# Retrieval-score detector with a clean-calibrated threshold  [§4.3] [app. D]
# [code: aligned_rag/defense_metrics.py#L12-L15]
def calibrate(clean_scores, fpr):
    # Threshold leaves about fpr of clean scores above it  [app. D]
    ordered = sorted(clean_scores)
    return ordered[ceil((1 - fpr) * len(ordered)) - 1]

def score_gap(retrieved_scores):
    # Highest minus lowest retrieval score in the top-k  [defense_suite.py#L152]
    return max(retrieved_scores) - min(retrieved_scores)

def is_flagged(context_score, threshold):
    # Strictly above threshold; flagged contexts are abstained on  [app. D]
    return context_score > threshold

Hyperparameters

NameValueWhereNotes
DatasetsHotpotQA and NQ, 100 PoisonedRAG queries each§3the split-knowledge probe uses 311 fixed HotpotQA candidates tab. 6
CorporaHotpotQA: full BEIR corpus, 5,233,329 passages; NQ: downloaded BEIR corpusapp. KNQ corpus fingerprint not documented; NQ support alignment not validated app. K
Retriever (main)Contriever, dot product§3 app. D
Retriever (split-knowledge)BGE-base-en-v1.5, cosine arm at 512 tokens§3 tab. 6
Top-k5§3 app. D protocol.pycode default is also 5
Passage max length (corpus)128 tokens (main arm)app. D models.py
Passage max length (queries and poisons)512 tokensapp. D models.py
Score functiondot product (main); cosine arm normalises embeddingsapp. D app. I models.py
Poison budget p1 and 5 injected passages per query group§4 app. Da group of 10 queries shares 10p injected passages app. D
Answering modelLlama-3-8B-Instruct (primary)§3
Larger victimLlama-3.1-70B-Instruct-AWQ, vLLM, 4-bit AWQ§3 app. Kvictim-scale checks reusing saved contexts; no full defense reruns app. K
Other familyMistral-7B-Instruct-v0.3§3 app. Bdetector-only check; pipeline defenses not rerun app. B
Decodingnot stated§2.1 §3code: greedy, see Paper vs code models.py
Max new tokensnot statednot statedcode default 150 models.py
Prompt templatenot statednot statedcode default is a short answer-from-documents prompt protocol.py
Pipeline defense: KeywordAggadapted from RobustRAG§3code thresholds: relative 0.3, absolute 3, abstention 1, corruption size 0 defense_backends.py defense_backends.py
Pipeline defense: TrustRAGfiltering plus conflict resolution, Llama only§3 app. Dfilter: k-means on CLS embeddings of a SimCSE BERT defense_backends.py
Pipeline defense: prompt controlprompt-only caution instruction, Llama only§3see Paper vs code
Pinned revisionsRobustRAG 9bc35b2, TrustRAG 11dcea0, PoisonedRAG f660d72app. Dadapters, not exact reproductions app. D
Detector calibration500 held-out clean queriesapp. D defense_suite.py
Detector held-out check200 clean queriesapp. D defense_suite.py
Nominal FPR1% main; 5%, 10%, 20% in sweepsapp. H defense_suite.pysweep points appear in the code, not in the paper text
Detector ruleflag if score strictly above thresholdapp. D defense_suite.py
Judge modelgpt-4o-mini-2024-07-18§3 app. D light_audit.py
Judge decodingnot stated in papernot statedcode: temperature 0, max 700 tokens light_audit.py
Bootstrapnonparametric, clustered by shared injection group§3 app. Dnumber of draws not stated; code uses 2,000 draws in one script and 10,000 in another
Intervals for rates95% Wilson§3 app. D
Intervals for paired effects95% bootstrap§3 app. D
HardwareNVIDIA L40S, 48 GB eachapp. E

Results

Author-reported. Setup: HotpotQA and NQ, PoisonedRAG passages, Contriever top-5 retrieval, Llama-3-8B-Instruct unless stated §3. Counts are out of 100 queries per cell.

SettingMetricUndefendedWith defenseSource
HotpotQA, p=1, TrustRAGtarget-positive, string6060 (20 prevented, 20 introduced)§4.1 tab. 2
HotpotQA, p=5, TrustRAGtarget-positive, string9632 (65 prevented, 1 introduced)§4.1 tab. 8
HotpotQA, p=5, KeywordAggtarget-positive, string9691tab. 8
HotpotQA, p=5, prompt controltarget-positive, string9694tab. 8
NQ, p=5, TrustRAGtarget-positive, string9214tab. 8
HotpotQA, p=1, TrustRAGjudged endorsement3511§4.2 tab. 10
NQ, p=1, KeywordAgg vs TrustRAGjudged endorsement20 (KeywordAgg)7 (TrustRAG)§4.2 tab. 10
HotpotQA, p=1, prompt controljudged endorsement3561§4.2 tab. 10
NQ, p=1, prompt controljudged endorsement4255§4.2 tab. 10
HotpotQA, p=1, prompt controlstring count6072§4.2 tab. 8

Net effect and clean side, one poison tab. 2. Intervals are 95% group bootstrap intervals.

DatasetPipelineNHΓMcNemar p
HotpotQA p=1KeywordAgg0 [−10, 9]−10 [−17, −3]−10 [−25, 3]1.000
HotpotQA p=1Prompt control−12 [−21, −4]−13 [−20, −6]−25 [−39, −13].029
HotpotQA p=1TrustRAG0 [−12, 13]12 [5, 21]12 [4, 22]1.000
NQ p=1KeywordAgg22 [15, 28]1 [0, 3]23 [16, 29]<.001
NQ p=1Prompt control−6 [−14, 1]−1 [−4, 2]−7 [−16, 1].263
NQ p=1TrustRAG18 [10, 26]9 [6, 12]27 [18, 36].002

Judged endorsement against TrustRAG, paired tab. 3. Negative differences mean TrustRAG has the lower judged endorsement rate (percentage points).

SettingvsQueries with decisive labelsDifference95% CI
HotpotQA p=1KeywordAgg79−26.6[−37.5, −15.4]
HotpotQA p=1Prompt80−53.8[−64.2, −42.5]
HotpotQA p=5KeywordAgg79−60.8[−74.3, −47.2]
HotpotQA p=5Prompt76−82.9[−93.1, −72.0]
NQ p=1KeywordAgg77−14.3[−24.3, −3.9]
NQ p=1Prompt78−55.1[−67.1, −43.9]
NQ p=5KeywordAgg79−54.4[−62.7, −47.0]
NQ p=5Prompt65−87.7[−94.1, −80.9]

Human audit of the judge tab. 11. 100 blinded outputs. Target endorsement: 69/100 agreement overall (κ = .39), 64/69 on decisive labels (κ = .74). Reference correctness: 50/98 overall (κ = .20), 50/68 decisive (κ = .40). Abstention: 67/99 overall (κ = .26), 67/71 decisive (κ = .72). Overall counts treat uncertain labels as disagreements tab. 11 §4.2.

Clean utility §4.2 app. C. The judge returned uncertain correctness for 147 of 200 TrustRAG answers on HotpotQA and 139 of 200 on NQ. The authors do not rank defenses on clean utility for this reason app. C.

Detectors at 1% FPR §4.3 tab. 14 app. H fig. 4.

DatasetPoisonsScore gap interceptedScore dispersion intercepted
HotpotQAp=159 of 6059 of 60
HotpotQAp=50 of 960 of 96
NQp=132 of 6436 of 64
NQp=51 of 921 of 92

Under a Mistral-7B victim, score-gap interception fell from 78 of 80 to 0 of 100 on HotpotQA and from 38 of 72 to 1 of 98 on NQ §4.3 app. B. The encoding controls keep the pattern: score gap 65 of 66 to 0 of 95 with 512-token dot-product encoding, and 61 of 63 to 0 of 97 with cosine app. I tab. 15.

Evidence dependence §4.4 tab. 4 tab. 5 tab. 6.

SettingPoisoned positivesNewly inducedSingle passage sufficient
HotpotQA, p=1, Llama-3-8B604657 of 60
HotpotQA, p=5, Llama-3-8B967996 of 96
NQ, p=1, Llama-3-8B646260 of 64
NQ, p=5, Llama-3-8B929091 of 92
HotpotQA, p=1, Llama-3.1-70B-AWQ57not stated54 of 57
HotpotQA, p=5, Llama-3.1-70B-AWQ95not stated95 of 95

Split-knowledge, 311 fixed HotpotQA candidates tab. 6: Contriever with Llama-3-8B gave 27 target-positive answers, 23 newly induced, 7 strict cases and 17 of 27 single-passage sufficient. BGE cosine (512) with Llama-3-8B gave 18 positives, 2 strict cases. Contriever with Llama-3.1-70B gave 18 positives and no strict cases §4.4.

Paper vs code

Official code: repository linked in the abstract, at commit b88ee78e201263598b101435c2376b22c24fca69. It is owned by the handle chxhana, which matches the first author's name. The repository has no LICENSE file; its pyproject.toml declares MIT. Nothing is quoted from it.

  • Strict case. Paper: target-positive with A+B+H, negative with the three subsets §2.2. Code also requires the legitimate and injected passages to be co-retrieved analysis.py.
  • Graph detectors. Paper names spectral connectivity, conductance and modularity §3 app. D. Code imports them from a file asr_risk_reduction.py that is not in the repository, so they cannot be checked. Code uses all-mpnet-base-v2 embeddings with 4 graph neighbours [code: aligned_rag/defense_suite.py#L137-L151].
  • Score detectors. Paper: minimum score, score gap and score dispersion §3. Code: minimum score is the negated lowest retrieval score; gap is the highest minus lowest; dispersion is the population standard deviation defense_suite.py. Orientation is fixed as "higher is more anomalous" before testing defense_suite.py.
  • Prompt control. Paper: an added instruction to be cautious about unreliable evidence §3. Code: uses the qa template of the RobustRAG repository defense_backends.py [code: aligned_rag/defense_backends.py#L86]. That template file is not in this repository, so the wording cannot be verified here.
  • Bootstrap draws. Paper: nonparametric bootstrap, count not stated §3. Code: 2,000 draws with seed 42 for the factorial estimates [code: aligned_rag/factorial.py#L174-L181]; 10,000 draws in the paired-inference script paired_inference.py. Which one produced the paper's intervals is not stated.
  • String label. Paper: case-insensitive substring §4.1. Code: lower-cases, strips one trailing period, then checks for the target as a substring protocol.py. The report refuses substring labels unless the user passes an explicit legacy flag, and tags their output as not endorsement defense_suite.py [code: aligned_rag/defense_suite.py#L280].
  • Decoding and length. Paper: decoding fixed but not given §2.1 app. D. Code: greedy decoding with a 150-token limit for the answering model [code: aligned_rag/models.py#L151] models.py. Pipeline internal calls allow 4,096 tokens defense_backends.py. Prompts over 8,192 tokens raise an error rather than being truncated [code: aligned_rag/models.py#L182-L183].
  • Answer prompt. Paper does not say which template the main runs used. Code has a short default protocol.py and an optional switch to PoisonedRAG's official prompt cli.py. The generalisation script checks for the official template [code: run_generalization.py#L66].
  • Judge settings. Paper: gpt-4o-mini-2024-07-18, decoding not stated §3 app. D. Code: temperature 0 and at most 700 output tokens [code: aligned_rag/light_audit.py#L75] light_audit.py. The judge rubric is defined in the same file light_audit.py.
  • KeywordAgg thresholds. Not in the paper. Code uses relative 0.3, absolute 3, abstention 1 and corruption size 0 [code: aligned_rag/defense_backends.py#L76] defense_backends.py.
  • TrustRAG filter. Paper: "filtering" §3. Code: k-means over CLS embeddings from a SimCSE BERT model, with n-gram filtering [code: aligned_rag/defense_backends.py#L103-L113].
  • No discrepancy was found in the detector flag rule (strictly above a threshold calibrated on clean scores) defense_suite.py.
  • I did not recompute the human-audit agreement from the completed audit file.

Pitfalls

  • Decoding, token budget and prompt are unstated in the paper. Pin them yourself, and note the code default of greedy decoding with 150 new tokens.
  • Substring labels are not endorsement. The paper's own argument is that a mention can be a rejection. Report the judge label, and keep uncertain labels as a separate count. Uncertain labels are not "safe" either §4.2.
  • Judge agreement is weak on some labels. Reference correctness agrees at κ = .20 overall and .40 on decisive labels. Do not use it to rank defenses on utility tab. 11 app. C.
  • Fixed thresholds across budgets. Detector thresholds are calibrated once per dataset and encoding arm and reused for all poison budgets app. D. The collapse at five poisons is therefore not a recalibration artefact app. H.
  • At five poisons every top-5 passage is injected app. H. A detector has no clean-versus-poison contrast left to compare against.
  • Groups of ten queries share poisons. Intervals must cluster by group; naive per-query intervals are too narrow § Limitations app. D.
  • Pipeline numbers were corrected. An early version of the adapter kept only the opening line of each KeywordAgg answer, which truncated those outputs. After recovery, HotpotQA KeywordAgg counts moved from 23 and 26 to 60 and 91 at one and five poisons. Any copy of the earlier numbers is wrong app. J.
  • Single-passage sufficiency is not universal. The text says a single passage suffices for all 95 HotpotQA positives with Llama-3.1-70B. That figure is the five-poison cell; the one-poison cell gives 54 of 57 tab. 5.
  • Mistral "at least 94%" is not traced. The claim for the Mistral settings §4.4 has no matching number in the appendix, which says "nearly all" app. B.
  • Strict composition is rare. Only 7 of 27 split-knowledge positives are strict, and none appear with the 70B victim §4.4. Do not read the probe as a general effect.
  • Undefined columns. The text does not define "Clean EM" in the split-knowledge table tab. 6, or the "Correct lost" and "Correct retained" columns of the clean-utility table tab. 7. The defense table's "Hit" and "Strict" columns are read here as cases suppressed, which fits the text and the arithmetic tab. 13.
  • Dangling reference. The appendix refers to "Finding 1", which is not in the main text app. E app. I.
  • Test not described. The table's McNemar p-values are not explained in the text tab. 2.
  • NQ provenance. The NQ corpus fingerprint is not documented and NQ support alignment was not validated app. K.
  • Pipeline adapters are not the originals. Shipped code uses adapted KeywordAgg, TrustRAG and prompt control on a common victim interface. The paper itself warns that these are not exact reproductions app. D § Limitations.

Lineage

  • Builds on: PoisonedRAG (the attack; passages at pinned revision f660d72) §3 app. D. Not in this wiki.
  • Compared against: RobustRAG's KeywordAgg and a prompt-only control; TrustRAG filtering with conflict resolution §3. Also the detector scores above.
  • Related card: 2610.33849, on query-time filters for RAG poisoning and how benchmark construction flatters them. Different paper, same threat model.
  • Superseded by: not stated.

Source and license

Adapted from “Prevented is Not Protected: What Attack Success Rate Hides about RAG” by Chahana Dahal, Zuobin Xiong (arXiv:2610.40742), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/chxhana/aligned-rag-eval at commit b88ee78e2012 (not stated (no LICENSE file; pyproject.toml declares MIT)). Any quoted code is shown under that license.