VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation

In plain words

A vision-language model is often shown several page images retrieved for one question. Models in this setting tend to make things up: they miss the relevant detail on one page, or they read something that is not there. EVisRAG asks the model to work in visible steps. It first looks at each image in turn and writes down what matters for the question, and it says so explicitly when an image has nothing relevant. Only then does it reason from those written notes and give an answer.

Training uses reinforcement learning, but the scoring is split into parts. One score checks whether the notes match the reference evidence. One checks whether the answer follows from the notes. One checks that the response has the required layout. Each score is applied only to the part of the response it judges, not to the whole response at once. The authors argue that this keeps the signals from pulling against each other.

The authors report that their 7B model beats its own base model and several larger models on five question-answering sets. All numbers are theirs, and the paper's own wording of its headline gains does not match its tables (see Pitfalls).

TL;DR

EVisRAG is a tagged, single-response answer pipeline for Qwen2.5-VL, run over the top-3 pages from the VisRAG retriever. It is trained with supervised fine-tuning on 60,000 teacher-written trajectories, then with RS-GRPO on 4,000 prompts. The paper reports 75.50 average accuracy for its 7B model against 54.23 for the backbone. The released code uses six reward channels where the paper describes three, and several training values differ from the paper's tables.

Problem

Visual retrieval-augmented generation returns page images to a vision-language model (VLM). With several images, the VLM often overlooks the relevant evidence or reasons about content that is not there §1. Existing remedies add tools or agents, which raise complexity, or train perception and reasoning with mixed rewards, which the authors say blurs credit and causes interference §1. The task also includes questions whose retrieved context is insufficient, where the model should abstain §4.

Method

Evidence-guided pipeline

EVisRAG answers a question in four stages: observe the images, record evidence per image, reason, then answer §3.1 fig. 2. The stages are written as conditional probabilities over the output tokens. Retrieval first selects the top-k pages with VisRAG-Ret eq. 2. The observation stage produces a page-aware note eq. 3. The evidence stage writes one record per page, conditioned on the question, the observation and that page eq. 4. The reasoning stage eq. 5 and the answer stage eq. 6 follow.

A record for an image either contains the evidence found or states that there is no relevant information §3.1 app. A.3. The released inference prompt asks for four tagged sections in one response: observe, evidence, think and answer prompt.py. Inference is a single generation call per question predict.py and the efficiency study describes it as single-step generation §5.6.

Two-stage training

Stage 1 is supervised fine-tuning as a cold start §3.2. The 60,000 SFT trajectories come from chains of thought generated by Qwen2.5-VL-72B and Qwen2.5-VL-7B. A trajectory is kept only when its final answer is correct and its cited evidence IDs match the truly relevant images §4 app. A.2. Stage 2 is RS-GRPO §3.2.

Reward scopes

The output is split by special tags into four scopes: the observation scope, the evidence scope, the reasoning scope and the answer scope §3.2. Each scope receives a set of reward channels eq. 7:

• perception reward on the observation and evidence scopes;
• format reward on all tokens;
• derivation reward on the reasoning and answer scopes.

A token's scope-averaged reward is the mean of the channels that apply to it eq. 8. The paper then normalises each token's reward across the group eq. 9 and optimises a clipped, token-averaged objective with lower and upper clip bounds ε_low and ε_high eq. 10. The authors incorporate the token-level advantages into DAPO §3.2.

The three rewards are defined in the appendix app. A.3:

R_{\text{perception}} = \frac{\sum_{i=1}^{n} r_i}{\sum_{i=1}^{n}\big(y_i k_{\text{pos}} + (1-y_i)\big)},\quad r_i = \begin{cases} k_{\text{pos}}\, f_1(e_i^{\text{pred}}, e_i^{\text{gold}}) & y_i = 1 \\ \mathbb{I}(e_i^{\text{pred}} = \text{"no relevant information"}) & y_i = 0 \end{cases} eq. 11

R_{\text{derivation}} = f_1(a^{\text{pred}}, a^{\text{gold}}), \quad a^{\text{gold}} = \begin{cases} a^{\text{gold}} & \text{sufficient context} \\ \text{"insufficient to answer"} & \text{insufficient context} \end{cases} eq. 12

R_{\text{format}} = \begin{cases} 1 & \text{format correct} \\ 0 & \text{otherwise} \end{cases} eq. 13

Here y_i is 1 when image i holds relevant information, n is the number of images, and k_pos is a scaling factor whose value the paper does not state app. A.3.

# Scope mapping and scope-averaged reward  [eq. 7] [eq. 8]
# Input:  tag of token t, in {observe, evidence, think, answer}
# Input:  channel_scores[m] for response i and channel m, one scalar each

def channels_for(tag):
    # Observation and evidence tokens get perception and format  [eq. 7]
    if tag in ("observe", "evidence"):
        return ["perception", "format"]
    # Reasoning and answer tokens get derivation and format  [eq. 7]
    if tag in ("think", "answer"):
        return ["derivation", "format"]
    return ["format"]

def scope_average(channel_scores, channels):
    # Mean over the channels that cover this token  [eq. 8]
    values = [channel_scores[m] for m in channels]
    return sum(values) / len(channels)

# RS-GRPO loss for one prompt with G sampled responses  [§3.2] [eq. 9] [eq. 10]
# Input:  per-token scope-averaged rewards, shape (G, T)
# Input:  per-token log-probs under the old and current policy, shape (G, T)
# Output: scalar loss to minimise

def rs_grpo_loss(token_reward, logps, old_logps, mask):
    # Standardise each token position across the group  [eq. 9]
    advantage = zscore_over_group(token_reward)             # (G, T)
    # Importance ratio per token  [eq. 10]
    ratio = exp(logps - old_logps)                          # (G, T)
    clipped = clip(ratio, 1 - eps_low, 1 + eps_high)        # (G, T)
    # Pessimistic clipped objective, averaged over all response tokens  [eq. 10]
    objective = min(ratio * advantage, clipped * advantage)
    return -sum(objective * mask) / sum(mask)

Hyperparameters

Results

Author-reported. Setup: five VQA sets; top-3 pages from VisRAG-Ret for every method; Qwen2.5-VL-7B-Instruct as the backbone of EVisRAG-7B §4. "Sufficient" and "insufficient" subsets follow a sufficiency test, and F1 is computed only on the sufficient subset §4 tab. 1. The insufficient-context correctness is judged by a model in the paper (GPT-5) §4, see Paper vs code.

Main comparison tab. 1 §5.1

Cells are Acc / F1 in percent. The table gives no variance.

The paper's text says the 7B model beats the backbone by 19% in accuracy and 27% in F1 §5.1. The paper does not say whether these are relative or absolute gains. From Table 1, the average gaps are 21.27 accuracy points (39% relative) and 51.13 F1 points.

Ablation tab. 2 §5.2

Accuracy in percent, mean ± standard deviation over five random seeds tab. 2. Only the ablation table reports variance.

The authors attribute the RS-GRPO gain to scoped credit assignment rather than to broadcasting all rewards over the whole sequence §5.2. Appendix A.7 reports bootstrap intervals (10,000 resamples over five runs) for a figure that is not in the LaTeX text app. A.7.

Compared with other prompting methods tab. 3 §5.7

Accuracy in percent, no training. The backbone for these prompt-only runs is not stated.

Dataset statistics tab. 4 app. A.1

Other results

• Transfer to InternVL3-8B tab. 5 app. A.4: average accuracy 57.72 without the prompt and 60.50 with the EVisRAG prompt. The gain is on four of five sets; InfoVQA falls from 59.47 to 55.29. No training was done on InternVL.
• Visual RAG generation baselines tab. 8 app. A.6: Page Concatenation averages 56.63 accuracy and Weighted Selection 29.35. The EVisRAG-3B and EVisRAG-7B rows here have different values from the main table (see Pitfalls).
• Natural images tab. 9 app. A.9: Visual Haystacks, 2/3/5 images in distribution and 10/50 out of distribution with CLIP top-3 retrieval. Qwen7b scores 64.67, 64.44, 62.00, 57.8 and 56.2 across these; EVisRAG scores 86.22, 85.33, 83.11, 71.6 and 64.5.
• Other retrievers tab. 10 app. A.11: on ViDoSeek, EVisRAG-7B reaches 50.79 accuracy with ColPali v1.3 and 49.37 with jina-embeddings-v4, against 52.10 with VisRAG-Ret.
• Top-k app. A.10: with up to 20 retrieved images, EVisRAG stays above Qwen2.5-VL and R1-Router in accuracy while using fewer tokens than R1-Router.
• Efficiency §5.6: on ViDoSeek, the backbone and OpenVLThinker take about 90–95 s and about 270–330 tokens and score below 44% accuracy. R1-Router takes about 120 s and over 400 tokens. EVisRAG scores over 52% at about 100 s and about 300 tokens.
• Training effect §5.5: before training, the backbone answers correctly in 24.32% of cases with correct retrieval and wrongly in 35.04%. The paper reports a substantial rise in correct answers after training; the exact after-training figures are only in a figure.
• Perception probes §5.3 §5.4: a visual evidence attention ratio, measured on annotated evidence boxes, is highest for EVisRAG, and higher ratios go with higher accuracy. Evidence density across top-1 to top-5 retrieval is reported as a figure only.

Paper vs code

Code: https://github.com/OpenBMB/VisRAG at commit 56f9785bf8385bce0c220f19ba7c82ddddc89ba0, Apache-2.0. The RS-GRPO trainer is in src/rsgrpo, the inference and evaluation scripts in src/evisrag and evisrag_scripts.

• Reward channels. The paper has three channels §3.2 app. A.3. The shipped reward uses six: accuracy (derivation), evidence (perception), format, overlong, isObserve and isThink function.py. Overlong, isObserve and isThink are absent from the compiled appendix; their paragraphs are commented out in the LaTeX.
• Penalty channels. isObserve gives −1 when the observation mentions an image index from 1 to 5, and isThink gives −1 when the think text equals the answer evidencecot.py. The overlong penalty uses a 1536-token limit with a 512-token buffer and gives −1 below 200 tokens evidencecot.py evidencecot.py.
• Channel weights. The shipped "overall" score is 3 × accuracy + 3 × evidence + overlong + format + isObserve + isThink evidencecot.py. The paper does not state these weights and does not use this combined score in its objective. Training normalises each channel separately, so a constant factor on one channel cancels there core_algos.py.
• Scope averaging. Paper eq. 8 averages channel rewards per token. The code gives each channel its own advantage and loss, applies them through masks and sums the channel losses. A token covered by two channels therefore gets both losses added, not their mean core_algos.py core_algos.py core_algos.py dp_actor.py.
• Advantage normalisation. Paper eq. 9 normalises per token position across the group. The code normalises one scalar per response and channel, then broadcasts it over the scope core_algos.py.
• Loss normalisation. Paper eq. 10 divides by the total number of response tokens in the group. The code averages each channel over its own in-scope tokens, weights by token share across the mini-batch, then divides by the count of non-zero channel losses core_algos.py dp_actor.py dp_actor.py.
• KL penalty. Neither the paper's objective nor the config has a KL term config.yaml. This matches.
• Perception reward. The paper uses F1 for relevant images app. A.3. The code uses F1 only when the gold evidence has at least five words and exact substring containment otherwise evidencecot.py. It weights relevant images 3 and "no relevant information" images 1.
• Format reward. The paper requires the four tags in order app. A.3. The code requires the whole response to match one pattern, with each tag appearing exactly once and no text outside the tags evidencecot.py.
• Accuracy reward. Both sides compute token F1 between normalised answers evidencecot.py. The code reads answers from <answer> tags in the reference string, so the training data must carry those tags.
• Scope masks. The code finds each scope boundary by searching for the tag tokens function.py. Derivation covers <think> to the end and perception covers the start to <think>. This matches eq. 7.
• Clip and learning rate. Clip bounds 0.2 and 0.28, learning rate 1e-6, weight decay 0.01, warm-up 0 and gradient clip 1.0 all match the paper's table tab. 7.
• Rollouts and temperature. The paper gives rollout temperature 1.2 tab. 7. The config uses 1.0 config.yaml. The paper does not give the number of rollouts; the config uses 8 config.yaml.
• Epochs. The paper gives 4 GRPO epochs tab. 7. The config sets 16 total epochs and the run script does not override it config.yaml run_rsgrpo.sh.
• SFT settings. The code uses 200 warm-up steps and an effective batch of 64 full_sft.sh. The paper gives a warm-up ratio of 0.1 and batch 32 tab. 6. The code also freezes the vision tower full_sft.sh and so does the GRPO config config.yaml. The paper does not mention freezing.
• Dynamic sampling. The paper cites DAPO-style filtering without thresholds §3.2. The code drops prompt groups whose mean "overall" score is at most 0.01 or at least 6.9 ray_trainer.py config.yaml.
• Evaluation. The paper says insufficient-context answers are judged by GPT-5 §4. The released eval.py uses string matching: it checks whether the gold tokens are contained in the prediction, computes exact match and token F1, and counts a refusal only when the prediction matches one of three fixed phrases eval.py eval.py. I found no GPT-5 judging code in the repository. The README mentions an OpenAI key only for the VisRAG generation script README.md.
• Not in the repository. The training data is referenced as data/Train/evidencecot_GRPO.jsonl, which is not in the clone; the README points to a Hugging Face collection instead README.md. The 72B trajectory generation code is not there either. The clone holds only one example parquet file 0.parquet.
• 3B model. The README says EVisRAG-3B was released README.md. Only the 7B training scripts are in the repository.

Pitfalls

• Headline gains do not match the table. The text says about 19% accuracy and 27% F1 over the backbone §1 §5.1. Table 1 gives 21.27 and 51.13 points on the averages, and the paper does not say what the percentages measure.
• "Consistently outperforms every comparator" is not true for every cell. On ViDoSeek accuracy, Qwen3-VL-8B scores 52.45 against 52.10 for EVisRAG-7B tab. 1.
• Several versions of the same numbers. EVisRAG-7B averages 75.50 in Table 1 tab. 1, 74.9 in the ablation's RS-GRPO row tab. 2 and 75.01 in the VisRAG-Gen table tab. 8. Its ViDoSeek F1 is 67.67 in Table 1 but 65.78 in Tables 8 and 10. The Qwen2.5-VL-7B ViDoSeek row in the retriever table gives accuracy 42.56 and F1 42.48 tab. 10, while Table 1 gives 42.38 and 29.55. The 3B average is 69.21 in Table 1 and 68.68 in Table 8. Treat the numbers as author-reported and check which version a claim uses.
• Ablation rows. StepGRPO appears in the ablation table tab. 2, but the paper does not define it and the code has no such trainer. The appendix text on bootstrap intervals refers to variants (without perception, without perception reward, without RS-GRPO) that are not rows of the compiled table app. A.7.
• Variance. Only the ablation table reports standard deviations tab. 2. The main table has single values, and the paper does not say whether they are means over seeds.
• Refusal scoring is string-based in the code. The paper describes a GPT-5 judge for insufficient-context refusals §4. The released scripts use fixed refusal phrases eval.py. Results obtained with a different judge will not be comparable.
• Accuracy is containment, not exact match. The released "acc" counts a prediction as correct when every gold token appears in it eval.py. This is looser than the paper's wording.
• Inference settings are unstated. Temperature is 0.1 in the code default, 0.0 in the example script and 0.1 in the README example. Repetition penalty 1.05 and 2048 maximum tokens are set in code predict.py. Paper numbers may depend on these choices.
• Scope fallback is silent. If a tag does not tokenise the way the search expects, the mask falls back to covering the whole response, which turns off the scoping for that sample function.py.
• Strict format reward. A response with any text outside the four tags, or with a repeated tag, scores 0 on format evidencecot.py.
• Undefined normalisation axis. Eq. 9 normalises across the group at each token index t. The paper does not say how positions are aligned when responses differ in length, and the code avoids the question by normalising per response core_algos.py.
• Length penalties change the reward. The code's overlong channel gives −1 for responses under 200 tokens and a linear penalty over the last 512 tokens before 1536 evidencecot.py. The compiled paper does not describe either limit.
• Ground truth must carry tags. The accuracy and evidence rewards split reference strings on <answer> and <evidence> tags evidencecot.py evidencecot.py. Any training data in a different format will break the reward silently or give zero.
• Data counts are unclear. The paper collects 30,000 QA samples and splits them 8:2, then reports 60,000 SFT and 4,000 GRPO samples §4 app. A.2. How 30,000 becomes 60,000 is not stated beyond retrieval-based augmentation. The appendix says 400 MMLongBench multi-hop examples are added to the 4,000 GRPO samples; whether they are inside that count is not stated. A manual check covered 75 answerable and 25 unanswerable cases §4.
• Teacher quality. The SFT labels come from Qwen2.5-VL-72B, and the paper says the evidence was checked on a small sample only §4. The 72B pipeline is not released.
• Dataset licences. The appendix says datasets without an explicitly open licence are restricted to non-commercial academic research app. A.13. The paper does not list the licence of each dataset. Check the source datasets before commercial reuse of data or weights.
• Numbering. Equation, table and figure numbers were counted from the LaTeX source, skipping commented-out blocks. The compiled PDF was not checked, so compare against it before quoting a number.
• Cost. RS-GRPO uses eight rollouts per prompt over up to five images config.yaml config.yaml. The paper reports 8 A100-80GB GPUs app. A.5. The paper gives no wall-clock training time.

Lineage

• Builds on: VisRAG and its retriever VisRAG-Ret §3.1 §4, which is not yet a card in this wiki. GRPO §3.2 and DAPO, whose arXiv id is 2503.14476 per the code comment core_algos.py. The curriculum filter follows Polaris app. A.2. Training frameworks are LLaMA-Factory and Easy-R1 §4.
• Compared against: general VLMs (Qwen2.5-VL, MiMo-VL, Qwen3-VL), RL-trained reasoning VLMs (Vision-R1, ThinkLite-VL, OpenVLThinker, Ocean-R1), and VRAG systems (ViDoRAG, R1-Router, MMSearch-R1, VRAG-RL) §4 tab. 1. Prompting baselines are COCoT, CCoT and DDCoT §5.7.
• Superseded by: not known from the paper or the repository.