Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

In plain words

Vision-language models sometimes describe objects that are not in the picture. They do this partly because they lean on what usually goes together in their training data, and partly because the language model behind them likes to finish sentences in familiar ways, whatever the image shows.

The paper's idea is to make the model compare two readings of the same question. One reading uses the real picture. The other uses a copy of the picture covered in random noise, so the visual evidence is weaker. Words that the model would produce mainly because of the bias or habit show up in both readings, so the method pushes the choice away from them and toward what the clear picture supports.

No extra training is needed, and no other model or tool is called. The change happens only at the moment the model picks each next word. The authors report that this reduces made-up objects on standard tests and also improves general benchmarks.

TL;DR

• Training-free change to decoding for LVLMs: at each token, contrast the logits from the original image with those from a Gaussian-noised copy of it §3.3.1 eq. 3.
• A plausibility cutoff keeps only tokens whose probability under the original image is at least β times the top probability §3.3.2 eq. 4. Default α=1, β=0.1 §4.1.
• Author-reported: up to +5.8 accuracy and +7.4 F1 on POPE, across three LVLMs and three sampling settings tab. 1.
• Costs a second forward pass per generated token. The paper does not quantify this §3.3.1.
• Relevant if you need fewer hallucinated objects from an existing LVLM without retraining it.

Problem

Object hallucination means an LVLM outputs content that is fluent but inconsistent with the objects in the image §1. The authors attribute it mainly to two causes: statistical bias from training data (frequent or co-occurring objects) and over-reliance on unimodal, mostly language, priors §1 §3.1. Existing remedies named in the paper need extra data, RLHF or a post-hoc revisor, which they consider costly §1.

Method

Decoding. The model is parameterised by θ, takes a text query x and a visual input v, and samples each token y_t from a softmax over its logits given (v, x, y_<t) §3.1 eq. 1. The authors argue hallucinations arise when probability goes to tokens that do not match v §3.1.

Visual uncertainty. The authors make the image less certain by adding Gaussian noise following a forward diffusion process. Each of T steps mixes the previous image with fresh noise, using a per-step parameter γ §3.2 eq. 2. Their observations (from their own experiments) are that more noise makes the model put more weight on language priors §3.2 fig. 2, that it favours frequent and co-occurring MSCOCO objects more §3.2 fig. 3, and that POPE performance drops as the noise steps grow for LLaVA-1.5 §4.3 fig. 6.

Visual contrastive decoding. For a query x and image v, the model produces two next-token distributions: one from v and one from the distorted image v'. The contrastive distribution is a softmax over (1+α)·logit(y | v, x) − α·logit(y | v', x) §3.3.1 eq. 3. With α=0 this reduces to regular decoding. Any standard sampler can be applied afterwards; the authors mention nucleus sampling and beam search §3.3.1. Their experiments use direct sampling §4.1.

Adaptive plausibility constraint. Contrasting can penalise tokens that are still reasonable under the distorted image. The authors follow an earlier contrastive decoding work §3.3.2 and keep only tokens whose probability under the original image is at least β times the maximum probability. All other tokens get probability zero §3.3.2 eq. 4. Larger β truncates more aggressively §3.3.2.

Full rule. Combining the two, each token is sampled from the contrastive softmax restricted to the plausible set §3.3.2 eq. 5.

# Visual contrastive decoding, one token  [§3.3] [eq. 3] [eq. 4] [eq. 5]
# Input:  image v, prompt x, tokens so far y_<t, T, gamma, alpha, beta
# Output: next token y_t

def vcd_next_token(image, prompt, tokens_so_far, T, gamma, alpha, beta):
    # 1. Distort the image with T Gaussian noising steps  [eq. 2] [§3.2]
    noised_image = image
    for step in range(T):
        noise = sample_gaussian_like(noised_image)
        noised_image = sqrt(1 - gamma) * noised_image + sqrt(gamma) * noise

    # 2. Next-token logits for the original and the distorted image  [§3.3.1]
    logits = model(image, prompt, tokens_so_far)                 # (V,)
    logits_noised = model(noised_image, prompt, tokens_so_far)   # (V,)

    # 3. Plausible set from the original image only  [eq. 4] [§3.3.2]
    probs = softmax(logits)
    plausible = probs >= beta * probs.max()                      # bool (V,)

    # 4. Contrast the two sets of logits  [eq. 3]
    contrast = (1 + alpha) * logits - alpha * logits_noised      # (V,)

    # 5. Drop implausible tokens and sample from the softmax  [eq. 5]
    contrast = where(plausible, contrast, -inf)
    probs_vcd = softmax(contrast)
    return sample(probs_vcd)

Pseudocode is written for this card. The paper has no algorithm box.

Hyperparameters

POPE set: MSCOCO, A-OKVQA and GQA images; 500 images per dataset per setting (random, popular, adversarial); 6 questions per image, 27,000 query-answer pairs in total §4.1. Negative objects are absent from the image, sampled at random, from a frequent pool, or from co-occurring objects, respectively §4.1. Segmentation with SEEM is used to find objects in A-OKVQA and GQA images §4.1.

MME: 10 perception and 4 cognition subtasks. The hallucination subset uses existence, count, position and color questions §4.1.

LLaVA-Bench: 24 images and 60 questions, used for qualitative cases and GPT-4V-judged open-ended answers §4.1 tab. 3.

Results

Author-reported. Setup: three LVLMs at 7B scale, regular decoding as baseline, 5 runs with standard deviation §4.1 tab. 1. VCD is bold (best) in every row of the POPE table tab. 1.

POPE tab. 1. Accuracy, precision, recall and F1, mean ± std over 5 runs.

Headline POPE gains as the paper states them: up to +5.8 accuracy and +7.4 F1 §4.2. These match the GQA Popular Qwen-VL row (accuracy 75.99 to 81.83, F1 74.84 to 82.23). Recall drives most of the F1 gain for LLaVA-1.5 and Qwen-VL; precision drives it for InstructBLIP §4.2.

MME hallucination subset tab. 2. Totals are sums of the four subtasks, mean ± std over 5 runs.

The paper says VCD gives a uniform gain on object-level items and on color, with little change on position §4.2. In the table, Regular beats VCD on position for Qwen-VL and InstructBLIP. The MME full-set perception and recognition results are in fig. 4 and app. C; the appendix tables are not transcribed here.

GPT-4V-judged open-ended answers on LLaVA-Bench tab. 3. Scores on a 0 to 10 scale, author-reported, judged by GPT-4V.

Ablations (LLaVA-1.5, MME hallucination, perception and recognition totals) app. B.

Sampling strategies on POPE random, LLaVA-1.5 app. B. The appendix text gives top-p 0.9, top-k 50 and temperature 1.5 and 0.7 with top-k 50; the table itself does not state these values.

13B variants on POPE app. B.

Paper vs code

Code: DAMO-NLP-SG/VCD at code_commit, Apache-2.0. Citations are relative to that repository. Discrepancies, each with both citations:

• α for InstructBLIP. Paper uses α=1 §4.1 app. A. The InstructBLIP eval script passes cd_beta but not cd_alpha object_hallucination_vqa_instructblip.py, so the sampler falls back to 0.5 vcd_sample.py. The LLaVA and Qwen-VL scripts pass α=1 by argument default object_hallucination_vqa_llava.py object_hallucination_vqa_qwenvl.py. Not verified by running; it follows from reading the call path.
• Noise schedule. Paper: T steps of Gaussian noise, each with a fixed γ=0.1 §3.2 eq. 2 app. A. Code: a fixed 1000-step schedule of betas sigmoid-spaced between 1e-5 and 5e-3, then one closed-form noising to step t = noise_step vcd_add_noise.py vcd_add_noise.py. The code never uses γ. Eq. 2 scales the signal by sqrt(1−γ) per step, so with γ=0.1 the signal shrinks geometrically over T steps; the code's schedule is not that.
• Noise steps. Paper: T=999 for POPE, 500 for MME and LLaVA-Bench app. A. Eval scripts default noise_step to 500 object_hallucination_vqa_llava.py object_hallucination_vqa_qwenvl.py object_hallucination_vqa_instructblip.py. The repository has no run scripts that set 999 for POPE.
• Noise space. The paper does not say in which space the noise is added. The code adds it to the processor output, which for LLaVA is the normalised pixel tensor object_hallucination_vqa_llava.py.
• Generation length. Not stated in the paper. The LLaVA script uses max_new_tokens=1024 object_hallucination_vqa_llava.py and the Qwen-VL script uses 20 object_hallucination_vqa_qwenvl.py.
• Default sampling. Paper baseline is direct sampling §4.1. Code defaults are temperature 1.0, top-p 1, top-k none object_hallucination_vqa_llava.py, which is consistent.
• Plausibility cutoff. No discrepancy. Paper keeps tokens with p ≥ β·max p eq. 4. Code applies logit < log(β) + max logit to the original logits, which is the same test because softmax is shift-invariant vcd_sample.py.
• Contrast. No discrepancy. Code computes (1+α)·original − α·noised on logits, matching eq. 3 vcd_sample.py.
• Beam search. The paper names beam search as a possible sampler §3.3.1. The patched decoding loop only samples with multinomial on the contrastive logits vcd_sample.py. Beam search with VCD is not implemented.
• Mechanism of patching. The code replaces transformers' internal sampling method at import time vcd_sample.py. Behaviour depends on the transformers version; the repository does not pin one in the code cited here.

No other discrepancies found in the parts read.

Pitfalls

• Two noise models. Eq. 2 with fixed γ and T steps is not the code's one-shot schedule. Pick one and say which in your implementation. The paper does not say which one produced the reported numbers.
• Noise scale depends on the input space. Gaussian noise in [0, 1] pixels and in normalised space have very different magnitudes. The paper is silent; the code uses the processor output object_hallucination_vqa_llava.py.
• Stochastic. Each call draws fresh noise, and the paper states no seed. Expect run-to-run variation; the reported std over 5 runs is the only guide.
• Cost. Each generated token needs a second forward pass on the noised image, with its own cached state vcd_sample.py vcd_sample.py. The introduction calls the overhead minimal §1; the paper gives no timing or memory numbers.
• Default β is not the paper's best β. In its own β sweep on MME, β=0.5 scores highest on all three subsets, while the default is 0.1 app. B. The paper does not explain the choice.
• Possible copy error in the 13B table. The InstructBLIP 13B VCD rows are identical to the InstructBLIP 7B VCD rows in all three settings tab. 1 app. B. Check before using either set.
• Intro MME figure. The introduction says "+18%" on MME §1 without naming the model or subset. The InstructBLIP hallucination total rises from 380.33 to 447.67, about +17.7%. LLaVA-1.5 rises about 7% and Qwen-VL about 2% tab. 2. The paper does not state the basis.
• Equation numbers. The appendix cites the source labels eq:1, eq:3 and eq:5 for γ, α and β app. A. In order of appearance those equations are numbered 2, 3 and 4 here, as cited on this card. Check the PDF's own numbering before citing.
• Ablation text names δ, table shows T. The MME ablation text describes a noise level δ with default 500 app. B. The table is labelled T.
• Table 2 is not uniform. VCD does not win Position for Qwen-VL or InstructBLIP tab. 2. The text says "minimal uplift" there §4.2.
• Sampling ablation values. The top-k value and the temperature values are in the appendix text only app. B. The "Greedy" row is also not explained in the paper.
• GPT-4V evaluation. Open-ended scores depend on GPT-4V, a proprietary judge model §4.2 app. E. Results cannot be reproduced without it, and the evaluation prompt is only in the supplement app. E.
• Only tested on three LVLMs, 7B scale plus 13B variants. No generalisation claim beyond these is made or tested §4.1 app. B.

Lineage

• Builds on: contrastive decoding for text (Li et al., 2022), followed for the plausibility constraint §3.3.2; classifier-free diffusion guidance, cited as an analogy §3.3.1; the diffusion forward process, cited for the noise model §3.2.
• Compared against: regular decoding (direct sampling) in every table tab. 1 tab. 2 tab. 3.
• Superseded by: not stated in the wiki.