Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking

In plain words

Vision-language models answer questions about images, and they can sound sure of themselves even when the picture played no part in the answer. The model may simply be guessing from what questions usually look like. Existing ways of asking a model how confident it is cannot tell these cases apart, because they only look at the answer.

BICR adds a check based on a simple contrast. The same question is asked twice: once with the real picture and once with a black square in its place. A small add-on, called a probe, learns to read the model's internal state and predict whether an answer is correct. It is trained so that, for correct answers, the real picture gives higher confidence than the black one. The model itself stays frozen and is never retrained.

The payoff is that the black-square pass is needed only during training. When the system is used, it asks the question once with the real picture and reads the probe's score, so it costs no more than an ordinary single-view probe.

TL;DR

• BICR is a confidence probe for frozen LVLMs, trained with three losses: a weighted classification loss, a calibration penalty, and a ranking loss on a blacked-out image §4 eq. 4.
• The ranking loss applies only to correct training answers and asks for a margin in probability between the real-image and blank-image scores §4 eq. 3.
• Inference needs one real-image forward pass and the probe, so it adds no inference cost over a single-view probe §4 eq. 5.
• Author-reported: the best cross-LVLM average on ECE, BS, AUCPR and AUROC among eight methods tab. 2. Removing the ranking loss costs 3.32 AUROC points tab. 3.
• Benchmark (VLCB) is built from seven public sources and is intended for non-commercial research use only app. B; the code is MIT README.

Problem

Large vision-language models can give fluent, confident and correct answers from language priors alone, with the image contributing nothing. Confidence estimators that watch normal inference cannot detect this, because they have no way to tell whether the image shaped the prediction §1.

Confidence estimation is also hard to evaluate fairly. The paper's benchmark trains the probe only on GQA and tests on six unseen datasets, so that an estimator that only works in-distribution does not pass §3.

Method

BICR keeps the LVLM frozen and trains a probe f on hidden states taken from the final decoder layer, at the last prompt token §4 app. G. Two views are extracted for each training sample.

• Base view: the real image with the question. The hidden state is h_base.
• Blank view: a solid black image (RGB 0,0,0) of the same spatial size, with the question unchanged. The hidden state is h_blank §4.

The probe is a multi-layer perceptron with ReLU activations and dropout. The same parameters are applied to both views during training, so the ranking loss shapes one shared function rather than two separate thresholds §4.

The training objective has three terms §4 eq. 4:

• A binary cross-entropy on the base view, with the positive class weighted by n_neg / n_pos §4 eq. 1.
• A Brier penalty on the base-view probability §4 eq. 2.
• A margin ranking loss, applied only to correct samples, that asks the base-view probability to exceed the blank-view probability by at least gamma §4 eq. 3.

The paper gives the objective as a sum with coefficients beta (Brier weight), lambda (rank weight) and gamma (margin), all selected by Optuna §4 app. G.

# BICR training objective  [§4]
# Input:  base_states, blank_states (B, d_h); labels (B,) in {0, 1}
# Output: scalar loss

def bicr_loss(base_states, blank_states, labels):
    # Shared probe f applied to both views  [§4 "Probe architecture"]
    logit_base = f(base_states)                  # (B,)
    logit_blank = f(blank_states)                # (B,)
    p_base = sigmoid(logit_base)                 # (B,)
    p_blank = sigmoid(logit_blank)               # (B,)

    # Positive-class weighted BCE on the base view  [§4] [eq. 1]
    n_pos = sum(labels)
    n_neg = count(labels == 0)
    w_pos = n_neg / n_pos
    loss_bce = bce_with_logits(logit_base, labels, pos_weight=w_pos)

    # Brier penalty on the base view  [§4] [eq. 2]
    loss_brier = mean((p_base - labels) ** 2)

    # Margin ranking loss, correct samples only  [§4] [eq. 3]
    hinge = relu(gamma - (p_base - p_blank))     # (B,)
    loss_rank = sum(hinge * labels) / (sum(labels) + 1e-8)

    # Combined objective  [§4] [eq. 4]
    return loss_bce + beta * loss_brier + lam * loss_rank

Inference uses only the real-image pass §4 eq. 5:

# Inference: one real-image forward pass  [§4] [eq. 5]
# Output: confidence in (0, 1) that the answer is correct

def confidence(question, image):
    # Frozen LVLM; hidden state at the last prompt token  [§4]
    h_base = last_prompt_hidden_state(frozen_lvlm, question, image)
    return sigmoid(f(h_base))

Training needs a second forward pass per sample for the blank view. The paper says this pass is computed once as preprocessing and reused across training runs and seeds §4. This is the only extra cost, and it is paid during training, not inference.

Figure 2 shows cross-LVLM reliability diagrams for all eight methods fig. 2.

Hyperparameters

Results

Author-reported. Setup: five open-weight LVLMs (Qwen3-VL-8B, LLaVA-NeXT-13B, InternVL3.5-14B, Gemma-3-27B, DeepSeek-VL2), evaluated on the VLCB test split §3 tab. 1. Trained methods report the mean over five seeds with 50 Optuna trials each tab. 2. Values are percentages, except the figure ECE values.

Cross-LVLM average tab. 2:

Per-LVLM: BICR leads on AUCPR and AUROC for all five LVLMs. It leads on ECE and BS for three of five (DeepSeek-VL2, LLaVA-NeXT, InternVL3.5) tab. 2. Against P(I Know), which uses the same Optuna search space, BICR is +2.1 AUROC, +1.2 AUCPR and −2.2 ECE on the cross-LVLM average §5 tab. 2.

Loss ablation, cross-LVLM average tab. 3:

Removing the ranking loss costs 3.32 AUROC points and 1.95 AUCPR points (p < 0.001) §5 app. I. Removing the Brier term has a smaller calibration effect §5 app. I.

Calibration: on the pooled reliability diagram, BICR has ECE 0.056, the best of the eight methods; P(True) has 0.366 and Self-Probing 0.281 fig. 2 §5. The pooled ECE in fig. 2 is not the same as the averaged ECE in tab. 2, because ECE is non-linear across bins fig. 2 caption.

Discrimination gains over trained baselines are reported as significant under pooled and cluster-aware tests. Calibration gains are significant against P(I Know), SAPLMA and CCPS §5 app. J.

Null-image choice (Qwen3-VL-8B only, five seeds): solid black gives the best mean on all six metrics. AUROC is 0.801 for black and between 0.678 and 0.722 for white, Gaussian noise, blurred and pixel-shuffled nulls app. I.

Paper vs code

Official code is the repository linked in the abstract, https://github.com/Ledengary/BICR [abstract]. The README gives the license as MIT for the code README.md. I could not clone the repository on this machine, so I read it through the GitHub web view. The commit is the head of main as shown on the web (code_commit), not one verified with git, and the citations below are file-level, without line numbers.

The loss matches the paper. models/BICR/BICR_train.py computes BCE with pos_weight = n_neg / n_pos on the base view; a Brier term averaged over the batch; and a hinge relu(margin - (p_base - p_blank)) that is summed over correct samples with a mask and divided by their count plus 1e-8 BICR_train.py. Ranges and defaults match the paper: beta in [0, 0.5], lambda in [0.01, 0.3], margin in [0.05, 0.25], learning rate and weight decay as stated, batch size 32, 200 epochs, patience 20, 50 trials, TPE sampler, Median Pruner with 5/10/5, and a composite score with AUROC weight 0.6 BICR_train.py app. F app. G.

Differences:

• Optimizer: the paper does not name one. The code uses Adam with gradient-norm clipping at 1.0 BICR_train.py.
• Blank-view reuse: the paper says the blank pass is computed once and reused across runs and seeds §4. The extraction script models/BICR/BICR_extraction.py runs the blank view as a separate forward pass for each sample, with a freshly generated black image of the same size as the resized real image, and I found no cache in that file. Whether features are reused across seeds depends on how the saved extraction files are used, which I did not verify BICR_extraction.py.
• Blank view failures: when the blank pass fails, the code stores a zero vector with mask_blank = 0 and excludes the sample from the ranking term. The paper does not describe this case BICR_extraction.py BICR_train.py.
• Position fallback: if the last prompt token is masked, the code uses the last unmasked token. The paper does not mention this BICR_extraction.py.
• Batching: a code comment calls training "full-batch", but the code uses shuffled mini-batches of size 32 BICR_train.py. Batch size 32 matches the paper app. G.
• Probe-architecture options are read from utils/optuna_configs.json, which I did not open, so I could not confirm the exact option list in the code.
• Not checked: models/BICR/null_ablation/, and the GQA and VLCB construction code under preprocessing/.

Pitfalls

• Qwen numbers differ slightly across tables. BICR on Qwen3-VL-8B is ECE 8.91, BS 17.40, AUCPR 90.26, AUROC 80.14 in tab. 2. The per-LVLM ablation app. I gives 8.9, 17.5, 90.1, 80.1, and the null-image table app. I gives 0.0886, 0.1747, 0.9014, 0.8008 for the black null. Use tab. 2 for comparisons and do not mix sources.
• Per-LVLM ECE is not monotonic with the ranking loss. On Qwen3-VL-8B, removing the ranking loss lowers ECE from 8.9 to 4.9, and on Gemma-3-27B removing the Brier term lowers it from 7.0 to 6.4 app. I. The paper's calibration claim applies to the cross-LVLM average.
• The blank pass costs a second LVLM forward pass per training sample §4. Only inference is free. Plan training compute accordingly.
• Margin is in probability space, not logit space §4 eq. 3. Copying a logit margin from another method will change the loss scale.
• Ranking loss ignores incorrect answers by design §4. Little positive mass in a batch makes the denominator small; the paper gives only the epsilon.
• The composite score uses validation ECE and AUROC and drives both checkpoint choice and the Optuna objective app. F. Test metrics are not used for selection, but reproduced numbers depend on the validation split.
• Fixed 0.5 threshold for Accuracy and F1 is the same for every method and not tuned app. E.
• Baseline budgets differ. P(I Know) gets the same Optuna search space as BICR. The other trained baselines keep their published configurations §5 app. G. The paper says the comparison with P(I Know) isolates the training objective.
• Seeds are not independent samples. The five seeds share the frozen weights and the test set, so the authors also report cluster-aware tests app. I.
• Single judge. Correctness labels come from one gpt-5-mini judge. The audit covers 350 items §3.
• Dataset license. VLCB is assembled from seven public sources. The paper says it is intended for non-commercial research use only and inherits ShareAlike terms from GMAI-MMBench. The code is MIT, but the assembled benchmark is not redistributed app. B. Reproduction needs the source datasets from their official distributors.
• Parameter count. BICR's average is 1.6M additional trainable parameters, against 11.3M for InternalInspector's ResNet18 encoder §5 app. H. The blank view adds no parameters at inference app. H.

Lineage

• Builds on: P(I Know), which trains the same kind of MLP probe on hidden states with BCE only §3 §5. Its arXiv id is 2207.05221 [references.bib: kadavath]. BICR's ranking term and blank view are new in this paper.
• Compared against: P(True), Self-Probing, Prompt Ensembles, SAPLMA, InternalInspector and CCPS §3. The calibration protocol follows the reliability-estimation benchmark of rmcb app. E [references.bib: rmcb].
• Superseded by: not stated.