Paper · 2026-03-01 · cs.CV, cs.AI
VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning
VisNec scores each instruction sample by how much its image lowers the response loss, comparing a blind pass (image masked) with the multimodal pass. It drops non-positive scores, clusters questions with K-means and keeps the top 15% per cluster. On LLaVA-665K it reaches 100.2% of full-data relative performance (author-reported), using 15% of the data.
Instruction tuningVisual instruction tuningDataset curationk-means clusteringdata-selectionmultimodalinstruction-tuningllava
arXivPDFCodeMarkdown10 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Instruction datasets for image-and-text models contain many examples that a model could answer from the question alone, and some where the picture contradicts the answer. Training on all of them is slow, and the bad ones can hurt. VisNec asks, for each example, how much the picture helps. It asks the model the same question twice, once with the image and once with the image hidden, and compares how surprised the model is by the correct answer in each case. Examples where the picture makes the answer less surprising are kept. Examples where the picture changes nothing, or makes the answer more surprising, are dropped.
To keep the kept set varied, the examples are first grouped by the kind of question they ask, and the best-scoring ones are taken from each group. The authors report that training on about 15% of the LLaVA-665K data picked this way performs about as well as training on all of it, on their benchmark average.
TL;DR
- VisNec is a per-sample score: the response loss without the image minus the response loss with it §3.1 eq. 5.
- Samples with a score of zero or below are removed; the rest are ranked and the top 15% of each question-type cluster is kept §3.2.
- Author-reported: 100.2% of full-data relative performance with 15% of LLaVA-665K (LLaVA-v1.5-7B, LoRA, 1 epoch) tab. 1; 115.8% on Vision-Flan-186K tab. 2.
- The official code is public, but it does not drop non-positive scores before the top-15% cut, which the paper describes §3.2 (see Paper vs code).
Problem
Large multimodal instruction sets contain examples that text alone answers and examples where the image contradicts the response §1. Training on all of them costs compute, and misaligned examples can degrade visual reasoning §1. Existing selectors use generic importance or diversity signals, gradient influence, or text-only loss, and so do not separate the image's own contribution §1 §2. A text-only loss score (IFD) cannot tell a hard text from an image that misleads the model §2. Gradient-based and API-based selectors are costly §2.
Method
Score. For sample (v, t, y), the paper defines the visual necessity score as the blind loss minus the multimodal loss §3.1 eq. 5. The multimodal loss is the negative log-likelihood of the response given image and text eq. 1. The blind loss uses the same response with the image removed eq. 4. A positive score means the image lowers the loss (vision-critical); about zero means the image adds nothing (redundant); negative means the image raises the loss (misaligned or noisy) §3.1.
Blind pass. Image token positions are replaced by padding and their attention mask is set to zero, so no image information reaches the response tokens. Image positions are excluded from the loss in both passes, so both losses average over the same response positions §3.1.
Two-stage selection. In the first stage every sample is scored with the target MLLM, and samples with non-positive scores are discarded §3.2 fig. 2. In the second stage the question part of each instruction (system prompt removed) is embedded and clustered with K-means into K clusters, K = 20 in the experiments §3.2 §4.1. Within each cluster the remaining samples are ranked by score and the top r% are kept, r = 15 in the main experiments §3.2 eq. 6 §4.1. The clustering keeps task types represented, which the authors argue a plain top-k by score does not §3.2 §4.6.
Fine-tuning. The selected subset is used to LoRA fine-tune LLaVA-v1.5-7B for one epoch at learning rate 2e-4, starting from the LLaVA-v1.5 stage-1 checkpoint §4.1. The authors state that all experiments follow the official training configuration §4.1.
Relative performance. Rel. is, for each benchmark, the model's score divided by the full-data model's score, averaged over benchmarks §4.1 eq. 7.
# Visual necessity of one sample [§3.1] [eq. 5]
def visnec_score(model, image, text, response):
# Multimodal pass: negative log-likelihood of the response [§3.1] [eq. 1]
loss_mm = response_nll(model, image, text, response)
# Blind pass: image slots become padding, attention to them masked [§3.1]
blind_inputs = pad_and_mask_image(image, text)
# Same response positions; image positions excluded from the loss [eq. 4]
loss_blind = response_nll(model, blind_inputs, response)
# Positive: the image lowers the loss [§3.1]
return loss_blind - loss_mm# Subset selection [§3.2] [eq. 6]
def select_subset(samples, scores, num_clusters=20, ratio=0.15):
# Embed the question part of each instruction [§3.2]
# (system prompt removed; embedding model not stated in the paper)
questions = [extract_question(s.instruction) for s in samples]
embeddings = embed(questions) # (N, d)
# Task-intent clusters from K-means [§3.2] [§4.1]
cluster_ids = kmeans(embeddings, num_clusters) # (N,)
selected = []
for k in range(num_clusters):
members = [i for i in range(len(samples)) if cluster_ids[i] == k]
# Remove non-positive scores before ranking [§3.2]
members = [i for i in members if scores[i] > 0]
# Rank by score, keep the top r% of the cluster [§3.2] [eq. 6]
# (whether r% counts the cluster before or after the filter is
# not stated; this follows the text order; rounding not stated)
members.sort(key=lambda i: scores[i], reverse=True)
keep = round(ratio * len(members))
selected += members[:keep]
return selectedKey equations (numbering as printed in the paper):
\mathcal{L}_{\text{MM}}(\theta; v, t, y) = -\sum_{j=1}^{L} \log P(y_j \mid y_{<j}, v, t; \theta) eq. 1
\mathcal{L}_{\text{Blind}}(\theta; t, y) = -\sum_{j=1}^{L} \log P(y_j \mid y_{<j}, t; \theta) eq. 4
S_{\text{VisNec}}(v, t, y) = \mathcal{L}_{\text{Blind}}(\theta; t, y) - \mathcal{L}_{\text{MM}}(\theta; v, t, y) eq. 5
\mathcal{D}_{\text{select}} = \bigcup_{k=1}^{K} \text{Top-}{r\%} \left( \{ (v_i, t_i, y_i) \in \mathcal{D} \mid \pi(q_i) = k \} ; S_{\text{VisNec}} \right) eq. 6
\text{Rel} = \frac{\text{The Model's Performance}}{\text{Full Fine-tuned Model's Performance}} \times 100\% eq. 7, per benchmark, then averaged
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Base model (experiments) | LLaVA-v1.5-7B, from the stage-1 pre-trained checkpoint | §4.1 | Feature-alignment checkpoint, as stated |
| Qwen2.5-VL models | 3B, 7B, 32B | §4.3 tab. 3 | Instruct vs base variant not stated |
| Fine-tuning method | LoRA | §4.1 | LoRA rank, alpha, dropout: not stated |
| Epochs | 1 | §4.1 | |
| Learning rate | 2e-4 | §4.1 | Scheduler, warmup, weight decay: not stated |
| Batch size, optimizer, max sequence length | not stated | §4.1 | Repo's 13B LoRA script uses batch 16 and max length 2048 finetune_lora.sh finetune_lora.sh; not confirmed for 7B |
| Budget ratio r | 15% (default) | §4.1 tab. 1 | Sweep 5% to 20% §4.4 fig. 4 |
| Subset size, LLaVA-665K | 98K samples | tab. 1 caption | 98K is about 14.7% of 665K; how it was derived is not stated |
| Subset size, Vision-Flan-186K | 28K samples | tab. 2 caption | About 15% of 186K |
| Number of clusters K | 20 (default) | §4.1 §3.2 | Stable from 10 to 30 within 1.3% Rel. §4.4 fig. 4 |
| Question embedder | not stated in paper | §3.2 | Code uses all-MiniLM-L6-v2 task_distribution.py |
| Clustering algorithm | K-means | §3.2 | Code: MiniBatchKMeans, random_state 42, batch 2048 task_distribution.py |
| Scoring batch size | 1 | Code only | cal_visnec.py |
| Scoring precision and attention | bf16, FlashAttention-2 | Code only | cal_visnec.py |
| Image resolution in scoring | 336 x 336, pad aspect ratio | Code only | cal_visnec.py |
| Scoring max length | 2048 | Code only | cal_visnec.sh |
Results
Author-reported. Ten evaluation sets cover visual question answering, knowledge-grounded QA, multiple-choice perception, text reading in images, open-ended chat and hallucination checks §4.1; column names are in tab. 1. Judge and evaluation toolkit for the open-ended benchmarks: not stated. Number of runs and variance: not stated; no error bars appear in any table.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| LLaVA-v1.5-7B, LLaVA-665K, 15% (98K) | Rel. over 10 benchmarks | Random 94.2%; best other 97.7% (PreSel, CoIDO) | 100.2% | tab. 1 |
| LLaVA-v1.5-7B, Vision-Flan-186K, 15% (28K) | Rel. | Random 96.7%; best other 103.6% (CoIDO) | 115.8% | tab. 2 |
| Qwen2.5-VL-3B / 7B / 32B, LLaVA-665K, 15% | Rel. | Random 98.8% / 98.7% / 97.9% | 103.8% / 104.0% / 102.4% | tab. 3 |
| LLaVA-v1.5-7B, loss ablation | Rel. | Text-only 95.6%; multimodal-only 94.0% | 100.2% | tab. 4 |
| LLaVA-v1.5-7B, clustering ablation | Rel. | Top-VisNec without clusters 97.0% | 100.2% | tab. 5 |
| LLaVA-665K cost | H100 GPU-hours | Full data 76.0 | 23.0 (12.0 selection + 11.0 fine-tuning) | tab. 6 |
Per-benchmark notes from tab. 1: VisNec scores above full data on LLaVA-Wild, MMBench-en, MMBench-cn and MM-Vet; it is below on VQAv2, GQA, SQA-I, TextVQA, MME-P and POPE §4.2. On Vision-Flan, VisNec is above full data on eight of the ten benchmarks and below on VQAv2 (65.0 vs 69.6) and POPE (82.0 vs 85.7) tab. 2.
Sensitivity (values in fig. 4 only, not in the text): clusters k from 10 to 30 keep Rel. above 98.9% (fluctuation 1.3%); the selection ratio gives 99.1% at 5%, 100.2% at 15% and 99.2% at 20% §4.4.
Ablations: text-only selection beats random, multimodal-only falls below random, and the combined score wins §4.5 tab. 4. Removing clusters costs 3.2 points §4.6 tab. 5.
Cost: VisNec's selection pass takes 12.0 GPU-hours; the paper's other selectors range from 9.0 (PreSel, plus API cost) to 73.5 (Self-Filter) tab. 6. The authors state VisNec uses no external LLM API §4.7.
Paper vs code
Repository: https://github.com/DMK041218/VisNec, linked from the paper's project
page at commit d894598ef9ba3e03da847a47464185c6262d230f. The owner is not
shown as an author; the link comes from the project page the paper names. The
abstract (v2) says code will be released upon acceptance, while the project
page and README now link the repository README.md.
Differences found by reading the scripts:
- Non-positive filter is missing. The paper removes samples with score
at or below zero before ranking §3.2.
task_distribution.pyranks the whole cluster and keeps the top 15% by score, with no sign test [code: scripts/task_distribution.py#L69-L81]. Eq. 6 as printed also has no filter eq. 6. If a cluster has fewer than 15% positive samples, the released code keeps non-positive ones. - Cluster size rule. The code keeps int(0.15 × cluster size), with a minimum of one task_distribution.py. The paper's rounding is not stated.
- Question extraction. The paper extracts the question and removes the
system prompt §3.2. The code takes the first human turn, removes
<image>and newlines, and does no system-prompt filtering [code: scripts/task_distribution.py#L22-L27]. - Embedding and clustering. The paper does not name an embedder or K-means variant §3.2. The code uses all-MiniLM-L6-v2 [code: scripts/task_distribution.py#L16] and MiniBatchKMeans with seed 42 [code: scripts/task_distribution.py#L62].
- Loss reduction. Eqs. 1 and 4 sum over response tokens, while the text says the losses are averaged over response tokens §3.1. The scoring script uses the model's returned loss cal_visnec.py and does not show which reduction it is.
- Blind pass length. The paper says the two passes differ only in visual
input §3.1. In the scoring script, the blind pass sets
images=Noneafter replacing image tokens with padding cal_visnec.py. LLaVA's forward returns before it expands the image placeholder into patch embeddings when images are None [code: LLaVA/llava/model/llava_arch.py#L150-L151]. So the two sequences are not the same length. This is inferred from the code and not tested here. - Scoring checkpoint. The paper says only "target MLLM" §3.2. The script
loads
llava-v1.5-7b_vicunawith a pretrainedmm_projector.bin[code: scripts/cal_visnec.sh#L10-L13]. - Vision-Flan path. The scoring script writes
dataset_visionflan_large_fixed_visnec_scores.csv[code: scripts/cal_visnec.py#L94]. The selection script readsresults/dataset_visnec_scores.csvtask_distribution.py and only handles the LLaVA JSON task_distribution.py. No Vision-Flan selection step is in the repository, and the file must be renamed to match. - Fine-tuning recipe. The README points to LLaVA's
finetune_lora.sh[code: README.md#L76-L79]. That script useslmsys/vicuna-13b-v1.5, LoRA rank 128, alpha 256, batch 16, learning rate 2e-4, one epoch, cosine schedule and warmup 0.03 finetune_lora.sh. The paper's 7B settings are not given beyond learning rate and epochs.
No discrepancies were checked for Qwen runs; the repository has no Qwen scripts.
Pitfalls
- Filter order. Follow the text (filter, then top-r%), not the released code, unless you want to match the released subsets. The two give different subsets when a cluster has few positive samples. Eq. 6 is ambiguous on the same point.
- Absolute cut under a relative claim. The paper says a bias from the out-of-distribution blind input cancels in relative ranking §3.1. The non-positive filter is an absolute threshold at zero, so a constant offset moves the cut. The paper does not test this.
- Reduction matters. A sum over response tokens makes scores grow with response length. A token mean does not. Pick one, and check it against the LLaVA training loss you use. The paper does not say which it used.
- Two full forward passes. Every pool sample needs two passes with the target model before selection fig. 2 §3.2. The paper reports 12.0 GPU-hours for this step tab. 6 and does not describe how that was measured.
- Budget count. 98K is about 14.7% of 665K, not 15% (about 99.75K) tab. 1. The paper does not explain the difference.
- Stated margins don't match the table. The text says VisNec beats Random by 5.8% and the second-best by 2.3% on LLaVA-665K §4.2; the table gives 6.0 and 2.5 points (100.2 − 94.2; 100.2 − 97.7) tab. 1. It says 18.4% over Random on Vision-Flan §4.2; the table gives 19.1 points (115.8 − 96.7) tab. 2. Use the table values. The 4.6% margin in §4.5 is 100.2 − 95.6 percentage points.
- Rel. is a mean of ratios. Recomputing from tab. 1 gives 100.2% and from tab. 3 (3B) gives 103.8%, so Rel. is the mean of the per-benchmark ratios eq. 7. A few benchmarks dominate: MM-Vet is 107% of full data in tab. 1.
- Open-ended judging. LLaVA-Wild and MM-Vet depend on a judge the paper does not name §4.1. Those two columns are the ones where VisNec is most above the full-data run.
- "Cross-architecture" means self-scoring. The Qwen experiment scores each Qwen model with itself and trains on its own subset §4.3 tab. 3. It does not test reusing one model's subset with another model.
- Single-run tables. No seeds, run counts or error bars are reported for the subset comparisons.
- Sensitivity values are only in a figure. The k and ratio claims §4.4 are checkable only against fig. 4.
- Notation. The paper uses K for cluster count in equations and k in §4.4.
- Code and data availability. The released repository has no LICENSE file at its root, so it cannot be quoted. The Hugging Face artifacts linked from the README were not inspected.
Lineage
- Builds on: V-usable information (Ethayarajh et al., 2025) for the "uniquely reduced uncertainty" view §2 eq. 2; LLaVA-v1.5 for the base model and training pipeline §4.1.
- Loss-based text-only selection it extends: IFD (Li et al., 2024) §2.
- Compared against (named in §4.1): Random, Self-Filter, EL2N, TypiClust, IFD, PreSel, XMAS, COINCIDE, CLIP-Score, CoIDO, and ICONS in the tables tab. 1 tab. 2.
- Related work from the same group named in the repository README: OnceSelect (arXiv:2605.26761) README.md. No card exists for it.
- Superseded by: not known from this wiki.
Source and license
Adapted from “VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning” by Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu (arXiv:2603.01195), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/DMK041218/VisNec at commit d894598ef9ba (none (no LICENSE file at the repository root; the vendored LLaVA/ folder is Apache-2.0)). Any quoted code is shown under that license.