Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

In plain words

Models that read text, look at video and listen to audio can give a correct answer for many reasons. Often it is hard to tell whether the answer came from the picture alone, from the sound alone, from the question text, or from the two senses working together. This paper asks how much each kind of information actually shapes the model's answer.

The tool is a way of splitting information into parts. One part is held by only one input, one part is held by both inputs equally, and one part only appears when the inputs are combined. The paper adapts this split to omni-modal models, keeps the text question fixed as context, and measures the parts from the model's answer probabilities, with one input at a time replaced by noise to get the partial cases.

The main finding is that, on audio-visual questions designed to need both senses, the answers still depend mostly on information that only one sense provides, and vision is the stronger of the two. The authors also test a training idea: give more weight to training examples where the two senses should work together, and less to examples where the text alone seems to decide. In their vision-language experiment this helps some models and not others.

TL;DR

• Sensory PID splits the information that video and audio carry about the answer into a video-only part, an audio-only part, a shared part and a joint (synergy) part, with the text instruction held fixed §2.1 eq. 1.
• The estimate is training-free: three answer distributions per sample, with the missing stream replaced by noise, plus the BATCH estimator §2.2 eq. 2 eq. 3.
• On three omni-modal models and MUSIC-AVQA, unique visual information is larger than unique audio information and synergy stays small, even on the audio-visual fusion subset tab. 1 §4.1.
• Modality shuffling supports the asymmetry: shuffling video hurts more than shuffling audio (author-reported, figure only) §4.1.2 fig. 2.
• PID-guided LoRA reweighting on vision-language models gives author-reported gains on Qwen2.5-VL-7B and LLaVA-OneVision-7B, not on Gemma3-12B tab. 2 tab. 3 §5.

Problem

Accuracy alone does not show which input a model uses, and attention or ablation analyses do not separate information that is unique to one input from information that needs both §1. A full three-source decomposition over video, audio and text would be large and would treat the instruction like a sensory input §2.1. The paper therefore conditions on the text and decomposes only the sensory information gain §2.1.

Method

Sensory PID decomposes the conditional information between the answer Y and the two sensory streams V and A, given the instruction T §2.1 eq. 1:

• U_vis: information only the video gives, given T.
• U_aud: information only the audio gives, given T.
• R_sens: information both streams give, given T.
• S_av: information that only appears when the two streams are used together, given T.

Three predictive distributions are needed: the full p(Y | V, A, T), and the two partial ones p(Y | V, T) and p(Y | A, T) §2.1 eq. 2. The partial ones are not native to these models. The paper approximates them by replacing one stream's embeddings with Gaussian noise whose per-dimension mean and variance come from a profiling set, while keeping the text §2.2 eq. 3. Each stream is first mean-pooled into one vector after projection into the shared token space, so that BATCH gets fixed-size inputs §2.2.

The candidate answers are handled as follows. Predictions are restricted to the candidate options with a selective softmax. The probability mass left on the candidates before renormalisation sets a reliability factor, and each distribution is shrunk toward uniform in proportion to how little mass it kept §2.2 eq. 4 eq. 5 eq. 6. The paper states that this continuous rule replaces the hard threshold used in earlier vision-language PID work §2.2. The target marginal is the average of the full-model distributions over the profiling set, and it is held fixed during fitting §2.2 eq. 7. BATCH then learns a Sinkhorn-normalised coupling that respects the source-to-answer marginals, and the four terms are read off that coupling §2.2 app. A.2.

# Sensory PID for one omni-modal model and one benchmark  [§2.1] [eq. 1]
# Input:  videos, audios, texts; candidate options; profiling set
# Output: U_vis, U_aud, R_sens, S_av at dataset level

def sensory_pid(videos, audios, texts, candidates, profile):
    # Three text-conditioned predictions  [§2.1] [eq. 2]
    p_vat = stabilise(model(videos, audios, texts), candidates)
    p_vt = stabilise(model(videos, mask(audios, profile), texts), candidates)
    p_at = stabilise(model(mask(videos, profile), audios, texts), candidates)
    # Mean-pool each sensory stream to a fixed-size vector  [§2.2]
    x_v = mean_pool(videos)
    x_a = mean_pool(audios)
    # Soft target marginal over the profiling set, kept fixed  [§2.2] [eq. 7]
    marginal = average(p_vat, over=profile)
    # Sinkhorn coupling fitted by BATCH  [§2.2] [app. A.2]
    coupling = batch_fit(x_v, x_a, p_vat, p_vt, p_at, marginal)
    # Conditional recovery formulas are not stated in the paper
    return recover_terms(coupling)

# Restrict to candidates, then shrink toward uniform  [§2.2] [eq. 4]-[eq. 6]
# Input:  full-vocabulary probs; candidate indices; tau
# Output: stabilised distribution over the candidates

def stabilise(probs, candidates, tau):
    # Mass that falls on the valid options, before renormalising  [eq. 4]
    mass = sum(probs[c] for c in candidates)
    # Renormalise over the valid options (selective softmax)  [§2.2]
    restricted = renormalise(probs, candidates)
    # Reliability in [0, 1]; tau is not stated for omni-modal  [eq. 5]
    rho = min(1, mass / tau)
    # Shrink toward the uniform distribution over the candidates  [eq. 6]
    return rho * restricted + (1 - rho) / len(candidates)

# PID-guided sample weights for VLM fine-tuning  [§2.3] [alg. 1]
# Input:  per-sample local terms s, u_txt, u_vis, r (K=50 batch draws);
#         frozen-model answer distributions p_v, p_t, p_vt; counts k_s, k_g
# Output: one training weight per sample

def pid_weights(s, u_txt, u_vis, r, p_v, p_t, p_vt, k_s, k_g):
    # Clip each local term at zero and add them  [eq. 8]
    info = clip(s) + clip(u_vis) + clip(u_txt) + clip(r)
    # Synergy ratio and shortcut score, eps = 1e-8  [eq. 9]
    sr = clip(s) / (info + 1e-8)
    sc = clip(u_txt) / (info + 1e-8)
    # Fusion potential from answer entropies  [eq. 10]
    fp = clip(min(entropy(p_v), entropy(p_t)) - entropy(p_vt))
    # Gap score: high fusion potential, low synergy, low shortcut  [eq. 11]
    gap = (1 - sr) * (1 - sc) * fp
    # Shortcut samples first, then gap samples among the rest  [eq. 12]
    shortcut = top_k(sc, k_s)
    gap_set = top_k(gap, k_g, exclude=shortcut)
    # Weights 3.0 for gap samples, 0.5 for shortcut samples, else 1.0  [§2.3]
    return where(gap_set, 3.0, where(shortcut, 0.5, 1.0))

The paper does not state a compute budget. For the omni-modal analysis, each sample needs three forward passes (full, and two with one stream masked) §2.1 §2.2.

Hyperparameters

Results

Author-reported. Setup: omni-modal runs on MUSIC-AVQA (three subsets, 1,800 samples each) with Qwen2.5-Omni 3B and 7B and VITA-1.5 7B, zero-shot §3 app. E. Baseline: none beyond the models themselves; the comparison is between subsets and between the four PID terms. Hardware not stated.

Sensory PID spectrum tab. 1. Bold marks the largest term in each model-subset pair, as in the paper. "Acc" is full audio-visual accuracy and "Acc1" is visual-only accuracy with audio removed, as the table caption states.

Modality shuffling §4.1.2 fig. 2: the paper reports that shuffling video causes larger accuracy drops than shuffling audio, on AV-Fusion, for Qwen2.5-Omni 7B. The Spearman values are in the figure and are not repeated in the text.

Layer-wise dynamics §4.2 fig. 3: unique visual information rises earlier across depth than unique audio information and synergy, for the three omni-modal models. The paper describes the trajectories only in words, and the figure carries the curves.

PID-guided reweighting, Qwen2.5-VL-7B, mean ± std over three seeds tab. 2 §5.1. Post-S_vl and post-U_txt are measured on a held-out MMStar subset.

Reported comparisons: C vs B is +2.3 pp on MMStar, 95% paired-bootstrap interval [1.4, 3.2] §5.1 app. H. The MMStar gain over random, accuracy-based and ablation-based selection is +2.8, +1.8 and +1.3 pp §5.1 app. H.

Cross-model tab. 3 app. H:

In the appendix, MMMU for LLaVA-OneVision-7B goes from 53.1 to 52.8 and for Gemma3-12B from 59.3 to 54.1 app. H.

The vision-language reference analysis over 20 models and six benchmarks is reported as broadly consistent with earlier work and is deferred to the appendix §3 app. F.

Paper vs code

No official code.

Searched: the LaTeX source (no code or project link; the only URL is an acknowledgement); the arXiv metadata in the repository (abstract only, no comments field); and GitHub repository search for "Sensory PID multimodal", "partial information decomposition mllm", "modality interaction omni" and "alvin chan pid". No repository belonging to the authors turned up, and the third-party hits were not used.

Pitfalls

• Conditional recovery is missing. Eq. 1 defines the four terms, and the estimator is described, but the appendix writes the recovery formulas only for the unconditioned vision-language case app. A.2. How U_vis, U_aud, R_sens and S_av are read from the coupling under the text condition is not stated. Do not assume the vision-language formulas carry over unchanged.
• Two stabilisation rules. The vision-language appendix describes a hard gate that replaces low-mass predictions with uniform at tau = 0.3 app. A.2. The main text says the hard gate does not suit Sensory PID and uses the continuous shrinkage instead §2.2. The omni-modal tau is not stated.
• Profiling set undefined for omni-modal runs. Its size and composition are not stated. The masking statistics, the target marginal and the BATCH fit all depend on it.
• Noise length not given. The masked stream is replaced with noise of length L_m, but the paper does not say how L_m is set or whether it matches the original token count.
• BATCH settings are sets, not values. Learning rate, epochs and batch size are given as sets app. D. The values used per run, the Sinkhorn iteration count and the stopping rule are not stated. Only the local-score runs fix batch size 256 and K = 50 app. A.3.
• Sensitivity checks cover only vision-language. Mean pooling and tau were varied on MMBench for three vision-language models app. G. No omni-modal sensitivity result is given.
• Audio-Focused breaks the visual skew. The text says unique visual information "generally" exceeds unique audio information §4.1. On the Audio-Focused subset, unique audio is the bold, larger term for VITA-1.5 (0.59 vs 0.54) and Qwen2.5-Omni 3B (0.64 vs 0.48) tab. 1. The skew holds on AV-Fusion and on the full selection.
• Share definition is inferred. The text quotes synergy shares of 67.5% and 73.9% without a formula. The numbers are reproduced by S / (S + U_txt + U_vis + R), which I checked against the appendix table (1.36 / 1.841 = 73.9%). The paper does not state this formula §5.2 app. H.
• Small arithmetic gaps in the appendix. The effective-percentage column in the training distribution table gives 34.0% and 11.1%, while 1,167 / 3,405 is 34.3% and 373 / 3,405 is 11.0% app. H. The shortcut count is 746 in the conditions table and about 750 in the ablation text. Use the counts, not the rounded percentages.
• Gemma3-12B is described as "no benefit", but one drop is outside the seed noise. MMStar falls from 56.2 to 54.3 with a reported std of ±0.5, and the appendix calls the +0.3 MMBench change within seed variation §5.3 tab. 3 app. H. The paper's own wording says the method does not create fusion capacity where the baseline lacks it §5.3.
• The reweighting experiment is vision-language, not omni-modal. Section 5 uses the two-source pipeline, and the omni-modal Sensory PID analysis does not feed the training weights §5. Porting the reweighting to omni-modal models would need new choices.
• Causality is explicitly not claimed. The synergy shift after LoRA-PID is described as consistent with the method, not as the cause of the accuracy gain §5.2.
• Three seeds, and bootstrap intervals only for MMStar. The 95% intervals are given only for the MMStar comparisons §5.1 app. H. MMBench and POPE gains have no interval.
• Local scores are heuristics. The paper says they are not pointwise PID atoms and are used only as ranking signals §2.3 app. A.3. The synergy ratio and shortcut score are normalised by the information mass, not derived from PID.
• Sample-level scores depend on batch composition. The Sinkhorn coupling changes with the batch, which is why K = 50 batch draws are averaged §2.3 app. A.3. The cost of this averaging is not reported.
• Small evaluation. Three omni-modal models and one benchmark (MUSIC-AVQA) support the findings. Claims such as "sensory synergy bottleneck" are limited to "the evaluated systems" §4.1.
• Figures carry numbers the text lacks. The shuffling Spearman values fig. 2 and the layer-wise curves fig. 3 are not in the text.

Lineage

• Builds on: the BATCH estimator of Liang et al. (2023) for the estimation step §2.2; masking by representation corruption, as in Meng et al. (2022) app. A.2.
• Prior work by the same authors: Fang et al. (2026), which used PID to inform representation alignment and multimodal intervention design §1 §5.
• Closest concurrent work: Xiu et al. (2026), decision-level PID for large vision-language models. The paper says its vision-language estimator choices closely follow that work §1 §2.2.
• Compared against (reweighting): random weights, accuracy-based difficulty mining, and modality-ablation selection §5.1.
• Superseded by: not stated.