Partial information decomposition

Overview

Partial information decomposition (PID) asks how the information that several inputs carry about one target is shared between them. Each part is one of four kinds: information only one input gives, information both give, and information that appears only when the inputs are used together (synergy) 2606.00959 §2.1. The paper uses this to ask which sense a multimodal model relies on when it answers a question about a video with sound.

The invariant all variants share is that the four terms are defined from three predictive distributions of the target: one from all inputs, and one from each input alone 2606.00959 §2.1 eq. 2. Variants differ in how those distributions are obtained and how the terms are estimated.

Variants

Sensory PID for omni-modal models

The target is the model's answer, the inputs are the video stream and the audio stream, and the text instruction is held fixed as context. This leaves a four-term split: video-only, audio-only, shared and synergy 2606.00959 §2.1 eq. 1. The text is not decomposed, because treating it as a third source would enlarge the analysis 2606.00959 §2.1. Card: Sensory PID for omni-modal models.

The single-stream distributions are not native to these models. The paper approximates them by replacing one stream's embeddings with Gaussian noise matched per dimension to a profiling set, and keeps the text unchanged 2606.00959 §2.2 eq. 3. The estimator is BATCH, a learned coupling fitted under the source-to-answer marginals 2606.00959 §2.2 app. A.2. The training-free estimate needs no change to the model.

PID-guided sample weighting for fine-tuning

The same paper reuses the local terms as per-sample scores for LoRA fine-tuning of vision-language models. Samples with high fusion potential and low synergy and shortcut scores get a larger weight, samples that a text-only path can answer get a smaller one 2606.00959 §2.3 alg. 1. The scores are heuristics for ranking, not pointwise PID atoms [arxiv:2606.00959 §2.3] app. A.3. Card: the same paper card.

When to use which

Constraints to check before choosing: the number of forward passes (three per sample for Sensory PID) 2606.00959 §2.1, the size of the profiling set (not stated) 2606.00959 §2.2, and whether the target answer is discrete and enumerable.

Implementation sketch

The common core is: obtain three answer distributions per sample, stabilise them, and fit a coupling that yields the four terms. Where variants differ is noted inline. The recovery of the four terms from the coupling under a text condition is not stated in the paper 2606.00959 app. A.2.

# Sensory PID core, one target and two input streams  [arxiv:2606.00959 §2.1-§2.2]
# Input:  inputs a, b; fixed context t; candidate answers; profiling set
# Output: four terms (unique a, unique b, shared, synergy)

def pid_terms(a, b, t, candidates, profile):
    # Full-input answer distribution  [§2.1] [eq. 2]
    p_ab = stabilise(model(a, b, t), candidates)
    # Single-stream distributions, other stream replaced by noise  [§2.2] [eq. 3]
    p_a = stabilise(model(a, noise_like(b, profile), t), candidates)
    p_b = stabilise(model(noise_like(a, profile), b, t), candidates)
    # Pool each stream to a fixed-size vector before estimation  [§2.2]
    x_a = mean_pool(a)
    x_b = mean_pool(b)
    # Target marginal, averaged over profiling data and then held fixed  [§2.2] [eq. 7]
    marginal = average(p_ab, over=profile)
    # Coupling fitted with BATCH (Sinkhorn-normalised)  [app. A.2]
    coupling = batch_fit(x_a, x_b, p_ab, p_a, p_b, marginal)
    # Read the four terms off the coupling; text-conditioned formulas not stated
    return recover_terms(coupling)

# Shrink a candidate-restricted distribution toward uniform  [§2.2] [eq. 4]-[eq. 6]
# Input:  full-vocabulary probs; candidate ids; reliability threshold tau
# Output: distribution over the candidates

def stabilise(probs, candidates, tau):
    # Mass that falls on the valid options  [eq. 4]
    mass = sum(probs[c] for c in candidates)
    restricted = renormalise(probs, candidates)
    # Reliability in [0, 1]; the omni-modal tau is not stated  [eq. 5]
    rho = min(1, mass / tau)
    return rho * restricted + (1 - rho) / len(candidates)

Contradictions

None known. Only one card covers this concept, so no paper disagreement can be checked here. Within the card, the paper reports two stabilisation rules: a hard gate in the vision-language appendix and a continuous one in the main text for Sensory PID 2606.00959 §2.2 app. A.2. The paper states the continuous rule is the one it uses for Sensory PID, so this is not a disagreement between results.

Papers

• Sensory PID for omni-modal models: defines Sensory PID for omni-modal models and PID-guided LoRA sample weighting; results are author-reported and on MUSIC-AVQA and vision-language benchmarks.