Mutual information maximization

Overview

The shared idea: a model is trained with a task loss plus a weighted term that asks its representation to keep mutual information I(X;Y) with another variable Y, such as the other input modalities 2109.00412 §1 2109.00412 §3.6. Mutual information cannot be computed exactly for continuous, high-dimensional vectors, so every variant replaces it with a tractable lower bound or estimator that is itself trained 2109.00412 §3.4 2109.00412 §3.5. The only card so far applies this at two places in a multimodal sentiment model: between pairs of input channels, and between the fused vector and each channel 2109.00412 §3.2.

Variants

Input-level bound: Barber-Agakov with a mixture entropy

The mutual information is lower-bounded by the expected log-density of a learned conditional q(y|x) plus the entropy H(Y) 2109.00412 §3.4. The predictor q is Gaussian, with mean and variance computed from x, and is fitted in its own stage before the main update 2109.00412 §3.6. The entropy term needs a density estimate for Y. The paper uses a two-class Gaussian mixture, one class per polarity of the sentiment label, and keeps a first-in-first-out memory of earlier batches so that each class has enough samples 2109.00412 §3.4 2109.00412 app. A.1. Text is the input X and the other modality is Y for each pair 2109.00412 §3.4. Card: MMIM.

Fusion-level bound: contrastive term on the fused vector

The fused vector is trained to predict each modality's vector through a predictor G, with an InfoNCE-style loss that treats the other items of the batch as negatives 2109.00412 §3.5. Both sides are normalised to unit length before the dot product. The paper says that without this the two vectors grow in norm to raise the score 2109.00412 §3.5. The three per-modality losses are summed 2109.00412 §3.5. Card: MMIM.

When to use which

The table compares what each term needs and what the paper's ablation shows. The ablation has one run per row and no spread 2109.00412 tab. 4, so differences of a few thousandths in MAE are not established.

Implementation sketch

The common core is a task loss plus two weighted MI terms, trained in two stages per epoch 2109.00412 §3.6 2109.00412 alg. 1. The input-level term enters with a minus sign, since the loss is the negative bound 2109.00412 eq. 8.

# Two-stage epoch with weighted MI terms  [§3.6] [alg. 1]
# Inputs: batches of (text, visual, acoustic, label); weights alpha, beta

def train_epoch(batches, alpha, beta, memory):
    # Stage 1: fit only the Gaussian predictors q  [eq. 3]
    for batch in batches:
        step_predictors(nll_of_pairs(batch))
    # Stage 2: every other parameter  [eq. 13]
    for batch in batches:
        fusion, y_hat = fusion_net(batch)
        # Task loss on the regression head  [eq. 12]
        loss = mean(abs(y_hat - batch.label))
        # Fusion-level contrastive term  [eq. 11]
        loss += alpha * contrastive_fusion(fusion, batch)
        # Input-level bound, loss = -I_BA  [eq. 8]
        loss += beta * -ba_bound_pairs(batch, memory)
        update_all_but_predictors(loss)
        # FIFO memory of visual and acoustic vectors  [alg. 1]
        memory.push(detached_visual_acoustic(batch))

Variant choices show up in three places: how q is parameterised, how the entropy is estimated, and the similarity used in the contrastive term 2109.00412 §3.4 2109.00412 §3.5. The card's "Paper vs code" section lists where the reference code departs from the paper. Read it before reimplementing either term.

Contradictions

None known across cards: only one card lists this concept. The paper's own text and its released code differ in details, such as the Gaussian log-likelihood and the direction of the contrastive softmax. These are listed on the paper card and are not conflicts between papers.

Papers

• MMIM: an input-level Barber-Agakov bound with a mixture entropy, and a fusion-level contrastive term, both added to a regression loss for sentiment on CMU-MOSI and CMU-MOSEI.