---
kind: concept
slug: mutual-information-maximization
title: Mutual information maximization
summary: >-
  Add a loss term that rewards a learned representation for sharing information with another variable. Mutual information is hard to compute exactly, so each variant uses a bound or estimator. One card so far: a Barber-Agakov bound on input pairs plus a contrastive term on the fused vector, for multimodal sentiment. Author-reported gains are small; the bound needs extra predictors and a memory.
tags: [mutual-information, representation-learning, multimodal, infonce]
papers: ["2109.00412"]
related: [contrastive-learning, multimodal-fusion, sentiment-analysis, variational-inference]
status: draft
updated_at: 2026-10-24
---

## Overview

The shared idea: a model is trained with a task loss plus a weighted term that
asks its representation to keep mutual information I(X;Y) with another
variable Y, such as the other input modalities [arxiv:2109.00412 §1] [arxiv:2109.00412 §3.6].
Mutual information cannot be computed exactly for continuous, high-dimensional
vectors, so every variant replaces it with a tractable lower bound or
estimator that is itself trained [arxiv:2109.00412 §3.4] [arxiv:2109.00412 §3.5].
The only card so far applies this at two places in a multimodal sentiment
model: between pairs of input channels, and between the fused vector and each
channel [arxiv:2109.00412 §3.2].

## Variants

### Input-level bound: Barber-Agakov with a mixture entropy

The mutual information is lower-bounded by the expected log-density of a
learned conditional q(y|x) plus the entropy H(Y) [arxiv:2109.00412 §3.4]. The
predictor q is Gaussian, with mean and variance computed from x, and is fitted
in its own stage before the main update [arxiv:2109.00412 §3.6]. The entropy
term needs a density estimate for Y. The paper uses a two-class Gaussian
mixture, one class per polarity of the sentiment label, and keeps a
first-in-first-out memory of earlier batches so that each class has enough
samples [arxiv:2109.00412 §3.4] [arxiv:2109.00412 app. A.1]. Text is the
input X and the other modality is Y for each pair [arxiv:2109.00412 §3.4].
Card: [MMIM](/papers/2109.00412).

### Fusion-level bound: contrastive term on the fused vector

The fused vector is trained to predict each modality's vector through a
predictor G, with an InfoNCE-style loss that treats the other items of the
batch as negatives [arxiv:2109.00412 §3.5]. Both sides are normalised to unit
length before the dot product. The paper says that without this the two
vectors grow in norm to raise the score [arxiv:2109.00412 §3.5]. The three
per-modality losses are summed [arxiv:2109.00412 §3.5]. Card: [MMIM](/papers/2109.00412).

## When to use which

The table compares what each term needs and what the paper's ablation shows. The ablation has one run per row and no spread
[arxiv:2109.00412 tab. 4], so differences of a few thousandths in MAE are not
established.

| Situation | Use | Why |
|---|---|---|
| A fused vector exists and an extra predictor per modality is acceptable | Fusion-level contrastive term | Needs only the predictor G on the fused vector [arxiv:2109.00412 §3.5]. Removing all three terms raised MOSEI MAE from 0.526 to 0.543 in the paper's one run [arxiv:2109.00412 tab. 4]. |
| Dependence between raw input channels should be controlled too | Input-level bound | Needs a Gaussian predictor per pair, a separate training stage, and a memory [arxiv:2109.00412 §3.4] [arxiv:2109.00412 §3.6]. Using all three pairs gave MOSEI MAE 0.527 against 0.541 with none [arxiv:2109.00412 tab. 4]. |
| Small batches or few samples per polarity class | Avoid the mixture entropy, or test it first | Without the history memory, the paper's run diverged to NaN in every metric [arxiv:2109.00412 tab. 4]. |
| Both terms wanted | Combine with weights α and β | The paper weights the fusion term by α and the input term by β, with separate values per dataset [arxiv:2109.00412 §3.6] [arxiv:2109.00412 §4.3]. |

## Implementation sketch

The common core is a task loss plus two weighted MI terms, trained in two
stages per epoch [arxiv:2109.00412 §3.6] [arxiv:2109.00412 alg. 1]. The
input-level term enters with a minus sign, since the loss is the negative
bound [arxiv:2109.00412 eq. 8].

```pseudocode
# Two-stage epoch with weighted MI terms  [§3.6] [alg. 1]
# Inputs: batches of (text, visual, acoustic, label); weights alpha, beta

def train_epoch(batches, alpha, beta, memory):
    # Stage 1: fit only the Gaussian predictors q  [eq. 3]
    for batch in batches:
        step_predictors(nll_of_pairs(batch))
    # Stage 2: every other parameter  [eq. 13]
    for batch in batches:
        fusion, y_hat = fusion_net(batch)
        # Task loss on the regression head  [eq. 12]
        loss = mean(abs(y_hat - batch.label))
        # Fusion-level contrastive term  [eq. 11]
        loss += alpha * contrastive_fusion(fusion, batch)
        # Input-level bound, loss = -I_BA  [eq. 8]
        loss += beta * -ba_bound_pairs(batch, memory)
        update_all_but_predictors(loss)
        # FIFO memory of visual and acoustic vectors  [alg. 1]
        memory.push(detached_visual_acoustic(batch))
```

Variant choices show up in three places: how q is parameterised, how the
entropy is estimated, and the similarity used in the contrastive term
[arxiv:2109.00412 §3.4] [arxiv:2109.00412 §3.5]. The card's "Paper vs code"
section lists where the reference code departs from the paper. Read it before
reimplementing either term.

## Contradictions

None known across cards: only one card lists this concept. The paper's own
text and its released code differ in details, such as the Gaussian
log-likelihood and the direction of the contrastive softmax. These are listed
on the paper card and are not conflicts between papers.

## Papers

- [MMIM](/papers/2109.00412): an input-level Barber-Agakov bound with a mixture entropy, and a fusion-level contrastive term, both added to a regression loss for sentiment on CMU-MOSI and CMU-MOSEI.