Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

In plain words

Videos of people talking carry sentiment in three channels at once: the words they say, their face and gestures, and the sound of their voice. A common way to predict sentiment is to squeeze these three signals into one combined summary and read the score from it. The risk is that the combining step can throw away, or pick up, details that matter for the score.

This paper adds an explicit check on that combining step. The model is rewarded when the combined summary still shares information with each of the original channels, and with the text in particular. The "shared information" is measured with mutual information, a standard quantity from information theory that is hard to compute exactly, so the paper uses approximations that can be trained with ordinary gradient descent.

The check is applied at two places: between pairs of raw channels, and between the combined summary and each channel. Training optimises the sentiment error together with these extra terms. The authors report results that are better than or comparable to the baselines they compare with, on two public datasets of video clips.

TL;DR

• MMIM predicts a sentiment score from text, visual and acoustic inputs and trains with two mutual-information terms added to the usual regression loss §3.2 §3.6.
• The input-level term is a Barber-Agakov lower bound on pairs (text, visual) and (text, acoustic), with an entropy estimate from a two-class Gaussian mixture and a memory of earlier batches §3.4.
• The fusion-level term is a contrastive loss that asks the fusion vector to predict each modality §3.5.
• Author-reported results beat or match the listed baselines on CMU-MOSI and CMU-MOSEI, with gains reported as small on several metrics tab. 2 §4.3.
• The official code is MIT-licensed and differs from the paper in several defaults and details; read "Paper vs code" before reimplementing.

Problem

Multimodal sentiment analysis (MSA) takes unaligned raw sequences of text, video features and audio features from one video segment and predicts a sentiment value §3.1 §4.3. Earlier methods either rely on the task loss alone or reshape the feature space (for example by matrix decomposition or Euclidean-distance losses) §1, and the paper argues that neither controls how much task-relevant information survives fusion §1.

The paper evaluates on CMU-MOSI, with 2199 segments labelled from -3 to +3, and CMU-MOSEI, a larger set of YouTube review clips §4.1 tab. 1. The evaluation metrics are MAE, Pearson correlation, seven-class accuracy (Acc-7), binary accuracy and F1, each in two variants for the binary case §4.1 tab. 2.

Method

Overview

Raw inputs become three fixed-size vectors: a BERT embedding for the text and LSTM outputs for the visual and acoustic streams §3.2 §3.3. A fusion network F maps the three vectors to a fusion vector Z, and a regression head maps Z to the prediction ŷ §3.2. Two parts train together: the task and the fusion network, and the mutual-information (MI) terms, which use predictors that are trained in a separate stage §3.2 §3.6 alg. 1. The architecture is shown in fig. 1.

Encoders §3.3 eq. 1

The text vector h_t is the first-token (head) output of the last BERT layer, with BERT fine-tuned §3.3. The visual and acoustic vectors h_v and h_a come from unidirectional LSTMs, one per stream §3.3. The paper calls these representations "unit-length" §3.2 without defining the term.

h_t = \mathrm{BERT}(X_t;\theta_t^{BERT}), \qquad h_m = \mathrm{sLSTM}(X_m;\theta_m^{LSTM}), \; m \in \{v, a\} eq. 1

Input-level MI: inter-modality bound §3.4

For the pairs (text, visual) and (text, acoustic) the text is X and the other modality is Y §3.4. The Barber-Agakov bound reads eq. 2:

I(X;Y) \ge \mathbb{E}_{p(x,y)}[\log q(y|x)] + H(Y) \triangleq I_{BA} eq. 2

The bound is tight when q equals the true conditional p(y|x) §3.4. The predictor q is a Gaussian whose mean and variance are functions of x §3.4. Its negative log-likelihood, summed over the two pairs, is the loss eq. 3:

\mathcal{L}_{lld} = -\frac{1}{N}\sum_{tv,ta}\sum_{i=1}^{N}\log q(y_i|x_i) eq. 3

The entropy H(Y) is estimated with a Gaussian mixture of two classes, the non-negative and the negative polarity §3.4. Each class has a mean and a covariance estimated from a batch, and the covariance uses the element-wise and outer-product notation in eq. 4 §3.4. The entropy of a multivariate normal is eq. 5. The mixture entropy is bounded below and above by eq. 6. With equal class weights, the paper takes the lower bound as the estimate eq. 7. The appendix shows the upper bound equals the lower bound plus 0.693 app. A.2.

\hat{\mu}_c = \frac{1}{N_c}\sum_{i=1}^{N_c} h^i_c, \qquad \hat{\Sigma}_c = \frac{1}{N_c}\sum_{i=1}^{N_c} h^i_c \odot h^i_c - \hat{\mu}_c^T\hat{\mu}_c eq. 4

H = \frac{1}{2}\log\left((2\pi e)^k\det(\Sigma)\right) eq. 5

\sum_c w_c h_c \le H(Y) \le \sum_c w_c(-\log w_c + h_c) eq. 6

H(Y) = \frac{1}{4}\left[\log\left(\det(\Sigma_1)\det(\Sigma_2)\right)\right] eq. 7

Because the class statistics would need a large batch, the paper keeps a first-in-first-out memory of earlier embeddings and estimates the mixture on the current batch plus the memory §3.4 app. A.1 alg. 1. The input-level loss is eq. 8:

\mathcal{L}_{BA} = -I_{BA}^{t,v} - I_{BA}^{t,a} eq. 8

Fusion-level MI: contrastive term §3.5

The fusion vector Z = F(X_t, X_v, X_a) is trained to predict each modality representation h_m through a predictor G with parameters φ §3.5. Both the prediction and the target are normalised to unit length before a dot product eq. 9. The paper states that without this normalisation the two vectors stretch to raise the score §3.5:

s(h_m, Z) = \exp\left(\overline{h_m}\,\overline{G_\phi(Z)}^{\,T}\right) eq. 9

The loss is an InfoNCE-style estimate that treats the other items of the batch as negatives eq. 10 §3.5:

\mathcal{L}_N(Z,\mathbf{H}_m) = -\mathbb{E}_{\mathbf{H}}\left[\log\frac{s(Z,h_m^i)}{\sum_{h_m^j\in\mathbf{H}_m}s(Z,h_m^j)}\right] eq. 10

\mathcal{L}_{CPC} = \mathcal{L}_{N}^{z,v} + \mathcal{L}_{N}^{z,a} + \mathcal{L}_{N}^{z,t} eq. 11

Training §3.6 alg. 1

Each epoch has two stages. In stage 1 only the predictors q are updated, by minimising eq. 3 alg. 1. In stage 2 all other parameters are updated, with the loss eq. 12 and eq. 13 §3.6 alg. 1:

\mathcal{L}_{task} = \mathrm{MAE}(\hat{y}, y) eq. 12

\mathcal{L}_{main} = \mathcal{L}_{task} + \alpha\,\mathcal{L}_{CPC} + \beta\,\mathcal{L}_{BA} eq. 13

α and β weight the two MI terms §3.6. The paper's training procedure is alg. 1; the pseudocode below restates it, with the equations it uses.

# MM-MIM training  [alg. 1] [§3.6]
# Input:  data D of (X_t, X_v, X_a, y); alpha, beta; memory M
# Output: fusion network F, regression head, predictors q

def train(dataset, alpha, beta, memory):
    for epoch in epochs:
        # Stage 1: fit the predictors q, nothing else  [alg. 1]
        for batch in subset_of(dataset):            # D_sub ⊆ D  [alg. 1]
            text, visual, acoustic = encode(batch)  # [eq. 1]
            # Negative log-likelihood of both pairs  [eq. 3]
            step_predictors_only(-(lld_tv + lld_ta))
        # Stage 2: update every parameter except q  [alg. 1]
        for batch in dataset:                       # D  [alg. 1]
            loss = stage2_loss(batch, memory, alpha, beta)
            update_all_but_predictors(loss)
        # Store this batch's visual and acoustic embeddings  [alg. 1]
        memory.push(batch_embeddings_v_a(batch))    # first in, first out

# Stage 2 loss for one batch  [eq. 8] [eq. 11] [eq. 12] [eq. 13]
# The code switches these terms on and off by flag. See "Paper vs code".

def stage2_loss(batch, memory, alpha, beta):
    # Encode each modality to a fixed-size vector  [eq. 1]
    text, visual, acoustic = encode(batch)
    # Fusion vector and prediction  [§3.2]
    fusion, y_hat = fusion_network(text, visual, acoustic)
    # Input-level bounds, text as X  [eq. 2]
    i_ba_tv = ba_bound(text, visual, labels, memory.tv)
    i_ba_ta = ba_bound(text, acoustic, labels, memory.ta)
    # Fusion-level contrastive terms  [eq. 11]
    l_cpc = sum(nce(fusion, h) for h in (text, visual, acoustic))
    # Task loss  [eq. 12]
    l_task = mean(abs(y_hat - labels))
    # L_BA = -I_BA for both pairs  [eq. 8]
    l_ba = -(i_ba_tv + i_ba_ta)
    return l_task + alpha * l_cpc + beta * l_ba  # [eq. 13]

# Barber-Agakov bound for one pair, text as X  [eq. 2] [§3.4]
# (x, y) are the pair's vectors, shape (B, d_x) and (B, d_y)

def ba_bound(x, y, labels, memory):
    # Gaussian predictor q(y|x): mean and log-variance  [§3.4]
    mean = predictor_mean(x)                     # (B, d_y)
    log_var = predictor_log_var(x)               # (B, d_y)
    # Gaussian log-density, averaged over the batch  [eq. 3]
    # The full density includes the log-variance term.
    # The code drops it. See "Paper vs code".
    log_q = -0.5 * (y - mean) ** 2 / exp(log_var) - 0.5 * log_var
    expected_log_q = mean(sum(log_q, dim=1))
    # Entropy of Y from the two-class mixture  [eq. 7]
    entropy = gmm_entropy(y, labels, memory)
    return expected_log_q + entropy              # I_BA  [eq. 2]

# Entropy of Y from two polarity classes  [§3.4] [eq. 4] [eq. 7]
# Uses the current batch plus the memory from earlier batches

def gmm_entropy(y, labels, memory):
    # Pool the memory with the current batch  [app. A.1] [alg. 1]
    values = concat(memory.values, y)
    signs = concat(memory.labels, labels)
    # Two classes: non-negative and negative  [§3.4]
    pos = values[signs >= 0]
    neg = values[signs < 0]
    # Full covariance per class, by maximum likelihood  [eq. 4]
    cov_pos = covariance(pos)
    cov_neg = covariance(neg)
    # Lower bound with equal weights, 1/4 factor  [eq. 7]
    return 0.25 * (logdet(cov_pos) + logdet(cov_neg))

# Contrastive term for one modality  [eq. 9] [eq. 10] [§3.5]

def nce(fusion, truth):
    # Predict the modality vector from the fusion  [§3.5]
    prediction = predictor_g(fusion)            # (B, d_m)
    # Unit-length vectors, so the score is a cosine  [eq. 9]
    prediction = prediction / norm(prediction, dim=1)
    truth = truth / norm(truth, dim=1)
    # Row i: fusion i. Column j: modality vector j  [eq. 10]
    scores = exp(prediction @ truth.T)          # (B, B)
    # Other batch items are negatives  [§3.5] [eq. 10]
    # Use log-sum-exp in practice; the paper does not say.
    return -mean(log(diag(scores) / sum(scores, dim=1)))

Hyperparameters

The paper gives the values in tab. 3 for the best run on each dataset, and the ranges it searched §4.3. The code defaults are shown separately, because the repository's config.py does not match the paper's table in several places (see "Paper vs code").

Other stated values: the paper ran the model five times under the same settings and averaged the results §4.3. It trained on a single RTX 2080Ti §4.3.

Results

Author-reported. Setup: CMU-MOSI and CMU-MOSEI, unaligned raw inputs, BERT as the text encoder for every compared model, one RTX 2080Ti §4.1 §4.3 tab. 2. The paper marks the baseline rows with † (numbers taken from the MISA paper), ‡ (numbers taken from the Self-MM paper) and * (numbers the authors reproduced from open-source code, using the hyperparameters of the original papers) tab. 2. The rows below copy the table; the source's bold best-value marks and significance symbols are left out. The table caption says the MMIM improvements marked as significant are better than the state of the art at p < 0.05 by a paired t-test tab. 2.

tab. 2

Dataset sizes as the paper gives them tab. 1:

tab. 1

Ablation, all on CMU-MOSEI, full model in the first row tab. 4. Columns are MAE, Corr, Acc-7, Acc-2 (non-neg / pos), F1 (non-neg / pos):

tab. 4

The paper's analysis section plots the losses over training on MOSEI and reads the best epoch's MI losses as a sign that the input-level term matters more than the fusion-level term §5 fig. 2. A case study lists CPC scores for four examples §5 tab. 5. Both are one-run illustrations, not statistics.

Paper vs code

Code: https://github.com/declare-lab/Multimodal-Infomax, MIT licence, checked at commit cd0774c5a712ca5f1a5497dbf27dde11cade7434 (main, shallow clone). The repository is linked from the paper's abstract and is owned by the DeCLaRe lab, to which the authors belong (abstract; LICENSE). The README gives the training command python main.py --dataset mosi --contrast README.md. Code cited below is under raw/code/2109.00412/.

• Gradient clip. Paper: 5.0 tab. 3. Code default: 1.0; its help text says 0.8 config.py.
• α and β. Paper, MOSI: α 0.3, β 0.1; MOSEI: α 0.1, β 0.05 tab. 3. Code defaults: α 0.1, β 0.1 for both config.py. The README command sets neither, so it runs MOSI with the MOSEI-like pair (0.1, 0.1).
• Predictor learning rate. Paper: η_lld 5e-3 for MOSI, 1e-3 for MOSEI tab. 3. Code default is 1e-3 for both config.py.
• Hidden sizes. Paper: visual LSTM hidden 32 (MOSI) and 64 (MOSEI); acoustic 32 and 16 tab. 3. Code default: 16 and 16 config.py.
• Batch size. Paper: 32 (MOSI), 64 (MOSEI) tab. 3. Code default: 32 for both, and the README does not mention a flag for MOSEI config.py.
• Memory. Paper: grid over one to three batches §4.3. Code: fixed to one batch, not exposed as an argument solver.py.
• Polarity classes. Paper: non-negative and negative §3.4. Code: positive is strictly greater than zero, negative strictly less than zero, so zero-labelled samples are in neither class encoders.py.
• Gaussian log-likelihood. Paper: the negative log-likelihood of a Gaussian with covariance σ²I §3.4 eq. 3. Code: keeps the squared-error term scaled by exp(-logvar) and drops the -½·logvar term encoders.py. The log-variance is also not bounded: the last activation argument (mmilb_last_activation, Tanh by default) is stored but never applied encoders.py, encoders.py.
• Entropy input. Paper: the entropy is of Y as given §3.4 eq. 7. Code: projects y through a learned linear layer and Tanh (output size one quarter of the input) before the mixture encoders.py, encoders.py.
• Covariance. Paper: eq. 4 is ambiguous between a diagonal and a full matrix. Code: full covariance from outer products encoders.py. The diagonal version is present but commented out encoders.py.
• Unused constant. a = 17.0795 is assigned and never used encoders.py.
• CPC direction. Paper: for each fusion vector Z, the denominator of eq. 10 sums over the candidate modality vectors. Code: for each modality vector, the log-sum-exp runs over the predictions made from the fusion vectors of the batch encoders.py. This is the transposed InfoNCE. Follow eq. 10 if you reimplement the paper.
• Predictor G. Paper: a neural network G_φ, architecture not stated §3.5. Code default: a single linear layer, no activation config.py, encoders.py.
• Fusion network. Paper: "stacked linear-activation layers" §3.2. Code: a dropout, then two tanh layers of width 128, with the second tanh layer as the fusion vector, and a linear head for the prediction encoders.py. The code also computes an unused intermediate fusion value encoders.py.
• Loss composition. Paper: L_main = L_task + αL_CPC + βL_BA eq. 13. Code: the CPC and likelihood terms are added only with the --contrast flag, and the entropy term is subtracted whenever the batch index exceeds the memory size. Default runs (no flag) therefore train only the task loss plus the entropy term, and that entropy passes through predictor-side weights that are never trained solver.py, config.py.
• Stage 1 subset. Paper: D_sub ⊆ D, size not stated alg. 1. Code: the predictor pass runs over all training batches, except that for MOSEI it stops at half of them. It runs only with --contrast solver.py, solver.py.
• Optimiser. Paper: plain gradient steps θ ← θ - η∇ alg. 1. Code: Adam with weight decay, separate groups for BERT, main and predictor parameters solver.py.
• Scheduler. Code creates a plateau scheduler for the predictors but never steps it, so their rate stays fixed solver.py, solver.py.
• Five runs. Paper: five runs averaged §4.3. Code: the --multiseed flag is defined but not used config.py, main.py. One seed (1111) per invocation.
• Model selection. Paper: not stated. Code: saves and reports an epoch only when the validation loss improves and the test MAE is also lower than the best so far solver.py. The test set chooses the reported epoch.
• Dataset size. Paper text: 23,454 MOSEI clips §4.1. Paper split table: 22856 in total tab. 1.
• Text length. Not stated in the paper. Code: BERT inputs are truncated to 50 tokens data_loader.py.
• Single-item batches. The LSTM encoder squeezes the final hidden state, which can give the wrong shape for a batch of one encoders.py.
• Feature extraction. The paper names COVAREP and P2FA for visual and acoustic features §4.3. The loader reads them from prepared pickles and its feature dimensions are data-dependent create_dataset.py, data_loader.py.

Checked and matching the paper: the sign and weighting of L_BA and L_main eq. 8 eq. 13, the MAE task loss eq. 12, the ¼ factor of the entropy estimate eq. 7, and the three-modality sum in L_CPC eq. 11 solver.py, encoders.py.

Pitfalls

• Test-set model selection. The reported numbers come from an epoch chosen with the test set. Use a separate evaluation pass if you compare to this paper solver.py. The paper does not say how the epoch was chosen §4.3.
• Unbounded log-variance. Without the -½·logvar term, the likelihood term rises toward zero as the predicted variance grows, whatever the mean. The code also leaves the variance unbounded. Expect the predictor's variance to drift, and watch for it in training encoders.py.
• Covariance is numerically fragile. The entropy takes the log-determinant of a covariance matrix with no epsilon encoders.py. The paper's ablation without the history memory diverged to NaN in every metric tab. 4 §4.4. A class with fewer samples than dimensions, or nearly collinear samples, gives a singular covariance.
• Stale memory. The memory stores detached embeddings from the previous batch, computed with the previous parameters solver.py. The paper describes the memory as "the nearest history" and does not mention staleness §3.4.
• Zero-labelled samples. On MOSI and MOSEI, labels can be exactly zero. The code leaves them out of both polarity classes, while the paper's classes cover all samples encoders.py.
• "Unit-length" representations. The paper's phrase §3.2 is not defined. The code does not normalise h_m; normalisation happens only inside the CPC score encoders.py.
• Predictor input size. For the text-side pairs the predictor reads the 768-dimensional BERT head. The paper gives its reason for choosing text as X §3.4.
• Cost per epoch. With --contrast, each epoch runs a predictor pass before the main pass, on half of MOSEI's batches or all of MOSI's solver.py, solver.py.
• Significance. The table caption gives a paired t-test at p < 0.05 tab. 2. It does not say which runs the test used, and the table shows no spread across runs.
• Reporting the binary metrics. Acc-2 and F1 come in two variants: non-negative versus negative, and positive versus negative tab. 2. Pick the variant the baseline uses before you compare numbers. The code reports both eval_metrics.py.
• Ablation spread. The ablation table gives one number per setting, with no spread shown tab. 4. The row with all three inter-modality pairs has a higher Acc-7 than the full model but lower binary accuracy tab. 4.

Lineage

• Builds on: contrastive predictive coding, cited in the paper as Oord et al. 2018 §3.5; the code cites its arXiv id 1807.03748 encoders.py. The Barber-Agakov bound §3.4 (Agakov 2004, no arXiv id given). The Gaussian predictor follows CLUB, cited in the code as arXiv 2006.12013 §3.4 encoders.py. The mixture entropy bounds are from Huber et al. 2008 §3.4.
• Uses: BERT for text §3.3; LSTM encoders for visual and acoustic streams, following MISA and Self-MM §3.3; COVAREP and P2FA features §4.3.
• Compared against: TFN, LMF, MFM, MulT, ICCN, MISA, MAG-BERT and Self-MM §4.2 tab. 2.
• Superseded by: not stated in the wiki.