Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis

In plain words

Words, tone of voice and facial expression all carry feeling, and the words usually carry the most of it. The common shortcut is to squash each signal into a vector and glue the vectors together. This paper instead lets the text vector interact with the audio vector and with the video vector. Every text number is multiplied by every audio number (and likewise for video), so the result keeps pairwise combinations rather than only the separate parts.

Those interaction grids go through small image-style networks, and the two networks are trained together so that their outputs agree as much as possible in a statistical sense (canonical correlation). The idea is that the audio side and the video side, each combined with the same sentence, should line up with each other. The aligned outputs are joined to the original text vector and fed to a simple predictor of sentiment or emotion.

The authors report good results on three public sentiment and emotion datasets. Their paper and released code disagree in several places, so the numbers are hard to reproduce exactly.

TL;DR

• ICCN (Interaction Canonical Correlation Network) builds text-based audio and text-based video features with outer products, then trains two 2D CNNs with a deep CCA loss.
• The final embedding is the two CNN outputs joined to the BERT text vector; it feeds a downstream classifier or regressor.
• Author-reported: best on CMU-MOSI in all five metrics; on CMU-MOSEI best on MAE and 7-class accuracy only; on IEMOCAP best in three of four emotions.
• The released code is partial and its model selection uses the test set; reproduce from the paper's equations, not the reported numbers.

Problem

Text features usually beat audio and video features on sentiment and emotion tasks. The authors give two reasons: text encoders are much more developed, and audio and video features are hand-engineered §1. Simple concatenation of the three modalities, or pairwise tensor products, ignores the correlation between the text-based audio and text-based video features §1, §2. Linear CCA is limited to linear maps, and DCCA adds non-linear networks but has no text interaction terms §2, §5.

Method

Subsections are unnumbered in the compiled paper, because the source sets secnumdepth to 1. They are cited here as the enclosing section plus the subsection title.

Overview: the paper's pipeline (Figure 1 fig. 1) extracts sentence-level features for each modality. Text is a single vector. Audio and video are frame sequences, each run through a 1D convolution and an LSTM whose final hidden state is the utterance feature [§3, "Text Based Audio Video Interaction Canonical Correlation"]. Each non-text feature is then combined with the text vector by an outer product, giving one interaction matrix per modality eq. 5. Two 2D CNNs turn the interaction matrices into vectors, and a DCCA-style loss trains them to be maximally correlated §3, "CCA and DCCA"; alg. 1. The concatenation [K_ta; H_t; K_tv] of the two CNN outputs and the text vector is the multimodal embedding §3, "Pipe-line for Downstream Tasks".

CCA background (paper's notation):

A^{*}, B^{*} = \arg\max_{A,B} \mathrm{corr}(A^{T}X, B^{T}Y) = \arg\max_{A,B} \frac{A^{T}S_{12}B}{\sqrt{A^{T}S_{11}A}\sqrt{B^{T}S_{22}B}} eq. 1

Closed form through the SVD of Z = S_{11}^{-1/2} S_{12} S_{22}^{-1/2}, with A^{*} = S_{11}^{-1/2}U, B^{*} = S_{22}^{-1/2}V, and the total correlation written in the paper as printed:

\mathrm{corr}(A^{*T}X, B^{*T}Y) = \mathrm{trace}(Z^{T}Z)^{1/2} eq. 2

DCCA objective, with networks f and g and their outputs F_X, F_Y:

\theta_f^{*}, \theta_g^{*} = \arg\max_{\theta_f,\theta_g} \mathrm{corr}(C^{*T}F_X, D^{*T}F_Y) eq. 3

CCA loss used for back-propagation, with E = R_{11}^{-1/2} R_{12} R_{22}^{-1/2} built from the covariances of F_X and F_Y:

\text{CCA Loss} = -\mathrm{trace}(E^{T}E)^{1/2} eq. 4

Interaction matrices, with H_t the text vector and H_{a2}, H_{v2} the LSTM utterance features:

H_{ta} = H_t \otimes H_{a2} \in \mathbb{R}^{d_t \times d_{a2}}, \quad H_{tv} = H_t \otimes H_{v2} \in \mathbb{R}^{d_t \times d_{v2}} eq. 5

Algorithm 1 gives the training loop in full-batch form alg. 1:

# Interaction Canonical Correlation Network training  [alg. 1]
# Input:  text (N, d_t); audio (N, l_a, d_a); video (N, l_v, d_v)
# Output: text-based audio K_ta and text-based video K_tv (N, k)

def train_iccn(text, audio, video, epochs, lr):
    # Initialize the audio branch (1D-CNN, LSTM, 2D-CNN) and the
    # video branch (1D-CNN, LSTM, 2D-CNN)  [alg. 1]
    audio_branch = init_branch()
    video_branch = init_branch()
    for epoch in range(epochs):
        # Encode each frame sequence  [alg. 1] [§3]
        audio_seq = conv1d_a(audio)              # (N, l_a, d_a1)
        video_seq = conv1d_v(video)              # (N, l_v, d_v1)
        # Keep the final LSTM hidden state as the utterance feature  [§3]
        audio_vec = lstm_a(audio_seq)            # (N, d_a2)
        video_vec = lstm_v(video_seq)            # (N, d_v2)
        # Text-based interaction matrices  [eq. 5] [alg. 1]
        text_audio = outer(text, audio_vec)      # (N, d_t, d_a2)
        text_video = outer(text, video_vec)      # (N, d_t, d_v2)
        # Map each interaction matrix with its 2D-CNN  [alg. 1]
        k_ta = cnn2d_ta(text_audio)              # (N, k)
        k_tv = cnn2d_tv(text_video)              # (N, k)
        # Maximize canonical correlation between the two outputs  [eq. 4]
        loss = cca_loss(k_tv, k_ta)
        # Gradient step on the weights of both branches  [alg. 1]
        grads = gradients(loss, audio_branch, video_branch)
        audio_branch = update(audio_branch, grads, lr)
        video_branch = update(video_branch, grads, lr)
    return k_ta, k_tv                            # [alg. 1]

The CCA loss itself, as implemented in the released code (not spelled out in the paper). The paper's eq. 4 corresponds to the commented "all singular values" branch in the code; the code's default branch keeps only the top k:

# CCA loss on two batches of paired outputs  [eq. 4] [§3]
def cca_loss(first, second, k):
    # Center each view over the batch  [code: objectives.py#L31-L32]
    first = first - mean(first, over_batch)
    second = second - mean(second, over_batch)
    # Covariances with 1e-3 * I on the diagonals  [code: objectives.py#L36-L40]
    cov_first = first.T @ first / (n - 1) + 1e-3 * eye
    cov_second = second.T @ second / (n - 1) + 1e-3 * eye
    cov_cross = first.T @ second / (n - 1)
    # Whitening via eigendecomposition; eigenvalues <= 1e-9 dropped
    # [code: objectives.py#L46-L66]
    whiten_first = inv_sqrt(cov_first)
    whiten_second = inv_sqrt(cov_second)
    t_matrix = whiten_first @ cov_cross @ whiten_second
    # Sum of the k largest singular values  [code: objectives.py#L78-L84]
    corr = sum(top_k_singular_values(t_matrix, k))
    return -corr

Pipeline for the downstream task: after the CNNs are trained, the paper concatenates the two CNN outputs with the text vector, [K_ta; H_t; K_tv], and feeds that to a simple downstream model (logistic regression or an MLP are named; the paper says "a simple MLP is used in this work") [§3, "Pipe-line for Downstream Tasks"; §4, Hyperparameter Tuning]. Test results use the setting picked by validation performance §4, Hyperparameter Tuning.

Figures: Figure 1 shows this pipeline fig. 1. Figures 2 and 3 plot mean canonical correlation and mean cosine similarity between the two CNN outputs on CMU-MOSI, once for each training loss. The authors read these curves as showing that canonical correlation and cosine similarity do not move together [fig. 2, fig. 3; §5, "Results of Ablation Studies"].

Hyperparameters

Values from the paper are given with their section. The paper's tuning ranges are listed, but it does not state the values finally chosen §4, Hyperparameter Tuning. Code values come from the released MOSEI export (mosei_dcca_exp (2).html) and the MOSI notebook (Get_CCA_embedding.ipynb); see Paper vs code.

Results

Author-reported, on CMU-MOSI, CMU-MOSEI and IEMOCAP. The setup is the paper's shared sentence features (BERT for MOSI and MOSEI, InferSent for IEMOCAP), with every baseline's embedding fed to the same downstream classifier or regressor §4, Baseline Methods; Hyperparameter Tuning. TFN, LMF and MFM were re-run by the authors on the same features. DCCA numbers marked with an asterisk are from the original DCCA paper tab. 1. Hardware is not stated.

Table 1: CMU-MOSI and CMU-MOSEI tab. 1 (higher is better except MAE; best per column in bold in the source)

Table 2: IEMOCAP, four emotions, binary accuracy and F-score tab. 2 (best in bold in the source)

Table 3: ablation on CMU-MOSI and CMU-MOSEI tab. 3 (ICCN row repeats Table 1)

Reading the headline numbers: ICCN is best on MOSI in all five metrics tab. 1. On MOSEI, MFM is higher on binary accuracy, F-score and correlation, while ICCN is best on MAE and 7-class accuracy tab. 1. On IEMOCAP, ICCN is best for happy, angry and sad, and MFM is best for neutral tab. 2. The paper's discussion says ICCN beats the uni-modal and concatenation baselines on every criterion, and that the CCA-based baselines do not improve the embedding §5, "Performance on Benchmark Datasets".

Paper vs code

Repository. zsun227/ICCN is a non-fork repository under the first author's GitHub account. The account name matches the e-mail prefix on the paper's author line. It has no LICENSE file, so only path-and-line citations are used and no code is quoted. Its README is one line, # ICCN README.md.

What the repository contains. There is no standalone ICCN training script. The evidence is in three places:

• Get_CCA_embedding.ipynb: the MOSI CCA stage with its random search and feature export Get_CCA_embedding.ipynb. It is not runnable as released: mosi_train, mosi_b_train and seen_settings are never assigned, and the continue at cell level has no loop around it in the notebook Get_CCA_embedding.ipynb.
• mosei_dcca_exp (2).html: a saved export of a MOSEI notebook. It contains the full ICCN model, the CCA training loop, the downstream MLP and the printed metrics mosei_dcca_exp (2).html mosei_dcca_exp (2).html. Line numbers below refer to this file.
• models.py, objectives.py, DeepCCAModels.py, main.py: a DCCA baseline and the CCA loss. main.py is a DCCA demo on noisy MNIST (784-d inputs, 10 outputs, batch 800, one epoch), not ICCN main.py.

Architecture.

• The audio and video projections are 1x1 Conv1d layers with no bias, not 1D convolutions with local support. The paper says the 1D convolution extracts local structure §3 mosei_dcca_exp (2).html.
• The LSTM is one unidirectional layer with hidden size 50, and the final hidden state is the utterance feature. This matches §3 mosei_dcca_exp (2).html.
• The outer product is the text vector (768) times the 50-d LSTM state, giving a 768 × 50 grid for each branch mosei_dcca_exp (2).html. The flattened CNN input is 6303 = 3 × 191 × 11 (the number of channels and the two conv/pool stages from the code), which matches fc_dim mosei_dcca_exp (2).html.
• The 2D CNN blocks use 3 channels, kernel 3, stride 2, padding 1, BatchNorm and MaxPool(2,1). None of these values is in the paper mosei_dcca_exp (2).html.
• The hidden_dim argument of MY is accepted but never used mosei_dcca_exp (2).html.
• The text vector is passed raw, so BERT is not fine-tuned, as the paper says mosei_dcca_exp (2).html.

CCA loss. objectives.py implements the loss with two branches, and the paper does not separate them.

• The branch that uses all singular values computes sqrt(trace(T^T T)), which is the Frobenius norm of the whitened cross-covariance objectives.py. The paper's eq. 4 is written as trace(E^T E)^{1/2}. That text can be read either as this Frobenius form or as the trace of the matrix square root (the sum of singular values). The code's two branches are not the same value.
• The default branch keeps only the top k singular values, with k = 20 in the MOSEI export and k = cov_cca_dim in the MOSI notebook. It is not in the paper objectives.py mosei_dcca_exp (2).html Get_CCA_embedding.ipynb.
• The covariance regularizer 1e-3 and the eigenvalue floor 1e-9 are not in the paper objectives.py.

Training.

• Algorithm 1 is full-batch with an epoch counter alg. 1. The code uses shuffled mini-batches inside each epoch mosei_dcca_exp (2).html and the MOSI notebook does the same Get_CCA_embedding.ipynb. One "epoch" therefore means many more updates than Algorithm 1 implies.
• The paper says "basic Grid-Search" §4, Hyperparameter Tuning. The code draws random settings and skips repeats mosei_dcca_exp (2).html.
• The learning-rate grid includes 2e-3 in the MOSEI export, which is outside the paper's range of 1e-5 to 1e-3 §4, Hyperparameter Tuning mosei_dcca_exp (2).html.
• Dropout is stored, but not applied. MY keeps self.p = p but the forward pass has no dropout layer, so the searched dropout1 values have no effect in the export mosei_dcca_exp (2).html. Dropout on the downstream model is applied after the second hidden layer mosei_dcca_exp (2).html.
• Both the export and the notebook use RMSprop with the learning rate only, so PyTorch defaults apply to the other arguments and weight decay is zero mosei_dcca_exp (2).html Get_CCA_embedding.ipynb.
• main.py passes weight_decay=reg_par for its DCCA demo, with reg_par = 1e-5 main.py. The ICCN export and notebook set no weight decay.

Feature order. The export concatenates [K_a, K_v] with the BERT vector, giving 868 = 50 + 50 + 768 inputs [code: mosei_dcca_exp (2).html#L13535-L13541, L13569]. The paper writes [K_ta; H_t; K_tv] §3. The notebook also puts the text vector last Get_CCA_embedding.ipynb.

Downstream model. The paper says "a simple MLP" §4, Hyperparameter Tuning. The export uses an 868-input regressor with two hidden ReLU layers, trained with L1 loss and Adam for up to 50 epochs. Its hidden width comes from {256, 768, 512} and its learning rate from {1e-3, 2e-3, 5e-4, 7e-4}. The 768 hidden width is above the paper's 512 limit mosei_dcca_exp (2).html mosei_dcca_exp (2).html.

Model selection uses the test set. Within a run, the reported test score is taken at the epoch with the best validation MAE mosei_dcca_exp (2).html. Across runs, the code keeps the single lowest test MAE, over every setting, seed, downstream width and learning rate, and prints that run's metrics mosei_dcca_exp (2).html. The paper says the best hyperparameters are chosen on validation performance §4, Hyperparameter Tuning. So the MOSEI numbers in the export are optimistic.

Saved output does not match Table 1. The printed best run in the export is hidden 512, learning rate 0.002, seed 3432: MAE 0.5693, correlation 0.7122, 7-class accuracy 0.5083, F-score 0.8433, binary accuracy 0.8432 mosei_dcca_exp (2).html. Table 1 reports MOSEI MAE 0.565, correlation 0.713, 7-class accuracy 51.58, F-score 84.15 and binary accuracy 84.18 tab. 1. The printed best line gives only the downstream width, downstream learning rate and seed. The CCA-stage tuple is printed separately for each setting, so the winning CCA-stage setting is not identified. Table 1 cannot be reproduced from the released log.

Metric definitions in code. The code gives definitions that the paper leaves open mosei_dcca_exp (2).html.

• Binary accuracy and F-score drop utterances whose label is zero, and count a prediction as positive when it is above zero. This resolves the paper's [-3, 0) / (0, 3] split §4, Evaluation Methods.
• F-score uses weighted averaging. The paper does not say which average it uses.
• Seven-class accuracy clips predictions to [-3, 3], rounds them, and compares to the rounded label. The paper says only that the classes come from the label distribution §4, Evaluation Methods.
• Correlation is the Pearson corrcoef over all utterances.

Dataset and inputs. The MOSEI export loads a pickle named mosei_bert_inf.pickle with BERT, vision and audio entries mosei_dcca_exp (2).html. The pickle is not in the repository, and neither is the feature extraction. The MOSI notebook refers to its own tensors, which are also absent Get_CCA_embedding.ipynb.

Dependencies. models.py evaluates np.float as a default argument at import time, so the module fails to import with NumPy 1.24 or later models.py. main.py uses r1 and r2 without defining them main.py, and its losses list is never filled, so test() returns the mean of an empty list main.py. main.py also imports linear_cca and utils, which are not in the repository main.py.

IEMOCAP. No IEMOCAP code is released. The IEMOCAP numbers in Table 2 cannot be checked against code tab. 2.

Pitfalls

• CCA loss has two readings. The paper's eq. 4 can mean the Frobenius norm of the whitened cross-covariance or the sum of its singular values. The code's full branch computes the Frobenius norm, and its default branch sums only the top k singular values. Pick one, state it, and check the implementation against the reported numbers before trusting the result.
• The top-k truncation is not in the paper. It changes the loss value and the gradient for the discarded components. The export fixes k at 20 while the CCA output has 50 dimensions mosei_dcca_exp (2).html.
• Covariances are estimated per mini-batch. The loss depends on batch size through the estimate, and the paper does not say whether the covariance is over the mini-batch or the full set. The code uses batches of 256 to 400 for MOSEI.
• Regularization values are code-only. The code adds 1e-3 to the covariance diagonals and drops eigenvalues at or below 1e-9 before whitening objectives.py. The paper gives neither value, and changing them changes the loss.
• Unpublished 2D-CNN shape. The fully connected input size is tied to the interaction-matrix shape: 3 × 191 × 11 = 6303 for a 768 × 50 input. A different audio or video width changes the shape, and the paper gives no formula for it.
• Unused dropout. dropout1 in the MOSEI grid has no effect in the export mosei_dcca_exp (2).html. The paper names no dropout for the CCA stage, so state whether you apply any.
• Test-set selection inflates the numbers. Select every setting on validation MAE, and report test MAE only for the selected setting. The export's best run is chosen on test MAE mosei_dcca_exp (2).html.
• Padding and masking are unspecified. Audio and video sequences have variable length, and the LSTM's final hidden state is taken without a mask. The paper says nothing about padding, and the feature pickle is not in the repository, so the utterance vector may include padding frames.
• Ablation inputs are unstated. "No text" (ICCN1) applies DCCA to audio and video directly, but the paper does not say how the two raw sequences are combined before DCCA §4, Ablation studies of ICCN; tab. 3.
• Ranges are not values. The paper gives tuning ranges but not the chosen learning rate, batch size, epochs, dimensions, or dropout. Table 1 cannot be reproduced from the paper alone.
• Feature dimensions are fixed in code. MY(768, 74, 35, 50, 6303, 50, ...) hard-codes the BERT, COVAREP, FACET and output sizes mosei_dcca_exp (2).html. Changing any of them requires recomputing fc_dim.
• Result claims. The paper calls ICCN state-of-the-art §6, but Table 1 shows MFM ahead on several MOSEI and IEMOCAP metrics tab. 1, tab. 2. Quote the table, not the summary.

Lineage

• Builds on: CCA §2, §3 and deep CCA, DCCA §2, §3, "CCA and DCCA", both cited as in the paper. Its outer-product interaction idea follows the tensor fusion work TFN and LMF §2, §4, Baseline Methods. Text encoders are BERT and InferSent §4, Multi-modal Features.
• Compared against: CCA, KCCA, GCCA, DCCA (the variant of Sun et al. 2019) §4, Baseline Methods, TFN, LMF and MFM §4, Baseline Methods, and unimodal and concatenation baselines §4, Baseline Methods.
• Superseded by: not stated in the paper. No later work is cited here.