Deep canonical correlation analysis

Overview

Canonical correlation analysis (CCA) finds one linear map per view so that the mapped views are as correlated as possible 1911.05544 eq. 1. Deep CCA (DCCA) keeps that objective but replaces the linear maps with neural networks. It trains them with a loss equal to the canonical correlation of their outputs 1911.05544 §3, eq. 3, eq. 4. The shared idea is that the training signal is a correlation between two sets of outputs, not a distance between paired outputs. The paper's cosine-loss variant tests exactly that difference 1911.05544 tab. 3.

The only source here is the ICCN paper. It uses DCCA inside a multimodal sentiment and emotion model. Its outer-product interaction follows the tensor fusion work it cites 1911.05544 §2, §4. The network forms a matrix of pairwise products between a text vector and each non-text summary, then applies DCCA to the two resulting matrices 1911.05544 §3, eq. 5.

Variants

ICCN: text-interaction DCCA (the paper's method)

Each audio or video sequence is summarised into one utterance vector. An outer product with the text vector gives one interaction matrix per modality. A 2D CNN maps each matrix to a vector, and the two outputs are trained with the CCA loss. The downstream embedding joins the two CNN outputs with the text vector 1911.05544 §3. It is best on CMU-MOSI in all five reported metrics 1911.05544 tab. 1. Card: Sun et al., ICCN (2019).

ICCN1: no text

The same two branches, with DCCA applied to the raw audio and video summaries and no text interaction. It scores 82.13 on MOSI binary accuracy and 83.01 on MOSEI, against 83.07 and 84.18 for ICCN 1911.05544 tab. 3. The paper does not say how the two raw sequences are combined before DCCA, so this variant cannot be rebuilt from the paper alone 1911.05544 §4.

ICCN2 and ICCN3: cosine loss

The CCA loss is replaced by a cosine similarity between the two outputs. ICCN2 keeps the text interaction and ICCN3 drops it. Binary accuracy is 82.32 and 81.49 on MOSI, and 82.98 and 82.59 on MOSEI 1911.05544 tab. 3. The paper plots canonical correlation and cosine similarity over training and reads the curves as not moving together 1911.05544 fig. 2, fig. 3, §5.

Linear CCA, KCCA, GCCA and DCCA as baselines

The baselines use the same text features and the same downstream model as ICCN 1911.05544 §4. On MOSI binary accuracy they score 79.45 (CCA), 79.82 (KCCA), 80.60 (DCCA) and 62.50 (GCCA). On MOSEI they score 82.94, 83.05, 83.62 and 75.12 1911.05544 tab. 1. The DCCA binary-accuracy and F-score entries are marked by the paper as taken from the original DCCA paper, not from its own runs 1911.05544 tab. 1.

When to use which

Implementation sketch

The paper's own training loop is full-batch 1911.05544 alg. 1. A mini-batch version is shown here, because the released code uses mini-batches [see card, Paper vs code]. Shapes are in comments.

# One DCCA training step for text-based interaction  [arxiv:1911.05544 §3]
# Inputs: text t (d_t); audio and video sequences
# Summarise each sequence into one utterance vector  [arxiv:1911.05544 §3]
a = encode_audio(audio_seq)      # (d_a)
v = encode_video(video_seq)      # (d_v)
# Outer product with the text vector  [arxiv:1911.05544 eq. 5]
m_a = outer(t, a)                # (d_t, d_a)
m_v = outer(t, v)                # (d_t, d_v)
# Map each interaction matrix to k numbers with a 2D CNN  [arxiv:1911.05544 §3]
k_a = cnn_audio(m_a)             # (k)
k_v = cnn_video(m_v)             # (k)
# Maximise canonical correlation of the two outputs  [arxiv:1911.05544 eq. 4]
loss = -cca_correlation(k_a, k_v)  # estimated on the batch
update(loss)                     # both branches
# After training: embed = concat(k_a, t, k_v)  [arxiv:1911.05544 §3]

Where variants differ: the cosine loss (ICCN2) replaces cca_correlation with a mean cosine similarity 1911.05544 tab. 3. The ICCN1 variant removes the outer products 1911.05544 tab. 3. How the CCA correlation is computed on a batch, with all singular values or only the top k, is not settled by the paper (see Contradictions).

Contradictions

• Reading of the CCA loss. The paper writes the objective as a trace of a matrix power 1911.05544 eq. 2, eq. 4. That can be the Frobenius norm of the whitened cross-covariance or the sum of its singular values. The card records that the released code's full branch computes the first and its default branch sums only the top k singular values, which are different values [see card, Paper vs code].
• "State of the art" versus Table 1. The paper calls ICCN state of the art 1911.05544 §6. Table 1 shows MFM ahead on MOSEI binary accuracy, F-score and correlation, and on IEMOCAP neutral [arxiv:1911.05544 tab. 1, tab. 2].
• CCA-based baselines. The paper says the CCA-based baselines do not improve the embedding 1911.05544 §5. In Table 1, DCCA's MOSEI correlation (0.707) is above the text-only row (0.681), and its MOSEI binary accuracy (83.62) is above it too 1911.05544 tab. 1. The accuracy numbers are the asterisked ones taken from the original DCCA paper, so only the correlation is from this paper's runs.
• Released log versus Table 1. The best MOSEI run in the released export has MAE 0.5693 and correlation 0.7122, while Table 1 reports 0.565 and 0.713 [see card, Paper vs code]. The paper says hyperparameters are chosen on validation data 1911.05544 §4. The export selects its reported run on test MAE, and its downstream hidden width of 768 is above the paper's range of 64 to 512 [see card, Paper vs code].

Papers

• Sun et al., ICCN (2019): ICCN, a text-interaction DCCA network for sentiment and emotion, with a cosine-loss ablation and author-reported results on CMU-MOSI, CMU-MOSEI and IEMOCAP.