---
kind: concept
slug: deep-canonical-correlation-analysis
title: Deep canonical correlation analysis
summary: >-
  Deep CCA trains two networks so that their outputs are as correlated as possible, with a loss computed over each batch. The paper's ICCN applies it to text-based audio and video interaction matrices for sentiment and emotion, and reports gains over text-only models. Its released code departs from the paper in several places, so the reported numbers are hard to reproduce.
tags: [canonical-correlation, multimodal, representation-learning, sentiment-analysis]
papers: ["1911.05544"]
related: [multimodal-fusion, sentiment-analysis, multi-view-representation, emotion-recognition-in-conversation]
status: draft
updated_at: 2026-10-31
---

## Overview

Canonical correlation analysis (CCA) finds one linear map per view so that the
mapped views are as correlated as possible [arxiv:1911.05544 eq. 1]. Deep CCA
(DCCA) keeps that objective but replaces the linear maps with neural networks.
It trains them with a loss equal to the canonical correlation of their outputs
[arxiv:1911.05544 §3, eq. 3, eq. 4]. The shared idea is that the training signal
is a correlation between two sets of outputs, not a distance between paired
outputs. The paper's cosine-loss variant tests exactly that difference
[arxiv:1911.05544 tab. 3].

The only source here is the ICCN paper. It uses DCCA inside a multimodal
sentiment and emotion model. Its outer-product interaction follows the tensor
fusion work it cites [arxiv:1911.05544 §2, §4]. The network forms a matrix of
pairwise products between a text vector and each non-text summary, then applies
DCCA to the two resulting matrices [arxiv:1911.05544 §3, eq. 5].

## Variants

### ICCN: text-interaction DCCA (the paper's method)

Each audio or video sequence is summarised into one utterance vector. An outer
product with the text vector gives one interaction matrix per modality. A 2D
CNN maps each matrix to a vector, and the two outputs are trained with the CCA
loss. The downstream embedding joins the two CNN outputs with the text vector
[arxiv:1911.05544 §3]. It is best on CMU-MOSI in all five reported metrics
[arxiv:1911.05544 tab. 1]. Card: [Sun et al., ICCN (2019)](/papers/1911.05544).

### ICCN1: no text

The same two branches, with DCCA applied to the raw audio and video summaries and
no text interaction. It scores 82.13 on MOSI binary accuracy and 83.01 on MOSEI,
against 83.07 and 84.18 for ICCN [arxiv:1911.05544 tab. 3]. The paper does not
say how the two raw sequences are combined before DCCA, so this variant cannot be
rebuilt from the paper alone [arxiv:1911.05544 §4].

### ICCN2 and ICCN3: cosine loss

The CCA loss is replaced by a cosine similarity between the two outputs. ICCN2
keeps the text interaction and ICCN3 drops it. Binary accuracy is 82.32 and
81.49 on MOSI, and 82.98 and 82.59 on MOSEI [arxiv:1911.05544 tab. 3]. The paper
plots canonical correlation and cosine similarity over training and reads the
curves as not moving together [arxiv:1911.05544 fig. 2, fig. 3, §5].

### Linear CCA, KCCA, GCCA and DCCA as baselines

The baselines use the same text features and the same downstream model as ICCN
[arxiv:1911.05544 §4]. On MOSI binary accuracy they score 79.45 (CCA), 79.82
(KCCA), 80.60 (DCCA) and 62.50 (GCCA). On MOSEI they score 82.94, 83.05, 83.62
and 75.12 [arxiv:1911.05544 tab. 1]. The DCCA binary-accuracy and F-score entries
are marked by the paper as taken from the original DCCA paper, not from its own
runs [arxiv:1911.05544 tab. 1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Text, audio and video features for sentiment or emotion, and the paper's reported numbers are the target | ICCN | Best on MOSI in all five metrics, best on MOSEI MAE and 7-class accuracy only, and best on three of four IEMOCAP emotions [arxiv:1911.05544 tab. 1, tab. 2]. The numbers are author-reported and the released code does not reproduce Table 1 (see Contradictions). |
| Text only, or text is the dominant signal | Text model first | Text alone reaches 80.80 on MOSI and 82.83 on MOSEI binary accuracy. ICCN adds 2.27 and 1.35 points [arxiv:1911.05544 tab. 1]. Weigh that gain against the cost of the audio and video pipelines. |
| Audio and video only, no text | ICCN1 or nothing from this paper | Only ICCN1 is reported without text, at 82.13 on MOSI [arxiv:1911.05544 tab. 3]. How the two sequences are combined is not stated, so the number cannot be reproduced. |
| The CCA loss is hard to optimise or a simpler objective is wanted | Cosine loss (ICCN2) | The paper's cosine variant loses 0.75 on MOSI and 1.20 on MOSEI binary accuracy [arxiv:1911.05544 tab. 3]. The table gives no spread across seeds, so gaps this small are not established as real. |
| Linear projection is enough, or the data is small | Linear CCA | On MOSI, linear CCA scores 79.45, below text alone at 80.80 [arxiv:1911.05544 tab. 1]. Use it as a sanity check, not a target. The paper gives no data-size results. |
| Changing the feature sizes (audio, video or text) | Re-derive the CNN input size first | The paper gives no formula for the 2D CNN input size. The released code fixes it for one interaction shape, so a new feature width needs a recomputed size [see card, Paper vs code]. |
| Budget for tuning | Search the paper's ranges on validation | The paper tunes the CCA learning rate, batch size, epochs, output dimension and downstream width over stated ranges, but does not give the chosen values [arxiv:1911.05544 §4]. |

## Implementation sketch

The paper's own training loop is full-batch [arxiv:1911.05544 alg. 1]. A
mini-batch version is shown here, because the released code uses mini-batches
[see card, Paper vs code]. Shapes are in comments.

```pseudocode
# One DCCA training step for text-based interaction  [arxiv:1911.05544 §3]
# Inputs: text t (d_t); audio and video sequences
# Summarise each sequence into one utterance vector  [arxiv:1911.05544 §3]
a = encode_audio(audio_seq)      # (d_a)
v = encode_video(video_seq)      # (d_v)
# Outer product with the text vector  [arxiv:1911.05544 eq. 5]
m_a = outer(t, a)                # (d_t, d_a)
m_v = outer(t, v)                # (d_t, d_v)
# Map each interaction matrix to k numbers with a 2D CNN  [arxiv:1911.05544 §3]
k_a = cnn_audio(m_a)             # (k)
k_v = cnn_video(m_v)             # (k)
# Maximise canonical correlation of the two outputs  [arxiv:1911.05544 eq. 4]
loss = -cca_correlation(k_a, k_v)  # estimated on the batch
update(loss)                     # both branches
# After training: embed = concat(k_a, t, k_v)  [arxiv:1911.05544 §3]
```

Where variants differ: the cosine loss (ICCN2) replaces `cca_correlation` with a
mean cosine similarity [arxiv:1911.05544 tab. 3]. The ICCN1 variant removes the
outer products [arxiv:1911.05544 tab. 3]. How the CCA correlation is computed on
a batch, with all singular values or only the top k, is not settled by the paper
(see Contradictions).

## Contradictions

- **Reading of the CCA loss.** The paper writes the objective as a trace of a
  matrix power [arxiv:1911.05544 eq. 2, eq. 4]. That can be the Frobenius norm of
  the whitened cross-covariance or the sum of its singular values. The card
  records that the released code's full branch computes the first and its
  default branch sums only the top k singular values, which are different values
  [see card, Paper vs code].
- **"State of the art" versus Table 1.** The paper calls ICCN state of the art
  [arxiv:1911.05544 §6]. Table 1 shows MFM ahead on MOSEI binary accuracy,
  F-score and correlation, and on IEMOCAP neutral [arxiv:1911.05544 tab. 1,
  tab. 2].
- **CCA-based baselines.** The paper says the CCA-based baselines do not improve
  the embedding [arxiv:1911.05544 §5]. In Table 1, DCCA's MOSEI correlation
  (0.707) is above the text-only row (0.681), and its MOSEI binary accuracy
  (83.62) is above it too [arxiv:1911.05544 tab. 1]. The accuracy numbers are the
  asterisked ones taken from the original DCCA paper, so only the correlation is
  from this paper's runs.
- **Released log versus Table 1.** The best MOSEI run in the released export has
  MAE 0.5693 and correlation 0.7122, while Table 1 reports 0.565 and 0.713
  [see card, Paper vs code]. The paper says hyperparameters are chosen on
  validation data [arxiv:1911.05544 §4]. The export selects its reported run on
  test MAE, and its downstream hidden width of 768 is above the paper's range of
  64 to 512 [see card, Paper vs code].

## Papers

- [Sun et al., ICCN (2019)](/papers/1911.05544): ICCN, a text-interaction DCCA network for sentiment and emotion, with a cosine-loss ablation and author-reported results on CMU-MOSI, CMU-MOSEI and IEMOCAP.