MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

In plain words

Videos of people talking carry three streams of signal at once: the words, the tone of voice and the face. Older systems mixed these streams together right away and hoped the mixing would sort out the differences between them. MISA first splits each stream into two parts. One part is pushed to look the same across the three streams, so that they can be compared directly. The other part is kept separate for each stream, so that what only the voice or only the face shows is not lost.

The model then combines all six parts with a small attention layer and makes one prediction, such as how positive or negative a sentence was said to be. The authors argue that cleaning up the streams before mixing them is what makes a simple combiner work well. The same setup is also tried on a humor dataset.

TL;DR

• MISA encodes text, visual and acoustic utterance sequences into one vector per modality, then projects each vector into a shared (modality-invariant) code and a private (modality-specific) code §3.3.
• Training combines a task loss with a CMD similarity loss, a soft-orthogonality difference loss and a reconstruction loss §3.5.
• A one-layer self-attention block over the six codes, followed by a small prediction head, produces the output §3.4.
• Author-reported: best results on CMU-MOSI and CMU-MOSEI, and on UR-FUNNY with BERT features tab. 1–3.
• Code is released (MIT); several reproduction details differ from the paper, see Paper vs code.

Problem

Multimodal sentiment analysis fuses language, visual and acoustic signals of an utterance. The authors argue that the signals have different distributions (a "modality gap"), so fusion has to bridge that gap, and that most prior methods leave it to the fusion step §1. The paper targets utterance-level prediction: each utterance is a short video, and the model sees only that utterance, without neighbouring utterances §2§3.1. Inputs are word-aligned low-level features from the benchmark toolkits §3.1§4.3. The task is either regression on a sentiment intensity score (MOSI labels lie in [−3, 3]) or a binary humor decision §3.1 §4.1.1 §4.1.3.

Method

Overview §3.2: each modality is first encoded to a vector of size d_h, then split into a shared code and a private code, both of size d_h. The six codes are fused by self-attention and concatenation, and a head predicts the label §3.3§3.4. Training uses four losses §3.5. The overall objective is a task loss plus three weighted regularisers, where α weights the similarity loss, β the difference loss and γ the reconstruction loss §3.5.

Utterance vector per modality §3.3 eq. 1: a stacked bidirectional LSTM reads the sequence U_m, and its end states of both layers are combined with a dense layer. For text, the paper replaces the LSTM with a pretrained BERT-base-uncased encoder, averaging its final hidden states over tokens §4.3.1.

Shared and private codes §3.3 eq. 2: a shared encoder E_c has the same parameters for all three modalities; each modality has its own private encoder E_p. The paper says these are simple feed-forward layers §3.3. Activation functions and layer norms are not stated in the text (see Paper vs code).

Fusion §3.4 eq. 5: the six codes are stacked into a 6 × d_h matrix, a multi-head self-attention layer (Transformer-style, scaled dot product) lets each code attend to the others, the outputs are concatenated into a 6·d_h vector, and a function G produces the prediction. The number of heads and the form of G are not stated in the text; the appendix gives the topology only as a figure fig. 6 app. D.

Losses §3.5:

• Similarity §3.5.1 eq. 8: CMD between the shared codes of each of the three modality pairs, averaged over the three pairs. CMD matches the means and the central moments up to order K, with K = 5 tab. 6.
• Difference §3.5.2: squared Frobenius norm of the cross-product of the zero-mean, unit-L2-row-normalised batch matrices, summed over (shared, private) pairs within each modality and over the three private–private pairs §3.5.2.
• Reconstruction §3.5.3: a decoder D reconstructs each utterance vector u_m from the sum of its shared and private codes; loss is the mean squared error averaged over the three modalities, scaled by 1/d_h §3.5.3.
• Task §3.5.4: cross-entropy for the classification task, mean squared error for regression §3.5.4.

# MISA training step  [§3.3] [§3.4] [§3.5]
# Input:  text, visual, acoustic sequences per utterance; label y
# Output: scalar training loss

def training_step(text, visual, acoustic, label):
    # Utterance vector of size d_h per modality  [§3.3] [eq. 1]
    u_text = project_text(encode_text(text))              # (B, d_h)
    u_visual = project_visual(encode_sequence(visual))    # (B, d_h)
    u_acoustic = project_acoustic(encode_sequence(acoustic))  # (B, d_h)

    # Shared (invariant) and private (specific) codes  [§3.3] [eq. 2]
    shared_text = E_shared(u_text)                        # (B, d_h)
    shared_visual = E_shared(u_visual)                    # (B, d_h)
    shared_acoustic = E_shared(u_acoustic)                # (B, d_h)
    private_text = E_private_text(u_text)                 # (B, d_h)
    private_visual = E_private_visual(u_visual)           # (B, d_h)
    private_acoustic = E_private_acoustic(u_acoustic)     # (B, d_h)

    # Reconstruct each utterance vector from its two codes  [§3.5.3]
    recon_text = D_text(shared_text + private_text)
    recon_visual = D_visual(shared_visual + private_visual)
    recon_acoustic = D_acoustic(shared_acoustic + private_acoustic)

    # Fuse the six codes with self-attention  [§3.4] [eq. 5]
    codes = stack(private_text, private_visual, private_acoustic,
                  shared_text, shared_visual, shared_acoustic)  # (6, B, d_h)
    attended = self_attention(codes)                      # (6, B, d_h)
    fused = concat(attended)                              # (B, 6 * d_h)
    prediction = G(fused)                                 # [§3.4]

    # Similarity: CMD over shared codes, K = 5  [§3.5.1] [eq. 8]
    # Difference: soft orthogonality over six code pairs  [§3.5.2]
    # Total objective  [§3.5]
    loss = task_loss(prediction, label)                   # [§3.5.4]
    loss = loss + alpha * similarity_loss(shared_codes)
    loss = loss + beta * difference_loss(shared_codes, private_codes)
    loss = loss + gamma * reconstruction_loss(recon, utterance)
    return loss

Key equations, as written in the paper:

\mathbf{u}_m = \text{sLSTM}\left(\mathbf{U}_m;\ \theta^{lstm}_m\right) eq. 1

\mathbf{h}_m^c = E_c(\mathbf{u}_m;\theta^c), \qquad \mathbf{h}_m^p = E_p(\mathbf{u}_m;\theta^p_m) eq. 2

\bar{\mathbf{M}} = \text{MultiHead}(\mathbf{M};\theta^{att}) = (\text{head}_1 \oplus \dots \oplus \text{head}_n)\,W^o eq. 5

\mathcal{L}_{sim} = \frac{1}{3}\sum_{(m_1,m_2)\in\{(l,a),(l,v),(a,v)\}} CMD_K(\mathbf{h}^c_{m_1},\mathbf{h}^c_{m_2}) eq. 8

The total objective and the difference and reconstruction losses are written in §3.5 as unlabelled equations; cite the section.

Hyperparameters

Final values per dataset tab. 6 app. C. The paper does not state the number of epochs, the number of seeds behind each table entry, or whether the reported numbers are single runs or averages.

Results

Author-reported. Setup: CMU-MOSI (93 videos, 2198 utterance segments, scores in [−3, 3]) §4.1.1; CMU-MOSEI (23453 segments, 5000 videos) §4.1.2; UR-FUNNY humor detection (binary) §4.1.3. The MISA rows use BERT text features (MISA (B)) unless stated. Hardware is not stated. Baselines are those in each table. Columns "neg./non-neg." and "neg./pos." are the two binary-accuracy conventions §4.2. Symbols: ⊗, ⊘ and ◇ mark values that the table note says were copied from the cited works, not re-run by the authors. The paper marks p < 0.05 under McNemar's test with †, against the publicly available models named in the table note.

Table 1, CMU-MOSI tab. 1:

Table 2, CMU-MOSEI tab. 2:

Table 3, UR-FUNNY humor detection, Acc-2 tab. 3:

The paper's Δ row reports +2.07 over the strongest baseline for MISA (BERT) tab. 3; the table's own baseline value of 68.57 gives +2.04 (see Pitfalls).

Ablation on the three datasets, MAE and Corr for MOSI and MOSEI, Acc-2 for UR-FUNNY tab. 4 §5.2:

The paper also reports a BERT-versus-GloVe comparison for UR-FUNNY §5.1.3 and qualitative plots of training losses fig. 3, t-SNE projections fig. 4 and fusion attention weights fig. 5. These are described, not reproduced here.

Headline claims §5.1: MISA is reported to beat every baseline on all metrics on MOSI and MOSEI, including ICCN, and to beat C-MFN on UR-FUNNY. The paper says significance is tested with McNemar's test only on the binary accuracies tab. 1 tab. 2.

Paper vs code

Official code: declare-lab/MISA (MIT), commit ec42faddde0d210cf7368aebf2118fe9570e7102. The paper links it in its conclusion §6. The repo has src/models.py, src/solver.py, src/config.py, src/train.py and data scripts. Everything below was checked by reading those files. The paper and code agree on: hidden size 128, batch sizes, learning rate 1e-4, Adam, patience 6, the CMD order 5, the choice of CMD over adversarial training, the three-pair similarity average, and the dataset sizes. The discrepancies are:

• Per-dataset settings are not the defaults. Table 6 gives different activation, dropout, α, β and γ for MOSEI and UR-FUNNY. In code, only num_classes and batch_size depend on the dataset; activation (ReLU), dropout (0.5), sim_weight (1.0), diff_weight (0.3) and recon_weight (1.0) are global defaults config.py config.py. Running --data mosi with the defaults reproduces the MOSI column of Table 6. For MOSEI and UR-FUNNY, the paper's values must be passed as flags. The repo does not include them.
• Frozen BERT layers. The paper does not freeze BERT. Code freezes encoder layers 0–8 for MOSEI and all BERT parameters for UR-FUNNY; MOSI trains BERT solver.py.
• Learning-rate decay has no effect on training. Code uses ExponentialLR with γ = 0.5 solver.py. It steps only when patience runs out solver.py. The same branch then breaks the loop solver.py, so the reduced learning rate is never used for an update. Training ends at the first patience exhaustion, and the best dev checkpoint is used for testing solver.py.
• Difference loss is a mean, not a Frobenius sum. The paper writes the squared Frobenius norm §3.5.2. Code takes torch.mean over the d_h × d_h matrix functions.py. The term is therefore divided by d_h² (16384 at d_h = 128), so β values do not transfer directly between the two forms. The normalising norms are also detached from the graph functions.py functions.py.
• Reconstruction target. The paper reconstructs the utterance vector u_m §3.5.3. Code reconstructs the output of the projection layer (after the activation and LayerNorm), and each decoder is a single linear layer without activation models.py models.py models.py models.py.
• CMD scaling. The paper's CMD includes 1/|b−a| factors for the bounded interval §3.5.1. Code omits them and uses the Euclidean norm of the moment differences functions.py.
• Private and shared encoders use sigmoid. The paper calls them "simple feed-forward" layers without an activation §3.3. Code applies Sigmoid after each linear layer models.py.
• Projection after the LSTM. The paper states a dense layer §3.3. Code applies a linear layer, the dataset activation and a LayerNorm models.py. It concatenates the end states of both LSTM layers, giving 4 × hidden inputs models.py models.py models.py.
• Fusion block. The paper describes multi-head self-attention followed by concatenation §3.4. Code runs one nn.TransformerEncoderLayer with 2 heads, the PyTorch defaults for feed-forward size and dropout, and residual connections and normalisation models.py models.py. The prediction head is a two-layer MLP with dropout and the dataset activation models.py.
• Gradient clipping is per-value clipping at 1.0 solver.py, not norm clipping. The paper lists the value only.
• Metric details. Acc-7 rounds predictions and labels after clipping to [−3, 3], using NumPy's round-half-to-even solver.py solver.py. Binary metrics for MOSI/MOSEI drop samples whose label is exactly zero solver.py solver.py. The F-score is weighted solver.py. The paper states none of these.
• Feature preprocessing. Visual and acoustic features are z-normalised per utterance create_dataset.py. The paper does not mention this.
• BERT input. Text is passed to BERT as the space-joined word list, with max_length set to the padded word count plus 2, so subword tokens may be truncated and no longer line up with the word-aligned features data_loader.py. Masked mean pooling includes [CLS] and [SEP] models.py.
• Unused flag. --runs 5 exists in config but train.py runs once with seed 336 config.py train.py. Fig. 3 caption says five seeds for the loss traces fig. 3.

Pitfalls

• Table 6 is not a single setting. Each dataset needs its own α, β, γ, dropout and activation. Copy them from tab. 6, not from the code defaults, which are the MOSI values.
• The paper and code disagree on the difference loss scale. If you implement the Frobenius form literally, β = 0.3 or 1.0 will over-weight the term relative to the code's mean form, by a factor of d_h² (16384). Pick one and document it.
• Model selection uses dev loss, not the reported metric. The checkpoint with the lowest dev MSE (regression) or cross-entropy (UR-FUNNY) is used for test solver.py. Tables that report several metrics come from that one checkpoint.
• Reported numbers may be copied, not measured. The rows marked ⊗, ⊘ and ◇ are taken from earlier papers tab. 1 tab. 2. Only the MISA and ICCN rows were re-run here, and the paper does not say whether ICCN was re-run.
• Δ rows are not always against the strongest baseline. On UR-FUNNY, 70.61 − 68.57 = 2.04, while the Δ row states 2.07 tab. 3. On MOSEI, Acc-2 neg./pos. Δ is +1.3 against ICCN (84.2), but MFM (B) reports 84.4 in that column tab. 2. Recompute Δ values yourself before quoting them.
• Unstated values to fix yourself: number of epochs (code caps at 500 config.py), number of attention heads (code: 2), decoder form, scheduler factor (code: 0.5), number of seeds, and the exact early-stopping trigger (code: first exhaustion ends training).
• Text-only signal dominates on MOSI/MOSEI. Removing language drops MOSI Corr from 0.761 to 0.041 tab. 4. Removing the visual or acoustic branch costs much less (MOSI MAE 0.798 and 0.849 against 0.783), so a bug in the text branch will show up far more than one elsewhere. The paper attributes this partly to manual transcripts and to BERT §5.2.1.
• UR-FUNNY transcripts were rebuilt. The dataset only provides GloVe embeddings. The authors map each embedding to its nearest GloVe vocabulary word, which they checked on 100 sampled utterances §4.3.1. BERT results on UR-FUNNY therefore rest on reconstructed text.
• Acc-2 conventions differ. The paper reports both neg./non-neg. (label ≥ 0) and neg./pos. (label ≠ 0) §4.2. Papers you compare against may use only one.
• Data availability. Pre-computed splits come from the CMU-MultimodalSDK and are not in the repo; the README says to download them or recreate them README.md.
• Licence. The paper is CC BY-SA 4.0, so any reuse of its figures or tables carries that licence. Figures are not used on this page.

Lineage

• Builds on: Domain Separation Networks, which introduced shared–private encoders for domain adaptation with orthogonality and reconstruction losses §2 §3.5.2 §3.5.3. MISA changes DSN in four ways: representations are learned per modality within an utterance, CMD replaces MMD and adversarial matching, a difference loss covers private codes too, and both shared and private codes are used for fusion §2.
• Builds on: CMD as a distribution-matching metric §3.5.1; the Transformer for self-attention fusion §3.4; BERT for text features §4.3.1.
• Compared against: MFN, MARN, MV-LSTM, RMFN, RAVEN, MulT, Graph-MFN, TFN, LMF, LMFN, HFFN, MCTN, ARGF and MFM (utterance-level); BC-LSTM, CH-Fusion, CIM-MTL, CIA and DFF-ATMF (contextual); ICCN as the MOSI/MOSEI state of the art and C-MFN for UR-FUNNY §4.4 app. A.