Decoupled Multimodal Distilling for Emotion Recognition

In plain words

Recognising the emotion in a video means combining what is said, how it sounds and what the face does. These three streams differ a lot. Words are usually the most informative of the three. A common trick is knowledge distillation: a strong stream teaches a weak one. Deciding by hand which stream should teach which, and how strongly, does not scale well.

The paper's method first splits each stream into two parts: one part that looks alike across streams, and one part that is specific to that stream. It then lets the streams teach each other on each part, with the strength of every teaching link learned from the data. The learned links are drawn as a small graph, and the paper shows that the graph has a readable pattern.

TL;DR

• DMD (decoupled multimodal distillation) handles emotion recognition from language, visual and acoustic inputs of a video clip §1.
• Each modality is split into a homogeneous part and a heterogeneous part, trained with reconstruction, cycle, margin and orthogonality losses §3.1.
• Two graph distillation units (GD-Units) learn the direction and weight of each cross-modal distillation link §3.2.
• Author-reported results beat the listed baselines on CMU-MOSI and CMU-MOSEI in both aligned and unaligned settings tab. 1 tab. 2.
• Several parts of the released code differ from the paper; read ## Paper vs code before reimplementing.

Problem

Heterogeneous modalities carry different amounts of information, and language is usually the most informative one §1. Direct cross-modal distillation needs a manual choice of direction and weights, and the feature distributions of the modalities differ enough to make direct distillation suboptimal §1. Earlier feature-decoupling methods split features into shared and specific parts, but do not distil knowledge between modalities inside those spaces §2.

Method

The model has three parts: multimodal feature decoupling, a homogeneous GD-Unit (HomoGD) and a heterogeneous GD-Unit (HeteroGD) §3. Symbols follow the paper's macros: com is the homogeneous (modality-irrelevant) part and prt is the heterogeneous (modality-exclusive) part eq. 1, drawn as X^com and X^prt in fig. 2.

1. Shallow encoding. Three 1D temporal convolutions map the language, visual and acoustic inputs to sequences of a common width d. The paper says the temporal length is kept, so aligned and unaligned inputs can be handled alike §3.1. See Paper vs code for the padding.
2. Decoupling. A shared encoder gives the homogeneous part of every modality. A private encoder per modality gives its heterogeneous part eq. 1.
3. Self-regression. A private decoder per modality rebuilds the modality from the concatenation of both parts eq. 2. The rebuilt sequence is re-encoded by that modality's private encoder, and the result must match the heterogeneous part eq. 3.
4. Margin loss. For a triplet with anchor i, a positive j (same emotion, other modality) and a negative k (same modality, other emotion), the similarity of i to j must exceed that of i to k by a margin α eq. 4.
5. Soft orthogonality. The cosine similarity between the homogeneous and heterogeneous parts of each modality is added to the loss to reduce redundancy eq. 5.
6. Decoupling loss. L_dec = L_rec + L_cyc + γ (L_mar + L_ort) eq. 6.
7. Graph distillation (GD-Unit). Each GD-Unit is a directed graph whose nodes are the three modalities. The edge from modality i to modality j gets a score from a fully connected layer f applied to both nodes' logits and features, followed by a fully connected layer g eq. 8. Edge scores are normalised with softmax. The distillation gap of an edge is the difference between the two logits §3.2. The loss is the weighted gap summed over edges, written as an L1 norm of the element-wise product W ⊙ E eq. 7 eq. 9.
8. HomoGD. The GD-Unit takes the homogeneous features and their logits §3.2.
9. HeteroGD. The heterogeneous features of a target modality are enhanced by cross-modal attention from the other two modalities. Query comes from the target and key and value from the source eq. 10. The enhanced features of all other modalities are concatenated for the target §3.2. A GD-Unit then runs on these enhanced features.
10. Fusion. The enhanced heterogeneous features and the homogeneous features are fused with adaptive weights, and the fused vector feeds the prediction §3.2. The form of the weights is not given in the paper.
11. Objective. L_total = L_task + λ1 L_dec + λ2 L_dtl eq. 11, with L_task described as a mean absolute error §3.3.

# One training step as the paper states it  [§3, eq. 1-11]
# Input:  text, audio, vision: per-modality sequences (B, T_m, d)
#         score: continuous sentiment label per clip (B,)
# Output: scalar loss

def training_step(text, audio, vision, score):
    # Shallow temporal convolution per modality to common width d  [§3.1]
    x = {"l": conv_l(text), "a": conv_a(audio), "v": conv_v(vision)}
    # Split into homogeneous (com) and heterogeneous (prt) parts  [eq. 1]
    com = {m: shared_encoder(x[m]) for m in x}
    prt = {m: private_encoder[m](x[m]) for m in x}
    # Self-regression: rebuild each modality from both parts  [eq. 2]
    recon = {m: decoder[m](cat(com[m], prt[m])) for m in x}
    loss_rec = sum(sq_frobenius(x[m] - recon[m]) for m in x)
    # Re-encode the rebuilt modality; its prt part must match  [eq. 3]
    loss_cyc = sum(
        sq_frobenius(prt[m] - private_encoder[m](recon[m])) for m in x
    )
    # Margin and orthogonality keep the two parts apart  [eq. 4, eq. 5]
    loss_mar = margin_loss(com, score)
    loss_ort = sum(cosine(com[m], prt[m]) for m in x)
    # Decoupling loss  [eq. 6]
    loss_dec = loss_rec + loss_cyc + gamma * (loss_mar + loss_ort)
    # Graph distillation on com and on cross-modally enhanced prt  [§3.2]
    enhanced = cross_modal_transformer(prt)
    loss_dtl = gd_loss(com) + gd_loss(enhanced)
    # Fuse and predict; task loss is MAE on the fused output  [§3.2, §3.3]
    output = fuse(enhanced, com)
    loss_task = mean_absolute_error(output, score)
    # Total objective  [eq. 11]
    return loss_task + lambda_1 * loss_dec + lambda_2 * loss_dtl

# Graph distillation over three modality nodes  [§3.2, eq. 7-9]
# Input:  feats: dict of modality -> (B, d) features
# Output: scalar distillation loss

def gd_loss(feats):
    # Logit per modality node, from its features  [§3.2]
    logits = {m: logit_head(feats[m]) for m in feats}
    # Score each directed edge i -> j from both endpoints  [eq. 8]
    for i, j in ordered_pairs(feats):     # i != j
        scores[i, j] = edge_fc(
            cat(logits[i], feats[i], logits[j], feats[j])
        )
    # Normalise the edge weights over all edges  [§3.2]
    weights = softmax(scores)              # (B, num_edges)
    # Logit difference per edge is the distillation gap  [§3.2]
    gaps = abs(logits[i] - logits[j]) for each edge (i, j)
    # Weighted sum over edges, then batch mean  [eq. 7, eq. 9]
    return mean(sum(weights * gaps))

# Margin loss on homogeneous features  [§3.1, eq. 4]
# Input:  com: dict of modality -> (B, d) features; score: (B,) labels
# Output: scalar loss

def margin_loss(com, score):
    # Triplets: anchor i, positive j, negative k  [eq. 4]
    loss = 0
    for i, j, k in triplets(score):
        # j shares the score but not the modality; k shares the modality
        loss += relu(alpha - cos(com[i], com[j]) + cos(com[i], com[k]))
    return loss / number_of_triplets

Hyperparameters

Values in the paper are in the first table group. The second group is what the released code uses where the paper is silent. Paper values are binding for reimplementation; code values are what the authors ran.

Results

Author-reported. Setup: CMU-MOSI (2,199 short monologue clips) and CMU-MOSEI (22,856 movie-review clips) §4. Metrics: 7-class accuracy (ACC7), binary accuracy (ACC2) and F1 §4. Model size, baseline re-runs and seed count are not stated. The numbers below were not reproduced for this card.

Setting: "Aligned" and "Unaligned" as the paper labels them. An asterisk marks rows whose input language features are BERT-based tab. 1 tab. 2.

CMU-MOSI tab. 1

CMU-MOSEI tab. 2

Ablation of components tab. 3 (FD: feature decoupling; CA: cross-modal attention; ✓ on, × off)

Unimodal accuracy with and without feature decoupling, MOSEI tab. 4 (ACC2 / F1, %)

Adding a GD-Unit to MulT tab. 5

Other reported results: an ablation of HomoGD on MOSEI gives ACC2 values of 80.9% vs. 82.4% for language, 56.5% vs. 61.8% for vision and 54.4% vs. 64.1% for audio; the paper reads these as gains from HomoGD §4.2. Visualisations fig. 3 fig. 4 show t-SNE plots of the two feature spaces; fig. 5 shows the learned edge weights, which the paper reads as language-to-vision and language-to-audio dominance in HomoGD and additional vision-to-audio flow in HeteroGD §4.2.

Paper vs code

Code at 516d4e116c22220e08873fa38df828d1a0d6f7e8 (MIT). The repository is linked in the abstract, so it is treated as the official code. The GitHub owner is not verified as an author. The paper's text and the code differ in the following ways.

• Fused prediction does not use the GD outputs. The paper says the refined features are fused for recognition §3. In the code, the output is built from the heterogeneous transformer states and a self-attended homogeneous feature, and the GD modules only add losses during training dmd.py DMD.py DMD.py.
• Homogeneous representation distillation is computed but not used. The homogeneous kernel returns a representation loss, which the training loss leaves out. Only the logit and regulariser terms are added get_distillation_kernel_homo.py DMD.py.
• Edge weights include a fixed prior. The code adds a hard-coded prior to the learned softmax weights and penalises the mean edge weights for deviating from it, with weight 10. The paper describes the weights as learned without a prior §3.2 run.py run.py get_distillation_kernel.py.
• Heterogeneous representation distillation uses a one-sided cosine loss. The paper's distillation term uses only logit differences §3.2. The code adds a representation term based on CosineEmbeddingLoss with target −1, which equals max(0, cos) under PyTorch's definition get_distillation_kernel.py misc.py.
• Margin loss differs from the paper's pair rules. The code treats two samples as positives when their continuous regression labels are exactly equal, and as negatives whenever the labels differ, including same-modality pairs. The margin is 0.15 times the label difference, not a constant α. The loss is applied to a learned linear projection of the homogeneous features, not to the features themselves HingeLoss.py DMD.py dmd.py.
• Orthogonality is one-sided. The paper writes a plain cosine sum eq. 5. The code uses the same CosineEmbeddingLoss with target −1 and so penalises only positive cosine DMD.py.
• Task loss has eight terms. The paper writes a single mean absolute error §3.3. The code sums L1 losses on the fused output, the three homogeneous logits, the three heterogeneous logits and the homogeneous self-attention logit DMD.py.
• Reconstruction and cycle losses are averages. The paper writes squared Frobenius norms eq. 2 eq. 3. The code's MSE divides the sum of squares by the number of elements DMD.py.
• Temporal length is shortened. The paper says each modality keeps its temporal dimension after the convolution §3.1. The code uses unpadded convolutions and computes lengths as len − kernel + 1 dmd.py dmd.py.
• Text features are BERT, and the default setting is aligned. The paper's implementation section says GloVe 300-d by default and BERT only for starred rows §4. The shipped config uses 768-d BERT features, fine-tuned, for the aligned data config.json config.json config.json. The paper's visual and acoustic dimensions (35 and 74) match the MOSEI config, but the MOSI config uses 20 and 5 config.json. Which setting produced each table row is not stated.
• Training length differs. The paper trains for 30 epochs until convergence §4. The code loops without a cap and stops after 10 epochs with no validation improvement DMD.py DMD.py config.json.
• Gradient accumulation is undocumented. The code takes one optimiser step per 10 batches of size 16 config.json DMD.py. A final group that does not fill up in an epoch is skipped DMD.py.
• Weight decay is configured but unused. The config lists weight decay per dataset config.json config.json. The optimiser is created without it DMD.py.
• Model selection uses validation loss. The paper selects λ on validation performance §4. The code keeps the checkpoint with the lowest validation loss and reports test metrics from that checkpoint config.json DMD.py run.py.
• Learning rate. The paper text gives no learning rate. The code uses 1e-4 config.json config.json. A commented-out table in the LaTeX source lists 5e-4, but it is not in the compiled paper, so it is not cited as paper content [tex: raw/src/2303.13802/main.tex#L266-L291].

No discrepancies were found for the paper's λ values: the code uses 0.1 on reconstruction and cycle, 0.01 on margin and orthogonality, and 0.05 on each GD term, consistent with §4 and eq. 11.

Pitfalls

• Edge patterns are partly set by the prior. The prior puts most mass on language-to-vision and language-to-audio in HomoGD, and on vision-to-audio in HeteroGD run.py run.py. These are the edges fig. 5 reports as dominant. The paper reads them as learned patterns §4.2. The prior is about 0.48 on each of the two language-sourced homogeneous edges and about 0.009 on the other four; in the heterogeneous graph it is about 0.33 on three edges (language to vision and audio, vision to audio) and about 0.006 on the other three. No ablation without the prior is reported, so learned and prior-driven weights cannot be separated.
• Margin positives depend on exact label equality. Regression labels are continuous floats metricsTop.py data_loader.py. Positives are therefore mostly the same clip's other modalities. This is consistent with the code's data, but it is not the "same emotion" rule the paper describes eq. 4.
• Margin argument is dead. HingeLoss.forward takes a margin argument that is overwritten by a label-based margin, so changing the argument has no effect HingeLoss.py HingeLoss.py.
• Fusion weights are not specified in the paper. Reimplementers must choose a form. The code's version is sigmoid gates on each modality's output, then concatenation, a residual MLP and a linear layer dmd.py.
• Sequence lengths are hard-coded. The code fixes the lengths per dataset and alignment mode, and the unaligned lengths of 375 and 500 frames appear only in code dmd.py.
• Model dimensions are in the supplementary. The paper defers the network configuration to a supplementary file §4; it is not in the arXiv source. The config values are the only available record.
• Seed and variance are unclear. The code's default run averages five seeds and reports mean and standard deviation run.py run.py. The single-seed train.py uses 1111 only train.py. The paper does not say how many seeds its tables use.
• Metric definitions need care. In the regression metrics, ACC7 rounds the clipped predictions and labels. ACC2 and F1 use only samples with non-zero labels metricsTop.py. The paper's "binary accuracy" does not specify this, so the code's definition is the one that matches the reported numbers.
• CUDA is required. Several places call .cuda() directly, including the margin and kernel code DMD.py get_distillation_kernel.py misc.py.
• Data is not in the repository. Processed MOSI and MOSEI files must be downloaded separately, and raw data is not provided [README.md#L41-L43].
• Paths are hard-coded. Test mode loads pt/mosi-aligned.pth regardless of dataset run.py.
• Fixed priors and softmax temperature are undocumented. The softmax on the edge scores uses a factor of 1/8 that the paper does not mention get_distillation_kernel.py.

Lineage

• Builds on: the multimodal transformer for crossmodal attention, feature decoupling into modality-invariant and modality-specific spaces, and graph-based distillation between teachers, as cited in the paper §1 §2.
• Compared against: Graph-MFN, MCTN, MulT, PMR, MICA, MISA, FDMER, MFM, RAVEN, LMF, TFN, EF-LSTM and LF-LSTM tab. 1 tab. 2.
• Superseded by: not known from this wiki.