Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

In plain words

A video of someone talking carries three kinds of signal at once: the words, the tone of voice, and the facial expression. A model that reads sentiment from all three has to combine them, but the three are not equally useful, and mixing them naively can drown out the useful one. This paper proposes a design that gives the spoken words a central role. The words are paired separately with the voice and with the face, and each pair is combined in its own branch.

Two things keep the branches from blurring together. First, a small side classifier at each step is trained to tell which modality a representation came from, which pushes the modalities to stay distinct. Second, learned gates decide, per dimension, how much of the attention result and how much of the original signal to keep. The authors report that this beats earlier fusion methods on sentiment-intensity and humor benchmarks.

TL;DR

Bi-Bimodal Fusion Network (BBFN) fuses text with visual and with acoustic input in two parallel, text-centred modules. Each module stacks gated Transformer layers with a layer-wise discriminator that keeps modality representations apart. The authors report the best MAE and accuracy among the compared baselines on CMU-MOSI, CMU-MOSEI and UR-FUNNY. The results are author-reported, the official code differs from the paper in several details (see Paper vs code), and the run script in the repository trains without the separator loss.

Problem

Multimodal sentiment analysis (MSA) predicts a sentiment score, a polarity or an emotion from text, visual and acoustic features of a video clip §3. Most earlier fusion models connect every pair of modalities symmetrically, and the authors argue this ignores that task information is unevenly spread across modalities; the text is often the most informative §1.

The authors also observe "feature space collapse": as the stacked Transformer layers fuse a modality pair, the representations of the two modalities drift together and lose independence, which hurts fusion §1 §3.3.

Method

BBFN has four parts: a sequence encoder per modality §3.2, two bimodal modules TV (text-visual) and TA (text-acoustic) that share the text input but have their own parameters §3.1 §3.4, a per-layer feature separator and a gated complementation Transformer inside each module §3.3, and an output layer §3.4.

1. Text encoding. Text is embedded with BERT-base-uncased after adding [CLS] and [SEP] tokens §3.2 §4.2.
2. Sequence encoding. Each modality sequence (text, visual, acoustic) goes through a single-layer bidirectional GRU and a linear projection to a common width, giving X^0 §3.2 eq. 1.
3. Two bimodal modules. Module TV and module TA each run two pipelines in parallel, one per modality of the pair. Layers are stacked, so each layer takes the outputs of the previous one §3.1 §3.3.
4. Feature separator (per layer). For each modality, a BiGRU runs over the layer input and its outputs are averaged over time to give one vector per sample §3.3. Consecutive groups of K such vectors are averaged, which the authors argue damps noise §3.3 eq. 2. A layer-specific discriminator takes both grouped vectors and predicts, for each group, which modality it came from §3.3 eq. 3. Its binary cross-entropy against pseudo labels is the separator loss.
5. Gates. The retain gate g_r and the compound gate g_c are sigmoids of a learned linear map applied to the concatenation of the two pooled vectors of that layer. Each is a vector of width d §3.3.
6. Gated complementation. Within a pipeline, the query comes from the "master" modality and the keys and values from the other modality, i.e. the heterogeneous multi-head attention §3.3. The result m is combined as layer-normalised (g_c ⊙ m + g_r ⊙ X), where X is the master's own input §3.3. A feed-forward block with residual and layer normalisation follows, giving the layer output eq. 4.
7. Heads. After the last layer, each module contributes the first time step of each of its two output sequences. The four head vectors (width d each) are concatenated §3.4.
8. Prediction and loss. The concatenated heads go to a feed-forward output network §3.4. The task loss is mean squared error for sentiment intensity and binary cross-entropy for humor detection. Separator losses are summed over layers and modules and added with weight λK/(2L), where λ is tunable §3.4.

# BBFN: one text-centred bimodal module  [§3.3] [eq. 1]-[eq. 4]
# Input:  seq_text, seq_other (B, n, d), one row per time step
# Output: head_text, head_other (B, d) and the separator loss (scalar)

def bimodal_module(seq_text, seq_other):
    separator_loss = 0.0
    for layer in range(num_layers):  # L is not stated in the paper
        # One vector per sample: BiGRU over time, then average  [§3.3]
        vec_text = mean_over_time(bigru_text[layer](seq_text))
        vec_other = mean_over_time(bigru_other[layer](seq_other))
        # Average consecutive groups of K vectors  [§3.3] [eq. 2]
        group_text = group_mean(vec_text, group_size)    # group size K
        group_other = group_mean(vec_other, group_size)
        # Layer discriminator predicts each group's modality  [§3.3] [eq. 3]
        pred_text, pred_other = discriminator[layer](group_text, group_other)
        separator_loss += bce(pred_text, pseudo_label_text)
        separator_loss += bce(pred_other, pseudo_label_other)
        # Both pipelines read this layer's inputs  [§3.3]
        new_text = complement(seq_text, seq_other, vec_text, vec_other, layer)
        new_other = complement(seq_other, seq_text, vec_other, vec_text, layer)
        seq_text, seq_other = new_text, new_other
    # Head: first time step of the last layer's outputs  [§3.4]
    return seq_text[:, 0], seq_other[:, 0], separator_loss

# Gated complementation for one pipeline  [§3.3] [eq. 4]
# master and other: (B, n, d); vec_master, vec_other: (B, d)

def complement(master, other, vec_master, vec_other, layer):
    # Retain and compound gates from both pooled vectors  [§3.3]
    pair = concat(vec_master, vec_other)                 # (B, 2d)
    gate_retain = sigmoid(retain_proj[layer](pair))      # (B, d)
    gate_compound = sigmoid(compound_proj[layer](pair))  # (B, d)
    # Query from master; keys and values from other  [§3.3]
    attended = multi_head_attention(master, other, other)
    # Gates applied over time steps (broadcast; see Pitfalls)  [§3.3]
    mixed = layer_norm(gate_compound * attended + gate_retain * master)
    # Feed-forward with residual, then layer norm  [§3.3] [eq. 4]
    return layer_norm(mixed + feed_forward(mixed))

# Full model and training objective  [§3.1] [§3.4]
# Regression targets use MSE; humor (binary) uses BCE

def forward(seq_text, seq_visual, seq_audio):
    # Both modules take the same text sequence  [§3.1]
    tv_text, tv_visual, sep_tv = bimodal_module(seq_text, seq_visual)
    ta_text, ta_audio, sep_ta = bimodal_module(seq_text, seq_audio)
    # Four heads of width d, concatenated into 4d  [§3.4]
    heads = concat(tv_text, tv_visual, ta_text, ta_audio)
    prediction = output_network(heads)  # details not stated
    return prediction, sep_tv + sep_ta


def training_loss(prediction, target, separator_loss):
    # Task term, averaged over the batch  [§3.4]
    task_loss = task_loss_fn(prediction, target)
    # Separator weight lambda * K / (2L)  [§3.4]
    weight = lam * group_size / (2 * num_layers)
    return task_loss + weight * separator_loss

Hyperparameters

Only values the paper states are cited to the paper. Values that appear only in the official code are marked "code default" and are the defaults of the argument parser, not values the paper reports. The paper gives no optimizer, learning rate, batch size, epoch count, hidden size, layer count or λ.

Results

Author-reported. Setup: CMU-MOSI and CMU-MOSEI test sets for sentiment intensity, UR-FUNNY test set for humor §4.1 tab. 1. Baselines are DFF-ATMF, LMF, TFN, MFM, ICCN, MulT and MISA §4.3. Metrics are MAE, Corr, Acc-7, Acc-2 and F1; UR-FUNNY reports Acc-2 only §4.3.

Table 1 tab. 1 (copied from the paper; markers as in the paper: △ taken from earlier papers, † reproduced with public code, ‡ paired t-test p < 0.05 against MISA).

The paper's summary states that the model beats the baselines on almost all metrics, with about 1% gain in binary accuracy and over 2.5% in 7-class accuracy against the strongest baseline, and the largest gain in MOSEI MAE (over 4%) §5.1. For the MOSEI test set, the paper also plots the error distribution against MISA; the BBFN curve has more peaks in the low-error region below 0.25 §5.1 fig. 4.

Ablation on CMU-MOSEI tab. 3 (test set; the separator and gate rows keep the other components).

Reading of the ablation §5.2 tab. 3: a single text pair (TV or TA) gives MAE of about 0.55, against 0.529 for the full model. The visual-acoustic pair alone is much weaker (MAE 0.816). The paper reads the close TV + VA result as showing that the architecture also works with other pairs. Adding a visual-acoustic module to make a ternary-symmetric model lowers the score. Removing the separator or the gates costs accuracy, and removing both costs more. The paper concludes that the overall two-module architecture gives the largest gain over the baselines. The input-variant rows used separators and gates §5.2.

Gate analysis. The paper plots the eight gate signals of the second layer on two test samples fig. 5. It reads a larger compound-gate weight as a modality that relies on its partner, and a larger retain-gate weight as a modality that carries most of the information §5.3.

Paper vs code

Official code: declare-lab/BBFN at commit 1a251bd2a82193902591a15ae2f28b902c295310, MIT licence (DeCLaRe Lab, the authors' lab). The paper's abstract links a multi-model repository of the same lab; its README points to this BBFN repository for this paper, which is the one read here. Checked: src/main.py, src/config.py, src/solver.py, src/models.py, src/modules/transformer.py, src/modules/encoders.py, scripts/run.sh. Not checked in detail: src/create_dataset.py and src/data_loader.py (feature extraction and alignment).

Discrepancies found:

• Text encoder. The paper uses BERT-base-uncased for all experiments §4.2. The code default uses frozen 300-d GloVe vectors, and BERT only with --use_bert main.py solver.py. The run script does not pass --use_bert run.sh. The GloVe file path is hard-coded to a local directory config.py.
• Sequence encoder. The paper specifies a BiGRU and linear projection §3.2. The code default projection is a 1-D CNN with kernel size 3 main.py encoders.py. The GRU branch exists encoders.py but is not the default.
• Separator grouping. The paper averages consecutive groups of K vectors before the discriminator §3.3 eq. 2. The code has no grouping step. The discriminator sees one vector per sample encoders.py.
• Separator layer 0 vs later layers. In the code, layer 0 applies a linear map to the pooled sequence vector with no BiGRU encoders.py encoders.py. Layers 1 and later run a BiGRU and masked averaging transformer.py encoders.py. The paper describes a BiGRU at every layer §3.3.
• Separator loss. The paper weights the summed per-layer losses by λK/(2L) §3.4 (unnumbered display). The code adds λ times one binary cross-entropy mean over all layers and both modules solver.py models.py transformer.py. The K/(2L) factor is absent, and the default λ is 0.1 main.py.
• Separator is off by default. The solver adds the discriminator loss only when --use_disc is passed, and that flag defaults to off main.py solver.py solver.py. The run script does not pass it run.sh. The discriminators are still built, because the GatedTransformer defaults use_disc=True and the model does not pass the flag transformer.py models.py. Without the loss, the discriminator parameters get no gradient and the separator adds nothing to training. The encoder outputs still feed the gates. So the default run resembles the paper's "without separator" ablation (MAE 0.533), not the full model.
• Gates. The form matches the paper: the attention output is scaled by the compound gate, and the residual input by the retain gate transformer.py. The gate inputs, however, are the concatenated DIV-encoder outputs, which are linear or GRU outputs with dropout, not the raw pooled BiGRU vectors transformer.py encoders.py.
• Layer norm placement. The paper normalises after the gated sum and after the feed-forward residual §3.3 eq. 4. The code sets normalize_before to true, so layer norm is applied to the attention query, keys and values and to the feed-forward input, and not after the residual sums transformer.py transformer.py transformer.py transformer.py transformer.py transformer.py transformer.py.
• Output network. The paper states the four heads go to a feed-forward network §3.4. The code adds a residual block (linear, ReLU, dropout, linear, plus skip) before the output layer models.py. The heads are the first time step of each output sequence models.py.
• Positional embedding. A sinusoidal positional embedding is created transformer.py, but the only method that adds it, forward_transformer, is not called from the forward path transformer.py transformer.py. The paper does not mention positional encoding.
• Humor loss. The paper uses binary cross-entropy §3.4. The code uses a two-logit cross-entropy solver.py with output size 2 main.py.
• Learning rate help text. The argument help says 1e-3, the default is 5e-4 main.py.
• Scheduler argument. --when is parsed and stored main.py main.py, but the solver uses its own hard-coded patience of 20 solver.py.
• Model selection uses test MAE. The validation loss decides when the best epoch is updated, but the checkpoint is written and the metrics are printed only when the test MAE also improves solver.py.
• Frozen parts. GloVe embeddings are frozen solver.py. With BERT, layers up to 8 are frozen for MOSI and MOSEI, and all BERT parameters for UR-FUNNY solver.py. The paper does not state any freezing.
• CUDA requirement. The attention mask builder allocates torch.cuda.BoolTensor transformer.py, and the sequence encoder calls .cuda() encoders.py. These paths are expected to fail on CPU-only machines. This was not tested here.

Pitfalls

• Separator loss is inactive in the shipped run. Reproducing the paper's separator needs --use_disc plus an explicit λ. Even then the code's loss normalisation and the missing K grouping differ from the paper (see above).
• Layer count and width are unstated. The paper never gives L, d, the number of attention heads per layer, or λ. The code defaults (5 layers, width 40, 5 heads) are not confirmed by the paper.
• Gates broadcast over time steps. The paper writes the gates as vectors of width d and multiplies them by sequences. Applying one gate vector per sample across all time steps is the only shape that fits the equations, but the paper does not say so. The code also uses one gate vector per sample across time transformer.py.
• Head = first time step. The heads are index 0 along the time axis models.py. With GloVe, index 0 is the first word, not a special token. With BERT, the text sequence starts with [CLS] §3.2. The code pads the visual and acoustic sequences with one step at each end models.py, so index 0 of those outputs is a padded position, not a feature step.
• Text and correlation claims disagree. The text says the model does not beat the baselines on correlation for CMU-MOSI §5.1. Table 1 gives BBFN 0.755 against MISA 0.748 and marks 0.755 as best tab. 1. This card reports the table. The ablation's TV + VA row has higher MOSEI correlation (0.775) than the full model (0.767) tab. 3.
• Significance is under-specified. The ‡ marker reports a paired t-test with p < 0.05 against MISA tab. 1, but the number of runs or seeds is not stated. The code's seed is 1111 main.py.
• Reported numbers come from test-set selection in the code. Because the checkpoint is chosen with test MAE solver.py, the reported test metrics are optimistically biased compared with selecting on validation alone. The paper does not say how the epoch was chosen.
• Alignment is not reproduced by the code read here. The paper uses P2FA for word-level alignment §4.2. The --aligned flag is parsed in the code main.py, but the data loader does not refer to it. How the features are aligned is in src/create_dataset.py, which was not checked in detail.
• Compute cost is not stated. The paper gives no parameter count, memory use, or training time. Each module has L layers with two BiGRUs, two attention blocks and a discriminator per layer, and there are two modules.
• Visual feature widths. The paper gives visual vector lengths of 47 (MOSI), 35 (MOSEI) and 75 (UR-FUNNY) §4.2. The code derives its input sizes from the dataset loader, which was not checked, so these were not confirmed.

Lineage

• Builds on: Transformer-based multimodal fusion with cross-modal attention, as in MulT (Tsai et al., 2019) §2 §4.3. The code names its model class MULTModel models.py, and the paper compares against MulT. The paper does not state that the code is derived from MulT.
• Separator idea: distance-based or distribution-based separators are cited from prior work (MISA, Hazarika et al., 2020; Sun et al., 2020); the paper argues a discriminator is preferable because it is parametric §3.3.
• Compared against: DFF-ATMF, LMF, TFN, MFM, ICCN, MulT and MISA §4.3 tab. 1.
• Superseded by: not stated in this card.