Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling

In plain words

A video review carries meaning in three channels at once: the words spoken, the tone of voice and the facial expression. Older systems squeeze the three sources into one long list of numbers and hand it to a classifier. That lets the sources contradict each other without any mechanism to sort it out.

The paper's idea is to combine the sources in stages. First it looks at two sources together, asking how the voice and the face agree or disagree for each part of the sound, and only then brings in the third source. Each stage also looks at the neighbouring sentences of the same clip, because a sentence's meaning often depends on what came before it. The authors report that this beats simply joining the lists, and that it also beats prior methods on two emotion and sentiment datasets.

TL;DR

• Text, audio and video features are first encoded per utterance, then run through a GRU over the clip.
• Each pair of modalities is fused per dimension with learned weights, contextualised with a GRU, and the three pairs are fused again.
• Reported gains are 1-2 points over early fusion and prior methods; the code is MIT and the paper and code disagree in many hyperparameters.

Problem

Early fusion concatenates the unimodal vectors. The authors argue this cannot filter conflicting or redundant information across modalities [§3.1, "Problems of early fusion"]. Context between utterances in a clip is also needed, since an utterance can be ambiguous alone §3.3. The method assumes that per-utterance features from all three modalities are available for every clip, with a fixed maximum number of utterances N, shorter clips padded with null vectors §3.3.

Method

Overview. Three unimodal streams are encoded, passed through a GRU for context, then fused hierarchically: unimodal to bimodal to trimodal, with a GRU after each fusion stage. The final vector goes to a softmax §3.1 fig. 4.

Unimodal features (§3.2).

• Text: each utterance is a sequence of 300-dimensional word2vec vectors, truncated or zero-padded to 50 words §3.2.1. Two convolution layers follow. The first has filters of width 3 and 4 with 50 maps each; the second has a filter of width 2 with 100 maps; each is followed by 2x2 max pooling. A fully connected layer of 500 ReLU units feeds a softmax, and the penultimate layer is used as the text feature §3.2.1.
• Audio: openSMILE at a 30 Hz frame rate with a 100 ms window, IS13-ComParE configuration, z-standardisation and voice-intensity thresholding. The card quotes 6392 features per segment §3.2.2.
• Video: a 3D convolution over the frames, with 32 feature maps and 5x5x5 filters, 3x3x3 max pooling, then a dense layer of 300 units. The dense activations are the video feature §3.2.3.

Context modelling (§3.3). Per modality m in {A, V, T}, a GRU with output size D_m runs over the utterance sequence of a clip, giving F_m. The paper's equations for the GRU are unnumbered §3.3. D_m is not stated. The authors say LSTM "does not perform well" as a footnote §3.3.

Common width (§3.4). Each context-aware vector is mapped to width D with a tanh dense layer, D = 400 in the best setting §3.4.

Bimodal fusion (§3.4). For each pair of modalities, every dimension l of the mapped vectors is combined by its own two weights and a bias, then tanh is applied eq. 1 eq. 2 eq. 3. The pair vectors feed a GRU with output size D2 = 500 §3.4. The paper says the penultimate layer serves as the bimodal feature §3.1.

Trimodal fusion (§3.4). The three pair vectors are combined per dimension, with three weights and a bias, and tanh §3.4. A GRU with output size D3 = 550 follows §3.4. alg. 1 writes the same steps as procedures.

Classification (§3.5). A softmax with C outputs gives the class. The text lists m = AV, VT, TA and AVT for the softmax input, so it may apply to pair features as well as the trimodal one §3.5. For CMU-MOSI, C = 2 §3.5. For IEMOCAP, C = 4 §4.4.2.

Training (§3.6). Categorical cross-entropy, Adam, up to 200 epochs with early stopping §3.6. Trainable parameters are listed as the union of GRU, mapping, fusion and softmax weights §3.6.

# Context-aware hierarchical fusion  [alg. 1] [§3.1]
# Input:  f_text (N, d_T), f_audio (N, d_A), f_video (N, d_V)
# Output: probs (N, C)

def forward(f_text, f_audio, f_video):
    # Context over each modality's utterance sequence  [§3.3]
    F_text = gru_text(f_text)                   # (N, D_text), D_text not stated
    F_audio = gru_audio(f_audio)                # (N, D_audio)
    F_video = gru_video(f_video)                # (N, D_video)
    # Map every modality to one common width D = 400  [§3.4]
    g_text = tanh(F_text @ W_text + b_text)     # (N, D)
    g_audio = tanh(F_audio @ W_audio + b_audio)  # (N, D)
    g_video = tanh(F_video @ W_video + b_video)  # (N, D)
    # Compare and combine each pair per dimension  [eq. 1] [eq. 2] [eq. 3]
    pair_va = pair_fuse(g_video, g_audio)       # (N, D)
    pair_at = pair_fuse(g_audio, g_text)        # (N, D)
    pair_vt = pair_fuse(g_video, g_text)        # (N, D)
    # Context over each pair sequence, width D2 = 500  [§3.4]
    bimodal_va = gru_va(pair_va)                # (N, D2)
    bimodal_at = gru_at(pair_at)                # (N, D2)
    bimodal_vt = gru_vt(pair_vt)                # (N, D2)
    # Combine the three pairs per dimension  [§3.4]
    fused = tri_fuse(bimodal_va, bimodal_at, bimodal_vt)  # (N, D2)
    # Context over the trimodal sequence, width D3 = 550  [§3.4]
    context = gru_avt(fused)                    # (N, D3)
    # Softmax over the classes  [§3.5]
    return softmax(context @ W_out + b_out)     # (N, C)


def pair_fuse(first, second):
    # Per dimension: own weights for each modality, then tanh  [eq. 1]
    return tanh(first * w_first + second * w_second + bias)  # (N, D)


def tri_fuse(first, second, third):
    # Per dimension: three weights, then tanh  [§3.4]
    combined = first * w_1 + second * w_2 + third * w_3 + bias
    return tanh(combined)                       # (N, D2)

Key equations:

J = -\frac{1}{N}\sum_{i=1}^{N}\sum_{j=0}^{C-1} y_{ij}\log \mathcal{P}_i[j] §3.6

Hyperparameters

Paper values come from the text; code values come from the pinned repository (hfusion.py, code_commit above). "not stated" means the paper does not say.

Results

Author-reported. Setup: CMU-MOSI with a person-independent split, 1447 training and 752 test utterances §4.1.1 tab. 1; IEMOCAP with four classes §4.1.2 tab. 1. Baselines: early fusion with an SVM §4.2; the MKL method of Poria et al., re-implemented by the authors §4.2; the LSTM context model the paper cites as the state of the art, rerun without its SVM §4.2; and the tensor fusion network of Zadeh et al., adapted by the authors to their data §4.2. Hardware and number of runs or seeds are not stated.

Headline claims, in the abstract and §1: on individual utterances, the hierarchical fusion beats concatenation by 1% (5% error reduction); on utterance-level clips with context, by up to 2.4% (about 10% error reduction).

CMU-MOSI, no context (HFusion), accuracy tab. 2:

Baselines are SOTA1 / SOTA2 for the Poria et al. feature set, and early fusion / SOTA2 for this paper's feature set tab. 2. Unimodal accuracy with this paper's features tab. 2: text 75.0%, video 55.3%, audio 56.9%.

CMU-MOSI and IEMOCAP, with context (CHFusion) tab. 3:

SOTA1 is the MKL method of Poria et al.; SOTA2 is the tensor fusion network of Zadeh et al. tab. 3. Accuracy is the reported metric on MOSI; IEMOCAP is reported with accuracy and F-score.

IEMOCAP trimodal (A+V+T), class-wise tab. 4: happy 74.3 accuracy / 81.4 F-score; sad 75.6 / 77.0; neutral 78.4 / 71.2; anger 79.6 / 77.6.

Context vs no context, CMU-MOSI §4.4.3: the context model (CHFusion) is 1-2 points better than HFusion for every modality combination. No table is given.

Paper vs code

Searched for the official repository: the paper's footnote links github.com/senticnet and a senticnet/hierarchical-fusion repository, which returns 404. A GitHub search returned declare-lab/hfusion (MIT). Its README names the paper's five authors and its code header is by S. Poria. It is therefore used as the official code. The README says the repository is no longer maintained and points to declare-lab/multimodal-deep-learning, which was not checked.

• Training is staged, not end-to-end. The paper says the method is end-to-end §2. The code trains each pair model with its own loss, saves the activations to bimodal.pickle, then trains the trimodal model on those frozen activations hfusion.py hfusion.py hfusion.py.
• Pair-level supervision. The code puts a 4-way softmax on each pair GRU and trains it with the utterance labels hfusion.py hfusion.py. The paper names no pair-level loss §3.5 §3.6.
• Common width. Paper D = 400 §3.4; code uses 450 hfusion.py hfusion.py hfusion.py.
• Bimodal GRU. Paper output width D2 = 500 directly from the GRU §3.4; code uses a 600-unit GRU and then a 500-unit tanh dense layer, and takes the dense output as the pair feature hfusion.py hfusion.py hfusion.py.
• Trimodal GRU. Paper D3 = 550 §3.4; code uses 500 hfusion.py.
• Classifier head. Paper applies a softmax directly to the context vector §3.5. The code concatenates the fused vector with the GRU output hfusion.py, then applies a 450-unit ReLU dense layer with dropout 0.86 hfusion.py hfusion.py, then a 4-way softmax hfusion.py.
• Dense mapping bias. The paper's mapping includes a bias §3.4. The code sets use_bias=False on the three mapping layers hfusion.py.
• Fusion rule matches. The per-dimension weighted sum and tanh used by the pair and trimodal fusion match the paper eq. 1 eq. 2 eq. 3 §3.4 hfusion.py hfusion.py.
• Epochs. Paper: up to 200 §3.6; code: 100 for the trimodal stage hfusion.py.
• Classes. The paper's CMU-MOSI setup is binary §3.5 §4.1.1. The code's output layers have four classes hfusion.py hfusion.py. The shipped pickle does not say which dataset it holds, so this cannot be confirmed from the repository.
• Feature extraction absent. The repository has no text CNN, openSMILE or 3D-CNN code. It loads unimodal.pickle hfusion.py. The visual width is taken from the pickle; the paper says 300 §3.2.3.
• Dummy utterances are zeroed under a mask, as the paper describes §3.3 hfusion.py.
• Test set during training. A callback prints test metrics every epoch hfusion.py hfusion.py. Early stopping uses the validation split, so the test metrics do not drive stopping. The paper does not say which epoch's numbers it reports.

Pitfalls

• Staged training. Reproducing the trimodal stage needs bimodal.pickle, which Bimodal() writes. The default __main__ runs only Trimodal() hfusion.py. Run Bimodal() first.
• Unimodal encoders are missing. Text, audio and video encoders are not in the repository, so the features cannot be reproduced from it alone hfusion.py.
• Heavy dropout. Dropout rates of 0.86 and 0.9 are in the code; the paper states none hfusion.py hfusion.py hfusion.py.
• Ambiguous classifier scope. §3.5 lists pair-level softmax inputs as well as the trimodal one, while §3.6 describes one loss. Pair-level supervision is what the code does §3.5 §3.6 hfusion.py.
• Dataset counts disagree within the paper. §4.1.2 gives 1083 angry, 1630 happy, 1083 sad and 1683 neutral "videos" §4.1.2. The IEMOCAP rows of tab. 1 sum to 1090 angry, 1627 happy, 1077 sad and 1704 neutral (train plus test). The units or the numbers differ; the paper does not resolve this.
• Small gains, no variance. The reported improvements are 1-2 points. The number of runs, seeds and any standard deviation are not stated §4.4.2.
• Baseline comparability. The early-fusion baseline uses an SVM, while HFusion does not §4.2. The tensor fusion baseline was adapted by the authors to their data §4.2.
• Imbalance. IEMOCAP is imbalanced; the paper therefore reports F-scores as well §4.4.2.
• Keras API. The README asks for Keras 2.0 or later and TensorFlow 1.7 or later. The code uses sample_weight_mode='temporal' and lr=, which newer Keras versions changed hfusion.py hfusion.py.
• Figure credit. The footnote says the trimodal figure is adapted from a master's thesis with permission §3.1 fig. 3. We do not reproduce figures.

Lineage

• Builds on: the LSTM context model that §2 cites as the state of the art, and the CNN-plus-MKL fusion of Poria et al. (SOTA1 in tab. 2 and tab. 3). The GRU context design follows the LSTM model §3.3.
• Compared against: early fusion with SVM §4.2; the tensor fusion network of Zadeh et al. §4.2.
• Superseded by: not known from content in this wiki.