Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines

In plain words

People post opinion videos where the words, the tone of voice and the face all carry information about how the speaker feels. This paper sets up simple, reproducible starting points for judging that feeling from all three channels together, and checks several things that earlier work often skipped.

The starting points are built in stages. Each utterance gets a text vector from a small text network, plus features for sound and for video. The three are glued into one long vector. A classic classifier then labels it. The last and strongest stage lets the model look at the utterances around the one being labelled, so the context of the conversation can help.

The authors also ask questions that matter in practice. Does the model still work when the test speakers were never seen in training? Which channel helps most? Does a model trained on one dataset transfer to another, in a different language? Their answers are mixed, and the paper says so.

TL;DR

• Three baselines on the same features: an SVM on concatenated features §3.4, and bc-LSTM, a bidirectional LSTM that adds neighbouring utterances as context §3.4.1.
• Features: a CNN over text, 3D-CNN over video, openSMILE over audio §3.1-§3.2.
• Main protocol is speaker-exclusive, with 5 or 10 folds §4.2. Table 4 is the consolidated comparison tab. 4.
• Best author-reported numbers, speaker-exclusive, all three modalities: bc-LSTM 76.1 (IEMOCAP, 4 classes), 68.1 (MOUD), 80.3 (MOSI); SVM 72.5, 66.1, 77.9 tab. 4.
• Cross-dataset (train MOSI, test MOUD) is weak: best 52.7% with bc-LSTM tab. 3.
• Several numbers in the paper disagree with each other. See Pitfalls.

Problem

Earlier multimodal sentiment papers mostly used datasets where the same speakers appear in training and test, so the scores overstate how well a model would work on new people §4.2. The paper also asks which modality matters most, whether context helps, and whether a model generalises across datasets §1. It states that these issues make methods hard to compare and sets a benchmark with fixed train/test splits §1, §4.1.

Assumptions: each utterance is short (about 5 s on average in MOUD §4.1) and has one label. Datasets have fixed splits; the paper's splits are in tab. 1. The text modality needs transcripts; the card does not cover how they were produced.

Method

The pipeline has three parts: unimodal feature extraction, fusion, and a classifier. The two classifiers are the baselines.

Unimodal features

Text §3.1. An utterance's words go through a CNN in the style of Kim (2014) §3.1. The network has three convolution widths (3, 4 and 5 tokens), each with 50 feature maps. Each map is max-pooled, passed through ReLU, and the results are concatenated. A dense layer then gives a 100-dimensional utterance vector. The network is trained on utterance-level emotion labels §3.1.

Audio §3.2. Acoustic features come from openSMILE §3.2. The paper follows bc-LSTM for the exact setup §3.2.

Video §3.2. Visual features come from a 3D-CNN, again following bc-LSTM §3.2.

Fusion

Feature-level fusion §3.3. For each utterance, the unimodal vectors of the chosen modalities are concatenated. No scaling, normalisation or learned merge is applied, on purpose §3.3. Any subset of {text, audio, video} can be used, giving the seven combinations in the tables.

Classifiers

SVM §3.4.2. The fused vector goes to a support vector machine with an RBF kernel. This is the non-contextual baseline.

bc-LSTM §3.4.1. Following bc-LSTM (Poria et al., 2017, cited as poria-EtAl:2017:Long) §3.4.1, a bidirectional LSTM reads the sequence of fused utterance vectors of a conversation. Its output for each utterance is a context-aware representation, which is then classified. The paper does not restate the LSTM hyperparameters; see Hyperparameters.

# Baseline pipeline for one utterance  [§3.1]-[§3.4]
# Input:  tokens, audio clip, video frames for one utterance
# Output: one label (emotion, or sentiment polarity)

def text_vector(tokens):
    # Three convolution widths, 50 maps each  [§3.1]
    maps = [convolve(tokens, width=w, n_maps=50) for w in (3, 4, 5)]
    # Max-pool each map, apply ReLU, concatenate  [§3.1]
    pooled = concatenate([relu(max_pool(m)) for m in maps])
    # Dense layer to a 100-dimensional vector  [§3.1]
    return dense(pooled, size=100)

def fuse(vectors):
    # Plain concatenation of the chosen modality vectors, no scaling  [§3.3]
    return concatenate(vectors)

def classify_svm(fused_vector):
    # RBF-kernel SVM, no context  [§3.4.2]
    return svm_rbf_predict(fused_vector)

def classify_bc_lstm(fused_sequence):
    # Bidirectional LSTM over the utterances of one conversation  [§3.4.1]
    context = bidirectional_lstm(fused_sequence)
    return classify(context)

Evaluation protocol

Speaker-exclusive evaluation §4.2. Each fold's test set contains speakers who do not appear in that fold's training set:

• IEMOCAP: one speaker per fold, 10 folds §4.2.
• MOUD: speakers split into 5 groups, one held out per fold §4.2.
• MOSI: speakers split into 5 groups, one held out per fold; about 75 speakers remain for training on average §4.2.

The reported score is the average over folds tab. 2, tab. 4 captions. Validation data is an 80/20 split of each training set tab. 1 caption, and for MOSI the text describes it explicitly §4.2.

The speaker-inclusive comparison in tab. 2 uses the same SVM setting with a train/test split that shares speakers.

Hyperparameters

Results

Author-reported. Setup: five-fold or ten-fold speaker-exclusive evaluation on IEMOCAP, MOUD and MOSI, with the features above and the training data of tab. 1 §4.2. Baselines are the SVM and bc-LSTM rows. No model size, hardware or run time is reported.

Dataset sizes tab. 1:

Speaker-inclusive vs speaker-exclusive, concatenation with SVM tab. 2. Accuracy in %. "--" means not reported. A = audio, V = video, T = text.

Cross-dataset: trained on MOSI, tested on MOUD tab. 3. Accuracy in %. The caption says the models use their earlier configurations, so this table is not a retrained comparison.

Consolidated speaker-exclusive comparison, SVM vs bc-LSTM tab. 4. Accuracy in %. IEMOCAP: 10-fold average, four classes. MOUD: 5-fold average, two classes. MOSI: 5-fold average, two classes.

Figures (described, not reproduced):

• fig. 1 has four panels: RMSE per modality on IEMOCAP and MOSI, the dataset class distribution, and TP-rate per modality for IEMOCAP and for MOSI.
• fig. 2 shows the accuracy of each modality on each dataset, with a line at the median accuracy.
• fig. 3 is a 2-D t-SNE of MOSI and of IEMOCAP, once for unimodal and once for multimodal features. The authors read the multimodal plots as less overlapping between classes.

Paper vs code

No official code. I could not complete the code search: GitHub returned HTTP 504 for every repository URL I tried in this run and HTTP 403 for the search API, and web search was not permitted. The paper links no code, and the arXiv metadata has no code link. Nothing here is taken from a repository, so no code field is set. Re-run /compile-paper 1803.07427 to search again; the likely place is the authors' lab GitHub organisation, but that is not verified.

Pitfalls

• Speaker-exclusive claim contradicted by tab. 2. §4.2.1 says audio is consistently better than video on MOSI and IEMOCAP. In tab. 2 MOSI speaker-exclusive, video (58.46) beats audio (57.14). tab. 4 shows audio ahead on MOSI for both classifiers.
• Same SVM setting, different numbers. The SVM rows in tab. 2 and tab. 4 do not match, for example MOSI audio speaker-exclusive 57.14 vs 58.5 and IEMOCAP T+A+V 71.59 vs 72.5. The captions do not explain the gap.
• Dataset counts disagree. §4.1 says MOUD has 80 videos and 448 utterances after dropping neutral. tab. 1 gives 437 utterances (322 + 115) and 79 videos (59 + 20).
• MOSI speaker count. §4.1 says 89 speakers; §4.2 says 93 people.
• Text labels. §3.1 trains the text CNN on emotion labels. For the sentiment datasets, the paper does not say which labels are used.
• Ties and "better in all experiments". §4.5 says bc-LSTM beats SVM "across all the experiments". tab. 4 supports that, but tab. 3 has a tie for T + V (49.8 vs 49.8).
• Cross-dataset test confounded by language. The authors attribute the poor MOUD results to MOUD being in Spanish and MOSI in English §4.4. This is their explanation; they do not test it.
• Unnormalised fusion. Concatenation with no scaling §3.3 means the modality with the largest feature scale can dominate the SVM's RBF distance. The paper does not measure this.
• Fold scores. Only fold averages are reported. Per-fold spread is not given, so small gaps (for example 1 to 2 points) may not be meaningful.
• Reproduction needs bc-LSTM details. The card lists them as not stated. Re-implementing the baseline from this paper alone is not possible.
• Version. The source compiled here is arXiv v2 (updated 2019-02-12). The licence and numbers in this card follow that version; v1 may differ.

Lineage

• Builds on: bc-LSTM, Poria et al. (2017), which is this paper's own baseline method §3.4.1. Its arXiv id is not given in this paper's bibliography, so it is not linked here. The text CNN follows Kim (2014) §3.1.
• Compared against: the SVM baseline on the same features §3.4.2.
• Superseded by: not stated in this card's sources.