UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition

In plain words

Sentiment (how positive or negative a clip feels, and how strongly) and emotion (joy, anger and similar states) are usually studied as two separate problems, each with its own labels, datasets and models. UniMSE treats them as one problem. It gives every training clip a single combined label. When a clip only has one of the two kinds of annotation, the missing part is borrowed from the most similar sentence in the other dataset.

A pretrained text model then writes that combined label out as text, so one model answers both questions. Audio and video features are fed in through small add-on layers near the top of the model, so the text model can take them into account. An extra training term pulls each text representation toward the audio and video of its own clip and away from the clips in the same batch. The released code leaves this term switched off by default.

TL;DR

• Sentiment intensity and emotion recognition are trained together as one generative task on T5-Base, with a shared label string §3.2.2 §3.5.
• Audio (Mel-spectrogram) and video (EfficientNet) features enter through adapter layers placed in the last three T5 Transformer blocks §3.3 §4.4.
• Author-reported: MOSI MAE 0.691 and Corr 0.809; MELD accuracy 65.09; IEMOCAP accuracy 70.56 tab. 2. The code departs from the paper in many details (see Paper vs code), so treat the paper's hyperparameters as the spec and the code as a reference for the gaps.

Problem

Most existing work treats multimodal sentiment analysis (MSA) and emotion recognition in conversation (ERC) as separate tasks. Sentiment is read as a longer-lasting stance and emotion as a short-lived state, and the two are evaluated on different datasets with different metrics, so knowledge is not shared between them §1 §4.5. The label formats differ: MSA predicts a real number for sentiment strength, while ERC predicts an emotion category for each utterance §3.2. The paper argues that features, labels and model can be shared across the two tasks §1 §3.1.

Method

Overview

The model has four parts. Each clip's label is rewritten into a universal label string §3.2.2. Audio and video are encoded by one LSTM per modality §3.1 §3.2.1. A T5 encoder reads the text, and fusion adapters after the feed-forward sub-layer of its last layers inject the audio and video §3.3 fig. 2. A contrastive term is added between the fused text states and each non-text modality §3.4. The decoder generates the universal label, which is then decoded to a number for MSA or to a category for ERC §3.5 alg. 1.

1. Text input: a window of utterances

The text input concatenates the current utterance with the two utterances before it and the two after it, and a segment id marks the current utterance §3.2.1 eq. 1. The paper's limitations section says this context is used only for MELD and IEMOCAP [Limitations]. See Pitfalls.

I^{t}_{i} = [u_{i-2}, u_{i-1}, u_{i}, u_{i+1}, u_{i+2}], \quad S^{t}_{i} = (0,0,1,0,0)\ \text{per utterance} eq. 1

2. Universal label

The target is a triple (polarity, intensity, emotion). Polarity is one of positive, neutral or negative. Intensity is a real number from -3 to +3 (the MSA score). Emotion is a predefined category (the ERC label) §3.2.2. Each dataset only fills part of the triple, so the other parts are completed as follows §3.2.2 fig. 3:

# Universal label completion  [§3.2.2] [fig. 3]
# Input:  MSA samples (intensity in -3..3), ERC samples (emotion category)
# Output: every sample labelled (polarity, intensity, emotion)

def complete_labels(msa_samples, erc_samples):
    # Split both sets by polarity: positive, neutral, negative  [§3.2.2]
    for polarity in ["positive", "neutral", "negative"]:
        msa_group = samples_with(msa_samples, polarity)
        erc_group = samples_with(erc_samples, polarity)
        # Embed each text with SimCSE; take the most similar partner  [§3.2.2]
        for sample in msa_group:
            partner = most_similar(sample, erc_group)
            sample.emotion = partner.emotion
        for sample in erc_group:
            partner = most_similar(sample, msa_group)
            sample.intensity = partner.intensity
    return msa_samples, erc_samples

The paper reports that a manual check of 80 completed MOSI labels was about 90% accurate §3.2.2. Decoding a generated triple back to a task output is described in alg. 1:

# Decoding for MSA or ERC  [alg. 1] [app. A.2]
# Input:  target task, generated triples Y = [(p, r, c), ...]
# Output: one prediction per triple

def decode(task, triples):
    # Keep the intensity field r for MSA, the emotion field c for ERC
    predictions = []
    for triple in triples:
        if task == "MSA":
            predictions.append(triple.intensity)
        if task == "ERC":
            predictions.append(triple.emotion)
    return predictions

3. Audio and video encoders

Audio is a Mel-spectrogram, extracted with librosa §3.2.1. Video uses a fixed number of frames per clip, each passed through an EfficientNet pretrained on VGGFace and AFEW §3.2.1; the frame count is not stated. Each modality goes through its own LSTM. The last hidden state of each LSTM is the clip-level vector used by the fusion layers, and the full sequence is used by the contrastive term §3.3 §3.4. The released code does not contain the extraction scripts; see Paper vs code.

4. Pre-trained modality fusion (PMF)

Fusion layers are inserted into the T5 blocks. Each one follows the feed-forward sub-layer of its block. It concatenates the current text state with the audio and video vectors, squeezes the result through a narrow sigmoid bottleneck, adds the bottleneck output back, and projects back to the text width §3.3 eq. 2.

F^{d}_{i} = \sigma(W^{d} F_{i} + b^{d}), \quad F^{u}_{i} = W^{u} F^{d}_{i} + b^{u}, \quad F^{(j)}_{i} = W\,(F^{u}_{i} \odot F^{(j-1)}_{i}) eq. 2

Here F_i is the concatenation of F^{(j-1)}_i with the last-step audio and video vectors, and \sigma is the sigmoid §3.3. The text describes \odot as element addition, while the symbol is a Hadamard product (see Pitfalls).

# One fusion adapter, after a T5 feed-forward sub-layer  [§3.3] [eq. 2]
# Input:  hidden (B, L, 768), audio_vec (B, 32), video_vec (B, 32)
# Output: fused hidden (B, L, 768)

def fusion_adapter(hidden, audio_vec, video_vec):
    # Repeat the clip-level audio and video vectors over all tokens
    audio_rep = repeat_over_tokens(audio_vec, L)   # (B, L, 32)
    video_rep = repeat_over_tokens(video_vec, L)   # (B, L, 32)
    # Concatenate the text state with both modalities  [eq. 2]
    joint = concat(hidden, video_rep, audio_rep)   # (B, L, 832)
    # Narrow sigmoid bottleneck  [eq. 2]
    bottleneck = sigmoid(down_project(joint))      # (B, L, 64)
    # Expand and add a residual over the joint input  [§3.3]
    expanded = up_project(bottleneck) + joint
    # Map back to the T5 width  [eq. 2]
    return linear(expanded)                        # (B, L, 768)

The paper places the fusion layers in the last three Transformer layers of the T5 encoder §4.4. The paper also says the same layer could sit in every encoder and decoder block, but that this would disturb text encoding and add parameters §3.3.

5. Contrastive term

For each fused layer j, the text representation is contrasted with the sequence of audio states and, separately, with the sequence of video states §3.4 eq. 3 eq. 4. Both sides are first mapped to a common length and width by a 1-D temporal convolution eq. 3. Within a mini-batch, the other samples supply the negatives §3.4.

L^{ta,j} = -\log \frac{\exp(\hat F^{(j)}_{i} \hat X^{a}_{i})}{\exp(\hat F^{(j)}_{i} \hat X^{a}_{i}) + \sum_{k=1}^{K} \exp(\hat F^{(j)}_{i} \hat X^{a}_{k})} eq. 4

L^{tv,j} has the same form with video in place of audio eq. 4.

# Inter-modality contrastive term for one fused layer  [§3.4] [eq. 4]
# Input:  fused (B, L, 768), modality_seq (B, T, 32), audio or video
# Output: scalar loss over the mini-batch

def contrastive_term(fused, modality_seq):
    # Map both sides to a common length and width  [§3.4] [eq. 3]
    text_side = conv1d_text(fused)                 # (B, L', k)
    other_side = conv1d_modality(modality_seq)     # (B, L', k)
    # Dot-product scores of each anchor against every sample  [eq. 4]
    scores = pairwise_dot(text_side, other_side)   # (B, B)
    # The matching sample is the positive; the rest are negatives  [eq. 4]
    return cross_entropy(scores, matching_index)

6. Training objective

The generation loss is combined with the two contrastive sums, each weighted by 0.5 §4.4 eq. 5.

L = L^{task} + \alpha \sum_{j} L^{ta,j} + \beta \sum_{j} L^{tv,j} eq. 5

# One training step  [§3.1] [§3.5] [eq. 5]
# Input:  text windows, audio (B, T_a, d_a), video (B, T_v, d_v), targets
# Output: scalar training loss

def training_loss(text, audio, video, target):
    # One LSTM per modality; keep the sequence and the last state  [§3.2.1]
    audio_seq, audio_vec = lstm_audio(audio)
    video_seq, video_vec = lstm_video(video)
    hidden = t5_embed(text)
    ta_terms = []
    tv_terms = []
    for index, block in enumerate(t5_encoder_blocks):
        # Standard T5 self-attention, then the feed-forward sub-layer  [§3.3]
        hidden = block.self_attention(hidden)
        hidden = block.feed_forward(hidden)
        # Fusion and contrastive terms in the last three blocks  [§4.4]
        if index >= num_blocks - 3:
            hidden = fusion_adapter(hidden, audio_vec, video_vec)
            ta_terms.append(contrastive_term(hidden, audio_seq))
            tv_terms.append(contrastive_term(hidden, video_seq))
    # Generation loss on the universal-label string  [§3.5]
    task_loss = t5_decoder_loss(hidden, target)
    return task_loss + 0.5 * sum(ta_terms) + 0.5 * sum(tv_terms)

Hyperparameters

Values stated in the paper are marked §4.4. Code values are the defaults in src/config.py of the commit listed in the frontmatter, not a recorded run.

Results

Author-reported. Setup: four public benchmarks §4.1 tab. 1, T5-Base with the training sets pooled §4.4, metrics as defined in §4.2. Baseline numbers for each row are as published by their authors in the baseline repositories tab. 2 caption. Many cells are "-" because the baseline reported other metrics only.

Dataset sizes, from tab. 1:

Metrics: MOSI and MOSEI use MAE, Corr, ACC-7, ACC-2 and F1; MELD and IEMOCAP use ACC and weighted F1 §4.2. Two-number cells such as 85.85/86.9 are the two binary splits of ACC-2 and F1 §4.2 tab. 2.

Sentiment, MOSI and MOSEI tab. 2:

Emotion recognition, MELD and IEMOCAP tab. 2:

In the paper's table, italic rows use only text, and asterisked rows were re-run by their authors in the official baseline code tab. 2 caption. The MSA-only and ERC-only baselines appear in the MSA and ERC tables respectively. The paper states that UniMSE improves on the previous best by 1.65, 1.16, 2.6 and 2.35 points on MOSI ACC-2, MOSEI ACC-2, MELD ACC and IEMOCAP ACC, and by 1.73, 1.29 and 2.48 points on MOSI F1, MOSEI F1 and IEMOCAP WF1 §4.5. Those comparisons are against the best baseline in each cell.

Ablation, evaluated on the MOSI test set tab. 3 §4.6:

The ablation section states that acoustic input matters more than visual input for MOSI, and that removing either PMF or CL raises MAE and lowers Corr §4.6. Visualizations of fused representations are qualitative §4.7 fig. 4.

Paper vs code

Code at commit 7518ac098357c88b70b53e10b822a7f776d058c9, repository https://github.com/LeMei/UniMSE. The paper links this repository in its contributions §1, which is why it is used. The repository has no LICENSE file; the card records code_license as none, so only path-and-line citations are used. The README says the source would be released later, but the source is present at this commit. The data files and pretrained checkpoints are not in the repository: the README points to Google Drive and Baidu Pan links [README]. I searched only the links the paper and README give. No other repository was searched.

• Batch size. Paper: 96 §4.4. Code default: 32 config.py. The repo contains no script that sets it.
• Acoustic and visual widths. Paper: 64 §4.4. Code defaults: 32 for hidden and output sizes config.py.
• Contrastive term is off by default. The flag defaults to False config.py, so the default run has no contrastive loss. The paper's experiments use it §3.4 §4.4.
• Where the contrastive term is computed. Paper: on the fused representation F^{(j)} §3.4 eq. 4. Code: on the input to each block, for blocks with index above 3 modeling_t5_prefix.py.
• Only one contrastive value is kept. The paper sums the terms over the layers eq. 5. In the code the tuple holding the term is overwritten on every block, so only the last block's value survives modeling_t5_prefix.py modeling_t5_prefix.py.
• Contrastive similarity. Paper: an unnormalized dot product with K sampled negatives eq. 4. Code: L2-normalized, mean-pooled over time, projected to 30 dimensions, divided by a temperature of 0.1, with the other batch samples as negatives InfoNCE.py InfoNCE.py InfoNCE.py InfoNCE.py InfoNCE.py.
• Decoder receives the same terms. The decoder stack is built with the same class and receives the modality inputs, and its contrastive values are added to the encoder's modeling_t5_prefix.py modeling_t5_prefix.py.
• Fusion placement. Paper: last three encoder layers §4.4. Code: last three blocks of both the encoder and the decoder stacks modeling_t5_prefix.py modeling_t5_prefix.py.
• Modality vectors passed to the fusion layer. Paper: the last hidden state of each LSTM §3.3 eq. 2. Code: that state passed through one more linear layer to the output width, which is 32 by default encoders.py model.py.
• Residual in the fusion layer. Paper: the symbol in eq. 2 is a Hadamard product, but the text calls it element addition §3.3. Code: adds the up-projection to the joint input, which includes the audio and video vectors, then applies a linear layer adapters.py.
• Adapter width. Not stated in the paper. Code: 64 adapters.py.
• Which T5 weights train. Not stated in the paper. Code freezes every T5 parameter except those whose names contain block.10 or block.11, which matches the last two blocks of both stacks. Adapter, LSTM and projection parameters train solver.py functions.py config.py.
• Learning-rate schedule. Not stated in the paper. Code creates a plateau scheduler with factor 0.5 and patience 20, but never calls its step method, so the learning rate stays constant solver.py.
• Gradient clipping and accumulation. Not stated in the paper. Code defines a clip of 1.0 and an accumulation factor of 5, but both uses are commented out, so the optimizer steps after every batch config.py config.py solver.py solver.py solver.py.
• Dropout on the audio and video heads. Paper: not stated. Code: the LSTM head uses dropout 0.3 when there is one layer, so the visual setting of 0.5 has no effect with default settings model.py model.py encoders.py.
• Loss weights. Paper: alpha and beta are 0.5 §4.4. Code: fixed at 0.5 in the loop, not read from config solver.py. Consistent, but not configurable.
• Model selection. Paper: validation sets choose hyperparameters §4.4. Code: test sets are evaluated every epoch, and the reported test results come from the last epoch whose validation loss improved. The test-loss check in that branch always passes, because mosi_best_mae is never updated solver.py solver.py.
• Decoding. Paper: a field-selection algorithm over triples alg. 1. Code: decodes the whole generated string, converts it to a float when it parses as a number, and pads one-field outputs with a default partner value solver.py solver.py solver.py.
• Universal-label matching. Paper: SimCSE, with no checkpoint named §3.2.2. Code: the supervised SimCSE BERT-base checkpoint, pooled output, cosine similarity and an argmax over the other dataset Sim_Process_v3.py Sim_Process_v3.py Sim_Process_v3.py. The code also matches MOSI against the restaurant and laptop ABSA sets, which the paper does not mention Sim_Process_v3.py.
• Default emotion for unmatched MOSI clips. Code: neutral when no match is found Sim_Process_v3.py. Not in the paper.
• Input prefix. Code prepends the string sst2 sentence: to every input data_loader.py data_loader.py. Not in the paper.
• Context windows. Paper: two utterances on each side §3.2.1 eq. 1, but the limitations say MOSI and MOSEI get no context [Limitations]. Code reads pre-built pickles whose names contain 6c_contexts; the window is not defined in the repo create_dataset.py.
• Feature extraction. Paper: librosa Mel-spectrogram and EfficientNet frames §3.2.1. Code: no extraction script; the missing-audio fallback is a 157 by 64 array of zeros preprocess.py.
• Universal-label files. The repo has the labelling and preprocessing scripts but not their outputs.

Pitfalls

• Label quality. About 90% accuracy was checked on 80 MOSI samples §3.2.2. The rest of the completed labels are unchecked, and ERC clips get a sentiment intensity only from their nearest MOSI neighbour.
• Polarity of ERC clips. The paper does not say how ERC clips get a polarity before the nearest-neighbour step §3.2.2. For MOSI and MOSEI the code uses the dataset's annotation column Sim_Process_v3.py; the source for MELD and IEMOCAP is not checked here.
• Emotion classes in MELD. The text lists six emotions §4.1. The MELD distribution table has no neutral column app. A.1 tab. 5, and its all-row total is 7,274 against 13,707 in tab. 1. The paper does not explain the gap.
• Emotion counts in IEMOCAP. The all-row total in the IEMOCAP distribution table is 7,380 against 7,532 in tab. 1. Not explained.
• MOSEI emotion labels are unused §4.1. Only the sentiment score is used there.
• Unstated but needed details. Epochs, patience, weight decay, dropout, decoding, the SimCSE checkpoint, the contrastive temperature, the adapter width, the frame count, the Mel settings and the context window are not in the paper. Implement the code defaults only if you accept the Paper vs code deviations above.
• Batch size and seeds. The paper uses batch size 96 §4.4 and reports no variance across seeds or repeated runs §4.5. The code's default batch size is 32, so a run with defaults will not match the paper's setting.
• Significance. The paper says UniMSE "significantly" outperforms prior work §4.5, but reports no significance test or variance across runs.
• Contrastive ablation setting. The paper does not say whether the main results use the contrastive term, although its ablation removes it §4.6. The code's default is off config.py.
• Cost. T5-Base has 220M parameters, and the fusion layers are described as about a thousandth of that size app. A.3. The paper's wording is "PML", which we read as PMF. The GPU count and training time are not stated.
• Compute on different datasets. Training pools four datasets into one model §4.4. Per-dataset gains are therefore not separable from the pooled setup.
• Data access. The data files come from external links [README]. The repo has no license for the code, and the datasets' terms are not stated here.
• Deployment. The authors state that accuracy on MELD (65%) is far from usable for emotion-aware products [Ethics Statement].

Lineage

• Builds on: T5 as the text backbone §3.3 and adapter-style modules after each feed-forward sub-layer, following the cited adapter work §3.3. The SimCSE sentence encoder is used for label matching §3.2.2. No cards for these are linked.
• Compared against: MSA baselines LMF, TFN, MFM, MTAG, SPC, ICCN, MulT, MISA, COGMEN, Self-MM, MAG-BERT and MMIM; ERC baselines DialogueGCN, DialogueCRN, DAG-ERC, ERMC-DisGCN, CoG-BART, Psychological, COSMIC, TODKAT, MMGCN and MM-DFN tab. 2.
• Superseded by: not known from content on this wiki.