MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations

In plain words

Emotion recognition in conversation means labelling each turn of a dialogue with the feeling the speaker is showing. Until now most datasets were either text only, covered just two people talking, or were not dialogues at all. MELD was built from scenes of the TV series Friends. It has about 13,000 short turns, each with a video clip, its sound and a transcript, and each turn is labelled with one of seven emotions and one of three sentiments.

The idea is that a turn's emotion depends on what was said before and on who said it. The face and voice can also settle cases where the words alone are unclear, such as "yeah". The authors' baseline models show that using the earlier turns and combining text with audio helps. Results are still weak for rare emotions such as disgust, fear and sadness.

TL;DR

MELD is a dataset of about 13,000 utterances from 1,433 dialogues of the TV series Friends, with audio, video and text for each utterance §1 tab. 8. Each utterance has one of seven emotion labels and one of three sentiment labels §3.2. The paper's baselines are text-CNN, bcLSTM and DialogueRNN; the best reported 7-way emotion score is a weighted F-score of 60.25 (DialogueRNN, text+audio) and the best sentiment score is 67.56 tab. 11 tab. 13. The paper states almost no training hyperparameters; the released bcLSTM code fills some gaps, and DialogueRNN's settings are not in this repository.

Problem

• Emotion in a conversation depends on the context, the emotions of earlier turns and the speakers involved §1.
• Multimodal emotion corpora such as IEMOCAP and SEMAINE are dyadic (two speakers) and small. EmotionLines has multi-party dialogues but no audio or video. MOSI, MOSEI and MOUD are not conversational §1 §3.3.
• Short turns are hard to label from text alone. About 42% of MELD utterances are under five words §1.
• EmotionLines annotators saw only transcripts. 2,772 of its utterances were left as "non-neutral" because the annotators could not agree on an emotion §3.1.

Method

Dataset construction

1. Start from the 1,000 dialogues of EmotionLines, which were sampled from Friends episodes §2 tab. 1.
2. Find each utterance's start and end time by crawling the episode subtitles. Two constraints apply: timestamps increase within a dialogue, and every utterance belongs to the same episode and scene. Dialogues that break these rules span scenes or episodes and are filtered out §3 tab. 2.
3. Re-annotate each utterance with three annotators who watched the video clip as well as reading the transcript. The final label is the majority vote. Utterances where all three annotators disagree are dropped: 89 utterances in 11 dialogues §3 §3.1. The Fleiss' kappa is 0.43, against 0.34 for the original EmotionLines annotation §3.1.
4. Derive sentiment from emotion. Anger, disgust, fear and sadness map to negative; joy maps to positive; neutral maps to neutral. Surprise is labelled positive or negative by the annotators. The sentiment annotation reaches a kappa of 0.91 §3.2.
5. Align each transcript with its audio using the Gentle forced aligner, which gives word-level times. This is needed because many subtitle lines share one timestamp §3.1.
6. Cut the audio-visual clip for each utterance and save its audio as 16-bit PCM WAV §3. The paper states that a legal office checked that very short clips fall under fair use §3, footnote.
7. Dyadic MELD: extract every contiguous sub-dialogue that involves only two speakers §3.1. The paper's reported results use only the multi-party variant §3.1.

Dataset counts tab. 8:

Emotion and sentiment counts tab. 4:

Baseline models

The paper evaluates three baselines and names a fourth in the tables §4.2 tab. 11.

• text-CNN applies a CNN to each utterance with no context §4.2.
• bcLSTM models context with a bidirectional RNN over the utterances of a dialogue. It first builds unimodal context vectors for each modality, then a bimodal model combines them and classifies §4.2. It does not track speakers §4.4.
• DialogueRNN tracks a state for each speaker, updates a global context state, and applies attention over earlier utterances. It has three GRUs: a global one, a per-speaker one and an emotion one §4.2. The paper gives no equations or hyperparameters for it §4.2.
• cMKL appears only in the tables tab. 11. The paper does not describe it.

Input features §4.1: GloVe vectors (300-d) per token, fed to a 1D CNN that outputs 100-d text features. Audio uses openSMILE with 6373 features, reduced with L2-based feature selection (sparse estimators such as SVMs) to a dense vector. The baselines use no visual features §4.1.

The pseudocode below is the text branch of bcLSTM. The paper describes this model only in words §4.2. The layer sizes, activations and optimiser come from the released code only baseline.py.

# bcLSTM, text modality  [§4.2]
# Layer sizes and optimiser are taken from the code only.
# Input:  dialogues (B, U, T) token ids, T = 50 tokens per utterance
# Output: class probabilities per utterance (B, U, C)

def bclstm_text(dialogues, glove_vectors):
    # Frozen GloVe lookup, 300-d per token  [§4.1]
    # [code: baseline/baseline.py#L124]
    embedded = frozen_lookup(glove_vectors, dialogues)     # (B, U, T, 300)
    # For each utterance: convolutions of widths 3, 4 and 5, 512 filters
    # each, max-pooled over time  [code: baseline/baseline.py#L115-L130]
    for each utterance:
        pooled = concat(
            [max_over_time(conv(embedded, width)) for width in (3, 4, 5)]
        )
        # Dense layer to 100 units with tanh, then dropout 0.5
        # [code: baseline/baseline.py#L131] [code: baseline/baseline.py#L168]
        features = dropout(dense_tanh(pooled, units=100), rate=0.5)
    # Keep the 100-d utterance vectors of each dialogue in order
    context_inputs = stack_over_utterances(features)       # (B, U, 100)
    # Two stacked bidirectional LSTMs, 300 units per direction, ReLU,
    # dropout 0.3  [code: baseline/baseline.py#L176-L177]
    context = bilstm(
        context_inputs, units=300, activation="relu", dropout=0.3
    )
    context = bilstm(
        context, units=300, activation="relu", dropout=0.3
    )
    # Softmax over classes for each utterance  [§4.2]
    # [code: baseline/baseline.py#L178]
    probabilities = softmax_per_utterance(dense(context, units=C))
    return probabilities


# Loss: cross-entropy per utterance, masked so padded slots are ignored
# [code: baseline/baseline.py#L223] [code: baseline/data_helpers.py#L147]
# No class weights are applied in this code.
def masked_loss(probabilities, labels, mask):
    per_utterance = cross_entropy(probabilities, labels)   # (B, U)
    return weighted_mean(per_utterance, mask)

Hyperparameters

Most rows come from the released code, because the paper defers to a link for them §4.2. The "Where" column names the source of each value.

Results

Author-reported. Setup: multi-party MELD, with the test split of 2,610 utterances tab. 8. Metric: weighted F-score on the test set tab. 11 tab. 13. Baselines: text-CNN, cMKL, bcLSTM and DialogueRNN. Hardware and number of runs are not stated.

7-way emotion, test-set weighted F-score tab. 11:

5-way emotion (disgust and fear removed), test-set weighted F-score tab. 12:

Sentiment, test-set weighted F-score tab. 13:

Other reported results §4.3 §4.4:

• Multimodal DialogueRNN beats its text-only version by about 1.4 points for sentiment and about 3 points for emotion §4.3.
• Audio alone is much weaker than text alone: about 17 points for sentiment and about 15 for emotion (DialogueRNN) §4.3 tab. 11 tab. 13.
• Context helps: the gain over text-CNN is 1.4 to 2.5 points §4.4.
• Emotion-shift recall is 66% for detecting that a shift happens, and 36.7% when the correct new emotion must also be found §4.4.
• For 63% (882/1381) of correct test predictions from the global-GRU attention, the highest historical attention falls on utterances from a different speaker §4.4.
• Attention often reaches past utterances on both sides of the target, so the paper suggests bidirectional models §4.4 fig. 4.

Paper vs code

The paper's "Baseline Models" section points to a GitHub repository for hyperparameters §4.2. That link, github.com/senticnet/meld, redirects to github.com/declare-lab/MELD, which the project page also links README.md [affective-meld.github.io]. The checked-out commit is e8cedf27b5d2877e198332c957127e16eb214afe.

• DialogueRNN. The paper reports DialogueRNN results and points to the repository for their hyperparameters §4.2. The repository only contains the bcLSTM baseline baseline.py. Its README points to a different repository for updated baselines README.md. That repository was not checked for this card.
• Class weights. The paper says training used class weights §4.2 and calls them hyperparameters §4.3. The README lists values for the emotion baseline README.md. But baseline.py never passes a class-weight argument. The only per-sample weight it sets is the padding mask baseline.py. So the bcLSTM release does not apply the listed class weights.
• Text features. The paper says a 1D CNN gives 100-d text features §4.1. The code's CNN has three filter widths, 512 filters each, max pooling, and a 100-unit tanh layer baseline.py. These details are in the code only.
• Audio features. The paper describes openSMILE with 6373 features and L2 feature selection §4.1. The README gives the selected sizes of 1611 (emotion) and 1422 (sentiment) for the released feature files README.md. The feature extraction and selection code is not in this repository, so the selection settings cannot be checked.
• Bimodal model. The text and audio branches have two BiLSTM layers baseline.py. The bimodal branch has one baseline.py. The paper describes the bimodal model only as concatenation of the two feature vectors §4.1 §5.
• Rounded statistics. Average utterance length is 8.0, 7.9 and 8.2 in the paper tab. 8. The README gives 8.03, 7.99 and 8.28 README.md. The difference is rounding.

Pitfalls

• Dialogue counts do not match the filtering story. EmotionLines has 1,000 dialogues §2 and MELD has 1,433 tab. 1 tab. 8. The text says that dialogues were filtered out §3 and does not say how the total went up. Do not map dialogue IDs between the two datasets.
• Class weights are unclear. The paper says they were used, and the README lists values, but the released code applies none of them baseline.py. It is not known which setup produced the reported numbers.
• Missing training details. The paper gives no learning rate, no number of runs, no seeds and no variance. The code sets none of these except the optimiser, so the Keras defaults apply baseline.py.
• Model selection. The code keeps the checkpoint with the lowest validation loss and stops after 10 epochs without improvement baseline.py baseline.py. The paper does not say which checkpoint is reported.
• Padding. Padded utterance slots get a dummy label, and the loss mask gives them zero weight data_helpers.py baseline.py. A reimplementation that leaves out the mask will train differently.
• Truncation and freezing. Utterances are cut to 50 tokens, and the GloVe embedding is frozen in the text model data_helpers.py baseline.py. The paper says neither.
• Metric. The "w-avg." column is scikit-learn's weighted F-score over the classes, computed on utterances with mask 1 baseline.py. Reproduce it with the same averaging.
• Text-CNN baseline. The bcLSTM text branch has a per-utterance CNN baseline.py. The paper does not say whether the text-CNN baseline uses the same network, and the standalone text-CNN code is not in this repository.
• cMKL. It appears in the tables tab. 11 and is not described anywhere in the paper text. It cannot be implemented from this card.
• Low agreement. The emotion kappa is 0.43 §3.1, so the labels are noisy. Sentiment agreement is higher (0.91) §3.2. Surprise has its own sentiment annotation step §3.2.
• Rare emotions. Disgust, fear and sadness score poorly. The paper blames class imbalance §4.3 tab. 4.
• Emotion shift. The paper uses "emotion shift" for a change between a speaker's successive utterances §3.2. It gives no formal definition, and the recall numbers in §4.4 depend on it.
• Licensing of the data. The paper's license is CC BY-SA 4.0 [raw metadata]. That covers the paper text only. The paper states that the video clips come from a TV series and cites a fair-use check §3, footnote. The dataset page does not state a licence [affective-meld.github.io]. The repository code is GPL-3.0. Do not assume the clips may be redistributed.
• Results are author-reported. They come from the authors' own runs and have not been reproduced here.

Lineage

• Builds on: EmotionLines (Chen et al., arXiv:1802.08379, https://arxiv.org/abs/1802.08379), the source of the dialogues and the original labels §2.
• Compared against: IEMOCAP, SEMAINE, MOSI, MOSEI and MOUD §1 §3.3. These are named in the paper and not summarised here.
• Baselines: bcLSTM and DialogueRNN are cited in §4.2. text-CNN and cMKL appear in tab. 11.