Emotion recognition in conversation

Overview

Emotion recognition in conversation (ERC) assigns an emotion label to each utterance in a dialogue. Most of the page covers the text-only setting. One variant, HFusion, also uses audio and video. A single line such as "okay" cannot be labelled reliably alone; the label depends on the lines around it and on who said them 1908.11540 §1. A 2019 survey gives the same framing: the label depends on the utterance, its speaker and the preceding conversation 1905.02947 §1. The survey lists the hard cases as emotion shift, sarcasm and talk with more than two people. It also says no public dataset labels why an emotion occurs, which it calls emotion reasoning 1905.02947 §2.

Earlier models ran a recurrent network over the utterances. The DialogueGCN authors argue that recurrent networks carry long-range context poorly, and that attention pooling over a recurrent state ignores which speaker wrote each context utterance and where it sits relative to the target [arxiv:1908.11540 §1]. Their working assumptions: the emotion of a speaker depends on the other party's utterances (inter-speaker influence) and on that speaker's own earlier utterances (self-dependency), and future utterances also carry signal 1908.11540 §1, §3. The DialogueRNN authors make the same point about speaker roles in dyadic talk, and model each party's state separately 1811.00405 §1.

The COSMIC authors make a different case. Context-independent embeddings, and recurrent or graph models over them, miss contextual cues and the difference between related emotion classes. Their proposed missing signal is commonsense knowledge about mental states and cause and effect [arxiv:2010.02795 §1, §2, §3.3].

A 2020 position paper on sentiment analysis lists conversational context as an open problem. It names two dependencies in a conversation: a speaker's dependence on their own earlier emotion, and dependence on the other participants 2005.00357 §3.3.2. It also says LSTM-based sequence models have been ineffective at modelling sentiment flow 2005.00357 §3.3.2. The paper runs no experiments, so these are hypotheses here, not results.

Four papers on wikillm do per-utterance ERC. The first three put a sequential model over the utterances, then a per-utterance classifier, and all take the speaker of each utterance as input. They differ in how context moves between utterances. HFusion also runs a sequential model over the utterances, but its input is text, audio and video, and its card does not state whether it uses speakers 1806.06228 §3.3. DialogueRNN keeps one recurrent state per speaker and a global state, and attends over earlier global states 1811.00405 §3.3. DialogueGCN adds a graph over a window of neighbouring utterances with typed edges 1908.11540 §3.3. COSMIC keeps no graph and updates per-speaker and per-listener recurrent states, with commonsense vectors as extra input 2010.02795 §3.4. The three text models report IEMOCAP results on different input features, so their numbers are not directly comparable. HFusion's IEMOCAP scores also use audio and video (see Contradictions).

The benchmark used most here is MELD: about 13,000 utterances from the TV series Friends, with text, audio and video, and seven emotion labels 1810.02508 §1 1810.02508 tab. 8. DialogueGCN and COSMIC both report results on it 1908.11540 tab. 4 2010.02795 tab. 4. The MELD paper says the video clips come from a TV series and cites a fair-use check for short clips. Do not assume the clips can be redistributed 1810.02508 §3.

Separately, CEM is an empathetic reply generator. Its emotion classifier outputs one label for the dialogue context, from a 32-label set, not a label for each utterance 2109.05739 §3.3 2109.05739 tab. 2. It is covered below as a variant with that limit stated, and its numbers are not comparable to the three above.

Variants

Per-speaker recurrent states with attention over history (DialogueRNN)

The model keeps a state for each participant and one global state for the whole conversation 1811.00405 §3.3. When an utterance arrives, attention over the earlier global states gives a context vector 1811.00405 eq. 2–4. Only the speaker's own state is updated from that context [arxiv:1811.00405 eq. 5]. An emotion state then follows the speaker's updated state, and a ReLU layer and a softmax give the label 1811.00405 eq. 8–11. The listener states in the paper are left unchanged by default. The authors found an active listener update slightly worse 1811.00405 §3.3.4 [arxiv:1811.00405 §5.2.1].

A bidirectional version runs two copies forwards and backwards and concatenates their emotion states. Its attention over all emotion states of a dialogue reads later turns 1811.00405 §3.4.2–§3.4.4. The plain model reads only earlier turns, by its design in §3.3. The paper does not test that streaming setting itself.

Reported results, IEMOCAP text, weighted F1 1811.00405 tab. 2:

• DialogueRNN scores 59.89, against 56.13 for CMN, the prior model in its table. The authors put the gain at 3.76 points 1811.00405 §5.1.1.
• The bidirectional model with emotion attention, BiDialogueRNN+Att, scores 62.75 1811.00405 tab. 2. It reads the whole dialogue.
• Without the party state the score is 55.56, and without the emotion GRU 57.38 1811.00405 tab. 4.
• On the four AVEC attributes, BiDialogueRNN+Att has the lowest mean absolute error of the methods in the table 1811.00405 tab. 2.

The paper's only multi-party claim is that the method should scale. That is future work, and only dyadic datasets are tested 1811.00405 §6. Card: DialogueRNN.

Windowed relational graph over a sequential encoder (DialogueGCN)

Each utterance is embedded by a CNN over pretrained word vectors, trained separately at the utterance level 1908.11540 §3.2. A bidirectional GRU then runs over the utterance sequence 1908.11540 §3.3.1. Each utterance becomes a node, linked to the 10 utterances before it and the 10 after, plus itself 1908.11540 §3.3.2. Edge weights come from a softmax over a bilinear score between the two nodes, normalised over each node's incoming edges 1908.11540 eq. 1. Each edge also has a relation type: the speaker of the source and the speaker of the target, combined with whether the source comes before the target. For two speakers that is 8 types, and in general 2M² for M speakers 1908.11540 §3.3.2 / tab. 1.

Two graph layers follow. The first is relation-specific, in the style of R-GCN 1908.11540 eq. 2. The second drops the relation types and uses an unweighted sum over the window 1908.11540 eq. 3. A classifier concatenates the sequential and graph features, applies attention over the whole dialogue, and outputs a label per utterance 1908.11540 eq. 4–9.

Reported results, text only, mean of 5 runs 1908.11540 §5.1:

• IEMOCAP, weighted F1 64.18, against 62.75 for DialogueRNN, the prior best on that table 1908.11540 tab. 3. The DialogueRNN value is discussed in Contradictions.
• MELD, F1 58.10, against 57.03 for DialogueRNN 1908.11540 tab. 4.
• AVEC, mean absolute error on valence, arousal, expectancy and power is the lowest of the compared methods on all four 1908.11540 tab. 4.

The authors also report that the window size, the speaker-pair relations and the sequential encoder each contribute. Measured on IEMOCAP:

Card: DialogueGCN.

Speaker-state recurrent model with commonsense features (COSMIC)

Each utterance is first turned into a fixed vector by RoBERTa-Large, fine-tuned on the utterance emotion labels. The vector is the mean of the [CLS] states from the last four layers, 1024-d 2010.02795 §3.2. The encoder has 355M parameters 2010.02795 §3.2, so this stage needs a fine-tuning run.

Each utterance also gets five commonsense vectors from COMET, a model trained on the ATOMIC knowledge base. The COMET encoder is run on the utterance followed by one relation phrase, and each vector is 768-d. The five relations are the speaker's intent, the effect on the speaker, the reaction of the speaker, the effect on the others and the reaction of the others 2010.02795 §3.3 2010.02795 tab. 1.

A set of GRU cells then runs over the conversation. One shared context state and one attention vector cover the whole dialogue. Each participant has an internal state, an external state and an intent state. The emotion state of the current utterance is built from the utterance, the speaker's states and the previous emotion state, and a softmax layer gives the label 2010.02795 §3.4 2010.02795 tab. 2. The paper's diagram uses bidirectional cells 2010.02795 fig. 2.

Reported results, text only, weighted F1 on IEMOCAP 2010.02795 tab. 4:

• 65.28, against 64.76 for RoBERTa DialogueRNN. The authors say the gain is smaller on IEMOCAP than on DailyDialog 2010.02795 §5.2.
• Removing the COMET features from both speaker and listener sides gives 63.05. Removing only the speaker side gives 63.27, and only the listener side 65.05 2010.02795 tab. 4 2010.02795 §5.3.
• The authors call the commonsense gain "not very substantial", and report no significant gain from four other COMET relations 2010.02795 §5.5.
• A case study on one IEMOCAP conversation shows the features helping with sudden shifts and with angry versus frustrated 2010.02795 fig. 3 2010.02795 §5.4. This is a single conversation.

Card: COSMIC.

Hierarchical fusion of text, audio and video with context (HFusion)

This variant adds audio and video. The authors argue that concatenating the unimodal vectors cannot filter conflicting or redundant information between modalities 1806.06228 §3.1. Each utterance is encoded per modality: text by a CNN over word2vec vectors, audio by openSMILE features and video by a 3D CNN 1806.06228 §3.2. A GRU per modality then runs over the utterances of a clip 1806.06228 §3.3. Each modality is mapped to a common width. Each pair of modalities is fused per dimension with learned weights and tanh, and each pair sequence gets its own GRU [arxiv:1806.06228 §3.4] 1806.06228 eq. 1–3. The three pair vectors are fused the same way, a last GRU runs over the result, and a softmax gives the label 1806.06228 §3.4 1806.06228 §3.5.

Reported results on IEMOCAP with context (CHFusion), four classes 1806.06228 tab. 3:

• Text, audio and video together give 76.5 accuracy and 76.8 F-score. The two baseline columns read 74.1 and 73.6. The card does not say which metric those baselines report.
• Per class, on the same setup, neutral has the lowest F-score, 71.2, against 77.0 to 81.4 for the other three classes 1806.06228 tab. 4.
• The context gain is shown on CMU-MOSI, a binary sentiment set, not on IEMOCAP. There the context model is 1 to 2 points above the same model without context, with no table 1806.06228 §4.4.3. The card gives no IEMOCAP result without context.

The public code (MIT) trains the pair-level models first and then the trimodal model on their saved activations, although the paper describes one end-to-end method 1806.06228 §2 hfusion.py. It does not include the unimodal encoders hfusion.py. Card: Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling.

Dialogue-level emotion head inside a reply generator (CEM)

CEM generates the next reply in an empathetic chat. Its emotion head is a side output, not a labeller of each utterance. Five commonsense relations are generated by COMET for the last user turn: xReact, xWant, xNeed, xIntent and xEffect 2109.05739 §3.2. The xReact vector goes through an affective encoder, and the other four through a cognitive encoder [arxiv:2109.05739 eq. 2-3]. The affective stream, with the xReact vector added to each context token, gives the single emotion label through a softmax 2109.05739 eq. 8-12. The emotion label set has 32 classes 2109.05739 §2.3.

The fused context is read by the decoder through cross-attention 2109.05739 eq. 15-16. Training sums three terms: the reply NLL, the emotion cross-entropy, and a frequency-weighted diversity loss that gives frequent tokens less weight, with weights 1, 1 and 1.5 2109.05739 §3.6 2109.05739 eq. 21. The diversity loss targets generic replies. The authors found some trigrams in at least half of the training replies 2109.05739 tab. 1.

Reported results on EmpatheticDialogues, 8:1:1 split, author-reported 2109.05739 tab. 2:

• Emotion accuracy 39.11, the highest in the table. The four baselines that report it range from 30.41 to 33.57, and the ablations from 33.76 to 38.82.
• Perplexity 36.11, against 36.93 for the best baseline. The ablation without the diversity loss is lower, at 35.60, although the text calls the full model the lowest 2109.05739 §4.3.
• Dist-1 0.66 and Dist-2 2.99, against best baselines of 0.47 and 2.10. The paper prints these without a unit.
• In crowd pairwise tests on 100 response pairs per comparison, three raters per pair, empathy preference was 52.0% for CEM and 38.0% for MoEL. Preference for CEM over MIME and EmpDG was 50.3% and 54.3% 2109.05739 tab. 3. The sign test marks the MoEL and MIME results at p < 0.05 [arxiv:2109.05739 tab. 3].

These numbers measure reply quality and one emotion head on a different label set. They say nothing about per-utterance ERC. The COMET step runs on each new last turn, so a new dialogue needs a COMET model at inference. The released code produces these inferences during data preparation [code: src/utils/data/loader.py#L66-L93]. Card: CEM.

When to use which

Implementation sketch

Common core of the windowed graph variant, language-agnostic. The sequential encoder, the window and the relation ids are the parts a variant would change. The COSMIC variant keeps the sequential step but replaces the graph with per-speaker recurrent states and adds COMET vectors as input; its per-utterance update is in the card 2010.02795 §3.4. DialogueRNN also uses per-speaker states, without a graph; its recurrence is in the card 1811.00405 §3.3.

# Input: utterance_vecs (N, D), speaker_of (N,), window p = f = 10
# 1. Speaker-agnostic sequential context  [arxiv:1908.11540 §3.3.1]
g = bigru(utterance_vecs)                      # (N, 2 * D_e)

# 2. Windowed directed edges, including the self-loop  [§3.3.2]
edges = [(i, j) for i in range(N)
         for j in range(max(0, i - p), min(N, i + f + 1))]

# 3. Relation id: speaker pair x (source before target?)  [tab. 1]
def relation(i, j):
    return rel_index(speaker_of[i], speaker_of[j], i < j)

# 4. Edge weights: softmax of bilinear scores over each window  [eq. 1]
alpha = softmax_over_window(g, edges, W_e)

# 5. First layer: relation-specific, weighted  [eq. 2]
h1[i] = relu(alpha[i, i] * W0 @ g[i] +
             sum(alpha[i, j] / c[i, r] * W_r[r] @ g[j]
                 for (i2, j) in edges if i2 == i
                 for r in [relation(i, j)]))

# 6. Second layer: unweighted, relation-free  [eq. 3]
h2[i] = relu(W0_2 @ h1[i] + sum(W2 @ h1[j] for (i2, j) in edges
                                if i2 == i and j != i))

# 7. Classifier on [g_i, h2_i] with attention over the dialogue  [eq. 4-8]
labels = classify(attend_over_dialogue([g, h2]))

Where the public code differs from the paper (card's Paper vs code section), as stated in model.py and model.py:

• The edge score is a learned vector per absolute position, not the bilinear form of eq. 1.
• The graph layers have no activation, although the paper writes σ in both.
• The sequential encoder is a 2-layer LSTM, not a GRU, in the default script and the README command.
• The windowed edges are built in Python loops per batch, which is slow for large batches model.py.

Contradictions

• Reported versus reproduced IEMOCAP F1 (DialogueGCN). The paper reports 64.18 1908.11540 tab. 3. The README reports 64.67 and 64.44 on two machines, with a different configuration: LSTM base, learning rate 3e-4, dropout 0.4, L2 set to zero, nodal attention and class weights on (the README of the official repository). The README attributes the spread to nondeterministic CUDA atomics. Neither side is resolved here.
• Model selection (DialogueGCN). The paper describes a validation split of 10% and a mean of 5 runs 1908.11540 tab. 2, §5.1. The public script uses no validation split and prints the maximum test F1 over all epochs train_IEMOCAP.py. The reported IEMOCAP numbers may therefore be optimistic.
• Model selection (DialogueRNN). The paper does not say which epoch it reports. The IEMOCAP script evaluates on the test split every epoch and keeps the epoch with the lowest test loss train_IEMOCAP.py. The script sets the validation split to zero, so the reported numbers may be optimistic, as with DialogueGCN.
• Loss weighting (DialogueRNN). The paper's loss is unweighted cross-entropy with an explicit L2 term 1811.00405 §3.3.7. The IEMOCAP script applies hard-coded per-class weights train_IEMOCAP.py, and the paper does not explain them. Accuracy and F1 will differ between the two.
• Model selection (COSMIC). The table caption says the scores are taken at the best validation score 2010.02795 tab. 4. The IEMOCAP script reports two test scores, one at the best validation loss and one at the best validation weighted F1 train_iemocap.py. The paper does not say which one it reports.
• Run count (COSMIC). The same table caption says "five runs" and then "10 runs" for two rows 2010.02795 tab. 4. The paper does not resolve this.
• DialogueRNN numbers across papers. The DialogueGCN table gives DialogueRNN 62.75 on IEMOCAP 1908.11540 tab. 3. The DialogueRNN card lists 62.75 for its bidirectional attention model and 59.89 for the plain model 1811.00405 tab. 2. The COSMIC table gives DialogueRNN 62.57, a GloVe-based row 2010.02795 tab. 4. The cards cannot say which DialogueRNN variant each table reports. The two papers also use different utterance features (a GloVe CNN in DialogueGCN, RoBERTa-Large in COSMIC), so the top scores, 64.18 and 65.28, are not directly comparable. The MELD DialogueRNN text-only value, 57.03, is the same in the DialogueGCN and COSMIC tables 1908.11540 tab. 4 2010.02795 tab. 4 and matches the MELD paper's text-only row 1810.02508 tab. 11. The MELD paper's best 7-way score, 60.25, uses text and audio, so it is not comparable with the text-only numbers here 1810.02508 tab. 11.
• Hyperparameter values (DialogueGCN). The paper does not state the learning rate, dropout or L2 weight. The script defaults (1e-4, 0.5, 1e-5) and the README values (3e-4, 0.4, 0) disagree train_IEMOCAP.py.
• Which COSMIC setting gives Table 4. The paper describes listener updates in its equations 2010.02795 §3.4, but the code turns them off by default train_iemocap.py. The paper does not say which setting produced Table 4.
• Label count (DialogueGCN). The problem definition lists 8 emotion labels 1908.11540 §3.1. IEMOCAP uses 6 and MELD 7 1908.11540 §4.1 1810.02508 §3.2. HFusion uses four IEMOCAP classes, happy, sad, neutral and anger 1806.06228 §4.4.2 1806.06228 tab. 4. Set the class count per dataset.
• Multi-party claim (DialogueRNN). The survey says DialogueRNN "can handle multi-party conversations at scale", unlike CMN and ICON [arxiv:1905.02947 §4]. The DialogueRNN paper says scaling to more than two parties is future work and that only dyadic datasets were tested 1811.00405 §6. The survey gives no test of its own, so the paper's statement is the one this page relies on.
• Survey numbers are not independent. The survey's DialogueRNN rows repeat the original paper's values: 62.75 weighted F1 on IEMOCAP [arxiv:1905.02947 tab. 3] and 57.03 on MELD 1905.02947 tab. 5. Three of the survey's four authors also wrote DialogueRNN, according to the two papers' author lists. Cite the original card for these numbers, not the survey.
• Metric and label set (CEM against the ERC papers). CEM's 39.11 is accuracy over 32 EmpatheticDialogues labels, one per dialogue 2109.05739 tab. 2 2109.05739 §2.3. The ERC papers report weighted F1 over IEMOCAP and MELD labels 1908.11540 tab. 3 2010.02795 tab. 4 1811.00405 tab. 2. The three figures do not measure the same task.
• Metric and input (HFusion against the text models). HFusion's IEMOCAP scores are accuracy and F-score, from text, audio and video [arxiv:1806.06228 tab. 3]. The text models report weighted F1 from text alone. HFusion's 76.5 accuracy is therefore not comparable with their 62.75 to 65.28.
• IEMOCAP counts (HFusion). §4.1.2 gives 1083 angry, 1630 happy, 1083 sad and 1683 neutral "videos". The IEMOCAP rows of tab. 1 sum to different counts 1806.06228 §4.1.2 1806.06228 tab. 1. The paper does not resolve this.
• Sequence models for sentiment flow. The position paper says LSTM-based sequence models have been ineffective at modelling sentiment flow 2005.00357 §3.3.2. The three text models and HFusion all run gated recurrent encoders, and each reports gains over earlier work on its own benchmark 1811.00405 tab. 2 1908.11540 tab. 3 2010.02795 tab. 4. The position paper gives no experiment, so the cards cannot settle the disagreement.
• Concatenation in multimodal fusion. The position paper says simple concatenation performs on par with most fusion methods, from the authors' experience and without numbers 2005.00357 §3.2.1. HFusion argues that concatenating unimodal vectors cannot filter conflicting or redundant information between modalities 1806.06228 §3.1. Neither side gives a controlled test here.
• MELD size and annotation. The position paper calls MELD too small, with weak inter-annotator agreement 2005.00357 §3.2.2. The MELD card is the source for the dataset's own size and labels 1810.02508 §1. The position paper gives no agreement figure, so this claim is unquantified.
• Ablation rounding (DialogueGCN). The text gives 36.7 for the no-encoder ablation, and the table gives 36.75 1908.11540 §5.3 / tab. 5. This is a rounding difference inside the same paper.

Papers

• Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances: a 2019 survey of ERC challenges, six public datasets and recent models. It proposes no method, and its DialogueRNN numbers repeat the original paper.
• DialogueRNN: An Attentive RNN for Emotion Detection in Conversations: per-speaker GRU states, a global state and an emotion GRU, with attention over earlier global states; bidirectional and attention variants for the whole dialogue.
• DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation: a windowed relational graph over a BiGRU encoder, with speaker-pair and before/after edge types, for text-only ERC.
• COSMIC: COmmonSense knowledge for eMotion Identification in Conversations: RoBERTa-Large utterance features plus five COMET commonsense vectors, tracked by speaker, listener, intent and emotion GRU states.
• Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling: text, audio and video fused in two stages per utterance clip, with a GRU for context at each stage; the one multimodal variant here.
• CEM: Commonsense-aware Empathetic Response Generation: COMET commonsense on the last user turn, fused into a Transformer reply generator, with a dialogue-level emotion head.
• Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research: a 2020 position paper with no experiments; its conversational-context and multimodal sections are cited above as open problems.
• MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations: about 13,000 utterances from a TV series with text, audio and video, labelled with seven emotions and three sentiments.