Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances

In plain words

Emotion recognition in conversation means reading a dialogue and tagging each message with the emotion the speaker is feeling, such as anger or happiness. It is harder than tagging single sentences. A short reply like "yeah" can be sarcastic, bored or happy depending on what was said just before, and people react differently from one another. Emotions also tend to stay the same for a while and then suddenly change.

This survey explains those difficulties, lists the public datasets people use for the task, and compares the main neural models published up to 2019. It also points to open problems. One is explaining why an emotion happens, which no current dataset labels. Another is handling conversations with more than two people.

The survey does not propose a new model. Most of its numbers come from earlier papers, mostly from one model, DialogueRNN, which three of the four survey authors co-wrote. Use it as a map of the field and a reading list, not as a source of independent benchmark results.

TL;DR

• A survey, not a method paper. It frames ERC as labelling each utterance with an emotion, using the utterance, its speaker and the preceding conversation §1, task definition.
• Challenges covered: emotion taxonomy, annotation basis, context modelling, speaker and listener modelling, emotion shift (with sarcasm and fine-grained cases), multiparty conversations and emotion reasoning §2.
• Public datasets: IEMOCAP, SEMAINE, EmotionLines, MELD, DailyDialog and EmoContext, with label and split statistics in tab. 1 and tab. 2 §3.
• The IEMOCAP comparison is in tab. 3. DialogueRNN reports 63.40 weighted accuracy and 62.75 weighted F1 there §4.
• Dialogue emotion-shift is the weak spot: the paper reports DialogueRNN at 47.5% on shifted utterances versus 69.2% without a shift §4.

Problem

Vanilla sentence-level emotion recognition fails on conversations because it ignores context, the order of turns and who is speaking §1, introduction. The survey argues that a context model has to handle two kinds of influence: the speaker's own emotional inertia (emotions persist) and the emotional influence of the other speaker, who may mirror or provoke the speaker [§2, conversational context modeling]. The paper also argues that a good model should use other factors, such as topic, personality, intent, viewpoint and argument logic §1, controlling variables.

The paper states no quantitative assumptions. Its claims about real-time use and multi-speaker scaling are qualitative §4, recent advances.

Method

The survey has no method of its own. It contributes three things: a task definition §1, task definition, a taxonomy of challenges §2, and a summary of existing datasets and models §3 §4.

Task definition §1, task definition

Input: a conversation of N utterances. Utterance i is a sequence of T words and is spoken by party p_i. Output: one emotion label e_i per utterance. The paper uses the same notation for all datasets, and it does not give a loss or a training procedure.

# ERC task interface  [§1, task definition]
# Input:  utterances [u_1 .. u_N], each a list of words
# Input:  speakers [p_1 .. p_N], one party label per utterance
# Output: emotions [e_1 .. e_N], one label per utterance

def recognize_emotions(utterances, speakers):
    # Each label may depend on the whole preceding history and on
    # who spoke each turn, not only on the utterance itself  [§2]
    ...

Factors the survey says drive an utterance's emotion §4

The paper says the emotion of a target utterance depends on three factors. The paper does not turn these into equations.

1. The utterance itself, plus its context: the preceding utterances, the intent and the topic of the conversation.
2. The speaker's state, which includes personality and argumentation logic.
3. The emotions expressed in the preceding utterances.

Its own pictorial model of these links (a dyadic dialogue with hidden and observed variables) is fig. 1. The paper uses it to motivate the factors. It does not fit or test it.

Models summarised in §4

These descriptions restate the survey's text. Architectural details are in the original papers, which the survey does not give.

• CMN (conversational memory network) §4: for dyadic dialogue. It keeps a separate memory per speaker, which models speaker-specific context. Neither CMN nor ICON uses the speaker information of the target utterance §4.
• ICON (interactive conversational memory network) §4: links the speaker memories, so the model captures self-influence and inter-speaker influence.
• IANN (interaction-aware attention network) §4: models inter-speaker relations and keeps distinct memories per speaker.
• DialogueRNN §4 fig. 8: uses the target speaker's identity. It models self and inter-speaker emotional influence in several RNN stages, with attention over the history. The survey says it can handle multi-party conversations at scale, unlike CMN and ICON. Its attention scores give more weight to nearby utterances fig. 8, and it uses past and future utterances about equally often fig. 9.
• bc-LSTM §4, EmoContext paragraph: an LSTM over the utterance sequence. The survey says most EmoContext work builds its context modelling on it.
• HRLCE §4 fig. 11: an utterance encoder that combines ELMo, GloVe and DeepMoji features, followed by a context encoder. The context encoder is a bc-LSTM followed by a multi-head attention layer. The survey applies it only to EmoContext.

Challenges (the paper's taxonomy) §2

• Emotion categories §2, categorization: categorical models (Plutchik's eight primary types fig. 4, Ekman's six basic emotions) or dimensional models (valence and arousal). Complex taxonomies are hard for annotators to tell apart, and simple ones cannot express complex emotions.
• Annotation basis §2, basis of annotation: self-annotation during a conversation would disturb it, and post-conversation self-annotation had not been done. Most datasets use third-party annotators who see the context.
• Context modelling §2, context modeling: earlier turns help, especially for short replies. Figures 5a and 5b show the same word "yeah" with negative and positive emotion depending on the context.
• Speaker and listener modelling §2, speaker-specific; listener-specific: speakers differ in how they express emotion, for example sarcasm. Listener modelling needs visual input and is useful mainly for continuous recognition (for example audience reaction) §2.
• Emotion shift §2, presence of emotion shift: emotions persist unless a stimulus changes them fig. 6 (Joey changes from neutral to anger).
• Fine-grained, multiparty, sarcasm §2: topic-level opinions fig. 7, more than two speakers, and sarcasm that only context reveals.
• Emotion reasoning §2, emotion reasoning: finding why an emotion occurs and what function the context utterances play. No dataset labels this.

Hyperparameters

Not applicable. The survey runs no experiments and gives no training settings. Settings for the models in §4 (for example DialogueRNN's) are not stated in this paper. Look them up in the cited papers, not here.

Results

Author-reported. The paper collects results from earlier work in tab. 3 to tab. 6. The ICON row in tab. 3 was re-run by the survey authors and is not the original paper's number tab. 3 note. The other table notes point to the DialogueRNN paper for details.

Datasets tab. 1 tab. 2 §3

Label counts from tab. 1:

Split sizes from tab. 2. The table covers five datasets; it omits EmoContext, which is in tab. 1. For IEMOCAP and SEMAINE the table has one merged cell for train and val, so the count is shared by the two:

IEMOCAP, DialogueRNN and baselines tab. 3

Accuracy (Acc.) and F1 per emotion, and weighted average. Bold marks the best per column, as in the paper. Note: ICON is re-run by the survey authors.

The text says DialogueRNN beats CMN and ICON on IEMOCAP §4. The weighted-average F1 gap is 62.75 against 58.54 for ICON, a 4.21 point margin.

SEMAINE (regression) tab. 4

Each cell is MAE (lower is better) and Pearson r (higher is better). The caption mentions accuracy, but the table has no accuracy column. Bold marks the best per column, as in the paper.

MELD F-score, emotion classification tab. 5

F-score per emotion, with weighted average (w-avg). The text says DialogueRNN improves little over bc-LSTM on MELD §4.

EmoContext tab. 6 §4

Single-row comparison of the two frameworks' F1 on EmoContext. The survey reports this as its own experiment ("in our experiments") §4.

Emotion shift §4

The text gives two accuracy figures for DialogueRNN, without naming the dataset: 47.5% on utterances whose emotion differs from the previous one, and 69.2% on utterances with no emotional shift. The paper also says CMN, ICON, IANN and DialogueRNN all perform poorly on shifted utterances §4.

Context usage §4 fig. 9

DialogueRNN's attention scores put most weight on nearby utterances. The histogram of the distance Δt between target and context utterance fig. 9 shows past and future utterances used about equally often. CMN and ICON use no future utterances. The paper says real-time systems cannot rely on future utterances, and that CMN, ICON and DialogueRNN with a fixed context window would suit them §4.

Paper vs code

No official code. This is a literature survey and implements nothing. The paper gives no repository link, and its own figures and tables are not software. The repo search was skipped for this card under the wiki's survey rule, so no repository was checked.

Pitfalls

• Shared authorship with the main source. The bibliography cites DialogueRNN as majumder2019dialoguernn (arXiv:1811.00405), with N. Majumder, S. Poria, D. Hazarika, R. Mihalcea, A. Gelbukh and E. Cambria as authors [bibliography]. Three of this survey's four authors are on it. Most numbers in tab. 3 to tab. 5 are DialogueRNN's comparisons, so check the original papers before relying on the ranking.
• Re-run rows. The ICON row in tab. 3 was re-run by the survey authors and differs from the original paper. The table note says the contextual feature extraction and pre-processing were disregarded in that run.
• Mixed setups across tables. The tables come from different papers and datasets. Treat them as separate evaluations and do not rank methods across datasets.
• Dataset count. §3 says "these five datasets" but names six (IEMOCAP, SEMAINE, EmotionLines, MELD, DailyDialog, EmoContext). tab. 2 leaves out EmoContext.
• Caption mismatch. tab. 4 says "Acc. = Accuracy", but the table only reports MAE and r.
• Dataset for the shift figures. The 47.5% vs 69.2% figures do not name a dataset §4.
• Unstated values. The paper gives no network sizes, learning rates, number of attention heads, context window lengths, or training procedure for any model. Implementers must get these from the cited originals.
• SEMAINE is regression. Its targets are continuous valence, arousal, expectancy and power §3. Its metrics cannot be compared with the classification F1 on other datasets.
• Figures from other papers. Figures 8 and 9 are marked in their captions as taken from DialogueRNN. Do not reuse them without that credit.

Lineage

• Context modelling with bc-LSTM §4, EmoContext paragraph. Earlier ERC work cited in the introduction includes EmotionLines and Zhou et al. §1.
• Compared against and discussed: CMN, ICON, IANN, DialogueRNN and HRLCE §4. DialogueRNN has a card: /papers/1811.00405.
• Superseded by: not stated in this paper.