DialogueRNN: An Attentive RNN for Emotion Detection in Conversations

In plain words

When people talk, how each person feels depends on what was said before and on who is speaking. Most emotion classifiers look at one sentence at a time, or at the conversation as one stream, and do not keep track of each speaker separately. This paper proposes a model that does.

The model reads a conversation one utterance at a time. It keeps a running memory for every participant, plus a shared memory of the whole conversation so far. When a new sentence arrives, the model looks back over the shared memory to find the earlier moments that matter most for it, and it updates only the memory of the person who just spoke. A final memory tracks the emotion of the current sentence, and a small classifier turns that into one emotion label.

A second version reads the conversation in both directions and lets each utterance attend to all the others. This gives it information from later turns as well as earlier ones, which helps classification but is not possible when the future is unknown. The authors report that this version does best on two datasets, one with six emotion labels and one with four continuous ratings.

TL;DR

DialogueRNN classifies the emotion of each utterance with three coupled GRUs: a global GRU for context, a party GRU per speaker, and an emotion GRU for the utterance. Attention over earlier global states supplies context. The bidirectional variant adds attention over all emotion states of the dialogue. Author-reported results beat the earlier CMN model on IEMOCAP (text) and AVEC, and on IEMOCAP with added audio and visual features. The shipped AVEC script does not match the paper's best AVEC model (see Paper vs code).

Problem

• Emotion labels are needed per utterance in conversations, for uses such as opinion mining over chat or support dialogue §1.
• The prior state of the art (CMN) does not model each party in a meaningful way and does not know who spoke each utterance §1 §2.
• The method assumes three factors drive an utterance's emotion: the speaker, the preceding utterances (context), and the emotions of the preceding utterances §3.3.
• In dyadic conversations the two parties have distinct roles, so the speaker's own earlier turns and the other party's turns both matter §1.
• The model is evaluated on dyadic data only. Scaling to more than two parties is named as future work and is not tested §6.

Method

Problem setting §3.1. There are M parties (M = 2 in the datasets). Each utterance u_t has one speaker s(u_t) and a label from six emotion classes. The input u_t is a fixed utterance vector from unimodal feature extractors §3.1 §3.2.

Unimodal features §3.2. Text: a CNN with filter sizes 3, 4 and 5, 50 maps each, max-pooling and ReLU, then a 100-dimensional dense layer trained on the utterance labels §3.2.1. Audio: openSMILE. Visual: a 3D-CNN. Both are stated to be identical to the CMN paper's extraction §3.2.2.

Three coupled states §3.3. Each party i keeps a party state q_i, initialised as null vectors. A global state g_t is shared by all parties. An emotion state e_t represents the current utterance §3.3.2 §3.3.1.

1. Global GRU §3.3.1 eq. 1. The global state moves with the utterance and the speaker's previous party state: g_t = GRU_G(g_{t-1}, [u_t ; q_{s(u_t), t-1}]).
2. Context attention §3.3.3 eq. 2 eq. 3 eq. 4. Scores over earlier global states g_1..g_{t-1} come from a bilinear match between u_t and each g_i through a learned matrix W_alpha (softmax, eq. 3). The context c_t is the score-weighted sum of those states (eq. 4). The paper does not say what c_1 is; the code uses zeros model.py.
3. Speaker (party) GRU §3.3.3 eq. 5. Only the speaker's party state is updated: q_{s(u_t), t} = GRU_P(q_{s(u_t), t-1}, [u_t ; c_t]).
4. Listener state §3.3.4 eq. 6 eq. 7. By default listener states are left unchanged (eq. 6). A variant updates each listener with a GRU over that party's visual cues and the context c_t (eq. 7). The authors found the simpler default sufficient and the variant slightly worse §3.3.4 §5.2.1.
5. Emotion GRU §3.3.5 eq. 8. The emotion state moves with the speaker's updated party state: e_t = GRU_E(e_{t-1}, q_{s(u_t), t}).
6. Classifier §3.3.6 eq. 9 eq. 10 eq. 11. A ReLU layer of width D_l, then a softmax over c = 6 classes, then argmax.

Training §3.3.7 eq. 12. Categorical cross-entropy summed over utterances, normalised by the number of utterances, plus an L2 term λ‖θ‖₂ over all trainable weights. Optimiser: Adam. Hyperparameters were chosen by grid search whose values the text says are in a supplement §3.3.7; that supplement is not in the compiled text (see Hyperparameters and Pitfalls).

Bidirectional model §3.4.2. Two DialogueRNNs read the sequence forwards and backwards. Their emotion states are concatenated per utterance, so each utterance sees past and future turns.

Emotion attention §3.4.3 §3.4.4 eq. 13 eq. 14. For each emotion state e_t, attention scores against all emotion states of the dialogue (eq. 13). The attended vector is the score-weighted sum of those states (eq. 14). The attended vector then goes to the classifier of step 6. The paper applies this to the unidirectional model (§3.4.3) and to the bidirectional one (§3.4.4).

# DialogueRNN, unidirectional, one utterance at a time  [eq. 1]-[eq. 8]
# Input:  utterances (T, D_m); speaker index per utterance (T,)
# Output: emotion states (T, D_e)

def dialogue_rnn(utterances, speakers):
    # Global state starts at zero; not stated in paper
    # [code: DialogueRNN/model.py#L124]
    global_state = zeros(D_g)
    # Party states start as null vectors  [§3.3.2]
    party_states = zeros(M, D_p)
    # Emotion state starts at zero; not stated in paper
    # [code: DialogueRNN/model.py#L150]
    emotion = zeros(D_e)
    history = []  # global states g_1..g_{t-1}
    emotions = []
    for utterance, speaker in zip(utterances, speakers):
        # Keep the speaker's state from before this utterance  [eq. 1] [eq. 5]
        previous_party = party_states[speaker]
        # Attention scores over earlier global states  [eq. 2] [eq. 3]
        if history:
            scores = softmax(utterance @ W_alpha @ stack(history).T)  # (t-1,)
            # Context: score-weighted sum of earlier global states  [eq. 4]
            context = scores @ stack(history)  # (D_g,)
        else:
            context = zeros(D_g)  # first utterance; not stated in paper
        # Update the speaker's party state only  [eq. 5]
        party_states[speaker] = GRU_P(
            previous_party, concat(utterance, context))
        # Emotion state from the speaker's updated party state  [eq. 8]
        emotion = GRU_E(emotion, party_states[speaker])
        emotions.append(emotion)
        # Global state uses the utterance and the previous party state  [eq. 1]
        global_state = GRU_G(
            global_state, concat(utterance, previous_party))
        history.append(global_state)
    return emotions


def classify(emotion):
    # Hidden layer with ReLU, then softmax over six classes  [eq. 9] [eq. 10]
    hidden = relu(W_l @ emotion + b_l)  # (D_l,)
    probabilities = softmax(W_smax @ hidden + b_smax)  # (6,)
    # Predicted label is the most likely class  [eq. 11]
    return argmax(probabilities)

The bidirectional emotion attention is a separate step that runs over all emotion states of the dialogue:

# Attention over all emotion states of a dialogue  [eq. 13] [eq. 14]
# Input:  emotions (N, 2 D_e) from the forward and backward models
# Output: attended states (N, 2 D_e)

def emotion_attention(emotions):
    attended = []
    for emotion in emotions:
        # Scores of this utterance against every utterance  [eq. 13]
        scores = softmax(emotion @ W_beta @ stack(emotions).T)  # (N,)
        # Weighted sum of all emotion states  [eq. 14]
        attended.append(scores @ stack(emotions))  # (2 D_e,)
    return attended

Hyperparameters

The compiled text gives few values. Code values are listed separately because the paper's text does not state them. Values from the DialogueRNN/ scripts at the commit in the front matter.

Results

Author-reported. Datasets §4.1 and tab. 1: IEMOCAP has six labels, two-person dyadic dialogues, 120 dialogues and 5810 utterances in train+val, and 31 dialogues and 1623 utterances in test. AVEC has four real-valued attributes, 63 and 32 dialogues, and 4368 and 1430 utterances. The split is about 80/20 with no speaker in both parts §4.1.1 §4.1.2. Baselines: c-LSTM, c-LSTM with attention, TFN, MFN, CNN, memory network, and CMN §4.2. Metric: weighted-average accuracy and weighted F1 on IEMOCAP; MAE and Pearson r on AVEC tab. 2.

IEMOCAP and AVEC, textual features tab. 2.

Source: tab. 2. Per-class values are the paper's; the Avg(w) columns are the weighted averages it reports.

Source: tab. 2, AVEC columns. The paper's text reports the headline gains as 2.77 points weighted accuracy and 3.76 points weighted F1 for DialogueRNN over CMN on IEMOCAP §5.1.1, and 6.62 points F1 for BiDialogueRNN+Att §5.2.4.

Multimodal (text + visual + audio, concatenated features) §5.3 tab. 3. IEMOCAP weighted F1 and AVEC Pearson r.

Source: tab. 3. The MFN arousal entry is printed as "25" in the source; it is probably a typo for 0.25, but it is reproduced here as printed.

Ablation on IEMOCAP tab. 4. Weighted F1 with the party state and emotion GRU removed one at a time: without party state, 55.56; without emotion GRU, 57.38; full model, 59.89 §5.6. The paper reads these as a 4.33-point drop for the party state and a 2.51-point drop for the emotion GRU §5.6.

Other analysis. On IEMOCAP, attention weight falls off with distance, so the strongest dependence is local. Even so, about 18% of test utterances attend to turns 20 to 40 steps away §5.4.3. For utterances where the same speaker's emotion changes, the model is correct 47.5% of the time, against 69.2% when it does not change §5.5.

Paper vs code

The official repository is linked from the paper's footnote §3.3. It is senticnet/conv-emotion at the commit in the front matter (MIT license, root LICENSE). Its README says the repository is now maintained under declare-lab/conv-emotion. The DialogueRNN code is in DialogueRNN/.

• Context attention. The paper's eq. 2 scores each earlier global state against u_t through W_alpha §3.3.3. The shipped IEMOCAP script uses the matching variant general as its default train_IEMOCAP.py model.py. The default inside the model classes and the AVEC script is simple, which scores the global states without u_t model.py model.py train_AVEC.py.
• Listener update. The paper's listener GRU takes visual cues and the context c_t §3.3.4 eq. 7. The code's listener GRU takes the utterance and the speaker's updated party state, with no visual features and no context model.py. It runs only with --active-listener train_IEMOCAP.py.
• Loss. The paper's loss is unweighted cross-entropy with an explicit L2 term §3.3.7 eq. 12. The IEMOCAP script uses per-class weights and normalises by the summed class weights of the labels train_IEMOCAP.py model.py. Regularisation is Adam weight decay, not an added λ‖θ‖₂ term train_IEMOCAP.py.
• Emotion attention bias. The paper's β score eq. 13 has no bias term. The code's attention transform for general2 has a bias model.py model.py.
• Dropout. The paper gives no dropout values. The code adds dropout after the ReLU layer and uses an extra 0.15 on the emotion states in the bidirectional model model.py model.py.
• AVEC model. The paper's best AVEC result is BiDialogueRNN+Att §5.2.4. The shipped AVEC script builds AVECModel, a unidirectional DialogueRNN with no emotion attention and one linear output train_AVEC.py model.py. Its loss is mean squared error train_AVEC.py model.py. The paper does not name its AVEC loss.
• Model selection. The paper does not say how the reported epoch was chosen. The code keeps the epoch with the lowest test loss and reports the test metrics for that epoch train_IEMOCAP.py train_AVEC.py.
• Feature files. Both scripts load precomputed feature pickles (IEMOCAP_features_raw.pkl, AVEC_features_<attr>.pkl) train_IEMOCAP.py train_AVEC.py. Neither file is in the DialogueRNN/ folder of this checkout, and I did not fetch the external downloads, so the feature pipeline is not checked here.
• Hyperparameters. The paper's text gives few of the values in the table above, only that they were grid-searched §3.3.7. The LaTeX source contains a commented-out supplementary table with different values (D_G 300 and 150, D_P 400 and 150, D_E 400 and 100, D_l 200 and 100, learning rate 1e-4, λ 1e-5) [main.tex#L981-L999]. It is not in the compiled paper, so this card does not use it. The code's IEMOCAP values differ from it.
• No discrepancies were found in the core equations: the global GRU input §3.3.1, the party GRU input §3.3.3, the emotion GRU input §3.3.5 and the classifier §3.3.6 match the code model.py model.py.

Pitfalls

• Epoch selection uses the test set. The IEMOCAP and AVEC scripts evaluate on the test split every epoch and keep the epoch with the lowest test loss train_IEMOCAP.py. Reported numbers may be optimistic. A faithful reproduction needs a validation split, which the scripts disable with valid=0.0 train_IEMOCAP.py.
• Class weights change the numbers. The IEMOCAP loss weights the six classes unevenly, and the weights are hard-coded, with no explanation in the code train_IEMOCAP.py. The paper's eq. 12 does not weight classes. Accuracy and F1 will differ between the two.
• Bidirectional and attention variants need the whole dialogue. BiDialogueRNN and the attention variants use future utterances §3.4.2 §5.2.2. They are not usable for real-time, one-turn-at-a-time classification.
• Listener state adds parameters for little gain. The authors found the listener GRU did not help and gave a slightly worse result §3.3.4 §5.2.1. The default is off, so the listener update is not part of the headline numbers.
• Attention masking. The masked attention in the code applies the mask to logits and then renormalises model.py. This is equivalent to a masked softmax over real utterances, but the padding logits are not set to minus infinity. Re-implementers should not copy the padding trick blindly.
• Context at the first utterance. The paper does not define c_1 or the initial global, party and emotion states. The code uses zeros for all of them model.py model.py.
• Equation numbers in the source differ from their labels. The LaTeX labels (for example eq:2 for the party update) are not the printed numbers. The numbers above count displayed rows in source order, so they match the compiled paper only if no row was added or removed. The compiled PDF was not checked.
• No seeds or variance reported. The paper does not state seeds or run-to-run variance. The IEMOCAP script seeds numpy only train_IEMOCAP.py, so torch runs are not reproducible from the code alone.
• Speaker split is not checked. The paper says the splits do not share speakers §4.1. The split comes from a pickle that is not in this checkout, so the claim cannot be verified here.
• AVEC labels are averaged. Each utterance's AVEC attribute is the average over its time span, because the original annotations come every 0.2 s §4.1.2. The exact averaging is not further described.
• Costs the abstract hides. The model keeps one GRU state per party and a growing list of global states, and attention is computed over the whole history, so cost grows with dialogue length §3.3.3. The paper does not report run time or memory.
• Multi-party claim is untested. The method is described as scalable to more than two speakers, but only dyadic datasets are used §6.

Lineage

• Builds on: GRU cells [DBLP:journals/corr/ChungGCB14], the CNN text extractor of Kim (2014) §3.2.1, and the feature pipeline of the CMN paper (Hazarika et al., 2018) §3.2.
• Compared against: c-LSTM and c-LSTM+Att (Poria et al., 2017), TFN, MFN, a CNN baseline, a memory network baseline, and CMN (Hazarika et al., 2018) §4.2 tab. 2 tab. 3.
• Superseded by: not stated in this card's sources.