COSMIC: COmmonSense knowledge for eMotion Identification in Conversations

In plain words

A short line in a chat, such as "okay", can mean someone is calm, annoyed or pleased. What came before it and who said it decide which. COSMIC gives each line extra background knowledge before it labels the emotion. The background comes from a model trained on everyday "if this happens, then..." statements. For each line, it guesses what the speaker probably intends and how the speaker and the other people are likely to feel and react.

These guesses go into a recurrent network that reads the conversation in order and keeps a running state for each person. A final layer then gives every line its emotion. The authors' claim is that the extra knowledge helps with sudden emotion shifts and with close emotions such as anger and frustration, and they show this on one example conversation. They also say the gain over the text-only models is modest.

TL;DR

COSMIC predicts an emotion label for each utterance in a dialogue, from text only. It fine-tunes RoBERTa-Large on utterance labels, adds five COMET commonsense vectors per utterance (ATOMIC relations), and runs GRU cells for context, participant states, intent and emotion. Author-reported: 65.28 weighted F1 on IEMOCAP and new best scores on DailyDialog, MELD and EmoryNLP tab. 4. Read Paper vs code before reproducing: the official scripts leave listener states unchanged by default, and the emotion-state recurrence and the classifier differ from the text.

Problem

Utterance-level emotion recognition in conversation needs context. The authors argue that context-independent word or sentence embeddings, and recurrent or graph models over them, miss contextual cues, emotion shifts across turns, and the difference between related emotion classes §1 §2. Their proposed source of missing information is commonsense knowledge about mental states, events and cause-effect relations §1 §3.3.

Assumptions stated in the paper: each utterance comes with its speaker §3.1, and the datasets are the four listed in tab. 3 with text transcripts only §4.1. The paper says the commonsense gain is "not very substantial" and that better knowledge sources may help §5.5.

Method

The framework has three stages §3. Stage 1 turns each utterance into a context-independent vector with a fine-tuned RoBERTa-Large. Stage 2 builds five commonsense vectors per utterance with COMET, a model trained on the ATOMIC knowledge base of if-then commonsense statements §3.3. Stage 3 is a recurrent context model that combines both sources §3.4. The architecture is drawn in fig. 2.

Stage 1: context-independent features §3.2

1. Fine-tune RoBERTa-Large for emotion classification of the utterances. The [CLS] token of each utterance goes through a small feed-forward classifier §3.2. RoBERTa-Large has 24 layers, 16 attention heads, hidden size 1024 and 355M parameters §3.2.
2. For each utterance, take the [CLS] activations of the final four layers and average them. The result is a 1024-dimensional vector x_t §3.2.

Stage 2: commonsense features §3.3 tab. 1

1. Use the COMET encoder only and discard its decoder §3.3.
2. For each utterance U, run the encoder on U followed by one of five relation phrases: intent of the speaker X, effect on X, reaction of X, effect on the others, and reaction of the others §3.3.
3. Take the encoder state at the final time step. Each vector is 768-dimensional §3.3. Table 1 lists the five features, their mental-state or event type, and their causal role tab. 1.

Notation used below: IS(u) intent of the speaker, ES(u) effect on the speaker, RS(u) reaction of the speaker, EL(u) effect on the listeners and RL(u) reaction of the listeners §3.4. The paper treats the other parties as listeners. The effect and reaction vectors for "others" are the listener features in Table 1.

Stage 3: commonsense conversational model §3.4

The model keeps one shared context state c_t and one attention vector a_t for the whole conversation. It also keeps three latent states per participant k: an internal state q_{k,t}, an external state r_{k,t} and an intent state i_{k,t} §3.4. Table 2 lists what each state depends on tab. 2. Five GRU cells update these: GRU_C for context, GRU_Q for the internal state, GRU_R for the external state, GRU_I for the intent state and GRU_E for the emotion state. In practice all five are bidirectional fig. 2. The equations below use unidirectional notation fig. 2 caption §3.4.

For an utterance u_t spoken by participant s:

# Per-utterance update, as written in the paper  [§3.4] [eq. 1-10]
# x: RoBERTa vector (1024,); cs: five COMET vectors (768,) each
# q, r, i: dict participant -> state; c_hist: earlier context states

def step(x, cs, speaker, listeners, c_hist, c_prev, e_prev):
    # Context state from the utterance and the speaker's last states  [eq. 1]
    c = gru_context(c_prev, concat(x, q[speaker], r[speaker]))
    # Attention pools earlier context states; see Paper vs code  [eq. 2]
    a = attend(c_hist, c)
    # Speaker internal state  [eq. 3]
    q[speaker] = gru_internal(q[speaker], concat(a, cs.effect_on_speaker))
    # Listener internal states, only if listener updates are enabled  [eq. 4]
    for j in listeners:
        q[j] = gru_internal(q[j], concat(a, cs.effect_on_listeners))
    # Speaker external state  [eq. 5]
    r[speaker] = gru_external(r[speaker], concat(a, x, cs.reaction_speaker))
    # Listener external states  [eq. 6]
    for j in listeners:
        r[j] = gru_external(r[j], concat(a, x, cs.reaction_listeners))
    # Speaker intent changes; listener intents stay the same  [eq. 7-8]
    i[speaker] = gru_intent(i[speaker], concat(cs.intent, q[speaker]))
    # Emotion state from the utterance and the speaker's three states  [eq. 9]
    e = gru_emotion(e_prev, concat(x, q[speaker], r[speaker], i[speaker]))
    return e, c

Attention, as written in the paper §3.4 eq. 2:

u_i = \tanh(W_s c_i + b_s), \quad \alpha_i = \frac{\exp(u_i^\top x_i)}{\sum_{j=1}^{t-1} \exp(u_j^\top x_j)}, \quad a_t = \sum_{i=1}^{t-1} \alpha_i c_i

Classification, as written in the paper §3.4 eq. 10:

P_t = \mathrm{softmax}(W_{smax} e_t + b_{smax}), \quad \hat{y}_t = \arg\max_k P_t[k]

Hyperparameters

Values stated in the paper are cited to the paper. Code defaults are listed per dataset; they are the argparse defaults in each training script, not values the paper reports. Where the paper gives no value, it says "not stated".

Results

Author-reported. All scores are test-set scores on the four benchmarks, taken at the best validation score tab. 4. Table 3 lists the splits and metrics tab. 3. Setup: textual input only; COSMIC uses RoBERTa-Large features and COMET commonsense features §4.1 §3.2 §3.3. The GloVe-based CNN and DialogueRNN rows come from an earlier paper, as the caption says tab. 4.

Source: tab. 4. "–" means the table gives no value. CSK is the commonsense component from COMET tab. 4 caption.

Points the paper makes about these numbers:

• On IEMOCAP, COSMIC is 65.28 against 64.76 for RoBERTa DialogueRNN, and the paper says the gain is smaller there than on DailyDialog tab. 4 §5.2.
• On MELD and EmoryNLP, the authors report the largest gain on the fine-grained 7-class setting §5.2.
• Removing the speaker-side commonsense features lowers the score more than removing the listener-side ones in five of the seven columns. The exceptions are DailyDialog macro F1 and MELD 3-class, where the listener-side removal is the larger drop. Removing the listener features slightly exceeds the full model on both EmoryNLP settings tab. 4 §5.3.
• In the IEMOCAP case study, the authors say the commonsense features help the model follow sudden shifts (neutral to frustrated and back) and separate angry from frustrated fig. 3 §5.4. This is a single conversation.

Paper vs code

Official code: github.com/declare-lab/conv-emotion, at code_commit above, under COSMIC/. The repository owner is the authors' lab. The repository is MIT-licensed. The code for the model is in COSMIC/erc-training/; the feature extraction is in COSMIC/feature-extraction/.

Discrepancies found:

1. Listener states are not updated by default. The paper updates the listener internal and external states with the attention vector and the listener features §3.4 eq. 4 eq. 6. The listener cells run only with --active-listener, which defaults to off in all four scripts [code: COSMIC/erc-training/train_iemocap.py#L120] train_iemocap.py. Without it, listeners keep their previous states [code: COSMIC/erc-training/commonsense_model.py#L118-L120]. When it is on, the code's inputs differ from the paper. The listener external state takes the utterance, the reaction-of-others vector and the speaker's updated external state, with no attention vector commonsense_model.py. The listener internal state takes the effect-on-others vector and the speaker's updated internal state commonsense_model.py. The paper does not say which setting produced Table 4.

2. Emotion-state recurrence. The paper says the emotion state depends on the speaker's previous emotion state §3.4 and writes e_{t-1} in eq. 9. The code passes the emotion state of the previous utterance to the next step, whoever spoke it commonsense_model.py commonsense_model.py.

3. Context attention. The paper's score is u_i^\top x_i with u_i = \tanh(W_s c_i + b_s) §3.4 eq. 2. The code uses the current utterance as the query over the history. For IEMOCAP and EmoryNLP, the default general2 applies a biased linear map to the query, takes a dot product with each earlier context state, applies tanh, then a masked softmax model.py model.py. For DailyDialog and MELD, the default simple uses a softmax over a learned scalar of each earlier context state, with no query model.py. The paper's x_i in the score is ambiguous; the code shows which quantity it means.

4. Classifier. The paper uses a fully connected layer and softmax on e_t §3.4 eq. 10. The code (a) runs forward and backward emotion cells and concatenates them, as the figure caption describes fig. 2 [code: COSMIC/erc-training/commonsense_model.py#L235-L238] [code: COSMIC/erc-training/commonsense_model.py#L318]; (b) during training and evaluation, applies an attention over all utterances in the dialogue commonsense_model.py [code: COSMIC/erc-training/train_iemocap.py#L67]; (c) adds a linear layer, ReLU and dropout before the output layer commonsense_model.py; (d) adds an optional residual connection, off by default [code: COSMIC/erc-training/commonsense_model.py#L336-L337]; (e) outputs log-softmax commonsense_model.py.

5. Unused auxiliary output. The code runs a bidirectional GRU over the effect-on-speaker vector and returns it, but the training loop discards it. The paper does not describe this layer commonsense_model.py commonsense_model.py train_iemocap.py.

6. Input normalisation. The paper describes no normalisation. The IEMOCAP default applies BatchNorm to each of the four RoBERTa vectors before averaging commonsense_model.py commonsense_model.py train_iemocap.py. The other three scripts use none.

7. RoBERTa learning rate. The paper says 1e-5 §4.2. The fairseq script the code provides uses 1e-6 roberta_train_iemocap.sh.

8. Residual connection in RoBERTa training. The paper says RoBERTa training uses a residual connection between the first and penultimate layer for MELD and EmoryNLP §4.2. I searched the RoBERTa scripts in feature-extraction/ and found no residual. The only residual in the code is the emotion-model option in point 4(d).

9. COMET input and pooling. The paper concatenates the utterance with a relation phrase §3.3. The code appends a special relation token to the event, and it splits each utterance into sentences and averages the sentence-level vectors csk_feature_extract.py csk_feature_extract.py. The paper says nothing about sentence splitting.

10. Relations computed. The extractor computes nine ATOMIC relations, and the emotion model uses five of them csk_feature_extract.py train_iemocap.py.

11. Missing artefacts. The COMET checkpoint path points to comet/pretrained_models/atomic_pretrained_model.pickle, which is not in the repository csk_feature_extract.py. The feature folders hold only a README. Reproducing the features needs a COMET checkpoint and the feature files from elsewhere. I did not find a download location in the repository.

12. Class weights. The loss uses class weights only with --class-weight, which defaults to off. The paper does not mention class weights [code: COSMIC/erc-training/train_iemocap.py#L188-L191].

Pitfalls

• Equation and table numbers above are counted from the LaTeX source in order. I did not check them against a compiled PDF. Section citations are reliable; equation numbers should be checked before quoting them elsewhere.
• Which setting gives Table 4? The paper does not say whether listener updates were on or off §3.4 tab. 4. The code defaults to off. Choose one and say which in your results.
• Two selection rules. The table caption says scores are at the best validation score tab. 4. The IEMOCAP script reports two test scores, one at the best validation loss and one at the best validation weighted F1 train_iemocap.py. The paper does not say which one it reports.
• Run count is inconsistent. The Table 4 caption says "average of five runs" and then "10 runs" for RoBERTa DialogueRNN and COSMIC tab. 4. The paper does not resolve this.
• Leakage-prone training features. RoBERTa is fine-tuned on the training labels §3.2, then the feature script runs that same model over the training, validation and test utterances roberta_feature_extract_iemocap.py. Training features therefore come from a model fitted to their own labels, while test features do not. The paper does not discuss this effect.
• Non-causal model. The backward pass reverses each dialogue, so each prediction uses future utterances commonsense_model.py commonsense_model.py. The paper's figure shows bidirectional cells fig. 2. It is not suitable for live prediction without changes.
• Fixed speaker width. The IEMOCAP loader encodes speaker as two columns dataloader.py. Check the multi-party loaders before reusing this code for more than two speakers.
• Modest commonsense gain. The authors say the gain is "not very substantial". They also tried four other COMET relations (attribute, need, and wanted by the speaker and the listeners) and report no significant gain from them §5.5. Test a no-commonsense baseline with the same RoBERTa features before attributing gains to the commonsense vectors.
• Compute cost. Stage 2 runs the COMET encoder over every utterance, for nine relations per sentence in the code [code: COSMIC/feature-extraction/comet/csk_feature_extract.py#L58-L85]. The paper gives no timing or memory figures.
• Baselines. The GloVe-based CNN and DialogueRNN rows come from an earlier paper, not from these experiments tab. 4 caption.

Lineage

• Builds on: DialogueRNN, a recurrent model with separate GRU networks for speaker, context and emotion §2 §5.1 tab. 4; COMET [bosselut2019comet], arXiv:1906.05317 §3.3; RoBERTa [liu2019roberta], arXiv:1907.11692 §3.2. These three are named, not summarised here; this card uses nothing from them beyond what the paper states.
• Compared against: CNN, ICON, KET, ConGCN, BERT DCR-Net, BERT+MTL, DialogueRNN and RoBERTa-based baselines §5.1 tab. 4.
• Related card: DialogueGCN, a graph model for the same task.
• Superseded by: not known from content already on this wiki.