DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation

In plain words

When someone in a chat says "okay", you can't tell on its own whether they're calm, fed up or pleased. You have to look at what was said around it, and also at who said what. DialogueGCN is a model that labels the emotion of every line in a conversation by looking at both.

It works in two passes. First, a recurrent network reads the conversation in order, so each line picks up some sense of the lines near it. Then the model draws a graph: every line is a node, and each node is linked to the ten lines before it and the ten after. Every link gets a type based on which two people spoke the lines and whether the neighbouring line came earlier or later. A graph network then passes information along these typed links. It can learn, for example, that what your partner just said affects you differently from your own earlier mood.

The result is a better context signal for each line. The authors report that it beats earlier recurrent models on two-person conversation datasets and does slightly better on a harder multi-party dataset built from a TV sitcom. Their tests show that both the speaker information and the before/after information on the links help.

TL;DR

DialogueGCN handles utterance-level emotion recognition in conversation (ERC) from text only. A bidirectional GRU produces sequential context features g_i. A directed graph links each utterance to its past 10 and future 10 utterances. Edge weights come from attention, and there are 2M^2 relation types (speaker of source × speaker of target × before/after). A two-layer graph convolution (R-GCN-style first layer, plain graph conv second) produces h_i^{(2)}, and the classifier uses [g_i, h_i^{(2)}] plus attention over the whole dialogue. The authors report 64.18 weighted F1 on IEMOCAP versus 62.75 for DialogueRNN, best MAE on all four AVEC dimensions, and 58.10 F1 on MELD. The official code differs from the paper in several places (edge attention formula, missing nonlinearities, LSTM instead of GRU, model selection on test F1), so read Paper vs code before reproducing anything.

Problem

Earlier ERC models run an RNN over the utterances. In practice, RNNs do not carry long-range context well, and IEMOCAP and AVEC dialogues average 50 and 72 utterances §1 §5.1. The attention variants of DialogueRNN pool over the conversation but ignore who spoke each context utterance and where it sits relative to the target §1. The authors argue that emotion depends on inter-speaker influence (counterparts change a speaker's emotion) and self-dependency (emotional inertia), and that future utterances can also help §1 §3. Assumptions: text modality only §4.1; per-utterance features are context-independent CNN features computed beforehand §3.2; speaker identity of each utterance is known §3.1.

Method

Input. A conversation u_1,\dots,u_N with speakers p_{s(u_i)} among M parties. u_i \in \mathbb{R}^{D_m} is a context-independent textual feature §3.1.

Step 0, utterance features §3.2. A Kim-style CNN over 300-d GloVe 840B vectors: one conv layer with filter widths 3, 4, 5 and 50 feature maps each, max-pooling with window 2, ReLU, concatenation, then a 100-d fully connected layer. It is trained separately at the utterance level on the emotion labels.

Step 1, sequential context encoder §3.3.1. A bidirectional GRU over u_1..u_N gives g_i = \overleftrightarrow{GRU_{\mathcal{S}}}(g_{i(+,-)1}, u_i) for each i. This step is speaker-agnostic.

Step 2, graph construction §3.3.2.

• Vertices: one per utterance, initialised with g_i.
• Edges: v_i connects to v_{i-p..i-1}, v_{i+1..i+f} and itself, with p=f=10 in all experiments. A fully connected graph is possible but costs O(N^2) edges.
• Edge weights, normalised so the incoming edges of each vertex sum to 1 eq. 1:

\alpha_{ij} = \mathrm{softmax}\big(g_i^{T} W_e [g_{i-p},\dots,g_{i+f}]\big), \quad j = i-p,\dots,i+f

• Relation of edge r_{ij}: the pair (speaker of u_i, speaker of u_j) combined with whether i<j. That gives up to 2M^2 relation types. tab. 1 lists all 8 relations for a 5-utterance dyadic example. There, self-edges (i,i) fall in the "not i<j" relation of their speaker pair.

tab. 1, reproduced (speakers p_1: u_1,u_3,u_5; p_2: u_2,u_4; fully connected graph):

Step 3, speaker-level encoder: two graph convolutions §3.3.2. The first is a relation-specific transform in the style of R-GCN eq. 2:

h_i^{(1)} = \sigma\Big(\sum_{r\in\mathcal{R}}\sum_{j\in N_i^r}\frac{\alpha_{ij}}{c_{i,r}} W_r^{(1)} g_j + \alpha_{ii} W_0^{(1)} g_i\Big)

c_{i,r} is either fixed as |N_i^r| or learned. \sigma is an activation "such as ReLU" §3.3.2. The second is a relation-free, unweighted neighbourhood transform eq. 3:

h_i^{(2)} = \sigma\Big(\sum_{j\in N_i^r} W^{(2)} h_j^{(1)} + W_0^{(2)} h_i^{(1)}\Big)

Step 4, classifier §3.3.2 eq. 4–9:

h_i = [g_i, h_i^{(2)}], \quad \beta_i = \mathrm{softmax}\big(h_i^T W_\beta [h_1,\dots,h_N]\big), \quad \tilde h_i = \beta_i [h_1,\dots,h_N]^T

l_i = \mathrm{ReLU}(W_l \tilde h_i + b_l), \quad \mathcal{P}_i = \mathrm{softmax}(W_{smax} l_i + b_{smax}), \quad \hat y_i = \arg\max_k \mathcal{P}_i[k]

Loss eq. 10: cross-entropy averaged over all utterances in the batch, plus an L2 term:

L = -\frac{1}{\sum_{s=1}^N c(s)}\sum_{i=1}^N\sum_{j=1}^{c(i)} \log \mathcal{P}_{i,j}[y_{i,j}] + \lambda \lVert\theta\rVert_2

Here N is the number of dialogues and c(i) the number of utterances in dialogue i. Optimiser: Adam. Hyperparameters were chosen by grid search §3.3.2.

# DialogueGCN forward pass for one dialogue, as written in the paper
# Input:  utterance_features (N, D_m), speaker_of (N,), window_past=10,
#         window_future=10
# Output: class probabilities (N, n_classes)

def dialogue_gcn(utterance_features, speaker_of, window_past, window_future):
    # Speaker-agnostic sequential context with a BiGRU  [§3.3.1]
    seq_context = bigru(utterance_features)            # (N, 2*D_e)

    # Build windowed directed edges, including the self-loop  [§3.3.2]
    edges = []
    for target in range(N):
        low = max(0, target - window_past)
        high = min(N - 1, target + window_future)
        for neighbour in range(low, high + 1):
            edges.append((target, neighbour))

    # Edge weights: softmax of bilinear scores over each window  [eq. 1]
    weight = {}
    for target in range(N):
        window = [n for (t, n) in edges if t == target]
        scores = [seq_context[target] @ W_e @ seq_context[n] for n in window]
        probs = softmax(scores)
        for n, p in zip(window, probs):
            weight[(target, n)] = p

    # Relation id from speaker pair and order; self-loop counts as "not i<j"
    # [§3.3.2] [tab. 1]
    def relation(i, j):
        before = 1 if i < j else 0
        return relation_index(speaker_of[i], speaker_of[j], before)

    # First graph conv, relation-specific, attention-weighted  [eq. 2]
    hidden1 = []
    for i in range(N):
        total = weight[(i, i)] * (W0_1 @ seq_context[i])
        for (t, j) in edges:
            if t != i:
                continue
            r = relation(i, j)
            norm = count_neighbours(i, r)              # c_{i,r} = |N_i^r|
            total += weight[(i, j)] / norm * (W_r1[r] @ seq_context[j])
        hidden1.append(relu(total))

    # Second graph conv, shared weights, unweighted sum  [eq. 3]
    hidden2 = []
    for i in range(N):
        total = W0_2 @ hidden1[i]
        for (t, j) in edges:
            if t == i:
                total += W_2 @ hidden1[j]
        hidden2.append(relu(total))

    # Concatenate sequential and speaker-level features  [eq. 4]
    node = [concat(seq_context[i], hidden2[i]) for i in range(N)]

    # Attention of every node over the whole dialogue  [eq. 5] [eq. 6]
    attended = []
    for i in range(N):
        beta = softmax([node[i] @ W_beta @ node[k] for k in range(N)])
        attended.append(sum(beta[k] * node[k] for k in range(N)))

    # Two-layer classifier  [eq. 7] [eq. 8]
    return [softmax(W_smax @ relu(W_l @ a + b_l) + b_smax) for a in attended]

Whether the sum in eq. 2 over j \in N_i^r also includes j=i is not stated, since the self term is written separately. In tab. 1 the self-loop has its own relation. The code puts the self-loop in the edge list and uses a root weight (see Paper vs code).

Hyperparameters

Datasets tab. 2:

IEMOCAP has 6 classes (happy, sad, neutral, angry, excited, frustrated) §4.1. AVEC has four real-valued targets (valence, arousal, expectancy, power), averaged over each utterance from 0.2 s annotations §4.1. MELD has 7 classes and is multi-party §4.1.

Results

Author-reported, text modality only, mean of 5 runs. The authors state significance by paired t-test, p < 0.05 §5.1.

IEMOCAP, accuracy / F1 per class and weighted average tab. 3:

AVEC (MAE, lower is better) and MELD (F1) tab. 4:

Memnet, CMN and ICON are not run on MELD because they only handle dyadic conversations §5.1.

Context window, IEMOCAP F1 §5.2: (10,10) 64.18; (8,8) 62.48; (4,4) 59.41; (0,0) 55.80. Window (0,0) leaves only self-edges. Larger windows were not tried because of compute §5.2.

Ablations on IEMOCAP, F1 tab. 5 tab. 6:

The §5.3 text gives the no-encoder F1 as 36.7, while tab. 5 gives 36.75 §5.3. Qualitatively, the authors report that edge weights let a target attend to the other speaker's utterances, and that a short "okay" is correctly labelled frustrated from context fig. 4 §5.4. Common confusions are frustrated vs angry/neutral and excited vs happy/neutral §5.5.

Paper vs code

Official code: DialogueGCN/ in the repository linked from the paper's footnote §3.3, at the commit above. It contains IEMOCAP and DailyDialog training scripts only, with no AVEC or MELD script. The repository also has DialogueGCN-mianzhang/, which the README credits to a third party. That folder is not used here.

• Edge weights. The paper scores g_i^T W_e g_j eq. 1. The code (attn1, marked as the paper's equation) uses a Linear(2*D_e, max_seq_len) layer instead, so the score for pair (i,j) is w_i^T g_j with one learned vector per absolute position i, independent of g_i. It applies a softmax over all padded positions j, masks to the window with 1e-10 elsewhere and renormalises model.py. Dialogues longer than max_seq_len=110 cannot be handled.
• Normalisation direction. The weights are normalised over j for each fixed first index of the edge pair (i, j) model.py. These pairs become the PyG edge_index rows model.py. Under PyG's default source-to-target flow, that index is the sender, so normalisation would run over outgoing edges rather than the incoming edges described in §3.3.2. Check this against your PyG version.
• Graph layers. The first layer is PyG RGCNConv(num_bases=30) with edge_norm = attention weights, and the second is PyG GraphConv, with no activation between or after them model.py. The paper applies \sigma in both eq. 2 eq. 3. No explicit 1/c_{i,r} appears in the repo, so any normalisation comes from the library's RGCNConv. The README pins PyTorch Geometric 1.3.
• Self term. Self-loops are in the edge list with relation "not i<j" model.py. On top of that, RGCNConv's root weight adds a separate self term.
• Sequential encoder. The paper uses a BiGRU §3.3.1. The script default and the README reproduction command use a 2-layer BiLSTM. GRU is also 2-layer model.py train_IEMOCAP.py.
• Classifier attention eq. 5 eq. 6 is off by default (--nodal-attention) and is turned on in the README command. It uses the general2 variant, which applies tanh to the scores before the softmax and masks padding model.py model.py. A linear layer of size 100 and dropout sit before the output layer model.py.
• Speakers. IEMOCAP speakers are encoded by gender (M → party 0, else party 1) dataloader.py. Relation types therefore mean "male/female" pairs across dialogues.
• Loss. nn.NLLLoss averages over the batch's utterances, and the L2 term is applied through Adam weight_decay train_IEMOCAP.py, matching eq. 10 in spirit.
• Evaluation. The script uses valid=0.0 (no validation split) and prints the maximum test F1 over all epochs train_IEMOCAP.py train_IEMOCAP.py train_IEMOCAP.py. That is model selection on the test set. The paper describes a 10% validation split tab. 2 and a mean of 5 runs §5.1.
• Reproduction note (README). The recommended IEMOCAP run turns on the graph model, nodal attention and class weights, with an LSTM base, learning rate 3e-4, dropout 0.4, L2 set to zero, 32 dialogues per batch, and runs on CPU. The README reports 64.67 and 64.44 weighted F1 on IEMOCAP on two machines and attributes run-to-run variation to nondeterministic CUDA atomics in PyG.
• Features. The IEMOCAP script loads precomputed 100-d text features from a pickle dataloader.py. The utterance CNN of §3.2 is not included for IEMOCAP.

Pitfalls

• Equation 1 and the code's attn1 are different models. If you implement the paper, use a bilinear form in g_i and g_j. If you want to match the reported numbers, you may need the code's per-position scorer and its max_seq_len limit.
• The paper never states \sigma for the graph layers, whether c_{i,r} is fixed or learned, the hidden sizes, the learning rate, dropout or epochs. All of these come from the code and the README, and the two disagree with each other (lr 1e-4 vs 3e-4, dropout 0.5 vs 0.4, L2 1e-5 vs 0).
• The reported IEMOCAP numbers may be optimistic: the public script picks the best epoch by test F1. For a fair comparison, use a held-out validation split.
• eq. 3 reuses the symbol N_i^r although the second layer has no relations. Read it as the full window neighbourhood, unweighted.
• Relation ids depend on a consistent speaker index. With M varying per dialogue (MELD often has more than 5 speakers §5.1), 2M^2 relation weights grow quickly. The paper gives no rule for mapping speakers to indices across dialogues.
• The sequential encoder's padding: the code runs the LSTM/GRU over padded batches without packing model.py, so the backward direction of shorter dialogues reads padding first.
• The windowed graph costs O(N(p+f)) edges per dialogue. The code builds edges in Python loops per batch model.py, which is slow for large batches.
• The problem definition lists 8 emotion labels §3.1, but IEMOCAP uses 6 and MELD 7 §4.1. Set n_classes per dataset.
• MELD gains are small (+1.07 F1 over DialogueRNN tab. 4). The authors attribute this to short utterances and many speakers who each say little §5.1.

Lineage

• Builds on: DialogueRNN (arXiv:1811.00405), the prior state of the art. Its speaker GRUs are replaced by a graph over a context window. The first graph layer follows R-GCN (arXiv:1703.06103) §3.3.2. bc-LSTM (Poria et al., 2017) is the sequential-context baseline.
• Compared against: CNN, Memnet, bc-LSTM, bc-LSTM+Att, CMN, ICON, DialogueRNN §4.2.