DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition

In plain words

Emotion recognition in conversation means reading a chat or a dialogue and labelling the feelings of each message: happy, angry, neutral and so on. What makes it hard is that the people in the conversation speak in turns, and the emotion of one turn depends on what the same person said earlier and on what the others said. Large language models read text well, but they were built for one long document, not for a dialogue where many speakers take turns.

DialogXL keeps a strong pre-trained model (XLNet) and changes two things instead of building a special hierarchical network on top. First, it keeps a memory of the earlier messages, stored as whole messages rather than cut-off chunks of text, so more of the conversation fits. Second, it changes which earlier messages each attention head is allowed to look at: some heads look at the whole conversation, some only at the last few messages, some only at what the same speaker said, and some only at what the other speakers said.

The authors report that this beats the earlier models on four public datasets. These are the authors' own numbers, and the card below lists their setup.

TL;DR

• DialogXL is XLNet-Base with an utterance-level memory and four kinds of masked self-attention for multi-party dialogue §1 §3.3 §3.4.
• It classifies the emotion of each query utterance from the final [CLS] state and is trained with cross-entropy §3.2 §3.5.
• Author-reported weighted F1 (or micro F1 for DailyDialog) beats all listed baselines on IEMOCAP, MELD, DailyDialog and EmoryNLP tab. 2.
• The official code is Apache-2.0. Several training details (classifier depth, loss normalisation, batching, clipping, per-dataset values) differ from the paper or are missing from it. See "Paper vs code".

Problem

Emotion recognition in conversation (ERC) labels each utterance of a dialogue §1. The paper argues that pre-trained language models such as BERT and XLNet do not fit ERC for two reasons. They do not model intra- and inter-speaker dependencies directly §1. And their input length limit forces truncation, which can lose distant history §1.

The paper also argues that XLNet's segment recurrence is a poor fit: a "segment" is a fixed-length window, not a whole utterance, and segments in one batch must have the same length, so the memory fills with padding §3.3 fig. 2. The authors report that segment recurrence wastes over 60% of its memory on padding for every memory length they tested §5.2 fig. 4.

The task is defined as: given the utterances before a query utterance u_t and their speaker labels, predict the emotion label of u_t §3.1.

Method

DialogXL reuses XLNet-Base (12 Transformer layers) and replaces two parts of it. The overall architecture has an embedding layer, 12 Transformer layers and a feed-forward classifier fig. 1 §3. The model handles one conversation step by step: at each step it identifies the emotion of the next query utterance, given the memory of the earlier utterances §3.

Input. The query utterance u_t gets a [CLS] token at its start. Only the word embedding is applied at the input, which is the only input layer used in DialogXL eq. 1 eq. 2 §3.2. The authors' footnote says that XLNet's permutation language modelling and two-stream self-attention are not used §3, footnote.

Utterance recurrence. At every Transformer layer l, the hidden states of the tokens of the current utterance are appended to a per-layer memory m. The [CLS] state and padding are not stored §3.3 eq. 3. The memory keeps only the most recent mem_len positions, so the oldest states are dropped first §4.1, footnote on memory threshold.

Dialog-aware self-attention. Each attention head of a layer takes its keys and values from the memory concatenated with the query, as in XLNet eq. 4 eq. 5 eq. 6. A mask s is added to the scores before the softmax eq. 7 eq. 8. The four head groups differ only in their masks §3.4:

• Global: no masking, so the query sees all memory and the query itself eq. 10 §3.4.1.
• Local: the query sees only the ω most recent historical utterances eq. 11 §3.4.2. The paper calls ω a hyperparameter and does not give its value §3.4.2.
• Speaker: the query sees only earlier utterances of its own speaker eq. 12 §3.4.3.
• Listener: the query sees only utterances of other speakers eq. 13 §3.4.4.

The outputs of all head groups are concatenated, normalised, and passed through a feed-forward network to form the layer output eq. 14 eq. 15 §3.4. The authors state that this adds no extra embeddings or parameters, only different masks §3.4 fig. 3.

Classification. The final [CLS] state at the last layer is the encoding of the query and its context eq. 16. It goes through a ReLU layer and a softmax output layer eq. 17 eq. 18. The predicted label is the argmax eq. 19 §3.5.

Loss. The training objective is a cross-entropy sum over all turns of all training conversations eq. 20 §3.5. The equation as printed uses P_t[y] without a logarithm; see "Pitfalls".

# DialogXL: encode one query utterance given the memory  [§3.2] [§3.3]
# Input:  query tokens (B, n); memory per layer (B, M, d); speaker ids (B,)
# Output: emotion logits (B, C)

def dialogxl_step(query_tokens, memory, memory_speakers, window_left,
                  speaker_ids):
    # Prepend the [CLS] token to the query utterance  [eq. 1]
    tokens = concat([CLS], query_tokens)           # (B, n+1)
    # Only word embedding at the input  [eq. 2]
    hidden = word_embedding(tokens)                # (B, n+1, d)
    for layer in transformer_layers:
        # Keys and values run over memory plus query  [eq. 4]
        context = concat(memory[layer], hidden)    # (B, M+n+1, d)
        # Each head has a mask: global, local, speaker or listener  [§3.4]
        scores = relative_attention(hidden, context)  # (B, H, n+1, M+n+1)
        # Masked keys get a large negative score before the softmax  [eq. 7]
        scores = scores - BIG * head_masks(memory_speakers, window_left,
                                           speaker_ids)  # (B, H, n+1, M+n+1)
        # Head outputs are concatenated, then normalised and fed forward
        # [eq. 14] [eq. 15]
        heads = concat_heads(softmax(scores), context)
        hidden = feed_forward(layer_norm(heads))
        # Cache this utterance's tokens, not [CLS], for later steps  [eq. 3]
        new_states = hidden[1 : n_tokens + 1]      # (B, n, d)
        memory[layer] = concat(memory[layer], new_states)[-mem_len:]
    # Classify from the final [CLS] state  [eq. 16] [eq. 17] [eq. 18]
    cls_state = hidden[:, 0]                       # (B, d)
    z = relu(linear_h(cls_state))                  # (B, hidden)
    return linear_z(z)                             # (B, C); softmax in the loss

The head masks for the query at step t are defined as follows (not written as code in the paper; the formulas are eq. 10 to eq. 13):

# Which earlier positions each head group may attend to  [eq. 10]-[eq. 13]
# Input:  memory speaker ids (M,); memory window counters (M,); query speaker
# Output: boolean mask (M,) per head group; True means masked

def head_masks(mem_speakers, window_left, query_speaker):
    # Global heads: only padding is masked  [eq. 10] [§3.4.1]
    global_masked = mem_speakers == 0
    # Local heads: keep the last omega utterances  [eq. 11] [§3.4.2]
    local_masked = window_left <= 0
    # Speaker heads: mask other speakers  [eq. 12] [§3.4.3]
    speaker_masked = global_masked | (mem_speakers != query_speaker)
    # Listener heads: mask the query's own speaker  [eq. 13] [§3.4.4]
    listener_masked = global_masked | (mem_speakers == query_speaker)
    return local_masked, global_masked, speaker_masked, listener_masked

Hyperparameters

The paper says that XLNet-Base initialises the model, AdamW is the optimiser, and that learning rate, the head split, the memory length and dropout are tuned on the validation set §4.1. It does not give the chosen values. The values below for the official code come from the repository; they are not from the paper.

Dataset sizes tab. 1:

Metrics: weighted F1 for IEMOCAP, MELD and EmoryNLP; micro F1 for DailyDialog §4.2. Only the textual modality is used §4.2.

Results

Author-reported. Setup: XLNet-Base initialisation, the four datasets above, baselines from the cited prior work, BERT and XLNet baselines fine-tuned with the same tuning as DialogXL, scores averaged over 5 runs §4.1 §4.3 tab. 2. The "*" marks baseline scores the authors re-implemented tab. 2.

Overall F1 tab. 2:

Ablation, F1 on IEMOCAP and MELD tab. 3 §5.3:

Speaker role embedding versus the speaker and listener attention tab. 4 §5.4:

Other reported findings: the memory-length sweep plateaus above about 700 tokens §5.2 fig. 4. Removing local self-attention costs the most on IEMOCAP, where conversations are long (about 70 utterances) §5.3. The authors say that in the error study, the model mispredicts 45% of emotional shifts (same speaker, consecutive utterances, different label) on IEMOCAP §5.5.

Paper vs code

Official code: shenwzh3/DialogXL (Apache-2.0), at commit 70a2d9f2d6f125f8be6def139c088495c48837d3. The paper's footnote links github.com/shen-wzh3/DialogXL, which returns "Repository not found". The repository above was found by search; its owner shenwzh3 matches the first author's email handle printed in the paper's affiliations. The code is read from the commit above.

• Classifier depth. Paper: one ReLU layer, then softmax eq. 17 eq. 18. Code: a 768-to-300 ReLU layer, two more 300-to-300 ReLU layers, then a 300-to-classes output layer, with dropout on the [CLS] state model.py model.py run.py.
• Loss. Paper: an unnormalised sum of P_t[y] eq. 20. Code: PyTorch CrossEntropyLoss with ignore_index=-1 (log-likelihood, averaged over the valid labels in each step) run.py trainer.py.
• Update schedule. The paper describes a loss over all turns of all conversations eq. 20. Code: the optimiser steps once per utterance index, over the batch of conversations at that index, and conversations are not shuffled trainer.py trainer.py dataloader.py.
• Batching. Paper: not stated. Code: conversations are sorted by length and cut into batches, in the same order every epoch dataloader.py.
• Memory reset. Code resets the memory at the start of each batch of conversations trainer.py. Paper: not stated.
• [CLS] token. Paper: [CLS]. Code: XLNet's <cls> token is prepended dataset.py.
• Padding. Code pads utterances with token id 5 and masks padding with 0 dataset.py. Paper: not stated.
• Truncation. Code truncates each step's utterances to max_sent_len dataset.py. Paper: not stated, though the code README sets it per dataset README.md.
• Gradient clipping at norm 5.0 trainer.py. Paper: not stated.
• Model selection. Code saves the checkpoint with the best validation F1 and reports the test F1 of that epoch run.py run.py. Paper: 5 runs averaged, selection not described §4.1.
• Seeds. Code uses seed 100, one run run.py. Paper averages 5 runs §4.1.
• Reproduction numbers. The README states its own test F1 values: IEMOCAP 65.88, MELD 62.67, DailyDialog 55.67, EmoryNLP 36.27 README.md. Three of the four differ from Table 2 (65.94, 62.41, 54.93, 34.73) tab. 2. The README also says the released checkpoints may not load with the current code README.md.
• Dropout on [CLS] output, and mean-pooling variant. The code can pool by mean instead of [CLS] when use_cls is False; the default uses [CLS] model.py. Paper: [CLS] only eq. 16.
• Head allocation order. Paper's Figure 3 lists global, local, speaker, listener fig. 3. Code's --num_heads order is local, global, speaker, listener run.py.

No discrepancies found in the masks: the four masks in create_mask_dialog_multipart match eq. 10 to eq. 13 modeling_xlnet.py and the memory update matches eq. 3 modeling_xlnet.py. The local window is a counter reset to windowp on each cache write and decremented on every later step; a memory entry is visible while the counter is above 0 modeling_xlnet.py.

Pitfalls

• Section and equation numbers. The class sets secnumdepth to 0, so the compiled PDF prints no section numbers [tex: Formatting-Instructions-LaTeX-2021.tex, line 83]. The § references here follow the source order of the headings. Equation numbers were counted from the source and not checked against a rendered PDF.
• Loss. Eq. 20 as printed is a negative sum of probabilities with no logarithm and no normalisation eq. 20. The code uses a mean cross-entropy run.py. An implementer should use cross-entropy; the normalisation changes the effective learning rate.
• Update granularity. The official code updates weights once per utterance index across a batch of conversations, not once per conversation or per epoch trainer.py. Inference is sequential over the utterances of each conversation, so a conversation cannot be processed in parallel. This is inferred from the loop structure in the code.
• Masks use large finite values, not -inf. Masked scores get −1e30 (−65500 in float16) modeling_xlnet.py. Implementers using −∞ get the same result only if no row is fully masked. The query tokens are never masked, so no row is fully masked in this code.
• Local window semantics. ω is not given in the paper. The code's window is the last windowp historical utterances, counted by the decrementing counter modeling_xlnet.py. A different counter convention shifts the window by one utterance.
• Dead path. In cache_window_mask, the non-batched branch reads prev_speaker_mask, which is not defined in that function modeling_xlnet.py. run.py always passes content_lengths, so this branch is not reached in the code's own use. An implementer should not copy it.
• Memory layout. Memory tensors are right-aligned and zero-filled to mem_len, and the padding slots are masked modeling_xlnet.py. The paper's "no padding in memory" claim §3.3 holds for the masked content, but the tensor still holds zeros.
• Memory cost. The memory grows with conversation length up to mem_len, and the memory-length sweep was run on IEMOCAP only §5.2. The paper gives no training memory or time figures.
• Speaker labels. DailyDialog has no speaker information; the paper treats each utterance turn as a speaker turn §4.2. The code's speaker ids come from the dataset's speaker vocabulary dataset.py.
• Dataset-specific values. The paper reports one set of results but tunes per dataset. The per-dataset values exist only in the README, not in the paper README.md.
• Ambiguous claim. Contributions say the memory caches "up to 1000 historical words" §1. The experiments use a 100 to 1000 sweep and the code's defaults are 400 to 900 §5.2 run.py.
• Error study. The 45% figure is for emotional shifts on IEMOCAP only §5.5.

Lineage

• Builds on: XLNet, which adds the segment recurrence that DialogXL replaces with utterance recurrence §3.3; Transformer-XL, the source of relative positional attention used by DialogXL §3.3 eq. 6.
• Compared against: CMN, DialogueRNN, HiGRU, DialogueGCN, TL-ERC, KET, BERT and XLNet tab. 2. The paper also compares speaker role embedding (DialogXL-emb) tab. 4.
• Superseded by: not stated.