MIME: MIMicking Emotions for Empathetic Response Generation

In plain words

When someone shares news, a good reply both understands how they feel and answers what they said. Many chatbots reply with generic sympathy that ignores the feeling in the message. MIME tries to fix this by first working out what emotion the other person is showing, then choosing a reply tone that echoes it.

The model sorts its emotion labels into a positive group and a negative group. If the person sounds upset, it picks a mix of negative emotions to echo, and it adds some positive emotion to comfort them. If the person sounds happy, it mostly picks positive emotions. Because the choice is sampled rather than fixed, the same message can get more than one reasonable reply.

Two versions of the emotional tone are built, one that mirrors the person and one that does not, and a gate blends them. A standard transformer decoder then writes the reply from this blended context. The authors target empathy as one of the main qualities, and they measure it with human ratings rather than word overlap.

TL;DR

MIME is a transformer dialogue model for empathetic replies that groups the 32 EmpatheticDialogues emotions into positive and negative sets, samples a mixture over each set, and blends a mimicking and a non-mimicking emotion-refined context. Its reported gains are in human empathy and relevance over MoEL, with fluency slightly lower. The results are author-reported on one dataset with small human evaluations. The released code departs from the paper in several places, listed under Paper vs code.

Problem

Earlier empathetic generators, such as MoEL, treat all the emotions in the input as one flat set §1. The authors argue that an empathetic reply often mirrors the user's emotion to a degree that depends on whether the emotion is positive or negative §1 §3.3. They also report that MoEL often gives generic, repetitive replies, which motivates adding randomness to the choice of response emotion §3.3.

Method

Overview. The model encodes the dialogue context, predicts the user's emotion, samples a response-emotion mixture for each polarity group, forms a mimicking and a non-mimicking refined context, fuses them with a gate, and decodes the reply. The emotion embeddings are shared by the emotion classifier and the response decoder §5.1.

1. Context encoding §3.2 eq. 1 eq. 2 eq. 3. Context utterances are concatenated. Each word is the sum of a word embedding, a positional embedding and a speaker embedding, where the speaker embedding marks the user or the agent. A CTX token is prepended, and a transformer encoder gives each position a context state. The state at CTX is the context vector c.
2. Emotion classification §3.2 eq. 4 eq. 5 eq. 6. c is projected by a weight matrix and scored against 32 learned emotion embeddings. A softmax over the scores gives the emotion distribution, trained with cross-entropy on the user's gold emotion.
3. Emotion grouping §3.3 tab. 1. The 32 emotions split into 13 positive and 19 negative groups, as listed in tab. 1.
4. Response-emotion sampling §3.3 eq. 7 eq. 8 eq. 9 eq. 10. For each group, a Gaussian prior over a latent z_g is computed from c through a ReLU layer, and z_g is sampled by reparameterisation eq. 11-15. A fully connected layer and a softmax turn z_g into weights d_g over the group's emotions. The group representation e_g is the weighted sum of those emotion embeddings eq. 10.
5. Emotion mimicry §3.3 eq. 18 eq. 19. m is the positive group representation when the user's emotion is positive, and the negative one otherwise. m_tilde is the other group.
6. Refined contexts §3.3 eq. 20 eq. 21 eq. 22 eq. 23. m is appended to every context state, and a transformer encoder produces the mimicking context M. The same is done with m_tilde to produce M_tilde.
7. Fusion §3.3 eq. 24 eq. 25 eq. 26 eq. 27. M and M_tilde are concatenated per position. A sigmoid gate weights the concatenation, and a fully connected layer maps it back to the model width, giving M_fused.
8. Decoding §3.3 eq. 28 eq. 29 eq. 30 eq. 31. A transformer decoder takes the embedded response prefix and uses M_fused as keys and values. The loss is the negative log-likelihood of the gold response.
9. Training objective §3.4 eq. 32. The loss is the classification loss, plus the two ELBO terms, plus the response loss, with all three weights set to 1 §3.4. Training uses a posterior that conditions on the emotion representations and is pulled towards the prior §3.3 eq. 16 eq. 17. At inference the group is chosen from the classifier, since the gold emotion is unknown §4.

Pseudocode of one training pass, following the paper's description:

# MIME, training loss for one batch (the paper's description)  [§3]
# Input:  context (B, k) word ids; speakers (B, k) user/agent ids
#         response (B, t) word ids; label (B,) emotion id in 0..31
# Output: scalar loss, all three weights 1  [§3.4]

def mime_loss(context, speakers, response, label):
    # Context = word embedding + speaker embedding  [§3.2] [eq. 1]
    inputs = word_emb(context) + speaker_emb(speakers)  # (B, k, D_emb)
    # Transformer encoder over the context  [§3.2] [eq. 2]
    H = context_encoder(inputs)  # (B, k, D_h)
    # Context vector is the CTX token state  [§3.2] [eq. 3]
    c = H[:, 0]  # (B, D_h)
    # Scores of c against the 32 emotion embeddings  [§3.2] [eq. 4-5]
    scores = emotion_match(c)  # (B, 32)
    # Cross-entropy on the user's emotion  [§3.2] [eq. 6]
    loss_cls = cross_entropy(scores, label)
    # Latent z_g sampled from the prior for each group  [§3.3] [eq. 7-8]
    z_pos = prior_sample(c, "pos")  # (B, D_h)
    z_neg = prior_sample(c, "neg")  # (B, D_h)
    # Softmax weights over each group's emotions  [§3.3] [eq. 9]
    d_pos = softmax(fc_pos(z_pos))  # (B, 13)
    d_neg = softmax(fc_neg(z_neg))  # (B, 19)
    # Group representation: weighted sum of emotion embeddings  [§3.3] [eq. 10]
    e_pos = d_pos @ emb_pos  # (B, D_h)
    e_neg = d_neg @ emb_neg  # (B, D_h)
    # Mimicry: m follows the polarity of the gold label  [§3.3] [eq. 18-19]
    m = where(is_positive(label), e_pos, e_neg)  # (B, D_h)
    m_tilde = where(is_positive(label), e_neg, e_pos)  # (B, D_h)
    # Training: KL between posterior Q and prior P  [§3.3] [eq. 16-17]
    kl = kl_posterior_prior(c, label, "pos")
    kl = kl + kl_posterior_prior(c, label, "neg")
    # Append m to each context state, then encode  [§3.3] [eq. 20-23]
    M = response_encoder(concat_last(H, m))  # (B, k, D_h)
    M_tilde = response_encoder(concat_last(H, m_tilde))  # (B, k, D_h)
    # Gate mixes the two refined contexts  [§3.3] [eq. 24-27]
    M_cat = concat_last(M, M_tilde)  # (B, k, 2 D_h)
    gate = sigmoid(fc_contrib(M_cat))  # (B, k, 2 D_h)
    M_fused = fc_fused(gate * M_cat)  # (B, k, D_h)
    # Teacher-forced decoding, M_fused as keys and values  [§3.3] [eq. 28-30]
    logits = response_decoder(
        word_emb(shift_right(response)), M_fused, M_fused)
    # Negative log-likelihood of the gold response  [§3.3] [eq. 31]
    loss_resp = nll(log_softmax(logits), response)
    # Total loss: classification + KL + response  [§3.4] [eq. 32]
    return loss_cls + kl + loss_resp

The prior_sample helper follows §3.3 eq. 11-15. The kl_posterior_prior helper is left undefined because the paper specifies the posterior only partly (see Pitfalls).

Hyperparameters

Results

Author-reported on EmpatheticDialogues, with 24,850 dyadic conversations and an 8:1:1 split defined by the dataset authors §4.1. Baselines are Multi-TR and MoEL §4.2. Human ratings use 128 test samples (four per label), three annotators, and a 1-5 scale §4.3. BLEU is kept only as a reference because the authors consider it weak for open-domain dialogue §4.3. Hardware and training time are not stated.

Overall results tab. 2:

Human A/B test, MIME as A tab. 3:

Positive and negative user emotions, human ratings tab. 4:

Ablation tab. 5. Rows are as the table lists them:

Emotion classification accuracy, top-1 / top-2 / top-5 §5.4: MoEL 38% / 63% / 74%, MIME 34% / 58% / 77%. The authors attribute MIME's lower top-1 score to the shared emotion embeddings.

Paper vs code

Code: declare-lab/MIME at commit 7c34f3ae6dc8f8b9e6fb89b5bfa016fbaa445018, MIT. The paper's abstract links the repository. We read the code but did not run it, and we did not reproduce any number.

Matches the paper:

• Emotion groups are the same 13 positive and 19 negative sets as in tab. 1 trainer.py VAE_noEmo_posterior.py data_loader.py.
• At inference the group comes from the classifier's top emotion, as §4 describes trainer.py.
• The gate has the form the paper gives: a sigmoid over the concatenation, then a linear map back to D_h complex_res_gate.py.
• The context vector is the CTX token's state, and the CTX token is the vocabulary id 6 config.py data_loader.py.

Differs from the paper:

• Learning rate. The paper states Adam with learning rate 0.0001 §3.4. The code builds a Noam schedule (factor 1, warmup 8000, model size 300), and the lr setting is used only when noam is off trainer.py common_layer.py. The --noam flag is store_true with default True, so it cannot be turned off from the command line config.py. From the formula, the rate peaks near 6.5e-4 at step 8000 and then decays. This is our arithmetic, not a value the code prints.
• Label smoothing. The paper gives a plain categorical cross-entropy for the response loss eq. 31. The code uses label smoothing of 0.1 [code: model/trainer.py#L95-L96], which can be turned off only by editing the code, since the flag is store_true with default True config.py.
• Reconstruction term. The paper's ELBO has a reconstruction term E[log P(e_g | z_g, C)] eq. 16. The code's training loss adds only the KL terms, and the classification loss trainer.py. The paper does not define the likelihood in the reconstruction term.
• Which emotions drive the decoder during training. The paper says the posterior Q is used for sampling during training only §3.3. The code computes the decoder's emotion-refined contexts from the prior samples trainer.py, then computes the posterior only for the KL terms trainer.py. The posterior sample is not used afterwards trainer.py.
• Posterior conditioning. The paper conditions Q on e_g, the concatenation of e_g to c §3.3. The code instead passes the mean-pooled M and M_tilde, computed from the prior samples, into the posterior [code: model/trainer.py#L189] VAE_noEmo_posterior.py.
• Negative-branch variance. prior() and posterior() each return the positive log-variance in the fourth slot, where the negative one belongs VAE_noEmo_posterior.py VAE_noEmo_posterior.py. The negative log-variance heads are computed and discarded [code: model/VAE_noEmo_posterior.py#L47] VAE_noEmo_posterior.py. Both the negative sampling and the negative KL therefore use the positive log-variance.
• KL prior at validation. During training the KL is taken against the learned prior. At validation it is taken against N(0, I) [code: model/trainer.py#L193-L196] VAE_noEmo_posterior.py. Validation perplexity returns only the negative log-likelihood, with no KL term and no classification term trainer.py. Model selection therefore ignores the KL.
• Reduction. The KL is summed over batch and dimension [code: model/VAE_noEmo_posterior.py#L159]. The response loss is a sum over tokens and batch common_layer.py. The classification loss is a mean trainer.py. The paper's unit weights therefore apply to differently scaled terms.
• Positional information. The paper writes the context input as word, positional and speaker embeddings summed eq. 1. The code adds a fixed sinusoidal timing signal after a linear projection of the word-plus-speaker input trainer.py transformer_mulexpert.py.
• Gradient clipping. The config defines a max norm of 2.0 [code: utils/config.py#L37], but no code path calls a clipping function.
• Patience. The paper's patience is 2 §3.4. The code breaks when the counter exceeds 2, so after three non-improving checks main.py. Checks run every 1000 steps, and no checkpoint is saved before step 10000 main.py main.py.
• BLEU decoding. The paper does not say which decoding gives the Table 2 BLEU. The code reports three values: greedy, beam with size 5, and top-3 sampling common_layer.py trainer.py.
• Test-time models. The README points to pretrained weights on an external drive. We did not fetch them.

No discrepancies found in: the 32-label set, the batch size, the hidden size of 300, and the beam size of 5.

Pitfalls

• Learning rate. A literal reading of §3.4 (constant Adam at 0.0001) is not what the code runs, and the code's default cannot be switched off. Decide which schedule you use and report it.
• Reductions and weights. Keep the paper's unit weights only if you use the same reductions. The code sums the KL over batch and dimension and sums the token loss, while the classification loss is averaged, so changing one reduction changes the balance the paper's weights assume.
• Negative-branch variance. Copying the code's prior() and posterior() reproduces the bug. Use the negative log-variance head for the negative group.
• Ambiguous posterior. Equation 10 defines e_g from z_g through d_g eq. 9 eq. 10, yet the posterior is conditioned on e_g §3.3. Q can therefore be read as depending on the sample it is meant to produce. The code sidesteps this by using M and M_tilde. Neither reading is confirmed by the paper.
• Undefined likelihood. The reconstruction term P(e_g | z_g, C) eq. 16 has no stated form. The code has no reconstruction term at all.
• Training versus test. In the code, the decoder sees prior samples in both training and inference. The paper's text describes the posterior as the training sample, so an implementation that follows the text differs from the released code.
• Group choice at test. The mimicry group comes from the classifier's top emotion §4, which the paper reports at 34% top-1 accuracy §5.4. The paper does not measure how often the group itself is right.
• Ablation attribution. Table 5 has no row that removes grouping alone. The paper's "effect of grouping" (empathy down 0.73) comes from comparing the full model with the row that also removes mimicry §5.2 tab. 5. Read the grouping effect as the combined effect of removing both.
• Statistical support. The human evaluation uses 128 samples and three annotators. The paper reports no significance test. Fluency is lower than MoEL's tab. 2 §5.1.
• Surprise label. The authors treat surprised as positive, and they acknowledge that negatively surprised users are misread §5.4.
• Unstated values. Layers, heads, key depth, filter size, dropout, word embedding size, GloVe version, decoding length, and clipping are not stated. The code values are in the Hyperparameters table. Use them only as a reference, and say so.
• Numbers not checked. We did not run the code and did not verify the parameter counts in tab. 2.

Lineage

• Builds on: MoEL, which the paper's emotion classifier and emotion-specific decoding follow §3.2 §2. The paper's VHRED reference is the source of the sampling idea §3.3. Neither is linked here, since no card exists for them.
• Compared against: Multi-TR, following the classification-by-encoder setup of the EmpatheticDialogues authors §4.2, and MoEL §4.2.
• Superseded by: not known from content in this wiki.