Adversarial Learning for Neural Dialogue Generation

In plain words

Chatbots trained the standard way learn to predict the most likely next reply. The most likely reply is often something bland like "I don't know", so these bots end up dull, generic and repetitive.

This paper borrows the idea of the Turing test. Besides the bot that writes replies (the generator), it trains a second network (the discriminator) whose only job is to tell human replies from machine replies. The generator is rewarded whenever the discriminator mistakes its reply for a human one. Because a reply is a sequence of discrete words, the reward cannot simply be backpropagated; it is treated as a reinforcement-learning signal instead.

Two tricks make this work. First, instead of giving every word in a reply the same reward, the system estimates how good each partial reply is by finishing it several times at random and asking the discriminator about the completions, so individual words get credit or blame. Second, training alone on the discriminator's feedback falls apart, so the generator is also regularly shown real human replies and trained on them the ordinary way.

The same idea serves as a test: train a fresh classifier to spot machine replies and measure how often the bot fools it. The paper shows this score can be gamed (random-looking replies fool weak classifiers) and adds sanity checks for it. In human judgments, the adversarially trained bot was preferred over a strong baseline.

TL;DR

A Seq2Seq-with-attention response generator G is fine-tuned with REINFORCE, where the reward is the probability, given by a hierarchical-LSTM discriminator D, that the (history, response) pair is human-generated. REGS gives per-token rewards via 5 Monte Carlo rollouts per prefix (or a discriminator trained on partial sequences). Training alternates 5 D-steps with 1 G-step, and each G-step is followed by an MLE update on the gold response, without which training diverges. The paper also defines AdverSuc (1 − evaluator accuracy) plus two reliability checks (ERE, machine-vs-random). Results are on OpenSubtitles; the MC variant is expensive (8 GPUs to parallelize sampling).

Problem

MLE-trained Seq2Seq dialogue models produce dull, generic, repetitive and short-sighted responses §1. Earlier RL approaches use hand-designed rewards (ease of answering, informativeness, coherence; keyword retrieval confidence), which cannot capture every aspect of a good reply §1. The paper instead asks the generator to produce responses indistinguishable from human ones, with a learned discriminator as the judge §1. Text is discrete, so the discriminator's error cannot be backpropagated into the generator directly, which motivates a policy-gradient formulation §2.

Setup: dialogue history x is approximated by concatenating the two preceding utterances (more context gave very small gains for Seq2Seq); the model generates a response y = \{y_1, \dots, y_T\} §3, footnote 1.

Method

Components §3.1

• Generator G: a Seq2Seq encoder-decoder RNN that defines the policy \pi(y \mid x); the source is encoded by a recurrent net and each target token is produced by a softmax. Attention is used ("Seq2Seq model with an attention mechanism") §3.3.
• Discriminator D: a binary classifier over the dialogue \{x, y\}. A hierarchical encoder maps each utterance to a vector with an LSTM, a second sentence-level LSTM maps the sequence of utterance vectors to one vector, and a 2-class softmax outputs Q_+(\{x,y\}) (human) and Q_-(\{x,y\}) (machine) §3.1, footnote 2.
• Critic (baseline): a separate network that estimates the expected reward of the state (the dialogue history) under the current policy, using a hierarchical network that outputs a scalar, trained with mean squared error against the observed reward §3.1, footnote 3.

Objective (vanilla adversarial REINFORCE) §3.1. G maximizes the expected discriminator score of its own samples:

J(\theta) = \mathbb{E}_{y \sim p(y \mid x)} \big( Q_+(\{x,y\}) \mid \theta \big) eq. 1

with the likelihood-ratio gradient, one sample per input:

\nabla J(\theta) \approx [Q_+(\{x,y\}) - b(\{x,y\})]\, \nabla \log \pi(y \mid x) = [Q_+(\{x,y\}) - b(\{x,y\})]\, \nabla \sum_t \log p(y_t \mid x, y_{1:t-1}) eq. 2

D is updated in parallel, with the human dialogue containing x as positive and the sampled one as negative §3.1.

REGS: reward for every generation step §3.2. In eq. 2 every token gets the same reward. REGS gives each prefix Y_t = y_{1:t} its own score Q_+(x, Y_t) and baseline b(x, Y_t):

\nabla J(\theta) \approx \sum_t \big(Q_+(x,Y_t) - b(x,Y_t)\big)\, \nabla \log p(y_t \mid x, Y_{1:t-1}) eq. 3

Two ways to get Q_+(x, Y_t) for a partial sequence §3.2:

1. Monte Carlo search. From prefix s_P, keep sampling until decoding ends; repeat N = 5 times; the mean D score of the N completions is the reward for s_P (similar to SeqGAN). This costs a full rollout set per prefix: a length-20 response needs 5×20 = 100 sampled sequences; a 128-example batch took about 1 minute on one GPU, so sampling was parallelized over 8 GPUs §3.2, footnote 4.
2. Partial-sequence discriminator. Train D on prefixes of human (\{y^+_{1:t}\}) and machine (\{y^-_{1:t}\}) responses. Using all prefixes overfits because early tokens appear in many examples, so per sequence pair only one random positive prefix and one random negative prefix are used (as in AlphaGo's value network training). Much faster than MC, but D becomes less accurate; MC performs better when training time is not a constraint §3.2.

Teacher forcing §3.2. Updating G only through eq. 1 is unstable for both vanilla REINFORCE and REGS: perplexity explodes after a few hours, even from a pre-trained Seq2Seq start. Once G degrades and D recognizes it easily, G only learns what is bad, not what is good. The fix is to also update G on human responses:

• Main variant: D assigns reward 1 (or another positive value) to the human response and G updates on it, which is identical to an ordinary MLE step. Training thus alternates adversarial and MLE updates; the paper relates this to professor forcing §3.2.
• Alternative: D scores the human response itself and G updates on it only if that reward exceeds the baseline, giving per-example, non-negative weights §3.2.

Algorithm: Adversarial REINFORCE training  [fig. 1]
Pre-train G with MLE (Seq2Seq + attention) on OpenSubtitles          [§3.3]
Pre-train D: positives = human data; negatives = decoded training data,
  half from beam search with MMI reranking, half from sampling       [§3.3]
for number of training iterations:
  repeat D-steps (= 5) times:
    sample (X, Y) from real data; sample Ŷ ~ G(·|X)
    update D with (X, Y) as positive and (X, Ŷ) as negative
  repeat G-steps (= 1) times:
    sample (X, Y) from real data; sample Ŷ ~ G(·|X)
    compute reward r for (X, Ŷ) with D   (whole-sequence or REGS)
    update G on (X, Ŷ) with r            (eq. 2 or eq. 3)
    teacher forcing: update G on (X, Y)  (MLE)

Data and decoding heuristics, used for both the adversarial model and the Seq2Seq baselines §3.3:

1. Drop training examples whose response is shorter than 5 tokens (said to improve quality significantly; short responses could get a separate model) §3.3, footnote 5.
2. Per-example learning rate weighted by the response's average tf-idf (each sentence is a document, stop words removed). With tf-idf scores t_1, \dots, t_N in a batch and base rate lr, example i uses N \cdot lr \cdot t_i / \sum_{i'} t_{i'}; each sentence's tf-idf is capped at L = 3 times the batch minimum §3.3, footnote 6.
3. Beam search with an intra-sibling ranking penalty for N-best diversity §3.3.
4. Penalize word types (excluding stop words) already generated, to avoid repetition and contradiction §3.3.

Adversarial evaluation §4. A separate evaluator, sharing no parameters with training, is trained to label dialogues human (positive) or machine (negative) and tested on held-out data.

• AdverSuc = 1 − evaluator accuracy; higher is better, 0.5 means indistinguishable §4.1.
• Evaluator reliability error (ERE): the evaluator's mean absolute deviation from the gold AdverSuc over four constructed tasks, equally weighted §4.2:
  1. human vs. human → gold AdverSuc 0.5;
  2. machine vs. machine → 0.5;
  3. human vs. dialogues with random responses → 0;
  4. human vs. dialogues whose response is replaced by the utterance that follows the true response → 0.
• Machine-vs-random accuracy: accuracy of telling the model's responses from randomly sampled human responses; higher is better. It catches models that fool the evaluator just by being random §4.3.

Evaluator variants compared §5.1: SVM on unigrams (SVM-Light); "Concat Neural" (softmax over concatenated sentence representations); "Hierarchical Neural" (same structure as D); and an SVM over unigrams, the hierarchical model's pre-softmax representation, forward likelihood \log p(t \mid s) and backward likelihood p(s \mid t).

Hyperparameters

Results

Author-reported. Data: OpenSubtitles §3.3. The evaluator in tab. 3 is the "Hierarchical Neural" model §5.1. Which adversarial variant (REINFORCE or REGS) is used in the human evaluation is not stated.

Evaluator reliability, lower is better tab. 2:

Adversarial evaluation, higher is better for both columns tab. 3:

Sampling has by far the highest AdverSuc but the lowest machine-vs-random, so its responses are hard to tell apart from random human ones as well; MMI−p(t) shows the same effect to a lesser degree §5.1. Among the baselines, MMI+p(t|s) > MLE-greedy > MLE+BS. Both adversarial models beat the baselines, and REGS is slightly better than vanilla REINFORCE §5.1.

Human pairwise judgments, adversarial model vs. the mutual-information (MMI) system tab. 4:

The authors contrast this with their earlier hand-crafted-reward RL system, which they say improved only multi-turn quality §5.2. They report no clear gain in preliminary machine-translation experiments and conjecture that adversarial training helps most when target entropy is high §6. Sample outputs are in tab. 1, tab. 5 and tab. 6.

Paper vs code

The repository is the first author's dialogue toolkit (Torch7/Lua); its README lists this paper and points to Adversarial/ for both the adversarial RL model and the adversarial evaluator. The paper does not link it.

• Critic architecture. Paper: the critic encodes the history with a hierarchical network and maps it to a scalar §3.1, footnote 3. Code: a single nn.Linear(dimension, 1), initialized to zero, on the generator encoder's last top-layer hidden state, trained with MSE at lr 0.0005 model.lua model.lua atten.lua.
• Per-step baseline. Paper's REGS uses b(x, Y_t) per prefix eq. 3. Code computes one baseline per example from the source only and repeats it across all time steps model.lua.
• Final-token reward in MC mode. The step that emits the end-of-sequence token reuses the MC reward of the last prefix, not the direct D score of the full response generative.lua. The direct D score of the sampled response is used only as the critic's regression target in this mode model.lua.
• REGS strategy 2 (partial-sequence discriminator with one random prefix per sequence) §3.2 is not implemented; only vanilla REINFORCE and MC rollouts exist (-vanillaReinforce, default false = MC) parse.lua.
• Teacher forcing. Code implements only the plain MLE variant (one gold-response update per outer iteration, after the G-steps) model.lua; the baseline-gated alternative §3.2 is absent.
• Gradient scaling. The REINFORCE signal is multiplied by Timeslr (0.5) and divided by the batch size model.lua, and the shared update divides gradients by the batch size again before norm clipping at 5 atten.lua. The MLE teacher-forcing step is not scaled by Timeslr. None of this is in the paper.
• Batching of steps. Each D-step and each G-step reads a new batch from the training file, and the D forward pass in a G-step runs with D in test mode, so D is not updated there model.lua.
• Heuristics from §3.3. The response-length filter and the tf-idf-weighted learning rate are not found in the repository (they may live in unreleased preprocessing). Diverse beam search is available in Decode as the DiverseBS setting README.md.
• Encoder backward placement. In the adversarial generator's model_backward, the backward pass through the encoder sits inside the loop over decoder time steps and overwrites the decoder's recurrent gradient d_output each step generative.lua, whereas the pre-training code runs it once after the decoder loop atten.lua. This looks like a bug; do not copy it.
• Discriminator input. The paper's history is two utterances concatenated §3, footnote 1; the discriminator code treats each dialogue as dialogue_length = 2 segments (source, response) with a shared word-level LSTM and a sentence-level LSTM dis_model.lua dis_parse.lua. Consistent with the paper.

Pitfalls

• Teacher forcing is not optional. Without interleaved MLE updates, perplexity blows up after a few hours §3.2. Even with it, the README warns that the model can still explode after tens of hours (it surfaces as a Torch sampling error) and recommends lowering Timeslr README.md. Monitor perplexity on gold data.
• MC REGS is expensive. It needs N rollouts for each prefix, i.e. about N·T full samples plus D evaluations per response; the authors needed 8 GPUs to make it tractable §3.2, footnote 4.
• The relative weight of adversarial and MLE updates is unspecified in the paper. In code it is set by Timeslr and a double division by batch size (see Paper vs code). Re-tune it in any re-implementation.
• Reward is a probability in [0, 1]. It is Q_+ from a softmax, not a log-probability; with a strong D, rewards for all samples approach 0, which is the collapse that teacher forcing guards against §3.2.
• AdverSuc can be gamed by randomness. Sampling decoding reaches AdverSuc 0.372 while being poor tab. 3. Always report machine-vs-random alongside AdverSuc, and check evaluator quality with ERE; an evaluator that outputs random or constant labels scores 0.5 AdverSuc §4.2 §4.3.
• Adversarial evaluator is separate. It must not share parameters with the training discriminator §4; otherwise the model is graded by the judge it was trained to fool.
• Unstated pieces: the exact OpenSubtitles split and size, number of adversarial iterations, generator learning rate during adversarial training, which variant was used for human evaluation, and how the baseline b(x, Y_t) is computed per step beyond "similar to vanilla REINFORCE" §3.2.
• Heuristic confounds. The §3.3 heuristics (length filter, tf-idf weighting, diversity penalties) are applied to the adversarial model and the Seq2Seq baselines alike §3.3, but they change outputs substantially, so comparisons with other papers' baselines are not like-for-like.
• Does not transfer to MT in the authors' preliminary tests §6.

Lineage

• Builds on: Seq2Seq with attention (Sutskever et al. 2014; Bahdanau et al.; Luong et al.); GANs (arXiv:1406.2661); SeqGAN (arXiv:1609.05473), which also uses policy gradient with MC rollouts for a sequence GAN; the authors' deep RL dialogue model (arXiv:1606.01541), whose hand-crafted rewards are replaced by a learned discriminator; MIXER-style RL for sequences with a learned baseline (arXiv:1511.06732).
• Adversarial evaluation follows Bowman et al. (arXiv:1511.06349) and Kannan & Vinyals.
• Compared against: MLE Seq2Seq (greedy, beam search, sampling) and MMI decoding (arXiv:1510.03055) tab. 3 tab. 4.