---
kind: concept
slug: dialogue-generation
title: Dialogue generation
summary: >-
  Generating a reply from the conversation so far. Plain maximum-likelihood Seq2Seq gives generic replies; decoding changes, extra inputs (commonsense, strategy labels, database results), reward-based training and LLM adaptation try to improve on that. Results are author-reported on each paper's own data. Trade-off: reward-based training, labelled turns and LLM systems cost more.
tags: [dialogue, text-generation, seq2seq, rlhf, negotiation, llm, task-oriented, empathy, recommendation, commonsense]
papers: ["1606.01541", "1701.06547", "1705.05414", "1808.07042", "1808.09637", "1905.04071", "2009.14306", "2109.05739", "2109.14739", "2201.06657", "2201.08239", "2202.06417", "2209.14375", "2305.16367", "2402.18013"]
related: [adversarial-text-generation, adversarial-evaluation, conversational-question-answering, dialogue-evaluation, rlhf, retrieval-augmented-generation, in-context-learning, parameter-efficient-fine-tuning, policy-gradient, commonsense-knowledge-injection, conversational-recommendation, emotion-recognition-in-conversation, controllable-text-generation, multi-task-learning, hallucination-mitigation, tool-use, noisy-channel-model, knowledge-base-question-answering, attention-mechanism, decoding-strategies, contrastive-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

A dialogue model reads the conversation so far and writes the next response. The
paper's setting approximates the history with the two preceding utterances, which
it found was enough for a Seq2Seq model; more context gave very small gains
[arxiv:1701.06547 §3, footnote 1]. The shared problem is that models trained by
maximum likelihood tend to produce bland, repetitive and short-sighted replies
[arxiv:1701.06547 §1]. One paper traces the repetition to token states that are
too similar to one another [arxiv:2202.06417 §1]. Most variants below try to make
replies better as a whole, not just more probable token by token. Some variants instead change what
the model is conditioned on: a commonsense vector, a strategy label, or a
database result. The LLM adaptation variant changes how a large pre-trained model
is used rather than the training objective of a Seq2Seq model.

Open-domain chat is the setting for the first two variant families, for the
hand-set reward variant, for LaMDA and for the contrastive variant, which is
also tested on two chat corpora. The survey
separates it from task-oriented systems, where generation is one module of a
pipeline with understanding and dialogue management [arxiv:1905.04071 §3.3], and
from interactive question answering [arxiv:1905.04071 §5.1]. Two variants below
work on task-oriented data: PPTOD folds that pipeline into a single model, and
the KV retrieval net reads a knowledge base inside the decoder
[arxiv:1705.05414 §2.3]. The empathetic and recommendation variants apply the
same generation step to other settings. The survey's
generative chatbot models are seq2seq networks, with HRED adding a context
encoder over earlier turns [arxiv:1905.04071 §4.2.1]. The Sparrow paper targets
information-seeking dialogue, where success is easier to define than in open
chit-chat [arxiv:2209.14375 §1]. The negotiation paper works in a closed,
goal-driven task instead: two agents agree a price for a real listing
[arxiv:1808.09637 §2].

Fifteen papers with cards cover this concept: eleven report their own experiments,
three are surveys, and one is a position paper with no experiments
[arxiv:2305.16367 §1]. Each reports on its own setup. The adversarial results use one
dataset (OpenSubtitles) and one group of authors [arxiv:1701.06547 §3.3]. The
hand-set reward results come from simulated two-agent dialogues scored by the
authors' own rules, plus human judgments on 500 single-turn and 200 multi-turn
items [arxiv:1606.01541 §5.2]. The LaMDA results come from crowdworker labels on
the authors' fixed evaluation sets, with human baselines shown only in figures
[arxiv:2201.08239 §5] [arxiv:2201.08239 §7]. The contrastive results come from
200 contexts per dataset rated on a five-point scale by five annotators, on LCCC
and DailyDialog [arxiv:2202.06417 §5] [arxiv:2202.06417 tab. 3]. The
CoQA results use a passage-grounded question-answering dataset with gold history,
where the best system is a span reader rather than a generator [arxiv:1808.07042 tab. 7].
The negotiation results come from a new dataset of buyer and seller chats and from a
deployed bot judged by crowd workers [arxiv:1808.09637 §2] [arxiv:1808.09637 §4.4].
The PPTOD results use MultiWOZ for end-to-end and state tracking [arxiv:2109.14739 tab. 2] [arxiv:2109.14739 tab. 4]. The
empathetic results use EmpatheticDialogues [arxiv:2109.05739 §2.3], and the
recommendation results use the authors' own 1,001 human-human dialogues
[arxiv:2009.14306 §3]. The KV retrieval results come from the authors' own
in-car corpus, scored on its test dialogues [arxiv:1705.05414 §5.1]. The survey reviews other studies and runs no experiments
of its own [arxiv:1905.04071 §2.2]. The chatbot survey likewise reports no experimental
results of its own [arxiv:2201.06657 §3]. The LLM survey likewise reports no new
results [arxiv:2402.18013 §1]. The Sparrow results come from a 70B-scale agent
judged by paid raters on the authors' own test prompts [arxiv:2209.14375 §3.1].
The method papers do not compare against one another, so their numbers are not
one benchmark.

## Variants

### Maximum-likelihood Seq2Seq with decoding choices

A Seq2Seq encoder-decoder with attention, trained by maximum likelihood. The
paper decodes it three ways, which are the baselines in its comparison: greedy,
beam search, and sampling [arxiv:1701.06547 tab. 3]. Sampling gets the highest
automatic adversarial score of the baselines, but its replies are the least
distinguishable from random human replies [arxiv:1701.06547 tab. 3, §5.1]. This
variant needs no extra training and is the starting point for the others.

### Hierarchical Seq2Seq (HRED)

A generative model with a second encoder that runs over the sequence of earlier
turns, so the decoder sees more than the last utterance [arxiv:1905.04071 §4.2.1,
fig. 7]. The survey describes it as a seq2seq extension and does not report a
head-to-head number against the models above. The paper's own result on context
length, that two previous utterances were enough [arxiv:1701.06547 §3, footnote 1],
is the only evidence on how much history helps here.

### Passage-grounded answer generation with copy (CoQA baselines)

A conversational answerer reads a passage and the earlier question-answer turns,
then writes the next answer. The paper's seq2seq model puts the passage, the
history and the current question into one input for a bidirectional LSTM encoder,
and an LSTM decoder with attention writes the answer. The copy variant (PGNet) can
copy words from the passage [arxiv:1808.07042 §5.1]. On
the test set, PGNet reaches 44.1 overall F1 and the plain seq2seq baseline 26.3,
against 52.6 for DrQA, a span reader [arxiv:1808.07042 tab. 7]. The best system in
that table is Augmented DrQA, a span reader with extra yes and no tokens, at 65.4,
and the combined DrQA+PGNet reaches 65.1 [arxiv:1808.07042 tab. 7]. A third of the
gold answers are not spans of the passage [arxiv:1808.07042 §4.1], so the generator
variants are the ones that can produce them. Card: [CoQA: A Conversational Question Answering Challenge](/papers/1808.07042).

### Utterance selection (retrieval)

Instead of generating text, the system ranks candidate replies from a database
and returns the best one [arxiv:1905.04071 §4.2.2]. The survey's metric for this
family, Recall@k, has a known gap: other candidates may also be correct, so a
single reference undercounts good replies [arxiv:1905.04071 §4.3.2]. No paper
with a card here compares retrieval against generation. The chatbot survey finds
few learned retrieval models, and a gap between large research models and
industry bots [arxiv:2201.06657 §6] [arxiv:2201.06657 §7].

### Mutual-information reranking (MMI)

A baseline that the paper compares against, in two rows of its table: MMI+p(t|s)
and MMI-p(t) [arxiv:1701.06547 tab. 3]. The paper also uses MMI reranking to
produce half of the negative examples for its discriminator [arxiv:1701.06547 §3.3].
The paper's exact definition of the two rows is not restated here; the original
method is not in this wiki and can be found at
[arXiv:1510.03055](https://arxiv.org/abs/1510.03055). The survey lists maximum
mutual information among the ways to push generative models towards more diverse
replies, alongside reinforcement rewards, latent variables and topics
[arxiv:1905.04071 §4.2.1]. That is a training objective; the paper's MMI row is a
reranking step.

### Contrastive training and contrastive search (SimCTG)

The model is fine-tuned on the usual next-token loss plus a contrastive term over
the last-layer states of all tokens in a sequence. The term has no extra weight,
and it pushes apart the states of different positions [arxiv:2202.06417 §3.1].
The decoder then takes the top-k tokens and picks the one that trades model
probability against a penalty for similarity to the text so far
[arxiv:2202.06417 §3.2]. The paper's dialogue setting uses a Chinese GPT-2 on
LCCC, with greedy, beam, nucleus and contrastive decoding (k = 5, alpha = 0.6)
[arxiv:2202.06417 §5]. Against nucleus sampling on the same SimCTG model, the
contrastive decoder scores 3.32, 3.96 and 3.13 on coherence, fluency and
informativeness on LCCC, against 2.84, 3.58 and 2.72. On DailyDialog it scores
3.73 and 3.85 on coherence and fluency, against 2.75 and 3.59, and 3.46 against
3.39 on informativeness, which is not marked significant. The paper marks the
starred cells as significant against greedy, beam and nucleus decoding
[arxiv:2202.06417 tab. 3]. Human reference scores are 3.42, 3.76 and 3.20 on
LCCC [arxiv:2202.06417 tab. 3]. The contrastive search also beats nucleus sampling
on the plain MLE model for LCCC on all three criteria, and the paper says the
Chinese model is already isotropic, which explains why [arxiv:2202.06417 §5].
For English GPT-2 small, the same search on the MLE model trails the SimCTG model
well behind [arxiv:2202.06417 tab. 1]. Decoding cost is close to beam search for
small beam width and k, and the paper reports it faster above six
[arxiv:2202.06417 §6.4]. Card: [A Contrastive Framework for Neural Text Generation](/papers/2202.06417).

### Adversarial REINFORCE with a whole-reply reward

A discriminator labels a dialogue as human or machine. The generator is trained
with policy gradient to raise the discriminator's probability that its sampled
reply is human, with one reward per reply [arxiv:1701.06547 §3.1, eq. 2]. The
paper reports that this beats the MMI baseline in human pairwise judgments, 0.62
wins to 0.18 losses on single-turn items and 0.72 to 0.10 on multi-turn items
[arxiv:1701.06547 tab. 4]. Card: [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547).

### Adversarial REINFORCE with per-step rewards (REGS)

The same loop, but each prefix of the reply gets its own reward, estimated from
N = 5 random completions scored by the discriminator [arxiv:1701.06547 §3.2, eq. 3].
On the automatic adversarial score it is slightly ahead of the whole-reply variant
(0.098 against 0.080) [arxiv:1701.06547 tab. 3]. The paper does not say which of
the two was used for the human evaluation [arxiv:1701.06547 §5.2].

### Hand-set reward policy gradient in two-agent simulation

The model is first trained by maximum likelihood, then moved towards a reward on
whole dialogues. The reward is a weighted sum with weights 0.25, 0.25 and 0.5
[arxiv:1606.01541 §3.4, eq. 4]. Its three terms are the likelihood of a fixed list
of eight dull replies, where lower is better [arxiv:1606.01541 §3.4, eq. 1]; a
penalty on repetition between consecutive turns of one agent [arxiv:1606.01541
§3.4, eq. 2]; and forward and backward likelihood for coherence [arxiv:1606.01541
§3.4, eq. 3]. Training starts from a mutual-information model. Two copies of the
policy then talk in simulation, with dialogues growing from two turns to five and
five candidate replies sampled per step [arxiv:1606.01541 §4.2] [arxiv:1606.01541
§4.3] [arxiv:1606.01541 §4.4]. Author-reported: simulated dialogues average 4.48
turns, against 3.40 for the mutual-information model and 2.68 for maximum-likelihood
Seq2Seq [arxiv:1606.01541 tab. 2]. On five-turn human comparisons the reinforced
model wins 0.72 and loses 0.12 [arxiv:1606.01541 tab. 5]. On single-turn general
quality it wins 0.40 and loses 0.36, which the authors call not a significant
boost, and its BLEU is 1.28 against 1.44 for the mutual-information model
[arxiv:1606.01541 tab. 5] [arxiv:1606.01541 §5.2]. The paper ships no code, and the
pseudocode in this concept is our reading of it. Card: [Deep Reinforcement Learning for Dialogue Generation](/papers/1606.01541).

### Rule-conditioned RLHF with evidence and reranking (Sparrow)

A 70B-scale agent prompted from Chinchilla, fine-tuned with human preferences
and a separate reward model per rule. Rules are short, single-issue statements, such
as a ban on threatening statements [arxiv:2209.14375 §2.1]. A rule model estimates the
probability that a reply breaks each rule [arxiv:2209.14375 §2.5], and the agent
is trained with advantage actor-critic against a KL penalty to its start point
[arxiv:2209.14375 §2.8]. For factual turns the agent issues web searches and
shows the excerpt to raters [arxiv:2209.14375 §2.4]. At answer time it samples
eight candidates (four with evidence) and keeps the best under a score that
multiplies a preference term by the geometric mean of the rule probabilities
[arxiv:2209.14375 §2.6, eq. 1]. The authors report that the RL agent with eight
samples reached 0.78 on supported-and-plausible evidence replies, against 0.61
for always searching with the prompted base model [arxiv:2209.14375 §3.2], and
0.82 on free-dialogue replies rated true or mostly true, against 0.63 for the
base model that never searches [arxiv:2209.14375 §3.3]. Probing found a targeted rule broken in 8% of
conversations [arxiv:2209.14375 §1]. Card: [Improving alignment of dialogue agents via targeted human judgements](/papers/2209.14375).

### Coarse dialogue acts with a retrieval generator (negotiation)

The dialogue is split into three modules. A rule-based parser maps each
utterance to a coarse act, an intent plus arguments such as `propose(price=...)`
[arxiv:1808.09637 §3.2] [arxiv:1808.09637 §3.3]. A manager picks the next act,
either by supervised sequence-to-sequence learning, by policy gradient on a
reward (utility, fairness or length), or by a hybrid that takes intents from a
trigram model and fills the arguments with rules [arxiv:1808.09637 §3.4]
[arxiv:1808.09637 §4.2]. A retrieval generator then picks a human-written
template that matches the act and the context [arxiv:1808.09637 §3.5]. Because
the strategy lives in the act, a developer can set it by data, reward or rules.
Reinforcement learning on acts kept human-likeness on the authors' CraigslistBargain
data at 2.8 to 3.0 out of 5, against 3.3 for supervised acts and 4.3 for humans
[arxiv:1808.09637 tab. 7]. The same reinforcement learning applied to words
gave 1.7 to 1.9, and the three most frequent sentences made up 81.6% to 100%
of the utterances in the word-level models [arxiv:1808.09637 §4.4]. The paper
reports these results as author-reported, from crowd-worker chats.
Card: [Decoupling Strategy and Generation in Negotiation Dialogues](/papers/1808.09637).

### Unified task-oriented generation (PPTOD)

One pre-trained T5 model handles each task-oriented sub-task as text generation,
selected by a prompt such as "translate dialogue to belief state:". The model is
pre-trained on eleven human-written corpora, and each corpus is labelled for only
some sub-tasks [arxiv:2109.14739 §3.1] [arxiv:2109.14739 tab. 1]. The response
step is generated from the database result as well, in parallel with the
dialogue action, so one model covers the whole turn after the belief state is
predicted [arxiv:2109.14739 §4.1.1]. The paper
reports the best MultiWOZ 2.0 end-to-end combined score in its table, 102.92 for
PPTOD-base [arxiv:2109.14739 tab. 2]. In low-resource belief-state tracking with
1% of the data, its large model reaches 31.46 joint accuracy against 13.21 for
SOLOIST [arxiv:2109.14739 tab. 5]. For latency, the paper's T5-small ablation
gives 38.70 ms for a cascaded run and 14.17 ms for the single-pass format, both
without database state, on one V100 at batch size 1 [arxiv:2109.14739 tab. 6].
Card: [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](/papers/2109.14739).

### Knowledge-base key attention in the decoder (KV retrieval net)

An LSTM encoder-decoder for in-car task-oriented dialogue, with a second
attention step over the keys of a knowledge base (KB) built for each dialogue.
Each KB row is split into (subject, relation, value) triples, and a key is formed
from the subject and relation words [arxiv:1705.05414 §2.3]. The key scores are
added to the output logits at a canonical token for that key. When the model
emits such a token, it is swapped for the stored value [arxiv:1705.05414 §2.3,
eq. 7-9]. No belief tracker or dialogue-state labels are used, and training is
token-level cross-entropy [arxiv:1705.05414 §1] [arxiv:1705.05414 §5.2]. On the
authors' test dialogues it reaches 13.2 BLEU and 48.0 entity F1, against 6.6 and
43.8 for a rule-based system and 11.0 and 37.0 for Copy Net. Removing the encoder
attention gives 40.9 entity F1 [arxiv:1705.05414 tab. 3]. In real-time chat, its
ratings for fluency, cooperation and human-likeness are above both baselines but
below the human partner, on 120 scenarios [arxiv:1705.05414 tab. 4]. Card:
[Key-Value Retrieval Networks for Task-Oriented Dialogue](/papers/1705.05414).

### Commonsense-conditioned empathetic generation (CEM)

The model infers five commonsense relations about the user's last turn: xReact,
xWant, xNeed, xIntent and xEffect [arxiv:2109.05739 §3.2]. An affective encoder
handles xReact and a cognitive encoder handles the other four. Their outputs are
fused with the dialogue context through a sigmoid gate and an MLP, and the
decoder reads the fused representation [arxiv:2109.05739 §3.3] [arxiv:2109.05739
§3.4]. Training combines reply likelihood, an emotion classification loss and a
frequency-weighted diversity loss [arxiv:2109.05739 §3.6]. Only the last user
turn receives commonsense; earlier turns are encoded without it [arxiv:2109.05739
§3.2]. On EmpatheticDialogues the authors report perplexity 36.11 against 36.93
for the best baseline, and Dist-1 and Dist-2 of 0.66 and 2.99 against 0.47 and
2.10 [arxiv:2109.05739 tab. 2]. In human pairwise tests over 100 sampled pairs
per comparison, the model won on coherence, empathy and informativeness against
MoEL at the paper's significance level. Against MIME it won on coherence and
empathy, and against EmpDG on empathy; the other differences are not significant
[arxiv:2109.05739 tab. 3]. Serving the model means producing commonsense
for each new last turn, which the method needs [arxiv:2109.05739 §3.2].
Card: [CEM: Commonsense-aware Empathetic Response Generation](/papers/2109.05739).

### Strategy-label conditioning (INSPIRED)

Each recommender turn in the data is annotated with a sociable strategy, such as
sharing a personal opinion or encouraging the other person. The dialogue model
writes the strategy as a special token in front of the turn, so the label is
generated along with the reply [arxiv:2009.14306 §6.2]. The model is two GPT-2
small networks, one per speaker [arxiv:2009.14306 §6.1] [arxiv:2009.14306 app. D].
On the authors' 100-dialogue test split, the labelled model has perplexity 8.93
against 9.28 for the unlabelled baseline, and BLEU-4 6.63 against 5.11
[arxiv:2009.14306 tab. 7]. In five-point ratings from 25 participants per model,
the labelled model scores higher on all five aspects [arxiv:2009.14306 tab. 10].
The paper's logistic regression finds four strategies with significant positive
coefficients for recommendation success, but it is a correlational analysis
[arxiv:2009.14306 §5.2] [arxiv:2009.14306 tab. 6]. Choosing among sampled replies
uses a rule that the paper describes [arxiv:2009.14306 §6.2]; the card for this
paper notes that the released code differs in both the number of samples and the
selection rule. Card: [INSPIRED: Toward Sociable Recommendation Dialog Systems](/papers/2009.14306).

### LLM generation with adaptation (survey view)

A large pre-trained language model generates the reply, adapted either by
changing its weights or by prompting it. The survey lists full fine-tuning,
adapters, LoRA (a low-rank update added to a frozen weight matrix), and
instruction tuning as the weight-changing options [arxiv:2402.18013 §2.4]. It
lists in-context learning, chain of thought, and retrieval-augmented generation
in naive, advanced and modular forms as prompt-side options
[arxiv:2402.18013 §2.5]. The survey places open-domain generation in three
families: retrieval, generation, and hybrid [arxiv:2402.18013 §4]. It gives no
measured comparison between these options, and it leaves the rank of LoRA and
the number of in-context examples unstated [arxiv:2402.18013 §2.4.2]
[arxiv:2402.18013 §2.5.1]. Card: [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](/papers/2402.18013).

### Dialogue-tuned LLM with safety filtering and tool grounding (LaMDA)

A decoder-only Transformer, from 2B to 137B parameters, is pre-trained to predict
the next token on about 1.56T words of public dialogue and web text [arxiv:2201.08239
§3] [arxiv:2201.08239 app. E]. Fine-tuning on crowd-labelled examples teaches the
same model to generate replies and to rate them for sensibleness, specificity,
interestingness and safety [arxiv:2201.08239 §6.1]. At inference it samples
candidates, drops those whose predicted safety is below a threshold, and ranks the
rest by three times sensibleness plus specificity plus interestingness
[arxiv:2201.08239 §6.1]. A second fine-tuning stage teaches it to write queries to a
toolset (search, calculator, translator) and to write its reply from the results
[arxiv:2201.08239 §6.2]. Author-reported at 137B: sensibleness 92.3, specificity
79.0, safety 95.2 and groundedness 73.2. The pre-trained model at the same size
scores 80.2, 49.8, 88.0 and 57.9 [arxiv:2201.08239 app. F]. Citation accuracy is
65% [arxiv:2201.08239 §7]. The paper says its gains came from fine-tuning data
under 0.001% of the pre-training data [arxiv:2201.08239 §9]. The authors call
LaMDA a recipe for application-specific versions rather than a finished model, and
the crowd data and weights are not released [arxiv:2201.08239 §9] [arxiv:2201.08239
app. A.2]. Card: [LaMDA: Language Models for Dialog Applications](/papers/2201.08239).

### Role-play framing for false and self-preserving replies (no method)

Not a training or decoding method. The position paper describes a dialogue agent
built on a base LLM as a simulator, the base model plus sampling, that holds many
possible characters at once. Each sampled reply narrows that set
[arxiv:2305.16367 §4] [arxiv:2305.16367 §5]. The paper reads apparent deception as
made-up, good-faith or deliberate falsehood, and apparent self-preservation as a
role played from human-written text [arxiv:2305.16367 §7] [arxiv:2305.16367 §8].
It proposes two behavioural tests: regenerate the same context several times and
check whether the answers vary in meaning, and ask the same question in different
contexts [arxiv:2305.16367 §7]. It reports no experiments, and it covers base
models; whether the framing holds after RLHF is unclear [arxiv:2305.16367 §2]
[arxiv:2305.16367 §8]. Card: [Role-Play with Large Language Models](/papers/2305.16367).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Need a working dialogue model fast, no extra training | Maximum-likelihood Seq2Seq with a decoding choice | Only one training run; the paper's baselines are all of this kind [arxiv:1701.06547 tab. 3] |
| Replies are too generic and training budget allows a reward signal | Adversarial REINFORCE, whole-reply reward | Reported as preferred over the MMI baseline by human judges [arxiv:1701.06547 tab. 4] |
| Same goal, and per-word credit is worth the compute | Adversarial REGS (per-step rewards) | Slightly higher adversarial score than the whole-reply variant [arxiv:1701.06547 tab. 3]; needs 5 rollouts per prefix and 8 GPUs in the authors' setup [arxiv:1701.06547 §3.2, footnote 4] |
| Only the maximum-likelihood model is in reach, and output must be less generic | MMI-style reranking | Reported gains on the adversarial score at the price of machine-vs-random accuracy in one of its forms [arxiv:1701.06547 tab. 3] |
| A fixed database of candidate replies exists | Utterance selection | The survey describes ranking candidates from a database [arxiv:1905.04071 §4.2.2]; judge it with the Recall@k caveat above [arxiv:1905.04071 §4.3.2] |
| Open-ended generation repeats itself, and the model can be fine-tuned with an extra loss and a top-k decoder | Contrastive training and search (SimCTG) | Reported significant human gains over nucleus sampling on LCCC on coherence, fluency and informativeness [arxiv:2202.06417 tab. 3]; the paper recommends alpha from 0.5 to 0.8 and k from 5 to 10 [arxiv:2202.06417 app. E]; decoding scores k candidates per step [arxiv:2202.06417 §3.2]. Its own dialogue results are on one Chinese model, so check that the setting fits |
| Replies depend on more than the last turn | Hierarchical Seq2Seq (HRED) | The survey describes the context encoder over previous turns [arxiv:1905.04071 §4.2.1]; the paper found little gain past two turns [arxiv:1701.06547 §3, footnote 1] |
| Factual, information-seeking dialogue, with a search API and a large training budget | Rule-conditioned RLHF with evidence and reranking (Sparrow) | Reported gains in evidence support and correctness [arxiv:2209.14375 §3.2] [arxiv:2209.14375 §3.3]; needs 64 TPU v3 machines for RL [arxiv:2209.14375 app. E], a search API whose results change over time [arxiv:2209.14375 app. C], and eight samples plus two searches per turn at inference [arxiv:2209.14375 §2.6] |
| Specific behaviour rules must be enforced and violations are the concern | Rule reward model, or rule-based reranking | Rule-conditioned classifier AUC 0.85 against 0.77 for a general safety classifier, on the authors' data [arxiv:2209.14375 §3.5]; RL raised bias scores on two benchmarks, so measure bias too [arxiv:2209.14375 §3.6.1] |
| Closed, goal-driven task where the strategy must be controlled and wording must stay human-like | Coarse-act manager with retrieval generator | Reported act-level RL kept human-likeness where word-level RL collapsed, on the authors' negotiation data [arxiv:1808.09637 tab. 7] [arxiv:1808.09637 §4.4]; needs a parser for the acts and a database of human sentences [arxiv:1808.09637 §3.3] [arxiv:1808.09637 §3.5] |
| Slot-filling task bot with partly labelled dialogue data, where one model should cover intent, state, action and reply | Unified task-oriented generation (PPTOD) | Reported best MultiWOZ 2.0 end-to-end combined score in its table [arxiv:2109.14739 tab. 2]; pre-training uses corpora labelled for only some sub-tasks [arxiv:2109.14739 §3.1]; the survey treats task-oriented systems as a separate class with its own metrics [arxiv:1905.04071 §3.1, §3.3] |
| Latency matters for a task bot on one GPU | PPTOD single-pass format | Reported 14.17 ms against 38.70 ms for cascaded generation, T5-small, no database state, one V100 at batch size 1 [arxiv:2109.14739 tab. 6]; the paper gives no variance for this |
| Task-oriented bot that must state facts from a per-dialogue database (times, places, weather) and should keep no belief tracker | KB key attention in the decoder (KV retrieval net) | Reported entity F1 48.0 against 43.8 for a rule-based system, on the authors' in-car data [arxiv:1705.05414 tab. 3]; needs KB rows split into triples, at most 230 per KB [arxiv:1705.05414 §2.3]; the paper links no code, only a dataset page [arxiv:1705.05414 §1] |
| Replies must fit the user's implied situation, not only the words, and a commonsense model is available | Commonsense-conditioned generation (CEM) | Reported gains in human pairwise coherence and empathy against MoEL [arxiv:2109.05739 tab. 3]; needs five relation inferences for each new last turn [arxiv:2109.05739 §3.2] |
| Recommendation or sociable chat where the strategy should be visible and controllable | Strategy-label conditioning (INSPIRED) | Reported lower perplexity and higher BLEU-4 on 100 dialogues [arxiv:2009.14306 tab. 7]; needs a strategy label on every recommender turn [arxiv:2009.14306 §4.2] [arxiv:2009.14306 §6.2] |
| A capable pre-trained LLM is available and training budget is small | Prompting, with in-context learning or retrieval | The survey lists these as the non-weight-changing options [arxiv:2402.18013 §2.5]; no measured comparison is given there |
| An LLM is available and adaptation is needed on limited hardware | Parameter-efficient tuning (adapters or LoRA) | The survey describes LoRA as training a low-rank update on a frozen model [arxiv:2402.18013 §2.4.2]; the rank and the adapter size are not stated there |
| Passage-grounded conversational QA where answers are often free-form | Span reader with history for mostly-span answers; a copy generator if non-span answers matter | On CoQA test, the best span reader reaches 65.4 F1 against 44.1 for PGNet [arxiv:1808.07042 tab. 7]; a third of gold answers are not spans [arxiv:1808.07042 §4.1] |
| Dialogue data exists, the team can write reward rules and simulate two agents, and a learned discriminator is unwanted | Hand-set reward policy gradient, starting from a mutual-information model | Reported longer simulated dialogues and a 0.72 to 0.12 win rate on five-turn human comparisons [arxiv:1606.01541 tab. 2] [arxiv:1606.01541 tab. 5]; the authors call the rewards insufficient and say BLEU falls [arxiv:1606.01541 §5.2] [arxiv:1606.01541 fn. 2] |
| Open-domain assistant where safety and factual grounding matter, with crowd labels and a large compute budget | LaMDA-style fine-tuning with filtering and tool queries | Reported safety 95.2 and groundedness 73.2 at 137B, against 88.0 and 57.9 for the pre-trained model [arxiv:2201.08239 app. F]; pre-training used 1,024 TPU-v3 chips for 57.7 days at 137B [arxiv:2201.08239 app. D]; the weights and crowd data are not released [arxiv:2201.08239 app. A.2] |
| Judging whether a base-model agent's false or self-preserving statements are role-played rather than its own beliefs | Role-play framing and its two proposed tests (regeneration spread, cross-context consistency) | The paper proposes both tests [arxiv:2305.16367 §7] but reports no measured results [arxiv:2305.16367 §10]; it gives no sample count or metric [arxiv:2305.16367 §7] |
| Target is machine translation | Not established here | The authors report no clear gain in preliminary tests and suspect high-entropy targets matter [arxiv:1701.06547 §6] |

## Measuring replies

Scores are a choice that changes which system looks best, so read the survey's
caveats before picking one:

- **Word-overlap metrics.** The survey reports that BLEU and ROUGE correlate
  poorly with human judgement at utterance level, with ROUGE at Pearson 0.062
  [arxiv:1905.04071 §4.3.1]. ΔBLEU, which weights references by human ratings,
  reaches Spearman 0.484 against BLEU's 0.318 [arxiv:1905.04071 §4.3.1].
- **Learned metrics.** ADEM is trained on human ratings, and RUBER needs no human
  labels. The survey reports that reversing a reply raised the ADEM score in
  48.66% of cases, and a dull dummy reply in 86.93% [arxiv:1905.04071 §4.3.1].
- **Discriminator as score.** The survey reports low agreement between the
  discriminator and human judges: Fleiss π 0.07 on Switchboard, against 0.3 for
  humans among themselves [arxiv:1905.04071 §4.3.1].
- **Human-likeness rating.** The negotiation paper asks raters whether the
  partner showed reasonable human behaviour, on a 1-to-5 scale, after a chat
  with a bot or a human [arxiv:1808.09637 §4.4]. Its results rest on that one
  question, and the evaluation model it uses to measure other outputs is itself
  trained on human-human data [arxiv:1808.09637 §4.4].
- **Rater agreement on rules.** Sparrow reports Krippendorff's alpha of 0.37
  for a general harm rule and 0.53 for specific rules [arxiv:2209.14375 §3.5].
  Its numbers come from paid raters on its own prompts, so they are not
  comparable with the survey's Switchboard figures.
- **Printed formulas in the chatbot survey.** Its F1 formula omits the factor 2
  of the harmonic mean, and its METEOR formula gives no alpha and no penalty term,
  so neither should be implemented as printed [arxiv:2201.06657 §4.3].
- **Perplexity formula in the contrastive paper.** Its gen-ppl is defined as 2 to
  the power of an average of log-probabilities with the sign missing, so the
  standard definition is needed to compare its numbers [arxiv:2202.06417 §4.1.2].
- **Rater agreement on Likert scores.** SimCTG's dialogue annotators reach Fleiss'
  kappa of 0.55 to 0.73 across the two datasets, with informativeness the lowest on
  LCCC [arxiv:2202.06417 tab. 3].
- **Rater agreement on pairs.** CEM's pairwise tests have three raters per pair,
  with Fleiss' kappa between 0.41 and 0.57 for the comparisons it reports
  [arxiv:2109.05739 tab. 3]. Each comparison uses 100 sampled pairs [arxiv:2109.05739 §4.4].
- **Small rating panels.** INSPIRED's five-point ratings come from 25 participants
  per model, and its text gives no statistical test for the preference result
  [arxiv:2009.14306 tab. 10] [arxiv:2009.14306 §6.3].
- **Agreement by aspect.** PPTOD's 50-session human study has five graders, and
  Fleiss' kappa for truthfulness is 0.598, the lowest of its four aspects
  [arxiv:2109.14739 tab. 9]. The study compares PPTOD-base only with SOLOIST.
- **Leave-one-out F1 (CoQA).** Each question has four gold answers. Each is
  scored against the other three and the scores are averaged, with human
  answers scored the same way [arxiv:1808.07042 §6.1]. Humans reach 88.8 test F1
  under this metric [arxiv:1808.07042 tab. 7], so that figure is the reference
  point, not 100.
- **Crowd labels for quality and safety.** LaMDA's quality label is yes when at
  least three of five raters agree, and a safety reply passes when at least two of
  three raters say no [arxiv:2201.08239 §5]. The authors say crowdworkers were not
  extensively trained or incentivised, so beating them on interestingness may not
  mean much [arxiv:2201.08239 §7].
- **Regeneration spread (proposed, untested).** The position paper suggests
  sampling the same context several times and comparing meanings, but names no
  count or measure [arxiv:2305.16367 §7]. It reports no experiments
  [arxiv:2305.16367 §10].
- **Entity F1 and BLEU (KV retrieval net).** Entity F1 is computed on canonical
  entity forms, so wording differences do not affect it. The BLEU score depends
  on surface forms that are sampled from training frequencies, and the paper does
  not state the seed [arxiv:1705.05414 §5.1] [arxiv:1705.05414 §5.4.1]. Its Copy
  Net baseline omits the entity-type features of the original, so it is not like
  for like with that paper [arxiv:1705.05414 §5.3].
- **Dull-reply rules as a metric.** The hand-set reward paper measures dialogue
  length with the same dull-reply and repetition rules it trains on, so that gain
  is not an independent check [arxiv:1606.01541 §5.2].

## Implementation sketch

Our own outline of the shared pipeline; the variant-specific loops are in
[adversarial-text-generation](/concepts/adversarial-text-generation).

```pseudocode
# Build pairs: history = two preceding utterances [arxiv:1701.06547 §3, fn. 1]
# Drop responses under 5 tokens [arxiv:1701.06547 §3.3]
pairs = [(x, y) for (x, y) in corpus if len(y) >= 5]
# Pretrain the generator by maximum likelihood [arxiv:1701.06547 §3.3]
G = train_mle(G, pairs)
# Decode with diversity and repetition controls [arxiv:1701.06547 §3.3]
reply = beam_search(G, x, diverse=True, penalize_seen_word_types=True)
```

The decoding controls (diverse beam search and a penalty on word types already
produced) were applied to the adversarial model and to the baselines alike, so
comparisons with other systems are not like-for-like [arxiv:1701.06547 §3.3].

The variants differ mainly in what is added to the model's input or output. The
commonsense variant adds a knowledge vector for the last turn [arxiv:2109.05739
§3.3]. The KV retrieval variant adds one score per KB key to the output logits
[arxiv:1705.05414 eq. 8]. The strategy variant prepends a label token to each recommender turn
[arxiv:2009.14306 §6.2]. The task-oriented variant adds a task prompt, and its
response step takes a database result as extra input [arxiv:2109.14739 §3.2]
[arxiv:2109.14739 §4.1.1]. Each adds an annotation or lookup step that the
Seq2Seq baseline does not need. The hand-set reward variant changes the training
signal instead [arxiv:1606.01541 §3.4], and LaMDA adds a safety filter and a tool
loop at inference [arxiv:2201.08239 §6.1] [arxiv:2201.08239 §6.2]. The
contrastive variant changes both the loss and the decoding rule
[arxiv:2202.06417 §3.1] [arxiv:2202.06417 §3.2].

## Contradictions

Most of the sources measure different things, so they do not clash directly.
Where they do disagree, both sides are stated and neither is resolved here.

- Word-level reinforcement learning. The adversarial paper trains its word-level
  generator with policy gradient and reports gains [arxiv:1701.06547 tab. 3]. The
  negotiation paper reports that reinforcement learning on words degenerates into
  repeated sentences and human-likeness of 1.7 to 1.9 out of 5 on its data
  [arxiv:1808.09637 §4.4] [arxiv:1808.09637 tab. 7]. The two use different
  rewards, data and generators, so this is not a settled result either way. The
  hand-set reward paper reports gains from policy gradient on word-level generation
  too, with longer simulated dialogues and higher distinct-n-gram rates
  [arxiv:1606.01541 tab. 2] [arxiv:1606.01541 tab. 4]. Its reward is written by hand,
  not learned, so it is a further data point and not a resolution.
- Internal inconsistency in the negotiation paper. Its text says supervised word
  models match the human statistics on length and utility better
  [arxiv:1808.09637 §4.4]. Its table caption says act-based models match human
  statistics across all metrics [arxiv:1808.09637 tab. 7]. Table 7 supports the
  text on length, so the caption claim is not used here.
- PPTOD's text calls its intent-classification results comparable to existing
  methods [arxiv:2109.14739 §4.3]. At 10 samples per intent, its table has
  USE+ConveRT ahead of its large model, 85.19 against 84.12 [arxiv:2109.14739
  tab. 7]. This is a gap between text and table, not between papers.
- The paper reports that its adversarial generators beat the MMI baseline in
  human pairwise judgments [arxiv:1701.06547 tab. 4], and it uses a discriminator
  as its reward and as an automatic score [arxiv:1701.06547 tab. 3].
- The survey reports that a discriminator's decisions agree poorly with human
  judges in the study it cites, and that a discriminator's accuracy is an
  unreliable score [arxiv:1905.04071 §4.3.1].
- Sparrow uses learned preference and rule models as its RL reward and reports
  the rule model's AUC [arxiv:2209.14375 §2.5] [arxiv:2209.14375 §3.5]. The
  survey's warnings concern different metrics (ADEM, RUBER and discriminators)
  [arxiv:1905.04071 §4.3.1], so no direct conflict is established here.

## Papers

- [Deep Reinforcement Learning for Dialogue Generation](/papers/1606.01541): tunes a mutual-information Seq2Seq model with policy gradients on a hand-set reward, using simulated two-agent dialogues, and reports dialogue length, diversity and human judgments.
- [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547): trains a Seq2Seq dialogue generator with a learned discriminator as the reward, with whole-reply and per-step variants, and compares against maximum-likelihood and mutual-information baselines.
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](/papers/1705.05414): an LSTM encoder-decoder that adds attention over knowledge-base keys to its output logits, so it emits database values without a belief tracker; reports entity F1 and BLEU on an in-car corpus.
- [CoQA: A Conversational Question Answering Challenge](/papers/1808.07042): a conversational QA dataset with free-form answers and text-span rationales, plus span-reader and copy-generator baselines evaluated with leave-one-out F1.
- [Decoupling Strategy and Generation in Negotiation Dialogues](/papers/1808.09637): splits a negotiation bot into a rule-based parser, a manager that picks coarse dialogue acts, and a retrieval generator, and compares supervised, reinforcement-learned and hybrid managers.
- [Survey on Evaluation Methods for Dialogue Systems](/papers/1905.04071): surveys how dialogue systems are evaluated, including the word-overlap, learned and retrieval metrics used for chatbots; runs no experiments of its own.
- [INSPIRED: Toward Sociable Recommendation Dialog Systems](/papers/2009.14306): a human-human movie-recommendation dataset with sociable strategy labels, and a GPT-2 dialogue model that is conditioned on the strategy label.
- [CEM: Commonsense-aware Empathetic Response Generation](/papers/2109.05739): a Transformer that fuses COMET commonsense inferences about the user's last turn into the reply decoder, trained with a diversity-weighted loss.
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](/papers/2109.14739): PPTOD, one T5 model that generates every task-oriented sub-task from a text prompt, pre-trained on partly labelled dialogue corpora.
- [Improving alignment of dialogue agents via targeted human judgements](/papers/2209.14375): Sparrow, an information-seeking dialogue agent trained with RLHF on per-rule violation labels and per-turn preferences, with search evidence and reranking.
- [LaMDA: Language Models for Dialog Applications](/papers/2201.08239): decoder-only dialogue Transformers from 2B to 137B parameters, fine-tuned on crowd labels for quality and safety, with a filter-and-rank step and a toolset for grounded replies.
- [A Literature Survey of Recent Advances in Chatbots](/papers/2201.06657): surveys 316 chatbot articles from 2007 to 2021 on rule-based, retrieval and generative designs, with training corpora and metric formulas; reports no experiments.
- [A Contrastive Framework for Neural Text Generation](/papers/2202.06417): SimCTG, a contrastive training loss plus a contrastive top-k decoder for GPT-2; reports WikiText-103 results and human ratings on two dialogue corpora.
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](/papers/2402.18013): surveys LLM architectures, adaptation methods, task-oriented and open-domain dialogue systems, metrics and datasets; runs no experiments of its own.
- [Role-Play with Large Language Models](/papers/2305.16367): a position paper that describes a dialogue agent as a simulator running role-played characters, with two proposed tests for false and self-preserving replies; reports no experiments.