Dialogue generation

Overview

A dialogue model reads the conversation so far and writes the next response. The paper's setting approximates the history with the two preceding utterances, which it found was enough for a Seq2Seq model; more context gave very small gains 1701.06547 §3, footnote 1. The shared problem is that models trained by maximum likelihood tend to produce bland, repetitive and short-sighted replies 1701.06547 §1. One paper traces the repetition to token states that are too similar to one another 2202.06417 §1. Most variants below try to make replies better as a whole, not just more probable token by token. Some variants instead change what the model is conditioned on: a commonsense vector, a strategy label, or a database result. The LLM adaptation variant changes how a large pre-trained model is used rather than the training objective of a Seq2Seq model.

Open-domain chat is the setting for the first two variant families, for the hand-set reward variant, for LaMDA and for the contrastive variant, which is also tested on two chat corpora. The survey separates it from task-oriented systems, where generation is one module of a pipeline with understanding and dialogue management 1905.04071 §3.3, and from interactive question answering 1905.04071 §5.1. Two variants below work on task-oriented data: PPTOD folds that pipeline into a single model, and the KV retrieval net reads a knowledge base inside the decoder 1705.05414 §2.3. The empathetic and recommendation variants apply the same generation step to other settings. The survey's generative chatbot models are seq2seq networks, with HRED adding a context encoder over earlier turns 1905.04071 §4.2.1. The Sparrow paper targets information-seeking dialogue, where success is easier to define than in open chit-chat 2209.14375 §1. The negotiation paper works in a closed, goal-driven task instead: two agents agree a price for a real listing 1808.09637 §2.

Fifteen papers with cards cover this concept: eleven report their own experiments, three are surveys, and one is a position paper with no experiments 2305.16367 §1. Each reports on its own setup. The adversarial results use one dataset (OpenSubtitles) and one group of authors 1701.06547 §3.3. The hand-set reward results come from simulated two-agent dialogues scored by the authors' own rules, plus human judgments on 500 single-turn and 200 multi-turn items 1606.01541 §5.2. The LaMDA results come from crowdworker labels on the authors' fixed evaluation sets, with human baselines shown only in figures 2201.08239 §5 2201.08239 §7. The contrastive results come from 200 contexts per dataset rated on a five-point scale by five annotators, on LCCC and DailyDialog 2202.06417 §5 2202.06417 tab. 3. The CoQA results use a passage-grounded question-answering dataset with gold history, where the best system is a span reader rather than a generator 1808.07042 tab. 7. The negotiation results come from a new dataset of buyer and seller chats and from a deployed bot judged by crowd workers 1808.09637 §2 1808.09637 §4.4. The PPTOD results use MultiWOZ for end-to-end and state tracking 2109.14739 tab. 2 2109.14739 tab. 4. The empathetic results use EmpatheticDialogues 2109.05739 §2.3, and the recommendation results use the authors' own 1,001 human-human dialogues 2009.14306 §3. The KV retrieval results come from the authors' own in-car corpus, scored on its test dialogues 1705.05414 §5.1. The survey reviews other studies and runs no experiments of its own 1905.04071 §2.2. The chatbot survey likewise reports no experimental results of its own 2201.06657 §3. The LLM survey likewise reports no new results 2402.18013 §1. The Sparrow results come from a 70B-scale agent judged by paid raters on the authors' own test prompts 2209.14375 §3.1. The method papers do not compare against one another, so their numbers are not one benchmark.

Variants

Maximum-likelihood Seq2Seq with decoding choices

A Seq2Seq encoder-decoder with attention, trained by maximum likelihood. The paper decodes it three ways, which are the baselines in its comparison: greedy, beam search, and sampling 1701.06547 tab. 3. Sampling gets the highest automatic adversarial score of the baselines, but its replies are the least distinguishable from random human replies 1701.06547 tab. 3, §5.1. This variant needs no extra training and is the starting point for the others.

Hierarchical Seq2Seq (HRED)

A generative model with a second encoder that runs over the sequence of earlier turns, so the decoder sees more than the last utterance [arxiv:1905.04071 §4.2.1, fig. 7]. The survey describes it as a seq2seq extension and does not report a head-to-head number against the models above. The paper's own result on context length, that two previous utterances were enough 1701.06547 §3, footnote 1, is the only evidence on how much history helps here.

Passage-grounded answer generation with copy (CoQA baselines)

A conversational answerer reads a passage and the earlier question-answer turns, then writes the next answer. The paper's seq2seq model puts the passage, the history and the current question into one input for a bidirectional LSTM encoder, and an LSTM decoder with attention writes the answer. The copy variant (PGNet) can copy words from the passage 1808.07042 §5.1. On the test set, PGNet reaches 44.1 overall F1 and the plain seq2seq baseline 26.3, against 52.6 for DrQA, a span reader 1808.07042 tab. 7. The best system in that table is Augmented DrQA, a span reader with extra yes and no tokens, at 65.4, and the combined DrQA+PGNet reaches 65.1 1808.07042 tab. 7. A third of the gold answers are not spans of the passage 1808.07042 §4.1, so the generator variants are the ones that can produce them. Card: CoQA: A Conversational Question Answering Challenge.

Utterance selection (retrieval)

Instead of generating text, the system ranks candidate replies from a database and returns the best one 1905.04071 §4.2.2. The survey's metric for this family, Recall@k, has a known gap: other candidates may also be correct, so a single reference undercounts good replies 1905.04071 §4.3.2. No paper with a card here compares retrieval against generation. The chatbot survey finds few learned retrieval models, and a gap between large research models and industry bots 2201.06657 §6 2201.06657 §7.

Mutual-information reranking (MMI)

A baseline that the paper compares against, in two rows of its table: MMI+p(t|s) and MMI-p(t) 1701.06547 tab. 3. The paper also uses MMI reranking to produce half of the negative examples for its discriminator 1701.06547 §3.3. The paper's exact definition of the two rows is not restated here; the original method is not in this wiki and can be found at arXiv:1510.03055. The survey lists maximum mutual information among the ways to push generative models towards more diverse replies, alongside reinforcement rewards, latent variables and topics 1905.04071 §4.2.1. That is a training objective; the paper's MMI row is a reranking step.

Contrastive training and contrastive search (SimCTG)

The model is fine-tuned on the usual next-token loss plus a contrastive term over the last-layer states of all tokens in a sequence. The term has no extra weight, and it pushes apart the states of different positions 2202.06417 §3.1. The decoder then takes the top-k tokens and picks the one that trades model probability against a penalty for similarity to the text so far 2202.06417 §3.2. The paper's dialogue setting uses a Chinese GPT-2 on LCCC, with greedy, beam, nucleus and contrastive decoding (k = 5, alpha = 0.6) 2202.06417 §5. Against nucleus sampling on the same SimCTG model, the contrastive decoder scores 3.32, 3.96 and 3.13 on coherence, fluency and informativeness on LCCC, against 2.84, 3.58 and 2.72. On DailyDialog it scores 3.73 and 3.85 on coherence and fluency, against 2.75 and 3.59, and 3.46 against 3.39 on informativeness, which is not marked significant. The paper marks the starred cells as significant against greedy, beam and nucleus decoding 2202.06417 tab. 3. Human reference scores are 3.42, 3.76 and 3.20 on LCCC 2202.06417 tab. 3. The contrastive search also beats nucleus sampling on the plain MLE model for LCCC on all three criteria, and the paper says the Chinese model is already isotropic, which explains why 2202.06417 §5. For English GPT-2 small, the same search on the MLE model trails the SimCTG model well behind 2202.06417 tab. 1. Decoding cost is close to beam search for small beam width and k, and the paper reports it faster above six 2202.06417 §6.4. Card: A Contrastive Framework for Neural Text Generation.

Adversarial REINFORCE with a whole-reply reward

A discriminator labels a dialogue as human or machine. The generator is trained with policy gradient to raise the discriminator's probability that its sampled reply is human, with one reward per reply 1701.06547 §3.1, eq. 2. The paper reports that this beats the MMI baseline in human pairwise judgments, 0.62 wins to 0.18 losses on single-turn items and 0.72 to 0.10 on multi-turn items 1701.06547 tab. 4. Card: Adversarial Learning for Neural Dialogue Generation.

Adversarial REINFORCE with per-step rewards (REGS)

The same loop, but each prefix of the reply gets its own reward, estimated from N = 5 random completions scored by the discriminator 1701.06547 §3.2, eq. 3. On the automatic adversarial score it is slightly ahead of the whole-reply variant (0.098 against 0.080) 1701.06547 tab. 3. The paper does not say which of the two was used for the human evaluation 1701.06547 §5.2.

Hand-set reward policy gradient in two-agent simulation

The model is first trained by maximum likelihood, then moved towards a reward on whole dialogues. The reward is a weighted sum with weights 0.25, 0.25 and 0.5 1606.01541 §3.4, eq. 4. Its three terms are the likelihood of a fixed list of eight dull replies, where lower is better 1606.01541 §3.4, eq. 1; a penalty on repetition between consecutive turns of one agent [arxiv:1606.01541 §3.4, eq. 2]; and forward and backward likelihood for coherence [arxiv:1606.01541 §3.4, eq. 3]. Training starts from a mutual-information model. Two copies of the policy then talk in simulation, with dialogues growing from two turns to five and five candidate replies sampled per step 1606.01541 §4.2 [arxiv:1606.01541 §4.3] 1606.01541 §4.4. Author-reported: simulated dialogues average 4.48 turns, against 3.40 for the mutual-information model and 2.68 for maximum-likelihood Seq2Seq 1606.01541 tab. 2. On five-turn human comparisons the reinforced model wins 0.72 and loses 0.12 1606.01541 tab. 5. On single-turn general quality it wins 0.40 and loses 0.36, which the authors call not a significant boost, and its BLEU is 1.28 against 1.44 for the mutual-information model 1606.01541 tab. 5 1606.01541 §5.2. The paper ships no code, and the pseudocode in this concept is our reading of it. Card: Deep Reinforcement Learning for Dialogue Generation.

Rule-conditioned RLHF with evidence and reranking (Sparrow)

A 70B-scale agent prompted from Chinchilla, fine-tuned with human preferences and a separate reward model per rule. Rules are short, single-issue statements, such as a ban on threatening statements 2209.14375 §2.1. A rule model estimates the probability that a reply breaks each rule 2209.14375 §2.5, and the agent is trained with advantage actor-critic against a KL penalty to its start point 2209.14375 §2.8. For factual turns the agent issues web searches and shows the excerpt to raters 2209.14375 §2.4. At answer time it samples eight candidates (four with evidence) and keeps the best under a score that multiplies a preference term by the geometric mean of the rule probabilities 2209.14375 §2.6, eq. 1. The authors report that the RL agent with eight samples reached 0.78 on supported-and-plausible evidence replies, against 0.61 for always searching with the prompted base model 2209.14375 §3.2, and 0.82 on free-dialogue replies rated true or mostly true, against 0.63 for the base model that never searches 2209.14375 §3.3. Probing found a targeted rule broken in 8% of conversations 2209.14375 §1. Card: Improving alignment of dialogue agents via targeted human judgements.

Coarse dialogue acts with a retrieval generator (negotiation)

The dialogue is split into three modules. A rule-based parser maps each utterance to a coarse act, an intent plus arguments such as propose(price=...) 1808.09637 §3.2 1808.09637 §3.3. A manager picks the next act, either by supervised sequence-to-sequence learning, by policy gradient on a reward (utility, fairness or length), or by a hybrid that takes intents from a trigram model and fills the arguments with rules 1808.09637 §3.4 1808.09637 §4.2. A retrieval generator then picks a human-written template that matches the act and the context 1808.09637 §3.5. Because the strategy lives in the act, a developer can set it by data, reward or rules. Reinforcement learning on acts kept human-likeness on the authors' CraigslistBargain data at 2.8 to 3.0 out of 5, against 3.3 for supervised acts and 4.3 for humans 1808.09637 tab. 7. The same reinforcement learning applied to words gave 1.7 to 1.9, and the three most frequent sentences made up 81.6% to 100% of the utterances in the word-level models 1808.09637 §4.4. The paper reports these results as author-reported, from crowd-worker chats. Card: Decoupling Strategy and Generation in Negotiation Dialogues.

Unified task-oriented generation (PPTOD)

One pre-trained T5 model handles each task-oriented sub-task as text generation, selected by a prompt such as "translate dialogue to belief state:". The model is pre-trained on eleven human-written corpora, and each corpus is labelled for only some sub-tasks 2109.14739 §3.1 2109.14739 tab. 1. The response step is generated from the database result as well, in parallel with the dialogue action, so one model covers the whole turn after the belief state is predicted 2109.14739 §4.1.1. The paper reports the best MultiWOZ 2.0 end-to-end combined score in its table, 102.92 for PPTOD-base 2109.14739 tab. 2. In low-resource belief-state tracking with 1% of the data, its large model reaches 31.46 joint accuracy against 13.21 for SOLOIST 2109.14739 tab. 5. For latency, the paper's T5-small ablation gives 38.70 ms for a cascaded run and 14.17 ms for the single-pass format, both without database state, on one V100 at batch size 1 2109.14739 tab. 6. Card: Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System.

Knowledge-base key attention in the decoder (KV retrieval net)

An LSTM encoder-decoder for in-car task-oriented dialogue, with a second attention step over the keys of a knowledge base (KB) built for each dialogue. Each KB row is split into (subject, relation, value) triples, and a key is formed from the subject and relation words 1705.05414 §2.3. The key scores are added to the output logits at a canonical token for that key. When the model emits such a token, it is swapped for the stored value [arxiv:1705.05414 §2.3, eq. 7-9]. No belief tracker or dialogue-state labels are used, and training is token-level cross-entropy 1705.05414 §1 1705.05414 §5.2. On the authors' test dialogues it reaches 13.2 BLEU and 48.0 entity F1, against 6.6 and 43.8 for a rule-based system and 11.0 and 37.0 for Copy Net. Removing the encoder attention gives 40.9 entity F1 1705.05414 tab. 3. In real-time chat, its ratings for fluency, cooperation and human-likeness are above both baselines but below the human partner, on 120 scenarios 1705.05414 tab. 4. Card: Key-Value Retrieval Networks for Task-Oriented Dialogue.

Commonsense-conditioned empathetic generation (CEM)

The model infers five commonsense relations about the user's last turn: xReact, xWant, xNeed, xIntent and xEffect 2109.05739 §3.2. An affective encoder handles xReact and a cognitive encoder handles the other four. Their outputs are fused with the dialogue context through a sigmoid gate and an MLP, and the decoder reads the fused representation 2109.05739 §3.3 [arxiv:2109.05739 §3.4]. Training combines reply likelihood, an emotion classification loss and a frequency-weighted diversity loss 2109.05739 §3.6. Only the last user turn receives commonsense; earlier turns are encoded without it [arxiv:2109.05739 §3.2]. On EmpatheticDialogues the authors report perplexity 36.11 against 36.93 for the best baseline, and Dist-1 and Dist-2 of 0.66 and 2.99 against 0.47 and 2.10 2109.05739 tab. 2. In human pairwise tests over 100 sampled pairs per comparison, the model won on coherence, empathy and informativeness against MoEL at the paper's significance level. Against MIME it won on coherence and empathy, and against EmpDG on empathy; the other differences are not significant 2109.05739 tab. 3. Serving the model means producing commonsense for each new last turn, which the method needs 2109.05739 §3.2. Card: CEM: Commonsense-aware Empathetic Response Generation.

Strategy-label conditioning (INSPIRED)

Each recommender turn in the data is annotated with a sociable strategy, such as sharing a personal opinion or encouraging the other person. The dialogue model writes the strategy as a special token in front of the turn, so the label is generated along with the reply 2009.14306 §6.2. The model is two GPT-2 small networks, one per speaker 2009.14306 §6.1 2009.14306 app. D. On the authors' 100-dialogue test split, the labelled model has perplexity 8.93 against 9.28 for the unlabelled baseline, and BLEU-4 6.63 against 5.11 2009.14306 tab. 7. In five-point ratings from 25 participants per model, the labelled model scores higher on all five aspects 2009.14306 tab. 10. The paper's logistic regression finds four strategies with significant positive coefficients for recommendation success, but it is a correlational analysis 2009.14306 §5.2 2009.14306 tab. 6. Choosing among sampled replies uses a rule that the paper describes 2009.14306 §6.2; the card for this paper notes that the released code differs in both the number of samples and the selection rule. Card: INSPIRED: Toward Sociable Recommendation Dialog Systems.

LLM generation with adaptation (survey view)

A large pre-trained language model generates the reply, adapted either by changing its weights or by prompting it. The survey lists full fine-tuning, adapters, LoRA (a low-rank update added to a frozen weight matrix), and instruction tuning as the weight-changing options 2402.18013 §2.4. It lists in-context learning, chain of thought, and retrieval-augmented generation in naive, advanced and modular forms as prompt-side options 2402.18013 §2.5. The survey places open-domain generation in three families: retrieval, generation, and hybrid 2402.18013 §4. It gives no measured comparison between these options, and it leaves the rank of LoRA and the number of in-context examples unstated 2402.18013 §2.4.2 2402.18013 §2.5.1. Card: A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems.

Dialogue-tuned LLM with safety filtering and tool grounding (LaMDA)

A decoder-only Transformer, from 2B to 137B parameters, is pre-trained to predict the next token on about 1.56T words of public dialogue and web text [arxiv:2201.08239 §3] 2201.08239 app. E. Fine-tuning on crowd-labelled examples teaches the same model to generate replies and to rate them for sensibleness, specificity, interestingness and safety 2201.08239 §6.1. At inference it samples candidates, drops those whose predicted safety is below a threshold, and ranks the rest by three times sensibleness plus specificity plus interestingness 2201.08239 §6.1. A second fine-tuning stage teaches it to write queries to a toolset (search, calculator, translator) and to write its reply from the results 2201.08239 §6.2. Author-reported at 137B: sensibleness 92.3, specificity 79.0, safety 95.2 and groundedness 73.2. The pre-trained model at the same size scores 80.2, 49.8, 88.0 and 57.9 2201.08239 app. F. Citation accuracy is 65% 2201.08239 §7. The paper says its gains came from fine-tuning data under 0.001% of the pre-training data 2201.08239 §9. The authors call LaMDA a recipe for application-specific versions rather than a finished model, and the crowd data and weights are not released 2201.08239 §9 [arxiv:2201.08239 app. A.2]. Card: LaMDA: Language Models for Dialog Applications.

Role-play framing for false and self-preserving replies (no method)

Not a training or decoding method. The position paper describes a dialogue agent built on a base LLM as a simulator, the base model plus sampling, that holds many possible characters at once. Each sampled reply narrows that set 2305.16367 §4 2305.16367 §5. The paper reads apparent deception as made-up, good-faith or deliberate falsehood, and apparent self-preservation as a role played from human-written text 2305.16367 §7 2305.16367 §8. It proposes two behavioural tests: regenerate the same context several times and check whether the answers vary in meaning, and ask the same question in different contexts 2305.16367 §7. It reports no experiments, and it covers base models; whether the framing holds after RLHF is unclear 2305.16367 §2 2305.16367 §8. Card: Role-Play with Large Language Models.

When to use which

Measuring replies

Scores are a choice that changes which system looks best, so read the survey's caveats before picking one:

• Word-overlap metrics. The survey reports that BLEU and ROUGE correlate poorly with human judgement at utterance level, with ROUGE at Pearson 0.062 1905.04071 §4.3.1. ΔBLEU, which weights references by human ratings, reaches Spearman 0.484 against BLEU's 0.318 1905.04071 §4.3.1.
• Learned metrics. ADEM is trained on human ratings, and RUBER needs no human labels. The survey reports that reversing a reply raised the ADEM score in 48.66% of cases, and a dull dummy reply in 86.93% 1905.04071 §4.3.1.
• Discriminator as score. The survey reports low agreement between the discriminator and human judges: Fleiss π 0.07 on Switchboard, against 0.3 for humans among themselves 1905.04071 §4.3.1.
• Human-likeness rating. The negotiation paper asks raters whether the partner showed reasonable human behaviour, on a 1-to-5 scale, after a chat with a bot or a human 1808.09637 §4.4. Its results rest on that one question, and the evaluation model it uses to measure other outputs is itself trained on human-human data 1808.09637 §4.4.
• Rater agreement on rules. Sparrow reports Krippendorff's alpha of 0.37 for a general harm rule and 0.53 for specific rules 2209.14375 §3.5. Its numbers come from paid raters on its own prompts, so they are not comparable with the survey's Switchboard figures.
• Printed formulas in the chatbot survey. Its F1 formula omits the factor 2 of the harmonic mean, and its METEOR formula gives no alpha and no penalty term, so neither should be implemented as printed 2201.06657 §4.3.
• Perplexity formula in the contrastive paper. Its gen-ppl is defined as 2 to the power of an average of log-probabilities with the sign missing, so the standard definition is needed to compare its numbers 2202.06417 §4.1.2.
• Rater agreement on Likert scores. SimCTG's dialogue annotators reach Fleiss' kappa of 0.55 to 0.73 across the two datasets, with informativeness the lowest on LCCC 2202.06417 tab. 3.
• Rater agreement on pairs. CEM's pairwise tests have three raters per pair, with Fleiss' kappa between 0.41 and 0.57 for the comparisons it reports 2109.05739 tab. 3. Each comparison uses 100 sampled pairs 2109.05739 §4.4.
• Small rating panels. INSPIRED's five-point ratings come from 25 participants per model, and its text gives no statistical test for the preference result 2009.14306 tab. 10 2009.14306 §6.3.
• Agreement by aspect. PPTOD's 50-session human study has five graders, and Fleiss' kappa for truthfulness is 0.598, the lowest of its four aspects 2109.14739 tab. 9. The study compares PPTOD-base only with SOLOIST.
• Leave-one-out F1 (CoQA). Each question has four gold answers. Each is scored against the other three and the scores are averaged, with human answers scored the same way 1808.07042 §6.1. Humans reach 88.8 test F1 under this metric 1808.07042 tab. 7, so that figure is the reference point, not 100.
• Crowd labels for quality and safety. LaMDA's quality label is yes when at least three of five raters agree, and a safety reply passes when at least two of three raters say no 2201.08239 §5. The authors say crowdworkers were not extensively trained or incentivised, so beating them on interestingness may not mean much 2201.08239 §7.
• Regeneration spread (proposed, untested). The position paper suggests sampling the same context several times and comparing meanings, but names no count or measure 2305.16367 §7. It reports no experiments 2305.16367 §10.
• Entity F1 and BLEU (KV retrieval net). Entity F1 is computed on canonical entity forms, so wording differences do not affect it. The BLEU score depends on surface forms that are sampled from training frequencies, and the paper does not state the seed 1705.05414 §5.1 1705.05414 §5.4.1. Its Copy Net baseline omits the entity-type features of the original, so it is not like for like with that paper 1705.05414 §5.3.
• Dull-reply rules as a metric. The hand-set reward paper measures dialogue length with the same dull-reply and repetition rules it trains on, so that gain is not an independent check 1606.01541 §5.2.

Implementation sketch

Our own outline of the shared pipeline; the variant-specific loops are in adversarial-text-generation.

# Build pairs: history = two preceding utterances [arxiv:1701.06547 §3, fn. 1]
# Drop responses under 5 tokens [arxiv:1701.06547 §3.3]
pairs = [(x, y) for (x, y) in corpus if len(y) >= 5]
# Pretrain the generator by maximum likelihood [arxiv:1701.06547 §3.3]
G = train_mle(G, pairs)
# Decode with diversity and repetition controls [arxiv:1701.06547 §3.3]
reply = beam_search(G, x, diverse=True, penalize_seen_word_types=True)

The decoding controls (diverse beam search and a penalty on word types already produced) were applied to the adversarial model and to the baselines alike, so comparisons with other systems are not like-for-like 1701.06547 §3.3.

The variants differ mainly in what is added to the model's input or output. The commonsense variant adds a knowledge vector for the last turn [arxiv:2109.05739 §3.3]. The KV retrieval variant adds one score per KB key to the output logits 1705.05414 eq. 8. The strategy variant prepends a label token to each recommender turn 2009.14306 §6.2. The task-oriented variant adds a task prompt, and its response step takes a database result as extra input 2109.14739 §3.2 2109.14739 §4.1.1. Each adds an annotation or lookup step that the Seq2Seq baseline does not need. The hand-set reward variant changes the training signal instead 1606.01541 §3.4, and LaMDA adds a safety filter and a tool loop at inference 2201.08239 §6.1 2201.08239 §6.2. The contrastive variant changes both the loss and the decoding rule 2202.06417 §3.1 2202.06417 §3.2.

Contradictions

Most of the sources measure different things, so they do not clash directly. Where they do disagree, both sides are stated and neither is resolved here.

• Word-level reinforcement learning. The adversarial paper trains its word-level generator with policy gradient and reports gains 1701.06547 tab. 3. The negotiation paper reports that reinforcement learning on words degenerates into repeated sentences and human-likeness of 1.7 to 1.9 out of 5 on its data 1808.09637 §4.4 1808.09637 tab. 7. The two use different rewards, data and generators, so this is not a settled result either way. The hand-set reward paper reports gains from policy gradient on word-level generation too, with longer simulated dialogues and higher distinct-n-gram rates 1606.01541 tab. 2 1606.01541 tab. 4. Its reward is written by hand, not learned, so it is a further data point and not a resolution.
• Internal inconsistency in the negotiation paper. Its text says supervised word models match the human statistics on length and utility better 1808.09637 §4.4. Its table caption says act-based models match human statistics across all metrics 1808.09637 tab. 7. Table 7 supports the text on length, so the caption claim is not used here.
• PPTOD's text calls its intent-classification results comparable to existing methods 2109.14739 §4.3. At 10 samples per intent, its table has USE+ConveRT ahead of its large model, 85.19 against 84.12 [arxiv:2109.14739 tab. 7]. This is a gap between text and table, not between papers.
• The paper reports that its adversarial generators beat the MMI baseline in human pairwise judgments 1701.06547 tab. 4, and it uses a discriminator as its reward and as an automatic score 1701.06547 tab. 3.
• The survey reports that a discriminator's decisions agree poorly with human judges in the study it cites, and that a discriminator's accuracy is an unreliable score 1905.04071 §4.3.1.
• Sparrow uses learned preference and rule models as its RL reward and reports the rule model's AUC 2209.14375 §2.5 2209.14375 §3.5. The survey's warnings concern different metrics (ADEM, RUBER and discriminators) 1905.04071 §4.3.1, so no direct conflict is established here.

Papers

• Deep Reinforcement Learning for Dialogue Generation: tunes a mutual-information Seq2Seq model with policy gradients on a hand-set reward, using simulated two-agent dialogues, and reports dialogue length, diversity and human judgments.
• Adversarial Learning for Neural Dialogue Generation: trains a Seq2Seq dialogue generator with a learned discriminator as the reward, with whole-reply and per-step variants, and compares against maximum-likelihood and mutual-information baselines.
• Key-Value Retrieval Networks for Task-Oriented Dialogue: an LSTM encoder-decoder that adds attention over knowledge-base keys to its output logits, so it emits database values without a belief tracker; reports entity F1 and BLEU on an in-car corpus.
• CoQA: A Conversational Question Answering Challenge: a conversational QA dataset with free-form answers and text-span rationales, plus span-reader and copy-generator baselines evaluated with leave-one-out F1.
• Decoupling Strategy and Generation in Negotiation Dialogues: splits a negotiation bot into a rule-based parser, a manager that picks coarse dialogue acts, and a retrieval generator, and compares supervised, reinforcement-learned and hybrid managers.
• Survey on Evaluation Methods for Dialogue Systems: surveys how dialogue systems are evaluated, including the word-overlap, learned and retrieval metrics used for chatbots; runs no experiments of its own.
• INSPIRED: Toward Sociable Recommendation Dialog Systems: a human-human movie-recommendation dataset with sociable strategy labels, and a GPT-2 dialogue model that is conditioned on the strategy label.
• CEM: Commonsense-aware Empathetic Response Generation: a Transformer that fuses COMET commonsense inferences about the user's last turn into the reply decoder, trained with a diversity-weighted loss.
• Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System: PPTOD, one T5 model that generates every task-oriented sub-task from a text prompt, pre-trained on partly labelled dialogue corpora.
• Improving alignment of dialogue agents via targeted human judgements: Sparrow, an information-seeking dialogue agent trained with RLHF on per-rule violation labels and per-turn preferences, with search evidence and reranking.
• LaMDA: Language Models for Dialog Applications: decoder-only dialogue Transformers from 2B to 137B parameters, fine-tuned on crowd labels for quality and safety, with a filter-and-rank step and a toolset for grounded replies.
• A Literature Survey of Recent Advances in Chatbots: surveys 316 chatbot articles from 2007 to 2021 on rule-based, retrieval and generative designs, with training corpora and metric formulas; reports no experiments.
• A Contrastive Framework for Neural Text Generation: SimCTG, a contrastive training loss plus a contrastive top-k decoder for GPT-2; reports WikiText-103 results and human ratings on two dialogue corpora.
• A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems: surveys LLM architectures, adaptation methods, task-oriented and open-domain dialogue systems, metrics and datasets; runs no experiments of its own.
• Role-Play with Large Language Models: a position paper that describes a dialogue agent as a simulator running role-played characters, with two proposed tests for false and self-preserving replies; reports no experiments.