Dialogue evaluation

Overview

A dialogue system is hard to judge because the usual method, having people use it and fill in a questionnaire, is slow and must be repeated after every change. The survey sets five wishes for an evaluation method: it should be automatic, repeatable, correlated with human judgement, able to separate strategies, and able to explain what matters 1905.04071 §2.2. It also states that no existing method meets all five, and most still need some human involvement 1905.04071 §2.2 1905.04071 §7.

The right family depends on the kind of system. The survey separates task-oriented systems, conversational agents and interactive question answering 1905.04071 §2.1 1905.04071 tab. 1. Each has its own measures. The variants below are listed by family, and the table in "When to use which" maps them to system kinds.

Four papers with cards cover this concept. Three are surveys and run no experiments of their own, so their numbers are figures reported from other studies. The 2019 survey predates large language model evaluation 1905.04071 §7. The 2022 chatbot survey reviews 316 articles and concludes that the field has no shared evaluation framework, and that automatic n-gram metrics are weak for dialogue 2201.06657 §4.3 2201.06657 §6. It gives no agreement figures. The 2024 survey covers LLM-based multi-turn systems and lists their metrics, but the card gives no agreement figures for them 2402.18013 §5. The fourth, LaMDA, is a model paper that scores its own replies with crowd labels, so its numbers are author-reported 2201.08239 §7.

Variants

Task success and dialogue cost

For task-oriented systems with a defined goal. Task success is scored against the goal, and the survey describes a confusion matrix per slot as one way to record it 1905.04071 §3.4 1905.04071 tab. 5. Dialogue cost counts turns, elapsed time, repair utterances and the turns needed to fill a slot 1905.04071 §3.4. This variant needs only logs. Card: Survey on Evaluation Methods for Dialogue Systems.

PARADISE-style satisfaction model

A linear regression from task success and cost to the questionnaire score of each dialogue 1905.04071 §3.4.1 1905.04071 fig. 5. The original study found only task success and the number of repetitions significant 1905.04071 §3.4.1. The survey reports a pooled test R² of 0.50 across three systems, and an R² of 0.55 when trained on one system and tested on another, but only 0.36 for a second pair and 0.04 when trained on novice and tested on expert users 1905.04071 tab. 6. The model must be fitted per system and user population 1905.04071 §3.4.1 1905.04071 §7.

Learned interaction-quality model

Experts rate each exchange, the label is the median of their ratings, and a support vector machine is trained on features from the system's logs 1905.04071 §3.4.1 1905.04071 fig. 6. Because the features come from logs, the model can score an exchange while the dialogue is running. The survey reports a Spearman ρ of 0.776 for the field-data target with automatic features 1905.04071 tab. 7, and 0.888 for the lab target with automatic, emotion and user features. The text gives 0.894 for the same case, so the survey disagrees with itself 1905.04071 tab. 7.

User simulation

A simulated user lets a dialogue manager be trained or tested without people. The agenda-based simulator keeps a hidden goal and a stack of user actions, and updates the stack after each turn 1905.04071 §3.4.2. A neural simulator works on surface text and can vary its goal 1905.04071 §3.4.2. The simulator can be checked directly, by precision and recall on dialogue acts or by perplexity, or indirectly, by the quality of the manager trained on it 1905.04071 §3.4.2. The survey also describes a workbench that checks whether simulated tests reach the same conclusions as a user study 1905.04071 §3.4.2. A simulator reproduces only the behaviour it encodes 1905.04071 §7.

Word-overlap metrics

BLEU and ROUGE were the early default for generated replies 1905.04071 §4.3.1. In a cited study, ROUGE reached a Pearson correlation of 0.062 with human judgement at utterance level and 0.268 at system level, and BLEU reached a Spearman ρ of 0.318 1905.04071 §4.3.1. ΔBLEU, which weights the score by human ratings of the reference replies, reached 0.484 in the same study 1905.04071 §4.3.1. The survey says BLEU is more competitive at corpus and system level 1905.04071 §7. The 2022 chatbot survey lists BLEU among its metrics and calls automatic n-gram scores weak for dialogue, without a correlation figure 2201.06657 §4.3. Its BLEU formula leaves the maximum n-gram order and the weights unstated, so use the original BLEU definition rather than that formula 2201.06657 §4.3 eq. 3 2201.06657 §4.3 eq. 4. Card: A Literature Survey of Recent Advances in Chatbots.

Learned reply metrics

ADEM is trained to predict human appropriateness ratings. It reached a Pearson correlation of 0.41 at utterance level and 0.954 at system level, and the two are not comparable 1905.04071 §4.3.1. RUBER needs no human labels. It combines similarity to the reference reply with relatedness to the context, and reached a Pearson correlation of 0.4594 with human judgement [arxiv:1905.04071 §4.3.1]. Both are fragile. In a cited re-evaluation, reversing a reply raised the ADEM score in 48.66% of cases, and a dull dummy reply raised it in 86.93% 1905.04071 §4.3.1.

Adversarial discriminator and Turing-style tests

A discriminator is trained to tell generated dialogue from real dialogue, and its accuracy is the score 1905.04071 §4.3.1. In the first cited study the discriminator reached 62.5% accuracy. Humans reached 64% to 67.7% on three corpora. On Switchboard the human-human agreement was a Fleiss π of 0.3, and discriminator-human agreement was 0.07 1905.04071 §4.3.1. The page adversarial-evaluation covers this in more depth.

Topic breadth and depth

Topic breadth counts the distinct topic keywords in a reply, and topic depth measures how long a run of same-topic replies is. Topics come from a Deep Averaging Network. A cited study reports Spearman ρ of 0.707 for depth and 0.512 for breadth against human ratings 1905.04071 §4.3.1.

Utterance-selection metrics

For systems that choose a reply from a set of candidates. Recall@k measures how often the correct reply is in the top k, but other candidates may be correct too 1905.04071 §4.3.2. Weak agreement counts a reply as appropriate if any annotator picked it. Voted appropriateness weights each reply by the number of annotators who picked it 1905.04071 §4.3.2. A cited study reports correlations of 0.479 (voted) and 0.485 (weak) at utterance level, and 0.893 and 0.803 at system level 1905.04071 §4.3.2.

Question answering metrics and questionnaires

Ranked open QA uses MAP, MRR, precision, recall and F1. Multiple choice uses accuracy. Extractive spans use exact match and token F1 1905.04071 §5.3. For interactive QA the survey reviews NASA TLX, a task questionnaire and a system questionnaire, and finds the task questionnaire best at telling systems apart 1905.04071 §5.3. It proposes multi-reference evaluation for the case where many answers are correct 1905.04071 §5.3.

LLM-era automatic metrics and human criteria

For LLM-based dialogue systems, the 2024 survey lists its evaluation metrics, grouped by system kind 2402.18013 §5.1. For task-oriented systems: joint goal accuracy, slot accuracy and average goal accuracy, a requested-slot macro F1, BLEU and entity F1 [arxiv:2402.18013 §5.1.1]. For open-domain systems: perplexity, BLEU, DIST-n and recall@K 2402.18013 §5.1.2. The recall@K formula as typeset uses an index that is never defined, so it cannot be implemented from the text alone 2402.18013 §5.1.2 eq. 32. For human evaluation the survey lists six criteria: diversity, creativity, grammar, faithfulness, coherence and consistency 2402.18013 §5.2. The card records no correlation with human judgement for any of these metrics, so this variant is a checklist of what to measure, not evidence about which measure to trust. The macro-F1 formula also uses a class count that the text does not define 2402.18013 §5.1.1 eq. 30. Card: A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems.

Crowd-labelled quality, safety and groundedness

For a dialogue model whose replies are scored on fixed test sets. The paper notes that automatic dialogue metrics may not correlate well with human judgement, so it relies on crowdworker labels instead 2201.08239 §2. Quality: five raters label each reply, and a label is "yes" if at least three agree. Specificity and interestingness are asked only after a "yes" on sensibleness 2201.08239 §5. Safety: three raters, and an objective is met if at least two say "no". A reply scores 1 only if every objective passes 2201.08239 §5. Groundedness: three raters, majority vote 2201.08239 §5. The test sets are a 1,477-item Mini-Turing Benchmark for quality, a 1,166-dialog holdout for safety, and 784 turns from the Wizard of Wikipedia set for groundedness 2201.08239 §5.

At 137B the paper reports sensibleness 92.3, specificity 79.0, safety 95.2 and groundedness 73.2 2201.08239 app. F 2201.08239 §7. Human baselines appear only in figures 2201.08239 fig. 4 [arxiv:2201.08239 fig. 5]. The authors say crowdworkers were not extensively trained or incentivised, so beating them on interestingness may mean little 2201.08239 §7. Safety is one binary score across all objectives, which cannot express degrees of harm 2201.08239 §9 2201.08239 app. A.1. The objectives and crowdworkers are US-based 2201.08239 §9 2201.08239 app. A.3, and the crowd datasets are not released 2201.08239 app. A.2, so the test sets cannot be rebuilt from the paper. Card: LaMDA: Language Models for Dialog Applications.

When to use which

The survey's cited results give little reason to use word-overlap metrics alone for single replies 1905.04071 §4.3.1. The 2022 chatbot survey reaches the same general view without numbers 2201.06657 §4.3 [arxiv:2201.06657 §6]. The 2019 survey's validity results do not cover large language models 1905.04071 §7, and the 2024 survey gives no validity results for its metrics 2402.18013 §5. The LaMDA card records labelling rules and a human-consensus reference, but no agreement statistic between its raters 2201.08239 §5. So no choice above is validated for LLM-based systems.

Implementation sketch

The common core is to test any automatic score against human ratings before trusting it. Averaging per system is reported to raise agreement, so keep the two levels apart.

# Check an automatic dialogue score against human judgement  [§2.2]
# Input:  per reply, an automatic score and a human rating
# Output: agreement at utterance level and at system level  [§4.3.1]

def utterance_agreement(auto_scores, human_ratings):
    # Rank correlation between the metric and the ratings  [§4.3.1]
    return spearman(auto_scores, human_ratings)

def system_agreement(system_auto, system_human):
    # Same check on per-system averages. Not comparable with the
    # utterance-level number  [§4.3.1]
    return spearman(system_auto, system_human)

def check_metric(data):
    # Also test reversed and dull replies: a fragile metric scores
    # them too high  [§4.3.1]
    reversed_gap = score_reversed(data) - score_reference(data)
    return reversed_gap

Contradictions

No conflicting numbers across the cards: the 2022 chatbot survey, the 2024 card and the LaMDA card give no correlation with human judgement to compare with the 2019 survey. Inside the 2019 survey, the IQ_lab correlation is 0.894 in the text and 0.888 in its table 1905.04071 tab. 7.

On word-overlap metrics the two surveys read the evidence differently. The 2019 survey says BLEU is more competitive at corpus and system level 1905.04071 §7. The 2022 survey calls automatic n-gram scores weak for dialogue and does not separate corpus or system level 2201.06657 §4.3. Both sides are reported here, and neither is resolved.

Papers

• Survey on Evaluation Methods for Dialogue Systems: surveys how task, conversational and interactive QA systems are evaluated, with the satisfaction models, simulators, learned metrics and datasets for each. It runs no experiments.
• A Literature Survey of Recent Advances in Chatbots: a 2007 to 2021 literature review of chatbot designs, datasets and evaluation. It runs no experiments, and it reports that no shared evaluation framework exists.
• LaMDA: Language Models for Dialog Applications: a family of dialogue models whose quality, safety and groundedness are scored with crowd labels. The numbers are author-reported, and the crowd datasets are not released.
• A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems: surveys LLM-based task-oriented and open-domain dialogue, with the metrics and datasets used for each. It runs no experiments, and its card flags several formula and table errors to check before use.