---
kind: concept
slug: dialogue-evaluation
title: Dialogue evaluation
summary: >-
  Scoring dialogue systems without a human in every test. Task bots: task success and cost, with satisfaction models fitted to ratings. Chatbots: word-overlap, learned and adversarial scores, none reliable alone, or crowd labels. A 2019 survey finds weak agreement with human judges. Trade-off: cheap automatic scores versus validity and labelling cost.
tags: [evaluation, dialogue, user-satisfaction, user-simulation, metrics, crowdsourcing]
papers: ["1905.04071", "2201.06657", "2201.08239", "2402.18013"]
related: [dialogue-generation, adversarial-evaluation, llm-evaluation, hallucination-mitigation]
status: draft
updated_at: 2026-11-01
---

## Overview

A dialogue system is hard to judge because the usual method, having people
use it and fill in a questionnaire, is slow and must be repeated after every
change. The survey sets five wishes for an evaluation method: it should be
automatic, repeatable, correlated with human judgement, able to separate
strategies, and able to explain what matters [arxiv:1905.04071 §2.2]. It also
states that no existing method meets all five, and most still need some human
involvement [arxiv:1905.04071 §2.2] [arxiv:1905.04071 §7].

The right family depends on the kind of system. The survey separates task-oriented
systems, conversational agents and interactive question answering
[arxiv:1905.04071 §2.1] [arxiv:1905.04071 tab. 1]. Each has its own measures.
The variants below are listed by family, and the table in "When to use which"
maps them to system kinds.

Four papers with cards cover this concept. Three are surveys and run no
experiments of their own, so their numbers are figures reported from other
studies. The 2019 survey predates large language model evaluation
[arxiv:1905.04071 §7]. The 2022 chatbot survey reviews 316 articles and
concludes that the field has no shared evaluation framework, and that
automatic n-gram metrics are weak for dialogue [arxiv:2201.06657 §4.3]
[arxiv:2201.06657 §6]. It gives no agreement figures. The 2024 survey covers
LLM-based multi-turn systems and lists their metrics, but the card gives no
agreement figures for them [arxiv:2402.18013 §5]. The fourth, LaMDA, is a
model paper that scores its own replies with crowd labels, so its numbers are
author-reported [arxiv:2201.08239 §7].

## Variants

### Task success and dialogue cost

For task-oriented systems with a defined goal. Task success is scored against
the goal, and the survey describes a confusion matrix per slot as one way to
record it [arxiv:1905.04071 §3.4] [arxiv:1905.04071 tab. 5]. Dialogue cost
counts turns, elapsed time, repair utterances and the turns needed to fill a
slot [arxiv:1905.04071 §3.4]. This variant needs only logs. Card:
[Survey on Evaluation Methods for Dialogue Systems](/papers/1905.04071).

### PARADISE-style satisfaction model

A linear regression from task success and cost to the questionnaire score of
each dialogue [arxiv:1905.04071 §3.4.1] [arxiv:1905.04071 fig. 5]. The original
study found only task success and the number of repetitions significant
[arxiv:1905.04071 §3.4.1]. The survey reports a pooled test R² of 0.50 across
three systems, and an R² of 0.55 when trained on one system and tested on
another, but only 0.36 for a second pair and 0.04 when trained on novice and
tested on expert users [arxiv:1905.04071 tab. 6]. The model must be fitted per
system and user population [arxiv:1905.04071 §3.4.1] [arxiv:1905.04071 §7].

### Learned interaction-quality model

Experts rate each exchange, the label is the median of their ratings, and a
support vector machine is trained on features from the system's logs
[arxiv:1905.04071 §3.4.1] [arxiv:1905.04071 fig. 6]. Because the features come
from logs, the model can score an exchange while the dialogue is running. The
survey reports a Spearman ρ of 0.776 for the field-data target with automatic
features [arxiv:1905.04071 tab. 7], and 0.888 for the lab target with automatic,
emotion and user features. The text gives 0.894 for the same case, so the
survey disagrees with itself [arxiv:1905.04071 tab. 7].

### User simulation

A simulated user lets a dialogue manager be trained or tested without people.
The agenda-based simulator keeps a hidden goal and a stack of user actions, and
updates the stack after each turn [arxiv:1905.04071 §3.4.2]. A neural simulator
works on surface text and can vary its goal [arxiv:1905.04071 §3.4.2]. The
simulator can be checked directly, by precision and recall on dialogue acts or
by perplexity, or indirectly, by the quality of the manager trained on it
[arxiv:1905.04071 §3.4.2]. The survey also describes a workbench that checks
whether simulated tests reach the same conclusions as a user study
[arxiv:1905.04071 §3.4.2]. A simulator reproduces only the behaviour it encodes
[arxiv:1905.04071 §7].

### Word-overlap metrics

BLEU and ROUGE were the early default for generated replies
[arxiv:1905.04071 §4.3.1]. In a cited study, ROUGE reached a Pearson correlation
of 0.062 with human judgement at utterance level and 0.268 at system level, and
BLEU reached a Spearman ρ of 0.318 [arxiv:1905.04071 §4.3.1]. ΔBLEU, which
weights the score by human ratings of the reference replies, reached 0.484 in
the same study [arxiv:1905.04071 §4.3.1]. The survey says BLEU is more
competitive at corpus and system level [arxiv:1905.04071 §7]. The 2022 chatbot
survey lists BLEU among its metrics and calls automatic n-gram scores weak for
dialogue, without a correlation figure [arxiv:2201.06657 §4.3]. Its BLEU
formula leaves the maximum n-gram order and the weights unstated, so use the
original BLEU definition rather than that formula [arxiv:2201.06657 §4.3 eq. 3]
[arxiv:2201.06657 §4.3 eq. 4]. Card:
[A Literature Survey of Recent Advances in Chatbots](/papers/2201.06657).

### Learned reply metrics

ADEM is trained to predict human appropriateness ratings. It reached a Pearson
correlation of 0.41 at utterance level and 0.954 at system level, and the two
are not comparable [arxiv:1905.04071 §4.3.1]. RUBER needs no human labels. It
combines similarity to the reference reply with relatedness to the context, and
reached a Pearson correlation of 0.4594 with human judgement [arxiv:1905.04071
§4.3.1]. Both are fragile. In a cited re-evaluation, reversing a reply raised the
ADEM score in 48.66% of cases, and a dull dummy reply raised it in 86.93%
[arxiv:1905.04071 §4.3.1].

### Adversarial discriminator and Turing-style tests

A discriminator is trained to tell generated dialogue from real dialogue, and
its accuracy is the score [arxiv:1905.04071 §4.3.1]. In the first cited study the
discriminator reached 62.5% accuracy. Humans reached 64% to 67.7% on three
corpora. On Switchboard the human-human agreement was a Fleiss π of 0.3, and
discriminator-human agreement was 0.07 [arxiv:1905.04071 §4.3.1]. The page
[adversarial-evaluation](/concepts/adversarial-evaluation) covers this in more
depth.

### Topic breadth and depth

Topic breadth counts the distinct topic keywords in a reply, and topic depth
measures how long a run of same-topic replies is. Topics come from a Deep
Averaging Network. A cited study reports Spearman ρ of 0.707 for depth and 0.512
for breadth against human ratings [arxiv:1905.04071 §4.3.1].

### Utterance-selection metrics

For systems that choose a reply from a set of candidates. Recall@k measures how
often the correct reply is in the top k, but other candidates may be correct
too [arxiv:1905.04071 §4.3.2]. Weak agreement counts a reply as appropriate if
any annotator picked it. Voted appropriateness weights each reply by the number
of annotators who picked it [arxiv:1905.04071 §4.3.2]. A cited study reports
correlations of 0.479 (voted) and 0.485 (weak) at utterance level, and 0.893
and 0.803 at system level [arxiv:1905.04071 §4.3.2].

### Question answering metrics and questionnaires

Ranked open QA uses MAP, MRR, precision, recall and F1. Multiple choice uses
accuracy. Extractive spans use exact match and token F1 [arxiv:1905.04071 §5.3].
For interactive QA the survey reviews NASA TLX, a task questionnaire and a system
questionnaire, and finds the task questionnaire best at telling systems apart
[arxiv:1905.04071 §5.3]. It proposes multi-reference evaluation for the case
where many answers are correct [arxiv:1905.04071 §5.3].

### LLM-era automatic metrics and human criteria

For LLM-based dialogue systems, the 2024 survey lists its evaluation metrics,
grouped by system kind [arxiv:2402.18013 §5.1]. For
task-oriented systems: joint goal accuracy, slot accuracy and average goal
accuracy, a requested-slot macro F1, BLEU and entity F1 [arxiv:2402.18013
§5.1.1]. For open-domain systems: perplexity, BLEU, DIST-n and recall@K
[arxiv:2402.18013 §5.1.2]. The recall@K formula as typeset uses an index that
is never defined, so it cannot be implemented from the text alone
[arxiv:2402.18013 §5.1.2 eq. 32]. For human evaluation the survey lists six
criteria: diversity, creativity, grammar, faithfulness, coherence and
consistency [arxiv:2402.18013 §5.2]. The card records no correlation with
human judgement for any of these metrics, so this variant is a checklist of
what to measure, not evidence about which measure to trust. The
macro-F1 formula also uses a class count that the text does not define
[arxiv:2402.18013 §5.1.1 eq. 30]. Card:
[A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](/papers/2402.18013).

### Crowd-labelled quality, safety and groundedness

For a dialogue model whose replies are scored on fixed test sets. The paper
notes that automatic dialogue metrics may not correlate well with human
judgement, so it relies on crowdworker labels instead [arxiv:2201.08239 §2].
Quality: five raters label each reply, and a label is "yes" if at least three
agree. Specificity and interestingness are asked only after a "yes" on
sensibleness [arxiv:2201.08239 §5]. Safety: three raters, and an objective is
met if at least two say "no". A reply scores 1 only if every objective passes
[arxiv:2201.08239 §5]. Groundedness: three raters, majority vote
[arxiv:2201.08239 §5]. The test sets are a 1,477-item Mini-Turing Benchmark for
quality, a 1,166-dialog holdout for safety, and 784 turns from the Wizard of
Wikipedia set for groundedness [arxiv:2201.08239 §5].

At 137B the paper reports sensibleness 92.3, specificity 79.0, safety 95.2 and
groundedness 73.2 [arxiv:2201.08239 app. F] [arxiv:2201.08239 §7]. Human
baselines appear only in figures [arxiv:2201.08239 fig. 4] [arxiv:2201.08239
fig. 5]. The authors say crowdworkers were not extensively trained or
incentivised, so beating them on interestingness may mean little
[arxiv:2201.08239 §7]. Safety is one binary score across all objectives, which
cannot express degrees of harm [arxiv:2201.08239 §9] [arxiv:2201.08239 app. A.1].
The objectives and crowdworkers are US-based [arxiv:2201.08239 §9]
[arxiv:2201.08239 app. A.3], and the crowd datasets are not released
[arxiv:2201.08239 app. A.2], so the test sets cannot be rebuilt from the paper.
Card: [LaMDA: Language Models for Dialog Applications](/papers/2201.08239).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Task bot with logs and some questionnaire ratings | Task success and cost, with a PARADISE-style model | Scores come from logs. The model needs ratings to fit, and transfer across systems was uneven [arxiv:1905.04071 §3.4.1] [arxiv:1905.04071 tab. 6] |
| Need a score for each exchange while the dialogue runs | Learned interaction-quality model | Built on log features. Needs expert ratings to train [arxiv:1905.04071 §3.4.1] |
| No users available to test a dialogue manager | User simulator | Tests without people, but only reproduces what it encodes. Check it against a user study [arxiv:1905.04071 §3.4.2] [arxiv:1905.04071 §7] |
| Open-ended chatbot with no human labels | Learned metric such as RUBER, checked on a human sample | Reported to correlate with human judgement more than word overlap does. Fragile to reversed and dull replies [arxiv:1905.04071 §4.3.1] |
| Comparing systems on one test set | System-level scores | Reported correlations are higher at system level than at utterance level [arxiv:1905.04071 §4.3.1] [arxiv:1905.04071 §4.3.2] |
| Reply selection from candidates | Voted appropriateness over annotator votes | Recall@k can count valid alternatives as errors [arxiv:1905.04071 §4.3.2] |
| Single-turn or interactive QA | Correctness metrics, plus a task questionnaire for interactive use | Correctness is scored per answer. The task questionnaire separated systems best in the survey [arxiv:1905.04071 §5.3] |
| LLM-based multi-turn system, task or open-domain | The metrics listed for its kind, plus human ratings on the six criteria | The LLM survey lists the metrics but gives no agreement figures for them. Validate any score on a human sample first [arxiv:2402.18013 §5.1] [arxiv:2402.18013 §5.2] |
| Quality, safety or groundedness of a dialogue model on fixed sets, with a labelling budget | Crowd labels with a stated rule per criterion | Each criterion has its own labelling rule. The paper's sets are not released, so labels must be collected again [arxiv:2201.08239 §5] [arxiv:2201.08239 app. A.2] |

The survey's cited results give little reason to use word-overlap metrics alone
for single replies [arxiv:1905.04071 §4.3.1]. The 2022 chatbot survey reaches
the same general view without numbers [arxiv:2201.06657 §4.3] [arxiv:2201.06657
§6]. The 2019 survey's validity results do not cover large language models
[arxiv:1905.04071 §7], and the 2024 survey gives no validity results for its
metrics [arxiv:2402.18013 §5]. The LaMDA card records
labelling rules and a human-consensus reference, but no agreement statistic
between its raters [arxiv:2201.08239 §5]. So no choice above is validated for
LLM-based systems.

## Implementation sketch

The common core is to test any automatic score against human ratings before
trusting it. Averaging per system is reported to raise agreement, so keep the
two levels apart.

```pseudocode
# Check an automatic dialogue score against human judgement  [§2.2]
# Input:  per reply, an automatic score and a human rating
# Output: agreement at utterance level and at system level  [§4.3.1]

def utterance_agreement(auto_scores, human_ratings):
    # Rank correlation between the metric and the ratings  [§4.3.1]
    return spearman(auto_scores, human_ratings)

def system_agreement(system_auto, system_human):
    # Same check on per-system averages. Not comparable with the
    # utterance-level number  [§4.3.1]
    return spearman(system_auto, system_human)

def check_metric(data):
    # Also test reversed and dull replies: a fragile metric scores
    # them too high  [§4.3.1]
    reversed_gap = score_reversed(data) - score_reference(data)
    return reversed_gap
```

## Contradictions

No conflicting numbers across the cards: the 2022 chatbot survey, the 2024
card and the LaMDA card give no correlation with human judgement to compare
with the 2019 survey. Inside the 2019 survey, the IQ_lab correlation is 0.894
in the text and 0.888 in its table [arxiv:1905.04071 tab. 7].

On word-overlap metrics the two surveys read the evidence differently. The 2019
survey says BLEU is more competitive at corpus and system level
[arxiv:1905.04071 §7]. The 2022 survey calls automatic n-gram scores weak for
dialogue and does not separate corpus or system level [arxiv:2201.06657 §4.3].
Both sides are reported here, and neither is resolved.

## Papers

- [Survey on Evaluation Methods for Dialogue Systems](/papers/1905.04071): surveys how task, conversational and interactive QA systems are evaluated, with the satisfaction models, simulators, learned metrics and datasets for each. It runs no experiments.
- [A Literature Survey of Recent Advances in Chatbots](/papers/2201.06657): a 2007 to 2021 literature review of chatbot designs, datasets and evaluation. It runs no experiments, and it reports that no shared evaluation framework exists.
- [LaMDA: Language Models for Dialog Applications](/papers/2201.08239): a family of dialogue models whose quality, safety and groundedness are scored with crowd labels. The numbers are author-reported, and the crowd datasets are not released.
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](/papers/2402.18013): surveys LLM-based task-oriented and open-domain dialogue, with the metrics and datasets used for each. It runs no experiments, and its card flags several formula and table errors to check before use.