Survey on Evaluation Methods for Dialogue Systems

In plain words

Building a chatbot or a voice assistant is easy compared with knowing whether it is any good. The usual answer is to have people talk to it and fill in a questionnaire, but that is slow and expensive, and it has to be repeated every time the system changes.

This survey goes through the ways researchers have tried to judge dialogue systems with less human effort. It splits the field into three kinds of system. Task bots, such as one that books a restaurant, can be judged by whether they got the user what they wanted and how many turns it took. Open-ended chatbots are much harder to judge, because nobody agrees on what a good reply is. Question-answering systems sit in between. For each kind, the survey explains the common measures, the models that try to predict human ratings from logs, the simulated users used for testing, and the datasets and public challenges. It ends with the open problems, such as the lack of a reliable automatic score for chatbots.

TL;DR

A survey, not a method paper: it runs no experiments of its own. It separates three classes of dialogue system (task-oriented, conversational, interactive QA) and reviews evaluation for each §2.1 §3 §4 §5. The main methods for task systems are user-satisfaction models, which predict questionnaire scores from logged features, and user simulators §3.4. For chatbots, word-overlap scores such as BLEU are reported to correlate poorly with human judgement, and learned metrics are reported to be fragile §4.3.1. Read it to pick an evaluation family for a dialogue class and to learn the known failure modes of each. Its source is arXiv v2 (2020-06-26), and it does not cover large language models.

Problem

The survey states an evaluation method should be automatic, repeatable, correlated with human judgements, able to tell dialogue strategies apart, and able to explain which features matter §2.2. It also states that no existing procedure meets all five requirements. Most methods still need some human involvement or substantial engineering §2.2 §7.

Method

The survey has no single method. Its structure is the taxonomy below. Table and figure numbers are those in the LaTeX source, in order of appearance.

Dialogue system classes §2.1 tab. 1

• Task-oriented systems have a clearly defined, measurable goal, a structured dialogue, a closed domain and a focus on efficiency §3.1.
• Conversational agents hold open-domain, unstructured dialogue, often with the aim of an engaging conversation §4.1.
• Interactive question answering systems answer questions, and may hold multi-turn dialogue with follow-up questions §5.1.

tab. 1 compares the three classes on task, structure, domain, turns, length, initiative and interface.

Task-oriented systems §3

Goal and constraints §3.4 fig. 4. A dialogue goal is split into constraints (the information to retrieve, such as the venue type or the route) and requests (the information the user wants back, such as the name, address and phone number) §3.4.

Task success §3.4 tab. 5. Success is scored against the goal. The survey describes a confusion matrix per slot key, with the expected value as row and the value the system produced as column tab. 5. It states that the Kappa coefficient can be applied to these matrices, and points to known weaknesses of Kappa. It gives no formula.

Dialogue efficiency §3.4. Cost measures such as the number of turns, the elapsed time, inappropriate repair utterances, and the turns needed to fill one slot.

Dialogue acts and structure §3.2 tab. 2 tab. 3 tab. 4. The domain ontology is a list of slots with types and allowed values tab. 2. Each system or user turn is a dialogue act with a type and arguments, such as inform(food=..., area=...) or request(phone) tab. 3 tab. 4.

Pipeline and end-to-end architectures §3.3. A pipeline is natural language understanding (NLU), dialogue state tracking (DST), a dialogue manager (DM) and natural language generation (NLG) fig. 2 fig. 3. End-to-end systems train one network instead §3.3.2. The survey notes the credit-assignment problem: when the whole system is scored, it is hard to say which module caused a reward §3.3.2.

User satisfaction modelling §3.4.1. Questionnaire ratings are the target. A model is fitted on objective features, such as task success and the word error rate of the speech recogniser §3.4. Three goals are named: explainability, automation and differentiation between strategies §3.4.1.

PARADISE §3.4.1 fig. 5. A linear regression on task success and dialogue costs predicts the questionnaire score for each dialogue. The survey says the original study found only task success and the number of repetitions significant §3.4.1.

# PARADISE-style satisfaction model  [§3.4.1] [fig. 5]
# Input:  per dialogue, objective features (N, k) and questionnaire scores
# Output: a linear model that predicts the score for unseen dialogues

def fit_paradise(features, satisfaction):
    # Target: the questionnaire score collected after each dialogue  [§3.4.1]
    model = linear_regression(features, satisfaction)
    return model

def predict_satisfaction(model, new_features):
    # Linear combination of task success and dialogue-cost measures  [§3.4.1]
    return model.predict(new_features)

Exchange-level Interaction Quality (IQ) §3.4.1 fig. 6. Experts rate each exchange, and the IQ label is the median of several expert ratings. A support vector machine is trained for each target, with features from the system's logs. The survey covers three targets: IQ_field, IQ_lab and US_lab §3.4.1 tab. 7.

# Interaction Quality labels and model  [§3.4.1] [fig. 6]
# Input:  per exchange: automatic features from the logs (AUTO),
#         ratings from several experts
# Output: an IQ score for each exchange, usable during the dialogue

def exchange_label(expert_ratings):
    # The IQ target is the median of the expert ratings  [§3.4.1]
    return median(expert_ratings)

def fit_iq_model(auto_features, iq_labels):
    # One SVM per target (IQ_field, IQ_lab, US_lab)  [§3.4.1] [tab. 7]
    return svm_fit(auto_features, iq_labels)

User simulation §3.4.2. Simulators are used either to train a reinforcement-learning dialogue manager or to test a policy §3.4.2. The most popular one is the agenda-based simulator (ABUS). It keeps a fixed hidden goal and a stack of user actions, and it adapts the stack after each turn §3.4.2. A neural user simulator, which works on surface text and varies its goal, is also described §3.4.2. Simulators can be checked directly (precision and recall on dialogue acts, perplexity) or indirectly (the quality of the dialogue manager trained on them) §3.4.2. The model-based workbench (MeMo) checks whether a simulation leads to the same conclusions as a user study §3.4.2.

Subsystem metrics §3.4.3. NLU: sentence-level semantic accuracy, slot error rate and F-measures. DST: accuracy and L2 distance, since the output is a distribution over states. NLG: F1 or slot error rate for content, and BLEU or ROUGE plus human ratings for surface quality §3.4.3.

Conversational agents §4

Modelling §4.2. Utterance selection ranks candidate replies from a database §4.2.2. Generative models are seq2seq networks, with the hierarchical HRED adding a context encoder over previous turns §4.2.1 fig. 7. Diversity is pursued through the loss (maximum mutual information), a reinforcement-learning reward, latent variables and topics §4.2.1.

Word-overlap metrics §4.3.1. BLEU and ROUGE are described as the early default. A cited study found neither correlates with human judgement. ΔBLEU weights the score by human ratings of the reference replies §4.3.1.

Learned metrics §4.3.1. ADEM is a network trained to predict human appropriateness ratings. RUBER needs no human labels: it combines similarity to the reference reply with relatedness to the context §4.3.1. The survey warns that learned metrics are not robust. A cited re-evaluation found that reversing a reply, or swapping in a dull dummy reply, often raised the ADEM score §4.3.1.

Human-likeness and adversarial evaluation §4.3.1. The Turing test is the classic idea §4.3.1. An adversarial discriminator is trained to tell generated from real dialogue, and its accuracy is used as the score §4.3.1. A cited study compared discriminator and human judgements on three corpora and found low human agreement §4.3.1.

Fine-grained metrics §4.3.1. Topic breadth (number of distinct topic keywords) and topic depth (average length of a same-topic run), with topics classified by a Deep Averaging Network §4.3.1.

Utterance selection metrics §4.3.2. Recall@k measures how often the correct reply is in the top k. Its drawback is that other candidates may also be correct. Weak agreement counts a reply as appropriate if any annotator picked it. Voted appropriateness weights each reply by the number of annotators who picked it §4.3.2.

Question answering systems §5

• Single-turn QA covers open QA and reading comprehension, with extractive, multiple-choice and generative variants §5.1.
• Context QA handles follow-up questions. Interactive QA adds a dialogue module that relaxes or adds constraints when results are too few or too many §5.1.
• Correctness metrics §5.3: MAP, MRR, precision, recall and F1 for ranked open QA; accuracy for multiple choice; exact match and token F1 for extractive spans.
• Dialogue evaluation §5.3: the survey reviews questionnaires for interactive QA (NASA TLX, a task questionnaire and a system questionnaire) and finds the task questionnaire best at telling systems apart. Simulated users are reviewed too. Multi-reference evaluation is proposed against the one-to-many problem §5.3.

Datasets §6

Tables list datasets by class. The survey notes that few public datasets cover a whole task-oriented system §6.1. Sizes are as printed in the tables; see Pitfalls for the typos.

Challenges named: DSTC (from 2013), ConvAI (second edition, 10,981 dialogues) and the Alexa Prize, which released no dataset §6.4.

Hyperparameters

Not applicable: the survey trains no models. It does not state the settings of the cited studies, for example the regression solver for PARADISE or the kernel and C of the IQ support vector machine §3.4.1.

Results

These are numbers the survey reports from the cited studies. They are author-reported in those studies, and the survey did not reproduce them. Setup is given where the survey states it.

Paper vs code

No official code. The arXiv abstract page lists no code, project page or related repository. A GitHub repository search for the exact title returned no results. The paper is a survey, so there is no method name to search for.

Pitfalls

• Numbers come from other studies. Each figure has the setup of its original study: for example, PARADISE on three systems and 544 dialogues §3.4.1, and Interaction Quality on the Let's Go bus system §3.4.1. Do not compare them across rows as if they were on one benchmark.
• Text and table disagree. The IQ_lab score with AUTO + EMO + USER is 0.894 in the text and 0.888 in tab. 7. The cross-model caption of tab. 8 says it reports ρ, κ and UAR, but only ρ appears in the table.
• Kappa has no formula here. The survey names the Kappa coefficient for task success §3.4 but gives no definition or variant. Pick and state one explicitly.
• Word-overlap metrics. Cited work found BLEU and ROUGE do not correlate with human judgement at the utterance level §4.3.1. The survey also says BLEU is more competitive at corpus or system level §7.
• Learned metrics are fragile. Reversing a reply raised the ADEM score in 48.66% of cases, and a dull dummy reply in 86.93% §4.3.1. ADEM's 0.954 correlation is at system level, over averaged scores, so it is not comparable with the utterance-level 0.41.
• Human agreement is low. Human judges agreed poorly in the adversarial study (Fleiss π = 0.3) §4.3.1, and users rate differently from experts §3.4.1. The case for Interaction Quality rests on its correlation with user ratings in one lab study §3.4.1.
• Satisfaction models are per system. The survey notes that a model fitted on one system can transfer to another, but not always. It can fail across user populations §3.4.1 §7.
• Simulators cover only what they encode. A simulator reproduces the behaviour in its corpus or rules and does not generalise to unseen behaviour §7.
• Dataset typos. The text gives 4,232 Twitter triples and 3.35M SubTle dialogues; the tables print 4,322 and 3,35M §6. Check the cited dataset papers for exact counts.
• Dataset counts are not uniform. Some counts are dialogues, some are passages or utterances, and the DSTC6 count is not stated §6.
• Age of the source. The survey is arXiv v2 (2020-06-26). It covers BERT-era QA and seq2seq chatbots, not large language model evaluation.

Lineage

• Builds on: the PARADISE framework §3.4.1 and the Interaction Quality study §3.4.1, both surveyed here, not compiled in this wiki.
• Compared against: word-overlap metrics (BLEU, ROUGE) §4.3.1.
• Superseded by: not stated in this survey.