A Literature Survey of Recent Advances in Chatbots

In plain words

This paper is a map of the chatbot field, written as a literature review. The authors collected a large set of research on chatbots, sorted it, and described how chatbots are built. They describe three main ways to build one. The oldest ones follow hand-written rules. Later ones pick a reply from a stored list of answers. The newest ones write a reply word by word with a neural network, and the Transformer design is the one the survey singles out as the current model of choice.

The review also covers what data these systems are trained on, how their quality is measured, and where chatbots are used, such as customer support, education and health. The authors find that the research side (large, general conversational models) and the industry side (small, task-specific bots) are far apart. Measuring chatbot quality is still unsettled: the usual text-overlap scores do not capture whether a reply makes sense, and human judging is slow and expensive.

TL;DR

• A survey of chatbot research from 2007 to 2021, with the search and screening steps spelled out §3.
• It compares rule-based, retrieval-based and generative chatbots, and treats Seq2Seq and Transformers as the main generative designs §4.1.
• It describes three open-domain training corpora (OpenSubtitles, Cornell, DailyDialog) and gives formulas for F1, perplexity, BLEU, METEOR and TER §4.2 §4.3.
• Its conclusions: no shared evaluation framework, a split between open-domain research and closed-domain industry bots, and few learned retrieval models §6 §7.
• It is a review, not a method: there is nothing to implement, but its formulas need checking before reuse (see Pitfalls).

Problem

The survey asks what current chatbot research looks like and where it is weak. It targets readers who need an overview of design choices, datasets and evaluation, which earlier surveys covered only in part §1 §5.

Method

The paper is a literature survey. It has two stages §3.

Stage one: information gathering §3.1.

1. Search IEEE, ScienceDirect, Springer, JSTOR, arXiv and Google Scholar, starting with the seed term "chatbot" §3.1 tab. 1. Scopus appears only in the publication-trend figure, not as a search source §3.1 fig. 1.
2. Add seven terms that appeared in the retrieved papers, each in quotation marks: "conversational modelling", "conversation systems", "conversational system", "conversational entities", "conversational agents", "embodied conversational agents" and "Human-Computer Conversational Systems" §3.1 tab. 1.
3. Use the databases the authors could access for free or through their institution §3.1.

Stage two: filtering and reviewing §3.2.

1. Keep articles published from 2007 to 2021. The authors justify 2007 by a publication peak in the Scopus counts §3.2.1 fig. 1.
2. Screen by title. Keep articles that show concrete technical implementations, and literature reviews or surveys. Drop marketing and social-science studies, and papers where the chatbot is only a means to an end §3.2.1.
3. Cover arXiv and the other sources, then de-duplicate with a reference manager. Then read the bibliographies of the collected papers and retrieve further papers through Google Scholar §3.2.1.
4. Read the abstracts of the 316 selected articles and record their main objectives. Then assign them to five groups: history and evolution, implementation, evaluation, applications, and datasets §3.2.2.

The section on methodology states that articles were classified on four aspects (design, implementation, application and evaluation), while the list of groups has five entries. Treat the five-group list as the operative one §3 §3.2.2.

Evaluation metrics as the survey defines them §4.3. These are reproduced here only in the form given in the paper. The survey does not state several parameters, marked "not stated".

• F1 eq. 1. The text calls it the harmonic mean of precision and recall, but the printed formula is precision times recall divided by their sum, without the factor 2 that a harmonic mean needs. Use 2PR/(P+R) for the harmonic mean; do not copy eq. 1 as printed.
• Perplexity over a test set of t tokens eq. 2: the inverse of the geometric mean of the model probabilities of each token given its history.
• BLEU eq. 3 eq. 4: the brevity penalty BP is 1 if the candidate length c exceeds the reference length r, and exp(1 − r/c) otherwise. BLEU is BP times the exp of a weighted sum of log modified n-gram precisions P_n. The maximum n-gram order N and the weights w_n are not stated.
• METEOR eq. 5: unigram precision P = m/t and recall R = m/r, where m is the number of mapped unigrams. The survey gives only the parameterised harmonic mean F_mean, with alpha not stated. It does not describe the penalty term or how the final score is formed.
• TER eq. 6: the number of edits divided by the average number of reference words. The survey does not say what one edit is or how several references are averaged.

# BLEU as written in the survey  [eq. 3] [eq. 4]
# Input:  precisions: P_n for n = 1..N (modified n-gram precisions)
#         weights: w_n, positive, sum to 1 (values not stated)
#         c: candidate length; r: reference corpus length
# Output: BLEU score

def bleu(precisions, weights, c, r):
    # Brevity penalty: 1 if the candidate is longer  [eq. 3]
    if c > r:
        bp = 1.0
    # Otherwise exp(1 - r / c)  [eq. 3]
    else:
        bp = exp(1 - r / c)
    # Weighted log-average of the precisions  [eq. 4]
    log_sum = sum(w * log(p) for w, p in zip(weights, precisions))
    return bp * exp(log_sum)

Dialogue model families described §4.1.

• Rule-based: pattern matching and hand-written rules, such as ELIZA and AIML-based systems, with limited coverage and high maintenance §2 §4.1.1.
• Retrieval-based: a knowledge base of question-answer pairs and an index over it; the reply is picked from the stored answers. The survey also covers a learned ranking that uses previous turns as context §4.1.2.
• Generative Seq2Seq: an encoder-decoder pair of recurrent networks. The encoder squeezes the input into one fixed-length context vector, which the decoder unrolls word by word. Decoding is either beam search or greedy §4.1.2.
• Transformers: attention-only models, with BERT and GPT as pretrained examples. Transformer-XL and Reformer are described as variants for longer context and lower cost §4.1.2.
• Meena: an end-to-end generative chatbot with an Evolved Transformer, trained on multi-turn dialogue with up to 7 context turns, and scored with the Sensibleness and Specificity Average (SSA) human metric §4.1.2. The survey does not give the SSA formula.

Hyperparameters

This is a survey: nothing is trained. The table lists the parameters of the review itself, which are needed to reproduce the search.

Results

Author-reported. The paper reports no experimental results. The numbers below are its counts and its descriptive statistics.

Search and selection counts tab. 1. The "selected" column sums to 316, matching the text: IEEE 22, ScienceDirect 20, Springer 16, arXiv 56, Google Scholar 201, JSTOR 1. The article counts are per source and term, as retrieved; the survey did not process the large totals (it reports one search engine returning over 45,000 articles) §3.2.1 tab. 1.

Training corpora tab. 2 §4.2.

For Cornell and DailyDialog the authors state that the source gives no token count, so they tokenised the corpora with the Keras text tokeniser §4.2.

Headline findings, as the authors state them.

• Open-domain corpora such as OpenSubtitles, Cornell and DailyDialog are the most used for training. No closed-domain dataset appears repeatedly §4.2.
• Chatbots are split: open-domain generative models are mostly social or companion bots, while task bots use retrieval or rules on small, often confidential datasets §4.4.
• No common evaluation framework exists. Automatic n-gram metrics are weak for dialogue, and human evaluation is costly and hard to compare §4.3 §6.
• Few learned retrieval models exist, and the survey sees a gap between large research models and industry needs §6 §7.

Paper vs code

No official code. The paper is a survey with no code, and its LaTeX source links no repository. The arXiv record links none either. Nothing to compare.

Pitfalls

• F1 formula is wrong as printed. eq. 1 omits the factor 2 of the harmonic mean. The text says harmonic mean. Implement 2PR/(P+R).
• METEOR is incomplete. eq. 5 is the parameterised harmonic mean only. The survey gives no alpha value and no penalty term. Do not implement a METEOR score from this card; use the original metric definition.
• BLEU parameters missing. N, the weights w_n and the handling of a zero precision (where log is undefined) are not stated.
• Perplexity notation. eq. 2 uses t without defining it beyond the test-set length, and writes P_M and PM interchangeably.
• Inconsistent category count. The methodology text names four aspects, the list has five categories §3 §3.2.2.
• Scopus contradiction. The survey says Scopus cannot search by keyword §3.1, yet fig. 1 reports Scopus counts for keyword queries. How those counts were produced is unclear.
• DailyDialog turn count conflicts. The text says about 8 turns per speaker §4.2; the table says 7.9 turns each on average tab. 2. The two may measure different units.
• Token counts look like vocabulary sizes. For Cornell and DailyDialog, the token count is smaller than the phrase count. That is impossible if every phrase has at least one token, unless the column is a vocabulary size or counts something else. The survey gives no more detail than the tokeniser named in §4.2. Verify before using these numbers.
• Selection is by title, then abstract. Screening was done by title and then by abstract, with no inclusion or exclusion log. The 316-article set cannot be reproduced exactly from the paper.
• Dates and sources drift. The arXiv access date is ambiguous §3.2.1. Google Scholar and arXiv totals are search-engine figures and may have changed since 2022.
• Survey claims are not measurements. Statements such as "no common framework" and the open-vs-closed split are the authors' reading of the selected papers §4.3 §6.

Lineage

• Builds on: nothing specific. The survey positions itself against earlier chatbot reviews, which it summarises in its Related Works section §5.
• Compared against: the reviews discussed in §5, among them a comparison of design techniques from nine papers, a discussion of chatbot shortcomings, a cloud-platform and programming review, a systematic literature review, a comparison of eleven application systems, a history-and-platforms overview, and a deep-learning tools overview.
• Superseded by: not stated in this card.