wikillm

Paper · 2018-09-21 · cs.CL

Towards Exploiting Background Knowledge for Building Conversation Systems

Holl-E is a crowdsourced set of about 9,000 movie chats in which each bot reply copies or lightly edits a sentence from a plot, review, comment or fact table about the movie. Baselines: HRED, a copy-or-generate model (GTTP) and span prediction (BiDAF). Best author-reported result: BiDAF on mixed-short resources, F1 45.73 single-reference and 51.35 multi-reference.

Dialogue generationExtractive question answeringSequence-to-sequence learningdatasetdialoguebackground-knowledgecopy-mechanismspan-prediction

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Most chatbot training data is a plain list of messages. A model learns to reply by imitating the messages before it, and it never sees any facts about the topic. People do not talk that way. When they discuss a film, they bring in the plot, a review or a fun fact they read earlier.

The authors built a dataset of movie chats in which every bot reply comes from written material about that film: a plot summary, reviews, online comments or a small fact sheet. The person writing the bot's replies was asked to copy a sentence from that material and change it a little so that it fits the conversation. Each reply therefore points back to the text it came from.

They then tried three kinds of model on this data. One writes replies without reading the material. One can copy words from the material or write new ones. The third only picks a stretch of the material as the reply. The span picker scored highest on the automatic metrics, but the authors judge all three as still far from good enough.

TL;DR

  • Holl-E (the name of the released repository; the paper gives no dataset name): 9,071 crowdsourced movie chats, 90,810 utterances, 921 movies. Bot turns copy or edit text from one of four resources: plot, review, comments or fact table §3 tab. 2.
  • Three baseline families: HRED (no background knowledge), a modified Get-To-The-Point copy-or-generate model (GTTP), and BiDAF span prediction §4.
  • Best author-reported result: BiDAF on the mixed-short resource, F1 45.73 single-reference and 51.35 multi-reference tab. 3.
  • The repository holds data-conversion and metric scripts but no model or training code. The baseline models are external projects. Several script details differ from the paper (see Paper vs code).

Problem

Large dialogue corpora contain only utterances and replies, with no explicit background text behind each reply §1 §2. Models trained on them learn to treat a chat as sequence-to-sequence text generation, and their output is often ungrammatical or off-topic §1. Earlier knowledge-aware dialogue work either targets templated, goal-oriented chats or adds knowledge after the dataset was built §2. The paper's goal is a dataset where each reply is explicitly linked to the background text it was made from §1 §2.

Two design problems shape the data. Live two-person chats on a crowdsourcing platform rarely synchronise, so one worker plays both speakers §3.4. A single reference reply undercounts the valid answers, so the test set gets extra references §5.5.

Method

Dataset collection §3. The stages, in order:

  1. Movies §3.1. 921 movies from popular recent titles, the IMDb top 250, popular genres and other public lists, spanning 22 genres. Movies were kept only when enough background text existed online.
  2. Background resources §3.2. Four per movie:
    • Review (R): two popular IMDb reviews per movie §3.2. Each chat's data record holds one picked review main_data.md.
    • Plot (P): the Plot section of the movie's Wikipedia page.
    • Comments (C): the first comment of each thread on the movie's official discussion page on Reddit; IMDb review titles if no such page exists.
    • Fact table (F): four fields only (box office, similar movies, awards and taglines), taken from IMDb and the Wikipedia infobox §3.2.
  3. Opening statements §3.3. Three questions per movie (favourite scene, favourite character, general opinion) were answered on Mechanical Turk by three workers each, giving nine openers per movie. A random opener starts each chat.
  4. Chats §3.4. One worker plays both speakers and gets three or four of the movie's resources. The worker adds at least eight utterances.
    • Speaker 1 writes freely.
    • Speaker 2 must copy or modify sentences from the resources. Speaker 2 may add words before or after the copied span.
    • Turns alternate, so the even-numbered utterances are the bot replies §5.2.
  5. Verification §3.5. An in-house evaluator checked each chat. A separate study had 15 evaluators rate 500 chats, each by three evaluators, on five criteria from 1 to 5 tab. 1.

Training instances §5.2. Each Speaker 2 turn becomes one instance of resource, context and response. The context is the query (the utterance just before the reply) plus the two utterances before it. BiDAF is trained on the copied span instead of the full reply.

Pseudocode
# Build instances from one chat  [§5.2] [§3.4]
def build_instances(chat, labels, spans, documents):
    instances = []
    # Replies are the Speaker 2 turns, at odd 0-based indices  [§5.2]
    for turn in range(1, len(chat), 2):
        # Turns that copy from no resource (label 4) are skipped  [§5.2]
        if labels[turn] == 4:
            continue
        # Context: the query and up to two utterances before it  [§5.2]
        context = " ".join(chat[max(0, turn - 3):turn])
        # Exact copied span; the example is dropped if it is not found
        # [§3.4]
        span = spans[turn]
        label = labels[turn]
        # Oracle: the labelled resource, cropped to 256 words  [§5.3]
        oracle = crop_around(documents[label], span, 256)
        # Mixed: all resources merged in proportion to length  [§5.3]
        mixed_short = merge_resources(documents, label, span, 256)
        mixed_long = merge_resources(documents, label, span, 1200)
        instances.append(
            (oracle, mixed_short, mixed_long, context, span, chat[turn]))
    return instances

Resource settings §5.3. Three ways to give a model the background text:

  • Oracle: only the resource the reply came from, cropped to 256 words so that the copied span stays inside the window. The crop start is random.
  • Mixed-short: all four resources merged into one document of 256 words. Each resource keeps a share of the window proportional to its length, and the window always contains the copied span.
  • Mixed-long: the same merge with a 1,200-word window, the maximum length of the merged resources.

Oracle is a test-time shortcut: it uses the gold resource label, so it does not measure resource selection.

Models §4:

  • HRED §4.1: the standard hierarchical encoder-decoder, not the variational variant. A lower RNN encodes each utterance, an upper RNN encodes the sequence of utterances, and a decoder writes the reply. HRED ignores the resource §5.2.
  • GTTP, copy-or-generate §4.2 app. Model details - GTTP: a pointer-generator adapted to a document and a dialogue context. Each has its own RNN, and the decoder attends over both. A learned gate decides, per step, between generating from the vocabulary and copying from the document. The equations below are the paper's eq. 1-eq. 4; x_t is the previous output word and s_t the decoder state after reading it.

e_i^t = v^\top \tanh(W_r h_i^r + U s_t + b_r), \quad a^t = \mathrm{softmax}(e^t), \quad r_t = \sum_i a_i^t h_i^r eq. 1

f_i^t = v^\top \tanh(W_c h_i^c + V s_t + b_c), \quad m^t = \mathrm{softmax}(f^t), \quad c_t = \sum_i m_i^t h_i^c eq. 2

p_{gen} = \sigma(w_r^\top r_t + w_c^\top c_t + w_s^\top s_t + w_x^\top x_t + b_g) eq. 3

P(w) = p_{gen} P_{vocab}(w) + (1 - p_{gen}) \sum_{i:\, w_i = w} a_i^t eq. 4

  • BiDAF, span prediction §4.3: the SQuAD reading-comprehension model, used without change. The context is the question and the resource is the document, and the model predicts the start and end of a span. The paper describes it as a six-layer architecture.

Metrics §5.4. HRED and GTTP use BLEU-4 and ROUGE-1, -2 and -L. BiDAF is scored by comparing its predicted span with the reference span using the same metrics, plus F1 in the SQuAD sense. Human judgements use 100 test responses per model and setup, each rated from 1 to 5 for fluency, appropriateness, humanness and specificity tab. 4.

Splits §5.1. Chats are split 80/10/10 by movie, so a movie appears in exactly one split. There are about 9.14 chats per movie.

Multi-reference test set §5.5. For each Speaker 2 turn in the test set, workers were asked for three reference replies from the same resources. The authors note that some turns had only one appropriate reply. 78.04% of test instances have more than one valid reply. Multi-reference scores take the maximum over the references tab. 3.

Dataset statistics tab. 2 §3.6:

ItemValue
Chats9,071
Movies921
Utterances90,810
Average utterances per chat10.01
Average words per utterance15.29
Average words per chat153.07
Average words in Plot / Review / Comments / Fact table186.10 / 384.44 / 123.81 / 33.47
Unique Plots / Reviews (paragraphs), unique Comments (sentences)5,157 / 1,817 / 12,740
Share of Speaker 2 replies from Reviews / Comments / Plots / Fact table41.2% / 34.6% / 16.1% / 8.1%

Dataset quality, five criteria, mean of 1 to 5 scores tab. 1:

CriterionRating (mean ± SD)Krippendorff's αCohen's κ
Intelligible4.47 ± 0.520.700.69
Coherent4.33 ± 0.930.570.71
Grammar4.41 ± 0.560.600.69
Two-person chat4.47 ± 0.460.640.70
On topic4.57 ± 0.430.720.70

Hyperparameters

Values as stated in the paper's appendix app. Hyper-parameters, unless a different source is given. The baseline values are the paper's reports of settings from the original baseline papers; the repository does not contain those training codes.

NameValueWhereNotes
HRED optimiserAdam, initial learning rate 0.001app. Hyper-parametersfrom the original HRED paper
HRED batch size / dropout16 / 0.25app. Hyper-parameters
HRED embeddingspre-trained GloVe, 300-dapp. Hyper-parameters
HRED RNNGRU, hidden size 300, all encoders and decoderapp. Hyper-parameters
HRED vocabulary20,000 most frequent wordsapp. Hyper-parametersrepository HRED export has no cap; see Paper vs code
GTTP optimiserAdagrad, initial learning rate 0.15, initial accumulator 0.1app. Hyper-parameters
GTTP batch size16app. Hyper-parameters
GTTP RNNLSTM, hidden size 256app. Hyper-parameters
GTTP gradient clippingmaximum norm 2app. Hyper-parameters
GTTP stoppingearly stopping on validation lossapp. Hyper-parameterspatience not stated
GTTP / BiDAF context limit65 tokensapp. Hyper-parametersrepository applies it to the BiDAF export only; see Paper vs code
BiDAF optimiserAdaDelta, initial learning rate 0.5app. Hyper-parameters
BiDAF batch size32app. Hyper-parameters
BiDAF RNNLSTM, hidden size 256, six-layer architectureapp. Hyper-parameters §4.3
BiDAF dropout0.2 on all LSTM layers and on the pre-softmax linear layerapp. Hyper-parameters
BiDAF embeddingspre-trained GloVe, 100-dapp. Hyper-parameters
Oracle / mixed-short window256 words§5.3repository counts tokens from an absent helper module
Mixed-long window1,200 words§5.3
Train / validation / test splitabout 80% / 10% / 10% of chats, by movie§5.1exact counts per split not stated
Training targetsSpeaker 2 turns (even-numbered)§5.25 instances for a 10-turn chat
Test referencesthree reference replies requested per turn§5.578.04% of test instances have multiple valid replies
Not statedepochs, learning-rate schedule, beam width, maximum decoding length, GloVe fine-tuning, GTTP and HRED dropout, hardware for GTTP and HRED

Results

Author-reported, on the Holl-E test set. Each cell gives the single-reference score followed by the multi-reference score tab. 3. The F1 column applies only to span prediction. "not reported" means the paper gives no value for that model.

ModelF1 (single / multi)BLEU-4 (single / multi)ROUGE-1 (single / multi)ROUGE-2 (single / multi)ROUGE-L (single / multi)
HREDnot reported5.23 / 5.3824.55 / 25.387.61 / 8.3518.87 / 19.67
GTTP (oracle)not reported13.92 / 16.4630.32 / 31.6017.78 / 21.2125.67 / 27.83
GTTP (mixed-short)not reported11.05 / 15.6829.66 / 31.7117.70 / 19.7225.13 / 27.35
GTTP (mixed-long)not reported7.51 / 8.7323.20 / 21.559.91 / 10.4217.35 / 18.12
BiDAF (oracle)39.69 / 47.1828.85 / 34.9839.68 / 46.4933.72 / 40.5835.91 / 42.64
BiDAF (mixed-short)45.73 / 51.3532.95 / 39.3945.69 / 50.7340.18 / 45.0143.46 / 46.95

Human judgements of 100 responses per model tab. 4, scale 1 to 5:

ModelHumannessAppropriatenessFluencySpecificity
HRED3.082.492.642.06
GTTP (oracle)4.103.734.033.33
GTTP (mixed-long)2.932.973.422.60
BiDAF (oracle)3.783.714.053.76
BiDAF (mixed-short)3.413.383.473.30

The authors' reading of these numbers §6:

  • Span prediction beats the generation models in the oracle setting, but the authors say generation is far from acceptable and that picked spans are not natural replies. GTTP (oracle) is rated more human-like than both BiDAF settings.
  • HRED and GTTP (mixed-long) score about the same, while GTTP (oracle) is well ahead. HRED's replies are very generic (specificity 2.06).
  • BiDAF scores better on mixed-short than on oracle, which the authors read as a hypothesis: the model learns to ignore the noisy sentences. They note it was not verified.
  • The copy-or-generate model copied 82.18% of output tokens in the oracle setting and 71.95% in mixed-long. Its copied spans often do not fit the context.
  • Multi-reference scoring raises every score, which the authors use to argue for metrics that allow several valid replies §6.

Paper vs code

The repository is nikitacs16/Holl-E at 06e8902, linked from the paper's footnote §3. Its owner name matches the first author, so it is taken as the authors' repository. It holds the data documentation, the data-conversion scripts and the metric scripts README.md. It contains no model or training code. The HRED, BiDAF and GTTP baselines link to separate projects README.md. The dataset itself is a Google Drive download README.md, which was not fetched, so the counts in the paper were not checked against released files.

Differences found by reading the scripts:

  • First reply's context. The paper defines the context as the previous two utterances plus the query §5.2. For the first reply, the script uses the query alone convert_main_data_to_raw_data.py.
  • GTTP and the 65-token limit. The paper says both GTTP and BiDAF use a 65-token context app. Hyper-parameters. The script truncates only the BiDAF (SQuAD-format) export convert_as_expt.py. The GTTP export keeps the full context convert_as_expt.py.
  • Response cap. Replies longer than 80 tokens are cut to 80 tokens while keeping the span. The paper does not say so convert_main_data_to_raw_data.py convert_main_data_to_raw_data.py.
  • HRED history cap. The HRED export keeps only the last 90 tokens of history plus query. The paper does not say so convert_as_expt.py.
  • Vocabulary. The paper gives 20,000 words for HRED. The HRED export writes no vocabulary convert_as_expt.py. The GTTP export's vocabulary file keeps the 25,000 most frequent words convert_as_expt.py.
  • Window units. The paper counts 256 and 1,200 words §5.3. The script counts tokens from a helper module that is not in the repository convert_main_data_to_raw_data.py convert_main_data_to_raw_data.py.
  • Multi-reference scoring. The paper takes the maximum over references §5.5. The ROUGE script picks the single reference with the highest ROUGE-L and reports that reference's ROUGE-1 and ROUGE-2 evaluate_multi_rouge.py. So ROUGE-1 and ROUGE-2 are not per-metric maxima.
  • BLEU aggregation. evaluate.py computes corpus-level BLEU evaluate.py. evaluate_multi_bleu.py computes a per-example maximum over references evaluate_multi_bleu.py. The paper does not say which one it used.
  • Broken average. evaluate_multi_bleu.py initialises count to 0 and never increments it, then divides the total by it. As written, the final print divides by zero evaluate_multi_bleu.py evaluate_multi_bleu.py.
  • Reference normalisation is one-sided. The scripts lower-case and word-tokenise the references only, not the predictions evaluate_multi_rouge.py evaluate_multi_bleu.py.
  • Line joining. evaluate.py joins the lines of each output file with no separator evaluate.py, so the last word of one line runs into the first word of the next.
  • Missing inputs. The conversion script imports token_utils, and the metric scripts import rouge and bleu. None of these is in the repository, and the multi-reference file multi_reference_test.json is not there either convert_main_data_to_raw_data.py evaluate.py evaluate_multi_bleu.py. The scripts cannot be rerun from this repository alone.
  • Python 2. The metric scripts use reload(sys) and setdefaultencoding evaluate_multi_bleu.py.
  • Span filter. Examples whose span cannot be located are dropped and only counted convert_main_data_to_raw_data.py. The paper does not report how many were dropped.

Pitfalls

  • Targets are not always the copied span. Speaker 2 may add words before or after the copied span §3.4. The released spans are only the copied part, so a span model cannot reproduce those added words. Full replies are therefore not recoverable from spans.
  • Gold resource at test time. The oracle setting crops around the true resource label, which a deployed system would not know. The paper suggests a two-stage model that first selects the resource §6, but does not evaluate one.
  • Random crops. The oracle crop start and the discarded words for proportional merging are drawn at random with seed 123 convert_main_data_to_raw_data.py convert_main_data_to_raw_data.py. Regenerating the data with different randomness changes the windows, so use the released files for comparable results.
  • Proportional merge rules. Shares are rounded down to integers. If the span is longer than the label's share, that share grows and the difference is taken from the first other resource with room convert_main_data_to_raw_data.py. The paper says only "proportional to length" §5.3.
  • Review source. The paper says two reviews per movie were fetched §3.2. The data field holds one picked review per chat main_data.md. Which of the two a chat uses is not stated.
  • BiDAF scores measure spans. Because the reply can contain added words §3.4, BiDAF's target is not always the full reply. Its BLEU and ROUGE are span-to-span scores, so they do not directly measure reply quality.
  • Copy vocabulary. The paper's copy equation sums attention over source positions eq. 4. It does not say whether source-only words can be emitted when they are missing from the output vocabulary.
  • Human evaluation coverage. The text says 100 responses for every model and every setup §5.4. Table 4 has no GTTP (mixed-short) row, and the number of raters per response is not stated.
  • Setup list mismatch. The Results text lists four configurations §5.3. Table 3 reports six, including GTTP (mixed-short).
  • F1. The paper reports F1 for BiDAF §5.4. The repository defines an F1 function evaluate_multi_rouge.py, but its only use is commented out evaluate_multi_rouge.py. The normaliser lower-cases and word-tokenises but does not drop punctuation or articles, though helpers for both exist and are unused evaluate_multi_rouge.py.
  • Movie list. The paper names 921 movies and points to an appendix list of source URLs. In the source, that list sits inside a disabled block, so it is not in the compiled paper §3.1.
  • Baselines unverifiable here. The GTTP, BiDAF and HRED hyperparameters come from the original papers and external repositories. This repository does not contain those codes, so the numbers cannot be checked against code.

Lineage

  • Builds on: HRED §4.1, the pointer-generator Get To The Point §4.2 (modified here), BiDAF §4.3, and the self-dialogue collection idea from earlier work in which one worker plays both sides §3.4 §2.
  • Compared against: the three baseline families above. The authors chose the modified copy-or-generate model over other knowledge-copy variants because it already has the extra encoder for the background document §4.2.
  • Superseded by: not stated in this wiki.

Source and license

Adapted from “Towards Exploiting Background Knowledge for Building Conversation Systems” by Nikita Moghe, Siddhartha Arora, Suman Banerjee, Mitesh M. Khapra (arXiv:1809.08205), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/nikitacs16/Holl-E at commit 06e8902b70b3 (none (no LICENSE file in repository)). Any quoted code is shown under that license.