Paper · 2018-09-21 · cs.CL
Towards Exploiting Background Knowledge for Building Conversation Systems
Holl-E is a crowdsourced set of about 9,000 movie chats in which each bot reply copies or lightly edits a sentence from a plot, review, comment or fact table about the movie. Baselines: HRED, a copy-or-generate model (GTTP) and span prediction (BiDAF). Best author-reported result: BiDAF on mixed-short resources, F1 45.73 single-reference and 51.35 multi-reference.
Dialogue generationExtractive question answeringSequence-to-sequence learningdatasetdialoguebackground-knowledgecopy-mechanismspan-prediction
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Most chatbot training data is a plain list of messages. A model learns to reply by imitating the messages before it, and it never sees any facts about the topic. People do not talk that way. When they discuss a film, they bring in the plot, a review or a fun fact they read earlier.
The authors built a dataset of movie chats in which every bot reply comes from written material about that film: a plot summary, reviews, online comments or a small fact sheet. The person writing the bot's replies was asked to copy a sentence from that material and change it a little so that it fits the conversation. Each reply therefore points back to the text it came from.
They then tried three kinds of model on this data. One writes replies without reading the material. One can copy words from the material or write new ones. The third only picks a stretch of the material as the reply. The span picker scored highest on the automatic metrics, but the authors judge all three as still far from good enough.
TL;DR
- Holl-E (the name of the released repository; the paper gives no dataset name): 9,071 crowdsourced movie chats, 90,810 utterances, 921 movies. Bot turns copy or edit text from one of four resources: plot, review, comments or fact table §3 tab. 2.
- Three baseline families: HRED (no background knowledge), a modified Get-To-The-Point copy-or-generate model (GTTP), and BiDAF span prediction §4.
- Best author-reported result: BiDAF on the mixed-short resource, F1 45.73 single-reference and 51.35 multi-reference tab. 3.
- The repository holds data-conversion and metric scripts but no model or training code. The baseline models are external projects. Several script details differ from the paper (see Paper vs code).
Problem
Large dialogue corpora contain only utterances and replies, with no explicit background text behind each reply §1 §2. Models trained on them learn to treat a chat as sequence-to-sequence text generation, and their output is often ungrammatical or off-topic §1. Earlier knowledge-aware dialogue work either targets templated, goal-oriented chats or adds knowledge after the dataset was built §2. The paper's goal is a dataset where each reply is explicitly linked to the background text it was made from §1 §2.
Two design problems shape the data. Live two-person chats on a crowdsourcing platform rarely synchronise, so one worker plays both speakers §3.4. A single reference reply undercounts the valid answers, so the test set gets extra references §5.5.
Method
Dataset collection §3. The stages, in order:
- Movies §3.1. 921 movies from popular recent titles, the IMDb top 250, popular genres and other public lists, spanning 22 genres. Movies were kept only when enough background text existed online.
- Background resources §3.2. Four per movie:
- Review (R): two popular IMDb reviews per movie §3.2. Each chat's data record holds one picked review main_data.md.
- Plot (P): the Plot section of the movie's Wikipedia page.
- Comments (C): the first comment of each thread on the movie's official discussion page on Reddit; IMDb review titles if no such page exists.
- Fact table (F): four fields only (box office, similar movies, awards and taglines), taken from IMDb and the Wikipedia infobox §3.2.
- Opening statements §3.3. Three questions per movie (favourite scene, favourite character, general opinion) were answered on Mechanical Turk by three workers each, giving nine openers per movie. A random opener starts each chat.
- Chats §3.4. One worker plays both speakers and gets three or four of
the movie's resources. The worker adds at least eight utterances.
- Speaker 1 writes freely.
- Speaker 2 must copy or modify sentences from the resources. Speaker 2 may add words before or after the copied span.
- Turns alternate, so the even-numbered utterances are the bot replies §5.2.
- Verification §3.5. An in-house evaluator checked each chat. A separate study had 15 evaluators rate 500 chats, each by three evaluators, on five criteria from 1 to 5 tab. 1.
Training instances §5.2. Each Speaker 2 turn becomes one instance of resource, context and response. The context is the query (the utterance just before the reply) plus the two utterances before it. BiDAF is trained on the copied span instead of the full reply.
# Build instances from one chat [§5.2] [§3.4]
def build_instances(chat, labels, spans, documents):
instances = []
# Replies are the Speaker 2 turns, at odd 0-based indices [§5.2]
for turn in range(1, len(chat), 2):
# Turns that copy from no resource (label 4) are skipped [§5.2]
if labels[turn] == 4:
continue
# Context: the query and up to two utterances before it [§5.2]
context = " ".join(chat[max(0, turn - 3):turn])
# Exact copied span; the example is dropped if it is not found
# [§3.4]
span = spans[turn]
label = labels[turn]
# Oracle: the labelled resource, cropped to 256 words [§5.3]
oracle = crop_around(documents[label], span, 256)
# Mixed: all resources merged in proportion to length [§5.3]
mixed_short = merge_resources(documents, label, span, 256)
mixed_long = merge_resources(documents, label, span, 1200)
instances.append(
(oracle, mixed_short, mixed_long, context, span, chat[turn]))
return instancesResource settings §5.3. Three ways to give a model the background text:
- Oracle: only the resource the reply came from, cropped to 256 words so that the copied span stays inside the window. The crop start is random.
- Mixed-short: all four resources merged into one document of 256 words. Each resource keeps a share of the window proportional to its length, and the window always contains the copied span.
- Mixed-long: the same merge with a 1,200-word window, the maximum length of the merged resources.
Oracle is a test-time shortcut: it uses the gold resource label, so it does not measure resource selection.
Models §4:
- HRED §4.1: the standard hierarchical encoder-decoder, not the variational variant. A lower RNN encodes each utterance, an upper RNN encodes the sequence of utterances, and a decoder writes the reply. HRED ignores the resource §5.2.
- GTTP, copy-or-generate §4.2 app. Model details - GTTP: a pointer-generator adapted to a document and a dialogue context. Each has its own RNN, and the decoder attends over both. A learned gate decides, per step, between generating from the vocabulary and copying from the document. The equations below are the paper's eq. 1-eq. 4; x_t is the previous output word and s_t the decoder state after reading it.
e_i^t = v^\top \tanh(W_r h_i^r + U s_t + b_r), \quad a^t = \mathrm{softmax}(e^t), \quad r_t = \sum_i a_i^t h_i^r eq. 1
f_i^t = v^\top \tanh(W_c h_i^c + V s_t + b_c), \quad m^t = \mathrm{softmax}(f^t), \quad c_t = \sum_i m_i^t h_i^c eq. 2
p_{gen} = \sigma(w_r^\top r_t + w_c^\top c_t + w_s^\top s_t + w_x^\top x_t + b_g) eq. 3
P(w) = p_{gen} P_{vocab}(w) + (1 - p_{gen}) \sum_{i:\, w_i = w} a_i^t eq. 4
- BiDAF, span prediction §4.3: the SQuAD reading-comprehension model, used without change. The context is the question and the resource is the document, and the model predicts the start and end of a span. The paper describes it as a six-layer architecture.
Metrics §5.4. HRED and GTTP use BLEU-4 and ROUGE-1, -2 and -L. BiDAF is scored by comparing its predicted span with the reference span using the same metrics, plus F1 in the SQuAD sense. Human judgements use 100 test responses per model and setup, each rated from 1 to 5 for fluency, appropriateness, humanness and specificity tab. 4.
Splits §5.1. Chats are split 80/10/10 by movie, so a movie appears in exactly one split. There are about 9.14 chats per movie.
Multi-reference test set §5.5. For each Speaker 2 turn in the test set, workers were asked for three reference replies from the same resources. The authors note that some turns had only one appropriate reply. 78.04% of test instances have more than one valid reply. Multi-reference scores take the maximum over the references tab. 3.
Dataset statistics tab. 2 §3.6:
| Item | Value |
|---|---|
| Chats | 9,071 |
| Movies | 921 |
| Utterances | 90,810 |
| Average utterances per chat | 10.01 |
| Average words per utterance | 15.29 |
| Average words per chat | 153.07 |
| Average words in Plot / Review / Comments / Fact table | 186.10 / 384.44 / 123.81 / 33.47 |
| Unique Plots / Reviews (paragraphs), unique Comments (sentences) | 5,157 / 1,817 / 12,740 |
| Share of Speaker 2 replies from Reviews / Comments / Plots / Fact table | 41.2% / 34.6% / 16.1% / 8.1% |
Dataset quality, five criteria, mean of 1 to 5 scores tab. 1:
| Criterion | Rating (mean ± SD) | Krippendorff's α | Cohen's κ |
|---|---|---|---|
| Intelligible | 4.47 ± 0.52 | 0.70 | 0.69 |
| Coherent | 4.33 ± 0.93 | 0.57 | 0.71 |
| Grammar | 4.41 ± 0.56 | 0.60 | 0.69 |
| Two-person chat | 4.47 ± 0.46 | 0.64 | 0.70 |
| On topic | 4.57 ± 0.43 | 0.72 | 0.70 |
Hyperparameters
Values as stated in the paper's appendix app. Hyper-parameters, unless a different source is given. The baseline values are the paper's reports of settings from the original baseline papers; the repository does not contain those training codes.
| Name | Value | Where | Notes |
|---|---|---|---|
| HRED optimiser | Adam, initial learning rate 0.001 | app. Hyper-parameters | from the original HRED paper |
| HRED batch size / dropout | 16 / 0.25 | app. Hyper-parameters | |
| HRED embeddings | pre-trained GloVe, 300-d | app. Hyper-parameters | |
| HRED RNN | GRU, hidden size 300, all encoders and decoder | app. Hyper-parameters | |
| HRED vocabulary | 20,000 most frequent words | app. Hyper-parameters | repository HRED export has no cap; see Paper vs code |
| GTTP optimiser | Adagrad, initial learning rate 0.15, initial accumulator 0.1 | app. Hyper-parameters | |
| GTTP batch size | 16 | app. Hyper-parameters | |
| GTTP RNN | LSTM, hidden size 256 | app. Hyper-parameters | |
| GTTP gradient clipping | maximum norm 2 | app. Hyper-parameters | |
| GTTP stopping | early stopping on validation loss | app. Hyper-parameters | patience not stated |
| GTTP / BiDAF context limit | 65 tokens | app. Hyper-parameters | repository applies it to the BiDAF export only; see Paper vs code |
| BiDAF optimiser | AdaDelta, initial learning rate 0.5 | app. Hyper-parameters | |
| BiDAF batch size | 32 | app. Hyper-parameters | |
| BiDAF RNN | LSTM, hidden size 256, six-layer architecture | app. Hyper-parameters §4.3 | |
| BiDAF dropout | 0.2 on all LSTM layers and on the pre-softmax linear layer | app. Hyper-parameters | |
| BiDAF embeddings | pre-trained GloVe, 100-d | app. Hyper-parameters | |
| Oracle / mixed-short window | 256 words | §5.3 | repository counts tokens from an absent helper module |
| Mixed-long window | 1,200 words | §5.3 | |
| Train / validation / test split | about 80% / 10% / 10% of chats, by movie | §5.1 | exact counts per split not stated |
| Training targets | Speaker 2 turns (even-numbered) | §5.2 | 5 instances for a 10-turn chat |
| Test references | three reference replies requested per turn | §5.5 | 78.04% of test instances have multiple valid replies |
| Not stated | epochs, learning-rate schedule, beam width, maximum decoding length, GloVe fine-tuning, GTTP and HRED dropout, hardware for GTTP and HRED |
Results
Author-reported, on the Holl-E test set. Each cell gives the single-reference score followed by the multi-reference score tab. 3. The F1 column applies only to span prediction. "not reported" means the paper gives no value for that model.
| Model | F1 (single / multi) | BLEU-4 (single / multi) | ROUGE-1 (single / multi) | ROUGE-2 (single / multi) | ROUGE-L (single / multi) |
|---|---|---|---|---|---|
| HRED | not reported | 5.23 / 5.38 | 24.55 / 25.38 | 7.61 / 8.35 | 18.87 / 19.67 |
| GTTP (oracle) | not reported | 13.92 / 16.46 | 30.32 / 31.60 | 17.78 / 21.21 | 25.67 / 27.83 |
| GTTP (mixed-short) | not reported | 11.05 / 15.68 | 29.66 / 31.71 | 17.70 / 19.72 | 25.13 / 27.35 |
| GTTP (mixed-long) | not reported | 7.51 / 8.73 | 23.20 / 21.55 | 9.91 / 10.42 | 17.35 / 18.12 |
| BiDAF (oracle) | 39.69 / 47.18 | 28.85 / 34.98 | 39.68 / 46.49 | 33.72 / 40.58 | 35.91 / 42.64 |
| BiDAF (mixed-short) | 45.73 / 51.35 | 32.95 / 39.39 | 45.69 / 50.73 | 40.18 / 45.01 | 43.46 / 46.95 |
Human judgements of 100 responses per model tab. 4, scale 1 to 5:
| Model | Humanness | Appropriateness | Fluency | Specificity |
|---|---|---|---|---|
| HRED | 3.08 | 2.49 | 2.64 | 2.06 |
| GTTP (oracle) | 4.10 | 3.73 | 4.03 | 3.33 |
| GTTP (mixed-long) | 2.93 | 2.97 | 3.42 | 2.60 |
| BiDAF (oracle) | 3.78 | 3.71 | 4.05 | 3.76 |
| BiDAF (mixed-short) | 3.41 | 3.38 | 3.47 | 3.30 |
The authors' reading of these numbers §6:
- Span prediction beats the generation models in the oracle setting, but the authors say generation is far from acceptable and that picked spans are not natural replies. GTTP (oracle) is rated more human-like than both BiDAF settings.
- HRED and GTTP (mixed-long) score about the same, while GTTP (oracle) is well ahead. HRED's replies are very generic (specificity 2.06).
- BiDAF scores better on mixed-short than on oracle, which the authors read as a hypothesis: the model learns to ignore the noisy sentences. They note it was not verified.
- The copy-or-generate model copied 82.18% of output tokens in the oracle setting and 71.95% in mixed-long. Its copied spans often do not fit the context.
- Multi-reference scoring raises every score, which the authors use to argue for metrics that allow several valid replies §6.
Paper vs code
The repository is nikitacs16/Holl-E at 06e8902, linked from the paper's
footnote §3. Its owner name matches the first author, so it is taken as the
authors' repository. It holds the data documentation,
the data-conversion scripts and the metric scripts README.md.
It contains no model or training code. The HRED, BiDAF and GTTP baselines link
to separate projects README.md. The dataset itself is a Google
Drive download README.md, which was not fetched, so the counts in
the paper were not checked against released files.
Differences found by reading the scripts:
- First reply's context. The paper defines the context as the previous two utterances plus the query §5.2. For the first reply, the script uses the query alone convert_main_data_to_raw_data.py.
- GTTP and the 65-token limit. The paper says both GTTP and BiDAF use a 65-token context app. Hyper-parameters. The script truncates only the BiDAF (SQuAD-format) export convert_as_expt.py. The GTTP export keeps the full context convert_as_expt.py.
- Response cap. Replies longer than 80 tokens are cut to 80 tokens while keeping the span. The paper does not say so convert_main_data_to_raw_data.py convert_main_data_to_raw_data.py.
- HRED history cap. The HRED export keeps only the last 90 tokens of history plus query. The paper does not say so convert_as_expt.py.
- Vocabulary. The paper gives 20,000 words for HRED. The HRED export writes no vocabulary convert_as_expt.py. The GTTP export's vocabulary file keeps the 25,000 most frequent words convert_as_expt.py.
- Window units. The paper counts 256 and 1,200 words §5.3. The script counts tokens from a helper module that is not in the repository convert_main_data_to_raw_data.py convert_main_data_to_raw_data.py.
- Multi-reference scoring. The paper takes the maximum over references §5.5. The ROUGE script picks the single reference with the highest ROUGE-L and reports that reference's ROUGE-1 and ROUGE-2 evaluate_multi_rouge.py. So ROUGE-1 and ROUGE-2 are not per-metric maxima.
- BLEU aggregation.
evaluate.pycomputes corpus-level BLEU evaluate.py.evaluate_multi_bleu.pycomputes a per-example maximum over references evaluate_multi_bleu.py. The paper does not say which one it used. - Broken average.
evaluate_multi_bleu.pyinitialisescountto 0 and never increments it, then divides the total by it. As written, the final print divides by zero evaluate_multi_bleu.py evaluate_multi_bleu.py. - Reference normalisation is one-sided. The scripts lower-case and word-tokenise the references only, not the predictions evaluate_multi_rouge.py evaluate_multi_bleu.py.
- Line joining.
evaluate.pyjoins the lines of each output file with no separator evaluate.py, so the last word of one line runs into the first word of the next. - Missing inputs. The conversion script imports
token_utils, and the metric scripts importrougeandbleu. None of these is in the repository, and the multi-reference filemulti_reference_test.jsonis not there either convert_main_data_to_raw_data.py evaluate.py evaluate_multi_bleu.py. The scripts cannot be rerun from this repository alone. - Python 2. The metric scripts use
reload(sys)andsetdefaultencodingevaluate_multi_bleu.py. - Span filter. Examples whose span cannot be located are dropped and only counted convert_main_data_to_raw_data.py. The paper does not report how many were dropped.
Pitfalls
- Targets are not always the copied span. Speaker 2 may add words before or after the copied span §3.4. The released spans are only the copied part, so a span model cannot reproduce those added words. Full replies are therefore not recoverable from spans.
- Gold resource at test time. The oracle setting crops around the true resource label, which a deployed system would not know. The paper suggests a two-stage model that first selects the resource §6, but does not evaluate one.
- Random crops. The oracle crop start and the discarded words for proportional merging are drawn at random with seed 123 convert_main_data_to_raw_data.py convert_main_data_to_raw_data.py. Regenerating the data with different randomness changes the windows, so use the released files for comparable results.
- Proportional merge rules. Shares are rounded down to integers. If the span is longer than the label's share, that share grows and the difference is taken from the first other resource with room convert_main_data_to_raw_data.py. The paper says only "proportional to length" §5.3.
- Review source. The paper says two reviews per movie were fetched §3.2. The data field holds one picked review per chat main_data.md. Which of the two a chat uses is not stated.
- BiDAF scores measure spans. Because the reply can contain added words §3.4, BiDAF's target is not always the full reply. Its BLEU and ROUGE are span-to-span scores, so they do not directly measure reply quality.
- Copy vocabulary. The paper's copy equation sums attention over source positions eq. 4. It does not say whether source-only words can be emitted when they are missing from the output vocabulary.
- Human evaluation coverage. The text says 100 responses for every model and every setup §5.4. Table 4 has no GTTP (mixed-short) row, and the number of raters per response is not stated.
- Setup list mismatch. The Results text lists four configurations §5.3. Table 3 reports six, including GTTP (mixed-short).
- F1. The paper reports F1 for BiDAF §5.4. The repository defines an F1 function evaluate_multi_rouge.py, but its only use is commented out evaluate_multi_rouge.py. The normaliser lower-cases and word-tokenises but does not drop punctuation or articles, though helpers for both exist and are unused evaluate_multi_rouge.py.
- Movie list. The paper names 921 movies and points to an appendix list of source URLs. In the source, that list sits inside a disabled block, so it is not in the compiled paper §3.1.
- Baselines unverifiable here. The GTTP, BiDAF and HRED hyperparameters come from the original papers and external repositories. This repository does not contain those codes, so the numbers cannot be checked against code.
Lineage
- Builds on: HRED §4.1, the pointer-generator Get To The Point §4.2 (modified here), BiDAF §4.3, and the self-dialogue collection idea from earlier work in which one worker plays both sides §3.4 §2.
- Compared against: the three baseline families above. The authors chose the modified copy-or-generate model over other knowledge-copy variants because it already has the extra encoder for the background document §4.2.
- Superseded by: not stated in this wiki.
Source and license
Adapted from “Towards Exploiting Background Knowledge for Building Conversation Systems” by Nikita Moghe, Siddhartha Arora, Suman Banerjee, Mitesh M. Khapra (arXiv:1809.08205), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/nikitacs16/Holl-E at commit 06e8902b70b3 (none (no LICENSE file in repository)). Any quoted code is shown under that license.