CoQA: A Conversational Question Answering Challenge

In plain words

Reading a passage and answering one question is not how people usually get information. They ask a question, hear the answer, and ask a follow-up that depends on what was just said, such as "Who?" after a question about a birthday. Most question-answering datasets contain only standalone questions, so they do not train or test that behaviour.

CoQA is a dataset built to fix this. Two people chat about a short text: one asks questions, the other answers. For each answer, the answerer first marks the part of the text that supports it, and then rewrites it into a natural sentence, so answers need not be copied spans. The chats cover seven kinds of text, and two of them are held back to test how well systems handle text they were not trained on.

The authors also tried simple systems on it. The best one reached about 65 F1 points, while people reached about 89, so there is a lot of room left.

TL;DR

CoQA is a conversational reading-comprehension dataset: 127k question-answer turns in 8k conversations, with free-form answers and a text-span rationale for each answer §1. Systems answer the next question given the passage and the earlier turns, and rationales are not given at test time §2. The best baseline reached 65.4 test F1 against 88.8 for humans, author-reported tab. 7. Use it as a benchmark or as a source of conversational QA data; the baseline code is MIT-licensed, but the dataset's passages carry several licenses (see Pitfalls).

Problem

Standard reading-comprehension sets use standalone questions. Their answers are spans that can be found in the passage §1, tab. 1. Questions in a conversation depend on earlier turns, for example "Who?" after "Who had a birthday?" §1, fig. 1. Such questions cannot be answered from the current turn alone, and span-only answers cannot express answers such as "Three" for a count that is not written in the text §1. Span-extraction models rely heavily on lexical overlap between the question and the passage §1, and only 29.8% of CoQA questions contain a content word from the rationale §4.2.

Method

The contribution has two parts: a data collection protocol and a set of baseline models with an evaluation metric.

Data collection §3

1. Two crowd workers are paired on a passage through Amazon Mechanical Turk and ParlAI §3. One is the questioner and the other is the answerer; both see the passage §3.1, §7. The answerer marks a rationale span first §3.1.
2. Free-form answers are made by editing the copied rationale. 78% of answers have at least one edit §3.1.
3. Questioners get a warning when they reuse words from the passage, to encourage paraphrase §3.1.
4. Passages come from seven domains: children's stories (MCTest), literature (Project Gutenberg), middle and high school exams (RACE), news (CNN), Wikipedia, Reddit (Writing Prompts) and science (AI2 Science Questions) §3.2. Passages are truncated to about 200 words and chosen to contain several entities and pronouns §3.2.
5. Splits: per in-domain domain, 100 passages each for development and test, the rest for training. Reddit and Science are test-only out-of-domain sets §3.2, tab. 2.
6. For development and test questions, three more answers are collected from new answerers. Each new answerer sees the conversation and is asked to predict the original answer, which keeps their turns consistent with the original conversation §3.3. In a pilot, this raised human F1 by 5.4% §3.3.

Dataset size: 8,399 passages, 127k question-answer pairs, 15.2 turns per passage on average tab. 2. Answers are 66.8% span-matches after ignoring punctuation and case, and 33.2% are not spans §4.1. Yes and No answers are 11.1% and 8.7% of questions tab. 4.

Baseline models §5

1. Conversational model (seq2seq with attention, with copy). The passage, the previous question-answer pairs and the current question are concatenated into one input, and a bidirectional LSTM encodes it. An LSTM decoder with attention generates the answer and can copy words from the passage §5.1. The copy variant is called PGNet §5.1. Pseudocode below.
2. Reading-comprehension model (DrQA). It predicts a start and end token in the passage §5.2. Gold spans for training are chosen as described below. The previous questions and answers are prepended to the question §5.2.
3. Augmented DrQA. Two extra tokens, "yes" and "no", are appended to the passage. When the gold answer is one of them, the model points to that token instead of a passage span §5.2.
4. Combined model (DrQA+PGNet). DrQA first points to the evidence, and PGNet turns that evidence into the answer. For DrQA, the model predicts the answer directly when it is a substring of the rationale, and predicts the rationale otherwise. PGNet receives the question and the DrQA span §5.3.

Evaluation §6.1

Macro-averaged word-overlap F1, as in SQuAD, with articles and punctuation removed. Each question has four gold answers, and each answer is compared with the other three. The F1 values are averaged over the four sets, for humans and models alike. Evaluation uses the gold answers of earlier turns as history, not the model's own earlier answers §6.1. Rationales are not used in evaluation §6.1.

Pseudocode 1: seq2seq encoder input §5.1

# Build the encoder input for turn i  [§5.1]
# Input:  passage; history as (question, answer) pairs; question q_i
# Output: token list for the bidirectional LSTM encoder

def build_encoder_input(passage, history, question, n_history):
    # Keep only the last n turns of history; n is the history size  [§5.1]
    recent = history[-n_history:] if n_history > 0 else []
    # The passage comes first  [§5.1]
    tokens = [passage]
    # Each past turn is tagged as question and answer  [§5.1]
    for past_question, past_answer in recent:
        tokens += ["<q>", past_question, "<a>", past_answer]
    # The current question ends the input  [§5.1]
    tokens += ["<q>", question]
    return tokens

Pseudocode 2: gold span for DrQA training §5.2

# Choose the training target span for DrQA  [§5.2]
# Input:  passage tokens; gold answer; rationale span
# Output: (start, end) token indices into the passage

def gold_span(passage, answer, rationale):
    # If the answer text occurs in the rationale, use that position  [§5.2]
    if answer in rationale.text:
        return locate_in_rationale(answer, rationale)
    # Otherwise take the span with the highest F1 against the answer  [§5.2]
    best = best_f1_span(passage, answer)
    # No token overlaps: the span falls back to the unknown token  [§5.2]
    return best

Pseudocode 3: per-question score §6.1

# Score one predicted answer against its four gold answers  [§6.1]
# Input:  predicted string; list of four gold strings
# Output: word-overlap F1, averaged over the leave-one-out sets

def question_f1(prediction, golds):
    # Build one set per gold answer, leaving that answer out  [§6.1]
    scores = []
    for held_out in range(len(golds)):
        references = golds[:held_out] + golds[held_out + 1:]
        # Take the best match among the three remaining answers  [§6.1]
        best = max(word_f1(prediction, ref) for ref in references)
        scores.append(best)
    # Average the sets to get the question score  [§6.1]
    return sum(scores) / len(scores)

Hyperparameters

Values in the paper and in the code at code_commit are both given. "Not stated" means neither gives a value.

Results

Author-reported. Setup: CoQA with gold history; the metric is F1 over four gold answers §6.1. Models are DrQA variants and seq2seq variants described above; hardware: not stated. In-domain test covers five domains, out-of-domain test covers Reddit and Science §3.2, tab. 7.

Source: tab. 7. Abbreviations: Child. = children's stories, Liter. = literature, Mid-High. = middle and high school, Wiki. = Wikipedia.

Other reported results, all on the development set tab. 8, tab. 9 unless noted:

• Fine-grained: humans find lexical-match questions easiest and pragmatic ones hardest §6.3, tab. 8. Models behave less consistently across these groups.
• History: DrQA is best at one previous turn (54.7); Augmented DrQA is best at two (67.2); DrQA+PGNet is best at one (66.2) tab. 9.
• Human history test: 300 short questions, human F1 is 19.9 with no history, 79.8 with the previous turn, 85.3 with the last two, and 86.4 with full history §6.3.
• Non-span answers: on answers that are not spans, Augmented DrQA scores 76.2 on Yes and 64.0 on No, and DrQA+PGNet scores 24.8 on Counting and 46.4 on Multiple choice tab. 10.

Paper vs code

Code: stanfordnlp/coqa-baselines at 693d05b39dc036a8ad4ef0f00e4e6068aafa9d02, MIT license. The repository is the official baselines repo; it is linked from the CoQA project page and is owned by the Stanford NLP organisation. Read through the clone: rc/main.py, rc/model.py, scripts/gen_drqa_data.py, scripts/gen_pipeline_data.py, scripts/evaluate-v1.0.py, and README.md. The seq2seq code is a git submodule (OpenNMT-py) that the clone does not include, so seq2seq hyperparameters are compared only through the README.

Discrepancies:

• Optimizer: the paper says Adam §6.2; the code's default is adamax main.py.
• Dropout: the paper uses 0.4 for LSTM layers §6.2; the code's default is 0.3 main.py, and the README training command does not override it README.md.
• Embeddings: the paper uses fastText for reading comprehension models §6.2; the README reading comprehension command loads GloVe 840B 300d README.md.
• Unknown token: the paper falls back to an unknown token when answer words do not occur in the passage §5.2. The code appends an unknown token to each passage gen_drqa_data.py and returns the last token when no word matches gen_drqa_data.py. These agree.
• Augmented DrQA: the paper describes adding "yes" and "no" tokens to the passage §5.2. No such tokens were found in rc/ or scripts/ at this commit (a text search for yes/no handling found only boolean argument parsing, main.py). Treat the Augmented DrQA recipe as not released here.
• Span selection: the paper uses the highest-F1 span §5.2. The code scores candidate start and end tokens whose text appears in the gold answer, keeps the first best span on ties gen_drqa_data.py, and locates the answer inside the rationale when it appears there gen_drqa_data.py.
• Combined model: the paper says PGNet is trained with oracle DrQA spans §5.3, footnote. The pipeline data script writes the gold answer span into the PGNet training input gen_pipeline_data.py.
• Evaluation: the paper's metric §6.1 matches the evaluator's normalisation and leave-one-out averaging evaluate-v1.0.py.
• Reported numbers: the README table gives dev F1 of 55.6 for DrQA and 65.0 for the pipeline at history 2 README.md. The paper gives 54.6 for DrQA at history 2 and 66.0 for DrQA+PGNet at history 2 tab. 9. These are not the same runs.

Pitfalls

• Adam versus adamax, and the learning rate. The paper says Adam §6.2. The code builds adamax and passes only weight decay, not the configured learning rate, so the step size is the framework default (inference from the code; the repo pins only torch>=0.4.0 in its README) model.py. The DrQA learning rate is not stated in the paper, so reproduce with the code and report what you used.
• Dropout and embeddings are not what the README runs. Use 0.4 and fastText if you follow the paper. The README runs 0.3 and GloVe 840B README.md.
• Gold history at evaluation. Test-time history uses gold answers §6.1. A system that uses its own earlier answers is evaluated differently. The paper does not report that setting.
• Rationales are training-only. They are optional evidence in collection and are not given at test time §2. Do not feed them to a test-time model.
• Unanswerable questions are rare. About 1.3% of CoQA answers are unanswerable tab. 4, so that case has little weight in the overall F1.
• Free-form answers and F1. Word overlap cannot credit an answer that means the same thing with different words. The paper notes that "she loved it" and "yes" are both accepted answers to one question §6.3.
• Evaluation depends on the four gold answers. Scores depend on the three extra answers collected for dev and test only §3.3. Training data has one answer per question.
• Data quality. 12% of training conversations were produced by one worker playing both roles, and are included in training only §3 footnote. Children's stories are scarce: MCTest has 660 passages, of which 200 are used for dev and test §6.3 footnote.
• Passage licenses are mixed. The paper does not state the dataset license. The CoQA project page lists per-domain licenses for the passages: CC BY-SA 4.0 for literature and Wikipedia, MSR-LA for children's stories, RACE's own license for exams, and Apache for news (CNN/DailyMail). This is outside the paper and its CC BY 4.0 license covers only the paper. Check the page before redistributing the data.
• Source version. Citations here refer to arXiv v2, the source that wikillm fetch retrieved.

Lineage

• Builds on: DrQA (Chen et al., 2017) for the span reader §5.2; attention seq2seq (Bahdanau et al., 2015) §5.1; the copy mechanism and pointer-generator (Gu et al., 2016; See et al., 2017) §5.1; OpenNMT for the seq2seq experiments §6.2; GloVe and fastText embeddings §6.2.
• Compared against: QuAC (Choi et al., 2018), a parallel conversational QA dataset whose answers are restricted to extractive spans and whose passages come only from Wikipedia articles about people §7.
• Superseded by: not stated in this wiki. The paper reports 82.8 test F1 for later BERT-based systems as of 8 Jan 2019 §7. Recent work named in §7: FlowQA (Huang et al., 2019) and SDNet (Zhu et al., 2018).