Conversational question answering

Overview

A conversational question answerer reads a passage and a chat so far, then answers the newest question. Questions often depend on earlier turns, so a system that sees only the current question cannot answer them. Standard reading-comprehension sets use standalone questions with span answers, which do not test this 1808.07042 §1, tab. 1. The CoQA paper is built around that gap: it collects a dataset of such chats and tests simple systems on it 1808.07042 §3, §5. The shared core of the variants is that each answer is conditioned on the passage and the previous question-answer pairs. They differ in whether the answer is a span copied from the passage, a generated string, or a span followed by generation 1808.07042 §5. Word overlap is a weak signal here: only 29.8% of CoQA questions share a content word with their rationale 1808.07042 §4.2.

Variants

Sequence-to-sequence with copy (PGNet)

An LSTM encoder reads the passage, the history and the question. An LSTM decoder with attention writes the answer and can copy words from the passage 1808.07042 §5.1. It is the only family here that writes free-form text without a span step. Test F1 is 44.1 overall, against 26.3 for a plain seq2seq without copy 1808.07042 tab. 7. Card: CoQA: A Conversational Question Answering Challenge.

Span reader (DrQA)

A reading-comprehension model predicts a start and an end token in the passage. The previous questions and answers are prepended to the current question 1808.07042 §5.2. It can only return text that appears in the passage. Test F1 is 52.6 overall 1808.07042 tab. 7. Card: CoQA: A Conversational Question Answering Challenge.

Span reader with yes/no tokens (Augmented DrQA)

Two tokens, "yes" and "no", are appended to the passage. When the gold answer is one of them, the model points to that token 1808.07042 §5.2. This is the best reported system: test F1 65.4 overall, and dev F1 67.2 [arxiv:1808.07042 tab. 7]. The released code does not contain this recipe at the paper's commit, so treat it as a paper-only result until you reproduce it (see the card's Paper vs code section).

Pipeline (DrQA+PGNet)

The span reader first points to the evidence, and the seq2seq model then writes the answer from the question and that span. When the answer is a substring of the rationale, the span reader is trained to predict the answer directly; otherwise it predicts the rationale 1808.07042 §5.3. Test F1 is 65.1 overall, close to Augmented DrQA 1808.07042 tab. 7. It also handles answers that are not spans: 24.8 F1 on counting and 46.4 on multiple choice 1808.07042 tab. 10. Augmented DrQA scores higher on Yes and No answers (76.2 and 64.0) 1808.07042 tab. 10.

Dataset only

CoQA itself has 127k question-answer pairs in about 8.4k passages from seven domains, and 15.2 turns per passage on average 1808.07042 tab. 2. Dev and test questions have four gold answers; training questions have one 1808.07042 §3.3. Use it as a benchmark or as training data for any of the families above.

When to use which

Implementation sketch

The common core is one function: take the passage, the history and the new question, and return an answer. The pipeline shows how the pieces fit. The history size is a parameter, tuned on dev; one previous turn gave the best DrQA and pipeline scores, and two gave the best Augmented DrQA score 1808.07042 tab. 9.

# Answer one turn of a conversation  [arxiv:1808.07042 §5.3]
# Input:  passage; history of (question, answer) pairs; new question
# Output: answer string

def answer_turn(passage, history, question, reader, generator, k):
    # Keep only the last k turns; k is tuned on dev  [arxiv:1808.07042 tab. 9]
    recent = history[-k:] if k > 0 else []
    # The span reader sees the previous turns and the question  [§5.2]
    evidence = reader.predict_span(passage, recent, question)
    # The generator writes the answer from the question and the evidence  [§5.3]
    return generator.generate(question, evidence)

Notes on the variants:

• For the seq2seq variant, the encoder input is the passage, then each past turn tagged as question and answer, then the current question 1808.07042 §5.1.
• For the span reader, training targets are the answer's position inside the rationale when it appears there, and otherwise the best-F1 span 1808.07042 §5.2.
• Augmented DrQA adds the yes/no tokens as extra pointer targets next to the passage spans 1808.07042 §5.2.

Pitfalls

• Gold history at test time. The paper evaluates with gold answers of earlier turns, not the model's own answers 1808.07042 §6.1. A system that feeds its own answers back is measured in a setting the paper does not report.
• Rationales are training-only. Do not give rationales to a test-time model 1808.07042 §2.
• Word F1 undercounts paraphrase. Different wordings of the same answer can both be correct, and word overlap does not credit that 1808.07042 §6.3.
• Dev and test scores depend on four gold answers. Training data has one answer per question 1808.07042 §3.3.
• Unanswerable answers are rare. About 1.3% of answers are unanswerable 1808.07042 tab. 4, so this case barely moves the overall F1.
• Passage licenses are mixed. The paper's CC BY 4.0 licence covers the paper only. The dataset's passages come from sources with different licences, which the card lists from the project page (not from the paper). Check them before redistributing the data.

Contradictions

None between papers: only one paper with a card covers this concept. The paper and its released code disagree on several settings (optimizer, dropout, embeddings, and the Augmented DrQA recipe). The card's Paper vs code section lists them.

Papers

• CoQA: A Conversational Question Answering Challenge: the CoQA dataset, its evaluation metric, and three baseline families with their author-reported scores.