Extractive question answering

Overview

An extractive question answerer returns a part of the passage. In the span form it predicts a start token and an end token, and the answer is the text between them 1808.07042 §5.2. The answer is always wording that already appears in the passage, so the model cannot write a count, a paraphrase or a yes/no answer unless the design adds a way to do so. Four cards cover this page. The attention-over-attention card gives a cloze reader that puts its probability on single passage words 1607.04423 §3. The CoQA card gives the conversational setting and span readers built for it 1808.07042 §4.1, §5.2. The LinkBERT card gives an encoder pretraining method, evaluated on extractive QA sets from MRQA 2203.15827 §5.2. The P-tuning v2 card keeps the encoder frozen and trains only prompt vectors and a span head, and reports SQuAD dev results 2110.07602 tab. 3.

Variants

Attention-over-attention pointer reader (AoA Reader)

This variant handles cloze questions: a sentence with one word blanked out, to be filled from a passage. A bi-directional GRU encodes the passage and the question, and a dot product scores every pair of passage and question positions 1607.04423 §3 eq. 5. A softmax over passage positions gives, for each question word, a map over the passage. A second softmax over question positions gives, for each passage word, a weight over the question words; averaging those over the passage gives one learned weight per question word eq. 6-9. The passage distribution is the weighted sum of the per-word maps eq. 10, and mass is summed by word identity, so only words that occur in the passage get probability eq. 11. The authors say the design adds a few steps and no weights to the AS Reader 1607.04423 §3. Single-model test accuracy is 72.0 on CBTest named entities and 74.4 on CNN 1607.04423 tab. 2. Re-ranking the 5-best answers with three n-gram-style features raises CBTest named entities to 74.0 1607.04423 tab. 4. Card: Attention-over-Attention Neural Networks for Reading Comprehension.

Span reader (DrQA)

A reading-comprehension model predicts the start and end token of the answer in the passage. The previous questions and answers are prepended to the current question, so the reader can use the conversation so far 1808.07042 §5.2. Test F1 is 52.6 overall, against 65.4 for the augmented variant below 1808.07042 tab. 7. Card: CoQA: A Conversational Question Answering Challenge.

Span reader with yes/no tokens (Augmented DrQA)

The word "yes" and the word "no" are appended to the passage as extra candidate positions. When the gold answer is one of them, the model points to that token instead of a passage span 1808.07042 §5.2. Test F1 is 65.4 overall and dev F1 is 67.2 1808.07042 tab. 7. On answers that are not spans, it scores 76.2 on Yes and 64.0 on No 1808.07042 tab. 10. The released code at the cited commit does not contain this recipe, so the result is paper-only until reproduced (see Contradictions).

Span reader followed by generation (DrQA+PGNet)

The span reader first points to the evidence, and a seq2seq model with copy then writes the final answer from the question and that span. When the answer is a substring of the rationale, the span reader is trained to predict the answer directly; otherwise it predicts the rationale 1808.07042 §5.3. Test F1 is 65.1 overall 1808.07042 tab. 7. This variant reports counting (24.8) and multiple-choice (46.4) F1 in 1808.07042 tab. 10, answer types that a span alone cannot produce when the text does not contain them.

For contrast, the copy-based generator alone (PGNet) is not extractive: it writes its answer and can copy words, and reaches 44.1 test F1 1808.07042 tab. 7.

Pretrained encoder fine-tuned on MRQA (LinkBERT)

An encoder is pretrained on inputs that place linked documents in one context, then fine-tuned on the MRQA question-answering sets. Average F1 over six sets is 81.0 for LinkBERT-large against 78.5 for BERT-large, and 77.8 against 75.2 at base size 2203.15827 tab. 1. The base model scores 78.2 on HotpotQA against 76.0 for BERT-base 2203.15827 tab. 1. The card does not describe the answer layer used in fine-tuning 2203.15827 §5.2, so the card supports the extractive-QA link only through its concept list. The MRQA dev sets were split in half to make the test split, so the numbers are not leaderboard results 2203.15827 §5.2. Card: LinkBERT: Pretraining Language Models with Document Links.

Frozen backbone with per-layer prompts (P-tuning v2)

The pretrained encoder stays frozen. Trainable prefix vectors are added at every transformer layer, and a linear head predicts a start and an end label for each token 2110.07602 app. A. Across the paper's experiments, the trainable share is 0.1% to 3% of the model 2110.07602 §1. On SQuAD 1.1 dev with RoBERTa-large, it scores 94.4 F1 against 94.6 for full fine-tuning 2110.07602 tab. 3. On SQuAD 2.0 dev with the same backbone, the gap is wider: 85.5 F1 against 89.4 2110.07602 tab. 3. With DeBERTa-xlarge, both reach 91.1 F1 on SQuAD 2.0 2110.07602 tab. 3. The input-layer prompt baseline reaches 12.0 F1 on SQuAD 1.1 with RoBERTa-large 2110.07602 tab. 3, in line with the paper's statement that earlier prompt tuning lagged far behind on extractive QA 2110.07602 §1. The released code has no QA run for that baseline, so it cannot be rerun from the release (see the card's Paper vs code section). Card: P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks.

When to use which

The CoQA card records no latency, memory or hardware figures for any span reader 1808.07042 tab. 7. The AoA card gives no latency or memory figures either 1607.04423 §5.1. The LinkBERT card records pretraining compute in GPU-days and no inference cost 2203.15827 §5.1. The P-tuning v2 card reports accuracy only, not latency or memory 2110.07602 tab. 3. The choice below therefore rests on answer types, training data, storage per task and the cost of pretraining.

Implementation sketch

Scoring for the cloze pointer reader. The paper gives the equations; the code that implements them is not reproduced here. The paper does not define the averaging constant in its word-weight equation, so the mean over passage positions is used here 1607.04423 §3.

# Score each passage word for a cloze question  [arxiv:1607.04423 §3]
# Input:  passage states (D, h); question states (Q, h)
# Output: probability over passage positions (D,)

def cloze_pointer_scores(passage, question):
    # Pair-wise dot products between passage and question positions  [eq. 5]
    match = passage @ question.T                  # (D, Q)
    # Per question word: softmax over passage positions  [eq. 6-7]
    per_word = softmax(match, axis=0)             # (D, Q)
    # Per passage position: softmax over question words  [eq. 8]
    per_pos = softmax(match, axis=1)              # (D, Q)
    # One learned weight per question word, averaged over passage positions  [eq. 9]
    word_weight = per_pos.mean(axis=0)            # (Q,)
    # Weighted sum of the per-word maps  [eq. 10]
    return per_word @ word_weight                 # (D,)

Training target for a span reader. The paper gives these rules; the code that implements them is not reproduced here.

# Choose the training target span for an extractive reader  [arxiv:1808.07042 §5.2]
# Input:  passage tokens; gold answer string; rationale span
# Output: (start, end) token indices into the passage, or a yes/no index

def gold_span(passage, answer, rationale):
    # Yes/no answers point to the appended token (augmented variant only)  [§5.2]
    if answer in ("yes", "no"):
        return yes_no_index(answer)
    # Prefer the position of the answer inside the rationale  [§5.2]
    if answer in rationale.text:
        return locate_in_rationale(answer, rationale)
    # Otherwise take the passage span with the highest F1 against the answer  [§5.2]
    span = best_f1_span(passage, answer)
    # If no token overlaps, the paper falls back to an unknown token  [§5.2]
    return span if span is not None else unknown_token_index(passage)

Inference for a span reader. The paper does not state a maximum answer length or the decoding rule for start and end, so set them and report what you used.

# Return an answer span for the current turn  [arxiv:1808.07042 §5.2]
# Input:  passage tokens; history of (question, answer) pairs; question
# Output: answer text taken from the passage (or "yes"/"no" if augmented)

def answer_turn(passage, history, question, reader):
    # Prepend earlier turns to the current question  [§5.2]
    query = flatten(history) + question
    # The reader scores start and end positions in the passage  [§5.2]
    start, end = reader.predict(query, passage)
    # Return the text between the chosen positions  [§5.2]
    return passage[start:end]

Contradictions

No cross-paper disagreement. The four cards report on different datasets (CBTest, CNN, CoQA, MRQA and SQuAD dev), so their numbers are not comparable. Within the attention-over-attention paper, the text gives gains of 4.1 and 3.7 points over CAS Reader on CNN validation and test, while the table gives 4.9 and 4.4 1607.04423 §5.2 tab. 2. The card uses the table. The paper reports re-ranking results only for CBTest, not CNN 1607.04423 tab. 2. The P-tuning v2 SQuAD numbers are dev scores, and the paper does not say how the reported epoch or seed was chosen 2110.07602 tab. 3. Within the P-tuning v2 release, the README's numbers differ from the paper's tables card: Paper vs code. Within the CoQA paper, the released baseline code does not match the paper in several places: the optimizer, the dropout value, the reader embeddings, and the augmented yes/no recipe, which is not in the code at the cited commit. The repository's README also reports dev F1 numbers that differ from the paper's table 9. Both sides are in the card's Paper vs code section. Within the LinkBERT paper, the MRQA test split is not the official one, and the tiny-size ablations are not repeated at other sizes 2203.15827 §5.2, tab. 5. Treat the paper's numbers as author-reported and reproduce before relying on them.

Papers

• Attention-over-Attention Neural Networks for Reading Comprehension: a cloze reader that weights query-to-passage attention by a learned passage-to-query attention, with optional n-gram re-ranking on CBTest.
• CoQA: A Conversational Question Answering Challenge: a conversational dataset, with span-reader baselines (DrQA, augmented DrQA) and a span-then-generate pipeline.
• LinkBERT: Pretraining Language Models with Document Links: an encoder pretraining method that links documents in one input, with MRQA results against BERT at three sizes.
• P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks: frozen-backbone prompt tuning with prefix vectors at every layer, with SQuAD dev results against full fine-tuning.