--- kind: concept slug: extractive-question-answering title: Extractive question answering summary: >- Answering a question by choosing a part of the passage instead of generating text. Span readers reach 52.6 test F1 on CoQA, or 65.4 with yes/no tokens. An attention-over-attention cloze reader reports 74.4 CNN test accuracy, single model. LinkBERT-large reports 81.0 average MRQA F1 against 78.5 for BERT-large. Author-reported, on different datasets. tags: [question-answering, reading-comprehension, span-extraction] papers: ["1607.04423", "1808.07042", "2203.15827", "2110.07602"] related: [conversational-question-answering, knowledge-base-question-answering, benchmark-validity, language-model-pretraining, masked-language-modeling, multi-hop-reasoning, soft-prompt-tuning, parameter-efficient-fine-tuning, attention-mechanism, recurrent-neural-network] status: draft updated_at: 2026-10-31 --- ## Overview An extractive question answerer returns a part of the passage. In the span form it predicts a start token and an end token, and the answer is the text between them [arxiv:1808.07042 §5.2]. The answer is always wording that already appears in the passage, so the model cannot write a count, a paraphrase or a yes/no answer unless the design adds a way to do so. Four cards cover this page. The attention-over-attention card gives a cloze reader that puts its probability on single passage words [arxiv:1607.04423 §3]. The CoQA card gives the conversational setting and span readers built for it [arxiv:1808.07042 §4.1, §5.2]. The LinkBERT card gives an encoder pretraining method, evaluated on extractive QA sets from MRQA [arxiv:2203.15827 §5.2]. The P-tuning v2 card keeps the encoder frozen and trains only prompt vectors and a span head, and reports SQuAD dev results [arxiv:2110.07602 tab. 3]. ## Variants ### Attention-over-attention pointer reader (AoA Reader) This variant handles cloze questions: a sentence with one word blanked out, to be filled from a passage. A bi-directional GRU encodes the passage and the question, and a dot product scores every pair of passage and question positions [arxiv:1607.04423 §3] [eq. 5]. A softmax over passage positions gives, for each question word, a map over the passage. A second softmax over question positions gives, for each passage word, a weight over the question words; averaging those over the passage gives one learned weight per question word [eq. 6-9]. The passage distribution is the weighted sum of the per-word maps [eq. 10], and mass is summed by word identity, so only words that occur in the passage get probability [eq. 11]. The authors say the design adds a few steps and no weights to the AS Reader [arxiv:1607.04423 §3]. Single-model test accuracy is 72.0 on CBTest named entities and 74.4 on CNN [arxiv:1607.04423 tab. 2]. Re-ranking the 5-best answers with three n-gram-style features raises CBTest named entities to 74.0 [arxiv:1607.04423 tab. 4]. Card: [Attention-over-Attention Neural Networks for Reading Comprehension](/papers/1607.04423). ### Span reader (DrQA) A reading-comprehension model predicts the start and end token of the answer in the passage. The previous questions and answers are prepended to the current question, so the reader can use the conversation so far [arxiv:1808.07042 §5.2]. Test F1 is 52.6 overall, against 65.4 for the augmented variant below [arxiv:1808.07042 tab. 7]. Card: [CoQA: A Conversational Question Answering Challenge](/papers/1808.07042). ### Span reader with yes/no tokens (Augmented DrQA) The word "yes" and the word "no" are appended to the passage as extra candidate positions. When the gold answer is one of them, the model points to that token instead of a passage span [arxiv:1808.07042 §5.2]. Test F1 is 65.4 overall and dev F1 is 67.2 [arxiv:1808.07042 tab. 7]. On answers that are not spans, it scores 76.2 on Yes and 64.0 on No [arxiv:1808.07042 tab. 10]. The released code at the cited commit does not contain this recipe, so the result is paper-only until reproduced (see Contradictions). ### Span reader followed by generation (DrQA+PGNet) The span reader first points to the evidence, and a seq2seq model with copy then writes the final answer from the question and that span. When the answer is a substring of the rationale, the span reader is trained to predict the answer directly; otherwise it predicts the rationale [arxiv:1808.07042 §5.3]. Test F1 is 65.1 overall [arxiv:1808.07042 tab. 7]. This variant reports counting (24.8) and multiple-choice (46.4) F1 in [arxiv:1808.07042 tab. 10], answer types that a span alone cannot produce when the text does not contain them. For contrast, the copy-based generator alone (PGNet) is not extractive: it writes its answer and can copy words, and reaches 44.1 test F1 [arxiv:1808.07042 tab. 7]. ### Pretrained encoder fine-tuned on MRQA (LinkBERT) An encoder is pretrained on inputs that place linked documents in one context, then fine-tuned on the MRQA question-answering sets. Average F1 over six sets is 81.0 for LinkBERT-large against 78.5 for BERT-large, and 77.8 against 75.2 at base size [arxiv:2203.15827 tab. 1]. The base model scores 78.2 on HotpotQA against 76.0 for BERT-base [arxiv:2203.15827 tab. 1]. The card does not describe the answer layer used in fine-tuning [arxiv:2203.15827 §5.2], so the card supports the extractive-QA link only through its concept list. The MRQA dev sets were split in half to make the test split, so the numbers are not leaderboard results [arxiv:2203.15827 §5.2]. Card: [LinkBERT: Pretraining Language Models with Document Links](/papers/2203.15827). ### Frozen backbone with per-layer prompts (P-tuning v2) The pretrained encoder stays frozen. Trainable prefix vectors are added at every transformer layer, and a linear head predicts a start and an end label for each token [arxiv:2110.07602 app. A]. Across the paper's experiments, the trainable share is 0.1% to 3% of the model [arxiv:2110.07602 §1]. On SQuAD 1.1 dev with RoBERTa-large, it scores 94.4 F1 against 94.6 for full fine-tuning [arxiv:2110.07602 tab. 3]. On SQuAD 2.0 dev with the same backbone, the gap is wider: 85.5 F1 against 89.4 [arxiv:2110.07602 tab. 3]. With DeBERTa-xlarge, both reach 91.1 F1 on SQuAD 2.0 [arxiv:2110.07602 tab. 3]. The input-layer prompt baseline reaches 12.0 F1 on SQuAD 1.1 with RoBERTa-large [arxiv:2110.07602 tab. 3], in line with the paper's statement that earlier prompt tuning lagged far behind on extractive QA [arxiv:2110.07602 §1]. The released code has no QA run for that baseline, so it cannot be rerun from the release (see the [card's Paper vs code section](/papers/2110.07602)). Card: [P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](/papers/2110.07602). ## When to use which The CoQA card records no latency, memory or hardware figures for any span reader [arxiv:1808.07042 tab. 7]. The AoA card gives no latency or memory figures either [arxiv:1607.04423 §5.1]. The LinkBERT card records pretraining compute in GPU-days and no inference cost [arxiv:2203.15827 §5.1]. The P-tuning v2 card reports accuracy only, not latency or memory [arxiv:2110.07602 tab. 3]. The choice below therefore rests on answer types, training data, storage per task and the cost of pretraining. | Situation | Use | Why | |---|---|---| | Cloze question with one blank, answer is a single word from the passage, and no extra weights are wanted over a base reader | Attention-over-attention pointer reader | The authors report no weights added over AS Reader [arxiv:1607.04423 §3]. Single-model CBTest named-entity test accuracy is 72.0, the highest single-model test score in the paper's table [arxiv:1607.04423 tab. 2]. Re-ranking adds 2.0 points but needs three language-model features, one trained per test sample [arxiv:1607.04423 §4] [tab. 4]. | | Answers are usually wording from the passage, and you want the simplest model | Span reader | 66.8% of CoQA gold answers are span matches after ignoring case and punctuation [arxiv:1808.07042 §4.1]. The reader reaches 52.6 test F1 [arxiv:1808.07042 tab. 7]. | | Yes/no questions are common | Span reader with yes/no tokens | Yes and No make up 11.1% and 8.7% of CoQA questions [arxiv:1808.07042 tab. 4]. The yes/no tokens lift test F1 from 52.6 to 65.4 [arxiv:1808.07042 tab. 7]. | | Answers include counts, choices or wording that is not in the passage | Span reader followed by generation | A span-only answer cannot give a count that is absent from the text [arxiv:1808.07042 §1]. The pipeline reports 24.8 counting and 46.4 multiple-choice F1 [arxiv:1808.07042 tab. 10]. | | You can afford continued pretraining of an encoder and have a document-link graph | Pretrained encoder with document links | Base gains over BERT on MRQA are 2.6 F1 on average [arxiv:2203.15827 tab. 1]. Base continued pretraining took 4 days on four A100 GPUs; the cost of building the link graph is not stated [arxiv:2203.15827 §5.1]. | | Labelled QA data is scarce | Pretrained encoder with document links | With 10% of the SQuAD fine-tuning data, base F1 is 82.8 against 79.6 for BERT-base [arxiv:2203.15827 tab. 4]. The same table gives gains on HotpotQA, TriviaQA and NaturalQ. | | Contexts may contain distracting documents | Pretrained encoder with document links | On SQuAD with distractors, base F1 is 89.6 against 85.9 for BERT-base [arxiv:2203.15827 tab. 3]. Without distractors the gap is 1.4 [arxiv:2203.15827 tab. 3]. | | You need one small stored file per QA task on a shared frozen model | Frozen backbone with per-layer prompts | Only prompt vectors and a head are trained [arxiv:2110.07602 §1]. SQuAD 1.1 dev F1 is 94.4 against 94.6 for fine-tuning, RoBERTa-large [arxiv:2110.07602 tab. 3]. The paper does not discuss the cost of long prefixes at inference. | | You want the highest SQuAD 2.0 F1 from RoBERTa-large and can store a full model per task | Full fine-tuning | 89.4 F1 against 85.5 for the prompt variant [arxiv:2110.07602 tab. 3]. | | You need answers to be short, verbatim strings for an evaluator | Span reader | Span output is verbatim by construction. Free-form answers are scored by word overlap, which can miss paraphrases [arxiv:1808.07042 §6.1, §6.3]. | | Training data has supporting spans (rationales) | Any span reader | Span training targets can be built from rationales [arxiv:1808.07042 §3.1, §5.2]. Rationales are not given at test time [arxiv:1808.07042 §2]. | ## Implementation sketch Scoring for the cloze pointer reader. The paper gives the equations; the code that implements them is not reproduced here. The paper does not define the averaging constant in its word-weight equation, so the mean over passage positions is used here [arxiv:1607.04423 §3]. ```pseudocode # Score each passage word for a cloze question [arxiv:1607.04423 §3] # Input: passage states (D, h); question states (Q, h) # Output: probability over passage positions (D,) def cloze_pointer_scores(passage, question): # Pair-wise dot products between passage and question positions [eq. 5] match = passage @ question.T # (D, Q) # Per question word: softmax over passage positions [eq. 6-7] per_word = softmax(match, axis=0) # (D, Q) # Per passage position: softmax over question words [eq. 8] per_pos = softmax(match, axis=1) # (D, Q) # One learned weight per question word, averaged over passage positions [eq. 9] word_weight = per_pos.mean(axis=0) # (Q,) # Weighted sum of the per-word maps [eq. 10] return per_word @ word_weight # (D,) ``` Training target for a span reader. The paper gives these rules; the code that implements them is not reproduced here. ```pseudocode # Choose the training target span for an extractive reader [arxiv:1808.07042 §5.2] # Input: passage tokens; gold answer string; rationale span # Output: (start, end) token indices into the passage, or a yes/no index def gold_span(passage, answer, rationale): # Yes/no answers point to the appended token (augmented variant only) [§5.2] if answer in ("yes", "no"): return yes_no_index(answer) # Prefer the position of the answer inside the rationale [§5.2] if answer in rationale.text: return locate_in_rationale(answer, rationale) # Otherwise take the passage span with the highest F1 against the answer [§5.2] span = best_f1_span(passage, answer) # If no token overlaps, the paper falls back to an unknown token [§5.2] return span if span is not None else unknown_token_index(passage) ``` Inference for a span reader. The paper does not state a maximum answer length or the decoding rule for start and end, so set them and report what you used. ```pseudocode # Return an answer span for the current turn [arxiv:1808.07042 §5.2] # Input: passage tokens; history of (question, answer) pairs; question # Output: answer text taken from the passage (or "yes"/"no" if augmented) def answer_turn(passage, history, question, reader): # Prepend earlier turns to the current question [§5.2] query = flatten(history) + question # The reader scores start and end positions in the passage [§5.2] start, end = reader.predict(query, passage) # Return the text between the chosen positions [§5.2] return passage[start:end] ``` ## Contradictions No cross-paper disagreement. The four cards report on different datasets (CBTest, CNN, CoQA, MRQA and SQuAD dev), so their numbers are not comparable. Within the attention-over-attention paper, the text gives gains of 4.1 and 3.7 points over CAS Reader on CNN validation and test, while the table gives 4.9 and 4.4 [arxiv:1607.04423 §5.2] [tab. 2]. The card uses the table. The paper reports re-ranking results only for CBTest, not CNN [arxiv:1607.04423 tab. 2]. The P-tuning v2 SQuAD numbers are dev scores, and the paper does not say how the reported epoch or seed was chosen [arxiv:2110.07602 tab. 3]. Within the P-tuning v2 release, the README's numbers differ from the paper's tables [card: Paper vs code](/papers/2110.07602). Within the CoQA paper, the released baseline code does not match the paper in several places: the optimizer, the dropout value, the reader embeddings, and the augmented yes/no recipe, which is not in the code at the cited commit. The repository's README also reports dev F1 numbers that differ from the paper's table 9. Both sides are in the [card's Paper vs code section](/papers/1808.07042). Within the LinkBERT paper, the MRQA test split is not the official one, and the tiny-size ablations are not repeated at other sizes [arxiv:2203.15827 §5.2, tab. 5]. Treat the paper's numbers as author-reported and reproduce before relying on them. ## Papers - [Attention-over-Attention Neural Networks for Reading Comprehension](/papers/1607.04423): a cloze reader that weights query-to-passage attention by a learned passage-to-query attention, with optional n-gram re-ranking on CBTest. - [CoQA: A Conversational Question Answering Challenge](/papers/1808.07042): a conversational dataset, with span-reader baselines (DrQA, augmented DrQA) and a span-then-generate pipeline. - [LinkBERT: Pretraining Language Models with Document Links](/papers/2203.15827): an encoder pretraining method that links documents in one input, with MRQA results against BERT at three sizes. - [P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](/papers/2110.07602): frozen-backbone prompt tuning with prefix vectors at every layer, with SQuAD dev results against full fine-tuning.