Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

In plain words

Reading-comprehension tests give a model a paragraph and a question, and the model must point at the answer in the text. Many such questions can be solved by matching words: the question and the answer share vocabulary with the sentence that contains it. Quoref was built to ask questions that only make sense if the model tracks which words refer to the same person or thing, for example when a pronoun or a second name points back to an entity mentioned earlier in the paragraph.

Writers were asked to write such questions. To keep out questions that a simple word-matching shortcut could answer, a BERT-based question answerer ran during writing. If its answer matched the writer's answer, the draft was rejected and the writer had to change it. The result is a benchmark for coreference-aware reading. Strong models score well below people on it, and the authors found that the models partly exploit shortcuts, such as picking the most frequent entity in the paragraph.

TL;DR

Quoref is a 2019 extractive reading-comprehension dataset (question, paragraph, span or set-of-span answers) whose questions require resolving coreference. Writers were checked by a live adversary, an uncased BERT QA model trained on SQuAD 1.1. The authors report that the best baseline (XLNet QA, base) reaches 70.5 test F1, against an estimated human 93.4 F1 §4.3. The paper is a useful benchmark reference and a warning about annotation artifacts. The released code is a leaderboard submission for XLNet QA, not the full training pipeline.

Problem

Long texts repeatedly refer to the same entities, and tracking those references is needed for full comprehension §1. Existing crowdsourced reading-comprehension datasets focus mostly on local predicate-argument structure, with very few questions that need long-distance entity tracking §1. Two obstacles: crowd workers find coreference hard to apply consistently, and even well-targeted questions often contain lexical cues that let a model answer without the intended reasoning §1.

Method

Dataset construction §2

1. Paragraphs come from Wikipedia articles on English-language films, art and architecture, geography, history and music. For films, the paragraphs are plot summaries of at least 40 tokens, drawn from the list of English-language films. For the other categories, they come from the featured-articles lists. About 40% of the paragraphs are movie plot summaries §2.
2. Crowd workers on Mechanical Turk must find two or more co-referring spans in a paragraph and write a question whose answer requires knowing that those spans refer to the same thing. Workers do not mark the coreferent spans. The answer spans they select need not be the coreferent spans themselves [§2, with footnote].
3. An adversary runs in the background while a worker types: an uncased base BERT QA model trained on SQuAD 1.1 §2. A question is accepted only if the adversary's prediction does not match the worker's answer §2. The paper does not define "match" (exact string, token F1 or another criterion).
4. Among models with acceptable latency, the authors chose uncased base BERT as the adversary on the basis of qualitative results §2, footnote.
5. Worker pool: 21 workers chosen with a two-stage qualification process. Each HIT had 4 paragraphs (movies) or 5 paragraphs (other domains), and workers wrote 10 questions per HIT in about 20 minutes, for US$7 per HIT app. A.4.

# Adversarial question collection  [§2] [app. A.4]
# Input:  paragraph; adversary = uncased base BERT QA trained on SQuAD 1.1
# Output: accepted (question, answer spans) pairs

def collect_question(paragraph, adversary):
    # Worker picks two or more co-referring spans; no span marking [§2]
    spans = worker_pick_coreferent_spans(paragraph)
    # Worker writes a question and selects answer spans  [§2]
    question, answer = worker_write_question(paragraph, spans)
    # Adversary answers in real time  [§2]
    predicted = adversary.answer(paragraph, question)
    # Reject if the adversary's prediction matches the answer  [§2]
    # ("match" is not defined in the paper)
    if matches(predicted, answer):
        return None
    return (question, answer)

Dataset statistics tab. 1

Manual analysis §3 tab. 2

The authors manually analysed 100 random paragraph-question pairs. 78% of the questions cannot be answered without resolving coreference. The other 22% involve coreference that does not have to be resolved to answer §3. Phenomena categories, which overlap tab. 2:

• pronominal resolution: 69%;
• nominal resolution: 54%;
• multiple resolutions (both kinds): 32%;
• commonsense reasoning: 10%.

Baselines §4

Metrics: exact match (the SQuAD implementation) and an F1 over bag-of-words overlap between gold and predicted answers, which the paper takes from DROP §4. The paper describes F1 as macro-averaged.

Models §4.1:

1. QANet, with the AllenNLP implementation modified to use the marginal objective of Clark and Gardner (2018) §4.1.
2. QANet + BERT: frozen BERT representations are concatenated to the original input embeddings §4.1.
3. BERT QA, base size (the large model did not fit the GPU memory) [§4.1, footnote]. This is the adversary used during collection §2.
4. XLNet QA, base size. It is pretrained on more text than BERT, including Giga5, ClueWeb 2012-B and Common Crawl §4.1.

Heuristic baseline §4.2: passage-only BERT QA and passage-only XLNet QA. They use the BERT QA and XLNet QA training setup, but the question is replaced by an empty string §4.2 app. B.

Span-extraction fine-tuning for BERT QA and XLNet QA app. B

# Fine-tune a span-extraction QA model  [app. B] [§4.1]
# Input:  train questions with answer spans; dev set
# Output: checkpoint with the best dev-set score

def fine_tune(model, train_set, dev_set):
    # Paragraph and question become windows of 512 wordpieces  [app. B]
    # consecutive windows overlap by a stride of 128  [app. B]
    features = sliding_windows(train_set, max_len=512, stride=128)
    # AdamW; the learning rate is typeset as 3^{-5}  [app. B]
    optimizer = adamw(model.parameters(), learning_rate=3e-5)
    for epoch in range(10):  # 10 epochs  [app. B]
        # Batch size 10  [app. B]
        for batch in batches(features, size=10):
            loss = model.span_loss(batch)  # loss not described in paper
            optimizer.step(loss)
        # Keep the best-on-dev checkpoint  [app. B]
        keep_if_best_on(dev_set, model)
    return best_checkpoint

Answer-level evaluation (the script in the official repo) [code: utils_squad_evaluate.py#L58-L113]

# Token F1 for one gold answer against one prediction
# [code: utils_squad_evaluate.py#L79-L92]
def f1(gold, predicted):
    # Lowercase; remove punctuation, articles and extra spaces  [code#L58-L70]
    gold_tokens = normalize(gold).split()
    pred_tokens = normalize(predicted).split()
    # Multiset overlap of tokens  [code#L82-L83]
    shared = overlap_count(gold_tokens, pred_tokens)
    if shared == 0:
        return 0.0
    precision = shared / len(pred_tokens)
    recall = shared / len(gold_tokens)
    return 2 * precision * recall / (precision + recall)

# Per question: the best score over all gold answer strings  [code#L110-L112]
def question_score(gold_answers, predicted):
    return max(f1(gold, predicted) for gold in gold_answers)

Hyperparameters

Results

Author-reported. Setup: Quoref train, dev and test splits, span-extraction models, base-size encoders §4.1, tab. 1. Human performance is estimated from the authors' answers to 400 test questions, scored with the same metric §4.3, footnote.

Source: tab. 3. Text: XLNet QA is about 4.12 F1 above BERT QA on test §4.3. BERT QA trained on SQuAD would score 0 F1 on Quoref, since SQuAD-trained BERT is the adversary §4.3.

Error analysis on the dev set §4.4: 51% of the questions that BERT QA answered correctly point at the first or the most frequent entity in the paragraph, against 12% of its wrong answers. For XLNet QA the figures are 48% and 11%. Nearest-entity answers: 17% of XLNet QA's correct answers and 4% of its wrong answers (BERT QA: 17% and 6%). This uses a manual analysis of 100 dev questions §4.4.

Paper vs code

The official repository is allenai/quoref-leaderboard-example (MIT), an XLNet QA leaderboard submission at commit a2ff145e. It contains inference only: run_model.sh downloads a trained checkpoint from S3 and predicts on the leaderboard input run_model.sh. Training uses the generic run_squad.py script, whose defaults differ from the paper.

• Learning rate: paper "3^{-5}" app. B; script default 5e-5 [code: run_squad.py#L376-L377]. The repo never sets a value.
• Epochs: paper 10 app. B; script default 3 run_squad.py.
• Batch size: paper 10 app. B; script default 8 per GPU [code: run_squad.py#L372-L373], so the total depends on GPU count.
• Sequence length: paper 512 app. B; the run script passes 512 [code: run_model.sh#L6], but the script default is 384 run_squad.py.
• Stride: 128 in both paper and run script app. B run_model.sh.
• Max question length: not in the compiled paper; script default 64 [code: run_squad.py#L360-L361], which the run script keeps.
• Lowercasing: paper says BERT is uncased and XLNet cased, with the data lowercased app. B. The run script passes --do_lower_case to xlnet-base-cased run_model.sh, which matches the paper's text.
• Metric: paper uses a bag-of-words F1 taken from DROP §4. The script takes a SQuAD-style token F1 and the maximum over gold answer strings per question utils_squad_evaluate.py. It does not implement a DROP-style multi-span alignment. How a multi-span answer is stored in the data file is not in this repo.
• Ambiguous gold answers: the appendix example lists several gold strings for one question app. A.3. The max over gold strings handles this [code: utils_squad_evaluate.py#L110-L112].
• Not in this repo: QANet, the passage-only models, and the SQuAD-trained BERT adversary used during collection. The paper cites the AllenNLP QANet implementation §4.1. No discrepancies were checked for these.
• Search performed: the paper's project page (allennlp.org/quoref) redirects to a different host, which I did not read further. GitHub search for "quoref" found allenai/quoref-leaderboard-example (MIT), owned by the authors' organisation, whose README links the leaderboard and the EMNLP 2019 paper. I used it as the official code.

Pitfalls

• Learning rate. The text and the appendix typeset it as "3^{-5}". The intended value is probably 3e-5, but it is not stated. Check the exponent base against the released training configuration before you use it.
• Adversary. Workers' questions are filtered by an uncased BERT QA trained on SQuAD 1.1 §2. The paper does not define "match" §2. The authors note that the BERT QA score on Quoref may partly reflect artifacts in the dataset §4.3, §4.4, so the adversary's family is not a neutral comparator.
• Shortcuts remain. The passage-only models reach 28 to 29 test F1 [tab. 3]. The paper reports they mostly predict the most frequent entity §4.3, and the error analysis finds the same bias for BERT QA and XLNet QA §4.4. Expect part of the score to come from annotation artifacts §4.4.
• Small manual samples. The 78% coreference figure comes from a 100-pair manual analysis §3. The paper does not say how many annotators did it or whether they agreed. The 17% vs 4% nearest-entity figures come from 100 dev questions §4.4.
• Human score. The 93.4 F1 human estimate uses the authors' answers to 400 test questions §4.3, footnote. It is not an inter-annotator study.
• Multi-span answers. About 10% of questions have multi-span answers in every split tab. 1. Metric details for those questions are only partly specified (see Paper vs code).
• Windowing. Long paragraphs are split into 512-wordpiece windows with stride 128 app. B. How window-level predictions are merged is not stated in the paper. The run script uses an n-best list of 20 with answers up to 30 tokens run_squad.py.
• Cost. Per HIT: 10 questions, about 20 minutes, US$7 app. A.4. Also note the two-stage selection of 21 workers app. A.4.
• Source hygiene. The LaTeX source contains a commented-out hyperparameter table with other values (for example the paragraph and question lengths and gradient clipping). Those values are not in the compiled paper and are not cited here.

Lineage

• Builds on: SQuAD 1.1 (the adversary's training data) §2; the DROP F1 metric §4; the adversarial-data-collection idea of DROP §2.
• Compared against: QANet §4.1, QANet + BERT §4.1, BERT QA (arXiv:1810.04805) §4.1, and XLNet QA (/papers/1906.08237) §4.1.
• Superseded by: not stated in the wiki's current content.