Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

In plain words

Most multi-step question answering benchmarks spell out the steps in the question itself. "Was Aristotle alive when the laptop was invented?" tells you to find two dates and compare them. StrategyQA asks questions where the steps are not written down, such as "Did Aristotle use a laptop?". To answer it, you have to work out a plan first, here a comparison of time periods, and then look up the facts that plan needs fig. 1.

The authors built the dataset with crowd workers. Each worker starts from a random Wikipedia term and writes a yes/no question about it. Models try to answer each question, and questions they get right too easily are thrown out, so the set keeps questions that need more than surface cues. Every kept question is then broken into steps, and each step is matched to a Wikipedia passage that answers it fig. 2.

The benchmark is useful for two things: testing whether a system can find the right plan for a question it has not seen spelled out, and testing whether retrieving the right passages helps. Current systems in the paper score well below people on it.

TL;DR

• StrategyQA is a boolean QA benchmark with 2,780 questions whose reasoning is implicit, so a strategy must be inferred §1.
• Each question has a step decomposition and Wikipedia evidence paragraphs for each retrieval step §2.2.
• Data collection primes workers with single Wikipedia terms and filters questions with pretrained and continuously fine-tuned solver models §3.1.
• Author-reported best baseline: 72.0% accuracy with gold paragraphs and gold decompositions; 63.6% with no context; humans 87% §5.2 tab. 9 tab. 7.
• The official code (MIT) reproduces the baselines on a 10% held-out split of the training data, not on the official test set README.md.

Problem

Multi-step QA benchmarks such as HotpotQA state the reasoning steps in the question, so the decomposition can be read off the text §1. Real questions often leave the steps implicit §1. For implicit questions, retrieval is harder because the question shares little vocabulary with its evidence, and the answer cannot be found by matching words in the question §1.

Earlier crowdsourced multi-hop datasets showed workers an entire context while writing questions. This produced high lexical overlap between question and context, and so reasoning shortcuts §1 §3.1. Writing many questions also led workers to repeat patterns, which limits diversity §1 §3.1.

Scope assumptions: the corpus is Wikipedia, assumed to be correct §2.1. Answers are restricted to yes or no to focus the work on question complexity rather than output format §2.1.

Method

Definitions

• A strategy question has at least two steps, uses implicit reasoning (a step needs words not in the question, judged by a vocabulary rule of thumb), and has a definitive answer that can be reached from the corpus §2.1.
• A decomposition is an ordered list of steps. A step may refer to the result of an earlier step through a placeholder. The last step gives the final answer §2.2.
• Each step is a retrieval step (needs a fact from the corpus), an operation (a logic step over earlier answers), or both §2.2.
• Each retrieval step has supporting evidence: one or more paragraphs that answer it §2.2.

Collection pipeline

The pipeline runs three crowd tasks, plus checks §3 fig. 2.

1. Creative question writing. The worker sees a Wikipedia term (a page title, filtered for popularity by number of contributors and backlinks §3.1), its short description, and an expected answer. They write a question and the facts needed to answer it §3.1. The expected answer is sampled to balance yes and no answers §3.1.
2. Model-in-the-loop filtering. Five pretrained solvers (Ptd) reject a question if at least four answer it correctly. Three solvers fine-tuned during collection (Fntd) reject it if all three answer correctly §3.1. Fntd is updated every 1K questions §4.
3. Decomposition. The decomposition is written before evidence is matched, to avoid bias from seeing context §3.2. The worker sees the question, the answer, and the facts from step 1 §3.2. For each step the worker names a Wikipedia page where they expect the answer §3.2.
4. Evidence matching. For each retrieval step, the worker searches a Wikipedia index built from the 11/05/2020 Cirrus dump §3.3 and drags paragraphs from search results into the step. Only indexed paragraphs can be chosen, at paragraph level §3.3. Each decomposition is matched by three workers §4.
5. Verification. Workers pass qualification tests, and authors review annotations §3.4. Automatic checks confirm that a question ends with a question mark and avoids explicit multi-hop phrasing §3.4. The decomposition must be a directed acyclic graph in which every step is reachable from the last §3.4. A subset is re-checked by an evidence verification task, in which workers answer each step from the matched paragraphs alone §3.4.

# Question collection loop  [§3.1] [§4]
# Input:  wikipedia_terms, pretrained_solvers, finetuned_solvers
# Output: accepted questions with expected answer and facts
def collect_questions(wikipedia_terms, pretrained_solvers, finetuned_solvers):
    # Accepted questions are kept in order of collection  [§4]
    accepted = []
    for term in wikipedia_terms:
        # Up to five questions are written per term  [§4]
        expected = sample_answer_balancing_yes_no(accepted)  # [§3.1]
        question, facts = worker_writes_question(term, expected)  # [§3.1]
        # Reject if at least 4 of 5 pretrained solvers answer correctly  [§3.1]
        if count_correct(pretrained_solvers, question) >= 4:
            continue
        # Reject if all 3 fine-tuned solvers answer correctly  [§3.1]
        if all_correct(finetuned_solvers, question):
            continue
        accepted.append((question, expected, facts))
    return accepted

Baseline models

• Base model. RoBERTa is fine-tuned in sequence on DROP, 20Q and BoolQ. The result is called RoBERTa* §5.1.
• No context. The model reads only the question and outputs a binary prediction from the classification token §5.1.
• With context. BM25 retrieves paragraphs from the corpus, with stop words removed from queries §5.1. Two retrieval modes are described: question-based (IR-Q), using the question as query and taking the top k=10; and decomposition-based (IR-D), using a separate query per gold or predicted step and concatenating results sorted by score §5.1. The model input is the question together with the context, truncated to 512 tokens §5.1.
• Decomposition model. BART is fine-tuned to generate the decomposition token by token §5.1.
• Oracle variants. Gold paragraphs (ORA-P), retrieval with gold decomposition (IR-ORA-D), and two answer-by-step variants that use gold decompositions and gold paragraphs §5.1.
• Solvers used in collection. Three no-context models and two question-retrieval models §5.1 tab. 8.

The paper states that exact training details are in the released codebase §5.1, footnote. The official code is the source for the values in the Hyperparameters section.

# Decomposition-based context for one question  [§5.1]
# Input:  question, steps (strings, gold or predicted)
# Output: at most 10 paragraphs, best score first
def decomposition_context(question, steps):
    # Stop words are removed from each step before the query  [§5.1]
    queries = [remove_stop_words(step) for step in steps]
    # One search per step  [§5.1]
    results = [bm25_search(query) for query in queries]
    # Merge all steps, sort by score, drop repeated paragraph ids  [§5.1]
    merged = sort_by_score(flatten(results))
    unique = drop_repeated_ids(merged)
    # Keep at most 10 paragraphs in total across all steps
    # [code: src/data/dataset_readers/strategy_qa_reader.py#L370-L387]
    return unique[:10]

Classifier input

# Input for the binary QA classifier  [§5.1]
# Input:  question, paragraphs (ordered by retrieval)
# Output: token ids of at most 512 tokens
def build_input(question, paragraphs):
    # Paragraphs are joined into one context string
    # [code: src/data/dataset_readers/strategy_qa_reader.py#L284]
    context = " ".join(paragraph.text for paragraph in paragraphs)
    # The context is truncated; the question is kept  [§5.1]
    # [code: src/data/dataset_readers/strategy_qa_reader.py#L219]
    return tokenize(context, question, truncation="only_first", max_tokens=512)

Hyperparameters

Values marked "paper" come from the paper text. Values marked "code" come from the official repository and are not stated in the paper.

Dataset statistics

Reproduced from tab. 4. Numbers are as the paper reports them.

Totals and process counts §4: 1,799 Wikipedia terms; 2,835 questions collected; 55 flagged for an incorrect answer during decomposition; 2,780 kept. 29 question writers, 19 decomposers, 54 evidence matchers. Authors state that 3,305 filtered questions are also released §4.1.

Reasoning complexity: most questions have three or more decomposition steps, of which about two retrieve external knowledge fig. 3 §4.1.

Data quality (sample of 100, expert authors) tab. 5

Agreement: κ = 0.73 for implicit and κ = 0.65 for multi-step §4.2. 94% of answers were agreed with on a web-enabled check of 100 questions, with 2% disagreeing §4.2.

Evidence coverage: 88.3% of questions were fully covered by some worker; 0.5% had no evidence from any of three workers §4.2. In a 217-step check, 93% of matched paragraphs were judged valid by majority vote, with κ = 0.42 §4.2.

Top strategies tab. 6

Share of 100 analysed examples: physical 13%, biological 11%, historical 10%, temporal 10%, definition 8%, cultural 5%, religious 5%, entertainment 4%, sports 4% tab. 6 §4.3. The nine listed categories cover 70% of the sample.

Human performance (100 questions, expert authors) tab. 7

Experts had Wikipedia access and could reveal the decomposition §4.4. Main error: failure to find evidence (10%); the rest (3%) came from ambiguous questions §4.4.

Results

Author-reported. Setup: StrategyQA test set tab. 9; RoBERTa-based models; BM25 retrieval over the Wikipedia index; standard deviation across 7 experiments per row tab. 9 §5.2. Baseline models and their training are described in §5.1 tab. 8.

Retrieval: question-based retrieval reaches Recall@10 0.174, predicted-decomposition retrieval 0.195, and gold-decomposition retrieval 0.282 tab. 9 §5.2. Human experts reach 0.586 Recall@10 with the decomposition and 0.513 without, on 100 questions §5.2.

Authors' reading: retrieval quality at this level does not beat the no-context model §5.2. Gold decompositions help retrieval more than question text does, but predicted decompositions do not yet beat question-based retrieval §5.2. Analysis of 50 failed steps points to BM25 word matching, since evidence often uses different words §5.2.

Paper vs code

Official code: eladsegal/strategyqa, MIT license, owner is author Elad Segal, the README calls it the official code of the paper. Checked at commit 1ba1e97452e189569357876f2854b01357ffbe37. Data included: data/strategyqa/train.json (2,061 entries) and dev.json (229 entries) README.md train.json.

• Splits. Paper: official train (2,290) and test (490), results on test tab. 9. Code: the 2,290 official training questions split 90/10 into train and dev. The README says the split is unofficial README.md. The test labels are not in the repository, so the code cannot reproduce the test numbers. Config evaluate_on_test: true names no test file in the repository base.jsonnet.
• Base model. Paper: RoBERTa, size not stated §5.1. Code: roberta-large base.jsonnet.
• DROP stage. Paper: fine-tune RoBERTa on DROP, then 20Q, then BoolQ §5.1. Code: the DROP stage is a download of a finished DROP checkpoint from an external location, not a training step in this repository train_RoBERTa_STAR.py. The 20Q and BoolQ stages are in the repository 1_twentyquestions.jsonnet 2_boolq.jsonnet.
• Decomposition retrieval cap. Paper: "concatenating the top k=10 results of all steps" §5.1. Code: results from all steps are merged, sorted, de-duplicated, then cut to 10 in total strategy_qa_reader.py strategy_qa_reader.py. The retrieval table is consistent with a 10-paragraph total §5.2.
• Gold paragraphs. Paper: "gold paragraphs" §5.1. Code: the union of evidence from all annotators, in step order strategy_qa_reader.py, then capped at 10 paragraphs strategy_qa_reader.py.
• Retrieval service. Paper: BM25 retrieval §5.1. Code: a retrieval URL that is empty in the repository elasticsearch_utils.py. The index script creates an Elasticsearch index, and no similarity setting is visible in it index_wikipedia.py. BM25 is therefore not confirmed by the code. The repository includes a query cache so that baselines can be rerun for the queries it holds queries_cache.json README.md.
• Hyperparameters. Paper: not stated, "released as part of our codebase" §5.1. Code: values in the Hyperparameters table.
• Checkpoint selection. Paper: not stated. Code: BART-Decomp is selected by SARI base.jsonnet.
• Training filter. Paper: no statement. Code: when training with retrieved context, examples with missing context are skipped; at evaluation, a blank context is used strategy_qa_reader.py strategy_qa_reader.py.
• Recall@10. Paper: maximum over three annotators, top 10 §5.2. Code: same recall@10.py. Questions with no evidence score 0 and stay in the average recall@10.py.
• Solvers and crowdsourcing. Paper: solver thresholds and Fntd updates §3.1 §4. Code: no solver or crowdsourcing code in the repository, so these are not verified.

Each row of the results table has a matching config file , and the oracle configs follow the paper's descriptions §5.1. The discrepancies above are the ones found on reading; the remaining code paths were not fully traced.

Pitfalls

• Test set not reproducible. Official test labels are needed for the reported numbers and are not in the repository. Use the leaderboard linked in the README README.md or accept the dev-split numbers. Do not compare the 10% dev split to the test table.
• Retrieval stack not included. The retrieval endpoint is blank, and cached results cover only the queries in the cache. A query missing from the cache returns no results elasticsearch_utils.py README.md. To reproduce retrieval, you need the corpus and an index with the same settings, and the paper does not state the index settings.
• Cap of 10 applies everywhere. The paragraph cap is applied after merging, so decomposition-based and gold-paragraph inputs are both trimmed to 10 paragraphs strategy_qa_reader.py. The paper's text on IR-D reads as if the cap is per step §5.1.
• Truncation. The paper says 512 tokens. The code truncates the context only, keeps the question, and sets no explicit maximum length in the configs strategy_qa_reader.py. The 512 limit therefore comes from the tokenizer's default for the model (not verified).
• Dropout is not applied. dropout = 0.1 is defined in the base configs but nothing in the configs references it base.jsonnet base.jsonnet. The effective dropout is whatever the pretrained model config sets (not verified).
• Pretraining is partly external. The DROP stage uses a downloaded checkpoint train_RoBERTa_STAR.py. Reproducing RoBERTa* from scratch requires the DROP training recipe, which this repository does not contain.
• Oracle numbers need the placeholder pipeline. The last-step rows depend on replacing decomposition placeholders with predicted answers. The no-placeholder files are generated by an iterative procedure §5.1 README.md. The paper does not give its exact settings beyond those in §5.1.
• Cost and labour. About $4 per example was paid, and each decomposition needed three evidence matchers §4. Writers were recruited continuously so that strategies would not repeat §4. Replicating the dataset is a large crowdsourcing effort, not a code run.
• Human baseline. Experts were authors with Wikipedia access and the option to see decompositions §4.4. They are not a crowd baseline.
• Dataset licence is not confirmed. The repository has no dataset licence, and the paper does not state one. The dataset page link resolved to a different site in our check. Do not copy the dataset files into this wiki. The statistics above are reported numbers only.
• Version. The arXiv record in raw/arxiv is v1 (2021-01-06). The LaTeX source has the accepted-paper style, and section and table numbers here follow that source. Check the compiled PDF before you cite a number, in case the published version differs.

Lineage

• Builds on: decomposition annotations inspired by BREAK, Wolfson et al. 2020 §2.2. Adversarial data collection follows Bartolo et al. 2020 §3.1. Not compiled in this wiki.
• Compared against: HotpotQA and ComplexWebQuestions for question length and n-gram statistics §4.1. Natural Questions and MS MARCO for real-user question collection §1. Not compiled in this wiki.
• Superseded by: not known from wiki content.