Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

In plain words

Many question-answering datasets that need several facts to answer a question only check the final answer. This paper builds a set of about 190,000 questions over Wikipedia and Wikidata. Each question also comes with the chain of facts that leads to its answer, so a reader can see how the model got there. Questions come from templates and are filtered so that they really need two steps, for example "Who is the grandchild of X?".

The chain of facts gives a way to check a model's reasoning, not only its answer. The authors' baseline does poorly on that chain, and human readers do much better. The paper also shows the dataset is harder than HotpotQA for a model of the kind HotpotQA was built for.

TL;DR

• A dataset of 192,606 English multi-hop questions in four types: comparison, inference, compositional and bridge-comparison §2.2 tab. 1.
• Each sample has a question, a context of 10 paragraphs, supporting sentences, an answer, and evidence triples (subject, relation, object) from Wikidata §2.1 §3.2.
• A HotpotQA-style baseline with an added evidence branch scores 5.41 joint F1 on test (author-reported) tab. 5.
• The paper releases a dataset, a baseline and an evaluation script; the repository carries no dataset-generation code (checked against the file list).

Problem

Existing multi-hop QA datasets give no complete explanation of how the answer follows from the question. HotpotQA's sentence-level supporting facts are a binary classification and cannot test reasoning skill §1. The authors also cite analyses finding that many HotpotQA examples do not need multi-hop reasoning §1. R4C has explanations but only 4,588 questions, too few for end-to-end training §1 tab. 8.

Method

Task and metrics §2.1

Input: a question Q and a set of documents D. Outputs: (1) an answer A, a textual span in D; (2) a set of supporting sentences; (3) a set of evidence triples E that describes the reasoning path from Q to A §2.1.

Answer and supporting facts are scored with exact match (EM) and F1. Joint scores multiply precision and recall across the three tasks, and joint EM is 1 only when all three tasks match exactly §2.1 eq. 1.

\mathrm{Joint\ F1} = \frac{2 P^{joint} R^{joint}}{P^{joint} + R^{joint}}, \quad P^{joint} = P^{ans} P^{sup} P^{evi}, \quad R^{joint} = R^{ans} R^{sup} R^{evi} eq. 1

Question types §2.2

• Comparison: compares two entities on one property, e.g. who was born first §2.2.
• Inference: two triples (e, r1, e1) and (e1, r2, e2) give a new triple (e, r, e2) through a logical rule. The question is built from (e, r) and the answer is e2 §2.2 app. A.3.
• Compositional: the same two triples, but no inference relation exists; the question names e and both relations §2.2 app. A.4.
• Bridge-comparison: a bridge entity is found first, then the answer is a comparison of a property of that entity §2.2 app. A.5.

Sources §3.1 app. A.1

Only the summary paragraph of each Wikipedia article is used as an entity's description §3.1. Triples come from Wikidata §3.1. The paper used the English Wikipedia dump of 1 Jan 2020 and the English Wikidata dump of 31 Dec 2019 app. A.1. The entities were grouped by the Wikidata value of "instance of" into 23,763 groups app. A.1. Comparison questions use the top-50 popular entities app. A.1 app. A.2; bridge-comparison questions use the human and film entity types app. A.5.

Generation §3.2 alg. 1 alg. 2 app. A.6

Generation has three steps: create templates, generate data, and post-process §3.2.

• Templates. Comparison templates were made by hand from named-entity patterns in the 17,456 HotpotQA comparison questions in its training data §3.2. There are 62 bridge-comparison templates app. A.5 and 799 compositional templates app. A.4. Inference templates are listed in tab. 11 app. A.3.
• Logical rules. The paper uses 28 logical rules over Wikidata relations, verified by hand from AMIE output §3.2; the rule table is tab. 10 app. A.3. Only relations with a single value per entity are used, so that each question has one answer app. A.3.
• Requirements. A generated sample must pass checks that the object of each triple appears in the relevant Wikipedia summary §3.2 alg. 1. For bridge questions, the first paragraph must mention the bridge entity, the second must mention the answer, the first must not mention the answer and the second must not mention the start entity §3.2 fig. 2.
• Distractors. Bigram tf-idf retrieves the top 50 paragraphs for the question; the top 8 of those with the gold paragraphs' entity types are kept (top 6 for bridge-comparison, with 4 gold paragraphs) and shuffled into a context of 10 paragraphs §3.2.
• Post-processing. Yes/no questions are balanced, since random pairing made many "no" samples §3.2 app. A.7. Ambiguous bridge samples, where a second triple shares the subject and relation, are discarded app. A.7. Comparison samples with equal numeric values are dropped app. A.7.

# Comparison question generation  [alg. 1] [app. A.6]
# Output: one sample (Q, A, C, SF, E) per call

def generate_comparison_sample(templates, group, wikipedia, wikidata):
    # Choose two entities from the same group  [alg. 1, line 1] [§3.2]
    entity_1, entity_2 = random_pair(group)
    # Fetch the Wikidata statements of both entities  [alg. 1, line 2]
    statements_1 = wikidata.statements(entity_1)
    statements_2 = wikidata.statements(entity_2)
    # Keep the relations both entities have  [alg. 1, line 3] [§3.2]
    shared = mutual_relations(statements_1, statements_2)
    for relation in shared:
        object_1 = statements_1[relation]
        object_2 = statements_2[relation]
        # Each object must appear in its own Wikipedia summary  [§3.2]
        if not appears(object_1, wikipedia[entity_1]):
            continue
        if not appears(object_2, wikipedia[entity_2]):
            continue
        # Fill a random template for this relation  [alg. 1, line 5]
        template = random_choice(templates[relation])
        question = fill(template, entity_1.label, entity_2.label)
        # Context is the two summaries, concatenated  [§3.2]
        context = wikipedia[entity_1] + wikipedia[entity_2]
        # Evidence is the two Wikidata triples  [§3.2]
        evidence = [(entity_1, relation, object_1),
                    (entity_2, relation, object_2)]
        # Supporting facts are the sentences holding the objects  [§3.2]
        supporting = sentence_indices(context, object_1, object_2)
        # Answer is the comparison result, e.g. the earlier date  [§3.2]
        answer = compare(relation, object_1, object_2)
        yield question, answer, context, supporting, evidence

# Bridge question generation  [alg. 2] [app. A.6]
# Output: one sample (Q, A, C, SF, E) per candidate chain

def generate_bridge_samples(first_relations, wikidata, wikipedia):
    # Pick a random start entity e  [alg. 2, line 1] [§3.2]
    start = random_entity()
    # One-hop statements, filtered to the predefined first relations
    # [alg. 2, line 3] [§3.2]
    hop_1 = [(r1, e1) for r1, e1 in wikidata.statements(start)
             if r1 in first_relations]
    for r1, bridge in hop_1:
        # Second hop from the bridge entity  [alg. 2, line 4] [§3.2]
        for r2, answer_entity in wikidata.statements(bridge):
            # Check the four paragraph requirements  [§3.2] [fig. 2]
            if not passes_bridge_checks(start, bridge, answer_entity,
                                        wikipedia):
                continue
            # The question template depends on (r1, r2)  [alg. 2, line 6]
            question = fill(random_choice(templates[r1, r2]), start)
            # Evidence is the two triples  [§3.2]
            evidence = [(start, r1, bridge), (bridge, r2, answer_entity)]
            yield question, answer_entity, evidence

Splits §3.2 tab. 1

A single-hop model (BERT, §5.1) was used to split the data. Five-fold cross-validation gave an average F1 of 86.7% §3.2. Questions the single-hop model solves form train-medium; the rest form train-hard, dev and test, with question types balanced across the three §3.2.

Baseline model §5.2 model.py

The baseline is adapted from the HotpotQA baseline and adds one block that predicts evidence §5.2. It is a stack of bidirectional GRUs with bi-attention over question and context model.py. The evidence branch predicts relations from the question, then uses the supporting-fact-weighted context to find a subject span and an object span for each predicted relation §5.2 model.py.

# Evidence branch of the baseline  [§5.2]
# [code: baseline_model/model.py#L136-L147]
# Shapes: question_states (B, Q, d); relation_logits (B, R)

def predict_relations(question_states):
    # Multi-label relation logits from the question only
    # [code: baseline_model/model.py#L137-L138]
    relation_logits = linear_relation(encode(question_states))
    # Independent sigmoid per relation; trained with BCE
    # [code: baseline_model/model.py#L140-L142]
    probabilities = sigmoid(relation_logits)
    # A relation is on when its probability is at least 0.5
    # [code: baseline_model/model.py#L143]
    active = probabilities >= 0.5
    return active

Hyperparameters

The paper states no training hyperparameters beyond the data split. The values below are the code defaults and the values in the README training command, which match each other main.py README.md.

Results

Author-reported. Setup: the 2WikiMultiHopQA dataset, a HotpotQA-style baseline, human annotators on 100 test samples §5.2 §5.3.

Split sizes tab. 1:

Question types and lengths tab. 2:

Dataset quality against HotpotQA, same multi-hop model tab. 4 §5.1:

Single-hop BERT: 64.6 F1 on HotpotQA and 55.9 F1 on this dataset, a gap of 8.7 F1 §5.1.

Baseline on the dev and test splits tab. 5 §5.2:

Baseline on the test split, per question type tab. 6 §5.2:

Model against human annotators on 100 random test samples; each sample had three graduate-student annotators, and the upper bound takes the best of them per sample tab. 7 §5.3:

The paper reports the evidence task as the largest gap (75.6 human vs 16.7 model F1) §5.3.

Paper vs code

• Loss. The paper gives no training loss. The code trains on the sum L_ans + 0.5 · L_sp + 0.5 · (L_subj + L_rel + L_obj), where L_ans is the start, end and answer-type cross-entropy summed per example and averaged over the batch [code: baseline_model/run.py#L127-L133] main.py.
• Answer types. The paper describes four question types §2.2. The answer-type head has three outputs (span, yes, no) [code: baseline_model/model.py#L90]. The data iterator also maps a label 3 for "noanswer" util.py, but the preprocessing code I read only emits yes (-1) and no (-2) labels, plus spans [code: baseline_model/prepro.py#L176-L186]. Label 3 is therefore not produced by the code I read.
• Evidence. The paper says the question predicts the relations, then the context gives the subject and object §5.2. The code matches this: relations come from the question encoding model.py, and the relation-selected context attention feeds subject and object pointers model.py. The predicted evidence strings are exact context spans, not Wikidata labels [code: baseline_model/run.py#L285-L299].
• Joint metric. Equation 1 matches the code, which multiplies answer, support and evidence precision and recall, and scores joint EM only when all three are present 2wikimultihop_evaluate_v1.1.py eq. 1.
• Updated results. The paper's Table 5 test row matches the README's leaderboard entry dated Oct 25, 2020 README.md. The README says tables 5, 6 and 7 were recomputed on Dec 8, 2020 with a new evaluation script and extra ID fields, and the updated tables are in update_results/ as PNG images README.md. I did not read those images, so the paper's numbers may differ from the latest release.
• Dataset generation. Algorithms 1 and 2 alg. 1 alg. 2 have no counterpart in the repository. The file list holds only the baseline, the evaluation scripts and the data links (checked against the repository file list). The generation pipeline cannot be checked against code.
• Environment. requirements.txt pins torch==1.1.0 [code: requirements.txt]. The code passes enforce_sorted=False to pack_padded_sequence model.py, which I believe needs a newer PyTorch. I did not run it. The code also calls .cuda() unconditionally run.py and uses the legacy volatile=True flag run.py.
• No discrepancies found in the joint metric or in the definition of the four question types.

Pitfalls

• Weight flag bug. --evi_lambda is declared with type=int [code: baseline_model/main.py#L67]. The default is 0.5, but any value passed on the command line is parsed as an integer, so a fractional value cannot be given. Set the weight in code or fix the type.
• Span length cap. The span mask keeps only spans of up to 16 tokens inclusive model.py. Longer answers or evidence spans can never be predicted. The paper does not mention this.
• Relation count. The paper does not give the number of evidence relations. The code defaults to 38 main.py, and its vocabulary is built from the training data prepro.py. Keep the two in step.
• Joint metric is a product. Joint precision and recall are products of the three tasks' values, so each is smaller than any single task's value. The baseline's test scores show the effect: answer F1 43.93, support F1 65.26, evidence F1 14.94, joint F1 5.41 tab. 5. Joint EM requires all three tasks to be exact at once: 36.53 answer EM falls to 0.35 joint EM tab. 5.
• Wikipedia and Wikidata do not always agree. A check of 100 training samples found 8 with a mismatch between the summary and the triple §5.4. The sentence chosen for support is the first sentence where the object appears, which the paper admits can be semantically wrong §5.4 app. A.1.
• Names are ambiguous. Wikidata can list several names for one entity, but the ground truth uses one of them §5.3. Evidence exact match is therefore noisy. The v1.1 evaluation uses IDs and aliases [code: 2wikimultihop_evaluate_v1.1.py#L141-L168].
• Yes/no balance. Random pairing made many "no" comparisons §3.2. The paper says post-processing balances yes and no, but the target ratio is not stated.
• Few inference questions. Only single-valued relations are used app. A.3, so the inference type has 7,478 examples tab. 2. The paper notes that several logical rules can be false in some cases and checks each new triple against Wikidata before use app. A.3.
• Split model. The single-hop BERT that split the data is described only by its F1 of 86.7% §3.2. Its training settings are not stated, and the repository does not include it.
• Dataset release. The paper's counts are from arXiv v2 (Nov 2020). The repository says the data gained ID fields in Dec 2020 and fixed sentence segmentation in Apr 2021 README.md. I did not check the released files against the counts in Table 1 and Table 2.
• Data format. Each sample has _id, question, answer, supporting_facts, context, evidences, type, and entity_ids, following HotpotQA's format. The test split has no answers or evidence README.md.
• GPU assumption. The baseline does not run on CPU as written [code: baseline_model/run.py#L85].

Lineage

• Builds on: HotpotQA §5.1 §5.2, whose distractor setting and baseline the paper adapts. The single-hop BERT of §5.1 is used for the split and for the reasoning test.
• Compared against: HotpotQA and R4C §6 tab. 8.
• Superseded by: not stated in this card.