Question Answering Over Temporal Knowledge Graphs

In plain words

A knowledge graph stores facts as links between things, such as a person holding a position. A temporal knowledge graph also stores the years in which each fact held, so a question like "who held this position in 2008?" needs both the links and the time. Before this work, most question-answering systems over knowledge graphs ignored time, and the few datasets for temporal questions were small.

The paper builds a large question set, CronQuestions, from a temporal graph derived from Wikidata, and writes the questions in natural language. It also proposes CronKGQA: a language model reads the question, and pre-trained embeddings that include time score every entity and every year as a possible answer at once.

The method does well on questions that need one fact and much less well on questions that need several facts chained together. All numbers come from the authors' own experiments.

TL;DR

• Use this card for temporal question answering over a knowledge graph, or as a source of a large synthetic temporal QA dataset.
• CronKGQA outperforms the baselines the authors tested on their dataset (author-reported) tab. 5.
• It needs gold entity and time annotations in the question, because entity linking is assumed to be solved §3.2.
• Its training recipe is only partly stated in the paper; the official code fills in the hyperparameters (see Hyperparameters).

Problem

Knowledge-graph question answering (KGQA) answers natural questions with entities from a graph. Temporal knowledge graphs add a start and end time to each edge, so a question can depend on when a fact held, for example "Who was the President of the USA during World War II?" §1.

Existing KGQA datasets are not temporal. TempQuestions, the temporal QA dataset the paper discusses, is built on a non-temporal FreeBase subset and has only 1,271 questions, useful only for evaluation §2.1. Recent KGQA algorithms work on non-temporal graphs, and extending them to timed facts is described as non-trivial §2.2. TEQUILA, the temporal method discussed, relies on pre-specified question templates §2.2.

The paper asks whether temporal KG embeddings can be used for temporal KGQA and how they compare with non-temporal embeddings and with language models alone §1.

Method

The paper contributes a dataset (CronQuestions) and a model (CronKGQA).

Dataset construction §3:

1. Temporal KG. Take all timed facts from a Wikidata subset used by the TNTComplEx work §3.1. Remove some instances of the predicate "member of sports team" to balance the graph, because it made up over 50 percent of facts. Discretise timestamps to years. Add event facts: entities with a start and end time, excluding game shows, movies and TV series, and excluding entities with fewer than 50 facts, stored as (event, significant event, occurred, start year, end year) §3.1. The subset before adding events has 323k facts, 125k entities and 203 relations; the final graph has 328k facts, of which 5k are event facts §3.1.
2. Templates. Use five frequent relations: member of sports team, position held, award received, spouse and employer §3.2. This gives 30 seed templates over five reasoning structures §3.2 tab. 2.
3. Paraphrases. Annotators paraphrase the slot-filled templates, giving 246 unique templates. A monolingual paraphraser generates more, and annotators verify them, leaving 654 templates §3.2 tab. 3.
4. Instantiation. Fill the templates with entity aliases from Wikidata to produce 410k question-answer pairs §3.2.
5. Split. Train 350,000, dev 30,000, test 30,000 tab. 4. Paraphrases of training questions do not appear in test. Test questions share no entities with training questions, though events may overlap §3.2.

Question types: a simple question needs one fact; a complex question is any other, split into before/after, first/last and time join §3.2.1 tab. 4. Each example has a paraphrased question, the set of entities and times in the question, and a set of gold answers, which are entities or years §3.2.

Temporal KG embeddings §4. CronKGQA uses TComplEx. Each entity and relation is a complex vector and each timestamp a complex vector too. The base ComplEx score is

\phi(s, r, o) = \Re\big(\langle u_s, v_r, \bar{u}_o \rangle\big) eq. 1

TComplEx adds the timestamp vector eq. 2 §4.2:

\phi(s, r, o, t) = \Re\big(\langle u_s, v_r, \bar{u}_o, w_t \rangle\big) eq. 2

TNTComplEx mixes a time-sensitive and a time-insensitive relation vector §4.2. TimePlex, with three relation representations and hyperparameters α, β, γ, is described §4.3 eq. 3 but has no row in the result tables.

CronKGQA §5 fig. 1. The method starts from a temporal KG embedding (TComplEx) and produces entity, relation and timestamp embeddings. It was built on EmbedKGQA, which used ComplEx and handled only non-temporal graphs and single entity questions §5. In CronKGQA:

• A pre-trained language model (BERT in the paper) encodes the question into one vector. Two projections turn it into a question embedding for entity answers and one for time answers §5 fig. 1.
• The entity score uses the subject, the question's time, and the candidate entity eq. 4. The time score uses the subject, the object, and the candidate timestamp eq. 5. The subject, object and time are taken from the question, with dummies when absent §5.
• Entity and timestamp embeddings are real-valued vectors of size 2D: the first half holds real parts and the second half imaginary parts §5.
• The two score vectors are concatenated and a softmax gives the answer distribution. The model is trained with cross-entropy loss §5.

# CronKGQA scoring  [§5] [eq. 4] [eq. 5]
# Input:  question text; subject s; time t; object o
# Output: scores for E entities, then T timestamps  # (E + T,)

def score_candidates(question, s, t, o):
    # Encode the question and keep the first ([CLS]) output vector  [§5]
    question_vector = language_model(question)[CLS]     # (768,)

    # Project to the complex embedding size 2D (real and imaginary parts)  [§5]
    relation = linear(question_vector)                  # (2D,)

    # Two separate projections, one per scoring function  [§5]
    entity_query = project_entity(relation)             # (2D,)
    time_query = project_time(relation)                 # (2D,)

    # Score every entity with the TComplEx product of s, t  [eq. 4]
    entity_scores = complex_product(s, entity_query, t)  # (E,)

    # Score every timestamp with the TComplEx product of s, o  [eq. 5]
    time_scores = complex_product_time(s, time_query, o)  # (T,)

    # One softmax over both sets; training uses cross-entropy  [§5]
    return concat(entity_scores, time_scores)           # (E + T,)

Baselines §6.1: BERT, RoBERTa and KnowBERT, each with a prediction head over the [CLS] token, and T5-3B. T5 is scored by exact match for year answers and by the closest entity alias (edit distance) for entity answers §6.1. The two Entities-as-Experts variants (T-EaE-add and T-EaE-replace) use TComplEx or TimePlex embeddings as the entity memory and a 6-layer transformer §6.1 app. A.1 app. A.2 eq. 6 eq. 7. The EmbedKGQA baseline uses ComplEx embeddings §5.

Hyperparameters

The paper states almost no training hyperparameters. The code's defaults and the values in its README command are given below, marked as code.

Results

Author-reported, on the CronQuestions dataset tab. 5. The paper does not state which split the table uses; the code's README evaluates the test split. Hits@1 and Hits@10 are the fraction of questions whose top-k predictions contain a gold answer train_qa_model.py.

Methods above the midrule in the paper use no KG embeddings; the rest use temporal or non-temporal ones tab. 5. Hits@10 is not available for T5-3B, which makes one prediction tab. 5.

Hits@1 by reasoning type tab. 6:

Hits@1 with ComplEx (CX) versus TComplEx (TCX) embeddings tab. 7:

The dataset itself tab. 4:

Training-size experiment fig. 2: described in words only. For T-EaE-add, more training data steadily raises performance on both simple and complex questions; for CronKGQA the trend holds for complex questions but not for simple ones §6.4.

Paper vs code

Official code: github.com/apoorvumang/CronKGQA at code_commit, MIT license. The paper links it in its final version §1. The repository was cloned and read at that commit.

• Question encoder. Paper: BERT §5. Code: DistilBERT, 768-dimensional, with the first output vector as the question embedding qa_models.py qa_models.py.
• Language model training. Paper §6.4 says CronKGQA "merely fine tune[s] BERT" §6.4. Code defaults to freezing the LM, and the README command passes --lm_frozen 1 train_qa_model.py README.md. Either the paper's runs unfroze it or the paper's description is loose; the paper does not say which.
• Dropout and batch norm. Not in the paper. Code applies batch norm and dropout 0.3 after each projection qa_models.py qa_models.py.
• Frozen embeddings. Paper does not say whether the KG embeddings are fixed. Code freezes them by default and in the README run qa_models.py train_qa_model.py.
• Dummy values. Paper: a dummy entity or time is used when one is missing §5. Code has no dummy entity: a missing object is replaced by the subject qa_datasets.py. A missing time becomes the first timestamp id, not a separate token qa_datasets.py.
• Paraphrases. Paper: each example has a paraphrased question §3.2. Code uses only the first paraphrase, for all splits qa_datasets.py.
• Gold answers in training. Paper: cross-entropy over the score vector §5. Code: each time a question is read, one gold answer is drawn at random as the target qa_datasets.py. Multi-answer questions therefore see different targets across epochs. Evaluation uses all gold answers train_qa_model.py.
• Stopping. Fig. 2 caption says training ran until validation Hits@10 stopped rising for 10 epochs fig. 2. Code trains for a fixed max_epochs and keeps the checkpoint with the best validation Hits@1, checked every valid_freq epochs train_qa_model.py train_qa_model.py.
• Entity-only scoring. Paper eq. 4 uses only the subject eq. 4. Code matches this by default; the alternative that pools subject and object is off unless combine_all_ents is set train_qa_model.py qa_models.py.
• Unused module. answer_type_embedding is created and frozen but not used in forward qa_models.py qa_models.py.
• Device. Model and data code call .cuda() directly, so the code needs a CUDA device as written qa_models.py train_qa_model.py.
• Embedding checkpoint. The README downloads the KG embedding and dataset archives from a shared drive; the checkpoint is not in the repository, so its rank and training settings cannot be checked from this code README.md.
• Dataset version. The README warns that the original dataset had an error where {tail2} was left in the question text, affecting 31,951 of 410,000 questions. The paper's numbers are from the original version; the README recommends the fixed one README.md.

No discrepancy was found in the scoring equations themselves: the code's time score and entity score follow eq. 4 and eq. 5 with the TComplEx product.

Pitfalls

• Gold annotations are needed at inference. Entity and time mentions in the question are assumed given, so the model is not an end-to-end system §3.2. The head entity is the first annotated entity by character position, found with str.find on the entity string, so the stored entity strings must occur verbatim in the question qa_datasets.py qa_datasets.py.
• Synthetic questions. Questions are generated from templates and paraphrases, so their distribution is artificial in a semantic sense; the authors call this a limitation §3.2.
• Checkpoint and split. The numbers depend on the original (non-fixed) dataset and on a validation-selected checkpoint; the split for each table is not stated. Use the fixed dataset and report which split you used.
• Complex questions remain weak. CronKGQA reaches 0.392 Hits@1 on complex questions, and before/after is the hardest type (best method 0.288) §6.3 tab. 6.
• Abstract arithmetic. The abstract says the dataset is 340× larger than the previous one; 410,000 divided by 1,271 is about 323. The abstract's "120% increase" over the next-best method does not match Table 5 either: 0.647 against 0.288 is about +125%. Use the table values.
• Text versus Table 6. §6.3 says every model except EmbedKGQA does best on time-join questions §6.3. Table 6 shows T-EaE-add and T-EaE-replace at 0.175 and 0.168 on time join, below their before/after and first/last scores. Only CronKGQA matches the sentence tab. 6.
• Temporal vs non-temporal. With ComplEx embeddings, time-answer Hits@1 falls to 0.057 for CronKGQA-CX (identical to EmbedKGQA) and 0.062 for T-EaE-replace-CX; TComplEx raises them to 0.549 and 0.231 tab. 7 §6.5.
• Unevaluated TimePlex. Section 4.3 describes TimePlex but no table reports it §4.3.
• Trainable size. CronKGQA trains only the question projection and two small heads, with the LM and KG embeddings frozen by default. The EaE baselines train a 6-layer transformer from scratch, which the paper links to their weaker gains at smaller training sizes §6.4 app. A.2 train_qa_model.py.

Lineage

• Builds on: EmbedKGQA (Saxena et al., 2020), the ComplEx-based KGQA method this work extends §5. No card in this wiki.
• Builds on: TComplEx and TNTComplEx (Lacroix et al., 2020) for temporal KG embeddings §4.2. No card in this wiki.
• Compared against: BERT, RoBERTa, KnowBERT, T5-3B, Entities-as-Experts (EaE) variants and EmbedKGQA §6.1 tab. 5. TEQUILA is discussed in §2.2 but not compared in the tables.
• Superseded by: not known from this wiki.