Knowledge base question answering

Overview

A knowledge base question answering system reads a question, finds the entities it mentions in a graph of facts, and returns the entities that answer it. When the facts carry time, the question can depend on when a fact held, for example "who held this position during a given year?" 2106.01515 §1. The standard non-temporal setting ignores that time dimension. Temporal KBQA adds it, and the answer can then be a year as well as an entity [arxiv:2106.01515 §3.2].

The field splits by the kind of question. A survey groups natural-language methods into single-turn questions, where each question stands alone, and multi-turn questions, where a user reformulates or follows up on earlier answers 2412.10390 §5.1 2412.10390 §5.2. The survey proposes no method and reports no numbers, so it gives the map and the names; the results below come from the CronKGQA paper alone.

The paper contributes a dataset and a method. CronQuestions has 410k question-answer pairs over a temporal graph derived from Wikidata, split into 350k training, 30k development and 30k test questions 2106.01515 §3.2 tab. 4. CronKGQA is the method that the paper proposes for this data 2106.01515 §5.

The same lookup idea also appears in task-oriented dialogue, where the system must state a fact from a small knowledge base inside its reply. There the answer is generated text rather than a ranked entity 1705.05414 §1 1705.05414 §2.3.

A second family maps the question to a logical form that the knowledge base executes, instead of ranking entities directly. Its test sets are split by how far the questions move from training: i.i.d., compositional and zero-shot 2011.07743 §2.2. The GrailQA paper is the example used here.

Variants

CronKGQA: language model plus temporal KG embeddings

A language model encodes the question into one vector. Two projections turn that vector into a query for entities and a query for years. Entities are scored with the subject entity and the question's time, and years with the subject and the object 2106.01515 §5 eq. 4 eq. 5. The graph embeddings come from TComplEx, a temporal embedding model 2106.01515 §4.2 eq. 2.

Reported result: overall Hits@1 0.647 and Hits@10 0.884 on CronQuestions, against 0.288 Hits@1 for the best baseline 2106.01515 tab. 5. Card: Question Answering Over Temporal Knowledge Graphs.

Logical-form parsing with BERT: GrailQA parser

The model reads the question together with candidate schema items and linked entities. BERT encodes them in chunks, and an LSTM decoder either writes a logical form token by token over a pruned vocabulary (transduction), or scores whole candidate queries (ranking) 2011.07743 §4.1 §4.2 §4.4. Transduction keeps only items within two hops of the linked entities 2011.07743 §4.2. Ranking enumerates candidates that use at most two relations and start from a linked entity, so it cannot return a query outside that set. The authors measure about 80% recall for these candidates 2011.07743 §4.4.

Reported results on the GrailQA test set, author-reported [arxiv:2011.07743 tab. 3]: ranking with BERT reaches 50.6 exact match (EM) and 58.0 F1 overall, and 48.6 EM and 55.7 F1 on zero-shot questions. Transduction reaches 33.3 EM and 36.8 F1 overall. With GloVe in place of BERT, ranking falls to 45.1 F1 overall and 33.8 F1 zero-shot, but on i.i.d. questions its EM is higher (62.2 against 59.9). Entity linking is a separate BERT-based step. Its errors account for 33% of sampled errors 2011.07743 §5.4.

Pre-training on GrailQA and then testing on WebQSP, with no WebQSP training data for the zero-shot setting, raises EM from 0.00 to 0.35. With 10% of WebQSP as training data, it raises EM from 0.44 to 0.55 [arxiv:2011.07743 §5.5]. These are single runs, and the card records that the dataset's own licence was not checked. Do not redistribute GrailQA without checking it. The paper states that all models use the same entity linker [arxiv:2011.07743 §5.2], but the released training code uses gold entities; see the card's Paper vs code section before comparing numbers. Card: Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases.

KB keys inside a dialogue decoder: KV Retrieval Net

The model is an LSTM encoder-decoder with attention over the dialogue context. Each knowledge-base entry is a (subject, relation, object) triple. Its key is built from the subject and relation embeddings, and a second attention scores these keys against the decoder state. The scores are added to the output logits at canonical tokens such as "meeting_time", and when the decoder emits one, it is replaced by the stored value 1705.05414 §2.3 eq. 7 eq. 8. The model has no belief tracker or dialogue-state labels and is trained with token-level cross-entropy 1705.05414 §5.2. Each knowledge base holds at most 230 triples 1705.05414 §2.3.

On the authors' in-car corpus (calendar, weather and points of interest), with 304 test dialogues, it reaches BLEU 13.2 and entity F1 48.0. The rule-based system gets 6.6 and 43.8 and Copy Net gets 11.0 and 37.0 [arxiv:1705.05414 tab. 3]. Human ratings on a 1-to-5 scale put it above the rule-based system on fluency, cooperation and humanlikeness, and below the human row [arxiv:1705.05414 tab. 4]. These are author-reported single results: the paper reports no variance and does not state the number of runs 1705.05414 §5.2. The corpus licence is not stated, and its download page could not be reached, so do not redistribute the data 1705.05414 §1. No official code was found, so an implementation must be written from the description. Card: Key-Value Retrieval Networks for Task-Oriented Dialogue.

Non-temporal baselines: EmbedKGQA with ComplEx

The paper's baseline EmbedKGQA uses ComplEx embeddings, which ignore time, and scores only entities 2106.01515 §5. It reaches 0.288 overall Hits@1 and 0.057 on time-answer questions 2106.01515 tab. 5 tab. 7. The paper also compares Entities-as-Experts models with TComplEx or TimePlex memories 2106.01515 §6.1. The source of EmbedKGQA has no card in this wiki, so its details are not used here. The survey also lists EmbedKGQA as a single-turn method that embeds the question and ranks answers by similarity 2412.10390 §5.1.

Single-turn ranking and retrieval (survey)

The survey names several single-turn families, each for one question at a time 2412.10390 §5.1:

• Subgraph retrieval: PullNet pulls a subgraph around candidate answers.
• Embedding similarity: KV-Mem and EmbedKGQA embed the question and rank answers.
• Candidate re-ranking: PrefNet re-ranks candidates.
• Query-graph generation: RnG-KBQA ranks candidate query graphs, generates the correct one, and alternates between the two steps.
• Joint training: BINET trains question answering and graph completion together.
• Path walking: reinforcement-learning agents walk the graph to find answer paths.

The survey gives no results for these, so compare them only with numbers from their own papers.

Multi-turn conversational KBQA (survey)

Multi-turn methods use the user's follow-up questions as signal. The survey names 2412.10390 §5.2:

• ConQuer: treats user reformulations as implicit feedback for a graph-walking reinforcement-learning agent.
• PRALINE: ranks graph paths with contrastive learning, marking paths that reach the correct answer as positives.
• CornNet: adds reformulations generated by a language model.
• Rewriting approaches: reformulation-based rewriting, logical-form generation and question rewriting in the style of CANARD.

See Conversational question answering for the dialogue setting in general.

LLM as a graph-walking agent (survey)

The survey describes one way to combine a language model with a graph: the model scores a node's neighbours from the topic entities and moves on when the current node lacks the answer (Think-on-Graph) 2412.10390 §6.2. The survey does not compare this with the methods above.

When to use which

Implementation sketch

The scoring step is the common core of CronKGQA. The text on each variant gives the details 2106.01515 §5. The survey gives no KBQA pseudocode; its only algorithm is for rule chaining, which is a different task 2412.10390 §3.1.1.

# Score every entity and every year for one question  [arxiv:2106.01515 §5]
# Input:  question text, subject s, question time t, object o
# Output: scores for E entities and T years  # (E + T,)

def score_candidates(question, s, t, o):
    # Encode the question into one vector with a pre-trained language model  [§5]
    question_vector = language_model(question)          # (d,)

    # Project it into the complex embedding space, once per answer type  [§5]
    entity_query = project_entity(question_vector)      # (2D,)
    time_query = project_time(question_vector)          # (2D,)

    # Entity score: TComplEx product of subject, entity query and time  [eq. 4]
    entity_scores = tcomplex(s, entity_query, t)        # (E,)

    # Year score: TComplEx product of subject, year query and object  [eq. 5]
    time_scores = tcomplex_time(s, time_query, o)       # (T,)

    # One softmax over both sets, trained with cross-entropy  [§5]
    return concat(entity_scores, time_scores)           # (E + T,)

Where the variants differ: the question encoder, whether the embeddings are frozen during training, and how missing subject, object or time values are filled in. The paper leaves several of these unstated; see the card's Pitfalls section.

KV Retrieval Net uses a different step. It does not rank entities. It adds one score per KB key to the decoder's output logits and then decodes greedily, replacing canonical tokens with stored values 1705.05414 §2.3 eq. 9.

GrailQA's ranking mode has a different core: it scores each enumerated logical form with the decoder and returns the best one. The paper does not state the scoring rule 2011.07743 §4.4.

Contradictions

No direct conflict between the papers, because the survey reports no numbers. CronKGQA reports Hits@1 on CronQuestions, KV Retrieval Net reports BLEU and entity F1 on an in-car corpus, and GrailQA reports EM and F1 on logical forms and answers. The numbers measure different things on different data and are not comparable 2106.01515 tab. 5 1705.05414 tab. 3 2011.07743 tab. 3. Within the CronKGQA paper, the text and the table disagree about time-join questions: the text says every model except EmbedKGQA does best on them, but the table shows both Entities-as-Experts baselines scoring lower on time-join questions than on before/after and first/last questions 2106.01515 §6.3 tab. 6.

Papers

• Key-Value Retrieval Networks for Task-Oriented Dialogue: an LSTM dialogue model that scores knowledge-base keys inside its reply decoder, with no belief tracker.
• Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases: a BERT seq2seq parser for KBQA in transduction and ranking modes, and the GrailQA dataset with three generalization test levels.
• Question Answering Over Temporal Knowledge Graphs: introduces the CronQuestions dataset and the CronKGQA method for temporal KBQA.
• Neural-Symbolic Reasoning over Knowledge Graphs: A Survey from a Query Perspective: a survey that groups single-turn and multi-turn natural-language KBQA methods and their LLM combinations. It proposes no method and reports no numbers.