A Survey of Knowledge-Enhanced Text Generation

In plain words

Text generation systems turn an input into output words: a reply in a chat, a summary, an answer, a headline. When the model sees only the input, its output is often generic or thin, because the input does not carry the facts, word choices or common sense that the output needs.

This survey collects the many ways researchers have added extra knowledge to such models. Some knowledge is pulled out of the input itself, such as its topics, keywords, grammar or word graph. Other knowledge comes from outside, such as a knowledge base of facts, a graph of concepts and their relations, or background documents. The survey sorts the techniques by how the knowledge reaches the model: as an extra attention input, as words the model may copy, as memory, as a graph encoding, through pre-training, through extra training targets or constraints, or by steering the model when it generates.

It is meant for researchers and practitioners. It also selects nine datasets for evaluating such methods. It runs no new experiments of its own.

TL;DR

• A survey, with no experiments of its own, of text generation that adds knowledge beyond the input text to encoder-decoder models §1.
• Organised as general architectures and training methods §2, internal knowledge (topic, keyword, linguistic features, open knowledge graphs) §3, and external knowledge (knowledge bases, knowledge graphs, grounded text) §4.
• Covers more than 80 papers from 2016 on, spread over seven applications §1.
• Selects nine benchmark datasets from four existing general NLG benchmarks §5.
• Use it as a map of approaches and datasets. For exact method details, go to the cited papers.

Problem

Encoder-decoder models that see only the input often fail on tasks that need facts, background or common sense. The survey frames the task as producing an output Y from an input X plus additional knowledge K, with the model exploiting the dependencies among X, K and Y §1. It names two challenges: obtaining the relevant knowledge from diverse sources, and understanding it well enough to use it in generation §1.

Method

The survey defines no method of its own. Its taxonomy, with section numbers of the paper, is:

1. Basic models §2.1. RNN, CNN and Transformer encoder-decoders, trained by maximum likelihood. The sequence probability factorises token by token §2.1, and the training loss is the negative log-likelihood eq. 5.
2. Architecture-level ways to use knowledge §2.2:
   • Knowledge attention: an extra attention over knowledge vectors (topic, keyword, knowledge base, graph or grounded-text representations). The knowledge context is merged with the usual context vector by an MLP §2.2.1.
   • Copy and pointing: CopyNet and the pointer-generator mix a generate distribution and a copy distribution over an extended vocabulary. Knowledge modes add further sources of tokens, such as knowledge-base entities §2.2.2.
   • Memory networks: knowledge is written to memory and read over several hops, used for knowledge bases, topics and dialogue history §2.2.3.
   • Graph networks: GNNs and graph-to-sequence models encode knowledge graphs or dependency graphs, with a readout over the final node states §2.2.4.
   • Pre-trained language models: knowledge enters either by injecting entity embeddings computed beforehand, or implicitly through knowledge-related pre-training tasks §2.2.5.
3. Learning and inference §2.3:
   • Knowledge as target (§2.3.1): auxiliary losses that predict knowledge (multi-task), or targets derived from knowledge as weak supervision.
   • Knowledge as condition (§2.3.1): the model learns p(Y | X, K).
   • Knowledge as constraint (§2.3.2): posterior regularisation adds a constraint term to the training objective.
   • Inference-time control (§2.3.3): plug-and-play steering of a fixed language model by gradient updates to its history state.
4. Internal knowledge §3:
   • Topic (§3.1): M1 topic words from generative topic models; M2 joint training with a CNN topic model; M3 neural topic models trained with variational inference.
   • Keyword (§3.2): M1 keyword assignment; M2 keyword extraction.
   • Linguistic features (§3.3): POS and NER tags, syntactic and semantic dependency graphs.
   • Open knowledge graphs (§3.4).
5. External knowledge §4:
   • Knowledge base (§4.1): M1 supervised tasks built around the KB for joint optimisation; M2 selecting KB facts to incorporate.
   • Knowledge graph (§4.2): M1 graph embeddings in generation; M2 triplet information transferred into a language model; M3 reasoning by path finding; M4 graph embeddings improved by GNNs.
   • Grounded text (§4.3): M1 guiding generation with retrieved information; M2 modelling background knowledge in response generation.
6. Benchmarks §5 and future directions §6.

The M1, M2 and so on labels restart in each section. An M1 in §3.1 is not the same method as an M1 in §4.1.

Knowledge-aware context for one decoding step, as described in §2.2.1:

# Knowledge-aware context for one decoding step  [§2.2.1]
# Input:  s_prev (d,); encoder states H (n, d); knowledge K (m, d)
# Output: knowledge-aware context c_tilde (d,)

def knowledge_aware_context(s_prev, H, K):
    # 1. Ordinary attention over the input states; the scorer eta is an
    #    MLP [§2.2.1]. attention_sum is a helper, not named in the paper.
    c_input = attention_sum(s_prev, H)               # (d,)
    # 2. Attention over the knowledge representations  [§2.2.1]
    c_knowledge = attention_sum(s_prev, K)           # (d,)
    # 3. Merge both contexts with an MLP  [§2.2.1]
    c_tilde = mlp(concat(c_input, c_knowledge))      # (d,)
    return c_tilde

Output mixing in the pointer-generator, as described in §2.2.2:

# Pointer-generator mixing over the extended vocabulary  [§2.2.2]
# Input:  p_gen (scalar in 0..1); vocab_dist (V,); copy_dist (n,)
# Output: final_dist (V_ext,)

def final_distribution(p_gen, vocab_dist, copy_dist, source_ids):
    # 1. Scale the generate-mode distribution by the switch probability
    #    p_m(g), which the paper says comes from an MLP  [§2.2.2]
    final_dist = p_gen * vocab_dist                  # (V_ext,)
    # 2. Add the copy-mode mass to the source tokens it points to  [§2.2.2]
    #    scatter_add is a helper, not named in the paper
    final_dist = scatter_add(final_dist, source_ids, (1 - p_gen) * copy_dist)
    return final_dist

Hyperparameters

The survey runs no experiments §1 and states no training settings of its own §1. Dataset split sizes are listed under Results §5. Any value not covered by the cited papers is "not stated".

Results

No new experiments. The survey's concrete output is its benchmark selection §5. Split sizes as given in the paper's table §5:

Selection rules §5: open download link; a spread of tasks, with at most three datasets per task; a mix of internal- and external-knowledge datasets; and a preference for multi-reference datasets.

Other reported figures are not the survey's own. The survey cites one summarisation paper for answering about 10% more topical questions correctly after adding a knowledge graph §1. That figure comes from the cited paper and was not checked here. The leaderboard tables in the appendix are not transcribed on this card.

Paper vs code

No official code. The paper's front matter links a GitHub repository that it describes as a collection of papers and code § front matter. That is a reading list of other people's work, not an implementation of anything in the survey, so it is not recorded as code. I checked only the paper's own links; the repository was not cloned.

Pitfalls

• Date window is stated two ways. The abstract says the survey covers the past five years [abstract]; the introduction says the paper set is work published or released in or after 2016 §1. Work after the last arXiv version (v4, 2022-01-22) is not covered.
• Selection, not exhaustive. The survey covers "more than 80" papers across seven applications §1. Missing methods are not evidence that a method is weak.
• Results are not comparable across papers. Within each table, the survey compares only "with knowledge" against "without knowledge" inside one paper, because the papers use different tasks and datasets [§3.1, §3.2 table captions]. Do not rank methods from the tables.
• Taxonomy lives in figures. The category diagrams are not reproduced here. Use the section list above.
• Posterior regularisation as typeset is ambiguous. The constrained problem in §2.3.2 puts the slack variable ξ both in the objective and in the constraint, and its sign and role are not explained. Check the cited posterior-regularisation source before implementing it.
• Pointer-generator switch. §2.2.2 says the generate-versus-copy probability is produced by an MLP, but does not give that MLP's inputs or size.
• Plug-and-play steering has unstated constants. For the gradient update in §2.3.3, the normalisation exponent γ, step size and number of update iterations are not stated. "Repeated multiple times" is the only guidance.
• Graph depth not given. §2.2.4 describes k-hop aggregation but states no value for k.
• Pre-trained-model details are not given. For the knowledge-injected pre-trained models in §2.2.5, the survey gives no training configuration.

Lineage

• Builds on: the Seq2Seq encoder-decoder line §1, §2.1; attention, copy and pointer mechanisms, memory networks and graph neural networks as used in generation §2.2; pre-trained models such as BERT and T5 §2.2.5. Only named in the text, with no arXiv ids given here.
• Compared against: general NLG surveys that do not cover knowledge enhancement (Garbacea et al., Gatt et al.), a survey of knowledge-graph techniques for NLG (Ji et al.), and a survey of structured knowledge for reading comprehension and retrieval (Wang et al.) §1.
• Superseded by: not stated.