Commonsense knowledge injection

Overview

A task model can receive knowledge from outside its own weights, and the knowledge is produced for each input. The four cards on this wiki that use this slug show two sources. In the first, a separate commonsense model writes a set of vectors for each input, and the task model reads them. COSMIC keeps them in a recurrent state 2010.02795 §3.3 2010.02795 §3.4; CEM gates them into an encoder that feeds a decoder 2109.05739 §3.2 2109.05739 §3.4. In the second, the system retrieves a subgraph of a knowledge graph around the question and a candidate answer, and a graph neural network runs over that subgraph together with a pretrained language model 2104.06378 §3 2201.08860 §4.1. The forms differ in where knowledge enters the model and in what it costs: at each utterance's state update (COSMIC) or in the encoder before decoding (CEM) in the first, and inside a joint graph for each candidate answer in the second.

Variants

Generated relation vectors fed into recurrent states

For each utterance, a COMET encoder (trained on the ATOMIC if-then knowledge base) reads the utterance followed by one of five relation phrases. The encoder state after the last token is a 768-dimensional vector. The five relations are the speaker's intent, the effect on the speaker, the reaction of the speaker, the effect on the others and the reaction of the others 2010.02795 §3.3 2010.02795 tab. 1. Card: COSMIC.

The task model keeps a shared context state and, for each participant, an internal state, an external state, an intent state and an emotion state. Each state is updated by a GRU that receives the commonsense vector for the matching role 2010.02795 §3.4 2010.02795 tab. 2. The knowledge therefore enters at the state update, not at the input layer. The paper updates the listener states with the listener-side vectors 2010.02795 §3.4; the card's Paper vs code section records that the released code leaves them off by default, so reproduce the setting you intend.

Generated relation vectors gated into a reply encoder

For the last user utterance, a COMET model trained on ATOMIC-2020 generates five inferences for each of five relations: xReact, xWant, xNeed, xIntent and xEffect. The paper drops xAttr on purpose, because it judges the other person 2109.05739 §3.2. The inferences for each relation are concatenated into one sequence. The affective relation xReact goes through its own encoder and its states are averaged. The four cognitive relations share a second encoder, and their first state is kept 2109.05739 §3.2 eq. 4-5. Each knowledge vector is concatenated to every token of the dialogue encoding, and the five refined streams are joined per token, passed through a sigmoid gate and mapped back to the model width by an MLP 2109.05739 §3.3 2109.05739 §3.4 eq. 13-15. The decoder attends to this fused context 2109.05739 §3.5. Only the last utterance receives knowledge; earlier turns are encoded without it 2109.05739 §3.2. Card: CEM.

Retrieved subgraph fused with a language model

QA-GNN takes the entities named in the question and the candidate answer, adds their two-hop neighbours from ConceptNet and keeps the 200 nodes that score highest for relevance, scored with the language model 2104.06378 §3.2 2104.06378 §4.2. It adds one node for the question and answer, linked to the question and answer entities, and runs five attention layers over the graph 2104.06378 §3.1 2104.06378 §4.3. Card: QA-GNN.

GreaseLM uses the same retrieval procedure 2201.08860 app. B.1. Its language model and graph network run side by side, with several fusion layers between them. In each fusion layer, one interaction token and one interaction node are the only states that pass between the two streams 2201.08860 §4.3. The number of fusion layers is 5 for CommonsenseQA, 6 for OpenbookQA and 3 for MedQA in the paper's table 2201.08860 tab. 7. Card: GreaseLM.

When to use which

The choice depends mainly on the input shape, the knowledge available and the compute budget. Figures are author-reported and come from different papers and datasets, so they show what each design reported, not a like-for-like ranking.

Implementation sketch

The common core is a lookup of knowledge for one input, an encoder for the text, and a fusion step that uses both. The variants differ in what the lookup returns and where the fusion happens. Details are in the cards.

# Common core: knowledge for one input, then a fused prediction
def predict_with_knowledge(text, candidates):
    # Generated vectors (COSMIC, CEM) or a retrieved subgraph (QA-GNN, GreaseLM)
    # [arxiv:2010.02795 §3.3] [arxiv:2109.05739 §3.2] [arxiv:2104.06378 §4.2]
    knowledge = get_knowledge(text)
    # Pretrained encoder for the text  [arxiv:2010.02795 §3.2] [arxiv:2201.08860 §4.2]
    text_vec = encoder(text)
    # Graph variants score each candidate separately  [arxiv:2201.08860 §4.1]
    scores = []
    for a in candidates:
        # Fuse text and knowledge: state updates, gated encoder or joint graph layers
        # [arxiv:2010.02795 §3.4] [arxiv:2109.05739 §3.4] [arxiv:2201.08860 §4.3]
        fused = fuse(text_vec, knowledge, a)
        scores.append(head(fused))
    return scores

Contradictions

No two cards report conflicting numbers for the same setting. The shared baselines agree: RoBERTa-large reaches 68.7 on the CommonsenseQA in-house test in both QA-GNN and GreaseLM 2104.06378 tab. 3 2201.08860 tab. 2, and the SapBERT-with-QA-GNN MedQA result of 38.0 appears in both papers 2104.06378 tab. 7 2201.08860 tab. 6.

The size of the gain is reported very differently. COSMIC's commonsense component adds about 2 points on IEMOCAP 2010.02795 tab. 4, while QA-GNN reports +4.7 over RoBERTa on CommonsenseQA 2104.06378 §4.5. The tasks differ, so these do not contradict each other, but a reader should not read them as one scale.

Inside GreaseLM, the text gives a gain of 0.9 over the LM+KG baselines on CommonsenseQA test, and the table gives 0.8 against QA-GNN 2201.08860 §6 2201.08860 tab. 2. This page uses the table value. COSMIC's Table 4 caption gives both five runs and ten runs for some rows, and the paper does not resolve this 2010.02795 tab. 4.

CEM's text says it "achieves the lowest perplexity" 2109.05739 §4.3, but its Table 2 lists the "w/o Div" ablation lower, at 35.60 against 36.11 2109.05739 tab. 2. This page uses the table values.

Papers

• COSMIC: COmmonSense knowledge for eMotion Identification in Conversations: adds five COMET relation vectors per utterance to a recurrent emotion model and reports gains on four dialogue datasets.
• CEM: Commonsense-aware Empathetic Response Generation: adds five COMET relation vectors for the last user turn to a Transformer reply model through a gated fusion, with a diversity loss, and reports gains over five baselines on EmpatheticDialogues.
• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering: joins a pretrained LM with a retrieved ConceptNet subgraph, scored and reasoned over by a GNN with a question-context node.
• GreaseLM: Graph REASoning Enhanced Language Models for Question Answering: runs an LM and a GNN side by side and exchanges information through an interaction token and node in several fusion layers.