--- kind: concept slug: commonsense-knowledge-injection title: Commonsense knowledge injection summary: >- Commonsense knowledge injection gives a task model knowledge from outside its weights, generated or retrieved per input. COSMIC adds COMET relation vectors to emotion states and CEM to reply encoders; QA-GNN and GreaseLM fuse a retrieved ConceptNet subgraph with a pretrained LM. Trade-off: extra per-input work, for author-reported gains from under 1 to about 5 points. tags: [commonsense, knowledge, dialogue, question-answering] papers: ["2010.02795", "2104.06378", "2109.05739", "2201.08860"] related: [emotion-recognition-in-conversation, dialogue-generation, graph-neural-network, knowledge-graph-reasoning, recurrent-neural-network, attention-mechanism] status: draft updated_at: 2026-10-19 --- ## Overview A task model can receive knowledge from outside its own weights, and the knowledge is produced for each input. The four cards on this wiki that use this slug show two sources. In the first, a separate commonsense model writes a set of vectors for each input, and the task model reads them. COSMIC keeps them in a recurrent state [arxiv:2010.02795 §3.3] [arxiv:2010.02795 §3.4]; CEM gates them into an encoder that feeds a decoder [arxiv:2109.05739 §3.2] [arxiv:2109.05739 §3.4]. In the second, the system retrieves a subgraph of a knowledge graph around the question and a candidate answer, and a graph neural network runs over that subgraph together with a pretrained language model [arxiv:2104.06378 §3] [arxiv:2201.08860 §4.1]. The forms differ in where knowledge enters the model and in what it costs: at each utterance's state update (COSMIC) or in the encoder before decoding (CEM) in the first, and inside a joint graph for each candidate answer in the second. ## Variants ### Generated relation vectors fed into recurrent states For each utterance, a COMET encoder (trained on the ATOMIC if-then knowledge base) reads the utterance followed by one of five relation phrases. The encoder state after the last token is a 768-dimensional vector. The five relations are the speaker's intent, the effect on the speaker, the reaction of the speaker, the effect on the others and the reaction of the others [arxiv:2010.02795 §3.3] [arxiv:2010.02795 tab. 1]. Card: [COSMIC](/papers/2010.02795). The task model keeps a shared context state and, for each participant, an internal state, an external state, an intent state and an emotion state. Each state is updated by a GRU that receives the commonsense vector for the matching role [arxiv:2010.02795 §3.4] [arxiv:2010.02795 tab. 2]. The knowledge therefore enters at the state update, not at the input layer. The paper updates the listener states with the listener-side vectors [arxiv:2010.02795 §3.4]; the card's Paper vs code section records that the released code leaves them off by default, so reproduce the setting you intend. ### Generated relation vectors gated into a reply encoder For the last user utterance, a COMET model trained on ATOMIC-2020 generates five inferences for each of five relations: xReact, xWant, xNeed, xIntent and xEffect. The paper drops xAttr on purpose, because it judges the other person [arxiv:2109.05739 §3.2]. The inferences for each relation are concatenated into one sequence. The affective relation xReact goes through its own encoder and its states are averaged. The four cognitive relations share a second encoder, and their first state is kept [arxiv:2109.05739 §3.2] [eq. 4-5]. Each knowledge vector is concatenated to every token of the dialogue encoding, and the five refined streams are joined per token, passed through a sigmoid gate and mapped back to the model width by an MLP [arxiv:2109.05739 §3.3] [arxiv:2109.05739 §3.4] [eq. 13-15]. The decoder attends to this fused context [arxiv:2109.05739 §3.5]. Only the last utterance receives knowledge; earlier turns are encoded without it [arxiv:2109.05739 §3.2]. Card: [CEM](/papers/2109.05739). ### Retrieved subgraph fused with a language model QA-GNN takes the entities named in the question and the candidate answer, adds their two-hop neighbours from ConceptNet and keeps the 200 nodes that score highest for relevance, scored with the language model [arxiv:2104.06378 §3.2] [arxiv:2104.06378 §4.2]. It adds one node for the question and answer, linked to the question and answer entities, and runs five attention layers over the graph [arxiv:2104.06378 §3.1] [arxiv:2104.06378 §4.3]. Card: [QA-GNN](/papers/2104.06378). GreaseLM uses the same retrieval procedure [arxiv:2201.08860 app. B.1]. Its language model and graph network run side by side, with several fusion layers between them. In each fusion layer, one interaction token and one interaction node are the only states that pass between the two streams [arxiv:2201.08860 §4.3]. The number of fusion layers is 5 for CommonsenseQA, 6 for OpenbookQA and 3 for MedQA in the paper's table [arxiv:2201.08860 tab. 7]. Card: [GreaseLM](/papers/2201.08860). ## When to use which The choice depends mainly on the input shape, the knowledge available and the compute budget. Figures are author-reported and come from different papers and datasets, so they show what each design reported, not a like-for-like ranking. | Situation | Use | Why | |---|---|---| | Dialogue with a speaker per turn, text input, intents and reactions matter | Generated relation vectors into recurrent states | Full model 65.28 weighted F1 on IEMOCAP against 63.05 with both commonsense sides removed [arxiv:2010.02795 tab. 4]. The authors call the gain "not very substantial" [arxiv:2010.02795 §5.5] | | Empathetic reply generation, where the last user turn implies a goal or a feeling the words do not state | Generated relation vectors gated into a reply encoder | CEM reports PPL 36.11 against 36.93 for the best baseline (MoEL), and human raters prefer CEM over MoEL on informativeness, 61.0% against 30.6% [arxiv:2109.05739 tab. 2] [arxiv:2109.05739 tab. 3]. Human tests use 100 sampled pairs per comparison, three raters each [arxiv:2109.05739 §4.4] | | Dist-n diversity is a target, and the commonsense share must be known | Use CEM's setup, but measure the diversity loss separately | Without the diversity loss, CEM's Dist-2 falls from 2.99 to 1.96, below the best baseline's 2.10. Without either commonsense branch it stays at 2.52 or 2.47 [arxiv:2109.05739 tab. 2]. No row removes both, so the table does not isolate the commonsense share | | Keep only some COMET relations | Use the five from the paper | The authors tested four other relations and report no significant gain from them [arxiv:2010.02795 §5.5] | | Multiple-choice QA over ConceptNet, with a RoBERTa-large encoder | Retrieved subgraph; GreaseLM if the top score matters | IHtest 74.2 for GreaseLM against 73.4 for QA-GNN [arxiv:2201.08860 tab. 2] [arxiv:2104.06378 tab. 3]. The 0.8-point gap is small, and the two papers use different training setups | | Gain over the plain language model is the question | Retrieved subgraph | QA-GNN reports +4.7 over RoBERTa-large on CommonsenseQA in-house test [arxiv:2104.06378 §4.5] | | Negated questions | Retrieved subgraph | QA-GNN reports 58.8 on negated IHtest questions, against 54.2 for RoBERTa [arxiv:2104.06378 tab. 10]. GreaseLM reports 69.9 on negated IHdev questions, against 66.2 for QA-GNN; the splits differ, so the numbers do not compare [arxiv:2201.08860 tab. 5] | | Medical multiple-choice QA on a UMLS and DrugBank graph | Retrieved subgraph with SapBERT | MedQA-USMLE test: GreaseLM 38.5, QA-GNN 38.0, SapBERT without a graph 37.2 [arxiv:2201.08860 tab. 6] [arxiv:2104.06378 tab. 7]. The margins are under one point and no standard deviations are given | | Compute or latency is tight | Budget per candidate answer | GreaseLM's input is question and candidate answer together, so each candidate gets its own pass [arxiv:2201.08860 §4.1]. QA-GNN reports about 20 hours on two RTX 2080 Ti for CommonsenseQA [arxiv:2104.06378 §4.3] | | The question-context node is part of the design | Keep the node and its edges to the question and answer entities | QA-GNN's ablation without edges from the context node to the question entities drops IHdev from 76.5 to 74.8 [arxiv:2104.06378 §4.6.1] | | Joint text-graph reasoning, not one vector from the LM | GreaseLM-style fusion | Removing the interaction between streams drops IHdev from 78.5 to 76.5 [arxiv:2201.08860 tab. 8] | | No graph or commonsense model is available | Neither variant applies as published | Both need a knowledge source; the baseline rows show what the same LM reaches without one [arxiv:2010.02795 tab. 4] [arxiv:2201.08860 tab. 2] | ## Implementation sketch The common core is a lookup of knowledge for one input, an encoder for the text, and a fusion step that uses both. The variants differ in what the lookup returns and where the fusion happens. Details are in the cards. ```pseudocode # Common core: knowledge for one input, then a fused prediction def predict_with_knowledge(text, candidates): # Generated vectors (COSMIC, CEM) or a retrieved subgraph (QA-GNN, GreaseLM) # [arxiv:2010.02795 §3.3] [arxiv:2109.05739 §3.2] [arxiv:2104.06378 §4.2] knowledge = get_knowledge(text) # Pretrained encoder for the text [arxiv:2010.02795 §3.2] [arxiv:2201.08860 §4.2] text_vec = encoder(text) # Graph variants score each candidate separately [arxiv:2201.08860 §4.1] scores = [] for a in candidates: # Fuse text and knowledge: state updates, gated encoder or joint graph layers # [arxiv:2010.02795 §3.4] [arxiv:2109.05739 §3.4] [arxiv:2201.08860 §4.3] fused = fuse(text_vec, knowledge, a) scores.append(head(fused)) return scores ``` ## Contradictions No two cards report conflicting numbers for the same setting. The shared baselines agree: RoBERTa-large reaches 68.7 on the CommonsenseQA in-house test in both QA-GNN and GreaseLM [arxiv:2104.06378 tab. 3] [arxiv:2201.08860 tab. 2], and the SapBERT-with-QA-GNN MedQA result of 38.0 appears in both papers [arxiv:2104.06378 tab. 7] [arxiv:2201.08860 tab. 6]. The size of the gain is reported very differently. COSMIC's commonsense component adds about 2 points on IEMOCAP [arxiv:2010.02795 tab. 4], while QA-GNN reports +4.7 over RoBERTa on CommonsenseQA [arxiv:2104.06378 §4.5]. The tasks differ, so these do not contradict each other, but a reader should not read them as one scale. Inside GreaseLM, the text gives a gain of 0.9 over the LM+KG baselines on CommonsenseQA test, and the table gives 0.8 against QA-GNN [arxiv:2201.08860 §6] [arxiv:2201.08860 tab. 2]. This page uses the table value. COSMIC's Table 4 caption gives both five runs and ten runs for some rows, and the paper does not resolve this [arxiv:2010.02795 tab. 4]. CEM's text says it "achieves the lowest perplexity" [arxiv:2109.05739 §4.3], but its Table 2 lists the "w/o Div" ablation lower, at 35.60 against 36.11 [arxiv:2109.05739 tab. 2]. This page uses the table values. ## Papers - [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](/papers/2010.02795): adds five COMET relation vectors per utterance to a recurrent emotion model and reports gains on four dialogue datasets. - [CEM: Commonsense-aware Empathetic Response Generation](/papers/2109.05739): adds five COMET relation vectors for the last user turn to a Transformer reply model through a gated fusion, with a diversity loss, and reports gains over five baselines on EmpatheticDialogues. - [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](/papers/2104.06378): joins a pretrained LM with a retrieved ConceptNet subgraph, scored and reasoned over by a GNN with a question-context node. - [GreaseLM: Graph REASoning Enhanced Language Models for Question Answering](/papers/2201.08860): runs an LM and a GNN side by side and exchanges information through an interaction token and node in several fusion layers.