wikillm

Paper · 2024-04-05 · cs.CL, cs.AI, cs.LG

Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction

Builds a knowledge graph from text in three prompted LLM steps: open triple extraction, a one-line definition per relation, then mapping each relation onto a target or self-built schema. A fine-tuned retriever adds schema relations for a second round. Author-reported: WebNLG Partial F1 rises from 0.767 (REGEN) to 0.820 (GPT-4, refined).

Relation extractionIn-context learningText embeddingRetrieval-augmented generationknowledge-graphrelation-extractionschema-canonicalizationinformation-extractionretrieval-augmented-generation

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A knowledge graph stores facts as triples, such as "this person, was born on, that date". Asking a language model for triples works well when you hand it the list of allowed relation names. Real collections have far too many relation types for that list to fit in the prompt, and sometimes no list exists at all.

The method splits the job into three questions the model answers one at a time. First it writes down whatever facts it sees, using its own words for the relations. Second it writes a short definition for each of those relation names. Third it matches each relation to the closest entry of a target list, or, when there is no list, it builds a short list from the names that come up. A second pass can hand the model a few candidate relation names from a search index, so it notices relations it would otherwise miss. The language model itself is never trained; the only trained part is that search index.

TL;DR

The Extract-Define-Canonicalize (EDC) pipeline prompts an LLM to extract open triples, define each relation in words, and canonicalize relations against a given schema or a self-built one. An optional refinement round feeds candidate entities and schema relations from a fine-tuned retriever back into extraction. The authors report higher Partial F1 than the trained baselines REGEN and GenIE on three benchmarks, with the largest gains on REBEL and Wiki-NRE. Relevant if you need relation-level KG extraction with large or absent schemas and can afford several LLM calls per input.

Problem

Prompt-based knowledge graph construction puts the schema into the prompt so the model only emits valid relations. Large schemas, such as Wikipedia-scale relation sets, exceed the context window or are partly ignored by the model §1. Many settings also have no fixed schema to begin with §1. The trained baselines are each built for one dataset §4.1. Automatic scores are also biased: the reference triples in REBEL and Wiki-NRE are incomplete, so correct extra triples are counted as errors §4.2.1.

Method

EDC runs three LLM-driven phases on each input text, followed by an optional refinement round §1 §3.1 §3.2. Only relations are canonicalized; entity and event types are left for future work §4.1 §3.2.

  1. Open information extraction. A few-shot prompt, with six worked examples per dataset, asks the LLM for triples of subject, relation and object. No schema is in the prompt §3.1 app. A.2. The worked example in §3.1 yields two triples for a sentence about an astronaut.

  2. Schema definition. A second few-shot prompt receives the text and the open triples, and returns a one-line definition for each relation that occurs. The definitions are kept as side information for step 3 §3.1.

  3. Schema canonicalization. Each relation definition is embedded with E5-Mistral-7B-instruct, used without modification §4.1. For each open triple, the 5 nearest schema entries by cosine similarity are retrieved app. A.2. The LLM sees the triple, the definition and these candidates as lettered options, plus a final option meaning none fits §3.1. If it picks a candidate, the relation is replaced; otherwise the handling depends on the setting below §3.1.

    • Target alignment (EDC-TA). With a given schema, a triple whose relation has no acceptable match is dropped §3.1.
    • Self-canonicalization (EDC-SC). Starting from an empty schema, an unmatched relation is added to the schema with its definition. The schema therefore grows while the triples are processed §3.1.
  4. Refinement (EDC+R). Refinement repeats the three phases, but the extraction prompt also receives a hint §1 §3.2:

    • Candidate entities: the entities from the previous round, merged with a fresh entity extraction by an LLM app. A.4.1.
    • Candidate relations: the relations from the previous round, plus the 10 schema relations the retriever ranks highest for the text §3.2 app. A.2 app. A.4.2. Each candidate carries its definition app. A.4.3.
    • One refinement round is reported in the main results. A second round gives a small further gain tab. 6 app. C.2.
  5. Schema retriever. A fine-tuned E5-Mistral-7B-instruct embeds the input text (with a fixed retrieval instruction prepended) and each relation definition. Relevance is cosine similarity. Training uses positive text and relation pairs and an InfoNCE objective with cosine similarity §3.2 app. A.3:

    \min \mathcal{L} = -\log \frac{\phi(t^+_{inst}, r^+)}{\phi(t^+_{inst}, r^+) + \sum_{n_i \in \mathbb{N}} \phi(t^+_{inst}, n_i)} app. A.3

    The training pairs are synthesised from TEKGEN, where Wikidata triples are aligned to Wikipedia sentences. The authors use 37,500 pairs, split evenly between positive and negative samples app. A.3.

The pseudocode below follows the method in order. It is a reading of the paper's description, not a copy of the released code.

Pseudocode
# EDC for one input text: extract, define, canonicalize  [§3.1]
# Input:  text (str); schema (dict: relation -> definition), may be empty
# Output: list of [subject, relation, object] triples

def edc(text, schema):
    # Phase 1: few-shot open extraction with no schema in the prompt  [§3.1]
    open_triples = llm_extract(text, six_examples)

    # Phase 2: one short definition per relation in the open triples  [§3.1]
    definitions = llm_define(text, open_triples)

    # Phase 3: map each open relation onto the schema  [§3.1]
    canonical = []
    for triple in open_triples:
        result = canonicalize(triple, definitions, schema)
        # Unmatched triples are dropped when a target schema is given,
        # and added to the schema when it is built from scratch  [§3.1]
        if result is not None:
            canonical.append(result)
    return canonical
Pseudocode
# Map one open relation onto the schema  [§3.1] [app. A.2]
# Input:  triple [s, r, o]; definitions: relation -> definition text
# Output: triple with a schema relation, or None if nothing fits

def canonicalize(triple, definitions, schema):
    relation = triple[1]
    definition = definitions[relation]
    # Embed the definition and rank schema entries by cosine similarity
    # [§4.1]; keep the 5 nearest  [app. A.2]
    candidates = top_k_by_similarity(schema, definition, k=5)
    # The LLM picks one candidate, or the final option "none"  [§3.1]
    choice = llm_choose(triple, definition, candidates)
    if choice is None:
        return None
    return [triple[0], choice, triple[2]]
Pseudocode
# One refinement round: hint from the last round, then extract again  [§3.2]
# Input:  text; last_triples (from the previous round); retriever
# Output: new open triples, which then go through the phases above

def refined_extract(text, last_triples, retriever, schema):
    # Candidate entities: last-round entities plus a fresh entity pass,
    # merged by the LLM  [app. A.4.1]
    entities = merge_entities(last_entities(last_triples),
                              llm_entities(text))
    # Candidate relations: last-round relations plus the 10 schema
    # relations ranked highest by the retriever  [§3.2] [app. A.2]
    relations = last_relations(last_triples)
    relations += retriever.top_k(text, k=10)
    # Put the hint and each relation's definition in the prompt; the model
    # may still use other relations  [app. A.4.3]
    return llm_extract_with_hint(text, entities, relations, schema)

Concrete order of work per input, from the paper's description: one open extraction call, one definition call, and one verification call per open triple, plus the refinement calls when enabled §3.1 §3.2 app. A.4. Reported token use for the definition and extraction steps fell from about 3k to about 2k tokens per example when the two were merged into one call, a variant the authors tested only on REBEL with GPT-3.5-turbo app. G.

Hyperparameters

NameValueWhereNotes
few-shot examples per prompt6app. A.2chosen empirically; the released prompt files also hold six examples per dataset oie_few_shot_examples.txt
canonicalization candidatestop-5app. A.2default in schema_canonicalization.py
refinement relations from retrievertop-10app. A.2 §3.2retriever default in schema_retriever.py; called without an argument at edc_framework.py
refinement rounds1 (main results)tab. 6 app. C.2a second round adds 0.003 to 0.005 Partial F1 on the three datasets with GPT-3.5
LLMs for open extractionGPT-4, GPT-3.5-turbo, Mistral-7B§4.1 app. A.1GPT sizes not stated; Mistral-7B run locally
LLMs for definition and canonicalizationGPT-3.5-turbo§4.1the main text names GPT-3.5-turbo for these modules
embedder for canonicalizationE5-Mistral-7B-instruct, unmodified§4.1
embedder for retrieverE5-Mistral-7B-instruct, fine-tuned§3.2 app. A.3fine-tuned weights not stated as released
retriever training pairs37,500, half positiveapp. A.3source: TEKGEN
retriever recall@10 (test)WebNLG 0.823; REBEL 0.663; Wiki-NRE 0.818app. A.3retriever quality only, not end-to-end
retriever lossInfoNCE on cosine similarityapp. A.3equation above
retriever optimiser, batch, epochsnot statedapp. A.3the released training script has defaults, see Paper vs code
human annotators2, agreement 0.94§4.2.2used for self-canonicalization only
compute4 × NVIDIA RTX A6000 (48 GB), 252 GB RAMapp. A.1for the open-weight models
temperature, decoding for LLMsnot statedapp. A.1see Paper vs code for the released defaults

Results

Author-reported. Setup: three benchmarks (WebNLG v3.0 semantic-parsing test with 1,165 pairs and 159 relation types; 1,000 sampled pairs each from REBEL, with 200 relation types, and Wiki-NRE, with 45 types) §4.1. Metric: Partial F1, from the WebNLG evaluation script; the Strict and Exact variants are in tab. 3, tab. 4 and tab. 5. Baselines: REGEN for WebNLG; GenIE for REBEL and Wiki-NRE §4.1. fig. 2 shows the same Partial F1 values as bar charts.

DatasetMethod (LLM for open extraction: GPT-4 / GPT-3.5 / Mistral-7B)Partial F1Baseline Partial F1Source
WebNLGEDC: 0.783 / 0.746 / 0.7280.767 (REGEN)tab. 3
WebNLGEDC+R: 0.820 / 0.794 / 0.7620.767 (REGEN)tab. 3
REBEL (1,000 sampled)EDC: 0.546 / 0.506 / 0.5160.385 (GenIE)tab. 4
REBEL (1,000 sampled)EDC+R: 0.601 / 0.559 / 0.5310.385 (GenIE)tab. 4
Wiki-NRE (1,000 sampled)EDC: 0.683 / 0.647 / 0.6470.484 (GenIE)tab. 5
Wiki-NRE (1,000 sampled)EDC+R: 0.713 / 0.693 / 0.6630.484 (GenIE)tab. 5

Ablation on the retriever, with GPT-3.5-turbo for open extraction tab. 1: removing the retrieved relations from the hint lowers Partial F1 by about 0.04 on each dataset (WebNLG 0.794 to 0.752; REBEL 0.559 to 0.517; Wiki-NRE 0.693 to 0.653). Refinement without retrieval still beats the unrefined EDC (0.746, 0.506 and 0.647).

Novel dataset of 50 fictional entries, Wiki-NRE schema, GPT-3.5-turbo: Partial F1 0.751 for EDC, 0.767 for EDC+R and 0.530 for GenIE tab. 9 app. D.

Small LLM-based comparison, GPT-3.5-turbo for all methods, Partial F1 tab. 10: CoNLL04 EDC 0.543, EDC+R 0.585, CodeKGC 0.545, ChatIE 0.468. SciERC EDC 0.395, EDC+R 0.451, CodeKGC 0.392, ChatIE 0.357. Wiki-NRE EDC 0.647, EDC+R 0.693, CodeKGC 0.612, ChatIE 0.571. The authors note that EDC without refinement does not beat the baselines on the two small-schema sets app. E.

Self-canonicalization, manual evaluation of GPT-3.5-turbo output tab. 2. Here "No. Rel." counts distinct relations and "Red." is the redundancy score, where lower is better for both:

DatasetMethodPrecisionNo. Rel.Redundancy
WebNLGEDC0.9562000.833
WebNLGCESI0.7242800.893
WebNLGopen triples0.9825290.927
REBELEDC0.8672250.831
REBELCESI0.5043070.854
REBELopen triples0.9036670.895
Wiki-NREEDC0.8981060.833
Wiki-NRECESI0.7531140.849
Wiki-NREopen triples0.9092040.881

Number of triples per sentence tab. 8: the references average 4.00 on REBEL and 2.86 on Wiki-NRE; GPT-4-based EDC averages 5.11 and 3.49. The authors use this to explain why the automatic scores are pessimistic app. C.4.

Paper vs code

Code: official repository https://github.com/clear-nus/edc, at commit 02aad17e2fc1e9845c84746b384e7fd3160edc1a, MIT license LICENSE. The paper links this repository §4.1 app. A.1; its license is held by "CLeAR", and both authors are at the National University of Singapore LICENSE; the title page lists both authors at the National University of Singapore. The card compares the arXiv v2 text (updated 2024-10-02) with that commit.

  • The trained schema retriever is not used by the released run path. run.py reads --sr_adapter_path, but EDC stores it and never loads it run.py edc_framework.py. The retriever is built from the base --sr_embedder, which defaults to the untuned E5-Mistral-7B-instruct, with finetuned_e5mistral=False run.py edc_framework.py. No adapter weights are in the repository. The retrieval step therefore reproduces only if the user supplies weights; the paper does not say whether the weights are released.
  • Instruction text differs. The paper's training template is a short retrieval instruction on the text side only §3.2. The code uses a different instruction, which asks for descriptions of the relations a text expresses, and applies it to the query text and to each relation definition schema_retriever.py schema_retriever.py.
  • Entity candidates use an extra LLM merge. The paper says the two entity sources are merged app. A.4.1. The code calls the LLM again to merge them into one list entity_extraction.py edc_framework.py.
  • Relation examples are added to the hint. The code appends, for each candidate relation, one example triple and sentence drawn at random from all earlier extractions in the same run. The paper describes no such example edc_framework.py edc_framework.py. No seed is set on the inference path (the only seed is in the training script), so repeated runs can differ.
  • Previous-round relations are filtered. Relations from the last round are kept only if they are in the schema edc_framework.py. The paper does not state this.
  • Known schema relations skip canonicalization. A triple whose relation is already in the schema is returned unchanged schema_canonicalization.py.
  • Self-canonicalization is order-dependent. One schema dictionary is shared across all inputs in a run and grows as relations are added edc_framework.py schema_canonicalization.py. The paper's description does not mention ordering.
  • Retriever training defaults are in the repository. The paper says the authors used an online implementation and hyperparameters app. A.3. The repository's training script sets maximum length 128, batch size 8, learning rate 5e-5, 3 epochs, a linear schedule and no warm-up e5_mistral_utils.py. The contrastive loss in the training loop uses temperature 0.02 with paired negatives e5_mistral_utils.py. Whether these match the authors' training run is not stated.
  • Training set size. The dataset builder defaults to 50,000 pairs and keeps at most three pairs per sentence collect_schema_retrieval_data.py collect_schema_retrieval_data.py. The paper reports 37,500.
  • Decoding settings are in the code. The paper does not state decoding. OpenAI calls use temperature 0 and at most 512 output tokens by default llm_utils.py. Local models use greedy decoding with at most 256 new tokens llm_utils.py. The canonicalization verifier is limited to one output token for OpenAI and five for local models schema_canonicalization.py. Embeddings are truncated at 4096 tokens llm_utils.py.
  • Entity extraction with an OpenAI model uses the wrong model name. When the entity model is an OpenAI model, the code instantiates the extractor with the schema-definition model's name edc_framework.py. The two are the same in the README commands but differ in general.
  • No discrepancies found for the OIE and definition prompts' few-shot count (six each) and for the candidate counts (5 and 10).

Pitfalls

  • Retriever weights. Without the fine-tuned adapter, the refinement step uses the base embedder with a different instruction (see above). It is then not the retriever the paper evaluates, so its EDC+R numbers do not transfer directly. The paper's retriever recall values app. A.3 describe the fine-tuned model only.
  • Crash when the open-extraction model is an OpenAI model not used elsewhere. This is read from the code, not run (no API key or GPU here). In oie(), when free_model is true, the code calls free_model(oie_model, ...), but oie_model is only assigned in the open-weights branch edc_framework.py. extract_kg sets free_model true on the last iteration whenever the open-extraction model name is not among the other module models edc_framework.py. The README command with --oie_llm gpt-4 and the other modules on gpt-3.5-turbo therefore reaches that call after the full extraction pass, which would raise UnboundLocalError. Using the same model name for all LLM modules avoids it.
  • Crash when enrichment meets a missing definition. When the definition output omits a relation and enrichment is on, the code reads the definition dictionary with that relation as the key, which raises KeyError schema_canonicalization.py schema_canonicalization.py. The definition parser drops lines without a colon, so this can happen on unusual outputs.
  • Enriched relations are embedded differently. Schema entries are embedded without a prompt schema_canonicalization.py, but relations added during enrichment are embedded with the query prompt, when the embedder defines one schema_canonicalization.py. Retrieval over the grown schema therefore mixes two encodings.
  • Scores depend on the metric. The authors say REBEL and Wiki-NRE references are incomplete, so correct extra triples are penalised under Partial F1 §4.2.1 app. C.4. Compare Strict and Exact tab. 4 tab. 5 before drawing conclusions.
  • Wiki-NRE numbers disagree across tables. The Exact F1 of EDC+R on Wiki-NRE is 0.687 in tab. 5 and tab. 10 but 0.657 in tab. 1 and tab. 6. For plain EDC, the Partial, Strict and Exact values in tab. 6 (0.644, 0.634, 0.637) do not match tab. 5 (0.647, 0.638, 0.640). The paper does not explain this.
  • Manual evaluation is small and under-described. Self-canonicalization was judged by two annotators with agreement 0.94 §4.2.2. The number of sampled triples and the embedding used for the redundancy score are not stated.
  • Cost grows with every triple. Canonicalization calls the LLM once per open triple, and refinement adds an entity pass, a merge and an extraction pass per round. Token counts are reported only for one variant app. G.
  • Relation-only scope. Only relations are canonicalized, so entity types and entity names are not normalised §4.1 §3.2. Combine with coreference and entity canonicalization if you need them app. F.
  • Small-schema sets. On CoNLL04, GPT-3.5-turbo with EDC alone scores 0.543 against CodeKGC's 0.545, so the pipeline's advantage depends on schema size tab. 10 app. E.

Lineage

  • Builds on: retrieval-augmented generation for the refinement hint §1 §3.2; open information extraction with LLMs, cited to the prior evaluations in §1; E5-Mistral-7B-instruct as the embedder §3.2.
  • Compared against: REGEN (T5-based text-to-graph) and GenIE (BART with constrained generation) §4.1; CESI for canonicalization §4.1; CodeKGC and ChatIE on small schemas app. E.
  • Superseded by: not stated.

Source and license

Adapted from “Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction” by Bowen Zhang, Harold Soh (arXiv:2404.03868), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/clear-nus/edc at commit 02aad17e2fc1 (MIT). Any quoted code is shown under that license.