--- kind: concept slug: relation-extraction title: Relation extraction summary: >- Finding which pairs of entity mentions in a text are linked, and by which relation type. Two variants are covered: a multi-type biomedical corpus with document-level and novelty labels, and a hybrid where a small supervised model filters easy cases and an LLM reranks hard ones. Choice depends on label budget, cost and whether the task is biomedical. tags: [information-extraction, relation-extraction, few-shot-learning, dataset, llm-reranking] papers: ["2204.04263", "2303.08559"] related: [named-entity-recognition, dataset-curation, inter-annotator-agreement, in-context-learning, few-shot-learning, chain-of-thought-prompting] status: draft updated_at: 2026-11-06 --- ## Overview Relation extraction (RE) takes a text in which entity mentions are already known, or are found first, and decides which pairs of mentions stand in a relation and what kind of relation it is. A system needs labelled examples of the relation types, and a benchmark needs to say whether the gold entities are given or must be found by the system [arxiv:2204.04263 §4.2]. Two papers in this wiki sit on different sides of the problem. One builds the labelled corpus that a system would be trained or scored on [arxiv:2204.04263 §3]. The other asks how a few-shot RE or information extraction system should combine a small supervised model with a large language model [arxiv:2303.08559 §4, §5]. Both must make the same basic choice: which spans or pairs get labels, and from where. ## Variants ### Multi-type biomedical corpus with novelty labels BioRED labels 600 PubMed abstracts with six entity types linked to database identifiers, plus 6,503 document-level relations of eight types between eight concept pairs [arxiv:2204.04263 tab. 4] [arxiv:2204.04263 §3.1]. Each relation also carries a novel-or-background label, which earlier corpora lacked [arxiv:2204.04263 §1]. Card: [BioRED: A Rich Biomedical Relation Extraction Dataset](/papers/2204.04263). The benchmark gives the gold concepts and asks for the relation. Scores are therefore not end-to-end. For a PubMedBERT classifier the reported F-score is 72.9 for the concept pair alone, 58.9 when the relation type is added, and 47.7 when the novelty label is added as well [arxiv:2204.04263 tab. 6]. The authors call the novelty part the open problem [arxiv:2204.04263 §1] [arxiv:2204.04263 §4.4]. ### Small model as filter, LLM as reranker The paper first finds that prompted LLMs are not, in general, strong few-shot extractors. In nine NER, RE, event-detection and event-argument settings they trail fine-tuned small models in most cases, and they cost more and run slower [arxiv:2303.08559 §3.4] [arxiv:2303.08559 tab. 1]. On the RE dataset TACREV, in-context prompting with CODEX reaches 59.1, 60.3 and 62.4 F1 at 20, 50 and 100 shots, against a small-model state of the art of 62.4, 68.5 and 72.6 [arxiv:2303.08559 tab. 3]. The proposed method is adaptive filter-then-rerank. The small model scores every candidate span. If its top probability is above a threshold, its label stands. Otherwise the top three labels plus None go to an LLM as a multiple-choice question, and the LLM answers after a short explanation [arxiv:2303.08559 §5.1] [arxiv:2303.08559 §5.2]. On TACREV, InstructGPT reranking reaches 66.8, 72.3 and 75.4 F1 at the same shot counts [arxiv:2303.08559 tab. 3]. The paper reports an average gain of about 2.4 F1 over three datasets and 80 to 90 percent lower budget and latency than direct prompting, both author-reported [arxiv:2303.08559 §5.4] [arxiv:2303.08559 fig. 7]. Card: [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](/papers/2303.08559). ## When to use which | Situation | Use | Why | |---|---|---| | Need many relation types across sentences, in biomedical text, with novelty labels | BioRED-style corpus as training or evaluation data | It is the only one of the two papers that labels these [arxiv:2204.04263 tab. 4]. Check the licence of the data itself before redistributing it [arxiv:2204.04263 §5]. | | Need an end-to-end biomedical system | BioRED for NER, then the RE benchmark on predicted entities | The RE scores assume gold entities, so they are not end-to-end scores. The paper does not measure RE on predicted entities [arxiv:2204.04263 §4.2]. | | Very few labelled examples and LLM access is acceptable | Filter-then-rerank over a small model you already have | Direct prompting trails the small model on TACREV at 20 to 100 shots [arxiv:2303.08559 tab. 3]. The hybrid recovers the gain on hard cases only. | | Tight latency or cost budget | Filter-then-rerank, not an LLM on every span | The LLM sees only uncertain spans. The paper's cost figures are from one API setup and come from a plot [arxiv:2303.08559 §5.4]. | | The small model's probabilities are poorly calibrated | Do not rely on the filter; fix calibration first or use direct prompting with a larger budget | The method assumes calibrated confidence, which the authors call an imperfect assumption [arxiv:2303.08559 Limitations]. | | Labels are plentiful and the fine-tuned model is strong | A fine-tuned small model alone | The paper's LLM advantage is mainly at very few samples and labels; the small-model curves rise more steeply with more data [arxiv:2303.08559 §3.4] [arxiv:2303.08559 fig. 2]. | ## Implementation sketch Hybrid filter-then-rerank for one candidate pair, in the style of the paper's method. The threshold and the reranker prompt are the choices to fix yourself: the paper does not state the threshold values it used [arxiv:2303.08559 tab. 3]. ```pseudocode # Filter-then-rerank for one candidate entity pair [arxiv:2303.08559 §5.1] # Input: sentence, pair, small_model, llm, demos, tau, top_n # Output: a relation label, possibly None def classify_pair(sentence, pair, small_model, llm, demos, tau, top_n): # 1. The supervised model gives a probability per relation label [§4.2] probs = small_model.label_probabilities(sentence, pair) # 2. Confidence is the largest probability [§4.2 eq. 2] confidence = max(probs.values()) # 3. Confident pairs keep the small model's answer [§5.1] if confidence >= tau: return max(probs, key=probs.get) # 4. Uncertain pairs: keep the top-N labels in model order [§5.2] ranked = sorted(probs, key=probs.get, reverse=True) options = ranked[:top_n] # 5. Always offer None when it is not already among the options [§5.2] if None not in options: options.append(None) # 6. Ask the LLM to choose one option, with an explanation first [§5.2] reply = llm.choose(demos, sentence, pair, options, explain=True) # 7. Unparseable replies fall back to the model's top label. Count these: # the paper does not discuss parse failures [code: reranker/re_reranker.py#L120-L123] return parse_choice(reply, options, fallback=ranked[0]) ``` For scoring a document-level dataset such as BioRED, match predicted relations to gold ones using a key that depends on the schema: the pair alone, the pair plus type, or the pair plus type plus novelty. Relation types are non-directional, so sort the pair first [arxiv:2204.04263 §3.1] [arxiv:2204.04263 §4.2]. ## Contradictions None known between the two papers. They measure different things. The BioRED scores assume gold entities, on biomedical abstracts. The filter-then-rerank scores are F1 on sentence-level NER, RE and event sets, with the LLM given the small model's candidates. Do not put the two sets of numbers in one table without saying this [arxiv:2204.04263 §4.2] [arxiv:2303.08559 §3.1]. Within a paper, the authors' own qualifications should be kept with the headline numbers. The BioRED single-type versus multi-type result does not hold for every type: single-type models win on disease and chemical [arxiv:2204.04263 tab. 7]. The LLM reranker's gain is not uniform: GPT-4 and InstructGPT move the hard-subset F1 in different directions on FewNERD, where InstructGPT lowers it by 3.1 points [arxiv:2303.08559 tab. 4]. ## Papers - [BioRED: A Rich Biomedical Relation Extraction Dataset](/papers/2204.04263): a multi-type, document-level biomedical RE corpus with novelty labels and benchmarks. - [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](/papers/2303.08559): shows prompted LLMs trail small models in few-shot IE, and proposes an adaptive filter-then-rerank over the small model's top-3 labels.