Relation extraction

Overview

Relation extraction (RE) takes a text in which entity mentions are already known, or are found first, and decides which pairs of mentions stand in a relation and what kind of relation it is. A system needs labelled examples of the relation types, and a benchmark needs to say whether the gold entities are given or must be found by the system 2204.04263 §4.2.

Two papers in this wiki sit on different sides of the problem. One builds the labelled corpus that a system would be trained or scored on 2204.04263 §3. The other asks how a few-shot RE or information extraction system should combine a small supervised model with a large language model 2303.08559 §4, §5. Both must make the same basic choice: which spans or pairs get labels, and from where.

Variants

Multi-type biomedical corpus with novelty labels

BioRED labels 600 PubMed abstracts with six entity types linked to database identifiers, plus 6,503 document-level relations of eight types between eight concept pairs 2204.04263 tab. 4 2204.04263 §3.1. Each relation also carries a novel-or-background label, which earlier corpora lacked 2204.04263 §1. Card: BioRED: A Rich Biomedical Relation Extraction Dataset.

The benchmark gives the gold concepts and asks for the relation. Scores are therefore not end-to-end. For a PubMedBERT classifier the reported F-score is 72.9 for the concept pair alone, 58.9 when the relation type is added, and 47.7 when the novelty label is added as well 2204.04263 tab. 6. The authors call the novelty part the open problem 2204.04263 §1 2204.04263 §4.4.

Small model as filter, LLM as reranker

The paper first finds that prompted LLMs are not, in general, strong few-shot extractors. In nine NER, RE, event-detection and event-argument settings they trail fine-tuned small models in most cases, and they cost more and run slower 2303.08559 §3.4 2303.08559 tab. 1. On the RE dataset TACREV, in-context prompting with CODEX reaches 59.1, 60.3 and 62.4 F1 at 20, 50 and 100 shots, against a small-model state of the art of 62.4, 68.5 and 72.6 2303.08559 tab. 3.

The proposed method is adaptive filter-then-rerank. The small model scores every candidate span. If its top probability is above a threshold, its label stands. Otherwise the top three labels plus None go to an LLM as a multiple-choice question, and the LLM answers after a short explanation 2303.08559 §5.1 2303.08559 §5.2. On TACREV, InstructGPT reranking reaches 66.8, 72.3 and 75.4 F1 at the same shot counts 2303.08559 tab. 3. The paper reports an average gain of about 2.4 F1 over three datasets and 80 to 90 percent lower budget and latency than direct prompting, both author-reported 2303.08559 §5.4 2303.08559 fig. 7. Card: Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!.

When to use which

Implementation sketch

Hybrid filter-then-rerank for one candidate pair, in the style of the paper's method. The threshold and the reranker prompt are the choices to fix yourself: the paper does not state the threshold values it used 2303.08559 tab. 3.

# Filter-then-rerank for one candidate entity pair  [arxiv:2303.08559 §5.1]
# Input:  sentence, pair, small_model, llm, demos, tau, top_n
# Output: a relation label, possibly None

def classify_pair(sentence, pair, small_model, llm, demos, tau, top_n):
    # 1. The supervised model gives a probability per relation label  [§4.2]
    probs = small_model.label_probabilities(sentence, pair)
    # 2. Confidence is the largest probability  [§4.2 eq. 2]
    confidence = max(probs.values())
    # 3. Confident pairs keep the small model's answer  [§5.1]
    if confidence >= tau:
        return max(probs, key=probs.get)
    # 4. Uncertain pairs: keep the top-N labels in model order  [§5.2]
    ranked = sorted(probs, key=probs.get, reverse=True)
    options = ranked[:top_n]
    # 5. Always offer None when it is not already among the options  [§5.2]
    if None not in options:
        options.append(None)
    # 6. Ask the LLM to choose one option, with an explanation first  [§5.2]
    reply = llm.choose(demos, sentence, pair, options, explain=True)
    # 7. Unparseable replies fall back to the model's top label. Count these:
    #    the paper does not discuss parse failures  [code: reranker/re_reranker.py#L120-L123]
    return parse_choice(reply, options, fallback=ranked[0])

For scoring a document-level dataset such as BioRED, match predicted relations to gold ones using a key that depends on the schema: the pair alone, the pair plus type, or the pair plus type plus novelty. Relation types are non-directional, so sort the pair first 2204.04263 §3.1 2204.04263 §4.2.

Contradictions

None known between the two papers. They measure different things. The BioRED scores assume gold entities, on biomedical abstracts. The filter-then-rerank scores are F1 on sentence-level NER, RE and event sets, with the LLM given the small model's candidates. Do not put the two sets of numbers in one table without saying this 2204.04263 §4.2 2303.08559 §3.1.

Within a paper, the authors' own qualifications should be kept with the headline numbers. The BioRED single-type versus multi-type result does not hold for every type: single-type models win on disease and chemical 2204.04263 tab. 7. The LLM reranker's gain is not uniform: GPT-4 and InstructGPT move the hard-subset F1 in different directions on FewNERD, where InstructGPT lowers it by 3.1 points 2303.08559 tab. 4.

Papers

• BioRED: A Rich Biomedical Relation Extraction Dataset: a multi-type, document-level biomedical RE corpus with novelty labels and benchmarks.
• Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!: shows prompted LLMs trail small models in few-shot IE, and proposes an adaptive filter-then-rerank over the small model's top-3 labels.