Revisiting Relation Extraction in the era of Large Language Models

In plain words

Relation extraction means reading text and listing the facts it states, such as "this person works for that organisation". This paper asks how far very large language models get at this when they are simply shown a few worked examples in the prompt and asked to write the facts out as text.

The answer is: close to the best specialised systems. The catch is grading. A model often writes a correct fact in different words, and a strict string-matching grader counts that as an error. The authors therefore had people re-check the disputed outputs, and many of them turned out to be right.

The paper also trains a much smaller, openly available model (Flan-T5) on short explanations that the large model wrote for each training example. The smaller model then outperforms the other systems in the comparison. The authors present it as a new baseline for the task.

TL;DR

• Relation extraction is cast as generating a linearized list of triplets from the input text §2.
• Few-shot GPT-3 comes close to fully supervised models, but only after human re-judging of exact-match errors §3.2 §3.3.
• Fine-tuning Flan-T5 Large on GPT-3-generated chain-of-thought explanations is reported as SOTA on CoNLL04, ADE and NYT tab. 2.
• All numbers are author-reported. No official code was found, and the card's pitfalls list several internal inconsistencies in the paper.

Problem

Standard supervised relation extraction tags entity spans and then classifies the relation between them §1. Recent work reaches near state-of-the-art by training a sequence-to-sequence model to output linearized triplets, but with moderately sized models such as BART §1. This paper asks whether much larger language models (GPT-3 and Flan-T5 Large) can do the task with few or no training examples §1 §2.

Generative models make evaluation hard. Their outputs can be correct but differ from the reference in wording, order or formatting, so exact matching of generated tuples underestimates performance §2. The authors report that the false positives and false negatives of exact matching are often not errors on inspection §3.2.

Without a grammar or an in-context format, few-shot outputs also do not follow the linearization scheme, so they cannot be parsed §3.3 §4.1.

Method

Task formulation. The input is a text. The output is a string that encodes the relation triplets. For one relation type (ADE) the output is a list of (drug, effect) tuples. For several relation types (CoNLL04, NYT) each triplet is (subject:subject type, relation type, object:object type), in the order in which the subject appears §2. The model scores the output token by token, conditioned on a context of example pairs and on the input (equation unnumbered) §2.

# Linearized target for a multi-relation input  [§2]
# Input:  text (str); triplets (list of (subj, subj_type, rel, obj, obj_type))
# Output: target string

def linearize(triplets):
    # Keep the order in which each subject appears in the text  [§2]
    ordered = sort_by_subject_position(triplets)
    # Render each triplet as "(subject:type, relation, object:type)"  [§2]
    parts = [format_triplet(t) for t in ordered]
    return "[" + ", ".join(parts) + "]"

Few-shot prompting with GPT-3 (text-davinci-002). Each dataset has an instructional prefix and a set of worked examples §3 §3.1:

1. ADE: the prefix "List all (drug: adverse effects) pairs in the following text" and 12 examples drawn at random from the training set §3.1.
2. CoNLL04: the prefix lists the entity types and the five relation types. The 12 examples were built by hand to include at least one example of each entity and relation type §3.1.
3. NYT: 24 relation types make a full prefix too long. The authors drop the relation descriptors from the prefix and use 20 examples covering all entity and relation types §3.1.

The new input is appended after the examples and the model continues the text §3.1. Prompt lengths are 755 tokens (ADE), 960 (CoNLL04) and 2095 (NYT) §3.1.

# Few-shot RE with GPT-3  [§3.1] [§2]
# Input:  prefix (str); examples (list of (text, target)); text (str)
# Output: list of triplets, or a flag for non-conforming output

def predict_few_shot(prefix, examples, text, lm):
    # Build the prompt: prefix, then each example, then the new text.
    # Separators between parts are not stated.  [§3.1]
    prompt = build_prompt(prefix, examples, text)
    # ADE used temperature 0.5 and at most 256 output tokens; the other
    # datasets' settings are not stated.  [§3.1]
    output = lm.generate(prompt, temperature=0.5, max_tokens=256)
    # Non-conforming strings are counted and not scored  [§4.1] [§3.3]
    return parse_triplets(output)

Human re-judging of "errors". Exact-match false positives and false negatives are shown to Mechanical Turk workers. For a false positive, the worker judges whether the relation can be derived from the text. For a false negative, the worker judges whether the reference triplet can be inferred from the generated set. A prediction counts as an error only when all three workers agree §3.2 app. D. The paper says the reported results "take into account" these judgements §3.2. The exact formula that applies them to precision and recall is not given.

Flan-T5 few-shot. The same prefixes are used with fewer examples: 7 for ADE and 7 for CoNLL04. NYT is excluded because the prompt cannot hold all types §4.1.

Chain-of-thought (CoT) for GPT-3. The few-shot prompts are repeated with one explanation per example, written by one of the authors. The explanation states how the gold triplets follow from the text §4.2.1.

CoT distillation into Flan-T5 (main method). The authors fine-tune Flan-T5 Large on the gold triplets together with explanations that GPT-3 generates for each training example. The GPT-3 prompt contains the input and its gold triplets, and asks for an explanation §4.2.2 fig. 3. For NYT, explanations were generated for a subset of 25k of the 56k training examples because of cost §4.2.2.

# Distillation targets from GPT-3 explanations  [§4.2.2] [fig. 3]
# Input:  train_set: list of (text, triplets) with gold triplets
#         shots: hand-written (text, triplets, explanation) examples
# Output: list of (source, target) pairs for fine-tuning Flan-T5

def build_cot_pairs(train_set, shots, gpt3):
    pairs = []
    for text, triplets in train_set:
        # Ask GPT-3 for an explanation that supports the gold triplets,
        # using the hand-written shots as a guide  [§4.2.2]
        explanation = gpt3.explain(shots, text, triplets)
        # Join the gold target and the explanation; the join format is
        # not stated  [§4.2.2]
        target = join_target(linearize(triplets), explanation)
        pairs.append((text, target))
    return pairs

Variant with GPT-3 labels. For CoNLL04 only, GPT-3 was given 12 training instances and their explanations, and it then produced both labels and explanations for the rest of the training set. Flan-T5 was fine-tuned on these generated targets §4.2.3.

Learned evaluator (exploratory). The authors train a BERT classifier on the human judgements. It takes "[CLS] input [SEP] potential FP" for false positives and "[CLS] input [SEP] potential FN [SEP] generated labels" for false negatives. It reaches an AUC of 0.88 on CoNLL false positives and 0.73 on CoNLL false negatives app. D fig. 4. Training details are not stated.

Hyperparameters

Fine-tuning settings for Flan-T5 Large, from tab. 4. Most other settings were left at defaults; the paper says some were tuned by hand but does not say which app. B.

Results

Author-reported. Metric: micro-F1 on each dataset's evaluation split. The baseline rows are reproduced from the paper. Values marked "-" are not reported. The paper's own footnote markers are kept.

Datasets tab. 1:

DocRED is listed in the table but not used in the experiments §2 app. A.

Main results tab. 2:

Precision, recall and F1 averaged over five seeds on the test sets (ADE: ten folds) tab. 3:

Human re-judging of exact-match errors §3.2 tab. 6:

• ADE: 51.67% of false positives judged correct (108 of 209). 32.61% of false negatives judged to be true negatives (136 of 417).
• CoNLL04: 50.27% of false positives judged correct (92 of 183). 36.6% of false negatives judged correct (56 of 152 in the table, which gives 36.8%).
• NYT, from Flan-T5 outputs: 36.9% of false positives and 22.97% of false negatives judged correct §3.2.

The paper gives these as the reason exact matching is pessimistic §3.2.

Effect of CoT for GPT-3 (text values) §4.2.1:

• Non-conforming outputs fall from 13.9% to 0.8% on ADE and from 12.5% to 1.1% on CoNLL04.
• The paper reports gains of "about 3" F1 on ADE and "about 2.2" on CoNLL04 in the text. The table gives 1.65 on CoNLL04 (78.18 vs 76.53) and no ADE value.

Generation costs in US dollars tab. 5:

Post-acceptance update §6, author-reported: on 200 Reuters headlines from June 2023, judged by people on June 4 2023, precision fell about 16 points for GPT-3 (78.31 to 62.01) and about 7 points for Flan-T5 (81.22 to 74.70). The authors cannot tell whether the cause is domain shift or data leakage.

Summary of claims §1 §6: few-shot GPT-3 is roughly comparable to fully supervised models with 10s to 20s of examples; Flan-T5 trained with GPT-3 explanations is 5 to 10 F1 points better, according to the conclusion. The per-dataset numbers in tab. 2 give 5.32 on CoNLL04, 9.96 on ADE and 3.23 on NYT over REBEL.

Paper vs code

No official code. The paper links a project page (https://sominw.com/ACL23LLMs), which returns HTTP 404. The author homepage (https://sominw.com/) lists only the arXiv entry. GitHub repository searches for "ACL23LLMs", for the title words and for the author name returned no results.

Pitfalls

• Two different GPT-3 NYT results. tab. 2 lists 61.79 F1 for in-context GPT-3 on NYT, and the text describes a 30-point drop §3.3. tab. 3 lists 68.55 for the same setting. The text also says no NYT prompt gave "reasonable few-shot results" §3.2. It is unclear which number is correct.
• Gains do not match the table. Text gives 9.97 (ADE), 5.42 (CoNLL04) and 3.37 (NYT) points over REBEL §4.2.2. tab. 2 gives 9.96, 5.32 and 3.23. The conclusion says "5-10 points" §6, which the NYT row does not reach. Reported gains may be averaged over folds or seeds, but this is not stated.
• Splits are inconsistent. tab. 3 says the results are on test sets, but the CoNLL04 few-shot prompts were evaluated on the validation split of Eberts et al. §3.1. The text for the 2.c row says it was evaluated on validation §4.2.3. The tab. 2 caption does not name the split.
• Wrong table reference. The text cites "Table 1 (1.f.)" for the CoT results §4.2.2. Table 1 is dataset statistics. The row belongs to tab. 2.
• Human-adjusted scores are not reproducible from the card. The rule for applying the judgements to precision and recall is not written out §3.2. The CoNLL04 false-negative share is 36.6% in the text and 36.8% from tab. 6.
• Unfilled placeholders. The post-acceptance paragraph refers to "Section xx" §6. The exact prompt for that update is therefore not recoverable from the paper.
• Exemplar choice is random for ADE only. The paper gives no seeds for the random ADE draw and reports no variance for GPT-3 results §3.1.
• Explanation quality is not evaluated. The paper says the quality of the GPT-3 explanations was not measured, and that they may affect the results [Limitations, unnumbered].
• CoT target format is ambiguous. The join between the gold triplets and the explanation is shown only by example §4.2.2. The fine-tuning instructional prefix is described for ADE only §4.2.2.
• Hidden cost. GPT-3 explanation generation on NYT cost about US$984 and covered only half the training set tab. 5 §4.2.2. Fine-tuning Flan-T5 took 99 to 109 minutes per epoch on NYT tab. 4.
• Few-shot Flan-T5 is weak. Few-shot Flan-T5 is about 20 F1 below GPT-3 on ADE under strict scoring §4.1. The paper does not give a table value for it.
• Non-conforming outputs are common. They reach 10.6% for few-shot GPT-3 on NYT §3.3, and 13.9% for Flan-T5 on ADE §4.1. How these are scored is not stated.
• Unnumbered equation. The probability model is unnumbered §2, so no equation number can be cited.
• Unexplained asterisk. The SpERT row is marked with an asterisk that has no footnote in the source tab. 2.

Lineage

• Builds on: sequence-to-sequence relation extraction by linearized triplets, with TANL §5 and REBEL §5, both named in the paper and not carded here. Also GPT-3 (Brown et al., 2020, cited as §1), chain-of-thought prompting (Wei et al., cited as §4.2.2) and Flan-T5 (arXiv:2210.11416, cited as §1).
• Compared against: fully supervised SpERT, TANL, TANL (MT), REBEL and fine-tuned Flan-T5 Large tab. 2.
• Superseded by: not stated in the paper.