Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!

In plain words

Large language models can often do a task after seeing just a few examples, so it is tempting to use them to pull names, relations and events out of text when labelled data is scarce. This paper tests that idea across nine datasets. Its main answer is no: a plain prompt to a large model is usually less accurate, slower and more expensive than a small model trained on the same few labels.

The useful part is where the large model does help. When the small model is unsure, the large model is often right. So the authors let the small model handle the easy cases and send only the uncertain ones onward. The large model does not write a free-form answer there. It sees a short multiple-choice question listing the small model's best guesses plus "none of these", and it picks one.

The result is a modest accuracy gain over the best small-model baseline, at a fraction of the cost of prompting the large model for every sentence.

TL;DR

• Prompted LLMs (CODEX, InstructGPT, ChatGPT, GPT-4, Vicuna-13B, LLaMA-13B) are not, in general, good few-shot information extractors: they trail fine-tuned small models in most of the nine NER, RE, event-detection and event-argument settings, and they are slower and costlier.
• LLMs do beat small models on hard samples, the ones the small model scores with low confidence.
• The proposed adaptive filter-then-rerank sends only low-confidence spans to an LLM, which chooses among the small model's top-3 labels plus None.
• Reported gain: about 2.4 F1 on average over three datasets, at 80-90% lower budget and latency than direct prompting (author-reported).
• The official code covers only the reranker. Several defaults and options differ from the paper; see Paper vs code.

Problem

Few-shot information extraction (IE) asks a model to find and type entity mentions, relations, event triggers or event arguments from a handful of labelled examples per label type §3.1. Two families compete: supervised small language models (SLMs), fine-tuned with few samples, and large language models used in-context, with no weight updates §3.2, §3.3. The paper asks whether prompted LLMs are competitive, especially when label types are many and samples are few §3.4, fig. 2. A second concern is cost: LLM inference is much slower and pricier than SLM inference §3.4, tab. 1.

Method

Empirical comparison (§3)

An LLM gets one prompt made of an instruction, a set of demonstrations picked from the labelled pool, and the test sentence. The output is parsed into structured results §3.3, eq. 1:

\mathcal{P}_{\mathcal{E}, I, f}(D, s) = [I; f(\mathcal{E}(D, s)); f(s)] \quad \text{[eq. 1]}

Here I is the instruction, f is a plain text template, and \mathcal{E} is the demonstration selector §3.3. The main experiments select demonstrations by sentence-embedding similarity to the test sentence, using SimCSE-RoBERTa-large §3.3, app. E. The paper also tried random selection and an efficient prompt retriever app. E, eq. 3, eq. 4.

Adaptive filter-then-rerank (§4, §5)

The small model is the filter. For every candidate span it gives a probability per label. Its confidence is the largest of these probabilities §4.2, eq. 2:

\mathrm{conf}(x) = \max_{l \in L} P_{SLM}(l \mid x; s) \quad \text{[eq. 2]}

Spans with confidence above a threshold \tau are easy; the small model's label is final. Spans below \tau are hard. For those, the top-N labels from the small model are shown to the LLM as options of a multiple-choice question, with None added when it is not already among them §5.1, §5.2. Each label is rendered as a short statement through a per-label template, for example "is a company" for an organisation label app. G. The prompt has four demonstrations with explanations, and the LLM answers after a brief explanation (chain of thought) §5.2. The paper sets N = 3 §5.2.

# Adaptive filter-then-rerank for one candidate span  [§5.1] [fig. 6]
# Input:  sentence, span, slm, llm, demos, tau (threshold), top_n
# Output: a label, possibly None

def classify_span(sentence, span, slm, llm, demos, tau, top_n):
    # 1. The supervised small model scores every label  [§4.2]
    label_probs = slm.label_probabilities(sentence, span)
    # 2. Difficulty is the highest of those probabilities  [§4.2] [eq. 2]
    confidence = max(label_probs.values())
    # 3. Easy spans keep the small model's answer  [§5.1]
    if confidence >= tau:
        return max(label_probs, key=label_probs.get)
    # 4. Hard spans keep the top-N labels, in the small model's order [§5.2]
    ranked = sorted(label_probs, key=label_probs.get, reverse=True)
    candidates = ranked[:top_n]
    # 5. Add None when the top-N do not already include it  [§5.2]
    if None not in candidates:
        candidates.append(None)
    # 6. Ask the LLM for one option, with an explanation first  [§5.2] [app. G]
    question = build_multiple_choice(sentence, span, candidates)
    reply = llm.answer(demos, question)
    # 7. Map the chosen letter to a label. If the reply cannot be parsed,
    #    keep the small model's label  [code: utils.py#L106-L114]
    return parse_choice(reply, candidates, fallback=ranked[0])

Building the few-shot data (appendix A)

For NER and ED the training and validation subsets are built by a greedy procedure that tries to give each label K sentences. Negative sentences (without any label) are added so that positive and negative sentences are in a 1:1 ratio app. A. Relation extraction takes exactly K sentences per label app. A.

# Greedy few-shot sampling for NER and ED  [alg. 1] [app. A]
# Input:  K (shots per label), pool of (sentence, labels) pairs
# Output: a sampled set with at least K sentences per label where possible

def greedy_sample(pool, label_set, K):
    # 1. Sort labels by ascending frequency in the pool  [alg. 1, line 1]
    labels = sort_by_frequency(label_set, pool)
    # 2. Start from an empty sample and zero counts  [alg. 1, lines 2-4]
    sample = []
    counts = {label: 0 for label in label_set}
    # 3. For each label, add sentences until it has K  [alg. 1, lines 5-10]
    for label in labels:
        while counts[label] < K:
            # the choice rule within the pool is not stated in the paper
            sentence = pick_containing(pool, label)
            pool.remove(sentence)
            sample.append(sentence)
            counts = recount(sample, label_set)
    # 4. Drop a sentence if all labels stay at or above K without it
    #    [alg. 1, lines 11-16]
    for sentence in list(sample):
        trial = [s for s in sample if s is not sentence]
        if all(recount(trial, label_set)[l] >= K for l in label_set):
            sample = trial
    return sample

Results are averaged over five sampled train/validation sets §3.1. Validation is a 10% split when a set has over 300 sentences; otherwise 5-fold cross-validation is used app. A.

Hyperparameters

Results

All numbers are author-reported. Setup: 9 datasets in four IE tasks (named entity recognition on CoNLL03, OntoNotes and FewNERD; relation extraction on TACRED and TACREV; event detection on ACE05, MAVEN and ERE; event argument extraction on ACE05, ERE and RAMS) §3.1, app. A. Full-dataset statistics:

app. A, tab. 6

Small versus large models (§3.4). LLMs beat SLMs mainly when the samples and labels are very few. The SLM curves rise more steeply with more samples fig. 2, §3.4. LLMs do relatively worse on fine-grained datasets with many labels. The 5-shot margin over the best SLM falls as the label count rises: for NER it is +25.3 F1 on CoNLL03 (4 labels), -0.3 on OntoNotes (18) and -3.9 on FewNERD (66) app. E, tab. 11. Figures are described in words here; the paper gives no numbers for the curves.

Inference cost (§3.4, tab. 1). Seconds for 500 sentences on one V100:

Main reranking results (§5.3, tab. 3). Micro-F1 at the listed shot counts. Each cell is a mean over 5 runs, except the InstructGPT and GPT-4 rows, which are single runs tab. 3 caption. Sample standard deviations in the paper range up to 5.2 points.

"Ensemble" combines two SLMs as the filter §5.2. Against the previous SLM state of the art in these nine cells, InstructGPT reranking without ensemble averages about 2.3 points by simple arithmetic on the table; the abstract's 2.4 figure is not tied to a named row set in the text. The paper adds SLM-only baselines (an ensemble, and an SLM reranker) to show that the gain is not only an ensembling effect. Those baselines gain less in most cells §5.2, tab. 3.

Where the reranking helps (§4.2, §5.4, tab. 4, fig. 5). On the hard subset, the reranker changes F1 as follows. Shares are the fraction of all samples that were reranked:

tab. 4 The paper's explanation is that GPT-4 reranks more aggressively, adding true positives, while InstructGPT removes more false positives §5.4.

Easy samples and negatives (§4.3, tab. 2). Most negative samples are easy for the small model. The ratio of negative to positive samples is higher among easy samples than overall, so LLM false positives hurt most on easy spans tab. 2. Overall versus easy versus hard negative-to-positive ratios: FewNERD 5.88 / 9.44 / 1.28; TACREV 3.03 / 3.21 / 2.68; ACE05 38.2 / 44.0 / 1.36.

Ablation (§5.5, tab. 5). Filter: ensembled SLMs. Reranker: GPT-4. Settings are 20-shot FewNERD, 100-shot TACREV and 20-shot ACE05 tab. 5:

Each removal is cumulative, read down the table. Without any filter the reranker is applied to everything, and results collapse §5.5.

Cost (§5.4, fig. 7). Relative to direct ICL, the paper reports 80-90% lower budget and latency, because the LLM is called only on hard samples and gets shorter prompts (fewer labels and demonstrations) §5.4, fig. 7. The exact cost figures are in the plot, not in the text.

Other findings. Sentence-embedding demonstration selection gave 57.6 / 49.6 / 42.9 F1 (FewNERD / TACREV / ACE05, 20-shot, ChatGPT), against 53.2 / 43.0 / 38.0 for random selection and 57.2 / 48.0 / 43.5 for the efficient prompt retriever app. E, tab. 13. Automatic chain-of-thought rationales hurt on all three tasks app. D, tab. 9. GPT-4 did not clearly beat the other LLMs except on OntoNotes and MAVEN, and still trailed supervised models there app. D, tab. 10.

Paper vs code

Official repository: mayubo2333/LLM-IE, linked from the abstract and owned by an author's handle (the same owner hosts the PAIE code the paper cites). The card follows arXiv v2 (updated 2023-10-21). Commit 400a50f, checked out with depth 1.

• Threshold tau. Paper: tuned on the validation set §5.2. Code: the default is a fixed upper bound of 0.6 run_reranker.py, and the shipped scripts set 0.6 for NER and 0.7 for RE and ED [code: scripts/run_ner_fewnerd.sh#L17] run_re_tacrev.sh [code: scripts/run_ed_ace.sh#L17]. The validation-tuning branch (--auto_th) reads the demonstration file instead of the validation results [code: run_reranker.py#L73]. The validation-results option --dev_res_path is declared but never read run_reranker.py. A user who turns on --auto_th gets a threshold tuned on the wrong data.
• Confidence. Paper: maximum label probability of the SLM §4.2, eq. 2. Code: reads a precomputed max_prob field and never computes it [code: run_reranker.py#L99-L100]. Only FewNERD small-model outputs are shipped [README]. TACREV and ACE05 small-model outputs must be produced by the user [README].
• Candidates and None. Consistent. Top three in SLM order, then None if absent ner_reranker.py [code: reranker/re_reranker.py#L37-L39].
• Demonstrations. Paper: four, chosen by no stated rule §5.2. Code: draws four at random, once per run, from the shipped demonstration file [code: run_reranker.py#L63-L67], seeded with 42 run_reranker.py.
• Chain of thought. Consistent: the flag adds an explanation before the answer run_reranker.py. Output length is capped at 128 tokens with explanations, 32 without ner_reranker.py. The paper does not state a reranker output limit.
• Models. Paper: InstructGPT, GPT-4 and Vicuna-13B as rerankers §5.2. Code: any Hugging Face causal model, and OpenAI chat models only [code: LLM_wrapper.py#L11-L30]. InstructGPT (text-davinci-003) is a completion model, so the paper's InstructGPT rerank rows cannot run through the shipped OpenAI path unchanged. The README reports tests on vicuna-7b-v1.5 and Mistral-7B-Instruct-v0.2, not on the 13B model in the paper [README].
• Model snapshots. Paper: gpt-3.5-turbo-0301, gpt-4-0314, text-davinci-003 §3.3 footnote, app. C. Code default: the alias gpt-4 [code: run_reranker.py#L18], which is not the snapshot the paper used.
• Parse failures. Not discussed in the paper. Code: any parse failure keeps the small model's prediction, silently ner_reranker.py re_reranker.py.
• Not in the repository. In-context learning baselines, SLM training code (FSLS, KnowPrompt, PAIE, UIE), the demonstration selectors, and the sampling procedure of appendix A. The paper's section 3 numbers cannot be checked against code [README].
• Label templates. Paper appendix G and the code dictionaries agree on the labels spot-checked (for example the artist-or-author and company templates) parameters.py.
• License. The repository has no LICENSE file, and the README states no license. The code is therefore not licensed for reuse here: no verbatim snippets are used, and the notes above paraphrase the code and cite lines.

Pitfalls

• Threshold is not given. The paper never states the tau values used, and the validation-tuning path is broken (see Paper vs code). Choose tau on your own validation split, and report it. The shipped 0.6 and 0.7 are not the paper's reported values.
• Calibration assumption. The filter needs well-calibrated probabilities. The authors call this an obviously imperfect assumption and leave better difficulty measures to future work [Limitations].
• Silent fallback. A failed parse keeps the SLM label. Count failures; they quietly lower the measured effect of the LLM.
• No output cap on the API path. The OpenAI call takes only model, messages and temperature LLM_wrapper.py, so the max_tokens value passed to generate_response is dropped. Explanations can run long. Add a cap if you port the code.
• Temperature above zero breaks the open-model path. do_sample is set only when temperature is zero or below LLM_wrapper.py, then used unconditionally LLM_wrapper.py. Keep temperature at 0.
• Old SDK. The OpenAI call uses the pre-1.0 ChatCompletion interface. The README pins openai 0.27.0, Transformers 4.36.0 and Python 3.8 [README].
• Demonstration draw. Four demonstrations are drawn at random once per run. The paper does not say whether its 5 runs re-draw them. Results depend on the draw.
• Option order. Options are shown in small-model rank order, so the top small-model label is always option (a) ner_reranker.py. The paper does not examine position bias. The --is_shuffle option shuffles all labels, not only the top N ner_reranker.py.
• In-place mutation. Label padding appends template labels to each record's candidate list until it has 26 entries ner_reranker.py. Re-running on the same records is not clean.
• Metric definition. The paper says micro-F1 without defining it for NER and ED. The code scores per sample: each mention or trigger counts as one item, and None is excluded from both sides utils.py. Match this, or your numbers will not compare.
• Noise versus gain. Table 3 standard deviations reach several points. Several gains are 1.2 to 1.5 points. GPT-4 and InstructGPT numbers are single runs. Compare with the same demonstrations and temperature 0.
• Test subset. Section 3 downsamples test sets to 250 or 500 sentences. The reranking section does not say which test subset it used.
• Cost figures are author-reported from one API setup. The 80-90% figure is read from the plot in figure 7; the text gives no price list.
• Few-shot sampling is ambiguous. The paper does not say how a sentence is picked from the pool at each step of the greedy procedure, and the loop in algorithm 1 changes the set it iterates over. Iterate over a copy.
• Open models get no chat template. The open-model path joins prompt parts with blank lines and no chat template LLM_wrapper.py. Instruction- tuned models such as Vicuna may do worse than with their own template. The paper does not specify a template.
• Reranking covers three datasets. Section 5 reranks only FewNERD, TACREV and ACE05, across nine shot settings §5.2, tab. 3. Other datasets are not reranked.

Lineage

• Builds on: multiple-choice prompting in the style of QA4RE, which the paper uses to turn IE into choice questions §4.1; SLM baselines FSLS, KnowPrompt, PAIE and UIE app. B; greedy sampling after Yang and Katiyar (2020) app. A; chain-of-thought prompting §5.2; retrieval of demonstrations following prior work §3.3.
• Compared against: in-context learning with CODEX, InstructGPT, ChatGPT, GPT-4, LLaMA-13B and Vicuna-13B §3.3; SLM ensembles and SLM reranking §5.2.
• Superseded by: not stated in the sources used for this card.