Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again

In plain words

Large language models can often learn a new task from a few examples pasted into the prompt, with no retraining. This paper asks whether that works for pulling structured facts out of medical and biology papers: finding the names of diseases, drugs and genes, and finding which pairs of entities are linked and how. Labelling data for these jobs is expensive, so prompting looks attractive.

The authors give the large model every fair chance. For each sentence to process, they pick examples from a small set of 100 labelled cases, try several prompt wordings, and nudge the model toward words that appear in the sentence. They compare it with smaller, older-style models that are trained on the same 100 cases.

In this study the smaller trained models come out ahead on nearly every dataset, and by a wide margin on entity extraction. The paper also finds a specific weakness: the large model handles sentences that contain nothing to extract poorly, so it tends to invent or miss items. The authors suggest that, for now, labelling a little data and fine-tuning a small model is the more reliable route.

TL;DR

• Compares GPT-3 (davinci) in-context learning with fine-tuning of BERT-sized models on biomedical NER and relation extraction (RE), using 100 labelled training examples and true few-shot model selection §2.2 tab. 2.
• GPT-3 with retrieved demonstrations, a +10 logit bias (NER) and contextual calibration (RE) scores below all three fine-tuned models on average: NER 51.5 vs 67.1-68.2 F1; RE 36.0 vs 39.9-47.4 F1 tab. 2.
• The authors attribute much of the gap to the null class: sentences or entity pairs with no target entity or relation §4.4.
• Author-reported; the paper is an empirical comparison, not a new method. No numbered equations or algorithms appear in it.

Problem

Biomedical IE needs many task-specific labels, and annotation is costly. In-context learning promises useful extraction with only a handful of examples, which would save that cost §1. The paper tests this under a true few-shot protocol: all choices (prompt, number of demonstrations, fine-tuning settings) are made on the same 100 training examples, not on a large validation set, so that few-shot scores are not inflated by model selection §2.2. Each dataset is evaluated on at most 1000 test examples, and every method sees the same test subset §3.2.

Method

The paper compares two paradigms on the same task splits §2.

(a) BERT-sized fine-tuning §2.3. For NER, each token gets a BIO tag from a linear layer on top of the encoder, trained with cross-entropy; one model per entity type. For RE, the subject and object are masked in the sentence and the [CLS] representation predicts the relation. The appendix describes the masking as new special tokens [ENT1] and [ENT2] app. A.2. Models: PubMedBERT-base, BioBERT-large and RoBERTa-large in the main table §3.2; base versions in tab. 11.

(b) GPT-3 in-context learning §2.4.

1. Task linearisation §2.4.1. NER becomes generation of a list: all entity strings of the sentence, separated by a fixed separator, each entity added once. RE becomes a short prompt containing the sentence and the two entities verbatim, with a label verbaliser that maps each relation type to a plain English word (for example "effect" or "none" in DDI) app. D, tab. 10.
2. Prompt design §2.4.2. Three parts: an overall instruction (may be empty), a sentence introduction, and a retrieval message placed right before the answer. For each dataset the authors compare 8 combinations of these parts and the number of demonstrations, chosen by leave-one-out cross-validation on the 100 examples app. D, tab. 10.
3. Logit bias for NER §2.4.3. Tokens of the input sentence, the separator and the newline token are raised by +10, and generated entities that are not in the sentence are removed afterwards.
4. Contextual calibration for RE §2.4.4. Each test prompt is paired with a null prompt in which the sentence and entities are replaced by "N/A". The null-prompt label probabilities are used to rescale the real ones.
5. Retrieval of demonstrations §2.4.5. For each test sentence, the k most similar training examples (k = 5 or 10 for NER, 5 for RE) are placed in the prompt, using a RoBERTa-large encoder and similarity on sentence embeddings.

Equations: none are numbered. The NER and RE formulations are given in words app. A.1, app. A.2.

# Retrieve demonstrations for one test sentence  [§2.4.5]
# Embeddings: RoBERTa-large first token, unit length
# [code: src/data_utils.py#L389-L414]
def retrieve(test_sentence, train_set, k):
    query = embed(test_sentence)                  # (d,)
    # train embeddings are precomputed once
    sims = [dot(s.embedding, query) for s in train_set]
    ranked = sort_descending(train_set, key=sims)
    # Most similar first, so the nearest example sits farthest from the query
    # [code: src/data_utils.py#L416-L428]
    return ranked[:k]

# Assemble the NER prompt  [§2.4.2] [code: src/data_utils.py#L235-L260]
def build_prompt(instruction, intro, message, demos, test_sentence):
    blocks = []
    if instruction != "":
        blocks.append(instruction)
    # Each demonstration: sentence, then its entity list
    for demo in demos:
        blocks.append(intro + " " + demo.sentence + "\n"
                      + message + " " + demo.entities)
    # Final block: the test sentence, answer left empty
    blocks.append(intro + " " + test_sentence + "\n" + message)
    return "\n\n".join(blocks)

# Logit bias for NER decoding  [§2.4.3] [code: src/gpt3_utils.py#L102-L142]
# Bias value +10 in the configs  [code: configs/gpt3/2/config.yaml#L10]
def ner_logit_bias(sentence, separator, value=10):
    biases = {}
    for token_id in gpt2_tokens(sentence):    # plus spaced variants
        biases[token_id] = value
    for token_id in gpt2_tokens(separator):
        biases[token_id] = value
    biases[NEWLINE_TOKEN_ID] = value          # id 198
    return biases

# Turn generated text into scoreable entities  [§2.4.3]
# [code: src/eval_utils.py#L112-L125]
def post_process(sentence, generated, separator):
    kept = []
    for item in generated.lower().split(separator):
        item = item.replace(",", "").strip()
        # Keep only strings that occur as whole words in the sentence
        if " " + item + " " in " " + sentence + " ":
            kept.append(item)
    return unique(kept)                       # first-occurrence order is lost

# Contextual calibration for RE  [§2.4.4] [code: src/gpt3_utils.py#L73-L89]
# p: label probabilities for the real prompt, softmax over the label
#    logprobs only (renormalised over labels)
# p_null: the same for the N/A prompt built from the same demonstrations
def calibrated_label(p, p_null, labels):
    # The code divides per label and takes the argmax of the ratio;
    # its reported probabilities are a softmax of that ratio
    ratio = p / p_null
    return labels[argmax(ratio)]

Hyperparameters

Results

Author-reported. Setup: BLURB datasets §3.1, 100 training examples per subset, three subsets (mean and standard deviation in the table), at most 1000 test examples per dataset tab. 2 §3.2.

Main results, test F1 (mean ± std over 3 subsets) tab. 2. Full precision and recall columns are in tab. 2 and tab. 11.

Source: tab. 2. The paper reports the average gap as 15.6 to 16.7 F1 for NER and 3.9 to 11.4 F1 for RE §4.1.

Dataset statistics tab. 1. Sizes are train / dev / test.

Ablations of GPT-3 tab. 3 tab. 4. Run on 250 validation examples from one dataset per task, with one 100-example training set. These are not test numbers, so they differ from tab. 2.

The authors report that random demonstrations hurt RE more than NER, and that removing the logit bias raises precision but lowers recall, because invalid strings are generated instead of valid spans §4.2.

Data efficiency fig. 3. On BC5CDR-disease and DDI, training sets of 100, 200 and 500 examples were compared. For NER, GPT-3 improves at about the same rate as the fine-tuned models, so the gap stays roughly constant. For RE, GPT-3 falls further behind as data grows. The curves are shown only as a figure; no values are tabulated in the text §4.3.

Null class tab. 5 tab. 6. On BC5CDR-disease with sentences without any disease removed, GPT-3 F1 rises from 43.6 to 59.8 (recall 35.0 to 59.3), while RoBERTa-large rises from 67.7 to 70.4 tab. 5. In a 2-shot vs 3-shot probe, the null-answer probability rose from 19.4 to 49.1 (+29.7, +153%) for zero-entity sentences and from 15.8 to 40.9 (+25.1, +159%) for sentences with one or more entities tab. 6 §4.4.1.

Computational cost app. B, tab. 8. RoBERTa-large: 354M parameters, 338 GPU hours. PubMedBERT-base: 100M parameters, 138 GPU hours. BioBERT-large: 345M parameters, 344 GPU hours. GPT-3 davinci: 175000M parameters, about $1500 in API cost.

Qualitative RE analysis tab. 7 fig. 4 app. F. LIME saliency on a DDI "none" example shows GPT-3 relying on surface phrases such as "expected to behave similarly", while RoBERTa-large uses the structure around the drug names §4.4.2. The DDI confusion matrices show GPT-3 rarely predicts the none class app. F.1, fig. 4.

Base models app. E, tab. 11. The base versions of BioBERT and RoBERTa are weaker than the large ones on most tasks. On RE, RoBERTa-base is the strongest of the base models, with mean F1 43.5 tab. 11.

Figures (described, not reproduced here): fig. 1 summarises the main findings and a feature comparison; fig. 2 shows the GPT-3 architecture and one-shot NER and RE prompts; fig. 3 shows data-efficiency curves; fig. 4 shows DDI confusion matrices.

Paper vs code

Code: official repository https://github.com/dki-lab/few-shot-bioIE, linked from the paper's abstract footnote. Commit 6cb38bd6d1beb41badc8f08622ff87f94354b1bf (shallow clone, default branch). License: MIT (LICENSE file at the repository root, "Copyright (c) 2022 OSU DKI Lab"). The repository was checked for the points below; the sweep result files (outputs/) are not included, so the numbers in tab. 2 cannot be traced to individual configs.

• GPT-3 engine. Paper: davinci for all GPT-3 experiments §3.2. Code: every GPT-3 config uses ada config.yaml; the benchmark script says to switch to davinci before a full run benchmark_gpt3_in_context.py.
• RE logit bias. Paper: logit bias is described for NER only §2.4.3. Code: the RE call also sets +100 on the first token of each label word, by default gpt3_utils.py.
• Calibration probabilities. Paper: a linear transformation of the label probabilities that equalises them under the null prompt §2.4.4. Code: the decision is the argmax of the ratio p / p_null over labels, and the reported probabilities are a softmax of that ratio, not a renormalised ratio gpt3_utils.py. The predicted label is the same either way; the probability values are not.
• RE metric. Paper: micro F1 tab. 2. Code: micro average restricted to the positive relation labels; the none class is excluded from the averaging label set eval_utils.py.
• Entity scoring. Code rebuilds the gold BIO tags from the gold entity list with the same function used for predictions, rather than reading the original tags eval_utils.py. Tags are compared on B/I/O only, types are dropped eval_utils.py.
• Entity matching. Predicted strings are lower-cased, have commas removed, and must match as whole-word substrings; duplicates are removed with a set eval_utils.py. The paper says only that non-matching entities are discarded §2.4.3.
• Demonstration order. Paper: not stated. Code: most similar demonstration first, so the closest one is furthest from the query data_utils.py.
• PLM hyperparameters. Paper: full grid in tab. 9. Code: sweep configs use narrower sets (for example learning rates 1e-5 and 2e-5, 5 epochs) config.yaml. Code defaults differ again: NER learning rate 3e-5 and 10 epochs run_ner_cv.py; RE learning rate 2e-5 and 5 epochs run_re_cv.py. Which values produced tab. 2 is not stated.
• PLM NER scoring. The training-time metric uses seqeval run_ner_cv.py run_ner_cv.py. The GPT-3 path uses a separate conlleval-style function eval_utils.py. Equivalence of the two is not checked here.
• Sub-word labels. Not in the paper. With label_all_tokens=True (the default), continuation sub-words are labelled as inside-entity, and the training metric keeps every sub-word position run_ner_cv.py run_ner_cv.py run_ner_cv.py.
• Max sequence length. Not in the paper. Inputs are truncated to 128 subword tokens run_ner_cv.py run_re_cv.py.
• RE entity markers. Paper: entities masked §2.3 and, in the appendix, replaced by special tokens app. A.2. Code adds the special tokens ENT1 and ENT2 run_re_cv.py.
• Test sampling. Paper: at most 1000 test examples, the same subset for all methods §3.2. Code: NER takes the first 1000 of a seed-42 shuffle; RE uses a stratified sample main_pipeline.py main_pipeline.py.
• Prompt search. Paper: leave-one-out CV on the 100 examples §2.4.2. Code: LOOCV implemented with cross_val=True, and the best F1 config is selected main_pipeline.py main_pipeline.py.

No discrepancy was found for the 8-prompt combination count in app. D: the NER config for BC5CDR-disease has 2 instructions x 2 intros x 2 demonstration counts, which gives 8 config.yaml.

Pitfalls

• Claim vs table. The text says fine-tuned models beat GPT-3 on all datasets §4.1. Table 2 shows PubMedBERT-base at 60.2 F1 on GAD against 66.0 for GPT-3 tab. 2. The NER claims hold on every dataset, and the RE averages hold; the "all datasets" wording does not hold for that one pair.
• Engine availability. The results use the original davinci engine, called through openai.Completion.create gpt3_utils.py. Exact reproduction needs that engine. Whether it is still served is not stated; the card does not assume it is.
• Demonstration order. The most similar demonstration is placed first, so the nearest one is furthest from the test sentence. The paper does not say this, and the results may depend on it data_utils.py.
• Retrieval ignores the entities. The retrieval embeds the whole sentence, so for RE the two entities do not influence which demonstrations are chosen. The authors name this as a limitation § Limitations data_utils.py.
• Logit bias is a token list. The code builds the bias from ids produced by the GPT-2 tokenizer gpt3_utils.py, plus the newline id 198 gpt3_utils.py. The paper does not state whether these ids match the engine's vocabulary, so check them against the engine before reusing the list.
• Post-processing changes the score. Entity strings that do not occur as whole words are dropped. This raises precision and lowers recall, as the ablation shows tab. 3 §4.2. Every occurrence of a predicted string is tagged, not just the one the model meant.
• Gold labels are rebuilt. The scorer rebuilds the gold BIO tags from the entity list, so the score depends on that rebuild, not on the original annotation eval_utils.py.
• RE F1 ignores true negatives. Correct predictions of the none class are not counted anywhere in the micro average. The paper's none-class analysis is therefore outside the reported F1 eval_utils.py.
• Calibration doubles the calls. Each RE test example needs a second model call for its null prompt. The null prompt reuses the same retrieved demonstrations, so it is built per example, not once gpt3_utils.py.
• Validation vs test. The ablations use 250 validation examples and one training set §4.2. The same dataset's test number is in tab. 2. Do not compare the two directly.
• Standard deviations are over subsets. The ± values come from three training subsets, not from repeated prompt runs tab. 2. The code sets temperature to 0 gpt3_utils.py; the paper does not discuss run-to-run variation.
• Costs. The GPT-3 runs cost about $1500 app. B, tab. 8. The repository's own estimator is a per-token formula gpt3_utils.py.
• Few-shot selection for PLMs. The PLM hyperparameters and stopping epoch are selected by 5-fold CV on the same 100 examples §2.3. The paper does not state which values were chosen per dataset.
• LIME is modified. The analysis uses a 3-token sliding window and the token BLANK for GPT-3, and keeps the query entities unmasked app. F.3. The saliency maps are therefore not standard LIME output.

Lineage

• Builds on: arXiv 2101.06804 (Liu et al., "What Makes Good In-Context Examples for GPT-3?") for the kNN demonstration retrieval used here §2.4.5 [raw/src/2203.08410/custom.bib].
• Builds on, not in the wiki: contextual calibration (Zhao et al., 2021) §2.4.4; true few-shot evaluation (Perez et al., 2021) §2.2; the BLURB benchmark and PubMedBERT (Gu et al., 2022) §3.1.
• Compared against: BioBERT-large and RoBERTa-large, fine-tuned under the same 100 examples §3.2.
• Superseded by: not stated in this card.