GPT-NER: Named Entity Recognition via Large Language Models

In plain words

Finding names of people, places and organisations in text (named entity recognition) is usually done by training a model that labels each word. A large language model writes text well, but it is not built to hand out a label to each word, so on this task it trails models trained for it.

GPT-NER changes the question into one the model can answer by writing. The model copies the sentence back and places special markers around each entity it finds. For a search for place names, "Columbus is a city" becomes "@@Columbus## is a city". Because the model only copies text and adds markers, it has less to get right.

The prompt holds a short task description, a few solved examples of the same format, and then the new sentence. In the best version, the examples are picked because they contain entities like the current one. The model is asked once for each entity type. Language models tend to mark things that are not entities, so a second prompt asks the model yes or no about each marked span, and the markers on rejected spans are removed. On the full benchmarks the result is close to trained systems on one dataset and further behind on another. It does best when labelled training data is scarce.

TL;DR

GPT-NER asks a GPT-3 model (davinci-003) to copy each sentence and wrap entities in @@ and ## markers, once per entity type §4.1. Demonstrations chosen by entity-level token embeddings beat random ones by a wide margin, and a yes/no self-verification pass adds a smaller gain tab. 2. All scores are author-reported. On full data it trails the best supervised model by about 4 F1 points on CoNLL2003, about 10 on OntoNotes5.0, and 15 to 19 on the nested datasets tab. 2 tab. 3. Every prediction costs API calls, so budget for them before choosing this over a trained tagger.

Problem

Named entity recognition assigns a type to each word of a sentence, which is a sequence-labelling task §3 eq. 1. Large language models are built for text generation, and the authors argue that this mismatch leaves them well below supervised baselines on NER §1.

A second problem is over-prediction. LLMs strongly tend to label text that contains no entity as one, even when examples are given §4.3.

Context windows also limit the approach. The authors cite a 4096-token prompt limit for GPT-3 and assume about 1.3 tokens per English word, so a prompt cannot hold the instructions and examples for every entity type at once §4.1.1.

The retrieval variants assume labelled training data is available as a datastore §4.2.2. The best variant also needs a fine-tuned NER model to embed entities §4.2.2.

Method

Overview. Three steps: build a prompt for each sentence and entity type, send it to the LLM, and convert the generated text into entity labels §4.

Prompt construction §4.1 fig. 1. The task description has three parts §4.1.1:

1. A constant opening sentence, "I am an excellent linguist", that asks for output in the style of linguistic knowledge §4.1.1.
2. A sentence naming the entity type to extract, for example location. Because each prompt covers one type, a sentence with N types needs N prompts, which turns an N-way classification into N binary decisions §4.1.1.
3. A sentence that introduces the demonstrations §4.1.1.

Demonstrations §4.1.2. The prompt then lists k solved pairs. Each pair is an input sentence followed by its labelled output, in the form "Input: ... Output: ..." §4.1.2. The examples do two jobs: they fix the output format, and they show the model what the task looks like §4.1.2.

Output format §4.1.2. If a sentence has no entity, the output copies the input unchanged. Otherwise each entity is wrapped in "@@" and "##". The authors compared this with a BMES tag-per-word format and with an entity-plus-position format, and both did far worse §6.1. The authors report that the tag-per-word format often produced output of the wrong length, and that the position format confused the position index §6.1.

Input sentence §4.1.3. The new sentence is appended, followed by an "Output:" cue where generation starts §4.1.3. Figure 1 shows a full example for the location type fig. 1.

Demonstration retrieval §4.2. Three options are compared:

• Random: k examples drawn at random from the training set §4.2.1. The authors note that nothing ties them to the input §4.2.1.
• Sentence-level kNN: embed each whole sentence with SimCSE and take the k nearest training sentences by cosine similarity §4.2.2.
• Entity-level kNN (the best variant): a fine-tuned NER model labels every training sentence, and each predicted entity span is stored with its embedding as a key that points back to its sentence §4.2.2 fig. 2. For a test sentence, the model finds predicted entities and retrieves, for each one, the nearest stored entity embeddings. The k best retrieved sentences become the demonstrations §4.2.2. The paper describes taking the top k from the K times N candidate tokens, where N is the number of entities in the test sentence §4.2.2. The implementation differs; see Paper vs code.

Self-verification §4.3 fig. 3. After extraction, each candidate entity is checked by a second prompt. The prompt gives a task description that tells the model to check whether the candidate word is an entity of the given type taken from the sentence, followed by demonstrations, the test sentence, and a question about the candidate word §4.3. The model answers "yes" or "no". A "yes" keeps the span. A "no" removes its markers §4.3. Each demonstration has three lines: the sentence, the question and the answer §4.3.

For verification demonstrations, the authors reuse the entity-level retrieval §4.3, "Demonstration Selection". A retrieved example is answered "Yes" only if its entity has the queried type, and "No" otherwise §4.3.

# GPT-NER for one sentence, one entity type at a time  [§4] [§4.1.1]
# Input:  words (list of str); label_types (list of str); k (int)
# Output: list of (span, label) pairs

def gpt_ner(words, label_types, k):
    spans = []
    for label in label_types:
        # Demonstrations: entity-level kNN, or random  [§4.2]
        demos = retrieve_demonstrations(words, label, k)
        # Task description, demonstrations, then the input  [§4.1]
        prompt = build_prompt(label, demos, words)
        # Output copies the sentence, marking entities with @@ and ##  [§4.1.2]
        completion = complete(prompt)
        candidates = parse_marked_spans(completion, words)
        for span in candidates:
            # Self-verification: keep only spans the model confirms  [§4.3]
            demos_v = retrieve_demonstrations(span, label, k)
            verdict = verify(words, span, label, demos_v)
            if verdict == "yes":
                spans.append((span, label))
    return spans

Pseudocode for the verification prompt is not separate: it is the prompt described above §4.3.

Hyperparameters

Results

All numbers are author-reported. Each row is one setting from the paper. "FS" means self-verification with few-shot demonstrations; "ZS" means zero-shot self-verification. "random", "sentence" and "entity" name the demonstration retrieval method. The table text was checked against the tex source, see Pitfalls for mismatches.

Sampled 100-instance test sets tab. 1 (GPT-3 davinci-003)

CoNLL2003 (sampled 100):

OntoNotes5.0 (sampled 100):

Full test sets, flat NER tab. 2

CoNLL2003 (full):

OntoNotes5.0 (full):

Full test sets, nested NER tab. 3

ACE2004 (full):

ACE2005 (full):

GENIA (full):

Headline comparisons (author-reported; read from the tables above)

• CoNLL2003, full: FS + entity 90.91 F1 against BERT-MRC 93.04 and ACE+document-context 94.6 tab. 2. The text says the gap to the SOTA is "94.6 v.s. 90.91" §5.1.1 (4).
• OntoNotes5.0, full: FS + entity 82.20 against BERT-MRC+DSC 92.07 tab. 2.
• Nested: FS + entity reaches 74.2 on ACE2004 against BINDER 88.7, 73.59 on ACE2005 against BINDER 89.5, and 64.42 on GENIA against BERT-MRC 83.75 tab. 3.
• Low resource §5.2.1: with 8 training sentences GPT-NER's F1 is "about 60", while the supervised baseline is near 0. The learning curves for 100, 1K and 10K sentences are in fig. 4 only; no values are given in the text or tables. The authors report that random retrieval barely improves with more data, while kNN retrieval improves §5.2.1 (2) fig. 4.
• Output format §6.1: on sampled CoNLL2003 with 32 few-shot examples, the proposed format scores 92.68 F1, BMES 29.75, and entity-plus-position 38.73. The retrieval method used for this ablation is not stated.
• Demonstration count §6.2 fig. 5: the authors report that F1 still rises at the largest k they could fit within GPT-3's 4096-token limit. They also report that kNN retrieval does worse than random at k = 2 and k = 4, because retrieved examples often contain no entities.

Paper vs code

The official repository is ShuheWang1998/GPT-NER, linked in the paper's footnote §5. Its owner name matches the first author. It has no LICENSE file, so no code is quoted here. Code is cited only for comparison, and every hyperparameter and algorithm step below was checked against it.

• Prompt wording. The paper opens with the linguist persona, names the entity type to find in the given sentence, and then says examples follow §4.1.1. The code uses one template for every dataset. It names OntoNotes5.0 in the opening line, although the same template runs on CoNLL2003 and the nested sets. The template also tells the model to copy the style of the examples get_results_mrc_knn.py. The paper's wording is commented out in the code get_results_mrc_knn.py.
• Demonstration layout. The paper uses "Input:" and "Output:" lines §4.1.2. The code uses "The given sentence: ..." and "The labeled sentence: ..." get_results_mrc_knn.py get_results_mrc_knn.py.
• Entity-type descriptions. The code's descriptions differ from the appendix in places. For example, NORP is described as excluding countries, cities and states in the code but not in app. A dataset_name.py.
• Demonstration count. The paper gives no k for the main results §5.1. The code defaults to 16 get_results_mrc_knn.py. The committed run script sets 8 access_ai.sh.
• Sentence-level retrieval model. The paper says "SimCSE" without naming a checkpoint §4.2.2. The code loads sup-simcse-roberta-large extract_mrc_knn.py.
• Entity-level candidate selection. The paper selects the top k from the K times N candidates §4.2.2. The code takes the first ceil(K/N) neighbours of each entity, concatenates them and keeps the first K extract_mrc_knn.py. The example script then keeps the first 8 get_results_mrc_knn.py.
• Sentences with no predicted entity. The paper does not describe this case. The code falls back to sentence-level SimCSE retrieval for them extract_mrc_knn.py.
• Datastore split. The paper says the datastore is the training set §4.2.2. The committed OntoNotes script uses the dev split as the source of demonstrations extract_mrc_knn.py. The CoNLL2003 word-embedding search uses train.dev find_verify_knn.py.
• Entity embeddings for verification. The paper says a fine-tuned NER model embeds entities §4.2.2. The repository reads precomputed word features, and its feature script only computes sentence-level SimCSE features find_verify_knn.py offline_comput_feature.py. The fine-tuned model and the word-feature extraction are not in the repository.
• Verification is skipped for PER. Spans of the PER type are never verified verify_results.py. The paper does not say this.
• What counts as a verification demonstration. The paper says a retrieved example is "Yes" if its entity has the queried type §4.3. The code also requires the retrieved span to be a gold entity at that position find_verify_knn.py verify_results.py.
• Unparseable verdicts. The paper gives only "yes" and "no" §4.3. The code removes markers only on an answer starting with "no"; anything else keeps the span verify_results.py. Any answer beginning with "no" counts as a rejection, including "Not sure".
• Verification prompt wording. The code's verification prompt opens with a linguist persona and a verification task for the named entity type, then adds a capitalised sentence about that type's definition verify_results.py. The paper's wording differs slightly §4.3.
• Decoding parameters. These match the paper: temperature 0, top_p 1, both penalties 0, best_of 1, 512 maximum tokens, davinci-003 get_results_mrc_knn.py verify_results.py.
• Retries and batching. The code retries failed calls with exponential backoff, up to 6 retries; rate-limit errors and other exceptions take the same path base_access.py base_access.py. It batches 4 extraction prompts per call get_results_mrc_knn.py and sends verification prompts one at a time verify_results.py. The paper does not discuss batching.
• F1 scoring. The scorer matches each predicted span to the sentence by its text, scanning forward from the previous match compute_f1.py. A predicted span whose text is not found is dropped and not counted as a false positive compute_f1.py.
• Reported numbers. The repository's README tables differ from the paper for several full-data OntoNotes5.0 and ACE2005 rows. Examples: OntoNotes5.0 entity FS F1 is 77.20 in the README and 82.20 in tab. 2; ACE2005 entity-level F1 is 73.96 in the README and 72.96 in tab. 3. The paper's tables are used here; the discrepancy is unexplained.

Pitfalls

• Verification crash on an empty neighbour list. When an entity's retrieved list is empty, get_knn returns None, and the caller's != "" check does not catch it. The string concatenation then raises a TypeError verify_results.py verify_results.py. Guard against empty lists when reimplementing.
• Demonstrations may all come from the first entity. With N predicted entities, each contributes ceil(K/N) neighbours, and the example slice keeps the first few. With K = 32 and k = 8, a sentence with two or more predicted entities gets all 8 examples from its first entity extract_mrc_knn.py get_results_mrc_knn.py. The paper says nothing about this.
• Verification demonstration labels depend on the gold annotations. A datastore example is labelled "Yes" only when its span is gold at that position, so annotation errors in the training data flow straight into the verification prompts find_verify_knn.py.
• The committed verification run is zero-shot. verify.sh does not pass --knn-file, so it does not run few-shot verification verify.sh. Few-shot needs the --knn-file option, and the --knn-num flag is commented out in that script. Its CLI default is 1 verify_results.py.
• Cost scales with sentences, types and candidate spans. Extraction costs one call per sentence and entity type. Verification costs one call per candidate span, sent one at a time verify_results.py. The paper reports no call counts.
• Position mapping depends on exact copies. The approach works only if the model copies the sentence exactly, because spans are recovered by searching the text. A changed or re-tokenised word can shift or drop a span compute_f1.py. Check the output format before trusting the scores.
• Text and table mismatches in the paper.
  • §5.1.1 (3) gives "84.97 v.s. 85.91" for few-shot. Neither pair matches a row in tab. 2; the table's FS + entity value on CoNLL2003 is 90.91.
  • §5.1.2 (2) gives "56.68 v.s. 69.06" on GENIA. tab. 3 gives sentence 51.68 and entity 64.06.
  • §6.2 refers to Figure 2 (fig:finetuned_method) for the k-shot results. The k-shot plot is Figure 5 fig. 5.
• F1 values that do not match their own precision and recall. Span-level micro F1 is the harmonic mean of P and R. For ACE2005 entity-level ZS, P 72.63 and R 75.39 give about 73.99, but the table says 73.46. For ACE2005 entity-level FS, P 72.77 and R 75.51 give about 74.11, and the table says 73.59. Recompute from the counts if you rely on these rows.
• Retrieval for the output-format ablation is not stated. The 92.68 F1 in §6.1 equals the sentence-level row of tab. 1, which suggests that retrieval, but the paper does not say so.
• The low-resource setup uses zero-shot verification only §5.2 Setups. Do not compare its numbers with the full-data FS rows.
• Several key components are not in the repository. The fine-tuned NER model, the word-feature extraction, the sampled test files and the MRC-processed data are missing. The README points to an external drive for the sampled data README.md. Plan to rebuild the entity-level embeddings yourself.
• Results are for one closed model. davinci-003 was accessed through a paid API with fixed decoding settings. The authors' own motivation is that access is expensive §5.1.1. Repeat runs may not match exactly, even at temperature 0; the paper does not measure this.

Lineage

• Builds on: in-context learning with GPT-3 §2, SimCSE sentence embeddings §4.2.2, and the MRC-NER data protocol for CoNLL2003 and OntoNotes5.0 §5.1.1. None of these has a card on this wiki.
• Compared against: BERT-Tagger, BERT-MRC, MRC+DSC, GNN-SL, ACE+document-context, Triaffine and BINDER, all as in tab. 2 and tab. 3. None has a card here.
• Superseded by: not stated in this wiki.