Named entity recognition

Overview

Named entity recognition (NER) finds the spans of a sentence that name an entity and gives each span a type such as person, location or organisation. The classic formulation is sequence labelling: each word gets a tag, usually in BIO form, and a model scores the whole tag sequence 2304.10428 §3 2404.09383 §2. The variants fall into three groups. The first trains a tagger on labelled sentences. Options include a CRF that can borrow labels from a related language 2404.09383 §2.3, §3.1, a biomedical pipeline with a transition-based tagger 1902.07669 §4, a German encoder fine-tuned on NER 2012.02110 §3, a SciBERT-based cascade for scientific entities 2606.29836 §3.2, a SciBERT encoder continued on materials text with a CRF or BiLSTM-CRF head 2109.15290 §2.4.1, and a SciBERT-BiLSTM tagger for method, dataset, tool and metric mentions 2606.31058 §3.3.1. The second group works on the data. It uses token-level edits to a small labelled set 2010.11683 §3, keeps only generated sentences that a tagger confirms 2606.29836 §3.3, or lets an LLM pre-label text that experts then correct 2312.02296 §4 2312.02296 §5. The third group asks an LLM for the labels directly. For span labelling, the model copies the input and marks each entity, with one prompt per entity type 2304.10428 §4.1.1. For de-identification, it replaces identifiers with a placeholder 2303.11032 §IV. The LLM prompting variants differ in the output format, how the in-prompt examples (demonstrations) are chosen, and whether the extracted spans are checked by a second prompt 2304.10428 §4.1.2, §4.2, §4.3. In the copy-and-mark format the output must reproduce the input text exactly, since spans are recovered by matching text 2304.10428 §4.1.2.

A tagger can also serve a privacy goal. It finds personal details in training text so that they can be masked before training. Tagger misses let some personal details through, and the same kind of tagger labels the generated text when leakage is measured 2302.00539 §II-C 2302.00539 §V.

Variants

Character-level neural CRF

Each word is read by a character LSTM, and its final state is joined with a word embedding. A sentence-level BiLSTM then gives each position a context vector. A CRF layer scores the tag transitions and the tag outputs, and all parameters are learned jointly 2404.09383 §2.3 2404.09383 eq. 3 2404.09383 eq. 5. The CRF normalises over all tag sequences of a sentence, so the tags are scored jointly rather than word by word 2404.09383 §2.1. With 10,000 labelled target sentences, it beats the log-linear CRF on all five monolingual targets, by 0.61 to 4.79 F1 2404.09383 tab. 2. Card: Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields.

Cross-lingual neural CRF with a related source language

The target and a related source language share the character encoder, the BiLSTM, the tag layers and the transition weights. Each language keeps its own word embeddings and a learned language vector 2404.09383 §3.1. Training sums the target log-likelihood and μ times the source log-likelihood, with μ > 0 2404.09383 eq. 7. With 100 target sentences, transfer raises F1 over the log-linear CRF in each of the ten listed pairs, by 3.97 to 9.80 points 2404.09383 tab. 2. For the same neural model, Marathi rises from 39.02 with no source to 58.92 with Urdu as the source 2404.09383 tab. 2. The authors describe the transfer effect with 10,000 target sentences as little to no effect 2404.09383 §5.2.

Transition-based tagger in a biomedical pipeline

The NER component is transition-based. It follows the chunking model of Lample et al., and each token is represented by embedded prefix, suffix, shape and lemma features 1902.07669 §4. The core model finds mentions and does not type them. It is trained on the mention spans of MedMentions [arxiv:1902.07669 §4.1]. Four typed packages each cover one set of labels and are trained on one dataset 1902.07669 §4.1. On nine NER datasets, the md model's F1 is above the Baseline column, which uses single models without extra resources, on six of them, and below it on three 1902.07669 tab. 5. The md package takes 33 ms per abstract on a 12-core CPU, and the sm package takes 32 ms. The speed table compares tools that do different tasks, so it is not a like-for-like comparison 1902.07669 tab. 2. Card: ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing.

Pretrained German encoder fine-tuned for NER

GottBERT is a RoBERTa-style encoder pretrained only on German text, in base and large sizes 2012.02110 §3. For NER it is fine-tuned on CoNLL 2003 (German) and on GermEval 2014, whose labels are finer and can nest 2012.02110 §3. Each task is tried with 24 settings of learning rate and batch size. The run with the best dev score is reported on test 2012.02110 §3 2012.02110 tab. 5. Among the base models, the filtered GottBERT models are best on both NER tasks. They score 87.59 F1 on GermEval 2014 (last checkpoint) and 86.14 on CoNLL 03 (best checkpoint), against 87.24 and 85.16 for GBERT base 2012.02110 tab. 3. Among the large models, no GottBERT model is best on either task. GBERT large leads CoNLL 03 and XLM-R large leads GermEval 2014 2012.02110 tab. 3. Filtering the pretraining corpus changed the best-checkpoint scores by at most 0.85 points 2012.02110 §4. Each pretraining was run once 2012.02110 §5. Card: GottBERT: a pure German Language Model.

Word, character and affix embeddings in a recurrent tagger

The survey's best neural family reads each word three ways. A word embedding is joined with a character-level summary of the word, and a sentence-level recurrent layer scores the labels. The affix variant adds learned embeddings for frequent prefixes and suffixes 1910.11470 §6.4.3 1910.11470 §6.4.4. The survey gives no threshold for "frequent" and no n-gram size, and it does not say how the three parts are combined; the joining is an inference from its figure 1910.11470 §6.4.4. The affix model is the Yadav et al. (2018) system, and the survey copies its scores from that paper 1910.11470 tab. 1. Card: A Survey on Recent Advances in Named Entity Recognition from Deep Learning models.

Cascaded start, end and type tagger for scientific entities

The tagger encodes each token with SciBERT. Two separate linear heads predict whether a token starts or ends an entity. A BiLSTM over the encoder output feeds a third head, which gives each token a type with an O class for non-entities 2606.29836 §3.2 2606.29836 eq. 1 2606.29836 eq. 2 2606.29836 eq. 6 2606.29836 eq. 7. The training loss is the unweighted sum of the three cross-entropy terms 2606.29836 eq. 9. The paper does not describe how the three outputs are decoded into spans 2606.29836 §3.2. The entity types are method, dataset, metric and tool, in ACL, EMNLP and NAACL text, split 8:1:1 into train, validation and test 2606.29836 §3.1 (3) 2606.29836 tab. 5. On the authors' 311-sentence test split, the cascade reaches 86.49 F1 against 86.04 for a plain SciBERT tagger 2606.29836 tab. 7. Each model is run once 2606.29836 §4.1. On SciERC the same model scores 69.14 F1, below 70.91 for a hierarchical transformer 2606.29836 tab. 8. Card: Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective.

Domain continued pretraining, then a tagger head

MatSciBERT starts from SciBERT's weights and keeps training with masked-word prediction on about 285M words of materials-science text. The corpus was built from 153,978 papers 2109.15290 §2.1 2109.15290 tab. 1. The authors trained it for about 10 days on two V100 GPUs 2109.15290 §2.2. The NER head is one of three: a linear softmax, a CRF, or a 2-layer BiLSTM with a CRF. Each word takes the encoder output of its first word piece 2109.15290 §2.4.1. On the SOFC-Slot test set, the BiLSTM-CRF head reaches 65.92 Macro-F1 with MatSciBERT against 61.68 with SciBERT and the same head 2109.15290 tab. 2. The gap is smaller on Matscholar. There, the BiLSTM-CRF Macro-F1 is tied at 85.66 2109.15290 tab. 5. Test scores average three seeds, and the paper gives no spread 2109.15290 §2.4.4. Card: MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction.

SciBERT-BiLSTM tagger for measurement studies

The tagger is a SciBERT encoder with a BiLSTM over it. It labels four entity types in NLP full text: method, dataset, tool and metric 2606.31058 §3.3.1 2606.31058 tab. 4. The training data are sentences from 50 ACL papers, 3,115 in all, annotated by five people 2606.31058 §3.3.1. On the authors' own test split it reaches 86.27 F1, against 85.65 for the best baseline, SciBERT or SciBERT+CRF 2606.31058 tab. 5. The paper does not state inter-annotator agreement 2606.31058 §3.3.1. In the paper the tagger is a means. It runs over 14,812 papers to produce 151,018 entities for a novelty measure 2606.31058 §3.3 2606.31058 tab. 6. Card: Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities.

Token-level data augmentation for taggers

The four methods build new labelled sentences from existing ones, with no external model 2010.11683 §3. Each one applies with a probability p:

• Label-wise token replacement: a token is swapped for another token that already carries the same full BIO tag, so the label sequence is unchanged 2010.11683 §3.
• Synonym replacement: a token is swapped for a WordNet synonym. When the token began a mention, the extra words of a multi-word synonym take I- tags 2010.11683 §3.
• Mention replacement: a whole mention is swapped for another training mention of the same type, and its BIO tags are rewritten for the new length 2010.11683 §3.
• Shuffle within segments: the tokens inside each mention, or inside each run of non-entity tokens, are reordered while the label row stays in place 2010.11683 §3.

The probability and the number of augmented copies are searched on the dev set 2010.11683 §4.3. The gains are concentrated in small training sets. With 50 sentences, each single method is significant on both data sets, for both the LSTM and the SciBERT tagger 2010.11683 tab. 3 2010.11683 §4.4. With all four methods, SciBERT on i2b2-2010 rises from 35.1 to 41.5 F1 2010.11683 tab. 3. On the full training set the gains are not reliable (see Contradictions). Only the SciBERT rows can be reproduced from the authors' release (see the card's Paper vs code section). Card: An Analysis of Simple Data Augmentation for Named Entity Recognition.

Model-based augmentation with a tagger check

The authors avoid edit-based augmentation, which they say can change entity spans 2606.29836 §3.3. Instead, sentences from unannotated abstracts that contain a known entity become candidates. A tagger trained on the training set labels each candidate, and a candidate is kept only if the tagger finds every known entity it matches. Kept sentences are added until the augmented set is as large as the training set 2606.29836 §3.3. On the authors' test split, the augmented cascade reaches 87.00 F1, against 86.49 without augmentation 2606.29836 tab. 7. That is one run, so the 0.51-point gain is not shown to exceed run-to-run variation 2606.29836 §4.1. The card's Paper vs code section notes that the released repository has no script for this step. Card: Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective.

Copy-and-mark generation with per-type prompts

Each prompt holds a task description for one entity type, the demonstrations, and the new sentence. The model copies the sentence and wraps every entity in @@ and ## markers. A sentence with no entity is copied unchanged 2304.10428 §4.1.2 2304.10428 fig. 1. This is the core of the method. A tag-per-word format and an entity-plus-position format were tried instead and scored far lower in the authors' ablation, so the marker format is the one to copy 2304.10428 §6.1. Card: GPT-NER: Named Entity Recognition via Large Language Models.

Demonstration retrieval

The authors compare three ways to pick the k solved examples 2304.10428 §4.2:

• Random: k examples drawn from the training set. Nothing ties them to the input 2304.10428 §4.2.1.
• Sentence-level kNN: the training sentences nearest to the input under sentence embeddings (SimCSE) 2304.10428 §4.2.2.
• Entity-level kNN: a fine-tuned NER model first tags the input. For each predicted entity, the nearest stored entity embeddings are retrieved, and their sentences become the demonstrations. This is the best variant in every full-data setting reported 2304.10428 §4.2.2 2304.10428 tab. 2 2304.10428 tab. 3.

Self-verification

After extraction, each candidate span goes through a second prompt that asks whether the span is an entity of the given type. The answer is "yes" or "no", and a "no" removes the span's markers 2304.10428 §4.3 2304.10428 fig. 3. The paper runs it zero-shot (ZS) or few-shot (FS). Its demonstrations reuse the entity-level retrieval, labelled "yes" when the retrieved entity has the queried type 2304.10428 §4.3. The gain is small compared with the gain from retrieval 2304.10428 tab. 2.

LLM pre-labelling with expert refinement

Here the LLM does not decide the final labels. It drafts them, and an expert corrects the draft. For medication fields in discharge summaries (name, dose, mode, frequency and duration), PaLM 2 was given a few worked examples and one chunk of text at a time, without any tuning 2312.02296 §5. Its output is parsed into structured entries 2312.02296 app. A.1. Two prompt schemas run separately: IOB tags per token with a group tag, and grouped YAML output with character positions 2312.02296 §5.3. The final draft merges the best configuration of each by span overlap 2312.02296 §5.4 2312.02296 app. A.4. Prompt versions and chunk sizes were chosen on a development set, with a score that weights recall above precision 2312.02296 §5.4 2312.02296 app. A.3. On 216 test documents, refinement of the LLM draft reaches phrase-level vertical F1 0.907, against 0.910 for refinement of human draft labels. The paper finds no significant difference between the two 2312.02296 tab. 2 2312.02296 tab. 4. Refinement time falls from 26.67 to 11.32 minutes per document 2312.02296 tab. 2. LLM compute time is not included in that figure, and the authors say it cannot be compared with rater time 2312.02296 app. B. Card: LLMs Accelerate Annotation for Medical Information Extraction.

Zero-shot redaction with a chat model

A chat model gets a system prompt with a task, a command to replace the listed identifiers with "[redacted]", and rules with examples. The note goes in the user message, and the reply is the redacted note. There are no examples and no training 2303.11032 §IV 2303.11032 fig. 7 2303.11032 fig. 8. The full prompt appears only as a figure. Prompt wording matters. On the 2014 i2b2/UTHealth test set, ChatGPT scores 0.686 with an implicit prompt and 0.929 with an explicit one 2303.11032 tab. 2. GPT-4 scores 0.99 with the explicit prompt, but the paper ran its 50 test notes by hand in the web interface, and gives no model version or date 2303.11032 §V. Fine-tuned ClinicalBERT scores 0.974 on the full test set 2303.11032 tab. 2. The score is the share of gold identifiers removed, not span F1. The paper's formula for it does not match its text 2303.11032 eq. 1 2303.11032 §V. The authors also point out that sending a note to an outside API passes patient data to a third party, and that hospitals need local models 2303.11032 §VI. Card: DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4.

Tagger-based scrubbing of training text

Each span the tagger finds is replaced by a mask token before training 2302.00539 alg. 1. The paper runs Flair's OntoNotes tagger together with a second tagger, and gives the Flair model's OntoNotes F1 as 90.93 (author-reported) 2302.00539 §IV-B. Scrubbing is imperfect. The tagger misses some personal information, and the mask still shows where a missed span sat 2302.00539 §II-C 2302.00539 §IV-B. Each replacement finds its span by splitting the sentence at the tagged text. This assumes the text occurs only once in the sentence 2302.00539 §II-C. On the ECHR sentence-level membership-inference test, the AUC is 0.96 undefended, 0.82 after scrubbing, and 0.505 with DP and scrubbing 2302.00539 §IV-H. That test measures whether a sentence was in the training data, not how many names leak, so it does not give a leakage rate. The same tagger labels the generated text, so its misses count as no leakage, and the paper's leakage figures are lower bounds 2302.00539 §V. Card: Analyzing Leakage of Personally Identifiable Information in Language Models.

When to use which

The numbers are author-reported. Each row's numbers come from one paper, and papers use different datasets, models and protocols, so numbers should not be compared across rows that cite different papers.

Implementation sketch

Two sketches follow, one per model family. Each is our own pseudocode of the paper's description, with the names chosen by us. The augmentation and pre-labelling variants are described in prose only.

The CRF sketch covers one training step. The paper gives the objective but not the batching, so the way the two batches are mixed is open 2404.09383 eq. 7.

# One training step, cross-lingual neural CRF  [arxiv:2404.09383 §3.1]
# Input:  target and source batches of (words, tags, language); mu (float)
# Output: scalar loss to minimise

def training_loss(target_batch, source_batch, mu):
    loss = 0.0
    # Target sentences enter with weight 1  [eq. 7]
    for words, tags, lang in target_batch:
        loss -= crf_log_likelihood(words, tags, lang)
    # Source sentences enter with weight mu > 0, which the paper does not set  [eq. 7]
    for words, tags, lang in source_batch:
        loss -= mu * crf_log_likelihood(words, tags, lang)
    return loss

The LLM sketch is the loop below. It follows the paper's description: one prompt per entity type, then a verification prompt per candidate span 2304.10428 §4. The paper gives no algorithm box.

# GPT-NER for one sentence, one entity type at a time  [arxiv:2304.10428 §4]
# Input:  words (list of str); label_types (list of str); k (int)
# Output: list of (span, label) pairs

def gpt_ner(words, label_types, k):
    spans = []
    for label in label_types:
        # Demonstrations: entity-level kNN, or random  [§4.2]
        demos = retrieve_demonstrations(words, label, k)
        # Instruction for this type, demonstrations, then the input  [§4.1]
        prompt = build_prompt(label, demos, words)
        # Output copies the sentence, marking entities with @@ and ##  [§4.1.2]
        completion = complete(prompt)
        candidates = parse_marked_spans(completion, words)
        for span in candidates:
            # Self-verification: keep only spans the model confirms  [§4.3]
            demos_v = retrieve_demonstrations(span, label, k)
            verdict = verify(words, span, label, demos_v)
            if verdict == "yes":
                spans.append((span, label))
    return spans

Where the LLM variants differ: the retrieval step (random, sentence-level or entity-level), whether verify runs, and the number of demonstrations k. The paper gives no k for its main results, and it reports that F1 still rose at the largest k that fits the context window 2304.10428 §5.1 2304.10428 §6.2. When the retrieved examples contain no entity, kNN retrieval did worse than random at k = 2 and k = 4 2304.10428 §6.2.

For the CRF variants, the paper does not say how the BiLSTM output maps to the 100-dimensional contextual representation it reports, and it does not state the character LSTM size, the language-vector size or μ 2404.09383 §2.3 fn. 1 2404.09383 §3.1. Choose these on your own validation data.

Contradictions

• Text and table disagree (GPT-NER). The text gives 84.97 against 85.91 for the CoNLL2003 few-shot comparison, but neither pair matches a row in 2304.10428 tab. 2, where the FS + entity value is 90.91 2304.10428 §5.1.1. On GENIA the text gives 56.68 against 69.06, while 2304.10428 tab. 3 gives 51.68 for sentence-level and 64.06 for entity-level 2304.10428 §5.1.2.
• Reported F1 is not the harmonic mean of its P and R (GPT-NER). For ACE2005 with entity-level ZS, P 72.63 and R 75.39 give about 73.99 F1, but the table reports 73.46 2304.10428 tab. 3. The same check fails for the FS row. Recompute from counts before relying on these rows.
• Text and table disagree (augmentation). The text says all methods beat the baseline on average 2010.11683 §4.4. On the full training set, several single-method cells are equal to or below the baseline, for example LSTM on MaSciP (76.3 and 76.0 against 76.4) and SciBERT on i2b2-2010 (87.5, 87.7, 87.7 and 87.2 against 87.8) 2010.11683 tab. 3.
• Biomedical gaps differ between the LLM papers. GPT-NER reports 64.42 F1 on GENIA, a biomedical nested set, against 83.75 for BERT-MRC 2304.10428 tab. 3. The benchmark reports prompted GPT-4 at 0.7979 to 0.8327 F1 on BC5CDR-chemical, against 0.9500 for a supervised baseline 2305.16326 tab. 3. The two papers use different datasets, models and prompt protocols, so this is not a direct disagreement. Measure the gap on your own domain.
• Text and table disagree (GottBERT). The NER paragraph gives 86.78 for the filtered large "last" model on CoNLL. The table gives 86.28 for that row. The value 86.78 matches the unfiltered large model and GELECTRA large in the table 2012.02110 §4 2012.02110 tab. 3.
• Text and table disagree (scispaCy). The authors describe the NER models as competitive with baselines on 5 of 9 datasets 1902.07669 §4.2. In their table, the md model is above the Baseline column on six rows: BC5CDR, JNLPBA, BioNLP13CG, BC4CHEMD, Linnaeus and NCBI-Disease. "Competitive" is not defined in the paper, so the text and the table cannot be checked against each other 1902.07669 tab. 5.
• Survey text and table disagree. The survey gives a 0.52-point English gain for Chiu and Nichols over the best word-based model. In its Table 1, the Chiu and Nichols score minus the Yang et al. score is 0.36 1910.11470 §7 1910.11470 tab. 1.
• No conflicting claims across the papers. The papers use different datasets, languages and models, so their results answer different questions. None reports a result that another contradicts.

Papers

• GPT-NER: Named Entity Recognition via Large Language Models: copy-and-mark prompting with entity-level kNN demonstrations and yes/no self-verification; reports 90.91 F1 on full CoNLL2003 and 82.20 on OntoNotes5.0 (author-reported).
• Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields: character-level neural CRF with a cross-lingual variant that shares most layers across languages; reports up to 9.8 F1 over the log-linear CRF with 100 target sentences (author-reported).
• An Analysis of Simple Data Augmentation for Named Entity Recognition: four token-level augmentations for NER; reports gains of up to 9.8 F1 on 50-sentence sets and mixed gains on the full sets (author-reported).
• Benchmarking large language models for biomedical natural language processing applications and recommendations: twelve biomedical tasks, including two NER sets, for four LLMs under zero-shot, few-shot and fine-tuned settings; supervised baselines lead on NER (author-reported).
• LLMs Accelerate Annotation for Medical Information Extraction: few-shot PaLM 2 drafts medication labels that experts refine; refined quality is close to the human-draft pipeline and rater time falls by 58% (author-reported).
• A Survey on Recent Advances in Named Entity Recognition from Deep Learning models: survey of 83 NER systems, grouped by family and compared on CoNLL 2002/2003 and DrugNER; the word, character and affix tagger has the best reported scores in three of four CoNLL languages (author-reported, copied from cited papers).
• ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing: spaCy-based biomedical pipeline with a transition-based NER component and typed NER packages; reports NER competitive with baselines on 5 of 9 datasets (author-reported).
• GottBERT: a pure German Language Model: German-only RoBERTa base and large, with a filtered and an unfiltered corpus; base models lead on GermEval 2014 and CoNLL 03 (author-reported).
• DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4: prompt-only removal of identifiers from clinical notes; reports 0.929 for ChatGPT with an explicit prompt and 0.99 for GPT-4 on 50 hand-run notes (author-reported).
• Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective: SciBERT-based cascade tagger for method, dataset, metric and tool entities, with model-based augmentation; reports 87.00 F1 on its own test split (author-reported).
• Analyzing Leakage of Personally Identifiable Information in Language Models: tagger-based scrubbing of GPT-2 training data, with DP-SGD and reconstruction and inference attacks; reports that scrubbing and DP cut but do not remove leakage (author-reported).
• MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction: SciBERT continued on materials text, with linear, CRF and BiLSTM-CRF NER heads; reports higher scores than SciBERT on nearly every test cell, with one tie (author-reported).
• Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities: SciBERT-BiLSTM tagger for four entity types in NLP papers, used to build a novelty score; reports 86.27 F1 on its own test split (author-reported).