--- kind: concept slug: named-entity-recognition title: Named entity recognition summary: >- Finding and typing entity mentions such as people or organisations. Covers trained taggers (CRF, encoders and domain encoders), token and model-based augmentation, LLM prompting for spans and PHI redaction, LLM pre-labelling for experts, and tagger-based scrubbing of training text. Trade-off: prompting suits small labelled sets but trails supervised taggers on benchmarks and costs API calls. tags: [named-entity-recognition, information-extraction, sequence-labeling, cross-lingual-transfer, low-resource, data-augmentation, annotation, llm, in-context-learning, few-shot, de-identification, biomedical, german, privacy, materials-science, domain-adaptation, scientometrics] papers: ["2010.11683", "2304.10428", "2305.16326", "2312.02296", "2404.09383", "1902.07669", "2012.02110", "2303.11032", "2606.29836", "1910.11470", "2302.00539", "2109.15290", "2606.31058"] related: [sequence-labeling, conditional-random-field, transfer-learning, data-augmentation, recurrent-neural-network, language-model-pretraining, masked-language-modeling, in-context-learning, prompt-engineering, few-shot-learning, hallucination-mitigation, llm-self-evaluation, llm-evaluation, retrieval-augmented-generation, text-de-identification, differentially-private-sgd, membership-inference, domain-adaptation, bibliometric-analysis] status: draft updated_at: 2026-10-31 --- ## Overview Named entity recognition (NER) finds the spans of a sentence that name an entity and gives each span a type such as person, location or organisation. The classic formulation is sequence labelling: each word gets a tag, usually in BIO form, and a model scores the whole tag sequence [arxiv:2304.10428 §3] [arxiv:2404.09383 §2]. The variants fall into three groups. The first trains a tagger on labelled sentences. Options include a CRF that can borrow labels from a related language [arxiv:2404.09383 §2.3, §3.1], a biomedical pipeline with a transition-based tagger [arxiv:1902.07669 §4], a German encoder fine-tuned on NER [arxiv:2012.02110 §3], a SciBERT-based cascade for scientific entities [arxiv:2606.29836 §3.2], a SciBERT encoder continued on materials text with a CRF or BiLSTM-CRF head [arxiv:2109.15290 §2.4.1], and a SciBERT-BiLSTM tagger for method, dataset, tool and metric mentions [arxiv:2606.31058 §3.3.1]. The second group works on the data. It uses token-level edits to a small labelled set [arxiv:2010.11683 §3], keeps only generated sentences that a tagger confirms [arxiv:2606.29836 §3.3], or lets an LLM pre-label text that experts then correct [arxiv:2312.02296 §4] [arxiv:2312.02296 §5]. The third group asks an LLM for the labels directly. For span labelling, the model copies the input and marks each entity, with one prompt per entity type [arxiv:2304.10428 §4.1.1]. For de-identification, it replaces identifiers with a placeholder [arxiv:2303.11032 §IV]. The LLM prompting variants differ in the output format, how the in-prompt examples (demonstrations) are chosen, and whether the extracted spans are checked by a second prompt [arxiv:2304.10428 §4.1.2, §4.2, §4.3]. In the copy-and-mark format the output must reproduce the input text exactly, since spans are recovered by matching text [arxiv:2304.10428 §4.1.2]. A tagger can also serve a privacy goal. It finds personal details in training text so that they can be masked before training. Tagger misses let some personal details through, and the same kind of tagger labels the generated text when leakage is measured [arxiv:2302.00539 §II-C] [arxiv:2302.00539 §V]. ## Variants ### Character-level neural CRF Each word is read by a character LSTM, and its final state is joined with a word embedding. A sentence-level BiLSTM then gives each position a context vector. A CRF layer scores the tag transitions and the tag outputs, and all parameters are learned jointly [arxiv:2404.09383 §2.3] [arxiv:2404.09383 eq. 3] [arxiv:2404.09383 eq. 5]. The CRF normalises over all tag sequences of a sentence, so the tags are scored jointly rather than word by word [arxiv:2404.09383 §2.1]. With 10,000 labelled target sentences, it beats the log-linear CRF on all five monolingual targets, by 0.61 to 4.79 F1 [arxiv:2404.09383 tab. 2]. Card: [Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields](/papers/2404.09383). ### Cross-lingual neural CRF with a related source language The target and a related source language share the character encoder, the BiLSTM, the tag layers and the transition weights. Each language keeps its own word embeddings and a learned language vector [arxiv:2404.09383 §3.1]. Training sums the target log-likelihood and μ times the source log-likelihood, with μ > 0 [arxiv:2404.09383 eq. 7]. With 100 target sentences, transfer raises F1 over the log-linear CRF in each of the ten listed pairs, by 3.97 to 9.80 points [arxiv:2404.09383 tab. 2]. For the same neural model, Marathi rises from 39.02 with no source to 58.92 with Urdu as the source [arxiv:2404.09383 tab. 2]. The authors describe the transfer effect with 10,000 target sentences as little to no effect [arxiv:2404.09383 §5.2]. ### Transition-based tagger in a biomedical pipeline The NER component is transition-based. It follows the chunking model of Lample et al., and each token is represented by embedded prefix, suffix, shape and lemma features [arxiv:1902.07669 §4]. The core model finds mentions and does not type them. It is trained on the mention spans of MedMentions [arxiv:1902.07669 §4.1]. Four typed packages each cover one set of labels and are trained on one dataset [arxiv:1902.07669 §4.1]. On nine NER datasets, the md model's F1 is above the Baseline column, which uses single models without extra resources, on six of them, and below it on three [arxiv:1902.07669 tab. 5]. The md package takes 33 ms per abstract on a 12-core CPU, and the sm package takes 32 ms. The speed table compares tools that do different tasks, so it is not a like-for-like comparison [arxiv:1902.07669 tab. 2]. Card: [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](/papers/1902.07669). ### Pretrained German encoder fine-tuned for NER GottBERT is a RoBERTa-style encoder pretrained only on German text, in base and large sizes [arxiv:2012.02110 §3]. For NER it is fine-tuned on CoNLL 2003 (German) and on GermEval 2014, whose labels are finer and can nest [arxiv:2012.02110 §3]. Each task is tried with 24 settings of learning rate and batch size. The run with the best dev score is reported on test [arxiv:2012.02110 §3] [arxiv:2012.02110 tab. 5]. Among the base models, the filtered GottBERT models are best on both NER tasks. They score 87.59 F1 on GermEval 2014 (last checkpoint) and 86.14 on CoNLL 03 (best checkpoint), against 87.24 and 85.16 for GBERT base [arxiv:2012.02110 tab. 3]. Among the large models, no GottBERT model is best on either task. GBERT large leads CoNLL 03 and XLM-R large leads GermEval 2014 [arxiv:2012.02110 tab. 3]. Filtering the pretraining corpus changed the best-checkpoint scores by at most 0.85 points [arxiv:2012.02110 §4]. Each pretraining was run once [arxiv:2012.02110 §5]. Card: [GottBERT: a pure German Language Model](/papers/2012.02110). ### Word, character and affix embeddings in a recurrent tagger The survey's best neural family reads each word three ways. A word embedding is joined with a character-level summary of the word, and a sentence-level recurrent layer scores the labels. The affix variant adds learned embeddings for frequent prefixes and suffixes [arxiv:1910.11470 §6.4.3] [arxiv:1910.11470 §6.4.4]. The survey gives no threshold for "frequent" and no n-gram size, and it does not say how the three parts are combined; the joining is an inference from its figure [arxiv:1910.11470 §6.4.4]. The affix model is the Yadav et al. (2018) system, and the survey copies its scores from that paper [arxiv:1910.11470 tab. 1]. Card: [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](/papers/1910.11470). ### Cascaded start, end and type tagger for scientific entities The tagger encodes each token with SciBERT. Two separate linear heads predict whether a token starts or ends an entity. A BiLSTM over the encoder output feeds a third head, which gives each token a type with an O class for non-entities [arxiv:2606.29836 §3.2] [arxiv:2606.29836 eq. 1] [arxiv:2606.29836 eq. 2] [arxiv:2606.29836 eq. 6] [arxiv:2606.29836 eq. 7]. The training loss is the unweighted sum of the three cross-entropy terms [arxiv:2606.29836 eq. 9]. The paper does not describe how the three outputs are decoded into spans [arxiv:2606.29836 §3.2]. The entity types are method, dataset, metric and tool, in ACL, EMNLP and NAACL text, split 8:1:1 into train, validation and test [arxiv:2606.29836 §3.1 (3)] [arxiv:2606.29836 tab. 5]. On the authors' 311-sentence test split, the cascade reaches 86.49 F1 against 86.04 for a plain SciBERT tagger [arxiv:2606.29836 tab. 7]. Each model is run once [arxiv:2606.29836 §4.1]. On SciERC the same model scores 69.14 F1, below 70.91 for a hierarchical transformer [arxiv:2606.29836 tab. 8]. Card: [Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective](/papers/2606.29836). ### Domain continued pretraining, then a tagger head MatSciBERT starts from SciBERT's weights and keeps training with masked-word prediction on about 285M words of materials-science text. The corpus was built from 153,978 papers [arxiv:2109.15290 §2.1] [arxiv:2109.15290 tab. 1]. The authors trained it for about 10 days on two V100 GPUs [arxiv:2109.15290 §2.2]. The NER head is one of three: a linear softmax, a CRF, or a 2-layer BiLSTM with a CRF. Each word takes the encoder output of its first word piece [arxiv:2109.15290 §2.4.1]. On the SOFC-Slot test set, the BiLSTM-CRF head reaches 65.92 Macro-F1 with MatSciBERT against 61.68 with SciBERT and the same head [arxiv:2109.15290 tab. 2]. The gap is smaller on Matscholar. There, the BiLSTM-CRF Macro-F1 is tied at 85.66 [arxiv:2109.15290 tab. 5]. Test scores average three seeds, and the paper gives no spread [arxiv:2109.15290 §2.4.4]. Card: [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](/papers/2109.15290). ### SciBERT-BiLSTM tagger for measurement studies The tagger is a SciBERT encoder with a BiLSTM over it. It labels four entity types in NLP full text: method, dataset, tool and metric [arxiv:2606.31058 §3.3.1] [arxiv:2606.31058 tab. 4]. The training data are sentences from 50 ACL papers, 3,115 in all, annotated by five people [arxiv:2606.31058 §3.3.1]. On the authors' own test split it reaches 86.27 F1, against 85.65 for the best baseline, SciBERT or SciBERT+CRF [arxiv:2606.31058 tab. 5]. The paper does not state inter-annotator agreement [arxiv:2606.31058 §3.3.1]. In the paper the tagger is a means. It runs over 14,812 papers to produce 151,018 entities for a novelty measure [arxiv:2606.31058 §3.3] [arxiv:2606.31058 tab. 6]. Card: [Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities](/papers/2606.31058). ### Token-level data augmentation for taggers The four methods build new labelled sentences from existing ones, with no external model [arxiv:2010.11683 §3]. Each one applies with a probability p: - **Label-wise token replacement:** a token is swapped for another token that already carries the same full BIO tag, so the label sequence is unchanged [arxiv:2010.11683 §3]. - **Synonym replacement:** a token is swapped for a WordNet synonym. When the token began a mention, the extra words of a multi-word synonym take I- tags [arxiv:2010.11683 §3]. - **Mention replacement:** a whole mention is swapped for another training mention of the same type, and its BIO tags are rewritten for the new length [arxiv:2010.11683 §3]. - **Shuffle within segments:** the tokens inside each mention, or inside each run of non-entity tokens, are reordered while the label row stays in place [arxiv:2010.11683 §3]. The probability and the number of augmented copies are searched on the dev set [arxiv:2010.11683 §4.3]. The gains are concentrated in small training sets. With 50 sentences, each single method is significant on both data sets, for both the LSTM and the SciBERT tagger [arxiv:2010.11683 tab. 3] [arxiv:2010.11683 §4.4]. With all four methods, SciBERT on i2b2-2010 rises from 35.1 to 41.5 F1 [arxiv:2010.11683 tab. 3]. On the full training set the gains are not reliable (see Contradictions). Only the SciBERT rows can be reproduced from the authors' release (see the card's Paper vs code section). Card: [An Analysis of Simple Data Augmentation for Named Entity Recognition](/papers/2010.11683). ### Model-based augmentation with a tagger check The authors avoid edit-based augmentation, which they say can change entity spans [arxiv:2606.29836 §3.3]. Instead, sentences from unannotated abstracts that contain a known entity become candidates. A tagger trained on the training set labels each candidate, and a candidate is kept only if the tagger finds every known entity it matches. Kept sentences are added until the augmented set is as large as the training set [arxiv:2606.29836 §3.3]. On the authors' test split, the augmented cascade reaches 87.00 F1, against 86.49 without augmentation [arxiv:2606.29836 tab. 7]. That is one run, so the 0.51-point gain is not shown to exceed run-to-run variation [arxiv:2606.29836 §4.1]. The card's Paper vs code section notes that the released repository has no script for this step. Card: [Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective](/papers/2606.29836). ### Copy-and-mark generation with per-type prompts Each prompt holds a task description for one entity type, the demonstrations, and the new sentence. The model copies the sentence and wraps every entity in `@@` and `##` markers. A sentence with no entity is copied unchanged [arxiv:2304.10428 §4.1.2] [arxiv:2304.10428 fig. 1]. This is the core of the method. A tag-per-word format and an entity-plus-position format were tried instead and scored far lower in the authors' ablation, so the marker format is the one to copy [arxiv:2304.10428 §6.1]. Card: [GPT-NER: Named Entity Recognition via Large Language Models](/papers/2304.10428). ### Demonstration retrieval The authors compare three ways to pick the k solved examples [arxiv:2304.10428 §4.2]: - **Random:** k examples drawn from the training set. Nothing ties them to the input [arxiv:2304.10428 §4.2.1]. - **Sentence-level kNN:** the training sentences nearest to the input under sentence embeddings (SimCSE) [arxiv:2304.10428 §4.2.2]. - **Entity-level kNN:** a fine-tuned NER model first tags the input. For each predicted entity, the nearest stored entity embeddings are retrieved, and their sentences become the demonstrations. This is the best variant in every full-data setting reported [arxiv:2304.10428 §4.2.2] [arxiv:2304.10428 tab. 2] [arxiv:2304.10428 tab. 3]. ### Self-verification After extraction, each candidate span goes through a second prompt that asks whether the span is an entity of the given type. The answer is "yes" or "no", and a "no" removes the span's markers [arxiv:2304.10428 §4.3] [arxiv:2304.10428 fig. 3]. The paper runs it zero-shot (ZS) or few-shot (FS). Its demonstrations reuse the entity-level retrieval, labelled "yes" when the retrieved entity has the queried type [arxiv:2304.10428 §4.3]. The gain is small compared with the gain from retrieval [arxiv:2304.10428 tab. 2]. ### LLM pre-labelling with expert refinement Here the LLM does not decide the final labels. It drafts them, and an expert corrects the draft. For medication fields in discharge summaries (name, dose, mode, frequency and duration), PaLM 2 was given a few worked examples and one chunk of text at a time, without any tuning [arxiv:2312.02296 §5]. Its output is parsed into structured entries [arxiv:2312.02296 app. A.1]. Two prompt schemas run separately: IOB tags per token with a group tag, and grouped YAML output with character positions [arxiv:2312.02296 §5.3]. The final draft merges the best configuration of each by span overlap [arxiv:2312.02296 §5.4] [arxiv:2312.02296 app. A.4]. Prompt versions and chunk sizes were chosen on a development set, with a score that weights recall above precision [arxiv:2312.02296 §5.4] [arxiv:2312.02296 app. A.3]. On 216 test documents, refinement of the LLM draft reaches phrase-level vertical F1 0.907, against 0.910 for refinement of human draft labels. The paper finds no significant difference between the two [arxiv:2312.02296 tab. 2] [arxiv:2312.02296 tab. 4]. Refinement time falls from 26.67 to 11.32 minutes per document [arxiv:2312.02296 tab. 2]. LLM compute time is not included in that figure, and the authors say it cannot be compared with rater time [arxiv:2312.02296 app. B]. Card: [LLMs Accelerate Annotation for Medical Information Extraction](/papers/2312.02296). ### Zero-shot redaction with a chat model A chat model gets a system prompt with a task, a command to replace the listed identifiers with "[redacted]", and rules with examples. The note goes in the user message, and the reply is the redacted note. There are no examples and no training [arxiv:2303.11032 §IV] [arxiv:2303.11032 fig. 7] [arxiv:2303.11032 fig. 8]. The full prompt appears only as a figure. Prompt wording matters. On the 2014 i2b2/UTHealth test set, ChatGPT scores 0.686 with an implicit prompt and 0.929 with an explicit one [arxiv:2303.11032 tab. 2]. GPT-4 scores 0.99 with the explicit prompt, but the paper ran its 50 test notes by hand in the web interface, and gives no model version or date [arxiv:2303.11032 §V]. Fine-tuned ClinicalBERT scores 0.974 on the full test set [arxiv:2303.11032 tab. 2]. The score is the share of gold identifiers removed, not span F1. The paper's formula for it does not match its text [arxiv:2303.11032 eq. 1] [arxiv:2303.11032 §V]. The authors also point out that sending a note to an outside API passes patient data to a third party, and that hospitals need local models [arxiv:2303.11032 §VI]. Card: [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](/papers/2303.11032). ### Tagger-based scrubbing of training text Each span the tagger finds is replaced by a mask token before training [arxiv:2302.00539 alg. 1]. The paper runs Flair's OntoNotes tagger together with a second tagger, and gives the Flair model's OntoNotes F1 as 90.93 (author-reported) [arxiv:2302.00539 §IV-B]. Scrubbing is imperfect. The tagger misses some personal information, and the mask still shows where a missed span sat [arxiv:2302.00539 §II-C] [arxiv:2302.00539 §IV-B]. Each replacement finds its span by splitting the sentence at the tagged text. This assumes the text occurs only once in the sentence [arxiv:2302.00539 §II-C]. On the ECHR sentence-level membership-inference test, the AUC is 0.96 undefended, 0.82 after scrubbing, and 0.505 with DP and scrubbing [arxiv:2302.00539 §IV-H]. That test measures whether a sentence was in the training data, not how many names leak, so it does not give a leakage rate. The same tagger labels the generated text, so its misses count as no leakage, and the paper's leakage figures are lower bounds [arxiv:2302.00539 §V]. Card: [Analyzing Leakage of Personally Identifiable Information in Language Models](/papers/2302.00539). ## When to use which The numbers are author-reported. Each row's numbers come from one paper, and papers use different datasets, models and protocols, so numbers should not be compared across rows that cite different papers. | Situation | Use | Why | |---|---|---| | Thousands of labelled target sentences (about 10,000 in the paper) | Monolingual character-level neural CRF | Beats the log-linear CRF on all five targets, by 0.61 to 4.79 F1 [arxiv:2404.09383 tab. 2] | | About 100 labelled target sentences and a related language with labels | Cross-lingual neural CRF, with that language as source | Transfer beat the log-linear CRF in all ten listed pairs, by 3.97 to 9.80 F1 [arxiv:2404.09383 tab. 2]. The paper gives no rule for choosing the source [arxiv:2404.09383 §5.1] | | About 100 labelled target sentences and no related labelled language | Log-linear CRF with standard features | Without transfer, it beat the neural CRF on all five targets [arxiv:2404.09383 tab. 2]. The paper tests no other low-resource monolingual model. The baseline features are not listed [arxiv:2404.09383 §2.2] | | Small labelled set (50 sentences in the paper) and a tagger you already train | Token-level augmentation, all four methods, with p and the copy count tuned on dev | Each single method is significant at 50 sentences. SciBERT on i2b2-2010 rises from 35.1 to 41.5 F1 with all methods [arxiv:2010.11683 tab. 3] | | Full labelled set and a tagger you already train | Augmentation is unlikely to help much; test on dev before keeping it | On the full set, several single-method cells are equal to or below the baseline. SciBERT on i2b2-2010 gives 87.2 with all methods against 87.8 without [arxiv:2010.11683 tab. 3] | | Labelled sentences exist and accuracy matters most (LLM route) | Entity-level kNN, with few-shot verification | Best row in every full-data table: CoNLL2003 90.91 F1 against 93.04 for BERT-MRC [arxiv:2304.10428 tab. 2] | | Only a few labelled sentences, 8 to about 100 (LLM route) | Entity-level kNN, zero-shot verification | With 8 training sentences the reported F1 is about 60, where the supervised baseline is near 0. Learning curves are only in a figure [arxiv:2304.10428 §5.2.1] [arxiv:2304.10428 fig. 4] | | No labelled data at all (LLM route) | Not tested in the paper | Every retrieval variant uses the training set as a datastore [arxiv:2304.10428 §4.2.2] | | Nested entities, ACE and GENIA (LLM route) | Entity-level kNN, but expect a large gap | 73.59 F1 on ACE2005 and 64.42 on GENIA, against 89.5 for BINDER and 83.75 for BERT-MRC [arxiv:2304.10428 tab. 3] | | Budget is tight (LLM route) | Entity-level kNN without verification, after counting calls | Verification adds 0.94 F1 on CoNLL2003 full, from 89.97 to 90.91 [arxiv:2304.10428 tab. 2]. Calls scale with sentences times entity types, plus one per verified span [arxiv:2304.10428 §4.1.1] [arxiv:2304.10428 §4.3]. The paper reports no call counts | | Entity-level retrieval but no fine-tuned NER model to embed entities (LLM route) | Not covered by the paper | The best variant needs a fine-tuned NER model to embed entities, and the paper does not name it [arxiv:2304.10428 §4.2.2] | | Long label sets (LLM route) | Check the prompt budget first | The authors cite a 4096-token limit for GPT-3, so the instructions and examples may not fit for every type at once [arxiv:2304.10428 §4.1.1] | | New domain, expert time is the bottleneck, experts can refine | LLM pre-labelling, then expert refinement | Refined LLM draft reaches 0.907 vertical F1 against 0.910 for refined human draft, a difference the paper finds not significant. Rater time falls from 26.67 to 11.32 minutes per document [arxiv:2312.02296 tab. 2] [arxiv:2312.02296 tab. 4]. Tested on medication fields only [arxiv:2312.02296 §3.1] | | Biomedical NER, no training, prompted GPT-4 | Expect a large gap; prefer a fine-tuned model if labels exist | Best prompted GPT-4 row: 0.8327 F1 on BC5CDR-chemical (1-shot) and 0.6389 on NCBI Disease (5-shot), against 0.9500 and 0.9090 for the cited supervised baselines. The baselines are copied from the literature, not re-run [arxiv:2305.16326 tab. 3] | | Biomedical NER with labelled data and GPUs | Fine-tune an open 13B model | Fine-tuned LLaMA 2 13B gets 0.9149 F1 on BC5CDR-chemical and 0.8682 on NCBI Disease, above every prompted row for those datasets [arxiv:2305.16326 tab. 3] | | Biomedical English, standard label sets, many abstracts on a CPU | scispaCy; md if you need word vectors (sm has none) | md is above the Baseline column on six of nine NER sets [arxiv:1902.07669 tab. 5], and takes about 33 ms per abstract on one CPU [arxiv:1902.07669 tab. 1] [arxiv:1902.07669 tab. 2]. The speed comparison mixes tasks, so do not read it as a ranking | | German text, base-size encoder, labelled NER data | GottBERT base, filtered or unfiltered | Best base model on GermEval 2014 (87.59) and CoNLL 03 (86.14) in the paper's table [arxiv:2012.02110 tab. 3]. Filtering moved best-checkpoint scores by at most 0.85 points [arxiv:2012.02110 §4]. Single pretraining run [arxiv:2012.02110 §5] | | German text, large encoder | Compare GBERT large and XLM-R large; GottBERT large was not best | GBERT large leads CoNLL 03 (87.19) and XLM-R large leads GermEval 2014 (88.83) [arxiv:2012.02110 tab. 3] | | Newswire text in Spanish, Dutch, English or German, no gazetteers, and you train a neural tagger yourself | Word, character and affix embeddings in a recurrent tagger | Survey table, 150 epochs: 87.26 (Spanish), 87.54 (Dutch), 79.01 (German) F1, the best row in each of these three languages. English is 90.86, below Agerri and Rigau at 91.36, which uses lexicons [arxiv:1910.11470 tab. 1]. Scores are copied from the cited papers, and the affix threshold is not stated [arxiv:1910.11470 §6.4.4] | | Materials or other specialised scientific text, a large unlabelled domain corpus, and about 10 days on two V100 GPUs | Continue pretraining SciBERT on the domain, then fine-tune a tagger head | BiLSTM-CRF head: 65.92 Macro-F1 on SOFC-Slot test against 61.68 for SciBERT [arxiv:2109.15290 tab. 2]. On Matscholar the gain is smaller and one Macro-F1 cell is tied [arxiv:2109.15290 tab. 5]. Hardware and duration [arxiv:2109.15290 §2.2] | | NLP full text, four entity types, labelled sentences from about 50 papers, and a test split to check against | SciBERT-BiLSTM tagger, as in the paper | 86.27 F1 on the authors' test split against 85.65 for the best baseline [arxiv:2606.31058 tab. 5]. Five annotators, agreement not stated [arxiv:2606.31058 §3.3.1] | | Scientific abstracts, typed method, dataset, metric and tool labels, about 2,500 training sentences, SciBERT available | Cascade tagger on SciBERT | 86.49 F1 against 86.04 for a plain SciBERT tagger on the authors' 311-sentence test split. One run [arxiv:2606.29836 tab. 7] [arxiv:2606.29836 §4.1]. On SciERC the same model scores 69.14, below 70.91 for a hierarchical transformer [arxiv:2606.29836 tab. 8] | | Same scientific setting, few labels, and you can run the tagger over unlabelled abstracts | Model-based augmentation with the every-entity check | 87.00 F1 against 86.49 without augmentation, on one run [arxiv:2606.29836 tab. 7] | | Clinical notes, identifiers must be removed, no outside API allowed | No result in the paper for this case; test a local model on your own notes | The authors say API-only models cannot be used in a hospital and name local models as needed [arxiv:2303.11032 §VI]. Open models in the table score 0.597 to 0.827 under zero-shot prompts, but their setup is not described in the text [arxiv:2303.11032 tab. 2] | | Clinical notes, external API allowed, labelled notes available | Fine-tuned ClinicalBERT or RoBERTa, rather than prompting | Fine-tuned ClinicalBERT scores 0.974 and RoBERTa 0.947, against 0.929 for ChatGPT with the explicit prompt, all on the full test set [arxiv:2303.11032 tab. 2]. The paper's own numbers favour the fine-tuned models | | Personal details must be removed from training text, and a tagger is available | Tagger-based scrubbing, then DP training if the threat needs it; test the tagger's misses on your own data | On ECHR, scrubbing alone gives membership-inference AUC 0.82 against 0.96 undefended, and DP with scrubbing gives 0.505 [arxiv:2302.00539 §IV-H]. Tagger misses remain [arxiv:2302.00539 §II-C] | | Measuring how many personal details a model leaks, with no labelled leak set | Off-the-shelf NER on the generated text, reported as a lower bound | Misses count as no leakage, so the figure understates it [arxiv:2302.00539 §V] | ## Implementation sketch Two sketches follow, one per model family. Each is our own pseudocode of the paper's description, with the names chosen by us. The augmentation and pre-labelling variants are described in prose only. The CRF sketch covers one training step. The paper gives the objective but not the batching, so the way the two batches are mixed is open [arxiv:2404.09383 eq. 7]. ```pseudocode # One training step, cross-lingual neural CRF [arxiv:2404.09383 §3.1] # Input: target and source batches of (words, tags, language); mu (float) # Output: scalar loss to minimise def training_loss(target_batch, source_batch, mu): loss = 0.0 # Target sentences enter with weight 1 [eq. 7] for words, tags, lang in target_batch: loss -= crf_log_likelihood(words, tags, lang) # Source sentences enter with weight mu > 0, which the paper does not set [eq. 7] for words, tags, lang in source_batch: loss -= mu * crf_log_likelihood(words, tags, lang) return loss ``` The LLM sketch is the loop below. It follows the paper's description: one prompt per entity type, then a verification prompt per candidate span [arxiv:2304.10428 §4]. The paper gives no algorithm box. ```pseudocode # GPT-NER for one sentence, one entity type at a time [arxiv:2304.10428 §4] # Input: words (list of str); label_types (list of str); k (int) # Output: list of (span, label) pairs def gpt_ner(words, label_types, k): spans = [] for label in label_types: # Demonstrations: entity-level kNN, or random [§4.2] demos = retrieve_demonstrations(words, label, k) # Instruction for this type, demonstrations, then the input [§4.1] prompt = build_prompt(label, demos, words) # Output copies the sentence, marking entities with @@ and ## [§4.1.2] completion = complete(prompt) candidates = parse_marked_spans(completion, words) for span in candidates: # Self-verification: keep only spans the model confirms [§4.3] demos_v = retrieve_demonstrations(span, label, k) verdict = verify(words, span, label, demos_v) if verdict == "yes": spans.append((span, label)) return spans ``` Where the LLM variants differ: the retrieval step (random, sentence-level or entity-level), whether `verify` runs, and the number of demonstrations k. The paper gives no k for its main results, and it reports that F1 still rose at the largest k that fits the context window [arxiv:2304.10428 §5.1] [arxiv:2304.10428 §6.2]. When the retrieved examples contain no entity, kNN retrieval did worse than random at k = 2 and k = 4 [arxiv:2304.10428 §6.2]. For the CRF variants, the paper does not say how the BiLSTM output maps to the 100-dimensional contextual representation it reports, and it does not state the character LSTM size, the language-vector size or μ [arxiv:2404.09383 §2.3 fn. 1] [arxiv:2404.09383 §3.1]. Choose these on your own validation data. ## Contradictions - **Text and table disagree (GPT-NER).** The text gives 84.97 against 85.91 for the CoNLL2003 few-shot comparison, but neither pair matches a row in [arxiv:2304.10428 tab. 2], where the FS + entity value is 90.91 [arxiv:2304.10428 §5.1.1]. On GENIA the text gives 56.68 against 69.06, while [arxiv:2304.10428 tab. 3] gives 51.68 for sentence-level and 64.06 for entity-level [arxiv:2304.10428 §5.1.2]. - **Reported F1 is not the harmonic mean of its P and R (GPT-NER).** For ACE2005 with entity-level ZS, P 72.63 and R 75.39 give about 73.99 F1, but the table reports 73.46 [arxiv:2304.10428 tab. 3]. The same check fails for the FS row. Recompute from counts before relying on these rows. - **Text and table disagree (augmentation).** The text says all methods beat the baseline on average [arxiv:2010.11683 §4.4]. On the full training set, several single-method cells are equal to or below the baseline, for example LSTM on MaSciP (76.3 and 76.0 against 76.4) and SciBERT on i2b2-2010 (87.5, 87.7, 87.7 and 87.2 against 87.8) [arxiv:2010.11683 tab. 3]. - **Biomedical gaps differ between the LLM papers.** GPT-NER reports 64.42 F1 on GENIA, a biomedical nested set, against 83.75 for BERT-MRC [arxiv:2304.10428 tab. 3]. The benchmark reports prompted GPT-4 at 0.7979 to 0.8327 F1 on BC5CDR-chemical, against 0.9500 for a supervised baseline [arxiv:2305.16326 tab. 3]. The two papers use different datasets, models and prompt protocols, so this is not a direct disagreement. Measure the gap on your own domain. - **Text and table disagree (GottBERT).** The NER paragraph gives 86.78 for the filtered large "last" model on CoNLL. The table gives 86.28 for that row. The value 86.78 matches the unfiltered large model and GELECTRA large in the table [arxiv:2012.02110 §4] [arxiv:2012.02110 tab. 3]. - **Text and table disagree (scispaCy).** The authors describe the NER models as competitive with baselines on 5 of 9 datasets [arxiv:1902.07669 §4.2]. In their table, the md model is above the Baseline column on six rows: BC5CDR, JNLPBA, BioNLP13CG, BC4CHEMD, Linnaeus and NCBI-Disease. "Competitive" is not defined in the paper, so the text and the table cannot be checked against each other [arxiv:1902.07669 tab. 5]. - **Survey text and table disagree.** The survey gives a 0.52-point English gain for Chiu and Nichols over the best word-based model. In its Table 1, the Chiu and Nichols score minus the Yang et al. score is 0.36 [arxiv:1910.11470 §7] [arxiv:1910.11470 tab. 1]. - **No conflicting claims across the papers.** The papers use different datasets, languages and models, so their results answer different questions. None reports a result that another contradicts. ## Papers - [GPT-NER: Named Entity Recognition via Large Language Models](/papers/2304.10428): copy-and-mark prompting with entity-level kNN demonstrations and yes/no self-verification; reports 90.91 F1 on full CoNLL2003 and 82.20 on OntoNotes5.0 (author-reported). - [Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields](/papers/2404.09383): character-level neural CRF with a cross-lingual variant that shares most layers across languages; reports up to 9.8 F1 over the log-linear CRF with 100 target sentences (author-reported). - [An Analysis of Simple Data Augmentation for Named Entity Recognition](/papers/2010.11683): four token-level augmentations for NER; reports gains of up to 9.8 F1 on 50-sentence sets and mixed gains on the full sets (author-reported). - [Benchmarking large language models for biomedical natural language processing applications and recommendations](/papers/2305.16326): twelve biomedical tasks, including two NER sets, for four LLMs under zero-shot, few-shot and fine-tuned settings; supervised baselines lead on NER (author-reported). - [LLMs Accelerate Annotation for Medical Information Extraction](/papers/2312.02296): few-shot PaLM 2 drafts medication labels that experts refine; refined quality is close to the human-draft pipeline and rater time falls by 58% (author-reported). - [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](/papers/1910.11470): survey of 83 NER systems, grouped by family and compared on CoNLL 2002/2003 and DrugNER; the word, character and affix tagger has the best reported scores in three of four CoNLL languages (author-reported, copied from cited papers). - [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](/papers/1902.07669): spaCy-based biomedical pipeline with a transition-based NER component and typed NER packages; reports NER competitive with baselines on 5 of 9 datasets (author-reported). - [GottBERT: a pure German Language Model](/papers/2012.02110): German-only RoBERTa base and large, with a filtered and an unfiltered corpus; base models lead on GermEval 2014 and CoNLL 03 (author-reported). - [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](/papers/2303.11032): prompt-only removal of identifiers from clinical notes; reports 0.929 for ChatGPT with an explicit prompt and 0.99 for GPT-4 on 50 hand-run notes (author-reported). - [Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective](/papers/2606.29836): SciBERT-based cascade tagger for method, dataset, metric and tool entities, with model-based augmentation; reports 87.00 F1 on its own test split (author-reported). - [Analyzing Leakage of Personally Identifiable Information in Language Models](/papers/2302.00539): tagger-based scrubbing of GPT-2 training data, with DP-SGD and reconstruction and inference attacks; reports that scrubbing and DP cut but do not remove leakage (author-reported). - [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](/papers/2109.15290): SciBERT continued on materials text, with linear, CRF and BiLSTM-CRF NER heads; reports higher scores than SciBERT on nearly every test cell, with one tie (author-reported). - [Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities](/papers/2606.31058): SciBERT-BiLSTM tagger for four entity types in NLP papers, used to build a novelty score; reports 86.27 F1 on its own test split (author-reported).