Sequence labeling

Overview

Sequence labeling assigns one label to each token of an input sequence. The labels can mark spans (where a phrase starts and ends) and attach a class to each span. A common recipe is to run an encoder over the tokens and put a small task layer on top that scores the labels for each position.

Eleven cards on wikillm use the concept, each in a different setting:

• arXiv:1910.00883 tags aspect terms and their sentiment in one pass, with a fine-tuned BERT-base encoder and one of five task heads 1910.00883 §2.
• arXiv:1707.09861 samples about 1,000 random BiLSTM configurations and trains each with a softmax or a CRF output. It also argues that single-run scores should be replaced by score distributions over seeds 1707.09861 §3 §5.
• arXiv:2010.11683 studies data augmentation for named entity recognition, where every token keeps a label. Its taggers are a BiLSTM and a SciBERT model, both with a CRF output 2010.11683 §4.2.
• arXiv:2204.07580 tags words one at a time by prompting a decoder-only language model with labelled examples. It trains nothing 2204.07580 §4.2.2.
• arXiv:2606.29836 tags method, dataset, metric and tool names in NLP papers with a SciBERT encoder. It predicts span starts and ends with separate heads, and it augments training data with a model-based check 2606.29836 §3.2 §3.3.
• arXiv:1902.07669 ships spaCy-based biomedical pipelines. Its NER is transition-based and its POS tagger is trained jointly with the parser 1902.07669 §3 §4.
• arXiv:1910.11470 is a survey of NER systems. Its best family reads each word as a whole, as characters and as affixes, then tags with a recurrent layer. The survey copies these scores from the cited papers 1910.11470 §6.4.4 tab. 1.
• arXiv:2110.07602 keeps the whole backbone frozen and trains prefix vectors at every layer, with a linear head per token for tagging 2110.07602 §3.2 app. A.
• arXiv:2606.26485 tags keyphrases in tweets with a BiLSTM-CRF whose word inputs include per-word eye-tracking and EEG features from a reading corpus 2606.26485 §3.2.1 §3.1.3.
• arXiv:1912.09582 pre-trains a Dutch BERT-base model and fine-tunes it for token-level named-entity recognition, among other tasks 1912.09582 §2 tab. 1.
• arXiv:2606.31058 tags four kinds of entity in NLP papers with a SciBERT encoder and a BiLSTM, then uses the tags to measure novelty 2606.31058 §3.3.1.

The first paper gives the core recipe; the others change the training data, the output structure, the pipeline, the model type, what is trained, or how results are reported. Every number below comes from one of these eleven papers.

Variants

Fine-tuned encoder with one task layer

The encoder is a pre-trained BERT-base (12 layers, hidden size 768). It gives a contextual state per input token 1910.00883 §2.1 eq. 1. One task layer then maps those states to tags. The paper compares five heads on the same encoder 1910.00883 §2.2:

• Linear: one softmax per token over the tag set 1910.00883 eq. 2.
• GRU: a single gated recurrent layer with layer norm on the gate inputs, then a softmax 1910.00883 eq. 3 eq. 4.
• SAN: one self-attention layer with a residual connection and layer norm 1910.00883 eq. 5.
• TFM: one transformer layer, attention plus a feed-forward block, each with residual and layer norm 1910.00883 eq. 6.
• CRF: a linear-chain conditional random field that scores whole tag sequences from per-token emissions plus tag-to-tag transitions, trained by likelihood and decoded with Viterbi 1910.00883 eq. 7 eq. 8. See conditional random field for the model itself.

The paper's tag set has 13 tags: B, I, E, S and O, with B, I, E and S crossed with POS, NEG and NEU 1910.00883 §2. Card: Exploiting BERT for End-to-End Aspect-based Sentiment Analysis.

Frozen versus fine-tuned encoder

The encoder can be fine-tuned together with the head, which is the paper's main setting 1910.00883 §3.1. The paper also trains with BERT frozen and reports the results as "far from satisfactory", so fine-tuning is needed in its setup. The text gives no numbers for the frozen run; they appear only in a figure 1910.00883 §3.4 fig. 3.

Monolingual encoder for Dutch token labels

The Dutch model has the BERT-base architecture, with 12 transformer blocks. It is pre-trained on about 2.4 billion tokens of Dutch text and uses sentence order prediction in place of next sentence prediction 1912.09582 §2.1 §2.2. For named-entity recognition it is fine-tuned for four epochs on the Dutch CoNLL-2002 data. It reaches 88.3 test F1, against 80.7 for multilingual BERT under the same setup 1912.09582 tab. 1 §4. On the token-level NER task built from SoNaR-1 it reaches 82.1 test F1, against 79.7 [arxiv:1912.09582 tab. 1]. The paper cites 90.9 test F1 for Wu and Dredze (2019) on CoNLL-2002, which the Dutch model does not reach 1912.09582 §4 tab. 1. Every cell is one run, and the paper states no run count or variance [arxiv:1912.09582 §4]. Card: BERTje: A Dutch BERT Model.

Frozen backbone with trainable prefixes

The backbone stays frozen. Trainable prefix vectors are added at every transformer layer as extra keys and values, and a linear classifier labels each token 2110.07602 §3.2 app. A. The tagging rows use RoBERTa-large and BERT-large on five benchmarks: CoNLL03, OntoNotes 5.0, CoNLL04, CoNLL12, and CoNLL05 on its WSJ and Brown splits 2110.07602 tab. 3. Only the first sub-token of each word is labelled; the others are set to -100, which the loss ignores dataset.py. Card: P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks.

On RoBERTa-large, prefix tuning (PT-2) reaches 92.8 F1 on CoNLL03 against 92.6 for full fine-tuning, and 88.4 against 88.8 on CoNLL04. On BERT-large it trails further: 90.2 against 92.8 on CoNLL03, and 83.2 against 84.9 on CoNLL12 2110.07602 tab. 3. The input-layer prompt baseline (PT) is far lower on the same rows, for example 76.2 against 88.8 on CoNLL04 with RoBERTa-large 2110.07602 tab. 3.

Read these as dev-set numbers. The split for the tagging rows is not stated 2110.07602 tab. 3. The code keeps the best epoch on dev trainer_base.py, and the paper does not say how many seeds each cell averages. The PT baseline has no code path for token classification, so those numbers cannot be rebuilt from the release utils.py.

Prefix length and learning rate are set per task. The paper says hard sequence labeling tends to prefer around 100 prefix positions 2110.07602 §3.3 fig. 4. The released scripts use 11 positions for CoNLL03 with RoBERTa-large run_conll03_roberta.sh and 224 for CoNLL05 with RoBERTa-large run_conll05_roberta.sh. Each prefix position adds keys and values at every layer, so attention cost grows with the prefix length, which the paper does not discuss.

Cascaded boundary tagging with a type head

Instead of one tag per token, this design uses three heads on a SciBERT encoder. Two linear heads predict whether each token is an entity start and whether it is an entity end, each as a binary label. A BiLSTM over the encoder output feeds a third head that gives each token an entity type, with an O class for non-entities 2606.29836 §3.2 eq. 2 eq. 3 eq. 6 eq. 7. The loss is the unweighted sum of three cross-entropy terms 2606.29836 eq. 9. The paper does not describe decoding. The reference code pairs each start with an end and takes a majority type over the subword tokens [arxiv:2606.29836 code: Code/ner-cascade.py#L53-L120].

The paper's four types are Method, Dataset, Metric and Tool [arxiv:2606.29836 §3.1]. On the authors' own test split of 311 sentences, the cascade reaches F1 86.49, against 86.04 for SciBERT with a softmax head and 85.42 for SciBERT with a CRF 2606.29836 tab. 7. Each model was run once, and the paper reports no variance 2606.29836 §4.1 (3). Card: Revealing the Technology Development of Natural Language Processing.

SciBERT with a BiLSTM tagger on full texts

A SciBERT encoder feeds a BiLSTM in a cascade, and the tagger marks four entity types in NLP papers: method, dataset, tool and metric 2606.31058 §3.3.1 tab. 4. It was trained on sentences from 50 ACL papers, annotated by five researchers, with a train, validation and test split of 8:1:1 [arxiv:2606.31058 §3.3.1]. The paper gives no training hyperparameters, such as the learning rate, optimizer, epochs or batch size 2606.31058 §3.3.1. On the authors' own annotated test data it reaches 86.27 F1 (precision 88.16, recall 84.45), against 85.65 for the best baseline, SciBERT or SciBERT with a CRF [arxiv:2606.31058 tab. 5]. The tagger was then run on 14,812 papers and produced 151,018 entities 2606.31058 §3.3.1 tab. 6. Card: Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities.

BIO tags with a CRF output layer

The NER study uses BIO tags: each token gets B, I or O, combined with an entity type 2010.11683 §4.2. The output layer is a CRF over those tags 2010.11683 §4.2. It compares two encoders: a BiLSTM over GloVe vectors, trained from scratch, and SciBERT, pre-trained on scientific articles and fine-tuned with the first sub-token of each word as its representation 2010.11683 §4.2. The reference code has only the BERT path, so only the SciBERT results can be rebuilt from it 2010.11683 code: models.py#L49-L52. Card: An Analysis of Simple Data Augmentation for Named Entity Recognition.

Transition-based NER and joint POS tagging in a spaCy pipeline

The pipeline is one package that runs tokenisation, POS tagging, dependency parsing, NER and sentence splitting. The NER component is transition-based and follows the chunking model of Lample et al. Each token is represented by hashed, embedded prefix, suffix, shape and lemma features 1902.07669 §4. The main NER model is trained on MedMentions mention spans and finds mentions without types. Four further packages add typed NER for specific datasets 1902.07669 §4.1. The POS tagger shares its convolutional token features with the parser 1902.07669 §3. The paper's models were built on spaCy 2.0.18 1902.07669 §2. Card: ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing.

Word-by-word tagging with a few-shot language model

No gradient updates are made. The model tags words left to right 2204.07580 §4.2.2. The prompt holds four demonstrations from the English training data, then the words already tagged with their tags, then the current word and a blank 2204.07580 §4.2.2. Each candidate tag fills the blank, and the tag with the lowest summed token loss is kept. The first word serves only as context 2204.07580 §4.2.2. The paper ran zero-shot too but reports only the 4-shot results 2204.07580 §4.2.2. The model is mGPT-1.3B or mGPT-13B, a decoder-only model with a GPT-3-style layout 2204.07580 §3.3. Card: mGPT: Few-Shot Learners Go Multilingual.

Word, character and affix embeddings in a recurrent tagger

The survey's most detailed neural family gives each word three learned inputs: a word embedding, a recurrent summary of its characters, and embeddings of its frequent prefixes and suffixes. A sentence-level recurrent layer then scores the labels for each word 1910.11470 §6.4.3 §6.4.4 fig. 4. The survey gives no pseudocode and does not say how the three inputs are joined; concatenation is inferred from the figure 1910.11470 §6.4.4 fig. 4. The affix frequency threshold and the n-gram size are not stated 1910.11470 §6.4.4.

The affix model's CoNLL F1, 150 epochs, is 87.26 (Spanish), 87.54 (Dutch), 90.86 (English) and 79.01 (German). The survey copies these from Yadav et al. (2018) 1910.11470 tab. 1 §6.4.4. The survey proposes no new model. Card: A Survey on Recent Advances in Named Entity Recognition from Deep Learning models.

BiLSTM tagger with a softmax or CRF output

The network takes pre-trained word embeddings, optionally adds casing features and a character CNN or LSTM, stacks one to three bidirectional LSTM layers, and ends with a per-token output: softmax or CRF 1707.09861 §4.2. The paper uses no transformer. Its search grid covers the optimizer, the dropout type and rate, the layer count and the unit count, and it samples configurations from that grid 1707.09861 §4.2 §5.

In the sampled configurations the CRF output was better on four of five tasks, and not on Events 1707.09861 §6.1. On NER it won in 90.5% of 232 configurations, with a smaller spread across seeds: σ 0.0060 against 0.0082 for softmax, Brown-Forsythe p < 10⁻³ 1707.09861 §5 tab. 3. The paper also finds that softmax gains more from extra stacked layers than CRF does 1707.09861 §6.1 fig. 3. Two stacked layers gave the best results overall 1707.09861 §6.7. The paper's difference is median(S − C), so a negative value favours CRF. The Events row prints a negative median under the CRF column, while softmax won most Events configurations; the paper does not explain the sign there 1707.09861 §5 tab. 3. Card: Reimers and Gurevych, 2017. See seed-variance evaluation for the protocol and dropout for the dropout variants.

BiLSTM-CRF with per-word reading features

The keyphrase study gives each word a 25-value vector: 17 eye-tracking features and 8 EEG features, averaged over every ZuCo occurrence of the word type 2606.26485 §3.1.1 §3.1.3 tab. 4 tab. 5. Words missing from ZuCo get an imputed value 2606.26485 §3.1. The vector is concatenated to the word embedding before the BiLSTM 2606.26485 §3.2.1 fig. 2. Tags are B, I, E, S and O 2606.26485 code: config.py#L14-L15. A soft-attention or self-attention layer can sit on top of the BiLSTM 2606.26485 §3.2.2.

Coverage is the main limit. ZuCo covers 8.579% of the Election-Trec vocabulary and 4.038% of General-Twitter 2606.26485 tab. 6 §3.1.2, so most words receive imputed values. Each word type has one fixed vector, so a word gets the same features in every tweet and for every reader 2606.26485 §3.1.3. Nothing in the cited results separates a word-type prior from a reading signal.

The paper's headline rows compare a plain BiLSTM with SATT-BiLSTM+CRF using all EEG features. Election-Trec goes from 61.67 to 67.82 F1 and General-Twitter from 70.67 to 82.46 2606.26485 tab. 4 §4.2. These comparisons change the model and the features together. The matched no-feature pair on General-Twitter is 81.63 for BiLSTM-CRF and 81.74 for SATT-BiLSTM+CRF, so attention alone adds 0.11 F1 there 2606.26485 tab. 4 §4.2. The authors report that EEG features help more than eye-tracking features, and that the combination lies between the two 2606.26485 §4.2 §4.4. With pre-trained encoders the gains shrink: BERT with no features scores 71.79 and with eye-tracking and EEG features 71.82 on Election-Trec 2606.26485 tab. 7 §4.5.

The released code does not match the paper in ways that matter for reproduction. The non-BERT models use only EEG features 5 and 6, so the All_EEG rows cannot be rebuilt from the repository [arxiv:2606.26485 code: models/BILSTMCRF.py#L48]. The released SATT-BiLSTM+CRF passes an argument the repository never defines 2606.26485 code: models/SATT_BILSTMCRF.py#L132. These points come from reading the source; the card's author did not run the code. The full list is in the card's Paper vs code section. Card: Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs. See also keyphrase extraction.

Label-preserving augmentation of token-labelled data

The NER study adapts four simple augmentations to token labels. None uses machine translation or parsing models; synonym replacement uses WordNet 2010.11683 §3. Each is applied with probability p, which the authors tune on the dev set over values from 0.1 to 0.7 2010.11683 §4.3.

• Label-wise token replacement: each token is swapped for another token that already carries the same full BIO label. The label sequence is unchanged 2010.11683 §3.
• Synonym replacement: each token is swapped for a WordNet synonym. A multi-word synonym gives its extra words I- labels when the token was B- 2010.11683 §3.
• Mention replacement: a whole mention is swapped for another mention of the same type from the training data, and its labels are rewritten to match 2010.11683 §3.
• Shuffle within segments: the tokens inside each mention, and inside each run of O tokens, are reordered. The label row stays the same 2010.11683 §3.

The paper's table shows that label-wise replacement and shuffling keep the label sequence, while synonym and mention replacement can change it 2010.11683 tab. 1. The authors also test all four together ("All") 2010.11683 §3.

Model-based augmentation with an entity check

The cascade study does not use easy text edits. It says such edits risk changing entity spans, and it uses a model-based procedure instead 2606.29836 §3.3:

1. Collect every entity annotated in the training set, and all unannotated abstracts.
2. Build candidate sentences from the abstracts that contain a known entity.
3. Tag each candidate with a model trained on the training set.
4. Keep a candidate only when every known entity it matches is also found by the model.
5. Add to the training set as many sentences as the training set has, drawn from the kept candidates 2606.29836 §3.3.

The augmented run reaches F1 87.00, against 86.49 without it. That is a gain of 0.51 F1 from one run each 2606.29836 tab. 7. The reference repository has no script for this procedure 2606.29836 code: Code/, so the steps above must be rebuilt from the text.

Comparing taggers across seeds

A single run can mislead. For one NER system the test F1 ranged from 89.99 to 91.00 over 86 seeds, and two systems ranked one way in their papers ranked the other way when re-run 1707.09861 §1 §3 tab. 1. Development-set selection did not fix this: for one system the Spearman correlation between dev and test score was 0.229 1707.09861 §3.

The paper's protocol compares score distributions over many seeds, using the median, the spread and a significance test (Kolmogorov-Smirnov for the distributions, Brown-Forsythe for the spread, a randomization test for a single gap) 1707.09861 §3 §1 footnote. Across random configurations, the median gap between the two seeds of one configuration is 0.17 to 0.72 F1 points, and the 95th percentile reaches 2.10 on ACE 2005 entities [arxiv:1707.09861 §3] tab. 2. Card: Reimers and Gurevych, 2017.

When to use which

The eleven papers use different datasets, tag sets and metrics, so the rows below compare results only within one paper's table, except where a row says otherwise.

Training settings the first paper states, for reference: learning rate 2e-5, 1500 steps, batch size 25 on LAPTOP and 16 on REST, and model selection on dev micro-F1 from step 1000, every 100 steps 1910.00883 §3.1. Five seeds are averaged in the paper 1910.00883 §3.1; the reference code runs one seed by default 1910.00883 code: fast_run.py#L5-L6.

Implementation sketch

Language-agnostic pseudocode for the common core. The head is the only part that changes between the fine-tuned variants.

# Sequence labeler: encoder, one task head, decode  [arxiv:1910.00883 §2]
# Input:  token ids for one sentence, T tokens
# Output: one tag per token from the BIEOS set

def label_sentence(token_ids):
    # Contextual state for each token from the pre-trained encoder  [§2.1]
    states = encoder(token_ids)               # (T, d)

    # Task head: linear, GRU, SAN, TFM or CRF  [§2.2]
    features = head(states)                   # (T, d)

    # Score every tag at every position  [§2.2] [eq. 2]
    scores = features @ W_out + b_out         # (T, 13)

    # Softmax heads: pick the best tag per token  [§2.2]
    # CRF head: Viterbi over the whole sequence  [eq. 8]
    return decode(scores)                     # (T,)

# Turn BIEOS tags back into labelled spans  [arxiv:1910.00883 code: seq_utils.py#L166-L172]
def spans_from_tags(tags):
    spans = []
    # Walk the tags; a span opens at B and closes at E, S is one token  [§2]
    for start, end, label in find_spans(tags):
        # Drop a span whose sentiment tags disagree or that has no closing E
        if label_consistent(tags, start, end):
            spans.append((start, end, sentiment_of(tags[start])))
    return spans

# Cascade: boundary heads and a type head, then pair starts with ends  [§3.2]
# Input:  per-token start, end and type scores from the three heads
# Output: labelled spans
# Simplified: the reference code also revises boundaries with two rules and
# falls back to the type sequence  [arxiv:2606.29836 code: Code/ner-cascade.py#L53-L120]

def spans_from_cascade(start_p, end_p, type_p):
    spans = []
    # Pair each start with an end; the pairing rule is simplified here  [code]
    for i in starts(start_p):
        j = next_end_after(i, end_p)
        if j is not None:
            # Majority type over the subword tokens  [code: ner-cascade.py#L53-L120]
            spans.append((i, j, majority_type(type_p, i, j)))
    return spans

Notes for implementers, each taken from the papers or their reference code:

• Freeze the backbone only when you train prefixes. The released sequence-classification model sets requires_grad = False on the backbone and trains the prompt table and the head [arxiv:2110.07602 code: model/sequence_classification.py#L336-L337]. This card did not read the token-classification path, so check it before you copy this for tagging.
• Fix the global batch size. The first paper's reference code multiplies the per-GPU batch by the GPU count, so the effective batch depends on hardware 1910.00883 code: main.py#L159.
• Label each word once. That paper's reference code gives the first word piece the word's tag and labels later pieces with an extra class, so the model has 14 classes in practice 1910.00883 code: glue_utils.py#L211-L212.
• Mask padding in attention heads. That code calls the SAN and TFM heads without a padding mask 1910.00883 code: absa_layer.py#L455-L456.
• Expect invalid tag sequences from unconstrained decoding; that code repairs or drops them when it rebuilds spans 1910.00883 code: absa_layer.py#L127.
• The first paper does not state the optimizer, schedule, gradient clipping, dropout or maximum sequence length. Treat any value you copy from its code as the authors' choice 1910.00883 code: main.py#L105-L110.
• Keep the B- and I- pools apart when replacing tokens by label. The reference code of the augmentation study draws from a pool keyed by the full BIO tag, so an I- token is never replaced by a B- token 2010.11683 code: augment.py#L139-L141.
• Regenerate augmented copies every epoch. That study's code makes new random draws each epoch from the original sentences 2010.11683 code: train.py#L156-L172.
• Fix the demonstration seed in few-shot tagging. The reference notebook accepts a seed but does not use it, and draws demonstrations with random.sample 2204.07580 code: evaluation/sequence_labeling/sequence_labeling_prediction.ipynb#L426-L429.
• Watch the context length. Its dataset truncates on the right 2204.07580 code: evaluation/sequence_labeling/list_docs_dataset.py#L14-L20, so a long demonstration block could cut off the candidate tag. This is an inference from the code and was not run.
• Seed every run and record the seed with its score. The BiLSTM code never sets a seed, so repeated runs will not match [arxiv:1707.09861 code: neuralnets/BiLSTM.py#L129] BiLSTM.py. Seeding Python, NumPy and the Keras backend yourself is our suggestion, not the authors'.
• Keep the tokenizer fixed between training and inference. Label boundaries follow token boundaries, and the scispaCy tokenizer adds prefix, suffix and infix rules and keeps listed abbreviations whole [arxiv:1902.07669 code: scispacy/custom_tokenizer.py#L48-L60] custom_tokenizer.py. Changing it changes which spans can be labelled. This advice is ours.
• Do not copy the scispaCy paper's feature list without checking. The paper lists lemma among the NER features, but the reference config has no lemma attribute 1902.07669 code: configs/base_ner.cfg#L51.
• Label subword pieces on purpose. The Dutch BERT fine-tuning script puts the word's label on the first piece and gives continuation pieces an O label, and it computes the loss at every position, padding included 1912.09582 code: finetuning/v1/run_benchmark.py#L157-L160 [#L176]. Mask the loss yourself if you want word-initial positions only. This is the card's reading of the script, and it was not run.
• Expect the scispaCy code to differ from the 2019 models. The repository requires spaCy 3.7 and its configs use the spaCy 3 format, while the paper's models use spaCy 2.0.18 1902.07669 code: requirements.in#L6. The paper states none of its optimizer, batch or epoch settings, so the repository configs are the only source for them.

Contradictions

• Augmentation gains: text versus table. The augmentation study says every method improves on the baseline on average 2010.11683 §4.4. Its table shows several full-set single-method cells equal to or below the baseline, for example SciBERT on i2b2-2010 full set: 87.5, 87.7, 87.8, 87.7 and 87.2 against 87.8 with no augmentation 2010.11683 tab. 3. The card reads the full-set column as no reliable gain.
• CRF on NER: two setups. The BiLSTM sweep finds CRF better on NER in 90.5% of sampled configurations 1707.09861 tab. 3. The cascade study finds the CRF below the softmax head for BERT, RoBERTa and SciBERT [arxiv:2606.29836 tab. 7]. The sweep varies the whole BiLSTM design, while the cascade study fixes one encoder and changes only the head, so the two results are not comparable.
• Easy edits versus model-based filtering. The augmentation study uses four label-preserving edits and reports gains, largest on small training sets 2010.11683 §4.4. The cascade study says easy text augmentations risk changing entity spans, and it uses model-based candidates instead 2606.29836 §3.3. The two papers test different data, tag sets and models, so neither result settles the other.
• Survey gains: text versus table. The survey says the best English word-based model beats the others by 0.52 1910.11470 §7. Its Table 1 gives 0.36 against Yang et al. (91.26) and 1.52 against Huang et al. with gazetteers (90.10) 1910.11470 tab. 1. The card lists this as a Pitfall. The table values are used here.
• CRF gains: small in one paper, mixed in another. The first paper finds a CRF slightly above the linear head on both data sets (60.78 against 60.43 on LAPTOP, 74.06 against 73.22 on REST) 1910.00883 tab. 2. The cascade study finds CRF lower than the softmax head for BERT, RoBERTa and SciBERT, and higher only for T5 (85.08 against 84.85) 2606.29836 tab. 7. The first paper uses BIEOS tags and the second uses BIOES, and the second runs one seed per model, so the results are not directly comparable.
• Scispacy NER: the authors' count and their table. The authors say the models are competitive with strong baselines on 5 of 9 data sets 1902.07669 §4.2. In the table's Baseline column, the md model scores higher on six data sets and lower on three (CRAFT, AnatEM and BC2GM) 1902.07669 tab. 5. The paper does not say which criterion gives five.
• Not comparable across papers. The first paper scores span F1 with exact boundaries and sentiment 1910.00883 code: glue_utils.py#L430. The augmentation study scores span F1 with exact token positions 2010.11683 code: train.py#L111-L122. The cascade study scores exact-match micro F1 over (type, start, end) spans 2606.29836 code: Code/ner-cascade.py#L206-L212. The mGPT paper reports accuracy for POS and F1 for NER, with few-shot prompting against fine-tuned baselines 2204.07580 §4.2.2. Compare numbers only within one paper's table.
• Frozen encoder: plain head versus prefixes. The BERT study reports a frozen encoder with a task layer as "far from satisfactory" 1910.00883 §3.4. The P-tuning v2 study also freezes the backbone, but trains prefix vectors at every layer, and reports dev scores near full fine-tuning on its tagging benchmarks 2110.07602 tab. 3. The two set-ups train different parts and use different data, so neither result settles the other.
• Reporting protocol. The BiLSTM sweep compares score distributions over many seeds 1707.09861 §3. The keyphrase study averages five runs, but it keeps each run's best of 20 epochs by test F1 and reports no variance or significance test 2606.26485 §4.1 SATT_BILSTMCRF.py. Test-set selection biases the reported F1 upward, so its gains and the sweep's findings are not comparable. Neither result settles the other.
• REST data. The first paper does not name its REST release. Its repository runs on a set that the README calls invalid, so the REST numbers cannot be checked against the code 1910.00883 code: train.sh#L2 1910.00883 code: README.md#L70-L71.

Papers

• Reimers and Gurevych, 2017: a BiLSTM sweep with softmax or CRF output, arguing for score distributions over seeds; CRF was better on NER in most sampled configurations.
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis: BERT-base with five task heads for joint aspect and sentiment tagging on LAPTOP and REST, with a linear head already above the non-BERT baselines.
• An Analysis of Simple Data Augmentation for Named Entity Recognition: four label-preserving augmentations for NER, tested with a BiLSTM and a SciBERT tagger on two specialist data sets, with gains largest on small training sets.
• mGPT: Few-Shot Learners Go Multilingual: word-by-word NER and POS tagging by prompting a decoder-only model, with results far below fine-tuned encoders on the paper's own benchmarks.
• Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective: a SciBERT cascade of start, end and type heads for four kinds of scientific entity, with a model-based augmentation step, reported as F1 87.00 on the authors' own test split.
• ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing: spaCy-based biomedical pipelines with transition-based NER and joint POS tagging and parsing, reported as competitive on NER and fast on CPU.
• P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks: deep prefix tuning on a frozen backbone with a linear token head, reported near full fine-tuning on several tagging benchmarks, on dev sets, with the token-classification baseline not in the release.
• Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs: per-word EEG and eye-tracking features added to BiLSTM-based keyphrase taggers for tweets; author-reported F1 gains, with released code that differs from the paper.
• BERTje: A Dutch BERT Model: a Dutch BERT-base model fine-tuned for token-level NER and POS tagging, reported above multilingual BERT on the paper's Dutch test sets, one run per cell.
• Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities: a SciBERT and BiLSTM tagger for four NLP entity types, reported at 86.27 F1 on its own test data and used to score paper novelty.
• A Survey on Recent Advances in Named Entity Recognition from Deep Learning models: a survey of 83 NER systems in four families, with CoNLL 2002/2003 and DrugNER tables; its affix-augmented word+character tagger gives the highest reported scores in three of four CoNLL languages.