--- kind: concept slug: sequence-labeling title: Sequence labeling summary: >- Give each token in a sentence one label, such as a span boundary or an entity type. Covered: fine-tuned encoders with task heads (English, Dutch and scientific BERT), BiLSTM taggers with softmax or CRF output, per-word reading features in a BiLSTM-CRF, cascaded tagging, a spaCy pipeline, data augmentation, few-shot prompting, prefix tuning and seed-distribution evaluation. tags: [sequence-labeling, tagging, aspect-based-sentiment-analysis, named-entity-recognition, data-augmentation, in-context-learning, bert, fine-tuning, cascade-tagging, biomedical, prefix-tuning, parameter-efficient-fine-tuning, bilstm, seed-variance, keyphrase-extraction, cognitive-signals, dutch] papers: ["1707.09861", "1902.07669", "1910.00883", "1910.11470", "1912.09582", "2010.11683", "2110.07602", "2204.07580", "2606.26485", "2606.29836", "2606.31058"] related: [named-entity-recognition, conditional-random-field, recurrent-neural-network, language-model-pretraining, masked-language-modeling, subword-tokenization, data-augmentation, in-context-learning, multilingual-evaluation, dependency-parsing, parameter-efficient-fine-tuning, soft-prompt-tuning, extractive-question-answering, seed-variance-evaluation, dropout, keyphrase-extraction, bibliometric-analysis] status: draft updated_at: 2026-11-01 --- ## Overview Sequence labeling assigns one label to each token of an input sequence. The labels can mark spans (where a phrase starts and ends) and attach a class to each span. A common recipe is to run an encoder over the tokens and put a small task layer on top that scores the labels for each position. Eleven cards on wikillm use the concept, each in a different setting: - [arXiv:1910.00883](/papers/1910.00883) tags aspect terms and their sentiment in one pass, with a fine-tuned BERT-base encoder and one of five task heads [arxiv:1910.00883 §2]. - [arXiv:1707.09861](/papers/1707.09861) samples about 1,000 random BiLSTM configurations and trains each with a softmax or a CRF output. It also argues that single-run scores should be replaced by score distributions over seeds [arxiv:1707.09861 §3] [§5]. - [arXiv:2010.11683](/papers/2010.11683) studies data augmentation for [named entity recognition](/concepts/named-entity-recognition), where every token keeps a label. Its taggers are a BiLSTM and a SciBERT model, both with a CRF output [arxiv:2010.11683 §4.2]. - [arXiv:2204.07580](/papers/2204.07580) tags words one at a time by prompting a decoder-only language model with labelled examples. It trains nothing [arxiv:2204.07580 §4.2.2]. - [arXiv:2606.29836](/papers/2606.29836) tags method, dataset, metric and tool names in NLP papers with a SciBERT encoder. It predicts span starts and ends with separate heads, and it augments training data with a model-based check [arxiv:2606.29836 §3.2] [§3.3]. - [arXiv:1902.07669](/papers/1902.07669) ships spaCy-based biomedical pipelines. Its NER is transition-based and its POS tagger is trained jointly with the parser [arxiv:1902.07669 §3] [§4]. - [arXiv:1910.11470](/papers/1910.11470) is a survey of NER systems. Its best family reads each word as a whole, as characters and as affixes, then tags with a recurrent layer. The survey copies these scores from the cited papers [arxiv:1910.11470 §6.4.4] [tab. 1]. - [arXiv:2110.07602](/papers/2110.07602) keeps the whole backbone frozen and trains prefix vectors at every layer, with a linear head per token for tagging [arxiv:2110.07602 §3.2] [app. A]. - [arXiv:2606.26485](/papers/2606.26485) tags keyphrases in tweets with a BiLSTM-CRF whose word inputs include per-word eye-tracking and EEG features from a reading corpus [arxiv:2606.26485 §3.2.1] [§3.1.3]. - [arXiv:1912.09582](/papers/1912.09582) pre-trains a Dutch BERT-base model and fine-tunes it for token-level named-entity recognition, among other tasks [arxiv:1912.09582 §2] [tab. 1]. - [arXiv:2606.31058](/papers/2606.31058) tags four kinds of entity in NLP papers with a SciBERT encoder and a BiLSTM, then uses the tags to measure novelty [arxiv:2606.31058 §3.3.1]. The first paper gives the core recipe; the others change the training data, the output structure, the pipeline, the model type, what is trained, or how results are reported. Every number below comes from one of these eleven papers. ## Variants ### Fine-tuned encoder with one task layer The encoder is a pre-trained BERT-base (12 layers, hidden size 768). It gives a contextual state per input token [arxiv:1910.00883 §2.1] [eq. 1]. One task layer then maps those states to tags. The paper compares five heads on the same encoder [arxiv:1910.00883 §2.2]: - **Linear:** one softmax per token over the tag set [arxiv:1910.00883 eq. 2]. - **GRU:** a single gated recurrent layer with layer norm on the gate inputs, then a softmax [arxiv:1910.00883 eq. 3] [eq. 4]. - **SAN:** one self-attention layer with a residual connection and layer norm [arxiv:1910.00883 eq. 5]. - **TFM:** one transformer layer, attention plus a feed-forward block, each with residual and layer norm [arxiv:1910.00883 eq. 6]. - **CRF:** a linear-chain conditional random field that scores whole tag sequences from per-token emissions plus tag-to-tag transitions, trained by likelihood and decoded with Viterbi [arxiv:1910.00883 eq. 7] [eq. 8]. See [conditional random field](/concepts/conditional-random-field) for the model itself. The paper's tag set has 13 tags: B, I, E, S and O, with B, I, E and S crossed with POS, NEG and NEU [arxiv:1910.00883 §2]. Card: [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](/papers/1910.00883). ### Frozen versus fine-tuned encoder The encoder can be fine-tuned together with the head, which is the paper's main setting [arxiv:1910.00883 §3.1]. The paper also trains with BERT frozen and reports the results as "far from satisfactory", so fine-tuning is needed in its setup. The text gives no numbers for the frozen run; they appear only in a figure [arxiv:1910.00883 §3.4] [fig. 3]. ### Monolingual encoder for Dutch token labels The Dutch model has the BERT-base architecture, with 12 transformer blocks. It is pre-trained on about 2.4 billion tokens of Dutch text and uses sentence order prediction in place of next sentence prediction [arxiv:1912.09582 §2.1] [§2.2]. For named-entity recognition it is fine-tuned for four epochs on the Dutch CoNLL-2002 data. It reaches 88.3 test F1, against 80.7 for multilingual BERT under the same setup [arxiv:1912.09582 tab. 1] [§4]. On the token-level NER task built from SoNaR-1 it reaches 82.1 test F1, against 79.7 [arxiv:1912.09582 tab. 1]. The paper cites 90.9 test F1 for Wu and Dredze (2019) on CoNLL-2002, which the Dutch model does not reach [arxiv:1912.09582 §4] [tab. 1]. Every cell is one run, and the paper states no run count or variance [arxiv:1912.09582 §4]. Card: [BERTje: A Dutch BERT Model](/papers/1912.09582). ### Frozen backbone with trainable prefixes The backbone stays frozen. Trainable prefix vectors are added at every transformer layer as extra keys and values, and a linear classifier labels each token [arxiv:2110.07602 §3.2] [app. A]. The tagging rows use RoBERTa-large and BERT-large on five benchmarks: CoNLL03, OntoNotes 5.0, CoNLL04, CoNLL12, and CoNLL05 on its WSJ and Brown splits [arxiv:2110.07602 tab. 3]. Only the first sub-token of each word is labelled; the others are set to -100, which the loss ignores [code: tasks/ner/dataset.py#L108-L120]. Card: [P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](/papers/2110.07602). On RoBERTa-large, prefix tuning (PT-2) reaches 92.8 F1 on CoNLL03 against 92.6 for full fine-tuning, and 88.4 against 88.8 on CoNLL04. On BERT-large it trails further: 90.2 against 92.8 on CoNLL03, and 83.2 against 84.9 on CoNLL12 [arxiv:2110.07602 tab. 3]. The input-layer prompt baseline (PT) is far lower on the same rows, for example 76.2 against 88.8 on CoNLL04 with RoBERTa-large [arxiv:2110.07602 tab. 3]. Read these as dev-set numbers. The split for the tagging rows is not stated [arxiv:2110.07602 tab. 3]. The code keeps the best epoch on dev [code: training/trainer_base.py#L52-L54], and the paper does not say how many seeds each cell averages. The PT baseline has no code path for token classification, so those numbers cannot be rebuilt from the release [code: model/utils.py#L73-L82]. Prefix length and learning rate are set per task. The paper says hard sequence labeling tends to prefer around 100 prefix positions [arxiv:2110.07602 §3.3] [fig. 4]. The released scripts use 11 positions for CoNLL03 with RoBERTa-large [code: run_script/run_conll03_roberta.sh#L5-L9] and 224 for CoNLL05 with RoBERTa-large [code: run_script/run_conll05_roberta.sh#L8]. Each prefix position adds keys and values at every layer, so attention cost grows with the prefix length, which the paper does not discuss. ### Cascaded boundary tagging with a type head Instead of one tag per token, this design uses three heads on a SciBERT encoder. Two linear heads predict whether each token is an entity start and whether it is an entity end, each as a binary label. A BiLSTM over the encoder output feeds a third head that gives each token an entity type, with an O class for non-entities [arxiv:2606.29836 §3.2] [eq. 2] [eq. 3] [eq. 6] [eq. 7]. The loss is the unweighted sum of three cross-entropy terms [arxiv:2606.29836 eq. 9]. The paper does not describe decoding. The reference code pairs each start with an end and takes a majority type over the subword tokens [arxiv:2606.29836 code: Code/ner-cascade.py#L53-L120]. The paper's four types are Method, Dataset, Metric and Tool [arxiv:2606.29836 §3.1]. On the authors' own test split of 311 sentences, the cascade reaches F1 86.49, against 86.04 for SciBERT with a softmax head and 85.42 for SciBERT with a CRF [arxiv:2606.29836 tab. 7]. Each model was run once, and the paper reports no variance [arxiv:2606.29836 §4.1 (3)]. Card: [Revealing the Technology Development of Natural Language Processing](/papers/2606.29836). ### SciBERT with a BiLSTM tagger on full texts A SciBERT encoder feeds a BiLSTM in a cascade, and the tagger marks four entity types in NLP papers: method, dataset, tool and metric [arxiv:2606.31058 §3.3.1] [tab. 4]. It was trained on sentences from 50 ACL papers, annotated by five researchers, with a train, validation and test split of 8:1:1 [arxiv:2606.31058 §3.3.1]. The paper gives no training hyperparameters, such as the learning rate, optimizer, epochs or batch size [arxiv:2606.31058 §3.3.1]. On the authors' own annotated test data it reaches 86.27 F1 (precision 88.16, recall 84.45), against 85.65 for the best baseline, SciBERT or SciBERT with a CRF [arxiv:2606.31058 tab. 5]. The tagger was then run on 14,812 papers and produced 151,018 entities [arxiv:2606.31058 §3.3.1] [tab. 6]. Card: [Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities](/papers/2606.31058). ### BIO tags with a CRF output layer The NER study uses BIO tags: each token gets B, I or O, combined with an entity type [arxiv:2010.11683 §4.2]. The output layer is a CRF over those tags [arxiv:2010.11683 §4.2]. It compares two encoders: a BiLSTM over GloVe vectors, trained from scratch, and SciBERT, pre-trained on scientific articles and fine-tuned with the first sub-token of each word as its representation [arxiv:2010.11683 §4.2]. The reference code has only the BERT path, so only the SciBERT results can be rebuilt from it [arxiv:2010.11683 code: models.py#L49-L52]. Card: [An Analysis of Simple Data Augmentation for Named Entity Recognition](/papers/2010.11683). ### Transition-based NER and joint POS tagging in a spaCy pipeline The pipeline is one package that runs tokenisation, POS tagging, dependency parsing, NER and sentence splitting. The NER component is transition-based and follows the chunking model of Lample et al. Each token is represented by hashed, embedded prefix, suffix, shape and lemma features [arxiv:1902.07669 §4]. The main NER model is trained on MedMentions mention spans and finds mentions without types. Four further packages add typed NER for specific datasets [arxiv:1902.07669 §4.1]. The POS tagger shares its convolutional token features with the parser [arxiv:1902.07669 §3]. The paper's models were built on spaCy 2.0.18 [arxiv:1902.07669 §2]. Card: [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](/papers/1902.07669). ### Word-by-word tagging with a few-shot language model No gradient updates are made. The model tags words left to right [arxiv:2204.07580 §4.2.2]. The prompt holds four demonstrations from the English training data, then the words already tagged with their tags, then the current word and a blank [arxiv:2204.07580 §4.2.2]. Each candidate tag fills the blank, and the tag with the lowest summed token loss is kept. The first word serves only as context [arxiv:2204.07580 §4.2.2]. The paper ran zero-shot too but reports only the 4-shot results [arxiv:2204.07580 §4.2.2]. The model is mGPT-1.3B or mGPT-13B, a decoder-only model with a GPT-3-style layout [arxiv:2204.07580 §3.3]. Card: [mGPT: Few-Shot Learners Go Multilingual](/papers/2204.07580). ### Word, character and affix embeddings in a recurrent tagger The survey's most detailed neural family gives each word three learned inputs: a word embedding, a recurrent summary of its characters, and embeddings of its frequent prefixes and suffixes. A sentence-level recurrent layer then scores the labels for each word [arxiv:1910.11470 §6.4.3] [§6.4.4] [fig. 4]. The survey gives no pseudocode and does not say how the three inputs are joined; concatenation is inferred from the figure [arxiv:1910.11470 §6.4.4] [fig. 4]. The affix frequency threshold and the n-gram size are not stated [arxiv:1910.11470 §6.4.4]. The affix model's CoNLL F1, 150 epochs, is 87.26 (Spanish), 87.54 (Dutch), 90.86 (English) and 79.01 (German). The survey copies these from Yadav et al. (2018) [arxiv:1910.11470 tab. 1] [§6.4.4]. The survey proposes no new model. Card: [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](/papers/1910.11470). ### BiLSTM tagger with a softmax or CRF output The network takes pre-trained word embeddings, optionally adds casing features and a character CNN or LSTM, stacks one to three bidirectional LSTM layers, and ends with a per-token output: softmax or CRF [arxiv:1707.09861 §4.2]. The paper uses no transformer. Its search grid covers the optimizer, the dropout type and rate, the layer count and the unit count, and it samples configurations from that grid [arxiv:1707.09861 §4.2] [§5]. In the sampled configurations the CRF output was better on four of five tasks, and not on Events [arxiv:1707.09861 §6.1]. On NER it won in 90.5% of 232 configurations, with a smaller spread across seeds: σ 0.0060 against 0.0082 for softmax, Brown-Forsythe p < 10⁻³ [arxiv:1707.09861 §5] [tab. 3]. The paper also finds that softmax gains more from extra stacked layers than CRF does [arxiv:1707.09861 §6.1] [fig. 3]. Two stacked layers gave the best results overall [arxiv:1707.09861 §6.7]. The paper's difference is median(S − C), so a negative value favours CRF. The Events row prints a negative median under the CRF column, while softmax won most Events configurations; the paper does not explain the sign there [arxiv:1707.09861 §5] [tab. 3]. Card: [Reimers and Gurevych, 2017](/papers/1707.09861). See [seed-variance evaluation](/concepts/seed-variance-evaluation) for the protocol and [dropout](/concepts/dropout) for the dropout variants. ### BiLSTM-CRF with per-word reading features The keyphrase study gives each word a 25-value vector: 17 eye-tracking features and 8 EEG features, averaged over every ZuCo occurrence of the word type [arxiv:2606.26485 §3.1.1] [§3.1.3] [tab. 4] [tab. 5]. Words missing from ZuCo get an imputed value [arxiv:2606.26485 §3.1]. The vector is concatenated to the word embedding before the BiLSTM [arxiv:2606.26485 §3.2.1] [fig. 2]. Tags are B, I, E, S and O [arxiv:2606.26485 code: config.py#L14-L15]. A soft-attention or self-attention layer can sit on top of the BiLSTM [arxiv:2606.26485 §3.2.2]. Coverage is the main limit. ZuCo covers 8.579% of the Election-Trec vocabulary and 4.038% of General-Twitter [arxiv:2606.26485 tab. 6] [§3.1.2], so most words receive imputed values. Each word type has one fixed vector, so a word gets the same features in every tweet and for every reader [arxiv:2606.26485 §3.1.3]. Nothing in the cited results separates a word-type prior from a reading signal. The paper's headline rows compare a plain BiLSTM with SATT-BiLSTM+CRF using all EEG features. Election-Trec goes from 61.67 to 67.82 F1 and General-Twitter from 70.67 to 82.46 [arxiv:2606.26485 tab. 4] [§4.2]. These comparisons change the model and the features together. The matched no-feature pair on General-Twitter is 81.63 for BiLSTM-CRF and 81.74 for SATT-BiLSTM+CRF, so attention alone adds 0.11 F1 there [arxiv:2606.26485 tab. 4] [§4.2]. The authors report that EEG features help more than eye-tracking features, and that the combination lies between the two [arxiv:2606.26485 §4.2] [§4.4]. With pre-trained encoders the gains shrink: BERT with no features scores 71.79 and with eye-tracking and EEG features 71.82 on Election-Trec [arxiv:2606.26485 tab. 7] [§4.5]. The released code does not match the paper in ways that matter for reproduction. The non-BERT models use only EEG features 5 and 6, so the All_EEG rows cannot be rebuilt from the repository [arxiv:2606.26485 code: models/BILSTMCRF.py#L48]. The released SATT-BiLSTM+CRF passes an argument the repository never defines [arxiv:2606.26485 code: models/SATT_BILSTMCRF.py#L132]. These points come from reading the source; the card's author did not run the code. The full list is in the card's Paper vs code section. Card: [Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs](/papers/2606.26485). See also [keyphrase extraction](/concepts/keyphrase-extraction). ### Label-preserving augmentation of token-labelled data The NER study adapts four simple augmentations to token labels. None uses machine translation or parsing models; synonym replacement uses WordNet [arxiv:2010.11683 §3]. Each is applied with probability p, which the authors tune on the dev set over values from 0.1 to 0.7 [arxiv:2010.11683 §4.3]. - **Label-wise token replacement:** each token is swapped for another token that already carries the same full BIO label. The label sequence is unchanged [arxiv:2010.11683 §3]. - **Synonym replacement:** each token is swapped for a WordNet synonym. A multi-word synonym gives its extra words I- labels when the token was B- [arxiv:2010.11683 §3]. - **Mention replacement:** a whole mention is swapped for another mention of the same type from the training data, and its labels are rewritten to match [arxiv:2010.11683 §3]. - **Shuffle within segments:** the tokens inside each mention, and inside each run of O tokens, are reordered. The label row stays the same [arxiv:2010.11683 §3]. The paper's table shows that label-wise replacement and shuffling keep the label sequence, while synonym and mention replacement can change it [arxiv:2010.11683 tab. 1]. The authors also test all four together ("All") [arxiv:2010.11683 §3]. ### Model-based augmentation with an entity check The cascade study does not use easy text edits. It says such edits risk changing entity spans, and it uses a model-based procedure instead [arxiv:2606.29836 §3.3]: 1. Collect every entity annotated in the training set, and all unannotated abstracts. 2. Build candidate sentences from the abstracts that contain a known entity. 3. Tag each candidate with a model trained on the training set. 4. Keep a candidate only when every known entity it matches is also found by the model. 5. Add to the training set as many sentences as the training set has, drawn from the kept candidates [arxiv:2606.29836 §3.3]. The augmented run reaches F1 87.00, against 86.49 without it. That is a gain of 0.51 F1 from one run each [arxiv:2606.29836 tab. 7]. The reference repository has no script for this procedure [arxiv:2606.29836 code: Code/], so the steps above must be rebuilt from the text. ## Comparing taggers across seeds A single run can mislead. For one NER system the test F1 ranged from 89.99 to 91.00 over 86 seeds, and two systems ranked one way in their papers ranked the other way when re-run [arxiv:1707.09861 §1] [§3] [tab. 1]. Development-set selection did not fix this: for one system the Spearman correlation between dev and test score was 0.229 [arxiv:1707.09861 §3]. The paper's protocol compares score distributions over many seeds, using the median, the spread and a significance test (Kolmogorov-Smirnov for the distributions, Brown-Forsythe for the spread, a randomization test for a single gap) [arxiv:1707.09861 §3] [§1 footnote]. Across random configurations, the median gap between the two seeds of one configuration is 0.17 to 0.72 F1 points, and the 95th percentile reaches 2.10 on ACE 2005 entities [arxiv:1707.09861 §3] [tab. 2]. Card: [Reimers and Gurevych, 2017](/papers/1707.09861). ## When to use which The eleven papers use different datasets, tag sets and metrics, so the rows below compare results only within one paper's table, except where a row says otherwise. | Situation | Use | Why | |---|---|---| | You want the simplest head on a pre-trained encoder for span tagging | Linear softmax on the encoder | Reaches 60.43 (LAPTOP) and 73.22 (REST) F1, above the best non-BERT baseline on both sets [arxiv:1910.00883 tab. 2] | | You want the best reported F1 on one dataset and can afford a slightly bigger head | GRU on LAPTOP (61.12) or SAN on REST (74.72) | Best entries in the paper's table [arxiv:1910.00883 tab. 2]; the margin over linear is 0.69 F1 on LAPTOP and 1.50 on REST | | Tags depend strongly on their neighbours and you accept slower decoding | CRF head | Scores whole sequences [arxiv:1910.00883 eq. 7] [eq. 8]; 60.78 (LAPTOP) and 74.06 (REST) F1, slightly above linear on both sets but below the best heads [arxiv:1910.00883 tab. 2] | | Your labelled set is very small (50 sentences in the paper) | Fine-tuned encoder with augmentation | With SciBERT, every single method beats no augmentation on the 50-sentence set for both data sets, and "All" gives +3.1 F1 on MaSciP and +6.4 F1 on i2b2-2010 [arxiv:2010.11683 tab. 3] | | Your labelled set is large, or the encoder is strong | Fine-tuned encoder without augmentation, or check on your data first | On the full set, SciBERT with "All" is 0.7 F1 above baseline on MaSciP and 0.6 F1 below it on i2b2-2010 [arxiv:2010.11683 tab. 3] | | Your entity types are domain-specific and you have a labelled set about the size of the paper's (2493 training sentences) | Cascaded boundary tagging, with or without model-based augmentation | On its 311-sentence test split, 86.49 F1 without augmentation and 87.00 with it, one run each [arxiv:2606.29836 tab. 7]; on SciERC it is below the Hierarchical Transformer, 69.14 against 70.91 [arxiv:2606.29836 tab. 8] | | You need POS tags, parses and NER from one CPU pipeline, and accept lower accuracy than the best tagger on each task | scispaCy models (`sm` or `md`) | The `md` model takes 33 ms per abstract on a 12-core CPU, but the table compares different tasks and is not like-for-like [arxiv:1902.07669 tab. 2]; its GENIA POS accuracy is 98.51, below 98.89 for the best BiLSTM-CRF with charCNN in the table [arxiv:1902.07669 tab. 3] | | You need a recurrent word-level tagger on CoNLL-style newswire data, without a pre-trained transformer | Word, character and affix embeddings in a recurrent tagger | Author-reported F1 is the highest of the survey's systems in Spanish, Dutch and German, and 0.76 below Chiu and Nichols on English (91.62) [arxiv:1910.11470 tab. 1]; the survey copies these from Yadav et al. (2018), so they are not re-run | | You want a recurrent tagger without a pre-trained transformer | BiLSTM with CRF output, Nadam, variational dropout and two stacked layers | The paper's most robust choices across its sampled configurations [arxiv:1707.09861 §6]; CRF was better on NER in 90.5% of 232 configurations [arxiv:1707.09861 tab. 3] | | Your text is Dutch and you have labelled NER or POS data | Monolingual Dutch BERT-base (BERTje) fine-tuned with a token head, compared with multilingual BERT on your data | On CoNLL-2002 test, 88.3 F1 against 80.7 for multilingual BERT, one run each [arxiv:1912.09582 tab. 1]; the same paper's CoNLL-2002 baseline of 90.9 is higher [§4] | | You need entity mentions from a large set of scientific papers, and the tagger's own test split is the only check | SciBERT with a BiLSTM tagger, trained on your annotations | 86.27 F1 on the authors' own test data, 0.62 above the best baseline [arxiv:2606.31058 tab. 5]; the paper gives no training hyperparameters [arxiv:2606.31058 §3.3.1] | | You have per-word reading data (eye-tracking or EEG) and a BiLSTM-CRF tagger on short, noisy text | Add per-word-type features to the word inputs, after checking vocabulary coverage | Gains are author-reported on tweets, but ZuCo covers about 4 to 9% of the tweet vocabulary [arxiv:2606.26485 tab. 6], and the released code cannot rebuild the All_EEG rows [arxiv:2606.26485 code: models/BILSTMCRF.py#L48] | | You want to show that a tagger change helps | Score distributions over several seeds, not one run | Single runs moved F1 by about a point and reversed a published ranking [arxiv:1707.09861 §1] [§3]; the paper gives the protocol, see the section above | | You have no labelled data and cannot train | Few-shot word-by-word prompting, with low expectations | mGPT-1.3B reaches 15.0 average NER F1 at 4 shots, against 78.2 to 79.7 for the fine-tuned baselines [arxiv:2204.07580 tab. 9]; the paper says these baselines are not like-for-like [arxiv:2204.07580 §4.2.2] | | The encoder must stay frozen, and each task should store only a small file | Frozen backbone with trainable prefixes (P-tuning v2), if a few points below full fine-tuning is acceptable | The paper reports 0.1%-3% trainable parameters [arxiv:2110.07602 §1]; on dev, PT-2 is 92.8 against 92.6 for fine-tuning on CoNLL03 with RoBERTa-large, but 90.2 against 92.8 with BERT-large [arxiv:2110.07602 tab. 3] | | The encoder must stay frozen, and only a task head is trained | Not supported by this paper's results | The paper calls frozen-encoder results "far from satisfactory" [arxiv:1910.00883 §3.4]; that run trains the head only, with no prefixes | | You need to reproduce REST numbers from released data | Use rest14, rest15, rest16 and state the choice | The paper's REST setting cannot be rebuilt from the released code and data [arxiv:1910.00883 code: train.sh#L2] [arxiv:1910.00883 code: README.md#L70-L71] | Training settings the first paper states, for reference: learning rate 2e-5, 1500 steps, batch size 25 on LAPTOP and 16 on REST, and model selection on dev micro-F1 from step 1000, every 100 steps [arxiv:1910.00883 §3.1]. Five seeds are averaged in the paper [arxiv:1910.00883 §3.1]; the reference code runs one seed by default [arxiv:1910.00883 code: fast_run.py#L5-L6]. ## Implementation sketch Language-agnostic pseudocode for the common core. The head is the only part that changes between the fine-tuned variants. ```pseudocode # Sequence labeler: encoder, one task head, decode [arxiv:1910.00883 §2] # Input: token ids for one sentence, T tokens # Output: one tag per token from the BIEOS set def label_sentence(token_ids): # Contextual state for each token from the pre-trained encoder [§2.1] states = encoder(token_ids) # (T, d) # Task head: linear, GRU, SAN, TFM or CRF [§2.2] features = head(states) # (T, d) # Score every tag at every position [§2.2] [eq. 2] scores = features @ W_out + b_out # (T, 13) # Softmax heads: pick the best tag per token [§2.2] # CRF head: Viterbi over the whole sequence [eq. 8] return decode(scores) # (T,) ``` ```pseudocode # Turn BIEOS tags back into labelled spans [arxiv:1910.00883 code: seq_utils.py#L166-L172] def spans_from_tags(tags): spans = [] # Walk the tags; a span opens at B and closes at E, S is one token [§2] for start, end, label in find_spans(tags): # Drop a span whose sentiment tags disagree or that has no closing E if label_consistent(tags, start, end): spans.append((start, end, sentiment_of(tags[start]))) return spans ``` ```pseudocode # Cascade: boundary heads and a type head, then pair starts with ends [§3.2] # Input: per-token start, end and type scores from the three heads # Output: labelled spans # Simplified: the reference code also revises boundaries with two rules and # falls back to the type sequence [arxiv:2606.29836 code: Code/ner-cascade.py#L53-L120] def spans_from_cascade(start_p, end_p, type_p): spans = [] # Pair each start with an end; the pairing rule is simplified here [code] for i in starts(start_p): j = next_end_after(i, end_p) if j is not None: # Majority type over the subword tokens [code: ner-cascade.py#L53-L120] spans.append((i, j, majority_type(type_p, i, j))) return spans ``` Notes for implementers, each taken from the papers or their reference code: - Freeze the backbone only when you train prefixes. The released sequence-classification model sets `requires_grad = False` on the backbone and trains the prompt table and the head [arxiv:2110.07602 code: model/sequence_classification.py#L336-L337]. This card did not read the token-classification path, so check it before you copy this for tagging. - Fix the global batch size. The first paper's reference code multiplies the per-GPU batch by the GPU count, so the effective batch depends on hardware [arxiv:1910.00883 code: main.py#L159]. - Label each word once. That paper's reference code gives the first word piece the word's tag and labels later pieces with an extra class, so the model has 14 classes in practice [arxiv:1910.00883 code: glue_utils.py#L211-L212]. - Mask padding in attention heads. That code calls the SAN and TFM heads without a padding mask [arxiv:1910.00883 code: absa_layer.py#L455-L456]. - Expect invalid tag sequences from unconstrained decoding; that code repairs or drops them when it rebuilds spans [arxiv:1910.00883 code: absa_layer.py#L127]. - The first paper does not state the optimizer, schedule, gradient clipping, dropout or maximum sequence length. Treat any value you copy from its code as the authors' choice [arxiv:1910.00883 code: main.py#L105-L110]. - Keep the B- and I- pools apart when replacing tokens by label. The reference code of the augmentation study draws from a pool keyed by the full BIO tag, so an I- token is never replaced by a B- token [arxiv:2010.11683 code: augment.py#L139-L141]. - Regenerate augmented copies every epoch. That study's code makes new random draws each epoch from the original sentences [arxiv:2010.11683 code: train.py#L156-L172]. - Fix the demonstration seed in few-shot tagging. The reference notebook accepts a seed but does not use it, and draws demonstrations with `random.sample` [arxiv:2204.07580 code: evaluation/sequence_labeling/sequence_labeling_prediction.ipynb#L426-L429]. - Watch the context length. Its dataset truncates on the right [arxiv:2204.07580 code: evaluation/sequence_labeling/list_docs_dataset.py#L14-L20], so a long demonstration block could cut off the candidate tag. This is an inference from the code and was not run. - Seed every run and record the seed with its score. The BiLSTM code never sets a seed, so repeated runs will not match [arxiv:1707.09861 code: neuralnets/BiLSTM.py#L129] [code: neuralnets/BiLSTM.py#L334]. Seeding Python, NumPy and the Keras backend yourself is our suggestion, not the authors'. - Keep the tokenizer fixed between training and inference. Label boundaries follow token boundaries, and the scispaCy tokenizer adds prefix, suffix and infix rules and keeps listed abbreviations whole [arxiv:1902.07669 code: scispacy/custom_tokenizer.py#L48-L60] [code: scispacy/custom_tokenizer.py#L121-L126]. Changing it changes which spans can be labelled. This advice is ours. - Do not copy the scispaCy paper's feature list without checking. The paper lists lemma among the NER features, but the reference config has no lemma attribute [arxiv:1902.07669 code: configs/base_ner.cfg#L51]. - Label subword pieces on purpose. The Dutch BERT fine-tuning script puts the word's label on the first piece and gives continuation pieces an `O` label, and it computes the loss at every position, padding included [arxiv:1912.09582 code: finetuning/v1/run_benchmark.py#L157-L160] [#L176]. Mask the loss yourself if you want word-initial positions only. This is the card's reading of the script, and it was not run. - Expect the scispaCy code to differ from the 2019 models. The repository requires spaCy 3.7 and its configs use the spaCy 3 format, while the paper's models use spaCy 2.0.18 [arxiv:1902.07669 code: requirements.in#L6]. The paper states none of its optimizer, batch or epoch settings, so the repository configs are the only source for them. ## Contradictions - **Augmentation gains: text versus table.** The augmentation study says every method improves on the baseline on average [arxiv:2010.11683 §4.4]. Its table shows several full-set single-method cells equal to or below the baseline, for example SciBERT on i2b2-2010 full set: 87.5, 87.7, 87.8, 87.7 and 87.2 against 87.8 with no augmentation [arxiv:2010.11683 tab. 3]. The card reads the full-set column as no reliable gain. - **CRF on NER: two setups.** The BiLSTM sweep finds CRF better on NER in 90.5% of sampled configurations [arxiv:1707.09861 tab. 3]. The cascade study finds the CRF below the softmax head for BERT, RoBERTa and SciBERT [arxiv:2606.29836 tab. 7]. The sweep varies the whole BiLSTM design, while the cascade study fixes one encoder and changes only the head, so the two results are not comparable. - **Easy edits versus model-based filtering.** The augmentation study uses four label-preserving edits and reports gains, largest on small training sets [arxiv:2010.11683 §4.4]. The cascade study says easy text augmentations risk changing entity spans, and it uses model-based candidates instead [arxiv:2606.29836 §3.3]. The two papers test different data, tag sets and models, so neither result settles the other. - **Survey gains: text versus table.** The survey says the best English word-based model beats the others by 0.52 [arxiv:1910.11470 §7]. Its Table 1 gives 0.36 against Yang et al. (91.26) and 1.52 against Huang et al. with gazetteers (90.10) [arxiv:1910.11470 tab. 1]. The card lists this as a Pitfall. The table values are used here. - **CRF gains: small in one paper, mixed in another.** The first paper finds a CRF slightly above the linear head on both data sets (60.78 against 60.43 on LAPTOP, 74.06 against 73.22 on REST) [arxiv:1910.00883 tab. 2]. The cascade study finds CRF lower than the softmax head for BERT, RoBERTa and SciBERT, and higher only for T5 (85.08 against 84.85) [arxiv:2606.29836 tab. 7]. The first paper uses BIEOS tags and the second uses BIOES, and the second runs one seed per model, so the results are not directly comparable. - **Scispacy NER: the authors' count and their table.** The authors say the models are competitive with strong baselines on 5 of 9 data sets [arxiv:1902.07669 §4.2]. In the table's Baseline column, the `md` model scores higher on six data sets and lower on three (CRAFT, AnatEM and BC2GM) [arxiv:1902.07669 tab. 5]. The paper does not say which criterion gives five. - **Not comparable across papers.** The first paper scores span F1 with exact boundaries and sentiment [arxiv:1910.00883 code: glue_utils.py#L430]. The augmentation study scores span F1 with exact token positions [arxiv:2010.11683 code: train.py#L111-L122]. The cascade study scores exact-match micro F1 over (type, start, end) spans [arxiv:2606.29836 code: Code/ner-cascade.py#L206-L212]. The mGPT paper reports accuracy for POS and F1 for NER, with few-shot prompting against fine-tuned baselines [arxiv:2204.07580 §4.2.2]. Compare numbers only within one paper's table. - **Frozen encoder: plain head versus prefixes.** The BERT study reports a frozen encoder with a task layer as "far from satisfactory" [arxiv:1910.00883 §3.4]. The P-tuning v2 study also freezes the backbone, but trains prefix vectors at every layer, and reports dev scores near full fine-tuning on its tagging benchmarks [arxiv:2110.07602 tab. 3]. The two set-ups train different parts and use different data, so neither result settles the other. - **Reporting protocol.** The BiLSTM sweep compares score distributions over many seeds [arxiv:1707.09861 §3]. The keyphrase study averages five runs, but it keeps each run's best of 20 epochs by test F1 and reports no variance or significance test [arxiv:2606.26485 §4.1] [code: models/SATT_BILSTMCRF.py#L217-L223]. Test-set selection biases the reported F1 upward, so its gains and the sweep's findings are not comparable. Neither result settles the other. - **REST data.** The first paper does not name its REST release. Its repository runs on a set that the README calls invalid, so the REST numbers cannot be checked against the code [arxiv:1910.00883 code: train.sh#L2] [arxiv:1910.00883 code: README.md#L70-L71]. ## Papers - [Reimers and Gurevych, 2017](/papers/1707.09861): a BiLSTM sweep with softmax or CRF output, arguing for score distributions over seeds; CRF was better on NER in most sampled configurations. - [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](/papers/1910.00883): BERT-base with five task heads for joint aspect and sentiment tagging on LAPTOP and REST, with a linear head already above the non-BERT baselines. - [An Analysis of Simple Data Augmentation for Named Entity Recognition](/papers/2010.11683): four label-preserving augmentations for NER, tested with a BiLSTM and a SciBERT tagger on two specialist data sets, with gains largest on small training sets. - [mGPT: Few-Shot Learners Go Multilingual](/papers/2204.07580): word-by-word NER and POS tagging by prompting a decoder-only model, with results far below fine-tuned encoders on the paper's own benchmarks. - [Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective](/papers/2606.29836): a SciBERT cascade of start, end and type heads for four kinds of scientific entity, with a model-based augmentation step, reported as F1 87.00 on the authors' own test split. - [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](/papers/1902.07669): spaCy-based biomedical pipelines with transition-based NER and joint POS tagging and parsing, reported as competitive on NER and fast on CPU. - [P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](/papers/2110.07602): deep prefix tuning on a frozen backbone with a linear token head, reported near full fine-tuning on several tagging benchmarks, on dev sets, with the token-classification baseline not in the release. - [Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs](/papers/2606.26485): per-word EEG and eye-tracking features added to BiLSTM-based keyphrase taggers for tweets; author-reported F1 gains, with released code that differs from the paper. - [BERTje: A Dutch BERT Model](/papers/1912.09582): a Dutch BERT-base model fine-tuned for token-level NER and POS tagging, reported above multilingual BERT on the paper's Dutch test sets, one run per cell. - [Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities](/papers/2606.31058): a SciBERT and BiLSTM tagger for four NLP entity types, reported at 86.27 F1 on its own test data and used to score paper novelty. - [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](/papers/1910.11470): a survey of 83 NER systems in four families, with CoNLL 2002/2003 and DrugNER tables; its affix-augmented word+character tagger gives the highest reported scores in three of four CoNLL languages.