Conditional random field

Overview

A conditional random field (CRF) puts one score on a whole structured output, not one score per item. The score is built from a term for each item and a pairwise term for each linked pair. The model normalises over all possible outputs, is trained by maximising the likelihood of the gold output, and predicts the highest-scoring one.

The cards use the name in two ways. In sequence labelling, the items are tokens and the pairs are neighbouring tags. This covers NER, keyphrase tagging and event trigger detection. The emission term scores each token's tag, the transition term scores each pair of neighbouring tags, and Viterbi search gives the best sequence 1910.00883 §2.2 1910.00883 eq. 7 1910.00883 eq. 8 2404.09383 eq. 1. In dense vision, the items are pixels and the pairs are pixel links. A deep network refines a depth map with these links, and the link weights are learned with attention instead of a probability model 2203.01502 §3.1 2203.01502 §3.3.

Within sequence labelling, the invariant all variants share is that the output is a sequence, and the tag at one position is scored together with the tags around it. Where a card states the objective, it is a likelihood over whole sequences 1910.00883 eq. 7 2104.05919 eq. 9. Where a card names a decoder, it is Viterbi 1910.00883 eq. 8 2404.09383 eq. 1. The event-trigger and keyphrase cards do not name a decoder. The cross-lingual model adds a source-language term to the objective 2404.09383 eq. 7. What changes is how the emission and transition scores are computed: from hand-built sparse features, from prototype distances, or from a neural encoder such as BERT or a character-plus-word BiLSTM. The dense-vision variant shares only the idea of pairwise links; it does not share the objective or the decoder.

Variants

Linear-chain CRF head on BERT

A single CRF layer over BERT token states, with 13 BIEOS-style tags for end-to-end aspect-based sentiment analysis. The emissions come from a linear map of the encoder states, and the transitions form a learned 13 by 13 matrix 1910.00883 §2.2 1910.00883 eq. 7. Reported F1: 60.78 on LAPTOP and 74.06 on REST, against 60.43 and 73.22 for the linear head on the same encoder 1910.00883 tab. 2. The paper's SAN head scores 74.72 on REST, the best in its table 1910.00883 tab. 2. Card: Exploiting BERT for End-to-End Aspect-based Sentiment Analysis.

The paper does not restrict transitions, and the released code keeps the constraint mask at all ones. Viterbi can therefore output tag sequences that break the BIEOS scheme, and the span rebuild silently drops or repairs them 1910.00883 §2 absa_layer.py.

CRF head on a domain-pretrained encoder

MatSciBERT tests three output layers on one encoder: a softmax (LM-Linear), a CRF on the encoder states (LM-CRF), and a 2-layer BiLSTM with hidden size 300 followed by a CRF (LM-BiLSTM-CRF). Each word takes the output at its first WordPiece 2109.15290 §2.4.1 2109.15290 §2.4.4. On the test sets, LM-CRF has a higher Macro-F1 than LM-Linear in all three NER tables: 65.00 against 63.63 on SOFC-Slot, 81.92 against 81.56 on SOFC, and 86.30 against 85.56 on Matscholar 2109.15290 tab. 2 2109.15290 tab. 3 2109.15290 tab. 5. The same holds for Micro-F1 and for the SciBERT encoder. Across these twelve test cells the gain runs from 0.36 to 1.83 points. The paper reports means over three seeds with no spread, so gaps near one point are hard to judge 2109.15290 tab. 2.

The BiLSTM layer helps on the two SOFC sets. On SOFC-Slot, LM-BiLSTM-CRF is 0.92 above LM-CRF in Macro-F1, and on SOFC it is 0.15 above. On Matscholar it is 0.64 below in Macro-F1 and 0.70 below in Micro-F1 2109.15290 tab. 2 2109.15290 tab. 3 2109.15290 tab. 5. Card: MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction.

CRF output layer under data augmentation

A CRF over BIO tags sits on two taggers, a BiLSTM with GloVe embeddings and SciBERT, in a study of simple token-level augmentation for NER 2010.11683 §4.2 2010.11683 tab. 3. The paper studies the augmentation, not the output layer. It does not compare the CRF with a softmax head, so its numbers say nothing about whether the CRF helps.

With all four augmentations combined, the mean gain over the no-augmentation baseline is +3.3 F1 for the LSTM tagger and +1.8 F1 for SciBERT, averaged over eight columns 2010.11683 tab. 3. The gain is largest on the smallest set: with 50 sentences, SciBERT rises from 68.1 to 71.2 on MaSciP and from 35.1 to 41.5 on i2b2-2010 2010.11683 tab. 3. On the full training set the effect is flat or negative in several cells; for SciBERT on i2b2-2010 it goes from 87.8 to 87.2 2010.11683 tab. 3. The text says all methods improve on the baseline 2010.11683 §4.4, which the full-set column does not support. Card: An Analysis of Simple Data Augmentation for Named Entity Recognition.

The paper does not state transition constraints for its CRF. The released code trains with an unconstrained transition matrix and decodes with constraints models.py models.py. Those two choices can give different outputs, so check which one your decoder uses.

Log-linear CRF with hand-built features

Each pairwise potential is the exponential of a dot product between a weight vector and a sparse feature vector. The features are built from the two tags, the sentence and the position. The paper uses the standard NER templates of Sha and Pereira, which it does not list 2404.09383 eq. 2 2404.09383 §2.2. The objective is convex, and training uses L-BFGS until convergence with the CRFsuite toolkit 2404.09383 §2.2 2404.09383 §5.2. With 100 target sentences, monolingual F1 runs from 42.76 (Marathi) to 62.71 (West Frisian) 2404.09383 tab. 2. Card: Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields.

A survey lists CRFs with hand-built features among the classic supervised NER systems 1910.11470 §6.3. Its CoNLL table has a CRF system of Passos et al. (2014) at 90.90 F1 on English, which uses a gazetteer [arxiv:1910.11470 tab. 1]. The survey copies that number and does not re-run it. It is on CoNLL 2003 English, not on the cross-lingual data above, so the two sets of numbers are not comparable. Card: A Survey on Recent Advances in Named Entity Recognition from Deep Learning models.

For cross-lingual transfer, the same paper adds an atomic language-ID feature and conjoins it with every template, which doubles the template count 2404.09383 §3.1. With Spanish as a source, Galician rises from 57.64 to 71.46 F1 2404.09383 tab. 2.

CRF output layer on a BiLSTM tagger, compared with softmax across seeds

The tagger reads pre-trained word embeddings, with optional casing and character features, then one to three stacked BiLSTM layers, then a softmax or a CRF output 1707.09861 §4.1 1707.09861 §4.2. The paper compares the two outputs over sampled configurations and many seeds, not on one hand-tuned setup 1707.09861 §5. On 232 NER configurations the CRF gave the higher score in 90.5% of them, with a median difference of −0.66 F1 points. The paper defines the difference as softmax minus CRF, so the sign favours the CRF 1707.09861 §5 1707.09861 tab. 3. The CRF also had a smaller spread on NER, σ = 0.0060 against 0.0082 for softmax (Brown-Forsythe p < 10⁻³) 1707.09861 §5. The CRF won on four of the five tasks. It did not win on Events, where softmax was better in 61.9% of configurations 1707.09861 tab. 3 1707.09861 §6.1. A softmax output gained more from extra stacked LSTM layers than a CRF did 1707.09861 §6.1 fig. 3. Card: Score spreads of LSTM sequence taggers, softmax against CRF output.

Neural CRF with a character-plus-word BiLSTM

The word input is the final state of a character LSTM over the word's characters, concatenated with a word embedding. A sentence-level BiLSTM runs over these inputs, and each tag-pair potential is the exponential of a transition weight plus a tag output vector times the hidden state at that position 2404.09383 eq. 3 2404.09383 eq. 4 2404.09383 eq. 5 2404.09383 §2.3. With 10,000 target sentences it beats the log-linear CRF on all five monolingual targets, by 0.61 to 4.79 F1 2404.09383 tab. 2. With 100 target sentences it loses on all five, by 0.75 to 8.45 F1 2404.09383 tab. 2. Card: Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields.

Cross-lingual neural CRF with shared parameters

The character LSTM, the sentence BiLSTM, the transition weights, the projection and the tag output vectors are shared across languages. Word embeddings and a learned language vector are language-specific, and the language vector is fed into the projection 2404.09383 §3.1 2404.09383 eq. 6. Training maximises the target log-likelihood plus μ times the source log-likelihood, with μ > 0 2404.09383 eq. 7. The paper does not report a value for μ. With 100 target sentences, this model beats the log-linear CRF with the same source in all ten transfer pairs, by 3.97 to 9.80 F1 2404.09383 tab. 2. For Galician, the neural CRF goes from 49.19 (monolingual) to 68.02 with French, while the log-linear CRF goes from 57.64 to 58.22 2404.09383 tab. 2. The choice of source matters: Spanish gives 76.40 for the neural CRF 2404.09383 tab. 2.

Prototype-emission linear-chain CRF for zero-shot trigger detection

The trigger model scores a tag sequence with a linear-chain CRF, but its emissions come from distances to class prototypes, not from a learned linear map. Emissions use a TapNet-style projection, solved in closed form from the class vectors and a set of reference vectors 2104.05919 §2.2 2104.05919 eq. 6 2104.05919 eq. 7. The transitions are two diagonal matrices, one for staying in the same event type and one for steps involving O, not a full matrix 2104.05919 eq. 8. Training minimises the negative log-probability of the tag sequences, plus a term that keeps the reference vectors close to orthonormal 2104.05919 eq. 9. For zero-shot use, the model first trains on pseudo-labels from cosine similarity between tokens and class vectors, and marks uncertain tokens as unknown 2104.05919 §2.2.

The paper's trigger results are for whole systems. TapKey, the trigger model, reaches 55.19 trigger identification F1 with no seen event types, against 3.19 for prototype networks 2104.05919 tab. 8. The table does not isolate the CRF layer, so these numbers do not show what the CRF adds. With 10 frequent seen types, TapKey plus the generator reaches 72.31 trigger identification F1 2104.05919 tab. 8. The released repository contains argument extraction only, not the trigger model, so these zero-shot results cannot be reproduced from it. Card: Document-Level Event Argument Extraction by Conditional Generation.

BiLSTM+CRF keyphrase tagger with cognitive features

Keyphrase extraction on tweets is tagged with B, I, E, S and O [code: config.py#L14-L15]. Baselines are a BiLSTM and a BiLSTM+CRF [arxiv:2606.26485 §3.2.1] 2606.26485 fig. 2. The attention variants add a soft-attention or self-attention layer on top, with or without the CRF 2606.26485 §3.2.2 2606.26485 fig. 3. The cognitive features are per-word averages of EEG and eye-tracking values from the ZuCo corpus, not per-reader values at inference 2606.26485 §3.1.3.

On General-Twitter, the paper's plain Bi-LSTM scores 70.67 F1, and its SATT-BiLSTM+CRF with all EEG features scores 82.46 2606.26485 tab. 4. Its BiLSTM+CRF with no cognitive features scores 81.63 on the same set, and SATT-BiLSTM+CRF without features scores 81.74 2606.26485 tab. 4. Read this way, most of the gain over the plain BiLSTM on this set comes with the CRF output, and the EEG features add 0.72 F1 over the no-feature SATT model. These figures come from the card's reading of tab. 4; check that table before relying on the split. On Election-Trec the plain BiLSTM to SATT-BiLSTM+CRF gap is 61.67 to 67.82 2606.26485 tab. 4. The hyperparameters were tuned on each test set and the best test epoch is kept, so these F1 values are optimistic 2606.26485 §4.1 SATT_BILSTMCRF.py. The released SATT-BiLSTM+CRF does not build as written SATT_BILSTMCRF.py. Card: Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs.

Windowed attention CRF for monocular depth (not a sequence model)

NeW CRFs use a decoder whose refinement step is a fully connected CRF energy, computed only inside non-overlapping windows of 7 by 7 patches. A second step uses windows shifted by half a window, so information crosses window borders 2203.01502 §3.2 2203.01502 §3.4. The pairwise weights come from multi-head attention with a relative position term, not from a fixed colour and position kernel 2203.01502 §3.3 2203.01502 eq. 9. The output is a depth map, trained with a scale-invariant log loss rather than a likelihood 2203.01502 eq. 11. Nothing here is a tag sequence, and no Viterbi search is used.

The pairwise cost per iteration is hw(N² − 1) for N × N windows, against hw(hw − 1) for a full CRF over h × w nodes 2203.01502 eq. 3. On the Eigen split, adding the neural CRFs to a UNet-style baseline lowers Abs Rel from 0.069 to 0.055, and the full model reaches 0.052 2203.01502 tab. 5. In a separate study on 352 × 352 crops, window size trades accuracy for cost: Abs Rel goes from 0.061 at window 2 to 0.054 at window 32, while decoder time goes from 121 ms to 192 ms; window 48 runs out of memory in training 2203.01502 tab. 6. Card: NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation.

When to use which

The gaps in this table come from author-reported numbers. The BiLSTM rows of 2404.09383 tab. 2 come from single runs with no variance. The keyphrase row tunes on the test set, and the event-trigger row reports whole systems 2606.26485 §4.1 2104.05919 tab. 8. The CRF row for BiLSTM taggers 1707.09861 tab. 3 is a distribution over configurations, not a single run, so its spread can be compared with the other rows. The BERT rows 1910.00883 tab. 2 are on two review datasets, and the BiLSTM rows are on a 15-language NER set 2404.09383 §5.1. The augmentation rows 2010.11683 tab. 3 use five seeds, but the paper's own text overstates the full-set gains. Treat the choice as open until you have run your own comparison.

Implementation sketch

Language-agnostic pseudocode of the common core of the sequence variants. The BERT head, the BiLSTM variants and the prototype variant differ in how emissions is computed, and the prototype variant also uses diagonal transitions. See the comment at the end.

# Linear-chain CRF over token emissions  [arxiv:1910.00883 eq. 7]
# emissions: (T, 13) per-token scores from a linear map of the encoder states
# transitions: (13, 13) learned score for each tag-to-tag step

def sequence_score(emissions, transitions, tags):
    # Sum of emission scores at the chosen tag of each token  [eq. 7]
    total = sum(emissions[t, tags[t]] for t in range(T))
    # Plus the transition score for each pair of neighbouring tags  [eq. 7]
    total += sum(transitions[tags[t], tags[t + 1]] for t in range(T - 1))
    return total

def training_loss(emissions, transitions, gold_tags):
    # Negative log-likelihood of the gold sequence  [§2.2] [eq. 7]
    # The normaliser sums over every possible tag sequence  [eq. 7]
    return logsumexp_over_all_sequences(...) - sequence_score(...)

def decode(emissions, transitions):
    # Highest-scoring tag sequence, found by Viterbi search  [eq. 8]
    return viterbi(emissions, transitions)    # (T,)

# BiLSTM variant: emissions[t] = u[tag] @ hidden_t, where hidden_t is the
# BiLSTM state at position t  [arxiv:2404.09383 eq. 3]. Log-linear variant:
# emissions come from a sparse feature dot product  [arxiv:2404.09383 eq. 2].
# Prototype variant: emissions come from distances to class prototypes, and
# transitions are two diagonal matrices instead of one full matrix
# [arxiv:2104.05919 eq. 6-8]. sequence_score, the loss and decode stay the same
# in the sketch; the prototype paper's own loss adds an orthonormality term
# [arxiv:2104.05919 eq. 9].

The paper's score sums from t = 0 1910.00883 eq. 7 and does not spell out start or end terms, so an implementation has to decide whether to add them 1910.00883 eq. 7. The released BERT code has none absa_layer.py.

The windowed depth variant is not a sequence model and does not use this sketch. Its refinement step is a windowed attention block over pixel features 2203.01502 §3.2 2203.01502 §3.3.

Contradictions

No direct disagreement on a shared setting. The papers measure different things, on different encoders. 1910.00883 tab. 2 compares a CRF head with a softmax head on BERT, and finds a gain of 0.35 F1 on LAPTOP and 0.84 on REST. 2606.26485 tab. 4 reports a larger gap on General-Twitter, 70.67 for a plain Bi-LSTM against 81.63 for a BiLSTM+CRF with no features. That baseline has no CRF, and the task is tweet keyphrase tagging, so it is not directly comparable with the BERT head above. 1707.09861 tab. 3 compares a CRF with a softmax output on BiLSTM taggers over many configurations, and reports a larger CRF advantage on NER, with a median difference of 0.66 F1 points. Both point the same way. The magnitudes are not comparable: the first is a single-setting comparison on BERT, and the second is a median over sampled configurations of a different encoder. The MatSciBERT gains of 0.36 to 1.83 test points 2109.15290 tab. 2 2109.15290 tab. 5 overlap the BERT gains above, but the encoders, tasks and seed counts differ. The paper's Events result goes the other way 1707.09861 tab. 3. 2404.09383 tab. 2 compares a neural CRF with a log-linear CRF on a BiLSTM. The second paper reports that the neural CRF loses to the log-linear CRF in the monolingual low-resource setting [arxiv:2404.09383 tab. 2], so the advantage of a neural CRF depends on how much labelled data you have. 2010.11683 tab. 3 does not compare CRF with softmax, so it does not add a conflicting number. 2203.01502 tab. 5 compares its decoder with a UNet-style baseline, not with any sequence CRF; the shared name does not make the numbers comparable.

Papers

• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis: uses a linear-chain CRF as one of five task layers over BERT-base for joint aspect extraction and sentiment tagging, and compares it with linear, GRU, self-attention and transformer heads.
• An Analysis of Simple Data Augmentation for Named Entity Recognition: tests four token-level augmentations for NER on a BiLSTM and a SciBERT tagger, both with a CRF output layer, on two specialist data sets.
• NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation: a depth decoder whose refinement step is a fully connected CRF energy inside shifted 7 by 7 patch windows, with learned attention weights.
• Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields: compares a log-linear CRF with a character-plus-word BiLSTM CRF, monolingual and cross-lingual, on low-resource NER.
• A Survey on Recent Advances in Named Entity Recognition from Deep Learning models: a survey that groups NER systems into families and copies CoNLL and DrugNER scores from the cited papers; it places CRF among the feature-engineered systems.
• Score spreads of LSTM sequence taggers, softmax against CRF output: compares softmax and CRF output layers on BiLSTM taggers across sampled configurations and seeds, and reports the score spread for each choice.
• MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction: compares linear, CRF and BiLSTM-CRF heads on a materials-domain BERT and on SciBERT for three NER sets, reporting test F1 averaged over three seeds.
• Document-Level Event Argument Extraction by Conditional Generation: its trigger model is a prototype-emission linear-chain CRF for zero-shot event detection; the released code covers argument extraction only.
• Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs: BiLSTM and BiLSTM+CRF keyphrase taggers on tweets, with per-word EEG and eye-tracking features added at three points.