A Survey on Recent Advances in Named Entity Recognition from Deep Learning models

In plain words

Named entity recognition (NER) means finding names in text and labelling them: people, places, organisations and, in specialised fields, drugs or proteins. This paper is a survey. Its authors went through 154 papers, kept 83, and compared how well the different kinds of NER systems did on shared-task datasets.

The main finding is that neural networks which learn their own features beat older systems that depend on hand-built features, lists of known names and rules. The strongest models read each word both as a whole and as a sequence of characters. The authors then add an older idea, the prefixes and suffixes of words, as learned embeddings. This gives the best scores in three of the four CoNLL languages.

For an engineer, the take-away is that a word-plus-character recurrent tagger with affix features is a strong starting point for entity tagging. The paper also shows where each family of systems stands, so you can judge a baseline against it.

TL;DR

• Survey of NER systems in four families: knowledge-based, unsupervised and bootstrapped, feature-engineered supervised, and neural §6.
• The best neural model beats the best feature-engineered model in all four CoNLL languages, though the English gap is smallest §7 tab. 1.
• The affix-augmented word+character model of Yadav et al. (2018) gives the best author-reported scores in three of four CoNLL languages and the best MedLine score on DrugNER, but not the best overall DrugNER score tab. 1 tab. 2.
• The survey proposes no new model. Numbers are copied from the cited papers. The affix model comes from Yadav, Sharp and Bethard (2018), which shares two authors with this survey [main.bib].

Problem

NER is an early step in question answering, information retrieval, co-reference resolution and topic modelling §1. Feature-engineered systems need domain lexicons, gazetteers and hand-written orthographic features, and they often cover one language or one domain §2. No survey at the time compared modern neural NER with feature-based systems across both several languages and several domains §2. This paper fills that gap §1 §2.

Method

The survey has two parts: how the papers were chosen §3, and how the systems were grouped and compared §4 to §7.

Selection procedure §3:

1. Search Google, Google Scholar and Semantic Scholar with NER-related phrases such as "named entity recognition" and "deep learning models for named entity recognition" §3.
2. For every query, rank the hits by citations and read at least the three highest-ranked §3.
3. Keep a paper if it introduces a neural architecture for NER or is a top-performing model on an NER dataset §3.
4. For an architecture, keep the paper that first introduced it. Trace citations back to the original source §3.
5. Also keep papers that apply these systems to another language or domain §3.

The result is 154 articles reviewed and 83 selected §3. The compiled text does not list the 83 papers.

Datasets §4 and metrics §5: the CoNLL 2002 and 2003 newswire data cover Spanish, Dutch, English and German, with four entity types: PER, LOC, ORG and MISC §4. Exact-match F1 is the CoNLL metric §5. The DrugNER task (SemEval 2013 Task 9) uses relaxed and strict F1 §5. The Yadav et al. DrugNER rows use the official evaluation script tab. 2.

System families §6:

• Knowledge-based systems (lexicons and rules) §6.1. Precision is high and recall is low when the lexicon is incomplete §6.1.
• Unsupervised and bootstrapped systems §6.2.
• Feature-engineered supervised systems (HMM, SVM, CRF, AdaBoost, Maximum Entropy) §6.3.
• Feature-inferring neural systems §6.4, grouped by the units they read:
  • word level: a word embedding per token, a recurrent layer and a label layer §6.4.1 fig. 1;
  • character level: a recurrent network over characters, with labels mapped back to words §6.4.2 fig. 2;
  • word+character: two types. In the first, each word vector combines a word embedding with a convolution over the word's characters. In the second, the word embedding is joined with a recurrent summary of the word's characters, which is the Lample et al. (2016) design §6.4.3 fig. 3;
  • word+character+affix: the second type, extended with learned prefix and suffix embeddings for frequent affixes §6.4.4 fig. 4.

Word+character+affix tagger (the model §6.4.4 and §7 single out, fig. 4). The survey gives no pseudocode, so the following is a reading of the text and figure:

# Input:  words, a list of strings
#         affix sets: prefixes and suffixes seen at least T times in training
# Output: one label per word  [§6.4.4] [fig. 4]

def tag_sentence(words):
    # Recurrent summary of each word's characters  [§6.4.3] [fig. 4]
    char_vectors = [char_summary(word) for word in words]   # (n, d_c)
    # Learned embedding per word from a lookup table  [§6.4.3]
    word_vectors = [embed(word) for word in words]          # (n, d_w)
    # Learned embeddings of the frequent prefixes and suffixes  [§6.4.4]
    affix_vectors = [affix_embed(word) for word in words]   # (n, d_a)
    # Join the three parts per word; joining by concatenation is assumed,
    # not stated  [fig. 4]
    inputs = [
        concat(word_vec, char_vec, affix_vec)
        for word_vec, char_vec, affix_vec in zip(
            word_vectors, char_vectors, affix_vectors
        )
    ]
    # Sentence-level recurrent layer over the joined vectors  [§6.4.3]
    context = sentence_rnn(inputs)                          # (n, h)
    # Per-word scores for each label  [§6.4.3]
    scores = output_layer(context)                          # (n, labels)
    # Softmax takes the top label per word; a CRF decodes the whole
    # sequence  [§6.4.3]
    return decode(scores)

Key claim from the paper: the affix embeddings add information that the character RNN does not capture, and only frequent affixes help §6.4.4 (see Pitfalls for the threshold).

Hyperparameters

The survey itself trains no model. The values below are the ones the compiled text states for the affix model, plus the setting of each table row.

Results

Author-reported. The survey copies results from the cited papers on the CoNLL 2002 and 2003 data tab. 1 and on the DrugNER data of SemEval 2013 Task 9 (MedLine and DrugBank test data) tab. 2. Hardware and random seeds are not stated. The affix rows come from Yadav et al. (2018) §6.4.4. The text does not say who ran the "re-implementation of Lample et al." row tab. 1.

CoNLL F1 (Table 1) — "-" means not reported in the table. Bold marks from the paper (best per category) are not reproduced here.

Source: tab. 1. Columns: SP = CoNLL 2002 Spanish, DU = CoNLL 2002 Dutch, EN = CoNLL 2003 English, GE = CoNLL 2003 German.

DrugNER (Table 2) — SemEval 2013 Task 9, MedLine and DrugBank test sets. Source: tab. 2. Scores are integers because the official script outputs no decimals for the Yadav et al. rows.

Headline numbers as stated in the text:

• The best feature-inferring neural system beats the best feature-engineered system in all four CoNLL languages §7 tab. 1.
• The word+character affix model reaches 87.26 (Spanish), 87.54 (Dutch), 90.86 (English) and 79.01 (German) at 150 epochs tab. 1.
• The affix model is within 1 point of the best English system §7.

Paper vs code

No official code. This is a survey and has no code of its own. The paper links code for individual systems §6: an ixa-pipe NER tool, CharNER, a TensorFlow word+character CNN tagger, Epitran, NeuroNER, the Lample tagger and SENNA. None of these is this paper's code, and the paper does not say these authors wrote them. The affix extension has a link to a repository under the first author's account §6.4.4. That code belongs to Yadav et al. (2018), not this survey, so it is not recorded as code here. Per the wiki rule for survey papers, no further GitHub search was run.

Discrepancies: the survey's numbers were not checked against the cited papers, which were not read. Inconsistencies inside the survey are listed under Pitfalls.

Pitfalls

• Intro gains do not all reproduce from Table 1. The Spanish (1.59) and German (2.34) gains match the Lample et al. (2016) word+character model against the best feature-engineered system, before the affix extension §1 tab. 1. The English (0.36) and Dutch (0.14) gains cannot be matched exactly: the Dutch gap between Yang et al. (85.19) and Agerri and Rigau (85.04) is 0.15 tab. 1.
• Discussion gaps that do not reproduce. The 0.52 English gain of Chiu and Nichols over the best word-based model §7 does not match Table 1: 91.62 minus 91.26 (Yang) is 0.36, and minus 90.10 (Huang with gazetteer) is 1.52.
• Character comparison uses a different system. The 5.12 (English) and 2.54 (German) gains over the "best character-based model" §7 equal the gaps to Gillick et al. BTS (86.50, 76.22), not to CharNER (84.52, 70.12), which the text cites tab. 1.
• DrugNER gains are against a weaker baseline. The 8.90 (MedLine) and 3.50 (overall) gains of the word+character model §7 are over Rocktäschel et al. (58.10, 71.50) tab. 2. Against the best feature-engineered system, Liu et al. (68.25 MedLine, 78.37 overall), the word+character row is 1.25 and 3.37 points lower.
• Epochs differ across rows. The affix model is 100 epochs in one row and 150 in the headline row tab. 1. Only the 150-epoch row is the headline number. Epoch counts for other systems are not stated.
• Feature-engineered vs. neural comparisons are not like for like. "Dict" marks systems that use gazetteers or lexicons, so the comparison mixes external resources with learned features tab. 1.
• Relaxed vs strict scores. The Chalapathy et al. DrugNER score is a "relaxed" result, and the paper notes the affix numbers are from an integer-output script tab. 2. Do not mix these with strict-F1 numbers.
• Affix threshold and n-gram size are not stated. The text says only frequent affixes count §6.4.4. The LaTeX source has a commented-out grid (n = 2, 3, 4; T from 10 to 200). That grid is not in the compiled paper and is not used as a stated value here.
• How affix vectors join the word vector is inferred. The text says affix embeddings are learned "alongside" word embeddings and character RNNs §6.4.4. Concatenation is inferred from fig. 4 and from the Lample design, not stated.
• Survey selection is not reproducible from the card. The procedure uses three search engines and citation-count sorting §3. The 83 papers are not listed in the compiled text, and no date cut-off is given.
• Numbers are copied, not re-run. Apart from the re-implementation row, the values come from the cited papers, each with its own splits, preprocessing and epochs. Compare them only as reported.

Lineage

• Builds on: Collobert et al. (2011) word-level convolution with a CRF §6.4.1 tab. 1; Lample et al. (2016) word+character tagger §6.4.3 tab. 1; Yadav et al. (2018) adds learned affix embeddings to the Lample design §6.4.4. No card for these papers exists in this wiki yet, and the compiled text gives no arXiv ids for them.
• Compared against: the feature-engineered systems of Agerri and Rigau (2016), Carreras et al. (2002) and Passos et al. (2014) tab. 1; the DrugNER systems of Liu et al. (2015) and Rocktäschel et al. (2013) tab. 2.
• Superseded by: not known from this wiki's content.