Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields

In plain words

Named entity recognition is the job of finding the names of people, places, organisations and other things in a sentence. Modern systems learn this from labelled example sentences, and for most languages there are hardly any. This paper asks whether a model can borrow knowledge from a related language that does have plenty of labels.

The idea is to train one model on the small target language and a large related language at the same time. Both languages share the parts of the network that read the letters of a word and that decide which labels fit together. Each language keeps its own word vectors and a small language tag tells the shared network which language it is reading. The authors compare this with a classic model built from hand-written features, which does better when the target language has very little data. With the shared model, the neural approach comes out ahead.

TL;DR

• Named entity recognition is cast as BIO tagging with 9 tags over 4 entity types §2.
• A neural CRF reads each word through a character LSTM and a sentence BiLSTM §2.3 eq. 3 eq. 4 eq. 5.
• For transfer, the character encoder, sentence BiLSTM, tag layers and transition weights are shared across languages, with a language vector §3.1 eq. 6.
• Training maximises the target-language log-likelihood plus a weighted source-language log-likelihood eq. 7.
• Author-reported: transfer raises F1 over the log-linear CRF in every low-resource case, by up to 9.8 points tab. 2.

Problem

Neural models need many labelled sentences before they beat feature-based CRFs. Labelling enough data for every language is too expensive, and CoNLL covers only four languages §1. The authors aim at a low-resource target language with 100 labelled sentences §5.1 and a related source language with 10,000 §5.1. Projection-based methods that need bitext are out of scope because they rely on an extra assumption that low-resource languages typically lack §4.

Method

Tagging scheme. Each token gets a BIO label. The authors use 9 tags for person, location, organisation and miscellaneous types: B and I for each type, plus O §2.

CRF. The model defines a globally normalised conditional distribution over tag sequences. It is a product of potentials over adjacent tag pairs divided by a partition function summed over all tag sequences eq. 1 §2.1.

Log-linear baseline. Each potential is an exponential of a dot product between a weight vector and a sparse feature vector built from the two tags, the sentence and the position eq. 2 §2.2. The features are standard NER templates from Sha and Pereira, not enumerated in the paper §2.2. The objective is convex.

Neural potential. The potential is the exponential of a transition weight between the two tags plus an output tag vector times a matrix times the word's contextual representation eq. 3 §2.3.

1. The contextual representation of word i is the concatenation of the forward and backward hidden states of a sentence-level BiLSTM at position i eq. 4.
2. The BiLSTM input for word i concatenates the final state of a character-level LSTM run over the word's characters with the word embedding eq. 5 §2.3.
3. All parameters (the sentence BiLSTM, the character LSTM and the word embeddings) are learned jointly §2.3.

Cross-lingual architecture. The potential is conditioned on a language label ℓ §3.1.

1. The language vector is a learned vector for ℓ. The contextual representation and the language vector are concatenated, projected by U with bias b, passed through tanh, and scored by a tag-specific output vector u(t_i) eq. 6.
2. Shared across languages: the transition weights a(·,·), the character LSTM, the sentence BiLSTM, the projection U, the tag output vectors u(t_i) and the bias b §3.1.
3. Language-specific: the word embeddings e(w_i) and the language vector §3.1.
4. The monolingual model also uses a single language vector, so monolingual and transfer models share one architecture and differ only in training data §3.1.

Transfer objective. Let D_τ be the target data and D_σ the source data eq. 7 §3.1.

1. Sum the log-likelihood of each target sentence given the target language.
2. Add μ times the sum of the log-likelihood of each source sentence given the source language, where μ > 0 is a trade-off parameter.
3. With several source languages, add one term per source language §3.1.

Log-linear cross-lingual variant. Add an atomic feature for the language ID and conjoin it with every existing feature template, doubling the template count §3.1.

# Shared char-level word encoder and neural CRF potential  [§2.3] [§3.1]
# Input:  sentence words; language id l; tags t
# Output: sum of log-likelihoods used in the transfer objective  [eq. 7]

def word_input(word, language):
    # Final state of a char LSTM over the word, shared across languages  [§3.1]
    char_summary = char_lstm(characters(word))
    # Word embedding is language-specific  [§2.3] [§3.1]
    word_vector = embedding[language][word]
    return concat(char_summary, word_vector)  # eq. 5

def contextual_states(words, language):
    # Sentence-level BiLSTM over the word inputs  [§2.3] [eq. 4]
    inputs = [word_input(w, language) for w in words]
    forward = lstm_forward(inputs)
    backward = lstm_backward(inputs)
    return [concat(f, b) for f, b in zip(forward, backward)]

def potential(tag_prev, tag, state, language):
    # Language-conditioned score for one tag transition  [eq. 6]
    hidden = tanh(U @ concat(state, language_vector[language]) + b)
    return exp(a[tag_prev, tag] + u[tag] @ hidden)

def transfer_objective(target_data, source_data, mu):
    # Target log-likelihood plus mu times source log-likelihood  [eq. 7]
    total = 0.0
    for words, tags in target_data:
        total += log_likelihood(tags, words, "target")
    for words, tags in source_data:
        total += mu * log_likelihood(tags, words, "source")
    return total

The log_likelihood helper is the usual CRF score minus the log partition function, computed over all tag sequences of the sentence eq. 1 §3.1. The paper does not give the forward or Viterbi algorithm explicitly.

Hyperparameters

Languages used tab. 1:

Source: tab. 1. The "Role" column is derived from the target and source lists in §5.1.

Results

Author-reported. Setup: a cross-lingual NER dataset from Ji et al. §5.1, 15 languages, 5 target languages with 100 training sentences in the low-resource setting and 10,000 in the high-resource setting §5.1. Metric: F1. Baseline: log-linear CRF with standard features §5.2. Each Δ column is neural minus log-linear tab. 2. The Δ values were checked against the table entries.

Source: tab. 2 (caption and table body). LR = low-resource (100 target sentences); HR = high-resource (10,000 target sentences). "—" = no source language (monolingual).

Headline numbers, from §5.2 and the abstract:

• Monolingual, high-resource: the neural CRF beats the log-linear CRF on all five targets tab. 2.
• Monolingual, low-resource: the log-linear CRF wins on all five targets tab. 2.
• Low-resource with transfer: the neural CRF wins in every listed pair tab. 2. The largest gain is +9.80 F1 for Galician with French tab. 2.
• High-resource with transfer: the effect is small, with positive Δ in every row tab. 2. The authors describe it as little to no effect §5.2.
• Remaining gap: the neural CRF trained on 10,000 target sentences still beats the transfer setting §5.2.

Paper vs code

No official code.

Searched: the paper and its LaTeX source contain no code or project link, and the arXiv metadata has no comments field. GitHub searches with the WebFetch tool found no results for: "cross-lingual character-level neural CRF NER" (title words), "neural crf cross-lingual ner", "crosslingual ner" within the rycolab organisation (an author's lab), and "cotterell ner crf". The paper does link the CRFsuite toolkit, which is a third-party tool and not the method's code.

Pitfalls

• Dimension mismatch. The footnote gives the sentence-level BiLSTM as 2 layers with 200 hidden units each, and r2 = 100 §2.3 fn. 1. The contextual representation is the forward and backward state concatenated eq. 4, which would be 400-dimensional if each direction has 200 units. The paper does not say how the BiLSTM output is mapped to r2 = 100, so pick a projection or confirm the per-direction size before implementing.
• Character LSTM size is not stated. The footnote describes only the sentence LSTM's size, so the character LSTM hidden size, the embedding sizes and the projection size q are open.
• μ is not given. The trade-off must be positive §3.1, and no value or search procedure is reported. The result depends on it, and it must be chosen on your own validation data.
• Validation split. The paper says a validation split exists §5.1 but does not say what it is used for (early stopping, μ, learning rate).
• Sampling and batching. The objective sums over target and source sentences with weight μ eq. 7, but the paper does not describe how they are mixed in each update, the batch size, or whether the losses are averaged or summed per sentence.
• Learning rate for AdaDelta. The paper gives a learning rate of 1.0 §5.2. Implement it as stated; the paper does not explain why a learning rate is needed for this optimiser.
• Linear-CRF features. "Standard feature templates" §2.2 are not listed. Reproducing the baseline requires the templates of Sha and Pereira, which the paper does not reproduce.
• Table caption. The caption of tab. 2 says the comparison is in the low-resource transfer setting, but the table also reports the monolingual and high-resource settings. Read the columns as the setting names given above.
• Single runs. The paper does not report seeds, repeated runs or variance. Differences of a few F1 points (for example the high-resource gains for Dutch or Frisian in tab. 2) may be within noise.
• Monolingual neural model still has a language vector §3.1. If you implement the baseline without it, you are not reproducing the paper's monolingual model.
• Abstract claim. The "up to 9.8 points" headline is over the log-linear CRF in the transfer setting [abstract] tab. 2. It is not a gain over the monolingual neural CRF.
• Source language choice. The paper pairs each target with a chosen related source §5.1 tab. 2. It gives no general rule for choosing the source, and only shows results for those pairs.
• Languages with no sentence-level training data are outside the experiments.
• Dataset access. The data comes from a cross-lingual NER dataset §5.1, cited as Ji et al. The paper does not state how to obtain it here.

Lineage

• Builds on: the BiLSTM-CRF architecture of Lample et al. (2016), which the paper uses as its starting point, with the character-level LSTM encoder and the cross-lingual extensions added §2.3.
• Compared against: a log-linear CRF with standard NER features, trained with L-BFGS §2.2 §5.2.
• Related work: character-level tagging, transfer schemes within a language (domain adaptation) and projection-based cross-lingual transfer §4.
• Superseded by: not stated.