Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks

In plain words

Tagging each word of a sentence, for example marking nouns, names of people or the span of a phrase, is usually done with a neural network that reads the sentence in both directions. Such a network has many settings: its size, which pre-trained word vectors it starts from, which optimizer trains it, and how the final labels are picked. Choosing these by hand from a few runs is unreliable, because one training run can land well above or below another just from its random start.

The authors instead randomly sampled a very large number of setups, trained them, and counted which choice tended to win across five tagging tasks. Some choices, such as the word vectors, the optimizer and the output layer, made a large difference. Others, such as the number of recurrent units, barely mattered. The result is a recommended configuration to start from on a new tagging task, not a guarantee for every dataset.

TL;DR

• Empirical study of a bidirectional LSTM tagger, with optional character encoder and optional CRF output, over five sequence tagging tasks and more than 50,000 sampled setups §1.
• Choices with large effects: the pre-trained word embeddings (Komninos et al. won most often), the optimizer (Nadam won most often), gradient normalization with threshold 1, variational dropout on both output and recurrent connections, and a CRF output layer §7.1 §7.3 §7.4 §7.6 §7.7.
• Choices with small effects: the number of LSTM layers (two was best most often), the number of recurrent units (about 100 per direction), the character encoder (CNN recommended for speed) and the tagging scheme (BIO recommended) §7.2 §7.5 §7.8 §7.9.
• The reported numbers are win shares and median differences, not absolute F1 for the recommended configuration tab. 4–tab. 19.
• Random seed alone changed one configuration's test F1 by up to 8.23 points on the Entities task tab. 2.

Problem

Good sequence taggers need many design choices and hyperparameters, and the choice often decides between mediocre and state-of-the-art results §1. Exhaustive grid search is impractical: the paper tuned 11 hyperparameters and would need more than 6 million combinations per task to cover even small value sets §2. Random search is cheaper but does not learn from earlier trials, and a single badly chosen value (such as a far too large learning rate) can ruin many trials §2. Bayesian optimization adapts, but adds implementation complexity §2.

The paper also argues that single-run comparisons are not meaningful. The random seed alone can change results enough to reverse a conclusion, so conclusions must come from distributions over many configurations §5 tab. 2.

Method

Network

Each token is represented by three parts, which are concatenated §3 fig. 1:

1. A pre-trained word embedding. Before lookup, the token is lowercased and numbers are replaced by a special token §6.
2. A capitalization feature with seven classes: numeric, mainly numeric (more than half of the characters are digits), all lower, all upper, initial upper, contains digit, other. The paper describes it as a seven-dimensional one-hot vector §3.
3. Optionally, a fixed-size character representation fig. 2:
   • CNN variant: each character gets a randomly initialised 30-dimensional embedding; 30 filters of width 3 (character trigrams) are applied, followed by max-over-time pooling, giving 30 dimensions §3.
   • BiLSTM variant: the same character embeddings are read by a BiLSTM with 25 units per direction; the last outputs of the two directions give 50 dimensions §3.

The concatenated vectors feed one or more stacked bidirectional LSTM layers. Forward and backward outputs of each layer are concatenated §3. Units can be set per layer, but a later layer may not be larger than an earlier one §6.1.

The output layer is either a dense softmax per token, which picks each token's tag independently §3, or a dense linear layer followed by a linear-chain CRF, which scores the whole tag sequence §3.

Dropout

Three schemes were compared §6.1 §7.7:

• No dropout.
• Naive dropout: a new mask at every time step; recurrent connections are not dropped §6.1.
• Variational dropout: the same mask is reused across all time steps of the sentence, and it is applied to the output units and to the recurrent units with separate rates §6.1 §7.7.

Dropout rates were sampled from {0.0, 0.05, 0.1, 0.25, 0.5} §6.1. The paper does not name a single recommended rate.

Gradient handling

Two strategies were compared §6.1 §7.4. For a threshold τ:

• Clipping: each gradient component g_ij becomes max(−τ, min(τ, g_ij)).
• Normalization: the gradient g becomes (τ / ‖g‖₂) · g whenever ‖g‖₂ > τ.

The paper states the normalization formula for a single gradient vector. It does not say whether the norm is taken over all parameters or per parameter tensor §6.1.

Training and model selection

Mini-batches of 32 sentences were used §6. Training stops when the development score has not improved for more than 5 epochs. The test score reported is the one from the epoch with the best development score §6. The paper does not state a maximum number of epochs.

Keras 1.2.2 was used §6. Backends were Theano 0.8.2 or TensorFlow 0.12.1 §7.11.

Post-processing of tags

Classifiers can emit invalid tags, such as an I- tag with no preceding B- tag, especially the softmax classifier §3 §7.6. The two repair options are to set such tags to O or to change them to B-. Setting them to O was usually slightly better §3.

Evaluation

Chunking requires an exact span match, with a single extra or missing token counted as an error §4. NER, entities and events are scored with F1 over entity spans §4. POS is scored by token accuracy §4. Tagging schemes compared: BIO, IOB and IOBES §6.1 §7.5.

Each sampled configuration was evaluated with every option of the factor under study, so options are compared on the same configuration §5. The paper reports, per task, the share of configurations in which an option was best, the median difference to the best option, and the standard deviation of scores §5.

Statistical tests §5:

• A two-sided binomial test at p < 0.01 decides whether one option wins significantly more often.
• A Brown-Forsythe test at p < 0.01 compares standard deviations.
• Options that tie with the best one are marked.

Pseudocode

The tagger forward pass, following the description in §3 and the layer order in the code:

# Tagger forward pass  [§3] [fig. 1]
# Input:  tokens (B, N), casing (B, N), chars (B, N, C)
# Output: per-token tag scores (B, N, T)

def tag_scores(tokens, casing, chars, params):
    # Pre-trained word vectors  [§6] [code: neuralnets/BiLSTM.py#L105]
    word_vecs = embedding_table[tokens]          # (B, N, d_word)
    # Capitalization class, 7 classes  [§3]
    case_vecs = casing_features(casing)          # (B, N, 7)
    # Character vector per word: CNN or BiLSTM  [§3] [fig. 2]
    char_vecs = char_encoder(chars, params)      # (B, N, d_char)
    # Concatenate the three parts of each token  [§3]
    x = concat(word_vecs, case_vecs, char_vecs)  # (B, N, d_in)
    # Stacked bidirectional LSTM layers  [§3] [§6.1]
    for units in params.lstm_units:
        x = bilstm(x, units, params.dropout)
    # Softmax per token, or linear layer then CRF  [§3]
    return output_layer(x, params.classifier)

Decoding and one training step:

# Decoding  [§3] [code: neuralnets/keraslayers/ChainCRF.py#L337-L341]
def decode(scores, classifier, transitions):
    # Softmax: most probable tag for each token  [§3]
    if classifier == "softmax":
        return argmax(scores, axis=-1)           # (N,)
    # CRF: best whole-sentence tag sequence (Viterbi)  [§3]
    return viterbi(scores, transitions)          # (N,)

# One optimizer step on a mini-batch  [§6] [§6.1]
# Input:  mini-batch of sentences, tags (B, N); threshold tau
# Output: updated parameters

def train_step(batch, params, optimizer, tau):
    # Loss: per-token cross-entropy or sentence NLL from the CRF  [§3]
    scores = tag_scores(batch.tokens, batch.casing, batch.chars, params)
    loss = tagging_loss(scores, batch.tags)
    # Gradient with respect to all trainable parameters  [§6.1]
    grads = gradients(loss, params)
    # Normalize when the norm exceeds tau; the norm scope is not stated
    # in the paper  [§6.1] [§7.4] [code: neuralnets/BiLSTM.py#L229-L230]
    grads = normalize_norm(grads, tau)
    # Update with the chosen optimizer; Nadam was best  [§7.3]
    optimizer.apply(grads)

Model selection:

# Early stopping and model selection  [§6]
for epoch in range(max_epochs):
    train_one_epoch(params)
    dev_score, test_score = evaluate(params)
    # Keep the test score of the epoch with the best dev score  [§6]
    if dev_score > best_dev:
        best_dev, best_test = dev_score, test_score
        since_best = 0
    else:
        since_best += 1
    # Paper: stop after more than 5 epochs without dev gain  [§6]
    # Code stops at 5 such epochs  [code: neuralnets/BiLSTM.py#L427-L428]
    if since_best >= 5:
        break

Hyperparameters

Recommended values from the paper's summary, followed by the sampled ranges and the code defaults. "Not stated" means the paper does not give a value.

Results

All numbers are the paper's own, reported on its sampled setups. They are not absolute benchmark results. The table below averages each option's share of wins over the five tasks. Source tables are listed in the last column.

Further results §7 tab. 2–tab. 19:

• Seed noise (same configuration, different seeds). Median difference per task: POS 0.17, chunking 0.17, NER 0.38, entities 0.72, events 0.43 points. Maximum difference: POS 1.55, chunking 0.81, NER 2.59, entities 8.23, events 1.73 tab. 2.
• Word embeddings. Komninos et al. won on POS with 93.5% of configurations. Its median accuracy advantage over GloVe2 on POS was 4.97 points §7.1 tab. 4. Levy dependency won chunking with 60.8%, and GloVe3 won NER with 43.6%; both differences from Komninos were not significant §7.1 tab. 4.
• Character representation. Character features were significant only for POS, chunking and events. CNN and BiLSTM did not differ significantly §7.2 tab. 5.
• Optimizers. Nadam won on average. RMSProp was up to 0.30 points below Adam or Nadam on average. SGD often failed to converge §7.3 tab. 6. Adam with the increased learning-rate schedule converged in fewer epochs and was better on POS and chunking §7.3 tab. 7.
• Gradient normalization. Clipping did not help any task. Normalization with τ = 1 gave the best result in most cases §7.4 tab. 8 tab. 9.
• Tagging schemes. IOB was clearly worst. BIO and IOBES were on par, except entities, where BIO was much better §7.5 tab. 10.
• Classifier. CRF beat softmax on POS, chunking, NER and entities; the text gives the CRF gain as 0.19 to 0.85 points §7.6 tab. 11. Softmax beat CRF on events, where the training data contains almost no multi-token events §7.6 tab. 11.
• Dropout. Variational dropout won on every task with a significant median gain over no dropout (up to 1.98 points on entities) §7.7 tab. 12 tab. 13.
• LSTM depth. Two layers won on chunking, NER, entities and events; one layer won on POS §7.8 tab. 14.
• Recurrent units. A quadratic fit of test score against units per direction gives per-task optima between 56 and 192 units; the text describes the optimum as around 100 §7.9 tab. 15. Moving 25 units away from the optimum changes the score by about 0.01 to 0.06 points for two-layer networks §7.9 tab. 15.
• Mini-batch size. For entities, batch size 1 was 2.83 points below batch size 32 (median). Best batch size depends on task §7.10 tab. 16.
• Backend. No difference in test scores. Theano took 10 to 30 times one epoch of compile time for complex networks §7.11 tab. 17.
• Multi-task learning. Adding POS as an auxiliary task improved chunking (74.0% of configurations, p < 10⁻¹¹ for the comparison) §8.2 tab. 18. Single-task learning was best for NER (65.8%) tab. 18. Supervising tasks at different levels beat same-level supervision on three of four tasks §8.3 tab. 19. Multi-task variance across configurations was higher than single-task variance §8 tab. 18.

Reference scores the paper cites for context, not its own results: POS accuracy 97.24% (WSJ, Toutanova et al.); chunking about 95% F1; NER 91.21% F1; TempEval3 events 81.05% F1 (best system) §4. Under the paper's reduced POS training set (first 500 sentences), accuracy fell to about 94–95% §4.

Paper vs code

Official code: the paper's abstract footnote links https://github.com/UKPLab/emnlp2017-bilstm-cnn-crf, owned by the UKP Lab of the authors. The repository is Apache-2.0 LICENSE, with a NOTICE that names "UKP Lab" NOTICE.txt. This card reads the clone at commit b709f580f11c33c0f9951a0afdc3e71a252c93fd. No other repository was searched.

Differences found by reading the code:

• Software versions. Requirements pin Keras 2.2.0 and TensorFlow 1.8.0 requirements.txt, and the README says the same README.md. The paper used Keras 1.2.2 with Theano 0.8.2 or TensorFlow 0.12.1 §6 §7.11. The author does not guarantee the same results with other versions README.md.
• Default optimizer. The code default is adam BiLSTM.py. The README documents nadam as the default README.md. The shipped training scripts do not set an optimizer, so they train with Adam Train_POS.py.
• Casing feature. The paper describes a seven-dimensional one-hot vector §3. The code embeds the feature with a trainable 10-dimensional table (addFeatureDimensions) BiLSTM.py BiLSTM.py. It defines eight rows including padding preprocessing.py.
• Word embedding matrix. The code freezes the pre-trained embedding layer (trainable=False) BiLSTM.py. Vectors for added words and for UNKNOWN are random and never updated. The paper does not say whether embeddings are trained.
• Adam learning-rate schedule. The paper describes the 0.01 / 0.005 / 0.001 Adam schedule §7.3. The code has a schedule only for SGD BiLSTM.py BiLSTM.py. The Adam schedule is not in the repository.
• SGD learning rate. The code starts SGD at 0.1 BiLSTM.py and lowers it at epochs 3 and 5 BiLSTM.py. The paper says only that the rate was tuned by hand §7.3.
• Dropout placement. The code passes the two dropout rates to the Keras LSTM arguments dropout and recurrent_dropout BiLSTM.py. In the variational branch, no dropout is applied after the last BiLSTM layer before the classifier BiLSTM.py BiLSTM.py. The naive branch adds a dropout layer after each LSTM BiLSTM.py. The paper's "output units" are not clearly mapped to the Keras arguments; the Keras meaning of dropout (on layer inputs) was not checked against the pinned version here.
• Early stopping. The code stops once the count of epochs without dev improvement reaches 5 BiLSTM.py. The paper says "more than 5" §6.
• Epoch cap. The example scripts call fit(epochs=25) Train_POS.py. The paper does not state a cap.
• Mini-batch composition. The code sorts sentences by length and forms batches only from sentences of the same length, split into bins no larger than miniBatchSize BiLSTM.py BiLSTM.py. Batches are shuffled BiLSTM.py. The paper does not mention length grouping.
• Invalid-tag repair. computeF1 scores both repairs, O and B-, and keeps the one with the higher F1 BiLSTM.py. This choice uses the gold labels of the split being scored, including test. See Pitfalls.
• Unknown-word initialisation. Frequent unseen words and UNKNOWN get vectors drawn uniformly from [−0.25, 0.25] preprocessing.py preprocessing.py. The paper says only "random word embedding" §6.
• Number normalization. The code maps any digit-and-punctuation token to NUMBER_TOKEN and also maps date and time strings to their own tokens WordEmbeddings.py. The paper mentions only numbers §6.
• Character initialisation and length. Character embeddings are drawn uniformly from ±√(3/d) BiLSTM.py. The default maximum word length is 25 characters BiLSTM.py, and the German NER script uses 50 Train_NER_German.py. The paper states neither.
• Gradient clipping defaults. Defaults are clipnorm = 1 and clipvalue = 0 BiLSTM.py. Keras applies clipnorm inside the optimizer; the norm scope was not checked against the pinned Keras source.
• POS data. The paper uses the first 500 WSJ training sentences §4. The example Train_POS.py uses Universal Dependencies (unidep_pos) Train_POS.py. Its results do not reproduce the paper's POS setting.
• Datasets in the repository. data/ contains conll2000_chunking, unidep_pos, unidep_pos_german and GermEval . The NER, entities and events corpora from the paper are not included, and the code does not reproduce those settings.
• Multi-task example. Train_MultiTask_Different_Levels.py gives chunking a 50-unit task LSTM on top of a shared 100-unit layer Train_MultiTask_Different_Levels.py. This is an example, not one of the 188 sampled configurations.
• Shipped results file. The repository includes hyperparameter_results.csv.gz hyperparameter_results.csv.gz. It was not analysed for this card.
• Speed claim. The README says grouping same-length sentences makes the implementation multiple times faster than the systems of Ma et al. and Lample et al. README.md.

Pitfalls

• Single runs are not reliable. Seed noise alone reaches 8.23 F1 on entities and 2.59 on NER in the paper's sample tab. 2. Compare distributions over many configurations, not one score §5.
• No absolute F1 for the recommended configuration. The results are win shares and median differences tab. 4–tab. 19. An implementer cannot reproduce absolute numbers from this card.
• The recommended configuration is not best on every task. Softmax won on events (61.6%) tab. 11. Same-level MTL was best for events tab. 19. Best mini-batch size and best embedding change by task tab. 4 tab. 16.
• Internal inconsistencies in the paper. The text says 229 chunking configurations were sampled, while the table says 230 §5 tab. 3. The recommendation says a mini-batch of 31 for NER and entities §7 overview table, but 31 is not in the tested set {1, 8, 16, 32, 64} §6.1 tab. 16.
• Adam learning-rate schedule is not in the code. Only the SGD schedule exists BiLSTM.py. Reproducing Table 7 requires writing this schedule yourself tab. 7.
• Optimizer default differs. The README says Nadam, the code uses Adam README.md BiLSTM.py. Shipped scripts train with Adam, not Nadam.
• Frozen embeddings. The code freezes the word embedding layer. Randomly initialised vectors (UNKNOWN and frequent OOV words) are never trained BiLSTM.py. Results from the embedding study depend on this choice, which the paper does not describe.
• Casing encoding. Seven-dimensional one-hot in the paper §3, trainable 10-dimensional embedding in the code BiLSTM.py.
• Dropout naming. "Output" dropout in the paper and the Keras dropout argument are not obviously the same. Under the variational setting the code applies no dropout between the last LSTM and the classifier BiLSTM.py. Check this against the paper's intent before reusing the code.
• Invalid-tag repair is chosen on the scored split. The code takes the better of the O and B- repairs using gold labels of that split, which can include test BiLSTM.py. Reported scores may therefore be slightly optimistic. The paper says O was usually slightly better but does not describe this selection §3.
• Norm scope for normalization is unstated. The paper gives one formula for a gradient vector §6.1. Whether it uses a global norm or per tensor changes the effective threshold.
• Early-stopping boundary. "More than 5" epochs in the paper, five in the code §6 BiLSTM.py.
• Recurrent-unit optimum is a fitted estimate. It comes from a quadratic fit to scores grouped by unit count §7.9. The formula for the optimum is not written out; the vertex of the fitted parabola is the implied maximum. For events with three layers the optimum lay outside the tested range, and the table shows no value tab. 15.
• Statistics on configurations, not seeds. The p-values compare options across sampled configurations §5. Some reported differences are described as not significant (for example p = 6.5% and p = 33.0% in §7.1). Do not treat "not significant" as "equal".
• Software versions. Results were produced with Keras 1.2.2 and Theano or TensorFlow 0.8 to 0.12 §6 §7.11. The repository pins Keras 2.2.0 and TensorFlow 1.8.0 requirements.txt. Expect different numbers.
• Dataset licences are separate. The paper's entities data comes from ACE 2005, which the paper links to the LDC catalogue §4. The code's license does not cover datasets. Check each dataset's terms before use.
• Compile cost. For complex networks, Theano's graph compilation sometimes took 10 to 30 times as long as one epoch §7.11. Plan for this if you use Theano.

Lineage

A condensed version appeared at EMNLP 2017, arXiv:1707.09861 [abstract].