Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

In plain words

Neural networks start from random numbers. Change the random seed and the same training recipe ends up in a different place, and on text tagging tasks such as finding names in sentences that place can change the score by about a point. A single reported score can therefore make a method look better or worse than it is.

The authors argue that papers should report how a method behaves over many runs with different seeds, not one lucky or unlucky run. They also use that approach to test many LSTM design choices. Some choices, such as using a CRF output layer and a certain type of dropout, give good scores that vary less when the other settings change. That makes them safer defaults for new tasks.

TL;DR

• Single-run scores for non-deterministic taggers are unreliable: for two NER systems the seed alone moved F1 by about one point §1.
• Compare score distributions over many seeds (medians, spread, significance tests) rather than one number §3.
• Across the sweep, a CRF output layer, Nadam, variational dropout and two stacked BiLSTM layers were the most robust choices §6.
• The official repository is a Keras BiLSTM(-CRF) tagger; it does not contain the seed or configuration sweep code README.md.

Problem

Published comparisons often rest on one training run. For sequence tagging with LSTMs, the random initialisation, the shuffling and the dropout masks all depend on the seed, and the training ends in a different local minimum for each seed §2. The authors show this for a state-of-the-art NER system: the test F1 of one system varied from 89.99% to 91.00% across 86 seeds §1. Two systems that the original papers ranked differently gave a different ranking when re-run §3. Development-set selection did not fix the problem: the Spearman correlation between dev and test score for one system was 0.229 §3.

The paper also asks which design choices matter, when each one is tested across many random configurations rather than one hand-tuned setup §5.

Method

The paper has two parts.

1. Seed study (§3). Take a fixed configuration, change only the random seed, train many times, and compare the resulting test-score distributions with a significance test (Kolmogorov-Smirnov for distributions, Brown-Forsythe for spread, and a randomization test for the single-score gap) §3 [§1 footnote]. The seed-variance tables use 1,830 random configurations, each run with two seeds §3 tab. 2.

2. Robust model evaluation (§5). Sample many configurations from a grid of design choices (§4.2). For every sampled configuration, train each option of one choice (for example, Softmax versus CRF) and compare the results, using the median of the paired differences and the standard deviation of the test score across configurations §5. The paper states the comparison as

\Delta F_1 = \text{median}(S_1 - C_1, \dots, S_{232} - C_{232})

where S_i and C_i are the test F1 of configuration i with Softmax and with CRF §5. The sweep covers 975 sampled configurations, each trained with both classifiers, i.e. 1,950 networks §5. The whole study is reported as 50,000 networks [abstract]; see Paper vs code.

Network used throughout (the baseline architecture for every experiment):

• Pre-trained word embeddings as the input layer, followed by optional casing features and optional character-level representations §4.1 §4.2.
• One to three stacked bidirectional LSTM layers §4.2.
• A per-token output layer: Softmax or CRF §4.2.
• Training with mini-batches of sentences of similar length [code: neuralnets/BiLSTM.py#L279-L338].

The training loop, as implemented in the official code:

# Length-bucketed mini-batches, reshuffled every epoch  [code: neuralnets/BiLSTM.py#L279-L338]
# Input: train_sentences (token sequences), batch_size (default 32), rng
def make_epoch_batches(train_sentences, batch_size, rng):
    # Sort by length and split each run of equal length into bins of at most
    # batch_size sentences  [code: neuralnets/BiLSTM.py#L289-L322]
    ranges = bucket_by_length(train_sentences, batch_size)
    # Shuffle the sentences inside each run of equal length  [code: neuralnets/BiLSTM.py#L331-L335]
    for run in ranges:
        shuffle_in_place(train_sentences, run, rng)
    # Shuffle the order of the mini-batches  [code: neuralnets/BiLSTM.py#L338]
    shuffle_in_place(ranges, rng)
    return ranges

# Seed-distribution protocol  [§3] [§5]
# Input: config (one fixed setting), seeds (e.g. 86 runs for one system)
def seed_scores(config, seeds):
    # Train once per seed; the seed is the only thing that changes  [§3]
    scores = []
    for seed in seeds:
        model = train(config, seed)
        # Keep the test F1 of the epoch with the best dev F1  [code: neuralnets/BiLSTM.py#L400-L412]
        scores.append(test_f1_at_best_dev(model))
    # Report the distribution (min, median, max, sigma), not one value  [§3] [tab. 1]
    return scores

Hyperparameters

The paper names the grid of design choices it searched. Values the paper does not state are marked "not stated". Code defaults and example scripts are given separately, because they are not the paper's sweep values.

Results

Author-reported. Setup: five tagging tasks (POS, chunking, NER, entity recognition and event detection), BiLSTM networks, the paper's own runs of two published NER systems plus the sweep over design choices §4 §5.

Seed effect on two NER systems §3 tab. 1

The Kolmogorov-Smirnov test gives p < 0.01 between the two distributions; the Lample et al. distribution has the higher quartiles and the lower spread (Brown-Forsythe p < 0.05) §3. The paper's conclusion is that the original reported ordering of the two systems does not hold once seeds vary §7.

Seed-variance across random configurations §3 tab. 2

Each row is 2 seeds per configuration; difference is the test score gap between the two seeds.

The largest gap is on ACE 2005 Entities: 82.5% F1 with one seed and 74.3% with another §3.

Classifier: Softmax versus CRF §5 §6.1 tab. 3

The CRF has a smaller standard deviation on NER: σ = 0.0060 against 0.0082 for Softmax (Brown-Forsythe p < 10⁻³) §5. The CRF gave the better result on four of five tasks; it did not win on Events §6.1. A Softmax output gains more from additional stacked LSTM layers than a CRF does; fig. 3 shows this for NER §6.1.

Optimizers §6.2

• NER test-F1 σ across configurations: SGD 0.1328; Adagrad 0.0139; Adadelta 0.0138; RMSProp 0.0096; Adam 0.0091; Nadam 0.0092.
• Nadam gave the best result in 453 of 882 configurations (51.4%) and was on average 0.82 percentage points better than Adagrad on NER.
• On NER, Nadam converged after 9 epochs on average; SGD needed 42 §6.2.

Word embeddings §6.3 tab. 4

Komninos embeddings were best for POS, Entities and Events. Levy dependency embeddings were slightly ahead of Komninos on Chunking (p = 0.025), and GloVe (Common Crawl) matched Komninos on NER (p = 0.391) §6.3. Embedding choice mattered more on small training sets; the POS median gap between Komninos and GloVe2 was 4.97% §6.3.

Cell values are the share of sampled configurations in which that embedding was best; Δ rows are median differences as printed tab. 4.

Character representation §6.4 tab. 5

A character representation gives a significant gain only on POS, Chunking and Events. CNN and LSTM character encoders did not differ significantly on any task §6.4.

Gradient clipping and normalization §6.5

• Value clipping at thresholds 1, 3, 5 and 10 gave no improvement on any task.
• Gradient normalization was clearly better than none: no normalization was best in only 5.6% of 492 configurations (20% expected under the null). Median gains ranged from 0.29 points (Chunking) to 0.82 points (POS).

Dropout §6.6

• Variational dropout was the best option in 83.5% of 479 configurations and gave the smallest spread §6.6.
• Median gain over no dropout: 0.31 points (POS) to 1.98 (Entities). Over naive dropout: 0.19 (POS) to 1.32 (Entities) §6.6.
• Applying dropout to both output and recurrent units was best in 62.6% of cases; median gains 0.05 to 0.82 points §6.6.

Other parameters §6.7

• BIO and IOBES were on par on four of five tasks; on Entities, BIO was significantly better in 88.7% of configurations (median ΔF1 −1.01%) §6.7.
• Two stacked LSTM layers gave the best results overall; one and two were on par for POS. A single layer worked best with about 150 units; two or three layers with about 100 units each. Unit count had a very small effect §6.7.
• Mini-batch: sizes 1 to 16 suit small training sets; 8 to 32 suit larger ones; 64 usually performed worst §6.7.

Paper vs code

Official code: UKPLab/emnlp2017-bilstm-cnn-crf at b709f580f11c33c0f9951a0afdc3e71a252c93fd (Apache-2.0) LICENSE.

• Training data for POS. The paper uses the Penn-Treebank-based setup of Toutanova et al. with the first 500 training sentences §4. The example script trains on the Universal Dependencies POS data [code: Train_POS.py#L35-L41]. The two are not the same benchmark.
• Dropout default. The paper's grid is 0.05 to 0.5 §4.2. The class default is (0.5, 0.5) BiLSTM.py, and the example scripts use (0.25, 0.25) Train_POS.py. The paper does not name one rate as best.
• Sweep and seed code. The paper samples 1,830 configurations for Table 2 and 975 for the classifier study §3 §5. The repository has no sampler, sweep driver or seed-variance script. It contains one training script per task, each for a single configuration.
• Seeds. The paper's premise is seed dependence §3. The code never sets a seed: np.random and Python's random are used unseeded [code: neuralnets/BiLSTM.py#L129] BiLSTM.py preprocessing.py. Repeated runs of the repository will not match, and the seed cannot be set from a config.
• Evaluation. The paper does not mention how invalid tags are scored. The code computes F1 twice, once with wrong tags left as O and once with them set to B, and reports the higher value BiLSTM.py.
• Dropout placement. The paper applies variational dropout to the LSTM output units, to the recurrent units or to both §6.6. The code's first tuple entry is passed to Keras as the LSTM dropout argument, which acts on the layer inputs, not its outputs BiLSTM.py. The naming differs from the paper; check the semantics before comparing results.
• Learning rate. The paper gives no learning-rate schedule. The code applies one to SGD only BiLSTM.py [code: neuralnets/BiLSTM.py#L263-L266].
• Re-runs of other systems. The Ma and Hovy and Lample et al. re-runs in §3 used their authors' repositories (footnotes to XuezheMax/LasagneNLP and glample/tagger). This repository does not contain those systems.
• Total networks. The abstract reports 50,000 networks § abstract. The text itemises 1,830 seed-pair runs §3, 975 configurations × 2 classifiers §5, and the optimizer, embedding and dropout comparisons §6, but no total that adds up to 50,000.

Pitfalls

• Seed control is absent. Nothing in the repository fixes the seed, so the variance the paper measures cannot be reproduced exactly from the code. To apply the protocol, add seeding for Python, NumPy and the Keras backend yourself, and record the seed with each score. The paper does not describe how its seeds were chosen.
• Test score at best dev epoch. The code keeps the test score from the epoch with the best dev score per task BiLSTM.py. A distribution over seeds should use the same rule, not the best test score across epochs.
• Early stopping counter. Patience is 5 epochs BiLSTM.py. Its effect on the spread across seeds is not analysed in the paper.
• Training order is not uniform. Sentences are sorted by length and only shuffled within each length run; batches are then shuffled [code: neuralnets/BiLSTM.py#L289-L338]. This differs from a plain random shuffle, so an implementation that shuffles all sentences will not reproduce the same batch statistics.
• Mini-batch sizes. The paper's range is 1 to 64 §4.2, and the best size depends on the task size §6.7. Within a length run, sentences are split into near-equal bins of at most the chosen size BiLSTM.py.
• Word normalization. Dates, times, numbers and repeated quotes are mapped to placeholder tokens, used as a fallback when a token is not found exactly or in lowercase WordEmbeddings.py preprocessing.py. The paper does not describe this step.
• Frozen random vectors. Tokens added from the training data and the UNKNOWN_TOKEN vector are random and, because the embedding layer is frozen, never trained preprocessing.py BiLSTM.py. Their quality is not reported in the paper, which says nothing about unknown words.
• Events row sign. The paper's Table 3 prints the Events median (−0.15%) under the CRF column, while Softmax wins most Events configurations §5. The sign convention for that row is not explained. Do not read the Δ column for Events without checking the convention.
• Optimizer settings. The optimizers were run with the defaults from their own papers §6.2, but the values are not listed. Compare against those papers before reusing the numbers.
• Cost. Each seed-distribution point is a full training run (up to 25 epochs in the example scripts Train_POS.py). The 1,950-network classifier sweep is far more expensive than a single run §5.
• Companion paper. The footnote points to a separate arXiv paper (1707.06799) for the full hyperparameter results [abstract] §6. That paper is not compiled here; nothing in this card comes from it.

Lineage

• Builds on: the BiLSTM-CRF tagger of Huang et al. (2015) and the character representations of Ma and Hovy (2016) and Lample et al. (2016), all cited in §4.1. Those papers are not compiled here.
• Compared against: the Ma and Hovy and Lample et al. NER systems §3.
• Companion: arXiv:1707.06799 (full hyperparameter study), not compiled here.
• Superseded by: not stated.