Seed-variance evaluation

Overview

A trained network depends on its random seed: the initialisation, the shuffling and the dropout masks all change with it. For a BiLSTM sequence tagger, changing only the seed moved test F1 by about one point, which is enough to reverse the ranking of two published NER systems 1707.09861 §1 1707.09861 §3. The shared idea of the variants below is to treat one training run as one sample, then compare samples in a way that accounts for their spread. Two systems are then "different" only if their score distributions differ, not if one run happened to be higher.

Variants

Seed distribution for one fixed setup

Keep the configuration fixed, change only the seed, train many times and report the minimum, median, maximum and standard deviation of the test score. For a named NER system the paper used 86 seeds, and for the other 41 seeds 1707.09861 §3 1707.09861 tab. 1. Two distributions are then compared with a Kolmogorov-Smirnov test for the whole distribution and a Brown-Forsythe test for spread 1707.09861 §3. The paper's ranking check shows the value of this view: Ma and Hovy reported 91.21% and Lample et al. 90.94%, but the seed medians were 90.64% and 90.81%, so the reported order did not hold 1707.09861 tab. 1 1707.09861 §7. Card: Reporting Score Distributions Makes a Difference.

Seed-pair noise floor across random configurations

Sample many random configurations from the design grid and train each with two seeds. The gap between the two runs is a noise estimate for that task. Across five tagging tasks the median gap ranged from 0.17 to 0.72 points, and the 95th percentile from 0.50 to 2.10 points; the largest single gap was 8.23 points on ACE 2005 entities 1707.09861 §3 1707.09861 tab. 2. A gain smaller than the typical gap for that task is hard to call real. Card: Reporting Score Distributions Makes a Difference.

Paired comparison of one design choice

Fix a random draw of all other design choices, train each option of one choice on that draw, and take the paired difference. Repeat over many draws. The paper reports the median of the paired differences, and the standard deviation of each option's test score across configurations [arxiv:1707.09861 §5]. For Softmax versus CRF output, the CRF was better in 90.5% of 232 NER configurations and the median difference was −0.66 points, with Softmax minus CRF as the sign convention 1707.09861 §5 1707.09861 tab. 3. Card: Reporting Score Distributions Makes a Difference.

When to use which

The design-choice results are the paper's findings on five tagging tasks, not a general rule. The most robust settings it reports were a CRF output layer, Nadam, variational dropout and two stacked BiLSTM layers 1707.09861 §6. For optimizers, Nadam was best in 51.4% of 882 configurations, but on NER Adam's standard deviation (0.0091) was marginally below Nadam's (0.0092) 1707.09861 §6.2. Treat the choice as a default to test, not a guarantee.

Implementation sketch

Pseudocode for the three variants. The official repository does not fix any seed, so seeding for Python, NumPy and the framework backend must be added before the variants can be reproduced; see the card's Paper vs code section Reporting Score Distributions Makes a Difference.

# Variant 1: score distribution for one fixed setup  [arxiv:1707.09861 §3]
# Input: train_fn(config, seed) -> run log, config (fixed), seeds (list)
def seed_distribution(train_fn, config, seeds):
    # Change only the seed; everything else in config stays fixed
    scores = []
    for seed in seeds:
        log = train_fn(config, seed)
        # Use the test score at the epoch with the best dev score
        scores.append(test_at_best_dev(log))
    # Report the spread, not one value  [arxiv:1707.09861 tab. 1]
    return {"min": min(scores), "median": median(scores),
            "max": max(scores), "sigma": stdev(scores)}

# Variant 2: noise floor across random configurations  [arxiv:1707.09861 §3]
# Input: sample_config() -> random draw from the grid, run_fn(config, seed)
def noise_floor(sample_config, run_fn, n_configs):
    gaps = []
    for _ in range(n_configs):
        config = sample_config()
        # Two seeds per configuration; the paper does not say how its
        # seeds were chosen, so any two fixed seeds will do here
        a = run_fn(config, seed=1)
        b = run_fn(config, seed=2)
        gaps.append(abs(a - b))
    # Typical and tail noise for this task  [arxiv:1707.09861 tab. 2]
    return median(gaps), percentile(gaps, 95), max(gaps)

# Variant 3: paired comparison of one design choice  [arxiv:1707.09861 §5]
# Input: sample_rest() -> random draw of the other choices, run_fn(config)
def paired_choice_effect(sample_rest, run_fn, opt_a, opt_b, n):
    deltas, scores_a, scores_b = [], [], []
    for _ in range(n):
        # Both options share the same draw of the other choices
        rest = sample_rest()
        a = run_fn({**rest, **opt_a})
        b = run_fn({**rest, **opt_b})
        deltas.append(a - b)
        scores_a.append(a)
        scores_b.append(b)
    # Median of paired differences, and spread of each option  [§5]
    return (median(deltas), stdev(scores_a), stdev(scores_b))

Use the same best-dev epoch rule in every variant, so that the test score is never picked to maximise the distribution. Record the seed with each score.

Contradictions

None known across papers; this slug has one source card. Within the paper: the abstract reports 50,000 networks, but the counts in the body (1,830 seed-pair runs in §3, 975 configurations times two classifiers in §5, and the comparisons in §6) do not add up to that total 1707.09861 §abstract 1707.09861 §3 1707.09861 §5. The Events row of the classifier table prints a median difference under the CRF column while Softmax won most Events configurations, and its sign convention is not explained 1707.09861 §5. Do not read that row without checking the convention.

Papers

• Reporting Score Distributions Makes a Difference: introduces the seed-distribution view for sequence tagging, reports seed noise on NER systems and five tasks, and compares design choices across sampled configurations.