---
kind: concept
slug: seed-variance-evaluation
title: Seed-variance evaluation
summary: >-
  How to tell whether a training-based score difference is real: report score distributions over random seeds rather than one run. Three variants: a seed distribution for one fixed setup, a seed-pair noise floor across random configurations, and a paired comparison of one design choice. The evidence comes from one paper on LSTM sequence tagging, and the official code does not fix seeds.
tags: [evaluation, reproducibility, statistics, sequence-labeling]
papers: ["1707.09861"]
related: [benchmark-validity, bootstrap-confidence-intervals, hyperparameter-optimization, sequence-labeling, conditional-random-field, dropout]
status: draft
updated_at: 2026-10-24
---

## Overview

A trained network depends on its random seed: the initialisation, the
shuffling and the dropout masks all change with it. For a BiLSTM sequence
tagger, changing only the seed moved test F1 by about one point, which is
enough to reverse the ranking of two published NER systems
[arxiv:1707.09861 §1] [arxiv:1707.09861 §3]. The shared idea of the variants
below is to treat one training run as one sample, then compare samples in a
way that accounts for their spread. Two systems are then "different" only if
their score distributions differ, not if one run happened to be higher.

## Variants

### Seed distribution for one fixed setup

Keep the configuration fixed, change only the seed, train many times and
report the minimum, median, maximum and standard deviation of the test score.
For a named NER system the paper used 86 seeds, and for the other 41 seeds
[arxiv:1707.09861 §3] [arxiv:1707.09861 tab. 1]. Two distributions are then
compared with a Kolmogorov-Smirnov test for the whole distribution and a
Brown-Forsythe test for spread [arxiv:1707.09861 §3]. The paper's ranking
check shows the value of this view: Ma and Hovy reported 91.21% and Lample et
al. 90.94%, but the seed medians were 90.64% and 90.81%, so the reported order
did not hold [arxiv:1707.09861 tab. 1] [arxiv:1707.09861 §7].
Card: [Reporting Score Distributions Makes a Difference](/papers/1707.09861).

### Seed-pair noise floor across random configurations

Sample many random configurations from the design grid and train each with two
seeds. The gap between the two runs is a noise estimate for that task. Across
five tagging tasks the median gap ranged from 0.17 to 0.72 points, and the
95th percentile from 0.50 to 2.10 points; the largest single gap was 8.23
points on ACE 2005 entities [arxiv:1707.09861 §3] [arxiv:1707.09861 tab. 2].
A gain smaller than the typical gap for that task is hard to call real.
Card: [Reporting Score Distributions Makes a Difference](/papers/1707.09861).

### Paired comparison of one design choice

Fix a random draw of all other design choices, train each option of one
choice on that draw, and take the paired difference. Repeat over many draws.
The paper reports the median of the paired differences, and the standard
deviation of each option's test score across configurations [arxiv:1707.09861
§5]. For Softmax versus CRF output, the CRF was better in 90.5% of 232 NER
configurations and the median difference was −0.66 points, with Softmax minus
CRF as the sign convention [arxiv:1707.09861 §5] [arxiv:1707.09861 tab. 3].
Card: [Reporting Score Distributions Makes a Difference](/papers/1707.09861).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Two systems on one task, a fixed recipe, budget for dozens of full runs | Seed distribution | Compares whole distributions; the paper used 41 to 86 seeds per system [arxiv:1707.09861 §3] |
| A new task, and you need to know how big a gain must be to count | Seed-pair noise floor | Two runs per configuration is cheap, and the gap sizes are task-specific [arxiv:1707.09861 tab. 2] |
| Choosing among design choices (classifier, optimizer, embeddings, dropout) for a new task | Paired comparison | Measures a choice's effect across many settings of the others, so one lucky setup does not decide it [arxiv:1707.09861 §5] |
| Only one training run is affordable | None of these | The paper's central claim is that one run is unreliable; report the limitation rather than a single number [arxiv:1707.09861 §1] |
| Selecting on dev scores to avoid the spread | Check first | The dev-test rank correlation for one system was only 0.229, so dev selection did not remove the seed effect [arxiv:1707.09861 §3] |

The design-choice results are the paper's findings on five tagging tasks, not
a general rule. The most robust settings it reports were a CRF output layer,
Nadam, variational dropout and two stacked BiLSTM layers [arxiv:1707.09861 §6].
For optimizers, Nadam was best in 51.4% of 882 configurations, but on NER
Adam's standard deviation (0.0091) was marginally below Nadam's (0.0092)
[arxiv:1707.09861 §6.2]. Treat the choice as a default to test, not a
guarantee.

## Implementation sketch

Pseudocode for the three variants. The official repository does not fix any
seed, so seeding for Python, NumPy and the framework backend must be added
before the variants can be reproduced; see the card's Paper vs code section
[Reporting Score Distributions Makes a Difference](/papers/1707.09861).

```pseudocode
# Variant 1: score distribution for one fixed setup  [arxiv:1707.09861 §3]
# Input: train_fn(config, seed) -> run log, config (fixed), seeds (list)
def seed_distribution(train_fn, config, seeds):
    # Change only the seed; everything else in config stays fixed
    scores = []
    for seed in seeds:
        log = train_fn(config, seed)
        # Use the test score at the epoch with the best dev score
        scores.append(test_at_best_dev(log))
    # Report the spread, not one value  [arxiv:1707.09861 tab. 1]
    return {"min": min(scores), "median": median(scores),
            "max": max(scores), "sigma": stdev(scores)}
```

```pseudocode
# Variant 2: noise floor across random configurations  [arxiv:1707.09861 §3]
# Input: sample_config() -> random draw from the grid, run_fn(config, seed)
def noise_floor(sample_config, run_fn, n_configs):
    gaps = []
    for _ in range(n_configs):
        config = sample_config()
        # Two seeds per configuration; the paper does not say how its
        # seeds were chosen, so any two fixed seeds will do here
        a = run_fn(config, seed=1)
        b = run_fn(config, seed=2)
        gaps.append(abs(a - b))
    # Typical and tail noise for this task  [arxiv:1707.09861 tab. 2]
    return median(gaps), percentile(gaps, 95), max(gaps)
```

```pseudocode
# Variant 3: paired comparison of one design choice  [arxiv:1707.09861 §5]
# Input: sample_rest() -> random draw of the other choices, run_fn(config)
def paired_choice_effect(sample_rest, run_fn, opt_a, opt_b, n):
    deltas, scores_a, scores_b = [], [], []
    for _ in range(n):
        # Both options share the same draw of the other choices
        rest = sample_rest()
        a = run_fn({**rest, **opt_a})
        b = run_fn({**rest, **opt_b})
        deltas.append(a - b)
        scores_a.append(a)
        scores_b.append(b)
    # Median of paired differences, and spread of each option  [§5]
    return (median(deltas), stdev(scores_a), stdev(scores_b))
```

Use the same best-dev epoch rule in every variant, so that the test score is
never picked to maximise the distribution. Record the seed with each score.

## Contradictions

None known across papers; this slug has one source card. Within the paper:
the abstract reports 50,000 networks, but the counts in the body (1,830
seed-pair runs in §3, 975 configurations times two classifiers in §5, and the
comparisons in §6) do not add up to that total [arxiv:1707.09861 §abstract]
[arxiv:1707.09861 §3] [arxiv:1707.09861 §5]. The Events row of the classifier
table prints a median difference under the CRF column while Softmax won most
Events configurations, and its sign convention is not explained
[arxiv:1707.09861 §5]. Do not read that row without checking the convention.

## Papers

- [Reporting Score Distributions Makes a Difference](/papers/1707.09861): introduces the seed-distribution view for sequence tagging, reports seed noise on NER systems and five tasks, and compares design choices across sampled configurations.