Ensembles

Overview

An ensemble combines several trained models, or several views of one input, into one answer. This page covers four ways to do it. They differ in what they return and what they cost at inference.

• Prediction ensembles run every member and combine their outputs. In the simplest form, the members differ only by random seed and the ensemble returns a mean Kratzert et al. 1907.08456 §2.6. In the variant from Nemani et al., each member also predicts a variance, so the ensemble returns a mean and an uncertainty for each input 2305.04933 §3.3.
• Weight-averaging ensembles merge the parameters of members that share an initialization into one network, so inference costs one model's forward pass. Matena and Raffel replace the plain mean of the parameters with a per-parameter weighted mean, where each weight is the diagonal Fisher information of that model 2111.09832 §2.4.
• View-aggregation ensembles run one model on several transformed copies of an input (test-time augmentation) and combine the outputs. Shanmugam et al. learn non-negative weights for the views on a held-out split, instead of averaging them equally 2011.11156 §4. The members are the views of one network, not separately trained models. Nori et al. use the same idea with a majority vote over calls to one LLM, each call with its answer options in a different order 2311.16452 §4.3.
• Sample ensembles run one stochastic generative model several times, each run from different initial noise. The runs are aligned for scale and shift, then merged by a pixel-wise median. Ke et al. use ten runs for monocular depth 2312.02145 §3.4, §4.1. The members share one set of weights, and they differ only in their starting noise.

In the prediction variant, the ensemble splits the total predictive variance into two parts. The average of the members' own variances is the aleatoric part, the noise in the data. The spread of the members' means is the epistemic part, the uncertainty that comes from limited data and training choices 2305.04933 §2.2, §3.3. The split comes from the law of total variance, which the paper states for a model with a posterior over its parameters 2305.04933 §2.2. The weight-averaging variant returns no uncertainty split. A seed ensemble has no variance head either, but member disagreement can still be read as model uncertainty. The survey describes mutual information over the members for this 2107.03342 §4.1.2.

The sources are author-reported. Nemani et al. is a tutorial with no new method. Its ensemble results come from two battery and turbofan case studies that the authors ran themselves [arxiv:2305.04933 §6.1, §6.2], and the authors say they do not claim to beat the published literature 2305.04933 §6.2. The Fisher-merging paper reports its ensembling results on GLUE checkpoints 2111.09832 §3.1. Nori et al. report GPT-4 results through an API on medical multiple-choice sets. The prompt choices were made on 80% of each dataset and checked on the other 20% 2311.16452 §3.1, §3.2. Kratzert et al. run the seed ensemble on 531 basins in their own experiments; the benchmark models come from other groups 1907.08456 §2.5. The survey by Gawlikowski et al. runs no experiments and reports the claims of the works it cites 2107.03342 §8.1. The Ke et al. depth results come from their own zero-shot evaluation on five real datasets, with the baselines taken from a published table 2312.02145 §4.2 tab. 1.

Variants

Seed ensemble of one architecture

Members share one architecture and differ only by random seed. Kratzert et al. train eight seeds of each setting and average their predictions at each time step 1907.08456 §2.6. Each member ends in one fully connected unit, so the ensemble returns a single value per day and no variance 1907.08456 §2.6. Card: Kratzert et al..

Gaussian-head neural network ensemble

Each member has a standard network body and a two-output head. One output is the mean μ̂ and the other is the variance σ̂². Each member is trained on the Gaussian negative log-likelihood, with the constant term dropped 2305.04933 §3.3. Members differ only through their random initialization and mini-batch order 2305.04933 §3.3. The paper's case studies use 15 members, chosen with an elbow plot of train RMSE and ECE 2305.04933 §6.1. Card: Nemani et al..

The combined prediction is the average of the member means. Its variance is the average of the member variances plus the spread of the means, which the paper gives as a single formula 2305.04933 §3.3. The paper reads the first term as aleatoric and the second as epistemic 2305.04933 §3.3.

The paper's case-study network has 113,332 trainable parameters per member 2305.04933 §6.1. With 15 members, the ensemble holds about 1.7 million parameters if every member shares that architecture. This is arithmetic on the paper's figures, not a number the paper states.

Fisher-weighted weight averaging

Members start from one shared initialization and are fine-tuned separately. The paper's ensembling experiment uses five fine-tuned BERT-base checkpoints of one model 2111.09832 §3.1. Their parameters are then merged into one set. Each merged parameter is a weighted average of the members' values, where a member's weight is its coefficient times its diagonal Fisher value for that parameter 2111.09832 §2.4 eq. 4. Setting every Fisher value to one gives the plain average, which the paper calls isotropic merging 2111.09832 §2.4. Card: Matena and Raffel.

The ensembling experiment sets each coefficient to 1/5 by the authors' reasoning rather than a search 2111.09832 §3.1. The card records that the Fisher data used for ensembling is not stated 2111.09832 §3.1. The released code adds a normalisation step that the paper's equations do not include; see the card's Paper vs code section before reimplementing.

Because the result is one network, its cost at inference matches a single model. The variant gives no per-input uncertainty by itself. Its cost is in computing the Fisher values, which needs a pass over training examples 2111.09832 §2.3.

Shuffled-option vote over LLM calls

One model is called several times on the same question. Each call sees the answer options in a different order, and the most frequent answer wins. The paper's standard setting uses five calls, with the options shuffled before each one 2311.16452 §4.3 alg. 1. Each answer is mapped back to its original option before the vote, so the method as described fits multiple-choice questions alg. 1. The paper calls the full recipe Medprompt. It also adds retrieved worked examples and model-written chains of thought. The vote is the ensemble step; those other parts are separate components 2311.16452 §4.1, §4.2. Card: Nori et al..

The members differ in option order. The paper does not state a sampling temperature, so whether the calls also differ by sampling is not stated 2311.16452 §4.4. Each question costs the five calls, and each prompt holds five retrieved examples alg. 1 2311.16452 §4.4. The paper contrasts this with Med-PaLM 2's ensemble refinement, which used as many as 44 inferences per question 2311.16452 §2.

Aligned median over diffusion samples

One trained generative model is sampled several times for the same input, each run from its own initial noise. Each run's output is correct only up to an unknown scale and shift, so every run gets its own scale and shift before the merge. The scale and shift are chosen to make the runs agree with each other, and the merged map is the pixel-wise median of the aligned runs 2312.02145 §3.4 eq. 4. The paper uses 10 runs, each with 50 DDIM steps 2312.02145 §4.1. The gain from ensembling is reported on monocular depth only 2312.02145 §4.3. Card: Ke et al..

Unlike the seed ensemble, the members do not differ in weights. They differ only in the noise that starts each run, which is why the alignment step is needed. The paper reports the spread across the runs on NYUv2 as a per-pixel standard deviation 2312.02145 app. B.3.

When to use which

The sources do not run the same comparison, so the rows below mix their evidence. Each row cites only what its paper reports. The prediction-ensemble results come from one backbone shared by the ensemble, MC dropout and SNGP, with GPR excluded from the turbofan study 2305.04933 §6.1, §6.2. The card records no timings for either paper, so no latency claim is made.

Calibration is not a reason to pick the prediction ensemble on its own. Its ECE was not the lowest in every case (see Results). Use NLL together with a reliability check when the choice depends on honest uncertainty.

Implementation sketch

Our own pseudocode for the Gaussian-head ensemble, with the variance split made explicit. The paper's formulas are in 2305.04933 §3.3.

# Gaussian-head ensemble  [arxiv:2305.04933 §3.3]
# Input:  training pairs (x, y), M members, one fixed architecture
# Output: M networks, each returning (mean, variance) for an input

def train_member(data, seed):
    # Only the seed changes: initial weights and batch order  [§3.3]
    net = build_net_with_mean_and_variance_heads(seed)
    # Keep the variance positive with a softplus-type transform  [§6.1]
    # Minimize the Gaussian negative log-likelihood  [§3.3]
    fit(net, data, loss=gaussian_nll)
    return net

def predict(members, x):
    # Each member gives its own (mean, variance) pair  [§3.3]
    mus, sig2s = unzip([m(x) for m in members])
    mu = mean(mus)
    # Aleatoric part: average of the members' variances  [§3.3]
    aleatoric = mean(sig2s)
    # Epistemic part: spread of the members' means  [§3.3]
    epistemic = mean(mus ** 2) - mu ** 2
    return mu, aleatoric + epistemic, aleatoric, epistemic

Two notes on where variants differ. The member count is a free choice: the paper uses 15, but the authors' notebooks try other counts too 2305.04933 §6.1. The number of training epochs per member is also set in the paper and differs from the code (see Contradictions).

The seed ensemble uses the same predict step without the variance terms: it averages the members' outputs only 1907.08456 §2.6.

Our own pseudocode for the weight-averaging ensemble. The merge rule is 2111.09832 §2.4 with eq. 4; the Fisher values are computed on training examples as in 2111.09832 §2.3.

# Fisher-weighted weight averaging  [arxiv:2111.09832 §2.4] [eq. 4]
# Input:  M checkpoints from one initialization, one Fisher set each
# Output: one merged network with the same architecture

def fisher_merge_ensemble(checkpoints, fishers, coeffs):
    # Coefficients sum to 1; the paper uses 1/M  [arxiv:2111.09832 §3.1]
    merged = {}
    for name in checkpoints[0]:
        # Weight at each element: coefficient times Fisher  [eq. 4]
        weights = [c * f[name] for c, f in zip(coeffs, fishers)]
        numerator = sum(w * p[name] for w, p in zip(weights, checkpoints))
        # Element-wise division gives the merged parameter  [eq. 4]
        merged[name] = numerator / sum(weights)
    return merged

Setting every Fisher to one turns the sketch into a plain average, which the paper uses as its baseline 2111.09832 §2.4.

Our own pseudocode for the shuffled-option vote. The steps follow the method in 2311.16452 §4.3 and alg. 1. The paper's retrieved examples and chains of thought are left out of the sketch.

# Shuffled-option vote over LLM calls  [arxiv:2311.16452 §4.3] [alg. 1]
# Input:  one question with options, a chat model, n calls
# Output: one answer, one of the original options

def shuffled_option_vote(question, llm, n_calls):
    votes = []
    for _ in range(n_calls):
        # Reorder the options; the paper says this order is random  [§4.3]
        order = random_permutation(question.options)
        position = llm(question.with_options(order))
        # Map the chosen position back to the original option  [alg. 1]
        votes.append(order[position])
    # Most frequent answer; the paper does not define a tie rule  [alg. 1]
    return most_common(votes)

The aligned-median ensemble has the same outer loop as the seed ensemble, with the run count and the sampler steps as settings. Its extra step aligns each run to the others before the median 2312.02145 §3.4 eq. 4. The paper's alignment formula leaves the penalty weight unstated, so the value must come from elsewhere; see the card's Paper vs code section before reimplementing.

Results

Author-reported. The papers use different setups, so their results are reported separately.

Seed ensemble (Kratzert et al.)

Author-reported. Benchmark comparison on the 447 basins that every benchmark model ran on, validation period. Single-model values are mean ± standard deviation as printed in the paper; ensemble values have no spread 1907.08456 tab. 3.

What the paper reports:

• The seed ensemble raises the EA-LSTM's mean and median NSE over its single members 1907.08456 tab. 3.
• Against the HBV (upper) ensemble, the paper reports p = 1.9e-4 for the single EA-LSTM and p = 6.2e-11 for the ensemble 1907.08456 §3.2.
• The LSTM with static inputs has a slightly higher mean NSE, both single and as an ensemble. The paper calls the EA-LSTM's cost "small" for the interpretability it gives 1907.08456 tab. 3, §3.1.

Prediction ensemble (Nemani et al.)

Values are mean ± standard deviation over 10 independent runs 2305.04933 §6.1, §6.2. The means below are copied from the paper's tables. The full ± values are on the card.

Sources: battery 2305.04933 §6.1, results table; turbofan 2305.04933 §6.2, results table. GPR is excluded from the turbofan study because of its scaling 2305.04933 §6.2.

What the paper reports:

• The ensemble has the lowest NLL on every test set in the battery study and on every turbofan unit reported 2305.04933 §6.1, §6.2.
• Its ECE is not the lowest in every case. GPR is lower on the battery primary test set, and SNGP is lower on turbofan unit 15 2305.04933 §6.1, §6.2.
• The ensemble and SNGP are underconfident on the battery data. The paper calls underconfidence the safer error for battery early-life prediction 2305.04933 §6.1.
• MC dropout is overconfident, and its run-to-run variation is the largest of the group 2305.04933 §6.1, §6.2.

Caveat for the battery table: its caption says "MSE ± standard deviation", but its row labels say RMSE. The labels are taken as correct here 2305.04933 §6.1.

Weight-averaging ensemble (Matena and Raffel)

Author-reported. Setup: BERT-base on GLUE, with ensembles of five fine-tuned checkpoints from HuggingFace 2111.09832 §3.1 app. A.

• On RTE, MRPC and SST-2, Fisher merging beats isotropic merging in every case and is comparable to averaging the predictions of the five models. It costs one model's inference instead of five 2111.09832 §3.1. The numbers appear only in the figure 2111.09832 fig. 2.

Shuffled-option vote (Nori et al.)

Author-reported. Setup: GPT-4 through an API on multiple-choice MultiMedQA sets. The snapshot and date are not stated 2311.16452 §3.1. The vote is one part of Medprompt, so these numbers do not isolate it.

• On MedQA (US, 4 options), GPT-4 with 5 fixed shots scored 81.4%. Medprompt, with 5 retrieved shots and 5 shuffled calls, scored 90.2% 2311.16452 tab. 1.
• In the MedQA ablation, each component was added in turn to a 5-shot baseline. Chain of thought gave +3.4 points, and few-shot selection and choice shuffling each gave +2.2. The paper says the order of addition is a choice, and it does not compute Shapley values 2311.16452 §5.2.
• With 20 retrieved shots and 11 calls, MedQA reached 90.6% 2311.16452 §5.2.
• The paper reports the best result on all nine MultiMedQA datasets 2311.16452 §5 tab. 1.
• The average on the 80% tuning split was 90.6%, and on the 20% held-out split 91.3%. The paper reads this as no sign of overfitting from prompt choices 2311.16452 §5.1.

Aligned median over diffusion samples (Ke et al.)

Author-reported. Setup: zero-shot monocular depth from a diffusion model trained on synthetic depth only. Metric is AbsRel, lower is better, on five real datasets 2312.02145 §4.2. Both rows below list the same training data tab. 1.

Source: 2312.02145 tab. 1. DPT is trained on real depth data, so it is a different kind of comparison.

What the paper reports:

• The 10-run ensemble has a lower AbsRel than a single run on all five real sets 2312.02145 tab. 1.
• Ten runs cut NYUv2 AbsRel by about 8%, and 20 runs by about 9.5%. The gain shrinks beyond 10 runs 2312.02145 §4.3.
• On DIODE the ensemble is worse than DPT, and the paper's "most cases" claim does not cover that set 2312.02145 tab. 1, §4.2.

Survey claims (Gawlikowski et al.)

The survey runs no experiments. These are its summaries of cited works, not results we re-checked 2107.03342 §3.3, §8.1:

• Deep ensembles are at least as good as MC dropout and Probabilistic Backpropagation on accuracy, calibration and out-of-distribution detection, in a study by Lakshminarayanan et al. 2107.03342 §3.3.
• Deep ensembles were the most robust to dataset shift among the methods compared, in a study by Ovadia et al. 2107.03342 §3.3.
• Bagging can worsen uncertainty estimation in ensembles, in work by Lakshminarayanan et al. and Lee et al. 2107.03342 §3.3.
• Five members were the size at which deep ensembles did best under dataset shift in the cited robustness study 2107.03342 §3.3.

Contradictions

• Turbofan ECE claim against the table. The paper's text says the ensemble has low ECE on the turbofan data 2305.04933 §6.2. The table gives 14.9% on unit 15, above SNGP's 8.5% and MC dropout's 11.5% [arxiv:2305.04933 §6.2, results table]. Both are reported; the table is the more specific source.
• Epochs and member count, text against code. The paper states 300 epochs per member and 15 members 2305.04933 §6.1. The authors' notebooks use other epoch counts and member counts in different cells. Which setting produced the published table is not determined 2305.04933 §6.1.
• Option shuffling, paper against the authors' code. The paper says the options are shuffled at random before each call 2311.16452 §4.3 alg. 1. The authors' released code, which the paper does not describe, uses a fixed set of five option orders instead. Both are reported; the paper's description is the one the results are attributed to.
• Ensemble distance, paper against the authors' code. Eq. 4 of the depth paper takes a square root of the mean squared pairwise distance between runs, but the released code sums a root-mean-square distance over pairs. The paper also leaves the penalty weight unstated. Both points are reported in the card's Paper vs code section 2312.02145 eq. 4; the equation is the one the results are attributed to.
• Across the sources. None known. They test different things: one measures uncertainty from member outputs, another the quality of a merged set of weights, a third the accuracy of a seed ensemble, a fourth the accuracy of a vote over option-shuffled LLM answers, and a fifth the accuracy of a median over aligned diffusion samples. No source compares its method with the others. The survey's ensemble claims are quoted from cited works and are not tested by Nemani et al. 2107.03342 §3.3. Nemani et al.'s battery table points the same way as the survey on MC dropout: the ensemble's ECE is 10.5% and MC dropout's is 24.4% on the primary test set 2305.04933 §6.1. That is consistency in one setting, not confirmation of the survey's claims.

Papers

• Nemani et al., uncertainty quantification tutorial: compares a Gaussian-head ensemble, MC dropout, SNGP and GPR on battery and turbofan prognostics, with calibration measured by ECE and NLL.
• Matena and Raffel, Fisher-weighted merging: averages the parameters of models from one initialization with per-parameter Fisher weights, and reports ensembling as one use case.
• Kratzert et al., regional rainfall-runoff with the EA-LSTM: an eight-seed ensemble of LSTM-type networks, compared with benchmark hydrology models on 447 basins.
• Nori et al., Medprompt: GPT-4 with retrieved few-shot examples, filtered chains of thought and a vote over option-shuffled calls, evaluated on medical multiple-choice sets without training.
• Ke et al., Marigold: a Stable Diffusion U-Net retrained for affine-invariant depth; its test-time ensemble aligns ten diffusion runs and takes their median.
• Gawlikowski et al., survey of uncertainty in deep neural networks: separates data and model uncertainty and reviews ensembles, Bayesian networks and test-time augmentation. It runs no experiments.