--- kind: concept slug: ensembles title: Ensembles summary: >- Combine several models, views of one input, or samples from one generative model. Gaussian-head ensembles also split uncertainty into aleatoric and epistemic parts. Fisher-weighted merging gives one model at one model's cost. Learned view weights beat the plain mean in most, not all, settings. Aligned diffusion samples help a little. An option-shuffled LLM vote is covered. All author-reported. tags: [uncertainty, calibration, ensembles, model-merging, test-time-augmentation, prompting, diffusion] papers: ["2305.04933", "2111.09832", "2011.11156", "1907.08456", "2107.03342", "2311.16452", "2312.02145"] related: [uncertainty-estimation, calibration, monte-carlo-dropout, model-merging, data-augmentation, ensemble-learning, test-time-augmentation, self-consistency, chain-of-thought-prompting, in-context-learning, latent-diffusion, monocular-depth-estimation] status: draft updated_at: 2026-10-24 --- ## Overview An ensemble combines several trained models, or several views of one input, into one answer. This page covers four ways to do it. They differ in what they return and what they cost at inference. - **Prediction ensembles** run every member and combine their outputs. In the simplest form, the members differ only by random seed and the ensemble returns a mean [Kratzert et al.](/papers/1907.08456) [arxiv:1907.08456 §2.6]. In the variant from [Nemani et al.](/papers/2305.04933), each member also predicts a variance, so the ensemble returns a mean and an uncertainty for each input [arxiv:2305.04933 §3.3]. - **Weight-averaging ensembles** merge the parameters of members that share an initialization into one network, so inference costs one model's forward pass. [Matena and Raffel](/papers/2111.09832) replace the plain mean of the parameters with a per-parameter weighted mean, where each weight is the diagonal Fisher information of that model [arxiv:2111.09832 §2.4]. - **View-aggregation ensembles** run one model on several transformed copies of an input (test-time augmentation) and combine the outputs. [Shanmugam et al.](/papers/2011.11156) learn non-negative weights for the views on a held-out split, instead of averaging them equally [arxiv:2011.11156 §4]. The members are the views of one network, not separately trained models. [Nori et al.](/papers/2311.16452) use the same idea with a majority vote over calls to one LLM, each call with its answer options in a different order [arxiv:2311.16452 §4.3]. - **Sample ensembles** run one stochastic generative model several times, each run from different initial noise. The runs are aligned for scale and shift, then merged by a pixel-wise median. [Ke et al.](/papers/2312.02145) use ten runs for monocular depth [arxiv:2312.02145 §3.4, §4.1]. The members share one set of weights, and they differ only in their starting noise. In the prediction variant, the ensemble splits the total predictive variance into two parts. The average of the members' own variances is the aleatoric part, the noise in the data. The spread of the members' means is the epistemic part, the uncertainty that comes from limited data and training choices [arxiv:2305.04933 §2.2, §3.3]. The split comes from the law of total variance, which the paper states for a model with a posterior over its parameters [arxiv:2305.04933 §2.2]. The weight-averaging variant returns no uncertainty split. A seed ensemble has no variance head either, but member disagreement can still be read as model uncertainty. The survey describes mutual information over the members for this [arxiv:2107.03342 §4.1.2]. The sources are author-reported. [Nemani et al.](/papers/2305.04933) is a tutorial with no new method. Its ensemble results come from two battery and turbofan case studies that the authors ran themselves [arxiv:2305.04933 §6.1, §6.2], and the authors say they do not claim to beat the published literature [arxiv:2305.04933 §6.2]. The Fisher-merging paper reports its ensembling results on GLUE checkpoints [arxiv:2111.09832 §3.1]. [Nori et al.](/papers/2311.16452) report GPT-4 results through an API on medical multiple-choice sets. The prompt choices were made on 80% of each dataset and checked on the other 20% [arxiv:2311.16452 §3.1, §3.2]. [Kratzert et al.](/papers/1907.08456) run the seed ensemble on 531 basins in their own experiments; the benchmark models come from other groups [arxiv:1907.08456 §2.5]. The [survey by Gawlikowski et al.](/papers/2107.03342) runs no experiments and reports the claims of the works it cites [arxiv:2107.03342 §8.1]. The [Ke et al.](/papers/2312.02145) depth results come from their own zero-shot evaluation on five real datasets, with the baselines taken from a published table [arxiv:2312.02145 §4.2] [tab. 1]. ## Variants ### Seed ensemble of one architecture Members share one architecture and differ only by random seed. [Kratzert et al.](/papers/1907.08456) train eight seeds of each setting and average their predictions at each time step [arxiv:1907.08456 §2.6]. Each member ends in one fully connected unit, so the ensemble returns a single value per day and no variance [arxiv:1907.08456 §2.6]. Card: [Kratzert et al.](/papers/1907.08456). ### Gaussian-head neural network ensemble Each member has a standard network body and a two-output head. One output is the mean μ̂ and the other is the variance σ̂². Each member is trained on the Gaussian negative log-likelihood, with the constant term dropped [arxiv:2305.04933 §3.3]. Members differ only through their random initialization and mini-batch order [arxiv:2305.04933 §3.3]. The paper's case studies use 15 members, chosen with an elbow plot of train RMSE and ECE [arxiv:2305.04933 §6.1]. Card: [Nemani et al.](/papers/2305.04933). The combined prediction is the average of the member means. Its variance is the average of the member variances plus the spread of the means, which the paper gives as a single formula [arxiv:2305.04933 §3.3]. The paper reads the first term as aleatoric and the second as epistemic [arxiv:2305.04933 §3.3]. The paper's case-study network has 113,332 trainable parameters per member [arxiv:2305.04933 §6.1]. With 15 members, the ensemble holds about 1.7 million parameters if every member shares that architecture. This is arithmetic on the paper's figures, not a number the paper states. ### Fisher-weighted weight averaging Members start from one shared initialization and are fine-tuned separately. The paper's ensembling experiment uses five fine-tuned BERT-base checkpoints of one model [arxiv:2111.09832 §3.1]. Their parameters are then merged into one set. Each merged parameter is a weighted average of the members' values, where a member's weight is its coefficient times its diagonal Fisher value for that parameter [arxiv:2111.09832 §2.4] [eq. 4]. Setting every Fisher value to one gives the plain average, which the paper calls isotropic merging [arxiv:2111.09832 §2.4]. Card: [Matena and Raffel](/papers/2111.09832). The ensembling experiment sets each coefficient to 1/5 by the authors' reasoning rather than a search [arxiv:2111.09832 §3.1]. The card records that the Fisher data used for ensembling is not stated [arxiv:2111.09832 §3.1]. The released code adds a normalisation step that the paper's equations do not include; see the card's Paper vs code section before reimplementing. Because the result is one network, its cost at inference matches a single model. The variant gives no per-input uncertainty by itself. Its cost is in computing the Fisher values, which needs a pass over training examples [arxiv:2111.09832 §2.3]. ### Shuffled-option vote over LLM calls One model is called several times on the same question. Each call sees the answer options in a different order, and the most frequent answer wins. The paper's standard setting uses five calls, with the options shuffled before each one [arxiv:2311.16452 §4.3] [alg. 1]. Each answer is mapped back to its original option before the vote, so the method as described fits multiple-choice questions [alg. 1]. The paper calls the full recipe Medprompt. It also adds retrieved worked examples and model-written chains of thought. The vote is the ensemble step; those other parts are separate components [arxiv:2311.16452 §4.1, §4.2]. Card: [Nori et al.](/papers/2311.16452). The members differ in option order. The paper does not state a sampling temperature, so whether the calls also differ by sampling is not stated [arxiv:2311.16452 §4.4]. Each question costs the five calls, and each prompt holds five retrieved examples [alg. 1] [arxiv:2311.16452 §4.4]. The paper contrasts this with Med-PaLM 2's ensemble refinement, which used as many as 44 inferences per question [arxiv:2311.16452 §2]. ### Aligned median over diffusion samples One trained generative model is sampled several times for the same input, each run from its own initial noise. Each run's output is correct only up to an unknown scale and shift, so every run gets its own scale and shift before the merge. The scale and shift are chosen to make the runs agree with each other, and the merged map is the pixel-wise median of the aligned runs [arxiv:2312.02145 §3.4] [eq. 4]. The paper uses 10 runs, each with 50 DDIM steps [arxiv:2312.02145 §4.1]. The gain from ensembling is reported on monocular depth only [arxiv:2312.02145 §4.3]. Card: [Ke et al.](/papers/2312.02145). Unlike the seed ensemble, the members do not differ in weights. They differ only in the noise that starts each run, which is why the alignment step is needed. The paper reports the spread across the runs on NYUv2 as a per-pixel standard deviation [arxiv:2312.02145 app. B.3]. ## When to use which The sources do not run the same comparison, so the rows below mix their evidence. Each row cites only what its paper reports. The prediction-ensemble results come from one backbone shared by the ensemble, MC dropout and SNGP, with GPR excluded from the turbofan study [arxiv:2305.04933 §6.1, §6.2]. The card records no timings for either paper, so no latency claim is made. | Situation | Use | Why | |---|---|---| | Per-input uncertainty is needed and M training runs are affordable | Gaussian-head ensemble | Lowest test NLL on every battery and turbofan set reported; separates aleatoric and epistemic parts [arxiv:2305.04933 §3.3, §6.1, §6.2] | | Accuracy is the goal, no per-input uncertainty is needed, and M training runs are affordable | Seed ensemble | On the EA-LSTM, eight seeds raised mean validation NSE from 0.674 to 0.705 over 447 basins. The paper's LSTM with static inputs reached 0.718 as an ensemble, so the seed ensemble is not the top model in that table [arxiv:1907.08456 tab. 3] | | Uncertainty is needed but the model has no variance head | Member disagreement (mutual information) | Reads only the model part of the uncertainty. It can be small on a wrong out-of-distribution input when all members agree, so pair it with a distribution-shift check [arxiv:2107.03342 §4.1.2, §4.1.3] | | Several fine-tuned checkpoints of one model exist and one model must be served | Fisher-weighted weight averaging | One merged network at one model's inference cost. Comparable to averaging five models' predictions on the paper's RTE, MRPC and SST-2 sets, and better than plain averaging [arxiv:2111.09832 §3.1, fig. 2]. Needs a shared initialization [arxiv:2111.09832 §2] | | A single forward pass per input is required, with uncertainty | SNGP, with caveats | Deterministic in one pass [arxiv:2305.04933 §3.4]. In the case studies it had higher NLL than the ensemble on every set reported, and it cannot separate the two uncertainty parts [arxiv:2305.04933 §3.4, §6.1, §6.2] | | Training data is small and an exact GP is wanted | GPR | Fits the battery training set exactly but generalizes poorly, with high test uncertainty [arxiv:2305.04933 §6.1]. Cost grows as O(N³), which ruled it out for the turbofan data [arxiv:2305.04933 §3.1, §6.2] | | A network already uses dropout and retraining M members is not an option | MC dropout | One network, several passes. The paper reports overconfidence and sensitivity to the dropout rate and training length [arxiv:2305.04933 §3.2, §6.1] | | A single hosted LLM answers a multiple-choice question, no training budget exists, and several calls per question are affordable | Shuffled-option vote | No training. Shuffling the options is meant to reduce position bias [arxiv:2311.16452 §4.3]. On MedQA the paper's ablation credits choice shuffling with +2.2 points, the same gain as few-shot selection [arxiv:2311.16452 §5.2] | | Dense depth from one RGB image is needed, a single stochastic sampler run is too noisy, and about ten sampling runs per input are affordable | Aligned median over diffusion samples | Lowers AbsRel by 0.2 to 0.6 points over a single run on the five real sets, with the best average rank in the paper's table. The gain shrinks after 10 runs. On DIODE the paper's model is weaker than DPT [arxiv:2312.02145 tab. 1, §4.3]. Ten runs at 50 steps each is the paper's cost [arxiv:2312.02145 §4.1] | Calibration is not a reason to pick the prediction ensemble on its own. Its ECE was not the lowest in every case (see Results). Use NLL together with a reliability check when the choice depends on honest uncertainty. ## Implementation sketch Our own pseudocode for the Gaussian-head ensemble, with the variance split made explicit. The paper's formulas are in [arxiv:2305.04933 §3.3]. ```pseudocode # Gaussian-head ensemble [arxiv:2305.04933 §3.3] # Input: training pairs (x, y), M members, one fixed architecture # Output: M networks, each returning (mean, variance) for an input def train_member(data, seed): # Only the seed changes: initial weights and batch order [§3.3] net = build_net_with_mean_and_variance_heads(seed) # Keep the variance positive with a softplus-type transform [§6.1] # Minimize the Gaussian negative log-likelihood [§3.3] fit(net, data, loss=gaussian_nll) return net def predict(members, x): # Each member gives its own (mean, variance) pair [§3.3] mus, sig2s = unzip([m(x) for m in members]) mu = mean(mus) # Aleatoric part: average of the members' variances [§3.3] aleatoric = mean(sig2s) # Epistemic part: spread of the members' means [§3.3] epistemic = mean(mus ** 2) - mu ** 2 return mu, aleatoric + epistemic, aleatoric, epistemic ``` Two notes on where variants differ. The member count is a free choice: the paper uses 15, but the authors' notebooks try other counts too [arxiv:2305.04933 §6.1]. The number of training epochs per member is also set in the paper and differs from the code (see Contradictions). The seed ensemble uses the same `predict` step without the variance terms: it averages the members' outputs only [arxiv:1907.08456 §2.6]. Our own pseudocode for the weight-averaging ensemble. The merge rule is [arxiv:2111.09832 §2.4] with [eq. 4]; the Fisher values are computed on training examples as in [arxiv:2111.09832 §2.3]. ```pseudocode # Fisher-weighted weight averaging [arxiv:2111.09832 §2.4] [eq. 4] # Input: M checkpoints from one initialization, one Fisher set each # Output: one merged network with the same architecture def fisher_merge_ensemble(checkpoints, fishers, coeffs): # Coefficients sum to 1; the paper uses 1/M [arxiv:2111.09832 §3.1] merged = {} for name in checkpoints[0]: # Weight at each element: coefficient times Fisher [eq. 4] weights = [c * f[name] for c, f in zip(coeffs, fishers)] numerator = sum(w * p[name] for w, p in zip(weights, checkpoints)) # Element-wise division gives the merged parameter [eq. 4] merged[name] = numerator / sum(weights) return merged ``` Setting every Fisher to one turns the sketch into a plain average, which the paper uses as its baseline [arxiv:2111.09832 §2.4]. Our own pseudocode for the shuffled-option vote. The steps follow the method in [arxiv:2311.16452 §4.3] and [alg. 1]. The paper's retrieved examples and chains of thought are left out of the sketch. ```pseudocode # Shuffled-option vote over LLM calls [arxiv:2311.16452 §4.3] [alg. 1] # Input: one question with options, a chat model, n calls # Output: one answer, one of the original options def shuffled_option_vote(question, llm, n_calls): votes = [] for _ in range(n_calls): # Reorder the options; the paper says this order is random [§4.3] order = random_permutation(question.options) position = llm(question.with_options(order)) # Map the chosen position back to the original option [alg. 1] votes.append(order[position]) # Most frequent answer; the paper does not define a tie rule [alg. 1] return most_common(votes) ``` The aligned-median ensemble has the same outer loop as the seed ensemble, with the run count and the sampler steps as settings. Its extra step aligns each run to the others before the median [arxiv:2312.02145 §3.4] [eq. 4]. The paper's alignment formula leaves the penalty weight unstated, so the value must come from elsewhere; see the card's Paper vs code section before reimplementing. ## Results Author-reported. The papers use different setups, so their results are reported separately. ### Seed ensemble (Kratzert et al.) Author-reported. Benchmark comparison on the 447 basins that every benchmark model ran on, validation period. Single-model values are mean ± standard deviation as printed in the paper; ensemble values have no spread [arxiv:1907.08456 tab. 3]. | Model | NSE mean, single | NSE mean, 8-model ensemble | |---|---|---| | EA-LSTM, NSE\* loss | 0.674 (±0.006) | 0.705 | | LSTM with static inputs, NSE\* loss | 0.685 (±0.015) | 0.718 | | HBV (upper), highest benchmark mean | 0.631 | not applicable | What the paper reports: - The seed ensemble raises the EA-LSTM's mean and median NSE over its single members [arxiv:1907.08456 tab. 3]. - Against the HBV (upper) ensemble, the paper reports p = 1.9e-4 for the single EA-LSTM and p = 6.2e-11 for the ensemble [arxiv:1907.08456 §3.2]. - The LSTM with static inputs has a slightly higher mean NSE, both single and as an ensemble. The paper calls the EA-LSTM's cost "small" for the interpretability it gives [arxiv:1907.08456 tab. 3, §3.1]. ### Prediction ensemble (Nemani et al.) Values are mean ± standard deviation over 10 independent runs [arxiv:2305.04933 §6.1, §6.2]. The means below are copied from the paper's tables. The full ± values are on the card. | Case, set | Metric | NNE (ensemble) | MC dropout | SNGP | GPR | |---|---|---|---|---|---| | Battery, primary test | NLL | 5.4 | 14.3 | 5.7 | 5.7 | | Battery, tertiary test | NLL | 5.7 | 9.2 | 5.9 | 6.4 | | Battery, primary test | ECE (%) | 10.5 | 24.4 | 21.5 | 6.9 | | Battery, tertiary test | ECE (%) | 9.8 | 22.6 | 9.3 | 8.0 | | Turbofan, unit 14 | RMSE (cycles) | 7.4 | 11.5 | 9.3 | excluded | | Turbofan, unit 14 | NLL | 2.2 | 4.2 | 4.4 | excluded | | Turbofan, unit 15 | ECE (%) | 14.9 | 11.5 | 8.5 | excluded | Sources: battery [arxiv:2305.04933 §6.1, results table]; turbofan [arxiv:2305.04933 §6.2, results table]. GPR is excluded from the turbofan study because of its scaling [arxiv:2305.04933 §6.2]. What the paper reports: - The ensemble has the lowest NLL on every test set in the battery study and on every turbofan unit reported [arxiv:2305.04933 §6.1, §6.2]. - Its ECE is not the lowest in every case. GPR is lower on the battery primary test set, and SNGP is lower on turbofan unit 15 [arxiv:2305.04933 §6.1, §6.2]. - The ensemble and SNGP are underconfident on the battery data. The paper calls underconfidence the safer error for battery early-life prediction [arxiv:2305.04933 §6.1]. - MC dropout is overconfident, and its run-to-run variation is the largest of the group [arxiv:2305.04933 §6.1, §6.2]. Caveat for the battery table: its caption says "MSE ± standard deviation", but its row labels say RMSE. The labels are taken as correct here [arxiv:2305.04933 §6.1]. ### Weight-averaging ensemble (Matena and Raffel) Author-reported. Setup: BERT-base on GLUE, with ensembles of five fine-tuned checkpoints from HuggingFace [arxiv:2111.09832 §3.1] [app. A]. - On RTE, MRPC and SST-2, Fisher merging beats isotropic merging in every case and is comparable to averaging the predictions of the five models. It costs one model's inference instead of five [arxiv:2111.09832 §3.1]. The numbers appear only in the figure [arxiv:2111.09832 fig. 2]. ### Shuffled-option vote (Nori et al.) Author-reported. Setup: GPT-4 through an API on multiple-choice MultiMedQA sets. The snapshot and date are not stated [arxiv:2311.16452 §3.1]. The vote is one part of Medprompt, so these numbers do not isolate it. - On MedQA (US, 4 options), GPT-4 with 5 fixed shots scored 81.4%. Medprompt, with 5 retrieved shots and 5 shuffled calls, scored 90.2% [arxiv:2311.16452 tab. 1]. - In the MedQA ablation, each component was added in turn to a 5-shot baseline. Chain of thought gave +3.4 points, and few-shot selection and choice shuffling each gave +2.2. The paper says the order of addition is a choice, and it does not compute Shapley values [arxiv:2311.16452 §5.2]. - With 20 retrieved shots and 11 calls, MedQA reached 90.6% [arxiv:2311.16452 §5.2]. - The paper reports the best result on all nine MultiMedQA datasets [arxiv:2311.16452 §5] [tab. 1]. - The average on the 80% tuning split was 90.6%, and on the 20% held-out split 91.3%. The paper reads this as no sign of overfitting from prompt choices [arxiv:2311.16452 §5.1]. ### Aligned median over diffusion samples (Ke et al.) Author-reported. Setup: zero-shot monocular depth from a diffusion model trained on synthetic depth only. Metric is AbsRel, lower is better, on five real datasets [arxiv:2312.02145 §4.2]. Both rows below list the same training data [tab. 1]. | Method | NYUv2 | KITTI | ETH3D | ScanNet | DIODE | Avg. rank | |---|---|---|---|---|---|---| | 10-run ensemble | 5.5 | 9.9 | 6.5 | 6.4 | 30.8 | 1.4 | | Single run | 6.0 | 10.5 | 7.1 | 6.9 | 31.0 | 2.5 | | DPT, for reference | 9.8 | 10.0 | 7.8 | 8.2 | 18.2 | 3.9 | Source: [arxiv:2312.02145 tab. 1]. DPT is trained on real depth data, so it is a different kind of comparison. What the paper reports: - The 10-run ensemble has a lower AbsRel than a single run on all five real sets [arxiv:2312.02145 tab. 1]. - Ten runs cut NYUv2 AbsRel by about 8%, and 20 runs by about 9.5%. The gain shrinks beyond 10 runs [arxiv:2312.02145 §4.3]. - On DIODE the ensemble is worse than DPT, and the paper's "most cases" claim does not cover that set [arxiv:2312.02145 tab. 1, §4.2]. ### Survey claims (Gawlikowski et al.) The survey runs no experiments. These are its summaries of cited works, not results we re-checked [arxiv:2107.03342 §3.3, §8.1]: - Deep ensembles are at least as good as MC dropout and Probabilistic Backpropagation on accuracy, calibration and out-of-distribution detection, in a study by Lakshminarayanan et al. [arxiv:2107.03342 §3.3]. - Deep ensembles were the most robust to dataset shift among the methods compared, in a study by Ovadia et al. [arxiv:2107.03342 §3.3]. - Bagging can worsen uncertainty estimation in ensembles, in work by Lakshminarayanan et al. and Lee et al. [arxiv:2107.03342 §3.3]. - Five members were the size at which deep ensembles did best under dataset shift in the cited robustness study [arxiv:2107.03342 §3.3]. ## Contradictions - **Turbofan ECE claim against the table.** The paper's text says the ensemble has low ECE on the turbofan data [arxiv:2305.04933 §6.2]. The table gives 14.9% on unit 15, above SNGP's 8.5% and MC dropout's 11.5% [arxiv:2305.04933 §6.2, results table]. Both are reported; the table is the more specific source. - **Epochs and member count, text against code.** The paper states 300 epochs per member and 15 members [arxiv:2305.04933 §6.1]. The authors' notebooks use other epoch counts and member counts in different cells. Which setting produced the published table is not determined [arxiv:2305.04933 §6.1]. - **Option shuffling, paper against the authors' code.** The paper says the options are shuffled at random before each call [arxiv:2311.16452 §4.3] [alg. 1]. The authors' released code, which the paper does not describe, uses a fixed set of five option orders instead. Both are reported; the paper's description is the one the results are attributed to. - **Ensemble distance, paper against the authors' code.** Eq. 4 of the depth paper takes a square root of the mean squared pairwise distance between runs, but the released code sums a root-mean-square distance over pairs. The paper also leaves the penalty weight unstated. Both points are reported in the card's Paper vs code section [arxiv:2312.02145 eq. 4]; the equation is the one the results are attributed to. - **Across the sources.** None known. They test different things: one measures uncertainty from member outputs, another the quality of a merged set of weights, a third the accuracy of a seed ensemble, a fourth the accuracy of a vote over option-shuffled LLM answers, and a fifth the accuracy of a median over aligned diffusion samples. No source compares its method with the others. The survey's ensemble claims are quoted from cited works and are not tested by [Nemani et al.](/papers/2305.04933) [arxiv:2107.03342 §3.3]. Nemani et al.'s battery table points the same way as the survey on MC dropout: the ensemble's ECE is 10.5% and MC dropout's is 24.4% on the primary test set [arxiv:2305.04933 §6.1]. That is consistency in one setting, not confirmation of the survey's claims. ## Papers - [Nemani et al.](/papers/2305.04933), uncertainty quantification tutorial: compares a Gaussian-head ensemble, MC dropout, SNGP and GPR on battery and turbofan prognostics, with calibration measured by ECE and NLL. - [Matena and Raffel](/papers/2111.09832), Fisher-weighted merging: averages the parameters of models from one initialization with per-parameter Fisher weights, and reports ensembling as one use case. - [Kratzert et al.](/papers/1907.08456), regional rainfall-runoff with the EA-LSTM: an eight-seed ensemble of LSTM-type networks, compared with benchmark hydrology models on 447 basins. - [Nori et al.](/papers/2311.16452), Medprompt: GPT-4 with retrieved few-shot examples, filtered chains of thought and a vote over option-shuffled calls, evaluated on medical multiple-choice sets without training. - [Ke et al.](/papers/2312.02145), Marigold: a Stable Diffusion U-Net retrained for affine-invariant depth; its test-time ensemble aligns ten diffusion runs and takes their median. - [Gawlikowski et al.](/papers/2107.03342), survey of uncertainty in deep neural networks: separates data and model uncertainty and reviews ensembles, Bayesian networks and test-time augmentation. It runs no experiments.