---
kind: concept
slug: hyperparameter-optimization
title: Hyperparameter optimization
summary: >-
  Automated search for settings fixed before training, such as learning rate, scored by each trial's validation result. Options run from grid and random search through successive halving and Bayesian optimization to hypergradients. The choice depends on the trial budget and on whether partial runs predict final results. Tuned scores are noisy, so compare methods over repeated runs.
tags: [automl, hyperparameter-optimization, search, training-recipe]
papers: ["1509.09292", "1612.08388", "1804.00247", "1810.03548", "1908.00709", "2103.03098", "2106.03253", "2305.14109", "2505.01912", "2605.25305", "2607.19153"]
related: [bayesian-optimization, differentiable-architecture-search, neural-architecture-search, meta-learning, transfer-learning, benchmark-validity, bootstrap-confidence-intervals, random-forest, metaheuristic-optimization, particle-swarm-optimization, time-series-forecasting]
status: draft
updated_at: 2026-10-31
---

## Overview

Hyperparameter optimization (HPO) searches for the settings that are fixed
before training. Each candidate setting is one trial: train with it, validate,
and keep the score. The survey writes the expensive step as "train and
validate" [arxiv:1908.00709 alg. 1], so the cost of HPO is the number of trials
times the cost of one training run.

The invariant shared by all variants is the trial loop: pick a setting, pay
for one evaluation, record the result, and let the record inform the next
pick. Variants differ in how the next setting is chosen and in whether a trial
must run to completion [arxiv:1908.00709 §4.3].

The survey treats HPO as one part of AutoML's model-generation step, next to
architecture search [arxiv:1908.00709 §1]. It has a dedicated HPO section
[arxiv:1908.00709 §4.3], and it returns to HPO in its discussion of joint
search [arxiv:1908.00709 §6.4] and in its open problems [arxiv:1908.00709 §7.6].

HPO also adds variance to a benchmark score. In the authors' five case studies,
the three HPO methods they compare add about as much variance on average as
weight initialization [arxiv:2103.03098 §2.2]. A tuned score is therefore one
noisy sample, not a fixed number [arxiv:2103.03098 §2.1].

A second card, [Training Tips for the Transformer Model](/papers/1804.00247),
is a practical study of one model's training settings: learning rate, warmup,
batch size, GPU count and sentence length. It is a training protocol, not a
search method [arxiv:1804.00247 §2]. Its value here is that it shows how
sensitive a training run is to these settings and how a trial should be scored
[arxiv:1804.00247 §2.1, §4].

HPO also runs inside larger pipelines. One predictive-maintenance study tunes
its generative models with Optuna, scoring each trial by the validation PR-AUC
of a downstream classifier [arxiv:2607.19153 §3.3.2]. The tuning objective
therefore depends on the classifier chosen for it, a point that returns in the
pitfalls below. An electricity-forecasting study tunes four regressors with a
genetic algorithm and particle swarm optimisation, scoring each trial by the
mean MAE over time-series cross-validation on the training months
[arxiv:2605.25305 §3.2.2]. The last 12 months are held out as the test set
and are not part of the search [arxiv:2605.25305 §3.2.3].

The survey's HPO statements summarise other work, so they are reported as the
survey's claims, not as measured results. The Transformer paper's findings are
measured on one model, one language pair and one framework version. The
benchmark papers' findings are measured on their own tasks and are not
transferred to others here.

## Variants

### Grid search

Every combination of a fixed set of values is tried. The survey notes that
the number of trials grows exponentially with the number of hyperparameters
[arxiv:1908.00709 §4.3.1]. Two refinements appear in the survey: a
coarse-to-fine grid, and a contracting grid that recentres on the best point
and halves the spacing each iteration until it converges [arxiv:1908.00709
§4.3.1]. Card: [AutoML: A Survey of the State-of-the-Art](/papers/1908.00709).

### Random search

Settings are drawn at random from the search space. The survey reports that,
for the same number of trials, random search covers more distinct values of
each parameter, which helps when only a few parameters matter [arxiv:1908.00709
§4.3.1]. Random search also serves as a baseline in the NAS literature the
survey reviews [arxiv:1908.00709 §4.2.5]. Card: [AutoML: A Survey of the
State-of-the-Art](/papers/1908.00709).

A clustering benchmark uses it over all parameters at once: 500 parameter sets
per method, each parameter drawn uniformly within bounds taken from its
one-at-a-time sweep [arxiv:1612.08388 §5.3]. For EM on the two-class,
two-feature datasets, 68.1% of draws beat the default settings, with a mean
gain among those draws of 22.1% [arxiv:1612.08388 tab. 5]. Card: [Clustering
Algorithms: A Comparative Approach](/papers/1612.08388).

A neural fingerprint study also uses random search, on three small regression
datasets. It draws 50 settings per cross-validation fold, and the search covers
the learning rate on a log scale, the initial weight scale, the L2 penalty, the
fingerprint length and depth, and the layer widths [arxiv:1509.09292 §4.2].
The paper gives "up to 6" as the depth range, while the released code searches
depths 1 to 4; the card's Paper vs code section records this. Card:
[Convolutional Networks on Graphs for Learning Molecular Fingerprints](/papers/1509.09292).

### One-at-a-time sweep

One parameter is varied while the others stay at their defaults, and the gain
over the default is summarised as a mean, a maximum and a standard deviation
[arxiv:1612.08388 §5.2]. The clustering benchmark uses this to find which
parameters matter before any joint search, and to set the bounds for random
search [arxiv:1612.08388 §5.3]. One setting can be most of the gain: changing
the EM model family gives a mean gain of 10.75% on the two-class, two-feature
datasets, against 7.36% for the spectral kernel parameter [arxiv:1612.08388
tab. 3]. Card: [Clustering Algorithms: A Comparative Approach](/papers/1612.08388).

### Successive halving and Hyperband

Many trials start, and each round drops the worse half so that the budget goes
to promising settings. The survey describes Hyperband as repeatedly dropping
the worse half of configurations early to save budget [arxiv:1908.00709
§4.3.1]. The survey also lists early stopping of runs
predicted to end badly, using learning-curve models or gradient statistics
[arxiv:1908.00709 §5.4]. Training on subsets or downsampled data is listed as a
low-fidelity way to evaluate a trial [arxiv:1908.00709 §5.1]. Card:
[AutoML: A Survey of the State-of-the-Art](/papers/1908.00709).

### Bayesian optimization

A probabilistic model is fitted to past trials, and an acquisition function
chooses the next setting from that model [arxiv:1908.00709 §4.3.2 alg. 1]. The
survey lists the open-source libraries and their models in [arxiv:1908.00709
tab. 2]: Gaussian processes, random forests (SMAC) and tree-structured Parzen
estimators (Hyperopt, BOHB). The full variant comparison is on the
[Bayesian optimization](/concepts/bayesian-optimization) page.

A benchmark that compares several tuned models uses it in practice. The tabular
benchmark runs Hyperopt's Bayesian optimization for 1,000 steps per model and
dataset, selecting the configuration with the smallest validation loss. The
estimator inside Hyperopt is not named [arxiv:2106.03253 §3.1.2]. Card:
[Tabular Data: Deep Learning is Not All You Need](/papers/2106.03253).

Optuna's TPE sampler can also be combined with a pruner that stops weak trials
early. The predictive-maintenance study uses TPE with 20 random start-up trials
and a median pruner, and runs 100 trials for its single generative model
[arxiv:2607.19153 §3.3.2] [§5.1]. The trial count for each per-type generator
in its multi-generator variant is not stated [arxiv:2607.19153 §3.2.5]. Card:
[Breaking the Homogeneity Assumption](/papers/2607.19153).

### Metaheuristic search (genetic algorithm and particle swarm)

A population of settings is evolved or moved through the space, and each
member is scored by one trial. The electricity study compares a custom
genetic algorithm (GA) with particle swarm optimisation (PSO) for each model.
Both use 750 evaluations per run: the GA runs 25 generations of 30 offspring
with mutation rate 0.5, and PSO runs 30 particles for 25 iterations
[arxiv:2605.25305 tab. 12]. The objective is the mean MAE over 5-fold
time-series cross-validation on the training data [arxiv:2605.25305 §3.2.2].
Each search is run with three seeds, and the best configuration across GA,
PSO and seeds is kept for each model [arxiv:2605.25305 §3.2.2]. The paper
reports that GA found the best configuration for every model, but it does not
report a timing comparison [arxiv:2605.25305 §3.2.2] [§4]. Card:
[Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble
Learning with SHapley Additive exPlanations](/papers/2605.25305). The evidence
is one pipeline on two campus datasets, and the paper does not compare this
variant with the Bayesian methods above.

### Multi-objective Bayesian optimization with an RL proposal step

Several objectives are tuned together: accuracy against ROM, RAM and FLOPs, under
microcontroller limits. The TinyML paper fits one Gaussian process per objective.
It proposes each next trial with an ensemble of augmented random search (ARS)
agents, trained on posterior samples of the GP rather than on real training runs
[arxiv:2305.14109 §4] [alg. 1]. The searched settings cover training, pruning
and quantization, with a budget of 150 samples, the first 10 drawn by
Latin-hypercube sampling [arxiv:2305.14109 §5] [§5.1]. The paper reports higher
hypervolume than ParEGO, TurBO, MorBO, NSGA-II and random sampling on two tasks.
These results are author-reported, the text gives no numeric values, and the
curves appear only in a figure [arxiv:2305.14109 §5.1] [fig. 2]. Card:
[Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for
TinyML](/papers/2305.14109).

### Warm starts from similar past tasks

Before any search, the best configurations of the most similar prior tasks are
evaluated first, and then used to seed a genetic search or Bayesian
optimisation [arxiv:1810.03548 §3.3]. A simpler form ranks a fixed portfolio of
configurations by their average rank over prior tasks and tries them in that
order [arxiv:1810.03548 §2.1]. Similarity is measured from task meta-features,
from relative landmarks (pairwise score differences), or from a surrogate fitted
per prior task [arxiv:1810.03548 §3.1] [§2.3.1] [§2.3.2]. Card: [Meta-Learning:
A Survey](/papers/1810.03548). The survey gives no benchmark numbers for these
methods, so the variant is described, not ranked.

### Gradient-based (hypergradients)

The validation loss is differentiated with respect to the hyperparameters, so
that they can be updated by gradient steps. The survey mentions a
reversible-dynamics memory tape that handles thousands of hyperparameters,
approximate hypergradients taken before training converges, and two modes:
reverse-mode, which stores the full training history, and forward-mode, which
updates in real time [arxiv:1908.00709 §4.3.3]. Hypergradient optimizers can
also tune Adam's $\beta_1$ and $\beta_2$ [arxiv:1908.00709 §4.3.3]. Card:
[AutoML: A Survey of the State-of-the-Art](/papers/1908.00709).

### Joint hyperparameter and architecture search

Hyperparameters and architecture are searched together. The survey cites BOHB
over a combined space [arxiv:1908.00709 §6.4]. AutoHAS expresses a continuous
hyperparameter as a weighted sum of categorical bases, for example a learning
rate built from {0.1, 0.2, 0.3} [arxiv:1908.00709 §6.4]. FBNetV3 trains an
accuracy predictor on the architecture embedding together with the training
recipe [arxiv:1908.00709 §6.4]. The survey lists joint search as an open
problem [arxiv:1908.00709 §7.6]. Card: [AutoML: A Survey of the
State-of-the-Art](/papers/1908.00709).

### Manual sweeps judged on learning curves

Settings are set by hand, one setting or a small group of them per experiment,
and each run is judged on its full learning curve rather than on one final
score. The Transformer paper reports BLEU against wall-clock time for every run
[arxiv:1804.00247 §2.1, §2.2]. Its sections vary the learning rate, warmup,
batch size, GPU count, model size and maximum sentence length in turn
[arxiv:1804.00247 §4]. Card: [Training Tips for the Transformer
Model](/papers/1804.00247). The paper does not frame this as a search method,
so the variant is listed for the trial-scoring rules it supports, not for
claims about search efficiency.

## When to use which

The survey gives few head-to-head comparisons of HPO methods, so the rules
below follow from the constraints the survey and the papers describe, not from
measured winners.

| Situation | Use | Why |
|---|---|---|
| Two or three parameters, a small range to refine | Coarse-to-fine or contracting grid | Refines around the best point; grid trials grow exponentially with dimensionality [arxiv:1908.00709 §4.3.1] |
| Many parameters, or only a few of them matter | Random search | Covers more distinct values of the important parameters [arxiv:1908.00709 §4.3.1] |
| Unknown which parameters matter, or whether defaults are good enough | One-at-a-time sweep first | Shows each parameter's gain alone; defaults were not always good and some methods gained from one changed setting [arxiv:1612.08388 §5.2] [tab. 3] |
| Any search, to check that a method beats a baseline | Random search with the same budget | Random search was a strong baseline in NAS studies the survey cites [arxiv:1908.00709 §4.2.5] |
| Several models, each with its own search, a time-ordered series and no gradient through the model | GA or PSO with a time-series cross-validation objective, using a fixed budget per model | The electricity study uses 750 evaluations per model and reports GA as best for every model in its setup [arxiv:2605.25305 tab. 12] [§3.2.2]; its results are one pipeline, so they are not a general ranking of GA against PSO |
| Several objectives with hard resource limits (flash, RAM, FLOPs) and a budget of about 150 trials | Multi-objective Bayesian optimization, if the RL-based proposal step is affordable to implement | Author-reported hypervolume gains over ParEGO, TurBO, MorBO, NSGA-II and random sampling at 150 samples [arxiv:2305.14109 §5.1] [fig. 2]; the gains are shown only in a figure, and the code is not checked here |
| Partial training runs are cheap and informative | Successive halving or Hyperband | Drops the worse half each round to save budget [arxiv:1908.00709 §4.3.1] |
| Each full trial is expensive and the budget allows a sequence of trials | Bayesian optimization | A surrogate chooses each next trial from the results so far [arxiv:1908.00709 §4.3.2] |
| Evaluations of many configurations exist for prior tasks similar to the new one | Warm start from the best configurations of the most similar prior tasks, then search | The survey lists this as the first search step in a warm start [arxiv:1810.03548 §3.3]; it gives no benefit when the new task is unrelated to the earlier ones [arxiv:1810.03548 §1] |
| Unsure whether tuning an algorithm on this task is worth its cost | Check whether tuning is likely to pay off before searching | Predicting whether tuning is worth the cost is a meta-learning family [arxiv:1810.03548 §3.6]; the survey gives no specific predictor |
| Many continuous parameters and a differentiable validation loss | Gradient-based hypergradients | Handles thousands of hyperparameters in the survey's description [arxiv:1908.00709 §4.3.3] |
| Two tuned methods to compare on one benchmark, with budget for k repeated runs | One search, then k runs that randomise every other source | Costs O(k+T) trainings instead of O(kT); the paper's estimator is biased but has lower standard error than seed-only averaging [arxiv:2103.03098 §3.2] [§3.3] |
| Comparing a new model with a tree baseline on tabular data, with equal tuning effort for each | Give every model the same search protocol and budget | With 1,000 Bayesian steps per model, XGBoost beat the deep models on 8 of the 11 datasets their own papers did not use [arxiv:2106.03253 §3.2] [tab. 3]. On Shrutime alone, it reached good performance in fewer search iterations [arxiv:2106.03253 §3.2] [fig. 2] |
| Tuning a generator whose output feeds a downstream classifier | Score each trial by the classifier's validation metric, and keep the test split for one final evaluation | The predictive-maintenance study does this with PR-AUC on validation and a single test pass [arxiv:2607.19153 §3.3.2] [§3.1] |
| Architecture and training recipe both unknown | Joint search, with the open problems in mind | Joint search is listed as open [arxiv:1908.00709 §7.6]; fixed training recipes can bias rankings [arxiv:1908.00709 §6.4] |
| Training a Transformer-style translation model on similar data, need a starting point | Manual sweep from the paper's defaults | The paper reports tested values and defaults for these settings [arxiv:1804.00247 §4.6, §4.8]; it does not claim they carry over to other languages or tasks [arxiv:1804.00247 §1] |
| Batch size or GPU count changes between trials | Re-tune the learning rate and warmup with it | With the learning rate held fixed, 8 GPUs raise the effective rate by the square root of 8 [arxiv:1804.00247 §4.8.3]; warmup has to be re-tuned [arxiv:1804.00247 §4.8.2] |
| Many small trials to run in parallel on a fixed cluster | Check the parallelism choice against the paper's advice | The authors recommend one multi-GPU run at a time over several smaller parallel runs for convergence in their setup [arxiv:1804.00247 §4.7] |

## Pitfalls when scoring trials

- **Single runs are noisy.** Changing a setting also changes the random
  initialization, so one-run differences can be noise [arxiv:1804.00247 §2.1].
  Patience-based early stopping gave high variance in both training time and
  final BLEU, even across seeds [arxiv:1804.00247 §2.1].
- **A one-run gap can be within noise.** The predictive-maintenance study
  reports PR-AUC 0.857 against 0.835 for its baseline from one run with one
  sampler seed, and does not test the gap [arxiv:2607.19153 §4.1] [tab. 4].
- **Learning curves fluctuate between checkpoints.** A threshold-based metric
  such as Time Till Score inherits this noise and needs a larger validation set
  to reduce it [arxiv:1804.00247 §2.3 footnote].
- **Checkpoint averaging is a cheap score smoother.** Averaging the last
  checkpoints gave a typical gain of about 0.2 BLEU and lower variance
  [arxiv:1804.00247 §4.10]. Comparisons of checkpoint counts were not tested
  for significance [arxiv:1804.00247 §4.10].
- **Divergence is a trial outcome, not a score.** In the paper's runs,
  divergence appeared in the first hours, when the effective learning rate is
  highest [arxiv:1804.00247 §4.6]. A trial loop should record it as a failed
  trial and stop it early, not compare it on the same scale.
- **Do not score with the framework's proxy metric.** The internal approximate
  BLEU ran 1.2 to 1.8 times the real BLEU and depends on the vocabulary, so
  the paper says not to report it [arxiv:1804.00247 §2.4].
- **Settings interact.** A maximum sentence length at or below 70 produced an
  unexplained BLEU drop at batch size 1500 but not at 2000 [arxiv:1804.00247 §4.4].
  A search that varies batch size and sentence length together needs both in
  the space, or a fixed batch size.
- **Do not select and report on the same test set.** The OOD study picks the
  best Chemprop configuration per property by its minimum OOD test RMSE
  [arxiv:2505.01912 fig. 6] [arxiv:2505.01912 app. C.4], so its reported gains
  are inflated by the selection. Its standard hyperparameter sweeps hold out
  10% of the training data for selection [arxiv:2505.01912 app. C.1].
- **Selection steps outside the search can leak the test set.** The electricity
  study tunes its models on training data only, but ranks its predictors by
  SHAP and prunes them before tuning. The paper does not say which rows scored
  the pruning curve [arxiv:2605.25305 §3.2.1]. The card's Paper vs code section
  records that the released code fits the pruning models on the full Palmas
  series, which includes the 2023 test year. Any feature or setting chosen
  before the search needs the same split discipline as the search itself.
- **A population search costs more than its trial count suggests.** Each
  objective evaluation is a 5-fold fit, so 750 evaluations are 3,750 model fits
  per optimiser run [arxiv:2605.25305 tab. 12] [§3.2.2]. The paper does not
  report run time, so the cost in wall-clock is not known from it.
- **Gains from search are uneven across targets.** Fifty random Chemprop
  configurations per property lowered OOD RMSE by 60% for density, 23% for
  heat of formation and 50% for zero-point vibrational energy. The paper does
  not report a broad gain [arxiv:2505.01912 §4.3] [arxiv:2505.01912 fig. 6].
- **Tuning the number of clusters is confounded.** The clustering benchmark's
  default-parameter results set K to the true number of classes, which an
  applied user would not know [arxiv:1612.08388 §5.1]. Its results for a
  wrong K are reported separately [arxiv:1612.08388 §5.2].
- **Search can overfit, by the authors' account.** The clustering paper states
  that parameter search is expensive and can overfit [arxiv:1612.08388 §1].
  The sweep bounds and the seeds used for random draws are not stated, so its
  random-search results cannot be matched exactly [arxiv:1612.08388 §5.3].
- **Shallow baseline tuning can understate the baselines.** The predictive-
  maintenance study tunes its Random Forest and SVM on shallow grids, while its
  generators get an Optuna search. The paper says the classical baselines may
  be understated. A deeper Random Forest grid crashed the system, and the paper
  gives no numbers for a deeper search [arxiv:2607.19153 §5.2].
- **The tuned generator may not transfer across classifiers.** The generators'
  objective is the Random Forest's validation PR-AUC. The paper does not say
  whether its SVM scenarios reuse the same tuned generators or retune them
  [arxiv:2607.19153 §3.3.2].
- **Draw seeding and selection are not in the paper.** The neural fingerprint
  study picks the configuration with the lowest validation loss at the last
  training step, then retrains on training and validation together [arxiv:1509.09292 §4.2].
  The paper does not say how the random draws are seeded, or whether every fold
  sees the same draws. The released code gives every fold the same 50 draws; the
  [card's Paper vs code section](/papers/1509.09292) records this, so a repeat
  of the search needs the same seed to match it.
- **Warm starts need related prior tasks.** The survey states that meta-learning
  gives no benefit when a new task is unrelated to the earlier ones, so task
  similarity is the key assumption [arxiv:1810.03548 §1]. Collaborative
  filtering over configurations needs some evaluations on the new task before
  it can predict anything [arxiv:1810.03548 §3.3].
- **Starting points and search spaces set the result.** The tabular benchmark
  starts each model from its original paper's configuration and searches
  ranges written in its appendix [arxiv:2106.03253 §3.1.2] [app. B]. The paper
  does not measure how much either choice moves the outcome, and its authors
  note that eleven datasets cannot cover all tabular data [arxiv:2106.03253 §1].
- **One search, then seeds.** The tabular benchmark runs one search per model
  and dataset, then reports mean ± SEM over four seeds of the tuned
  configuration [arxiv:2106.03253 §3.1.2] [tab. 2]. Those error bars cover
  training seeds only. The variance study treats HPO as a separate source of
  benchmark variance [arxiv:2103.03098 §2.2].
- **A tuned proposer has its own tuning cost.** The TinyML paper tunes PPO
  with Optuna over 20 configurations of 20 samples each, and that search is not
  counted in its PPO-versus-ARS comparison. With default PPO settings, the
  hypervolume was no better than random sampling on ResNet-18
  [arxiv:2305.14109 §5.4.1] [fig. 5]. The comparison is therefore not
  budget-matched.
- **Check the search ranges and the objective signs before reuse.** The TinyML
  paper lists momentum as log-uniform on [0.7, 0.99] and weight decay as
  log-uniform on [0.6, 0.99] [arxiv:2305.14109 tab. 1]. It does not say how the
  objectives are sign-flipped or normalised before scalarization, although
  accuracy is maximised and ROM, RAM and FLOPs are minimised [arxiv:2305.14109
  §3] [§5.1].
- **Check the text's win count against the table.** The TinyML paper's text says
  its method beats ParEGO on 7 of 10 datasets, with one tie. Its hypervolume
  column gives 6 wins, 2 ties and 2 losses [arxiv:2305.14109 §5.1] [tab. 4].

## Reporting a tuned result

When the question is "is method A better than method B after tuning", a single
search with a single run is not enough. The variance study compares estimators
for a benchmark that includes HPO [arxiv:2103.03098 §3.2].

- **Ideal estimator.** Each of the k runs repeats the whole pipeline with fresh
  seeds, including a new HPO, so the variance falls as 1/k. It costs O(k·T)
  trainings for T trials per search [arxiv:2103.03098 §3.2.1]. For k = 100 in
  the paper's setup, this is 1,070 hours [arxiv:2103.03098 §3.3].
- **Recommended estimator.** Run HPO once, then run k trainings that randomise
  every source except HPO: data split, initialization, data order and
  augmentation. The paper calls this FixHOptEst(k, All). It costs O(k+T)
  trainings, and in its setup 21 hours for k = 100 [arxiv:2103.03098 §3.2]
  [§3.3]. The HPO step is still biased, but the paper reports lower standard
  error than seed-only averaging [arxiv:2103.03098 §3.2].
- **Decision rule.** Over k paired runs of A and B, compute the fraction of
  paired runs in which A scores higher, and build a bootstrap interval for it
  [arxiv:2103.03098 §4.1]. The appendix form declares A significant if the
  lower bound exceeds 0.5 and meaningful if the upper bound exceeds 0.75
  [arxiv:2103.03098 app. Statistical test with P(A>B)]. The main text states
  these bounds with a different expression, so use the appendix form
  [arxiv:2103.03098 §4.1] [arxiv:2103.03098 app. Statistical test with P(A>B)].
- **Sample size.** With α = 0.05, β = 0.05 and the 0.75 threshold, the paper's
  Noether formula gives 29 paired runs [arxiv:2103.03098 app. Sample size].
- **Report every configuration inside the bounds, not only the top one**
  [arxiv:2103.03098 §5].

Error rates for the decision rule are simulation results on five case studies.
They are author-reported and not validated on other tasks [arxiv:2103.03098 §4.2].

## Implementation sketch

Successive halving, the core of Hyperband, in generic form. The survey
describes the halving step [arxiv:1908.00709 §4.3.1] but not the budget
schedule, so the sizes below are assumptions, not the survey's values.

```pseudocode
# sample n settings at random from the space [arxiv:1908.00709 §4.3.1]
# n is not stated in the survey; set it from the total budget
configs = sample_random(space, n)
budget = b_min          # b_min is not stated in the survey
while len(configs) > 1 and budget <= b_max:
    # train each survivor for the current budget, then validate [arxiv:1908.00709 alg. 1]
    scores = [validate(train(c, budget)) for c in configs]
    # keep the better half; Hyperband drops the worse half each round [arxiv:1908.00709 §4.3.1]
    configs = top_half(configs, scores)
    # doubling the budget per round is an assumption, not from the survey
    budget = budget * 2
return best(configs)
```

Where variants differ: how the settings are proposed (random draw, grid,
surrogate or gradient step) and how much budget each trial gets. BOHB combines
TPE-based Bayesian optimization with Hyperband [arxiv:1908.00709 §4.3.2].

## Contradictions

Early stopping. The survey lists early stopping of runs predicted to end badly
as a way to cut evaluation cost, using learning-curve models or gradient
statistics [arxiv:1908.00709 §5.4]. The Transformer paper reports that early
stopping with a patience rule gave high variance in both training time and
final BLEU [arxiv:1804.00247 §2.1]. The two sources use different stopping
rules (predicted curves versus patience) and do not test the same method, so
the disagreement is not resolved here.

Gains from random search. The clustering benchmark finds that random draws
improve several methods, with a mean gain among improved draws of up to 30.6%
for hierarchical clustering [arxiv:1612.08388 tab. 5]. The OOD molecular study
finds gains on three of its ten properties and no broad gain
[arxiv:2505.01912 §4.3]. The tasks, metrics and draw counts (500 versus 50) differ,
so the two results are not compared here.

Otherwise, none known. The survey does not compare HPO methods head to head.
Its warning that methods are compared under different training setups
[arxiv:1908.00709 §6.1.2] concerns NAS results.

## Papers

- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](/papers/1509.09292): learned molecular fingerprints tuned by random search with 50 draws per fold, where the paper's depth range differs from its code.
- [Meta-Learning: A Survey](/papers/1810.03548): organises warm starts, configuration transfer and meta-features by the meta-data they use, with no new experiments.
- [Training Tips for the Transformer Model](/papers/1804.00247): manual studies of learning rate, warmup, batch size and GPU count for one translation model, with guidance on scoring noisy runs.
- [AutoML: A Survey of the State-of-the-Art](/papers/1908.00709): surveys grid, random, Bayesian and gradient-based HPO, early stopping, and joint HPO with architecture search.
- [Clustering Algorithms: A Comparative Approach](/papers/1612.08388): compares seven clustering methods at defaults, one parameter at a time and under random parameter search, on synthetic datasets.
- [Tabular Data: Deep Learning is Not All You Need](/papers/2106.03253): tunes four deep models and XGBoost with one Bayesian protocol on 11 tabular datasets, with search spaces in the appendix.
- [Accounting for Variance in Machine Learning Benchmarks](/papers/2103.03098): shows that HPO adds benchmark variance and proposes a repeated-run estimator and a decision rule for comparing tuned methods.
- [BOOM](/papers/2505.01912), a benchmark of out-of-distribution molecular property prediction: random search over Chemprop settings for out-of-distribution molecular property prediction, with a warning about selecting on the test set.
- [Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations](/papers/2605.25305): GA and PSO tuning of four regressors on time-series cross-validation, with SHAP pruning and a booster; one test year per campus.
- [Breaking the Homogeneity Assumption: Specialized Multi-Generator Adversarial Learning for Rare Failure Detection in Predictive Maintenance](/papers/2607.19153): tunes one WGAN-GP per failure type with Optuna on a Random Forest's validation PR-AUC, and notes that its classical baselines were tuned shallowly.
- [Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML](/papers/2305.14109): multi-objective Bayesian optimization for microcontroller networks, with ARS agents proposing trials on GP samples, under a 150-sample budget.