Hyperparameter optimization

Overview

Hyperparameter optimization (HPO) searches for the settings that are fixed before training. Each candidate setting is one trial: train with it, validate, and keep the score. The survey writes the expensive step as "train and validate" 1908.00709 alg. 1, so the cost of HPO is the number of trials times the cost of one training run.

The invariant shared by all variants is the trial loop: pick a setting, pay for one evaluation, record the result, and let the record inform the next pick. Variants differ in how the next setting is chosen and in whether a trial must run to completion 1908.00709 §4.3.

The survey treats HPO as one part of AutoML's model-generation step, next to architecture search 1908.00709 §1. It has a dedicated HPO section 1908.00709 §4.3, and it returns to HPO in its discussion of joint search 1908.00709 §6.4 and in its open problems 1908.00709 §7.6.

HPO also adds variance to a benchmark score. In the authors' five case studies, the three HPO methods they compare add about as much variance on average as weight initialization 2103.03098 §2.2. A tuned score is therefore one noisy sample, not a fixed number 2103.03098 §2.1.

A second card, Training Tips for the Transformer Model, is a practical study of one model's training settings: learning rate, warmup, batch size, GPU count and sentence length. It is a training protocol, not a search method 1804.00247 §2. Its value here is that it shows how sensitive a training run is to these settings and how a trial should be scored 1804.00247 §2.1, §4.

HPO also runs inside larger pipelines. One predictive-maintenance study tunes its generative models with Optuna, scoring each trial by the validation PR-AUC of a downstream classifier 2607.19153 §3.3.2. The tuning objective therefore depends on the classifier chosen for it, a point that returns in the pitfalls below. An electricity-forecasting study tunes four regressors with a genetic algorithm and particle swarm optimisation, scoring each trial by the mean MAE over time-series cross-validation on the training months 2605.25305 §3.2.2. The last 12 months are held out as the test set and are not part of the search 2605.25305 §3.2.3.

The survey's HPO statements summarise other work, so they are reported as the survey's claims, not as measured results. The Transformer paper's findings are measured on one model, one language pair and one framework version. The benchmark papers' findings are measured on their own tasks and are not transferred to others here.

Variants

Grid search

Every combination of a fixed set of values is tried. The survey notes that the number of trials grows exponentially with the number of hyperparameters 1908.00709 §4.3.1. Two refinements appear in the survey: a coarse-to-fine grid, and a contracting grid that recentres on the best point and halves the spacing each iteration until it converges [arxiv:1908.00709 §4.3.1]. Card: AutoML: A Survey of the State-of-the-Art.

Random search

Settings are drawn at random from the search space. The survey reports that, for the same number of trials, random search covers more distinct values of each parameter, which helps when only a few parameters matter [arxiv:1908.00709 §4.3.1]. Random search also serves as a baseline in the NAS literature the survey reviews 1908.00709 §4.2.5. Card: AutoML: A Survey of the State-of-the-Art.

A clustering benchmark uses it over all parameters at once: 500 parameter sets per method, each parameter drawn uniformly within bounds taken from its one-at-a-time sweep 1612.08388 §5.3. For EM on the two-class, two-feature datasets, 68.1% of draws beat the default settings, with a mean gain among those draws of 22.1% 1612.08388 tab. 5. Card: Clustering Algorithms: A Comparative Approach.

A neural fingerprint study also uses random search, on three small regression datasets. It draws 50 settings per cross-validation fold, and the search covers the learning rate on a log scale, the initial weight scale, the L2 penalty, the fingerprint length and depth, and the layer widths 1509.09292 §4.2. The paper gives "up to 6" as the depth range, while the released code searches depths 1 to 4; the card's Paper vs code section records this. Card: Convolutional Networks on Graphs for Learning Molecular Fingerprints.

One-at-a-time sweep

One parameter is varied while the others stay at their defaults, and the gain over the default is summarised as a mean, a maximum and a standard deviation 1612.08388 §5.2. The clustering benchmark uses this to find which parameters matter before any joint search, and to set the bounds for random search 1612.08388 §5.3. One setting can be most of the gain: changing the EM model family gives a mean gain of 10.75% on the two-class, two-feature datasets, against 7.36% for the spectral kernel parameter [arxiv:1612.08388 tab. 3]. Card: Clustering Algorithms: A Comparative Approach.

Successive halving and Hyperband

Many trials start, and each round drops the worse half so that the budget goes to promising settings. The survey describes Hyperband as repeatedly dropping the worse half of configurations early to save budget [arxiv:1908.00709 §4.3.1]. The survey also lists early stopping of runs predicted to end badly, using learning-curve models or gradient statistics 1908.00709 §5.4. Training on subsets or downsampled data is listed as a low-fidelity way to evaluate a trial 1908.00709 §5.1. Card: AutoML: A Survey of the State-of-the-Art.

Bayesian optimization

A probabilistic model is fitted to past trials, and an acquisition function chooses the next setting from that model 1908.00709 §4.3.2 alg. 1. The survey lists the open-source libraries and their models in [arxiv:1908.00709 tab. 2]: Gaussian processes, random forests (SMAC) and tree-structured Parzen estimators (Hyperopt, BOHB). The full variant comparison is on the Bayesian optimization page.

A benchmark that compares several tuned models uses it in practice. The tabular benchmark runs Hyperopt's Bayesian optimization for 1,000 steps per model and dataset, selecting the configuration with the smallest validation loss. The estimator inside Hyperopt is not named 2106.03253 §3.1.2. Card: Tabular Data: Deep Learning is Not All You Need.

Optuna's TPE sampler can also be combined with a pruner that stops weak trials early. The predictive-maintenance study uses TPE with 20 random start-up trials and a median pruner, and runs 100 trials for its single generative model 2607.19153 §3.3.2 §5.1. The trial count for each per-type generator in its multi-generator variant is not stated 2607.19153 §3.2.5. Card: Breaking the Homogeneity Assumption.

Metaheuristic search (genetic algorithm and particle swarm)

A population of settings is evolved or moved through the space, and each member is scored by one trial. The electricity study compares a custom genetic algorithm (GA) with particle swarm optimisation (PSO) for each model. Both use 750 evaluations per run: the GA runs 25 generations of 30 offspring with mutation rate 0.5, and PSO runs 30 particles for 25 iterations 2605.25305 tab. 12. The objective is the mean MAE over 5-fold time-series cross-validation on the training data 2605.25305 §3.2.2. Each search is run with three seeds, and the best configuration across GA, PSO and seeds is kept for each model 2605.25305 §3.2.2. The paper reports that GA found the best configuration for every model, but it does not report a timing comparison 2605.25305 §3.2.2 §4. Card: Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations. The evidence is one pipeline on two campus datasets, and the paper does not compare this variant with the Bayesian methods above.

Multi-objective Bayesian optimization with an RL proposal step

Several objectives are tuned together: accuracy against ROM, RAM and FLOPs, under microcontroller limits. The TinyML paper fits one Gaussian process per objective. It proposes each next trial with an ensemble of augmented random search (ARS) agents, trained on posterior samples of the GP rather than on real training runs 2305.14109 §4 alg. 1. The searched settings cover training, pruning and quantization, with a budget of 150 samples, the first 10 drawn by Latin-hypercube sampling 2305.14109 §5 §5.1. The paper reports higher hypervolume than ParEGO, TurBO, MorBO, NSGA-II and random sampling on two tasks. These results are author-reported, the text gives no numeric values, and the curves appear only in a figure 2305.14109 §5.1 fig. 2. Card: Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML.

Warm starts from similar past tasks

Before any search, the best configurations of the most similar prior tasks are evaluated first, and then used to seed a genetic search or Bayesian optimisation 1810.03548 §3.3. A simpler form ranks a fixed portfolio of configurations by their average rank over prior tasks and tries them in that order 1810.03548 §2.1. Similarity is measured from task meta-features, from relative landmarks (pairwise score differences), or from a surrogate fitted per prior task 1810.03548 §3.1 §2.3.1 §2.3.2. Card: Meta-Learning: A Survey. The survey gives no benchmark numbers for these methods, so the variant is described, not ranked.

Gradient-based (hypergradients)

The validation loss is differentiated with respect to the hyperparameters, so that they can be updated by gradient steps. The survey mentions a reversible-dynamics memory tape that handles thousands of hyperparameters, approximate hypergradients taken before training converges, and two modes: reverse-mode, which stores the full training history, and forward-mode, which updates in real time 1908.00709 §4.3.3. Hypergradient optimizers can also tune Adam's \beta_1 and \beta_2 1908.00709 §4.3.3. Card: AutoML: A Survey of the State-of-the-Art.

Joint hyperparameter and architecture search

Hyperparameters and architecture are searched together. The survey cites BOHB over a combined space 1908.00709 §6.4. AutoHAS expresses a continuous hyperparameter as a weighted sum of categorical bases, for example a learning rate built from {0.1, 0.2, 0.3} 1908.00709 §6.4. FBNetV3 trains an accuracy predictor on the architecture embedding together with the training recipe 1908.00709 §6.4. The survey lists joint search as an open problem 1908.00709 §7.6. Card: AutoML: A Survey of the State-of-the-Art.

Manual sweeps judged on learning curves

Settings are set by hand, one setting or a small group of them per experiment, and each run is judged on its full learning curve rather than on one final score. The Transformer paper reports BLEU against wall-clock time for every run 1804.00247 §2.1, §2.2. Its sections vary the learning rate, warmup, batch size, GPU count, model size and maximum sentence length in turn 1804.00247 §4. Card: Training Tips for the Transformer Model. The paper does not frame this as a search method, so the variant is listed for the trial-scoring rules it supports, not for claims about search efficiency.

When to use which

The survey gives few head-to-head comparisons of HPO methods, so the rules below follow from the constraints the survey and the papers describe, not from measured winners.

Pitfalls when scoring trials

• Single runs are noisy. Changing a setting also changes the random initialization, so one-run differences can be noise 1804.00247 §2.1. Patience-based early stopping gave high variance in both training time and final BLEU, even across seeds 1804.00247 §2.1.
• A one-run gap can be within noise. The predictive-maintenance study reports PR-AUC 0.857 against 0.835 for its baseline from one run with one sampler seed, and does not test the gap 2607.19153 §4.1 tab. 4.
• Learning curves fluctuate between checkpoints. A threshold-based metric such as Time Till Score inherits this noise and needs a larger validation set to reduce it 1804.00247 §2.3 footnote.
• Checkpoint averaging is a cheap score smoother. Averaging the last checkpoints gave a typical gain of about 0.2 BLEU and lower variance 1804.00247 §4.10. Comparisons of checkpoint counts were not tested for significance 1804.00247 §4.10.
• Divergence is a trial outcome, not a score. In the paper's runs, divergence appeared in the first hours, when the effective learning rate is highest 1804.00247 §4.6. A trial loop should record it as a failed trial and stop it early, not compare it on the same scale.
• Do not score with the framework's proxy metric. The internal approximate BLEU ran 1.2 to 1.8 times the real BLEU and depends on the vocabulary, so the paper says not to report it 1804.00247 §2.4.
• Settings interact. A maximum sentence length at or below 70 produced an unexplained BLEU drop at batch size 1500 but not at 2000 1804.00247 §4.4. A search that varies batch size and sentence length together needs both in the space, or a fixed batch size.
• Do not select and report on the same test set. The OOD study picks the best Chemprop configuration per property by its minimum OOD test RMSE 2505.01912 fig. 6 2505.01912 app. C.4, so its reported gains are inflated by the selection. Its standard hyperparameter sweeps hold out 10% of the training data for selection 2505.01912 app. C.1.
• Selection steps outside the search can leak the test set. The electricity study tunes its models on training data only, but ranks its predictors by SHAP and prunes them before tuning. The paper does not say which rows scored the pruning curve 2605.25305 §3.2.1. The card's Paper vs code section records that the released code fits the pruning models on the full Palmas series, which includes the 2023 test year. Any feature or setting chosen before the search needs the same split discipline as the search itself.
• A population search costs more than its trial count suggests. Each objective evaluation is a 5-fold fit, so 750 evaluations are 3,750 model fits per optimiser run 2605.25305 tab. 12 §3.2.2. The paper does not report run time, so the cost in wall-clock is not known from it.
• Gains from search are uneven across targets. Fifty random Chemprop configurations per property lowered OOD RMSE by 60% for density, 23% for heat of formation and 50% for zero-point vibrational energy. The paper does not report a broad gain 2505.01912 §4.3 2505.01912 fig. 6.
• Tuning the number of clusters is confounded. The clustering benchmark's default-parameter results set K to the true number of classes, which an applied user would not know 1612.08388 §5.1. Its results for a wrong K are reported separately 1612.08388 §5.2.
• Search can overfit, by the authors' account. The clustering paper states that parameter search is expensive and can overfit 1612.08388 §1. The sweep bounds and the seeds used for random draws are not stated, so its random-search results cannot be matched exactly 1612.08388 §5.3.
• Shallow baseline tuning can understate the baselines. The predictive- maintenance study tunes its Random Forest and SVM on shallow grids, while its generators get an Optuna search. The paper says the classical baselines may be understated. A deeper Random Forest grid crashed the system, and the paper gives no numbers for a deeper search 2607.19153 §5.2.
• The tuned generator may not transfer across classifiers. The generators' objective is the Random Forest's validation PR-AUC. The paper does not say whether its SVM scenarios reuse the same tuned generators or retune them 2607.19153 §3.3.2.
• Draw seeding and selection are not in the paper. The neural fingerprint study picks the configuration with the lowest validation loss at the last training step, then retrains on training and validation together 1509.09292 §4.2. The paper does not say how the random draws are seeded, or whether every fold sees the same draws. The released code gives every fold the same 50 draws; the card's Paper vs code section records this, so a repeat of the search needs the same seed to match it.
• Warm starts need related prior tasks. The survey states that meta-learning gives no benefit when a new task is unrelated to the earlier ones, so task similarity is the key assumption 1810.03548 §1. Collaborative filtering over configurations needs some evaluations on the new task before it can predict anything 1810.03548 §3.3.
• Starting points and search spaces set the result. The tabular benchmark starts each model from its original paper's configuration and searches ranges written in its appendix 2106.03253 §3.1.2 app. B. The paper does not measure how much either choice moves the outcome, and its authors note that eleven datasets cannot cover all tabular data 2106.03253 §1.
• One search, then seeds. The tabular benchmark runs one search per model and dataset, then reports mean ± SEM over four seeds of the tuned configuration 2106.03253 §3.1.2 tab. 2. Those error bars cover training seeds only. The variance study treats HPO as a separate source of benchmark variance 2103.03098 §2.2.
• A tuned proposer has its own tuning cost. The TinyML paper tunes PPO with Optuna over 20 configurations of 20 samples each, and that search is not counted in its PPO-versus-ARS comparison. With default PPO settings, the hypervolume was no better than random sampling on ResNet-18 2305.14109 §5.4.1 fig. 5. The comparison is therefore not budget-matched.
• Check the search ranges and the objective signs before reuse. The TinyML paper lists momentum as log-uniform on [0.7, 0.99] and weight decay as log-uniform on [0.6, 0.99] 2305.14109 tab. 1. It does not say how the objectives are sign-flipped or normalised before scalarization, although accuracy is maximised and ROM, RAM and FLOPs are minimised [arxiv:2305.14109 §3] §5.1.
• Check the text's win count against the table. The TinyML paper's text says its method beats ParEGO on 7 of 10 datasets, with one tie. Its hypervolume column gives 6 wins, 2 ties and 2 losses 2305.14109 §5.1 tab. 4.

Reporting a tuned result

When the question is "is method A better than method B after tuning", a single search with a single run is not enough. The variance study compares estimators for a benchmark that includes HPO 2103.03098 §3.2.

• Ideal estimator. Each of the k runs repeats the whole pipeline with fresh seeds, including a new HPO, so the variance falls as 1/k. It costs O(k·T) trainings for T trials per search 2103.03098 §3.2.1. For k = 100 in the paper's setup, this is 1,070 hours 2103.03098 §3.3.
• Recommended estimator. Run HPO once, then run k trainings that randomise every source except HPO: data split, initialization, data order and augmentation. The paper calls this FixHOptEst(k, All). It costs O(k+T) trainings, and in its setup 21 hours for k = 100 2103.03098 §3.2 §3.3. The HPO step is still biased, but the paper reports lower standard error than seed-only averaging 2103.03098 §3.2.
• Decision rule. Over k paired runs of A and B, compute the fraction of paired runs in which A scores higher, and build a bootstrap interval for it 2103.03098 §4.1. The appendix form declares A significant if the lower bound exceeds 0.5 and meaningful if the upper bound exceeds 0.75 2103.03098 app. Statistical test with P(A>B). The main text states these bounds with a different expression, so use the appendix form 2103.03098 §4.1 2103.03098 app. Statistical test with P(A>B).
• Sample size. With α = 0.05, β = 0.05 and the 0.75 threshold, the paper's Noether formula gives 29 paired runs 2103.03098 app. Sample size.
• Report every configuration inside the bounds, not only the top one 2103.03098 §5.

Error rates for the decision rule are simulation results on five case studies. They are author-reported and not validated on other tasks 2103.03098 §4.2.

Implementation sketch

Successive halving, the core of Hyperband, in generic form. The survey describes the halving step 1908.00709 §4.3.1 but not the budget schedule, so the sizes below are assumptions, not the survey's values.

# sample n settings at random from the space [arxiv:1908.00709 §4.3.1]
# n is not stated in the survey; set it from the total budget
configs = sample_random(space, n)
budget = b_min          # b_min is not stated in the survey
while len(configs) > 1 and budget <= b_max:
    # train each survivor for the current budget, then validate [arxiv:1908.00709 alg. 1]
    scores = [validate(train(c, budget)) for c in configs]
    # keep the better half; Hyperband drops the worse half each round [arxiv:1908.00709 §4.3.1]
    configs = top_half(configs, scores)
    # doubling the budget per round is an assumption, not from the survey
    budget = budget * 2
return best(configs)

Where variants differ: how the settings are proposed (random draw, grid, surrogate or gradient step) and how much budget each trial gets. BOHB combines TPE-based Bayesian optimization with Hyperband 1908.00709 §4.3.2.

Contradictions

Early stopping. The survey lists early stopping of runs predicted to end badly as a way to cut evaluation cost, using learning-curve models or gradient statistics 1908.00709 §5.4. The Transformer paper reports that early stopping with a patience rule gave high variance in both training time and final BLEU 1804.00247 §2.1. The two sources use different stopping rules (predicted curves versus patience) and do not test the same method, so the disagreement is not resolved here.

Gains from random search. The clustering benchmark finds that random draws improve several methods, with a mean gain among improved draws of up to 30.6% for hierarchical clustering 1612.08388 tab. 5. The OOD molecular study finds gains on three of its ten properties and no broad gain 2505.01912 §4.3. The tasks, metrics and draw counts (500 versus 50) differ, so the two results are not compared here.

Otherwise, none known. The survey does not compare HPO methods head to head. Its warning that methods are compared under different training setups 1908.00709 §6.1.2 concerns NAS results.

Papers

• Convolutional Networks on Graphs for Learning Molecular Fingerprints: learned molecular fingerprints tuned by random search with 50 draws per fold, where the paper's depth range differs from its code.
• Meta-Learning: A Survey: organises warm starts, configuration transfer and meta-features by the meta-data they use, with no new experiments.
• Training Tips for the Transformer Model: manual studies of learning rate, warmup, batch size and GPU count for one translation model, with guidance on scoring noisy runs.
• AutoML: A Survey of the State-of-the-Art: surveys grid, random, Bayesian and gradient-based HPO, early stopping, and joint HPO with architecture search.
• Clustering Algorithms: A Comparative Approach: compares seven clustering methods at defaults, one parameter at a time and under random parameter search, on synthetic datasets.
• Tabular Data: Deep Learning is Not All You Need: tunes four deep models and XGBoost with one Bayesian protocol on 11 tabular datasets, with search spaces in the appendix.
• Accounting for Variance in Machine Learning Benchmarks: shows that HPO adds benchmark variance and proposes a repeated-run estimator and a decision rule for comparing tuned methods.
• BOOM, a benchmark of out-of-distribution molecular property prediction: random search over Chemprop settings for out-of-distribution molecular property prediction, with a warning about selecting on the test set.
• Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations: GA and PSO tuning of four regressors on time-series cross-validation, with SHAP pruning and a booster; one test year per campus.
• Breaking the Homogeneity Assumption: Specialized Multi-Generator Adversarial Learning for Rare Failure Detection in Predictive Maintenance: tunes one WGAN-GP per failure type with Optuna on a Random Forest's validation PR-AUC, and notes that its classical baselines were tuned shallowly.
• Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML: multi-objective Bayesian optimization for microcontroller networks, with ARS agents proposing trials on GP samples, under a 150-sample budget.