---
kind: concept
slug: bayesian-optimization
title: Bayesian optimization
summary: >-
  Sequential model-based search for the best hyperparameters or architecture when each evaluation is expensive: fit a surrogate to the trials so far, pick the next trial with an acquisition function, train it, repeat. The surrogate is usually a Gaussian process, a random forest or a TPE model; the cost of the surrogate and the number of trials are the main trade-offs.
tags: [automl, hyperparameter-optimization, nas, surrogate-model, thompson-sampling, ensemble-learning, multi-objective, tinyml, safe-exploration, robot-control, exoskeleton]
papers: ["1908.00709", "2205.14090", "2305.14109", "2411.12573", "2412.20350"]
related: [gaussian-process, hyperparameter-optimization, neural-architecture-search, surrogate-modeling, thompson-sampling, network-pruning, trust-region-method, human-in-the-loop]
status: draft
updated_at: 2026-10-25
---

## Overview

Bayesian optimization (BO) finds a good configuration theta for an objective
f(theta) that is costly to evaluate, for example validation accuracy after a
full training run. It keeps a record of (theta, y) pairs, fits a probabilistic
model of f to that record, and uses the model to choose the next theta to try
[arxiv:1908.00709 §4.3.2]. The survey calls this sequential model-based
optimization (SMBO) [arxiv:1908.00709 alg. 1].

The loop shares one invariant across all variants: every expensive call to f
adds one observation, and the model is refit before the next choice. The
choice of the next point is made by an acquisition function that scores
candidates by their expected usefulness under the current model, so the search
balances trying promising regions with exploring uncertain ones
[arxiv:1908.00709 §4.3.2]. One variant below replaces that scoring step with
reinforcement-learning agents, but keeps the same outer loop.

The survey's scope is AutoML, so the page covers BO for hyperparameters and
for neural architectures. In NAS, the same loop is called a surrogate
model-based method [arxiv:1908.00709 §4.2.4].

This page draws on five cards. [AutoML: A Survey of the
State-of-the-Art](/papers/1908.00709) surveys BO for AutoML and runs no
experiments of its own; its statements about BO come from the works it cites.
[Surrogate modeling for Bayesian optimization beyond a single Gaussian
process](/papers/2205.14090) proposes an ensemble-of-GPs variant and reports
its own experiments. [Combining Multi-Objective Bayesian Optimization with
Reinforcement Learning for TinyML](/papers/2305.14109) applies a multi-objective
variant to microcontroller networks and reports its own experiments. [Safe
Bayesian Optimization for the Control of High-Dimensional Embodied
Systems](/papers/2412.20350) proposes a safety-constrained variant for
controllers with thousands of parameters. [Locomotion Mode Transitions](/papers/2411.12573)
uses BO to tune the thresholds of an exoskeleton transition detector for one
user. The algorithm below is the survey's generic description, not a tested
implementation.

## Variants

The variants differ in the probabilistic model M, in how the next trial is
chosen from that model, and in whether they also control how much budget each
trial gets.

### Gaussian process (GP) surrogates

The survey lists GP-based libraries first: Spearmint, MOE, PyBO, Bayesopt,
SkGP and GPyOpt [arxiv:1908.00709 tab. 2]. GP inference cost grows cubically
with the number of observations, and the survey notes that GPs handle
variable-length architectures poorly [arxiv:1908.00709 §4.2.4]. A GP is
therefore a natural fit for a modest number of continuous hyperparameters and
a poor fit for large architecture spaces. Card: [AutoML: A Survey of the
State-of-the-Art](/papers/1908.00709). The multi-objective variant below also
uses one GP per objective [arxiv:2305.14109 §4].

### Ensemble of GPs with Thompson sampling (EGP-TS)

EGP-TS replaces the single GP with one GP per kernel in a fixed dictionary.
Each GP gets a weight that rises when it fits the new observations well. Each
query is chosen by Thompson sampling: draw a component from the weights, draw
one function from that component's posterior, and query where that function is
highest [arxiv:2205.14090 §3, eq. 10, eq. 12]. Random features (50 per kernel
in the experiments) keep these draws cheap [arxiv:2205.14090 §3.1, §5]. The
paper's motivation is that a fixed kernel is usually picked by hand, and that
parallel evaluation with upper-confidence-bound or expected-improvement rules
needs extra parameters or diversity rules, while Thompson sampling needs none
[arxiv:2205.14090 §1]. Card: [Surrogate modeling for Bayesian optimization
beyond a single Gaussian process](/papers/2205.14090).

The results are the authors' own. The paper reports lower simple regret than
single-kernel GP Thompson sampling, GP-EI and two bandit ensembles on four
synthetic functions and on robot pushing; the comparisons are read from plots
[arxiv:2205.14090 §5.1, §5.2, fig. 1–3]. On hyperparameter tuning of three
model families, the text says EGP-TS is best in most cases, with per-task
numbers only in figures [arxiv:2205.14090 §5.3, fig. 4–9]. The paper also says
EGP-TS needs slightly more run time than single-GP Thompson sampling; its
runtime table has no unit [arxiv:2205.14090 §9.3, tab. 2].

The regret bound is sublinear for exact GPs under a smoothness assumption
[arxiv:2205.14090 §4, Theorem 1]. The paper does not give a bound for the
random-feature version it runs in practice [arxiv:2205.14090 §4, §7]. The
paper's released notebook differs from its equations in several places, for
example refits off by default and fixed component lengthscales. The paper
card lists them, so implement from the paper and check that list before
reusing any code [arxiv:2205.14090 §3, §5].

### Multi-objective BO with RL-trained proposals (ARS-MOBOpt)

The TinyML paper optimizes several objectives at once for microcontroller
networks: accuracy, ROM, RAM and FLOPs [arxiv:2305.14109 §1, §5]. It fits one GP
per objective to the trials so far [arxiv:2305.14109 alg. 1]. It then replaces
the acquisition step with a small ensemble of augmented random search (ARS)
agents. Each agent starts from a different point on the feasible Pareto front,
and is trained on rewards computed from GP posterior samples, so no real
network is trained inside the inner loop. The best rollout of each agent is
the proposal [arxiv:2305.14109 §4, alg. 1]. The paper combines the objectives
into one scalar with an augmented Chebyshev scalarization, using a random
weight vector per sample [arxiv:2305.14109 §4, eq. 7]. It argues that expected
improvement clamps negative improvements to zero, and it keeps negative
improvements at a scaled-down value instead [arxiv:2305.14109 §4, eq. 8]. Card:
[Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for
TinyML](/papers/2305.14109).

The paper reports a 150-sample budget with five seeds, and a 10-sample
Latin-hypercube prior [arxiv:2305.14109 §5.1]. Its authors report the highest
hypervolume after about 50 to 75 samples on both tasks, against ParEGO, TurBO,
MorBO, NSGA-II and random sampling. The values are read from a figure
[arxiv:2305.14109 §5.1, fig. 2]. On a set of time-series datasets, the text
claims a win over ParEGO on most datasets, but the hypervolume column of the
paper's own table does not show that count [arxiv:2305.14109 §5.1, tab. 4]. The
paper also compares the ARS agent with PPO. Default PPO settings were no better
than random sampling on ResNet, and only a tuned PPO beat ARS there; the authors
chose ARS for its simplicity [arxiv:2305.14109 §5.4.1, fig. 5]. The code is
linked as a Zenodo archive [arxiv:2305.14109 §1]. This page did not check it
against the paper's equations, so implement from the paper and check the
details listed in its pitfalls first.

### Safe trust-region BO in a latent space (HdSafeBO)

HdSafeBO tunes controllers with hundreds to thousands of parameters while a
safety function g must stay above a threshold during the search. It fits one
GP to the objective f and one to g [arxiv:2412.20350 §3]. Inputs are mapped to
a low-dimensional latent space by PCA, or by an isometric VAE for the
neural-stimulation task [arxiv:2412.20350 §4.3, app. B.2]. Around the best safe
latent point it forms a box, keeps the points whose optimistic safety bound
μ + βσ passes the threshold [arxiv:2412.20350 alg. 1, line 7], and draws the
next point by Thompson sampling over that safe set [arxiv:2412.20350 alg. 1,
line 8]. The box grows after a run of successes and shrinks after failures
[arxiv:2412.20350 §4.2, alg. 2]. The paper chooses β so that Φ(β) ≤ 1 − α, which
gives a one-step probability bound on safety [arxiv:2412.20350 §4.1]. Its
experiments use β = 2 [arxiv:2412.20350 app. B.3]. Card: [Safe Bayesian
Optimization for the Control of High-Dimensional Embodied
Systems](/papers/2412.20350).

The results are the authors' own. On 1000-dimensional synthetic GP functions,
with 500 evaluations and 100 runs, the paper reports an objective of 3.96 and a
cumulative violation of 27.42, against 3.07 and 36.59 for the best baseline
(LineBO) [arxiv:2412.20350 tab. 1]. It says SafeOpt did not expand its safe
region from the initial points and is not in the table [arxiv:2412.20350 §5.1].
On a musculoskeletal task and a neural-stimulation task the results are only in
figures [arxiv:2412.20350 fig. 2, fig. 3]. The probability threshold α is not
stated in the paper [arxiv:2412.20350 app. B.3], so the guarantee cannot be
evaluated for a given setup without choosing it.

### Rule-threshold BO against labelled data (exoskeleton transitions)

This is an application of BO, not a new surrogate. A finite-state detector of
walk-to-sit and stair transitions has two thresholds per transition pair. For
one user, BO searches those thresholds with a GP surrogate (RBF kernel) fitted
to the cost of each candidate pair, and minimizes an acquisition of the form k
times the mean plus or minus (1 − k) times the standard deviation, with k = 0.5
[arxiv:2411.12573 §2.5.2, eq. 8, eq. 9]. The cost is computed on labelled
transitions from that user [arxiv:2411.12573 alg. 2]. The authors run 30 BO
iterations, against 100 grid-search evaluations of the same space
[arxiv:2411.12573 §4.3, fig. 8], and report BO as about four times faster than
grid search on average [arxiv:2411.12573 fig. 10]. BO trains on two transitions
and evaluates on three [arxiv:2411.12573 §2.5.2]. Card: [Locomotion Mode
Transitions: Tackling System- and User-Specific Variability in Lower-Limb
Exoskeletons](/papers/2411.12573).

The authors report that on autonomyo, BO raised four transition accuracies
from 77.5, 75, 77.5 and 75% to 90, 90, 87.5 and 82.5%, with the values read
from a figure [arxiv:2411.12573 §4.2, fig. 6]. On eWalk the text says BO gave
"the same increases" as rescaling for three transitions, without exact values,
and reports 96 and 98% for the two stair-descent transitions [arxiv:2411.12573
§4.2, fig. 6]. The statistics-based rescaling,
which needs no labels, gave no improvement on autonomyo [arxiv:2411.12573 §4.2].

### Random-forest surrogates (SMAC)

SMAC uses a random forest as its model [arxiv:1908.00709 tab. 2]. The survey
reports that random forests do better than GPs in high-dimensional settings
[arxiv:1908.00709 §4.2.4]. It also lists random forests as one of the
surrogates for NAS, alongside TPE [arxiv:1908.00709 §4.2.4]. Card:
[AutoML: A Survey of the State-of-the-Art](/papers/1908.00709).

### Tree-structured Parzen estimator (TPE)

Hyperopt and BOHB use TPE [arxiv:1908.00709 tab. 2]. The survey does not
describe TPE's internals; it names it as a BO model [arxiv:1908.00709 §4.2.4].

### Multi-fidelity BO (BOHB, FABOLAS)

BOHB combines TPE-based BO with Hyperband, which drops the worse half of the
configurations early to save budget [arxiv:1908.00709 §4.3.1, §4.3.2].
FABOLAS instead models both loss and training time as functions of dataset
size, and the survey reports it as 10–100× faster than other BO methods
[arxiv:1908.00709 §4.3.2]. The 10–100× figure is the survey's citation of
that work, not a result reproduced here. Card:
[AutoML: A Survey of the State-of-the-Art](/papers/1908.00709).

### BO for architecture search

Auto-Keras uses BO to guide network morphisms, which grow a trained parent
into a deeper or wider child [arxiv:1908.00709 §4.1.4]. Joint search over
hyperparameters and architecture with BOHB is described in
[arxiv:1908.00709 §6.4]. In both cases the survey treats BO as one of several
optimizers rather than the default for NAS.

## When to use which

The survey gives few direct comparisons, so most rules below are constraints
it states, not measured winners.

| Situation | Use | Why |
|---|---|---|
| Few continuous hyperparameters, cheap-enough trials | GP surrogate (e.g. Spearmint, GPyOpt) | Six of the nine libraries the survey lists use a GP [arxiv:1908.00709 tab. 2] |
| Kernel choice is unclear and you can afford several GPs per step | EGP-TS ([card](/papers/2205.14090)) | Authors report lower simple regret than single-kernel GP-TS and GP-EI, from plots [arxiv:2205.14090 §5.1, fig. 1–3] |
| Parallel workers, and no tuned exploration parameter wanted | Thompson sampling (single GP or EGP-TS) | Thompson sampling needs no extra diversity rule for parallel queries [arxiv:2205.14090 §1] |
| High-dimensional or mixed categorical space | Random-forest surrogate (SMAC) or TPE (Hyperopt) | Survey reports RF does better than GP in high dimensions [arxiv:1908.00709 §4.2.4] |
| Several objectives at once (for example accuracy against memory or FLOPs) with a fixed trial budget | Multi-objective BO with a GP per objective; ARS-MOBOpt for the RL-trained proposal step ([card](/papers/2305.14109)) | Authors report higher hypervolume than five baselines within 150 samples on two tasks, from a figure [arxiv:2305.14109 §5.1, fig. 2] |
| Proposal step driven by a deep RL policy such as PPO | Budget the tuning, or keep ARS | Default PPO did no better than random sampling on one task, and the tuning cost is not counted in that comparison [arxiv:2305.14109 §5.4.1, fig. 5] |
| Architecture search with variable-length encodings | Not GP; use a tree-based or neural surrogate | GP handles variable-length architectures poorly [arxiv:1908.00709 §4.2.4] |
| Controller with hundreds to thousands of parameters, and a safety limit that must hold during tuning | HdSafeBO: latent trust region with an optimistic safe set ([card](/papers/2412.20350)) | Authors report objective 3.96 against 3.07 for LineBO on 1000-D synthetic functions [arxiv:2412.20350 tab. 1]; the probability threshold is not stated [arxiv:2412.20350 app. B.3] |
| A few thresholds to personalize per user, and labelled transitions from that user are available | BO over the thresholds against the labelled cost ([card](/papers/2411.12573)) | Authors use 30 BO iterations against 100 grid evaluations [arxiv:2411.12573 §4.3, fig. 8]; the cost needs labelled data [arxiv:2411.12573 alg. 2] |
| Same, but no labelled data for the user | Statistics-based rescaling of thresholds ([card](/papers/2411.12573)) | Needs no labels [arxiv:2411.12573 §2.5.1]; on one device it gave no improvement [arxiv:2411.12573 §4.2] |
| Each trial is expensive but can be run on a subset of data or epochs | Multi-fidelity BO (BOHB, or FABOLAS) | Early dropping and dataset-size modelling reduce budget [arxiv:1908.00709 §4.3.1, §4.3.2] |
| Need a baseline before trusting any optimizer | Random search with the same budget | Survey cites studies where random search with early stopping matched ENAS and where SOTA NAS was not significantly better than random search [arxiv:1908.00709 §4.2.5] |

## Implementation sketch

The loop from the survey's algorithm 1 [arxiv:1908.00709 alg. 1], written
generically. The model, acquisition function and the number of iterations are
the choices that vary between variants.

```pseudocode
# initial random or space-filling trials [arxiv:1908.00709 alg. 1]
D = init_samples(f, theta_space)

# repeat until the time or compute budget T is spent [arxiv:1908.00709 §4.3.2]
for i in 1..T:
    # refit the surrogate to every observation so far [arxiv:1908.00709 alg. 1]
    model = fit_model(M, D)
    # pick the candidate with the best acquisition score [arxiv:1908.00709 alg. 1]
    theta_i = argmax over theta_space of acq(theta, model)
    # expensive step: train and validate [arxiv:1908.00709 alg. 1]
    y_i = f(theta_i)
    D = D + [(theta_i, y_i)]
```

Where variants differ: the fit_model step (GP, random forest, TPE), the acq
function, and whether f(theta) runs on a reduced budget, as in BOHB and FABOLAS
[arxiv:1908.00709 §4.3.2]. In EGP-TS, fit_model keeps one GP per kernel plus
mixture weights, and the acquisition step is one Thompson draw from that
mixture [arxiv:2205.14090 §3, alg. 1]. In ARS-MOBOpt, fit_model fits one GP per
objective, and the acquisition step is replaced by ARS agents trained on GP
posterior samples, whose best rollout is the proposal [arxiv:2305.14109 §4,
alg. 1]. In HdSafeBO, fit_model is one GP for f and one for g on latent points,
and the acquisition is one posterior sample of f, maximized over the optimistic
safe set inside the trust region [arxiv:2412.20350 §4, alg. 1]. In the
exoskeleton BO, the cost is the detector's error on labelled transitions, and
the acquisition is the k-weighted mean and standard deviation rule, minimized
over the threshold bounds [arxiv:2411.12573 §2.5.2, eq. 8, eq. 9].

## Contradictions

The survey gives no BO-versus-BO numbers and says nothing about which BO variant
wins [arxiv:1908.00709 §4.3.2]. The EGP-TS paper compares its method only with
GP-EI, single-kernel GP-TS, fully Bayesian GP-TS and two bandit ensembles, all
author-reported [arxiv:2205.14090 §5]. It does not compare against random
forests or TPE, so the survey's claim that random forests beat GPs in high
dimensions [arxiv:1908.00709 §4.2.4] is untested by the EGP-TS paper. The
ARS-MOBOpt paper compares against ParEGO, TurBO, MorBO, NSGA-II and random
sampling, and does not include EGP-TS [arxiv:2305.14109 §5.1].

The HdSafeBO paper compares against LineBO, SCBO, CONFIG, constrained EI and
CMA-ES, all in its own runs [arxiv:2412.20350 tab. 1]. Neither the HdSafeBO
paper nor the EGP-TS paper includes a random-forest or TPE baseline, so the
survey's high-dimensional claim for random forests [arxiv:1908.00709 §4.2.4] is
not tested in either. The HdSafeBO paper's trust-region success rule is
described in its card as differing from its own algorithm 2 in the released
code [arxiv:2412.20350 §4.2, alg. 2]; check that before implementing the rule.

The exoskeleton paper's headline gain of up to 80% is not in a table. The
closest in-text case is one eWalk transition, read from a figure
[arxiv:2411.12573 §4.2, fig. 7]. Its BO and grid-search comparison is on one
subject [arxiv:2411.12573 §4.3], so it does not show the gain across users.

Random search: the survey's reading that random search is competitive for NAS
[arxiv:1908.00709 §4.2.5, §6.1] is a general observation. The ARS-MOBOpt paper
reports that its BO method beats random sampling on hypervolume for two
TinyML tasks [arxiv:2305.14109 §5.1, fig. 2]. The two claims concern different
problems (single-objective NAS against multi-objective TinyML design), and
neither paper settles the other.

Within the TinyML paper: the text claims ARS-MOBOpt beats ParEGO on 7 of 10
time-series datasets with at least 70% accuracy, and on par on one. The
hypervolume column of the same table does not show that count, and the text
does not say which metric it uses [arxiv:2305.14109 §5.1, tab. 4]. The
introduction gives a RAM limit of 512 Kb, and the experiments use 256 KB
[arxiv:2305.14109 §1, §5]. The paper does not explain the gap.

## Papers

- [AutoML: A Survey of the State-of-the-Art](/papers/1908.00709): surveys BO for hyperparameters and NAS, lists the open-source BO libraries and their surrogate models, and gives the generic SMBO loop.
- [Surrogate modeling for Bayesian optimization beyond a single Gaussian process](/papers/2205.14090): ensemble of GPs with Thompson sampling (EGP-TS); author-reported results, and the released code differs from the paper.
- [Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML](/papers/2305.14109): multi-objective BO with one GP per objective and ARS agents as the proposal step (ARS-MOBOpt), for microcontroller networks; author-reported results, and the released code was not checked against the paper.
- [Safe Bayesian Optimization for the Control of High-Dimensional Embodied Systems](/papers/2412.20350): HdSafeBO, a trust-region Thompson-sampling search in a PCA or VAE latent space with an optimistic safety set; author-reported results, and the released code departs from the algorithm in its trust-region update.
- [Locomotion Mode Transitions: Tackling System- and User-Specific Variability in Lower-Limb Exoskeletons](/papers/2411.12573): BO of two thresholds per transition for a finite-state detector on two exoskeletons, compared with statistics-based rescaling and grid search; author-reported results, with the headline gain not in a table.