AutoML: A Survey of the State-of-the-Art

In plain words

Building a good deep learning model involves many choices made by hand: which data to collect and how to clean and augment it, which features to build, how many layers the network has and how they connect, and which learning rate and batch size to train with. AutoML tries to automate those choices so that a computer, not an expert, runs the trial and error.

This survey walks through that pipeline step by step and spends most of its pages on neural architecture search (NAS), where an algorithm designs the network itself. It explains the three ingredients every NAS method has: a space of allowed architectures (for example, "stack copies of one small repeated cell"), a strategy for searching it (evolution, reinforcement learning, gradient descent, a cheap predictive model, or plain random sampling), and a way to judge each candidate without paying for full training every time (fewer epochs, smaller data, or one big network whose weights all candidates share).

Its most useful parts for a practitioner are two big comparison tables. They show early NAS methods spending thousands of GPU-days, while later weight-sharing and gradient-based methods reach similar CIFAR-10 accuracy in under a day on one GPU, and random search with the same tricks is often nearly as good. The survey also warns that these numbers come from different training setups, so the tables are a map of the field rather than a fair contest.

TL;DR

A breadth-first survey (v6, April 2021), not a method paper; it has no experiments or code of its own. It structures AutoML as data preparation → feature engineering → model generation (search space + optimizer for hyperparameters or architecture) → model evaluation §1 fig. 1. For NAS it defines four search-space families (entire-structured, cell-based, hierarchical, morphism-based) §4.1, six optimizer families (EA, RL, gradient descent, surrogate-model, grid/random, hybrids) §4.2, and four evaluation speed-ups (low fidelity, weight sharing, surrogates, early stopping) §5. It compiles reported accuracy, parameter counts and GPU-days for about 50 NAS methods on CIFAR-10 tab. 3 and ImageNet tab. 4. Read it for orientation, the DARTS/Gumbel/mixed-level equations, and the one-shot vs one-stage distinction §6.2 §6.3; go to the cited primary papers before implementing anything.

Problem

Well-performing deep networks were designed by experts through costly trial and error; the survey cites VGG-16 at over 130 million parameters, about 500 MB and 15.3 billion FLOPs for one 224×224 image §1. AutoML aims to build an ML pipeline automatically within a limited compute budget §1. Earlier surveys covered only NAS, or covered HPO and feature engineering but little NAS §1 tab. 1; this one claims to cover data preparation, feature engineering, HPO and NAS together, aiming for breadth rather than depth in each sub-topic §1.

Method

The "method" of a survey is its taxonomy. Section numbering of the compiled paper: §1 Introduction, §2 Data Preparation, §3 Feature Engineering, §4 Model Generation (§4.1 search space, §4.2 architecture optimization, §4.3 HPO), §5 Model Evaluation, §6 NAS Discussion, §7 Open Problems, §8 Conclusions. No appendix. Tables: tab. 1 survey comparison, tab. 2 BO libraries, tab. 3 NAS on CIFAR-10, tab. 4 NAS on ImageNet, tab. 5 AutoML applications. One algorithm: alg. 1 SMBO. Equations 1–14 are numbered in order of appearance.

Data preparation §2

• Collection §2.1: search the web for data, then filter mismatched results, self-label (active learning, self-training, co-training, co-learning), fine-tune on web data to bridge distribution shift, and rebalance with SMOTE-style minority oversampling §2.1.1; or synthesise data with simulators (e.g. OpenAI Gym, Unreal Engine 4 for the IRS stereo dataset) or GANs for images, tables and text §2.1.2.
• Cleaning §2.2: from crowdsourced systems (Katara) to cleaning only a subset (SampleClean, ActiveClean), to automated search over cleaning operations (BoostClean as boosting, AlphaClean as HPO over pipelined cleaning operators), meta-learning, and continuous cleaning of streaming data §2.2.
• Augmentation §2.3 fig. 3: image transforms grouped as affine, elastic (colour/blur/channel shuffle), advanced (random erasing, blending, Cutout, mixup) and neural (adversarial noise, style transfer, GANs); text and audio augmentation via synonym insertion, back-translation, RNN noising, NLPAug. Automated policy search starts with AutoAugment (RL, almost 500 GPU hours per search) and continues with gradient-based, Bayesian, online, greedy and random-search variants, plus search-free UniformAugment §2.3.

Feature engineering §3

• Selection §3.1 fig. 4: an iterative loop of subset generation, evaluation, stop check and validation. Search strategies: complete (exhaustive or BFS/branch-and-bound/beam/best-first), heuristic (SFS, SBS, bidirectional), random (simulated annealing, GAs). Evaluation: filter (score each feature by variance, correlation, chi-square, mutual information, then threshold), wrapper (classifier accuracy on the subset), embedded (regularization, trees, deep nets).
• Construction §3.2: expand the feature space with operators chosen by feature type (Boolean: conjunction/disjunction/negation; numeric: min/max/add/subtract/mean; nominal: Cartesian product, M-of-N); automated via decision trees, GAs or annotation-based methods, then filtered with feature selection.
• Extraction §3.3: dimensionality-reducing maps such as PCA, ICA, isomap, LDA, pretrained-network hidden units, and autoencoder variants.

NAS search spaces §4.1

A network is a DAG of B ordered nodes (tensors) whose edges are operations from a candidate set O (convolution, pooling, activations, skip, concat, add, and human-designed modules such as depthwise-separable or dilated convolutions and SE blocks) §4.1:

Z_k = \sum_{i=1}^{N_k} o_i(I_i), \quad o_i \in O eq. 1

where N_k is the in-degree of node Z_k.

1. Entire-structured §4.1.1 fig. 6: a chain of layers, optionally with arbitrary skip connections. Simple, but expensive to search for deep nets and does not transfer from small to large datasets.
2. Cell-based §4.1.2 fig. 7: search one normal cell (keeps spatial size) and one reduction cell (halves height and width, doubles channels), then stack them; the number of normal-cell repeats is set by hand. Following NASNet, a cell has B blocks; each block is a tuple (I_1, I_2, O_1, O_2, C): two inputs chosen from the previous two cells' outputs and earlier blocks in the same cell, one operation per input, and a combine op (add or concat). For the first cell, both inputs are the image. Implementation details the survey calls out: insert a 1×1 "calibration" convolution on every node input so channel counts match, with stride 2 when input and output resolutions differ and stride 1 otherwise; all blocks themselves use stride 1 §4.1.2. Search-space size with M candidate ops, L layers and B blocks per cell: N_{entire} = M^L \times 2^{\frac{L(L-1)}{2}} eq. 2 One cell has (M^B \times (B+2)!)^2 options; with normal and reduction cells: N_{cell} = (M^B \times (B+2)!)^4 eq. 3 With M=5, L=10, B=3: N_{entire} = 3.44\times10^{20} versus N_{cell} = 5.06\times10^{16} §4.1.2. Two-stage gap: DARTS searches with 8 cells (to save GPU memory) and evaluates with 20, so the best shallow cell may not be best when deep. P-DARTS grows the searched network over stages (5 → 11 → 17 cells) while pruning candidate ops (5 → 3 → 2) §4.1.2 fig. 8.
3. Hierarchical §4.1.3: search the network level too (Auto-DeepLab's down-sampling path fig. 9); build higher-level cells from lower-level ones, each defined by an upper-triangular adjacency matrix G with G_{ij}=k meaning op o_k sits between nodes i and j (HierNAS) fig. 10; grow cells block by block with a surrogate ranking the top-k (PNAS); or let each stage have its own block structure repeated a variable number of times (MnasNet's factorized hierarchical space, used by FBNet and ProxylessNAS) fig. 11.
4. Morphism-based §4.1.4 fig. 12: grow a trained parent into a deeper or wider child while preserving its function (Net2Net IdMorph). Network morphism extends this to non-identity layers, arbitrary nonlinearities and simultaneous depth/width/kernel-size changes; Auto-Keras guides morphisms with Bayesian optimization.

Architecture optimization (AO) §4.2

Evolutionary algorithms §4.2.1 fig. 13. Encoding is direct (fixed-length binary connection strings as in Genetic CNN; DAG/CGP/NEAT encodings for variable length) or indirect (generation rules such as cellular encoding, or function-preserving morphism operators as in Lemonade). Loop:

Evolutionary NAS loop  [§4.2.1] [fig. 13]
1. Selection, one of:
   - fitness-proportional: P(h_i) = Fitness(h_i) / sum_j Fitness(h_j)
   - rank selection: probability proportional to relative rank instead of raw fitness
   - tournament: draw k networks at random, sort by performance; pick the best with
     probability p, the second with p*(1-p), and so on
2. Crossover: combine two parents into a child inheriting half of each
   (bit-level one/multi-point; whole "stages" as units in Genetic CNN;
    sub-tree swap for cellular encoding; historical markings in NEAT)
3. Mutation: point mutation (flip bits); toggle connections; add/remove skips;
   predefined operators such as changing the learning rate
4. Update (population control), one of:
   - remove the worse of two random networks
   - remove the oldest (AmoebaNet)
   - discard all at regular intervals
   - never remove (HierNAS)
   - EENA: remove the worst with probability lambda, else the oldest

Reinforcement learning §4.2.2 fig. 14 fig. 15: an RNN controller emits actions that describe an architecture; the environment trains and evaluates it and returns accuracy as reward. Zoph & Le used policy gradient to emit a whole-network string; NASNet used PPO and a cell-based space; MetaQNN used Q-learning with ε-greedy exploration and experience replay. Costs cited: 28 days on 800 K40 GPUs (Zoph & Le), 10 days on 10 GPUs (MetaQNN), 20 hours on 1 GPU (BlockQNN with distributed async search and early stopping), about 10 hours on 1 GPU for ENAS via parameter sharing in a supernet §4.2.2.

Gradient descent §4.2.3 fig. 16. DARTS relaxes each edge to a softmax-weighted mix of the K candidate ops:

\bar{o}_{i,j}(x) = \sum_{k=1}^{K} \frac{\exp(\alpha^{k}_{i,j})}{\sum_{l=1}^{K} \exp(\alpha^{l}_{i,j})} o^k(x) eq. 4

Architecture \alpha is fit on validation data and weights \theta on training data, alternately:

\min_{\alpha} \mathcal{L}_{val}(\theta^*, \alpha) \quad \text{s.t.} \quad \theta^* = \arg\min_\theta \mathcal{L}_{train}(\theta, \alpha) eq. 5

In the survey's DARTS description a cell has N ordered nodes (N=4 in the figure), every edge starts as an equal-weight mixture, and the final cell keeps only the highest-weight op on each edge §4.2.3 fig. 16. Alternatives to the bilevel problem:

\min_{\theta,\alpha} \mathcal{L}_{train}(\theta, \alpha) eq. 6 (single-level; the survey notes \alpha tends to overfit the training set)

\min_{\alpha,\theta} \left[\mathcal{L}_{train}(\theta^*, \alpha) + \lambda \mathcal{L}_{val}(\theta^*, \alpha)\right] eq. 7 (mixed-level, MiLeNAS; \lambda \ge 0, \lambda=0 recovers eq. 6)

DARTS memory grows linearly with the number of candidate ops because every op runs on every edge. GDAS, SNAS, FBNet and others instead sample with Gumbel-softmax:

\bar{o}^k_{i,j}(x) = \sum_{k=1}^{K} \frac{\exp\left((\log \alpha^{k}_{i,j} + G^{k}_{i,j})/\tau\right)}{\sum_{l=1}^{K} \exp\left((\log \alpha^{l}_{i,j} + G^{l}_{i,j})/\tau\right)} o^k(x) eq. 8

with G^k_{i,j} = -\log(-\log(u^k_{i,j})), u uniform, \tau the temperature. GDAS runs only the argmax op in the forward pass and backpropagates through eq. 8; ProxylessNAS binarizes path weights into gates so one path is active §4.2.3. Known DARTS failure: skip-connections take over late in search, giving shallow, worse cells. DARTS+ stops the search once a normal cell has two or more skip-connects; P-DARTS applies op-level dropout to limit them §4.2.3.

Surrogate model-based optimization §4.2.4: keep a record of evaluated architectures, fit a model of the objective, and query it for the next candidate. Surrogates are BO models (GP, RF, TPE) or neural nets (LSTM in PNAS and EPNAS, an MLP in NAO, an ensemble predicting mean and variance in BANANAS). GP inference cost is cubic in the number of observations and GPs handle variable-length architectures poorly; RF is reported to do better in high dimensions §4.2.4.

Grid and random search §4.2.5: a cited study found random search with early stopping matched ENAS, and another found SOTA NAS methods not significantly better than random search §4.2.5.

Hybrids §4.2.6: EA+RL (reinforced mutations; Evo-NAS), EA+GD (CARS: population shares one supernet, inherited across generations, 0.4 GPU-days), EA+SMBO (RF predicts fitness), GD+SMBO (NAO: encoder to a continuous space, accuracy predictor, decoder back to an architecture) §4.2.6.

Hyperparameter optimization §4.3

• Grid vs random search §4.3.1 fig. 17: grid trials grow exponentially with dimensionality; random search covers more distinct values of the important parameters. Refinements: coarse-to-fine grid; contracting grid that recentres on the max-likelihood point and halves the spacing each iteration until convergence; Hyperband, which repeatedly drops the worse half of configurations early to save budget.
• Bayesian optimization §4.3.2:

Algorithm: Sequential Model-Based Optimization  [alg. 1]
Input: objective f, search space Theta, acquisition function S, probabilistic model M
D <- INITSAMPLES(f, Theta)
for i in 1..T:
    p(y | theta, D) <- FITMODEL(M, D)
    theta_i <- argmax_{theta in Theta} S(theta, p(y | theta, D))
    y_i <- f(theta_i)            # expensive: train + validate
    D <- D ∪ {(theta_i, y_i)}

T is set by the available time or compute §4.3.2. BOHB combines TPE-based BO with Hyperband; FABOLAS models loss and training time as functions of dataset size and is reported 10–100× faster than other BO methods §4.3.2. Libraries are listed in tab. 2 (reproduced below).

• Gradient-based HPO §4.3.3: reversible-dynamics memory tape for thousands of hyperparameters; approximate hypergradients before convergence; reverse-mode (stores full training history) vs forward-mode (real-time updates) hypergradients; hypergradient optimizers that also tune Adam's \beta_1, \beta_2.

Model evaluation speed-ups §5

1. Low fidelity §5.1: train on subsets (FABOLAS) or downsampled images (ImageNet 64×64, 32×32, 16×16), or with fewer filters per layer; TSE ensembles several low-fidelity estimators. The survey cites Zela et al. for the conclusion that long searches per configuration are unnecessary.
2. Weight sharing §5.2: reuse weights across candidates (ENAS, morphism inheritance, single-path NAS with shared kernels).
3. Surrogates §5.3: predict performance instead of training; SemiNAS trains its predictor on a few labelled (architecture, accuracy) pairs and then adds generated pairs.
4. Early stopping §5.4: stop runs predicted to end badly, using learning-curve models (weighted combination of parametric curves) or gradient statistics that need no validation set.

NAS discussion §6

• Cost metric: \text{GPU Days} = N \times D (number of GPUs × days of search) eq. 9 §6.1. For methods that search on CIFAR-10 and transfer to ImageNet, the same search cost appears in both tables §6.1.
• Kendall tau for search-vs-evaluation ranking consistency §6.1.1: \tau = \frac{N_C - N_D}{N_C + N_D} eq. 10 (N_C/N_D = concordant/discordant pairs; 1 identical ranking, −1 reversed, 0 unrelated).
• NAS-Bench §6.1.2: NAS-Bench-101 (423,624 unique networks with CIFAR-10 results), NAS-Bench-201 (different space; CIFAR-10, CIFAR-100, ImageNet-16-120), NAS-Bench-NLP.
• Two-stage vs one-stage §6.2 fig. 18: two-stage searches and then retrains the winner; one-stage outputs trained weights directly (OFA's progressive shrinking, 1,200 V100 GPU-hours, models under 600M FLOPs; BigNAS 200M–1G FLOPs; AtomNAS; DSNAS).
• One-shot ≠ one-stage §6.3 fig. 19: one-shot means candidates share supernet weights; one-stage is about the workflow. Coupled one-shot (ENAS, DARTS, GDAS, ProxylessNAS, SNAS) biases toward fast-converging subnetworks and suffers "multimodel forgetting" (addressed by the NSAS loss). Decoupled one-shot first trains the supernet (random single-path sampling, path dropout with increasing rate, GreedyNAS multipath sampling, hypernetworks in SMASH/GHN), then searches it: random ranking, EA over inference-only evaluations (SPOS), SETN's estimator, or ranking by symmetrized KL between the candidate's and the one-shot model's predictions: D_{SKL} = D_{KL}(p\|q) + D_{KL}(q\|p), \quad D_{KL}(p\|q) = \sum_{i=1}^{n} p_i \log\frac{p_i}{q_i} eq. 11 computed on 64 random training examples in the cited work §6.3.
• Joint hyperparameter + architecture optimization §6.4: BOHB over a combined space (Zela et al.); AutoHAS expresses continuous hyperparameters as weighted sums of categorical bases, e.g. lr = w_1 \cdot 0.1 + w_2 \cdot 0.2 + w_3 \cdot 0.3; FBNetV3 trains an accuracy predictor on [architecture embedding ; training recipe].
• Resource-aware NAS §6.5: add parameters, MACs, FLOPs or measured latency to the objective. MnasNet: \max_m \; ACC(m) \times \left[\frac{LAT(m)}{T}\right]^w eq. 12 w = \alpha \text{ if } LAT(m) \le T, \text{ else } \beta eq. 13 FBNet (latency from a per-operator lookup table): \mathcal{L}(a, \theta_a) = \mathrm{CE}(a, \theta_a) \cdot \alpha \log(\mathrm{LAT}(a))^{\beta} eq. 14 SNAS makes the cost linear in the one-hot sampling variables so it stays differentiable §6.5.

Open problems §7

Search spaces still encode human bias (AutoML-Zero builds algorithms from basic math ops) §7.1; NAS lags hand-designed models on NLP, e.g. PTB perplexity §7.2 fig. 20; weak interpretability §7.3; reproducibility (unreported seeds, compute cost) §7.4; robustness to label noise and adversarial data §7.5; joint HPO+NAS §7.6; complete pipelines (TPOT, Auto-WEKA, Auto-sklearn, Auto-Keras, NNI, Vega) §7.7; lifelong learning via meta-learning NAS, unsupervised NAS (UnNAS) and incremental learning (LwF, iCaRL) §7.8. tab. 5 lists AutoML applications across CV (medical imaging, detection, segmentation, re-ID, super-resolution, restoration, GANs, disparity, video), NLP (translation, LM, NER, classification, sequence labelling, keyword spotting) and other areas (compression, GNNs, federated learning, loss/activation search, captioning, TTS, recommendation).

Hyperparameters

The survey trains nothing itself. These are the concrete values it states, mostly quoting other papers.

Results

All numbers are as reported by the original papers and collected by the survey; it ran no experiments. Different methods use different training hyperparameters and augmentation, so the survey itself calls the comparison not quite fair §6.1.2. "c/o" = Cutout. GPU-days per eq. 9.

CIFAR-10 tab. 3 (accuracy column is labelled "Top-1 Acc (%)"):

ImageNet tab. 4:

Open-source BO libraries tab. 2:

The survey's own reading of the tables §6.1: early EA and RL methods chased accuracy at huge cost (AmoebaNet 3,150 GPU-days on 450 GPUs); weight sharing brought RL search to one GPU and under a day (ENAS); gradient-based methods cut cost further while reaching SOTA; random-search methods come close, and RS with weight sharing was reported to beat ENAS and DARTS. Other cited figures: P-DARTS 2.50% vs DARTS 2.83% CIFAR-10 test error §4.1.2; network morphism used one-fifteenth of the training time and beat the original VGG16 §4.1.4; CARS 0.4 GPU-days vs AmoebaNet 3,150 §4.2.6; PNAS evaluated over five times as many models as ENAS and was eight times faster in total compute §5.3.

Paper vs code

No official code.

Pitfalls

• Temperature direction in eq. 8 is stated backwards. The text says the Gumbel-softmax approaches one-hot as \tau \to \infty §4.2.3. From eq. 8 itself, dividing the logits by a large \tau pushes them all toward zero and the weights toward uniform; the near-one-hot limit is \tau \to 0. Anneal \tau downward if you want discrete samples.
• Eq. 8 reuses k as both the output index on the left-hand side and the summation variable on the right. Read it as the mixed output of edge (i,j), as in eq. 4.
• Eq. 7 condition is incomplete. The text says \lambda=0 gives single-level optimization, then says "in contrast" it becomes bilevel without stating for which \lambda §4.2.3. It also writes \theta^* in both terms while minimizing over \theta. Take the exact update rule from the MiLeNAS paper, not from this summary.
• Eq. 14 is ambiguous about whether \beta applies to \log(\mathrm{LAT}) or to \mathrm{LAT}; neither \alpha nor \beta is given for FBNet §6.5. Check FBNet before implementing.
• Eq. 13 sign convention: with w = -0.07, models slower than the target T are penalized and faster ones get a small bonus; the survey recommends the same value for both branches §6.5, which makes the objective a soft constraint, not a hard one.
• Symbol overloading: B is the number of DAG nodes in eq. 1 §4.1 but the number of blocks per cell in eq. 3 §4.1.2; N is the DARTS node count §4.2.3 and the GPU count in eq. 9. The per-cell count (M^B \times (B+2)!)^2 is stated without derivation §4.1.2.
• Tournament selection probabilities p, p(1-p), \dots do not sum to 1 for finite k; how the remainder is assigned is not stated §4.2.1.
• The tables are not a fair benchmark. Methods differ in training recipe, Cutout use, epochs and GPU type; GPU-days ignore GPU generation (K40 vs V100 vs 2080Ti) eq. 9 §6.1.2. ImageNet search costs for transferred architectures are just the CIFAR-10 costs §6.1. Use NAS-Bench-101/201 when you need a controlled comparison §6.1.2.
• Suspicious table entries; verify against the primary paper before citing: DARTS ImageNet top-5 of 81.3 sits about 10 points below every other method with similar top-1 tab. 4; RENASNet's 91.12% on CIFAR-10 is far below other methods of its size tab. 3; NASNet-A is listed as CVPR18 in tab. 3 but ICLR17 in tab. 4; among the manual ImageNet baselines, DenseNet-201 has the highest top-1 but the lowest top-5, and ResNet-152 is listed at 230M parameters tab. 4. The CIFAR-10 column is labelled "Top-1" although CIFAR-10 is single-label accuracy tab. 3.
• Wrong figure reference: the one-shot paragraph points to the two-/one-stage figure §6.3; the intended illustration is fig. 19.
• Search/evaluation depth gap: a cell picked in an 8-cell supernet is deployed in a 20-cell network §4.1.2; low Kendall tau between search-phase and final rankings is the symptom to measure §6.1.1. Kendall tau as given in eq. 10 has no tie handling.
• Coupled weight sharing distorts rankings: fast-converging ops (notably skip-connect) get more updates and dominate §4.2.3 §6.3; one cited study found weight sharing degrades individual architectures and the true ranking §6.3. Always run random search with the same supernet as a baseline §4.2.5.
• DARTS memory scales with the number of candidate ops, since every op runs on every edge §4.2.3; single-path samplers (GDAS, ProxylessNAS) avoid this but re-couple architecture and weights §6.3.
• Fixed hyperparameters bias NAS rankings: most methods use one training recipe for all candidates, which can favour architectures that happen to suit it §6.4.
• Unreported seeds are cited as a major reproducibility problem in NAS §7.4.
• Snapshot in time: the last version is April 2021 (v6); nothing later is covered.

Lineage

• Builds on: earlier surveys focused on NAS only ("NAS Survey", "A Survey on NAS", "NAS Challenges") or on AutoML with little NAS ("A Survey on AutoML", AutoML Challenges, AutoML Benchmark) tab. 1. None of these has a card here.
• Compared against: no methods of its own; it compiles results of ENAS, DARTS, P-DARTS, GDAS, SNAS, ProxylessNAS, PC-DARTS, AmoebaNet, NASNet, PNAS, MnasNet, EfficientNet, OFA and others tab. 3 tab. 4.
• Superseded by: not known from content in this wiki.