Neural architecture search

Overview

Neural architecture search (NAS) lets an algorithm choose the structure of a network instead of an expert. Every method has three parts: a search space of allowed architectures, a strategy that proposes the next candidate, and a way to estimate how good a candidate is without paying for full training 1908.00709 §4.1 1908.00709 §4.2 1908.00709 §5.

A network in this framing is a directed acyclic graph whose edges each carry one operation from a candidate set 1908.00709 §4.1, eq. 1. The search decides which operation sits on each edge and how the edges connect. The variants below differ in how each of the three parts is done.

One later paper, EfficientFormerV2, applies the same three parts to a hybrid vision transformer. It searches depth, width and feed-forward expansion, and scores each candidate on both parameter count and measured phone latency 2212.08059 §4.2. Its method is covered under the search strategy and objective variants below.

Its predecessor, EfficientFormer, uses a Gumbel-softmax supernet and a phone latency table to choose depth, width and block types 2206.01191 §4.2. Both are covered below.

Two more papers sit at the edges of the design space. WANN searches topologies whose connections all share one value and trains no weight 1906.04358 §3. BayesNAS reads the choice of operations as sparse Bayesian learning over one over-parameterised network 1905.04919 §3.

Two other papers search on the deployment target rather than a proxy task. ProxylessNAS runs the search on ImageNet and on a chosen device, with latency in the loss 1812.00332 §3.3.1. Once-for-All trains one super-network and then picks a sub-network for each device by search, with no new training of the super-network per device 1908.09791 §3.4.

Four more papers cover narrower settings. One proposes candidate networks with a language model and filters them on MAC and SRAM budgets before any training 2504.09685 §3.1. One extends an intermittent-aware search to batteryless microcontrollers with about 2 KB of SRAM 2407.14514 §4.2. One searches training and pruning settings with a Gaussian-process surrogate, and keeps the network structure fixed 2305.14109 §4 tab. 1. The TinyML survey places NAS among its model-design techniques [arxiv:2309.11932 §5.2.1].

The survey that this page draws on runs no experiments of its own. Its numbers are copied from the original papers and compared in tables, so they show the field's shape rather than a controlled ranking [arxiv:1908.00709 §6.1.2]. Check the primary paper before implementing any method.

Variants

Search spaces

Entire-structured. A chain of layers, optionally with skip connections between any layers. It is simple, but the survey says it is expensive to search for deep networks and does not transfer well from small to large datasets 1908.00709 §4.1.1.

Cell-based. The search finds one normal cell, which keeps the spatial size, and one reduction cell, which halves height and width and doubles channels. The network stacks these cells, and the repeat count is set by hand 1908.00709 §4.1.2. This is the most common space in the survey's tables. It shrinks the space a great deal: with 5 candidate operations, 10 layers and 3 blocks per cell, the entire-structured space holds about 3.44×10²⁰ options against about 5.06×10¹⁶ for the cell-based one 1908.00709 §4.1.2, eq. 2, eq. 3.

The cell design creates a depth gap. DARTS searches with 8 cells to save GPU memory and then evaluates with 20, so the best shallow cell is not guaranteed to be best when deep. P-DARTS grows the searched network in stages (5, 11, then 17 cells) while pruning candidate operations (5, then 3, then 2) 1908.00709 §4.1.2, fig. 8.

Hierarchical. The space also searches the network level, or builds higher-level cells from lower-level ones. Examples in the survey include a down-sampling path for segmentation, cells defined by an adjacency matrix whose entries pick operations, and a factorised space where each stage has its own block structure repeated a variable number of times 1908.00709 §4.1.3, fig. 9, fig. 10, fig. 11.

Morphism-based. A trained parent network grows into a deeper or wider child while keeping its function, so the child starts from the parent's accuracy. Network morphism extends this to more layer types and to changes in depth, width and kernel size at once 1908.00709 §4.1.4, fig. 12.

Search strategies

Evolutionary. A population of networks is mutated and recombined. Selection can be fitness-proportional, rank-based or tournament-based, and the population can drop its oldest members (AmoebaNet) or its worst ones 1908.00709 §4.2.1, fig. 13. Early EA runs were very expensive: the survey lists AmoebaNet-B at 3,150 GPU-days on 450 K40 GPUs for CIFAR-10 1908.00709 tab. 3.

Weight-agnostic evolution. WANN evolves minimal topologies with three operators: insert a node, add a connection, and change an activation. It ranks networks by mean reward and connection count, using Pareto ranking, and scores each one with a single shared weight tried over six values. No weight is trained 1906.04358 §3 tab. 1. On swing-up, the author-reported result with one tuned shared weight is 723 against 8 for a fixed network, and on car racing it is 608 against -37 1906.04358 tab. 1. The baselines are hand-designed networks, not searched ones 1906.04358 §4. The authors also say the networks fail when individual weights are drawn at random, so they are not independent of their weight values [arxiv:1906.04358 §4]. The card's cost figures are rollout counts derived from the configuration files, not GPU-days reported by the paper 1906.04358 app. A.4.

Reinforcement learning. An RNN controller emits a description of an architecture, the child network is trained, and its accuracy is the reward 1908.00709 §4.2.2, fig. 14, fig. 15. The survey reports costs from 28 days on 800 K40 GPUs (the original controller work) down to about 20 hours on one GPU for BlockQNN with asynchronous search and early stopping 1908.00709 §4.2.2. Parameter sharing brought this to about 10 hours on one GPU for ENAS 1908.00709 §4.2.2.

Gradient descent. The discrete choice is relaxed into a softmax-weighted mixture of all candidate operations on each edge, and the mixture weights are trained by gradient descent 1908.00709 §4.2.3, eq. 4. The full method, with its bilevel form and its memory and skip-connection issues, is on its own page. The survey lists DARTS at 1.5 GPU-days (first order) and 4 GPU-days (second order) on CIFAR-10 1908.00709 tab. 3. ProxylessNAS keeps one path per edge active. It samples binary gates from the softmax of its architecture parameters, so only the sampled path's activations are stored 1812.00332 §3.2, eq. 2, eq. 3. Its architecture update replaces the path-probability gradient with the gate gradient. The paper motivates this approximation but does not measure its bias [arxiv:1812.00332 §3.2.1, eq. 4]. Gumbel-softmax samplers such as GDAS take a similar single-path step 1908.00709 §4.2.3, eq. 8. See the ProxylessNAS card for where the code departs from the paper.

Bayesian pruning. BayesNAS puts a hierarchical sparse prior on each architecture weight. The prior's variance combines the switches of the predecessor edges with the edge's own switch, and a Laplace approximation with a cheap Hessian updates it 1905.04919 §4.1 §5.2. An edge is removed when its prior variance is at most 1/(2πe), the point where the entropy of a zero-mean Gaussian is no longer positive 1905.04919 §5.3. The paper argues that pruning on architecture-weight magnitude is questionable, because magnitude depends on weight scale 1905.04919 §1 §2. For the most sparse CIFAR-10 setting (λ_w = 0.005), it reports 2.81% test error with cutout 1905.04919 tab. 1. The search cost is discussed under Contradictions.

Surrogate model-based. The method keeps a record of evaluated architectures, fits a model of accuracy, and uses it to pick the next candidate. The surrogate can be a Gaussian process, a random forest, TPE, or a neural network 1908.00709 §4.2.4. A Gaussian process scales cubically with the number of observations and handles variable-length architectures poorly, while the survey reports random forests as better in high dimensions 1908.00709 §4.2.4.

Multi-objective surrogate with RL proposals (ARS-MOBOpt). Each objective gets a Gaussian-process surrogate. For each proposal, one augmented random search agent is trained per start point on the feasible Pareto front, using rewards drawn from posterior samples of the surrogate. The best rollout of each agent becomes a candidate, which is then trained, pruned and quantised for real 2305.14109 §4 alg. 1. The search variables are training and pruning settings, and the constraints are ROM under 1 MB, RAM under 256 KB and 1e9 FLOPs 2305.14109 tab. 1 §5. Replacing ARS with PPO needed per-problem tuning, and the tuning budget is not counted in the comparison 2305.14109 §5.4.1.

Grid and random. This is the baseline to beat. The survey cites one study where random search with early stopping matched ENAS, and another where state of the art NAS methods were not significantly better than random search 1908.00709 §4.2.5. Random search with weight sharing is reported to beat ENAS and DARTS 1908.00709 §6.1. See Contradictions below.

Greedy shrinking from a supernet. A supernet is trained once, sampling four sub-networks per step (smallest, two random, largest): the sandwich rule 2212.08059 §4.2.2 2212.08059 alg. 1. The search then starts at the largest sub-network and, at each step, takes the shrinking action with the smallest accuracy loss per unit of score gained. Accuracy is read from the supernet on an ImageNet-1K validation split, about 4 GPU-minutes per check; parameter and latency changes come from layer shapes and a device latency table 2212.08059 §4.2.3. The chosen network is then trained from scratch 2212.08059 §4.2.3. The survey's taxonomy calls a supernet trained before it is searched "decoupled one-shot" 1908.00709 §6.3. The authors report 37 A100 GPU-days for the supernet 2212.08059 app.. The paper reports no rank-agreement check between supernet and stand-alone accuracy, so how far the shared weights distort the ranking is not stated.

Latency-table slimming (EfficientFormer). This search trains a Gumbel-softmax supernet and trims it against a latency target instead of a score. Block types and the identity path are sampled while the supernet trains, and each choice gets an importance score from its architecture weight 2206.01191 §4.2, eq. 7, eq. 8. A table of measured phone latency, built for each block type and each width in steps of 16, drives the trimming. At each step the search applies the action with the lowest accuracy drop per millisecond saved: skip the least important block, remove the first attention block, or shrink the least important stage width. It stops once the summed table latency meets the target 2206.01191 §4.2, alg. 1, app. A. The paper does not say how the accuracy drop is estimated, whether by retraining or by a proxy 2206.01191 §4.2. It calls the slimming step simple and fast but gives no cost figure, and notes that an enumeration-based search may do better at much higher cost 2206.01191 §5.3. The EfficientFormer card records that the released repository has no search code, so only the final models can be rebuilt from it.

Predictor and evolutionary search over a supernet (Once-for-All). A single elastic super-network is trained with progressive shrinking and then left fixed 1908.09791 §3.3. Its space covers depth {2, 3, 4} per unit, expand ratio {3, 4, 6}, kernel {3, 5, 7} and input size from 128 to 224 in steps of 4 1908.09791 §3.2. For each device, the method samples 16K sub-networks, measures their accuracy on 10K training images and fits a predictor with three hidden layers of 400 units 1908.09791 §3.4 [app. A]. An evolutionary search then uses that predictor and a per-device latency table to find a sub-network under the limit 1908.09791 §3.4. The authors report a predictor RMSE of 0.21% on their test set [arxiv:1908.09791 app. A]. Progressive shrinking carries much of the accuracy: without it, a 235M-MAC network reaches 72.4%, against 76.0% at 230M MACs with it 1908.09791 tab. 1.

Searched detectors are a further use of NAS. DAMO-YOLO uses MAE-NAS-based search, and YOLO-NAS uses AutoNAC with QSP and QCI blocks for 8-bit quantization. The survey reports 50.0 AP for DAMO-YOLO and 52.2 AP for YOLO-NAS at 16-bit precision 2304.00501 §15 §18. These are the survey's figures, not checked against the original papers.

Language-model proposals with Pareto feedback. An open-weight LLM writes each candidate as a five-stage description. Each proposal is checked for duplication, for a MAC range of 70 to 350 million, and for an int8 peak-SRAM estimate. Rejected proposals come back with a reason, and surviving candidates are trained for 30 epochs. The non-dominated set on accuracy, MACs and parameters is summarised back to the LLM for its next proposal 2504.09685 §3.1 §3.2 fig. 2. The paper's SRAM limit is 320 KB and its target is an STM32H743 2504.09685 §4.1.4. The table lists search costs in days, 1.5 to 3.5 for the three LLM runs, against 12.5 for MCUNet, as quoted from the cited works 2504.09685 tab. 2.

Hybrids. The survey lists combinations such as EA with RL, EA with gradient descent (CARS, which shares one supernet across generations), EA with a surrogate, and gradient descent with a surrogate (NAO) [arxiv:1908.00709 §4.2.6].

Cutting the evaluation cost

Each candidate still needs a score. The survey groups four ways to make that cheaper 1908.00709 §5:

• Low fidelity. Train on a subset of the data, on downsampled images, or with fewer filters per layer 1908.00709 §5.1.
• Weight sharing. Candidates reuse weights from one shared network, as in ENAS, or inherit weights from a morphed parent 1908.00709 §5.2.
• Surrogates. A learned predictor estimates performance without training 1908.00709 §5.3.
• Early stopping. Runs predicted to end badly are stopped early, using learning-curve models or gradient statistics that need no validation set 1908.00709 §5.4.

Staging and sharing

Two-stage vs one-stage. A two-stage method searches, then retrains the winner from scratch. A one-stage method outputs trained weights directly. The Once-for-All is one-stage in this sense: its sub-networks inherit trained weights from the super-network, and nothing is retrained per device 1908.09791 §3.4. Its reported total is about 1,200 V100 GPU-hours for the full network and the shrinking stages 1908.09791 tab. 1. The 80.0% headline result needs a 75-epoch fine-tune on top of that [arxiv:1908.09791 §4.2]. BayesNAS is two-stage: after the search it retrains the pruned network from scratch on its own schedule 1905.04919 §6.1. The LLM pipeline is also two-stage. Its chosen network is trained for 120 epochs and then distilled from a ViT-B/16 teacher, and the authors' table shows a gain of about 1.3 to 1.4 points from distillation 2504.09685 §4.1.3 tab. 4. WANN is a third case. It trains no weights during the search, and its champion is judged with a shared weight 1906.04358 §3 tab. 1.

One-shot. Candidates share the weights of one supernet. The survey separates two kinds. In coupled one-shot methods (ENAS, DARTS, GDAS, ProxylessNAS, SNAS), architecture and weights are trained together, which favours fast-converging subnetworks 1908.00709 §6.3. In decoupled one-shot methods, the supernet is trained first, for example with random single-path sampling, and only then searched 1908.00709 §6.3.

Resource-aware objectives

Latency, FLOPs or parameter counts can be added to the score. MnasNet multiplies accuracy by a penalty on how far measured latency is from a target 1908.00709 §6.5, eq. 12, eq. 13. FBNet uses a per-operator latency lookup table in its loss 1908.00709 §6.5, eq. 14. The survey notes that MnasNet's exponent is a soft constraint, not a hard limit 1908.00709 §6.5.

ProxylessNAS puts measured latency straight into the search loss. Each candidate's latency comes from a predictor, and the expected latency is the probability-weighted sum over blocks, added to the cross-entropy with a scale λ2 1812.00332 §3.3.1, eq. 5, eq. 6, eq. 7. A REINFORCE variant covers rewards that are not differentiable, such as latency measured on a device 1812.00332 §3.3.2. The author-reported ImageNet result at 78 ms on mobile is 74.6% top-1 1812.00332 tab. 2. A survey of edge vision lists MnasNet and ProxylessNAS as latency-aware NAS and says the search itself is costly 2605.06714 §4.5.

Power-law size and latency score (EfficientFormerV2). The score is a constant times a product of two power-law terms, one on parameter count and one on measured iPhone 12 latency, each relative to a unit reference and each with its own exponent 2212.08059 §4.2.1. The exponents set a trade-off, not a free gain. At a fixed 3.5M parameters, the authors' chosen weights give 75.7% at 0.9 ms, while other weights give 77.0% at 1.1 ms and 77.3% at 1.3 ms 2212.08059 app.. As with MnasNet, the penalty is smooth, so it is a soft trade-off rather than a hard limit 2212.08059 §4.2.1 1908.00709 §6.5.

Power-cycle budgets (intermittent NAS). For batteryless devices, the search also asks whether a candidate can finish within each power-on period. The baseline, iNAS, runs a design explorer for each child and a performance model that estimates energy per power cycle and end-to-end latency. A child is trained only if a feasible design meets the latency requirement 2407.14514 §4.1. The extension shrinks a distilled teacher before the search and uses a larger batch per power cycle. The paper says this allows much larger networks, but it gives no batch size 2407.14514 §4.2 §4.3. The author reports 1.33 times the peak memory of HW-NAS on CIFAR-10 and 12% more latency than iNAS 2407.14514 §5.5 §5.6. The reward as written adds the latency ratio, which favours slower designs. The card treats this as a likely typo 2407.14514 §4.1.

When to use which

The survey's tables give GPU cost and accuracy but do not control for training recipe, so use the cost columns as orders of magnitude 1908.00709 §6.1.2.

Implementation sketch

The loop below is shared by every variant. The variants differ only in what propose, estimate and update do. For ProxylessNAS's binarized one-shot step, propose samples one path per edge and update alternates weight and architecture steps 1812.00332 §3.2.1. For a supernet, estimate reads accuracy from the shared weights, and propose can be a predictor-guided evolutionary search 1908.09791 §3.4.

# Define the search space: allowed ops and the cell or network layout [§4.1]
space = define_search_space()

# Set up optimizer state: population, controller, surrogate or alpha [§4.2]
state = optimizer.init(space)

for trial in range(budget):
    # Propose a candidate: EA mutation, RL sample, gradient step on alpha,
    # surrogate argmax or a random draw [§4.2]
    arch = optimizer.propose(state, space)

    # Score it cheaply: fewer epochs or data, shared weights, a predictor
    # or early stopping [§5]
    score = estimate(arch)

    # Feed the score back so the next proposal can improve [§4.2]
    state = optimizer.update(state, arch, score)

# Retrain the winner (two-stage). One-stage methods skip this [§6.2]
best = optimizer.best(state)
final = train_full(best)

Contradictions

• Weight sharing as a help or a harm. The survey reports that random search with weight sharing beats ENAS and DARTS 1908.00709 §6.1. It also cites a separate study that found weight sharing degrades individual architectures and the true ranking 1908.00709 §6.3. The survey does not resolve this. WANN scores its topologies with one shared weight and reports results against hand-designed networks, not against weight-sharing NAS methods 1906.04358 §4.
• BayesNAS search cost. The abstract and tab. 1 give 0.2 GPU days for the CIFAR-10 search. The experiments section says the search takes about 3 hours on a single GPU 1905.04919 §6.1, which is about 0.125 GPU days. The paper does not reconcile the two.
• Random search versus state of the art. One cited study found random search with early stopping matched ENAS, and another found state of the art methods not significantly better than random search [arxiv:1908.00709 §4.2.5]. These are two separate findings, not the same test.
• Random search versus a joint size-speed search. The survey reports random search as close to the state of the art 1908.00709 §4.2.5. EfficientFormerV2's authors report their search beating two random sub-networks of the same size by about 0.7 to 1.0 points at 3.5M parameters 2212.08059 §5.3. These are single-paper comparisons in different spaces and recipes, so they are not a direct test of the same claim. EfficientFormer's authors make a similar comparison: their L1 network at 1.3 GMACs reaches 79.2% top-1, against 73.3% to 78.3% for five random sub-networks of the same compute, trained with the same recipe 2206.01191 tab. 5. This is also a single-paper comparison.
• Search cost is not one quantity. ProxylessNAS gives about 200 GPU hours for its ImageNet search, without GPU type or count 1812.00332 tab. 2. Once-for-All gives about 1,200 V100 GPU-hours for its super-network and 40 GPU-hours for the per-device predictor data 1908.09791 tab. 1 §3.4. The figures cover different work, so this page does not rank the two methods by them.
• PNASNet numbers in the edge survey. The survey gives PNASNet a CIFAR-10 error of 3.41%, and in the same section a CIFAR-10 accuracy of 82.9%. Both cannot hold for one setup. It also gives a search cost of 400 GPU hours against 2,000 for NASNet 2605.06714 §4.5. Check the original paper before using any of these.
• Headline split versus released code. The LLM paper reports 74.50% on CIFAR-100, but the paper does not say which split is used for selection. The card's review of the authors' code finds that candidates are ranked and picked on test accuracy 2504.09685 §5.1. The headline is therefore not a held-out score (card).
• SRAM limit, paper versus code. The paper gives 320 KB as a hard constraint 2504.09685 §4.1.4, while the released search admits candidates up to about 512 KB (card).
• Reported numbers in the survey's tables. The survey itself flags several entries as suspicious and needing checks against the primary papers: a DARTS ImageNet top-5 score far below its peers, a RENASNet CIFAR-10 score far below methods of similar size, and a NASNet-A venue that differs between the two tables 1908.00709 tab. 3, tab. 4, §6.1.2.

Papers

• AutoML: A Survey of the State-of-the-Art: survey of AutoML that structures NAS into search spaces, optimizers and evaluation speed-ups, and compiles reported costs and accuracies for about 50 methods.
• Weight Agnostic Neural Networks: evolves topologies scored with one shared weight and no weight training, with author-reported results on swing-up, biped, car racing and MNIST.
• BayesNAS: A Bayesian Approach for Neural Architecture Search: one-shot search with a hierarchical sparse prior whose variance decides which edges are pruned, with author-reported CIFAR-10 and ImageNet results.
• Rethinking Vision Transformers for MobileNet Size and Speed: EfficientFormerV2, a hybrid vision transformer family found by a joint parameter and phone-latency search over depth, width and FFN expansion. The search is not in the released code.
• EfficientFormer: Vision Transformers at MobileNet Speed: the predecessor, a pure vision transformer whose depth, width and block types come from a Gumbel supernet trimmed against a phone latency table; author-reported ImageNet and iPhone 12 results.
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware: searches ImageNet and CIFAR-10 directly, keeps one sampled path per edge in memory and puts expected latency in the loss; author-reported results on GPU, CPU and mobile.
• Once-for-All: Train One Network and Specialize it for Efficient Deployment: one elastic super-network trained by progressive shrinking, then specialised per device by an accuracy predictor and latency table; author-reported ImageNet results, some needing fine-tuning.
• A Comprehensive Review of YOLO Architectures in Computer Vision: YOLO survey that reports the NAS-designed DAMO-YOLO and YOLO-NAS detectors.
• Edge Deep Learning in Computer Vision and Medical Diagnostics: edge-deployment survey that reviews NAS and latency-aware search at a high level; it runs no experiments of its own.
• A Machine Learning-oriented Survey on Tiny Machine Learning: PRISMA review of TinyML that places NAS in its model-design taxonomy, with popularity labels and cited-work counts; it reports no experiments.
• Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML: multi-objective Bayesian optimization whose proposal step uses augmented random search agents; it tunes training and pruning settings for microcontroller networks, with author-reported results on two tasks.
• Accelerate Intermittent Deep Inference: extends intermittent-aware NAS to batteryless microcontrollers with about 2 KB of SRAM, using distillation and a larger batch per power cycle; author-reported results on CIFAR-10 and Tiny ImageNet, and no official code.
• Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?: an LLM proposes five-stage CNNs, which are checked for MACs and SRAM and guided by Pareto feedback; author-reported CIFAR-100 results on an STM32H743 target.