--- kind: concept slug: neural-architecture-search title: Neural architecture search summary: >- Automatic design of network structure: a search space, a strategy for proposing candidates and a cheap way to score them. Evolution and RL cost hundreds to thousands of GPU-days; weight-sharing and gradient methods report under a few GPU-days on CIFAR-10. Trade-offs: search cost, a gap between searched and deployed depth, and shared weights that can distort rankings. tags: [automl, nas, search-space, weight-sharing, latency] papers: ["1908.00709", "2212.08059", "2206.01191", "1906.04358", "1905.04919", "1812.00332", "1908.09791", "2304.00501", "2605.06714", "2309.11932", "2305.14109", "2407.14514", "2504.09685"] related: [differentiable-architecture-search, bayesian-optimization, hyperparameter-optimization, efficient-vision-backbone, vision-transformer, network-pruning, knowledge-distillation, intermittent-computing, hardware-aware-training] status: draft updated_at: 2026-10-25 --- ## Overview Neural architecture search (NAS) lets an algorithm choose the structure of a network instead of an expert. Every method has three parts: a search space of allowed architectures, a strategy that proposes the next candidate, and a way to estimate how good a candidate is without paying for full training [arxiv:1908.00709 §4.1] [arxiv:1908.00709 §4.2] [arxiv:1908.00709 §5]. A network in this framing is a directed acyclic graph whose edges each carry one operation from a candidate set [arxiv:1908.00709 §4.1, eq. 1]. The search decides which operation sits on each edge and how the edges connect. The variants below differ in how each of the three parts is done. One later paper, EfficientFormerV2, applies the same three parts to a hybrid vision transformer. It searches depth, width and feed-forward expansion, and scores each candidate on both parameter count and measured phone latency [arxiv:2212.08059 §4.2]. Its method is covered under the search strategy and objective variants below. Its predecessor, EfficientFormer, uses a Gumbel-softmax supernet and a phone latency table to choose depth, width and block types [arxiv:2206.01191 §4.2]. Both are covered below. Two more papers sit at the edges of the design space. WANN searches topologies whose connections all share one value and trains no weight [arxiv:1906.04358 §3]. BayesNAS reads the choice of operations as sparse Bayesian learning over one over-parameterised network [arxiv:1905.04919 §3]. Two other papers search on the deployment target rather than a proxy task. ProxylessNAS runs the search on ImageNet and on a chosen device, with latency in the loss [arxiv:1812.00332 §3.3.1]. Once-for-All trains one super-network and then picks a sub-network for each device by search, with no new training of the super-network per device [arxiv:1908.09791 §3.4]. Four more papers cover narrower settings. One proposes candidate networks with a language model and filters them on MAC and SRAM budgets before any training [arxiv:2504.09685 §3.1]. One extends an intermittent-aware search to batteryless microcontrollers with about 2 KB of SRAM [arxiv:2407.14514 §4.2]. One searches training and pruning settings with a Gaussian-process surrogate, and keeps the network structure fixed [arxiv:2305.14109 §4] [tab. 1]. The TinyML survey places NAS among its model-design techniques [arxiv:2309.11932 §5.2.1]. The survey that this page draws on runs no experiments of its own. Its numbers are copied from the original papers and compared in tables, so they show the field's shape rather than a controlled ranking [arxiv:1908.00709 §6.1.2]. Check the primary paper before implementing any method. ## Variants ### Search spaces **Entire-structured.** A chain of layers, optionally with skip connections between any layers. It is simple, but the survey says it is expensive to search for deep networks and does not transfer well from small to large datasets [arxiv:1908.00709 §4.1.1]. **Cell-based.** The search finds one normal cell, which keeps the spatial size, and one reduction cell, which halves height and width and doubles channels. The network stacks these cells, and the repeat count is set by hand [arxiv:1908.00709 §4.1.2]. This is the most common space in the survey's tables. It shrinks the space a great deal: with 5 candidate operations, 10 layers and 3 blocks per cell, the entire-structured space holds about 3.44×10²⁰ options against about 5.06×10¹⁶ for the cell-based one [arxiv:1908.00709 §4.1.2, eq. 2, eq. 3]. The cell design creates a depth gap. DARTS searches with 8 cells to save GPU memory and then evaluates with 20, so the best shallow cell is not guaranteed to be best when deep. P-DARTS grows the searched network in stages (5, 11, then 17 cells) while pruning candidate operations (5, then 3, then 2) [arxiv:1908.00709 §4.1.2, fig. 8]. **Hierarchical.** The space also searches the network level, or builds higher-level cells from lower-level ones. Examples in the survey include a down-sampling path for segmentation, cells defined by an adjacency matrix whose entries pick operations, and a factorised space where each stage has its own block structure repeated a variable number of times [arxiv:1908.00709 §4.1.3, fig. 9, fig. 10, fig. 11]. **Morphism-based.** A trained parent network grows into a deeper or wider child while keeping its function, so the child starts from the parent's accuracy. Network morphism extends this to more layer types and to changes in depth, width and kernel size at once [arxiv:1908.00709 §4.1.4, fig. 12]. ### Search strategies **Evolutionary.** A population of networks is mutated and recombined. Selection can be fitness-proportional, rank-based or tournament-based, and the population can drop its oldest members (AmoebaNet) or its worst ones [arxiv:1908.00709 §4.2.1, fig. 13]. Early EA runs were very expensive: the survey lists AmoebaNet-B at 3,150 GPU-days on 450 K40 GPUs for CIFAR-10 [arxiv:1908.00709 tab. 3]. **Weight-agnostic evolution.** WANN evolves minimal topologies with three operators: insert a node, add a connection, and change an activation. It ranks networks by mean reward and connection count, using Pareto ranking, and scores each one with a single shared weight tried over six values. No weight is trained [arxiv:1906.04358 §3] [tab. 1]. On swing-up, the author-reported result with one tuned shared weight is 723 against 8 for a fixed network, and on car racing it is 608 against -37 [arxiv:1906.04358 tab. 1]. The baselines are hand-designed networks, not searched ones [arxiv:1906.04358 §4]. The authors also say the networks fail when individual weights are drawn at random, so they are not independent of their weight values [arxiv:1906.04358 §4]. The card's cost figures are rollout counts derived from the configuration files, not GPU-days reported by the paper [arxiv:1906.04358 app. A.4]. **Reinforcement learning.** An RNN controller emits a description of an architecture, the child network is trained, and its accuracy is the reward [arxiv:1908.00709 §4.2.2, fig. 14, fig. 15]. The survey reports costs from 28 days on 800 K40 GPUs (the original controller work) down to about 20 hours on one GPU for BlockQNN with asynchronous search and early stopping [arxiv:1908.00709 §4.2.2]. Parameter sharing brought this to about 10 hours on one GPU for ENAS [arxiv:1908.00709 §4.2.2]. **Gradient descent.** The discrete choice is relaxed into a softmax-weighted mixture of all candidate operations on each edge, and the mixture weights are trained by gradient descent [arxiv:1908.00709 §4.2.3, eq. 4]. The full method, with its bilevel form and its memory and skip-connection issues, is on [its own page](/concepts/differentiable-architecture-search). The survey lists DARTS at 1.5 GPU-days (first order) and 4 GPU-days (second order) on CIFAR-10 [arxiv:1908.00709 tab. 3]. ProxylessNAS keeps one path per edge active. It samples binary gates from the softmax of its architecture parameters, so only the sampled path's activations are stored [arxiv:1812.00332 §3.2, eq. 2, eq. 3]. Its architecture update replaces the path-probability gradient with the gate gradient. The paper motivates this approximation but does not measure its bias [arxiv:1812.00332 §3.2.1, eq. 4]. Gumbel-softmax samplers such as GDAS take a similar single-path step [arxiv:1908.00709 §4.2.3, eq. 8]. See the [ProxylessNAS card](/papers/1812.00332) for where the code departs from the paper. **Bayesian pruning.** BayesNAS puts a hierarchical sparse prior on each architecture weight. The prior's variance combines the switches of the predecessor edges with the edge's own switch, and a Laplace approximation with a cheap Hessian updates it [arxiv:1905.04919 §4.1] [§5.2]. An edge is removed when its prior variance is at most 1/(2πe), the point where the entropy of a zero-mean Gaussian is no longer positive [arxiv:1905.04919 §5.3]. The paper argues that pruning on architecture-weight magnitude is questionable, because magnitude depends on weight scale [arxiv:1905.04919 §1] [§2]. For the most sparse CIFAR-10 setting (λ_w = 0.005), it reports 2.81% test error with cutout [arxiv:1905.04919 tab. 1]. The search cost is discussed under Contradictions. **Surrogate model-based.** The method keeps a record of evaluated architectures, fits a model of accuracy, and uses it to pick the next candidate. The surrogate can be a Gaussian process, a random forest, TPE, or a neural network [arxiv:1908.00709 §4.2.4]. A Gaussian process scales cubically with the number of observations and handles variable-length architectures poorly, while the survey reports random forests as better in high dimensions [arxiv:1908.00709 §4.2.4]. **Multi-objective surrogate with RL proposals (ARS-MOBOpt).** Each objective gets a Gaussian-process surrogate. For each proposal, one augmented random search agent is trained per start point on the feasible Pareto front, using rewards drawn from posterior samples of the surrogate. The best rollout of each agent becomes a candidate, which is then trained, pruned and quantised for real [arxiv:2305.14109 §4] [alg. 1]. The search variables are training and pruning settings, and the constraints are ROM under 1 MB, RAM under 256 KB and 1e9 FLOPs [arxiv:2305.14109 tab. 1] [§5]. Replacing ARS with PPO needed per-problem tuning, and the tuning budget is not counted in the comparison [arxiv:2305.14109 §5.4.1]. **Grid and random.** This is the baseline to beat. The survey cites one study where random search with early stopping matched ENAS, and another where state of the art NAS methods were not significantly better than random search [arxiv:1908.00709 §4.2.5]. Random search with weight sharing is reported to beat ENAS and DARTS [arxiv:1908.00709 §6.1]. See Contradictions below. **Greedy shrinking from a supernet.** A supernet is trained once, sampling four sub-networks per step (smallest, two random, largest): the sandwich rule [arxiv:2212.08059 §4.2.2] [arxiv:2212.08059 alg. 1]. The search then starts at the largest sub-network and, at each step, takes the shrinking action with the smallest accuracy loss per unit of score gained. Accuracy is read from the supernet on an ImageNet-1K validation split, about 4 GPU-minutes per check; parameter and latency changes come from layer shapes and a device latency table [arxiv:2212.08059 §4.2.3]. The chosen network is then trained from scratch [arxiv:2212.08059 §4.2.3]. The survey's taxonomy calls a supernet trained before it is searched "decoupled one-shot" [arxiv:1908.00709 §6.3]. The authors report 37 A100 GPU-days for the supernet [arxiv:2212.08059 app.]. The paper reports no rank-agreement check between supernet and stand-alone accuracy, so how far the shared weights distort the ranking is not stated. **Latency-table slimming (EfficientFormer).** This search trains a Gumbel-softmax supernet and trims it against a latency target instead of a score. Block types and the identity path are sampled while the supernet trains, and each choice gets an importance score from its architecture weight [arxiv:2206.01191 §4.2, eq. 7, eq. 8]. A table of measured phone latency, built for each block type and each width in steps of 16, drives the trimming. At each step the search applies the action with the lowest accuracy drop per millisecond saved: skip the least important block, remove the first attention block, or shrink the least important stage width. It stops once the summed table latency meets the target [arxiv:2206.01191 §4.2, alg. 1, app. A]. The paper does not say how the accuracy drop is estimated, whether by retraining or by a proxy [arxiv:2206.01191 §4.2]. It calls the slimming step simple and fast but gives no cost figure, and notes that an enumeration-based search may do better at much higher cost [arxiv:2206.01191 §5.3]. The [EfficientFormer card](/papers/2206.01191) records that the released repository has no search code, so only the final models can be rebuilt from it. **Predictor and evolutionary search over a supernet (Once-for-All).** A single elastic super-network is trained with progressive shrinking and then left fixed [arxiv:1908.09791 §3.3]. Its space covers depth {2, 3, 4} per unit, expand ratio {3, 4, 6}, kernel {3, 5, 7} and input size from 128 to 224 in steps of 4 [arxiv:1908.09791 §3.2]. For each device, the method samples 16K sub-networks, measures their accuracy on 10K training images and fits a predictor with three hidden layers of 400 units [arxiv:1908.09791 §3.4] [app. A]. An evolutionary search then uses that predictor and a per-device latency table to find a sub-network under the limit [arxiv:1908.09791 §3.4]. The authors report a predictor RMSE of 0.21% on their test set [arxiv:1908.09791 app. A]. Progressive shrinking carries much of the accuracy: without it, a 235M-MAC network reaches 72.4%, against 76.0% at 230M MACs with it [arxiv:1908.09791 tab. 1]. Searched detectors are a further use of NAS. DAMO-YOLO uses MAE-NAS-based search, and YOLO-NAS uses AutoNAC with QSP and QCI blocks for 8-bit quantization. The survey reports 50.0 AP for DAMO-YOLO and 52.2 AP for YOLO-NAS at 16-bit precision [arxiv:2304.00501 §15] [§18]. These are the survey's figures, not checked against the original papers. **Language-model proposals with Pareto feedback.** An open-weight LLM writes each candidate as a five-stage description. Each proposal is checked for duplication, for a MAC range of 70 to 350 million, and for an int8 peak-SRAM estimate. Rejected proposals come back with a reason, and surviving candidates are trained for 30 epochs. The non-dominated set on accuracy, MACs and parameters is summarised back to the LLM for its next proposal [arxiv:2504.09685 §3.1] [§3.2] [fig. 2]. The paper's SRAM limit is 320 KB and its target is an STM32H743 [arxiv:2504.09685 §4.1.4]. The table lists search costs in days, 1.5 to 3.5 for the three LLM runs, against 12.5 for MCUNet, as quoted from the cited works [arxiv:2504.09685 tab. 2]. **Hybrids.** The survey lists combinations such as EA with RL, EA with gradient descent (CARS, which shares one supernet across generations), EA with a surrogate, and gradient descent with a surrogate (NAO) [arxiv:1908.00709 §4.2.6]. ### Cutting the evaluation cost Each candidate still needs a score. The survey groups four ways to make that cheaper [arxiv:1908.00709 §5]: - **Low fidelity.** Train on a subset of the data, on downsampled images, or with fewer filters per layer [arxiv:1908.00709 §5.1]. - **Weight sharing.** Candidates reuse weights from one shared network, as in ENAS, or inherit weights from a morphed parent [arxiv:1908.00709 §5.2]. - **Surrogates.** A learned predictor estimates performance without training [arxiv:1908.00709 §5.3]. - **Early stopping.** Runs predicted to end badly are stopped early, using learning-curve models or gradient statistics that need no validation set [arxiv:1908.00709 §5.4]. ### Staging and sharing **Two-stage vs one-stage.** A two-stage method searches, then retrains the winner from scratch. A one-stage method outputs trained weights directly. The Once-for-All is one-stage in this sense: its sub-networks inherit trained weights from the super-network, and nothing is retrained per device [arxiv:1908.09791 §3.4]. Its reported total is about 1,200 V100 GPU-hours for the full network and the shrinking stages [arxiv:1908.09791 tab. 1]. The 80.0% headline result needs a 75-epoch fine-tune on top of that [arxiv:1908.09791 §4.2]. BayesNAS is two-stage: after the search it retrains the pruned network from scratch on its own schedule [arxiv:1905.04919 §6.1]. The LLM pipeline is also two-stage. Its chosen network is trained for 120 epochs and then distilled from a ViT-B/16 teacher, and the authors' table shows a gain of about 1.3 to 1.4 points from distillation [arxiv:2504.09685 §4.1.3] [tab. 4]. WANN is a third case. It trains no weights during the search, and its champion is judged with a shared weight [arxiv:1906.04358 §3] [tab. 1]. **One-shot.** Candidates share the weights of one supernet. The survey separates two kinds. In *coupled* one-shot methods (ENAS, DARTS, GDAS, ProxylessNAS, SNAS), architecture and weights are trained together, which favours fast-converging subnetworks [arxiv:1908.00709 §6.3]. In *decoupled* one-shot methods, the supernet is trained first, for example with random single-path sampling, and only then searched [arxiv:1908.00709 §6.3]. ### Resource-aware objectives Latency, FLOPs or parameter counts can be added to the score. MnasNet multiplies accuracy by a penalty on how far measured latency is from a target [arxiv:1908.00709 §6.5, eq. 12, eq. 13]. FBNet uses a per-operator latency lookup table in its loss [arxiv:1908.00709 §6.5, eq. 14]. The survey notes that MnasNet's exponent is a soft constraint, not a hard limit [arxiv:1908.00709 §6.5]. ProxylessNAS puts measured latency straight into the search loss. Each candidate's latency comes from a predictor, and the expected latency is the probability-weighted sum over blocks, added to the cross-entropy with a scale λ2 [arxiv:1812.00332 §3.3.1, eq. 5, eq. 6, eq. 7]. A REINFORCE variant covers rewards that are not differentiable, such as latency measured on a device [arxiv:1812.00332 §3.3.2]. The author-reported ImageNet result at 78 ms on mobile is 74.6% top-1 [arxiv:1812.00332 tab. 2]. A survey of edge vision lists MnasNet and ProxylessNAS as latency-aware NAS and says the search itself is costly [arxiv:2605.06714 §4.5]. **Power-law size and latency score (EfficientFormerV2).** The score is a constant times a product of two power-law terms, one on parameter count and one on measured iPhone 12 latency, each relative to a unit reference and each with its own exponent [arxiv:2212.08059 §4.2.1]. The exponents set a trade-off, not a free gain. At a fixed 3.5M parameters, the authors' chosen weights give 75.7% at 0.9 ms, while other weights give 77.0% at 1.1 ms and 77.3% at 1.3 ms [arxiv:2212.08059 app.]. As with MnasNet, the penalty is smooth, so it is a soft trade-off rather than a hard limit [arxiv:2212.08059 §4.2.1] [arxiv:1908.00709 §6.5]. **Power-cycle budgets (intermittent NAS).** For batteryless devices, the search also asks whether a candidate can finish within each power-on period. The baseline, iNAS, runs a design explorer for each child and a performance model that estimates energy per power cycle and end-to-end latency. A child is trained only if a feasible design meets the latency requirement [arxiv:2407.14514 §4.1]. The extension shrinks a distilled teacher before the search and uses a larger batch per power cycle. The paper says this allows much larger networks, but it gives no batch size [arxiv:2407.14514 §4.2] [§4.3]. The author reports 1.33 times the peak memory of HW-NAS on CIFAR-10 and 12% more latency than iNAS [arxiv:2407.14514 §5.5] [§5.6]. The reward as written adds the latency ratio, which favours slower designs. The card treats this as a likely typo [arxiv:2407.14514 §4.1]. ## When to use which The survey's tables give GPU cost and accuracy but do not control for training recipe, so use the cost columns as orders of magnitude [arxiv:1908.00709 §6.1.2]. | Situation | Use | Why | |---|---|---| | Few GPUs, a day or less of search | Weight-sharing or gradient methods | Several report under 1 GPU-day on CIFAR-10 [arxiv:1908.00709 tab. 3] | | Large budget, search space unknown | Evolution or RL | Both reach strong accuracy, but the survey's tables list them at thousands of GPU-days [arxiv:1908.00709 tab. 3] | | Deploy to a latency or FLOPs target | Resource-aware objective | The score includes the measured cost directly [arxiv:1908.00709 §6.5]; ProxylessNAS adds a per-device latency term to the loss [arxiv:1812.00332 §3.3.1] | | Many devices, each with its own latency or MAC limit | One super-network, then a search per device (Once-for-All) | Each device costs a search, not a new training run; the authors report 40 GPU-hours for predictor data, constant as devices grow [arxiv:1908.09791 §3.4] [tab. 1]. The 80.0% headline needs fine-tuning [arxiv:1908.09791 tab. 1] | | One target dataset and device, and the search must run on the real task (no proxy) | Direct search with binarized paths (ProxylessNAS) | Author-reported ImageNet search of about 200 GPU hours, with 74.6% top-1 at 78 ms on mobile [arxiv:1812.00332 tab. 2]; the GPU type and count are not stated | | Want trained weights without a retrain step | One-stage method | Outputs trained weights directly rather than retraining the winner [arxiv:1908.00709 §6.2] | | Supernet memory is the limit | Single-path sampler (GDAS, ProxylessNAS) | DARTS memory grows with the number of candidate operations [arxiv:1908.00709 §4.2.3] | | Target is much deeper than the search | Expect a gap; search at target depth or measure transfer | Cells searched at 8 cells were deployed at 20 [arxiv:1908.00709 §4.1.2] | | Any shared-weight method | Also run random search with the same supernet | Shared weights can distort the ranking [arxiv:1908.00709 §4.2.5, §6.3] | | Phone or edge target with both size and latency limits | Size-and-latency score on a measured device table | The score combines parameters and iPhone 12 latency; the chosen search reached 75.7% at 0.9 ms and 3.5M parameters [arxiv:2212.08059 §4.2.1] [arxiv:2212.08059 tab. 2] | | Phone latency target, a measured per-block latency table, and the search should stop at the target | Latency-table slimming (EfficientFormer) | The authors report 79.2% top-1 at 1.6 ms on the iPhone 12 NPU for L1 [arxiv:2206.01191 tab. 1]; the paper gives no search cost [arxiv:2206.01191 §5.3]; the search code is not released, so reuse the published models ([card](/papers/2206.01191)) | | Only a model in the released size class is needed | Use the released fixed networks, not a new search | The paper's code repository has the four fixed networks but not the search [arxiv:2212.08059 §4.2]; see its [card](/papers/2212.08059) | | Want to know what an architecture can do before any weight is trained | Weight-agnostic topology search (WANN) | Author-reported tasks where a shared weight beats the fixed baseline without training [arxiv:1906.04358 tab. 1]; random per-weight values make the networks fail [arxiv:1906.04358 §4] | | Want a principled prune rule instead of weight magnitude, on a short search | Bayesian pruning (BayesNAS) | Reports 2.81% CIFAR-10 error with cutout and a search cost stated two ways [arxiv:1905.04919 tab. 1] [§6.1]; see Contradictions | | Microcontroller with a few hundred KB of SRAM, LLM access, and no budget for a long search | LLM proposals with Pareto feedback | Author-reported 74.50% on CIFAR-100 at 86M MACs, with a 1.5-day search cost for the chosen network [arxiv:2504.09685 tab. 2]. The headline is selected on test accuracy, so re-select on a validation split before reporting ([card](/papers/2504.09685)) | | Batteryless device that loses power during inference, about 2 KB of SRAM | Intermittent-aware search with distillation | The only variant here that models power cycles in the search. The author reports +19.1% accuracy against HW-NAS on CIFAR-10, with the unit not stated [arxiv:2407.14514 §5.4] [fig. 11]; no official code [card](/papers/2407.14514) | | MCU limits on ROM, RAM and FLOPs, each trial a full training run, and a fixed network family | Multi-objective BO with ARS agents | Author-reported higher hypervolume than five baselines on two tasks, with a front after 150 samples [arxiv:2305.14109 §5.1] [fig. 2]. The search variables are training and pruning settings, so this does not search topology | | Need a map of TinyML and its model-design techniques | Survey | Lists NAS with reinforcement, evolutionary and gradient-based variants [arxiv:2309.11932 §5.2.1]; reports no experiments of its own [arxiv:2309.11932 §1] [§8] | | Need a controlled comparison | Benchmark tables such as NAS-Bench-101 or 201 | Same setup for every candidate [arxiv:1908.00709 §6.1.2] | | Only hyperparameters to tune, architecture fixed | Not NAS | The survey covers joint hyperparameter and architecture tuning as its own topic [arxiv:1908.00709 §6.4] | ## Implementation sketch The loop below is shared by every variant. The variants differ only in what `propose`, `estimate` and `update` do. For ProxylessNAS's binarized one-shot step, `propose` samples one path per edge and `update` alternates weight and architecture steps [arxiv:1812.00332 §3.2.1]. For a supernet, `estimate` reads accuracy from the shared weights, and `propose` can be a predictor-guided evolutionary search [arxiv:1908.09791 §3.4]. ```pseudocode # Define the search space: allowed ops and the cell or network layout [§4.1] space = define_search_space() # Set up optimizer state: population, controller, surrogate or alpha [§4.2] state = optimizer.init(space) for trial in range(budget): # Propose a candidate: EA mutation, RL sample, gradient step on alpha, # surrogate argmax or a random draw [§4.2] arch = optimizer.propose(state, space) # Score it cheaply: fewer epochs or data, shared weights, a predictor # or early stopping [§5] score = estimate(arch) # Feed the score back so the next proposal can improve [§4.2] state = optimizer.update(state, arch, score) # Retrain the winner (two-stage). One-stage methods skip this [§6.2] best = optimizer.best(state) final = train_full(best) ``` ## Contradictions - **Weight sharing as a help or a harm.** The survey reports that random search with weight sharing beats ENAS and DARTS [arxiv:1908.00709 §6.1]. It also cites a separate study that found weight sharing degrades individual architectures and the true ranking [arxiv:1908.00709 §6.3]. The survey does not resolve this. WANN scores its topologies with one shared weight and reports results against hand-designed networks, not against weight-sharing NAS methods [arxiv:1906.04358 §4]. - **BayesNAS search cost.** The abstract and [tab. 1] give 0.2 GPU days for the CIFAR-10 search. The experiments section says the search takes about 3 hours on a single GPU [arxiv:1905.04919 §6.1], which is about 0.125 GPU days. The paper does not reconcile the two. - **Random search versus state of the art.** One cited study found random search with early stopping matched ENAS, and another found state of the art methods not significantly better than random search [arxiv:1908.00709 §4.2.5]. These are two separate findings, not the same test. - **Random search versus a joint size-speed search.** The survey reports random search as close to the state of the art [arxiv:1908.00709 §4.2.5]. EfficientFormerV2's authors report their search beating two random sub-networks of the same size by about 0.7 to 1.0 points at 3.5M parameters [arxiv:2212.08059 §5.3]. These are single-paper comparisons in different spaces and recipes, so they are not a direct test of the same claim. EfficientFormer's authors make a similar comparison: their L1 network at 1.3 GMACs reaches 79.2% top-1, against 73.3% to 78.3% for five random sub-networks of the same compute, trained with the same recipe [arxiv:2206.01191 tab. 5]. This is also a single-paper comparison. - **Search cost is not one quantity.** ProxylessNAS gives about 200 GPU hours for its ImageNet search, without GPU type or count [arxiv:1812.00332 tab. 2]. Once-for-All gives about 1,200 V100 GPU-hours for its super-network and 40 GPU-hours for the per-device predictor data [arxiv:1908.09791 tab. 1] [§3.4]. The figures cover different work, so this page does not rank the two methods by them. - **PNASNet numbers in the edge survey.** The survey gives PNASNet a CIFAR-10 error of 3.41%, and in the same section a CIFAR-10 accuracy of 82.9%. Both cannot hold for one setup. It also gives a search cost of 400 GPU hours against 2,000 for NASNet [arxiv:2605.06714 §4.5]. Check the original paper before using any of these. - **Headline split versus released code.** The LLM paper reports 74.50% on CIFAR-100, but the paper does not say which split is used for selection. The card's review of the authors' code finds that candidates are ranked and picked on test accuracy [arxiv:2504.09685 §5.1]. The headline is therefore not a held-out score ([card](/papers/2504.09685)). - **SRAM limit, paper versus code.** The paper gives 320 KB as a hard constraint [arxiv:2504.09685 §4.1.4], while the released search admits candidates up to about 512 KB ([card](/papers/2504.09685)). - **Reported numbers in the survey's tables.** The survey itself flags several entries as suspicious and needing checks against the primary papers: a DARTS ImageNet top-5 score far below its peers, a RENASNet CIFAR-10 score far below methods of similar size, and a NASNet-A venue that differs between the two tables [arxiv:1908.00709 tab. 3, tab. 4, §6.1.2]. ## Papers - [AutoML: A Survey of the State-of-the-Art](/papers/1908.00709): survey of AutoML that structures NAS into search spaces, optimizers and evaluation speed-ups, and compiles reported costs and accuracies for about 50 methods. - [Weight Agnostic Neural Networks](/papers/1906.04358): evolves topologies scored with one shared weight and no weight training, with author-reported results on swing-up, biped, car racing and MNIST. - [BayesNAS: A Bayesian Approach for Neural Architecture Search](/papers/1905.04919): one-shot search with a hierarchical sparse prior whose variance decides which edges are pruned, with author-reported CIFAR-10 and ImageNet results. - [Rethinking Vision Transformers for MobileNet Size and Speed](/papers/2212.08059): EfficientFormerV2, a hybrid vision transformer family found by a joint parameter and phone-latency search over depth, width and FFN expansion. The search is not in the released code. - [EfficientFormer: Vision Transformers at MobileNet Speed](/papers/2206.01191): the predecessor, a pure vision transformer whose depth, width and block types come from a Gumbel supernet trimmed against a phone latency table; author-reported ImageNet and iPhone 12 results. - [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](/papers/1812.00332): searches ImageNet and CIFAR-10 directly, keeps one sampled path per edge in memory and puts expected latency in the loss; author-reported results on GPU, CPU and mobile. - [Once-for-All: Train One Network and Specialize it for Efficient Deployment](/papers/1908.09791): one elastic super-network trained by progressive shrinking, then specialised per device by an accuracy predictor and latency table; author-reported ImageNet results, some needing fine-tuning. - [A Comprehensive Review of YOLO Architectures in Computer Vision](/papers/2304.00501): YOLO survey that reports the NAS-designed DAMO-YOLO and YOLO-NAS detectors. - [Edge Deep Learning in Computer Vision and Medical Diagnostics](/papers/2605.06714): edge-deployment survey that reviews NAS and latency-aware search at a high level; it runs no experiments of its own. - [A Machine Learning-oriented Survey on Tiny Machine Learning](/papers/2309.11932): PRISMA review of TinyML that places NAS in its model-design taxonomy, with popularity labels and cited-work counts; it reports no experiments. - [Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML](/papers/2305.14109): multi-objective Bayesian optimization whose proposal step uses augmented random search agents; it tunes training and pruning settings for microcontroller networks, with author-reported results on two tasks. - [Accelerate Intermittent Deep Inference](/papers/2407.14514): extends intermittent-aware NAS to batteryless microcontrollers with about 2 KB of SRAM, using distillation and a larger batch per power cycle; author-reported results on CIFAR-10 and Tiny ImageNet, and no official code. - [Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?](/papers/2504.09685): an LLM proposes five-stage CNNs, which are checked for MACs and SRAM and guided by Pareto feedback; author-reported CIFAR-100 results on an STM32H743 target.