Automated Machine Learning: State-of-The-Art and Open Challenges

In plain words

Building a good machine learning model means choosing a learning method and then tuning its many settings, such as how fast it learns or how large it is. Each choice affects accuracy, and the right answer changes from one dataset to the next. Experts make these choices from experience, which does not scale when there is more data than people to analyse it.

This survey looks at the work that tries to automate those choices. It groups the methods by idea: reuse results from earlier datasets, let a search design the network, or search the settings with cheap partial runs. It then lists the software that implements them and the steps before and after model training that still need people. It ends with the problems that remain open.

TL;DR

A 2019 survey of AutoML for the combined algorithm selection and hyper-parameter tuning (CASH) problem, plus the pipeline steps around it. Use it as a map of the field and of its named frameworks, not as evidence for any one method: it runs no experiments and most of its performance claims are quoted from other papers.

Problem

Given a set of candidate learning algorithms and a dataset split into training and validation parts, find the algorithm and its settings with the lowest validation loss, within a user-set time budget §1. The paper gives no formula for the time budget beyond stating it as a constraint §1. It argues that this search is hard for non-expert users and that the wider pipeline (data understanding, validation, preparation and deployment) also needs automation §1 §6.

Method

This is a survey, so the "method" is the taxonomy the paper uses. It covers four areas of the CASH problem and then the surrounding pipeline. Figures are described in words and cited; no figure is reproduced.

Meta-learning §2. Warm-start the search from earlier tasks. The paper names three groups §2: learning from task properties (meta-features such as instance and feature counts, statistics, landmark-learner results and entropy of class labels), learning from previous model evaluations (relative landmarks, surrogate models and warm-started multi-task learning), and learning from pretrained models (transfer learning and few-shot learning). The taxonomy is in fig. 3. Similarity between a new task and a past task is measured by the distance between their meta-feature vectors, which the paper says should be normalised or reduced first §2.

Neural architecture search §3 fig. 4. The paper sorts NAS into five families: random search, reinforcement learning (for example a recurrent controller trained with REINFORCE §3), gradient-based methods (continuous relaxation of the architecture, as in DARTS §3), evolutionary methods, and Bayesian optimisation §3.

Hyper-parameter optimisation §4 fig. 5. Two families:

• Black-box methods §4.1. Grid search, random search §4.1, Bayesian optimisation with a Gaussian process or a random-forest surrogate §4.1, tree-structured Parzen estimators §4.1, simulated annealing §4.1 and genetic algorithms §4.1.
• Multi-fidelity methods §4.2. Cheap low-fidelity evaluations (a data subset) are used in place of full runs. The paper covers learning-curve extrapolation with early termination §4.2, successive halving §4.2 and HyperBand, which runs successive halving over several budget splits §4.2.

The random-forest surrogate is described step by step in §4.1: grow a set of regression trees, each on a bootstrap sample of the observed configurations, split on randomly chosen parameters, then take the forest's mean and variance as the prediction for a new configuration.

Successive halving as the paper describes it §4.2:

# Successive halving  [§4.2]
# Input:  configs (list), initial_budget (number)
# Output: one surviving configuration

def successive_halving(configs, initial_budget):
    # Evaluate every surviving configuration with the current budget [§4.2]
    survivors = configs
    budget = initial_budget
    while len(survivors) > 1:
        scores = [evaluate(config, budget) for config in survivors]
        # Rank the survivors by score and keep the better half [§4.2]
        ranked = sort_by_score(survivors, scores, descending=True)
        survivors = ranked[: len(ranked) // 2]
        # Double the budget for the next round [§4.2]
        budget = 2 * budget
    return survivors[0]

Tools and frameworks §5. Three groups plus one for neural networks:

• Centralised frameworks run on one machine on top of Weka or scikit-learn, and are suited to small and medium datasets §5.1 tab. 1. Auto-Weka is named as the first such framework §5.1. Auto-sklearn adds meta-learning initialisation and ensembles §5.1.
• Distributed frameworks run on clusters. MLbase is the first named §5.2. ATM runs parallel workers that share a model store, with a hybrid of Bayesian optimisation and multi-armed bandits §5.2 fig. 7 tab. 2.
• Cloud-based services (Google AutoML, Azure AutoML, Amazon SageMaker) are described in prose only, with no citations §5.3.
• Neural-network automation covers Auto-Keras, Auto-Net, NNI, ENAS, NAO, DARTS and LEAF §5.4 tab. 3.

Other pipeline steps §6 fig. 8 fig. 9. Pre-modeling covers data understanding (sanity checks, feature-based slicing, data life-cycle tracking) §6.1.1, data validation and automatic repair §6.1.2, and data preparation and feature synthesis §6.1.3. Post-modeling covers experiment tracking, packaging and deployment, with MLflow, ModelChimp and datmo named §6.2.

Open challenges §7. Scalability of the distributed side; no single best optimiser; the time budget as a user decision; composing pipelines across frameworks; user-friendliness; continuous delivery; automatic data repair; and automatic feature engineering and deployment §7.

Hyperparameters

The survey gives few settings of its own. Values below are the ones stated in the text. Anything not listed is "not stated".

Results

Author-reported. The survey runs no experiments of its own; each row is a claim it quotes, with the source the paper gives. The setup is as stated in the survey, and is often thin.

Framework feature summary, centralised tools, from tab. 1:

Paper vs code

No official code. The paper's footnote links a repository, DataSystemsGroupUT/AutoML_Survey. Checked on GitHub: it is a literature list of the survey's references with a Figures folder and no code. It shows no license. It is not method code, so no code field is recorded and it was not cloned. Discrepancy check: not applicable. A survey has no implementation of its own to compare against the text.

Pitfalls

• Dated snapshot. The framework tables describe the state in 2019. Release dates, languages and open-source status of the listed tools change; check the project before relying on any row.
• Quoted numbers are not checked here. Most results are cited claims from other papers. The survey gives setup details only for some, such as the Yeast and Abalone examples, and the Abalone example names no source for its 25% figure §7.
• Cloud section has no evidence. The claims about Google, Azure and Amazon services are not cited and give no benchmark §5.3.
• Successive halving is underspecified. The text says half the candidates are removed and the budget doubles, but gives no initial budget, no rule for an odd count and no general factor η §4.2.
• HyperBand is described loosely. The text says it "optimises the search space", and the split of budget across brackets is not given §4.2.
• Inconsistent notation. The CASH definition gives an argmin over a set of algorithms that is not indexed the same way in the text §1, and the pipeline is described with variable names that change between sections.
• Tool descriptions overlap. Auto-Weka's default optimiser (SMAC) and the choice of cross-validation are described in §5.1 and again in §7 with differing emphasis. Check the source framework for the actual behaviour.
• Typos in tool names. "Hyperpot" appears for Hyperopt §5.1; the NAS table caption is garbled ("n]Neural Architecture Search") tab. 3. These do not change any claim.

Lineage

• Builds on: the CASH formulation of Auto-Weka and Auto-sklearn, and the successive-halving and HyperBand papers it cites §1 §4.2.
• Compared against: the centralised, distributed, cloud and NAS frameworks listed in tab. 1–tab. 3.
• Superseded by: not stated in this paper; the survey does not discuss later work.