Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

In plain words

An AutoML tool searches for a good machine-learning pipeline on your data within a time limit. Each candidate is a model with its settings, and each one costs time to train and score. The tool has to choose how to run that search, and the best choice depends on the dataset and on how much time there is.

This paper's system automates most of those choices. It starts from a fixed list of pipelines that worked well on many other datasets, so it does not have to measure the new data first. It runs the search in rounds: weak candidates are dropped early and strong ones get more training time. A small learned model then looks at two basic facts about the dataset, its size and its number of columns, and picks one of the search settings to use. On the authors' benchmark, the result beats the previous version of the system under the same time limit.

TL;DR

Auto-Sklearn 2.0 combines three changes to Auto-sklearn: a precomputed portfolio of starting pipelines, successive halving to stop weak candidates early, and a per-dataset policy selector trained on meta-data from many datasets. The selector uses only two meta-features and falls back to a simple policy when no meta-training dataset is at least as large on both. Results are author-reported on 39 AutoML benchmark datasets at 10 and 60 minutes, with a comparison to other AutoML systems. The experiment code is public but has no license file.

Problem

Auto-sklearn-style systems search for a pipeline under a time budget T: the chosen pipeline must come from evaluations whose times sum to less than T §2.1 eq. 5. Two design decisions matter a lot and depend on the dataset and the budget: how to estimate generalization error (holdout or k-fold cross-validation) and how to allocate budget across pipelines (full budget per pipeline, or successive halving) §1 §3.2. The authors show that no single choice wins on every dataset or horizon §4 fig. 4. The earlier system warm-started its search with a k-nearest-dataset method that needs meta-features and can run pipelines that time out on large datasets §3.1 §3.5.1. The authors aim to automate these choices and to make the search start faster §1.

Method

The system has two parts. Part I (PoSH Auto-sklearn) changes how the search starts and how budget is spent. Part II adds a learned selector that chooses the search policy for each new dataset §1.

Notation and objective

The generalization error of a pipeline is the expected loss on new data, which is estimated from a finite training set with holdout or K-fold cross-validation §2 eq. 1–4. An AutoML system is treated as a function from a dataset to a trained pipeline, and its quality is averaged over a set of meta-training datasets §2.2 eq. 6–7.

\mathrm{GE}(\mathcal{M}_\lambda) = \mathbb{E}_{(x,y)\sim P_d}\left[\mathcal{L}(\mathcal{M}_\lambda(x), y)\right] eq. 1

\widehat{\mathrm{GE}}(\mathcal{M}_\lambda, \mathcal{D}_d) = \frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(\mathcal{M}_\lambda(x_i), y_i) eq. 2

\mathcal{M}_{\lambda^*} \in \arg\min_{\lambda\in\Lambda} \widehat{\mathrm{GE}}(\mathcal{M}_\lambda, \mathcal{D}_{\text{train}}) eq. 3

\widehat{\mathrm{GE}}_{\text{CV}}(\mathcal{M}_\lambda, \mathcal{D}_{\text{train}}) = \frac{1}{K}\sum_{k=1}^{K}\widehat{\mathrm{GE}}\left(\mathcal{M}_\lambda^{\mathcal{D}^{(\text{train},k)}_{\text{train}}}, \mathcal{D}^{(\text{val},k)}_{\text{train}}\right) eq. 4

The time-bounded goal is to find the best pipeline among those whose evaluation times sum to less than the horizon T:

\mathcal{M}_{\lambda^*} \in \arg\min_{\lambda\in\Lambda} \widehat{\mathrm{GE}}(\mathcal{M}_\lambda, \mathcal{D}_{\text{train}}) \quad \text{s.t.} \quad \sum_i t_{\lambda_i} < T eq. 5

The meta-level goal averages over datasets eq. 6–7:

\mathrm{GE}(\mathcal{A}) = \mathbb{E}_{\mathcal{D}_d\sim P(\mathcal{D})}\left[\widehat{\mathrm{GE}}(\mathcal{A}(\mathcal{D}_d), \mathcal{D}_d)\right] eq. 6

\widehat{\mathrm{GE}}(\mathcal{A}, \mathbf{D}_{\text{meta}}) = \frac{1}{|\mathbf{D}_{\text{meta}}|}\sum_{d=1}^{|\mathbf{D}_{\text{meta}}|}\widehat{\mathrm{GE}}(\mathcal{A}(\mathcal{D}_d), \mathcal{D}_d) eq. 7

Here an optimization policy π bundles the system's hyperparameters and components, so that A_π is a configured system §2.2. The selector is a function Ξ from a dataset to a policy, and it is scored with the same average eq. 10 §4.1.

Part I: portfolio and successive halving

Step 1, candidate pipelines (offline). For each meta-dataset, run Auto-sklearn without meta-learning and without ensembling. Keep one candidate per dataset. Evaluate every candidate on every meta-dataset to build a performance matrix of candidates by datasets §3.1.1 §3.4.2. For meta-data generation, the authors run the system three times per dataset, keep the best pipeline on the dataset's test split, and give each run ten times the horizon §3.4.2.

Step 2, greedy portfolio (offline) alg. 1. Start with an empty portfolio. Repeatedly add the candidate that most reduces the estimated error summed over all meta-datasets, and remove it from the candidate set, until the portfolio has the target size alg. 1 §3.1.1. Each dataset's error is the error of its best member, chosen by the model selection rule eq. 8, and the summed error is eq. 9. The portfolio size used in the experiments is 32 §3.5.1. Errors are first converted per dataset to a distance to the best observed value, scaled to [0, 1] and computed separately for each (model selection, budget allocation) pair. The average of this distance is called ADTM §3.1.1.

\mathcal{M}^{\mathcal{D}}_{\lambda^*} = S(\widehat{\mathrm{GE}}, \mathcal{P}, \mathcal{D}) \in \arg\min_{\mathcal{M}^{\mathcal{D}}\in\mathcal{P}}\widehat{\mathrm{GE}}(\mathcal{M}^{\mathcal{D}}, \mathcal{D}) eq. 8

\widehat{\mathrm{GE}}_S(\mathcal{P}, \mathbf{D}_{\text{meta}}) = \sum_{d=1}^{|\mathbf{D}_{\text{meta}}|}\widehat{\mathrm{GE}}\left(S(\widehat{\mathrm{GE}}, \mathcal{P}, \mathcal{D}_d), \mathcal{D}^{\text{val}}_d\right) eq. 9

The paper's Equation 9 uses holdout validation. The greedy construction in alg. 1 and the guarantees in §3.1.2 use the test split of the meta-datasets instead. The text notes that this is allowed as long as the new datasets are kept separate §3.1.1 §3.1.2.

Step 3, online warm start. For a new dataset, evaluate the portfolio members one after another first, ordered from the ones that do well on average to the more specialized ones §3.1. While portfolio members are left, they are suggested instead of asking the optimizer for a new point §3.2.1. Afterwards Bayesian optimization (SMAC, random-forest surrogate) continues from the evaluated pipelines §3.2.1.

Step 4, budget allocation with successive halving. Successive halving (SH) starts a fixed number of pipelines at the smallest budget. It then keeps the best 1/η of them, multiplies their budget by η, and repeats until one pipeline is left or the largest budget is spent. Full budget (FB) gives every pipeline the largest budget §3.2.1. In this paper SH replaces holdout with full budget §3.2.1.

# Successive halving over pipelines  [§3.2.1]
# Input:  pipelines, smallest budget b_min, largest budget b_max, factor eta
# Output: survivors, one or more pipelines

def successive_halving(pipelines, b_min, b_max, eta):
    # Start every pipeline at the smallest budget  [§3.2.1]
    budget = b_min
    survivors = pipelines
    while True:
        # Train and score each survivor at the current budget  [§3.2.1]
        errors = [train_and_score(p, budget) for p in survivors]
        # Stop at one pipeline or at the largest budget  [§3.2.1]
        if len(survivors) == 1 or budget >= b_max:
            return survivors
        # Keep the best 1/eta of them; rounding is not stated  [§3.2.1]
        survivors = keep_best(survivors, errors, len(survivors) / eta)
        # Multiply the budget for the survivors by eta  [§3.2.1]
        budget = budget * eta

Step 5, BO with SH. Following the BO-with-SH method the paper cites as Falkner et al. §3.2.1, Bayesian optimization proposes pipelines that are evaluated at the lowest budget until a fixed number of them have been evaluated; then SH runs as above. The surrogate is fitted on the highest budget that has enough data points. The paper sets "enough" to n/2, where n is the number of hyperparameters. The original requirement was n+1 finished pipelines §3.2.1. The paper uses only SH and not Hyperband, because SH cuts the budget per pipeline more aggressively at very small time limits §3.2.1 (footnote).

Step 6, practical restrictions (PoSH). The search space is restricted to iterative algorithms, with no feature preprocessing. This leaves extremely randomized trees, random forests, histogram gradient boosting, a passive-aggressive linear model, SGD and an MLP §3.3 app. D.2. Because the algorithms are iterative, partial models are stored at 2, 4, 8, ... iterations, so that a prediction exists if a run times out or runs out of memory §3.3 app. B.1.

Part II: automated policy selection

Step 7, policies. Each policy combines a model selection strategy (3-fold, 5-fold, 10-fold cross-validation or holdout) with a budget allocation (SH or FB), and a portfolio built for that combination. There are eight policies §4 §4.1.

Step 8, offline training of the selector. For every meta-dataset, run every policy to obtain a realistic performance estimate §4.1. Compute meta-features §4.1.1. Train the selector on these estimates. To avoid optimistic estimates, an inner 5-fold cross-validation over the meta-datasets is used: each policy is built on four fifths of the meta-datasets and evaluated on the held-out fifth. The final system uses a portfolio built on all meta-datasets §4.1.

Step 9, selector. The main selector is pairwise. For each pair of policies (π_A, π_B), a random forest predicts whether π_A beats π_B from the meta-features §4.1.1. Each meta-observation is weighted by the loss difference between the two policies, after a per-dataset scaling so that errors are comparable across datasets §4.1.1. At prediction time, the pairwise models vote, and the resulting win counts are normalized to a distribution over the policies §4.1.1 policy_selectors.py. Ties are broken by a fixed order and then at random [code: experiment_scripts/autoauto/policy_selectors.py#L354-L371].

# Per-dataset policy selector, pairwise voting  [§4.1.1] [tab. 4]
# Input:  meta_features (2,) = (number of datapoints, number of features)
# Output: one score per policy, summing to 1

def select_policy(meta_features, pairwise_models):
    # Backup first: no meta-dataset dominates this dataset  [§4.1.1]
    if not dominated_by_some_meta_dataset(meta_features):
        return backup_scores  # paper: holdout with SH; see Paper vs code
    wins = zeros(num_policies)
    # One random forest per pair of policies: does policy a beat b?  [§4.1.1]
    for a, b in policy_pairs:
        # Soft voting adds the probability of a winning  [tab. 4]
        prob_a_wins = pairwise_models[a][b].predict_proba(meta_features)
        wins[a] += prob_a_wins
        wins[b] += 1 - prob_a_wins
    # Normalize the win counts into a distribution  [§4.1.1]
    return wins / sum(wins)

Step 10, backup. If no meta-dataset has at least as many datapoints and at least as many features as the new dataset, the selector is not trusted and the system falls back to holdout with SH, described as the most aggressive and cheapest policy §4.1.1.

Step 11, hyperparameters of the selector. Seven hyperparameters are tuned with Bayesian optimization, minimizing the cross-validation error on the meta-data §4.1.1 tab. 4. Five belong to the random forest, one sets the pairwise voting rule, and one sets how errors are scaled to weight datasets §4.1.1 tab. 4. A tree depth of 0 is described as a single leaf that equals the single-best policy §4.1.1. The same hyperparameters are shared across all pairwise models §4.1.1.

Step 12, runtime. For a new dataset, compute the two meta-features §4.1.1, pick a policy with the selector, run the AutoML system with that policy on the training split, and return the best pipeline. The test split is used only at the end §4.1.

Single-best baseline. Instead of a selector, one policy is chosen as the best on average over the meta-datasets (per-set algorithm selection). This is the automated version of picking SH and holdout by hand §4.1.1.

Implementation facts used by the experiments

The experiments use the Auto-sklearn package version 0.12.6 for the baseline and version 0.12.7 for Auto-sklearn 2.0, with SMAC3 extended for asynchronous parallel runs §5.1 §D.1 tab. 17. Each pipeline evaluation has a time limit of one tenth of the horizon and a memory limit of 4 GB §3.4. The experiments use ensembles of size 50 with replacement §3.4.3. Balanced error rate (one minus balanced accuracy) is the metric §3.4.

Hyperparameters

The table lists the values the paper states, with their source. Values that the paper does not state are marked "not stated" in the notes, and any code value is cited separately.

Selector search space tab. 4:

The code's seven scaling options are all, all1, binary, binary1, y, rank and None policy_selectors.py. The paper describes raw errors, min-max scaling over a pair or over all policies, and rank differences §4.1.1. The code's "1" variants fix the upper bound at 1 policy_selectors.py.

Configuration space of Auto-sklearn 2.0, for iterative models and preprocessing only tab. 18. The final column says whether the search is on log10 scale.

The paper states the Table 18 values as the search space of Auto-sklearn 2.0. The code's ranges are in the same file as the selector and are not reproduced here.

Results

Author-reported. Setup: 39 AutoML benchmark datasets (balanced error rate, normalized per dataset to a 0–1 regret scale, averaged over 10 repetitions), the two horizons, and a comparison with the earlier Auto-sklearn 1.0 and with the intermediate PoSH system §3.4 §3.5 §4.2.

Main result tab. 5. Mean normalized balanced error, lower is better. The values are ADTM averages with standard deviations.

The authors report a relative error reduction against Auto-sklearn 1.0 of 78% at 10 minutes, which they describe as a factor of 4.5, and 65% at 60 minutes §4.2. Several large datasets drive much of the gap, because Auto-sklearn 1.0 times out or hits its memory limit on them §4.2. Removing those datasets still leaves Auto-sklearn 2.0 ahead §4.2.

PoSH vs Auto-sklearn 1.0 tab. 3: PoSH 4.11 ± 0.09 and 3.19 ± 0.12, against Auto-sklearn 1.0 at 16.21 ± 0.27 and 7.17 ± 0.30 §3.5.2.

Portfolio results tab. 2. Normalized balanced error, with bold for the best per row. Italics mark values the paper underlines as not significantly different from the best (Wilcoxon, α = 0.05).

Selector ablation tab. 6. Normalized balanced error for the selector and its variants, trained on different meta-data (P = portfolio performance table, BO = Bayesian optimization, E = ensembles). The oracle is the best policy per dataset on the test set. Random chooses a policy at random §4.3.1. Italics mark values the paper underlines as not significantly different from the best.

Which model selection strategies to offer tab. 7. Selector, random and oracle, on subsets of the eight policies:

Warm starting with a portfolio tab. 8:

Comparison with other AutoML systems tab. 9. Running on 39 datasets with 1 hour, 8 cores, five repetitions. The metric is log loss on multiclass tasks and one minus AUC on binary tasks, and smaller is better. Bold marks the lowest error per dataset. The aggregate rows are the mean rank, best count, wins/losses against Auto-sklearn 2.0, and p-values of a binomial sign test against it. Columns: AS 2.0 = Auto-sklearn 2.0, AS 1.0 = Auto-sklearn 1.0, AW = Auto-WEKA, TPOT, H2O = H2O AutoML, TunedRF = tuned random forest.

Setup for the comparison §5.1: the AutoML benchmark's 1h8c setting, 10 folds, five repetitions, 32 GB of RAM per run, and Singularity images instead of Docker. Package versions are in tab. 17. The paper states that Auto-sklearn 2.0 has the lowest average rank and the most best-performance counts, and that both Auto-sklearn versions give results on all datasets, including the largest §5.2. The H2O p-value is .053 tab. 9, so that comparison is not significant at 0.05.

Auto-sklearn 2.0 without ensembles tab. 12 (appendix B.2): 10 min 5.01 ± 0.18 and 60 min 3.18 ± 0.31, against PoSH 5.76 and 3.97 and Auto-sklearn 1.0 23.24 and 8.68.

Iterative search space and results retrieval tab. 10 (appendix B.1): Auto-sklearn 1.0 at 16.21 and 7.17; with the iterative search space only, 18.10 and 9.57; with iterative space and intermittent results, 5.29 and 3.98; with the portfolio added, 3.70 and 3.08.

Per-dataset raw numbers are in tab. 13–tab. 16, and the meta-train dataset list is in tab. 19. They are not reproduced here. The without-ensembles portfolio table is tab. 11.

Paper vs code

Code: https://github.com/automl/ASKL2.0_experiments at commit 7373041248711052f7af853e946ce48692669800. Its readme says it is provided as-is and that the scripts will be ported to the main Auto-sklearn repository [code: readme.md]. It has no LICENSE file, so the code is cited by path and line only, with no quoted text. Its readme cites arXiv v2, and the paper text compiled here is v3 (JMLR 2022 format), so differences may partly reflect the version. The final Auto-sklearn 2.0 implementation is in the Auto-sklearn package (version 0.12.7 per §D.1). That package is not in this repository, and I did not check it. The behaviour below is taken from the experiment scripts.

• Greedy tie-breaking. The paper says ties go to the model trained first alg. 1. The code shuffles the candidates with the random generator and takes the first minimum in the shuffled order greedy_portfolio.py greedy_portfolio.py. The tie break is random, not by training order.
• Early stop in the greedy loop. The code stops adding members when the best score is at most 0 greedy_portfolio.py. The paper runs until the target size is reached alg. 1 §3.1.1. A portfolio built this way can be shorter than 32.
• Selection on test data. Portfolio construction uses the test loss when consider_validation is not set, which is the default greedy_portfolio.py greedy_portfolio.py. The build script does not set it build_portfolio.py. This matches the paper's choice §3.1.1.
• Successive-halving budgets. The budgets are fractions of the full budget, 6.25% to 100% with η = 4 build_portfolio.py. The bracket code works out to N_i = [16, 4, 1] and R_i = [6.25, 25, 100] in percent fidelity_strategies.py. This is the same 16× range as the paper's 32 to 512 and 64 to 1024 iterations app. D.3: 32/512 and 64/1024 are both 6.25%.
• Backup policy. The paper says the fallback is holdout with SH §4.1.1. The code returns a weight vector over all eight policies, not a single choice. The weights are proportional to (7, 6, ..., 0) for the eight policies in the order holdout+SH, holdout+FB, 3CV+SH, and so on [code: experiment_scripts/autoauto/policy_selectors.py#L480-L484] run_autoauto.py. Holdout+SH gets the largest share, 7/28 = 25%, but not all of it. The script scores outputs as a weighted regret run_autoauto.py, so whether the runtime picks an argmax is in the autoauto_experiments package, which is not in this repository.
• Depth-0 selector. The paper says depth 0 gives a single leaf with all data §4.1.1. The code instead fits a constant predictor for the pair, chosen from the summed weights, rather than a random forest with one leaf policy_selectors.py. Pairs where all labels are equal also get a constant predictor policy_selectors.py.
• Random forest size and selector search. The paper does not state the number of trees or the number of selector search iterations. The code uses 500 trees, or 10 in test mode run_autoauto.py run_autoauto.py, and 200 SMAC iterations, or 10 in test mode run_autoauto.py.
• Ensemble size. The paper says ensembles of size 50 §3.4.3. The ensemble step uses ensemble_size=50 run_autosklearn.py run_autosklearn.py. The main AutoML call passes ensemble_size=1 when post-hoc ensembles are on run_autosklearn.py run_autosklearn.py. How these two interact is inside the Auto-sklearn package, which I did not check.
• Selector hyperparameter ranges. The paper's Table 4 gives max features as [1, 2]. The code uses [1, n_features], where n_features is the number of meta-features, which is 2 policy_selectors.py run_autoauto.py. This matches. The code also searches min samples split and min samples leaf on a log scale policy_selectors.py, which Table 4 does not say.
• Time and memory limits. The code uses 60 s per pipeline for the 10-minute horizon and 4000 MB master_calls.py, which matches the one-tenth rule and 4 GB §3.4 within rounding.
• Portfolio size. The code builds portfolios of size 32 [code: experiment_scripts/master_calls.py#L748], consistent with §3.5.1.

No discrepancies found for the SH bracket beyond the units noted above.

Pitfalls

• Test-set selection is what makes the guarantees hold. The submodularity argument, and the 1 − 1/e and 99% bounds, apply when the portfolio is chosen on meta-dataset test losses §3.1.2 app. C.2. Choosing on the validation split breaks submodularity and monotonicity, shown with a counterexample for holdout app. C.3, and also for SH app. C.4. Eq. 9 is written with validation loss eq. 9, so implementing it literally changes the guarantees.
• Bounds from the greedy algorithm. The 63% and 99% statements are inherited from cited work §3.1.2. They describe the portfolio's loss on the meta-data, not on new datasets.
• Rounding in successive halving. The paper says "1/η of the pipelines" and does not say how to round or how many pipelines start §3.2.1. The code's bracket fixes n = 16 and N_i = floor(n η^{-i}) [code: experiment_scripts/portfolio/fidelity_strategies.py#L173-L186].
• BO and SH together have no guarantee. The authors say so §7 and ask for approaches that raise SH's smallest budget over time §7.
• Number of BO pipelines before SH. The paper says "a fixed number" without giving it §3.2.1.
• Surrogate data threshold. The paper uses n/2 points for the BO surrogate, instead of the n+1 the original model needs §3.2.1. Check this before you copy the rule into another system.
• Only two meta-features. The authors say the two-feature selector can be seen as learning the manual fallbacks from Part I §4.1.1. Other meta-features may do better, and the authors leave that open §7.
• Backup changes the result. Without the backup, the selector is worse on huge datasets it cannot extrapolate to, and on the 60-minute horizon the single best policy is better than the selector without backup §4.3.1. The backup is part of the reported result.
• Meta-data with ensembles is not always better. P+BO slightly beats P+BO+E for the selector. The authors trace this to one dataset where the selector picks a worse policy than the single best, and they suggest ensembles lower meta-data quality §4.3.1. The main results use P+BO+E tab. 5.
• Oracle is not zero. Normalization uses the single-best test loss, so the oracle can be above zero §4.3.1 (footnote).
• Meta-data cost. Building the selector's training data is expensive. The portfolio table is the cheap option, but it gives the worst selector §4.3.1.
• Tie-breaking in the selector. Ties are broken by a fixed order first, then at random policy_selectors.py. The paper does not describe the tie rule.
• Large datasets dominate the average. Several large datasets drive much of the gap to Auto-sklearn 1.0, because its KND warm start proposes pipelines that time out §4.2. Read the averages together with the per-dataset results.
• Ensembles. The paper uses ensemble selection with 50 members and reports results without ensembles in the appendix §3.4.3 app. B.2. It says the ensemble improves the error in every case app. B.2, so compare like with like.
• Benchmark significance. The H2O comparison has p = .053 tab. 9, which is not significant at 0.05.

Lineage

• Builds on: Auto-sklearn 1.0, with k-nearest-dataset warm starting and post-hoc ensembles §1 §3.1. The intermediate system, PoSH Auto-sklearn, is the authors' winning entry to the second AutoML challenge §3.3. Successive halving is from the works the paper cites in §3.2.
• Uses: SMAC for Bayesian optimization §3.2.1; HydraMIP's pairwise selector design §4.1.1; per-set algorithm selection §4.1.1.
• Compared against: Auto-WEKA, TPOT, H2O AutoML, a tuned random forest, and Auto-sklearn 1.0 §5 tab. 9.
• Superseded by: not stated in the paper, and not checked.