---
kind: concept
slug: automated-machine-learning
title: Automated machine learning
summary: >-
  Finding a model and its settings for a new dataset with little human design. Auto-sklearn 2.0 searches pipelines under a time limit, using a warm start, budget halving and a learned policy selector. CellForge has LLM agents design the model itself, bounded by discussion rounds and repair attempts instead of a time limit. Pick by time and compute budget, and by how much repeatability matters.
tags: [automl, meta-learning, hyperparameter-optimization, algorithm-selection, llm-agents]
papers: ["2007.04074", "2508.02276"]
related: [algorithm-selection, hyperparameter-optimization, bayesian-optimization, meta-learning, successive-halving, llm-agent, multi-agent-systems]
status: draft
updated_at: 2026-11-01
---

## Overview

An AutoML system picks a model and its settings for one dataset with little
human design. This page covers two systems that do it in different ways.

Auto-sklearn 2.0 searches over pipelines (a model plus its settings) under a
time budget T. The chosen pipeline must come from evaluations whose times add
up to less than T [arxiv:2007.04074 eq. 5]. It answers the design questions
(how to estimate error, how to split the budget, where to start, which
algorithms to search) with rules learned offline, plus a selector that picks one
of eight search policies for each new dataset [arxiv:2007.04074 §1]
[arxiv:2007.04074 §2.1].

CellForge has no time budget. A team of LLM agents formulates the task, debates
candidate model designs in a panel and then writes and trains the model
[arxiv:2508.02276 § Methods]. Its limits are the number of discussion rounds and
the number of code repair attempts.

The first five variants below are parts of Auto-sklearn 2.0. The last one is
CellForge. Other systems and papers are not covered yet.

## Variants

### Warm start from a portfolio

Before searching, the system tries a fixed list of pipelines built offline. The
list is chosen greedily: at each step, add the candidate that most reduces the
summed error over the meta-datasets, until the list has 32 members
[arxiv:2007.04074 alg. 1] [arxiv:2007.04074 §3.5.1]. The portfolio is evaluated
one member at a time before Bayesian optimization takes over
[arxiv:2007.04074 §3.1] [arxiv:2007.04074 §3.2.1]. Card: [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](/papers/2007.04074).

It replaces the older warm start, which used k-nearest datasets and needed
meta-features. That older method could propose pipelines that time out on large
datasets [arxiv:2007.04074 §3.1] [arxiv:2007.04074 §3.5.1].

### Successive halving for budget

Successive halving starts many pipelines at the smallest budget. It keeps the
best 1/η, multiplies their budget by η, and repeats until one pipeline is left
or the largest budget is spent. The paper uses η = 4 [arxiv:2007.04074 app. D.3].
The alternative is full budget: every pipeline gets the largest budget
[arxiv:2007.04074 §3.2.1]. Hyperband was not used; the authors chose
successive halving because it cuts the budget more aggressively when the time
limit is very small [arxiv:2007.04074 §3.2.1].

### Restricted search space

The search is limited to iterative algorithms without feature preprocessing: two
forest models, histogram gradient boosting, a passive-aggressive model, SGD and
an MLP. Partial models are stored at doubling iteration counts, so a prediction
exists if a run times out [arxiv:2007.04074 §3.3] [arxiv:2007.04074 app. B.1].
The restriction alone did not help at 10 minutes: the error rose from 16.21 to
18.10, and the gains came from storing intermediate results and from the
portfolio [arxiv:2007.04074 tab. 10].

### Model selection strategy

Each policy pairs one model selection strategy (holdout, 3-, 5- or 10-fold
cross-validation) with one budget rule (successive halving or full budget). That
gives eight policies [arxiv:2007.04074 §4] [arxiv:2007.04074 §4.1].

### Per-dataset policy selector with backup

For each new dataset, a selector scores the eight policies from two
meta-features: the number of datapoints and the number of features. It is a set
of pairwise random forests whose votes are normalized into a distribution over
policies [arxiv:2007.04074 §4.1.1]. If no meta-dataset has at least as many
datapoints and at least as many features, the selector is not used, and the
system falls back to holdout with successive halving [arxiv:2007.04074 §4.1.1].
Its seven hyperparameters are tuned by Bayesian optimization on the
meta-data [arxiv:2007.04074 §4.1.1] [arxiv:2007.04074 tab. 4].

### Ensembling

The main results combine the searched pipelines into an ensemble of 50 members
with replacement [arxiv:2007.04074 §3.4.3]. Without ensembles, the authors report
5.01 at 10 minutes and 3.18 at 60 minutes, and they say the ensemble improved
the error in every case [arxiv:2007.04074 app. B.2].

### Agents design the model per dataset

CellForge does not search a fixed algorithm list. Its pipeline has three
stages: formulation, ideation and execution [arxiv:2508.02276 § Methods]. In
ideation, expert agents that share one LLM backend write plans. Peers review
each other's plans, a critic writes a meta-review with a score, and each
expert's confidence is updated with weights 0.3, 0.4 and 0.3
[arxiv:2508.02276 eq. 1]. The discussion stops when every confidence is at least
0.8 and the gap between them is below 0.03, or after 20 rounds
[arxiv:2508.02276 § Discussion protocol]. The critic then selects one plan.
Execution writes training code and gets up to five repair attempts
[arxiv:2508.02276 § Code generation and repair]. Card: [AI Scientists for Building Virtual-Cell Models](/papers/2508.02276).

The confidence score is not a performance measure. The paper says it is neither
a calibrated probability nor a measure of how well the model predicts
[arxiv:2508.02276 § Critic-mediated expert panel]. The reported results are on
held-out perturbations: PCC 0.9883 on Adamson CRISPRi with CPA-X, and MSE_DE
0.1736 on Norman CRISPRa with scGen-X [arxiv:2508.02276 § Discussion]. The
baselines appear only in figures [arxiv:2508.02276 fig. 3] [arxiv:2508.02276 fig. 4].

## When to use which

The first rows rest on the Auto-sklearn 2.0 ablations. The last rows rest on
the CellForge setup and reported costs. They are not general rules.

| Situation | Use | Why |
|---|---|---|
| The time limit is fixed and the search rules should be fixed in advance | Auto-sklearn 2.0-style search | Its budget rule, portfolio and selector are set offline, and results are reported at 10 and 60 minutes [arxiv:2007.04074 tab. 8] |
| You can build a portfolio from past results on other datasets | Warm start from it | With it, the selector scores 3.58 and 2.47; without it, 5.63 and 3.42 [arxiv:2007.04074 tab. 8] |
| You can only build part of the system | Intermittent results first, then the portfolio | Error at 10 min went 16.21 → 5.29 → 3.70 as those parts were added [arxiv:2007.04074 tab. 10] |
| The new dataset is larger than every meta-dataset | Backup policy, not the selector | Without the backup, the selector scores 4.79 and 5.43; with it, 3.58 and 2.47 [arxiv:2007.04074 tab. 6] |
| Building a selector is too costly | A single best policy, chosen on average over meta-datasets | At 60 min, the single best scores 2.44 against 2.47 for the selector [arxiv:2007.04074 tab. 8]. At 10 min the selector is ahead, 3.58 against 3.69 [arxiv:2007.04074 tab. 6]. The selector's training data is the expensive part [arxiv:2007.04074 §4.3.1] |
| Holdout is the only validation you want to offer | Still offer all four strategies to the selector | A holdout-only selector scores 4.03 and 3.18, against 3.58 and 2.47 with all eight policies [arxiv:2007.04074 tab. 7] |
| Picking the budget rule | Compare both per regime | No single rule wins across validation strategies and horizons [arxiv:2007.04074 tab. 2] |
| No fixed model family suits the task and you can pay for LLM calls and hours of GPU time | Agent design (CellForge) | Per run the paper's table lists 40,000 to 80,000 prompt tokens and 200,000 to 400,000 completion tokens [arxiv:2508.02276 tab. 2]. Wall-clock is roughly 4 to 8 hours of GPU time [arxiv:2508.02276 supp. Cost interpretation] |
| You need a design you can rerun and trust | Neither as a single run | The authors used three runs per dataset and say that is too few to describe the spread of designs [arxiv:2508.02276 § Discussion] |

## Implementation sketch

The first two blocks are the common core of Auto-sklearn 2.0. The offline part
builds the portfolio. The online part runs once per new dataset. The third block
is CellForge's discussion loop. It is a different kind of search and shares no
code with the first two.

```pseudocode
# Offline: greedy portfolio over meta-datasets  [arxiv:2007.04074 alg. 1]
# Input:  candidates, error table over meta-datasets, target size
# Output: portfolio, a list of pipelines

def build_portfolio(candidates, error_table, size):
    portfolio = []
    while len(portfolio) < size:
        # Add the candidate that most reduces the summed error  [alg. 1]
        best = min(candidates - set(portfolio),
                   key=lambda c: summed_error(portfolio + [c], error_table))
        portfolio.append(best)
        # Stop only at the target size  [§3.1.1]
    return portfolio
```

```pseudocode
# Online: one run on a new dataset  [arxiv:2007.04074 §4.1] [§3.1] [§3.2.1]

def run_automl(train, budget, meta_data, portfolio):
    # Two meta-features: datapoints and features  [§4.1.1]
    meta = (num_datapoints(train), num_features(train))
    # Pick a policy; back up to holdout + SH if not covered  [§4.1.1]
    policy = select_policy(meta, meta_data)
    # Warm start: evaluate portfolio members first  [§3.1]
    history = [evaluate(p, policy) for p in portfolio]
    # Then Bayesian optimization proposes pipelines  [§3.2.1]
    # The policy decides the budget rule: SH or full budget  [§3.2.1]
    history = bayes_opt(history, policy, budget)
    # Test split is used only for the final score  [§4.1]
    return best_of(history)
```

```pseudocode
# Agent design: discussion over model plans  [arxiv:2508.02276 § Discussion protocol]
# Stops when all confidences reach 0.8 with a gap below 0.03, or after 20 rounds

def design_model(experts, report, evidence):
    # Every expert starts at confidence 0.5  [§ Discussion protocol]
    conf = {e: 0.5 for e in experts}
    for _ in range(20):  # T_max = 20 rounds  [§ Discussion protocol]
        # Each expert writes a plan  [§ Discussion protocol, step 1]
        plans = {e: e.write_plan(report, evidence) for e in experts}
        # Peers score each other's plans  [§ Discussion protocol, step 2]
        peer = {(i, j): j.review(plans[i]) for i in experts
                for j in experts if j != i}
        # Critic writes a meta-review and a score per plan  [step 3]
        critic = {i: critic_review(plans[i], peer) for i in experts}
        # Confidence weights 0.3, 0.4, 0.3  [arxiv:2508.02276 eq. 1]
        for i in experts:
            mean_peer = mean(peer[i, j] for j in experts if j != i)
            conf[i] = (0.3 * conf[i] + 0.4 * critic[i]
                       + 0.3 * mean_peer)
        low, high = min(conf.values()), max(conf.values())
        if low >= 0.8 and high - low < 0.03:  # convergence test
            break
    # The critic picks one final plan  [§ Discussion protocol]
    return critic_select_final(plans, conf)
```

## Contradictions

None on the same claim. The two papers measure different things on different
tasks, so their numbers are not comparable. Auto-sklearn 2.0 reports normalized
balanced error [arxiv:2007.04074 tab. 8]. CellForge reports PCC and MSE for
held-out perturbations [arxiv:2508.02276 § Discussion]. Inside the Auto-sklearn
2.0 paper, the portfolio objective is written with validation loss
[arxiv:2007.04074 eq. 9], but the greedy construction and its guarantees use the
test split of the meta-datasets [arxiv:2007.04074 §3.1.1] [arxiv:2007.04074
§3.1.2]. The paper says this is allowed as long as the new datasets are kept
separate.

## Papers

- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](/papers/2007.04074): a portfolio warm start, successive halving and a learned per-dataset policy selector, with a pipeline search restricted to iterative models; author-reported results at 10 and 60 minutes on 39 datasets.
- [AI Scientists for Building Virtual-Cell Models](/papers/2508.02276): an LLM agent team that debates and then implements a perturbation-prediction model per dataset; author-reported results on held-out perturbations, with most baselines only in figures.