Automated machine learning

Overview

An AutoML system picks a model and its settings for one dataset with little human design. This page covers two systems that do it in different ways.

Auto-sklearn 2.0 searches over pipelines (a model plus its settings) under a time budget T. The chosen pipeline must come from evaluations whose times add up to less than T 2007.04074 eq. 5. It answers the design questions (how to estimate error, how to split the budget, where to start, which algorithms to search) with rules learned offline, plus a selector that picks one of eight search policies for each new dataset 2007.04074 §1 2007.04074 §2.1.

CellForge has no time budget. A team of LLM agents formulates the task, debates candidate model designs in a panel and then writes and trains the model 2508.02276 § Methods. Its limits are the number of discussion rounds and the number of code repair attempts.

The first five variants below are parts of Auto-sklearn 2.0. The last one is CellForge. Other systems and papers are not covered yet.

Variants

Warm start from a portfolio

Before searching, the system tries a fixed list of pipelines built offline. The list is chosen greedily: at each step, add the candidate that most reduces the summed error over the meta-datasets, until the list has 32 members 2007.04074 alg. 1 2007.04074 §3.5.1. The portfolio is evaluated one member at a time before Bayesian optimization takes over 2007.04074 §3.1 2007.04074 §3.2.1. Card: Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning.

It replaces the older warm start, which used k-nearest datasets and needed meta-features. That older method could propose pipelines that time out on large datasets 2007.04074 §3.1 2007.04074 §3.5.1.

Successive halving for budget

Successive halving starts many pipelines at the smallest budget. It keeps the best 1/η, multiplies their budget by η, and repeats until one pipeline is left or the largest budget is spent. The paper uses η = 4 2007.04074 app. D.3. The alternative is full budget: every pipeline gets the largest budget 2007.04074 §3.2.1. Hyperband was not used; the authors chose successive halving because it cuts the budget more aggressively when the time limit is very small 2007.04074 §3.2.1.

Restricted search space

The search is limited to iterative algorithms without feature preprocessing: two forest models, histogram gradient boosting, a passive-aggressive model, SGD and an MLP. Partial models are stored at doubling iteration counts, so a prediction exists if a run times out 2007.04074 §3.3 2007.04074 app. B.1. The restriction alone did not help at 10 minutes: the error rose from 16.21 to 18.10, and the gains came from storing intermediate results and from the portfolio 2007.04074 tab. 10.

Model selection strategy

Each policy pairs one model selection strategy (holdout, 3-, 5- or 10-fold cross-validation) with one budget rule (successive halving or full budget). That gives eight policies 2007.04074 §4 2007.04074 §4.1.

Per-dataset policy selector with backup

For each new dataset, a selector scores the eight policies from two meta-features: the number of datapoints and the number of features. It is a set of pairwise random forests whose votes are normalized into a distribution over policies 2007.04074 §4.1.1. If no meta-dataset has at least as many datapoints and at least as many features, the selector is not used, and the system falls back to holdout with successive halving 2007.04074 §4.1.1. Its seven hyperparameters are tuned by Bayesian optimization on the meta-data 2007.04074 §4.1.1 2007.04074 tab. 4.

Ensembling

The main results combine the searched pipelines into an ensemble of 50 members with replacement 2007.04074 §3.4.3. Without ensembles, the authors report 5.01 at 10 minutes and 3.18 at 60 minutes, and they say the ensemble improved the error in every case 2007.04074 app. B.2.

Agents design the model per dataset

CellForge does not search a fixed algorithm list. Its pipeline has three stages: formulation, ideation and execution 2508.02276 § Methods. In ideation, expert agents that share one LLM backend write plans. Peers review each other's plans, a critic writes a meta-review with a score, and each expert's confidence is updated with weights 0.3, 0.4 and 0.3 2508.02276 eq. 1. The discussion stops when every confidence is at least 0.8 and the gap between them is below 0.03, or after 20 rounds 2508.02276 § Discussion protocol. The critic then selects one plan. Execution writes training code and gets up to five repair attempts 2508.02276 § Code generation and repair. Card: AI Scientists for Building Virtual-Cell Models.

The confidence score is not a performance measure. The paper says it is neither a calibrated probability nor a measure of how well the model predicts 2508.02276 § Critic-mediated expert panel. The reported results are on held-out perturbations: PCC 0.9883 on Adamson CRISPRi with CPA-X, and MSE_DE 0.1736 on Norman CRISPRa with scGen-X 2508.02276 § Discussion. The baselines appear only in figures 2508.02276 fig. 3 2508.02276 fig. 4.

When to use which

The first rows rest on the Auto-sklearn 2.0 ablations. The last rows rest on the CellForge setup and reported costs. They are not general rules.

Implementation sketch

The first two blocks are the common core of Auto-sklearn 2.0. The offline part builds the portfolio. The online part runs once per new dataset. The third block is CellForge's discussion loop. It is a different kind of search and shares no code with the first two.

# Offline: greedy portfolio over meta-datasets  [arxiv:2007.04074 alg. 1]
# Input:  candidates, error table over meta-datasets, target size
# Output: portfolio, a list of pipelines

def build_portfolio(candidates, error_table, size):
    portfolio = []
    while len(portfolio) < size:
        # Add the candidate that most reduces the summed error  [alg. 1]
        best = min(candidates - set(portfolio),
                   key=lambda c: summed_error(portfolio + [c], error_table))
        portfolio.append(best)
        # Stop only at the target size  [§3.1.1]
    return portfolio

# Online: one run on a new dataset  [arxiv:2007.04074 §4.1] [§3.1] [§3.2.1]

def run_automl(train, budget, meta_data, portfolio):
    # Two meta-features: datapoints and features  [§4.1.1]
    meta = (num_datapoints(train), num_features(train))
    # Pick a policy; back up to holdout + SH if not covered  [§4.1.1]
    policy = select_policy(meta, meta_data)
    # Warm start: evaluate portfolio members first  [§3.1]
    history = [evaluate(p, policy) for p in portfolio]
    # Then Bayesian optimization proposes pipelines  [§3.2.1]
    # The policy decides the budget rule: SH or full budget  [§3.2.1]
    history = bayes_opt(history, policy, budget)
    # Test split is used only for the final score  [§4.1]
    return best_of(history)

# Agent design: discussion over model plans  [arxiv:2508.02276 § Discussion protocol]
# Stops when all confidences reach 0.8 with a gap below 0.03, or after 20 rounds

def design_model(experts, report, evidence):
    # Every expert starts at confidence 0.5  [§ Discussion protocol]
    conf = {e: 0.5 for e in experts}
    for _ in range(20):  # T_max = 20 rounds  [§ Discussion protocol]
        # Each expert writes a plan  [§ Discussion protocol, step 1]
        plans = {e: e.write_plan(report, evidence) for e in experts}
        # Peers score each other's plans  [§ Discussion protocol, step 2]
        peer = {(i, j): j.review(plans[i]) for i in experts
                for j in experts if j != i}
        # Critic writes a meta-review and a score per plan  [step 3]
        critic = {i: critic_review(plans[i], peer) for i in experts}
        # Confidence weights 0.3, 0.4, 0.3  [arxiv:2508.02276 eq. 1]
        for i in experts:
            mean_peer = mean(peer[i, j] for j in experts if j != i)
            conf[i] = (0.3 * conf[i] + 0.4 * critic[i]
                       + 0.3 * mean_peer)
        low, high = min(conf.values()), max(conf.values())
        if low >= 0.8 and high - low < 0.03:  # convergence test
            break
    # The critic picks one final plan  [§ Discussion protocol]
    return critic_select_final(plans, conf)

Contradictions

None on the same claim. The two papers measure different things on different tasks, so their numbers are not comparable. Auto-sklearn 2.0 reports normalized balanced error 2007.04074 tab. 8. CellForge reports PCC and MSE for held-out perturbations 2508.02276 § Discussion. Inside the Auto-sklearn 2.0 paper, the portfolio objective is written with validation loss 2007.04074 eq. 9, but the greedy construction and its guarantees use the test split of the meta-datasets 2007.04074 §3.1.1 [arxiv:2007.04074 §3.1.2]. The paper says this is allowed as long as the new datasets are kept separate.

Papers

• Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning: a portfolio warm start, successive halving and a learned per-dataset policy selector, with a pipeline search restricted to iterative models; author-reported results at 10 and 60 minutes on 39 datasets.
• AI Scientists for Building Virtual-Cell Models: an LLM agent team that debates and then implements a perturbation-prediction model per dataset; author-reported results on held-out perturbations, with most baselines only in figures.