Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

In plain words

Training a small language model involves many settings, such as how deep the network is, how big each batch is, and how fast the weights learn. Finding good values is called hyperparameter optimization. The usual tools are mathematical search methods that try a setting, measure the result, and use that history to pick the next try. A newer option is to hand the job to a language-model agent that reads the history and writes its own guesses, or even edits the training code directly.

This paper runs a fair race between both kinds on one small training task, with the same time budget for every method. In a fixed list of settings, the classical search methods did better than the language-model agents. The agents did better when they were allowed to rewrite the code, but they still did not beat the best classical method. A common failure was running out of GPU memory, and the agents were poor at remembering which regions had caused that.

The authors then built a hybrid. It keeps one classical search method (CMA-ES) running, and occasionally lets a language model look at that method's internal state and override its suggestion. The hybrid gave the best average result in their runs. Their conclusion is that language models are most useful as a helper to a classical optimizer, not as its replacement.

TL;DR

• Nine hyperparameter optimizers (four classical, four LLM-based, one hybrid) are compared on one small language-model training task, with equal 24-hour GPU budgets and 3 seeds §1 tab. 2.
• In a fixed 14-hyperparameter search space, classical methods beat pure LLM agents; LLM agents that edit training code get closer but do not overtake the best classical method §5.1 §5.2.
• Centaur shares CMA-ES's mean, step size and covariance with an LLM and lets the LLM override the proposal on 30% of trials §4; its best mean is 0.9739 val_bpb with Claude Opus 4.6 and 0.9763 with Qwen3.5-27B tab. 3.
• All numbers are author-reported on one task with three seeds. Gaps among the top methods are often smaller than the seed spread tab. 3.

Problem

Hyperparameter optimization (HPO) is a central part of AutoML, and it covers settings from architecture to optimizer and data augmentation §1. The paper asks two questions: how do classical HPO methods perform on the autoresearch task, and can LLM-based HPO methods beat them §1. The task is fixed-budget: each trial runs for five minutes on one GPU, and the score is validation bits-per-byte, where lower is better §3. Out-of-memory (OOM) failures are a major cost: the paper treats a crashed trial as a failure scored 100.0, and OOM rates separate the strong methods from the weak ones §3 §5.1.

Earlier LLM-for-HPO methods use the LLM as a surrogate model, as a direct suggester, or as a tuner of CMA-ES's own settings. None shares the classical optimizer's full internal state with the LLM §2.

Method

Benchmark. The optimizee is a nanochat-style decoder-only transformer trained on FineWeb, scored by validation bits-per-byte (val_bpb). Each trial runs for five minutes on one H200 GPU with training VRAM capped at 80 GB §3. Each method runs for 24 hours of cumulative training wall-time, excluding LLM inference, with three seeds §3 §5. A failed trial (out of memory) is recorded as val_bpb = 100.0 §3.

Search space. Fourteen hyperparameters are extracted from the training script train.py by parsing it: every top-level assignment with an ALL_CAPS name and a literal value is a candidate. The ranges are set by hand, and the defaults are the paper's starting configuration §3 tab. 1. The extractor and range table are in search_space.py.

Methods compared. Fixed-space methods: TPE, CMA-ES, SMAC, Random, two LLAMBO variants, and an LLM agent that suggests values for the 14 hyperparameters. Unconstrained: an LLM agent that rewrites train.py as a full replacement. Hybrid: Centaur tab. 2.

Centaur. Centaur is a hybrid of CMA-ES and an LLM. Each trial, CMA-ES proposes a candidate from its multivariate Gaussian, defined by mean vector μ, step size σ and covariance matrix C §4. On about 30% of trials, the LLM is shown the proposal together with μ, σ, C, the five best configurations so far and the last 20 trials §4. The LLM may override the proposal. It did so on 100% of these trials with the 27B model and 95% with the 0.8B model §4 §5.3. CMA-ES updates from every result, including overridden ones §4 alg. 1. The paper has no numbered equations; the update rules are standard CMA-ES, delegated to Optuna in the code centaur_backend.py.

The LLM receives the search space, a CMA-ES analysis block, the last 20 evaluations and the current incumbent, and must answer with a JSON object that maps each HP name to a value app. A.11. The prompt text is reproduced in the appendix app. A.11.

# Centaur loop  [alg. 1] [§4]
# Input: search space S, trials T, LLM ratio r (0.3)
# Output: configuration with the lowest val_bpb

def centaur(space, trials, ratio):
    # Start CMA-ES on the search space; history begins empty  [alg. 1]
    optimizer = cma_es(space)
    history = []
    for trial in range(trials):
        # CMA-ES always proposes a candidate from its Gaussian  [§4]
        proposal = optimizer.ask()
        # Paper: LLM turn with probability r. Code: fixed schedule.
        # See "Paper vs code"  [alg. 1] [code: centaur_backend.py#L222-L228]
        if llm_turn(trial, ratio):
            # LLM sees mu, sigma, C, top-5 configs and last 20 trials  [§4]
            config = llm_suggest(optimizer.mean, optimizer.sigma,
                                 optimizer.cov, proposal, history)
        else:
            config = proposal
        # Train for the fixed budget; a failed run is scored 100.0  [§3]
        value = evaluate(config)
        # CMA-ES learns from every trial, overridden or not  [alg. 1] [§4]
        optimizer.tell(config, value)
        history.append((config, value))
    return best(history)

Baseline. All methods start from the same baseline, Karpathy's starting configuration (the defaults in tab. 1), with val_bpb ≈ 0.991 §5. In the code, the baseline is trial 0 and is also seeded into Centaur runner.py.

Search-space details (CMA-ES). CMA-ES handles the 13 continuous hyperparameters; the categorical WINDOW_PATTERN is excluded from the covariance analysis app. A.7 centaur_backend.py.

Hyperparameters

Search space, auto-extracted and bounded by hand §3 tab. 1:

The ranges and log flags match the code's metadata table search_space.py, and the defaults match its starting configuration search_space.py.

Run and method settings:

Results

Author-reported. Setup: nanochat-style transformer on FineWeb, H200 GPU, 5-minute trials, 24-hour budget per method, 3 seeds, Karpathy's starting configuration as baseline §3 §5. Values are mean best val_bpb over 3 seeds, lower is better. The full table, including OOM rates, is tab. 3.

Headline claims, as stated: classical methods beat pure LLM agents in the fixed space; the best result was Centaur with Claude Opus 4.6, and Centaur with a 0.8B model already beat all classical and pure-LLM methods §1 §5.1 §5.3. The paper also reports that pure-LLM code editing reaches 0.9770 with Opus 4.6 but still does not beat TPE's 0.9768 §5.2.

Scaling: the 0.8B model is insufficient for code editing, but scale does not help the fixed-space LLM agent (0.9904 at 27B versus 0.9908 at 0.8B) §5.2 tab. 5. Ratio ablation (2 seeds, default with 3): the 0.8B model is best at r = 0.1 (0.9744), and the 27B model is best at r = 0.5 (0.9746); r = 0.8 degrades both app. A.4 tab. 4.

Diversity: CMA-ES and TPE keep OOM rates at 16% and 11%, against 56% for random search and 48–61% for the LLAMBO variants §5.1 tab. 3.

Paper vs code

Code: official repository ferreirafabio/autoresearch-automl (MIT, owned by the first author) at commit 150cd1418b7bd65c3821a844a91598110d3b36b8. The abstract and the appendix link it app. A.11.

• Mean μ is not in the LLM prompt. The paper says the LLM receives μ, σ, C, top-5 configs and last 20 trials §4, and the appendix paragraph lists "mean" among the CMA-ES state app. A.11. The code extracts the mean centaur_backend.py and stores it centaur_backend.py, but the prompt formatter prints σ, CMA-ES's next proposal, top-5 configs and the covariance matrix, and never the mean centaur_backend.py.
• Covariance is shown in full, not as a summary. The appendix says "covariance summary" app. A.11; the code prints the full matrix with rows labelled by alphabetically sorted continuous HP names centaur_backend.py.
• LLM trials are deterministic, not random. Algorithm 1 draws an LLM turn with probability r alg. 1. The code uses a fixed schedule: after warmup, an LLM trial whenever the trial count is divisible by round(1/r) centaur_backend.py. For r = 0.3 that is every third trial (about 33%, not 30%). Ratios 0.5 and 0.8 become every second and every trial respectively (round(1/0.8) = 1).
• Warmup is not in the paper. The code runs 10 counted trials before the first LLM trial, and the seeded baseline counts as one centaur_backend.py centaur_backend.py centaur_backend.py.
• Output token limit. The paper says fixed-HP methods use 2048 tokens and code editing 16384 §3. Centaur is a fixed-space method, but its code requests 16384 tokens, with a comment that Gemini counts thinking tokens against the limit centaur_backend.py.
• Snapping to powers of two is applied to all backends and is not in the paper. The runner rounds HEAD_DIM, DEVICE_BATCH_SIZE and TOTAL_BATCH_SIZE to powers of two and enforces TOTAL ≥ DEVICE × 2048 before every non-source-override trial runner.py search_space.py. The appendix credits the LLM with "hardware-friendly rounding" (61 → 64, 133143 → 131072) app. A.6; the same rounding applies to CMA-ES's own suggestions.
• CMA-ES is told about the unsnapped configuration. The study's stored parameters are set at ask time (or replaced by the LLM's config), and tell passes only the objective value centaur_backend.py centaur_backend.py. Snapping happens afterwards in the runner, so the configuration CMA-ES records can differ from the one evaluated.
• Missing LLM keys take defaults, and invalid categoricals are not checked. Missing HPs are filled with defaults centaur_backend.py. Numeric values are clamped to bounds, but categorical values are only skipped in the study update when invalid centaur_backend.py centaur_backend.py. The paper does not describe either behaviour.
• Failures are penalized, not only OOM. The paper calls the 100.0 score an OOM penalty §3. The code applies it to any failed run, including timeouts and assertion errors; OOM is only a label on the error objective.py centaur_backend.py.
• The 14 HPs are a hard-coded list. The paper describes automatic extraction from every ALL_CAPS literal §3. The code builds the space only from the 14 entries in its metadata table; an extra literal without metadata gets a range inferred from its default search_space.py search_space.py.
• Dataset name. Section 3 says the model is trained on FineWeb §3; the appendix prompts and the code's prompt say climbmix-400b-shuffle app. A.11 centaur_backend.py.
• Not verified here. train.py and prepare.py come from the autoresearch git submodule .gitmodules, which the shallow clone did not fetch. The training code and data were therefore not checked.

No discrepancy was found between the Centaur prompt in the appendix and the code's SUGGEST_PROMPT template app. A.11 centaur_backend.py.

Pitfalls

• Read the ratio labels as realised schedules. Under the code's schedule, r = 0.8 means every post-warmup trial is an LLM trial, and r = 0.5 means every second trial. The ablation's "higher ratios degrade" result therefore compares schedules, not the stated fractions §5.3 tab. 4.
• Top margins are small. Centaur (0.8B) is 0.9766 against TPE's 0.9768, a gap of 0.0002, while TPE's std is 0.0019 over 3 seeds tab. 3. "Outperforms all classical and pure LLM methods" §1 rests on means within the seed spread. The paper calls the gap to the best LLAMBO variant "significant" with no test reported §5.1. The repository includes scripts/wilcoxon_tests.py, but the paper does not report its results wilcoxon_tests.py.
• Internal contradiction on 0.8B versus 27B. Section 5.3 says Centaur with 0.8B outperformed Centaur with 27B §5.3. Table 3 gives 0.9766 for 0.8B and 0.9763 for 27B, where lower is better tab. 3. The 27B model is marginally better. The paper also reports a plateau for open-weight models.
• Cost is not reported. LLM inference time is excluded from the 24-hour training budget §5, and the paper gives no inference cost, latency or token totals for any LLM method §5.3. A Centaur run takes longer in wall-clock time than its training budget suggests.
• Frontier results rest on one prompt. Each frontier result uses one prompt template and 3 seeds. The paper attributes the Opus gain to better suggestions rather than OOM avoidance, since its OOM rate is similar to Qwen's §5.3 tab. 3, and it reports no prompt-sensitivity analysis.
• Standard deviation is not defined. The paper reports mean ± std over 3 seeds tab. 3 and does not say whether it is sample or population std.
• Covariance row order. The code labels rows of C alphabetically centaur_backend.py. Whether Optuna's internal order matches was not checked. If it does not, the LLM sees the matrix with wrong labels.
• OOM detection is by text. The code classifies OOM by searching stderr for "OutOfMemoryError" or "out of memory" objective.py. A crash with other wording is recorded under another error label, and the paper's OOM rates depend on this match.
• Optuna settings are not in the paper. sigma0, population size and the independent sampler for the categorical are not stated. The code leaves Optuna's defaults in place centaur_backend.py.
• Reproduce with the submodule. Running the benchmark needs the autoresearch submodule and a GPU, so the 24-hour budgets cannot be checked from this clone.

Lineage

• Builds on: CMA-ES, the evolution-strategy method cited in §2 (no arXiv id in this wiki); Karpathy's autoresearch repository, the task itself (github.com/karpathy/autoresearch, linked in the title) §1.
• Compared against: TPE, SMAC and random search, which are classical HPO methods §2 (not cards in this wiki); LLAMBO [liu2024llambo], SLLMBO, LLaMA-ES and HOLLM §2 (not cards in this wiki); AlphaEvolve and ShinkaEvolve, which the paper names as broader agentic search methods §2.
• Superseded by: not known from content already in this wiki.