LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics

In plain words

Designing a new search method for hard optimisation problems, such as an algorithm that looks for the lowest point of an unknown bumpy landscape, is usually done by hand. Researchers tweak known methods and publish variants. LLaMEA asks a language model to write such a method as Python code, tests the code on a standard set of benchmark landscapes, and sends the score and the code back to the model. The model then edits the best code it has so far, or rewrites it, and the loop repeats for a fixed number of rounds.

The paper's point is that this loop can produce a method that beats a strong hand-tuned baseline on the benchmark it was shaped for. The result holds for one dimensionality only, and the reported gains come from the authors' own runs. The generated methods often look like known evolutionary methods with extra parts added.

TL;DR

LLaMEA is a loop in which a language model writes a Python optimiser, the optimiser is run on 24 noiseless BBOB test functions in 5 dimensions, and the mean anytime score (AOCC) is fed back to the model with the code. The best configuration (GPT-4, (1+1) selection) produced ERADS, a differential-evolution-like method with AUC 0.733 against 0.742 for an optimised CMA-ES and 0.703 for default CMA-ES tab. I. Generalisation to 10-D and 20-D is poor §VI-B. Use it if you want to see the design and evaluation loop; treat the numbers as author-reported on a single benchmark setting.

Problem

Many black-box optimisers are built by hand, and modular design spaces still need experts to choose the modules §I. The paper targets the continuous, box-constrained problem of minimising a function over a box eq. 1. It asks whether an LLM can generate such optimisers from a text description, with an automatic quality measure from benchmark runs, and whether the generated methods can match or beat state-of-the-art baselines §I §IV-C. It also notes that evaluation is costly: each candidate needs many benchmark runs, and the paper reports 4,500 commercial LLM calls in total §V.

Method

LLaMEA is a one-parent evolution strategy. Each iteration creates one offspring by asking the LLM to change the selected algorithm, runs it on the benchmark, and keeps the better of parent and offspring. The paper also describes an unnamed population variant in its outlook §VII (not implemented in the experiments).

Initialisation §III-A §III-B alg. 1 l.1-6. The first prompt gives the model an expert role, a description of the BBOB task with box constraints, a required class interface, a simple random-search example, and a response format with a name and code. The code and name are extracted with regular expressions; in the experiments 100% of responses parsed §III-B. If a response cannot be parsed, the format is asked for again in the next prompt. Code that fails to load or throws at run time gets the lowest score, zero, and its error text is kept for the next prompt §III-B §IV-C.

Evaluation §III-C §IV-B eq. 2 eq. 3 eq. 4. Each algorithm runs on the 24 BBOB functions, three instances per function and three seeds per instance, with budget 10,000 evaluations in 5 dimensions §IV-B. Each run is scored by the normalised area over the convergence curve (AOCC). The score used for selection is the mean AOCC over functions and instances, averaged over k = 5 repeated runs eq. 4 §IV-C. The standard deviation over runs is kept for the feedback §III-C.

Feedback prompt §III-D alg. 1 l.9-11. The prompt for the next offspring holds (i) the task description, (ii) the names and mean scores of all algorithms generated so far, (iii) the full code, mean score, standard deviation and error text of the selected parent, and (iv) a one-line instruction: either refine the algorithm or redesign it. The parent is the best-so-far algorithm under (1+1) selection and the most recent algorithm under (1,1) selection §III-D. The paper treats the refine-or-redesign choice as left to the model; it compares it to a restart or a large mutation in evolution strategies §III-D.

Loop alg. 1. The loop runs T = 100 LLM calls per run §IV-B. The paper gives the following algorithm for the (1+1) and (1,1) variants.

# LLaMEA loop, (1+1) or (1,1) variant  [alg. 1]
# Input:  task prompt S, evaluator f, budget T, flag elitist
# Output: best algorithm code a_b and its mean quality y_b

def llamea(task_prompt, f, budget, elitist):
    # Initialise: generate the first algorithm from S  [alg. 1 l.1-6]
    last_code = generate(task_prompt)
    last_mean, last_std, last_error = f(last_code)
    best_code, best_mean = last_code, last_mean
    best_std, best_error = last_std, last_error
    history = [(name(last_code), last_mean)]
    for t in range(budget):
        # Choose the parent: best under (1+1), last under (1,1)  [alg. 1 l.9-11]
        if elitist:
            parent_code, parent_mean = best_code, best_mean
            parent_std, parent_error = best_std, best_error
        else:
            parent_code, parent_mean = last_code, last_mean
            parent_std, parent_error = last_std, last_error
        # Build the feedback prompt F from S, history, parent and F0  [§III-D]
        prompt = build_prompt(task_prompt, history, parent_code,
                              parent_mean, parent_std, parent_error)
        # One LLM call gives the offspring  [alg. 1 l.13]
        child_code = generate(prompt)
        child_mean, child_std, child_error = f(child_code)
        # A runtime or compile error scores zero  [alg. 1 l.14-15]
        if child_error is not None:
            child_mean = 0
        # Update best; compares with the last score, as printed  [alg. 1 l.17]
        if child_mean >= last_mean:
            best_code, best_mean = child_code, child_mean
            best_std, best_error = child_std, child_error
        last_code, last_mean = child_code, child_mean
        last_std, last_error = child_std, child_error
        history.append((name(child_code), child_mean))
    return best_code, best_mean

Hyper-parameters of the search itself. Three LLM families were used (see the table below). The paper states no tuning of the LLaMEA loop itself; the choices are listed as stated §IV-A §IV-B.

Hyperparameters

Results

Author-reported. Setup: BBOB 24 functions, d = 5, budget 10,000 evaluations, the 25-run evaluation of §VI, AUC of the empirical attainment function (EAF) as the score in tab. I. Baselines are CMA-ES, a tuned CMA-ES ("CMA-best") and differential evolution (DE), all taken from IOHanalyzer data §VI §VI-B.

The best generated algorithm is shown below in pseudocode, with its parameters alg. 2. The paper notes that its name ("quantum flux") is not explained by the code and that the algorithm resembles JADE and DE with a memory term §VI-A.

# ERADS_QuantumFluxUltraRefined: DE with a memory term  [alg. 2]
# Input:  objective f on (-5, 5)^d, budget B
# Output: best point x_opt and its value f_opt

def erads(f, B, d):
    # Population of N = 50 uniform random points  [alg. 2]
    population = uniform(-5.0, 5.0, (50, d))
    fit = f(population)
    best_index = argmin(fit)
    f_opt = fit[best_index]
    x_opt = population[best_index]
    # Mutation memory starts at zero  [alg. 2]
    memory = zeros(d)
    t = 50                      # evaluations used so far
    while t < B:
        # Scale factor rises linearly with the fraction of budget used  [alg. 2]
        F = 0.55 + (0.85 - 0.55) * (t / B)
        for i in range(50):
            # Three distinct partners other than i  [alg. 2]
            r1, r2, r3 = three_distinct_indices_except(i)
            best = population[best_index]
            # Mutant: rand-to-best plus difference plus memory term  [alg. 2]
            mutant = population[r1] + F * (
                best - population[r1]
                + population[r2] - population[r3]
                + 0.3 * memory)
            mutant = clip(mutant, -5.0, 5.0)
            # Binomial crossover with probability CR = 0.95  [alg. 2]
            trial = where(rand(d) < 0.95, mutant, population[i])
            f_trial = f(trial)
            t += 1
            if f_trial < fit[i]:
                population[i] = trial
                fit[i] = f_trial
                if f_trial < f_opt:
                    f_opt = f_trial
                    x_opt = trial
                    best_index = i
                # Memory update; uses the new population[i]  [alg. 2]
                memory = 0.7 * memory + 0.3 * F * (mutant - population[i])
            if t >= B:
                break
    return x_opt, f_opt

Paper vs code

Official code: the repository XAI-liacs/LLaMEA (MIT). It is owned by the LIACS lab organisation, and its README names both authors and cites the TEVC version of this paper. The paper links only an anonymised Zenodo archive (v1, 2024, DOI 10.5281/zenodo.11358117) which I did not download; the code cloned here is the current main branch (commit f896094aff09b021e42b70f33a68de63bd2795c5, 2026-10-29). It is a later state of the framework and not verified as the code that produced the paper's numbers. Discrepancies below are checked against this commit.

• Output format. The paper asks for # Name: and # Code: §III-A. The default code asks for # Description: and # Code: [code: llamea/llamea.py#L241-L248]. The name is the first def or class name in the code llm.py.
• Task prompt. The paper's prompt uses __call__(self, f) §III-A. The default code uses __init__(self, budget, dim) and __call__(self, func), and states the bounds llamea.py. The random-search example differs llamea.py.
• Refine-or-redesign instruction. The paper uses "either refine or redesign" §III-D. The default operator only says to refine the selected solution, with no redesign option llamea.py.
• Feedback history. The paper lists the names and mean scores of all algorithms generated so far §III-D. The code lists the current population llamea.py.
• Population shape. The paper's loop is one parent, one offspring alg. 1. The default is 5 parents and 5 offspring with elitism [code: llamea/llamea.py#L68-L77]. The shipped example uses 2 parents and 5 offspring black-box-optimization.py. Selection with elitism keeps the parents first in a stable sort, so on ties the parent is kept llamea.py. The paper's update test accepts an offspring on a tie alg. 1 l.17.
• Budget counting. The paper counts T loop calls §IV-B. The loop continues while the run history, which includes the initial population, is below the budget llamea.py.
• Error score. The paper uses 0 for fatal errors §IV-C. The default maximising worst value is minus infinity llamea.py. Ordering is unchanged for scores that are never negative, but the value shown in feedback may differ.
• Benchmark budget in the example. The paper uses B = 10,000 §IV-B. The example evaluator uses 200 × d = 1,000 at d = 5 [code: examples/black-box-optimization.py#L38-L39]. It also uses three instances and three seeds, and averages over those nine runs, not over five repeats black-box-optimization.py.
• AOCC upper bound. The paper uses ub = 1e2 §IV-C. The example passes 1e2 black-box-optimization.py, but the logger's default is 1e8 utils.py. The logger log10-transforms and clips the best-so-far value utils.py.
• Sampling defaults. The paper uses temperature 0.8 and top_p 1 §IV-A. The OpenAI wrapper defaults to temperature 0.8 and sets no top_p [code: llamea/llm.py#L264]. The Gemini wrapper uses temperature 1 and top_p 0.95 llm.py.
• ERADS memory update. The paper's prose says the memory moves toward the mutant minus the old individual §VI-A. Algorithm 2 and the code update it with the new population entry, which is the trial vector alg. 2 [code: llamea/bbobalgs/ERADS_QuantumFluxUltraRefined.py#L76-L81]. Because CR = 0.95, most coordinates of the trial equal the mutant, so the update is small. The paper also says the memory update fires only on improvement of the best; Algorithm 2 and the code fire on improvement of the individual §VI-A [alg. 2] ERADS_QuantumFluxUltraRefined.py.
• ERADS crossover. The paper calls the crossover classical binomial §VI-A. The code draws each coordinate independently with probability CR and does not force one coordinate from the mutant [code: llamea/bbobalgs/ERADS_QuantumFluxUltraRefined.py#L60-L63].

I searched for other official code: the paper links only the Zenodo archive; the GitHub search returned the XAI-liacs repository as the only match owned by the authors' lab; several forks and student projects were ignored.

Pitfalls

• Update test in the printed algorithm. Algorithm 1 compares a new offspring with the previous offspring's score, not with the best score, when it decides to update the best alg. 1 l.17. Under (1+1) the parent shown to the model is the best, so the printed rule can replace the best with a weaker offspring. The code keeps the top parents instead (see Paper vs code). Pick one rule explicitly.
• Repeat counts disagree. The text gives 3 instances times 3 seeds, or 9 runs per function §IV-B. The score in the equation is averaged over k = 5 independent runs eq. 4 §IV-C. The paper does not say which set feeds the model's feedback. Decide and state it.
• Search and test instances overlap. The search uses instances 1 to 3 §V. The final comparison uses five instances §VI. The paper does not say whether the test instances include the search ones. Generalisation claims need separate instances.
• The abstract is broader than the table. The abstract says several generated algorithms outperform CMA-ES [abstract]. The text says one algorithm beats CMA-ES, and Table I lists only ERADS above CMA-ES's 0.703 §V tab. I. The text also says two algorithms beat CMA-best after the full budget, while Table I lists the second one (0.643) below CMA-best (0.742) §VI-A tab. I. The table caption says "best 3 per model", but the table has nine generated rows and three baselines tab. I.
• Scores are not comparable across models. The first algorithm differs per LLM, so initial points differ in the convergence plots fig. 2.
• Errors are common. About 18.7% of runs had a run-time error and score zero; zero scores are not plotted §V fig. 2. The paper does not show how many of those would have passed with a retry.
• Only 5-D was optimised. The paper says the algorithm was built for the 5-D case and is not expected to scale §VI-B. ERADS fails to generalise to 10-D and 20-D §VI-B fig. 9.
• Tuned baseline. CMA-best was tuned on the same BBOB 5-D setting at 10,000 evaluations §VI. Its advantage over default CMA-ES is shown in higher dimensions §VI-B. A fair comparison needs the same tuning budget.
• Hidden costs. 4,500 LLM calls and 9.72 × 10^9 benchmark evaluations in total §V. The paper gives no dollar cost. Each candidate costs 216 runs of 10,000 evaluations each (2.16 million evaluations) §V.
• Name is not evidence. The model's algorithm name is not checked against the code, and the paper calls the "quantum" part unexplained, possibly a hallucination §VI-A. Manual checks say names match the code in most cases §V-A.
• Supplement numbers. The hyper-parameter tuning table in the supplement is SMAC3 with 5,000 evaluations; the authors report that only the 20-D tuned setting beats the default [supp. tab. II] [supp. §IV].

Lineage

• Builds on: EvoLLM (arXiv:2402.18381), which uses an LLM as a proposal distribution; AEL (arXiv:2311.15249), which uses an LLM for evolutionary crossover and mutation of heuristics; STOP (arXiv:2310.02304), which has an LLM improve its own scaffolding; and EoH, which the paper calls the closest prior method (ICML 2024, no arXiv id in the paper's bibliography) §II. Named and linked only; no content used.
• Compared against: EoH and random search, both with GPT-4 §IV-D; CMA-ES, tuned CMA-ES and DE from prior benchmark data §VI.
• Superseded by: not known from this wiki.

Figures and tables cited

Figures are described in words only; no images are published. fig. 1 is the framework overview. fig. 2 shows mean best-so-far curves for all configurations. fig. 3 compares the best configuration with EoH and random search. fig. 4 is a word cloud of algorithm name parts. fig. 5 shows code difference ratios. fig. 6 shows Jaro name similarity. fig. 7 shows the EAF at 5-D. fig. 8 shows per-function curves for ERADS and the baselines. fig. 9 shows the EAF at 10-D and 20-D. tab. I is the AUC table. The supplement has two tables: BBOB groups, and ERADS tuning, cited as [supp. tab. II].