--- kind: concept slug: evolutionary-algorithm title: Evolutionary algorithm summary: >- Candidates are scored, the better ones become parents, and their children are recombined or mutated, for a fixed number of rounds. Three cards apply this loop to bagged trees, to LLM-written optimisers and to LLM-agent solution trees. The trade-off: every round needs fresh fitness evaluations, and with an LLM in the loop, paid model calls. tags: [search, optimization, metaheuristics, llm] papers: ["2208.02400", "2405.20132", "2511.07262"] related: [metaheuristic-optimization, neuroevolution, neural-architecture-search, prompt-optimization, bagging, random-forest, multi-agent-systems] status: draft updated_at: 2026-11-08 --- ## Overview An evolutionary algorithm keeps a set of candidate solutions, scores each one with a fitness function, keeps the better ones as parents, and makes children from them by recombination or mutation. The loop then repeats for a fixed number of rounds or until a stopping rule fires. The three cards below share this loop. They differ in what a candidate is, how a child is made, and what the fitness costs to compute. The invariant the variants share: each round replaces part of the population with children of selected parents, and the best-scoring member is what the method returns [arxiv:2208.02400 §3.1] [arxiv:2405.20132 §III-D] [arxiv:2511.07262 §2]. ## Variants ### Evolving the rows of an ensemble A candidate is one bag: a bootstrap sample of training rows, with one decision tree fitted to it. Crossover swaps the rows each parent gets wrong, a fixed share of fresh random bags enters each round, and a few bags are mutated by swapping rows [arxiv:2208.02400 §3.1] [arxiv:2208.02400 §3.3] [arxiv:2208.02400 §3.4]. Fitness is the bag's performance times a size term (K + bag size) / K [arxiv:2208.02400 §3.2]. The paper leaves unstated the data on which the performance term is measured [arxiv:2208.02400 §3.2]. Card: [Evolutionary bagging](/papers/2208.02400). ### Evolving code written by a language model A candidate is a Python optimiser. The language model writes the first one from a task prompt, and each later child is a rewrite of the selected parent, guided by the parent's code, its score and its error text [arxiv:2405.20132 §III-D]. The score is the normalised area over the convergence curve (AOCC), averaged over benchmark functions, instances and repeated runs [arxiv:2405.20132 §IV-C]. The paper's main loop is one parent and one child per round, with (1+1) or (1,1) selection [arxiv:2405.20132 §III-D]. Card: [LLaMEA](/papers/2405.20132). ### Evolving solution trees with agents A candidate is a complete scientific-ML solution, stored in a tree with its score and an analysis report. The best node is always a parent. Three more parents are picked by a majority vote of three selector models [arxiv:2511.07262 §2] (Ensemble-Guided Parent Selection). Each child comes from a debate between a proposer and a critic, then an engineer edits the parent's code, and a debugger repairs it [arxiv:2511.07262 §2]. Card: [AgenticSciML](/papers/2511.07262). ## When to use which The choice depends on what a fitness evaluation costs, what the mutation operator costs, and how much of the search space you can express in code. | Situation | Use | Why | |---|---|---| | Improve a bagged ensemble of trees, where each fitness call is one tree fit | Row-level evolutionary bagging [arxiv:2208.02400 §3.1] | Author-reported: best or tied test accuracy on 10 of 12 datasets [arxiv:2208.02400 tab. 4]. Cost per iteration is about N tree fits, so I iterations cost about I times bagging's training [arxiv:2208.02400 §5]. | | Continuous black-box function, and you can afford many benchmark runs | Hand-tuned CMA-ES as the reference | On 5-D BBOB, the best LLM-generated method reached AUC 0.733, below tuned CMA-ES at 0.742 and above default CMA-ES at 0.703 [arxiv:2405.20132 tab. I]. | | You want a language model to write new optimiser code, and can pay for thousands of calls | One-parent LLM loop with benchmark feedback | The paper used 4,500 LLM calls and 216 benchmark runs per candidate [arxiv:2405.20132 §V]. About 18.7% of runs hit a run-time error and scored zero [arxiv:2405.20132 §V]. | | The target is one dimensionality only | Any of the above, with a check at the other dimensions | The generated method did not generalise to 10-D and 20-D [arxiv:2405.20132 §VI-B]. | | Design a physics-ML model with expensive training, and LLM calls are affordable | Tree search with a multi-agent mutation step | Reported LLM cost per run was $2.07 to $11.30 [arxiv:2511.07262 §3.5]. GPU time exceeded LLM time in five of six runs [arxiv:2511.07262 §3.5]. Each problem was run once [arxiv:2511.07262 §3]. | Notes on reading the numbers: - All results are author-reported. None of the three papers reports seeds or repeated runs for the headline comparison, except that LLaMEA averages five runs per configuration [arxiv:2405.20132 §V]. - The benchmarks differ, so the scores cannot be compared across rows of the table. The bagging result uses tabular datasets, LLaMEA uses noiseless BBOB functions, and AgenticSciML uses six PDE and operator-learning tasks [arxiv:2208.02400 §4.2] [arxiv:2405.20132 §IV-B] [arxiv:2511.07262 §3]. - If fitness is computed on the same rows a model was trained on, the score is optimistic. The evolutionary bagging fitness can work this way, and its paper does not say which data is used [arxiv:2208.02400 §3.2]. - In the agent card, selection and reporting use one validation set, so the champion number is optimistic [arxiv:2511.07262 §S1.1.1]. Re-evaluate on held-out data before comparing methods. ## Implementation sketch The loop shared by the three variants. Each variant changes the `vary` and `select` steps, and the cost is in `fitness` and `vary`. ```pseudocode # Common loop of the three variants (generic sketch, not a released function) def evolve(population, fitness, select, vary, rounds): # Score every member. The score is a training accuracy, a benchmark # AOCC or a validation score, depending on the variant [arxiv:2208.02400 §3.2] scores = [fitness(m) for m in population] for _ in range(rounds): # Choose parents. Top-ranked [arxiv:2405.20132 §III-D], or the best # node plus votes [arxiv:2511.07262 §2] parents = select(population, scores) # Make children by recombination, mutation or an LLM rewrite # [arxiv:2208.02400 §3.4] [arxiv:2405.20132 §III-D] [arxiv:2511.07262 §2] children = [vary(p) for p in parents] # Score the children. This is the costly step in every variant child_scores = [fitness(c) for c in children] # Keep the best of the old and new members. Ties are handled # differently in the code than in the printed algorithm, see below population, scores = keep_best(population + children, scores + child_scores) return best_of(population, scores) ``` Where the variants differ: - **Crossover.** The row-level variant gives each parent's correct rows to itself and passes its wrong rows to the other parent [arxiv:2208.02400 §3.4]. - **Mutation.** The row-level variant swaps a fixed number of rows in a few bags [arxiv:2208.02400 §3.4]. The LLM variants change the candidate by asking a model for an edit [arxiv:2405.20132 §III-D]. - **Ties.** The printed update rule in the LLM paper accepts an offspring on a tie [arxiv:2405.20132 alg. 1]. The released code keeps the parent on a tie [see the card's Paper vs code section](/papers/2405.20132). ## Contradictions - **Parent selection in the bagging paper.** The text names rank selection in one place [arxiv:2208.02400 §3.4] and probabilistic selection by fitness in another [arxiv:2208.02400 §3.3]. The released code takes the top-ranked bags. - **Two-spiral gain in the bagging paper.** The text gives 11.6% over bagging and 18.3% over random forest [arxiv:2208.02400 §4.3]. The main table gives 8.4 and 12.0 percentage points [arxiv:2208.02400 tab. 4]. Both are reported; this page uses the table. - **Ensemble size from the test set.** The bagging paper chooses the number of bags on the test set [arxiv:2208.02400 §4.2]. The released code uses cross-validation on the training data [see the card's Paper vs code section](/papers/2208.02400). - **Generalisation of the best LLM method.** The abstract of the LLM paper says several generated algorithms beat CMA-ES [arxiv:2405.20132 §I]. The text says two algorithms beat tuned CMA-ES after the full budget, but Table I lists the second one below it [arxiv:2405.20132 §VI-A] [arxiv:2405.20132 tab. I]. - **Model line-up in the agent paper.** The cost tables label the critic and debugger as GPT-4o-Mini, and the configuration tables say GPT-5 Mini [arxiv:2511.07262 tab. S2] [arxiv:2511.07262 tab. S1]. Both are reported here; the card's Pitfalls section lists the rest of the discrepancies. - **Benefit of the knowledge base.** The agent paper reports one ablation with one run per setting and no stated seed [arxiv:2511.07262 §S2.2]. No other result in the cited set tests the same effect. ## Papers - [Evolutionary bagging](/papers/2208.02400): evolves the training rows of each bagged tree with crossover, mutation and fresh bags. - [LLaMEA](/papers/2405.20132): a language model writes and rewrites Python optimisers, scored on 5-D BBOB. - [AgenticSciML](/papers/2511.07262): an LLM-agent tree search for scientific-ML models, with ensemble-chosen parents.