---
kind: concept
slug: neuroevolution
title: Neuroevolution
summary: >-
  Evolving networks with gradient-free search. Topologies scored with one shared weight, no weight training, beat fixed baselines on three control tasks. A linear controller evolved with CMA-ES over a learned world model reaches 906 on car racing. Co-evolved bodies with PPO learners learn new tasks faster. Author-reported. Trade-offs: large search budgets, fixed-weight scoring, feed-forward only.
tags: [evolution, topology-search, weight-agnostic, evolution-strategies, neural-architecture-search, morphology, quality-diversity]
papers: ["1706.01040", "1708.09251", "1803.10122", "1809.01999", "1906.04358", "2102.02202"]
related: [neural-architecture-search, metaheuristic-optimization, policy-gradient, model-based-reinforcement-learning, variational-autoencoder, recurrent-neural-network, robot-simulation, quality-diversity]
status: draft
updated_at: 2026-11-01
---

## Overview

Neuroevolution uses evolutionary search, not gradient descent, to shape a
neural network. The variants below differ in what the search evolves. One
evolves the wiring and never trains the weights during the search
[arxiv:1906.04358 §3]. Another evolves the weights of a small controller and
scores them with the real reward [arxiv:1809.01999 §2] [arxiv:1809.01999 app.
A.5]. A third evolves an agent's body, and each candidate's neural controller
is trained with PPO before its fitness is known [arxiv:2102.02202 §DERL]. A
fourth keeps a whole collection of different good solutions rather than one
best, and its genotypes are parameter vectors, not described as networks
[arxiv:1708.09251 §3] [arxiv:1708.09251 §4]. A position paper proposes an
evolutionary update rule but gives no method and no results [arxiv:1706.01040
§3].

Every number on this page is author-reported and comes from the cited tables.
The three lines of work use different tasks, inputs and metrics, so their
scores are not directly comparable (see Contradictions). The two world-model
papers by the same authors describe the same controller and report the same
Car Racing and Take Cover scores [arxiv:1803.10122 tab. 2] [arxiv:1803.10122
tab. 4] [arxiv:1809.01999 tab. 1] [arxiv:1809.01999 tab. 2]. Those numbers come
from one line of work and are not independent replications.

## Variants

### Topology search with a shared weight

Every connection in a candidate takes the same value, so a candidate's score
depends on its structure and on that one value [arxiv:1906.04358 §3]. The
search starts from minimal networks [arxiv:1906.04358 §3]. Each generation it
scores every candidate over six shared weight values, -2, -1, -0.5, 0.5, 1 and
2, and averages the rollouts at each value [arxiv:1906.04358 §3]. Parents are
picked by tournament selection. Each child receives exactly one change from
three operators: insert a node on a connection, add a feed-forward connection,
or change a hidden node's activation [arxiv:1906.04358 §3]. Candidates are
ranked by Pareto dominance, using mean reward against connection count in most
generations and against maximum reward in the rest [arxiv:1906.04358 §3]. Card:
[Weight Agnostic Neural Networks](/papers/1906.04358).

Reported results, averaged over 100 trials of each champion [arxiv:1906.04358
tab. 1]:

| Task | Condition | Searched network | Fixed baseline |
|---|---|---|---|
| Swing-up | Random shared weight | 515 ± 58 | 7 ± 2 |
| Swing-up | Tuned shared weight | 723 ± 16 | 8 ± 1 |
| Biped | Random shared weight | 51 ± 108 | -107 ± 12 |
| Biped | Tuned shared weight | 261 ± 58 | -35 ± 23 |
| Car racing | Random shared weight | 375 ± 177 | -85 ± 27 |
| Car racing | Tuned shared weight | 608 ± 161 | -37 ± 36 |

The caption does not define the ± [arxiv:1906.04358 tab. 1]. Random shared
weights are drawn from U(-2, 2), and the tuned value is the best in that
range [arxiv:1906.04358 §4]. The introduction reports about 92% MNIST test
accuracy with random weights [arxiv:1906.04358 §1]. The paper does not state
the test protocol behind that figure.

The paper also says that its networks fail when their individual weights are
drawn at random, so random-weight rows are not a measure of the method without
training [arxiv:1906.04358 §4].

#### Fine-tuning the champion's weights

After the search, the paper trains the champion's individual weights with
population-based REINFORCE [arxiv:1906.04358 app. A.6]. For swing-up and
biped it uses 384 agents with 16 rollouts each. For car racing it uses 64
agents with 4 rollouts each. Training runs for 3000 generations
[arxiv:1906.04358 app. A.6].

With trained weights the searched networks are close to the fixed baselines,
and the order depends on the task. Swing-up gives 932 ± 6 against 918 ± 7,
biped 332 ± 1 against 347 ± 1, and car racing 893 ± 74 against 906 ± 21
[arxiv:1906.04358 tab. 1]. The authors read these as similar to
state-of-the-art performance [arxiv:1906.04358 §4].

### Evolving a controller over a learned world model

A convolutional VAE encodes each frame into a latent z. An LSTM with a
mixture-density output predicts the next z from the current z, the action and
its hidden state [arxiv:1809.01999 §2]. Both are trained on frames from
10,000 random rollouts and never see the reward [arxiv:1809.01999 §3]
[arxiv:1809.01999 alg. 1]. The controller is a single linear layer from z and
the LSTM hidden state to the action. It has 867 parameters for car racing
[arxiv:1809.01999 §2] [arxiv:1809.01999 app. A.1] [arxiv:1803.10122 tab. 1].
CMA-ES evolves those weights against the real episode reward, with a
population of 64 and 16 rollouts per candidate [arxiv:1809.01999 app. A.5]
[arxiv:1803.10122 app. A.4]. Cards: [World Models](/papers/1803.10122) (the
original arXiv paper) and [World Models Facilitate Policy
Evolution](/papers/1809.01999).

On car racing the full model reaches 906 ± 21, against 838 ± 11 for the Gym
Leader baseline and 591 ± 45 for A3C continuous. The controller on z alone
reaches 632 ± 251 [arxiv:1809.01999 tab. 1] [arxiv:1809.01999 §3.1]. The
original paper reports the same full-model score and the same baselines
[arxiv:1803.10122 tab. 2]. The best agent over 1024 rollouts reaches 900.46
after 1800 generations [arxiv:1809.01999 app. A.5].

On DoomTakeCover the controller is trained only inside the learned model and
then run in the real game. At sampling temperature τ = 1.15 it scores 1092 ±
556 on the real task, against 820 ± 58 for the Gym Leader baseline
[arxiv:1809.01999 tab. 2] [arxiv:1809.01999 §4.2] [arxiv:1803.10122 tab. 4].
The temperature sweep shows the trade-off. At τ = 0.10 the same procedure
scores 2086 ± 140 in the model but 193 ± 58 in the real game [arxiv:1809.01999
tab. 2] [arxiv:1803.10122 tab. 4].

### Evolving bodies with lifetime learning

Here the search changes an agent's body: a kinematic tree with a spherical
head and motor-driven limbs, simulated in MuJoCo [arxiv:2102.02202 §DERL].
Mutations add or delete limbs and change limb size, density, joint axis, joint
limits or motor gear [arxiv:2102.02202 app. A]. Each new body starts with a
controller of random weights, which is trained with PPO for five million
agent-environment steps. Its fitness is the final average reward [arxiv:2102.02202
§DERL] [arxiv:2102.02202 app. A] [arxiv:2102.02202 tab. 2]. The controller is
trained by gradient-based PPO and is not evolved, so this variant is
neuroevolution only in its outer loop.

The search is asynchronous. Four random agents are drawn, the fittest becomes
the parent, and one mutation is applied to a copy of its body. The child is
trained and added, and the oldest member leaves, so the population stays at
576 [arxiv:2102.02202 §DERL] [arxiv:2102.02202 app. A]. Card: [Embodied
Intelligence via Learning and Evolution](/papers/2102.02202).

Reported results, from one simulator with three evolutionary runs per
environment [arxiv:2102.02202 app. D]:

- Bodies evolved in manipulation on variable terrain beat bodies evolved on
  flat ground on most test tasks. Bodies evolved on variable terrain beat them
  on five of six agility and stability tasks [arxiv:2102.02202 §Environmental
  complexity] [arxiv:2102.02202 fig. 4]. The count of test tasks differs
  between text and figure (see Contradictions).
- For a flat-ground lineage, generation 8 reaches about twice the final reward
  of generation 1, and reaches generation 1's final reward in about a fifth of
  the time [arxiv:2102.02202 §Baldwin effect] [arxiv:2102.02202 fig. 5].
- Learning time to a 75th-percentile starting fitness halves within 10
  generations, although selection never rewards learning speed [arxiv:2102.02202
  §Baldwin effect] [arxiv:2102.02202 fig. 5].
- Bodies from manipulation on variable terrain have a lower cost of work than
  flat-ground bodies [arxiv:2102.02202 §Environmental complexity] [arxiv:2102.02202
  fig. 4]. Across bodies, lower cost of work goes with higher reward and faster
  learning. This is a correlation [arxiv:2102.02202 §Mechanistic underpinning]
  [arxiv:2102.02202 fig. 6].

### Quality-diversity search over parameter vectors

This variant keeps a container of solutions and tries to fill it with
solutions that behave differently, instead of returning one best [arxiv:1708.09251
§1]. The container is either a grid with one solution per cell, or an
unstructured archive with an insertion rule based on exclusive
epsilon-dominance [arxiv:1708.09251 §3.1.1] [arxiv:1708.09251 §3.1.2]. Parents
are chosen with a curiosity score: a parent gains 1 when its offspring is added
to the container and loses 0.5 when it is not [arxiv:1708.09251 §3.2.3]. The
paper's genotypes are parameter vectors: eight real values for a planar arm,
and 36 sampled values for a six-legged robot [arxiv:1708.09251 tab. 2]. The
card does not describe them as neural networks. Card: [Quality and Diversity
Optimization](/papers/1708.09251).

Reported results, simulation only, with 10 to 20 replications per condition
[arxiv:1708.09251 §4]:

- On the arm, curiosity selection over the whole collection beats
  population-based selection on all four collection metrics [arxiv:1708.09251
  §4.2.2] [arxiv:1708.09251 fig. 4].
- On hexapod turning, curiosity selection beats NSLC (p < 0.0047)
  [arxiv:1708.09251 §4.3.2].
- On hexapod gaits, curiosity reaches the 20,000-batch quality of random
  selection at 11,000 batches with an archive and at 13,500 with a grid
  [arxiv:1708.09251 §4.4.2] [arxiv:1708.09251 fig. 7].
- On the same gait task, population-based fitness selection gives the best
  maximal quality (p < 0.0017), so the curiosity result is about collection
  quality and speed, not peak score [arxiv:1708.09251 §4.4.2].

### Brain Intelligence (proposal only)

The position paper proposes a network of small neural sub-networks, updated by
an "S-system" of reproduction, selection and mutation [arxiv:1706.01040 §3].
It gives no equations, algorithm, hyperparameters or experiments
[arxiv:1706.01040 §1–§4], and it does not define the S-system. Its claims that
the design would solve four AI limitations are not tested [arxiv:1706.01040
§4]. Card: [Brain Intelligence: Go Beyond Artificial Intelligence](/papers/1706.01040).
Use it as background only, not as a method to implement.

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need a network whose structure you can inspect, and the task is small enough for hand-checking | Topology search with a shared weight | The champions are small: 52 connections for swing-up, 210 for biped, 245 for car racing [arxiv:1906.04358 fig. 5] [arxiv:1906.04358 §4] |
| The fixed baseline needs tuning and you can afford to tune one scalar | Topology search with a tuned shared weight | Tuned shared weights beat the fixed baselines in all three control tasks [arxiv:1906.04358 tab. 1] |
| You can train weights and want the best score on a task | Fine-tune the champion | Tuned-weight rows are close to the baselines, and the biped baseline is higher [arxiv:1906.04358 tab. 1] |
| You can afford a learned model and want a small controller that search can handle | Controller over a learned world model | The controller has 867 parameters and scores 906 ± 21 on car racing [arxiv:1809.01999 §2] [arxiv:1809.01999 tab. 1]. The model itself has about 4.8 million parameters for car racing [arxiv:1809.01999 tab. 3] |
| The task is a game where a learned model can be exploited | Choose the sampling temperature by real score | At τ = 0.10 the model score is 2086 but the real score is 193 [arxiv:1809.01999 tab. 2] |
| You have a large compute budget for the search | Topology search, full settings | The paper uses populations of 192 to 960 over 1024 to 4096 generations [arxiv:1906.04358 app. A.4] |
| The task needs memory across steps | Controller over a learned world model, or not the topology method | The topology search builds feed-forward networks only; recurrent connections are future work [arxiv:1906.04358 §4] [arxiv:1906.04358 §5]. The world-model controller reads the LSTM hidden state [arxiv:1809.01999 §2] |
| You need a classifier with a trusted test score | Treat MNIST with care | The 92% figure has no stated test protocol [arxiv:1906.04358 §1]. The search scores networks on training samples [arxiv:1906.04358 app. A.3] |
| You can run a large simulation cluster and need bodies that learn new tasks fast | Evolve bodies with lifetime learning | The full run used 1152 CPUs for under 16 hours of wall-clock time [arxiv:2102.02202 §DERL] [arxiv:2102.02202 app. A]. No single-machine estimate is given |
| You need energy-efficient bodies | Evolve in manipulation on variable terrain, and measure cost of work | Those bodies have a lower cost of work, but selection used only forward reward, so efficiency is a side effect [arxiv:2102.02202 app. B] [arxiv:2102.02202 fig. 4] |
| You need many distinct good solutions, not one optimum | Quality-diversity search, with curiosity selection | On the arm task it beats population-based selection on coverage and quality [arxiv:1708.09251 §4.2.2]. The arm collection has about 6,000 solutions [arxiv:1708.09251 §4.2.2] |
| You need the single highest score on a task | Not quality-diversity; keep population-based fitness selection | On gaits, population-based fitness selection gives the best maximal quality [arxiv:1708.09251 §4.4.2] |
| You must choose between a grid and an archive container | Either; check the descriptor size for yourself | The paper tries both containers on the 2-D arm and the 6-D gait task, and gives no rule for the choice [arxiv:1708.09251 tab. 2] [arxiv:1708.09251 §4.2.2] |
| Each candidate evaluation must be cheap | Avoid lifetime-learning search | Each child trains for five million steps from random weights, so the search is as slow as its learner [arxiv:2102.02202 app. A] [arxiv:2102.02202 tab. 2] |

The search cost is high. The topology paper gives no wall-clock figures, so
budget it by population and generation count [arxiv:1906.04358 app. A.4]. The
world-model variant reaches about 1.8 million episodes at 1800 generations,
derived from its population and rollout counts [arxiv:1809.01999 app. A.5].

## Implementation sketch

Language-agnostic pseudocode of the common core: score candidates, select, and
vary. The variants differ in what a candidate is, how it is scored, and the
update rule.

```pseudocode
# Score one topology over the shared weights  [arxiv:1906.04358 §3]
# Input:  net, six shared weights W, rollouts per weight
# Output: (mean reward, best reward) over W

def score(net, W, reps):
    per_weight = []
    for w in W:
        # Every existing connection takes the value w; absent ones stay out
        # [arxiv:1906.04358 §3]
        rewards = [rollout(net, shared_weight=w) for _ in range(reps)]
        per_weight.append(mean(rewards))
    return mean(per_weight), max(per_weight)
```

```pseudocode
# Search loop  [arxiv:1906.04358 §3]
# Input:  pop, generations, W, reps, k (tournament size)
# Output: final population of topologies

def wann_search(pop, generations, W, reps, k):
    # Start from minimal networks  [arxiv:1906.04358 §3]
    population = minimal_networks(pop)
    for _ in range(generations):
        scored = [(net, *score(net, W, reps)) for net in population]
        # Pareto ranking; the second objective is connection count in most
        # generations and maximum reward in the rest  [arxiv:1906.04358 §3]
        ranked = pareto_sort(scored)
        children = []
        while len(children) < pop:
            # Tournament selection picks the parent  [arxiv:1906.04358 §3]
            parent = tournament(ranked, k)
            # Exactly one change per child: insert node, add connection or
            # change activation  [arxiv:1906.04358 §3]
            children.append(one_change(parent))
        population = children
    return population
```

```pseudocode
# Evolve controller weights with CMA-ES  [arxiv:1809.01999 app. A.5]
# Input:  num_params, generations, population, rollouts per candidate
# Output: mean of the search distribution (controller weights)

def evolve_controller(num_params, generations, population, rollouts):
    search = cma_init(num_params, popsize=population)
    for _ in range(generations):
        # Sample candidate weight vectors  [arxiv:1809.01999 app. A.5]
        candidates = search.ask()
        # Fitness is the mean real return over several seeds
        # [arxiv:1809.01999 app. A.5]
        returns = [mean_return(w, rollouts) for w in candidates]
        search.tell(returns)
    return search.current_mean()
```

```pseudocode
# Asynchronous tournament over bodies  [arxiv:2102.02202 §DERL]
# Input:  population of evaluated agents, search budget
# Output: population after the budget is spent

def body_tournament(population, budget, pop_size=576, k=4):
    searched = 0
    while searched < budget:
        # Draw k agents; the fittest is the parent  [arxiv:2102.02202 app. A]
        parent = max(sample(population, k), key=fitness)
        # Mutate a copy of the parent body  [arxiv:2102.02202 app. A]
        child = Agent(mutate(parent.body), controller=random_weights())
        # Lifetime learning with PPO; fitness is the final reward
        # [arxiv:2102.02202 app. B] [arxiv:2102.02202 tab. 2]
        child.fitness = train_ppo(child, steps=5_000_000)
        # Add the child and keep the newest pop_size members
        # [arxiv:2102.02202 §DERL]
        population = (population + [child])[-pop_size:]
        searched += 1
    return population
```

## Contradictions

Nothing in the sources directly disagrees, but the car racing numbers need
care. The world-model papers report 906 ± 21 for the controller
[arxiv:1809.01999 tab. 1] [arxiv:1803.10122 tab. 2]. The topology paper uses
it as its fixed baseline for car racing [arxiv:1906.04358 tab. 1]. Its searched
network scores 608 ± 161 with a tuned shared weight and 893 ± 74 with trained
weights [arxiv:1906.04358 tab. 1]. These are not comparable as methods: the
topology network sees only the 16-dimensional VAE latent, not the recurrent
state [arxiv:1906.04358 §4] [arxiv:1906.04358 app. A.4].

The world-model paper's own numbers disagree with its Doom setup. The paper
trains the Doom controller at τ = 1.15 [arxiv:1809.01999 app. A.6], but the
released code's default is 1.25. The code does not show which value produced
the reported numbers [arxiv:1809.01999 §4.2].

The DERL paper disagrees with itself on the number of test tasks. Its main
comparison is described as using 7 test tasks, while the suite and the figure
caption list 8 [arxiv:2102.02202 §Environmental complexity] [arxiv:2102.02202
fig. 4]. Check the figure before quoting a count. Its scores come from MuJoCo
locomotion and manipulation tasks, so they cannot be set against the car racing
or swing-up numbers above.

The quality-diversity paper's code differs from its paper. Algorithm 1
credits each offspring's actual parent with the curiosity change. The released
code pairs offspring with parents by index, so unless a random permutation is
the identity, credit may go to a different individual [arxiv:1708.09251 alg.
1] [arxiv:1708.09251 §Paper vs code]. Its archive insertion test also differs
from the paper's in the second-neighbour test and the tolerance term
[arxiv:1708.09251 §3.1.2] [arxiv:1708.09251 §Paper vs code]. The card reads
these from the code and does not run it, so check them before implementing
either version.

## Papers

- [Quality and Diversity Optimization](/papers/1708.09251): one loop with a swappable container (grid or archive) and selection rule; adds an epsilon-dominance archive rule and curiosity selection, tested on simulated robot tasks.
- [Weight Agnostic Neural Networks](/papers/1906.04358): evolves minimal topologies that are scored over shared weight values, with no weight training during search.
- [World Models](/papers/1803.10122): the original three-part agent (VAE, MDN-RNN, linear controller) with the controller evolved by CMA-ES, plus training inside the learned model.
- [World Models Facilitate Policy Evolution](/papers/1809.01999): evolves a linear controller with CMA-ES over a VAE and LSTM model, scored on the real reward.
- [Embodied Intelligence via Learning and Evolution](/papers/2102.02202): co-evolves limb-tree bodies with PPO controllers trained from scratch for each candidate; reports that bodies evolved in harder terrain learn new tasks faster.
- [Brain Intelligence: Go Beyond Artificial Intelligence](/papers/1706.01040): position paper proposing an evolutionary S-system update over sub-networks; no method or results.