Embodied Intelligence via Learning and Evolution

In plain words

Animals are shaped by evolution, and their bodies make learning to move easier. This paper asks whether a computer can do the same. It builds a simulator where each agent is a small creature made of a head and connected limbs, and where both the body and the brain (a neural controller) are changed over time. The body gets mutated, like a gene copy with small errors, and each new body has to learn its walking or grabbing skills from scratch before its quality is known.

The search is run as many small contests instead of generation-by-generation replacement. Four random bodies compete, the best one is copied and slightly changed, and the copy is trained. Because the training is slow, the contests run in parallel on many machines.

The main finding is that bodies evolved in harder terrain end up easier to teach. They learn a set of new tasks faster and with less energy, and across generations the bodies learn faster even though nothing in the search rewards fast learning directly. This matters because it suggests that the shape of an agent can do part of the work a learning algorithm would otherwise have to do.

TL;DR

• DERL evolves agent morphologies (limb trees with motor joints in MuJoCo) and trains each candidate's PPO controller from scratch, with asynchronous tournament selection §DERL app. A.
• Evolution in variable terrain and with manipulation produces bodies that learn eight test tasks better and faster than bodies evolved on flat ground §Environmental complexity fig. 4.
• Learning time to a fitness criterion halves within 10 generations, without any selection pressure on learning speed §Baldwin effect fig. 5.
• All performance figures are author-reported and come from one simulator; the paper's own setup uses 1152 CPUs §DERL.

Problem

Learning a controller for each candidate morphology is the expensive part of evolving bodies. Previous work either restricted the body search to small spaces (cuboid bodies, few degrees of freedom) or kept the body fixed and only tuned a controller, and in both cases learned controllers from few parameters §Introduction. The cost of evaluating fitness also meant that controllers could not easily learn from raw sensory input §Introduction.

Generational evolution, where the whole population is replaced each round, scales poorly here because every member must be trained before any further evolution can occur §DERL. Learning to walk on variable terrain is also hard: the paper cites a prior 9-DoF planar walker that needed about 10^7 agent-environment interactions even with curriculum learning and a morphology-specific reward §UNIMAL app. A, Variable terrain.

The paper's central assumption is that fitness is the reward a morphology reaches at the end of its own lifetime learning, with the controller starting from random weights in every generation app. A.

Method

DERL has two nested loops. The outer loop evolves morphologies; the inner loop trains each morphology's controller with reinforcement learning fig. 1.

1. Morphology representation. A body is a kinematic tree. The root is a sphere (the head) and the other nodes are cylinders (limbs) joined by motor-driven hinge joints. Bodies are bilaterally symmetric, so every mutation that changes a limb also changes its mirror §UNIMAL app. A.

2. Mutation. Mutations either grow limbs (at most two at a time), delete leaf limbs, change limb size or density, or change joint axis, angle limits or gear. A grown limb is kept only if the centre of mass stays in the sagittal plane and the new limb does not collide with other limbs, checked with a short simulation app. A, Grow limb(s).

3. Population and tournaments. Each run starts with P = 576 random bodies with distinct topologies. A worker draws four agents, takes the fittest as the parent, mutates a copy, trains the child, and adds it. The oldest member is removed, so the population stays at P §DERL app. A.

4. Lifetime learning. Each child is trained with PPO for 5 million agent-environment interactions. Its fitness is the average final reward §DERL app. A.

5. Controller. Each observation type (proprioceptive, exteroceptive, and task information such as goal or object position) goes through a two-layer MLP with hidden sizes 64 and 64. The encodings are concatenated, passed through a 64-unit layer, and produce a Gaussian action policy and a value estimate app. B, Policy Architecture. Hidden layers use tanh [app. B, Policy Architecture].

6. Reward for walking, flat and variable terrain. Forward velocity minus a small torque penalty app. B, Rewards:

   r_t = w_x v_x - w_c \lVert a \rVert^2 app. B, Rewards

   with w_x = 1 and w_c = 0.001. Selection uses only the forward term, so there is no explicit pressure to save energy app. B, Rewards.

7. Reward for manipulation in variable terrain. The agent is rewarded for approaching the box and for moving the box toward the goal, plus sparse bonuses app. B, Rewards. The paper's formula is:

   r_t = w_{ao} d_{ao} + w_{og} d_{og} - w_c \lVert a \rVert^2 app. B, Rewards

   with w_ao = w_og = 100 and w_c = 0.001.

8. Early termination. An episode ends when the head drops below 50% of its starting height. The authors report that without this, almost all agents fall immediately and move in a snake-like gait app. B, Rewards.

9. Evaluation. The best morphologies are retrained with new seeds and then used in eight test tasks, each learned from scratch, to measure how well a body learns new tasks §Environmental complexity app. C app. D.

# Asynchronous tournament evolution  [§DERL] [app. A]
# Input:  population of trained agents (P = 576)
# Output: population after the search budget is spent

def evolution_worker(population):
    # Repeat until the total number of searched morphologies reaches 4000
    while searched_count < SEARCH_BUDGET:
        # Draw 4 random agents and keep the fittest as the parent  [app. A]
        contestants = draw_random(population, size=4)
        parent = max(contestants, key=fitness)
        # Copy the parent body and apply one mutation  [app. A]
        child_body = mutate(parent.body)
        # The child controller starts from a random initialisation  [app. A]
        child = Agent(child_body, controller=random_init())
        # Learn with PPO; fitness is the final average reward  [app. A]
        child.fitness = train_ppo(child, iterations=5_000_000)
        # Add the child and drop the oldest member to keep P fixed  [§DERL]
        population.append(child)
        population.pop_oldest()
        searched_count += 1
    return population

# Per-step reward, manipulation in variable terrain  [app. B, Rewards]
# dist_prev is the distance at the previous step, dist the current one.
def manipulation_reward(agent_pos, box_pos, goal_pos, action):
    # Reward moving the agent toward the box, weight 100  [app. B]
    reach_gain = dist_prev(agent_pos, box_pos) - dist(agent_pos, box_pos)
    # Reward moving the box toward the goal, weight 100  [app. B]
    push_gain = dist_prev(box_pos, goal_pos) - dist(box_pos, goal_pos)
    # Sparse bonus of 10 near the box start and near the goal  [app. B]
    bonus = 10.0 * (near_start_event + near_goal_event)
    # Small torque penalty, weight 0.001  [app. B]
    torque_cost = 0.001 * sum(action ** 2)
    return 100.0 * reach_gain + 100.0 * push_gain + bonus - torque_cost

Hyperparameters

Source column: tab. 1 = UNIMAL design space table, tab. 2 = PPO table. Section citations are to the appendix unless stated.

Evolution and environment

Morphology design space

Controller and PPO

Rewards and evaluation tasks

Key equations are given in the Method section. Cost of work, which the paper uses as an energy measure, is defined as:

COW = \frac{E}{M g r} app. D, Cost of Work

Results

Author-reported. Setup: MuJoCo simulation; evolution in three environments (flat terrain FT, variable terrain VT, manipulation in variable terrain MVT); 3 evolutionary runs per environment; the top 100 morphologies per environment are used for the figures; eight test tasks, each learned from scratch with 5 seeds (median over n = 50); baselines are the first-generation bodies (Baldwin effect), or bodies evolved on flat ground (morphological intelligence) §DERL §Environmental complexity app. D. Numeric values for test-task rewards are shown only in the figures, not in the text, so they are left out here.

Paper vs code

Code: official repository agrimgupta92/derl (owner is the first author), at commit 91e077e355b27bb74fa920c708d606ff9f4d611e, dated 2021-10-09. It is later than arXiv v1. The repository's description refers to the Nature Communications version, which may differ from this text. The repository has no LICENSE file, so its code is cited here by path and line only and is not quoted.

Search for the code: the paper and abstract give no repository link, and the arXiv metadata has no comment field with one. The author's GitHub account has a repository with this title, and the description matches. Found by searching the short name "derl" on GitHub.

• Fitness window. Paper: average reward over about the last 100,000 iterations app. A. Code: the mean of the last 100 logged values, where each value is the mean over the last 10 episodes at one PPO update [code: derl/algos/ppo/ppo.py#L65] ppo.py [code: tools/evo_single_proc.py#L81-L83]. Each update is 128 steps × 32 environments = 4096 steps, so 100 updates is about 409,600 steps. If "iterations" means environment steps, the code uses about four times the paper's window. The value is also truncated to an integer evo_single_proc.py.
• Tournament sampling. Paper: a group of 4 from the population app. A. Code: random.choices draws with replacement, so one agent can appear twice in a tournament evo.py. The winner is chosen from the Pareto front over the selection criteria; repository configs use only one criterion, so the winner is the forward-fittest, with ties broken at random [code: derl/utils/evo.py#L74-L81] ft.yml. The package default lists two criteria, including a standing reward [code: derl/config.py#L449-L454].
• Replacement. Paper: oldest member removed after each child §DERL. Code: the default "aging" tournament removes nothing; the population is the newest 576 metadata files evo.py config.py. The result matches the paper. The "vanilla" option instead deletes a dominated agent evo.py and is not what the paper describes.
• Initial population. Paper: distinct random topologies app. A. Code: samples 10 × P candidates, groups them by graph isomorphism (connected components of a species graph), and keeps one per group until P is reached [code: tools/evolution.py#L123-L176].
• Mutation choice. Paper: three classes of mutation app. A. Code: seven operations (grow, delete, joint angle, DoF, gear, limb parameters, density), chosen uniformly from those allowed for the current body [code: derl/config.py#L439-L447] morphology.py.
• Seeds. Paper: the same seed for all lifetime learning in a run app. D. Code: ppo_train reseeds with the global seed for every child [code: tools/evo_single_proc.py#L41]. This matches the paper, but the per-iteration seed set in tournament_evolution evo_single_proc.py is overwritten before training.
• Manipulation distances. Paper: geodesic distance app. B. Code: Euclidean distance manipulation.py [code: derl/envs/tasks/manipulation.py#L69].
• Push reward gate. Paper: the box-to-goal term is always active app. B. Code: it is added only after the agent has reached the box [code: derl/envs/tasks/manipulation.py#L67-L70].
• Sparse bonus. Paper: bonus when the object is within 0.75 m of its initial position app. B. Code: bonus when the agent is within the success margin of the object, once per episode manipulation.py.
• Forward reward. Matches the paper: forward displacement divided by the time step, minus the torque penalty locomotion.py locomotion.py unimal.py.
• Energy. Matches the paper's torque sum, but the code truncates to an integer unimal.py.
• Cost of work, stability test, beneficial-mutation threshold. Paper gives these as analyses app. D. None of the three was found in the repository. A search for the strings "400", "stab" and "cost of" found nothing.
• Cartwheel termination. Not in the paper. It exists in code but is off by default config.py hfield.py.
• Standing reward. Not in the paper. Weight 0 by default [code: derl/config.py#L123].
• PPO early stop. Not in the paper. If the approximate KL is above 0.2, the whole update stops, not only the current minibatch [code: derl/algos/ppo/ppo.py#L162-L163].
• Advantage normalisation. Not in the paper. Advantages are normalised per update ppo.py.
• Initialisation and architecture. Not in the paper. Actor and critic use separate observation encoders model.py. The log standard deviation starts at zero model.py. Layers use orthogonal initialisation model.py.
• Learning-rate schedule. Max iterations are 5e6 // 128 // 32 = 1220, so the run takes 4,997,120 steps, not exactly 5 × 10^6 [code: tools/evolution.py#L55-L60].
• Frame skip. Not in the paper. Each action is applied for 4 simulator steps unimal.py.
• Terrain sizes. Consistent with the paper: the size field is a half-extent, so FT is 150 × 150 m, VT 100 × 100 m and MVT 60 × 40 m [code: derl/envs/modules/terrain.py#L23] terrain.py.
• Joint and gear ranges. Consistent with the paper's table [code: derl/config.py#L45-L61].
• Ball radius. Paper: 0.2 m app. C. Repository default: 0.15 m [code: derl/config.py#L251]. The manipulate-ball eval config was not checked.
• Setup. The repository requires a MuJoCo license file README.md.

Pitfalls

• Unnumbered sections and equations. The LaTeX source has no section numbers and no numbered equations. Sections are cited by title, for example §DERL, §UNIMAL, §Environmental complexity, §Baldwin effect and [§Mechanistic underpinning]. The introduction is cited as §Introduction.
• Figure numbers. Assigned in the order the figures appear in the LaTeX source (1 to 8). Check them against the compiled PDF before relying on them.
• Fitness is noisy. Each morphology is trained with one seed per run, and the paper's own robustness step is to retrain the best agents with 5 new seeds app. D. The reported per-generation gains are averages over agents, not per-agent guarantees.
• Computational cost. The full run uses 1152 CPUs and takes under 16 hours §DERL app. A. The paper does not give a single-machine time estimate.
• Hidden costs of the tournament. Every child starts from random weights, so learning is slow for each candidate app. A. The tournament result is only as good as the 5-million-step learning budget.
• Test-task count. The text says "across 7 test tasks" for the main comparison, while the suite has 8 tasks §Environmental complexity fig. 4. The figure caption lists 8 tasks.
• Obstacle design. Terrain obstacles are sampled per episode; obstacle parameter ranges are given in the appendix app. A, Variable terrain. Ranges used in code are the defaults and match the paper for steps, hills and rubble config.py.
• Repository has no license. The code can be read and cited, but it may not be copied or redistributed. Do not copy snippets from it into other projects.
• Reward gating. The manipulation reward is gated in code (push only after reaching the box), which changes learning behaviour early in an episode. Reimplementations that follow the paper's formula will learn differently app. B.

Lineage

• Builds on: PPO, arXiv:1707.06347, which the paper uses for the controller app. B. Asynchronous tournament evolution follows regularised evolution from neural architecture search, cited in the paper without an arXiv id in the source §DERL.
• Compared against: earlier evolution-and-learning simulations (cited in §Introduction), which used limited body spaces, few controller parameters or Lamarckian inheritance.
• Superseded by: not known from content already in this wiki.