Learning-based legged locomotion; state of the art and future perspectives

In plain words

Legged robots, such as four-legged dogs and two-legged humanoids, are hard to control. Their bodies are unstable, they touch the ground at changing points, and their models are never exact. For decades, engineers wrote control rules or solved heavy optimisation problems at every step. In recent years, learning from trial and error has become a strong alternative. The training runs in a physics simulator, and the learned controller is then moved onto the real robot.

This survey is a guide to that work. It explains how the field developed, the choices a designer must make (what the robot senses, what it is rewarded for, how its commands are shaped), the training methods that make learning work, and how to close the gap between simulation and reality. It also covers how learning can be combined with classic control, and what is still open, such as walking on two legs and the risks of deploying these robots.

TL;DR

A survey of learning-based legged locomotion, focused on quadrupeds, with a shorter part on bipeds and humanoids. It covers the reinforcement-learning setup, observation and reward design, curriculum, hierarchical and privileged (teacher-student) training, sim-to-real methods, hybrids with model-based control, and open problems. It contains no new method and no experiments of its own, so use it as a map and for its design pitfalls, not as a recipe.

Problem

Legged robots are nonlinear, hybrid and inherently unstable systems §1. The survey states that model-based optimal control handles multi-contact motion, but real-time solution of high-dimensional non-convex problems is costly, uncertainty in contact is hard to handle, and separate estimation and control modules are used §1.3. Learning moves most of the computation offline into simulation. At run time the policy is one network evaluation per control step §1.3. Scope: the survey concentrates on locomotion, with manipulation covered only where it meets locomotion §1.

Method

This is a survey, so there is no algorithm of its own. The section numbers below follow the paper.

1. Background §1 §2. Hardware is dominated by proprioceptive actuators, which give high torque density and backdrivability §1.1. The paper names the Unitree A1 and Go1/Go2 as popular research platforms §1.1. Learning is cast as a Markov decision process: states, actions, transitions, a reward, and a discount factor. The goal is the policy that maximises expected return §2.1 eq. 1. Common algorithms are summarised in tab. 1. The on-policy methods TRPO and PPO are the usual choice for locomotion §2.2. Behaviour cloning and GAIL cover imitation §2.3.
2. MDP components for locomotion §3. Simulators model rigid contact with complementarity conditions and friction cones §3.1.1. Observations split into proprioception, which needs state estimation and a short history §3.2.1, exteroception such as elevation maps, depth or RGB §3.2.2, and task inputs such as velocity or pose commands §3.2.3. The reward is usually a weighted sum of terms §3.3 tab. 2, either hand-shaped §3.3.1 or imitative, from motion capture or model-based references §3.3.2. The action is usually a joint position target sent to a PD controller §3.4.1. Structured actions include task-space control, residual RL and oscillator primitives §3.4.2.
3. Learning frameworks §4. End-to-end RL with on-policy methods is the baseline §4.1 fig. 5. Curriculum learning raises difficulty through terrain parameters or disturbances §4.2. Hierarchical learning splits a high-level and a low-level policy §4.3. Privileged learning trains a teacher on simulator-only information and then a student on measurements the robot actually has §4.4.
4. Sim-to-real §5. Good system design means avoiding simulator exploits and choosing gains and observations that transfer §5.1. System identification fits actuator and contact models §5.2. Domain randomisation varies dynamics during training §5.3. Domain adaptation infers the environment, explicitly or as a latent code, and conditions the policy on it §5.4 fig. 6.
5. Combining control and learning §6. Four patterns: learning parameters of a model-based controller §6.1; a learned high-level policy that sets commands for it §6.2; learning to cache expensive MPC solutions §6.3; and model-based controllers that guide RL exploration §6.4.
6. Bipeds and open problems §7 §8. Biped and humanoid learning is traced from early RL and the DARPA-era optimal-control work §7. Open problems: unsupervised skill discovery §8.1, differentiable simulators §8.2, challenging terrain §8.3, safety §8.4, wheeled-legged robots §8.5, loco-manipulation §8.6 and foundation models §8.7. §9 covers societal impact and §10 concludes.

The privileged-learning recipe in §4.4 can be sketched as follows. The paper gives no imitation loss and no history length, so those lines are marked as not stated.

# Privileged learning: teacher, then student  [§4.4]
# Input:  simulator; measurement history from the robot
# Output: student policy for the real robot

def train_teacher(simulator):
    # Teacher observes noiseless states and privileged terms, e.g. friction
    # coefficient or terrain information  [§4.4]
    # Trained in simulation; the RL algorithm is not fixed here  [§4.4] [§4.1]
    teacher = train_policy(simulator, teacher_observations)
    return teacher

def train_student(simulator, teacher):
    # Student sees only measurements the robot has, as a history  [§4.4]
    # Sequence model such as an RNN or TCN  [§4.4]
    # History length: not stated.  Imitation loss: not stated.
    student = sequence_model()
    student = imitate(student, teacher, simulator)
    return student

PD joint-position action, as defined in the paper §3.4.1:

# PD joint-position action  [§3.4.1]
# Input:  policy action (target joint angle); measured theta, theta_dot
# Output: joint torque tau

def pd_torque(action, theta, theta_dot, kp, kd):
    # Gains kp and kd are not given numerically in the paper
    # Desired joint velocity is zero  [§3.4.1]
    tau = kp * (action - theta) - kd * theta_dot
    return tau

Hyperparameters

Not applicable: the survey reports no experiments of its own. The numerical settings it mentions are listed below, all taken from other works. Anything marked "not stated" is missing from the paper.

Results

No results of its own. The survey quotes these figures from cited works, and they are not checked here:

The paper gives no benchmark numbers, so it cannot rank methods.

Reward terms commonly used for quadruped locomotion tab. 2. "*" marks a target quantity. The paper writes the reward as a weighted sum of such terms, and each term is passed through a norm or kernel:

Paper vs code

No official code. The LaTeX source has no repository or project-page link, and the abstract and arXiv metadata have none either. A GitHub repository search for the title words returned no results. The survey itself links third-party codebases, for example for PyBullet, Isaac Sim and PPO §4.5. They are not the authors' code and are not used here. No discrepancies found, because there is nothing to compare.

Pitfalls

• Survey, not a method. Nothing here can be implemented as stated. Read the cited work before building anything from a claim.
• Reward notation is inconsistent. The general reward is written as r(s_t, s_{t+1}, s_t) = Σ_i c_i r_i(s_t, s_{t+1}, a_t) §3.3. The first argument list repeats s_t, and the inner terms take a_t. Read it as r(s_t, a_t, s_{t+1}) and treat the printed form as a typo.
• Discount missing from eq. 1. The MDP tuple includes a discount factor γ §2.1, but the return in eq. 1 sums over the horizon with no γ. The paper does not say which form was used in practice.
• States and observations share one symbol. The paper says it uses s for both, which is a "slight abuse of notation" §2.1. Be explicit about which one the policy receives.
• Torque-form term definitions are loose. Joint mechanical power is written as Σ τ · q̇ with no sign or magnitude handling tab. 2. Norms and kernels are left to the designer.
• Many gains and ranges are missing. PD gains, reward coefficients, randomisation ranges, curriculum bounds and history length are all unstated §3.3.1 §3.2.1 §4.2 §5.3. Expect to tune them yourself.
• Simulator popularity is a rough count. fig. 4 comes from keyword searches on Google Scholar. It shows usage, not quality.
• Dated snapshot. The paper is arXiv v2 (updated 2024-11-22). Its humanoid and foundation-model parts are dated, and the survey's own list of recent humanoid work §7 is not complete. The source also contains older drafts in an old_files folder; the card uses only the main manuscript.

Lineage

• Builds on: nothing of its own. It places itself against related surveys: deep RL for quadrupedal locomotion (Zhang, He and Wang, 2022) §1.4, deep RL for bipedal locomotion (arXiv:2401.16889) §1.4, and optimisation-based control for dynamic legged robots (Wensing et al., 2023) §1.4.
• Compared against: not applicable.
• Superseded by: not known from the content of this wiki.