Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations

In plain words

A walking robot needs to know what is under its feet. This paper teaches a bipedal robot (the Digit robot) to walk over stairs, hills, slopes and square steps. Instead of feeding the learning system a full, detailed map of the ground in front of the robot, the authors first squeeze that map into a short list of 16 numbers with a small image compressor. The walking policy then chooses where to put each foot and how fast to go, using only that short list and a simple model of the robot's body.

The authors argue that the length of this list matters. Too short and the policy misses the details that matter, such as the edge of a step. Too long and learning becomes slow, because the policy has more to sort through. They also show how to learn the same short list from depth cameras on the robot, rather than from a height map that only exists in simulation.

The result is a walking controller that learns from scratch with fewer training samples than the end-to-end approaches it compares against. The numbers come from the authors' own simulations, not from tests on the physical robot.

TL;DR

• A convolutional VAE encodes a 20x40 local terrain height map into a 16-value latent vector §II.B eq. 1 eq. 2.
• A hierarchical PPO policy takes that latent, a reduced-order state and the last action, and outputs six task-space commands; a TSID whole-body controller tracks them §III eq. 5 eq. 6.
• Author-reported: policies above 80% success on all tested terrains, at least 2x better sample efficiency than end-to-end baselines §IV.A §IV.E tab. 1.
• A distillation step learns the same latent from two depth cameras, and a history-aware variant is tested §IV.C §IV.G.
• No official code. Hardware deployment is listed as future work §V.

Problem

Blind walking policies handle slopes and hills, but they fall when the foot meets the edge of a stair or a square step §IV.E. Perceptive policies that take the whole height map as input learn slowly and reach lower reward: the full 800-element height map gives a larger network and a harder learning problem §II.A fig. 9. End-to-end perceptive work on Cassie reports roughly 60 million samples, and those approaches rely on pretrained blind policies or on heavy perception networks tab. 1 §IV.A. Using a height map from depth cameras needs a costly, noise-sensitive pipeline §IV.C.

The paper's question is how much terrain information a locomotion policy needs, and what compression level is enough §I.

Method

The method has three parts: a terrain encoder, a hierarchical policy with a reduced-order state, and a low-level controller. A distillation stage and a history-aware variant extend the first part.

1. Terrain data. In a MuJoCo environment, the authors generate sloped planes, hills, square steps and stairs (up and down) §II.A. They move the robot's base kinematically with random velocities, not with a walking controller, and record the local height map at each step. The data set has 60,000 samples per terrain type and 360,000 in total §II.A.

2. Terrain encoder (CNN-VAE). The encoder has three convolutional layers and two fully connected layers that output a mean and a spread for the latent. The latent is sampled with the reparameterisation trick §II.B eq. 1 eq. 2. The decoder maps the latent back to the height map §II.C eq. 3. The loss is a reconstruction term plus a beta-weighted KL term §II.C eq. 4. The latent size is chosen by an ablation over several sizes; m = 16 is selected §II.B fig. 5 fig. 6.

3. Reduced-order state. The policy state combines the LIP-like state of the base, the base height, the speed error, the commanded speed, the swing foot position and velocity, the latent z, and the last action §III.B eq. 5.

4. Action. The policy outputs six values: the landing position of the swing foot relative to the base (three values), the swing clearance, and two speed offsets §III.C eq. 6. The action is updated at 33 Hz §III.C.

5. Trajectory generation. The swing foot's x and y positions follow a minimum-jerk curve from touchdown to the landing target. Its z position follows a Bezier curve with five control points, whose peak is the swing clearance §III.C. Initial foot positions are set at each touchdown §III.C.

6. Low-level controller. A task-space inverse dynamics problem, solved as a quadratic program, tracks the trajectories §III.D. The constrained dynamics are eq. 7 to eq. 9. The QP objective and constraints are eq. 10 and the unnumbered constraints in §III.D. The feedback loop runs at 1 kHz §III.F. The base position is not tracked; roll, pitch and their rates are zero, and the yaw follows the heading §III.D.

7. Reward. The reward is a weighted sum of seven tracking terms and an action-smoothness term eq. 11. Each term has the form of a Gaussian in the distance between measured and desired values eq. 12 eq. 13. The weights are listed in §III.E. Foothold targets are estimated from the commanded speed, with a lateral offset of ±0.1 (unit not stated); the swing clearance target is 5 cm above the terrain §III.E.

8. RL training. PPO trains an MLP with two hidden layers of 128 tanh units §III.F. The episode is at most 300 steps (stated as 9 s) and ends early if torso pitch or roll exceed 1 rad or the base height relative to the stance foot falls below 0.4 m §III.F. The training terrains are hills, slopes, random stairs, square steps and stairs up §III.F. A fixed step duration of 0.4 s is used §IV.D.

9. Depth-camera distillation. A second encoder is trained on depth images from two D435i cameras (torso and pelvis). Its latent is pulled toward the height-map latent by an MSE loss, and the VAE loss is kept as a regulariser §IV.C. Simulated depth images are post-processed (cropping, blur, edge shadowing, clipping at 2 m, inpainting, pixel dropout, leg masking) to match the real sensor §IV.C fig. 12.

10. History-aware encoder. The CNN-VAE takes n consecutive height maps stacked as channels and either reconstructs all n (n-to-n) or only the latest (n-to-1) §II.D. The paper tests 4-frame variants with latent sizes 32, 64 and 128 §IV.G fig. 17.

Key equations (all numbered equations of the paper):

\mathbf{z}=g_{\theta}(\epsilon, \mathbf{x}) eq. 1

\mathbf{z}= \mathbf{\mu} + \mathbf{\sigma} \mathbf{\epsilon} eq. 2

\hat{\mathbf{x}} = d_{\phi}(\mathbf{z}) eq. 3

\mathcal{L} = \text{MSE}(\mathbf{x}^{(i)}, \hat{\mathbf{x}}^{(i)}) + \beta D_{\text{KL}}(q(\mathbf{z}^{(i)}|\mathbf{x}^{(i)})||p(\mathbf{z}^{(i)})) eq. 4

\mathbf{s} = (\mathbf{x}_\text{b}, h_\text{b}, \mathbf{e}_{\bar{v}}, v_x^d, v_y^d, \mathbf{p_{sw}}, \mathbf{v_{sw}}, \mathbf{z}, \mathbf{a}_{t-1}) eq. 5

\mathbf{a} = (\bar{p}_{X}, \bar{p}_{Y}, \bar{p}_{Z}, h_{\mathrm{sw}}, \dot{x}_{\mathrm{off}}, \dot{y}_{\mathrm{off}}) eq. 6

M(q)\ddot{q} + H(q,\dot{q}) = B\tau + J_c^T(q) f_c + N(q)\lambda eq. 7

N(q)\ddot{q} + \dot{N}(q,\dot{q})\dot{q} = 0 eq. 8

J_c(q)\ddot{q} + \dot{J}(q,\dot{q})\dot{q} = 0 eq. 9

\min_{\ddot{q}, f_c, \lambda} \sum_{i} \| J_i \ddot{q} + \dot{J}_i \dot{q} - \mathcal{A}_i \|_{Q_i} + \| f_c - f_c^* \|_{R_f} + \| \lambda \|_{R_\lambda} eq. 10

\mathbf{r} = \mathbf{w}^T [ r_{v_x}, r_{v_y},r_{sw_x},r_{sw_y},r_{sw_z},r_{sw_h},r_{sw_f}, r_{a}]^T eq. 11

r_{a} = \exp(-\lVert \mathbf{a}_k - \mathbf{a}_{k-1} \rVert^2) eq. 12

r_{\square} = \exp(-\lVert \square - \square^d \rVert^2) eq. 13

The unnumbered equations in §III.D are the TSID constraints (contact, loop-closure, friction cone and torque limits) and the desired acceleration \mathcal{A}_i = \mathcal{A}_i^* + K_p \log(X_{m,i}^T X_i^*) + K_d (\mathcal{V}_i^* - \mathcal{V}_{m,i}) §III.D.

Training pipeline, step by step. The paper gives no algorithm box, so this is a reading of §II, §III and §IV:

# Terrain encoder training  [§II.A] [§II.B] [eq. 1] [eq. 2] [eq. 4]
# Input:  height_maps (B, 20, 40), 360,000 samples in total  [§II.A]
# Output: encoder and decoder with latent size m = 16  [§II.B]
def train_terrain_vae(height_maps, latent_size):
    # Repeat for 40 epochs over batches of 256  [§II.B]
    for batch in batches(height_maps, 256):
        # Encoder: 3 conv layers (32, 64, 128 channels, kernel 4,
        # stride 2, padding 1), then 2 linear layers  [§II.B]
        mu, sigma = encoder(batch)             # (B, m) each
        # sigma is used as a std in eq. 2; std vs log-variance not stated
        eps = sample_standard_normal(256, latent_size)
        z = mu + sigma * eps                   # (B, m)
        # Decoder reconstructs the height map  [§II.C] [eq. 3]
        recon = decoder(z)                     # (B, 20, 40)
        # beta is not stated; prior p(z) is read as N(0, I)  [eq. 4]
        loss = mse(batch, recon) + beta * kl(mu, sigma)
        # Adam, learning rate 0.001  [§II.B]
        update(encoder, decoder, loss)
    return encoder

# Depth-camera distillation  [§IV.C]
# Teacher: height-map encoder (frozen)  [§IV.C]
# Student: encoder on two depth images from torso and pelvis cameras  [§IV.C]
def distill_step(depth_pair, height_map, teacher, student):
    # Target latent from the height map  [§IV.C]
    target_latent = teacher(height_map)        # (B, m)
    # Latent from the depth images  [§IV.C]
    student_latent = student(depth_pair)       # (B, m)
    # MSE between the two latents  [§IV.C]
    latent_loss = mse(student_latent, target_latent)
    # VAE regularisation term  [§IV.C] [eq. 4]
    # Which input it reconstructs, and the weighting, are not stated
    vae_loss = vae_objective(student, depth_pair)
    return vae_loss + latent_loss

# One high-level step at 33 Hz; low level tracks at 1 kHz  [§III.C] [§III.F]
def high_level_step(policy, state, prev_action, latent):
    # State: base LIP state, speed error, swing foot, latent, last action
    # [eq. 5]
    obs = build_observation(state, latent, prev_action)
    # MLP, 2 hidden layers of 128 tanh units, inputs normalised  [§III.F]
    action = policy(normalise(obs))            # 6 values  [eq. 6]
    # Landing target, swing clearance and speed offset  [§III.C]
    targets = action_to_task_space(action)
    # TSID tracks the targets  [§III.D] [eq. 7] [eq. 8] [eq. 9] [eq. 10]
    run_low_level(targets)
    return action

# Per-step reward  [eq. 11] [eq. 12] [eq. 13]
def reward(measured, desired, action, prev_action, weights):
    # Gaussian-style terms exp(-||measured - desired||^2)  [eq. 13]
    # Smoothness term exp(-||a_k - a_{k-1}||^2)  [eq. 12]
    # Weight order: vx, vy, swx, swy, swz, swh, swf, a  [§III.E]
    # Forms of the swing-height and swing-force terms are not given
    terms = reward_terms(measured, desired, action, prev_action)
    return dot(weights, terms)                 # scalar  [eq. 11]

Pseudocode is written from the text. The paper's own algorithm listing does not exist.

Hyperparameters

Terrain parameter ranges used for data collection, training and evaluation tab. 2:

The "Ext Limits" column is the out-of-distribution range §IV.F tab. 2.

Results

Author-reported. Setup: Digit robot in the authors' simulation, with terrains from tab. 2; 200 or more Monte Carlo trials per terrain type with a confidence interval of at least 95% §IV.E. Sample counts compare training from scratch against published end-to-end baselines tab. 1.

Comparison table as published tab. 1:

The paper's text says "at least 2x increase in sample efficiency" §IV.A.

Paper vs code

No official code. I checked the paper's text, abstract and metadata for a repository link and found none (the only external link is a video, §IV.D). I searched GitHub for "terrain aware bipedal locomotion", "digit bipedal rl" and "castillo bipedal locomotion digit", which found nothing relevant. The account of an author, Ayonga Hereid (ayonga), has no terrain, VAE or RL repository (its cyber-frost repository holds only a README and licence file). Accounts named gacastillo showed no matching public repositories. No code discrepancies can be checked.

Pitfalls

• Units typo in the grid resolution. §II.A says "5 cm^2". The matrix size, the caption and the surrounding text all give 5 cm per cell. Use 5 cm.
• Sigma: std or variance? §II.B calls sigma the "variance" and then uses it as a multiplier in z = mu + sigma * eps eq. 2. Whether the encoder outputs a standard deviation or a log-variance is not stated. Choose the standard deviation reading, which the equation implies, and check the reconstruction loss.
• VAE beta and reductions. The beta value is not stated eq. 4. Neither is whether the MSE and KL are summed or averaged over elements. Their relative scale decides how much the latent is regularised, and the paper's latent-size ablation depends on it.
• Prior. The prior p(z) is described as "parameterised by the Gaussian epsilon" §II.C. Reading it as N(0, I) is inferred from the epsilon definition.
• Latent-size choice. Loss curves favour 32 to 64 fig. 5 and the text says "64 and 32 clearly balance" §II.B. The paper selects 16 on the basis of the t-SNE plot fig. 6 and the RL ablation §IV.A. No reconstruction numbers are given in the text.
• Table 1 contradicts the text on baseline inputs. Table 1 lists [vanmarum2023learning] with a full height map and [duan2023learning] with sampled heights. The text says [duan2023learning] uses a full height map at 5 cm §IV.B and [vanmarum2023learning] samples 318 points §IV.B. Check the text and the table before you cite either.
• Gadde sample count. Table 1 prints "60 x 1.4^9". Read literally, 1.4^9 is about 20.7, which would give about 1.2 x 10^9 samples. This is probably a typesetting error; do not use the number.
• "Hardware validation" is sensor-only. The abstract's "preliminary hardware validation" compares simulated and real depth images fig. 12. No policy ran on hardware, and the conclusion lists hardware deployment as future work §V.
• Simulator for RL training is not named. MuJoCo is named for terrain generation §II.A. The simulator used for RL training is not. The Agility Robotics simulator is used only for the single-policy test §IV.F.
• Stairs-down terrain. Data collection includes stairs down §II.A, but the RL terrain list omits it §III.F. Check which terrains a reproduction should train on.
• Out-of-distribution range. The text says upper bounds rose by 20% §IV.F. Table 2's extended limits do not follow that rule: several upper limits rise by 10% (square step size 0.5 to 0.55, slope 0.2 to 0.22) and some lower limits move by 50% (stair step height 0.05 to 0.025). The table is more specific; check which one a reproduction needs.
• Hills octaves. Octaves are sampled from a normal distribution (mean 10, std 2) but the count must be an integer. Rounding is not stated.
• Reward terms. The swing-height and swing-force terms have no explicit formula. Only the generic exp form eq. 13 is given. The reward for x and y foothold uses T, which is not given a value. Reading T as the 0.4 s step duration §IV.D is inferred.
• Sign of the lateral offset. The foothold offset is ±0.1 m. The rule for which foot gets which sign is not stated.
• Samples per PPO iteration. "56,000 samples with six epochs per iteration" does not say whether 56,000 is the batch collected per iteration or the total. Assuming a per-iteration batch is an inference.
• Speed command range. The speed is sampled per episode §III.E, but the range is not given. The only stated command is 0.5 m/s for evaluation §IV.D.
• Initial state spread. The initial state is drawn from a normal distribution around the double-support pose §III.F, with no spread given.
• Policy state dimension. The state in eq. 5 has 37 entries if m = 16 and the action has 6 entries. This count is derived from eq. 5 and eq. 6, not stated in the paper.
• History-aware policy input. The paper does not say explicitly whether the history latent replaces z in the state eq. 5. Reading it that way is an inference.
• Velocity tracking is a soft constraint. The paper says tracking error is larger on hard terrains and that the policy trades speed for stability §IV.D. Plan for tracking error, not perfect speed matching.
• Sim-to-sim only. The AR simulator test covers policies A and C only §IV.F. No real-robot success rate is reported.

Lineage

• Builds on: the authors' earlier hierarchical framework, which is blind and sample-efficient §I (castillo2023template in the bibliography); reduced-order model walking speed control §III.B (gong2021onestep); and the authors' work on data-driven latent spaces §I (castillo2023datadriven).
• Compared against: [vanmarum2023learning], [duan2023learning] and [gadde2025learning] for end-to-end and teacher-student perceptive locomotion tab. 1 §IV.A; the blind policy and a full-height-map variant of their own method §IV.E.
• Related work: [miki2022learning] for latent terrain representation on quadrupeds §I; [long2025learning], [he2025attention] and [zhang2025distillationppo] are cited as recent perceptive locomotion methods that the paper does not compare with §IV.A.
• Superseded by: not known.