Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads

In plain words

A two-legged robot can learn to walk in a simulator and then work on the real machine. This paper asks what happens when the robot carries something that moves on its own, such as a box on a tray, a cart it drags, a pole with swinging weights at both ends, or a jug of sloshing water. The robot is not told about the load. It only feels its own joints and body, and it learns to walk while the load is part of the simulated robot.

The authors find that training with the load attached is enough. A policy trained on one load works well on that load, and one policy trained on all loads at once gets close to the specialised ones. Starting training from a policy that already walks without a load also speeds learning up a lot. The learned walks run on the real robot, but the gap between simulation and hardware is bigger than for an unloaded robot, and the real robot is much slower than the simulated one.

TL;DR

• Proprioception-only PPO policies, with an LSTM and a periodic clock input, walk a bipedal robot under unloaded and four dynamic loads §III-A §III-B.
• Training on the specific load recovers most of the lost performance. The carrying-pole pass rate goes from 0.0014 (base) to 0.9578 (specific) tab. 2.
• A general policy trained on all models is close, and bootstrapping from a walking policy learns faster without a performance loss §IV-B §IV-C.
• Hardware transfer works, but top speed is far below simulation for every load policy §V.

Problem

Robots that carry objects need to walk with loads that move. Earlier loco-manipulation work either held the load fixed, which makes it a rigid part of the robot, or used slow, statically stable gaits §II-B. The authors want fast dynamic walking with loads and no extra load sensing §I. The base controller they start from is a reinforcement-learned walker trained without loads, which they adapt rather than redesign §III-A.

Method

Base framework §III-A. The base walker follows an earlier RL framework for Cassie [the Siekmann et al. paper, arXiv:2103.14295; not compiled here].

1. The controller is an LSTM with two 128-unit recurrent hidden layers §III-A.
2. Inputs: pelvis orientation and rotational velocity, joint positions and velocities, a sine and cosine clock whose period is the desired cycle time, and the desired forward velocity §III-A.
3. Outputs: joint position set points at 40 Hz, tracked by a PD controller with fixed gains at 2 kHz §III-A.
4. Training uses PPO with the clipped objective and gradient clipping §III-A.
5. The reward is the earlier framework's reward: tracking error on commanded velocity and orientation, smoothing terms (pelvis acceleration, torque, action difference) and foot costs that alternate swing and stance §III-A.
6. The foot cost weights come from a piecewise-linear periodic clock instead of the earlier probabilistic function. Anchor points are set by the swing ratio and cycle time, and values are interpolated linearly §III-A fig. 2.
7. Gait parameters are set per speed by a fixed table; the phase offset is 0.5 §III-A tab. 1.
8. Dynamics randomization covers joint damping, joint mass and ground friction §III-A.

Load models §III-B fig. 3. The study uses five models, one unloaded and four with loads. The load is only changed in the simulator.

• Unloaded Cassie: the base model.
• Tray Box: a freely moving box on a tray on the pelvis.
• Cart: a cart dragged behind on ropes.
• Carrying Pole: a pole on the pelvis with a swinging weight at each end.
• Water Jug: a box rigidly attached to the pelvis; sloshing is modelled as a weight on springs along each axis.

The policy input does not change with the load §III-B.

Reward and termination changes §III-B.

• All loads except Tray Box use the same reward as the base walker.
• Tray Box adds a reward term that penalises the distance between the box and the middle of the tray.
• Tray Box also ends an episode when the box's z height falls below 50 cm.

Training variants §III-B.

• Specific policies are trained from scratch or bootstrapped from the base policy.
• A general policy is trained on all five models. Each iteration samples 50,000 timesteps, split evenly across the models. It uses the base reward on every model, including Tray Box, but keeps the box-falling termination.
• The general policy is trained both from scratch and bootstrapped.

# Build the simulated robot for one load  [§III-B]
def make_env(load_model):
    # Only the simulated robot model changes; policy I/O stays the same  [§III-B]
    env = cassie_env(model=load_model)
    # Training speed commands cover 0 to 4 m/s  [§III-A]
    env.speed_command_range = (0.0, 4.0)  # (m/s)
    return env

# One PPO iteration of the general load policy  [§III-B]
def general_iteration(policy, models, timesteps=50000):
    # Split the per-iteration budget evenly across the five models  [§III-B]
    per_model = timesteps // len(models)
    batch = []
    for model in models:
        # Use the base walking reward on every model  [§III-B]
        batch += rollout(policy, make_env(model), per_model)
    # Clipped PPO update with gradient clipping  [§III-A]
    return ppo_update(policy, batch)

Equations: the source has no numbered equations; the reward is described in words §III-A.

Hyperparameters

Results

Author-reported. Setup: Cassie in simulation; each policy trained from scratch unless noted; the Unloaded column is the base model §IV tab. 2. Specific policies are evaluated only on the model they were trained on; Base and General are evaluated on every model tab. 2.

Table II, full (reproduced from the paper tab. 2):

Source: tab. 2. Speed error is reported only when pass rate is at least 50 % §IV. The table gives N/A for Base Carry Pole, whose pass rate is 0.0014.

Bootstrapped vs. scratch, full (reproduced from the paper tab. 3):

Source: tab. 3. Learning curves: fig. 4. Bootstrapped policies are described as needing on the order of 100 million fewer samples to reach the same reward §IV-C fig. 4.

Hardware, top speed in walk-in-place ramp tests §V:

Source for hardware values: §V. Hardware general-policy speeds are not stated. The text says the from-scratch general policy failed on the Carry Pole and the bootstrapped one walked §V.

Paper vs code

No official code.

Searched: the paper text, the abstract page and comments on arXiv (they name only a video link), and GitHub repository searches for "cassie sim-to-real loads", "cassie bipedal dynamic loads reinforcement learning", "cassie mujoco rl", and the authors' usernames. None returned a repository owned by the authors.

Pitfalls

• Many values are not stated. PPO hyperparameters, PD gains, the dynamics randomization ranges, the water-jug spring constants, the cart's mass and rope parameters, the reward weights, and the number of training iterations are missing. The paper defers most of the base framework to the earlier Siekmann et al. paper §III-A. Its values must come from that paper, which this wiki does not yet hold.
• Load count. The text says "5 qualitatively different loaded Cassie robot models" §III-B, but its list has five models including Unloaded, so four are loads. The conclusion says "four different dynamic loads" §VI.
• Section cross-reference. The Tray Box paragraph says "Section II A" for the reward §III-B. The base reward is in §III-A, and §II is Related Work. The section numbers here follow the LaTeX section order, because the PDF was not checked.
• Text vs table, force. The text says the Cart base policy resists 60 N more than on the unloaded model §IV-A. Table II gives 208.89 N minus 168.89 N, which is 40 N tab. 2.
• Text vs table, force gain. The text says load-specific policies resist 10 to 20 N more §IV-A. Table II gives +22.5 N for Tray Box and +126 N for Carry Pole tab. 2. The Carry Pole gain is described separately as "more than doubles", which matches 95 to 221.11 N.
• Pass-rate wording. "Over 95 % pass rate improvement" for Carry Pole §IV-A means 0.0014 to 0.9578, a gain of about 95.6 percentage points tab. 2.
• Unloaded "Specific" column. Its values equal the Base column in every row tab. 2. The text does not say whether this is the same policy.
• Speed schedule ambiguity. The gait table gives values for three speed ranges tab. 1, and the text describes linear interpolation within 1 to 3 m/s §III-A. The cycle time is taken as the inverse of the step frequency; the paper does not say this explicitly.
• Clock ramp shape. The ramp uses 20 % of swing time in total fig. 2. The anchor placement and ramp shape are not stated in the compiled text.
• Evaluation protocol gaps. The sampling of the first command, the step size for the max-speed search, and the set of push directions are not stated. The push sweep is described as every 10° §IV, without saying whether it covers a full circle.
• Max speed is not a fixed protocol. The max-speed test is described only as the fastest speed at which the policy does not fall §IV.
• Hardware is qualitative. The hardware protocol is walking in place and raising speed until a fall; no repeat count or variance is reported §V. A video link is given in the text §V.
• Hardware load setup. The paper does not describe how loads are mounted on the hardware, so the sim-to-real comparison cannot be reproduced from the text §V.
• Gap widens. The paper expects dynamics randomization alone to fall short with loads §V §VI. This is a finding, not a fix.
• Bootstrapping sample counts. The "100 million fewer samples" figure is stated in text §IV-C and read from fig. 4. The figure is not a table, so use it only as a rough reading.

Lineage

• Builds on: the unloaded Cassie walking framework of Siekmann et al. (arXiv:2103.14295, no card yet). The load work keeps its inputs, outputs, PPO training and reward, and changes only the simulated model. Tray Box adds a reward term and a termination condition §III-A §III-B.
• Compared against: the base walking policy trained without loads §IV tab. 2.
• Sim-to-real reference: the conclusion points to a more sophisticated sim-to-real method, citing the paper with bib key Mozifian2019 (arXiv:1906.00410, no card yet), which the authors expect to be needed for full simulation performance on hardware §VI.