Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots

In plain words

A humanoid robot can stand, walk, copy a person's arm movements or follow a joystick, but usually each of these is handled by a separate controller. This paper asks for one controller that takes a single kind of command for all of them. The command is a target motion written as a sequence of body poses, where the parts the user does not care about are left blank. The controller fills in the blanks itself. An arm-only motion becomes "move the arms and keep the legs balanced", and a joystick input becomes "walk this way and let the legs do the work".

The controller is learned by trial and error in a physics simulator, with randomised settings so that it does not depend on one exact model of the robot. It was then run on a real Digit V3 humanoid. The authors report that it mostly works on hardware, with more falls and larger tracking errors than in simulation.

TL;DR

• The Masked Humanoid Controller (MHC) is one learned policy for the Digit V3 humanoid. It takes the robot's state and a masked motion directive, and outputs PD setpoints for all 20 actuated joints §3 §4.2.
• Unspecified joints are masked to zero and the policy fills them in. Joystick walking, upper-body imitation and full motion copying all use this one interface §3 §4.4.
• Training uses PPO in MuJoCo, a three-stage curriculum and domain randomization §4 §4.4 §4.5.
• Author-reported: 0.450% failures in simulated mimicry, against 5% on hardware for 20 motions. Walking and standing are competitive with the Stand-and-Walk baseline, though one standing case is worse tab. 2 tab. 3 §5.6.
• No code is released for this paper.

Problem

Whole-body humanoid controllers are usually built for one skill at a time, such as standing, walking or mimicry. Each one uses its own input format, so combining them, or adding a new motion source such as video or motion capture, needs more controllers or an interpreter §1 §2. The paper also points out that policies trained in simulation often transfer poorly to hardware §1.

The problem is stated as a learning task. Given a dataset of humanoid motions, learn a controller that matches target motion directives that may be only partly specified. For example, the controller should accept upper-body joint trajectories alone, or torso velocity alone, and fill in the lower body §3.

Method

Formulation. A motion is a sequence of poses over H steps. Each pose holds joint angles, joint angular velocities, root planar velocity, turn rate and base orientation with height §3. A motion directive d = (\hat{q}_{1:H}, I_{1:H}) pairs a masked motion with binary masks. Masked entries of \hat{q} are set to zero, and each mask bit says which dimensions are constrained §3. At each step the policy \pi reads the proprioceptive state s_t and the current directive step, and outputs a_t = \pi(s_t, d), the PD setpoints for all motors §3.

s_t = (r\theta_t, \dot{r\theta}_t, \omega_t) §3

Reference motions. The training set mixes motion capture (AMASS, Reallusion), video demonstrations and optimized spring-mass trajectories for Digit §4.1. Each frame is retargeted to Digit with inverse kinematics. The problem is a nonlinear program built with Drake's IK module and solved with SNOPT. Stance foot contact and Digit's closed leg chain are task-space constraints, and upper-body targets are soft costs. Frames with no feasible solution are dropped, and the remaining states are linearly interpolated in time §4.1.

Network. A single feed-forward layer encodes the directive step into a 160-dimensional embedding. This is concatenated with the state and passed to an LSTM with two 64-unit recurrent layers. A linear head outputs one offset per actuated joint. The action adds this offset to the actuated joint values of the directive step. For masked joints those values are zero, so the offset is the full PD setpoint §4.2. The policy runs at 50 Hz and feeds a 2 kHz PD controller §4.2.

# Policy step: directive step -> PD setpoints  [§4.2]
# Input:  proprio (proprioceptive state), directive_step (masked joints + mask)
# Output: pd_setpoints (20 actuated joints), next hidden state

def mhc_action(proprio, directive_step, hidden_state):
    # Encode the masked directive step into a 160-d embedding  [§4.2]
    embedding = encoder(directive_step)              # (160,)
    # Concatenate with the proprioceptive state  [§4.2]
    features = concat(proprio, embedding)
    # Run the two-layer LSTM, 64 units each  [§4.2]
    features, hidden_state = lstm(features, hidden_state)
    # Linear head: one offset per actuated joint  [§4.2]
    offset = linear(features)                        # (20,)
    # Masked joints are zero, so the offset becomes the setpoint  [§4.2]
    pd_setpoints = directive_step.joints + offset    # (20,)
    return pd_setpoints, hidden_state

Episodes. Each episode starts with the robot standing. The sampler draws a command window length w and a directive d from the distribution of the current curriculum stage. The directive is cycled until w steps have passed. Sampling repeats until the robot falls or the stage's maximum length is reached. Torso perturbations are applied during the episode, following the stage's distribution §4.3.

# Training episode: switch directives until a fall or the length limit  [§4.3]
# Input:  stage (curriculum stage), policy
# Output: list of (state, action, reward) transitions

def run_episode(stage, policy):
    # Every episode starts with the robot standing  [§4.3]
    state = reset_to_standing()
    transitions = []
    elapsed = 0
    # Repeat until a fall or the stage's maximum episode length  [§4.3] [§4.4]
    while not has_fallen(state) and elapsed < max_length(stage):
        # Draw a command window w and a directive d for this stage  [§4.3]
        window = sample_window(stage)
        directive = sample_directive(stage)
        # Cycle through the directive for w policy steps  [§4.3]
        for step in range(window):
            # Apply torso perturbations for this stage  [§4.3] [§4.4]
            apply_perturbation(stage)
            action = policy.act(state, directive.at(step))
            state, reward = env_step(action, directive.at(step))
            transitions.append((state, action, reward))
            elapsed += 1
    return transitions

Curriculum. Training has three stages §4.4.

• Stage 1, locomotion. Only locomotion directives are used, which mask everything except root velocity, turn rate and base orientation. Single torso pushes of 80 to 800 N hit one policy step each, with probability 1% per step. Command windows are 40 to 100 policy steps and episodes last 300 policy steps (6 s) §4.4.
• Stage 2, stability. Continuous torso pushes of 20 to 150 N last 20 to 50 policy steps. Episodes grow to 800 steps (16 s) and command windows to 100 to 400 steps §4.4.
• Stage 3, whole-body directives. Episodes stay at 800 steps and windows at 100 to 400. The directive can be fully specified (whole-body imitation), or one of three partial forms: locomotion only (joystick), upper body only while standing (VR or video imitation), or upper body plus locomotion §4.4. The paper does not state the perturbations for this stage.

Domain randomization. Dynamics are randomised in all stages. The randomised quantities are joint damping, link masses, centre-of-mass positions, encoder noise and ground friction. The ranges are the same as those of prior Digit locomotion work §4.5 (the numerical ranges are not given here).

Reward. The reward starts from the Stand-and-Walk (SaW) controller of van Marum et al.: a task term for command following, a style term for natural posture, and a regularization term for smooth motion and low torque. The definitions of these three terms are deferred to that paper §4.6. This paper adds a tracking term over the masked joints §4.6, unnumbered display:

r_{track} = \exp\left(-1.5 \cdot \lVert (\theta - \hat{\theta}_i) \cdot I_i \rVert\right) §4.6, unnumbered display

Here \theta is the current joint vector, \hat{\theta}_i the target and I_i the mask. If the lower body is masked, the reward is task + style + regularization

• tracking. If the lower body is unmasked, the style term is dropped, because its posture preferences can conflict with the lower-body targets §4.6.

# Per-step reward: SaW terms plus masked tracking  [§4.6]
# Input:  state, directive_step (target joints), mask (per joint)
# Output: scalar reward

def reward(state, directive_step, mask):
    # Task and regularization terms, defined in the SaW paper  [§4.6]
    r_task = task_term(state)
    r_reg = regularization_term(state)
    # Tracking error over masked joints only, weight 1.5  [§4.6]
    error = norm((joints(state) - directive_step.joints) * mask)
    r_track = exp(-1.5 * error)
    # Drop the style term when the lower body is unmasked  [§4.6]
    if lower_body_unmasked(mask):
        return r_task + r_reg + r_track
    # Style term applies when the lower body is masked  [§4.6]
    r_style = style_term(state)
    return r_task + r_style + r_reg + r_track

Hyperparameters

Locomotion ranges (tab. 1), for reference: v_x in [-0.5, 2.0] m/s, v_y in [-0.5, 0.5] m/s, turn rate in [-0.5, 0.5] rad/s, torso x and y orientation in [-0.314, 0.314] rad, torso height in [0.5, 1.0] m, and a standing bit in {0, 1} tab. 1.

Results

Author-reported. Setup: MuJoCo simulation of the Digit V3 model for the baseline, generalization and curriculum tables, and the real Digit V3 for the sim-to-real table. Baselines are described in §5.3. Figures are described, not reproduced: fig. 1 shows the controller's inputs and outputs, and fig. 2 shows three real-world demonstrations.

Walking and standing against the SaW controller, trained on setA §5.3:

Mimicry against the OffsetBlind and OffsetAware baselines, trained on setA §5.3:

Sim-to-real, same policy, setA, 20 motions with 3 runs each §5.6:

Generalization across training sets §5.4. MEPE and root drift in metres:

Curriculum ablation, mimicry averaged over partial and whole-body directives on setA §5.5:

The headline claims are a 0.450% simulated failure rate on mimicry tab. 3, about 5% failures over 20 motions on hardware (3 runs each) §5.6, and better walking and mimicry than the no-curriculum variant tab. 5.

Paper vs code

No official code. The paper gives no code link. Its project page (masked-humanoid.github.io/mhc) links the paper, a video and a social-media post, and no repository. The GitHub repository search for "masked humanoid controller" and "masked humanoid" found osudrl/masked-humanoid-controller. Its README describes the earlier ECCV 2024 system of Shrestha et al. on SMPL and Reallusion models, built on Isaac Gym, and not this Digit V3 MuJoCo controller. Its license is an NVIDIA license limited to non-commercial use, so it is not used here. Only the GitHub search was run; no organisation-level search was done. The repository was not used for any figures, numbers or code.

Pitfalls

• Section numbers. The source sets secnumdepth to 0, so the compiled paper prints no section numbers. The § numbers on this card are ordinal counts of the LaTeX sections. The equations in §3 and §4.6 are unnumbered. The tables and figures are numbered in order of appearance.
• Reward terms are deferred. The task, style and regularization terms, and their weights, are defined only in the SaW paper §4.6. The norm in r_{track} is not specified §4.6. An implementer needs the SaW reward first.
• Orientation representation. The proprioceptive state uses the torso orientation as a quaternion §3. The directive pose uses Euler angles for the base orientation §3. How the two are converted is not stated.
• Mask convention. Masked joints are zero in the directive, and the output head adds its offset to the directive values. For masked joints the offset is therefore the full setpoint §4.2. Mixing the two conventions would produce wrong setpoints.
• Directive cycling. "The MHC cycles through d until reaching w steps" §4.3 does not say what happens when the window and the directive length differ. Wrapping around is an assumption.
• Episode length in Stage 1. "Training lasts for 300 policy steps" §4.4 could mean episode length or total run length. The card reads it as the episode length, which the 6 s figure supports.
• Stage 3 unspecified. Stage 3 gives no perturbation schedule and no mix of directive types §4.4. The sampling distribution is referred to but not given §4.3.
• Timing. 300 steps at 50 Hz equals 6 s, and 800 steps equals 16 s §4.4 §4.2, so the policy step is 20 ms.
• Retargeting limits. Digit's one-DoF rotational joints cannot express the spherical joints of motion capture, so some retargeted motions are only approximate §4.1. Infeasible frames are dropped, and the remaining frames are interpolated §4.1.
• Text overstates one table. The text says MHC_A "excels on setA" §5.4, but in tab. 4 MHC_AB is best on setA for both MEPE and root drift (0.09 against 0.10, and 0.09 against 0.14).
• Standing is mixed. The text calls the MHC "on par" with SaW §5.3. In tab. 2, SaW drifts less under 50 N (0.109 against 0.256), while MHC drifts less under 100 N and when walking. Check the rows before quoting the claim.
• Curriculum comparison is not compute-matched. MHC_NC trained until the loss plateaued, for longer than MHC §5.5. Total training steps are not reported for either.
• No variance. The tables give single numbers, and the text reports no error bars or seed counts §5. The hardware result rests on 3 runs per motion over 20 motions §5.6.
• Sim-to-real gap. The authors report higher errors on hardware than in simulation §5.6. The Future Work section lists sim-to-real gaps for wide foot placements and single-foot stances §6.
• Simulator. Training uses MuJoCo, and the paper says the Digit V3 model is not yet supported in Isaac Sim §6. Results depend on the MuJoCo model, and the paper does not report transfer to other simulators.
• Compute not stated. Training hours, GPUs, number of environments and total simulation steps are not given.
• Real-robot setup. Real-robot trials are not described in detail beyond motions and repetitions §5.6. Fig. 2 shows examples, not a protocol fig. 2.

Lineage

• Builds on: the SaW Stand-and-Walk controller of van Marum et al. for the reward structure, the domain-randomization ranges and the standing test §2 §4.5 §4.6 §5.2. Its arXiv id is not given in this paper, and it has no card here.
• Related prior work: an earlier MHC by Shrestha et al. §2 (cited there as shrestha2024mhc). Its arXiv id is not given in this paper.
• Related work cited: Versatile Motion Priors §2 and HOVER §2.
• Compared against: SaW tab. 2; OffsetBlind and OffsetAware baselines tab. 3; MHC_NC, the no-curriculum variant tab. 5.
• Superseded by: not known from content in this wiki.