PDP: Physics-Based Character Animation via Diffusion Policy

In plain words

A simulated human-like character has to walk, recover from pushes and follow instructions without falling over. A useful way to teach it is to copy motions that a trained expert performed, but a copy-only learner drifts off course after one small slip and never learns what to do from there.

PDP first trains a set of expert controllers, each good at a small group of motions. It then lets these experts act with some random jitter, which pushes the character into slightly wrong poses. The expert's correct action for each of those wrong poses is recorded. A diffusion model, the same kind of model that generates images, is trained on these records to produce the right action from the current pose and the requested motion. Because the training data covers imperfect poses, the single learned policy can steer the character back after a slip.

TL;DR

• PDP trains one diffusion policy for physics-based character control by behaviour cloning on RL expert data §3.
• Data uses noisy states paired with clean expert actions; the noise is applied to the executed action, and the clean action is stored §3.2.
• Author-reported results cover push recovery on the Bump'em data, motion tracking on AMASS, and text-conditioned motion §5.
• No official code was found, so implementation details come from the paper alone.

Problem

Training a diffusion policy with plain behaviour cloning on clean expert trajectories fails for high-frequency, under-actuated control such as bipedal locomotion. Small errors accumulate and move the agent away from the states seen in training §1. Diffusion policies can represent several valid behaviours, which the paper says matters for perturbation recovery and text-conditioned motion §1 §2.

The paper assumes that the expert RL policies can be trained per small task set, and that motion data for the simulated character (for example AMASS or the Bump'em data) is available §3.1 §4.1.1 §4.2.1.

Method

The method has three stages §3:

1. Expert policy training §3.1. The task set is split into subsets, and one RL expert is trained per subset. The paper states that the split strategy is not critical as long as the experts produce usable state-action trajectories §3.1.
2. Stochastic data collection §3.2. Each expert is rolled out on sampled motions. The executed action is the expert action plus noise, but the stored training target is the clean expert action for the state the character actually reached. This gives noisy-state, clean-action pairs, which the authors attribute to the DASS strategy §3.2 §1. PDP extends this with short recovery episodes that start the character with a random root position and orientation offset and let it return to the reference motion over several steps §3.2. The paper also reports that pairing noisy states with noisy actions gave less robust policies §3.2.
3. Behaviour cloning with a diffusion policy §3.3. A transformer-based denoiser predicts the noise on an action sequence, conditioned on an observation history, the task information and the diffusion step §3.3.1 eq. 2. The architecture for text-conditioned applications adds a CLIP text branch with FiLM modulation §3.3.2 fig. 2.

Denoising follows a Langevin-style update with parameters α, γ and σ eq. 1. The training objective is a mean-squared error between the true noise and the predicted noise eq. 2. Action sequences are length T, and observation sequences end at the current timestep §3.3.1.

Pseudocode below follows the paper's three stages. Names are ours.

# Stage 1: one RL expert per task subset  [§3.1]
# Expert training details beyond this are not stated in the compiled text.
experts = [train_rl_expert(subset) for subset in task_subsets]

# Stage 2: noisy states, clean actions  [§3.2]
# The stored action is the expert's clean action; the executed one is noisy.
dataset = []
for expert, subset in zip(experts, task_subsets):
    # Sample a motion from this expert's subset  [§3.2]
    motion = sample_motion(subset)
    obs = reset_character(motion)
    while not done(obs):
        # Expert's clean action for the observation  [§3.2]
        clean_action = expert(obs, motion)
        # Executed action carries noise to widen the visited states  [§3.2]
        noisy_action = clean_action + noise_sample()
        dataset.append((obs, motion, clean_action))
        obs = simulate_step(noisy_action)

# Stage 3: behaviour cloning with a diffusion policy  [§3.3.1] [eq. 2]
def training_step(batch):
    # Observation sequence, task vector and clean action sequence  [§3.3.1]
    obs_seq, task, clean_seq = batch
    # Sample a diffusion step and Gaussian noise  [§3.3.1]
    step = sample_diffusion_step()
    eps = gaussian_like(clean_seq)
    # Noised input as written in eq. 2; the schedule is not stated  [eq. 2]
    noised = clean_seq + eps
    # Predict the noise from observations, task and step  [§3.3.1] [eq. 2]
    pred = eps_theta(noised, obs_seq, task, step)
    return mse(eps, pred)

# Sampling: denoise from pure noise  [§3.3.1] [eq. 1]
def act(obs_history, task, steps):
    # Start from Gaussian noise over the action horizon  [§3.3.1]
    actions = gaussian_like(horizon_shape())
    for step in reversed(range(steps)):
        # Update with alpha, gamma, sigma (values not stated)  [eq. 1]
        pred = eps_theta(actions, obs_history, task, step)
        noise = gaussian_like(actions)
        actions = alpha * (actions - gamma * pred + sigma * noise)
    # Execute the first action, then re-plan at the next step  [§5.2.1]
    return actions[0]

Key equations, as written in the paper:

\bm{A}_{t}^{k-1} = \alpha \left( \bm{A}_{t}^k - \gamma\, \epsilon_{\theta}(\bm{A}_{t}^k, \bm{O}_{t}, \bm{\tau}_t, k) + \mathcal{N}(0, \sigma^2 \bm{I}) \right) \quad [\text{eq. 1}]

\mathcal{L} = \mathrm{MSE}\left( \epsilon^k,\; \epsilon_{\theta}(\bm{A}_{t}^0 + \epsilon^k, \bm{O}_{t}, \bm{\tau}_t, k) \right) \quad [\text{eq. 2}]

\mathrm{FPC} = \frac{1}{N} \sum_{i=1}^{N} \min_{j} \sqrt{(x_i - \bar{x}_j)^2 + (y_i - \bar{y}_j)^2} \quad [\text{eq. 3}]

Per-task observation and action spaces, as stated in the experiments:

• Perturbation recovery §4.1.2: the observation is a tuple of body centre-of-mass positions, body rotations, their velocities, and a binary flag saying whether a perturbation is present. The paper says the flag does not reveal force magnitude or direction. Actions are not described in the compiled text.
• Motion tracking §4.2.2: the observation is the difference between the reference and simulated poses and velocities, plus the reference positions, rotations and finite-difference velocities. All quantities are in the character frame (origin at the root, x-axis along the facing direction, z up).
• Text-to-motion §4.3.2: the observation is the joint positions, linear velocities, joint rotations and rotational velocities in the character frame. The task vector is the CLIP embedding of the text annotation §4.3.1.

Hyperparameters

Values from the compiled paper. Anything marked "not stated" is absent from the paper and its figures and tables.

Source-only values, not in the compiled paper. They are in appendix.tex, which the main file comments out (% \input{appendix}). Treat them as the authors' own notes, not published values:

• PDP transformer: width 512 (AMASS tasks) or 256 (Bump'em); 10 or 8 layers; 8 heads; attention dropout 0.3; AdamW with learning rate 1e-4 and weight decay 1e-3; 1000 warm-up steps; cosine β schedule; 100 diffusion steps [appendix.tex].
• Expert tracking policies: Isaac Lab for AMASS and MuJoCo for Bump'em; discount 0.99; λ 0.95; clip 0.2; learning rates and network sizes differ per task [appendix.tex].

Results

Author-reported. Each setup is listed below. Numbers were not reproduced.

Sampling strategy (perturbation on Bump'em; tracking trained on KIT, tested on AMASS test) tab. 1 §5.1

* denotes PDP's sampling strategy tab. 1.

Motion tracking on full AMASS (train and test splits follow PHC) tab. 2 §5.3

Perturbation baselines on Bump'em tab. 3 §5.2

Noise level and action horizon on Bump'em (perturbation) tab. 4 §5.2.1

Headline numbers (all author-reported):

• Perturbation recovery on Bump'em: 100.0% in-distribution and 96.3% out-of-distribution success for PDP, against 91.3% OOD for the best C-VAE and 81.0% for the MLP tab. 3 §5.2.1.
• Foot placement correctness, lower is better: 2.79 for PDP and 4.97 for the best C-VAE tab. 3 §5.2.2 eq. 3.
• Motion tracking on AMASS test: 97.1% success for PDP and 96.4% for PHC tab. 2. The text reports 96.4% for PDP; see Pitfalls.
• Text-to-motion: 57.1% of 42 prompts for PDP against 11.9% for an MLP. No table is given §5.4.

Setup for the headline numbers: Bump'em uses one participant with 14 usable trajectories §4.1.1 §5.1. AMASS evaluation uses the split from PHC §4.2.1. Text-to-motion uses KIT and HumanML3D text labels §4.3.1.

Paper vs code

No official code. The paper, its project links and its bibliography contain no repository link. GitHub repository searches for the method name, for title words, and for the first author's name returned no results. Nothing was cloned.

Pitfalls

• Noise is underspecified. The stored action is the clean expert action, but the executed action is noisy §3.2. The compiled paper does not state the noise distribution, its units, or the magnitude for tracking and text-to-motion. The perturbation value 0.12 comes from tab. 4 and fig. 4 only.
• Recovery episodes are underspecified. Their root offset distribution, length and recovery schedule are not given in the compiled paper §3.2.
• The training objective is simplified. Equation 2 adds the noise directly to the clean action sequence. It has no diffusion-schedule coefficients. The β schedule and the number of steps are not stated in the compiled text. An implementer has to pick a schedule, and results depend on it.
• Sampling constants are missing. α, γ and σ in eq. 1 are not stated in the compiled text eq. 1.
• Horizon and re-planning. Training predicts several actions, but the policy executes only the first and re-plans at every simulation step. The paper reports that horizon 1 was best for perturbation and that long horizons reduced success §5.2.1 tab. 4. It says this is probably because the loss does not weight near-term actions more heavily §5.2.1.
• Perturbation sample counts disagree. Section 4.1.1 says 16 motions were recorded and 2 were dropped, which leaves 14. Section 4.1.2 says 15 motions were sampled. Section 5.1 says the clean-state strategy used 14 unique trajectories §4.1.1 §4.1.2 §5.1.
• Table 1 and the text disagree. The text gives 66.9% success for noisy-state noisy-action on perturbation §5.1. Table 1 gives 59.5% tab. 1.
• AMASS test success disagrees. The text gives PDP 96.4% on the AMASS test split §5.3. Table 2 gives PDP 97.1%, and 96.4% is PHC's value tab. 2.
• Which PDP row is which is not labelled. Table 3 has a PDP row with 2.79 FPC. Table 1 has a PDP row with 2.77 FPC for the same perturbation task. Table 4's noise 0.12 and horizon 1 row matches Table 3's values. The horizon of the Table 3 row is not labelled.
• MLP comparison is mixed. The text says the MLP outperforms PDP and PHC on tracking §5.3. In Table 2 the MLP has higher success on the test split (97.8% against 97.1%). On the test split, PDP has lower error than the MLP on every metric except velocity, where they tie tab. 2.
• Text-to-motion is weakly measured. Success is binary per prompt, over 42 prompts, and no variance is reported §5.4. The paper says that composed prompts such as "walk then jump" cannot be executed, because the policy lacks memory of the first action §5.4.
• Tracking results use different training sets. The sampling-strategy table trains on KIT and tests on the AMASS test split tab. 1. The motion-tracking table trains on the full AMASS train split tab. 2. Do not compare their numbers directly.
• Costs the abstract hides. Inference takes K times the cost of the MLP, which the authors name as their main limitation §6. Expert RL policies are trained first, and individual policies are added for motions that PHC fails on §3.1 §4.2.1.
• Source-only material. results.tex and experiments_old.tex are in the source archive but are not included in the main file. results.tex contains a DASS ablation (52%, 8%, 99%) and a handstand fine-tuning experiment; experiments_old.tex holds superseded text. They are not part of the published paper and are not used here.

Lineage

• Builds on: DASS for noisy-state, clean-action sampling §3.2 (cited as Xie et al., 2020; no arXiv id in the bibliography). Diffusion Policy for the denoising policy architecture §3.3 (Chi et al., 2023).
• Compared against: PHC, the motion-tracking baseline and expert for tracking §4.2.1 (arXiv 2305.06456). C-VAE, following the setup of Character Control VAE §5.2. MLP, a deterministic baseline with the same architecture §4.1.2 §5.2.
• Concurrent: DiffuseLoco, a diffusion policy distilled from an offline dataset for simple locomotion gaits on a real robot §2 (arXiv 2404.19264).
• Superseded by: not known from the content of this wiki.