Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion

In plain words

A walking robot can be driven by a hand-written controller that plans over a short time window and picks the forces on its feet. That controller is too heavy to run in real time on the robot itself, and it struggles when the ground is bumpy or slippery, because its built-in model of the robot is wrong there. Learning-based controllers adapt to such ground, but they usually need many reward terms and careful tuning before the walk looks smooth.

The idea here is to use both. The hand-written controller first runs in simulation and produces examples of good behaviour. A small neural network learns to copy those examples, and it is also corrected on the mistakes it makes itself. Reinforcement learning then fine-tunes the copy on harder ground, with small exploration settings so that it keeps the original style. The result is a light network that walks more symmetrically than an agent trained from scratch with a simple reward, and that needs far less computation than the original controller.

TL;DR

• The paper builds a model predictive controller (MPC) from differential dynamic programming (DDP) and Raibert foot placement, and uses it as an expert §III-A.
• An MLP (512, 256, 64 units) is trained by imitation with DAgger, then finetuned with PPO on harder terrains with limited exploration §III-B §III-C.
• In simulation and on hardware, the author-reported results favour the finetuned policy over the MPC expert on robustness and over vanilla RL on tracking and gait symmetry §IV-B §IV-C §IV-D.
• Hardware results are for the Mini Cheetah robot, with 5 to 10 trials per setting, so small differences may not be meaningful §IV-A §IV-C.
• No official code was found; the card is based on the LaTeX source only.

Problem

Model-based MPC gives an interpretable, robust controller for quadrupeds, but it depends on a predefined model: a fixed contact sequence and a reference swing-leg trajectory. These limit it on rough and dynamic terrain that needs quick changes of foothold §I §II. Raibert-style foot placement cannot guarantee stability when a foot slips a lot or gets stuck §II.

Deep RL can handle such terrain without a model, but typically needs up to ten reward terms to get smooth, symmetric, periodic gaits, and can otherwise learn jerky or asymmetric gaits that do not transfer to hardware §I §II. The paper's aim is to pre-train a policy from an MPC expert and then fine-tune it, so that less reward shaping is needed and the RL stage keeps the expert's style §I.

Method

The framework has three stages, shown in fig. 2 §III:

1. Stage 0, expert. An expert policy \pi^E is built from DDP with friction-pyramid constraints for stance legs, and Raibert foot placement with task-space impedance for swing legs. A reference generator turns the velocity command (linear x and y, yaw rate) into a reference state trajectory §III-A eq. 1 eq. 2 eq. 3. The expert runs only in simulation, to create training data, because the formulation cannot run in real time on hardware §III-A.
2. Stage 1, imitation. The expert is cloned into a neural policy \pi^I by DAgger: roll out the current policy, label the visited states with the expert's action, add them to the dataset, and retrain §III-B eq. 6.
3. Stage 2, reinforcement learning. The cloned policy is finetuned with PPO and an asymmetric actor-critic on a terrain curriculum, with a minimal reward and limited exploration §III-C.

Expert details §III-A:

• The MPC state is body Z-Y-X Euler angles, position, angular velocity and linear velocity; the control is the four ground reaction forces eq. 1.
• The cost is a weighted sum of state-tracking and control terms over the horizon eq. 2. Weight matrices Q and R are not stated.
• The control step is 0.01 s and the horizon is 26 steps §III-A.
• Swing legs use task-space PD torques, \tau_i = J_i^T[K_p^t(p - \bar p) + K_d^t(v - \bar v)], with K_p^t = 150 and K_d^t = 5 eq. 3. The foot target comes from the Raibert heuristic, and the foot path is a cubic Bézier curve from the current foot to that target.
• The gait is a fixed swing of 0.13 s and stance of 0.13 s, with no flight phase §III-A.

Action conversion §III-A: the expert gives torques, but RL and imitation policies are more usual with joint targets, and torque policies vibrate on hardware. The paper therefore assumes a joint PD law, \tau = K_p^j(q - \bar q) - K_d^j \dot q with K_p^j = 17 and K_d^j = 0.4 eq. 4, and inverts it to get the target angle \bar q = \frac{1}{K_p^j}(K_p^j q - K_d^j \dot q - \tau) eq. 5. The policy output is an offset from the initial pose, \bar q = q_{init} + \sigma a §III-B.

Stage 1, imitation §III-B: the student is an MLP with three hidden layers (512, 256, 64 units) and leaky ReLU. The objective is a squared-error loss between expert and student outputs over the dataset, summed over D eq. 6. The input contains body orientation, body angular velocity, joint angles and velocities, the previous action, a joint angle and velocity history, the predefined contact sequence and phase, and the user command. Body linear velocity is deliberately left out, because the authors found a noisy state estimator can hurt hardware performance §III-B. Imitation uses flat terrain only, but the joint history is kept because RL finetuning needs it §III-B.

# Stage 1: DAgger imitation of the expert  [§III-B] [eq. 6]
# Expert: DDP MPC (simulation only). Student: MLP 512-256-64.
def dagger(expert, student, env, num_iterations):
    # Start with an empty dataset  [§III-B]
    dataset = []
    for iteration in range(num_iterations):
        # Roll out 10 trajectories with the current student  [§III-B]
        rollouts = collect_rollouts(student, env, trajectories=10)
        for trajectory in rollouts:
            for state in trajectory:
                # End a rollout on non-foot contact or a 70 deg flip  [§III-B]
                if terminated(state):
                    break
                # Label the visited state with the expert's action  [§III-B]
                dataset.append((observe(state), expert_action(state)))
        # Retrain on everything collected so far, 10 epochs  [§III-B] [eq. 6]
        student = fit_squared_error(student, dataset, epochs=10)
    return student

Stage 2, RL finetuning §III-C: PPO is used with an asymmetric actor-critic. The actor is the cloned MLP, with a learnable standard deviation acting as a Gaussian noise. The critic has the same 512-256-64 shape. Its input includes body linear and angular velocity, body height, joint position and velocity, the previous action, contact sequence and phase, and the user command. It leaves out the observation noise and the joint history.

• Tasks. Four terrains are used as downstream tasks: rough, discrete rough, step and cliff fig. 3. Terrain difficulty is set by a "terrain factor", which is the scaled maximum height. Rough terrain is Perlin noise with the stated fractal parameters §III-C.
• Curriculum. The terrain factor starts at 0.01 and changes exponentially every 5 iterations; a flat terrain is sampled about 5% of the time to keep the pre-trained motion §III-C.
• Reward, simple (SR). Three terms: track the commanded (v_x, v_y, \omega_z), penalise motion in the other directions (v_z, \omega_x, \omega_y), and regularise torque §III-C. Per episode, the initial state and the command are sampled, with the command drawn from U(-1, 1) §III-C.
• Reward, contact (CR). SR plus a contact-biasing reward that encourages a predefined contact sequence §III-C.
• Domain randomisation. Observation noise, motor friction, PD gains, ground friction and motor command delay are randomised, after Ji et al. Ground friction is drawn from [0.3, 1.0], which is wider than in Ji et al. §III-C. The paper models about 10 ms of real latency by setting the PD target after a uniform delay in [0, 10] ms §III-C.
• Constrained exploration. The initial standard deviation is 1.0 (2.0 in CR), the policy learning rate is 10^{-5}, the PPO clip is 0.05 (0.1 in CR), and the critic learning rate is 10^{-3} §III-C.

# Stage 2: PPO finetuning with a terrain curriculum  [§III-C]
# Actor starts from the cloned policy; critic is asymmetric.
def finetune(actor, critic, env, num_iterations):
    for iteration in range(num_iterations):
        # Difficulty follows the schedule; direction is ambiguous  [§III-C]
        terrain_factor = curriculum_factor(iteration)
        # Mostly the chosen terrain, about 5% flat ground  [§III-C]
        terrain = sample_terrain(terrain_factor, flat_probability=0.05)
        # Collect rollouts with the actor, with domain randomisation  [§III-C]
        batch = collect_rollouts(actor, critic, env, terrain)
        # PPO update with clip 0.05 (0.1 in CR), policy LR 1e-5  [§III-C]
        actor, critic = ppo_update(actor, critic, batch)
    return actor

Hyperparameters

Values are as stated in the source. "not stated" means the paper gives no value for it.

Results

All numbers are author-reported. The setup is flat-terrain velocity tracking on the Mini Cheetah robot, with MPC/WBIC, DAgger, Vanilla RL (SR and CR) and Concurrent RL (Ji et al.) as baselines §IV-A. Lower is better for every column of tab. 1 and tab. 2. The ± values in the source are not defined in the text.

Table 1: flat-terrain tracking, simulation and hardware tab. 1

Table 2: hardware step-climbing success and slippery-ground tracking tab. 2

Headline results, with the setup in brackets:

• Robustness, simulation. IFM improves on MPC and DAgger on rough and slippery terrain (both in training) and on a conveyor belt (unseen in training). The metric is survival time until a fall. Values appear only in the figure fig. 4. IFM(SR) is comparable to, and sometimes slightly worse than, Vanilla RL(SR) on out-of-distribution tasks §IV-B.
• Compute. MPC optimisation time falls from 0.0154 s to 0.0010 s §IV-B.
• Learning speed. IFM reaches its plateau in about 2000 iterations and Vanilla RL in about 4000, from curves averaged over three trials fig. 6 §IV-D.
• Hardware step climbing. IFM(CR) crosses 7.5 cm in 9 of 10 trials, the best of the set. Vanilla RL(CR) manages 8 of 10 tab. 2 §IV-C.
• Hardware slippery ground (friction 0.22, 1 m/s). IFM(SR) has lower tracking error and pass time than Vanilla RL(SR). IFM(CR) passes, while Vanilla RL(CR) fails tab. 2 §IV-C.
• Gait quality. IFM(SR) and IFM(CR) have CoT and PPI close to MPC/WBIC. Vanilla RL shows much higher CoT and PPI. IFM(SR) has nearly overlapping left and right knee phase portraits, which the authors read as symmetry fig. 8 tab. 1 §IV-D.

Qualitative comparison tab. 3 (no numbers): IFM is listed as imitation-based, with a target-joint-angle action space, under 1 ms computation, contact adaptation, improvement of its MPC, and demo terrains flat, rough, slippery, conveyor and step. The compared works are contact-adaptive MPC, GLiDE and MPC-net §IV-E.

Paper vs code

No official code.

Searched: the arXiv abstract page (no comments field, no project page link, no code link), the LaTeX source (no URL to code or project page), a GitHub repository search for the title and method name ("imitating finetuning model predictive control", "IFM mini cheetah", "IFM quadruped locomotion", "imitating mpc legged", "finetuning mpc mini cheetah"), a GitHub user search for Donghoon Youm (no results), and the public repositories of Sehoon Ha (15 listed, none for this method). The method searches returned no repositories.

Pitfalls

• Computation time is reported three ways. §III-A says the DDP MPC takes 12.382 ms per optimisation. §IV-B gives 0.0154 s for MPC and 0.0010 s for IFM. tab. 3 says IFM is "< 1 ms", which a 0.0010 s value only just meets. The hardware, load and definition of each measurement are not stated.
• The hardware "MPC" is not the expert. On hardware the MPC baseline is a convex MPC with Whole Body Impulse Control, tuned for the robot §IV-A. The expert used to label training data is the DDP MPC, which runs only in simulation §III-A. The hardware MPC rows therefore do not test the expert that IFM clones.
• Eq. 6 has an unstated target type. The loss compares \pi^E(x) and \pi^I(x) eq. 6. The text says the expert torque is converted to a target joint angle to fit the imitation policy §III-A. Implementers should assume both sides are joint targets, after eq. 5, and check the output scale. The paper does not state this.
• Curriculum direction is ambiguous. The text says the terrain factor "increase[s] exponentially by a factor of 0.96 every 5 iterations" §III-C. Multiplying by 0.96 shrinks it. The sign and the exact form of the schedule are not stated, so the card does not choose one.
• Friction used for the slippery test is outside the randomised range. Ground friction is randomised over [0.3, 1.0] §III-C, and the hardware slippery test uses 0.22 §IV-C. Simulation results on slippery ground (§IV-B) also rely on "training" terrain, yet the training terrains in §III-C are rough, discrete rough, step and cliff, with no slippery terrain listed. The two statements conflict. The slippery friction used in simulation is not given.
• Sample sizes are small. Each step-height cell is 10 trials, so one trial moves the rate by 10 points. Hardware tracking uses 5 trials §IV-C. Simulation trial counts are not stated. The ± values have no stated definition.
• The learning-curve iteration count is ambiguous. It is unclear whether the "about 2000 iterations" for IFM includes the DAgger stage §IV-D. The reward axis has no stated units fig. 6.
• Baseline identity is partly unverifiable. The paper calls Concurrent RL "the exact work from Ji et al." §IV-A. This card does not check that implementation. Vanilla RL uses the IFM terrain curriculum but not the constrained exploration §IV-A.
• Expert failures are dropped, which biases the data. Rollouts that hit a non-foot contact or flip over 70° are cut short §III-B, so the imitation data covers mainly states where the expert did not fail. The paper does not test the effect.
• Constrained exploration is tight. The PPO clip (0.05) and policy learning rate (10^{-5}) are small §III-C, and no sensitivity study is reported. Results on other robots or reward scales are not known.
• Velocity observation is asymmetric. The actor omits body linear velocity, and the critic includes it §III-B §III-C. An implementer should keep this split, because the authors say the actor avoids a noisy state estimator.
• Symmetry and efficiency claims have thin evidence. The phase portraits are shown as a single plot per method fig. 8, and the number of runs behind them is not stated. The PPI definition is in a cited reference that this card does not reproduce.
• Several inputs have no stated size. The joint-history length, the phase encoding of the contact sequence and the action scale \sigma are missing §III-B. Implementers must choose these, and the choice will affect the result.

Lineage

• Builds on: DDP for trajectory optimisation (Tassa et al.) §III-A; the Raibert foot-placement heuristic (Raibert) §III-A; the reference generator of Di et al. §III-A; DAgger (Ross et al.) §III-B; PPO (Schulman et al.) §III-C; the asymmetric actor-critic (Pinto et al.) §III-C; domain randomisation and the Concurrent RL baseline (Ji et al.) §III-C §IV-A; the WBIC hardware controller (Kim et al.) §IV-A; the Mini Cheetah robot (Katz et al.) §IV-A.
• Compared against: Vanilla RL with and without contact reward (this paper's baselines); Concurrent RL (Ji et al.); MPC-net (Carius et al.); GLiDE (Xie et al.); contact-adaptive MPC (cited as da2021learning; author names not in the card's source text) tab. 3.
• Superseded by: not known.