Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

In plain words

The authors wanted a cheap, small humanoid robot to play soccer in a fast, agile way instead of walking stiffly through pre-written moves. They taught it in a computer simulation first: a neural network watched the robot's joints, its own motion and the ball and opponent positions, and it learned by trial and error to score goals. Learning everything at once did not work well, so training happened in two steps. First, the robot learned two separate skills, kicking toward a goal and getting up from the ground. Second, those skills were merged into one player that trained against earlier copies of itself, so the opponent kept getting harder.

To make the simulated player work on a real robot, the authors made the simulator a little messy on purpose: friction, joint offsets, added weight and small delays changed from episode to episode, and the robot was randomly nudged during training. They also rewarded staying upright and penalised strain on the knees, so the robot would not break itself. The result, according to the authors, is a robot that moves faster than a hand-built controller, gets up faster, and takes its own initiative, such as blocking a shot before it is taken.

TL;DR

• A 20-joint OP3 miniature humanoid learns 1v1 soccer in simulation with Distributional MPO (DMPO) and transfers zero-shot to real robots §Introduction §Materials and Methods: Policy Optimization.
• Two stages: separate soccer and get-up skill policies, then distillation into one agent with KL regularisation to the skills plus self-play §Materials and Methods: Training.
• Transfer depends on a 40 Hz control loop, actuator identification, domain randomisation, random pushes and shaping rewards §Materials and Methods: Sim-to-Real Transfer.
• Author-reported gains over a scripted OP3 controller: walking 181% faster, turning 302% faster, get-up 63% less time §Results.
• No official code. Read the card's Pitfalls before implementing: several reward and parameter definitions are inconsistent or missing.

Problem

Low-cost humanoids are fragile and have few actuators, so learning-based control on them has been limited to basic skills such as walking or jumping. The state of the art in humanoid control relied on model-based predictive control, which the authors describe as limiting generality §Introduction. Learning end-to-end on the full 1v1 soccer task was reported to produce degenerate solutions: either rolling on the ground, or standing up without learning to score §Introduction §Materials and Methods: Ablations. The method assumes that simulation is good enough for zero-shot transfer, which the authors achieved only with several sim-to-real measures and a motion-capture setup on the real pitch §Materials and Methods: Sim-to-Real Transfer §Materials and Methods: Robot Hardware and Motion Capture.

Method

The task is modelled as a partially observable MDP. Policy outputs are 20-dimensional, continuous joint position targets. The return is a weighted sum of reward components §Materials and Methods: Policy Optimization.

The policy is a feed-forward network that outputs the mean and a diagonal covariance of a Gaussian. It is trained with MPO, an off-policy actor-critic algorithm, with a distributional critic (DMPO). The critic is n-step (n = 5) categorical Q-learning, and policy improvement reweights sampled actions by exp(Q/η) and then fits the policy to them under a KL bound §Materials and Methods: Policy Optimization §Supplementary Methods: Agent Training and Architecture.

Training is in two stages §Materials and Methods: Training:

1. Skill training. The soccer skill is trained against an untrained opponent that falls almost at once. An episode ends when the agent falls, leaves the pitch, enters the goal penalty area, the opponent scores, or 50 s pass. The get-up skill is trained to reach target poses interpolated between three key poses taken from a scripted OP3 get-up trajectory. Actor and critic are both conditioned on the target joint angles and the target gravity direction in the egocentric frame. Target poses are resampled at exponentially distributed intervals with a mean of 1.5 s §Materials and Methods: Get-Up Skill Training.
2. Distillation and self-play. One agent plays against increasingly strong opponents, starting from the two skill policies. Episodes end when either side scores or after 50 s. Starting poses are lying face-down, lying face-up, or standing, with equal probability. While the robot is lying down, outside the pitch, or inside its penalty area, it receives a fixed penalty each step, and its positive reward terms are dropped for that step §Materials and Methods: Training.

Distillation. In each state the policy is regularised to one skill: the soccer skill when upright, the get-up skill otherwise. The policy-improvement objective becomes a mix of predicted Q and a KL term to that skill. The weight λ is adapted by gradient descent on λ(E[Q] − Q_threshold), so that λ rises toward 1 while predicted Q is below the threshold and falls to 0 above it. This lets the agent surpass the skills. Stage 2 stops distilling once performance passes a preset threshold §Materials and Methods: Distillation §Materials and Methods: Training.

Self-play. The opponent is sampled uniformly from a pool made of the first quarter of the saved snapshots plus an untrained agent. The authors found that using all snapshots gave less stable learning §Materials and Methods: Self-Play fig. 7. The critic receives an integer ID for the opponent, because the opponent's identity otherwise makes value learning hard §Materials and Methods: Self-Play.

Observations. The table below lists the observations. Proprioceptive channels are stacked over the five most recent steps. Game-state channels are not stacked, except the agent's own velocity §Materials and Methods: Environment §Supplementary Methods: Environment Details.

Total: 267 values (our sum of the table entries). All positions and velocities are planar, in the agent's frame; velocities come from finite differences §Supplementary Methods: Environment Details.

Control. The agent acts at 40 Hz. Actions are clipped to the joint ranges and passed through an exponential filter, u_t = 0.8·u_{t−1} + 0.2·a_t. PID controllers then drive the joints (torques in simulation, voltages on the robot) §Materials and Methods: Environment.

Sim-to-real. Actuator parameters were fitted to a sinusoidal test. The model is position-controlled with torque feedback, with damping, armature, friction, maximum torque and proportional gain as free parameters. The fitted values are given in the Hyperparameters section below. The authors found that direct current control gave too large a gap for transfer §Materials and Methods: System Identification.

Domain randomisation is re-sampled per episode and held fixed within it. Random pushes are applied during training §Materials and Methods: Domain Randomization and Perturbations. Their values are in the Hyperparameters section below.

Safety shaping. An extra penalty on knee torque peaks reduces gear breakage. An upright reward keeps the lean angle small, to protect the real robot from falling forward §Materials and Methods: Regularization for Safe Behaviors.

Get-up reward (as printed).

# Get-up skill reward  [§Materials and Methods: Get-Up Skill Training]
# Inputs: joint_angles (20,), target_joints (20,),
#         gravity (3,), target_gravity (3,)
# Output: scalar reward (printed sign; see Pitfalls)
def get_up_reward(joint_angles, target_joints, gravity, target_gravity):
    # Joint similarity scaled to [0, 1]; 1 means no error  [§M&M: Get-Up]
    joint_error = norm(target_joints - joint_angles)
    joint_score = (pi - joint_error) / pi
    # Gravity similarity scaled to [0, 1]; 1 means aligned  [§M&M: Get-Up]
    # Clip the dot product to [-1, 1] before arccos (numerical guard;
    # not stated in the paper)
    gravity_angle = arccos(dot(gravity, target_gravity))
    gravity_score = (pi - gravity_angle) / pi
    # Printed as the negative of the product of the two scores  [§M&M: Get-Up]
    return -(joint_score * gravity_score)

Distillation score (per state).

# Policy-improvement score with skill regularisation  [§M&M: Distillation]
# Inputs: state, actions (K, action_dim) sampled from the current policy
# Output: score per state (higher is better)
def distillation_score(state, actions, critic, policy, skill, weight):
    # Mean predicted Q over sampled actions; the estimator is not stated
    # beyond sampling  [§M&M: Distillation]
    expected_q = mean(critic(state, actions))
    # KL from the current policy to the skill for this state:
    # soccer skill if upright, get-up skill otherwise  [§M&M: Distillation]
    kl = kl_divergence(policy(state), skill(state))
    # Mix of value and pull toward the skill  [§M&M: Distillation]
    return (1 - weight) * expected_q - weight * kl

Distillation weight update.

# Adapt the skill-regularisation weight  [§M&M: Distillation]
# Goal: weight near 1 while predicted Q is below Q_threshold, near 0 above
def update_weight(raw_weight, mean_q, q_threshold, learning_rate):
    # The cost lambda * (E[Q] - Q_threshold) has this derivative in lambda
    # [§M&M: Distillation]
    gradient = mean_q - q_threshold
    raw_weight = raw_weight - learning_rate * gradient
    # Softplus then clip to [0, 1]; exact order and form not stated
    weight = clip(softplus(raw_weight), 0.0, 1.0)
    return raw_weight, weight

MPO policy improvement (E-step).

# MPO E-step: reweight sampled actions  [§Supp. Methods: Policy Improvement]
# Inputs: q_values (K,) for K sampled actions; eta is the temperature
# K is 20 actions per state in the hyperparameter table
def improved_action_weights(q_values, eta):
    # Unnormalised weight is exp(Q / eta); normalise over the samples
    # [§Supp. Methods: Policy Improvement]
    return softmax(q_values / eta)

# Dual objective for eta, with epsilon = 0.1  [§Supp. Methods: Policy Impr.]
# The integral over pi_t is estimated from the K sampled actions
def dual_loss(eta, q_values, epsilon):
    # eta * epsilon + eta * log( mean over samples of exp(Q / eta) )
    log_mean = logsumexp(q_values / eta) - log(num_samples)
    return eta * epsilon + eta * log_mean

Hyperparameters

Values from the paper's supplementary tables app. Hyperparameters. Units are as printed.

Agent and training §Supplementary Methods: Agent Hyperparameters

Control and episodes

Domain randomisation and perturbations §Materials and Methods: Domain Randomization and Perturbations

Actuator fit §Materials and Methods: System Identification

Reward components §Supplementary Methods: Reward Functions

Joint limits (applied per joint, zero = T-pose) §Supplementary Methods: Environment Details

The table lists ID 16 twice (see Pitfalls).

Training cost §Supplementary Methods: Training Curves

Training was distributed over parallel simulated copies. Number of workers and hardware are not stated.

Results

Author-reported. Setup: OP3 robots (51 cm, 3.5 kg, 20 servos), 1v1 play on a 5 m × 4 m pitch, and a motion-capture system for the real pitch. Baseline: the scripted OP3 controllers, with walking and turning tuned by grid search on a real robot. Kick and get-up have no tunable parameters §Results: Comparison to Scripted Baseline Controllers.

Notes:

• Real-robot kick from standing is about 3% slower than the scripted kick. The "34% faster" figure applies only to the run-up kick §Results: Comparison.
• The learned walking speed is 181% faster than the scripted walker in the text. Dividing the table values gives about 185%; the table values are the primary source.
• The stated 13% walking-speed drop from sim to real does not match the table: 0.57 / 0.51 is about 12%.
• Ablations use five seeds (95% CI). Learning curves for skills use ten seeds fig. 7 §Supplementary Methods: Training Curves.
• Turning comparison: the learned policy fell in 3 of 13 trials; the scripted controller stayed upright for 10 consecutive trials §Supplementary Methods: Behavior Analysis Experiment Details.

Paper vs code

No official code.

Search performed for this card:

• Paper LaTeX source: no code link, no GitHub or GitLab address. The data availability statement points only to a Zenodo dataset for the quantitative figures §Acknowledgements.
• Project page (sites.google.com/view/op3-soccer, linked in the Introduction): no code repository or code release.
• GitHub repository search for "op3 soccer": three results (ifawwaz/RobotisSoccer, iwandwip/RobotisSoccer2, jadstrike/op3-probabilistic-ball-tracking), all third-party, none with a license. Not used.
• GitHub search within google-deepmind for "soccer", "op3", and "dmpo mpo": zero results.

Pitfalls

• Section citations. The paper's sections are unnumbered. This card cites them by name (e.g. §Materials and Methods: Distillation). Equation numbers are not used because the source has unlabeled alignments; cite the named equation instead.
• Get-up reward sign. The text defines the reward as the negative of the product of two scores, each of which is 1 at zero error and 0 at maximal error §Materials and Methods: Get-Up Skill Training §Supplementary Methods: Reward Functions. Read literally, that rewards the worst match. The intended sign is probably the opposite (or the scores are error terms), but no code is available to settle it. Test before trusting it.
• Gravity score scaling. The reward table says the gravity term is 0 at an error of π/2. The formula in the text is 0 at an angle of π. Pick one and check it against the behaviour you want.
• Interference penalty. The table calls it a penalty equal to the cosine of the angle between velocity and opponent heading, with weight 1. The sign and the exact velocity definition are not stated.
• Torque penalty. The Methods text describes a penalty on the time integral of torque peaks above 5 N·m on the constraint forces of the targeted joints. The reward table instead defines a knee-torque magnitude penalty with weight 0.01. The 5 N·m threshold does not appear in the table. Which definition the authors used is not stated.
• Push units. The impulse is written as "5 to 15 N·m", which is a torque unit for what is described as a force impulse. The units are ambiguous.
• Joint-limit table. ID 16 is listed for both l_ank_pitch and r_ank_pitch. The ankle and knee ranges have mirrored signs. Verify the limits against the robot model before use.
• Missing parameters. The Q thresholds Q_s and Q_g, the λ learning rate, the softplus form, the number of snapshots in self-play, and the snapshot interval are not stated. The target update period units are not stated. The η initialisation is not stated beyond a warm-start.
• Control rate. The text says 40 Hz. The Discussion says the nominal step was 25 ms (which matches 40 Hz) but that on the robot the agent often did not produce an action within that time. Plan the real-time loop with that gap in mind.
• Sim vs real control. Simulation uses PID with torque. The real servos are position-controlled with proportional gain only. The authors say this hides model mismatch. Expect the sim-to-real gap to show up in actuator response and battery sag.
• Observation stacking. Game-state channels are not stacked. The authors note that a full observation history may matter for reacting to the opponent.
• Episode rules differ by stage. In the skill stage a fall terminates the episode. In the full-agent stage a fall gives a per-step penalty and the episode continues until a goal or 50 s.
• Self-play stability. The authors report unstable learning in self-play. Using all snapshots, instead of the first quarter, made it worse fig. 7.
• Reward weights. The weights come from an "extensive hyperparameter search" that is not described §Discussion: Limitations.
• Hardware fragility. Performance degraded over time (loose hips, encoder miscalibration). Robots needed regular maintenance, and behaviour was sensitive to battery charge, with 5 to 10 minutes of operation per charge §Discussion: Limitations.
• Motion capture dependence. Ball tracking with markers was difficult (facing angle, occlusion by walls). The vision-based variant is preliminary: trained in a NeRF-based simulation, and 6 of 10 real penalty kicks scored §Discussion: Future Work §Supplementary Methods: Playing Soccer from Raw Vision.
• Small sample sizes. The baseline comparisons use about 10 episodes per behaviour. The set piece uses 50 episodes per setting. Treat the differences as indicative, not as precise.
• Baseline tuning. The scripted walker was grid-searched on the real robot. The learned policy got no equivalent tuning, and the authors say the baseline does not correct heading. The comparison is therefore not symmetric.
• Unlabelled equations in the source. The MPO equations in the appendix are unnumbered. The E-step and dual forms in this card are transcribed from the text and should be checked against the paper.

Lineage

• Builds on: Maximum a posteriori Policy Optimization (MPO), Abdolmaleki et al. (cited as the base algorithm) §Materials and Methods: Policy Optimization. Distributional critic (Bellemare et al., 2017). Policy distillation (Rusu et al., 2015; Parisotto et al., 2015) §Materials and Methods: Distillation. Distillation weight adjustment follows Abdolmaleki et al. (2021), a Lagrangian-style constraint §Materials and Methods: Distillation.
• Compared against: the scripted OP3 walk, turn, kick and get-up controllers from the OP3 driver §Results: Comparison to Scripted Baseline Controllers. The authors also compare to a parameterised walker evaluated only in simulation (Bestmann et al.) §Results: Comparison to Scripted Baseline Controllers.
• Related work: humanoid soccer in simulation (Liu et al.), which the authors suggest for population-based training and for automatic skill selection §Discussion: Limitations.
• Superseded by: not known from content in this wiki.