---
kind: concept
slug: proximal-policy-optimization
title: Proximal policy optimization (PPO)
summary: >-
  PPO is a policy-gradient method that clips the new-to-old probability ratio, so one update cannot move the policy far. Twenty-two cards use it as learner, fine-tuner or reward partner, from team learning and evolutionary search to diffusion policies, motion latents, physics rewards, FPGA accelerators and LLM agents.
tags: [reinforcement-learning, policy-optimization]
papers: ["2011.09533", "2102.02202", "2303.03381", "2204.04340", "2508.11849", "2512.12993", "2306.06766", "2409.00588", "2409.03930", "2411.06128", "2412.13621", "2312.06639", "2412.15587", "2412.14172", "2412.16201", "2412.18865", "2406.01152", "2311.02304", "2509.20322", "2610.28699", "2103.14274", "2604.18401"]
related: [policy-gradient, actor-critic, credit-assignment, llm-agent, mixture-of-experts, variational-autoencoder, trust-region-method, multi-agent-reinforcement-learning, centralized-training-decentralized-execution, sim-to-real-transfer, neuroevolution, robot-simulation, hierarchical-reinforcement-learning, state-space-model, variational-autoencoder, long-short-term-memory, domain-randomization, reward-design, digital-twin, diffusion-models, action-chunking, deep-q-learning, dijkstra-algorithm, recurrent-neural-network, motion-retargeting, robotic-grasp-generation, model-predictive-control, partially-observable-markov-decision-process, multi-robot-navigation, vision-language-pretraining, imitation-learning, on-policy-distillation]
status: draft
updated_at: 2026-11-01
---

## Overview

PPO optimises a policy with on-policy samples. Its invariant is that the
update is capped: the ratio of the new to the old action probability is
clipped to a band of width epsilon around 1, and the pessimistic minimum of the
raw and clipped surrogate is used [arxiv:2011.09533 eq. 3]. Eight cards state
epsilon = 0.2 in their settings [arxiv:2011.09533 tab. 1]
[arxiv:2102.02202 tab. 2] [arxiv:2508.11849 tab. 2]
[arxiv:2306.06766 tab. 2] [arxiv:2412.13621 §III-A]
[arxiv:2412.14172 app. B.6] [arxiv:2412.16201 tab. 1]
[arxiv:2412.18865 tab. III]. One card uses 0.1 [arxiv:2312.06639 app. RL
training]. The DPPO card lists 0.01, annealed, for its diffusion variant
[arxiv:2409.00588 app. C.10]. The imitation-then-RL card uses 0.05 for its
main runs and 0.1 for one reward variant [arxiv:2311.02304 §III-C], and 0.2 for
its vanilla RL baseline [arxiv:2311.02304 §IV-A]. Six cards do not state
epsilon for their experiments [arxiv:2204.04340 Hyperparameters]
[arxiv:2303.03381 § Policy learning] [arxiv:2512.12993 §III.F]
[arxiv:2412.15587 Hyperparameters] [arxiv:2409.03930 §3.2]
[arxiv:2509.20322 Hyperparameters]. The warehouse card gives "about 0.2" as a
general statement only [arxiv:2411.06128 §IV]. The FPGA survey gives the
clipped objective without a numeric epsilon, and lists the clip range as not
stated [arxiv:2610.28699 §3.8] [arxiv:2610.28699 Hyperparameters].

The VisualMimic card's "clipping" is a clip on the generator's output actions
to a band around the human-motion mean, not the PPO ratio clip
[arxiv:2509.20322 §3.3]. Do not read its clip value as an epsilon.

Two cards clip the value loss as well [arxiv:2011.09533 eq. 6]
[arxiv:2102.02202 tab. 2]. The LocoMamba card uses a plain value regression
term in its objective [arxiv:2508.11849 eq. 20]. The wireless-navigation code
clips the value loss too, although the paper does not say so
[arxiv:2306.06766 Paper vs code].

The clipping is the part the variants share. The IPPO authors argue that it is
also what makes independent learners workable, because it limits how much one
agent's policy can change between updates [arxiv:2011.09533 §1]. Across the
other variants, what changes is the network that produces the action, the
extra loss or reward term, or what sits around PPO: a latent interface, a
teacher, a denoising chain, a planner, or a vision-language score. A survey of
learning-based legged locomotion names TRPO and PPO as the usual on-policy
choices for that task [arxiv:2406.01152 §2.2]. Several cards take some settings
from library defaults, for example the Stable Baselines3 values in
[arxiv:2412.18865 tab. III].

One card changes what is clipped. CAPO clips one ratio per complete agent
action rather than one ratio per token [arxiv:2604.18401 §3.5] [eq. 5].

## Variants

The first four variants are about legged locomotion or multi-agent learning.
The later ones add manipulation, navigation, fine-tuning and reward shaping.

### Independent PPO for cooperative teams (IPPO)

Each agent runs its own PPO update on its own observation history. The team
reward replaces each agent's reward, and the actor and critic weights are shared
across agents [arxiv:2011.09533 §4]. The authors report that IPPO matches or
beats MAPPO, QMIX, MAVEN and IQL on several hard SMAC maps, with mild per-map
tuning [arxiv:2011.09533 §5.1]. Their 10M-step table reports 80% vs 0% on
corridor and 60% vs 8.95% on 6h vs 8z, but the MAPPO numbers are taken from a
cited benchmark and were not re-measured [arxiv:2011.09533 fig. 2]. Card:
[Independent PPO on StarCraft micromanagement (SMAC)](/papers/2011.09533).

### PPO as the inner learner of evolutionary search (DERL)

Here PPO is not the research object. It is the fixed inner loop that trains
each candidate body's controller from random weights before its fitness is
known [arxiv:2102.02202 app. A]. Each child gets 5 million interactions, and
its fitness is the average final reward [arxiv:2102.02202 app. A]. The PPO table
lists Adam, initial learning rate 3e-4 with linear decay, 4 epochs, batch size
512, entropy coefficient 0.01 and gradient norm clip 0.5 [arxiv:2102.02202
tab. 2]. The full run uses 1152 CPUs and is reported to take under 16 hours
[arxiv:2102.02202 §DERL]. Card: [Embodied Intelligence via Learning and Evolution](/papers/2102.02202).

### PPO with a KL term to a teacher (sim-to-real walking)

The student's loss adds a KL term to a teacher that sees the full simulator
state, next to the usual PPO loss. The KL weight is annealed to zero, reaching
zero near the middle of training, so the student can later outperform the
teacher [arxiv:2303.03381 eq. 2] [arxiv:2303.03381 § Joint optimization with
reinforcement learning]. The paper states that PPO is used with separate actor
and critic networks [arxiv:2303.03381 § Policy learning]. The PPO settings
themselves are in a supplement that is not in the arXiv source. Card:
[Real-World Humanoid Locomotion with Reinforcement Learning](/papers/2303.03381).

### PPO fine-tuning after imitating an MPC expert (IFM)

A small MLP with 512, 256 and 64 units first copies a DDP-based model
predictive controller, which runs only in simulation, using DAgger
[arxiv:2311.02304 §III-B] [eq. 6]. PPO then fine-tunes that copy on harder
terrain. The critic is asymmetric: it sees more state than the actor
[arxiv:2311.02304 §III-C]. The clip is 0.05, or 0.1 in the contact-reward
variant, with a policy learning rate of 1e-5 and a critic learning rate of
1e-3 [arxiv:2311.02304 §III-C]. The paper's vanilla PPO baseline uses clip 0.2
and a different initial standard deviation [arxiv:2311.02304 §IV-A]. Discount,
GAE, epochs, minibatch size and the number of RL environments are not stated
[arxiv:2311.02304 Hyperparameters]. Author-reported: in simulation the
fine-tuned policy survives longer than the MPC expert and DAgger on rough and
slippery ground, and on a conveyor belt that was not in training
[arxiv:2311.02304 §IV-B] [fig. 4]. On hardware, the contact-reward variant
crosses a 7.5 cm step in 9 of 10 trials, against 8 of 10 for vanilla RL with
the same reward [arxiv:2311.02304 tab. 2] [§IV-C]. Card:
[Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion](/papers/2311.02304).

### Recurrent PPO policy with a clock input (bipedal walking under load)

The controller is an LSTM with two 128-unit recurrent layers. Its inputs are
pelvis orientation and rotational velocity, joint positions and velocities, a
sine and cosine clock whose period is the desired cycle time, and the desired
forward speed. It outputs joint position set points at 40 Hz, which a PD
controller tracks at 2 kHz [arxiv:2204.04340 §III-A]. Training uses the clipped
PPO objective with gradient clipping [arxiv:2204.04340 §III-A]. The learning
rate, clip threshold, batch sizes, epochs and discount are not stated
[arxiv:2204.04340 Hyperparameters]. The base reward comes from an earlier
framework that the paper does not reproduce [arxiv:2204.04340 §III-A]. Card:
[Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads](/papers/2204.04340).

### Hierarchical PPO over a learned terrain latent (Digit)

A CNN-VAE compresses a 20 by 40 terrain height map into a 16-value latent
[arxiv:2512.12993 §II.B]. A PPO policy takes that latent, a reduced-order robot
state and the last action, and outputs six task-space commands at 33 Hz. A
whole-body controller tracks them [arxiv:2512.12993 §III] [eq. 5] [eq. 6]
[§III.C]. The policy is an MLP with two hidden layers of 128 tanh units, and
the paper uses 6 epochs over 56,000 samples per iteration [arxiv:2512.12993
§III.F]. Whether 56,000 is per iteration or total is not stated
[arxiv:2512.12993 §III.F]. The authors report success above 80% on all tested
terrains in simulation, and at least 2x better sample efficiency than
end-to-end baselines [arxiv:2512.12993 §IV.A] [§IV.E]. Card:
[Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations](/papers/2512.12993).

### PPO teachers distilled into depth-based students (VisualMimic)

The framework has two levels, and PPO trains the teacher at each. A motion
tracker teacher sees the full reference motion for the next 2 seconds and
privileged proprioception. It is trained with PPO on a reward for body
positions, root velocity, jitter and foot slip, and its student sees only
proprioception and the command. The student is distilled with DAgger
[arxiv:2509.20322 §3.1]. A task generator teacher sees object state and
proprioception, and is trained with PPO on a task reward. Its student uses a
depth CNN and is distilled with DAgger [arxiv:2509.20322 §3.2]. The paper
gives the tracker and student as three-layer MLPs, and does not state the PPO
learning rate, batch size, epochs or environment count [arxiv:2509.20322 §3.1]
[arxiv:2509.20322 Hyperparameters]. Author-reported: zero-shot transfer to a Unitree G1 for four
real tasks, with qualitative results only [arxiv:2509.20322 §4.2]. The public
code is a MuJoCo sim-to-sim demo with four checkpoints, and the training code
is not released [arxiv:2509.20322 Paper vs code]. Card:
[VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation](/papers/2509.20322).

### PPO with a state-space backbone (LocoMamba)

The PPO objective is the standard clipped one with GAE advantages, a value
term and an entropy bonus [arxiv:2508.11849 eq. 16–20]. The change is in the
encoder: proprioceptive and depth-image tokens are fused by two Mamba layers
rather than self-attention [arxiv:2508.11849 §3.2] [tab. 1]. Each iteration
collects 16,384 samples and runs 3 minibatch epochs [arxiv:2508.11849 tab. 2].
The authors report a return of 762.34 against 511.96 for a Transformer baseline
on the training terrain, in PyBullet [arxiv:2508.11849 tab. 4]. The
state-space layer's internal size is not stated [arxiv:2508.11849 §3.2].
Card: [LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba](/papers/2508.11849).

### PPO over a motion latent for character control (Motion VAE)

A motion VAE is first fitted to about 17 minutes of motion capture. It is then
frozen and its encoder is discarded [arxiv:2103.14274 §3.4] [§4.1]. A PPO
policy takes the current pose and a task vector, and outputs a 32-value latent
that the decoder turns into the next pose [arxiv:2103.14274 §5] [fig. 6]. The
four locomotion tasks are reaching a target, following a joystick heading and
speed, following a figure-eight path, and exploring a maze [arxiv:2103.14274
§6.1] [§6.2] [§6.3] [§6.4]. Sampling-based control without RL works modestly
for the target task and fails on joystick and path following [arxiv:2103.14274
§4.2]. Author-reported foot skate is 0.067 to 0.30 cm per frame across three
MVAE variants, against 0.10 in the motion capture data [arxiv:2103.14274 tab.
1]. The RL results are shown as figures and videos, with no success rates
[arxiv:2103.14274 fig. 7] [fig. 8] [fig. 9]. The paper does not state the PPO
discount, GAE, clip or epoch count, and the effort penalty weight is missing
too [arxiv:2103.14274 §5] [§5.2]. The released controller uses 0.99, 0.95, 0.2
and 10 epochs [arxiv:2103.14274 Paper vs code]. Card:
[Character Controllers Using Motion VAEs](/papers/2103.14274).

### PPO with physics terms in the reward (wireless indoor navigation)

A global PPO policy picks a waypoint from radio input, and a local policy with
vision drives the robot there [arxiv:2306.06766 §IV] [fig. 6]. The reward
keeps a distance cost and adds three terms from radio physics: a penalty for
the link state rising along a path, a penalty for heading away from the
strongest path's arrival angle, and an SNR-gradient penalty in the worst link
state [arxiv:2306.06766 §III-B] [eq. 10]. The paper says this reward is not
potential-based, so its optimal policy need not be the shortest path
[arxiv:2306.06766 §III-B]. Training uses a wireless digital twin of the
building [arxiv:2306.06766 §III-B]. The PPO settings in the paper are a
discount of 0.99, clip 0.2, learning rate 3e-4 and batch size 64
[arxiv:2306.06766 tab. 2]. Author-reported normalised path length on unseen
maps P to U in the 2+-NLOS state is 2.46 to 2.82 for the proposed method and
4.50 to 5.14 for the end-to-end RL baseline [arxiv:2306.06766 tab. 3]. The
released reward has no link-state and no SNR term [arxiv:2306.06766
Hyperparameters]. Card: [Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning](/papers/2306.06766).

### PPO fine-tuning of a diffusion policy (DPPO)

DPPO starts from a behaviour-cloned diffusion policy and fine-tunes it with
PPO [arxiv:2409.00588 §4.2]. The denoising chain is written as an inner MDP
inside the environment MDP, and each denoising step has a Gaussian likelihood,
so the clipped PPO objective applies to every step [arxiv:2409.00588 §4.1]
[eq. 6] [eq. 8]. The advantage is scaled by a discount on the denoising index
[eq. 10], and the value function sees only the environment state [eq. 11].
Fine-tuning only the last few denoising steps is recommended, with a frozen
copy for the earlier ones [arxiv:2409.00588 §4.3]. Author-reported: over 90%
success on Robomimic Transport from state, and 80% (16 of 20) zero-shot on
FurnitureBench One-Leg, against 0% for a fine-tuned Gaussian policy
[arxiv:2409.00588 §5.1] [§5.4]. The cost is about six times the wall-clock
per iteration of a Gaussian policy, with 10 denoising steps [arxiv:2409.00588
app. D]. Card: [Diffusion Policy Policy Optimization](/papers/2409.00588).

### PPO with an asymmetric actor-critic for drones carrying a payload

Three cable-linked hex-rotors learn to find and carry an unknown payload in a
simulated room, using one camera [arxiv:2409.03930 §3.1] [§4]. The critic sees
privileged state in training, and the actor sees only observations at
deployment [arxiv:2409.03930 §3.2]. Author-reported Gazebo success is 0.824 to
0.903 across three payload classes, against 0.502 to 0.786 for tabular
Q-learning, DQN and Sarsa [arxiv:2409.03930 tab. 1]. The policy objective as
printed has no clip range, and the reward, observations, actions, network and
optimiser are not stated [arxiv:2409.03930 §3.2]. Use it as an idea, not as a
recipe. Card: [DRAL, cable-suspended drones carrying an unknown payload](/papers/2409.03930).

### PPO for crossing narrow pipes with a privileged height scan (Go2)

A Unitree Go2 learns to cross a narrow pipe with PPO. The policy takes a
bidirectional height scan that is available only in simulation, and its reward
has seven terms, including centreline distance and contact penalties
[arxiv:2412.13621 §II-B] [§II-C] [tab. II]. Training uses 6144 robots, clip
0.2 and a learning rate of 1e-3 [arxiv:2412.13621 §III-A] [§III-B]. The
curriculum runs three stages of 15,000, 8,000 and 8,000 iterations, with
friction and base mass randomised [arxiv:2412.13621 §III-A] [§III-B]. In
simulation, stage 3 reaches 100% success on 0.3 m and 0.2 m pipes without
obstacles, and 93.75% with obstacles at 0.2 m [arxiv:2412.13621 tab. IV]. On
three real pipes the robot crossed without obstacles in every trial, and with
obstacles it reached 46.5% on one pipe [arxiv:2412.13621 tab. V]. Card:
[Learning Quadrupedal Robot Locomotion for Narrow Pipe Inspection](/papers/2412.13621).

### PPO for mobile manipulation with a recurrent visual policy (HarmonicMM)

One policy takes two RGB views and five arm proprioception values, and
outputs base and arm commands together [arxiv:2312.06639 §IV] [§IV-B]. A
frozen DINOv2 backbone feeds a recurrent network [arxiv:2312.06639 app.
Network Architecture]. Training uses DD-PPO in AllenAct with 20 parallel
environments, 4 epochs per update and clip 0.1 [arxiv:2312.06639 app. RL
training] [§V-A]. The paper gives 10^7 steps and three seeds [arxiv:2312.06639
§V-A], but the released stage budgets sum to 10^9 [arxiv:2312.06639 Paper vs
code]. Author-reported average success over four ProcTHOR tasks is 47%
[arxiv:2312.06639 §V-B]. Real-robot trials in one unseen apartment reach 66.6%
on cleaning, 60.0% on pulling and 70.0% on pushing [arxiv:2312.06639 tab. III].
Card: [Harmonic Mobile Manipulation](/papers/2312.06639).

### PPO refining a grasp on point clouds (dexterous articulated objects)

An AnyGrasp-based grasp pose on the object's functional part sets the start
state. PPO then refines the hand's motion from a point cloud, robot
proprioception and the goal pose [arxiv:2412.15587 §3.1] [§3.4.1]. The 22-value
action has six arm end-effector velocities, converted to joint angles with
IKFast, and 16 hand joint targets [arxiv:2412.15587 §3.4.2]. The point-cloud
encoder is pre-trained on four part categories before PPO [arxiv:2412.15587
§3.4.1]. Author-reported success on the bucket task in simulation is 94.17%,
against 79.17% for DexArt [arxiv:2412.15587 tab. 1]. The paper gives no PPO
hyperparameters [arxiv:2412.15587 Hyperparameters]. Card:
[Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge](/papers/2412.15587).

### Goal-conditioned PPO tracker for humanoid motion (UH-1)

Retargeted human motion is tracked by a goal-conditioned PPO policy in Isaac
Gym, which follows keypoints, joint angles and root motion [arxiv:2412.14172
§3.2] [app. B.6]. The output is target joint positions for a PD controller
[arxiv:2412.14172 app. E.1]. The paper's settings are clip 0.2, discount 0.99,
GAE 0.95, five epochs with four minibatches each, learning rate 1e-3 and a
rollout of 21 steps [arxiv:2412.14172 app. B.6]. The released configuration
differs on some of these, for example the learning rate and the rollout
length [arxiv:2412.14172 app. B.6].
Real-robot success is reported per command over 10 trials each
[arxiv:2412.14172 tab. 3]. Card:
[Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172).

### PPO with a CLIP-based reward term (unsignalized intersection)

A CLIP image-text score for slow-down, proceed and speed-up prompts is added
to the reward of a DQN or a PPO agent turning left at an intersection
[arxiv:2412.16201 §2.4] [§3.1]. The PPO settings are clip 0.2, GAE 0.95,
discount 0.99, 8000 steps per update, learning rate 5e-4, batch 64 and 10
epochs [arxiv:2412.16201 tab. 1]. Author-reported success with one other
vehicle is 96% for CLIP-DQN against 70% for DQN, and 38% for CLIP-PPO against
78% for plain PPO [arxiv:2412.16201 tab. 2]. The paper reports no variance and
no seeds in its table. The CLIP fine-tuning data and weights are not in the
released repository [arxiv:2412.16201 TL;DR]. Card:
[CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning](/papers/2412.16201).

### PPO combined with Dijkstra's shortest path (warehouse robots)

Dijkstra's algorithm plans a path on the static warehouse layout, and PPO
learns actions that adapt when the environment changes [arxiv:2411.06128 §VI]
[fig. 4]. The PPO objective is written as a clipped term plus value and
entropy terms, with epsilon about 0.2 in the text [arxiv:2411.06128 §IV]
[eq. 1-3]. The paper does not say how the Dijkstra path enters the PPO agent
[arxiv:2411.06128 §VI]. On a single test set it reports accuracy 96.42%,
recall 95.83%, F1 94.88% and robustness 75.62%, above DQN and A*
[arxiv:2411.06128 tab. II]. Setup, training details and baseline
implementations are not stated, so the result cannot be reproduced from the
card. Card: [Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout](/papers/2411.06128).

### PPO chosen from four algorithms for a farm robot

A four-wheel-steering field robot follows crop rows and moves between waypoints
in Gazebo. Stable Baselines3 provides SAC, TD3, A2C and PPO, and PPO gives the
shortest episodes and the highest reward in the paper's runs [arxiv:2412.18865
§IV-B] [fig. 9] [fig. 10]. The PPO table lists rollout 2048, batch 64, 10
epochs, learning rate 3e-4, discount 0.99, GAE 0.95 and clip 0.2
[arxiv:2412.18865 tab. III]. Author-reported success is 351 of 420 randomised
trials, 83.57% [arxiv:2412.18865 tab. IV]. The paper has simulation results
only [arxiv:2412.18865 §V]. Card:
[Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning](/papers/2412.18865).

### PPO accelerators on FPGA and integrated circuits (as surveyed)

This is a survey's view of PPO in hardware, not a new method. The survey
breaks PPO into a ratio block, a clip-and-select stage, rollout buffers and
repeated minibatch passes, and notes that the clip and ratio logic can take
more run time than the network layers [arxiv:2610.28699 §3.8]. It lists two
PPO designs with author-reported gains: one on an Alveo U200 with up to 26x
throughput, and one on a ZCU106 with 1.5x reward and a 30% speed-up
[arxiv:2610.28699 tab. 4] [§3.8]. The survey also reports uniform 8-bit
quantisation in one PPO pipeline, without saying what is quantised
[arxiv:2610.28699 Hyperparameters]. It warns that baselines differ by study, so
the numbers cannot be ranked against each other [arxiv:2610.28699 §3.9]. The
two accelerator papers have no card here, so their settings are not given.
Card: [Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey](/papers/2610.28699).

### Action-level PPO for multi-turn LLM agents (CAPO)

In an LLM agent, one step is a whole message with a short reasoning note and a
tool call, and the environment answers only after that message ends. CAPO
treats the step as the decision unit [arxiv:2604.18401 §3.2]. A critic reads
the value at the last state token before each step. GAE then gives the step a
single advantage, which every token in the step shares [§3.3] [eq. 2]. The
token log-ratios inside the step are centred on a batch mean, summed, and
divided by the square root of the step length. The result is one ratio, and it
is clipped as a whole [§3.4] [eq. 4] [§3.5] [eq. 5]. The authors argue that
multiplying or averaging the token ratios makes both the weighting and the
clipping depend on step length [§3.4] [fig. 3]. Author-reported: CAPO is best
in every column on HotpotQA, 2Wiki, MuSiQue, a paper-search task, ALFWorld and
WebShop, for both Qwen3-1.7B and Qwen3-4B, over three seeds [arxiv:2604.18401
tab. 1]. In the Qwen3-4B ablation, the centred square-root ratio scores 78.12
on WebShop, against 76.18 without the centring and 73.80 for token-wise ratios
[tab. 2]. The clip is 0.001 for the action ratio [§4.1] [app. B]. The released
repository does not contain the action-aware ratio, so a reproduction needs a
new implementation [arxiv:2604.18401 Paper vs code]. Card:
[CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities](/papers/2604.18401).

## Settings across the cards

| Card | Clip ε | Discount γ, GAE λ | Learning rate | Epochs per update |
|---|---|---|---|---|
| [IPPO, SMAC](/papers/2011.09533) | 0.2 [arxiv:2011.09533 tab. 1] | 0.99, 0.95 [arxiv:2011.09533 §4] | 1e-4, or 5e-4 on two maps [arxiv:2011.09533 tab. 1] | 1 or 4, per map [arxiv:2011.09533 tab. 1] |
| [DERL, morphology search](/papers/2102.02202) | 0.2 [arxiv:2102.02202 tab. 2] | 0.99, 0.95 [arxiv:2102.02202 tab. 2] | 3e-4, linear decay [arxiv:2102.02202 tab. 2] | 4 [arxiv:2102.02202 tab. 2] |
| [Teacher-student walking](/papers/2303.03381) | not stated in the text [arxiv:2303.03381 § Policy learning] | not stated | not stated | not stated |
| [Cassie LSTM](/papers/2204.04340) | not stated [arxiv:2204.04340 Hyperparameters] | not stated | not stated | not stated |
| [Digit, hierarchical](/papers/2512.12993) | not stated [arxiv:2512.12993 §III.F] | discount 0.95; GAE not stated [arxiv:2512.12993 §III.F] | not stated | 6 [arxiv:2512.12993 §III.F] |
| [LocoMamba](/papers/2508.11849) | 0.2 [arxiv:2508.11849 tab. 2] | 0.99, 0.95 [arxiv:2508.11849 tab. 2] | 2e-4 [arxiv:2508.11849 tab. 2] | 3 [arxiv:2508.11849 tab. 2] |
| [Wireless navigation](/papers/2306.06766) | 0.2 [arxiv:2306.06766 tab. 2] | discount 0.99 [arxiv:2306.06766 tab. 2]; value target is a k-step return [arxiv:2306.06766 eq. 29] | 3e-4 [arxiv:2306.06766 tab. 2] | not stated in the paper [arxiv:2306.06766 §IV-B] |
| [DPPO, diffusion policy](/papers/2409.00588) | 0.01, annealed [arxiv:2409.00588 app. C.10] | discount 0.999 on Robomimic [arxiv:2409.00588 app. C.10]; GAE [arxiv:2409.00588 §4.2] | actor 1e-4 or 1e-5, by task [arxiv:2409.00588 app. C.10] | not stated |
| [Drones, asymmetric critic](/papers/2409.03930) | not stated [arxiv:2409.03930 §3.2] | not stated | not stated | not stated |
| [Warehouse PP-D](/papers/2411.06128) | about 0.2, general statement [arxiv:2411.06128 §IV] | not stated | not stated | not stated |
| [Go2 pipes](/papers/2412.13621) | 0.2 [arxiv:2412.13621 §III-A] | 0.99, 0.95 [arxiv:2412.13621 §III-A] | 1e-3 [arxiv:2412.13621 §III-A] | not stated [arxiv:2412.13621 §III-A] |
| [HarmonicMM](/papers/2312.06639) | 0.1 [arxiv:2312.06639 app. RL training] | discount 0.99 [arxiv:2312.06639 app. RL training]; GAE not stated in the paper | 5e-5 [arxiv:2312.06639 app. RL training] | 4 [arxiv:2312.06639 app. RL training] |
| [Grasp refinement](/papers/2412.15587) | not stated [arxiv:2412.15587 Hyperparameters] | not stated | not stated | not stated |
| [UH-1 tracker](/papers/2412.14172) | 0.2 [arxiv:2412.14172 app. B.6] | 0.99, 0.95 [arxiv:2412.14172 app. B.6] | 1e-3 [arxiv:2412.14172 app. B.6] | 5 [arxiv:2412.14172 app. B.6] |
| [CLIP reward](/papers/2412.16201) | 0.2 [arxiv:2412.16201 tab. 1] | discount 0.99, GAE 0.95 [arxiv:2412.16201 tab. 1] | 5e-4 [arxiv:2412.16201 tab. 1] | 10 [arxiv:2412.16201 tab. 1] |
| [Farm robot](/papers/2412.18865) | 0.2 [arxiv:2412.18865 tab. III] | 0.99, 0.95 [arxiv:2412.18865 tab. III] | 3e-4 [arxiv:2412.18865 tab. III] | 10 [arxiv:2412.18865 tab. III] |
| [IFM, MPC imitation then PPO](/papers/2311.02304) | 0.05, or 0.1 in CR [arxiv:2311.02304 §III-C] | not stated [arxiv:2311.02304 Hyperparameters] | policy 1e-5, critic 1e-3 [arxiv:2311.02304 §III-C] | not stated [arxiv:2311.02304 Hyperparameters] |
| [VisualMimic, teachers and students](/papers/2509.20322) | not stated [arxiv:2509.20322 Hyperparameters] | not stated | not stated | not stated |
| [FPGA survey, PPO hardware](/papers/2610.28699) | not stated [arxiv:2610.28699 Hyperparameters] | not stated [arxiv:2610.28699 Hyperparameters] | not stated | not stated |
| [Motion VAE latent control](/papers/2103.14274) | not stated [arxiv:2103.14274 Hyperparameters] | not stated [arxiv:2103.14274 Hyperparameters] | decaying schedule [arxiv:2103.14274 §5.1] | not stated [arxiv:2103.14274 Hyperparameters] |
| [CAPO, LLM agents](/papers/2604.18401) | 0.001, on the action ratio [arxiv:2604.18401 §4.1] [app. B] | discount 0.99, GAE 1.0 [arxiv:2604.18401 §4.1] | actor 1e-6, critic 1e-5 [arxiv:2604.18401 §4.1] [tab. 7] | not stated [arxiv:2604.18401 Hyperparameters] |

Most of these numbers are per-paper choices, not a shared recipe. Use the
card's own table, not a value copied from another row. Where a cell says
"not stated", the paper's text or table does not give the value. It does not
mean the value is zero or the default.

## When to use which

| Situation | Use | Why |
|---|---|---|
| One learner, on-policy samples are cheap | Clipped PPO as in [arxiv:2011.09533 eq. 3] | It is the base objective of most variants. The Digit hierarchical card does not state a clip value [arxiv:2512.12993 §III.F]. |
| Many agents with local observations, a joint action space is too large | IPPO with shared weights | On the reported SMAC maps it matches or beats joint-value baselines [arxiv:2011.09533 §5.1]. Results are author-reported and rely on a benchmark for MAPPO. |
| Centralised critic with full state is available | Test both; the IPPO authors' own centralised-critic runs were worse on four maps [arxiv:2011.09533 §5.2] [arxiv:2011.09533 fig. 3] | The paper says the value of central state is still unclear on these maps [arxiv:2011.09533 §6]. |
| Each candidate design must be trained before it is scored, and you have a large cluster | PPO as the inner loop of an evolutionary search | The paper's search used 4000 candidates on 1152 CPUs [arxiv:2102.02202 app. A]. Each candidate learns from scratch, so cost grows with the search budget. |
| Sim-to-real locomotion with a privileged teacher available in simulation | PPO plus a KL term to the teacher, weight annealed to zero | The authors report no falls over a week of outdoor tests [arxiv:2303.03381 § Outdoor deployment]. The ablation that compares joint and single-term objectives is shown only in a figure [arxiv:2303.03381 fig. 8]. |
| Proprioception-only walking, and the policy must track a gait phase | Recurrent PPO with a sine and cosine clock input | The clock sets the cycle time and the LSTM carries state [arxiv:2204.04340 §III-A]. The card reports no numeric PPO settings, so start from a base framework or obtain them from the cited earlier paper [arxiv:2204.04340 Pitfalls]. |
| Perception input is high-dimensional and a low-dimensional command interface is acceptable | Hierarchical PPO over a learned latent, with a whole-body controller below | The authors report over 80% success and at least 2x sample efficiency against end-to-end baselines [arxiv:2512.12993 §IV.A] [§IV.E]. The VAE beta is not stated [arxiv:2512.12993 eq. 4], so reproducing the latent is not possible from the card alone. |
| Long-range context from image and proprioceptive tokens, and attention is too costly | PPO with a Mamba encoder in place of attention | The authors report higher return and fewer collisions than a Transformer fusion baseline, in PyBullet only [arxiv:2508.11849 tab. 4]. Compute gains are small: 104.68 vs 98.87 samples per second [arxiv:2508.11849 tab. 6]. |
| You need the exact PPO settings of the teacher-student setup | Obtain the supplement, or treat the settings as unknown | The PPO settings are not in the arXiv source [arxiv:2303.03381 § Policy learning]. |
| A behaviour-cloned diffusion or other multi-step policy must be improved with reinforcement learning | PPO on the denoising chain, fine-tuning only the last steps (DPPO) | The authors report 80% zero-shot hardware success where a fine-tuned Gaussian policy got 0% [arxiv:2409.00588 §5.4]. The cost is roughly six times the wall-clock per iteration of a Gaussian policy [arxiv:2409.00588 app. D], and the clip is far smaller than the usual 0.2 [arxiv:2409.00588 app. C.10]. |
| Robot must cross narrow passages, and a height scan is available only in simulation | PPO with a privileged height scan and a centreline reward | The authors report 100% simulated success on the larger pipes without obstacles [arxiv:2412.13621 tab. IV]. Real success was lower with obstacles, 46.5% on one pipe [arxiv:2412.13621 tab. V]. |
| Mobile base and arm must act together from cameras and arm readings | One recurrent PPO policy over two cameras, trained with DD-PPO | Reported average success is 47% across four tasks [arxiv:2312.06639 §V-B]. The 10^7-step budget does not match the released config [arxiv:2312.06639 Paper vs code], so budget the training from the paper and check the code. |
| A grasp pose is available and the last part of the motion is hard to plan | PPO refinement on a point cloud, after a grasp generator | The authors report 94.17% on bucket against 79.17% for DexArt [arxiv:2412.15587 tab. 1]. The PPO hyperparameters are not given [arxiv:2412.15587 Hyperparameters], so they must be set from scratch. |
| Humanoid must follow retargeted human motion on hardware | Goal-conditioned PPO tracker in simulation, with a PD controller on the output | The paper gives the full PPO settings [arxiv:2412.14172 app. B.6]. The released code differs on learning rate and rollout length [arxiv:2412.14172 app. B.6]. |
| Learned behaviour needs an extra semantic signal and a vision-language model is available | Add a CLIP score to the reward; compare DQN and PPO on the same task | In the authors' tests the CLIP-DQN version is best, and CLIP-PPO has lower success than plain PPO [arxiv:2412.16201 tab. 2]. Choose the algorithm by the test, not by the method name. |
| Known static map, dynamic obstacles, and a planner is acceptable | A planner for the global path, with PPO for local actions (PP-D) | The authors report higher scores than DQN and A* on one test set [arxiv:2411.06128 tab. II]. The settings and baselines are not stated, so treat this as a weak signal. |
| Wheeled field robot in simulation, and the choice of algorithm is open | Run PPO first; in the authors' runs it beat SAC, TD3 and A2C | PPO gave the shortest episodes and the highest reward [arxiv:2412.18865 §IV-B]. The results are simulation only [arxiv:2412.18865 §V]. |
| Robot needs a reward from physics, and a digital twin of the environment is available | PPO with physics-derived reward terms, trained in the twin | The authors report 2.46 to 2.82 normalised path length in 2+-NLOS against 4.50 to 5.14 for the end-to-end baseline [arxiv:2306.06766 tab. 3]. The released reward does not include two of the three terms [arxiv:2306.06766 Hyperparameters]. |
| A model-based expert controller exists, but it is too slow to run onboard or fails on rough ground | Clone the expert with DAgger, then fine-tune with PPO at a small clip and a low policy learning rate | The authors report better simulated survival than the expert on rough and slippery ground, with values only in a figure [arxiv:2311.02304 §IV-B] [fig. 4]. MPC optimisation time falls from 0.0154 s to 0.0010 s [arxiv:2311.02304 §IV-B]. The clip and learning rate are not swept, so they are not tested beyond the paper's setting [arxiv:2311.02304 Pitfalls]. |
| Humanoid loco-manipulation from a depth camera, and object state is available in training | Teacher trained with PPO on privileged state, then distilled with DAgger into a depth-based student | Author-reported zero-shot transfer for four real tasks, with qualitative results only [arxiv:2509.20322 §4.2]. The training code is not released, so the recipe cannot be rerun from the card [arxiv:2509.20322 Paper vs code]. |
| PPO must run on an FPGA or integrated circuit, and throughput or power is the goal | Hardware PPO as the survey describes it: ratio block, clip-and-select stage, rollout buffers | The survey reports up to 26x throughput for one PPO design [arxiv:2610.28699 tab. 4], but its text says 27x [arxiv:2610.28699 §3.8]. Baselines differ by study [arxiv:2610.28699 §3.9], so check the original accelerator papers before choosing a design. |
| The chip is reprogrammable, and the RL algorithm is still open | Compare algorithm families for the hardware budget, not PPO alone | The survey concludes that no single RL paradigm dominates, and that the algorithm should follow both the task and the hardware budget [arxiv:2610.28699 §7]. |
| Character must follow goals with natural-looking motion, and motion-capture clips are available | Fit a generative motion model, freeze it, and train PPO on its latent | The authors report that sampling-based control fails on joystick and path tasks [arxiv:2103.14274 §4.2]. The RL results are figures without success rates [arxiv:2103.14274 fig. 7] [fig. 8] [fig. 9], and the motion-quality numbers are kinematic [arxiv:2103.14274 tab. 1]. |
| Multi-turn LLM agent with tool calls, where one step is a whole message | Action-level credit with a critic (CAPO), in place of per-token advantages | Author-reported gains over PPO and GRPO on four agent settings with Qwen3 models [arxiv:2604.18401 tab. 1]. The released code does not contain the ratio, so the method must be implemented from the paper [arxiv:2604.18401 Pitfalls]. Some margins are a few standard deviations of a 3-seed estimate [arxiv:2604.18401 Pitfalls]. |

## Implementation sketch

The core is the clipped surrogate. The variants differ in what the advantage
and the critic are computed from, in the extra loss term (the teacher KL,
for example), and in the network that produces the action (an LSTM, a Mamba
encoder, or a latent-command head). In the diffusion variant the same loss is
applied to each denoising step, with a smaller epsilon [arxiv:2409.00588 eq. 8].
Coefficients differ by card, see the table above. CAPO keeps the clipped form
but applies it to one action-level ratio built from the token log-ratios,
instead of to each token [arxiv:2604.18401 eq. 4] [eq. 5].

```pseudocode
# Clipped PPO policy loss, one batch of on-policy samples  [arxiv:2011.09533 eq. 3]
# Inputs: log-probs of taken actions (new and old), advantages, epsilon
# Output: scalar loss to minimise

def ppo_policy_loss(logp_new, logp_old, advantages, eps=0.2):
    # Ratio of the new policy to the policy that collected the batch  [eq. 3]
    ratio = exp(logp_new - logp_old)
    # Pessimistic minimum of the raw and clipped surrogate  [eq. 3]
    surrogate = minimum(ratio * advantages,
                        clip(ratio, 1 - eps, 1 + eps) * advantages)
    # The surrogate is maximised, so negate it for a minimiser  [eq. 3]
    return -surrogate.mean()

def ppo_value_loss(values, old_values, targets, eps=0.2):
    # Keep the critic within eps of its previous estimate  [eq. 6] [tab. 2]
    clipped = old_values + clip(values - old_values, -eps, eps)
    # Take the larger squared error; the paper writes a minimum, see Contradictions
    return maximum((values - targets) ** 2, (clipped - targets) ** 2).mean()

# Advantages from GAE over the rollout, gamma 0.99, lambda 0.95  [§4] [tab. 2]
advantages = gae(rewards, values, gamma=0.99, lam=0.95)
# Teacher-student variant only: KL weight decays to zero  [arxiv:2303.03381 eq. 2]
loss = ppo_policy_loss(...) + c_v * ppo_value_loss(...) - c_e * entropy
```

## Contradictions

Most cards measure different tasks, robots and simulators, so their numbers are
not in conflict. The points below are where a card's own text, table or code
disagree, or where the settings differ enough to need a decision.

- **Value clipping form.** The IPPO card's equation 6 takes a minimum of the
  two squared errors, which is the reverse of the usual pessimistic form. The
  paper does not say whether this was intended, and no code was located to
  check it [arxiv:2011.09533 eq. 6].
- **Value clipping evidence.** The IPPO ablation finds value clipping helps on
  two maps [arxiv:2011.09533 §5.3] [arxiv:2011.09533 fig. 4]. DERL uses clipped
  values but gives no ablation [arxiv:2102.02202 tab. 2]. The LocoMamba card
  uses a value regression term without a stated clip [arxiv:2508.11849 eq. 20].
  The wireless-navigation code clips values, and the paper does not
  [arxiv:2306.06766 Paper vs code].
- **Shared weights.** IPPO shares actor and critic weights across agents
  [arxiv:2011.09533 §4]. The teacher-student paper uses separate networks for
  actor and critic [arxiv:2303.03381 § Policy learning]. These are different
  settings, not conflicting claims.
- **Discount on the Digit tasks.** The hierarchical card uses a discount of
  0.95 [arxiv:2512.12993 §III.F]. The DERL and LocoMamba cards use 0.99
  [arxiv:2102.02202 tab. 2] [arxiv:2508.11849 tab. 2], and so do several
  newer cards. The tasks differ, so this is not a conflict, but the value does
  not transfer without a check.
- **Sample efficiency claims.** The hierarchical card's "at least 2x" claim is
  against end-to-end baselines [arxiv:2512.12993 §IV.A]. The card notes that
  the paper's text does not say whether 56,000 samples is per iteration
  [arxiv:2512.12993 §III.F], so the absolute budget is unclear.
- **Training budget.** The HarmonicMM paper gives 10^7 steps [arxiv:2312.06639
  §V-A]. Its released stage budgets sum to 10^9 [arxiv:2312.06639 Paper vs
  code]. The card says the paper's number cannot be reproduced from the
  code's defaults.
- **Reward terms.** The wireless-navigation paper's reward has three physics
  terms. The released reward has no link-state and no SNR term, and it gives
  the policy the ground-truth link state [arxiv:2306.06766 Hyperparameters].
  Reported success numbers come from the paper's own runs.
- **Clip value across cards.** HarmonicMM uses 0.1 [arxiv:2312.06639 app. RL
  training] and most others use 0.2. DPPO lists 0.01, annealed, for a
  per-denoising-step ratio [arxiv:2409.00588 app. C.10]. IFM uses 0.05 for
  its main runs, against 0.2 for its vanilla RL baseline [arxiv:2311.02304
  §III-C] [§IV-A]. These are different settings with different roles, so do
  not copy one into another.
- **"Clipping" in VisualMimic.** The VisualMimic card clips the generator's
  output actions to a band around the human-motion mean [arxiv:2509.20322
  §3.3]. This is an action clip, not the PPO ratio clip. The card's ablation
  says removing it makes returns collapse late in training [arxiv:2509.20322
  §4.5], which is a result about the action clip alone.
- **CLIP card: text against table.** The CLIP card's text says CLIP-PPO beats
  plain PPO on training reward [arxiv:2412.16201 §3.1]. Its test table shows
  plain PPO with higher success at every vehicle count [arxiv:2412.16201 tab. 2].
  The card records both and does not resolve them.
- **CLIP card: discount values.** The paper gives a DQN discount of 0.95 and a
  PPO discount of 0.99 [arxiv:2412.16201 tab. 1]. The released code uses 0.99
  for DQN and a different PPO value in a commented-out block
  [arxiv:2412.16201 Paper vs code]. Check the PPO value against the code you
  actually run.
- **Epochs per update.** The cards use 4, 5 and 10 epochs for the same
  objective [arxiv:2312.06639 app. RL training] [arxiv:2412.14172 app. B.6]
  [arxiv:2412.16201 tab. 1], and the Go2, grasp, IFM and VisualMimic cards do
  not state the count [arxiv:2412.13621 §III-A] [arxiv:2412.15587
  Hyperparameters] [arxiv:2311.02304 Hyperparameters] [arxiv:2509.20322
  Hyperparameters]. These are per-paper settings, not disagreements.
- **PPO accelerator speed-up.** The FPGA survey's Table 4 gives up to 26x
  throughput for one PPO design, while its PPO section gives 27x
  [arxiv:2610.28699 tab. 4] [arxiv:2610.28699 §3.8]. The survey says to check
  the original paper [arxiv:2610.28699 Pitfalls]. The card records both values
  and does not resolve them.
- **Motion VAE KL weight, paper against code.** The generative model's KL
  weight is 0.2 in the paper [arxiv:2103.14274 §3.4] and 1.0 in the shipped
  training script [code: vae_motion/train_mvae.py#L127]. This concerns the
  motion model, not PPO, but check it before reproducing the latent.
- **CAPO clip, paper against code.** The paper gives a clip of 0.001 for the
  action ratio [arxiv:2604.18401 §4.1] [app. B]. The step-level scripts set
  0.0003 [code: examples/webshop/run_steppo.sh#L13-L14], the value the paper
  gives for its GSPO baseline [app. B]. The card records both and does not
  resolve them.
- **Clipped unit.** CAPO clips one ratio per action, at 0.001. The per-token
  clips in most other cards are about 0.2 [arxiv:2604.18401 §3.5]
  [§4.1]. The two numbers are not the same quantity, so do not compare them.

## Papers

- [Independent PPO on StarCraft micromanagement (SMAC)](/papers/2011.09533): independent PPO with shared weights on SMAC; ablations of clipping and value clipping.
- [Embodied Intelligence via Learning and Evolution](/papers/2102.02202): PPO as the controller learner inside an evolutionary search over bodies.
- [Real-World Humanoid Locomotion with Reinforcement Learning](/papers/2303.03381): PPO with a teacher KL term for zero-shot humanoid walking in simulation and on hardware.
- [Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads](/papers/2204.04340): LSTM PPO walking policy for Cassie with a clock input, trained for load carrying.
- [LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba](/papers/2508.11849): PPO with a two-layer Mamba encoder for quadruped locomotion in PyBullet.
- [Character Controllers Using Motion VAEs](/papers/2103.14274): PPO outputs the latent of a frozen motion VAE to steer a character through locomotion tasks.
- [Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations](/papers/2512.12993): hierarchical PPO over a VAE terrain latent, with a whole-body controller, for Digit.
- [Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning](/papers/2306.06766): PPO with three radio-physics reward terms and a hierarchical global and local policy.
- [Diffusion Policy Policy Optimization](/papers/2409.00588): PPO fine-tuning of a behaviour-cloned diffusion policy, applied per denoising step.
- [DRAL, cable-suspended drones carrying an unknown payload](/papers/2409.03930): PPO with an asymmetric actor-critic for drones carrying a payload; the method is under-specified.
- [Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout](/papers/2411.06128): Dijkstra path planning combined with a PPO agent that adapts to changes.
- [Learning Quadrupedal Robot Locomotion for Narrow Pipe Inspection](/papers/2412.13621): PPO for a Unitree Go2 crossing narrow pipes, with a privileged height scan.
- [Harmonic Mobile Manipulation](/papers/2312.06639): one recurrent PPO policy for base and arm motion, trained with DD-PPO.
- [Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge](/papers/2412.15587): PPO refines a grasp on point clouds for dexterous hands.
- [Learning from Massive Human Videos for Universal Humanoid Pose Control](/papers/2412.14172): a goal-conditioned PPO tracker for humanoid motion retargeted from human clips.
- [CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning](/papers/2412.16201): a CLIP-based reward term added to DQN and PPO agents.
- [Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning](/papers/2412.18865): PPO against SAC, TD3 and A2C for a farm robot in simulation.
- [Learning-based legged locomotion; state of the art and future perspectives](/papers/2406.01152): survey that names TRPO and PPO as the usual on-policy choices for locomotion.
- [Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion](/papers/2311.02304): DAgger clone of an MPC expert, then PPO fine-tuning with a small clip.
- [VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation](/papers/2509.20322): PPO teachers distilled with DAgger into depth-based students for a humanoid.
- [Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey](/papers/2610.28699): survey of RL on FPGAs and integrated circuits; its PPO section covers the clipped objective and PPO accelerators.
- [CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities](/papers/2604.18401): PPO-style updates with one advantage and one clipped ratio per agent action.

The original PPO paper (Schulman et al., 2017) has no card in this wiki, so it
is named here and not used as a source.