Proximal policy optimization (PPO)

Overview

PPO optimises a policy with on-policy samples. Its invariant is that the update is capped: the ratio of the new to the old action probability is clipped to a band of width epsilon around 1, and the pessimistic minimum of the raw and clipped surrogate is used 2011.09533 eq. 3. Eight cards state epsilon = 0.2 in their settings 2011.09533 tab. 1 2102.02202 tab. 2 2508.11849 tab. 2 2306.06766 tab. 2 2412.13621 §III-A 2412.14172 app. B.6 2412.16201 tab. 1 2412.18865 tab. III. One card uses 0.1 [arxiv:2312.06639 app. RL training]. The DPPO card lists 0.01, annealed, for its diffusion variant 2409.00588 app. C.10. The imitation-then-RL card uses 0.05 for its main runs and 0.1 for one reward variant 2311.02304 §III-C, and 0.2 for its vanilla RL baseline 2311.02304 §IV-A. Six cards do not state epsilon for their experiments 2204.04340 Hyperparameters 2303.03381 § Policy learning 2512.12993 §III.F 2412.15587 Hyperparameters 2409.03930 §3.2 2509.20322 Hyperparameters. The warehouse card gives "about 0.2" as a general statement only 2411.06128 §IV. The FPGA survey gives the clipped objective without a numeric epsilon, and lists the clip range as not stated 2610.28699 §3.8 2610.28699 Hyperparameters.

The VisualMimic card's "clipping" is a clip on the generator's output actions to a band around the human-motion mean, not the PPO ratio clip 2509.20322 §3.3. Do not read its clip value as an epsilon.

Two cards clip the value loss as well 2011.09533 eq. 6 2102.02202 tab. 2. The LocoMamba card uses a plain value regression term in its objective 2508.11849 eq. 20. The wireless-navigation code clips the value loss too, although the paper does not say so 2306.06766 Paper vs code.

The clipping is the part the variants share. The IPPO authors argue that it is also what makes independent learners workable, because it limits how much one agent's policy can change between updates 2011.09533 §1. Across the other variants, what changes is the network that produces the action, the extra loss or reward term, or what sits around PPO: a latent interface, a teacher, a denoising chain, a planner, or a vision-language score. A survey of learning-based legged locomotion names TRPO and PPO as the usual on-policy choices for that task 2406.01152 §2.2. Several cards take some settings from library defaults, for example the Stable Baselines3 values in 2412.18865 tab. III.

One card changes what is clipped. CAPO clips one ratio per complete agent action rather than one ratio per token 2604.18401 §3.5 eq. 5.

Variants

The first four variants are about legged locomotion or multi-agent learning. The later ones add manipulation, navigation, fine-tuning and reward shaping.

Independent PPO for cooperative teams (IPPO)

Each agent runs its own PPO update on its own observation history. The team reward replaces each agent's reward, and the actor and critic weights are shared across agents 2011.09533 §4. The authors report that IPPO matches or beats MAPPO, QMIX, MAVEN and IQL on several hard SMAC maps, with mild per-map tuning 2011.09533 §5.1. Their 10M-step table reports 80% vs 0% on corridor and 60% vs 8.95% on 6h vs 8z, but the MAPPO numbers are taken from a cited benchmark and were not re-measured 2011.09533 fig. 2. Card: Independent PPO on StarCraft micromanagement (SMAC).

PPO as the inner learner of evolutionary search (DERL)

Here PPO is not the research object. It is the fixed inner loop that trains each candidate body's controller from random weights before its fitness is known 2102.02202 app. A. Each child gets 5 million interactions, and its fitness is the average final reward 2102.02202 app. A. The PPO table lists Adam, initial learning rate 3e-4 with linear decay, 4 epochs, batch size 512, entropy coefficient 0.01 and gradient norm clip 0.5 [arxiv:2102.02202 tab. 2]. The full run uses 1152 CPUs and is reported to take under 16 hours 2102.02202 §DERL. Card: Embodied Intelligence via Learning and Evolution.

PPO with a KL term to a teacher (sim-to-real walking)

The student's loss adds a KL term to a teacher that sees the full simulator state, next to the usual PPO loss. The KL weight is annealed to zero, reaching zero near the middle of training, so the student can later outperform the teacher 2303.03381 eq. 2 [arxiv:2303.03381 § Joint optimization with reinforcement learning]. The paper states that PPO is used with separate actor and critic networks 2303.03381 § Policy learning. The PPO settings themselves are in a supplement that is not in the arXiv source. Card: Real-World Humanoid Locomotion with Reinforcement Learning.

PPO fine-tuning after imitating an MPC expert (IFM)

A small MLP with 512, 256 and 64 units first copies a DDP-based model predictive controller, which runs only in simulation, using DAgger 2311.02304 §III-B eq. 6. PPO then fine-tunes that copy on harder terrain. The critic is asymmetric: it sees more state than the actor 2311.02304 §III-C. The clip is 0.05, or 0.1 in the contact-reward variant, with a policy learning rate of 1e-5 and a critic learning rate of 1e-3 2311.02304 §III-C. The paper's vanilla PPO baseline uses clip 0.2 and a different initial standard deviation 2311.02304 §IV-A. Discount, GAE, epochs, minibatch size and the number of RL environments are not stated 2311.02304 Hyperparameters. Author-reported: in simulation the fine-tuned policy survives longer than the MPC expert and DAgger on rough and slippery ground, and on a conveyor belt that was not in training 2311.02304 §IV-B fig. 4. On hardware, the contact-reward variant crosses a 7.5 cm step in 9 of 10 trials, against 8 of 10 for vanilla RL with the same reward 2311.02304 tab. 2 §IV-C. Card: Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion.

Recurrent PPO policy with a clock input (bipedal walking under load)

The controller is an LSTM with two 128-unit recurrent layers. Its inputs are pelvis orientation and rotational velocity, joint positions and velocities, a sine and cosine clock whose period is the desired cycle time, and the desired forward speed. It outputs joint position set points at 40 Hz, which a PD controller tracks at 2 kHz 2204.04340 §III-A. Training uses the clipped PPO objective with gradient clipping 2204.04340 §III-A. The learning rate, clip threshold, batch sizes, epochs and discount are not stated 2204.04340 Hyperparameters. The base reward comes from an earlier framework that the paper does not reproduce 2204.04340 §III-A. Card: Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads.

Hierarchical PPO over a learned terrain latent (Digit)

A CNN-VAE compresses a 20 by 40 terrain height map into a 16-value latent 2512.12993 §II.B. A PPO policy takes that latent, a reduced-order robot state and the last action, and outputs six task-space commands at 33 Hz. A whole-body controller tracks them 2512.12993 §III eq. 5 eq. 6 §III.C. The policy is an MLP with two hidden layers of 128 tanh units, and the paper uses 6 epochs over 56,000 samples per iteration [arxiv:2512.12993 §III.F]. Whether 56,000 is per iteration or total is not stated 2512.12993 §III.F. The authors report success above 80% on all tested terrains in simulation, and at least 2x better sample efficiency than end-to-end baselines 2512.12993 §IV.A §IV.E. Card: Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations.

PPO teachers distilled into depth-based students (VisualMimic)

The framework has two levels, and PPO trains the teacher at each. A motion tracker teacher sees the full reference motion for the next 2 seconds and privileged proprioception. It is trained with PPO on a reward for body positions, root velocity, jitter and foot slip, and its student sees only proprioception and the command. The student is distilled with DAgger 2509.20322 §3.1. A task generator teacher sees object state and proprioception, and is trained with PPO on a task reward. Its student uses a depth CNN and is distilled with DAgger 2509.20322 §3.2. The paper gives the tracker and student as three-layer MLPs, and does not state the PPO learning rate, batch size, epochs or environment count 2509.20322 §3.1 2509.20322 Hyperparameters. Author-reported: zero-shot transfer to a Unitree G1 for four real tasks, with qualitative results only 2509.20322 §4.2. The public code is a MuJoCo sim-to-sim demo with four checkpoints, and the training code is not released 2509.20322 Paper vs code. Card: VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation.

PPO with a state-space backbone (LocoMamba)

The PPO objective is the standard clipped one with GAE advantages, a value term and an entropy bonus 2508.11849 eq. 16–20. The change is in the encoder: proprioceptive and depth-image tokens are fused by two Mamba layers rather than self-attention 2508.11849 §3.2 tab. 1. Each iteration collects 16,384 samples and runs 3 minibatch epochs 2508.11849 tab. 2. The authors report a return of 762.34 against 511.96 for a Transformer baseline on the training terrain, in PyBullet 2508.11849 tab. 4. The state-space layer's internal size is not stated 2508.11849 §3.2. Card: LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba.

PPO over a motion latent for character control (Motion VAE)

A motion VAE is first fitted to about 17 minutes of motion capture. It is then frozen and its encoder is discarded 2103.14274 §3.4 §4.1. A PPO policy takes the current pose and a task vector, and outputs a 32-value latent that the decoder turns into the next pose 2103.14274 §5 fig. 6. The four locomotion tasks are reaching a target, following a joystick heading and speed, following a figure-eight path, and exploring a maze [arxiv:2103.14274 §6.1] §6.2 §6.3 §6.4. Sampling-based control without RL works modestly for the target task and fails on joystick and path following [arxiv:2103.14274 §4.2]. Author-reported foot skate is 0.067 to 0.30 cm per frame across three MVAE variants, against 0.10 in the motion capture data [arxiv:2103.14274 tab. 1]. The RL results are shown as figures and videos, with no success rates 2103.14274 fig. 7 fig. 8 fig. 9. The paper does not state the PPO discount, GAE, clip or epoch count, and the effort penalty weight is missing too 2103.14274 §5 §5.2. The released controller uses 0.99, 0.95, 0.2 and 10 epochs 2103.14274 Paper vs code. Card: Character Controllers Using Motion VAEs.

PPO with physics terms in the reward (wireless indoor navigation)

A global PPO policy picks a waypoint from radio input, and a local policy with vision drives the robot there 2306.06766 §IV fig. 6. The reward keeps a distance cost and adds three terms from radio physics: a penalty for the link state rising along a path, a penalty for heading away from the strongest path's arrival angle, and an SNR-gradient penalty in the worst link state 2306.06766 §III-B eq. 10. The paper says this reward is not potential-based, so its optimal policy need not be the shortest path 2306.06766 §III-B. Training uses a wireless digital twin of the building 2306.06766 §III-B. The PPO settings in the paper are a discount of 0.99, clip 0.2, learning rate 3e-4 and batch size 64 2306.06766 tab. 2. Author-reported normalised path length on unseen maps P to U in the 2+-NLOS state is 2.46 to 2.82 for the proposed method and 4.50 to 5.14 for the end-to-end RL baseline 2306.06766 tab. 3. The released reward has no link-state and no SNR term [arxiv:2306.06766 Hyperparameters]. Card: Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning.

PPO fine-tuning of a diffusion policy (DPPO)

DPPO starts from a behaviour-cloned diffusion policy and fine-tunes it with PPO 2409.00588 §4.2. The denoising chain is written as an inner MDP inside the environment MDP, and each denoising step has a Gaussian likelihood, so the clipped PPO objective applies to every step 2409.00588 §4.1 eq. 6 eq. 8. The advantage is scaled by a discount on the denoising index eq. 10, and the value function sees only the environment state eq. 11. Fine-tuning only the last few denoising steps is recommended, with a frozen copy for the earlier ones 2409.00588 §4.3. Author-reported: over 90% success on Robomimic Transport from state, and 80% (16 of 20) zero-shot on FurnitureBench One-Leg, against 0% for a fine-tuned Gaussian policy 2409.00588 §5.1 §5.4. The cost is about six times the wall-clock per iteration of a Gaussian policy, with 10 denoising steps [arxiv:2409.00588 app. D]. Card: Diffusion Policy Policy Optimization.

PPO with an asymmetric actor-critic for drones carrying a payload

Three cable-linked hex-rotors learn to find and carry an unknown payload in a simulated room, using one camera 2409.03930 §3.1 §4. The critic sees privileged state in training, and the actor sees only observations at deployment 2409.03930 §3.2. Author-reported Gazebo success is 0.824 to 0.903 across three payload classes, against 0.502 to 0.786 for tabular Q-learning, DQN and Sarsa 2409.03930 tab. 1. The policy objective as printed has no clip range, and the reward, observations, actions, network and optimiser are not stated 2409.03930 §3.2. Use it as an idea, not as a recipe. Card: DRAL, cable-suspended drones carrying an unknown payload.

PPO for crossing narrow pipes with a privileged height scan (Go2)

A Unitree Go2 learns to cross a narrow pipe with PPO. The policy takes a bidirectional height scan that is available only in simulation, and its reward has seven terms, including centreline distance and contact penalties 2412.13621 §II-B §II-C tab. II. Training uses 6144 robots, clip 0.2 and a learning rate of 1e-3 2412.13621 §III-A §III-B. The curriculum runs three stages of 15,000, 8,000 and 8,000 iterations, with friction and base mass randomised 2412.13621 §III-A §III-B. In simulation, stage 3 reaches 100% success on 0.3 m and 0.2 m pipes without obstacles, and 93.75% with obstacles at 0.2 m 2412.13621 tab. IV. On three real pipes the robot crossed without obstacles in every trial, and with obstacles it reached 46.5% on one pipe 2412.13621 tab. V. Card: Learning Quadrupedal Robot Locomotion for Narrow Pipe Inspection.

PPO for mobile manipulation with a recurrent visual policy (HarmonicMM)

One policy takes two RGB views and five arm proprioception values, and outputs base and arm commands together 2312.06639 §IV §IV-B. A frozen DINOv2 backbone feeds a recurrent network [arxiv:2312.06639 app. Network Architecture]. Training uses DD-PPO in AllenAct with 20 parallel environments, 4 epochs per update and clip 0.1 [arxiv:2312.06639 app. RL training] §V-A. The paper gives 10^7 steps and three seeds [arxiv:2312.06639 §V-A], but the released stage budgets sum to 10^9 [arxiv:2312.06639 Paper vs code]. Author-reported average success over four ProcTHOR tasks is 47% 2312.06639 §V-B. Real-robot trials in one unseen apartment reach 66.6% on cleaning, 60.0% on pulling and 70.0% on pushing 2312.06639 tab. III. Card: Harmonic Mobile Manipulation.

PPO refining a grasp on point clouds (dexterous articulated objects)

An AnyGrasp-based grasp pose on the object's functional part sets the start state. PPO then refines the hand's motion from a point cloud, robot proprioception and the goal pose 2412.15587 §3.1 §3.4.1. The 22-value action has six arm end-effector velocities, converted to joint angles with IKFast, and 16 hand joint targets 2412.15587 §3.4.2. The point-cloud encoder is pre-trained on four part categories before PPO [arxiv:2412.15587 §3.4.1]. Author-reported success on the bucket task in simulation is 94.17%, against 79.17% for DexArt 2412.15587 tab. 1. The paper gives no PPO hyperparameters 2412.15587 Hyperparameters. Card: Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge.

Goal-conditioned PPO tracker for humanoid motion (UH-1)

Retargeted human motion is tracked by a goal-conditioned PPO policy in Isaac Gym, which follows keypoints, joint angles and root motion [arxiv:2412.14172 §3.2] app. B.6. The output is target joint positions for a PD controller 2412.14172 app. E.1. The paper's settings are clip 0.2, discount 0.99, GAE 0.95, five epochs with four minibatches each, learning rate 1e-3 and a rollout of 21 steps 2412.14172 app. B.6. The released configuration differs on some of these, for example the learning rate and the rollout length 2412.14172 app. B.6. Real-robot success is reported per command over 10 trials each 2412.14172 tab. 3. Card: Learning from Massive Human Videos for Universal Humanoid Pose Control.

PPO with a CLIP-based reward term (unsignalized intersection)

A CLIP image-text score for slow-down, proceed and speed-up prompts is added to the reward of a DQN or a PPO agent turning left at an intersection 2412.16201 §2.4 §3.1. The PPO settings are clip 0.2, GAE 0.95, discount 0.99, 8000 steps per update, learning rate 5e-4, batch 64 and 10 epochs 2412.16201 tab. 1. Author-reported success with one other vehicle is 96% for CLIP-DQN against 70% for DQN, and 38% for CLIP-PPO against 78% for plain PPO 2412.16201 tab. 2. The paper reports no variance and no seeds in its table. The CLIP fine-tuning data and weights are not in the released repository 2412.16201 TL;DR. Card: CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning.

PPO combined with Dijkstra's shortest path (warehouse robots)

Dijkstra's algorithm plans a path on the static warehouse layout, and PPO learns actions that adapt when the environment changes 2411.06128 §VI fig. 4. The PPO objective is written as a clipped term plus value and entropy terms, with epsilon about 0.2 in the text 2411.06128 §IV eq. 1-3. The paper does not say how the Dijkstra path enters the PPO agent 2411.06128 §VI. On a single test set it reports accuracy 96.42%, recall 95.83%, F1 94.88% and robustness 75.62%, above DQN and A* 2411.06128 tab. II. Setup, training details and baseline implementations are not stated, so the result cannot be reproduced from the card. Card: Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout.

PPO chosen from four algorithms for a farm robot

A four-wheel-steering field robot follows crop rows and moves between waypoints in Gazebo. Stable Baselines3 provides SAC, TD3, A2C and PPO, and PPO gives the shortest episodes and the highest reward in the paper's runs [arxiv:2412.18865 §IV-B] fig. 9 fig. 10. The PPO table lists rollout 2048, batch 64, 10 epochs, learning rate 3e-4, discount 0.99, GAE 0.95 and clip 0.2 2412.18865 tab. III. Author-reported success is 351 of 420 randomised trials, 83.57% 2412.18865 tab. IV. The paper has simulation results only 2412.18865 §V. Card: Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning.

PPO accelerators on FPGA and integrated circuits (as surveyed)

This is a survey's view of PPO in hardware, not a new method. The survey breaks PPO into a ratio block, a clip-and-select stage, rollout buffers and repeated minibatch passes, and notes that the clip and ratio logic can take more run time than the network layers 2610.28699 §3.8. It lists two PPO designs with author-reported gains: one on an Alveo U200 with up to 26x throughput, and one on a ZCU106 with 1.5x reward and a 30% speed-up 2610.28699 tab. 4 §3.8. The survey also reports uniform 8-bit quantisation in one PPO pipeline, without saying what is quantised 2610.28699 Hyperparameters. It warns that baselines differ by study, so the numbers cannot be ranked against each other 2610.28699 §3.9. The two accelerator papers have no card here, so their settings are not given. Card: Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey.

Action-level PPO for multi-turn LLM agents (CAPO)

In an LLM agent, one step is a whole message with a short reasoning note and a tool call, and the environment answers only after that message ends. CAPO treats the step as the decision unit 2604.18401 §3.2. A critic reads the value at the last state token before each step. GAE then gives the step a single advantage, which every token in the step shares §3.3 eq. 2. The token log-ratios inside the step are centred on a batch mean, summed, and divided by the square root of the step length. The result is one ratio, and it is clipped as a whole §3.4 eq. 4 §3.5 eq. 5. The authors argue that multiplying or averaging the token ratios makes both the weighting and the clipping depend on step length §3.4 fig. 3. Author-reported: CAPO is best in every column on HotpotQA, 2Wiki, MuSiQue, a paper-search task, ALFWorld and WebShop, for both Qwen3-1.7B and Qwen3-4B, over three seeds [arxiv:2604.18401 tab. 1]. In the Qwen3-4B ablation, the centred square-root ratio scores 78.12 on WebShop, against 76.18 without the centring and 73.80 for token-wise ratios tab. 2. The clip is 0.001 for the action ratio §4.1 app. B. The released repository does not contain the action-aware ratio, so a reproduction needs a new implementation 2604.18401 Paper vs code. Card: CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities.

Settings across the cards

Most of these numbers are per-paper choices, not a shared recipe. Use the card's own table, not a value copied from another row. Where a cell says "not stated", the paper's text or table does not give the value. It does not mean the value is zero or the default.

When to use which

Implementation sketch

The core is the clipped surrogate. The variants differ in what the advantage and the critic are computed from, in the extra loss term (the teacher KL, for example), and in the network that produces the action (an LSTM, a Mamba encoder, or a latent-command head). In the diffusion variant the same loss is applied to each denoising step, with a smaller epsilon 2409.00588 eq. 8. Coefficients differ by card, see the table above. CAPO keeps the clipped form but applies it to one action-level ratio built from the token log-ratios, instead of to each token 2604.18401 eq. 4 eq. 5.

# Clipped PPO policy loss, one batch of on-policy samples  [arxiv:2011.09533 eq. 3]
# Inputs: log-probs of taken actions (new and old), advantages, epsilon
# Output: scalar loss to minimise

def ppo_policy_loss(logp_new, logp_old, advantages, eps=0.2):
    # Ratio of the new policy to the policy that collected the batch  [eq. 3]
    ratio = exp(logp_new - logp_old)
    # Pessimistic minimum of the raw and clipped surrogate  [eq. 3]
    surrogate = minimum(ratio * advantages,
                        clip(ratio, 1 - eps, 1 + eps) * advantages)
    # The surrogate is maximised, so negate it for a minimiser  [eq. 3]
    return -surrogate.mean()

def ppo_value_loss(values, old_values, targets, eps=0.2):
    # Keep the critic within eps of its previous estimate  [eq. 6] [tab. 2]
    clipped = old_values + clip(values - old_values, -eps, eps)
    # Take the larger squared error; the paper writes a minimum, see Contradictions
    return maximum((values - targets) ** 2, (clipped - targets) ** 2).mean()

# Advantages from GAE over the rollout, gamma 0.99, lambda 0.95  [§4] [tab. 2]
advantages = gae(rewards, values, gamma=0.99, lam=0.95)
# Teacher-student variant only: KL weight decays to zero  [arxiv:2303.03381 eq. 2]
loss = ppo_policy_loss(...) + c_v * ppo_value_loss(...) - c_e * entropy

Contradictions

Most cards measure different tasks, robots and simulators, so their numbers are not in conflict. The points below are where a card's own text, table or code disagree, or where the settings differ enough to need a decision.

• Value clipping form. The IPPO card's equation 6 takes a minimum of the two squared errors, which is the reverse of the usual pessimistic form. The paper does not say whether this was intended, and no code was located to check it 2011.09533 eq. 6.
• Value clipping evidence. The IPPO ablation finds value clipping helps on two maps 2011.09533 §5.3 2011.09533 fig. 4. DERL uses clipped values but gives no ablation 2102.02202 tab. 2. The LocoMamba card uses a value regression term without a stated clip 2508.11849 eq. 20. The wireless-navigation code clips values, and the paper does not 2306.06766 Paper vs code.
• Shared weights. IPPO shares actor and critic weights across agents 2011.09533 §4. The teacher-student paper uses separate networks for actor and critic 2303.03381 § Policy learning. These are different settings, not conflicting claims.
• Discount on the Digit tasks. The hierarchical card uses a discount of 0.95 2512.12993 §III.F. The DERL and LocoMamba cards use 0.99 2102.02202 tab. 2 2508.11849 tab. 2, and so do several newer cards. The tasks differ, so this is not a conflict, but the value does not transfer without a check.
• Sample efficiency claims. The hierarchical card's "at least 2x" claim is against end-to-end baselines 2512.12993 §IV.A. The card notes that the paper's text does not say whether 56,000 samples is per iteration 2512.12993 §III.F, so the absolute budget is unclear.
• Training budget. The HarmonicMM paper gives 10^7 steps [arxiv:2312.06639 §V-A]. Its released stage budgets sum to 10^9 [arxiv:2312.06639 Paper vs code]. The card says the paper's number cannot be reproduced from the code's defaults.
• Reward terms. The wireless-navigation paper's reward has three physics terms. The released reward has no link-state and no SNR term, and it gives the policy the ground-truth link state 2306.06766 Hyperparameters. Reported success numbers come from the paper's own runs.
• Clip value across cards. HarmonicMM uses 0.1 [arxiv:2312.06639 app. RL training] and most others use 0.2. DPPO lists 0.01, annealed, for a per-denoising-step ratio 2409.00588 app. C.10. IFM uses 0.05 for its main runs, against 0.2 for its vanilla RL baseline [arxiv:2311.02304 §III-C] §IV-A. These are different settings with different roles, so do not copy one into another.
• "Clipping" in VisualMimic. The VisualMimic card clips the generator's output actions to a band around the human-motion mean [arxiv:2509.20322 §3.3]. This is an action clip, not the PPO ratio clip. The card's ablation says removing it makes returns collapse late in training [arxiv:2509.20322 §4.5], which is a result about the action clip alone.
• CLIP card: text against table. The CLIP card's text says CLIP-PPO beats plain PPO on training reward 2412.16201 §3.1. Its test table shows plain PPO with higher success at every vehicle count 2412.16201 tab. 2. The card records both and does not resolve them.
• CLIP card: discount values. The paper gives a DQN discount of 0.95 and a PPO discount of 0.99 2412.16201 tab. 1. The released code uses 0.99 for DQN and a different PPO value in a commented-out block 2412.16201 Paper vs code. Check the PPO value against the code you actually run.
• Epochs per update. The cards use 4, 5 and 10 epochs for the same objective 2312.06639 app. RL training 2412.14172 app. B.6 2412.16201 tab. 1, and the Go2, grasp, IFM and VisualMimic cards do not state the count 2412.13621 §III-A [arxiv:2412.15587 Hyperparameters] 2311.02304 Hyperparameters [arxiv:2509.20322 Hyperparameters]. These are per-paper settings, not disagreements.
• PPO accelerator speed-up. The FPGA survey's Table 4 gives up to 26x throughput for one PPO design, while its PPO section gives 27x 2610.28699 tab. 4 2610.28699 §3.8. The survey says to check the original paper 2610.28699 Pitfalls. The card records both values and does not resolve them.
• Motion VAE KL weight, paper against code. The generative model's KL weight is 0.2 in the paper 2103.14274 §3.4 and 1.0 in the shipped training script train_mvae.py. This concerns the motion model, not PPO, but check it before reproducing the latent.
• CAPO clip, paper against code. The paper gives a clip of 0.001 for the action ratio 2604.18401 §4.1 app. B. The step-level scripts set 0.0003 run_steppo.sh, the value the paper gives for its GSPO baseline app. B. The card records both and does not resolve them.
• Clipped unit. CAPO clips one ratio per action, at 0.001. The per-token clips in most other cards are about 0.2 2604.18401 §3.5 §4.1. The two numbers are not the same quantity, so do not compare them.

Papers

• Independent PPO on StarCraft micromanagement (SMAC): independent PPO with shared weights on SMAC; ablations of clipping and value clipping.
• Embodied Intelligence via Learning and Evolution: PPO as the controller learner inside an evolutionary search over bodies.
• Real-World Humanoid Locomotion with Reinforcement Learning: PPO with a teacher KL term for zero-shot humanoid walking in simulation and on hardware.
• Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads: LSTM PPO walking policy for Cassie with a clock input, trained for load carrying.
• LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba: PPO with a two-layer Mamba encoder for quadruped locomotion in PyBullet.
• Character Controllers Using Motion VAEs: PPO outputs the latent of a frozen motion VAE to steer a character through locomotion tasks.
• Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations: hierarchical PPO over a VAE terrain latent, with a whole-body controller, for Digit.
• Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning: PPO with three radio-physics reward terms and a hierarchical global and local policy.
• Diffusion Policy Policy Optimization: PPO fine-tuning of a behaviour-cloned diffusion policy, applied per denoising step.
• DRAL, cable-suspended drones carrying an unknown payload: PPO with an asymmetric actor-critic for drones carrying a payload; the method is under-specified.
• Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout: Dijkstra path planning combined with a PPO agent that adapts to changes.
• Learning Quadrupedal Robot Locomotion for Narrow Pipe Inspection: PPO for a Unitree Go2 crossing narrow pipes, with a privileged height scan.
• Harmonic Mobile Manipulation: one recurrent PPO policy for base and arm motion, trained with DD-PPO.
• Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge: PPO refines a grasp on point clouds for dexterous hands.
• Learning from Massive Human Videos for Universal Humanoid Pose Control: a goal-conditioned PPO tracker for humanoid motion retargeted from human clips.
• CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning: a CLIP-based reward term added to DQN and PPO agents.
• Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning: PPO against SAC, TD3 and A2C for a farm robot in simulation.
• Learning-based legged locomotion; state of the art and future perspectives: survey that names TRPO and PPO as the usual on-policy choices for locomotion.
• Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion: DAgger clone of an MPC expert, then PPO fine-tuning with a small clip.
• VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation: PPO teachers distilled with DAgger into depth-based students for a humanoid.
• Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey: survey of RL on FPGAs and integrated circuits; its PPO section covers the clipped objective and PPO accelerators.
• CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities: PPO-style updates with one advantage and one clipped ratio per agent action.

The original PPO paper (Schulman et al., 2017) has no card in this wiki, so it is named here and not used as a source.