Policy gradient

Overview

A policy-gradient method samples an action or a whole output from the current policy, scores it, and raises the log-probability of that output in proportion to its weight 1609.05473 eq. 6. The reward enters only as a multiplier, so it need not be differentiable. The cards use discriminator scores 1701.06547 eq. 1, simulator returns 2108.10470 §6.1.1, exact-match verifiers 2503.16219 §2, solver execution results 2505.11792 eq. 6, learned video-text similarity 2206.08853 §4.2 and ROUGE-L overlap with approved headlines 2607.06818 §4.1.

The estimator is noisy, so every variant subtracts a typical value from the return, or normalises it, before the update. Variants differ in four places:

1. The baseline or advantage. A learned value function [arxiv:2102.02202 app. A], the mean over a group of rollouts for the same prompt 2503.01785 eq. 4, a running average of past returns 1808.09637 eq. 1, a greedy rollout 2012.10638 eq. 16–18, a per-token KL term 2605.07865 eq. 10, or nothing sequence_gan.py.
2. What counts as an action. A token 1609.05473 eq. 6, a denoising step of a diffusion policy 2409.00588 app. C.10, an act-level decision above a word generator 1808.09637 §3.4, a guide chosen per turn 2508.04451 eq. 1, a candidate architecture 1812.00332 eq. 8, or the parameters of a masked-diffusion language model 2507.08838 §3.1.
3. Stabilisers. A probability-ratio clip 2102.02202 tab. 2, a KL penalty to a reference model 2505.11792 tab. 6, an entropy term 2102.02202 tab. 2, or a replay buffer of past successes 2510.02245 §4.2.
4. Where the rollouts come from. The real environment, a world model 2301.04104 tab. 4, a cached tool sandbox 2509.21842 tab. 7, or a teacher's per-token log-probabilities 2605.07865 §3.1.

Four cards are not method sources for this page. Two are surveys or position papers that discuss the method without an experiment 2003.01200 §3.3 2105.01767 §5.1. Two are simulation frameworks that use PPO as a tool 2301.04195 §5.1 2108.10470 §4. One is a textbook chapter 2610.17391 app. B. Their content is used only for the throughput and training-setup notes in the Variants section.

Variants

Learned value baseline with GAE (PPO-style)

The policy is updated with a clipped surrogate and a learned value function that supplies the advantage through generalised advantage estimation. This is the most common setting in robot learning and simulation. Settings from the cards:

• Massively parallel simulation. Isaac Gym runs 4096 to 16384 environments on the GPU, with GAE gamma 0.99, lambda 0.95 and clip 0.2 [arxiv:2108.10470 §4] 2108.10470 tab. 17. The paper reports an Ant reward of 3000 in about 20 seconds 2108.10470 §6.1.1. The speed-ups compare different hardware and algorithms, with no iso-compute test 2108.10470 §6.4.1. Card: Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning.
• Co-evolved bodies and controllers. DERL trains each child's PPO controller from random weights for 5 million interactions before its fitness is known 2102.02202 app. A. Its PPO table uses batch 512, 4 epochs, clip 0.2, Adam at 3e-4 with linear decay 2102.02202 tab. 2. The full run is reported at 1152 CPUs and under 16 hours 2102.02202 §DERL. Card: Embodied Intelligence via Learning and Evolution.
• Frozen visual encoder. A frozen masked-autoencoder feature feeds a small PPO actor-critic. The paper reports success on 7 of 8 pixel tasks against a supervised ViT-S, by up to 80 points 2203.06173 §5.1. Unfreezing the encoder gives zero success for both random and MAE initialisation 2203.06173 §5.3. Card: Masked Visual Pre-training for Motor Control.
• Learned dense reward. MineDojo uses PPO with self-imitation and a video-text model that scores every step. That model has about 150 million parameters and is queried at every step; caching reduces the cost but does not remove it 2206.08853 §4.2. The paper reports 83.5 ± 7.1 on Hunt Cow against 48.3 for a manual reward 2206.08853 tab. 1, and 1.3% zero-shot success on novel tasks 2206.08853 tab. 5. Card: MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge.
• Humanoid locomotion with an asymmetric critic. The policy is trained with PPO, 4096 agents on one RTX 4090 2601.07718 app. B. On hardware, stair ascent succeeds 10 of 10 times with an edge penalty against 8 of 10 without it 2601.07718 tab. 1. Card: Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids. The paper is CC BY 4.0, but the linked code is CC BY-NC, so no code is cited here.
• Diffusion policy fine-tuning. DPPO treats each denoising step as a Gaussian policy inside a nested MDP. The advantage is GAE with the environment discount, scaled by a per-denoising-step discount [arxiv:2409.00588 app. C.10]. The paper reports 80% (16/20) zero-shot sim-to-real on One-Leg, against 0% for a fine-tuned Gaussian policy 2409.00588 §5.4. Each environment step costs K network evaluations 2409.00588 app. D. Card: Diffusion Policy Policy Optimization.
• Imagination. DreamerV3 trains its actor by REINFORCE on 15-step imagined rollouts, with a learned critic as the baseline [arxiv:2301.04104 tab. 4]. The paper uses the same hyperparameters for every benchmark 2301.04104 tab. 4. It reports 9.1 Minecraft diamond return at 100 million steps, against 5.1 for PPO 2301.04104 tab. 5, using 8.9 A100 GPU-days 2301.04104 tab. 2. Card: Mastering Diverse Domains through World Models.
• Web agents. WebShop first imitates a search model and a choice model, then fine-tunes the choice model with on-line policy gradient using a learned value baseline 2207.01206 §4.3. The best result is score 62.4 and success 28.7% 2207.01206 §5.2 2207.01206 tab. 2. Starting RL from BERT without imitation scores below the rule baseline [arxiv:2207.01206 §5.2]. Card: WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents.
• Edge computation offloading. An asynchronous actor-critic chooses per-slot offloading actions. Its reported cost is lowest among the compared methods, but the values are read from plots 2011.08430 fig. 4. The learning rate of 0.001 gives the lowest curve 2011.08430 fig. 8. Card: Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks.
• Critic as baseline for REINFORCE. Dialogue generation with a discriminator reward uses a learned critic as the baseline 1701.06547 eq. 2. Card: Adversarial Learning for Neural Dialogue Generation.

Group-relative baseline without a critic (GRPO family)

For each prompt, sample a group of G outputs, score them, and normalise the rewards inside the group. No value network is trained. The advantage is the group-normalised reward 2503.01785 eq. 4 2504.06958 eq. 1. This is the main variant for language and vision-language models with a verifiable reward.

• Verifiable rewards on small models. Open-RS trains a 1.5B reasoning model with G = 6, learning rate 1e-6 and a 500-step cap on four A40 GPUs for 24 hours 2503.16219 §3 2503.16219 tab. 5. Quality peaks at 50 to 100 steps and then degrades 2503.16219 §3 2503.16219 fig. 4. The paper reports AIME24 at 46.7 after 50 steps, against 28.8 for the start model 2503.16219 tab. 1. Card: Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't.
• Detection and grounding with rule rewards. Visual-RFT uses an IoU reward for boxes and an accuracy reward for classes 2503.01785 §3.2.1. The paper leaves the group size and the KL weight out; the release script uses 8 2B_base65cate_6k.sh. Card: Visual-RFT: Visual Reinforcement Fine-Tuning.
• Video grounding. VideoChat-R1 sums an IoU reward, an answer reward, a format reward and a caption-recall reward per task 2504.06958 §3.1. It reports Charades-STA mIoU from 29.0 to 60.8 2504.06958 tab. 1. The paper does not write out the objective 2504.06958 §3.1. Card: VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning.
• Agentic planning with replay. DeepTravel samples 8 rollouts per query 2509.21842 §4 and uses a clipped, group-normalised objective with a replay step 2509.21842 eq. 3 2509.21842 §3.3.2. Groups whose reward spread is at most 0.1 are dropped. Its online pass rate for the 32B model is 82.58% 2509.21842 tab. 4. Card: DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents.
• Experience replay for RLVR. ExGRPO uses K = 8 rollouts and batch 128 questions 2510.02245 §5.1. Half of each batch replays a stored low-entropy correct answer 2510.02245 §4.2. It reports a math average of 48.3 against 45.3 for on-policy GRPO 2510.02245 tab. 1, at 19% more step time 2510.02245 app. E.5. Card: ExGRPO: Learning to Reason from Experience.
• Large group, no KL. The Conductor trains a 7B orchestrator with G = 64 rollouts per question, 200 iterations and KL weight 0 2512.04388 app. A 2512.04388 §4.1. Its unconstrained LiveCodeBench v6 score is 83.93, against 82.90 for GPT 5 2512.04388 tab. 1. Card: Learning to Orchestrate Agents in Natural Language with the Conductor.
• Ratio-free and reference-free. wd1 turns group-relative advantages into softmax weights on a positive and a negative weighted log-likelihood 2507.08838 §3.1 2507.08838 eq. 9. It needs one likelihood estimate per update and reports 1280 rollouts to its best model, against 30000 for d1 2507.08838 tab. 8. Its likelihood is biased 2507.08838 §2.3. Card: wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models.
• Batch-normalised REINFORCE++. SIRL normalises advantages across the mini-batch rather than within a group 2505.11792 §3.2 [arxiv:2505.11792 eq. 4]. It places the outcome reward on the end-of-sequence token and applies KL only to the model and code sections 2505.11792 §3.2. Its partial KL gives 51.7 against 47.3 with no KL on MAMO Complex 2505.11792 tab. 3. Card: Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling.
• Learned step skipping with a composite reward. Low-entropy reasoning steps are first replaced by [SKIP] in training traces, then GRPO teaches the model to skip on its own. The reward adds correctness, a skip-ratio tier, a skip-count penalty and a response-length penalty [arxiv:2508.03346 §3.3] 2508.03346 eq. 4. The paper uses a group of 14 samples per prompt 2508.03346 app. C. In its reward ablation, correctness plus skip ratio alone gives MATH500 accuracy 51.00, against 85.00 with all four terms 2508.03346 tab. 5. The full SFT and GRPO recipe reports 35% fewer MATH500 tokens at 85.00 accuracy, against 88.17 for the base model 2508.03346 tab. 3. Card: step-entropy compression of reasoning traces.

Single rollout per prompt

Some agentic tasks cannot afford a group of long episodes. SAO uses one rollout per prompt and corrects for the policy lag with the rollout engine's own log-probabilities 2607.07508 §3.1. It updates a critic twice per batch, with attention frozen 2607.07508 §3.2. On AIME2025 it reports 97.3, against 84.2 for GRPO with a Python tool and 93.5 for GRPO with DIS 2607.07508 tab. 1. Its ablation reports 95.0 with one critic update per batch 2607.07508 tab. 3. The value model needs a large pretraining corpus 2607.07508 §3.2, and throughput and memory were not measured 2607.07508 §1. The results come from one training seed [arxiv:2607.07508 §4.1]. Card: Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning.

Running mean, greedy rollout or teacher-based baselines

• Self-critical greedy baseline for a masked language model. For each campaign, SC-MLM samples one headline and also decodes a greedy headline from the same model. The update weights the sample's log-probability by the difference of their ROUGE-L F1 rewards 2607.06818 §3.3 [arxiv:2607.06818 eq. 9]. The masking rate and the exact greedy decode are not stated 2607.06818 §3.1 2607.06818 §3.3. The ROUGE-L table reports 6.33 for the full model against 5.08 for the same model without the RL step, both as differences from a bi-LSTM baseline 2607.06818 tab. 1. Because the reward is the metric in that table, a crowd audit is the independent check: 6.53% more headlines are rated 3 of 3 than human-submitted ones 2607.06818 tab. 2. Card: Ad Headline Generation using Self-Critical Masked Language Model.
• Running mean. The negotiation act manager is trained by policy gradient on a reward of utility, fairness or length. Each token's log-likelihood is weighted by the reward minus the average return so far [arxiv:1808.09637 eq. 1] 1808.09637 §3.4. Card: Decoupling Strategy and Generation in Negotiation Dialogues.
• Greedy-rollout baseline with several decoders. MDAM trains five decoders with REINFORCE. The baseline is the shortest greedy tour among the decoders of the previous best model 2012.10638 eq. 16–18. A KL term on first-step distributions, with weight 0.01, pushes the decoders apart 2012.10638 §4 2012.10638 eq. 19. It reports a TSP100 gap of 0.38% with batch 50 2012.10638 tab. 1. The KL weight was tuned on TSP20 only 2012.10638 §4. Card: Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems.
• Per-token baseline from the forward pass. On-policy distillation uses the teacher-minus-student log-probability as reward. The baseline is the detached per-token negative reverse KL, which needs no critic [arxiv:2605.07865 §3.1] 2605.07865 eq. 10. The top-k variant computes the KL over the student's top 20 tokens 2605.07865 §3.2 2605.07865 §4.1. It reports a 1.7B math average of 33.0, against 29.8 for plain on-policy distillation 2605.07865 tab. 2, and up to 57.7% less wall-clock than the full-vocabulary version 2605.07865 fig. 3. Card: KL for a KL: On-Policy Distillation with Control Variate Baseline.

No baseline (plain REINFORCE on sampled returns)

• Sequence GAN. The LSTM generator is a policy over tokens. Action values come from Monte Carlo roll-outs, which give every token a value 1609.05473 eq. 3 1609.05473 eq. 4. The code uses no baseline sequence_gan.py. The repository has no license, so only the path is cited. The paper reports synthetic NLL of 8.736 against 9.038 for MLE 1609.05473 tab. 1. The discriminator-step count behind that table is ambiguous: one step in the compared settings, five in the code 1609.05473 fig. 3. Card: SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.
• Hierarchical generator with a goal. LeakGAN gives a Manager a policy-gradient update weighted by its Q-value and the cosine between the feature change and the goal 1709.08624 eq. 7. The Worker gets REINFORCE with an intrinsic cosine reward 1709.08624 eq. 8 [arxiv:1709.08624 eq. 9]. At length 40 it reports synthetic NLL of 7.191, against 10.310 for SeqGAN 1709.08624 tab. 1. The appendix says a fully connected layer instead of the hierarchy was unstable 1709.08624 app. Discussions. Card: Long Text Generation via Adversarial Training with Leaked Information.
• Dialogue with a dialogue-level reward. The two-agent stage sums rewards over simulated turns with no baseline and no discount [arxiv:1606.01541 eq. 8] 1606.01541 eq. 9. The mutual-information stage uses a neural baseline 1606.01541 eq. 7. The paper reports a five-turn quality win rate of 0.72 against 0.12 1606.01541 tab. 5, with no significance test 1606.01541 §5.2. Card: Deep Reinforcement Learning for Dialogue Generation.
• Editor learning in graphs. Learned edge editors create virtual environments, trained by REINFORCE with the variance of the GNN's risks as reward 2202.02466 app. A. No baseline is recorded for the editors. The paper reports up to 7.0% ROC-AUC gain on Twitch 2202.02466 §5.2. Each editor is a dense matrix over the graph's nodes [code: temp_arxiv/main.py#L42]. Card: Handling Distribution Shifts on Graphs: An Invariance Perspective.
• Architecture gates. ProxylessNAS uses REINFORCE for the non-differentiable latency reward 1812.00332 eq. 8. The gate gradient it uses by default is biased and unmeasured 1812.00332 §3.2.1. Card: ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware.
• Tool-augmented LLM. LLM-Augmenter trains a T5 policy with REINFORCE, but only in a learning curve 2302.12813 eq. 1 2302.12813 fig. 3. Its main results use a rule policy. Card: Check Your Facts and Try Again.

Variance-reduced, Bayesian and theory variants

• Bayesian policy gradient. The gradient integrand gets a Gaussian-process prior, and the posterior mean is computed by Bayesian quadrature with a Fisher kernel 2604.27563 §4. On the LQR task, BPG and its natural-gradient form beat MC policy gradient at 10, 20 and 40 trajectories, but not at 5 2604.27563 §6.2.2 2604.27563 fig. 3. Other tasks are simulation only 2604.27563 fig. 6. The card records no official code. Card: Bayesian policy gradient and actor-critic algorithms.
• Defensive variance reduction. DEF-PG is a PAGE-style recursive gradient without a baseline 2610.03165 §3 2610.03165 alg. 2. It claims O(ε^-3) stopped trajectories under two assumptions, against O(ε^-4) for GPOMDP 2610.03165 §4 Thm. 6 2610.03165 §4 Prop. 3. The score bound it needs generally fails for Gaussian policies [arxiv:2610.03165 §4 Rem. 2]. It has no experiments 2610.03165 §7. Card: DEF-PG, a theory paper on sample complexity.
• Self-play game agents. The AlphaGo reinforcement-learning stage is REINFORCE on self-play games 2610.17391 app. B. The reported RL policy without search wins 85% against Pachi, against 11% for the supervised-only policy 2610.17391 app. B. These numbers come from the original papers and were not rerun. Card: Learning to Play: Reinforcement Learning and Games.

Not a score-function method

The DDAC actor for lane keeping takes the gradient of the critic with respect to the action and multiplies it by the actor's gradient 1704.02532 §3.2. It is a deterministic actor-critic, not a policy gradient in the sense used here. The paper reports no metric values 1704.02532 §5. Card: Deep Reinforcement Learning framework for Autonomous Driving.

Model-based value expansion is also a deterministic actor-critic, built on the deterministic policy gradient 1803.00101 §2.1 1803.00101 §2.2. It rolls a learned one-step model forward H steps and bootstraps from the critic 1803.00101 eq. 2. Its horizon is set by hand per task: H = 10 on cheetah and swimmer, H = 3 on walker 1803.00101 fig. 3 caption. The results are learning curves on three MuJoCo tasks with 4 seeds for the learned model 1803.00101 §4 1803.00101 app. A. The paper does not state a discount. Card: Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning.

When to use which

Choose by what you can afford per update and what the reward looks like.

Two cautions apply across rows. Group and single-rollout results come from one training seed or a few seeds, and several cards flag that [arxiv:2607.07508 §4.1] 2503.16219 §3. Most headline numbers in these cards are author-reported, and the comparison baselines often differ in compute.

Implementation sketch

The loop below covers the common core. The lines that change between variants are the advantage and the optional clip and KL terms.

# Sample a group of G outputs for each prompt [arxiv:2503.01785 eq. 4]
outputs = [policy.sample(prompt) for _ in range(G)]
# Score each output with the reward: verifier, judge or learned model
rewards = [reward(prompt, o) for o in outputs]
# Choose the advantage: group mean, critic with GAE, running mean or none
adv = advantage(rewards, outputs)  # see Variants
# Optionally divide by the standard deviation [arxiv:2504.06958 eq. 1]
adv = adv / (std(rewards) + 1e-4)
# Score the sampled tokens under the current policy
logp = policy.logprob(outputs)
# Add a KL penalty to a reference model if the variant uses one
loss = -mean(adv * logp) + beta * kl(policy, ref)
# PPO-style variants clip the probability ratio here [arxiv:2102.02202 tab. 2]

Contradictions

No two cards report incompatible results on the same comparison. Several pairs differ in setting, so the cards do not test each other's claims.

• KL to a reference. SIRL finds that a partial KL term helps, 51.7 against 47.3 2505.11792 tab. 3. Conductor trains with KL weight 0 2512.04388 app. A. The two cards use different tasks and different reward designs, so neither result tests the other.
• Critic against group baseline. SAO reports that a single rollout with a critic beats GRPO on AIME2025 2607.07508 tab. 1. The group-relative cards use no critic and do not run this comparison 2503.16219 §3 2512.04388 app. A.
• Within-paper conflicts. Several cards show conflicts inside one paper, and the page does not resolve them. These include the DPPO batch-size tables 2409.00588 app. C.10, the DreamerV3 horizon [arxiv:2301.04104 tab. 4], and the wd1 Sudoku gain in the text and table 2507.08838 §5.1.

Papers

• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient: token-level REINFORCE with Monte Carlo action values for text generation.
• Deep Reinforcement Learning for Dialogue Generation: two-stage policy gradient for dialogue, with a neural baseline in one stage only.
• Adversarial Learning for Neural Dialogue Generation: discriminator reward with a learned critic baseline and Monte Carlo rewards per prefix.
• Deep Reinforcement Learning framework for Autonomous Driving: deterministic actor-critic for lane keeping in simulation; not a score-function method.
• Long Text Generation via Adversarial Training with Leaked Information: hierarchical Manager and Worker with a goal vector and intrinsic reward.
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning: deterministic actor-critic with H-step model value expansion and the TD-k critic trick; learning curves only.
• Decoupling Strategy and Generation in Negotiation Dialogues: act-level policy gradient with a running-mean baseline.
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware: REINFORCE as the alternative to gradient-based gates for latency-aware search.
• Natural Language Processing Advancements By Deep Learning: A Survey: survey; mentions policy gradient in one section, no method.
• Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks: asynchronous actor-critic for per-slot offloading.
• Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems: REINFORCE with a greedy-rollout baseline across several decoders.
• Embodied Intelligence via Learning and Evolution: PPO as the inner learner for co-evolved bodies and controllers.
• Challenges and Outlook in Robotic Manipulation of Deformable Objects: survey and position paper; no policy-gradient content beyond a list item.
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning: GPU simulation with PPO and GAE at very large environment counts.
• Handling Distribution Shifts on Graphs: An Invariance Perspective: REINFORCE for learned edge editors with no recorded baseline.
• Masked Visual Pre-training for Motor Control: frozen masked-autoencoder features feeding a PPO actor-critic.
• MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge: PPO with self-imitation and a dense learned reward.
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents: imitation followed by on-line policy gradient with a learned value baseline.
• Mastering Diverse Domains through World Models: actor trained by REINFORCE on imagined rollouts with a critic baseline.
• Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments: simulation framework with PPO wrappers; no method of its own.
• Check Your Facts and Try Again: REINFORCE policy shown only in a learning curve for a tool-augmented LLM.
• Diffusion Policy Policy Optimization: PPO over the denoising chain of a diffusion policy.
• Visual-RFT: Visual Reinforcement Fine-Tuning: group-relative policy gradient with rule rewards for vision-language models.
• Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't: critic-free group policy gradient on a 1.5B reasoning model.
• VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning: group-relative objective with IoU, answer and format rewards for video.
• Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling: REINFORCE++ with a batch-normalised advantage and partial KL.
• wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models: ratio-free softmax-weighted policy optimisation for masked diffusion LLMs.
• Step-entropy compression of reasoning traces: GRPO with a composite reward that teaches a model to emit [SKIP] for low-entropy reasoning steps.
• There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning: two-level PPO for multi-turn red teaming with token-level credit.
• DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents: group-normalised objective with replay for a travel-planning agent.
• ExGRPO: Learning to Reason from Experience: GRPO with experience replay of correct rollouts.
• Learning to Orchestrate Agents in Natural Language with the Conductor: group-relative training of a 7B orchestrator with KL weight 0.
• Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids: PPO with an asymmetric critic for end-to-end humanoid parkour.
• Bayesian policy gradient and actor-critic algorithms: Gaussian-process prior on the gradient, computed by Bayesian quadrature.
• KL for a KL: On-Policy Distillation with Control Variate Baseline: per-token control-variate baseline for on-policy distillation without a critic.
• Ad Headline Generation using Self-Critical Masked Language Model: masked-LM headline generator fine-tuned with a self-critical greedy baseline and ROUGE-L reward.
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning: one rollout per prompt with a critic and importance ratio from the rollout engine.
• DEF-PG, a theory paper on sample complexity: theory for a recursive variance-reduced estimator without experiments.
• Learning to Play: Reinforcement Learning and Games: textbook chapter on the AlphaGo reinforcement-learning stage.