--- kind: concept slug: policy-gradient title: Policy gradient summary: >- Training a policy by raising the probability of sampled actions in proportion to a return or advantage, so the reward need not be differentiable. Variants differ in the baseline: a learned critic with GAE, a group of rollouts per prompt, a running mean, a greedy rollout, or none. Costs: extra rollouts, a critic, or a reward model queried at every step. tags: [reinforcement-learning, optimization, text-generation] papers: ["1606.01541", "1609.05473", "1701.06547", "1704.02532", "1709.08624", "1803.00101", "1808.09637", "1812.00332", "2003.01200", "2011.08430", "2012.10638", "2102.02202", "2105.01767", "2108.10470", "2202.02466", "2203.06173", "2206.08853", "2207.01206", "2301.04104", "2301.04195", "2302.12813", "2409.00588", "2503.01785", "2503.16219", "2504.06958", "2505.11792", "2507.08838", "2508.03346", "2508.04451", "2509.21842", "2510.02245", "2512.04388", "2601.07718", "2604.27563", "2605.07865", "2607.06818", "2607.07508", "2610.03165", "2610.17391"] related: [actor-critic, baseline-subtraction, proximal-policy-optimization, trust-region-method, importance-weighting, experience-replay, reinforcement-learning-verifiable-rewards, reward-modeling, reward-design, model-based-reinforcement-learning, world-model, variance-reduced-gradient, on-policy-distillation, knowledge-distillation, neural-architecture-search, generative-adversarial-network, discrete-diffusion, bayesian-quadrature, offline-reinforcement-learning, reinforcement-learning-evaluation, seed-variance-evaluation, chain-of-thought-compression, masked-language-modeling] status: draft updated_at: 2026-11-01 --- ## Overview A policy-gradient method samples an action or a whole output from the current policy, scores it, and raises the log-probability of that output in proportion to its weight [arxiv:1609.05473 eq. 6]. The reward enters only as a multiplier, so it need not be differentiable. The cards use discriminator scores [arxiv:1701.06547 eq. 1], simulator returns [arxiv:2108.10470 §6.1.1], exact-match verifiers [arxiv:2503.16219 §2], solver execution results [arxiv:2505.11792 eq. 6], learned video-text similarity [arxiv:2206.08853 §4.2] and ROUGE-L overlap with approved headlines [arxiv:2607.06818 §4.1]. The estimator is noisy, so every variant subtracts a typical value from the return, or normalises it, before the update. Variants differ in four places: 1. **The baseline or advantage.** A learned value function [arxiv:2102.02202 app. A], the mean over a group of rollouts for the same prompt [arxiv:2503.01785 eq. 4], a running average of past returns [arxiv:1808.09637 eq. 1], a greedy rollout [arxiv:2012.10638 eq. 16–18], a per-token KL term [arxiv:2605.07865 eq. 10], or nothing [code: sequence_gan.py#L146-L147]. 2. **What counts as an action.** A token [arxiv:1609.05473 eq. 6], a denoising step of a diffusion policy [arxiv:2409.00588 app. C.10], an act-level decision above a word generator [arxiv:1808.09637 §3.4], a guide chosen per turn [arxiv:2508.04451 eq. 1], a candidate architecture [arxiv:1812.00332 eq. 8], or the parameters of a masked-diffusion language model [arxiv:2507.08838 §3.1]. 3. **Stabilisers.** A probability-ratio clip [arxiv:2102.02202 tab. 2], a KL penalty to a reference model [arxiv:2505.11792 tab. 6], an entropy term [arxiv:2102.02202 tab. 2], or a replay buffer of past successes [arxiv:2510.02245 §4.2]. 4. **Where the rollouts come from.** The real environment, a world model [arxiv:2301.04104 tab. 4], a cached tool sandbox [arxiv:2509.21842 tab. 7], or a teacher's per-token log-probabilities [arxiv:2605.07865 §3.1]. Four cards are not method sources for this page. Two are surveys or position papers that discuss the method without an experiment [arxiv:2003.01200 §3.3] [arxiv:2105.01767 §5.1]. Two are simulation frameworks that use PPO as a tool [arxiv:2301.04195 §5.1] [arxiv:2108.10470 §4]. One is a textbook chapter [arxiv:2610.17391 app. B]. Their content is used only for the throughput and training-setup notes in the Variants section. ## Variants ### Learned value baseline with GAE (PPO-style) The policy is updated with a clipped surrogate and a learned value function that supplies the advantage through generalised advantage estimation. This is the most common setting in robot learning and simulation. Settings from the cards: - **Massively parallel simulation.** Isaac Gym runs 4096 to 16384 environments on the GPU, with GAE gamma 0.99, lambda 0.95 and clip 0.2 [arxiv:2108.10470 §4] [arxiv:2108.10470 tab. 17]. The paper reports an Ant reward of 3000 in about 20 seconds [arxiv:2108.10470 §6.1.1]. The speed-ups compare different hardware and algorithms, with no iso-compute test [arxiv:2108.10470 §6.4.1]. Card: [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](/papers/2108.10470). - **Co-evolved bodies and controllers.** DERL trains each child's PPO controller from random weights for 5 million interactions before its fitness is known [arxiv:2102.02202 app. A]. Its PPO table uses batch 512, 4 epochs, clip 0.2, Adam at 3e-4 with linear decay [arxiv:2102.02202 tab. 2]. The full run is reported at 1152 CPUs and under 16 hours [arxiv:2102.02202 §DERL]. Card: [Embodied Intelligence via Learning and Evolution](/papers/2102.02202). - **Frozen visual encoder.** A frozen masked-autoencoder feature feeds a small PPO actor-critic. The paper reports success on 7 of 8 pixel tasks against a supervised ViT-S, by up to 80 points [arxiv:2203.06173 §5.1]. Unfreezing the encoder gives zero success for both random and MAE initialisation [arxiv:2203.06173 §5.3]. Card: [Masked Visual Pre-training for Motor Control](/papers/2203.06173). - **Learned dense reward.** MineDojo uses PPO with self-imitation and a video-text model that scores every step. That model has about 150 million parameters and is queried at every step; caching reduces the cost but does not remove it [arxiv:2206.08853 §4.2]. The paper reports 83.5 ± 7.1 on Hunt Cow against 48.3 for a manual reward [arxiv:2206.08853 tab. 1], and 1.3% zero-shot success on novel tasks [arxiv:2206.08853 tab. 5]. Card: [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](/papers/2206.08853). - **Humanoid locomotion with an asymmetric critic.** The policy is trained with PPO, 4096 agents on one RTX 4090 [arxiv:2601.07718 app. B]. On hardware, stair ascent succeeds 10 of 10 times with an edge penalty against 8 of 10 without it [arxiv:2601.07718 tab. 1]. Card: [Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids](/papers/2601.07718). The paper is CC BY 4.0, but the linked code is CC BY-NC, so no code is cited here. - **Diffusion policy fine-tuning.** DPPO treats each denoising step as a Gaussian policy inside a nested MDP. The advantage is GAE with the environment discount, scaled by a per-denoising-step discount [arxiv:2409.00588 app. C.10]. The paper reports 80% (16/20) zero-shot sim-to-real on One-Leg, against 0% for a fine-tuned Gaussian policy [arxiv:2409.00588 §5.4]. Each environment step costs K network evaluations [arxiv:2409.00588 app. D]. Card: [Diffusion Policy Policy Optimization](/papers/2409.00588). - **Imagination.** DreamerV3 trains its actor by REINFORCE on 15-step imagined rollouts, with a learned critic as the baseline [arxiv:2301.04104 tab. 4]. The paper uses the same hyperparameters for every benchmark [arxiv:2301.04104 tab. 4]. It reports 9.1 Minecraft diamond return at 100 million steps, against 5.1 for PPO [arxiv:2301.04104 tab. 5], using 8.9 A100 GPU-days [arxiv:2301.04104 tab. 2]. Card: [Mastering Diverse Domains through World Models](/papers/2301.04104). - **Web agents.** WebShop first imitates a search model and a choice model, then fine-tunes the choice model with on-line policy gradient using a learned value baseline [arxiv:2207.01206 §4.3]. The best result is score 62.4 and success 28.7% [arxiv:2207.01206 §5.2] [arxiv:2207.01206 tab. 2]. Starting RL from BERT without imitation scores below the rule baseline [arxiv:2207.01206 §5.2]. Card: [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](/papers/2207.01206). - **Edge computation offloading.** An asynchronous actor-critic chooses per-slot offloading actions. Its reported cost is lowest among the compared methods, but the values are read from plots [arxiv:2011.08430 fig. 4]. The learning rate of 0.001 gives the lowest curve [arxiv:2011.08430 fig. 8]. Card: [Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks](/papers/2011.08430). - **Critic as baseline for REINFORCE.** Dialogue generation with a discriminator reward uses a learned critic as the baseline [arxiv:1701.06547 eq. 2]. Card: [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547). ### Group-relative baseline without a critic (GRPO family) For each prompt, sample a group of G outputs, score them, and normalise the rewards inside the group. No value network is trained. The advantage is the group-normalised reward [arxiv:2503.01785 eq. 4] [arxiv:2504.06958 eq. 1]. This is the main variant for language and vision-language models with a verifiable reward. - **Verifiable rewards on small models.** Open-RS trains a 1.5B reasoning model with G = 6, learning rate 1e-6 and a 500-step cap on four A40 GPUs for 24 hours [arxiv:2503.16219 §3] [arxiv:2503.16219 tab. 5]. Quality peaks at 50 to 100 steps and then degrades [arxiv:2503.16219 §3] [arxiv:2503.16219 fig. 4]. The paper reports AIME24 at 46.7 after 50 steps, against 28.8 for the start model [arxiv:2503.16219 tab. 1]. Card: [Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't](/papers/2503.16219). - **Detection and grounding with rule rewards.** Visual-RFT uses an IoU reward for boxes and an accuracy reward for classes [arxiv:2503.01785 §3.2.1]. The paper leaves the group size and the KL weight out; the release script uses 8 [code: src/scripts/2B_base65cate_6k.sh#L31]. Card: [Visual-RFT: Visual Reinforcement Fine-Tuning](/papers/2503.01785). - **Video grounding.** VideoChat-R1 sums an IoU reward, an answer reward, a format reward and a caption-recall reward per task [arxiv:2504.06958 §3.1]. It reports Charades-STA mIoU from 29.0 to 60.8 [arxiv:2504.06958 tab. 1]. The paper does not write out the objective [arxiv:2504.06958 §3.1]. Card: [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](/papers/2504.06958). - **Agentic planning with replay.** DeepTravel samples 8 rollouts per query [arxiv:2509.21842 §4] and uses a clipped, group-normalised objective with a replay step [arxiv:2509.21842 eq. 3] [arxiv:2509.21842 §3.3.2]. Groups whose reward spread is at most 0.1 are dropped. Its online pass rate for the 32B model is 82.58% [arxiv:2509.21842 tab. 4]. Card: [DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents](/papers/2509.21842). - **Experience replay for RLVR.** ExGRPO uses K = 8 rollouts and batch 128 questions [arxiv:2510.02245 §5.1]. Half of each batch replays a stored low-entropy correct answer [arxiv:2510.02245 §4.2]. It reports a math average of 48.3 against 45.3 for on-policy GRPO [arxiv:2510.02245 tab. 1], at 19% more step time [arxiv:2510.02245 app. E.5]. Card: [ExGRPO: Learning to Reason from Experience](/papers/2510.02245). - **Large group, no KL.** The Conductor trains a 7B orchestrator with G = 64 rollouts per question, 200 iterations and KL weight 0 [arxiv:2512.04388 app. A] [arxiv:2512.04388 §4.1]. Its unconstrained LiveCodeBench v6 score is 83.93, against 82.90 for GPT 5 [arxiv:2512.04388 tab. 1]. Card: [Learning to Orchestrate Agents in Natural Language with the Conductor](/papers/2512.04388). - **Ratio-free and reference-free.** wd1 turns group-relative advantages into softmax weights on a positive and a negative weighted log-likelihood [arxiv:2507.08838 §3.1] [arxiv:2507.08838 eq. 9]. It needs one likelihood estimate per update and reports 1280 rollouts to its best model, against 30000 for d1 [arxiv:2507.08838 tab. 8]. Its likelihood is biased [arxiv:2507.08838 §2.3]. Card: [wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models](/papers/2507.08838). - **Batch-normalised REINFORCE++.** SIRL normalises advantages across the mini-batch rather than within a group [arxiv:2505.11792 §3.2] [arxiv:2505.11792 eq. 4]. It places the outcome reward on the end-of-sequence token and applies KL only to the model and code sections [arxiv:2505.11792 §3.2]. Its partial KL gives 51.7 against 47.3 with no KL on MAMO Complex [arxiv:2505.11792 tab. 3]. Card: [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](/papers/2505.11792). - **Learned step skipping with a composite reward.** Low-entropy reasoning steps are first replaced by `[SKIP]` in training traces, then GRPO teaches the model to skip on its own. The reward adds correctness, a skip-ratio tier, a skip-count penalty and a response-length penalty [arxiv:2508.03346 §3.3] [arxiv:2508.03346 eq. 4]. The paper uses a group of 14 samples per prompt [arxiv:2508.03346 app. C]. In its reward ablation, correctness plus skip ratio alone gives MATH500 accuracy 51.00, against 85.00 with all four terms [arxiv:2508.03346 tab. 5]. The full SFT and GRPO recipe reports 35% fewer MATH500 tokens at 85.00 accuracy, against 88.17 for the base model [arxiv:2508.03346 tab. 3]. Card: [step-entropy compression of reasoning traces](/papers/2508.03346). ### Single rollout per prompt Some agentic tasks cannot afford a group of long episodes. SAO uses one rollout per prompt and corrects for the policy lag with the rollout engine's own log-probabilities [arxiv:2607.07508 §3.1]. It updates a critic twice per batch, with attention frozen [arxiv:2607.07508 §3.2]. On AIME2025 it reports 97.3, against 84.2 for GRPO with a Python tool and 93.5 for GRPO with DIS [arxiv:2607.07508 tab. 1]. Its ablation reports 95.0 with one critic update per batch [arxiv:2607.07508 tab. 3]. The value model needs a large pretraining corpus [arxiv:2607.07508 §3.2], and throughput and memory were not measured [arxiv:2607.07508 §1]. The results come from one training seed [arxiv:2607.07508 §4.1]. Card: [Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning](/papers/2607.07508). ### Running mean, greedy rollout or teacher-based baselines - **Self-critical greedy baseline for a masked language model.** For each campaign, SC-MLM samples one headline and also decodes a greedy headline from the same model. The update weights the sample's log-probability by the difference of their ROUGE-L F1 rewards [arxiv:2607.06818 §3.3] [arxiv:2607.06818 eq. 9]. The masking rate and the exact greedy decode are not stated [arxiv:2607.06818 §3.1] [arxiv:2607.06818 §3.3]. The ROUGE-L table reports 6.33 for the full model against 5.08 for the same model without the RL step, both as differences from a bi-LSTM baseline [arxiv:2607.06818 tab. 1]. Because the reward is the metric in that table, a crowd audit is the independent check: 6.53% more headlines are rated 3 of 3 than human-submitted ones [arxiv:2607.06818 tab. 2]. Card: [Ad Headline Generation using Self-Critical Masked Language Model](/papers/2607.06818). - **Running mean.** The negotiation act manager is trained by policy gradient on a reward of utility, fairness or length. Each token's log-likelihood is weighted by the reward minus the average return so far [arxiv:1808.09637 eq. 1] [arxiv:1808.09637 §3.4]. Card: [Decoupling Strategy and Generation in Negotiation Dialogues](/papers/1808.09637). - **Greedy-rollout baseline with several decoders.** MDAM trains five decoders with REINFORCE. The baseline is the shortest greedy tour among the decoders of the previous best model [arxiv:2012.10638 eq. 16–18]. A KL term on first-step distributions, with weight 0.01, pushes the decoders apart [arxiv:2012.10638 §4] [arxiv:2012.10638 eq. 19]. It reports a TSP100 gap of 0.38% with batch 50 [arxiv:2012.10638 tab. 1]. The KL weight was tuned on TSP20 only [arxiv:2012.10638 §4]. Card: [Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems](/papers/2012.10638). - **Per-token baseline from the forward pass.** On-policy distillation uses the teacher-minus-student log-probability as reward. The baseline is the detached per-token negative reverse KL, which needs no critic [arxiv:2605.07865 §3.1] [arxiv:2605.07865 eq. 10]. The top-k variant computes the KL over the student's top 20 tokens [arxiv:2605.07865 §3.2] [arxiv:2605.07865 §4.1]. It reports a 1.7B math average of 33.0, against 29.8 for plain on-policy distillation [arxiv:2605.07865 tab. 2], and up to 57.7% less wall-clock than the full-vocabulary version [arxiv:2605.07865 fig. 3]. Card: [KL for a KL: On-Policy Distillation with Control Variate Baseline](/papers/2605.07865). ### No baseline (plain REINFORCE on sampled returns) - **Sequence GAN.** The LSTM generator is a policy over tokens. Action values come from Monte Carlo roll-outs, which give every token a value [arxiv:1609.05473 eq. 3] [arxiv:1609.05473 eq. 4]. The code uses no baseline [code: sequence_gan.py#L146-L147]. The repository has no license, so only the path is cited. The paper reports synthetic NLL of 8.736 against 9.038 for MLE [arxiv:1609.05473 tab. 1]. The discriminator-step count behind that table is ambiguous: one step in the compared settings, five in the code [arxiv:1609.05473 fig. 3]. Card: [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](/papers/1609.05473). - **Hierarchical generator with a goal.** LeakGAN gives a Manager a policy-gradient update weighted by its Q-value and the cosine between the feature change and the goal [arxiv:1709.08624 eq. 7]. The Worker gets REINFORCE with an intrinsic cosine reward [arxiv:1709.08624 eq. 8] [arxiv:1709.08624 eq. 9]. At length 40 it reports synthetic NLL of 7.191, against 10.310 for SeqGAN [arxiv:1709.08624 tab. 1]. The appendix says a fully connected layer instead of the hierarchy was unstable [arxiv:1709.08624 app. Discussions]. Card: [Long Text Generation via Adversarial Training with Leaked Information](/papers/1709.08624). - **Dialogue with a dialogue-level reward.** The two-agent stage sums rewards over simulated turns with no baseline and no discount [arxiv:1606.01541 eq. 8] [arxiv:1606.01541 eq. 9]. The mutual-information stage uses a neural baseline [arxiv:1606.01541 eq. 7]. The paper reports a five-turn quality win rate of 0.72 against 0.12 [arxiv:1606.01541 tab. 5], with no significance test [arxiv:1606.01541 §5.2]. Card: [Deep Reinforcement Learning for Dialogue Generation](/papers/1606.01541). - **Editor learning in graphs.** Learned edge editors create virtual environments, trained by REINFORCE with the variance of the GNN's risks as reward [arxiv:2202.02466 app. A]. No baseline is recorded for the editors. The paper reports up to 7.0% ROC-AUC gain on Twitch [arxiv:2202.02466 §5.2]. Each editor is a dense matrix over the graph's nodes [code: temp_arxiv/main.py#L42]. Card: [Handling Distribution Shifts on Graphs: An Invariance Perspective](/papers/2202.02466). - **Architecture gates.** ProxylessNAS uses REINFORCE for the non-differentiable latency reward [arxiv:1812.00332 eq. 8]. The gate gradient it uses by default is biased and unmeasured [arxiv:1812.00332 §3.2.1]. Card: [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](/papers/1812.00332). - **Tool-augmented LLM.** LLM-Augmenter trains a T5 policy with REINFORCE, but only in a learning curve [arxiv:2302.12813 eq. 1] [arxiv:2302.12813 fig. 3]. Its main results use a rule policy. Card: [Check Your Facts and Try Again](/papers/2302.12813). ### Variance-reduced, Bayesian and theory variants - **Bayesian policy gradient.** The gradient integrand gets a Gaussian-process prior, and the posterior mean is computed by Bayesian quadrature with a Fisher kernel [arxiv:2604.27563 §4]. On the LQR task, BPG and its natural-gradient form beat MC policy gradient at 10, 20 and 40 trajectories, but not at 5 [arxiv:2604.27563 §6.2.2] [arxiv:2604.27563 fig. 3]. Other tasks are simulation only [arxiv:2604.27563 fig. 6]. The card records no official code. Card: [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563). - **Defensive variance reduction.** DEF-PG is a PAGE-style recursive gradient without a baseline [arxiv:2610.03165 §3] [arxiv:2610.03165 alg. 2]. It claims O(ε^-3) stopped trajectories under two assumptions, against O(ε^-4) for GPOMDP [arxiv:2610.03165 §4 Thm. 6] [arxiv:2610.03165 §4 Prop. 3]. The score bound it needs generally fails for Gaussian policies [arxiv:2610.03165 §4 Rem. 2]. It has no experiments [arxiv:2610.03165 §7]. Card: [DEF-PG, a theory paper on sample complexity](/papers/2610.03165). - **Self-play game agents.** The AlphaGo reinforcement-learning stage is REINFORCE on self-play games [arxiv:2610.17391 app. B]. The reported RL policy without search wins 85% against Pachi, against 11% for the supervised-only policy [arxiv:2610.17391 app. B]. These numbers come from the original papers and were not rerun. Card: [Learning to Play: Reinforcement Learning and Games](/papers/2610.17391). ### Not a score-function method The DDAC actor for lane keeping takes the gradient of the critic with respect to the action and multiplies it by the actor's gradient [arxiv:1704.02532 §3.2]. It is a deterministic actor-critic, not a policy gradient in the sense used here. The paper reports no metric values [arxiv:1704.02532 §5]. Card: [Deep Reinforcement Learning framework for Autonomous Driving](/papers/1704.02532). Model-based value expansion is also a deterministic actor-critic, built on the deterministic policy gradient [arxiv:1803.00101 §2.1] [arxiv:1803.00101 §2.2]. It rolls a learned one-step model forward H steps and bootstraps from the critic [arxiv:1803.00101 eq. 2]. Its horizon is set by hand per task: H = 10 on cheetah and swimmer, H = 3 on walker [arxiv:1803.00101 fig. 3 caption]. The results are learning curves on three MuJoCo tasks with 4 seeds for the learned model [arxiv:1803.00101 §4] [arxiv:1803.00101 app. A]. The paper does not state a discount. Card: [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](/papers/1803.00101). ## When to use which Choose by what you can afford per update and what the reward looks like. | Situation | Use | Why | |---|---|---| | Dense reward, many cheap parallel simulator environments | PPO with GAE and a learned value | Isaac Gym uses 4096 to 16384 environments [arxiv:2108.10470 tab. 17] | | Verifiable answer, no room for a critic in memory | GRPO-style group baseline | G = 6 ran in 24 hours on four GPUs [arxiv:2503.16219 §3] | | One long episode costs as much as a group | Single rollout with a critic | 97.3 vs 84.2 on AIME2025 in one comparison [arxiv:2607.07508 tab. 1], one seed | | Real environment is slow and a world model is feasible | Learned-model imagination | 8.9 A100 GPU-days for 100 million steps [arxiv:2301.04104 tab. 2] | | Per-step learned reward must be queried | Budget for the reward model | About 150 million parameters, queried at every step [arxiv:2206.08853 §4.2] | | Discrete construction with a cheap greedy rollout | REINFORCE with a greedy baseline | TSP100 gap 0.38% at batch 50 [arxiv:2012.10638 tab. 1] | | Pretrained diffusion policy to fine-tune online | Diffusion PPO | K network evaluations per environment step [arxiv:2409.00588 app. D] | | Teacher with per-token log-probabilities | Control-variate on-policy distillation | Baseline without a critic [arxiv:2605.07865 §3.1] | | Diffusion language model, no likelihood ratio | wd1-style softmax weights | 1280 vs 30000 rollouts to best model [arxiv:2507.08838 tab. 8] | | Reward is a discriminator on long sequences | Hierarchical goal (LeakGAN) or plain Monte Carlo (SeqGAN) | LeakGAN NLL 7.191 vs 10.310 at length 40 [arxiv:1709.08624 tab. 1] | | Need a convergence bound | Theory only | DEF-PG has no experiments [arxiv:2610.03165 §7] | | Masked or non-autoregressive text generator with a cheap greedy decode and a metric reward | Self-critical baseline from the greedy output | ROUGE-L 6.33 vs 5.08 without the RL step, author-reported [arxiv:2607.06818 tab. 1]; proprietary data | | Learned model is only accurate for a few steps | Short-horizon value expansion with a deterministic actor | H = 3 needed on walker, H = 10 on cheetah and swimmer [arxiv:1803.00101 fig. 3 caption] | | Thinking tokens dominate inference cost | GRPO with skip-count and length terms in the reward | Without them, MATH500 accuracy is 51.00 against 85.00 [arxiv:2508.03346 tab. 5] | Two cautions apply across rows. Group and single-rollout results come from one training seed or a few seeds, and several cards flag that [arxiv:2607.07508 §4.1] [arxiv:2503.16219 §3]. Most headline numbers in these cards are author-reported, and the comparison baselines often differ in compute. ## Implementation sketch The loop below covers the common core. The lines that change between variants are the advantage and the optional clip and KL terms. ```pseudocode # Sample a group of G outputs for each prompt [arxiv:2503.01785 eq. 4] outputs = [policy.sample(prompt) for _ in range(G)] # Score each output with the reward: verifier, judge or learned model rewards = [reward(prompt, o) for o in outputs] # Choose the advantage: group mean, critic with GAE, running mean or none adv = advantage(rewards, outputs) # see Variants # Optionally divide by the standard deviation [arxiv:2504.06958 eq. 1] adv = adv / (std(rewards) + 1e-4) # Score the sampled tokens under the current policy logp = policy.logprob(outputs) # Add a KL penalty to a reference model if the variant uses one loss = -mean(adv * logp) + beta * kl(policy, ref) # PPO-style variants clip the probability ratio here [arxiv:2102.02202 tab. 2] ``` ## Contradictions No two cards report incompatible results on the same comparison. Several pairs differ in setting, so the cards do not test each other's claims. - **KL to a reference.** SIRL finds that a partial KL term helps, 51.7 against 47.3 [arxiv:2505.11792 tab. 3]. Conductor trains with KL weight 0 [arxiv:2512.04388 app. A]. The two cards use different tasks and different reward designs, so neither result tests the other. - **Critic against group baseline.** SAO reports that a single rollout with a critic beats GRPO on AIME2025 [arxiv:2607.07508 tab. 1]. The group-relative cards use no critic and do not run this comparison [arxiv:2503.16219 §3] [arxiv:2512.04388 app. A]. - **Within-paper conflicts.** Several cards show conflicts inside one paper, and the page does not resolve them. These include the DPPO batch-size tables [arxiv:2409.00588 app. C.10], the DreamerV3 horizon [arxiv:2301.04104 tab. 4], and the wd1 Sudoku gain in the text and table [arxiv:2507.08838 §5.1]. ## Papers - [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](/papers/1609.05473): token-level REINFORCE with Monte Carlo action values for text generation. - [Deep Reinforcement Learning for Dialogue Generation](/papers/1606.01541): two-stage policy gradient for dialogue, with a neural baseline in one stage only. - [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547): discriminator reward with a learned critic baseline and Monte Carlo rewards per prefix. - [Deep Reinforcement Learning framework for Autonomous Driving](/papers/1704.02532): deterministic actor-critic for lane keeping in simulation; not a score-function method. - [Long Text Generation via Adversarial Training with Leaked Information](/papers/1709.08624): hierarchical Manager and Worker with a goal vector and intrinsic reward. - [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](/papers/1803.00101): deterministic actor-critic with H-step model value expansion and the TD-k critic trick; learning curves only. - [Decoupling Strategy and Generation in Negotiation Dialogues](/papers/1808.09637): act-level policy gradient with a running-mean baseline. - [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](/papers/1812.00332): REINFORCE as the alternative to gradient-based gates for latency-aware search. - [Natural Language Processing Advancements By Deep Learning: A Survey](/papers/2003.01200): survey; mentions policy gradient in one section, no method. - [Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks](/papers/2011.08430): asynchronous actor-critic for per-slot offloading. - [Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems](/papers/2012.10638): REINFORCE with a greedy-rollout baseline across several decoders. - [Embodied Intelligence via Learning and Evolution](/papers/2102.02202): PPO as the inner learner for co-evolved bodies and controllers. - [Challenges and Outlook in Robotic Manipulation of Deformable Objects](/papers/2105.01767): survey and position paper; no policy-gradient content beyond a list item. - [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](/papers/2108.10470): GPU simulation with PPO and GAE at very large environment counts. - [Handling Distribution Shifts on Graphs: An Invariance Perspective](/papers/2202.02466): REINFORCE for learned edge editors with no recorded baseline. - [Masked Visual Pre-training for Motor Control](/papers/2203.06173): frozen masked-autoencoder features feeding a PPO actor-critic. - [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](/papers/2206.08853): PPO with self-imitation and a dense learned reward. - [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](/papers/2207.01206): imitation followed by on-line policy gradient with a learned value baseline. - [Mastering Diverse Domains through World Models](/papers/2301.04104): actor trained by REINFORCE on imagined rollouts with a critic baseline. - [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](/papers/2301.04195): simulation framework with PPO wrappers; no method of its own. - [Check Your Facts and Try Again](/papers/2302.12813): REINFORCE policy shown only in a learning curve for a tool-augmented LLM. - [Diffusion Policy Policy Optimization](/papers/2409.00588): PPO over the denoising chain of a diffusion policy. - [Visual-RFT: Visual Reinforcement Fine-Tuning](/papers/2503.01785): group-relative policy gradient with rule rewards for vision-language models. - [Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't](/papers/2503.16219): critic-free group policy gradient on a 1.5B reasoning model. - [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](/papers/2504.06958): group-relative objective with IoU, answer and format rewards for video. - [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](/papers/2505.11792): REINFORCE++ with a batch-normalised advantage and partial KL. - [wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models](/papers/2507.08838): ratio-free softmax-weighted policy optimisation for masked diffusion LLMs. - [Step-entropy compression of reasoning traces](/papers/2508.03346): GRPO with a composite reward that teaches a model to emit `[SKIP]` for low-entropy reasoning steps. - [There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning](/papers/2508.04451): two-level PPO for multi-turn red teaming with token-level credit. - [DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents](/papers/2509.21842): group-normalised objective with replay for a travel-planning agent. - [ExGRPO: Learning to Reason from Experience](/papers/2510.02245): GRPO with experience replay of correct rollouts. - [Learning to Orchestrate Agents in Natural Language with the Conductor](/papers/2512.04388): group-relative training of a 7B orchestrator with KL weight 0. - [Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids](/papers/2601.07718): PPO with an asymmetric critic for end-to-end humanoid parkour. - [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563): Gaussian-process prior on the gradient, computed by Bayesian quadrature. - [KL for a KL: On-Policy Distillation with Control Variate Baseline](/papers/2605.07865): per-token control-variate baseline for on-policy distillation without a critic. - [Ad Headline Generation using Self-Critical Masked Language Model](/papers/2607.06818): masked-LM headline generator fine-tuned with a self-critical greedy baseline and ROUGE-L reward. - [Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning](/papers/2607.07508): one rollout per prompt with a critic and importance ratio from the rollout engine. - [DEF-PG, a theory paper on sample complexity](/papers/2610.03165): theory for a recursive variance-reduced estimator without experiments. - [Learning to Play: Reinforcement Learning and Games](/papers/2610.17391): textbook chapter on the AlphaGo reinforcement-learning stage.