Actor-critic

Overview

An actor-critic method trains two learned parts together. The actor is the policy that picks actions. The critic estimates how good a state, or a state-action pair, is under the current behaviour. The actor is then updated from the critic's scores. Some variants use the critic as a baseline and follow an advantage 2301.04104 § Actor learning 2607.07508 § 3.2. Others follow the gradient of the critic's score with respect to the action 1904.12304 eq. 6 2206.07568 § 4.3 2106.06860 eq. 3.

The twenty papers below share this two-part structure but differ in what the critic is, where its training data comes from, and how the actor uses it. Two of them are surveys, not new methods. The offline RL survey treats an actor-critic learner plus modifications as the general form of an offline method 2203.01387 § III, and the FPGA survey groups actor-critic hardware designs by RL family 2610.28699 § 3.6:

• The critic is a value function over imagined rollouts 2301.04104 § Critic learning.
• The critic is a twin off-policy Q-function fed from a replay buffer of thousands of parallel simulated environments. Its target averages the two critics, and the actor maximises entropy-regularised Q 2512.01996 § 2.1 2512.01996 alg. 1.
• The critic estimates the value of goal-driven behaviour inside a world model. A fixed controller turns each goal into an action, and the actor is trained by backpropagating value gradients through imagined sequences 2512.02417 § 4.2.1 2512.02417 § 4.2.3.
• The critic is a Gaussian-process posterior over Q with a Fisher kernel on state-action pairs, fitted from observed transitions with a Bellman-type observation model 2604.27563 § 7.1 2604.27563 § 7.2.
• The critic is a value model over the tokens of an LLM answer 2607.07508 § 3.2.
• The critic is read once per complete agent action, at the last token of the state, and the whole action gets one advantage 2604.18401 § 3.3.
• The critic is a contrastive score that stands in for a goal-conditioned Q-function 2206.07568 § 4.2.
• The critic is a state-action Q-function trained with a Bellman target, for an actor that emits one scalar per input 1904.12304 § 3.3.
• The critic is a twin Q-function over logged offline actions, and the actor trades Q against imitation of those actions 2106.06860 eq. 3.
• The critic is a Q-function over continuous driving actions, used as the planner head of an end-to-end driving pipeline 1704.02532 § 3.2 1704.02532 § 4.3.
• The critic is a local value function over each agent's own observation history, trained on a shared team reward, with actor and critic weights shared across agents 2011.09533 § 4.
• The actor-critic is only the learner in each column of a network that grows with the task count. The paper's own contribution is the column architecture 1606.04671 § 2 1606.04671 § 5.1.
• The critic is a Q-function that also reads privileged simulator maps, which the camera-only actor never sees. The actor's and critic's inputs differ on purpose 2605.08713 § 4.2.
• The learner is trained by TD on relabelled transitions, with each step's reward taken from an optimal-transport score against one or two demonstrations 2410.21795 eq. 1 2410.21795 eq. 5.
• The critic is the value estimate behind PPO's advantage. The advantage uses generalised advantage estimation with λ = 0.95, which needs a learned value function 2412.18865 tab. III.
• The critic approximates the costate of a distributed MPC problem over each robot's neighbourhood error, and the actor is linear in its own basis. Both are fitted online, inside each prediction interval 2412.19669 eq. 11 2412.19669 eq. 14.
• The critic is a Q-function over the actions of all base stations, and each base station's actor is trained through it 2011.09902 § 4.2 2011.09902 eq. 22.
• The critic's bootstrap target is a model-based value expansion. A learned one-step model is rolled forward H steps under the policy, the true reward is summed along the way, and the critic is evaluated at the end 1803.00101 § 3 1803.00101 eq. 2.

Variants

Imagined-rollout critic (DreamerV3)

A world model is learned from replayed experience. The actor and critic are then trained only on trajectories the model imagines, 15 steps long from each replayed start state 2301.04104 § Critic learning 2301.04104 tab. 4. The critic outputs a categorical distribution over bins and reads its value as the expectation under that distribution 2301.04104 § Robust predictions. The paper does not state the number of bins. Returns are λ-weighted with γ ≈ 0.997 (discount horizon 333) and λ = 0.95 2301.04104 tab. 4. The critic also regresses on replayed trajectories at a lower loss scale of 0.3 2301.04104 tab. 4. The actor is trained by REINFORCE on the return minus the critic's value, scaled by a running return range, with an entropy bonus of weight 3e-4 2301.04104 § Actor learning 2301.04104 tab. 4. Card: Mastering Diverse Domains through World Models.

Single-rollout token-level critic (SAO)

For LLM reinforcement learning with one rollout per prompt, the paper drops GRPO's group baseline and learns a value model 2607.07508 § 3.2. The policy update uses rollout log-probabilities as the behaviour proxy, so no separate old-policy model is kept, and it masks tokens whose importance ratio leaves a fixed interval 2607.07508 § 3.1 2607.07508 eq. 3. The value model gets two updates per policy update, and its attention parameters are frozen during RL 2607.07508 § 3.2 2607.07508 § 4.1. Environment-feedback tokens are left out of the advantage recursion 2607.07508 § 3.2. The discount factor, the optimiser and the size of the value-pretraining corpus are not stated 2607.07508 § 3.2 2607.07508 § 4.1. Card: Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning.

Action-level critic with an action-aware ratio (CAPO)

For multi-turn LLM agents, the environment changes only after a whole action, which is a reasoning note plus a tool call. CAPO therefore uses a critic at each action boundary. The critic's value is read at the last token of the state, which is conditioned on the state prefix but not on the action, so no extra encoder is needed 2604.18401 § 3.3. Each action gets one GAE advantage from action-level TD residuals, and that advantage is shared by all tokens of the action 2604.18401 § 3.3 2604.18401 eq. 2. The token probability ratios inside an action are combined into one ratio. The log-ratios are centred on a token-weighted batch mean μ̂, summed, and divided by the square root of the action length L. The result is clipped as one unit with interval [1 − ε, 1 + ε] 2604.18401 § 3.4 2604.18401 eq. 4 2604.18401 eq. 5. The paper treats the result as a length-calibrated ratio, not an exact likelihood ratio, and says a TRPO-style guarantee does not apply 2604.18401 app. A. When an action has one token, the ratio reduces to the ordinary token ratio 2604.18401 § 3.5. The paper's experiments use ε = 0.001 2604.18401 § 4.1. How μ̂ is computed, over a mini-batch or the rollout batch, is not stated 2604.18401 app. A. Per-action loss averaging is also not stated. Its ablation on WebShop with Qwen3-4B gives a score of 78.12 for the full ratio, against 73.80 for token-wise ratios and 76.18 for plain square-root aggregation without centring 2604.18401 tab. 2. Action-level GAE with γ = 0.99 and λ = 1.00 scores 78.12, against 71.86 for token-level GAE at the same γ 2604.18401 tab. 3. The paper reports 58.20 WebShop success for CAPO against 46.00 for PPO, with 3 seeds, and all results are author-reported 2604.18401 tab. 1. The critic is still trained, so each iteration costs about the same as PPO's, with a single measurement of 189.54 s against 195.37 s 2604.18401 tab. 6 2604.18401 app. F. Card: CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities.

Goal-conditioned contrastive critic (contrastive RL)

The critic is a binary contrastive classifier over (state, action, future state) triples, with its score as an inner product of two learned encoders 2206.07568 § 4.2 2206.07568 § 4.4. At the optimum, the exponential of the score equals the goal-conditioned Q-function divided by the goal's marginal density. For a fixed goal, the goal-dependent factor is the same for every action, so the actor can rank actions by the score alone 2206.07568 § 4.2. The actor is trained to maximise the critic's score at a commanded goal, with no target networks and no auxiliary losses 2206.07568 § 4.3 2206.07568 § 4.5. Card: Contrastive Learning as Goal-Conditioned Reinforcement Learning.

Deterministic critic with a one-step GAN-seed actor (DDPG)

A DDPG agent 1904.12304 § 3.3 chooses one scalar, the seed of a latent GAN, from a 128-value encoding of a partial point cloud. The critic Q(s, a) is trained with a Bellman target. The actor follows the deterministic policy gradient, which passes the critic's gradient with respect to the action back into the actor 1904.12304 eq. 6. Each episode is a single action, and the replay batch is 100 1904.12304 supp. RL Agent Details. The reward is a weighted sum of negated losses, with weights 100, 10 and 0.01 1904.12304 eq. 5 1904.12304 supp. RL Agent Details. Card: RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion.

Offline deterministic actor with a behaviour-cloning term (TD3+BC)

The method starts from TD3 and changes two things. The actor maximises λ times the critic's Q-value at the policy's action, minus the squared error between the policy's action and the logged action 2106.06860 eq. 3. The scale λ is α divided by the batch mean of |Q|, with α = 2.5, and it is not differentiated 2106.06860 eq. 5 2106.06860 § 5. States are normalised with statistics from the dataset 2106.06860 § 5. The rest is TD3 with its standard settings: twin critics, target policy smoothing, delayed actor updates and soft target updates 2106.06860 app. B 2106.06860 tab. 4. The paper reports a total normalised score of 979.3 over 15 D4RL MuJoCo v0 dataset and environment pairs, against 974.6 for Fisher-BRC and 764.3 for CQL. Training is reported at about 39 minutes, against 2h12m and 4h11m for those two methods 2106.06860 tab. 2 2106.06860 tab. 3. These numbers are author-reported. Card: A Minimalist Approach to Offline Reinforcement Learning.

Continuous-action actor-critic head (DDAC, driving)

The planner of an end-to-end driving pipeline uses a deep deterministic actor-critic head for continuous actions. The actor outputs steering, gear, brake and acceleration, and the critic scores them 1704.02532 § 3.2 1704.02532 § 5. The actor's gradient is the critic's gradient with respect to the action, multiplied by the actor's gradient with respect to its weights 1704.02532 § 3.2. The critic is trained as in DQN, against a one-step target 1704.02532 § 3.2. The paper tests lane keeping in the TORCS simulator. It describes the DDAC actions as smoother than tiled DQN actions, but reports no metrics, and it does not state the hyperparameters 1704.02532 § 5. Card: Deep Reinforcement Learning framework for Autonomous Driving.

Per-column A3C learner (progressive networks)

Progressive networks add one column per task. Each column has its own policy and value heads, and it is trained with asynchronous advantage actor-critic (A3C) using 16 workers 1606.04671 app. 3 1606.04671 § 5.1. While a new column trains, the earlier columns are frozen, and the new column reads their hidden activations through lateral connections 1606.04671 § 2 1606.04671 eq. 1. The paper does not say whether the value head gets lateral inputs 1606.04671 § 5.1 1606.04671 app. 3, and it does not state the discount factor, the n-step length or the value-loss weight 1606.04671 app. 3. The paper's contribution is the column structure; A3C is the learner it trains each column with. Card: Progressive Neural Networks.

Asymmetric SAC with a privileged critic (REAP)

The actor is a camera-only SAC policy. Its surround-view images are lifted to bird's-eye-view (BEV) features, and the target slot is given as a BEV heatmap 2605.08713 § 3 2605.08713 § 4.1. The critic gets the simulator's artifact-free BEV maps (occupancy, parking-slot detection and target), encoded by an autoencoder, together with the actor's action feature. Its loss is the autoencoder loss plus a value loss 2605.08713 § 4.2. The actor's loss is the standard SAC form with an entropy term weighted by a temperature α, whose value the paper does not state 2605.08713 eq. 5 2605.08713 § 4.2. The deployed actor sees only images, so the critic cannot run on a real car 2605.08713 § 4.2.

The actor is also pulled toward a rule-based planner. Where a Reeds-Shepp planner action exists, behaviour cloning with mean squared error replaces the SAC loss with probability p. The probability falls linearly from 1 to 0 over 8,000 epochs 2605.08713 eq. 4 2605.08713 eq. 6 2605.08713 § 5.3. Experience is kept in two replay buffers, one for all transitions and one for collision-free successful episodes. The sampling ratio between them is not stated 2605.08713 § 4.2. The reward combines sparse episode terms with dense collision terms 2605.08713 eq. 7 2605.08713 eq. 8. The paper reports 83.5% parking success in simulation on standard slots, with author-reported ablations. Removing the critic's auxiliary tasks drops success to 40.5%, and removing the second training stage drops it to 0.0% 2605.08713 tab. 2. Card: REAP: Reinforcement-Learning End-to-End Autonomous Parking with Gaussian Splatting Simulator for Real2Sim2Real Transfer.

Per-agent independent PPO critic (IPPO)

Each agent runs PPO on its own local observation history, with a local critic V(z^a) 2011.09533 § 4. The agents share one actor and one critic weight set, and each agent's advantage is computed from the team reward rather than from a per-agent reward 2011.09533 § 4. The advantage is a truncated generalised-advantage sum over h steps, with γ = 0.99 and λ = 0.95 2011.09533 eq. 4 2011.09533 § 4. The actor's loss is the clipped PPO surrogate, applied per agent, and the critic's loss is clipped in the same way 2011.09533 eq. 5 2011.09533 eq. 6. The clip width ε = 0.2 is listed in the table caption 2011.09533 tab. 1. The paper gives no algorithm box; its update is written in equations and prose 2011.09533 § 4. The paper's main claim is that this independent setup matches or beats joint-critic methods on several SMAC maps, with mild per-map tuning 2011.09533 § 5.1. On three maps the author-reported median win rates are 100% on 3m and 2m vs 1z, and 90% on 3s5z vs 3s6z 2011.09533 fig. 2. The paper reports that IPPO exceeds MAPPO and QMIX on 3s5z vs 3s6z, corridor and 6h vs 8z 2011.09533 § 5.1. The MAPPO values in its figure come from a benchmarking study that the paper cites, not from runs in the paper 2011.09533 fig. 2. The ablations remove the clipping terms, and the authors report that policy clipping is essential. A lower learning rate for the unclipped variant does not reproduce the gain 2011.09533 § 5.3. The paper's own hypothesis is that clipping may limit the effect of other agents' changing behaviour on each learner's update 2011.09533 § 1. The paper also reports that replacing the local critics with a centralised critic on full state performs substantially worse on four maps 2011.09533 § 5.2. Its theory section notes that independent learners lose the usual convergence guarantees under non-stationarity, and gives a relative-overgeneralisation example in which IPPO is prone to failure 2011.09533 § 6 2011.09533 fig. 6. All results are author-reported. Card: Independent PPO for SMAC.

Bayesian GP critic (BAC)

The critic is GPTD, a Gaussian-process model of Q. Its prior is a sum of a state kernel and a Fisher kernel u(z)^T G^{-1} u(z') on state-action pairs 2604.27563 § 7.2. Each observed reward is modelled as Q(z_i) minus gamma times Q(z_{i+1}), plus noise, so the critic is fitted from transitions rather than from a separate target network 2604.27563 § 7.1. The posterior gives both a Q estimate and its covariance 2604.27563 § 7.1. The actor's gradient is the posterior mean U_t alpha_t, where U_t stacks the score vectors of the visited pairs. The covariance is G − U_t C_t U_t^T 2604.27563 prop. 6. The update can be preconditioned by the inverse Fisher matrix for a natural-gradient step 2604.27563 alg. 4. The Fisher matrix is estimated from data. For the discounted state distribution, which the paper uses, its G-EST estimator applies 2604.27563 § 7.2 2604.27563 alg. 3. The observation unit is a single transition, so the Markov property is used, unlike whole-trajectory policy gradients 2604.27563 § 7.2. The paper gives a sparse form with a dictionary of state-action pairs 2604.27563 § 7.3 2604.27563 prop. 7. The authors report that BAC reaches better policies than Monte-Carlo policy gradient on a random walk, mountain car and ship steering 2604.27563 § 8.1 2604.27563 § 8.2 2604.27563 § 8.3. These results appear in figures only, and they are author-reported. Card: Bayesian policy gradient and actor-critic algorithms.

Off-policy twin critic at massive parallel scale (FastSAC)

The learner is FastSAC, an off-policy method with a SAC-style actor. The recipe runs it on thousands of parallel simulated humanoids and keeps old transitions in a replay buffer 2512.01996 § 2.1. The critic is distributional (C51), because the authors found quantile regression too costly at large batch sizes 2512.01996 § 2.1. The target averages the two critics instead of taking their minimum, which the authors report as harmful with layer normalisation 2512.01996 § 2.1 2512.01996 alg. 1. The initial entropy temperature is 0.001 and it is tuned automatically, with a target entropy of 0.0 for locomotion 2512.01996 § 2.1. The maximum standard deviation of the pre-tanh action is 1.0 2512.01996 § 2.1. The paper tests velocity tracking on Unitree G1 and Booster T1 humanoids under strong domain randomisation, and whole-body motion tracking on G1 2512.01996 § 3.1 2512.01996 § 3.2. It reports locomotion policies trained in 15 minutes on one RTX 4090 2512.01996 § 3.1. These numbers are author-reported, and the learning curves appear only in figures 2512.01996 § 3.1. Card: Learning Sim-to-Real Humanoid Locomotion in 15 Minutes.

Vehicle-dynamics world-model critic (VDD)

A Dreamer-style world model is split into two parts. One is an ego-vehicle dynamics model with physical parameters. The other is an environment model that sees only the ego state 2512.02417 § 4.1. The actor outputs a goal, and a PID controller turns each goal into an action, so the vehicle's response is handled by feedback 2512.02417 § 4.2.1. The critic uses symlog-transformed returns, two-hot targets and a bin-weighted readout 2512.02417 § 4.2.2. The actor loss has three parts: a return term divided by max(1, S), an entropy term and a reachability penalty 2512.02417 eq. 19. Two mechanisms handle a changed car. PAD adjusts the trained policy's actions at deployment, and PAT trains the policy on imagined trajectories under shifted vehicle parameters 2512.02417 § 4.3 2512.02417 § 4.4. PAD cannot help when the changed car cannot reach the needed states, for example with a smaller steering limit 2512.02417 § 4.3. The paper reports roundabout success of 0.70 against 0.53 for DreamerV3, and slightly lower success on intersections, 0.88 against 0.91 2512.02417 tab. 1. The mass and steering robustness results are figure-only 2512.02417 § 5.2.4. The experiments use the MetaDrive and CARLA simulators, not real vehicles 2512.02417 § 5.1.1 2512.02417 § 6. Card: Vehicle Dynamics Embedded World Models for Autonomous Driving.

Demonstration-based optimal-transport reward with a TD learner (TemporalOT)

The paper changes the reward the learner sees, and leaves the learner's structure as it is. After each rollout, the trajectory is scored against one or two expert video demonstrations with an optimal-transport plan, and each step receives the negative cost it carries under that plan 2410.21795 eq. 5. The frame cost is a cosine cost between fixed ResNet50 features. Two changes make it order-aware: the cost averages over a short window of consecutive frames 2410.21795 eq. 6, and the plan is restricted to a band around the diagonal 2410.21795 eq. 9. The agent is then updated with a TD loss on the relabelled transitions 2410.21795 eq. 1 2410.21795 alg. 1. The pixel agent follows the DrQ-v2 model of ADS 2410.21795 § 4.6. The paper says the base OT reward depends on the whole trajectory, which makes it non-stationary during training 2410.21795 § 3.1.2. Author-reported: average success 61.1 over nine Meta-World tasks with two demonstrations, against 35.9 for ADS and 20.8 for the sparse task reward 2410.21795 tab. 1. The method assumes the demonstrator moves at about the agent's speed; success drops at 3x and 4x demonstrator speed on several tasks 2410.21795 tab. 3. Card: Robot Policy Learning with Temporal Optimal Transport Reward.

Advantage critic for PPO on a simulated mobile robot (crop-row navigation)

The learner is PPO from Stable Baselines3, with settings stated in full: 2,048 rollout steps, batch size 64, 10 epochs, γ = 0.99, GAE λ = 0.95 and clip range 0.2 2412.18865 tab. III. The paper does not state the critic's architecture 2412.18865 tab. III. The actor outputs one continuous value per step, which sets the turn rate of a four-wheel-steering robot 2412.18865 § III-C 2412.18865 tab. I. The reward is a sum of six terms: distance to the nearest waypoint, camera alignment, a waypoint bonus, a goal term, control smoothness and a time penalty 2412.18865 § III-C. Four learners were compared, SAC, TD3, A2C and PPO. The paper reports PPO as best, but only from reward and episode-length plots 2412.18865 § IV-B 2412.18865 fig. 9 2412.18865 fig. 10. The success rate of 351 of 420 randomised trials is author-reported, in simulation only 2412.18865 tab. IV 2412.18865 § V. Card: Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning.

Online actor-critic for distributed MPC (explicit policies)

This actor-critic is not trained on rollouts. In each prediction interval, each robot's critic is fitted to a desired costate from the value update 2412.19669 eq. 9a 2412.19669 eq. 12. The critic approximates the costate as a linear combination of basis functions of the neighbourhood error 2412.19669 eq. 11. The actor is a linear combination of its own basis, pulled toward a target built from the neighbours' costates 2412.19669 eq. 14 2412.19669 eq. 15. Both weight sets take gradient steps, with rates 0.4 for the critic and 0.2 for the actor 2412.19669 eq. 13 2412.19669 eq. 16 2412.19669 tab. I. The sweep repeats until the weight change is small, with a cap of 30 sweeps 2412.19669 alg. 1 2412.19669 tab. I, and the robot applies the actor's output for one step 2412.19669 alg. 1. The learned policy need not lower the cost at each step. The paper therefore compares it with a baseline whose cost decreases, and re-initialises when the check fails 2412.19669 eq. 17 2412.19669 eq. 18. Under two conditions on the learning rates and basis norms, the weight errors are uniformly ultimately bounded 2412.19669 app. A. The safe variant for state and control limits adds log-barrier terms and a force-field policy 2412.19669 eq. 21. Its algorithm and stability proof are in an auxiliary file that is not in the source package 2412.19669 § V-B 2412.19669 § VI. Reported costs are per step for all robots on one machine. Deployment takes 9.6e-4 s per step at six robots, against 40.2 s for IPOPT at 30 iterations. Learning takes 0.084 s per step at six robots and 14.57 s at 10,000 robots 2412.19669 tab. III. Collision-avoidance success is 100% at 1 and 1.5 m/s and falls at higher speeds, read from a plot 2412.19669 fig. 7. Card: Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC.

Multi-agent DDPG, one actor per base station (edge association)

Each base station is an agent that runs DDPG, and all agents share one reward. The paper casts the choice of twin association, per-twin batch sizes and sub-channel bandwidth as a Markov decision process, with one agent per base station 2011.09902 § 4 2011.09902 § 4.1. Each agent's action is its twin count, its batch sizes and its bandwidth shares 2011.09902 § 4.1. The reward is the negative time cost, and every agent receives the same value, because the total time is set by the slowest base station 2011.09902 § 4.1 2011.09902 eq. 19.

The actor is deterministic, with Ornstein-Uhlenbeck noise added to its output for exploration 2011.09902 § 4.2 2011.09902 eq. 21. The actor gradient is taken through a critic that reads the actions of all agents 2011.09902 eq. 22. The critic's TD error is written for one agent's action, while its gradient is written over all actions 2011.09902 eq. 23. The paper does not say whether each agent has its own critic or they share one. The twin counts must be integers 2011.09902 eq. 18b, but the paper does not say how a continuous actor output becomes an integer count. Learning rates, the target update rate, the replay size and the mini-batch size are not stated 2011.09902 § 4.2. Card: Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks.

Model-based value expansion critic with TD-k (MVE)

The critic is trained as in DDPG, but its target is not a one-step Bellman backup. The target sums H imagined steps under the policy, using a learned one-step model and the true reward, and then adds the critic's value at the imagined state after H steps 1803.00101 § 3 1803.00101 eq. 2. Setting H = 0 gives the plain actor-critic method 1803.00101 § 3.2. The TD-k trick trains the critic on imagined states too. Each imagined step at offset t gets a target with horizon H − t, and the actor is updated on the real transition 1803.00101 § 3.2 1803.00101 alg. 1. Imagined transitions are not stored, because the actor changes during training 1803.00101 § 3.2. The dynamics model is a fully connected network with 8 layers of 128 units, trained to predict the change in state at a learning rate of 10⁻³ 1803.00101 § 3.2. The paper's assumptions are a deterministic environment, a known reward and a model accurate for H steps 1803.00101 § 3. Its results are learning curves on three continuous-control tasks. The author-reported gain over DDPG and an imagination-buffer baseline is smaller on walker 1803.00101 fig. 3. The paper reports no final-score table. Card: Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning.

When to use which

Notes that come from the cited results:

• The papers' own numbers do not show that one variant is better across tasks. Each paper compares against baselines on its own benchmarks 2301.04104 § Results 2607.07508 § 4.2 2206.07568 § 5.1 1904.12304 § 4.1 1606.04671 § 5.1.
• The per-agent PPO variant's lower-learning-rate control applies to IAC only. The authors say other baselines were not tested with the same control, so the learning-rate ablation does not show that other methods are untuned 2011.09533 § 5.3.
• The imagined-rollout and token-level variants both report that the critic needs careful handling. DreamerV3 uses a twohot readout and a return normalisation 2301.04104 § Robust predictions. SAO freezes the critic's attention layers for stability 2607.07508 § 3.2.
• The contrastive variant depends on batch size. The other batch rows act as negatives, so the batch is part of the method 2206.07568 tab. 2.
• The DDPG variant's reward weights are tied to the paper's own loss definitions. The paper warns that they cannot be copied onto a loss with a different scale without a check 1904.12304 supp. RL Agent Details.
• The TD3+BC variant is weakest on random data. It scores 10.2 on HalfCheetah-random, below CQL (21.7) and Fisher-BRC (32.2) 2106.06860 tab. 2, and the paper calls this a concern 2106.06860 app. C.1. Its α was tuned on Hopper tasks that are also evaluation tasks 2106.06860 app. B.
• The progressive-network mean is sensitive to outliers. On Labyrinth the 2-column mean is 491% against a median of 115% 1606.04671 tab. 1 1606.04671 § 5.2. Compare medians when the gap is large.
• The asymmetric variant's real-car results come from one vehicle and one seed, with no confidence intervals. Each trial moves a scenario's success rate by 5 points 2605.08713 tab. 4.
• The Bayesian variant's Fisher matrix is estimated from data in its experiments. The paper's closed forms are exact only for the true Fisher matrix, so the posterior moments are approximate in practice 2604.27563 app. A 2604.27563 § 4.3.
• The FastSAC variant's results are figure-only, apart from the 15-minute training time. The paper gives no numbers for its own ablations in the text 2512.01996 § 3.1 2512.01996 § 2.1.
• The VDD variant has no official code. Its controller gains, reachability horizon, entropy scale and critic bin count are not stated, so an implementer must set them and tune them 2512.02417 § 4.2 2512.02417 eq. 19 2512.02417 § 5.2.3.
• The TemporalOT variant reports five seeds, and some standard deviations are large, for example 78.4 (12.4) on Door-open and 42.2 (44.5) for ADS on Basketball 2410.21795 tab. 1. Compare the spread, not just the mean.
• The PPO variant's only comparison with a classical controller is a PD controller on a skid-steer robot, a different platform. Its distance and time are 5.97 m and 49.4 s against 5.29 m and 18.0 s, and the PD gains and the number of runs are not stated 2412.18865 tab. V. It does not show that the learner beats tuned classical control.
• The distributed MPC variant compares a horizon of 20 steps with baselines that use 10 2412.19669 § VI. The solver comparison in its table is therefore not at equal horizons. At two robots with obstacles, its cost is 13.5 against 5.0 for IPOPT, so the cost favours IPOPT there 2412.19669 tab. III.
• The multi-agent DDPG variant compares against two simple association rules, both set up by the authors, and reports no repeated runs or error bars 2011.09902 § 5. Its gains are read from figures, and the text gives no numbers 2011.09902 fig. 5 2011.09902 fig. 7.
• The MVE variant's analysis assumes a deterministic environment and a known reward 1803.00101 § 3. Its experiments use 4 seeds with a learned model and 6 seeds with an oracle model 1803.00101 app. A, so the learned-model curves rest on few runs.
• The offline survey's heat map collects scores the authors found in the literature. It orients the reader and does not rank methods on one task 2203.01387 fig. 5.
• The FPGA survey's A3C advantage is a one-step TD error, while its PPO section names GAE without defining it 2610.28699 eq. 7 2610.28699 § 3.8. Do not treat the two as the same estimator.

Implementation sketch

The shared core: fit the critic on collected data, then update the actor from the critic. The variants differ in how the data is gathered, how the critic target is built, and what the actor's objective is. Not every variant uses an advantage.

# Shared actor-critic update; each variant fills in the marked steps
# [arxiv:2301.04104 § Critic learning] [arxiv:2607.07508 § 3.2]
# [arxiv:2011.09533 § 4] [arxiv:2011.09533 eq. 5] [arxiv:2011.09533 eq. 6]
# [arxiv:2206.07568 § 4.4] [arxiv:1904.12304 § 3.3]
# [arxiv:2106.06860 § 5] [arxiv:1704.02532 § 3.2] [arxiv:1606.04671 § 2]
# [arxiv:2604.27563 § 7.1] [arxiv:2604.27563 prop. 6]
# [arxiv:2512.01996 alg. 1] [arxiv:2512.02417 § 4.2.2] [arxiv:2512.02417 eq. 19]

def actor_critic_step(batch, actor, critic, lr_critic, lr_actor):
    # Step 1: source of the batch differs by variant
    #   DreamerV3: imagined rollouts from a learned world model
    #   SAO: one rollout per prompt, with rollout log-probs logged
    #   contrastive RL: replayed (state, action, future state) triples
    #   DDPG: one-step transitions, with a reward and no future state
    #   TD3+BC: logged offline (state, action, reward, next state) tuples
    #   DDAC driving: simulator transitions with continuous actions
    #   REAP: sim rollouts; the critic also sees privileged BEV maps
    #   progressive nets: A3C rollouts on the new task, earlier columns frozen
    #   IPPO: each agent's own observation history, with the team reward
    #   BAC: single transitions, with the score vector of each state-action pair
    #   FastSAC: replay of parallel simulated transitions  [alg. 1]
    #   TemporalOT: replayed transitions relabelled with the OT reward  [eq. 5]
    #   PPO crop navigation: on-policy rollouts of 2,048 steps  [tab. III]
    #   distributed MPC: no rollouts; a model propagates neighbourhood errors
    #                    [eq. 3]
    #   VDD: imagined rollouts from replayed latent starts  [eq. 14] [eq. 15]
    #   edge association: one agent per base station, shared reward  [§4.1] [eq. 19]
    #   MVE: real start states, then H imagined steps from a learned model  [alg. 1]
    # Step 2: critic target and critic update
    #   DreamerV3: lambda-returns, cross-entropy on twohot targets
    #   SAO: skip-observation GAE; two critic updates, attention frozen
    #   contrastive RL: binary cross-entropy on the batch score matrix
    #   DDPG: Bellman target for Q(s, a)
    #   TD3+BC: clipped double-Q target from two target critics
    #   DDAC driving: one-step target, as in DQN
    #   REAP: autoencoder loss plus value loss on the critic's inputs
    #   progressive nets: A3C value loss, n-step length not stated
    #   IPPO: per-agent clipped value loss on a GAE target  [eq. 6]
    #   BAC: GPTD posterior over Q; no Bellman regression target  [§7.1]
    #   FastSAC: twin critics regress to the averaged target  [alg. 1]
    #   VDD: two-hot cross-entropy on symlog lambda-returns  [§4.2.2]
    #   TemporalOT: TD target from the relabelled transitions  [eq. 1]
    #   PPO crop navigation: GAE advantage targets, lambda 0.95  [tab. III]
    #   distributed MPC: least-squares fit to the desired costate  [eq. 12]
    #   edge association: DDPG TD target, y_t not written out  [§4.2] [eq. 23]
    #   MVE: TD-k loss, averaged over imagined steps, with horizon H - t  [§3.2]
    critic_loss = critic_target_loss(batch, critic)
    update(critic, critic_loss, lr_critic)
    # Step 3: actor update from the critic
    #   DreamerV3, SAO: policy gradient on an advantage [§ Actor learning]
    #   contrastive RL: maximise the critic score at the commanded goal
    #   DDPG: deterministic policy gradient through Q(s, a)  [eq. 6]
    #   TD3+BC: lambda * Q(s, pi(s)) minus squared error to the logged action  [eq. 3]
    #   DDAC driving: critic gradient w.r.t. the action through the actor  [§ 3.2]
    #   REAP: SAC loss, or MSE to the planner action with probability p  [eq. 4]
    #   progressive nets: A3C policy gradient with an entropy term  [app. 3]
    #   IPPO: per-agent clipped PPO surrogate, summed over agents  [eq. 5] [eq. 7]
    #   BAC: posterior mean U_t alpha_t, optionally with inverse Fisher  [prop. 6]
    #   FastSAC: entropy-weighted Q on a reparameterised sample  [alg. 1]
    #   VDD: return term, entropy and reachability penalty on goals  [eq. 19]
    #   TemporalOT: TD-trained actor, DrQ-v2 model in the pixel setup  [§ 4.6]
    #   PPO crop navigation: clipped objective on the advantage  [tab. III]
    #   distributed MPC: actor fitted to neighbours' costate  [eq. 15]
    #   edge association: deterministic policy gradient through the critic  [eq. 22]
    #   MVE: usual actor loss on the real transition  [alg. 1 line 9]
    actor_loss = actor_objective(batch, actor, critic)
    update(actor, actor_loss, lr_actor)

Contradictions

None known between the twenty papers' reported results on a common benchmark. Their benchmarks and metrics differ, so their numbers are not directly comparable. Within single papers, the following disagree:

• Bayesian GP critic: the LQR rewards are costs and the LQR is minimised, but the update rules are written as ascent, theta + beta * delta 2604.27563 alg. 2 2604.27563 alg. 4. The authors explain the cost convention in a footnote 2604.27563 § 6.2 but do not show the sign change in the update.
• Bayesian GP critic: the text says BPG is similar to BAC at M = 100 2604.27563 § 8.1, while its random-walk policy runs use M = 1, 25, 50 and 75 2604.27563 § 8.1 2604.27563 tab. 4. The card treats the M = 100 claim as unverified.
• TD3+BC: the paper's λ averages |Q| over the logged actions 2106.06860 eq. 5, while the released code averages |Q| over the policy's own actions TD3_BC.py. An implementation that follows only the equation computes a different λ.
• Per-agent PPO: the text gives the convolution strides as (2, 1, 1), while the table caption says all convolutions use stride 3 2011.09533 § 4 2011.09533 tab. 1. The paper does not say which is used.
• Per-agent PPO: the text says 16 SMAC maps, while the table lists 17 2011.09533 § 5 2011.09533 tab. 1. The count is not reconciled.
• TD3+BC: the printed Fisher-BRC total is 974.6, but summing its 15 per-row scores gives 975.2 2106.06860 tab. 2.
• DreamerV3: the text gives the critic's prediction horizon as 16 steps 2301.04104 § Critic learning, while its hyperparameter table gives an imagination horizon of 15 2301.04104 tab. 4.
• SAO: the introduction claims stable training for "around one thousand" steps 2607.07508 § 1. The explicit curves and text reach about 400 steps 2607.07508 § 4.2.
• SAO: the ablation's running-mean baseline uses a window of 8 recent rewards per prompt 2607.07508 § 4.3, while the online writing simulation uses 128 2607.07508 § 4.5. These are two different baselines, so the window is not one setting.
• REAP: the simulated success rate for its own 3DGS simulator is 82.5% in 2605.08713 tab. 3 and 83.5% for the same method on standard slots in 2605.08713 tab. 1. The paper does not explain the gap.
• FastSAC: Algorithm 1 and §2.1 give a target that averages the two target Q-values 2512.01996 alg. 1 2512.01996 § 2.1. The released code builds one projected target per critic and does not average across them fast_sac_agent.py. The paper does not say which one produced its figures 2512.01996 fig. 3.
• Distributed MPC: Table III gives a per-step cost V that is lower for the proposed method without obstacles, 0.73 against 1.46 at two robots 2412.19669 tab. III. The text gives a cumulative cost over 180 steps at two robots of 8.9 for DMPC and 16.9 for the proposed method 2412.19669 § VI. The paper argues that closed-loop performance matters more, and does not test a third metric.
• TemporalOT: Table 3 gives Button-press at 1x speed as 89.0 (3.8), which is the ADS entry in Table 1. Table 1 gives TemporalOT 92.4 (3.6) for the same task 2410.21795 tab. 3 2410.21795 tab. 1. The paper does not explain the mismatch.
• Multi-agent DDPG (edge association): the text says each agent's reward uses T_i 2011.09902 § 4.1, while eq. 17 defines one global T as maxima over base stations 2011.09902 eq. 17. The paper gives every agent the same −T 2011.09902 eq. 19. Which time is meant per agent is not stated.
• Multi-agent DDPG (edge association): the text says agents train in parallel 2011.09902 § 4.2, while Algorithm 1 nests the episode loop inside a loop over base stations 2011.09902 alg. 1. The paper does not explain the order.
• Multi-agent DDPG (edge association): as printed, eqs. 24 and 25 have θ on both sides of the target update, so the target equals the primary network 2011.09902 eq. 24 2011.09902 eq. 25. The text does not say which soft update is intended.
• CAPO: the paper sets the clip width to ε = 0.001 2604.18401 § 4.1, while the step-level scripts in the released repository set 0.0003 run_steppo.sh. The same scripts use the GSPO ratio, which takes the mean of token log-ratios, and the action-aware ratio of 2604.18401 eq. 4 is not in the repository core_algos.py. The paper does not say which code produced its tables 2604.18401 § 4.1.

Papers

• Independent PPO for SMAC: independent PPO, with a per-agent local critic, a shared team reward and shared weights, compared with joint-critic methods on SMAC.
• Mastering Diverse Domains through World Models: an imagined-rollout actor and critic that use one fixed configuration across many domains.
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning: a token-level critic for LLM RL with one rollout per prompt, and skip-observation advantages.
• Contrastive Learning as Goal-Conditioned Reinforcement Learning: a contrastive critic whose score is a goal-conditioned Q-function up to a constant, with an actor that maximises it.
• Deep Reinforcement Learning framework for Autonomous Driving: a driving pipeline whose planner is a DDAC actor-critic head for continuous actions, shown on lane keeping without metrics.
• RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion: a DDPG agent that picks a latent GAN seed in one step per shape.
• A Minimalist Approach to Offline Reinforcement Learning: TD3 plus a behaviour-cloning term in the actor loss, with state normalisation, for offline D4RL data.
• Progressive Neural Networks: one column per task with frozen earlier columns and lateral connections, trained with A3C. The actor-critic is the learner here, not the contribution.
• REAP: Reinforcement-Learning End-to-End Autonomous Parking with Gaussian Splatting Simulator for Real2Sim2Real Transfer: a camera-only SAC parking actor with a privileged BEV critic, and a planner behaviour-cloning term annealed to zero.
• Robot Policy Learning with Temporal Optimal Transport Reward: an optimal-transport reward with a context window and a temporal band, used to relabel trajectories for a TD learner.
• Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning: PPO from Stable Baselines3 for crop-row and waypoint navigation in simulation, compared with SAC, TD3 and A2C.
• Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC: an online actor-critic that learns explicit distributed MPC policies, with a barrier variant for state and control limits.
• Bayesian policy gradient and actor-critic algorithms: a GPTD critic with a Fisher kernel, whose gradient posterior is closed-form, compared with Monte-Carlo policy gradient.
• Learning Sim-to-Real Humanoid Locomotion in 15 Minutes: off-policy FastSAC on massively parallel simulation, with a recipe of action bounds, layer norm, averaged critics and a minimal reward.
• Vehicle Dynamics Embedded World Models for Autonomous Driving: a Dreamer-style world model that separates ego dynamics from the scene, with a goal-output actor and PAD and PAT for changed vehicles.
• Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks: one DDPG agent per base station, choosing twin association, batch sizes and bandwidth to cut training time, with a shared reward.
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning: model-based value expansion with a TD-k critic loss on imagined states, in a DDPG-style actor-critic. Results are learning curves only.
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems: a survey that treats offline actor-critic as a base learner plus modification classes. It proposes no method.
• Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey: a survey of RL on FPGAs and integrated circuits, with actor-critic and A3C hardware pipelines. It runs no experiments.
• CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities: a PPO-style method for multi-turn LLM agents with one critic value and one advantage per action, and an action-aware clipped ratio.