--- kind: concept slug: actor-critic title: Actor-critic summary: >- Two learned parts: a critic that scores states or actions, and an actor trained on those scores. Twenty papers show the design space, from imagined rollouts and token-level critics to agent-step critics, model-based value expansion, offline TD3+BC, per-agent PPO, edge-network DDPG, online MPC policies and Bayesian GP critics. Pick by data, compute and action structure. tags: [reinforcement-learning, policy-optimization, value-estimation] papers: ["1606.04671", "1704.02532", "1803.00101", "1904.12304", "2011.09533", "2011.09902", "2106.06860", "2206.07568", "2203.01387", "2301.04104", "2410.21795", "2412.18865", "2412.19669", "2512.01996", "2512.02417", "2604.27563", "2605.08713", "2607.07508", "2610.28699", "2604.18401"] related: [credit-assignment, llm-agent, policy-gradient, contrastive-learning, goal-conditioned-reinforcement-learning, model-based-reinforcement-learning, offline-reinforcement-learning, deep-q-learning, rlhf, imitation-learning, continual-learning, sim-to-real-transfer, gaussian-process, multi-agent-systems, world-model, domain-randomization, reward-design, optimal-transport, learning-from-demonstration, proximal-policy-optimization, model-predictive-control, multi-agent-reinforcement-learning, centralized-training-decentralized-execution, digital-twin] status: draft updated_at: 2026-11-01 --- ## Overview An actor-critic method trains two learned parts together. The actor is the policy that picks actions. The critic estimates how good a state, or a state-action pair, is under the current behaviour. The actor is then updated from the critic's scores. Some variants use the critic as a baseline and follow an advantage [arxiv:2301.04104 § Actor learning] [arxiv:2607.07508 § 3.2]. Others follow the gradient of the critic's score with respect to the action [arxiv:1904.12304 eq. 6] [arxiv:2206.07568 § 4.3] [arxiv:2106.06860 eq. 3]. The twenty papers below share this two-part structure but differ in what the critic is, where its training data comes from, and how the actor uses it. Two of them are surveys, not new methods. The offline RL survey treats an actor-critic learner plus modifications as the general form of an offline method [arxiv:2203.01387 § III], and the FPGA survey groups actor-critic hardware designs by RL family [arxiv:2610.28699 § 3.6]: - The critic is a value function over imagined rollouts [arxiv:2301.04104 § Critic learning]. - The critic is a twin off-policy Q-function fed from a replay buffer of thousands of parallel simulated environments. Its target averages the two critics, and the actor maximises entropy-regularised Q [arxiv:2512.01996 § 2.1] [arxiv:2512.01996 alg. 1]. - The critic estimates the value of goal-driven behaviour inside a world model. A fixed controller turns each goal into an action, and the actor is trained by backpropagating value gradients through imagined sequences [arxiv:2512.02417 § 4.2.1] [arxiv:2512.02417 § 4.2.3]. - The critic is a Gaussian-process posterior over Q with a Fisher kernel on state-action pairs, fitted from observed transitions with a Bellman-type observation model [arxiv:2604.27563 § 7.1] [arxiv:2604.27563 § 7.2]. - The critic is a value model over the tokens of an LLM answer [arxiv:2607.07508 § 3.2]. - The critic is read once per complete agent action, at the last token of the state, and the whole action gets one advantage [arxiv:2604.18401 § 3.3]. - The critic is a contrastive score that stands in for a goal-conditioned Q-function [arxiv:2206.07568 § 4.2]. - The critic is a state-action Q-function trained with a Bellman target, for an actor that emits one scalar per input [arxiv:1904.12304 § 3.3]. - The critic is a twin Q-function over logged offline actions, and the actor trades Q against imitation of those actions [arxiv:2106.06860 eq. 3]. - The critic is a Q-function over continuous driving actions, used as the planner head of an end-to-end driving pipeline [arxiv:1704.02532 § 3.2] [arxiv:1704.02532 § 4.3]. - The critic is a local value function over each agent's own observation history, trained on a shared team reward, with actor and critic weights shared across agents [arxiv:2011.09533 § 4]. - The actor-critic is only the learner in each column of a network that grows with the task count. The paper's own contribution is the column architecture [arxiv:1606.04671 § 2] [arxiv:1606.04671 § 5.1]. - The critic is a Q-function that also reads privileged simulator maps, which the camera-only actor never sees. The actor's and critic's inputs differ on purpose [arxiv:2605.08713 § 4.2]. - The learner is trained by TD on relabelled transitions, with each step's reward taken from an optimal-transport score against one or two demonstrations [arxiv:2410.21795 eq. 1] [arxiv:2410.21795 eq. 5]. - The critic is the value estimate behind PPO's advantage. The advantage uses generalised advantage estimation with λ = 0.95, which needs a learned value function [arxiv:2412.18865 tab. III]. - The critic approximates the costate of a distributed MPC problem over each robot's neighbourhood error, and the actor is linear in its own basis. Both are fitted online, inside each prediction interval [arxiv:2412.19669 eq. 11] [arxiv:2412.19669 eq. 14]. - The critic is a Q-function over the actions of all base stations, and each base station's actor is trained through it [arxiv:2011.09902 § 4.2] [arxiv:2011.09902 eq. 22]. - The critic's bootstrap target is a model-based value expansion. A learned one-step model is rolled forward H steps under the policy, the true reward is summed along the way, and the critic is evaluated at the end [arxiv:1803.00101 § 3] [arxiv:1803.00101 eq. 2]. ## Variants ### Imagined-rollout critic (DreamerV3) A world model is learned from replayed experience. The actor and critic are then trained only on trajectories the model imagines, 15 steps long from each replayed start state [arxiv:2301.04104 § Critic learning] [arxiv:2301.04104 tab. 4]. The critic outputs a categorical distribution over bins and reads its value as the expectation under that distribution [arxiv:2301.04104 § Robust predictions]. The paper does not state the number of bins. Returns are λ-weighted with γ ≈ 0.997 (discount horizon 333) and λ = 0.95 [arxiv:2301.04104 tab. 4]. The critic also regresses on replayed trajectories at a lower loss scale of 0.3 [arxiv:2301.04104 tab. 4]. The actor is trained by REINFORCE on the return minus the critic's value, scaled by a running return range, with an entropy bonus of weight 3e-4 [arxiv:2301.04104 § Actor learning] [arxiv:2301.04104 tab. 4]. Card: [Mastering Diverse Domains through World Models](/papers/2301.04104). ### Single-rollout token-level critic (SAO) For LLM reinforcement learning with one rollout per prompt, the paper drops GRPO's group baseline and learns a value model [arxiv:2607.07508 § 3.2]. The policy update uses rollout log-probabilities as the behaviour proxy, so no separate old-policy model is kept, and it masks tokens whose importance ratio leaves a fixed interval [arxiv:2607.07508 § 3.1] [arxiv:2607.07508 eq. 3]. The value model gets two updates per policy update, and its attention parameters are frozen during RL [arxiv:2607.07508 § 3.2] [arxiv:2607.07508 § 4.1]. Environment-feedback tokens are left out of the advantage recursion [arxiv:2607.07508 § 3.2]. The discount factor, the optimiser and the size of the value-pretraining corpus are not stated [arxiv:2607.07508 § 3.2] [arxiv:2607.07508 § 4.1]. Card: [Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning](/papers/2607.07508). ### Action-level critic with an action-aware ratio (CAPO) For multi-turn LLM agents, the environment changes only after a whole action, which is a reasoning note plus a tool call. CAPO therefore uses a critic at each action boundary. The critic's value is read at the last token of the state, which is conditioned on the state prefix but not on the action, so no extra encoder is needed [arxiv:2604.18401 § 3.3]. Each action gets one GAE advantage from action-level TD residuals, and that advantage is shared by all tokens of the action [arxiv:2604.18401 § 3.3] [arxiv:2604.18401 eq. 2]. The token probability ratios inside an action are combined into one ratio. The log-ratios are centred on a token-weighted batch mean μ̂, summed, and divided by the square root of the action length L. The result is clipped as one unit with interval [1 − ε, 1 + ε] [arxiv:2604.18401 § 3.4] [arxiv:2604.18401 eq. 4] [arxiv:2604.18401 eq. 5]. The paper treats the result as a length-calibrated ratio, not an exact likelihood ratio, and says a TRPO-style guarantee does not apply [arxiv:2604.18401 app. A]. When an action has one token, the ratio reduces to the ordinary token ratio [arxiv:2604.18401 § 3.5]. The paper's experiments use ε = 0.001 [arxiv:2604.18401 § 4.1]. How μ̂ is computed, over a mini-batch or the rollout batch, is not stated [arxiv:2604.18401 app. A]. Per-action loss averaging is also not stated. Its ablation on WebShop with Qwen3-4B gives a score of 78.12 for the full ratio, against 73.80 for token-wise ratios and 76.18 for plain square-root aggregation without centring [arxiv:2604.18401 tab. 2]. Action-level GAE with γ = 0.99 and λ = 1.00 scores 78.12, against 71.86 for token-level GAE at the same γ [arxiv:2604.18401 tab. 3]. The paper reports 58.20 WebShop success for CAPO against 46.00 for PPO, with 3 seeds, and all results are author-reported [arxiv:2604.18401 tab. 1]. The critic is still trained, so each iteration costs about the same as PPO's, with a single measurement of 189.54 s against 195.37 s [arxiv:2604.18401 tab. 6] [arxiv:2604.18401 app. F]. Card: [CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities](/papers/2604.18401). ### Goal-conditioned contrastive critic (contrastive RL) The critic is a binary contrastive classifier over (state, action, future state) triples, with its score as an inner product of two learned encoders [arxiv:2206.07568 § 4.2] [arxiv:2206.07568 § 4.4]. At the optimum, the exponential of the score equals the goal-conditioned Q-function divided by the goal's marginal density. For a fixed goal, the goal-dependent factor is the same for every action, so the actor can rank actions by the score alone [arxiv:2206.07568 § 4.2]. The actor is trained to maximise the critic's score at a commanded goal, with no target networks and no auxiliary losses [arxiv:2206.07568 § 4.3] [arxiv:2206.07568 § 4.5]. Card: [Contrastive Learning as Goal-Conditioned Reinforcement Learning](/papers/2206.07568). ### Deterministic critic with a one-step GAN-seed actor (DDPG) A DDPG agent [arxiv:1904.12304 § 3.3] chooses one scalar, the seed of a latent GAN, from a 128-value encoding of a partial point cloud. The critic Q(s, a) is trained with a Bellman target. The actor follows the deterministic policy gradient, which passes the critic's gradient with respect to the action back into the actor [arxiv:1904.12304 eq. 6]. Each episode is a single action, and the replay batch is 100 [arxiv:1904.12304 supp. RL Agent Details]. The reward is a weighted sum of negated losses, with weights 100, 10 and 0.01 [arxiv:1904.12304 eq. 5] [arxiv:1904.12304 supp. RL Agent Details]. Card: [RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion](/papers/1904.12304). ### Offline deterministic actor with a behaviour-cloning term (TD3+BC) The method starts from TD3 and changes two things. The actor maximises λ times the critic's Q-value at the policy's action, minus the squared error between the policy's action and the logged action [arxiv:2106.06860 eq. 3]. The scale λ is α divided by the batch mean of |Q|, with α = 2.5, and it is not differentiated [arxiv:2106.06860 eq. 5] [arxiv:2106.06860 § 5]. States are normalised with statistics from the dataset [arxiv:2106.06860 § 5]. The rest is TD3 with its standard settings: twin critics, target policy smoothing, delayed actor updates and soft target updates [arxiv:2106.06860 app. B] [arxiv:2106.06860 tab. 4]. The paper reports a total normalised score of 979.3 over 15 D4RL MuJoCo v0 dataset and environment pairs, against 974.6 for Fisher-BRC and 764.3 for CQL. Training is reported at about 39 minutes, against 2h12m and 4h11m for those two methods [arxiv:2106.06860 tab. 2] [arxiv:2106.06860 tab. 3]. These numbers are author-reported. Card: [A Minimalist Approach to Offline Reinforcement Learning](/papers/2106.06860). ### Continuous-action actor-critic head (DDAC, driving) The planner of an end-to-end driving pipeline uses a deep deterministic actor-critic head for continuous actions. The actor outputs steering, gear, brake and acceleration, and the critic scores them [arxiv:1704.02532 § 3.2] [arxiv:1704.02532 § 5]. The actor's gradient is the critic's gradient with respect to the action, multiplied by the actor's gradient with respect to its weights [arxiv:1704.02532 § 3.2]. The critic is trained as in DQN, against a one-step target [arxiv:1704.02532 § 3.2]. The paper tests lane keeping in the TORCS simulator. It describes the DDAC actions as smoother than tiled DQN actions, but reports no metrics, and it does not state the hyperparameters [arxiv:1704.02532 § 5]. Card: [Deep Reinforcement Learning framework for Autonomous Driving](/papers/1704.02532). ### Per-column A3C learner (progressive networks) Progressive networks add one column per task. Each column has its own policy and value heads, and it is trained with asynchronous advantage actor-critic (A3C) using 16 workers [arxiv:1606.04671 app. 3] [arxiv:1606.04671 § 5.1]. While a new column trains, the earlier columns are frozen, and the new column reads their hidden activations through lateral connections [arxiv:1606.04671 § 2] [arxiv:1606.04671 eq. 1]. The paper does not say whether the value head gets lateral inputs [arxiv:1606.04671 § 5.1] [arxiv:1606.04671 app. 3], and it does not state the discount factor, the n-step length or the value-loss weight [arxiv:1606.04671 app. 3]. The paper's contribution is the column structure; A3C is the learner it trains each column with. Card: [Progressive Neural Networks](/papers/1606.04671). ### Asymmetric SAC with a privileged critic (REAP) The actor is a camera-only SAC policy. Its surround-view images are lifted to bird's-eye-view (BEV) features, and the target slot is given as a BEV heatmap [arxiv:2605.08713 § 3] [arxiv:2605.08713 § 4.1]. The critic gets the simulator's artifact-free BEV maps (occupancy, parking-slot detection and target), encoded by an autoencoder, together with the actor's action feature. Its loss is the autoencoder loss plus a value loss [arxiv:2605.08713 § 4.2]. The actor's loss is the standard SAC form with an entropy term weighted by a temperature α, whose value the paper does not state [arxiv:2605.08713 eq. 5] [arxiv:2605.08713 § 4.2]. The deployed actor sees only images, so the critic cannot run on a real car [arxiv:2605.08713 § 4.2]. The actor is also pulled toward a rule-based planner. Where a Reeds-Shepp planner action exists, behaviour cloning with mean squared error replaces the SAC loss with probability p. The probability falls linearly from 1 to 0 over 8,000 epochs [arxiv:2605.08713 eq. 4] [arxiv:2605.08713 eq. 6] [arxiv:2605.08713 § 5.3]. Experience is kept in two replay buffers, one for all transitions and one for collision-free successful episodes. The sampling ratio between them is not stated [arxiv:2605.08713 § 4.2]. The reward combines sparse episode terms with dense collision terms [arxiv:2605.08713 eq. 7] [arxiv:2605.08713 eq. 8]. The paper reports 83.5% parking success in simulation on standard slots, with author-reported ablations. Removing the critic's auxiliary tasks drops success to 40.5%, and removing the second training stage drops it to 0.0% [arxiv:2605.08713 tab. 2]. Card: [REAP: Reinforcement-Learning End-to-End Autonomous Parking with Gaussian Splatting Simulator for Real2Sim2Real Transfer](/papers/2605.08713). ### Per-agent independent PPO critic (IPPO) Each agent runs PPO on its own local observation history, with a local critic V(z^a) [arxiv:2011.09533 § 4]. The agents share one actor and one critic weight set, and each agent's advantage is computed from the team reward rather than from a per-agent reward [arxiv:2011.09533 § 4]. The advantage is a truncated generalised-advantage sum over h steps, with γ = 0.99 and λ = 0.95 [arxiv:2011.09533 eq. 4] [arxiv:2011.09533 § 4]. The actor's loss is the clipped PPO surrogate, applied per agent, and the critic's loss is clipped in the same way [arxiv:2011.09533 eq. 5] [arxiv:2011.09533 eq. 6]. The clip width ε = 0.2 is listed in the table caption [arxiv:2011.09533 tab. 1]. The paper gives no algorithm box; its update is written in equations and prose [arxiv:2011.09533 § 4]. The paper's main claim is that this independent setup matches or beats joint-critic methods on several SMAC maps, with mild per-map tuning [arxiv:2011.09533 § 5.1]. On three maps the author-reported median win rates are 100% on 3m and 2m vs 1z, and 90% on 3s5z vs 3s6z [arxiv:2011.09533 fig. 2]. The paper reports that IPPO exceeds MAPPO and QMIX on 3s5z vs 3s6z, corridor and 6h vs 8z [arxiv:2011.09533 § 5.1]. The MAPPO values in its figure come from a benchmarking study that the paper cites, not from runs in the paper [arxiv:2011.09533 fig. 2]. The ablations remove the clipping terms, and the authors report that policy clipping is essential. A lower learning rate for the unclipped variant does not reproduce the gain [arxiv:2011.09533 § 5.3]. The paper's own hypothesis is that clipping may limit the effect of other agents' changing behaviour on each learner's update [arxiv:2011.09533 § 1]. The paper also reports that replacing the local critics with a centralised critic on full state performs substantially worse on four maps [arxiv:2011.09533 § 5.2]. Its theory section notes that independent learners lose the usual convergence guarantees under non-stationarity, and gives a relative-overgeneralisation example in which IPPO is prone to failure [arxiv:2011.09533 § 6] [arxiv:2011.09533 fig. 6]. All results are author-reported. Card: [Independent PPO for SMAC](/papers/2011.09533). ### Bayesian GP critic (BAC) The critic is GPTD, a Gaussian-process model of Q. Its prior is a sum of a state kernel and a Fisher kernel u(z)^T G^{-1} u(z') on state-action pairs [arxiv:2604.27563 § 7.2]. Each observed reward is modelled as Q(z_i) minus gamma times Q(z_{i+1}), plus noise, so the critic is fitted from transitions rather than from a separate target network [arxiv:2604.27563 § 7.1]. The posterior gives both a Q estimate and its covariance [arxiv:2604.27563 § 7.1]. The actor's gradient is the posterior mean U_t alpha_t, where U_t stacks the score vectors of the visited pairs. The covariance is G − U_t C_t U_t^T [arxiv:2604.27563 prop. 6]. The update can be preconditioned by the inverse Fisher matrix for a natural-gradient step [arxiv:2604.27563 alg. 4]. The Fisher matrix is estimated from data. For the discounted state distribution, which the paper uses, its G-EST estimator applies [arxiv:2604.27563 § 7.2] [arxiv:2604.27563 alg. 3]. The observation unit is a single transition, so the Markov property is used, unlike whole-trajectory policy gradients [arxiv:2604.27563 § 7.2]. The paper gives a sparse form with a dictionary of state-action pairs [arxiv:2604.27563 § 7.3] [arxiv:2604.27563 prop. 7]. The authors report that BAC reaches better policies than Monte-Carlo policy gradient on a random walk, mountain car and ship steering [arxiv:2604.27563 § 8.1] [arxiv:2604.27563 § 8.2] [arxiv:2604.27563 § 8.3]. These results appear in figures only, and they are author-reported. Card: [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563). ### Off-policy twin critic at massive parallel scale (FastSAC) The learner is FastSAC, an off-policy method with a SAC-style actor. The recipe runs it on thousands of parallel simulated humanoids and keeps old transitions in a replay buffer [arxiv:2512.01996 § 2.1]. The critic is distributional (C51), because the authors found quantile regression too costly at large batch sizes [arxiv:2512.01996 § 2.1]. The target averages the two critics instead of taking their minimum, which the authors report as harmful with layer normalisation [arxiv:2512.01996 § 2.1] [arxiv:2512.01996 alg. 1]. The initial entropy temperature is 0.001 and it is tuned automatically, with a target entropy of 0.0 for locomotion [arxiv:2512.01996 § 2.1]. The maximum standard deviation of the pre-tanh action is 1.0 [arxiv:2512.01996 § 2.1]. The paper tests velocity tracking on Unitree G1 and Booster T1 humanoids under strong domain randomisation, and whole-body motion tracking on G1 [arxiv:2512.01996 § 3.1] [arxiv:2512.01996 § 3.2]. It reports locomotion policies trained in 15 minutes on one RTX 4090 [arxiv:2512.01996 § 3.1]. These numbers are author-reported, and the learning curves appear only in figures [arxiv:2512.01996 § 3.1]. Card: [Learning Sim-to-Real Humanoid Locomotion in 15 Minutes](/papers/2512.01996). ### Vehicle-dynamics world-model critic (VDD) A Dreamer-style world model is split into two parts. One is an ego-vehicle dynamics model with physical parameters. The other is an environment model that sees only the ego state [arxiv:2512.02417 § 4.1]. The actor outputs a goal, and a PID controller turns each goal into an action, so the vehicle's response is handled by feedback [arxiv:2512.02417 § 4.2.1]. The critic uses symlog-transformed returns, two-hot targets and a bin-weighted readout [arxiv:2512.02417 § 4.2.2]. The actor loss has three parts: a return term divided by max(1, S), an entropy term and a reachability penalty [arxiv:2512.02417 eq. 19]. Two mechanisms handle a changed car. PAD adjusts the trained policy's actions at deployment, and PAT trains the policy on imagined trajectories under shifted vehicle parameters [arxiv:2512.02417 § 4.3] [arxiv:2512.02417 § 4.4]. PAD cannot help when the changed car cannot reach the needed states, for example with a smaller steering limit [arxiv:2512.02417 § 4.3]. The paper reports roundabout success of 0.70 against 0.53 for DreamerV3, and slightly lower success on intersections, 0.88 against 0.91 [arxiv:2512.02417 tab. 1]. The mass and steering robustness results are figure-only [arxiv:2512.02417 § 5.2.4]. The experiments use the MetaDrive and CARLA simulators, not real vehicles [arxiv:2512.02417 § 5.1.1] [arxiv:2512.02417 § 6]. Card: [Vehicle Dynamics Embedded World Models for Autonomous Driving](/papers/2512.02417). ### Demonstration-based optimal-transport reward with a TD learner (TemporalOT) The paper changes the reward the learner sees, and leaves the learner's structure as it is. After each rollout, the trajectory is scored against one or two expert video demonstrations with an optimal-transport plan, and each step receives the negative cost it carries under that plan [arxiv:2410.21795 eq. 5]. The frame cost is a cosine cost between fixed ResNet50 features. Two changes make it order-aware: the cost averages over a short window of consecutive frames [arxiv:2410.21795 eq. 6], and the plan is restricted to a band around the diagonal [arxiv:2410.21795 eq. 9]. The agent is then updated with a TD loss on the relabelled transitions [arxiv:2410.21795 eq. 1] [arxiv:2410.21795 alg. 1]. The pixel agent follows the DrQ-v2 model of ADS [arxiv:2410.21795 § 4.6]. The paper says the base OT reward depends on the whole trajectory, which makes it non-stationary during training [arxiv:2410.21795 § 3.1.2]. Author-reported: average success 61.1 over nine Meta-World tasks with two demonstrations, against 35.9 for ADS and 20.8 for the sparse task reward [arxiv:2410.21795 tab. 1]. The method assumes the demonstrator moves at about the agent's speed; success drops at 3x and 4x demonstrator speed on several tasks [arxiv:2410.21795 tab. 3]. Card: [Robot Policy Learning with Temporal Optimal Transport Reward](/papers/2410.21795). ### Advantage critic for PPO on a simulated mobile robot (crop-row navigation) The learner is PPO from Stable Baselines3, with settings stated in full: 2,048 rollout steps, batch size 64, 10 epochs, γ = 0.99, GAE λ = 0.95 and clip range 0.2 [arxiv:2412.18865 tab. III]. The paper does not state the critic's architecture [arxiv:2412.18865 tab. III]. The actor outputs one continuous value per step, which sets the turn rate of a four-wheel-steering robot [arxiv:2412.18865 § III-C] [arxiv:2412.18865 tab. I]. The reward is a sum of six terms: distance to the nearest waypoint, camera alignment, a waypoint bonus, a goal term, control smoothness and a time penalty [arxiv:2412.18865 § III-C]. Four learners were compared, SAC, TD3, A2C and PPO. The paper reports PPO as best, but only from reward and episode-length plots [arxiv:2412.18865 § IV-B] [arxiv:2412.18865 fig. 9] [arxiv:2412.18865 fig. 10]. The success rate of 351 of 420 randomised trials is author-reported, in simulation only [arxiv:2412.18865 tab. IV] [arxiv:2412.18865 § V]. Card: [Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning](/papers/2412.18865). ### Online actor-critic for distributed MPC (explicit policies) This actor-critic is not trained on rollouts. In each prediction interval, each robot's critic is fitted to a desired costate from the value update [arxiv:2412.19669 eq. 9a] [arxiv:2412.19669 eq. 12]. The critic approximates the costate as a linear combination of basis functions of the neighbourhood error [arxiv:2412.19669 eq. 11]. The actor is a linear combination of its own basis, pulled toward a target built from the neighbours' costates [arxiv:2412.19669 eq. 14] [arxiv:2412.19669 eq. 15]. Both weight sets take gradient steps, with rates 0.4 for the critic and 0.2 for the actor [arxiv:2412.19669 eq. 13] [arxiv:2412.19669 eq. 16] [arxiv:2412.19669 tab. I]. The sweep repeats until the weight change is small, with a cap of 30 sweeps [arxiv:2412.19669 alg. 1] [arxiv:2412.19669 tab. I], and the robot applies the actor's output for one step [arxiv:2412.19669 alg. 1]. The learned policy need not lower the cost at each step. The paper therefore compares it with a baseline whose cost decreases, and re-initialises when the check fails [arxiv:2412.19669 eq. 17] [arxiv:2412.19669 eq. 18]. Under two conditions on the learning rates and basis norms, the weight errors are uniformly ultimately bounded [arxiv:2412.19669 app. A]. The safe variant for state and control limits adds log-barrier terms and a force-field policy [arxiv:2412.19669 eq. 21]. Its algorithm and stability proof are in an auxiliary file that is not in the source package [arxiv:2412.19669 § V-B] [arxiv:2412.19669 § VI]. Reported costs are per step for all robots on one machine. Deployment takes 9.6e-4 s per step at six robots, against 40.2 s for IPOPT at 30 iterations. Learning takes 0.084 s per step at six robots and 14.57 s at 10,000 robots [arxiv:2412.19669 tab. III]. Collision-avoidance success is 100% at 1 and 1.5 m/s and falls at higher speeds, read from a plot [arxiv:2412.19669 fig. 7]. Card: [Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC](/papers/2412.19669). ### Multi-agent DDPG, one actor per base station (edge association) Each base station is an agent that runs DDPG, and all agents share one reward. The paper casts the choice of twin association, per-twin batch sizes and sub-channel bandwidth as a Markov decision process, with one agent per base station [arxiv:2011.09902 § 4] [arxiv:2011.09902 § 4.1]. Each agent's action is its twin count, its batch sizes and its bandwidth shares [arxiv:2011.09902 § 4.1]. The reward is the negative time cost, and every agent receives the same value, because the total time is set by the slowest base station [arxiv:2011.09902 § 4.1] [arxiv:2011.09902 eq. 19]. The actor is deterministic, with Ornstein-Uhlenbeck noise added to its output for exploration [arxiv:2011.09902 § 4.2] [arxiv:2011.09902 eq. 21]. The actor gradient is taken through a critic that reads the actions of all agents [arxiv:2011.09902 eq. 22]. The critic's TD error is written for one agent's action, while its gradient is written over all actions [arxiv:2011.09902 eq. 23]. The paper does not say whether each agent has its own critic or they share one. The twin counts must be integers [arxiv:2011.09902 eq. 18b], but the paper does not say how a continuous actor output becomes an integer count. Learning rates, the target update rate, the replay size and the mini-batch size are not stated [arxiv:2011.09902 § 4.2]. Card: [Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks](/papers/2011.09902). ### Model-based value expansion critic with TD-k (MVE) The critic is trained as in DDPG, but its target is not a one-step Bellman backup. The target sums H imagined steps under the policy, using a learned one-step model and the true reward, and then adds the critic's value at the imagined state after H steps [arxiv:1803.00101 § 3] [arxiv:1803.00101 eq. 2]. Setting H = 0 gives the plain actor-critic method [arxiv:1803.00101 § 3.2]. The TD-k trick trains the critic on imagined states too. Each imagined step at offset t gets a target with horizon H − t, and the actor is updated on the real transition [arxiv:1803.00101 § 3.2] [arxiv:1803.00101 alg. 1]. Imagined transitions are not stored, because the actor changes during training [arxiv:1803.00101 § 3.2]. The dynamics model is a fully connected network with 8 layers of 128 units, trained to predict the change in state at a learning rate of 10⁻³ [arxiv:1803.00101 § 3.2]. The paper's assumptions are a deterministic environment, a known reward and a model accurate for H steps [arxiv:1803.00101 § 3]. Its results are learning curves on three continuous-control tasks. The author-reported gain over DDPG and an imagination-buffer baseline is smaller on walker [arxiv:1803.00101 fig. 3]. The paper reports no final-score table. Card: [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](/papers/1803.00101). ## When to use which | Situation | Use | Why | |---|---|---| | Real interactions are expensive, and you can learn a model of the environment | Imagined-rollout critic | Most practice happens in imagination. DreamerV3 reports 0.1 to 16.1 A100 GPU days per benchmark, with one fixed configuration [arxiv:2301.04104 tab. 2] | | You train an LLM with feedback on one answer per prompt | Token-level critic with single rollout | GRPO needs several answers per prompt, which one-answer-per-prompt tasks cannot supply [arxiv:2607.07508 § 1] [arxiv:2607.07508 § 2] | | A multi-turn LLM agent acts in whole steps, with the environment reacting once per step, and you can afford a critic | Action-level critic with action-aware ratio (CAPO) | The advantage and clipping work per step rather than per token, and the ratio does not grow with action length. The paper reports 58.20 WebShop success against 46.00 for PPO on Qwen3-4B, author-reported [arxiv:2604.18401 § 3.2] [arxiv:2604.18401 tab. 1] | | Goal reaching from images or from a state with a goal given | Goal-conditioned contrastive critic | No hand-written reward is needed, and the paper reports that contrastive RL beats auxiliary-loss methods without augmentation [arxiv:2206.07568 § 1] [arxiv:2206.07568 § 5.2] | | Offline data with no environment access | Goal-conditioned contrastive critic, offline variant | The paper adds a behavioural-cloning term to the actor, with coefficient λ [arxiv:2206.07568 § 5.5] | | Offline logged data with continuous actions, and you already run TD3 | TD3+BC | Adds one term and one hyperparameter, α = 2.5. Author-reported total of 979.3 on D4RL MuJoCo v0, with about 39 minutes of training [arxiv:2106.06860 tab. 2] [arxiv:2106.06860 tab. 3] | | Each decision is one continuous action that picks an input to a fixed generator | DDPG critic | One action per episode, with a learned Q-function and a deterministic actor. The paper reports about 1 ms per action for the agent [arxiv:1904.12304 § 4.1] | | Sequential tasks where earlier skills must not be overwritten, and the model may grow with each task | Per-column A3C learner (progressive networks) | Frozen earlier columns cannot be damaged by later training. The 2-column net reaches 209% mean transfer on Pong Soup against 181% for finetuning, but on Atari it is slightly below finetuning (132% against 133%) [arxiv:1606.04671 tab. 1] | | A simulator can provide state that the deployed policy will not see, such as BEV ground truth, and the policy must use camera input only | Asymmetric SAC with a privileged critic | The critic can use what the actor lacks, and the actor stays deployable. The critic's auxiliary tasks matter: removing them drops simulated success from 83.5% to 40.5% [arxiv:2605.08713 tab. 2] | | Expert planner available, but RL from scratch explores poorly | Asymmetric SAC with a planner behaviour-cloning term | The term is switched off over training. Without the RL stage, simulated success is 0.0% [arxiv:2605.08713 § 4.2] [arxiv:2605.08713 tab. 2] | | Few real episodes, single-transition data, and you can estimate a Fisher matrix and pick a state kernel | Bayesian GP critic (BAC) | The critic comes with a posterior covariance, and the gradient posterior is closed-form given the score vectors [arxiv:2604.27563 prop. 6]. The paper reports BAC policies beating Monte-Carlo policy gradient on three simulated tasks, in figures [arxiv:2604.27563 § 8] | | Several cooperating agents, each with only a local view, and a joint value function would have an exponentially growing joint action space | Per-agent independent PPO critic (IPPO) | Each agent learns from its own history with a shared team reward, and no joint critic is needed. The paper reports IPPO at or above MAPPO and QMIX on several SMAC maps, with mild tuning, but the MAPPO figures come from a cited benchmark [arxiv:2011.09533 § 5.1] [arxiv:2011.09533 fig. 2] | | Simulation is the bottleneck: you can run thousands of environments on one GPU and need a fast sim-to-real loop for humanoid locomotion | Off-policy twin critic at massive parallel scale (FastSAC) | Old transitions are reused, and the paper reports locomotion policies in 15 minutes on one RTX 4090, against PPO in its figures [arxiv:2512.01996 § 2.1] [arxiv:2512.01996 § 3.1] | | The vehicle's mass or steering limits may differ from training, and a learned world model is available | Vehicle-dynamics world-model critic (VDD) | The ego dynamics are separated from the scene, so the policy can be adjusted at deployment (PAD) or trained across shifted parameters (PAT). PAD fails when the new car cannot reach the needed states. Roundabout success is 0.70 against 0.53 for DreamerV3 [arxiv:2512.02417 § 4.3] [arxiv:2512.02417 § 4.4] [arxiv:2512.02417 tab. 1] | | Only one or two expert demonstrations, no task reward, and expert videos of the task | Demonstration-based optimal-transport reward with a TD learner (TemporalOT) | The reward is order-aware, and the paper reports 61.1 average success on nine Meta-World tasks against 35.9 for ADS, with the same two demos [arxiv:2410.21795 tab. 1]. Demonstrator speed must stay near the agent's [arxiv:2410.21795 tab. 3] | | Simulated mobile robot where you can write a per-step reward and run many short episodes | PPO with a value baseline (crop-row navigation) | Settings are given in full [arxiv:2412.18865 tab. III]. The paper's 351 of 420 randomised trials succeed in simulation only, and its algorithm comparison is in plots [arxiv:2412.18865 tab. IV] [arxiv:2412.18865 fig. 9] | | Many robots in a formation, a solver per step is too slow on onboard hardware, and the nonlinear error model is known | Online actor-critic for distributed MPC | Deployment is one formula per step: 9.6e-4 s at six robots, against 40.2 s for IPOPT at 30 iterations. Learning time is extra, and the stability check needs a baseline cost [arxiv:2412.19669 tab. III] [arxiv:2412.19669 eq. 18] | | Several nodes each set part of a shared allocation (assignment, batch size, bandwidth), they share one time-based reward, and a simulator can run the whole system | Multi-agent DDPG, one actor per agent | The paper's agents are the base stations, with one shared reward [arxiv:2011.09902 § 4.1]. Its time-cost gain is shown in figures, against two simple association rules [arxiv:2011.09902 fig. 5]. Integer assignments need a step the paper does not state [arxiv:2011.09902 eq. 18b] | | Real interactions are expensive, a deterministic model with a known reward is available, and it is accurate for a short horizon | MVE critic with TD-k | The horizon H sets how far the model is trusted, and TD-k trains the critic on imagined states. The paper's gains are learning curves only [arxiv:1803.00101 § 3.1] [arxiv:1803.00101 fig. 3] | | Offline logged data, and you need to choose a fix for the critic or actor | Offline actor-critic with one modification | The survey lists the modification classes: policy constraints, regularisation, uncertainty, one-step and others. It states the trade-offs in its own table, without a single ranking [arxiv:2203.01387 tab. 1] [arxiv:2203.01387 § III] | | You cannot afford a value model of the same size as the policy | Not stated | None of the papers on this page reports the memory cost of its critic against the policy it trains [arxiv:2607.07508 § 3.2] | Notes that come from the cited results: - The papers' own numbers do not show that one variant is better across tasks. Each paper compares against baselines on its own benchmarks [arxiv:2301.04104 § Results] [arxiv:2607.07508 § 4.2] [arxiv:2206.07568 § 5.1] [arxiv:1904.12304 § 4.1] [arxiv:1606.04671 § 5.1]. - The per-agent PPO variant's lower-learning-rate control applies to IAC only. The authors say other baselines were not tested with the same control, so the learning-rate ablation does not show that other methods are untuned [arxiv:2011.09533 § 5.3]. - The imagined-rollout and token-level variants both report that the critic needs careful handling. DreamerV3 uses a twohot readout and a return normalisation [arxiv:2301.04104 § Robust predictions]. SAO freezes the critic's attention layers for stability [arxiv:2607.07508 § 3.2]. - The contrastive variant depends on batch size. The other batch rows act as negatives, so the batch is part of the method [arxiv:2206.07568 tab. 2]. - The DDPG variant's reward weights are tied to the paper's own loss definitions. The paper warns that they cannot be copied onto a loss with a different scale without a check [arxiv:1904.12304 supp. RL Agent Details]. - The TD3+BC variant is weakest on random data. It scores 10.2 on HalfCheetah-random, below CQL (21.7) and Fisher-BRC (32.2) [arxiv:2106.06860 tab. 2], and the paper calls this a concern [arxiv:2106.06860 app. C.1]. Its α was tuned on Hopper tasks that are also evaluation tasks [arxiv:2106.06860 app. B]. - The progressive-network mean is sensitive to outliers. On Labyrinth the 2-column mean is 491% against a median of 115% [arxiv:1606.04671 tab. 1] [arxiv:1606.04671 § 5.2]. Compare medians when the gap is large. - The asymmetric variant's real-car results come from one vehicle and one seed, with no confidence intervals. Each trial moves a scenario's success rate by 5 points [arxiv:2605.08713 tab. 4]. - The Bayesian variant's Fisher matrix is estimated from data in its experiments. The paper's closed forms are exact only for the true Fisher matrix, so the posterior moments are approximate in practice [arxiv:2604.27563 app. A] [arxiv:2604.27563 § 4.3]. - The FastSAC variant's results are figure-only, apart from the 15-minute training time. The paper gives no numbers for its own ablations in the text [arxiv:2512.01996 § 3.1] [arxiv:2512.01996 § 2.1]. - The VDD variant has no official code. Its controller gains, reachability horizon, entropy scale and critic bin count are not stated, so an implementer must set them and tune them [arxiv:2512.02417 § 4.2] [arxiv:2512.02417 eq. 19] [arxiv:2512.02417 § 5.2.3]. - The TemporalOT variant reports five seeds, and some standard deviations are large, for example 78.4 (12.4) on Door-open and 42.2 (44.5) for ADS on Basketball [arxiv:2410.21795 tab. 1]. Compare the spread, not just the mean. - The PPO variant's only comparison with a classical controller is a PD controller on a skid-steer robot, a different platform. Its distance and time are 5.97 m and 49.4 s against 5.29 m and 18.0 s, and the PD gains and the number of runs are not stated [arxiv:2412.18865 tab. V]. It does not show that the learner beats tuned classical control. - The distributed MPC variant compares a horizon of 20 steps with baselines that use 10 [arxiv:2412.19669 § VI]. The solver comparison in its table is therefore not at equal horizons. At two robots with obstacles, its cost is 13.5 against 5.0 for IPOPT, so the cost favours IPOPT there [arxiv:2412.19669 tab. III]. - The multi-agent DDPG variant compares against two simple association rules, both set up by the authors, and reports no repeated runs or error bars [arxiv:2011.09902 § 5]. Its gains are read from figures, and the text gives no numbers [arxiv:2011.09902 fig. 5] [arxiv:2011.09902 fig. 7]. - The MVE variant's analysis assumes a deterministic environment and a known reward [arxiv:1803.00101 § 3]. Its experiments use 4 seeds with a learned model and 6 seeds with an oracle model [arxiv:1803.00101 app. A], so the learned-model curves rest on few runs. - The offline survey's heat map collects scores the authors found in the literature. It orients the reader and does not rank methods on one task [arxiv:2203.01387 fig. 5]. - The FPGA survey's A3C advantage is a one-step TD error, while its PPO section names GAE without defining it [arxiv:2610.28699 eq. 7] [arxiv:2610.28699 § 3.8]. Do not treat the two as the same estimator. ## Implementation sketch The shared core: fit the critic on collected data, then update the actor from the critic. The variants differ in how the data is gathered, how the critic target is built, and what the actor's objective is. Not every variant uses an advantage. ```pseudocode # Shared actor-critic update; each variant fills in the marked steps # [arxiv:2301.04104 § Critic learning] [arxiv:2607.07508 § 3.2] # [arxiv:2011.09533 § 4] [arxiv:2011.09533 eq. 5] [arxiv:2011.09533 eq. 6] # [arxiv:2206.07568 § 4.4] [arxiv:1904.12304 § 3.3] # [arxiv:2106.06860 § 5] [arxiv:1704.02532 § 3.2] [arxiv:1606.04671 § 2] # [arxiv:2604.27563 § 7.1] [arxiv:2604.27563 prop. 6] # [arxiv:2512.01996 alg. 1] [arxiv:2512.02417 § 4.2.2] [arxiv:2512.02417 eq. 19] def actor_critic_step(batch, actor, critic, lr_critic, lr_actor): # Step 1: source of the batch differs by variant # DreamerV3: imagined rollouts from a learned world model # SAO: one rollout per prompt, with rollout log-probs logged # contrastive RL: replayed (state, action, future state) triples # DDPG: one-step transitions, with a reward and no future state # TD3+BC: logged offline (state, action, reward, next state) tuples # DDAC driving: simulator transitions with continuous actions # REAP: sim rollouts; the critic also sees privileged BEV maps # progressive nets: A3C rollouts on the new task, earlier columns frozen # IPPO: each agent's own observation history, with the team reward # BAC: single transitions, with the score vector of each state-action pair # FastSAC: replay of parallel simulated transitions [alg. 1] # TemporalOT: replayed transitions relabelled with the OT reward [eq. 5] # PPO crop navigation: on-policy rollouts of 2,048 steps [tab. III] # distributed MPC: no rollouts; a model propagates neighbourhood errors # [eq. 3] # VDD: imagined rollouts from replayed latent starts [eq. 14] [eq. 15] # edge association: one agent per base station, shared reward [§4.1] [eq. 19] # MVE: real start states, then H imagined steps from a learned model [alg. 1] # Step 2: critic target and critic update # DreamerV3: lambda-returns, cross-entropy on twohot targets # SAO: skip-observation GAE; two critic updates, attention frozen # contrastive RL: binary cross-entropy on the batch score matrix # DDPG: Bellman target for Q(s, a) # TD3+BC: clipped double-Q target from two target critics # DDAC driving: one-step target, as in DQN # REAP: autoencoder loss plus value loss on the critic's inputs # progressive nets: A3C value loss, n-step length not stated # IPPO: per-agent clipped value loss on a GAE target [eq. 6] # BAC: GPTD posterior over Q; no Bellman regression target [§7.1] # FastSAC: twin critics regress to the averaged target [alg. 1] # VDD: two-hot cross-entropy on symlog lambda-returns [§4.2.2] # TemporalOT: TD target from the relabelled transitions [eq. 1] # PPO crop navigation: GAE advantage targets, lambda 0.95 [tab. III] # distributed MPC: least-squares fit to the desired costate [eq. 12] # edge association: DDPG TD target, y_t not written out [§4.2] [eq. 23] # MVE: TD-k loss, averaged over imagined steps, with horizon H - t [§3.2] critic_loss = critic_target_loss(batch, critic) update(critic, critic_loss, lr_critic) # Step 3: actor update from the critic # DreamerV3, SAO: policy gradient on an advantage [§ Actor learning] # contrastive RL: maximise the critic score at the commanded goal # DDPG: deterministic policy gradient through Q(s, a) [eq. 6] # TD3+BC: lambda * Q(s, pi(s)) minus squared error to the logged action [eq. 3] # DDAC driving: critic gradient w.r.t. the action through the actor [§ 3.2] # REAP: SAC loss, or MSE to the planner action with probability p [eq. 4] # progressive nets: A3C policy gradient with an entropy term [app. 3] # IPPO: per-agent clipped PPO surrogate, summed over agents [eq. 5] [eq. 7] # BAC: posterior mean U_t alpha_t, optionally with inverse Fisher [prop. 6] # FastSAC: entropy-weighted Q on a reparameterised sample [alg. 1] # VDD: return term, entropy and reachability penalty on goals [eq. 19] # TemporalOT: TD-trained actor, DrQ-v2 model in the pixel setup [§ 4.6] # PPO crop navigation: clipped objective on the advantage [tab. III] # distributed MPC: actor fitted to neighbours' costate [eq. 15] # edge association: deterministic policy gradient through the critic [eq. 22] # MVE: usual actor loss on the real transition [alg. 1 line 9] actor_loss = actor_objective(batch, actor, critic) update(actor, actor_loss, lr_actor) ``` ## Contradictions None known between the twenty papers' reported results on a common benchmark. Their benchmarks and metrics differ, so their numbers are not directly comparable. Within single papers, the following disagree: - Bayesian GP critic: the LQR rewards are costs and the LQR is minimised, but the update rules are written as ascent, theta + beta * delta [arxiv:2604.27563 alg. 2] [arxiv:2604.27563 alg. 4]. The authors explain the cost convention in a footnote [arxiv:2604.27563 § 6.2] but do not show the sign change in the update. - Bayesian GP critic: the text says BPG is similar to BAC at M = 100 [arxiv:2604.27563 § 8.1], while its random-walk policy runs use M = 1, 25, 50 and 75 [arxiv:2604.27563 § 8.1] [arxiv:2604.27563 tab. 4]. The card treats the M = 100 claim as unverified. - TD3+BC: the paper's λ averages |Q| over the logged actions [arxiv:2106.06860 eq. 5], while the released code averages |Q| over the policy's own actions [code: TD3_BC.py#L140-L141]. An implementation that follows only the equation computes a different λ. - Per-agent PPO: the text gives the convolution strides as (2, 1, 1), while the table caption says all convolutions use stride 3 [arxiv:2011.09533 § 4] [arxiv:2011.09533 tab. 1]. The paper does not say which is used. - Per-agent PPO: the text says 16 SMAC maps, while the table lists 17 [arxiv:2011.09533 § 5] [arxiv:2011.09533 tab. 1]. The count is not reconciled. - TD3+BC: the printed Fisher-BRC total is 974.6, but summing its 15 per-row scores gives 975.2 [arxiv:2106.06860 tab. 2]. - DreamerV3: the text gives the critic's prediction horizon as 16 steps [arxiv:2301.04104 § Critic learning], while its hyperparameter table gives an imagination horizon of 15 [arxiv:2301.04104 tab. 4]. - SAO: the introduction claims stable training for "around one thousand" steps [arxiv:2607.07508 § 1]. The explicit curves and text reach about 400 steps [arxiv:2607.07508 § 4.2]. - SAO: the ablation's running-mean baseline uses a window of 8 recent rewards per prompt [arxiv:2607.07508 § 4.3], while the online writing simulation uses 128 [arxiv:2607.07508 § 4.5]. These are two different baselines, so the window is not one setting. - REAP: the simulated success rate for its own 3DGS simulator is 82.5% in [arxiv:2605.08713 tab. 3] and 83.5% for the same method on standard slots in [arxiv:2605.08713 tab. 1]. The paper does not explain the gap. - FastSAC: Algorithm 1 and §2.1 give a target that averages the two target Q-values [arxiv:2512.01996 alg. 1] [arxiv:2512.01996 § 2.1]. The released code builds one projected target per critic and does not average across them [code: src/holosoma/holosoma/agents/fast_sac/fast_sac_agent.py#L445-L459]. The paper does not say which one produced its figures [arxiv:2512.01996 fig. 3]. - Distributed MPC: Table III gives a per-step cost V that is lower for the proposed method without obstacles, 0.73 against 1.46 at two robots [arxiv:2412.19669 tab. III]. The text gives a cumulative cost over 180 steps at two robots of 8.9 for DMPC and 16.9 for the proposed method [arxiv:2412.19669 § VI]. The paper argues that closed-loop performance matters more, and does not test a third metric. - TemporalOT: Table 3 gives Button-press at 1x speed as 89.0 (3.8), which is the ADS entry in Table 1. Table 1 gives TemporalOT 92.4 (3.6) for the same task [arxiv:2410.21795 tab. 3] [arxiv:2410.21795 tab. 1]. The paper does not explain the mismatch. - Multi-agent DDPG (edge association): the text says each agent's reward uses T_i [arxiv:2011.09902 § 4.1], while eq. 17 defines one global T as maxima over base stations [arxiv:2011.09902 eq. 17]. The paper gives every agent the same −T [arxiv:2011.09902 eq. 19]. Which time is meant per agent is not stated. - Multi-agent DDPG (edge association): the text says agents train in parallel [arxiv:2011.09902 § 4.2], while Algorithm 1 nests the episode loop inside a loop over base stations [arxiv:2011.09902 alg. 1]. The paper does not explain the order. - Multi-agent DDPG (edge association): as printed, eqs. 24 and 25 have θ on both sides of the target update, so the target equals the primary network [arxiv:2011.09902 eq. 24] [arxiv:2011.09902 eq. 25]. The text does not say which soft update is intended. - CAPO: the paper sets the clip width to ε = 0.001 [arxiv:2604.18401 § 4.1], while the step-level scripts in the released repository set 0.0003 [code: examples/webshop/run_steppo.sh#L13-L14]. The same scripts use the GSPO ratio, which takes the mean of token log-ratios, and the action-aware ratio of [arxiv:2604.18401 eq. 4] is not in the repository [code: verl/verl/trainer/ppo/core_algos.py#L1037-L1056]. The paper does not say which code produced its tables [arxiv:2604.18401 § 4.1]. ## Papers - [Independent PPO for SMAC](/papers/2011.09533): independent PPO, with a per-agent local critic, a shared team reward and shared weights, compared with joint-critic methods on SMAC. - [Mastering Diverse Domains through World Models](/papers/2301.04104): an imagined-rollout actor and critic that use one fixed configuration across many domains. - [Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning](/papers/2607.07508): a token-level critic for LLM RL with one rollout per prompt, and skip-observation advantages. - [Contrastive Learning as Goal-Conditioned Reinforcement Learning](/papers/2206.07568): a contrastive critic whose score is a goal-conditioned Q-function up to a constant, with an actor that maximises it. - [Deep Reinforcement Learning framework for Autonomous Driving](/papers/1704.02532): a driving pipeline whose planner is a DDAC actor-critic head for continuous actions, shown on lane keeping without metrics. - [RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion](/papers/1904.12304): a DDPG agent that picks a latent GAN seed in one step per shape. - [A Minimalist Approach to Offline Reinforcement Learning](/papers/2106.06860): TD3 plus a behaviour-cloning term in the actor loss, with state normalisation, for offline D4RL data. - [Progressive Neural Networks](/papers/1606.04671): one column per task with frozen earlier columns and lateral connections, trained with A3C. The actor-critic is the learner here, not the contribution. - [REAP: Reinforcement-Learning End-to-End Autonomous Parking with Gaussian Splatting Simulator for Real2Sim2Real Transfer](/papers/2605.08713): a camera-only SAC parking actor with a privileged BEV critic, and a planner behaviour-cloning term annealed to zero. - [Robot Policy Learning with Temporal Optimal Transport Reward](/papers/2410.21795): an optimal-transport reward with a context window and a temporal band, used to relabel trajectories for a TD learner. - [Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning](/papers/2412.18865): PPO from Stable Baselines3 for crop-row and waypoint navigation in simulation, compared with SAC, TD3 and A2C. - [Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC](/papers/2412.19669): an online actor-critic that learns explicit distributed MPC policies, with a barrier variant for state and control limits. - [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563): a GPTD critic with a Fisher kernel, whose gradient posterior is closed-form, compared with Monte-Carlo policy gradient. - [Learning Sim-to-Real Humanoid Locomotion in 15 Minutes](/papers/2512.01996): off-policy FastSAC on massively parallel simulation, with a recipe of action bounds, layer norm, averaged critics and a minimal reward. - [Vehicle Dynamics Embedded World Models for Autonomous Driving](/papers/2512.02417): a Dreamer-style world model that separates ego dynamics from the scene, with a goal-output actor and PAD and PAT for changed vehicles. - [Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks](/papers/2011.09902): one DDPG agent per base station, choosing twin association, batch sizes and bandwidth to cut training time, with a shared reward. - [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](/papers/1803.00101): model-based value expansion with a TD-k critic loss on imagined states, in a DDPG-style actor-critic. Results are learning curves only. - [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](/papers/2203.01387): a survey that treats offline actor-critic as a base learner plus modification classes. It proposes no method. - [Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey](/papers/2610.28699): a survey of RL on FPGAs and integrated circuits, with actor-critic and A3C hardware pipelines. It runs no experiments. - [CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities](/papers/2604.18401): a PPO-style method for multi-turn LLM agents with one critic value and one advantage per action, and an action-aware clipped ratio.