Offline reinforcement learning

Overview

Offline reinforcement learning (offline RL) learns a policy from a fixed dataset collected by some unknown behaviour process, without trying new actions in the environment 2106.06860 §1 §3 2601.10471 §1. Off-policy algorithms can run on such data, but they tend to fail because value estimates for actions missing from the dataset are wrong, and the policy learns to prefer them 2106.06860 §1 §3 2601.10471 §2.1.

The invariant that the approaches share is to keep the learned policy tied to the dataset's actions, though the five cards do this in different ways. TD3+BC enforces it directly with a behaviour-cloning term 2106.06860 §5. DeFlow enforces it indirectly: the policy is a logged-action flow model plus a learned shift, and the size of the shift is penalised 2601.10471 §3.2. QPHIL extracts its low-level policies by advantage-weighted regression on logged actions 2411.07760 §4.3 eq. 2. HDP adds a small Q term to a behaviour-cloning loss 2411.12982 §3.2 eq. 9. DexDiff has no value term at all: its policy is trained with a denoising loss only 2409.05493 §3.2 eq. 4. The methods with a value term trade off how much they reward value improvement against how far they move from the log.

A survey gives a wider map of the field. It groups offline methods by the part of the learning problem they change: the learned policy, the policy gradient, the Q-function or policy objective, the reward, the number of evaluation steps, the training data, or the trajectory distribution 2203.01387 tab. 1. It places TD3+BC among the policy-constraint methods 2203.01387 §IV-A. The survey proposes no method and runs no experiments of its own 2203.01387 §I.

Variants

TD3+BC

TD3 with two changes and nothing else: a behaviour-cloning term in the actor objective, weighted by a normalized Q term, and normalization of the dataset's states 2106.06860 §5 app. B. The only hyperparameter it adds is α = 2.5 2106.06860 §5 app. B. On D4RL MuJoCo v0 (15 dataset and environment pairs) it reports a total normalized score of 979.3 ± 33.4, against 974.6 ± 108.3 for Fisher-BRC and 764.3 ± 61.5 for CQL 2106.06860 tab. 2. Its training took about 39 minutes, against 2h 12m for Fisher-BRC and 4h 11m for CQL, on one GPU and without evaluation 2106.06860 tab. 3. All scores are author-reported. Card: A Minimalist Approach to Offline Reinforcement Learning.

Weak spots reported in the same paper: on random data it scores 10.2 on HalfCheetah, below CQL (21.7) and Fisher-BRC (32.2) 2106.06860 tab. 2, and on the AntMaze large tasks its scores are near zero 2106.06860 tab. 8.

DeFlow

Keeps a multi-step flow-matching policy as a fixed behaviour prior and trains a small residual network on top of it 2601.10471 §3.2 alg. 1. The flow is fitted only to logged actions and receives no Q-learning gradient, so the prior stays unchanged 2601.10471 §2.2 §3.2. The residual sees the state and one sampled base action, outputs a shift, and the executed action is the base plus the shift, clipped to the action bounds 2601.10471 §3.2. The residual is trained to raise a Q term normalised by the batch mean of |Q|, with a penalty on the mean squared shift. The penalty weight is raised by a dual step whenever the shift exceeds a per-task budget δ 2601.10471 §3.2 app. B.1. The paper argues that the gradient never passes through the ODE solver, which is the cost it wants to avoid for multimodal generators 2601.10471 §1 §3.1.

The paper contrasts this with one-step or few-step shortcut policies such as FQL and SORL, which it says can compress a multimodal flow 2601.10471 §1 §3.1. Author-reported results: 40 ± 5 on OGBench cube-double against 29 ± 2 for FQL and 25 ± 3 for SORL, and 43 ± 4 on puzzle-3x3 against 30 ± 1 for FQL 2601.10471 tab. 1. On the OGBench antmaze-large row, SORL leads at 89 ± 2, and DeFlow reports 81 ± 3 2601.10471 tab. 1. A diagnostic on cube-double task 1 shows the base flow alone at 10 ± 2 and the full policy at 62 ± 12, so the residual carries most of the gain there 2601.10471 app. C.3. Card: DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction.

The budget δ is an average squared displacement per action coordinate, not a radius, and it is enforced on minibatches only 2601.10471 §3.2. Per-task values in the paper's table run from 0.0005 for puzzle tasks to 0.015 for D4RL antmaze 2601.10471 app. B.1. Changing it changes the result: on a manipulation diagnostic, δ = 0.0005 gives 55 ± 8 and δ = 0.005 gives 27 ± 4 2601.10471 app. C.1. The paper's puzzle budget differs from the one in the authors' public code, so check the card before reproducing puzzle numbers 2601.10471 app. B.1.

QPHIL

An offline goal-conditioned method for navigation, with hierarchical planning over discrete labels. A VQ-VAE maps each state to one of a fixed set of landmark tokens, trained with reconstruction, commitment and temporal triplet losses 2411.07760 §4.1. A transformer predicts the landmark sequence for a goal, and two IQL-style policies follow it: a landmark policy for each token and a goal policy for the final approach 2411.07760 §4.2 §4.3. The paper argues that discrete planning avoids the noise that HIQL's continuous subgoals pick up as the subgoal horizon grows 2411.07760 §3 app. A.

Stitching augmentation splices two trajectories that share two landmarks, and the paper assumes the low-level policy can reach any state of a landmark from any other 2411.07760 §4.2. Author-reported: on AntMaze-ultra diverse, 70.3 ± 6.9 without augmentation, the best value in that row, against 52.9 ± 17 for HIQL with representation 2411.07760 tab. 1. On the paper's extreme map, QPHIL with augmentation reaches 50.0 ± 6.9 on play, against 22.8 ± 7.9 for HIQL without representation 2411.07760 fig. 5. The paper says the advantage is largest on very large mazes 2411.07760 §5.3. Card: Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning.

The card records a text-table conflict on the larger maps and a seed-count conflict; see Contradictions.

HDP

HDP is framed as imitation learning, but its actor carries an offline Q-learning term. Two conditional diffusion models run in sequence. A Guider predicts the next object contact, and an Actor denoises a 16-step action chunk toward that contact 2411.12982 §3.2 tab. 8. The Actor loss is a denoising mean-squared error minus η times a Q value, with η = 0.001 [arxiv:2411.12982 eq. 9] tab. 9. The Q value is taken at a one-step denoised action, which the paper calls snapshot gradient optimisation. At 10 denoising steps the authors report about 4× faster training and 24% less GPU memory for this form 2411.12982 §4.4 eq. 11.

The critic is trained on demonstration data with noise-augmented actions 2411.12982 §4.3 eq. 15. Its target uses the next tuple from the demonstration, not the Actor's prediction 2411.12982 §3.2 eq. 13. Author-reported: an average improvement of 20.8% over Diffusion Policy across six tasks, with the simulation baseline reduced in size and reproduced by the authors 2411.12982 §6 tab. 2. Card: Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance.

The effect of the Q term in the paper's own tables is mixed. Comparing η = 0.001 with η = 0 on the maximum-success column, η = 0.001 is higher in six rows, lower in six and equal in six 2411.12982 tab. 2 tab. 3. Its object-point-cloud input and single-object assumption also limit where it applies 2411.12982 §4.2 §6.

DexDiff

DexDiff is on this page because its low-level policy is trained only on a fixed set of demonstrations and is conditioned on return-to-go values. The paper does not present it as offline RL. The GCAD policy is a diffusion model that predicts a chunk of actions, conditioned on observations and on return-to-go values 2409.05493 §3.2. Its training loss is the denoising mean-squared error 2409.05493 eq. 4. It has no Q-function and no value-maximising step.

The return-to-go values come from a vision-language plan, via goal-conditioned grasp poses and relational tuples. The paper gives no formula for them 2409.05493 §3.1 §3.2. Author-reported: 70% average success over four simulated layouts, against 23% for the Finetuned-VLM with Diffusion Policy-T. Every method in that table uses the same VLM planner, so only the low-level policy is compared 2409.05493 §4.2 tab. 1. On a real UR5e across four setups it reports 65% (26 of 40), against 35% for a heuristic planner with the same policy 2409.05493 tab. 2. The paper says the approach needs a lot of demonstration data and is computationally expensive 2409.05493 §5. Card: DexDiff: Towards Extrinsic Dexterity Manipulation of Ungraspable Objects in Unrestricted Environments.

Other methods the papers compare against

CQL, Fisher-BRC, BRAC, AWAC and the Decision Transformer are the baselines in 2106.06860 §6 tab. 2 and 2106.06860 app. C.3. DeFlow compares against BC, IQL, ReBRAC, FAWAC, FBRAC, IFQL, FQL, SORL, Cal-QL and RLPD 2601.10471 §4.1 tab. 1 tab. 2. QPHIL compares against GCBC, IQL and GC-IQL, G-ADT, TAP, TT, PT, HGCBC and HIQL 2411.07760 §5.1. HDP compares against LSTM-GMM, IBC, BET and Diffusion Policy in simulation 2411.12982 §6.2 tab. 2. DexDiff compares against behaviour cloning, Decision Transformer and Diffusion Policy as low-level baselines 2409.05493 §4.1. None of these has a card on this site, so this page does not describe their mechanisms beyond what the cards say about them. The TD3+BC paper says that CQL and Fisher-BRC add implementation-level and algorithmic changes to SAC, and that removing the implementation-level changes lowers their performance on many tasks 2106.06860 §4 tab. 1 fig. 1.

When to use which

The paper's α sweep shows that α = 2 and α = 3 perform about the same, and that some tasks lose performance as α moves toward 1 or 4 2106.06860 fig. 6 app. B. The paper tuned α on Hopper tasks that are also used for evaluation 2106.06860 app. B.

Implementation sketch

A TD3+BC-style actor update, in our own words. Critic updates are unchanged from TD3 2106.06860 §5. The actor and targets update on a delay 2106.06860 tab. 4.

# Offline actor update on top of TD3 [arxiv:2106.06860 §5]
# States are normalized once, with dataset statistics
def actor_update(batch, alpha):
    # Q term scaled by the batch's mean |Q|, so its size is
    # independent of reward scale [arxiv:2106.06860 eq. 5]
    q_pi = critic(batch.states, actor(batch.states))
    lam = alpha / mean(abs(q_pi)), not differentiated
    # Maximize scaled Q while staying close to logged actions
    # [arxiv:2106.06860 eq. 3]
    loss = -lam * mean(q_pi) + mean((actor(batch.states) - batch.actions)^2)
    update(actor, loss)

The paper's equation computes the mean over dataset actions, while the authors' code uses the policy's actions. See Contradictions.

DeFlow keeps the same critic-and-Q-term shape but changes the actor. The action is a frozen flow sample plus a residual, the penalty is on the residual's squared size rather than on distance to the logged action, and the penalty weight follows a dual update toward budget δ 2601.10471 §3.2 alg. 1. The card has the full update.

HDP's actor loss has the same outer shape: a denoising error minus η times a Q value, evaluated at a one-step denoised action 2411.12982 eq. 9 eq. 11. The paper's equation has no Q-scale normalisation, but the authors' code divides the Q term by a detached mean of the other critic's |Q| hiera_diffusion_policy.py. So η is not on the same scale as TD3+BC's λ or DeFlow's normalised term, and the values should not be copied between methods. See Paper vs code on the HDP card.

Contradictions

• Within TD3+BC: the λ normalizer is defined over dataset actions 2106.06860 eq. 5, while the authors' code averages over policy actions. Results may differ depending on which is used 2106.06860 §5.
• Within TD3+BC: the behaviour-cloning term is written as a squared error without saying whether it is summed or averaged over action dimensions 2106.06860 eq. 3. This changes its weight relative to Q.
• Within QPHIL, larger maps: the text says QPHIL "outperforms all other methods on the larger maps" 2411.07760 §5.2. In its Table 1, HIQL with representation scores 88.2 ± 5.3 on large diverse, above both QPHIL variants (80.25 and 80.5) 2411.07760 tab. 1. The claim does not hold for that row.
• Within QPHIL, seeds: Section 5.1 says QPHIL and HIQL ran 8 seeds 2411.07760 §5.1. The reproducibility statement names the "same 8 seeds" but lists 7 values 2411.07760 §8. Which seeds were used is unclear.
• Within HDP, the Q term: the text says η = 0.001 is best for most tasks tested 2411.12982 §4.4. In the maximum-success column of the paper's tables it is higher than η = 0 in six of 18 rows, lower in six, and equal in six 2411.12982 tab. 2 tab. 3.
• Within DexDiff, data source: Section 4.1 says the simulated trajectories combine reinforcement learning with human demonstration 2409.05493 §4.1. The appendix says the simulated data comes only from Soft Actor-Critic policies 2409.05493 app. A.1.
• Between papers, AntMaze: TD3+BC reports near-zero scores on the large D4RL AntMaze tasks 2106.06860 tab. 8. DeFlow reports 81 ± 3 on OGBench antmaze-large and 83 ± 2 averaged over six D4RL antmaze tasks 2601.10471 tab. 1. QPHIL reports 80.25 ± 3.3 with augmentation and 80.5 ± 7.4 without on large diverse 2411.07760 tab. 1. The benchmarks and aggregation differ, so these are not a direct conflict, but the AntMaze picture depends on which benchmark is used.
• Between papers, benchmarks: the five cards report on different task sets. TD3+BC uses D4RL MuJoCo; DeFlow uses OGBench, D4RL antmaze and adroit, and visual tasks; QPHIL uses AntMaze with D4RL play and diverse datasets, including ultra and extreme maps; HDP uses robomimic, Tilt and Push-T in simulation and a real UR5e Move-T task; DexDiff uses simulated tabletop layouts and a real UR5e. Their scores cannot be compared directly.
• Between papers, Q-term scale: TD3+BC scales Q by the batch mean |Q| 2106.06860 eq. 5, DeFlow divides Q by a stopped batch mean of |Q| 2601.10471 §3.2, and HDP's equation has no such scale 2411.12982 eq. 9. Coefficients from one method do not carry over to another.

Papers

• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems: a taxonomy of offline RL method modifications, a review of benchmarks and off-policy evaluation, and open problems; no new method.
• A Minimalist Approach to Offline Reinforcement Learning: TD3+BC, a one-term change to TD3 with state normalization, evaluated on D4RL MuJoCo.
• DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction: a frozen multi-step flow prior with a small residual trained under an adaptive trust-region penalty, evaluated on OGBench and D4RL.
• Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning: discrete landmark planning with a VQ-VAE and two IQL-style policies, evaluated on AntMaze.
• Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance: a contact-guided diffusion actor with a small Q term, evaluated on robomimic, Tilt, Push-T and a real robot.
• DexDiff: Towards Extrinsic Dexterity Manipulation of Ungraspable Objects in Unrestricted Environments: a VLM plan turned into return-to-go values for a diffusion policy, evaluated in simulation and on a UR5e.