--- kind: concept slug: offline-reinforcement-learning title: Offline reinforcement learning summary: >- Learning a control policy from a fixed log of past experience, without new environment interaction. Variants keep the policy near logged actions in different ways: TD3+BC adds a behaviour-cloning term, DeFlow learns a residual on a frozen flow prior, QPHIL plans over learned landmarks. Gains are author-reported, on different benchmarks. tags: [reinforcement-learning, offline-rl] papers: ["2106.06860", "2601.10471", "2411.07760", "2411.12982", "2409.05493", "2203.01387"] related: [actor-critic, imitation-learning, flow-matching, residual-learning, goal-conditioned-reinforcement-learning, hierarchical-reinforcement-learning, diffusion-models, return-conditioned-sequence-modeling, vector-quantization, model-based-reinforcement-learning] status: draft updated_at: 2026-11-01 --- ## Overview Offline reinforcement learning (offline RL) learns a policy from a fixed dataset collected by some unknown behaviour process, without trying new actions in the environment [arxiv:2106.06860 §1] [§3] [arxiv:2601.10471 §1]. Off-policy algorithms can run on such data, but they tend to fail because value estimates for actions missing from the dataset are wrong, and the policy learns to prefer them [arxiv:2106.06860 §1] [§3] [arxiv:2601.10471 §2.1]. The invariant that the approaches share is to keep the learned policy tied to the dataset's actions, though the five cards do this in different ways. TD3+BC enforces it directly with a behaviour-cloning term [arxiv:2106.06860 §5]. DeFlow enforces it indirectly: the policy is a logged-action flow model plus a learned shift, and the size of the shift is penalised [arxiv:2601.10471 §3.2]. QPHIL extracts its low-level policies by advantage-weighted regression on logged actions [arxiv:2411.07760 §4.3] [eq. 2]. HDP adds a small Q term to a behaviour-cloning loss [arxiv:2411.12982 §3.2] [eq. 9]. DexDiff has no value term at all: its policy is trained with a denoising loss only [arxiv:2409.05493 §3.2] [eq. 4]. The methods with a value term trade off how much they reward value improvement against how far they move from the log. A survey gives a wider map of the field. It groups offline methods by the part of the learning problem they change: the learned policy, the policy gradient, the Q-function or policy objective, the reward, the number of evaluation steps, the training data, or the trajectory distribution [arxiv:2203.01387 tab. 1]. It places TD3+BC among the policy-constraint methods [arxiv:2203.01387 §IV-A]. The survey proposes no method and runs no experiments of its own [arxiv:2203.01387 §I]. ## Variants ### TD3+BC TD3 with two changes and nothing else: a behaviour-cloning term in the actor objective, weighted by a normalized Q term, and normalization of the dataset's states [arxiv:2106.06860 §5] [app. B]. The only hyperparameter it adds is α = 2.5 [arxiv:2106.06860 §5] [app. B]. On D4RL MuJoCo v0 (15 dataset and environment pairs) it reports a total normalized score of 979.3 ± 33.4, against 974.6 ± 108.3 for Fisher-BRC and 764.3 ± 61.5 for CQL [arxiv:2106.06860 tab. 2]. Its training took about 39 minutes, against 2h 12m for Fisher-BRC and 4h 11m for CQL, on one GPU and without evaluation [arxiv:2106.06860 tab. 3]. All scores are author-reported. Card: [A Minimalist Approach to Offline Reinforcement Learning](/papers/2106.06860). Weak spots reported in the same paper: on random data it scores 10.2 on HalfCheetah, below CQL (21.7) and Fisher-BRC (32.2) [arxiv:2106.06860 tab. 2], and on the AntMaze large tasks its scores are near zero [arxiv:2106.06860 tab. 8]. ### DeFlow Keeps a multi-step flow-matching policy as a fixed behaviour prior and trains a small residual network on top of it [arxiv:2601.10471 §3.2] [alg. 1]. The flow is fitted only to logged actions and receives no Q-learning gradient, so the prior stays unchanged [arxiv:2601.10471 §2.2] [§3.2]. The residual sees the state and one sampled base action, outputs a shift, and the executed action is the base plus the shift, clipped to the action bounds [arxiv:2601.10471 §3.2]. The residual is trained to raise a Q term normalised by the batch mean of |Q|, with a penalty on the mean squared shift. The penalty weight is raised by a dual step whenever the shift exceeds a per-task budget δ [arxiv:2601.10471 §3.2] [app. B.1]. The paper argues that the gradient never passes through the ODE solver, which is the cost it wants to avoid for multimodal generators [arxiv:2601.10471 §1] [§3.1]. The paper contrasts this with one-step or few-step shortcut policies such as FQL and SORL, which it says can compress a multimodal flow [arxiv:2601.10471 §1] [§3.1]. Author-reported results: 40 ± 5 on OGBench cube-double against 29 ± 2 for FQL and 25 ± 3 for SORL, and 43 ± 4 on puzzle-3x3 against 30 ± 1 for FQL [arxiv:2601.10471 tab. 1]. On the OGBench antmaze-large row, SORL leads at 89 ± 2, and DeFlow reports 81 ± 3 [arxiv:2601.10471 tab. 1]. A diagnostic on cube-double task 1 shows the base flow alone at 10 ± 2 and the full policy at 62 ± 12, so the residual carries most of the gain there [arxiv:2601.10471 app. C.3]. Card: [DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction](/papers/2601.10471). The budget δ is an average squared displacement per action coordinate, not a radius, and it is enforced on minibatches only [arxiv:2601.10471 §3.2]. Per-task values in the paper's table run from 0.0005 for puzzle tasks to 0.015 for D4RL antmaze [arxiv:2601.10471 app. B.1]. Changing it changes the result: on a manipulation diagnostic, δ = 0.0005 gives 55 ± 8 and δ = 0.005 gives 27 ± 4 [arxiv:2601.10471 app. C.1]. The paper's puzzle budget differs from the one in the authors' public code, so check the card before reproducing puzzle numbers [arxiv:2601.10471 app. B.1]. ### QPHIL An offline goal-conditioned method for navigation, with hierarchical planning over discrete labels. A VQ-VAE maps each state to one of a fixed set of landmark tokens, trained with reconstruction, commitment and temporal triplet losses [arxiv:2411.07760 §4.1]. A transformer predicts the landmark sequence for a goal, and two IQL-style policies follow it: a landmark policy for each token and a goal policy for the final approach [arxiv:2411.07760 §4.2] [§4.3]. The paper argues that discrete planning avoids the noise that HIQL's continuous subgoals pick up as the subgoal horizon grows [arxiv:2411.07760 §3] [app. A]. Stitching augmentation splices two trajectories that share two landmarks, and the paper assumes the low-level policy can reach any state of a landmark from any other [arxiv:2411.07760 §4.2]. Author-reported: on AntMaze-ultra diverse, 70.3 ± 6.9 without augmentation, the best value in that row, against 52.9 ± 17 for HIQL with representation [arxiv:2411.07760 tab. 1]. On the paper's extreme map, QPHIL with augmentation reaches 50.0 ± 6.9 on play, against 22.8 ± 7.9 for HIQL without representation [arxiv:2411.07760 fig. 5]. The paper says the advantage is largest on very large mazes [arxiv:2411.07760 §5.3]. Card: [Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](/papers/2411.07760). The card records a text-table conflict on the larger maps and a seed-count conflict; see Contradictions. ### HDP HDP is framed as imitation learning, but its actor carries an offline Q-learning term. Two conditional diffusion models run in sequence. A Guider predicts the next object contact, and an Actor denoises a 16-step action chunk toward that contact [arxiv:2411.12982 §3.2] [tab. 8]. The Actor loss is a denoising mean-squared error minus η times a Q value, with η = 0.001 [arxiv:2411.12982 eq. 9] [tab. 9]. The Q value is taken at a one-step denoised action, which the paper calls snapshot gradient optimisation. At 10 denoising steps the authors report about 4× faster training and 24% less GPU memory for this form [arxiv:2411.12982 §4.4] [eq. 11]. The critic is trained on demonstration data with noise-augmented actions [arxiv:2411.12982 §4.3] [eq. 15]. Its target uses the next tuple from the demonstration, not the Actor's prediction [arxiv:2411.12982 §3.2] [eq. 13]. Author-reported: an average improvement of 20.8% over Diffusion Policy across six tasks, with the simulation baseline reduced in size and reproduced by the authors [arxiv:2411.12982 §6] [tab. 2]. Card: [Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance](/papers/2411.12982). The effect of the Q term in the paper's own tables is mixed. Comparing η = 0.001 with η = 0 on the maximum-success column, η = 0.001 is higher in six rows, lower in six and equal in six [arxiv:2411.12982 tab. 2] [tab. 3]. Its object-point-cloud input and single-object assumption also limit where it applies [arxiv:2411.12982 §4.2] [§6]. ### DexDiff DexDiff is on this page because its low-level policy is trained only on a fixed set of demonstrations and is conditioned on return-to-go values. The paper does not present it as offline RL. The GCAD policy is a diffusion model that predicts a chunk of actions, conditioned on observations and on return-to-go values [arxiv:2409.05493 §3.2]. Its training loss is the denoising mean-squared error [arxiv:2409.05493 eq. 4]. It has no Q-function and no value-maximising step. The return-to-go values come from a vision-language plan, via goal-conditioned grasp poses and relational tuples. The paper gives no formula for them [arxiv:2409.05493 §3.1] [§3.2]. Author-reported: 70% average success over four simulated layouts, against 23% for the Finetuned-VLM with Diffusion Policy-T. Every method in that table uses the same VLM planner, so only the low-level policy is compared [arxiv:2409.05493 §4.2] [tab. 1]. On a real UR5e across four setups it reports 65% (26 of 40), against 35% for a heuristic planner with the same policy [arxiv:2409.05493 tab. 2]. The paper says the approach needs a lot of demonstration data and is computationally expensive [arxiv:2409.05493 §5]. Card: [DexDiff: Towards Extrinsic Dexterity Manipulation of Ungraspable Objects in Unrestricted Environments](/papers/2409.05493). ### Other methods the papers compare against CQL, Fisher-BRC, BRAC, AWAC and the Decision Transformer are the baselines in [arxiv:2106.06860 §6] [tab. 2] and [arxiv:2106.06860 app. C.3]. DeFlow compares against BC, IQL, ReBRAC, FAWAC, FBRAC, IFQL, FQL, SORL, Cal-QL and RLPD [arxiv:2601.10471 §4.1] [tab. 1] [tab. 2]. QPHIL compares against GCBC, IQL and GC-IQL, G-ADT, TAP, TT, PT, HGCBC and HIQL [arxiv:2411.07760 §5.1]. HDP compares against LSTM-GMM, IBC, BET and Diffusion Policy in simulation [arxiv:2411.12982 §6.2] [tab. 2]. DexDiff compares against behaviour cloning, Decision Transformer and Diffusion Policy as low-level baselines [arxiv:2409.05493 §4.1]. None of these has a card on this site, so this page does not describe their mechanisms beyond what the cards say about them. The TD3+BC paper says that CQL and Fisher-BRC add implementation-level and algorithmic changes to SAC, and that removing the implementation-level changes lowers their performance on many tasks [arxiv:2106.06860 §4] [tab. 1] [fig. 1]. ## When to use which | Situation | Use | Why | |---|---|---| | Choosing among method families for the first time, or placing a new method | Read the survey's taxonomy first | It separates methods by what they modify, such as policy constraints, regularization, model use or trajectory optimization [arxiv:2203.01387 tab. 1]; it runs no experiments, so use the method cards for numbers | | Fixed dataset, MuJoCo locomotion, a TD3 codebase already exists | TD3+BC | Only the actor loss and state normalization change; one new hyperparameter [arxiv:2106.06860 §5] [tab. 4] | | Training time or implementation effort is the constraint | TD3+BC | About 39 minutes against 2h 12m and 4h 11m on the paper's setup [arxiv:2106.06860 tab. 3]; DeFlow adds a flow model, a residual and a dual update [arxiv:2601.10471 §3.2] | | Multimodal demonstrations, and a flow or diffusion prior is acceptable | DeFlow | The paper argues that unimodal actors cannot represent such data while flow policies can [arxiv:2601.10471 §1]; reported gains are on cube-double and puzzle-3x3 [tab. 1] | | Fixed dataset with a single hyperparameter and no per-task sweep | TD3+BC | One value, α = 2.5, applied across the paper's tasks [arxiv:2106.06860 §5] [app. B]. DeFlow needs a per-task δ [arxiv:2601.10471 app. B.1] | | Online fine-tuning is planned after the offline phase | DeFlow, if its results on your task family transfer | Offline-to-online cube-double 48 ± 13 → 93 ± 4 and puzzle-4x4 4 ± 4 → 100 ± 0 [arxiv:2601.10471 tab. 2]; on humanoidmaze-medium IFQL reaches 82 ± 20 and DeFlow 65 ± 12 [tab. 2] | | Inference cost matters | TD3+BC over DeFlow, or measure both | DeFlow samples M flow steps and then runs the residual for each action [arxiv:2601.10471 tab. 3]; the paper's timings are 4.5 ± 0.5 ms for DeFlow and 4.2 ± 0.7 ms for FQL on one GPU, for its own implementation only [tab. 4] | | Dataset is mostly random or low-quality | Check the evidence before choosing TD3+BC | It is weakest on random data [arxiv:2106.06860 tab. 2] [app. C.1]; DeFlow reports no such case | | Goal reaching in large mazes, with trajectories that can be stitched at shared regions | QPHIL, after checking its zone assumption | Gains are reported largest on very large mazes [arxiv:2411.07760 §5.3]; the method assumes any state in a landmark can be reached from any other [§4.2]; each run takes about 10 GPU-hours on V100s [app. B.1] | | Contact-rich manipulation from demonstrations, with object point clouds and one object | HDP | Needs object point clouds [arxiv:2411.12982 §4.2] and assumes one robot and one object [§6]; the Q term's effect is mixed in the paper's own tables [tab. 2] [tab. 3] | | Language-planned, multi-step manipulation of large flat objects | DexDiff | Needs VLM fine-tuning on about 1.6k simulated and 1.4k real images [arxiv:2409.05493 §3.1] and a lot of demonstration data [§5]; the comparison holds the planner fixed [tab. 1] | | AntMaze-style sparse, long-horizon tasks | QPHIL, if its zone assumption holds | TD3+BC scores near zero on the large D4RL tasks [arxiv:2106.06860 tab. 8]; DeFlow is competitive with FQL on D4RL antmaze (83 ± 2 vs 84 ± 3) but behind SORL on OGBench antmaze-large [arxiv:2601.10471 tab. 1]; QPHIL's own table has the top AntMaze-ultra scores, while HIQL leads on large diverse [arxiv:2411.07760 tab. 1] | | Hard guarantee that actions stay in the logged support | None of the cards | DeFlow's budget holds on minibatches only, with no pointwise bound [arxiv:2601.10471 §3.2]; the other methods rely on soft weighting or on a denoising objective. No card claims a hard guarantee | | Comparing methods on the same schedule | Any; report the averages | Single checkpoints vary widely. TD3+BC averages the last 10 evaluations over 5 seeds [arxiv:2106.06860 fig. 2] [fig. 3] [tab. 2]; DeFlow uses 8 seeds on state tasks and 4 on pixel tasks [arxiv:2601.10471 §4.1]; HDP averages the last 10 checkpoints over three seeds [arxiv:2411.12982 §6.2]; QPHIL reports 8 seeds [arxiv:2411.07760 §5.1] | The paper's α sweep shows that α = 2 and α = 3 perform about the same, and that some tasks lose performance as α moves toward 1 or 4 [arxiv:2106.06860 fig. 6] [app. B]. The paper tuned α on Hopper tasks that are also used for evaluation [arxiv:2106.06860 app. B]. ## Implementation sketch A TD3+BC-style actor update, in our own words. Critic updates are unchanged from TD3 [arxiv:2106.06860 §5]. The actor and targets update on a delay [arxiv:2106.06860 tab. 4]. ```pseudocode # Offline actor update on top of TD3 [arxiv:2106.06860 §5] # States are normalized once, with dataset statistics def actor_update(batch, alpha): # Q term scaled by the batch's mean |Q|, so its size is # independent of reward scale [arxiv:2106.06860 eq. 5] q_pi = critic(batch.states, actor(batch.states)) lam = alpha / mean(abs(q_pi)), not differentiated # Maximize scaled Q while staying close to logged actions # [arxiv:2106.06860 eq. 3] loss = -lam * mean(q_pi) + mean((actor(batch.states) - batch.actions)^2) update(actor, loss) ``` The paper's equation computes the mean over dataset actions, while the authors' code uses the policy's actions. See Contradictions. DeFlow keeps the same critic-and-Q-term shape but changes the actor. The action is a frozen flow sample plus a residual, the penalty is on the residual's squared size rather than on distance to the logged action, and the penalty weight follows a dual update toward budget δ [arxiv:2601.10471 §3.2] [alg. 1]. The card has the full update. HDP's actor loss has the same outer shape: a denoising error minus η times a Q value, evaluated at a one-step denoised action [arxiv:2411.12982 eq. 9] [eq. 11]. The paper's equation has no Q-scale normalisation, but the authors' code divides the Q term by a detached mean of the other critic's |Q| [code: hiera_diffusion_policy/policy/hiera_diffusion_policy.py#L409-L414]. So η is not on the same scale as TD3+BC's λ or DeFlow's normalised term, and the values should not be copied between methods. See Paper vs code on the HDP card. ## Contradictions - **Within TD3+BC:** the λ normalizer is defined over dataset actions [arxiv:2106.06860 eq. 5], while the authors' code averages over policy actions. Results may differ depending on which is used [arxiv:2106.06860 §5]. - **Within TD3+BC:** the behaviour-cloning term is written as a squared error without saying whether it is summed or averaged over action dimensions [arxiv:2106.06860 eq. 3]. This changes its weight relative to Q. - **Within QPHIL, larger maps:** the text says QPHIL "outperforms all other methods on the larger maps" [arxiv:2411.07760 §5.2]. In its Table 1, HIQL with representation scores 88.2 ± 5.3 on large diverse, above both QPHIL variants (80.25 and 80.5) [arxiv:2411.07760 tab. 1]. The claim does not hold for that row. - **Within QPHIL, seeds:** Section 5.1 says QPHIL and HIQL ran 8 seeds [arxiv:2411.07760 §5.1]. The reproducibility statement names the "same 8 seeds" but lists 7 values [arxiv:2411.07760 §8]. Which seeds were used is unclear. - **Within HDP, the Q term:** the text says η = 0.001 is best for most tasks tested [arxiv:2411.12982 §4.4]. In the maximum-success column of the paper's tables it is higher than η = 0 in six of 18 rows, lower in six, and equal in six [arxiv:2411.12982 tab. 2] [tab. 3]. - **Within DexDiff, data source:** Section 4.1 says the simulated trajectories combine reinforcement learning with human demonstration [arxiv:2409.05493 §4.1]. The appendix says the simulated data comes only from Soft Actor-Critic policies [arxiv:2409.05493 app. A.1]. - **Between papers, AntMaze:** TD3+BC reports near-zero scores on the large D4RL AntMaze tasks [arxiv:2106.06860 tab. 8]. DeFlow reports 81 ± 3 on OGBench antmaze-large and 83 ± 2 averaged over six D4RL antmaze tasks [arxiv:2601.10471 tab. 1]. QPHIL reports 80.25 ± 3.3 with augmentation and 80.5 ± 7.4 without on large diverse [arxiv:2411.07760 tab. 1]. The benchmarks and aggregation differ, so these are not a direct conflict, but the AntMaze picture depends on which benchmark is used. - **Between papers, benchmarks:** the five cards report on different task sets. TD3+BC uses D4RL MuJoCo; DeFlow uses OGBench, D4RL antmaze and adroit, and visual tasks; QPHIL uses AntMaze with D4RL play and diverse datasets, including ultra and extreme maps; HDP uses robomimic, Tilt and Push-T in simulation and a real UR5e Move-T task; DexDiff uses simulated tabletop layouts and a real UR5e. Their scores cannot be compared directly. - **Between papers, Q-term scale:** TD3+BC scales Q by the batch mean |Q| [arxiv:2106.06860 eq. 5], DeFlow divides Q by a stopped batch mean of |Q| [arxiv:2601.10471 §3.2], and HDP's equation has no such scale [arxiv:2411.12982 eq. 9]. Coefficients from one method do not carry over to another. ## Papers - [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](/papers/2203.01387): a taxonomy of offline RL method modifications, a review of benchmarks and off-policy evaluation, and open problems; no new method. - [A Minimalist Approach to Offline Reinforcement Learning](/papers/2106.06860): TD3+BC, a one-term change to TD3 with state normalization, evaluated on D4RL MuJoCo. - [DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction](/papers/2601.10471): a frozen multi-step flow prior with a small residual trained under an adaptive trust-region penalty, evaluated on OGBench and D4RL. - [Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](/papers/2411.07760): discrete landmark planning with a VQ-VAE and two IQL-style policies, evaluated on AntMaze. - [Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance](/papers/2411.12982): a contact-guided diffusion actor with a small Q term, evaluated on robomimic, Tilt, Push-T and a real robot. - [DexDiff: Towards Extrinsic Dexterity Manipulation of Ungraspable Objects in Unrestricted Environments](/papers/2409.05493): a VLM plan turned into return-to-go values for a diffusion policy, evaluated in simulation and on a UR5e.