A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

In plain words

Offline reinforcement learning (RL) teaches a controller only from a fixed log of past experience, with no new trial and error. That matters where trying things is slow, costly or dangerous, such as healthcare, education, robotics and driving. The main difficulty is that a learned policy can choose actions the log rarely or never shows, and its estimates for those actions can be wrong.

This survey sorts the many offline RL methods into a small set of building blocks, so a reader can see what each method changes. It explains each block in one shared notation, reviews the datasets used to test these methods and how to score a policy without running it, and lists open questions. It does not propose a new algorithm.

TL;DR

A survey of offline RL with a taxonomy of method modifications tab. 1, a review of algorithms in one notation §IV, a review of benchmarks and off-policy evaluation §V §VI, and open problems §VIII. It proposes no new algorithm and reports no new experiments. Its one performance figure fig. 5 is built from scores the authors found in the literature, so it orients the reader rather than ranking methods on a task.

Problem

Offline RL learns from a fixed dataset of transitions, with no further interaction with the environment §II-C. The authors motivate it with robotics, education, healthcare and autonomous driving, where online data collection is impractical §I. Standard off-policy RL methods were designed assuming that further interaction can correct errors, so they often fail on purely offline data §I.

The core obstacle is distributional shift. The learned policy visits states and takes actions the dataset covers poorly, so value estimates there can be wrong and the error compounds §II-C2. The Bellman error is computed under the learned policy, but the data come from a different behavior policy eq. 4.

Offline RL has several desirable properties in practice: generalisation, filtering of good behaviour, and stitching of good parts of different trajectories §II-C1. The authors also list the challenges that make it hard, such as the inability to try new actions to find high-return regions §II-C2.

The survey states that the field's publications have grown rapidly, and that few works have reviewed it. The most notable earlier review is the tutorial by Levine et al., arXiv:2005.01643 §I. The authors propose a taxonomy under one notation, plus a benchmark review and a discussion of OPE §I.

Method

This is a survey, so its "method" is a classification and a review.

Taxonomy. Every offline RL algorithm is a base actor-critic (or imitation, or trajectory) learner plus modifications. Table 1 lists the modification types and what each one changes tab. 1. Some modifications change the algorithm's class ("is-a" relationships, e.g. one-step), and others can be added to any class ("has-a" relationships, e.g. policy constraints) tab. 1 §III.

tab. 1 §III-A to §III-H

Classes and their core equations. The survey describes each class in §III and reviews the methods in §IV. The equations below are the ones an implementer needs to recognise each class. The survey's notation is used throughout: π_β is the behavior policy, π_θ or π_off the learned policy, and D the dataset.

• Policy constraints §III-A §IV-A. Constrained improvement maximises expected Q under the learned policy, subject to a divergence bound from the behavior policy eq. 5. Its Lagrangian eq. 6 has a closed-form nonparametric solution, which is projected back onto the policy class by a weighted maximum-likelihood step eq. 7. Implicit constraints skip the estimate of π_β. Example: BCQ samples candidate actions from a generative model of π_β, perturbs them within a bound and maximises Q over them eq. 16 §IV-A. TD3+BC adds a behaviour-cloning term to the TD3 actor loss eq. 20.
• Importance sampling §III-B §IV-B. The policy gradient is reweighted by products of importance ratios over the horizon eq. 8. Per-decision weighting shortens the product eq. 21. Doubly robust estimators add a Q control variate eq. 22. The state-marginal ratio ρ^π avoids the exponential weights and is learned from a Bellman-style constraint eq. 23 §IV-B.
• Regularization §III-C §IV-C. Policy regularization adds an objective term such as the policy entropy eq. 9 eq. 25. Value regularization changes the Bellman objective eq. 10. Conservative Q-learning (CQL) pushes down Q at actions from a distribution μ and up at dataset actions eq. 27. With μ taken as the softmax of Q, this becomes a log-sum-exp term eq. 28.
• Uncertainty estimation §III-D §IV-D. The objective subtracts an uncertainty penalty on the Q distribution eq. 11. An ensemble of Q-functions estimates that distribution eq. 29. REM uses random convex combinations of ensemble members eq. 30. The reward can be penalised by an uncertainty measure eq. 12.
• Model-based §III-E §IV-E. A dynamics model is learned from D. MOReL penalises state-action pairs where ensemble models disagree beyond a threshold eq. 31 eq. 32. MOPO uses the Frobenius norm of the model covariance as the uncertainty eq. 33 eq. 34. COMBO uses a conservative value penalty over model rollouts and dataset points, mixed by a fraction f eq. 36 eq. 37 eq. 38. BREMEN trains a policy inside an ensemble of learned models under a trust-region constraint eq. 35.
• One-step §III-F §IV-F. One policy evaluation step is followed by one policy improvement step, avoiding iterative evaluation on out-of-distribution actions. Implicit Q-learning (IQL) fits Q to the reward plus V of the next state eq. 39. It fits V to an upper expectile of Q over dataset actions eq. 40 eq. 41. The expectile loss weights residuals by τ on one side and by 1 − τ on the other eq. 41. V approximates the maximum of Q over actions with π_β(a|s) above a threshold ε eq. 42. The policy-extraction step is described as similar to advantage-weighted regression and is not detailed in the survey §IV-F.
• Imitation learning §III-G §IV-G. Behaviour cloning minimises an f-divergence between the learned and behavior policies eq. 13. Conditional imitation learns π_θ(a|s, ω) from an outcome ω sampled from the rest of the trajectory eq. 14. BAIL fits V to returns, weights errors by w(s_t) eq. 43 eq. 44, and keeps the pairs with return above ρ·V(s_t) for cloning eq. 45. ABM filters by an n-step advantage and an indicator eq. 46 eq. 47. CRR uses a max-normalised advantage eq. 48. RvS conditions on a goal outcome eq. 49 or a reward outcome eq. 50.
• Trajectory optimization §III-H §IV-H. A sequence model is trained on trajectories of (return-to-go, state, action) tokens eq. 51. Trajectory Transformer (TT) plans with beam search. Decision Transformer (DT) instead minimises the error on actions and conditions on a target return.
• Off-policy evaluation (OPE) §V. Three estimators are described: a model rollout estimate eq. 52, importance sampling eq. 53, and fitted Q evaluation, which averages Q over states and actions of the evaluation dataset eq. 54.

Benchmarks §VI. D4RL covers OpenAI Gym MuJoCo tasks, mazes, Adroit, FrankaKitchen, CARLA and traffic flow, with random, medium and expert data in some environments §VI-A2 tab. 3. RL Unplugged covers DeepMind Control, DM locomotion, Atari and real-world RL suite tasks §VI-A2 tab. 3. The survey lists eight desirable dataset properties: narrow data, undirected multitask data, sparse rewards, suboptimal data, non-representable behavior policies, non-Markovian behavior policies, realistic domains and nonstationarity §VI-A1. The DOPE benchmark is the OPE benchmark reviewed §VI-B.

Algorithm pseudocode. The survey gives no code, so this is a sketch of the IQL update from its equations.

# Implicit Q-learning (IQL), one update  [§IV-F] [eq. 39-41]
# Input:  batch of (s, a, r, s') from D; expectile tau in [0, 1]
# Output: value and Q losses for this batch

def iql_losses(batch):
    # Fit V toward the upper expectile of Q over dataset actions  [eq. 40-41]
    diff = v(s) - q(s, a)                          # (B,)
    weight = where(diff > 0, 1 - tau, tau)         # (B,)
    v_loss = mean(weight * diff ** 2)
    # Fit Q to reward plus V(next state), not a policy action  [eq. 39]
    target = r + gamma * v(s_next)                 # (B,)
    q_loss = mean((q(s, a) - target) ** 2)
    return v_loss, q_loss

Hyperparameters

The survey has no hyperparameters of its own. The table lists the values it states for the methods it reviews. "not stated" means the survey gives none.

Results

Author-reported, from the survey's own review. The survey runs no experiments.

fig. 5 shows, for each method and each taxonomy class, a score on each of the eight dataset properties. Light gray cells mean the method was not evaluated on any dataset with that property. Class scores are the maximum over the methods in that class. The raw scores are in the supplementary materials the paper links fig. 5 §I.

Paper vs code

No official code.

The survey gives no implementation. Its footnote 1 links supplementary materials, including a publication-count figure and the raw scores behind fig. 5, to a GitHub repository §I fig. 5. The paper presents these as supplementary material, not as code. This card did not open that repository and does not use it.

Pitfalls

• This is a survey, not a method. Nothing here can be reproduced as a model. Use the equations to recognise a method's class, and go to the original paper for its hyperparameters and training details.
• The performance figure is aggregated. fig. 5 takes scores from different papers, on different environments and protocols, then takes class maxima. It shows coverage and tendency, not a controlled ranking §VI-C fig. 5. The paper notes that the D4RL and RL Unplugged evaluation protocols differ §VI-A2.
• Classes overlap. A single algorithm can belong to several classes. Table II marks "has-a" modifications with a plus sign and the class with a check mark tab. 2. The authors call their classification non-exhaustive §IV.
• Behavior-policy estimation is a shared weak point. Methods that estimate π_β inherit its errors, and a unimodal fit to multimodal data can fail badly §III-A. Implicit constraints do not need this estimate §III-A.
• Ambiguous points in the equations. IQL's V-update eq. 40 and policy-extraction step are not fully specified in the survey §IV-F. BAIL's weight K is described only as "much larger than 1" §IV-G. ABM uses an indicator where AWR uses an exponential weight eq. 47 §IV-G.
• Importance-sampling weights blow up. The vanilla weight product is exponential in the horizon H §III-B eq. 8. Per-decision, doubly robust and marginalised estimators reduce but do not remove this eq. 21 eq. 22 eq. 23.
• COMBO's data fraction is a hidden knob. The mix f decides how much of the penalised distribution comes from dataset points, which changes how conservative the Q-function is eq. 38 §IV-E.
• Heat map and benchmarks lack coverage. Non-stationary, stochastic and multi-agent datasets are largely missing, so results on these properties are not available §VI-A3. The authors say hyperparameter tuning and OPE still lack satisfying solutions §VIII.
• Version. The PDF read is the accepted manuscript of IEEE TNNLS (DOI 10.1109/TNNLS.2023.3250269) p. 1. The arXiv version is v3, updated 2023-04-19.

Lineage

• Reviews and extends: the tutorial by Levine et al., arXiv:2005.01643 §I.
• Points readers to: the deep RL surveys by Arulkumaran et al. and Wang et al. for general RL background §II-B.
• Superseded by: not known from this wiki.