A Minimalist Approach to Offline Reinforcement Learning

In plain words

Offline reinforcement learning means learning a control policy only from a fixed log of past experience, without trying anything new in the environment. The known difficulty is that the learner can over-rate actions that never appear in the log, and then the policy chases those inflated values. Many existing fixes add a lot of machinery: extra settings to tune, generative models, and changes to the underlying learning algorithm.

This paper asks whether a standard deep RL algorithm can work offline with almost no changes. The proposed answer, TD3+BC, takes the existing TD3 algorithm and adds one imitation term that keeps the learned policy close to the actions in the log. The log's states are also rescaled to a standard range. The imitation term is scaled by a normalizing factor, so that a single setting works across different datasets.

On a standard benchmark of simulated locomotion tasks, the authors report results comparable to more complex offline methods, at less than half their training time. The method is presented as a simple baseline to compare against.

TL;DR

• Offline RL learns from a fixed dataset; most existing methods keep the policy near the dataset's actions, usually with extra machinery.
• TD3+BC changes TD3 in two ways: a behaviour-cloning term in the actor loss, and state normalization. Its only new hyperparameter is α = 2.5 §5 app. B.
• On D4RL MuJoCo v0 it matches Fisher-BRC's total score, beats CQL, and trains in about 39 minutes against 2h12m (Fisher-BRC) and 4h11m (CQL) tab. 2 tab. 3. All numbers are author-reported.

Problem

Offline RL must learn from data collected by an unknown behaviour process, without environment interaction §1 §3. Off-policy algorithms run on such data but tend to fail because of extrapolation error: value estimates of actions not in the dataset are wrong, and the policy learns to prefer them §1 §3. Most offline algorithms counter this by constraining or regularizing the policy toward dataset actions, and they differ in how that closeness is implemented §1 §2.

The paper's concern is cost. Offline algorithms built on top of an online one often add hyperparameters, generative models or extra computation §1 §4. Its own study of CQL and Fisher-BRC finds that removing their implementation-level changes (architecture, learning rates, pre-training, reward bonus, and similar) significantly lowers performance on many tasks fig. 1 tab. 1. CQL and Fisher-BRC also cost much more training time than their base algorithm tab. 3. The paper further reports high variance of offline-trained policies across evaluation episodes and across evaluation points fig. 2 fig. 3 §4. The setting assumed is D4RL MuJoCo, where the authors take SAC and TD3 to be strong base algorithms §4.

Method

TD3+BC is TD3 with two changes, and nothing else §5 app. B:

1. Behaviour-cloning term in the actor objective. TD3's deterministic policy maximizes the expected Q value over dataset states eq. 1. TD3+BC instead maximizes a Q term scaled by λ, minus the squared difference between the policy's action and the dataset action eq. 3. The scale λ is set per batch from α = 2.5 and the mean absolute Q value, and it is not differentiated eq. 5 §5.
2. State normalization. Each state feature in the dataset is shifted to mean 0 and scaled to standard deviation 1, with a small constant ε = 10⁻³ added to the standard deviation eq. 4 §5. The same dataset statistics are used for evaluation main.py.

The rest is TD3 with its standard hyperparameters: twin critics, target policy smoothing noise, delayed actor updates and soft target updates §5 app. B, tab. 4. The paper states that the modifications are the only changes made to the deep RL algorithm §1.

The following pseudocode follows the authors' code, with the paper's equations cited. It is an implementation guide, not a copy.

# TD3+BC, one training step  [§5] [eq. 3] [eq. 4] [eq. 5]
# Inputs: batch of normalized states, actions, rewards, next states,
# not_done flags; step counts from 1

def td3_bc_step(states, actions, rewards, next_states, not_done, step):
    # Target action: target actor plus clipped Gaussian noise
    # [tab. 4] [code: TD3_BC.py#L109-L117]
    noise = clip(
        policy_noise * random_normal(actions),
        -noise_clip,
        noise_clip,
    )
    next_actions = clip(
        target_actor(next_states) + noise,
        -max_action,
        max_action,
    )
    # Clipped double-Q target: smaller of the two target critics
    # [tab. 4] [code: TD3_BC.py#L119-L122]
    target_q = min(
        target_critic_1(next_states, next_actions),
        target_critic_2(next_states, next_actions),
    )
    target_q = rewards + not_done * discount * target_q
    # Critic update: both critics regress onto the same target
    # [code: TD3_BC.py#L125-L133]
    critic_loss = (
        mse(critic_1(states, actions), target_q)
        + mse(critic_2(states, actions), target_q)
    )
    update_critics(critic_loss)
    # Actor and target networks update only every policy_freq steps
    # [tab. 4] [code: TD3_BC.py#L136]
    if step % policy_freq != 0:
        return
    pi = actor(states)
    q_pi = critic_1(states, pi)
    # lambda = alpha / mean |Q| over the batch; not differentiated
    # [eq. 5] [code: TD3_BC.py#L141]
    lam = alpha / mean(abs(q_pi)).detach()
    # Maximize lam * Q minus the behaviour-cloning error
    # [eq. 3] [code: TD3_BC.py#L143]
    actor_loss = -lam * mean(q_pi) + mse(pi, actions)
    update_actor(actor_loss)
    # Polyak averaging of both target networks with rate tau
    # [tab. 4] [code: TD3_BC.py#L150-L155]
    soft_update(target_critics, critics, tau)
    soft_update(target_actor, actor, tau)

Equations, as printed in the paper:

\pi = \operatorname*{argmax}_\pi \mathbb{E}_{(s,a) \sim \mathcal{D}} \left[ \lambda Q(s,\pi(s)) - (\pi(s) - a)^2 \right] eq. 3

s_i = \frac{s_i - \mu_i}{\sigma_i + \epsilon}, \quad \epsilon = 10^{-3} eq. 4 §5

\lambda = \frac{\alpha}{\frac{1}{N} \sum_{(s_i,a_i)} |Q(s_i,a_i)|}, \quad \alpha = 2.5 eq. 5 §5

Hyperparameters

Everything not listed inherits TD3's defaults app. B tab. 4. The paper states that a single setting of α works across all datasets app. B.

Results

Author-reported. Setup: D4RL v0 MuJoCo locomotion tasks (HalfCheetah, Hopper, Walker2d) with five dataset types §6 tab. 2. Score is the average normalized return over the final 10 evaluations and 5 seeds tab. 2. CQL and Fisher-BRC were re-run with the authors' code, BRAC and AWAC are taken from published results, and BC is the authors' own implementation §6 tab. 2.

Headline totals over the 15 pairs tab. 2: TD3+BC 979.3 ± 33.4; Fisher-BRC 974.6 ± 108.3; CQL 764.3 ± 61.5; BC 595.3 ± 91.5; BRAC-p 284.1. AWAC has no total because three of its entries are missing tab. 2.

tab. 2

Training time (one GTX 1080, training only, no evaluation) tab. 3:

tab. 3. The authors describe the TD3+BC overhead as effectively zero and the total as less than half of the prior methods' cost §6 tab. 3.

Learning curves. TD3+BC learns at a similar speed and reaches a similar final performance to Fisher-BRC, without pre-training a generative model fig. 4.

Ablations (percent difference from the full method) fig. 5: the paper says both the BC term and the TD3 part are needed for strong performance. Removing state normalization has the smallest effect, but it still helps on many tasks. The paper says the full method never underperforms vanilla BC, including on expert data §6 fig. 5.

Sensitivity to α fig. 6: α = 2 and α = 3 perform about the same. Performance on some tasks drops as α moves toward 1 (more imitation) or 4 (more RL) fig. 6 app. B.

Additional datasets: 50 % random + 50 % expert tab. 7:

tab. 7 app. C.1. TD3+BC is comparable to Fisher-BRC here, and both are below CQL on Walker2d app. C.1.

AntMaze (state normalization not used here; other hyperparameters unchanged) tab. 8 app. C.1:

Decision Transformer comparison, D4RL v2 datasets tab. 9 fig. 8 app. C.3: DT results are taken from its paper (3 seeds). TD3+BC was re-run on v2 with no extra tuning. Over the nine pairs that DT reports, DT totals 672.6 ± 16.6 and TD3+BC totals 677.4 ± 45.6. Over all 15 pairs, TD3+BC totals 1013.2 ± 57.4 tab. 9. Wall-clock training on one million steps is faster for TD3+BC than DT trained for 100k steps, by more than a factor of two fig. 8.

Table 9 values (D4RL v2; "-" where DT has no result):

Paper vs code

Compared at code_commit 8791ad7 (the sfujim/TD3_BC repository, MIT License) LICENSE.

• λ uses different actions. The paper's λ averages |Q(s_i, a_i)| over dataset pairs, with dataset actions eq. 5 §5. The code computes the mean over the minibatch of |Q(s, π(s))|, using the policy's own actions TD3_BC.py. The paper also says the mean is estimated over mini-batches §5, which the code does. The paper's text and code therefore differ on which actions enter the mean.
• Behaviour-cloning term is a mean, not a sum. The paper writes the penalty as (π(s) − a)², without saying whether the squared norm is summed over action dimensions eq. 3. The code uses mse_loss, which averages over the batch and over action dimensions TD3_BC.py.
• Noise scaled by max action. The paper gives policy noise 0.2 and clipping (−0.5, 0.5) tab. 4. The code multiplies both by the maximum action value main.py. The two agree when the action range is [−1, 1].
• Target networks update only on delayed steps. The paper does not say when target networks update. The code updates them only inside the delayed actor-update branch TD3_BC.py.
• Hyperparameters match. Learning rates, batch size, γ, τ, policy frequency, layer widths and α = 2.5 in the code match the tables tab. 4 main.py TD3_BC.py TD3_BC.py.

Pitfalls

• λ depends on which actions enter the mean. The paper's equation uses the dataset's actions, but the code uses the policy's actions. An implementation that follows only the paper's equation computes a different λ, so its results may not match the reported ones. Choose the code's version, or test both.
• The relative weight of the BC term depends on action dimension. Averaging the squared error over action dimensions shrinks the BC term by a factor of the action dimension compared with summing it TD3_BC.py. The paper does not say which was used, so an implementation that sums will effectively weight imitation more strongly.
• Normalization statistics are fixed at training time. The code computes mean and standard deviation once on the whole dataset and applies them to states, next states and evaluation observations utils.py main.py main.py. Any deployed agent must reuse the same statistics. The constant ε is added to the standard deviation, not the variance eq. 4 utils.py.
• α was tuned, on Hopper. The paper tuned α on Hopper-medium-v0 and Hopper-expert-v0 app. B, which are also evaluation tasks. The paper says the seed used for tuning is excluded from the final results app. B. The α sweep fig. 6 is the only robustness evidence given.
• Random data is a weak spot. The paper calls the poor performance on random data a concern app. C.1. TD3+BC scores 10.2 on HalfCheetah-random, below CQL (21.7) and Fisher-BRC (32.2) tab. 2. On the 50/50 random + expert mix it falls far below CQL on Walker2d tab. 7, and AntMaze large tasks score near zero tab. 8.
• Single-checkpoint results are unstable. The paper reports large variation across episodes at the last evaluation fig. 2 and across the last 10 evaluations fig. 3, and does not solve it. Headline scores are averages over 10 evaluations and 5 seeds tab. 2. Compare methods on the same schedule, not on the best checkpoint.
• The printed Fisher-BRC total does not match its rows. Summing the 15 Fisher-BRC scores in tab. 2 gives 975.2, not the printed 974.6. The paper does not say how it rounded. Use the per-row values if the difference matters.
• Training-time breakdown does not add up. In tab. 3, the implementation and algorithmic columns do not sum to the printed totals, and the caption does not say what makes up the rest. Treat the comparison as indicative. The runtime was measured on one GPU type, and Fisher-BRC was re-implemented in PyTorch for the runtime experiment only §6.
• Exploration noise is not used. The offline training loop does not add exploration noise. The code accepts an expl_noise argument but never uses it main.py main.py.
• No reward scaling. The code uses D4RL rewards as stored main.py utils.py. The paper says nothing about reward scaling.
• Pinning the code. The paper pins the TD3 code (commit 6a9f7610) and the CQL and Fisher-BRC code, but not the TD3+BC repository app. B. The code_commit above is the HEAD at the time of compilation.
• Figure results are read from text and captions. Figures 1 to 8 are plots without printed values. This card describes them only as the text and captions do, and gives numbers only from the printed tables.

Lineage

• Builds on: TD3 (cited as fujimoto2018addressing in the paper), which supplies the base algorithm and all its hyperparameters except α §5 app. B. The paper does not state an arXiv id, so none is linked here.
• Compared against: CQL (kumar2020conservative) §6 tab. 2, Fisher-BRC (kostrikov2021offline) §6 tab. 2, BRAC (wu2019behavior) and AWAC (nair2020accelerating) §6 tab. 2, and the Decision Transformer (chen2021decision) app. C.3. The paper also names a SAC+BC baseline from nair2020accelerating §2.
• Related approach: Table 1 lists the implementation and algorithmic changes that CQL and Fisher-BRC make to SAC tab. 1 §4.
• Superseded by: not stated in the paper.