Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inference

In plain words

Imagine an A/B test that runs for a month, but the business cares about what happens over years. Short experiments can say little about long-run effects, because the people in the test drift into situations that the test rarely showed. This paper asks how to estimate the long-run value of a policy from data gathered by a short experiment, treating each user's life as a sequence of states and actions.

The standard tool for this is a method called double reinforcement learning. It works well in principle, but it becomes unstable when the policy you want to evaluate spends time in situations the experiment barely covered. The paper's idea is to assume a simple shape for the value function that the method learns, such as a linear form in some features. If that assumption is right, the estimate becomes more precise. If it is wrong, the paper still defines the target, as the closest fit to the true equations under a chosen weighting. The estimator is built to remain valid even then, and the paper shows that choosing the weighting well recovers the best achievable precision when the assumption holds.

The practical message is that partial structure on the value function can trade a little bias or model risk for much more stable inference, and that the weighting is a real design choice rather than a detail.

TL;DR

Works with a working semiparametric restriction on the infinite-horizon Q-function. Its target is the weighted Bellman projection, which equals the original functional when the restriction is correct. The autoDRL estimator uses two Bellman-image nuisances and a projection-residual correction, both fitted by minimax criteria without estimating the Bellman operator. Under correct specification, weighting by the inverse conditional variance of the Bellman residual attains the restricted-model efficiency bound §4.1 §5.3. Evidence is synthetic simulation only, author-reported.

Problem

Long-term effects from short experiments need the policy's long-run state-action occupancy. Fully nonparametric double reinforcement learning is efficient only if that occupancy ratio exists with finite variance; the efficiency bound scales with its square, so weak intertemporal overlap inflates variance and undercovers §3. Randomisation guarantees overlap for the assigned action, not for future states visited under sustained use §1 §3. Linear MDP structure can help, but it requires correct structure on both rewards and transitions §1. This paper instead restricts the Q-function directly, so that only the Q-function's Bellman image is constrained §1.

Method

Setting. Observed data are i.i.d. one-step transitions Z=(S,A,Y,S') with X=(S,A), from a time-homogeneous MDP §2.1. The target is a policy \pi, with discount \gamma\in[0,1), and the estimand is \psi_0 = E_0\{m(X,q_0)\} for a linear map m §2.2. The policy value is the case m(x,q)=\int q(s,a')\pi(a'\mid s)\,\nu(da') §2.2. The A/B-test example takes m(\tilde x,q)=q(s,1)-q(s,0) on an augmented state (s,d) with d the arm label §2.2, example A/B test.

Bellman equation. The Q-function solves q_0(X) = E_0\!\left[ Y+\gamma V^\pi(q_0)(S') \mid X \right] \qquad [\text{eq. 1}] with V^\pi(q)(s') := \int q(s',a')\pi(a'\mid s')\,\nu(da'). Written as an operator equation, \mathcal T_P(q_P)=r_P, where \mathcal T_P(h)(X) = h(X)-\gamma E_P[V^\pi(h)(S')\mid X] and r_P(X)=E_P(Y\mid X) eq. 2 §2.2.

Step 1: Projection target §4.1. Restrict q to a linear working class H (possibly infinite-dimensional), with L^2(P_{0,X})-closure \overline H. Let w be a weight function. The weighted Bellman projection is the minimiser of weighted Bellman-residual error: q_{P,H,w} = \arg\min_{q\in\overline H} E_P\!\left[ w(X)\{Y-\mathcal T_P(q)(X)\}^2 \right] \qquad [\text{eq. 4}] The projection parameter is \Psi_{H,w}(P) = E_P\{m(X, q_{P,H,w})\} §4.1. The default is w\equiv 1 §4.1. Under correct specification the target does not depend on w; under misspecification it does §4.1.

Step 2: Riesz representer §4.2. Under the boundedness conditions, a unique \alpha_{0,H,w}\in\overline H exists such that, for every q\in\overline H, E_0\{m(X,q)\} = E_0\!\left[ w(X)\,\mathcal T_0(\alpha_{0,H,w})(X)\,\mathcal T_0(q)(X) \right] \qquad [\text{eq. 5}] Two Bellman-image nuisances are then defined §4.2:

• \tau_{0,H,w} := \mathcal T_0(\alpha_{0,H,w}), which gives the reward-weighting form \Psi_{H,w}(P_0) = E_0\{w\,\tau_{0,H,w}\,Y\} §4.2;
• e_{0,H,w} := \mathcal T_0(q_{0,H,w}) - r_0, the projection residual §4.2.

For the unrestricted policy-value problem, w\,\tau_{0,H,w} reduces to the occupancy ratio d_0 §4.2 §5.2.

Step 3: autoDRL estimator §4.2. Given fitted nuisances (q_n, \alpha_n, \tau_n, e_n), the estimator is the sample average of the terms below. The first correction is the usual DRL Bellman-residual correction. The second corrects the extra first-order bias from misspecification:

# autoDRL estimate of the projection target  [§4.2]
# Input:  transitions (s_i, a_i, y_i, s'_i), i = 1..n
#         nuisances q_n, alpha_n, tau_n, e_n (fitted on independent data)
#         weight function w, discount gamma
# Output: scalar estimate psi

def autodrl_estimate(transitions, q_n, alpha_n, tau_n, e_n, w, gamma):
    # Plug-in term: m evaluated at the fitted Q-function  [§4.2]
    plug_in = mean(m(x_i, q_n) for each transition i)
    # Bellman-residual correction, reward weight w * tau_n  [§4.2]
    residual_q = y_i + gamma * V(q_n)(s'_i) - q_n(x_i)
    correction_q = mean(w(x_i) * tau_n(x_i) * residual_q)
    # Projection-residual correction for misspecification  [§4.2]
    residual_alpha = alpha_n(x_i) - gamma * V(alpha_n)(s'_i) - tau_n(x_i)
    correction_e = mean(w(x_i) * e_n(x_i) * residual_alpha)
    # Sum: plug-in + correction_q - correction_e  [§4.2]
    return plug_in + correction_q - correction_e

Under correct specification, e_0=0, so the last correction may be set to zero §4.2. The paper recommends cross-fitting §4.2.

Step 4: Nuisance fitting by minimax criteria §4.3. The paper avoids estimating the Bellman operator or the reward regression. Instead, two saddle-point problems recover all four nuisances from observed one-step residuals §4.3: \min_{e\in L^2_w(P_{0,X})} \sup_{q\in H} E_0\!\left[ \tfrac12 w(X)e(X)^2 + w(X)e(X)\{Y-q(X)+\gamma V^\pi(q)(S')\} \right] \min_{\tau\in L^2_w(P_{0,X})} \sup_{\alpha\in H} E_0\!\left[ \tfrac12 w(X)\tau(X)^2 - w(X)\tau(X)\{\alpha(X)-\gamma V^\pi(\alpha)(S')\} + m(X,\alpha) \right] The first recovers (e_0, q_0) and the second (\tau_0, \alpha_0) §4.3. Both components enter the final estimator §4.3. For finite-dimensional H, profiling the inner supremum gives constrained quadratic programmes with empirical Bellman moment constraints. For high-dimensional linear or RKHS classes, a quadratic penalty on the inner variable yields ridge-type or kernel least-squares systems §4.3.

Step 5: Optimal weight §5.3. Under correct specification, all weights give the same target, but different efficiency. The best weight equals one over the Bellman residual's conditional variance: w_{\rm opt}(X) = \{\sigma_0^2(X)\}^{-1}, \quad \sigma_0^2(X) = E_0\!\left[ \{Y+\gamma V^\pi(q_0)(S')-q_0(X)\}^2 \mid X \right] For \gamma=0 this is the usual heteroskedastic least-squares weight §5.3. The weight must be estimated in practice; the simulations estimate the conditional variance and use inverse-variance weights app. simulation design.

Theory in brief.

• The estimand is pathwise differentiable with an efficient influence function \varphi^*_{0,H,w} that adds a projection-residual term to the usual DRL form §5.1.
• Under sample splitting, the product-rate conditions in §5.2 and the bounded-nuisance condition, the estimator is asymptotically linear with influence function \varphi^*_{0,H,w} §5.2.
• Projection bias is second order: the target equals the original \Psi(P_0) when either q_0\in\overline H or the unrestricted Riesz representer lies in \overline H §5.2.
• Under correct specification and the non-degeneracy condition, the optimally weighted estimator attains the restricted-model bound for \Psi §5.3.

Hyperparameters

Only simulation settings are stated. No real-data settings, learning rates, optimisers or basis sizes for the nuisance classes are given in the main text.

Results

Author-reported. Setup: synthetic finite-state A/B test with 81 states, \gamma=0.9, 200 replications per setting app. simulation design. Baselines are the nonparametric DRL estimator with the exact (oracle) occupancy ratio and tabular Q-fits, which is not deployable app. simulation design. The paper uses no real dataset §7, data availability.

Paper vs code

No official code. The paper's TeX source gives no repository link, and the arXiv record has no comments with one [raw/src main TeX]. Searches: the GitHub repository search for "semiparametric double reinforcement learning" returned 0 results; for "autoDRL" it returned two repositories by unrelated owners (bach137here/AutoDRL, Anakonkai01/AutoDRL), which are not by the authors and are not used; for "laan double reinforcement learning" it returned 0 results. No project page is linked from the paper.

The appendix's Code section gives a Python listing of fitted Q-calibration with isotonic regression through monotone-constrained XGBoost app. Code. That is a different method from this card's main estimator (the Bellman-calibration variant in the appendix), and it is printed in the paper, not released as a repository. It is not used as code.

Pitfalls

• Weight changes the target under misspecification. Under misspecification, w defines the projection norm, so \Psi_{H,w}(P_0) differs across weights §4.1. Compare variances only within one weight. The paper omits optimal-weight estimators from the misspecification experiment for this reason §6.
• Optimal weight needs an estimated variance. w_{\rm opt}=\sigma_0^{-2} depends on q_0 through the conditional variance of the Bellman residual §5.3. The main text does not give the estimator for \sigma^2, and simulations only say "estimate the conditional Bellman-residual variance" app. simulation design. Expect estimation error to affect efficiency; the paper reports no sensitivity analysis.
• Keep both correction terms. Under misspecification the projection-residual correction (the e term) is needed to remove first-order bias §4.2. Dropping it is valid only when q_0\in\overline H. The nuisance \alpha and its outputs \tau must be fitted together through the same saddle problem; the paper notes that retaining only one component is not what it does §4.3.
• Nuisance approximation dominates misspecified coverage. With the same target and estimator, changing the Bellman-image nuisance from an additive basis to state indicators moves coverage at \beta=1.2 from 0.880 to 0.950 tab. S1. The choice of nuisance class is a first-order decision, not a tuning detail.
• Product-rate conditions are conditions on the nuisances, not on the learners. Asymptotic linearity needs o_p(n^{-1/2}) for both products in §5.2. The paper gives no rate for any learner, and no regularisation strength. Treat the theorem as a checklist, not a guarantee.
• Cross-fitting is recommended but not analysed. The theory is a single-split analysis §4.2. Number of folds is not stated. Use cross-fitting in practice only with that caveat.
• Bounded outcomes and weights are assumed. The conditions assume |Y|\le C almost surely and bounded weights §4.1, C1. Heavy-tailed rewards need separate checks.
• Weak overlap is handled by restriction, not by estimating the ratio. The estimator avoids direct occupancy-ratio estimation §4.3. The paper's deployable DualDICE stress test is poor app. fig. S1. Do not conclude that a plug-in ratio works in practice.
• Simulation only. All numbers are from a synthetic finite-state design with a known transition matrix app. simulation design; no real A/B-test data is analysed.
• Scope of this card. Sections on sieve or adaptive model selection, Bellman calibration, and data fusion in the supplementary appendix are not condensed here.

Lineage

• Builds on: the time-homogeneous MDP approach to long-term effects of Tran et al. (2023), which defines the A/B-test estimand §1 §2.2; double reinforcement learning for nonparametric policy values §1 §3. Named by reference key only; no arXiv ids are given in the paper text.
• Compared against: nonparametric DRL with the oracle occupancy ratio and tabular Q-fits §6; linear-MDP structure on rewards and transitions §1; DualDICE as a deployable ratio estimator app. simulation design.
• Superseded by: not stated.