Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds

In plain words

Policy gradient methods improve a policy by estimating, from sampled episodes, which direction in parameter space raises the expected reward. Plain estimates are noisy, so they need a lot of episodes. Variance-reduced methods reuse estimates from the previous policy to steady the new one, which needs fewer episodes. Until now, the fast versions reweighted old episodes to fit the new policy, and those weights could explode unless one assumed they never do.

This paper avoids that assumption. It mixes episodes drawn from the old and the new policy, so each weight is capped by construction. The result is a method, called DEF-PG, with the same fast rate as the best earlier methods and without the extra assumption. The paper also proves lower bounds in a simplified black-box model, which suggest that the two rates it reaches cannot be beaten at that level. These bounds do not yet apply to ordinary reinforcement learning environments.

TL;DR

DEF-PG is a PAGE-style variance-reduced policy gradient whose correction term uses defensive importance sampling, a mixture of the old and new policy. Its importance weights are bounded without any assumption on the variance of the ordinary weights §3, §4. Under bounded rewards and bounded, mean-square continuous policy scores, it needs O(eps^-3) stopped trajectories to find an eps-stationary point §4, Thm. 6. Those score conditions exclude Gaussian policies in general §4, Rem. 2, and the rates are oracle-level results.

Problem

Plain REINFORCE/GPOMDP needs O(eps^-4) samples to reach a point whose gradient norm is at most eps §1, §4 Prop. 3. Variance-reduced methods such as SVRPG, SRVR-PG, STORM-PG and PAGE-PG reach O(eps^-3) §1. Their analyses, however, reuse trajectories from one policy to estimate the gradient of another, which requires importance weights. Those analyses assume the variance of the ordinary importance weights is uniformly bounded by a constant W for all parameter pairs §2, Assumption 0. The paper points out that this can fail even for simple Gaussian policies with a constant mean §2, Example 1.

The paper asks whether a first-order method can reach O(eps^-3) without that assumption and without second-order information §1. The stated target is an eps-first-order stationary point, meaning E||grad J(theta_OUT)|| <= eps §1.

Method

Setting. The problem is a discounted MDP with parametric policies pi_theta, and J(theta) is the expected discounted return §2. Each episode is stopped after every transition with probability 1 - gamma. The stopped return then has the same expectation as J, with no truncation bias. Its length H has E[H] = 1/(1-gamma) §2, Rem. 1.

Estimator. The GPOMDP estimator of a trajectory tau is the sum over steps h of the reward R_{h+1} times the sum of score terms up to h §2, eq. 1. The paper uses it without a baseline §2.

Off-policy weight. For a trajectory from pi_theta', the importance weight for pi_theta is the product of per-step policy ratios over the stopped length. Geometric stopping cancels out [§2, footnote 1 and the unnumbered display after eq. 1]. Its variance equals a chi-square divergence, which can be infinite §2.

Defensive mixture. For parameters theta_1, theta_2 and alpha in (0, 1), the sampling distribution is q = alpha p(. | theta_1) + (1 - alpha) p(. | theta_2). A trajectory from q comes from pi_theta_1 with probability alpha and from pi_theta_2 otherwise §3. The two defensive weights are the ratios of the policy density to q §3, eq. 3 and eq. 4. For any alpha in (0, 1], the variance of p/q is at most (1 - alpha)/alpha §3.

Oracle-level algorithm. Coupled-PAGE alg. 1 refreshes the estimate with N on-policy gradient samples with probability p. Otherwise it applies a recursive update that uses B coupled pairs at (theta_{t+1}, theta_t). The output is one iterate chosen uniformly from theta_0 to theta_{T-1} alg. 1.

DEF-PG. The paper instantiates Coupled-PAGE with GPOMDP and defensive sampling alg. 2. The refresh step draws N trajectories at theta_{t+1} and averages their GPOMDP estimates alg. 2. The recursive step draws B trajectories, each from the mixture q of theta_{t+1} and theta_t, and adds the difference of the two weighted GPOMDP terms, averaged over the B trajectories alg. 2. The pseudocode below follows alg. 2 with the defensive weights of eq. 3 and eq. 4. The paper does not show the helper functions, so they are written here in the paper's terms.

# GPOMDP estimate for one stopped trajectory  [§2] [eq. 1]
def gpomdp(theta, tau):
    grad = zeros_like(theta)
    score_sum = zeros_like(theta)
    for h in range(len(tau.rewards)):
        # Add the score of the action taken at step h  [eq. 1]
        score_sum = score_sum + grad_log_policy(theta, tau, h)
        # Weight the running score sum by the reward R_{h+1}  [eq. 1]
        grad = grad + score_sum * tau.rewards[h]
    return grad

# DEF-PG  [alg. 2]
# Inputs: theta_0, batch N, batch B, step eta, refresh p, alpha, T

def def_pg(theta_0, N, B, eta, p, alpha, T):
    # Initial large batch at theta_0  [alg. 2, lines 2-4]
    trajectories = sample_trajectories(theta_0, N)
    v = mean_gpomdp(theta_0, trajectories)
    theta = theta_0
    iterates = []
    for t in range(T):
        # Record the iterate, then step along the estimate  [alg. 2, line 5]
        iterates.append(theta)
        theta_next = theta + eta * v
        if bernoulli(p) == 1:
            # Large-batch refresh at the new parameters  [alg. 2, lines 7-9]
            trajectories = sample_trajectories(theta_next, N)
            v = mean_gpomdp(theta_next, trajectories)
        else:
            # Defensive recursive update  [alg. 2, lines 11-12]
            v = v + defensive_correction(theta_next, theta, B, alpha)
        theta = theta_next
    # Return one iterate chosen uniformly at random  [alg. 2]
    return uniform_choice(iterates)

# Defensive correction term  [alg. 2] [eq. 3] [eq. 4]
# Mixture: theta_next with prob. alpha, theta with prob. 1 - alpha

def defensive_correction(theta_next, theta, B, alpha):
    total = zeros_like(theta)
    for i in range(B):
        # Draw from the defensive mixture  [alg. 2, lines 13-17]
        if bernoulli(alpha) == 1:
            tau = sample_trajectory(theta_next)
        else:
            tau = sample_trajectory(theta)
        # Weight for the new parameters  [eq. 3]
        ratio_old_new = likelihood_ratio(theta, theta_next, tau)
        w_new = 1 / (alpha + (1 - alpha) * ratio_old_new)
        # Weight for the previous parameters  [eq. 4]
        ratio_new_old = 1 / ratio_old_new
        w_old = 1 / (alpha * ratio_new_old + (1 - alpha))
        # Difference of the two weighted GPOMDP terms  [alg. 2, line 19]
        total = total + (
            w_new * gpomdp(theta_next, tau)
            - w_old * gpomdp(theta, tau)
        )
    return total / B

The likelihood ratio is p(tau | theta) / p(tau | theta_next). Geometric stopping cancels from it, so it is the product of per-step policy ratios over the trajectory §2.

Key lemma. Lemma 4 §4 bounds the variance of each defensive weight by a constant times ||theta_1 - theta_2||^2, using the score bound C_1 and the moments M_k of the stopped length H. Under bounded rewards it also bounds the mean squared difference between the two weighted GPOMDP terms by C_alpha times ||theta_1 - theta_2||^2 §4, Lemma 4. Each weighted term is unbiased for the gradient at its own parameter §4, Lemma 4.

Rate. Lemma 5 §4 gives the complexity of Coupled-PAGE for any N, B, p, eta and T that satisfy its conditions. Theorem 6 §4 then applies it to DEF-PG with the constants below. With B = 1, the expected number of oracle calls is O(eps^-3) §4, Lemma 5 and Thm. 6.

Lower bounds. Definitions 1 to 3 §5 describe a black-box model. The learner queries policy parameters and receives reward and score records instead of states and actions. It can also query pairs of parameters with shared randomness. Theorem 9 §5 shows Omega(eps^-4) for one-policy feedback with bounded variance. Theorem 10 §5 shows Omega(eps^-3) for coupled feedback. Both use a Gaussian policy class embedding §5, Prop. 7 and Prop. 8.

Hyperparameters

The paper has no experiments in this version. The experiments section and the numerical remarks in the conclusion are commented out in the source §7. The values below are the theoretical settings from the analysis, not tuned values.

Results

Author-reported. This version has no empirical results. All results are oracle-level sample-complexity statements, and the table below compares them with the rates they are measured against §1, §4, §5, tab. 1.

The GPOMDP route gives O((1 - gamma)^-6 eps^-4) trajectories when L = O((1 - gamma)^-2) and Delta <= 2 R_max / (1 - gamma) §4, after Prop. 3. No (1 - gamma) dependence is stated for DEF-PG §4.

Paper vs code

No official code. The paper and arXiv metadata link no code, and no project page is listed [raw/src/2610.03165/main.tex]. A GitHub search for "DEF-PG", "defensive policy gradient", "variance reduced policy gradient importance sampling" and "defensive importance sampling policy" found one repository, paczyg/defpg (MIT, copyright 2023, no README, no link to the paper). Its owner is not verifiably an author, and it predates the paper's arXiv date, so it is not used. Searching the author name "Paczolay" returned no repositories.

Pitfalls

• Score assumption excludes Gaussians. Assumption 6(i) bounds the score norm uniformly, and the paper says it "generally fails for Gaussian policies" §4, Rem. 2. The O(eps^-3) guarantee therefore does not cover the common Gaussian policy for continuous actions. Only the oracle-level result covers other policy classes §4, Thm. 6.
• Bounded rewards are needed for the concrete result. Assumption 4 is only used in the GPOMDP and DEF-PG instantiations. The oracle-level upper bound does not need it §4.
• Variance depends on the step size. The defensive variance bounds in Lemma 4 scale with ||theta_{t+1} - theta_t||^2. The step-size rule in Lemma 5 therefore depends on L_pair, and L_pair depends on M_4 and M_5. These moments are not given in closed form, so the step size cannot be computed from the paper alone §4, Lemma 4 and Lemma 5.
• Cost per recursive step. Each of the B trajectories is scored at both theta_{t+1} and theta_t, and both policy densities are needed for the weights alg. 2. The weights are not computed from stored returns alone.
• Geometric stopping is part of the estimator. The analysis uses trajectories that end after each transition with probability 1 - gamma §2, Rem. 1. A fixed-horizon implementation is a different estimator, and the paper's bounds do not cover it. If transitions are counted instead of trajectories, each call costs (1 - gamma)^-1 transitions in expectation §2, Rem. 1.
• Output is a random iterate. The theorem bounds the expected gradient norm of an iterate drawn uniformly from theta_0 to theta_{T-1} alg. 2, §4. The paper does not analyse the last iterate.
• Alpha choice. alpha = 1/2 minimises the stated bound, which depends on 1 / min{alpha, 1 - alpha} §4, Cor. 1 and Rem. 3. The bound is not evidence about practical performance, since no experiments are reported.
• Lower bounds are oracle-level. Theorems 9 and 10 use a black-box model in which the reward may depend on the queried parameter §5, Def. 1. They do not establish lower bounds for the standard MDP protocol §5, §7. The paper states that this remains open §7.
• Attribution to earlier work. The GPOMDP variance bound is imported from Papini et al. (2022), Lemma 29, and not re-derived here §4, Prop. 2. The related-work footnote also questions a proof in Yang et al. (2022) §6. The paper's claim about that proof is the authors' view, not an established result.
• Version. This card is based on arXiv v1 [raw/arxiv/2610.03165.json]. A later version may change constants and numbering.

Lineage

• Builds on: PAGE and PAGE-PG (Li et al. 2021; Gargiani et al. 2022), which supply the recursive estimator §3, eq. 2. Defensive importance sampling (Owen and Zhou 2000) supplies the bounded weights §3. SVRPG (Papini et al. 2018) introduced the policy-gradient variance-reduction setting §1.
• Compared against: SRVR-PG (Xu et al. 2020), STORM-PG (Yuan et al. 2020), TSIVR-PG (Zhang et al. 2021), HAPG (Shen et al. 2019), SHARP and N-PG-IGT (Fatkhullin et al. 2023) §1, §6.
• Superseded by: not stated.