Importance weighting

Overview

The problem is to estimate an average under a distribution you cannot sample from directly, to estimate a total from a small subset of its items, or to reuse old data for a new gradient. The shared idea is a weight per sample. Each sample's loss or value is multiplied by a ratio, so the weighted average matches the target quantity in expectation. The ratio is a density ratio when the distributions differ 2606.04665 eq. 5, an inverse inclusion probability when items were sampled non-uniformly 2410.01917 §3.2, a policy ratio when the data came from an older policy 2607.07508 eq. 2, or a ratio against a mixture of two policies 2610.03165 §3. One variant uses heuristic weights with no target distribution behind them 2606.00959 §2.3. It is listed here because it uses the same mechanism, but it is not an unbiased estimator.

The variants below differ in what the ratio is for. The variance of the weights is the common weak point. The variance bound for density-ratio weights grows with the Rényi divergence between the two distributions 2606.04665 eq. 6, and the papers handle it differently: a control variate, a better sampling distribution, a mixture, masking or clipping extreme ratios, or shaping the ratio.

Variants

Density-ratio weights for validation under covariate shift

When labels exist only in the source domain, the source validation loss, weighted by the density ratio q/p, is unbiased for the target risk 2606.04665 eq. 5. Deep Embedded Validation (DEV) estimates the ratio in feature space instead of input space, using a domain classifier trained on source-training and unlabelled target features 2606.04665 §4.2 2606.04665 eq. 7. It then adds a control variate with a fitted coefficient to reduce variance 2606.04665 eq. 9 [arxiv:2606.04665 eq. 10]. The authors report a selected model at 72.42% mean accuracy on VisDA against 72.55% for selection with target labels 2606.04665 tab. 2. Card: Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation.

Inverse-inclusion weights for sampled Shapley values

Exact Shapley values need one model call per feature subset. Leverage SHAP draws subsets with a sampling distribution based on leverage scores, which have a closed form here 2410.01917 §3.1. Each sampled subset is paired with its complement 2410.01917 §3.2. Its regression row is weighted by the kernel weight divided by the realised inclusion probability 2410.01917 alg. 1. The authors report lower error than Optimized Kernel SHAP at most budgets. Their error ratio lies between about 0.05 and 0.75 where the budget is below 2^n 2410.01917 tab. 2. Card: Provably Accurate Shapley Value Estimation via Leverage Score Sampling.

Policy ratios with masking for off-policy RL

When an agent trains on answers written by a slightly older model, the per-token ratio of the current policy to the rollout policy corrects the mismatch 2607.07508 eq. 2. Single-rollout Asynchronous Optimization (SAO) takes this ratio from the rollout engine's logged log-probabilities, so no copy of the old policy is kept 2607.07508 §3.1. It masks tokens whose ratio falls outside an open interval rather than clipping them 2607.07508 eq. 3. The authors report 97.3 on AIME2025 for Qwen3-30B-A3B, against 84.2 for GRPO with Python tools [arxiv:2607.07508 tab. 1]. Card: Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning.

Replayed correct rollouts with a shaped ratio

ExGRPO stores the correct answers from earlier batches, bucketed by how often each question was solved, and puts one stored answer into about half of each new batch 2510.02245 §4.1 2510.02245 §4.2. Questions that were always solved are retired and never replayed. For the replayed answer, the ratio of the current policy to the policy that wrote it is not used directly. It is passed through f(w) = w/(w+0.1), which amplifies low-probability tokens and damps high-probability ones 2510.02245 §4.2. Shaping replaces the plain ratio, so the replayed term is no longer the plain importance-weighted term; the main text states no unbiasedness result for it. The paper's mixed-phase objective keeps the clip 2510.02245 eq. 4, but the released code does not clip there (see Contradictions). On Qwen2.5-Math-7B the authors report 48.3 average over six math benchmarks against 45.3 for on-policy RLVR, and 41.2 without the shaping 2510.02245 tab. 1 [arxiv:2510.02245 app. E.4]. The step time rose by 19.0% on 8 GPUs 2510.02245 app. E.5. Card: ExGRPO: Learning to Reason from Experience.

Defensive mixture sampling for variance-reduced policy gradient

Variance-reduced policy gradient methods reuse trajectories from the previous policy, which needs ratio weights. Their analyses usually assume those weights have bounded variance. DEF-PG draws the correction trajectories from the mixture q = αp(·|θ_1) + (1−α)p(·|θ_2) of the new and old parameters. For any α in (0, 1], the variance of each ratio p/q is at most (1−α)/α, with no separate variance assumption 2610.03165 §3. Under bounded rewards and mean-square continuous policy scores, the paper's bound is O(ε⁻³) stopped trajectories to reach an ε-stationary point 2610.03165 Thm. 6. The score condition generally fails for Gaussian policies [arxiv:2610.03165 §4 Rem. 2]. The paper reports no experiments, and its step size needs constants it does not estimate 2610.03165 §4 Lemma 5. Card: Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds.

Heuristic sample weights for VLM fine-tuning

This variant does not estimate a target quantity. Each training sample gets a weight of 3.0 if it is in a "synergy gap" set, 0.5 if it is in a "shortcut" set, and 1.0 otherwise. The two sets come from ranking scores built from answer entropies and local information terms 2606.00959 alg. 1 2606.00959 eq. 8 2606.00959 eq. 12. The paper says these local scores are heuristics, used only as ranking signals 2606.00959 §2.3 2606.00959 app. A.3. On Qwen2.5-VL-7B, LoRA with these weights reaches +2.3 pp on MMStar over uniform weights, with a 95% paired-bootstrap interval of [1.4, 3.2] 2606.00959 §5.1. On Gemma3-12B the MMStar score falls from 56.2 to 54.3 with the same weights 2606.00959 tab. 3. Card: Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition.

When to use which

Implementation sketch

The common core is a weighted mean with an optional control variate. The control variate uses the weights' known mean of 1 under the sampling distribution, which holds only for the exact ratio 2606.04665 eq. 8. This sketch is not from any one paper; the steps are cited to their source.

# Weighted risk under a shift, with a control variate  [eq. 5] [eq. 9]
# Input:  per-sample losses (n,), per-sample weights (n,) >= 0
# Output: scalar estimate of the target risk

def weighted_risk(losses, weights):
    # Weight each loss by the estimated density ratio  [eq. 5] [eq. 7]
    weighted = weights * losses
    # Coefficient that minimises the variance of the estimate  [eq. 10]
    eta = -cov(weighted, weights) / var(weights)
    # Control variate: the weights have mean 1 under the sampling
    # distribution when the ratio is exact  [eq. 8] [eq. 9]
    return mean(weighted) + eta * (mean(weights) - 1)

For sampled estimators such as Leverage SHAP the weight is the kernel weight over the inclusion probability instead of a density ratio [arxiv:2410.01917 alg. 1]. For RL the weight is the token ratio, and tokens whose ratio falls outside the interval are zeroed 2607.07508 eq. 1 [arxiv:2607.07508 eq. 3]. ExGRPO applies w/(w+0.1) to the ratio of its replayed answer instead of a plain ratio 2510.02245 §4.2. For the defensive mixture, the weights for each draw come from the two parameter sets and the mixture coefficient 2610.03165 eq. 3 2610.03165 eq. 4.

Contradictions

• Headline average for Leverage SHAP. The paper's "50%" figure is the mean of the 38 Table 2 entries below 2^n 2410.01917 tab. 2. Averaging all 48 entries, with the ten ratios of 1 included, gives about 0.61, or about 39% lower error (average computed here from the table). Quote the budget range with the figure.
• Density-ratio classifier, within DEV. The paper describes a two-layer logistic regression 2606.04665 §4.3. The released code uses a multilayer perceptron with its regularisation picked on held-out data, a difference the paper card records. Whether this moves the reported numbers is not known.
• Theorem and released sampler, within Leverage SHAP. The guarantee covers an independent sampler, while the released estimator uses fixed per-size counts 2410.01917 §3.2. The paper's experiments use the released version.
• Clip interval, within SAO. The paper's clip values are named ε_low and ε_high, but the interval is written with other symbols [arxiv:2607.07508 §4.1] 2607.07508 §3.1. Reading them as lower and upper bounds gives (0.7, 6.0) for math, a reading the paper does not confirm.
• Clipping in the replay phase, within ExGRPO. The mixed-phase objective keeps the clip 2510.02245 eq. 4, and the appendix theory assumes it is removed 2510.02245 app. D. The released code removes it in the mixed phase. The paper card records this as a paper-versus-code difference.
• Bucket weight scale, within ExGRPO. The paper's Gaussian is on accuracy in [0, 1] 2510.02245 alg. 1 2510.02245 §4.1. The code applies it to success counts, which makes the question sampler much narrower. The paper does not say which scale produced the reported numbers.

Papers

• Provably Accurate Shapley Value Estimation via Leverage Score Sampling: leverage-score sampling for Kernel SHAP with inverse-inclusion weights and a regression guarantee on the objective.
• ExGRPO: Learning to Reason from Experience: replay of stored correct rollouts with a shaped importance ratio and no clip in the mixed phase, for RL with verifiable rewards.
• Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition: PID-based sample weights for VLM fine-tuning; the weights are heuristic, not density ratios.
• Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation: DEV, density-ratio weights in feature space with a control variate, for selecting unsupervised domain adaptation models.
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning: token-level policy ratios from logged log-probabilities, with interval masking, for off-policy agentic RL.
• Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds: DEF-PG, defensive mixture sampling for a variance-reduced policy gradient, with oracle-level rates.