---
kind: concept
slug: importance-weighting
title: Importance weighting
summary: >-
  Reweighting samples so their average estimates a quantity under another distribution, or so a sample stands in for the items it represents. Covers density-ratio validation under covariate shift, inverse-inclusion Shapley weights, off-policy RL ratios and replay, defensive mixture sampling for policy gradients, and heuristic weights for VLM fine-tuning. Author-reported results only.
tags: [sampling, reweighting, distribution-shift, off-policy, variance-reduction]
papers: ["2410.01917", "2510.02245", "2606.00959", "2606.04665", "2607.07508", "2610.03165"]
related: [feature-attribution, active-learning, domain-adaptation, policy-gradient, actor-critic, experience-replay, reinforcement-learning-verifiable-rewards, multimodal-language-model]
status: draft
updated_at: 2026-10-19
---

## Overview

The problem is to estimate an average under a distribution you cannot sample
from directly, to estimate a total from a small subset of its items, or to
reuse old data for a new gradient. The shared idea is a weight per sample.
Each sample's loss or value is multiplied by a ratio, so the weighted average
matches the target quantity in expectation. The ratio is a density ratio when
the distributions differ [arxiv:2606.04665 eq. 5], an inverse inclusion
probability when items were sampled non-uniformly [arxiv:2410.01917 §3.2], a
policy ratio when the data came from an older policy [arxiv:2607.07508 eq. 2],
or a ratio against a mixture of two policies [arxiv:2610.03165 §3]. One variant
uses heuristic weights with no target distribution behind them
[arxiv:2606.00959 §2.3]. It is listed here because it uses the same mechanism,
but it is not an unbiased estimator.

The variants below differ in what the ratio is for. The variance of the
weights is the common weak point. The variance bound for density-ratio weights
grows with the Rényi divergence between the two distributions
[arxiv:2606.04665 eq. 6], and the papers handle it differently: a control
variate, a better sampling distribution, a mixture, masking or clipping extreme
ratios, or shaping the ratio.

## Variants

### Density-ratio weights for validation under covariate shift

When labels exist only in the source domain, the source validation loss,
weighted by the density ratio q/p, is unbiased for the target risk
[arxiv:2606.04665 eq. 5]. Deep Embedded Validation (DEV) estimates the ratio in
feature space instead of input space, using a domain classifier trained on
source-training and unlabelled target features [arxiv:2606.04665 §4.2]
[arxiv:2606.04665 eq. 7]. It then adds a control variate with a fitted
coefficient to reduce variance [arxiv:2606.04665 eq. 9] [arxiv:2606.04665
eq. 10]. The authors report a selected model at 72.42% mean accuracy on VisDA
against 72.55% for selection with target labels [arxiv:2606.04665 tab. 2].
Card: [Towards Accurate Model Selection in Deep Unsupervised Domain
Adaptation](/papers/2606.04665).

### Inverse-inclusion weights for sampled Shapley values

Exact Shapley values need one model call per feature subset. Leverage SHAP
draws subsets with a sampling distribution based on leverage scores, which
have a closed form here [arxiv:2410.01917 §3.1]. Each sampled subset is
paired with its complement [arxiv:2410.01917 §3.2]. Its regression row is
weighted by the kernel weight divided by the realised inclusion probability
[arxiv:2410.01917 alg. 1]. The authors report lower error than Optimized
Kernel SHAP at most budgets. Their error ratio lies between about 0.05 and 0.75
where the budget is below 2^n [arxiv:2410.01917 tab. 2]. Card:
[Provably Accurate Shapley Value Estimation via Leverage Score
Sampling](/papers/2410.01917).

### Policy ratios with masking for off-policy RL

When an agent trains on answers written by a slightly older model, the
per-token ratio of the current policy to the rollout policy corrects the
mismatch [arxiv:2607.07508 eq. 2]. Single-rollout Asynchronous Optimization
(SAO) takes this ratio from the rollout engine's logged log-probabilities, so
no copy of the old policy is kept [arxiv:2607.07508 §3.1]. It masks tokens
whose ratio falls outside an open interval rather than clipping them
[arxiv:2607.07508 eq. 3]. The authors report 97.3 on AIME2025 for
Qwen3-30B-A3B, against 84.2 for GRPO with Python tools [arxiv:2607.07508
tab. 1]. Card: [Single-Rollout Asynchronous Optimization for Agentic
Reinforcement Learning](/papers/2607.07508).

### Replayed correct rollouts with a shaped ratio

ExGRPO stores the correct answers from earlier batches, bucketed by how often
each question was solved, and puts one stored answer into about half of each
new batch [arxiv:2510.02245 §4.1] [arxiv:2510.02245 §4.2]. Questions that were
always solved are retired and never replayed. For the replayed answer, the
ratio of the current policy to the policy that wrote it is not used directly.
It is passed through f(w) = w/(w+0.1), which amplifies low-probability tokens
and damps high-probability ones [arxiv:2510.02245 §4.2]. Shaping replaces the
plain ratio, so the replayed term is no longer the plain importance-weighted
term; the main text states no unbiasedness result for it. The paper's
mixed-phase objective keeps the clip [arxiv:2510.02245 eq. 4], but the released
code does not clip there (see Contradictions). On Qwen2.5-Math-7B the authors
report 48.3 average over six math benchmarks against 45.3 for on-policy RLVR,
and 41.2 without the shaping [arxiv:2510.02245 tab. 1] [arxiv:2510.02245
app. E.4]. The step time rose by 19.0% on 8 GPUs [arxiv:2510.02245 app. E.5].
Card: [ExGRPO: Learning to Reason from Experience](/papers/2510.02245).

### Defensive mixture sampling for variance-reduced policy gradient

Variance-reduced policy gradient methods reuse trajectories from the previous
policy, which needs ratio weights. Their analyses usually assume those weights
have bounded variance. DEF-PG draws the correction trajectories from the
mixture q = αp(·|θ_1) + (1−α)p(·|θ_2) of the new and old parameters. For any
α in (0, 1], the variance of each ratio p/q is at most (1−α)/α, with no
separate variance assumption [arxiv:2610.03165 §3]. Under bounded rewards and
mean-square continuous policy scores, the paper's bound is O(ε⁻³) stopped
trajectories to reach an ε-stationary point [arxiv:2610.03165 Thm. 6]. The
score condition generally fails for Gaussian policies [arxiv:2610.03165 §4
Rem. 2]. The paper reports no experiments, and its step size needs constants
it does not estimate [arxiv:2610.03165 §4 Lemma 5]. Card: [Sample complexity of
variance-reduced policy gradient: weaker assumptions and lower
bounds](/papers/2610.03165).

### Heuristic sample weights for VLM fine-tuning

This variant does not estimate a target quantity. Each training sample gets a
weight of 3.0 if it is in a "synergy gap" set, 0.5 if it is in a "shortcut"
set, and 1.0 otherwise. The two sets come from ranking scores built from
answer entropies and local information terms [arxiv:2606.00959 alg. 1]
[arxiv:2606.00959 eq. 8] [arxiv:2606.00959 eq. 12]. The paper says these local
scores are heuristics, used only as ranking signals [arxiv:2606.00959 §2.3]
[arxiv:2606.00959 app. A.3]. On Qwen2.5-VL-7B, LoRA with these weights reaches
+2.3 pp on MMStar over uniform weights, with a 95% paired-bootstrap interval
of [1.4, 3.2] [arxiv:2606.00959 §5.1]. On Gemma3-12B the MMStar score falls
from 56.2 to 54.3 with the same weights [arxiv:2606.00959 tab. 3]. Card:
[Towards Understanding Modality Interaction in Multimodal Language Models via
Partial Information Decomposition](/papers/2606.00959).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Labels only in the source domain, a deep feature extractor, and you need to rank candidate models | Density-ratio weights with a control variate (DEV) | The paper reports close agreement with target-label selection on its benchmarks [arxiv:2606.04665 tab. 2] [arxiv:2606.04665 tab. 3]. Expect bias from the classifier estimate [arxiv:2606.04665 §4.3] |
| Source and target input distributions are far apart | Feature-space ratio, with a support check first | The paper argues feature-space ratios are better behaved [arxiv:2606.04665 §4.2], but gives no test for support overlap |
| Exponentially many items, each costly to evaluate, and a fixed budget of evaluations | Leverage-score sampling with inverse-inclusion weights | Its closed-form scores avoid listing all subsets [arxiv:2410.01917 §3.1]. The error ratio against Optimized Kernel SHAP varies with the budget and dataset, and is 1 where the budget reaches 2^n [arxiv:2410.01917 tab. 2] |
| Small feature count where the budget reaches 2^n | Exact computation | At m ≥ 2^n both methods are exact and the ratio is 1 [arxiv:2410.01917 tab. 2] |
| Asynchronous RL with one rollout per prompt, a learned critic, and logged rollout probabilities | Token-level ratio with interval masking (SAO) | It removes the need for group sampling and an old-policy copy [arxiv:2607.07508 §3.1] [arxiv:2607.07508 §3.2]. Its critic must be trained twice per batch |
| Verifiable-reward RL where past correct answers can be stored and replayed, and about 19% more step time is acceptable | Replay with the shaped ratio (ExGRPO) | Reported gain of 3.0 average points over on-policy RLVR on one 7B model [arxiv:2510.02245 tab. 1]. The shaping is part of the gain [arxiv:2510.02245 app. E.4]. Replay needs log-probs recorded at rollout time, see the card's Pitfalls |
| Variance-reduced policy gradient, where a theory without a bounded-variance assumption on the weights is the goal, and no empirical guidance is needed | Defensive mixture sampling (DEF-PG) | Its rates are oracle-level and rest on score and reward assumptions [arxiv:2610.03165 Thm. 6] [arxiv:2610.03165 tab. 1]. It reports no experiments [arxiv:2610.03165 §7] |
| Fine-tuning a vision-language model on a fixed set, a frozen model for scoring, and no density-ratio estimate | Heuristic weights (2606.00959) | Gains were reported on one model family and not the other [arxiv:2606.00959 tab. 3]. The weights are not an estimate of any target distribution [arxiv:2606.00959 §2.3] |
| Need an unbiased correction with no estimated ratio and no threshold | Not established by these papers | Each variant above relies on an estimated ratio, a sampling scheme, a mixture, a masking or shaping rule, or a heuristic score |

## Implementation sketch

The common core is a weighted mean with an optional control variate. The
control variate uses the weights' known mean of 1 under the sampling
distribution, which holds only for the exact ratio [arxiv:2606.04665 eq. 8].
This sketch is not from any one paper; the steps are cited to their source.

```pseudocode
# Weighted risk under a shift, with a control variate  [eq. 5] [eq. 9]
# Input:  per-sample losses (n,), per-sample weights (n,) >= 0
# Output: scalar estimate of the target risk

def weighted_risk(losses, weights):
    # Weight each loss by the estimated density ratio  [eq. 5] [eq. 7]
    weighted = weights * losses
    # Coefficient that minimises the variance of the estimate  [eq. 10]
    eta = -cov(weighted, weights) / var(weights)
    # Control variate: the weights have mean 1 under the sampling
    # distribution when the ratio is exact  [eq. 8] [eq. 9]
    return mean(weighted) + eta * (mean(weights) - 1)
```

For sampled estimators such as Leverage SHAP the weight is the kernel weight
over the inclusion probability instead of a density ratio [arxiv:2410.01917
alg. 1]. For RL the weight is the token ratio, and tokens whose ratio falls
outside the interval are zeroed [arxiv:2607.07508 eq. 1] [arxiv:2607.07508
eq. 3]. ExGRPO applies w/(w+0.1) to the ratio of its replayed answer instead
of a plain ratio [arxiv:2510.02245 §4.2]. For the defensive mixture, the
weights for each draw come from the two parameter sets and the mixture
coefficient [arxiv:2610.03165 eq. 3] [arxiv:2610.03165 eq. 4].

## Contradictions

- **Headline average for Leverage SHAP.** The paper's "50%" figure is the mean
  of the 38 Table 2 entries below 2^n [arxiv:2410.01917 tab. 2]. Averaging all
  48 entries, with the ten ratios of 1 included, gives about 0.61, or about
  39% lower error (average computed here from the table). Quote the budget
  range with the figure.
- **Density-ratio classifier, within DEV.** The paper describes a two-layer
  logistic regression [arxiv:2606.04665 §4.3]. The released code uses a
  multilayer perceptron with its regularisation picked on held-out data, a
  difference the [paper card](/papers/2606.04665) records. Whether this moves
  the reported numbers is not known.
- **Theorem and released sampler, within Leverage SHAP.** The guarantee covers
  an independent sampler, while the released estimator uses fixed per-size
  counts [arxiv:2410.01917 §3.2]. The paper's experiments use the released
  version.
- **Clip interval, within SAO.** The paper's clip values are named ε_low and
  ε_high, but the interval is written with other symbols [arxiv:2607.07508
  §4.1] [arxiv:2607.07508 §3.1]. Reading them as lower and upper bounds gives
  (0.7, 6.0) for math, a reading the paper does not confirm.
- **Clipping in the replay phase, within ExGRPO.** The mixed-phase objective
  keeps the clip [arxiv:2510.02245 eq. 4], and the appendix theory assumes it
  is removed [arxiv:2510.02245 app. D]. The released code removes it in the
  mixed phase. The [paper card](/papers/2510.02245) records this as a
  paper-versus-code difference.
- **Bucket weight scale, within ExGRPO.** The paper's Gaussian is on accuracy
  in [0, 1] [arxiv:2510.02245 alg. 1] [arxiv:2510.02245 §4.1]. The code applies
  it to success counts, which makes the question sampler much narrower. The
  paper does not say which scale produced the reported numbers.

## Papers

- [Provably Accurate Shapley Value Estimation via Leverage Score
  Sampling](/papers/2410.01917): leverage-score sampling for Kernel SHAP with
  inverse-inclusion weights and a regression guarantee on the objective.
- [ExGRPO: Learning to Reason from Experience](/papers/2510.02245): replay of
  stored correct rollouts with a shaped importance ratio and no clip in the
  mixed phase, for RL with verifiable rewards.
- [Towards Understanding Modality Interaction in Multimodal Language Models via
  Partial Information Decomposition](/papers/2606.00959): PID-based sample
  weights for VLM fine-tuning; the weights are heuristic, not density ratios.
- [Towards Accurate Model Selection in Deep Unsupervised Domain
  Adaptation](/papers/2606.04665): DEV, density-ratio weights in feature space
  with a control variate, for selecting unsupervised domain adaptation models.
- [Single-Rollout Asynchronous Optimization for Agentic Reinforcement
  Learning](/papers/2607.07508): token-level policy ratios from logged
  log-probabilities, with interval masking, for off-policy agentic RL.
- [Sample complexity of variance-reduced policy gradient: weaker assumptions
  and lower bounds](/papers/2610.03165): DEF-PG, defensive mixture sampling
  for a variance-reduced policy gradient, with oracle-level rates.