Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation

In plain words

A model trained on labelled data from one setting often gets used on a new setting where no labels exist. Picking which trained model to keep is then hard, because the usual check needs labels from the new setting. Checks that use only the old labelled data can mislead when the two settings differ a lot, and checks that use some new labelled data break the premise of learning without labels.

The paper proposes a check that stays label-free for the new setting. It looks at the model's internal features, trains a classifier to tell old examples from new ones, and uses that classifier to give each old validation example a weight, so the old set looks more like the new one. A second adjustment reduces how much the estimate jumps around. The authors report that the model it picks is close to the one that a check with new labels would pick.

TL;DR

• DEV estimates the target-domain error of a candidate model with labelled source validation data, reweighted by a density ratio computed in feature space §4.2, §4.3.
• The density ratio comes from a domain classifier trained on source-training and unlabelled target features; the classifier is trained from the model's own features alg. 1.
• A control variate reduces the variance of the weighted estimate §4.4, eq. 9, eq. 10.
• Author-reported: DEV is within 0.13 to 0.41 points of target-label selection on the four tables, and beats source-risk and IWCV baselines in each tab. 2, tab. 3, tab. 4, tab. 5.
• Read the pitfalls before use: the paper's domain classifier is described differently in the paper and the code, and the toy experiment in the code validates on training data.

Problem

Deep unsupervised domain adaptation (UDA) trains on labelled source data and unlabelled target data. The target labels are unavailable during training and during model selection, so the usual hold-out check cannot be run on the target §1. The paper lists the alternatives and their problems §1, §2, tab. 1:

• Fixed hyperparameters borrowed from other tasks; not usable when only one task is at hand §1.
• Source risk (error on source validation data); biased when the domain gap is large §1, §5.1.
• Target risk (error on a labelled slice of target data); unbiased, but it uses target labels that UDA assumes are missing §1.
• Importance-weighted cross-validation (IWCV); unbiased when the density ratio is known, but its variance is unbounded and its usual density-ratio fit is poor §2.2, §4.1.
• Transfer cross-validation (TrCV); needs labelled target data §2.2.

The gap the paper targets is a label-free estimator that is unbiased, has bounded variance, and works on deep features rather than raw inputs §2.2.

Method

The estimator is the importance-weighted validation risk, computed on features and wrapped in a control variate. Each candidate model is scored separately alg. 2.

Importance weighting (IWCV)

If the density ratio w = q/p between target and source inputs is known, the mean of weighted source losses equals the target risk §4.1, eq. 5:

\mathbb{E}_{x \sim p}\, w(x)\,\ell(g(x), y) = \mathcal{R}(g) \qquad \text{[eq. 5]}

The variance of that estimator is bounded by a Rényi divergence between p and q, with a term that depends on \alpha §4.1, eq. 6; definitions in §3.1, eq. 1:

\mathrm{Var}_{x \sim p}[\ell_w] \le d_{\alpha+1}(q \| p)\, \mathcal{R}(g)^{1 - 1/\alpha} - \mathcal{R}(g)^2 \qquad \text{[eq. 6]}

The paper's key move is that these two quantities depend on the input distributions. Those distributions do not shrink during adaptation, so IWCV's variance bound does not improve as the model adapts §4.2.

Deep Embedded Validation: feature-space weights

DEV replaces the input-space density ratio with one in the feature space f = F(x), where F is the candidate model's feature extractor §4.2. The argument: adaptation makes source and target features closer, so the feature-space divergence is typically smaller than the input-space one. The paper states that the same unbiasedness and variance bound carry over to feature-space weights w_f §4.2.

The feature-space ratio is estimated from a domain classifier. Bayes' rule rewrites the ratio as a constant times a posterior ratio §4.3, eq. 7:

w_f(x) = \frac{n_s}{n_t}\, \frac{J_f(d = 0 \mid x)}{J_f(d = 1 \mid x)} \qquad \text{[eq. 7]}

Here d = 1 marks source samples and d = 0 marks target samples. The constant comes from sample counts; the posterior ratio comes from a classifier trained on the domain labels, which are known for both sets §4.3.

Variance reduction by a control variate

Because \mathbb{E}_{x \sim p_f}[w_f] = 1 §4.4, eq. 8, the weights can serve as a control variate with no unknown expectation §3.2, §4.4. The DEV risk is §4.4, eq. 9:

\mathcal{R}_{\text{DEV}} = \frac{1}{n_s} \sum_{i=1}^{n_s} \left[ \ell_{w_f} + \eta\,(w_f(x_i^s) - 1) \right] \qquad \text{[eq. 9]}

The coefficient that minimises the variance is §4.4, eq. 10; §3.2, eq. 4:

\eta = -\frac{\widehat{\mathrm{Cov}}(\ell_{w_f}, w_f)} {\widehat{\mathrm{Var}}[w_f]} \qquad \text{[eq. 10]}

Procedure

The paper gives the procedure as two algorithms. GetRisk computes the DEV risk of one candidate model alg. 1. DEV ranks candidates by their risk and returns the minimiser alg. 2. The pseudocode below follows GetRisk, with the notation of the code dev.py.

# DEV risk of one candidate model  [alg. 1]
# Input:  features of source train (N_tr, d), target test (N_ts, d) and
#         source validation (N_v, d); per-sample error on validation (N_v,)
# Output: scalar DEV risk

def dev_risk(f_train, f_test, f_val, error_val):
    # Train a domain classifier: source = 1, target = 0  [§4.3] [alg. 1]
    classifier = fit_domain_classifier(f_train, f_test)
    # Density-ratio weight per validation sample  [eq. 7] [alg. 1]
    prob_target = classifier.prob_target(f_val)   # (N_v,)
    prob_source = classifier.prob_source(f_val)   # (N_v,)
    weights = len(f_train) / len(f_test) * prob_target / prob_source
    # Weighted error per validation sample  [alg. 1]
    weighted_error = weights * error_val
    # Control-variate coefficient  [eq. 10] [alg. 1]
    eta = -cov(weighted_error, weights) / var(weights)
    # DEV risk  [eq. 9] [alg. 1]
    return mean(weighted_error) + eta * mean(weights) - eta

Hyperparameters

Results

Author-reported. Setup: each row is one target model selected by the given method; the value is target-domain accuracy in percent. "Original" is the number from the original paper, as the authors list it §5 clarifications. "Target Risk" uses target labels and is an upper bound §5.2, §5.3.

Other reported results:

• Toy regression: over 1,000 repetitions, source risk is biased, and DEV and IWCV track the target risk. DEV has a clearly smaller spread than IWCV fig. 1, §5.1. No numbers are given in the text.
• Ablation on VisDA: the control variate adds just over 2 points to DEV (70.22 to 72.42) §5.6, tab. 2.
• Digits: DEV improves the original GTA result by about 1.4 points (92.83 to 94.22) §5.4, tab. 4.
• Density ratio on partial adaptation (PADA, A to W): the estimated ratios are higher for shared classes than for source-only classes, with one exception the authors explain fig. 2, §5.5.
• Table 1 compares the methods qualitatively on five properties tab. 1.

Paper vs code

Official code: thuml/Deep-Embedded-Validation at commit abf9c96c6bf653d919073f7fbe7f3213c56731b9, MIT licence, copyright of an author LICENSE. The repository has dev.py (DEV risk and density ratio) and toy.py (toy experiment). The experiment code for MCD, CDAN, GTA and PADA is not in the repository, so those tables cannot be checked against code.

Differences found:

• Domain classifier. The paper says it trains a "two-layer logistic regression" §4.3 alg. 1. The code uses scikit-learn's MLPClassifier with three hidden layers of widths (d, d, 2), ReLU activations, and nine L2 strengths from 1e-1 to 1e-5. The strength is picked by accuracy on a 20% held-out split of the features, and the split has no fixed seed dev.py. The code therefore differs from the paper in architecture, model selection and randomness.
• Density-ratio formula. Consistent. The code returns P(target | f) / P(source | f) times N_source / N_target dev.py, which equals eq. 7 and the weight line of alg. 1.
• DEV risk formula. Consistent. The code computes mean(weighted error) + η·mean(weight) − η dev.py, as in alg. 1 and eq. 9. It uses sample covariance and variance with one degree of freedom (ddof = 1); the paper does not state this.
• Loss. The paper uses a generic loss ℓ alg. 1. The code's docstring says the error is typically 0 for a correct and 1 for a wrong classification dev.py. The paper does not say which loss the classification experiments use.
• Toy λ grid. The text says λ runs from 0.0 to 1.0 §5.1. The code uses λ = 0.0, 0.1, …, 0.9 toy.py.
• Toy validation set. The code computes the "IWCV" and DEV risks on the training points themselves, not on held-out points toy.py. The paper's §5 describes hold-out validation for all experiments, and the toy section does not say which set it uses.
• Toy loss. The toy code uses squared error toy.py. The footnote on Figure 1 says the error rate is used to select models fig. 1. Toy is a regression task, so the footnote's wording does not match the code.
• Toy density parameters. The paper writes N(x | 1, (1/2)²) and N(x | 2, (1/4)²). The code uses standard deviations of 0.5 and 0.25 toy.py. Read the paper's second argument as a standard deviation.
• Toy sample sizes. 150 training and 150 test points toy.py; the paper does not state these numbers.
• Toy DEV. The toy uses the analytic density ratio, not the classifier toy.py, as the paper says §5.1.

Pitfalls

• Unbiasedness assumes the true ratio. Eqs. 5 and 9 are unbiased for the exact density ratio. DEV replaces it with a classifier estimate eq. 7. The paper gives no error bound for the estimate; its evidence is the qualitative plot of the density ratio on PADA fig. 2. Expect bias in practice.
• Support condition. Eq. 5 needs the source support to contain the target support §4.1. If it fails, the weights can grow without limit §4.2. The paper says the feature space makes this more likely to hold, but gives no test for it.
• No clipping. The code divides by P(source | f) and does not clip or normalise the weights dev.py. A validation sample that the classifier finds very source-like gets a near-zero divisor; a sample that looks target-like gets a large weight.
• Degenerate variance. η divides by the variance of the weights dev.py. Constant weights give a division by zero, and the code has no guard.
• Non-determinism. The domain-classifier split has no random seed dev.py. Two runs can give different risks for the same model. Fix the seed before comparing candidates.
• Cost per candidate. The code fits nine MLPs per candidate model, one per L2 value dev.py. The width of each hidden layer equals the feature dimension d, so cost grows with d. The paper does not report this cost.
• Choice of the held-out split. The paper does not give the source train/validation ratio. The code takes the split as input and does not make it dev.py.
• Rényi order. §3.1 allows α ≥ 0 with α ≠ 1, but eq. 6 contains the exponent 1 − 1/α, which needs α > 0 §3.1, §4.1. The experiments do not state an α, so the variance bound cannot be evaluated from the paper.
• Bounded variance is theoretical. The paper gives a bound on variance eq. 6 but does not measure it on the real datasets. The only variance evidence is the toy plot and the VisDA ablation.
• Small gaps and single runs. The gaps between DEV and target-label selection are 0.13 (VisDA), 0.41 (CDAN), 0.28 (GTA) and 0.19 (PADA) points tab. 2–5. The real-dataset tables give one number per method and no run count or spread, so these gaps may be within noise.
• Baseline variant. The paper's own description of IWCV fits a Gaussian to estimate the density ratio §2.2. In the experiments the authors use a classifier on pre-trained ResNet-50 features instead §5 clarifications. Read the IWCV rows as this stronger variant, not the original method.
• Candidate grids are not fully stated. The MCD and GTA tuning spaces are only partly given §5.2, §5.4, so the set of candidates that DEV ranks cannot be rebuilt exactly.
• Algorithm input. Alg. 1 takes the validation error per sample. Use the same loss for all candidates, or the risks are not comparable.

Lineage

• Builds on: importance-weighted cross-validation (Sugiyama et al., JMLR 2007), used for model selection under covariate shift §2.2, §4.1; discriminative density-ratio estimation (Bickel et al., 2007) §4.3; control variates, a standard variance-reduction method §3.2.
• Compared against: source risk, target risk (uses target labels), IWCV, and TrCV (named in §2.2 and tab. 1 but not in the results tables) tab. 1, §5.
• Applied to: MCD (VisDA), CDAN (Office-31), GTA (Digits), PADA (partial Office-31) §5. These base methods have no cards in this wiki.
• Superseded by: not known from the content of this wiki.