Model selection without target labels

Overview

Training produces several candidate models, and one must be kept. The usual rule is to score each candidate on a labelled hold-out set from the distribution where the model will be used. Under unsupervised domain adaptation that set does not exist: the target domain has no labels during training or selection, so the check cannot run there 2606.04665 §1.

Each option below is a different way to estimate the target error without target labels, or to give up on target labels entirely. They differ in whether they are unbiased, how much variance they carry, and whether they need access to the model's features. The only paper so far that compares them on deep UDA models is arxiv:2606.04665. Its results are author-reported and come from one run per method on the real benchmarks 2606.04665 §5.

Variants

Fixed hyperparameters from other tasks

Reuse the settings that worked on a different task. The paper notes this is unusable when only one task is at hand 2606.04665 §1. It is listed for completeness and is not scored in the results tables.

Source risk

Score each candidate by its error on labelled source validation data. It uses no target data at all, but it is biased when the source and target differ a lot 2606.04665 §1, §5.1. On the paper's VisDA run it selects MCD models at 66.36% mean accuracy, against 72.42% for DEV on the same run 2606.04665 tab. 2.

Importance-weighted cross-validation (IWCV)

Reweight the labelled source validation errors by the density ratio between target and source inputs. If the ratio is known, the weighted mean is an unbiased estimate of target risk 2606.04665 §4.1, eq. 5. Its variance is bounded through a Rényi divergence between the two input distributions 2606.04665 eq. 6, and that bound does not shrink as the model adapts, because the input distributions stay fixed 2606.04665 §4.2. The paper's experiments fit the ratio with a classifier on pre-trained ResNet-50 features rather than the Gaussian fit in the original method 2606.04665 §2.2, §5. On the paper's VisDA run it selects MCD models at 54.72% 2606.04665 tab. 2.

Deep Embedded Validation (DEV)

Estimate the density ratio in the candidate's feature space instead of the input space. A domain classifier, trained on source-training and unlabelled target features, gives the ratio through Bayes' rule [arxiv:2606.04665 §4.3, eq. 7]. A control variate built from the same weights reduces the variance of the weighted error 2606.04665 §4.4, eq. 9, eq. 10. The paper reports DEV at 72.42% on the VisDA MCD run, against 72.55% for target-label selection 2606.04665 tab. 2. Its ablation shows the control variate adds about 2.2 points on that run (70.22 to 72.42) 2606.04665 §5.6.

Target risk (uses target labels)

Score each candidate by its error on a labelled slice of the target domain. It is unbiased but breaks the label-free premise of UDA [arxiv:2606.04665 §1]. The paper uses it as the reference that the label-free methods are measured against 2606.04665 §5.2, §5.3.

Transfer cross-validation (TrCV)

Named in the paper as a method that needs labelled target data 2606.04665 §2.2, tab. 1. It is not a label-free option and is not evaluated in the results tables.

When to use which

The table rests on one paper. The gaps between DEV and target-label selection are single numbers without a run count or spread, so they may be within noise 2606.04665 tab. 2–5.

Implementation sketch

Language-agnostic pseudocode of the DEV risk for one candidate model. Inputs are the candidate's features on labelled source training data, unlabelled target data and source validation data, and its per-sample validation error 2606.04665 alg. 1.

# Fit a domain classifier: source = 1, target = 0  [arxiv:2606.04665 §4.3]
classifier = fit_domain_classifier(features_src_train, features_tgt)

# Density-ratio weight per validation sample  [arxiv:2606.04665 eq. 7]
weights = (n_src / n_tgt) * p_tgt(features_val) / p_src(features_val)

# Control-variate coefficient, fitted on the same samples  [eq. 10]
eta = -cov(weights * error_val, weights) / var(weights)

# DEV risk  [eq. 9]
risk = mean(weights * error_val) + eta * (mean(weights) - 1)

Rank the candidates by this risk and keep the lowest 2606.04665 alg. 2. Use the same loss for every candidate, and fix any random seed in the classifier so repeated runs agree. The paper states neither point; its code has no fixed seed 2606.04665 alg. 1.

Contradictions

Only one paper covers this concept, so no two papers disagree. Within the paper's own materials, the density-ratio classifier is described as a two-layer logistic regression 2606.04665 §4.3, while the official code uses a multilayer perceptron. The card lists the differences and the other paper-code mismatches in its Paper vs code section.

Papers

• Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation: introduces DEV, a label-free model selection method for UDA that reweights source validation data with a feature-space density ratio and a control variate.