---
kind: concept
slug: model-selection
title: Model selection without target labels
summary: >-
  Choosing which trained model to keep when the usual hold-out check needs labels you do not have, such as labels from the new domain under unsupervised domain adaptation. Options are source risk, importance-weighted validation and feature-space Deep Embedded Validation, which the authors report comes close to target-label selection. One paper so far.
tags: [model-selection, validation, domain-adaptation, importance-weighting]
papers: ["2606.04665"]
related: [domain-adaptation, importance-weighting, hyperparameter-optimization]
status: draft
updated_at: 2026-10-18
---

## Overview

Training produces several candidate models, and one must be kept. The usual
rule is to score each candidate on a labelled hold-out set from the
distribution where the model will be used. Under unsupervised domain
adaptation that set does not exist: the target domain has no labels during
training or selection, so the check cannot run there [arxiv:2606.04665 §1].

Each option below is a different way to estimate the target error without
target labels, or to give up on target labels entirely. They differ in
whether they are unbiased, how much variance they carry, and whether they
need access to the model's features. The only paper so far that compares
them on deep UDA models is arxiv:2606.04665. Its results are author-reported
and come from one run per method on the real benchmarks [arxiv:2606.04665 §5].

## Variants

### Fixed hyperparameters from other tasks

Reuse the settings that worked on a different task. The paper notes this is
unusable when only one task is at hand [arxiv:2606.04665 §1]. It is listed for
completeness and is not scored in the results tables.

### Source risk

Score each candidate by its error on labelled source validation data. It
uses no target data at all, but it is biased when the source and target
differ a lot [arxiv:2606.04665 §1, §5.1]. On the paper's VisDA run it selects
MCD models at 66.36% mean accuracy, against 72.42% for DEV on the same run
[arxiv:2606.04665 tab. 2].

### Importance-weighted cross-validation (IWCV)

Reweight the labelled source validation errors by the density ratio between
target and source inputs. If the ratio is known, the weighted mean is an
unbiased estimate of target risk [arxiv:2606.04665 §4.1, eq. 5]. Its variance
is bounded through a Rényi divergence between the two input distributions
[arxiv:2606.04665 eq. 6], and that bound does not shrink as the model adapts,
because the input distributions stay fixed [arxiv:2606.04665 §4.2]. The
paper's experiments fit the ratio with a classifier on pre-trained ResNet-50
features rather than the Gaussian fit in the original method
[arxiv:2606.04665 §2.2, §5]. On the paper's VisDA run it selects MCD models at
54.72% [arxiv:2606.04665 tab. 2].

### Deep Embedded Validation (DEV)

Estimate the density ratio in the candidate's feature space instead of the
input space. A domain classifier, trained on source-training and unlabelled
target features, gives the ratio through Bayes' rule [arxiv:2606.04665 §4.3,
eq. 7]. A control variate built from the same weights reduces the variance
of the weighted error [arxiv:2606.04665 §4.4, eq. 9, eq. 10]. The paper
reports DEV at 72.42% on the VisDA MCD run, against 72.55% for target-label
selection [arxiv:2606.04665 tab. 2]. Its ablation shows the control variate
adds about 2.2 points on that run (70.22 to 72.42) [arxiv:2606.04665 §5.6].

### Target risk (uses target labels)

Score each candidate by its error on a labelled slice of the target domain.
It is unbiased but breaks the label-free premise of UDA [arxiv:2606.04665
§1]. The paper uses it as the reference that the label-free methods are
measured against [arxiv:2606.04665 §5.2, §5.3].

### Transfer cross-validation (TrCV)

Named in the paper as a method that needs labelled target data
[arxiv:2606.04665 §2.2, tab. 1]. It is not a label-free option and is not
evaluated in the results tables.

## When to use which

| Situation | Use | Why |
|---|---|---|
| No target labels, feature access to the candidates, unlabelled target inputs available | DEV | Best label-free result in the paper's tables, within 0.13 to 0.41 points of target-label selection on the four benchmarks [arxiv:2606.04665 tab. 2–5] |
| No target labels, and features or unlabelled target inputs are not accessible | Source risk, with the caveat that it is biased under a large domain gap | Its numbers are below DEV on every table, by about 6 points on VisDA [arxiv:2606.04665 tab. 2–5] |
| Target labels can be spent on a held-out slice | Target risk | Unbiased; the paper uses it as the upper bound for the label-free methods [arxiv:2606.04665 §5.2] |
| Density ratio can be estimated well and variance matters | IWCV, with the variance bound in mind | The bound is unbiased only with the true ratio and grows with the divergence [arxiv:2606.04665 §4.1, eq. 6]. The paper's IWCV rows are weaker than DEV on VisDA [arxiv:2606.04665 tab. 2] |

The table rests on one paper. The gaps between DEV and target-label
selection are single numbers without a run count or spread, so they may be
within noise [arxiv:2606.04665 tab. 2–5].

## Implementation sketch

Language-agnostic pseudocode of the DEV risk for one candidate model. Inputs
are the candidate's features on labelled source training data, unlabelled
target data and source validation data, and its per-sample validation error
[arxiv:2606.04665 alg. 1].

```pseudocode
# Fit a domain classifier: source = 1, target = 0  [arxiv:2606.04665 §4.3]
classifier = fit_domain_classifier(features_src_train, features_tgt)

# Density-ratio weight per validation sample  [arxiv:2606.04665 eq. 7]
weights = (n_src / n_tgt) * p_tgt(features_val) / p_src(features_val)

# Control-variate coefficient, fitted on the same samples  [eq. 10]
eta = -cov(weights * error_val, weights) / var(weights)

# DEV risk  [eq. 9]
risk = mean(weights * error_val) + eta * (mean(weights) - 1)
```

Rank the candidates by this risk and keep the lowest [arxiv:2606.04665 alg. 2].
Use the same loss for every candidate, and fix any random seed in the
classifier so repeated runs agree. The paper states neither point; its code
has no fixed seed [arxiv:2606.04665 alg. 1].

## Contradictions

Only one paper covers this concept, so no two papers disagree. Within the
paper's own materials, the density-ratio classifier is described as a
two-layer logistic regression [arxiv:2606.04665 §4.3], while the official
code uses a multilayer perceptron. The card lists the differences and the
other paper-code mismatches in [its Paper vs code section](/papers/2606.04665).

## Papers

- [Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation](/papers/2606.04665): introduces DEV, a label-free model selection method for UDA that reweights source validation data with a feature-space density ratio and a control variate.