Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives

In plain words

A model trained on one collection of data often performs worse when it meets data that looks different, for example a diagnostic model applied to a population other than the one it was trained on. Unsupervised domain adaptation tries to close that gap by using the labels you already have for the old data, plus unlabelled examples from the new setting.

This review sorts the many ways to do that into a few families. Some make the statistics of the two data sets match. Some train a network to fool a domain classifier. Some keep separate normalisation statistics per domain, some translate images between domains, some let the model label the new data itself, and some add auxiliary tasks. It then shows where these methods have been used, from images and video to language, medical scans and time series, and it lists open problems.

TL;DR

A 2022 survey of deep unsupervised domain adaptation (UDA). It covers seven method families, applications in six areas, and eight open directions. It runs no experiments of its own and publishes no code, so use it as a map of the field and not as a source of benchmark numbers.

Problem

Deep networks usually need a large labelled training set and assume that training and test data are independent and identically distributed §1. When the target data comes from a different distribution, performance can drop sharply, and labelling enough target data is often expensive §1. For example, a disease diagnostic system applied to a population that differs from its training population cannot be expected to perform reliably §1. UDA assumes labelled source data and unlabelled target data, and aims to improve the model's accuracy on the target domain §1, §2.

The review gives four kinds of distribution shift §2, fig. 3:

• Covariate shift: the input distribution p(x) changes.
• Conditional shift: p(x|y) changes, which is more fine-grained than covariate shift.
• Label shift (target shift): the class proportions change.
• Concept shift: the meaning of a label changes between domains.

The review's main focus is covariate shift, the most studied case. The other three are discussed as open problems §2, §5.1.

Method

This is a survey, so there is no single method. Section 2 sets up the notation and motivates UDA with a bound on the target error §2. Section 3 then groups the published methods into nine families §3.1-§3.9, fig. 4.

Setup §2. The source distribution is p_s(x, y) and the target is p_t(x, y). The labelled source set is drawn i.i.d. from p_s(x, y), and the unlabelled target set is drawn i.i.d. from the marginal p_t(x). The goal is to generalise to the target domain by learning from both sets. For discriminative tasks the label set is {1, ..., c}. For generative tasks the label can be a continuous value, a sentence or an image §2.

Bound §2, eq. 1, Theorem 1. For a hypothesis h, the review states that the target loss is bounded by the source loss, plus a divergence between the two input distributions, plus a third term that it calls negligible §2:

\mathcal{L}_t(h) \le \mathcal{L}_s(h) + d[p_\mathcal{S}, p_\mathcal{T}] + \min\big[\mathbb{E}_{x\sim p_s}|p_s(y|x)-p_t(y|x)|,\ \mathbb{E}_{x\sim p_t}|p_s(y|x)-p_t(y|x)|\big] eq. 1

The divergence d is left general; the review names the Jensen-Shannon divergence as the one used in classic adversarial UDA §2. Its reasoning is that UDA should reduce the divergence, while the source loss is minimised with ordinary supervised training §2.

Families §3.1-§3.9.

• Statistic divergence alignment §3.1, fig. 5(a). Choose a divergence between source and target features and minimise it. The review lists MMD (with multi-kernel and joint variants), CORAL and its extensions, contrastive domain discrepancy (CDD), Wasserstein distance (JDOT and DeepJDOT) and a graph-matching loss.
• Adversarial learning §3.2, fig. 5(b), fig. 6(b). Learn the divergence with a domain discriminator, using the objectives in eqs. 2-4 below. The review describes DANN (gradient reversal layer), ADDA, Wasserstein-based discriminators and two-discriminator variants.
• Normalization statistics §3.3, fig. 7. Adapt batch-norm statistics per domain. AdaBN keeps all weights fixed and swaps the statistics. AutoDIAL retrains with alignment layers. OSUDA adapts without source data.
• Generative domain mapping §3.4, fig. 6. Render target-style images from source images with GANs or cycle-consistent translation, then train on them. The review notes that cycle reconstruction can distort local structure, and describes a modality-independent neighbourhood descriptor as one fix.
• Self-training §3.5. Alternate between pseudo-labelling the target domain and retraining on the selected pseudo-labels. See the pseudocode below.
• Self-supervision §3.6. Add pretext tasks such as image rotation or jigsaw prediction, or reconstruction of target images, to the training loss.
• Low-density target boundary §3.7. Push class boundaries into low-density regions of the target features, for example with virtual adversarial training or conditional entropy losses.
• Other methods §3.8. Model selection (DEV), prototype and subtype alignment, mixup regularisation, domain randomisation, mean teachers and graph reasoning.
• Combinations §3.9. Families are often combined, for example feature-level adversarial alignment with generative mapping (CyCADA), or BN alignment with entropy minimisation for source-free UDA §3.9.

Adversarial objectives §3.2, eqs. 2-4. Let f be the feature extractor, C(f(x), y) the classifier's probability for class y, Dis the discriminator with output in (0, 1), and lambda > 0 a balance weight. The review writes three objectives to be maximised. Each updates one module §3.2:

\max_{Cls}\ \mathbb{E}_{x\sim p_s} \log C(f(x), y) eq. 2

\max_{Dis}\ \mathbb{E}_{x\sim p_s} \log\big(1 - Dis(f(x))\big) + \mathbb{E}_{x\sim p_t} \log Dis(f(x)) eq. 3

\max_{f}\ \mathbb{E}_{x\sim p_s} \log C(f(x), y) + \lambda\, \mathbb{E}_{x\sim p_t} \log\big(1 - Dis(f(x))\big) eq. 4

The discriminator is trained to output 0 for source features §3.2. The review calls the pair of eqs. 3 and 4 a minimax game §3.2.

Pseudocode: one adversarial update §3.2, eqs. 2-4. This restates the three objectives. It returns values to maximise; for gradient descent, negate them. The order in which the three updates are applied is not stated.

# One adversarial UDA update  [§3.2] [eq. 2] [eq. 3] [eq. 4]
# Input:  source batch (x_s, y_s); unlabelled target batch x_t; weight lam
# Output: three objectives to maximise, one per module

def adversarial_uda_step(x_s, y_s, x_t, lam):
    # Shared feature extractor applied to both domains  [§3.2]
    feat_s = f(x_s)                              # (B_s, K)
    feat_t = f(x_t)                              # (B_t, K)

    # Classifier: maximise the source log-likelihood  [eq. 2]
    cls_objective = mean(log(C(feat_s, y_s)))

    # Discriminator: source features are pushed to 0, target to 1  [eq. 3]
    dis_objective = (mean(log(1 - Dis(feat_s)))
                     + mean(log(Dis(feat_t))))

    # Feature extractor: classify source and fool Dis on target  [eq. 4]
    f_objective = (mean(log(C(feat_s, y_s)))
                   + lam * mean(log(1 - Dis(feat_t))))

    return cls_objective, dis_objective, f_objective

Pseudocode: round-based self-training §3.5. The review describes only two steps per round: pseudo-label the target data, then retrain on the pseudo-labels §3.5. The starting model, the number of rounds and the confidence cut-off are not stated.

# Round-based self-training for UDA  [§3.5]
# Input:  starting model; unlabelled target images; number of rounds
# Output: model after the last round

def self_training(model, target_images, rounds):
    for round in range(rounds):
        # Step 1: pseudo-label the target domain  [§3.5]
        probs = predict(model, target_images)    # (N, c)
        pseudo_labels = argmax(probs, axis=1)
        confidence = max(probs, axis=1)

        # Keep confident predictions only; the review suggests the
        # maximum softmax value as the confidence  [§3.5]
        keep = confidence > threshold            # threshold: not stated

        # Step 2: retrain on the selected pseudo-labelled target data  [§3.5]
        # Whether source data is kept during retraining is not stated.
        model = fit(model, target_images[keep], pseudo_labels[keep])

    return model

Hyperparameters

The review proposes no method and gives no training settings, so there are no hyperparameters to copy. The free parameters that the text names are below.

Results

The review reports no new results. It names benchmarks and makes comparative statements, and it gives no metrics. Dataset sizes are quoted as the review states them. For the cited works' setups, see the original papers.

Benchmarks named in the review.

Comparative claims made in the review (no numbers given).

• Compared with ten earlier surveys, this review marks all seven topic columns in its own coverage table: deep UDA, generative mapping, normalisation, ensemble, self-training, self-supervision and low-density boundaries tab. 1.
• Self-training has, in several deep UDA tasks, surpassed adversarial training §3.5.
• Covariate-shift alignment methods and conditional-shift-blind methods have been outperformed in most benchmarks by self-training, dropout-based and moment-matching methods §5.1.

Paper vs code

No official code. The review proposes no implementation.

I searched the arXiv metadata and the LaTeX source for repository links. The source contains only a JMLR citation URL and two dataset URLs (Kaggle and the IXI dataset) §4.2, §4.2 footnotes. A GitHub repository search for the title terms returned no results.

Pitfalls

• No experiments. Every number on this card comes from the cited papers, and this card has not checked any of them. Verify against the source papers before using a figure.
• The bound's third term is called negligible without argument. The review calls the min term in eq. 1 "a negligible value" §2. It gives no size estimate. Read the bound as the review's framing, not as a quantified result.
• Eqs. 3 and 4 are typeset with unbalanced parentheses. The source writes log(1 - Dis(f(x))) without a closing parenthesis in both equations §3.2. The pseudocode above uses the balanced form that the text describes.
• Adversarial details are missing. The text gives no value for lambda, no update order for the three objectives, and no gradient-reversal scaling for DANN §3.2. An implementation needs these from the original papers.
• Self-training details are missing. The threshold, round count, starting model and whether source data is kept during retraining are not stated §3.5. The review also names a soft-label smoothing alternative to hard pseudo-labels, without a formula §3.5.
• Covariate-only scope. The review focuses on covariate shift §2. Its claims about benchmark outcomes §5.1 concern covariate-shift methods, and label and concept shift remain open §5.1.
• Coverage marks are the review's own judgement. The coverage table is not a measured result. One cell marks the self-training coverage of one cited survey as "no deep" tab. 1.
• The applications are examples, not a full survey. The review says it highlights trends and does not aim for completeness §4.
• Dataset counts need a source check. The GTA5 and Cityscapes counts §4.1.3 and the DomainNet numbers §4.1.1 are given as stated. Check them against each dataset's own documentation before use.

Lineage

• Builds on: none. This is a survey of the methods named in sections 3 and 4.
• Compared against: the ten earlier surveys in tab. 1.
• Superseded by: not known from the content of this wiki.