Generalizing to Unseen Domains: A Survey on Domain Generalization

In plain words

A model is usually tested on data that looks like the data it was trained on. When the test data comes from a different source, such as photos instead of sketches, accuracy often drops. Domain generalization is the setting where a model is trained on several related data sources, none of them the target, and must still work on a new source it has never seen.

This survey collects the ways researchers have tried to do that. Some change the training data, for example by adding varied or synthetic copies. Some change the features the model learns, so that they do not depend on the source. Some change how training itself is run, for example by simulating a shift between sources during training. The paper also explains the theory behind the problem, lists the benchmark datasets, and runs a set of common methods on two image benchmarks, so that they can be compared under one code base.

TL;DR

Domain generalization (DG) trains on one or more source domains and tests on an unseen target domain whose data is never used in training or model selection §II.A. The survey groups methods into data manipulation, representation learning and learning strategies §IV. Its own benchmark (DeepDG, ResNet-18) finds that the ERM baseline is acceptable and that the best gains over it are small §VII.C. Use it as a map of the field and as a starting point for the code, not as proof that any one method works.

Problem

Standard learning assumes training and test data share one distribution. When the test distribution differs, performance often falls §I. Collecting data from every possible domain is expensive or impossible §I. Domain generalization asks for a predictor h that has low error on an unseen test domain, given M source domains whose joint distributions are pairwise different §II.A.

The test domain is not available during training, not even unlabelled. This is what separates DG from domain adaptation, which may use unlabelled target data §III.A. The paper argues DG is harder but closer to practical use §II.B.

Assumptions: the sources and the target are related but not identical, and the label space is the same across domains §II.A. The theory in §III.B uses a covariate-shift assumption, where the labelling function is shared across domains §III.B.

Method

This is a survey, so the card describes the categories and the objectives the paper writes down, not a single algorithm. For each method family, the card gives the paper's section and the main named methods. The paper itself uses a fixed taxonomy (its Figure "Taxonomy of domain generalization methods" §IV).

Formal objective. The target risk is the quantity to minimise, but it can only be estimated on sources eq. 1 §II.A:

\min_{h} \ \mathbb{E}_{(x,y) \in S_{test}} [\ell(h(x), y)] eq. 1

Data manipulation (§IV.A). Add manipulated copies of the inputs to the training set. The objective adds a loss on M(x), a function that changes the input §IV.A:

\min_{h} \mathbb{E}_{x,y}[\ell(h(x),y)] + \mathbb{E}_{x',y}[\ell(h(x'),y)] §IV.A

with x' = M(x). Two families:

1. Augmentation. Standard transforms (flip, rotation, scaling, cropping, noise), and domain randomisation that varies object placement, texture, illumination and camera view §IV.A.1.
2. Generation. VAEs, GANs, and Mixup, which interpolates two samples and their labels with a Beta-distributed weight. Mixup can be applied in input space or feature space §IV.A.2.

Adversarial augmentation (CrossGrad, iterative fictitious-target augmentation) chooses the perturbation with an optimisation goal, at the cost of adversarial training §IV.A.1.

Representation learning (§IV.B). Learn g so that the classifier f∘g works on all sources and the target. Written as a loss plus a regulariser §IV.B:

\min_{f,g} \mathbb{E}_{x,y}[\ell(f(g(x)),y)] + \lambda \ell_{reg} §IV.B

• Domain-invariant representations (§IV.B.1). Kernel methods (DICA, SCA), domain-adversarial training (DANN and class-conditional variants), explicit alignment with MMD, second-order statistics or Wasserstein distance, and feature normalisation (IN, BIN, SNR). The paper says normalisation methods can be combined with the others §IV.B.1.
• Invariant risk minimisation (§IV.B.1). Learn a representation whose best classifier is the same in every domain. The paper gives a practical surrogate that penalises the gradient norm of a dummy classifier, and notes its linear assumption and its failure cases §IV.B.1.
• Feature disentanglement (§IV.B.2). Split features into domain-shared and domain-specific parts, with a reconstruction loss so no information is lost:

\min_{g_c,g_s,f} \mathbb{E}[\ell(f(g_c(x)),y)] + \lambda \ell_{reg} + \mu \ell_{recon} §IV.B.2

Three types: multi-component analysis (UndoBias, low-rank models), generative modelling (DIVA, UFDN) and causality-inspired methods §IV.B.2.

Learning strategy (§IV.C). General training schemes applied to DG:

• Ensemble learning (§IV.C.1): per-domain classifiers or experts, combined with learned weights (Mancini et al.; DAEL). Costs more memory and compute.
• Meta-learning (§IV.C.2): split the sources into meta-train and meta-test to simulate a shift. Examples: MLDG, MetaReg, MetaVIB. The paper gives a bi-level formulation, but its update rule uses α and β in a way that does not match its own text (see Pitfalls).
• Gradient operations (§IV.C.3): RSC removes dominant features during training; gradient inner-product maximisation (GIP) and gradient-match variants such as Fishr align per-domain gradients.
• Distributionally robust optimisation (§IV.C.4): optimise the worst-case training domain (GroupDRO, VREx, AdaRNN). GroupDRO needs group labels, which later work reduced to a validation subset.
• Self-supervised learning (§IV.C.5): jigsaw pretext tasks and contrastive objectives, which can be added to other methods. Costs compute.
• Other (§IV.C.6): metric learning, random forests, calibration, lottery ticket substructures, and weight averaging for flat minima (SWAD).

Baseline implementation, as in the code. The benchmark code trains every method with the same loop. The ERM step below is the reference that all other DeepDG methods build on ERM.py:

# One training step of ERM, the DeepDG baseline  [code: alg/algs/ERM.py#L25-L35]
# Input:  one minibatch (x, y) per training domain
# Output: scalar loss

def erm_step(minibatches):
    # Stack the minibatches of all training domains into one batch  [ERM.py#L26]
    all_x = concat([x for x, y in minibatches])
    all_y = concat([y for x, y in minibatches])
    # Plain cross-entropy over all source domains  [ERM.py#L28]
    loss = cross_entropy(model(all_x), all_y)
    # SGD with Nesterov momentum and weight decay  [code: alg/opt.py#L51-L52]
    sgd_update(loss)
    return loss

The Mixup variant used in the benchmark mixes pairs of minibatches instead Mixup.py:

# Mixup across pairs of source minibatches  [code: alg/algs/Mixup.py#L14-L35]
# Input:  one minibatch per training domain; alpha = mixupalpha
# Output: scalar loss

def mixup_loss(minibatches, alpha):
    objective = 0
    # Pair the minibatches; the pairing rule is in the helper  [Mixup.py#L17]
    for (x_i, y_i), (x_j, y_j) in random_pairs(minibatches):
        # Draw one mixing weight per pair from Beta(alpha, alpha)  [Mixup.py#L18]
        lam = beta_sample(alpha, alpha)
        # Interpolate the inputs  [Mixup.py#L20]
        mixed = lam * x_i + (1 - lam) * x_j
        predictions = model(mixed)
        # Weight both labels by lam  [Mixup.py#L24-L26]
        objective += lam * cross_entropy(predictions, y_i)
        objective += (1 - lam) * cross_entropy(predictions, y_j)
    # Average over the minibatches  [Mixup.py#L28]
    return objective / len(minibatches)

Hyperparameters

The paper states the setting for its own benchmark only in part. Values marked "code" are the defaults of the DeepDG code at the pinned commit, not values the paper reports. The paper does not say which of them produced its benchmark table §VII.C.

Per-task hyperparameter values for PACS and Office-Home with ResNet-18 and ResNet-50 are in scripts/paramsref.md paramsref.md. They are not in the paper.

Results

Author-reported. The paper's benchmark uses its DeepDG code, trains each method on the training domains, selects the model on the 20% validation split, and tests on the held-out domain §VII.C. Setup: ResNet-18 backbone; PACS and Office-Home §VII.C. The table gives one number per cell with no seed count or deviation stated.

Source: §VII.C (benchmark table). The Office-Home columns are labelled A, C, P, R in the paper. OH = Office-Home.

Dataset overview from the paper, with the domain and class counts it gives §VII.A:

Source: §VII.A (dataset table). The paper lists 18 datasets; the last ten come from WILDS and are listed there with their own counts.

Other evaluation settings the paper names: GvHD flow cytometry, Parkinson's telemonitoring, semantic segmentation (Cityscapes, GTA5, Synthia), person re-identification, and OpenAI Gym control tasks §VII.A.

Paper vs code

Official code: the DeepDG toolkit in the authors' transferlearning repository, which the paper links in its list of contributions §I. The owner is the first author, Jindong Wang. Code commit d77ef50a2e7ecdd4f3354148205666970ba809b6, license MIT LICENSE.

Discrepancies between the paper's text and table, and between the paper and the code:

• Backbone. Paper: ResNet-18 §VII.C. Code default: resnet50 train.py. The run script and parameter reference use ResNet-18 run.sh paramsref.md. The paper does not say which setting produced its table.
• Learning rate. Paper: not stated. Code default: 1e-2 train.py. Parameter reference: 0.001 or 0.005 paramsref.md.
• Method count. Paper: nine methods in DeepDG, namely ERM, Mixup, DDC, DANN, CORAL, MLDG, RSC, GroupDRO and ANDMask §VII.C. Its table has only seven of them (no DDC or MLDG). Code README lists eleven, including VREx and DIFEX readme.md.
• RSC as best. Paper text: RSC is best on both datasets §VII.C. Table: RSC and CORAL both average 82.0 on PACS §VII.C. RSC is best only on Office-Home.
• DANN and ANDMask. Paper text: they "even have worse performance" than ERM §VII.C. Table: DANN is above ERM on PACS (81.4 vs 81.2). Both are below ERM on Office-Home.
• Validation. Paper: 20% of each source for validation §VII.C. Code: rate = 0.2, stratified per source domain getdataloader.py getdataloader.py. Consistent.

Pitfalls

• The benchmark is not a leaderboard. One run per cell, no seed count and no deviation. Differences of 0.5 or less between methods (for example 81.2 vs 81.4 for ERM and DANN) are within what one seed can move. The paper's own text calls the RSC gains "unremarkable" §VII.C.
• Hyperparameters are not in the paper. For the benchmark, the paper gives the backbone and validation rule only. An implementer who wants to match the table must take values from the code, and the code has two different learning rates documented (1e-2 default, 0.001 or 0.005 in the parameter reference).
• Meta-learning notation is inconsistent. The meta update uses α as the outer step, but the text says η and β are the outer and inner learning rates §IV.C.2. Use the MAML/MLDG papers for the exact rule, and check the code before copying any number.
• Inner-loop learning rate in MLDG. The code exposes an inner_lr flag with default 1e-2 train.py, which is separate from lr. The paper does not state either value.
• Mixup is applied per minibatch pair, not per sample. The code draws one λ per pair of domain minibatches and divides by the number of minibatches Mixup.py. Descriptions of Mixup in the paper treat it as sample-level interpolation §IV.A.2.
• Evaluation protocol. The paper calls training-domain validation the most common protocol, and says it "cannot achieve the best performance for some time" §VII.B. Test-domain validation is more accurate but uses target data, so it does not fit DG.
• Cost notes. The paper warns that ensembles multiply memory and compute §IV.C.1, that adversarial augmentation is hard to optimise §IV.A.1, and that self-supervised objectives need more compute §IV.C.5.
• Theory assumptions. The DG bounds in §III.B assume covariate shift and, for some results, a kernel or VC-dimension setting §III.B. The first bound is on the average risk over domains drawn from a hyper-distribution. The second assumes the target distribution lies in the convex hull of the sources.
• Snapshot. The arXiv record is v7 (updated 2022-05-24). The code and benchmark cover image classification only §VII.C.

Lineage

• Builds on: domain adaptation theory of Ben-David et al. and Blanchard et al. (kernel-based DG). The paper reproduces their bounds in §III §III.A §III.B. Original DA work it cites: DANN (Ganin et al.), CORAL, MMD-based alignment §IV.B.1. Meta-learning: MAML, MLDG §IV.C.2.
• Compared against: DomainBed (Gulrajani et al.), which the paper names as the prior benchmark it contrasts DeepDG with §VII.C.
• Concurrent surveys named by the paper: Zhou et al. (computer vision DG), Shen et al. (OOD generalization with causality), and Yang et al. (OOD detection) §I.
• Superseded by: not stated in this paper.