Domain generalization

Overview

A model is often tested on data that differs from its training data, and its accuracy drops when the difference is large. Domain generalization (DG) is the setting where training uses related source domains, none of them the target, and the model must work on a new domain it has never seen 2103.03097 §I 2103.03097 §II.A. Most methods need several sources, but some normalization-based methods work with one source 2103.15597 §1. A text-side method on a frozen vision-language model uses no source images at all: its training data is template prompts and class names 2311.16445 §4.2.

The shared invariant is that the target domain is unavailable during training, not even unlabelled. This is the main difference from domain adaptation, which may use unlabelled target data 2103.03097 §III.A. The sources and the target are assumed to be related but not identical, with the same label space 2103.03097 §II.A.

Some data has no domain labels at all. For node-level prediction on one graph, with no environment for each node, the method has to create its own training environments. 2202.02466 §2.1 2202.02466 §3.2 describes one way to do this, by editing the graph (see Data manipulation).

The objective is to minimise the risk on the unseen target. Since that risk cannot be measured during training, every method is a proxy for it 2103.03097 §II.A 2103.03097 eq. 1.

Variants

The survey sorts methods into three groups. The paper's own taxonomy is in its §IV; the names below are the groups the page uses. Two methods on this page are not among the survey's named methods: RobustNet 2103.15597 is placed under Representation learning, Fish 2104.09937 under Learning strategy, and CLAP 2311.16445 under Representation learning. CLAP's training data is text, so its augmentation is in the spirit of data manipulation, but its learned component is a feature map.

Data manipulation

Add modified copies of the inputs to the training set, so the model sees more variation than the sources contain 2103.03097 §IV.A.

• Augmentation. Standard transforms such as flips, rotations, scaling, cropping and noise, plus domain randomisation that varies object placement, texture, illumination and camera view 2103.03097 §IV.A.1. Card: Generalizing to Unseen Domains.
• Generation. Samples from VAEs or GANs, and Mixup, which interpolates two samples and their labels. Mixup can run in input space or feature space 2103.03097 §IV.A.2.
• Adversarial augmentation. Perturbations chosen by an optimisation goal, for example CrossGrad and iterative fictitious-target augmentation. These need adversarial training and are hard to optimise 2103.03097 §IV.A.1. A graph variant learns the edits. Editors add and delete edges of one graph to create K virtual environments, and the GNN minimises the mean plus the variance of its risks on them. The editors are trained with REINFORCE and the variance as reward 2202.02466 §3.2 2202.02466 app. A. The paper calls this EERM. Its single-level form, with known environments, is the objective REx uses 2202.02466 §3.2. Card: Handling Distribution Shifts on Graphs.

Representation learning

Learn a feature map g so that one classifier f works on all sources and the target. The general form is a task loss plus a regulariser on g 2103.03097 §IV.B.

• Domain-invariant representations. Kernel methods (DICA, SCA), domain adversarial training (DANN and class-conditional variants), explicit alignment with MMD, second-order statistics or Wasserstein distance, and feature normalisation (IN, BIN, SNR). The survey says normalisation can be combined with the other methods 2103.03097 §IV.B.1.
• Selective whitening. Penalise only the channel-covariance entries of a normalisation layer that move under colour and blur change, and leave the others free. RobustNet's instance selective whitening (ISW) does this in three instance-normalisation layers. It picks the entries after a warm-up, with k-means on their variance 2103.15597 §4.3 2103.15597 §4.4. Full whitening removes both style and content correlations, and its exact form is slow and blocks gradients 2103.15597 §3. The method trains with one source domain 2103.15597 §1.
• Invariant risk minimisation. Learn a representation whose best classifier is the same in every domain. The survey gives a gradient-norm surrogate and names its linear assumption and failure cases 2103.03097 §IV.B.1.
• Feature disentanglement. Split features into a domain-shared part and a domain-specific part, with a reconstruction loss so that information is not lost 2103.03097 §IV.B.2. Families: multi-component analysis, generative modelling (DIVA, UFDN) and causality-inspired methods. Related page: causal-inference.
• Text-side style removal. For a frozen CLIP-style model, a small residual MLP is trained on text features only. Each training prompt is paired with a copy in which style words are dropped or reordered, and a contrastive loss pulls the pair together and pulls the copy toward the bare class name. The MLP is then reused on image features at inference, with a scalar weight on its residual branch 2311.16445 §4.1.2 2311.16445 §4.2. The paper argues that text is easier to edit for style than images 2311.16445 §4.2. Card: CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts. Related pages: zero-shot-learning, contrastive-learning.

Learning strategy

Change how training runs, without changing the data or the feature map 2103.03097 §IV.C.

• Ensemble learning. One classifier per source domain or expert, combined with learned weights (Mancini et al., DAEL). Costs more memory and compute 2103.03097 §IV.C.1. Related page: ensembles.
• Meta-learning. Split the sources into meta-train and meta-test sets to simulate a shift between domains during training. Examples: MLDG, MetaReg, MetaVIB 2103.03097 §IV.C.2. Related page: meta-learning.
• Gradient operations. RSC removes dominant features during training. Gradient inner-product maximisation and gradient-matching methods such as Fishr align per-domain gradients 2103.03097 §IV.C.3. Fish is a first-order way to reach a similar objective. Each meta-step copies the weights, takes one gradient step on the copy for each source domain in random order, and then moves the weights part of the way toward the copy. No second derivatives are needed 2104.09937 §3.4 2104.09937 alg. 1. Its inner loop has the shape of Reptile's, but it takes one batch from each domain instead of several from one task 2104.09937 app. A.1. Card: Gradient Matching for Domain Generalization. Related page: invariant-learning.
• Distributionally robust optimisation. Optimise the worst-case training domain, for example GroupDRO, VREx and AdaRNN 2103.03097 §IV.C.4.
• Self-supervised learning. Jigsaw pretext tasks and contrastive objectives, which can be added to other methods. Costs extra compute 2103.03097 §IV.C.5.
• Other. Metric learning, random forests, calibration, lottery-ticket substructures, and weight averaging for flatter minima (SWAD) 2103.03097 §IV.C.6.

When to use which

The survey does not give a single best method. Its benchmark uses one backbone and two image datasets, so the results below apply only to that setting 2103.03097 §VII.C. RobustNet's results cover urban-scene segmentation only 2103.15597 §5.1.1.

Benchmark averages from the survey's table, one run per cell:

• PACS: RSC and CORAL both average 82.0; ERM 81.2; DANN 81.4; ANDMask 79.9 2103.03097 §VII.C.
• Office-Home: RSC averages 63.7; CORAL 63.3; ERM 62.9; ANDMask 61.6 2103.03097 §VII.C.

RobustNet's table, author-reported and mIoU in percent. Each value is an average over three runs 2103.15597 §5.2.1. It is trained on GTAV with DeepLabV3+ and ResNet-50, and scored on Cityscapes (C), BDD-100K (B) and Mapillary (M) 2103.15597 tab. 1:

• ISW: C 36.58, B 35.20, M 40.33. Baseline: C 28.95, B 25.14, M 28.18.
• IBN-Net (re-implemented by the authors): C 33.85, B 32.30, M 37.75.

Fish's DomainBed table, author-reported test accuracy averaged over seven datasets, 5 trials, training-domain validation 2104.09937 tab. 4:

• Coral 67.2; Fish 67.1; Mixup 67.1; ERM 67.0.

Fish's main Wilds table, test set, author-reported 2104.09937 tab. 3:

• CivilComments worst-group accuracy: Fish 72.8; ERM 56.0.
• Camelyon17 average accuracy: Fish 74.7; ERM 70.3.

CLAP's zero-shot table, ViT-B/16 CLIP, averaged over PACS, VLCS, OfficeHome and DomainNet, author-reported 2311.16445 tab. 3. Bare class-name prompt: CLIP 77.4, Im.Aug 76.1, CLAP 79.9. Noised prompt: CLIP 70.4, Im.Aug 69.9, CLAP 76.1. The baseline is zero-shot CLIP, not a model trained on the sources, so these rows are not comparable with the ERM rows above.

These numbers come from different papers and settings: the survey's image benchmark, RobustNet's segmentation benchmark, Fish's DomainBed and Wilds runs, and CLAP's zero-shot runs. The graph numbers below come from the EERM paper, which uses its own setting. All are the authors' own runs:

• OGB-Arxiv, inductive, GPRGNN, 2014–2016 test interval: ERM 47.25 ± 0.55, EERM 49.88 ± 0.49 2202.02466 tab. 3. For GraphSAGE on the same interval, EERM is 41.55 against 42.09 for ERM 2202.02466 tab. 3.

Differences of half a point or less are within what one seed can move. The paper's own text calls the RSC gains unremarkable 2103.03097 §VII.C. On DomainBed, Fish's average is 0.1 points above ERM 2104.09937 tab. 4.

Implementation sketch

The common core is pooled training over the source domains, plus a term that each method adds to the loss. The paper's general form is a task loss plus a weighted regulariser on the feature map 2103.03097 §IV.B:

# Common core of DG training  [arxiv:2103.03097 §IV.B]
# Input:  minibatches from each source domain, weight lambda
# Output: scalar loss for one step

def dg_step(source_batches, lam):
    # Pool the batches of all source domains  [arxiv:2103.03097 §II.A]
    x, y, d = pool(source_batches)
    # Encode inputs with the shared feature map g  [arxiv:2103.03097 §IV.B]
    z = g(x)
    # Task loss on the shared classifier f  [arxiv:2103.03097 §IV.B]
    loss = cross_entropy(f(z), y)
    # Add the method's regulariser: alignment, invariance penalty,
    # gradient matching, a disentanglement term, or a masked covariance
    # penalty  [arxiv:2103.03097 §IV.B.1-2] [arxiv:2103.15597 eq. 18]
    loss = loss + lam * regulariser(z, y, d)
    return loss

Where the variants differ: the regulariser (alignment, invariance penalty, reconstruction, or a covariance penalty with a fixed mask), the batch sampling (pooled, per-domain, or worst-domain), the update rule (Fish replaces the single step with an inner loop over domains on a copy of the weights 2104.09937 alg. 1), and extra training stages for meta-learning 2103.03097 §IV.B 2103.03097 §IV.C. A selective whitening method computes its mask once, after a warm-up stage, and then keeps it fixed 2103.15597 §4.3. With no domain labels, the "sources" are edited copies of one graph, and the loss is taken over the copies' risks 2202.02466 §3.2. A text-only variant (CLAP) changes the inputs in a different way: it computes its contrastive terms on frozen text features of a prompt, an augmented copy of it, and the class name, and it reads no images during training 2311.16445 §4.2.

Contradictions

• RSC as best. The text says RSC is best on both benchmark datasets. The table shows RSC and CORAL tied on PACS at 82.0, so RSC is best only on Office-Home 2103.03097 §VII.C.
• DANN and ANDMask versus ERM. The text says they perform worse than ERM. The table shows DANN above ERM on PACS (81.4 vs 81.2), and both below ERM on Office-Home 2103.03097 §VII.C.
• Meta-learning notation. The meta update uses α as the outer step, but the text names η and β as the outer and inner learning rates. The survey points to the MAML and MLDG papers for the exact rule 2103.03097 §IV.C.2.
• Mapillary claim (RobustNet). The text says ISW is best on BDD-100K, Mapillary and SYNTHIA when trained on Cityscapes 2103.15597 §5.2.1. The Cityscapes table gives ISW 58.64 on Mapillary, below IW (58.90) and IRW (59.20) 2103.15597 tab. 2.
• IBN-Net claim (RobustNet). The supplement says ISW beats IBN-Net for every number of clusters k 2103.15597 app. A.2. On SYNTHIA, IBN-Net scores 27.90 2103.15597 tab. 1. Only k = 3 (28.30) is above it; k = 2, 5, 10 and 20 score 27.70, 27.52, 27.68 and 27.47 2103.15597 tab. 7.
• Fish's own tables. The main Wilds table gives Fish 74.7 on Camelyon17 and 72.8 on CivilComments worst-group accuracy. The appendix gives 79.5 and 74.2 for the same method and test sets, and gives ERM 73.3 and 58.0 where the main table has 70.3 and 56.0 2104.09937 tab. 3 2104.09937 app. C.2 2104.09937 app. C.4. Both places put Fish above ERM on these two datasets, but the size of the gap differs.
• Gain size across papers. The survey finds its best image method 0.8 points above ERM 2103.03097 §VII.C. Fish is 0.1 points above ERM on DomainBed 2104.09937 tab. 4 and several points above it on some Wilds datasets 2104.09937 tab. 3. EERM reports gains of up to 7.2% on Facebook-100 and 8.1% relative on OGB-Arxiv 2202.02466 §5.2 2202.02466 §5.3. RobustNet reports gains of 7.63, 10.06 and 12.15 points over its own baseline on C, B and M 2103.15597 tab. 5. The settings and baselines differ, so the numbers do not conflict directly. The survey's gains should not be read as typical for graphs or segmentation. CLAP's average zero-shot gain over CLIP is 2.5 points on four of the same benchmark datasets 2311.16445 tab. 3, but its baseline is a pretrained zero-shot model, not an ERM model trained on the sources.

Pitfalls

• Benchmark numbers are few runs. The survey reports one number per cell, with no seed count or deviation 2103.03097 §VII.C. RobustNet averages three runs but reports no spread 2103.15597 §5.2.1. Fish uses three seeds, and ten on Camelyon17 2104.09937 §4.2. Treat small differences as noise.
• Source accuracy can drop. ISW lowers the source column: 72.10 against 73.45 on GTAV 2103.15597 tab. 1, and 76.41 against 77.51 on Cityscapes 2103.15597 tab. 2. The paper says ISW is best on every target set except the source 2103.15597 §5.2.1.
• Training cost is not measured. The paper reports unchanged inference time and FLOPs 2103.15597 tab. 6. It does not measure training time, and training adds a warm-up pass, a variance estimate over photometric pairs, and the loss on each of three layers 2103.15597 §5.2.3.
• Mask depends on the statistics images. The selected entries depend on which images and photometric pairs are used. The paper does not give the number of pairs or trials. See the card's Paper vs code section.
• Model selection uses source data only. Under the training-domain protocol, the validation split comes from the sources. The survey says this protocol cannot always reach the best performance, while test-domain validation uses target data and so does not fit DG 2103.03097 §VII.B.
• Theory assumptions are narrow. The DG bounds assume covariate shift and, for some results, a kernel or VC-dimension setting. One bound assumes the target lies in the convex hull of the sources 2103.03097 §III.B.
• Image tasks only. The survey's benchmark covers image classification. RobustNet covers urban-scene segmentation. The survey lists other settings, but does not benchmark them 2103.03097 §VII.A 2103.03097 §VII.C.
• Dense editor memory (graph variant). The editor's edit matrix is over node pairs, and the paper gives no sparse or subsampled alternative 2202.02466 §3.2 2202.02466 app. A. The reference code keeps one n × n matrix per editor. At OGB-Arxiv size (169,343 nodes 2202.02466 tab. 1) the card estimates about 115 GB per editor in float32. Any large-graph use needs a design the paper does not describe. See the card's Paper vs code section.
• No code reuse for the graph variant. The authors' repository has no license file, so do not copy its code. Implement from the paper's equations 2202.02466 §3.2.
• Unstated graph settings. The number of environments K, the number of epochs and the optimisers are not stated in the paper, and the card lists where the reference code's values differ from the paper's 2202.02466 app. F.2 2202.02466 app. F.3.
• Reference code differs from the paper. The RobustNet repository uses other covariance normalisation, a different warm-up count, and per-setting cluster counts. Keep the code's conventions when reproducing its reported numbers see the card's Paper vs code section.
• Fish's update is not the paper's literal rule in the code. Algorithm 1 applies the outer rate directly and uses plain gradient steps in the inner loop 2104.09937 alg. 1. The released code divides the outer rate by the number of meta-steps and carries the ERM optimiser state across inner loops see the card's Paper vs code section. Match the code if you want its reported numbers, and say which one you used.
• The alignment holds only in a limit. Fish drops the second-order term on purpose. The paper's alignment result holds as the inner rate goes to zero, and the paper gives no error bound at the rates it uses 2104.09937 §3.4 2104.09937 app. A 2104.09937 tab. 14.
• Fish rates differ by dataset. Across the Wilds datasets the inner rate runs from 2e-6 to 1e-3 and the outer rate from 0.01 to 0.1 2104.09937 tab. 14. Copy them per dataset, not as one global value.
• Fish speed claim is unmeasured. The paper says Fish is about ten times faster than direct IDGM, but gives no timing table 2104.09937 §4.1. Each meta-step costs one gradient step per source domain it visits 2104.09937 alg. 1.
• Pretraining matters for Fish too. Several Wilds runs start from an ERM checkpoint, and results change with how far that checkpoint converged 2104.09937 app. E 2104.09937 tab. 15. Give the baseline the same starting point.
• CLAP's α was chosen on evaluation data. The paper does not say how the inference weight α was picked. The reference code's search scores zero-shot accuracy on the test environments, so the reported gains may be optimistic. Use a held-out split see the card's Paper vs code section 2311.16445 app. A.1.
• CLAP's OfficeHome and DomainNet settings are appendix-only. The temperature of 0.3 and the class-term weight of 0.0001 for those datasets are in the appendix, and the shipped training config covers VLCS only 2311.16445 app. A.1. Carry the dataset values explicitly.
• CLAP's loss differs from its equation in code. The code uses a two-directional InfoNCE with same-view negatives, not the one-directional form in 2311.16445 eq. 2. With few classes, a batch repeats class names, and the code masks only the diagonal, so same-class pairs count as negatives see the card's Paper vs code section.
• No code reuse for CLAP. The repository has no LICENSE file, so implement from the paper's equations 2311.16445 §4.2.

Papers

• Generalizing to Unseen Domains: A Survey on Domain Generalization: survey of DG methods in three groups, the theory of the problem, the dataset list, and a benchmark of seven methods on PACS and Office-Home with ResNet-18.
• RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening: one-source DG for urban-scene segmentation, penalising the style-sensitive covariance entries of instance-normalised layers (ISW) during training only.
• Gradient Matching for Domain Generalization: Fish, a first-order approximation of inter-domain gradient matching that runs one inner gradient step per domain on a copy of the weights; evaluated on Wilds and DomainBed.
• Handling Distribution Shifts on Graphs: An Invariance Perspective: node-level DG on graphs, with learned edge edits that create virtual environments (EERM) and a mean-plus-variance risk objective.
• CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts: a text-only residual MLP on frozen CLIP features, trained contrastively on prompts and style-dropped copies, then reused on image features at inference.