Domain adaptation

Overview

A classifier trained on one data distribution often degrades on another: a digit reader trained on street-number photos misreads clean handwritten digits, and a sentiment model trained on restaurant reviews misreads laptop reviews. Domain adaptation tries to carry the labeled knowledge across without labeling the new domain. The source domain has labels, the target domain may have none 1611.02200 §1.

The cards share one idea: some signal links the two domains, and the method uses it. In the translation approach it is a fixed feature function f that should give the same output before and after the mapping [arxiv:1611.02200 §1]. In the segmentation approaches the signal is the unlabeled target image itself, through a loss on its predictions 1909.13589 §3.2. In the disentangling approach it is the image's layout, which is assumed to be shared across domains while texture is not 1903.12212 §1. In the language-model approaches it is unlabeled domain text, used for further pre-training 2402.10373 §3.2 2311.00176 §2.2 [arxiv:2005.07503 §2] 2311.16079 §4.1 2109.15290 §2.2. In the self-supervision approach it is the unlabeled images themselves, through puzzles whose labels the method makes up 1909.11825 §4. In the aerial translation approach it is the unpaired target images that train the translator 1905.03198 §4.1. In the transport approach it is the match between source and target minibatches, scored by features and predicted labels 2103.03606 §5.2. In the federated approach it is each user's own data, whose feature statistics are aligned to the shared model's [arxiv:1907.09173 §3.4]. The fault-diagnosis and manifold cards use the unlabeled target features directly, through a maximum mean discrepancy (MMD) penalty on pseudo-labels 1804.07265 §III-A or on a kernel fitted to both domains [arxiv:1807.07258 §3.4]. The source-free card uses the unlabeled target images, with the source model's weights as an anchor and no source data 2502.19316 §3.2. A contrastive test-time card uses the same kind of input, the unlabeled target stream, and refines its pseudo labels in a memory queue [arxiv:2204.10377 §3.1]. The SciFact card is the exception. It needs no unlabeled target signal: it trains on a general fact-checking set and then on the labeled scientific set 2004.14974 tab. 3. In the content-moderation approach it is the unlabeled production traffic, which a Wasserstein critic compares with labeled public data on the encoder's embeddings 2208.03274 §3.4. In the vehicle-perception approach it is unlabeled real LiDAR from the target vehicles, which a domain-adversarial head compares with labeled simulated scans 2303.07601 §4.3. What differs between variants is where the adaptation happens: in the data, the loss, the features, the fine-tuning recipe, the merged weights, or at test time for each input.

The thirty-three cards cover the approaches below. Their evidence is uneven. Some papers test on one benchmark pair, some report only internal data, two are surveys with no experiments of their own, one survey runs a single experiment, one dataset paper runs one Sim2Real adaptation benchmark, and one is a benchmark that compares training sources but proposes no adaptation method. Read the "When to use which" table before choosing a variant.

Domain generalization is a neighbouring setting. A model sees several source domains and no data from the target, not even unlabeled, so it cannot adapt to the target directly 2103.03097 §II.A §III.A. The cards keep it separate; see the section below.

Variants

Data-based: translate source samples into the target domain

The Domain Transfer Network (DTN) learns a map G from the source domain S to the target domain T. It uses unpaired samples from both domains and a fixed, pretrained feature function f, and it is trained so that f(x) and f(G(x)) stay close 1611.02200 §1, §3. G is built as g after f, so only the trainable part g decodes f's representation 1611.02200 §4. Training combines a three-way discriminator, a feature-constancy loss on source samples, an identity loss on target samples and a smoothness term 1611.02200 eq. 3–7. Adaptation then works as follows: G carries the labeled source set into T, the source labels are kept, and a classifier is fit on the translated samples. The paper uses a nearest-neighbour classifier for this step 1611.02200 §2, §5.1.

On SVHN to MNIST, the author-reported accuracy is 84.44% when the train split of the source is transferred. The paper's comparison points are 73.85% for DANN, 59.32% for SA, and 79.72% when transferring the SVHN test split 1611.02200 tab. 2. Ablations on the same task show that removing the identity loss drops accuracy to 88.40% and removing the feature-constancy loss to 74.55%, with 90.66% for the full model 1611.02200 tab. 1. The paper also reports that the target set must contain every class. When class 3 is absent from T, accuracy on that class falls to 40.13% [arxiv:1611.02200 tab. 3]. Card: Unsupervised Cross-Domain Image Generation.

Data-based: pseudo-label target pairs with a slow, accurate model

Augmented SBERT (AugSBERT) is a text version of the same idea, but the labels come from a model rather than a generator. A BERT cross-encoder is fine-tuned on a labeled source domain. It then scores unlabeled target-domain sentence pairs, and a fast bi-encoder is trained on those scores 2010.08240 §3.2. The scores are soft labels, not hard ones 2010.08240 §3.1. Which pairs are scored matters. Random pairs are mostly negatives 2010.08240 §3.1. BM25 gives the smallest silver sets and the best average score, which the paper credits to its computational efficiency 2010.08240 §6.1.

On the domain-adaptation table, AugSBERT's AUC(0.05) for Quora → Sprint is 0.875, against 0.505 for SBERT trained on the source only. That is the "up to 37 points" headline 2010.08240 tab. 6. It is not a uniform gain. For Quora → AskUbuntu it reaches 0.602, against 0.715 for SBERT trained in-domain on the target 2010.08240 tab. 6. On Sprint as target, the paper's own Bi-LSTM baselines are level with or above it, the exception the paper itself notes 2010.08240 §6.2. The public example scripts train on the English STS Benchmark and label Quora pairs, so they do not reproduce the paper's setup README.md. Card: Augmented SBERT.

Data-based: cycle-consistent translation, then fine-tune the segmenter

Benjdira et al. adapt a segmenter trained on Potsdam aerial tiles to Vaihingen tiles. A CycleGAN-style pair of translators, one for each direction, is trained on unpaired patches from both sites 1905.03198 §4.1. Its objective has an adversarial term per direction and a reconstruction term that asks each round trip to return the input 1905.03198 eq. 8. The source-to-target translator then restyles every labelled source image, and the source labels are kept 1905.03198 §4.2. The segmenter is fine-tuned on the restyled images, starting from its source weights 1905.03198 §4.3. Unlike the DTN pipeline above, no new classifier is fitted on the translated set. The source model is adapted in place.

The source segmenter is a BiSeNet with a ResNet-101 feature extractor, trained with Adam at a learning rate of 0.0001 1905.03198 §5.2.1. The translators use 512 by 512 patches, and the paper does not state a total loss weight or an identity term 1905.03198 §5.2.2 §4.3.

On the target, average accuracy rises from 0.35 before adaptation to 0.52 after, and IoU from 0.17 to 0.30 1905.03198 tab. 3. The gain is uneven by class. Building accuracy goes from 0.23 to 0.71 and Tree from 0.06 to 0.51, while Low vegetation falls from 0.38 to 0.27 1905.03198 tab. 4. The text says the other classes are unaffected 1905.03198 §6, which the Low vegetation row does not support. The paper measures the target after each epoch and reports the gain at epoch 8 1905.03198 §5.2.4. That choice uses target test labels, so the reported best-epoch numbers are optimistic. Results are for one source and one target pair with one segmenter. Card: Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images.

Loss-based: maximum squares on unlabeled target pixels

For segmentation with a labeled synthetic source (GTA5 or SYNTHIA) and an unlabeled real target (Cityscapes), the objective is the source cross-entropy plus a target term weighted by λ_T 1909.13589 §3.1 eq. 1. The common target term is entropy minimization. Its gradient is largest on the pixels the model already predicts confidently, so those pixels dominate the update 1909.13589 §1 §3.2. The paper replaces that term with the negative sum of squared target probabilities, normalised per image [arxiv:1909.13589 §3.2] eq. 6. It needs no discriminator network 1909.13589 §1.

Two add-ons follow. Image-wise class weights use the predicted class counts of each target image, with a hyperparameter α 1909.13589 §3.3 [eq. 13]. A second, low-level head is trained on confident guidance labels built from both heads' predictions, with threshold δ and weight λ_low [arxiv:1909.13589 §3.4] §4.2.

On GTA5 to Cityscapes (mIoU over 19 classes, DeepLabv2 with ResNet-101), the author-reported numbers are 42.2 for their entropy-minimization reproduction, 44.3 for maximum squares, 45.2 with image-wise weights and 46.4 with the low-level head 1909.13589 tab. 2 tab. 4. On SYNTHIA to Cityscapes, the full method reaches 41.4 over 16 classes, against 38.0 for entropy minimization 1909.13589 tab. 3. The paper also reports classification with ResNet-50 on Office-31: 85.6 average for maximum squares, 84.1 for entropy minimization, 82.2 for DANN and 76.1 for the source-only model 1909.13589 tab. 1. The paper calls those classification results uncompetitive with the state of the art 1909.13589 §4.3. Card: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss.

Loss-based: label-free puzzles on both domains

Sun et al. train one shared network on the labeled source task and on self-supervised tasks that use both domains. The tasks are rotation (4-way), vertical flip (binary) and patch location, and each head is a linear layer on the shared features 1909.11825 §3 §4. The source labels are dropped in the puzzle losses, and the target images only feed those losses 1909.11825 eq. 1. The total loss is an unweighted sum of the source task and the puzzle losses 1909.11825 eq. 2. The paper uses no minimax or adversarial term §4. The puzzle heads are discarded at test time §4.

The appendix trains one puzzle at a time, one gradient step per task on balanced batches of 64 source and 64 target images, then one supervised step on source 1909.11825 app. B. Per source batch, the cost is one update for each puzzle and one for the classifier.

The paper reports best accuracy on four of seven digit and natural-image benchmarks. On MNIST to MNIST-M, rotation alone reaches 98.9 against 44.9 for source-only. On MNIST to USPS it reaches 96.5 against 94.7 [arxiv:1909.11825 tab. 2]. On CIFAR-10 to STL-10 the full set of tasks reaches 82.1, against 75.6 for source-only 1909.11825 tab. 2. The paper states that its method fails on the SVHN benchmarks, where rotation gives trivial solutions because the crop shows neighbouring digits 1909.11825 §5.1. On GTA5 to Cityscapes the mIoU is 28.9 against 25.3 for source-only, and 39.5 for CyCADA 1909.11825 tab. 3. The epoch is picked by a score that adds the normalised feature-mean distance on unlabeled data to the normalised source error 1909.11825 §4.1 eq. 3. The public code selects the checkpoint on the test splits, not on validation data train.py. Card: Unsupervised Domain Adaptation through Self-Supervision.

Loss-based: minibatch unbalanced optimal transport (JUMBOT)

Fatras et al. align source and target minibatches with an optimal transport plan and train the network on the plan-weighted cost 2103.03606 §5.2 eq. 10. Each pair of samples costs a squared feature distance plus the cross-entropy of the source label under the target prediction 2103.03606 eq. 9. Balanced transport must send all mass, so inside a small minibatch it pairs samples that do not correspond. The method replaces it with unbalanced transport, which charges a KL penalty on the marginals and adds entropic regularisation 2103.03606 §2 eq. 2. The paper shows that the minibatch estimator stays unbiased and that its gradients behave under this change 2103.03606 §4.1 §4.3. The paper reports state-of-the-art results with one pair of minibatches per step (k = 1) 2103.03606 §5.2. Source minibatches are class-stratified 2103.03606 app. C.

The settings differ by task. For Office-Home the paper uses feature weight 0.01, label weight 0.5, transfer weight 1, τ = 0.5, ε = 0.01 and minibatch 65. For digits it uses τ = 1, ε = 0.1 and minibatch 500 2103.03606 app. C. The same method covers partial domain adaptation, where the target has fewer classes 2103.03606 §5.3.

On Office-Home DA, the end-of-training average is 70.0, against 66.6 for CDAN-E and 63.5 for DeepJDOT, both reproduced by the authors 2103.03606 tab. 1. On digits the average is 98.0, against 96.5 for ALDA 2103.03606 tab. 2. On VisDA the score is 72.5, against 70.5 for ALDA 2103.03606 tab. 3. The appendix reports Office-Home again with the best score over epochs, where JUMBOT gets 70.1 and CDAN-E 67.0 2103.03606 app. C, Table 4. That protocol is not the one of the main table, so the two should not be mixed. Card: Unbalanced minibatch Optimal Transport; applications to Domain Adaptation.

Loss-based: MMD penalty on features, marginal and per class (JDA in a CNN)

Han et al. call their model a deep transfer network (DTN). It is not the Domain Transfer Network above. A CNN is trained on labeled source data. Its loss then adds lambda times a joint distribution adaptation (JDA) term 1804.07265 eq. 11. The JDA term is the maximum mean discrepancy (MMD) between all source and target features, plus one MMD for each class between source samples of that class and target samples with that pseudo-label 1804.07265 eq. 10. The pseudo-labels come from the current network and are recomputed each round 1804.07265 alg. 1. The paper writes the marginal and class-wise terms with weight 1 each, so lambda is the only scale 1804.07265 eq. 10.

The paper also tests the marginal-only version, called DTN with MDA. It drops the class-wise terms 1804.07265 §III-D. Across ten fault-diagnosis transfer tasks on three data sets, the average accuracy is 97.5 for DTN with JDA, 87.4 for DTN with MDA and 80.0 for the plain CNN [arxiv:1804.07265 tab. III]. These are the authors' numbers, on one fault-diagnosis setting. The source and target label spaces are assumed equal 1804.07265 §III-A. The lambda grid is listed in the card's hyperparameters, but the paper does not say which data chose it. Treat the figures as optimistic until that is clear 1804.07265 §IV-A. The paper names no code repository, and a search found no official code 1804.07265 §IV-A. Card: Deep Transfer Network with Joint Distribution Adaptation.

Feature-based: split structure from texture

DISE (Domain-Invariant Structure Extraction) encodes each image into two codes. A structure code, shared by both domains, drives the segmentation classifier. A texture code, private to each domain, carries colour and surface look 1903.12212 §3.1. A decoder rebuilds images from both codes. Swapping the texture codes between a game image and a real one gives a translated image that keeps the game layout but has the real look 1903.12212 §3.1. Those translated images carry the source labels as pseudo-labels, because segmentation is assumed to depend only on structure 1903.12212 §3.1 §3.2. The target labels are never used in the loss 1903.12212 §3.2. The model has five sub-networks, plus discriminators on the output space and on the translated images 1903.12212 §3.3 §3.2.

The author-reported result on GTA5 to Cityscapes is 45.4 mIoU with a DeepLab-v2 ResNet-101 base, against 42.4 for the output-space adaptation baseline in the same table 1903.12212 tab. 2. The loss weights are not stated in the paper 1903.12212 eq. 1. Card: All about Structure.

Feature-based: gradient-reversal domain loss across two modalities

Dey et al. train two encoders, one for sketches and one for photos, into a shared embedding space 1904.03451 §4.2. A small classifier tries to tell which modality an embedding came from. A gradient-reversal layer sits in front of it, so the encoders learn embeddings the classifier cannot separate 1904.03451 §4.3 eq. 2. The reversal weight is zero for the first few steps and then ramps up 1904.03451 alg. 1. The method is domain adaptation in the same adversarial form, but the two domains are two input types, not a labelled source and an unlabelled target. The test categories are unseen in training, so the setting is zero-shot retrieval 1904.03451 §4.1.

In the paper's ablation, adding the domain term to the attention model moves mAP from 0.3207 to 0.3256 on Sketchy-Extended, from 0.0729 to 0.0845 on TU-Berlin-Extended and from 0.0456 to 0.0651 on QuickDraw-Extended 1904.03451 tab. 3. The gain is largest on the new QuickDraw benchmark. The full model, which adds the semantic loss, reaches 0.3691 on Sketchy-Extended 1904.03451 tab. 3. The TU-Berlin discussion says results move a lot across random splits 1904.03451 §5.2. The card records that the released code restarts the reversal schedule each epoch, while the paper ramps it over all iterations 1904.03451 §4.3. Card: Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval.

Feature-based: Wasserstein domain critic on production traffic

Markov et al. adapt a content classifier from labeled public data to unlabeled production traffic. A transformer encoder feeds a classification head and a critic head. The critic maps each embedding to a real number, and the training objective adds the gap between the critic's mean output on the two domains to the classification loss 2208.03274 §3.4. This is Wasserstein domain-adversarial training (WDAT). The weight on the gap is λ = 0.01, and the critic weights are clipped to [-0.01, 0.01] to keep the critic Lipschitz 2208.03274 §3.4. The authors chose it over a classifier-based domain adversary, reporting more stable training and better production results. The paper does not state the critic's architecture beyond a domain critic head, and the optimizer and schedule are also not stated 2208.03274 §4.1.

The effect is measured on a production validation set, with three seeds and no spread reported 2208.03274 §4.4. With public data only (PUB, about 90k labeled examples), WDAT raises average AUPRC in all eight categories, by 0.03 (S) to 0.29 (V2). The largest gains are self-harm, from .063 to .281, and V2, from .165 to .453 2208.03274 tab. 5. When 69k curated synthetic examples are added (SYN), WDAT is higher in five categories. It is lower for H, from .551 to .476, which is a larger drop than the paper's "slightly hurt" wording suggests 2208.03274 §4.4 tab. 5. When 60k labeled production examples are also added (MIX), WDAT is lower for S, H and V and higher for the other five, by 0.003 to 0.09 2208.03274 tab. 5. So the gain is largest when labeled in-distribution data is scarce, and it is small or negative once that data is present. This matches the paper's own reading 2208.03274 §4.4.

The paper's active-learning loop picks which production samples to label. That is a separate choice from the adversarial term, and it belongs to the active-learning concept. Its numbers are author-reported, and the production test set is not shared 2208.03274 §4.2 §4.3 tab. 4. Card: A Holistic Approach to Undesired Content Detection in the Real World.

Feature-based, Sim2Real: domain-adversarial training for cooperative LiDAR detection

V2V4Real releases a real two-vehicle LiDAR dataset and defines a Sim2Real benchmark on it. The source is the labeled simulated OPV2V data and the target is the unlabeled real LiDAR of the training split 2303.07601 §4.3. The adapted model adds two discriminators to a cooperative detector, in the style of a domain-adversarial network 2303.07601 §4.3. One sits on the fused feature map and has two 3×3 convolutions. The other sits on the classification score map and has three linear layers with ReLU. Both use binary cross-entropy and a gradient-reversal layer 2303.07601 app. C.3. The paper does not state the discriminator loss weight or the reversal scalar 2303.07601 app. C.3.

The benchmark reports AP@0.5 on the labeled real test split. The no-adaptation baseline trains on OPV2V and tests on real data. On this comparison, the domain-adversarial variant raises CoBEVT from 32.6 to 40.2 AP@0.5, a gain of 7.6 2303.07601 tab. 5. Gains differ by fusion method: 0.9 for AttFuse, 13.7 for F-Cooper, 3.1 for V2VNet and 12.1 for V2X-ViT [arxiv:2303.07601 tab. 5]. The text reports an average gain of 7.46 points, while the table values average 7.48, so the rounded table values should not be quoted as the paper's average 2303.07601 §5.4. The authors also compare the adapted CoBEVT with a no-fusion model trained on the real data, 40.2 against 39.8 2303.07601 §5.4 tab. 3. That comparison uses one run per row with no spread reported, so it is a close call and not a clear win.

The paper's scope is narrow. The benchmark uses LiDAR only and one merged vehicle class, and the source simulator's version and split are not given 2303.07601 §4.1 §4.3. Its numbers are for cooperative detection only. The dataset is not openly licensed, so check its terms before reuse 2303.07601 §3. Card: V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception.

Feature-based: manifold alignment with a closed-form classifier (MEDA)

Manifold Embedded Distribution Alignment (MEDA) works on fixed feature vectors, such as SURF or DeCAF6 features. It first maps both domains onto a geodesic flow kernel between their subspaces 1807.07258 §3.3 eq. 2. Then it mixes a marginal MMD and a sum of per-class MMDs with a weight mu 1807.07258 eq. 3. The mu is estimated from A-distances on the unlabeled target, and it is recomputed each round 1807.07258 §3.4 [eq. 5] eq. 6. The classifier is fitted in one closed-form linear solve, not by a training loop 1807.07258 §3.5 eq. 16. The target labels start from a base classifier and are refined over T rounds 1807.07258 alg. 1. The paper sets T = 10 1807.07258 §4.3.

Author-reported accuracy over 28 tasks is 73.2, against 69.7 for JGSA, the best listed baseline 1807.07258 §4.4. The gain varies by feature set. On Office+Caltech10 with SURF it is 52.7 against 50.0 for JGSA [arxiv:1807.07258 tab. 2]. On USPS/MNIST and ImageNet/VOC it is 75.9 against 72.8 for ARTL 1807.07258 tab. 3. On Office+Caltech10 with DeCAF6 it is 92.8 against 90.7 for ARTL 1807.07258 tab. 4. Removing the manifold step lowers the accuracy in each of the six tasks of that table, and on C→A with SURF the average rises from 44.9 to 56.5 1807.07258 tab. 5. The paper does not show a seed spread for the headline numbers. Its standard deviations are over mu, not over runs 1807.07258 tab. 5.

Cost grows fast with data size. The method solves a system of size n plus m each round. The paper reports 2,931.7 seconds for ImageNet to VOC with 10,717 samples, against 2,648.8 for ARTL on the same task 1807.07258 tab. 7. It gives no memory figure.

The paper's equations are not all the released code's choices. The MATLAB code normalises the mixed matrix, uses a different Laplacian, clips mu to [0, 1] and starts from 1-NN labels 1807.07258 eq. 10 eq. 14 §3.4. The paper does not state these choices, and the repository has no license, so the card quotes no code. Card: Visual Domain Adaptation with Manifold Embedded Distribution Alignment.

Test-time: adapt to each input

Test-time training (TTT) with a masked autoencoder (MAE) takes a different route. There is no pool of target data. The paper assumes the shift is unknown in advance and that the test input is the only information about it 2209.07522 §1 §5. For each test image the procedure resets the pretrained weights, runs 20 SGD steps on masked-patch reconstruction of that one image, predicts on the unmasked image, and then discards the adapted weights 2209.07522 §3 §4.1. The main SGD step uses momentum 0.9 and a fixed learning rate of 5e-3 at batch 128, where the batch is 128 copies of the same image with independent masks 2209.07522 §4.1.

On ImageNet-C at severity level 5, the author-reported accuracy improves over the fixed ViT-probing baseline on all 15 corruptions. For JPEG it goes from 51.2% to 63.0% 2209.07522 tab. 2. The paper also reports that the earlier rotation-based TTT hurts on rotation-invariant classes, which the masked-reconstruction task does not 2209.07522 §2 tab. 1. The cost is paid at every prediction: 20 steps over 128 copies per image, and the paper gives no wall-clock time 2209.07522 §4.1 [Limitations]. Card: Test-Time Training with Masked Autoencoders.

Generator adaptation from text, with no target images

StyleGAN-NADA shifts a pretrained StyleGAN2 generator to a new domain from a pair of text prompts, such as "dog" to "cat" 2108.00946 tab. 2. A frozen copy of the generator supplies source images, and a trained copy is updated. Both share one frozen mapping network 2108.00946 §4.1 fig. 2. The loss is one minus the cosine between the CLIP-space change from source to target image and the change from source to target text 2108.00946 eq. 2. A global CLIP loss on the target text alone is rejected, because it leads to adversarial pixel patterns and mode collapse 2108.00946 §4.1 app. B. For larger shape changes, the trained layers are chosen at each iteration by a short latent optimisation, and only the top k layers are updated 2108.00946 §4.2 app. I.

The authors' default recipe uses Adam at lr 0.002 with batch size 2, about 300 iterations for texture changes (roughly three minutes on one V100), and 2000 iterations plus a latent mapper for animal changes [arxiv:2108.00946 app. I]. On FID with 10 target images, the Ojha et al. few-shot baseline goes from 45.13 to 41.22 with zero-shot pre-training, and with 100 images from 27.54 to 16.17 2108.00946 tab. 2. The quality user study reports a preference for this method over most 5- and 10-shot baselines, without per-method percentages in the text 2108.00946 §5.1 fig. 12. The paper says the method works best for style and fine detail, and may struggle with large geometry or scale changes 2108.00946 §6. Card: StyleGAN-NADA.

Source-free: a class-conditional generator on the source model alone

3C-GAN adapts a classifier when its source data is not available at all. The inputs are the source-trained classifier and unlabeled target images 2502.19316 §1 §3. A class-conditional generator and a discriminator are trained against the target images. The generator must make images that look like the target and that the classifier assigns to the conditioning label 2502.19316 §3.1 eq. 2 eq. 3. The classifier is then trained on those labeled generated samples 2502.19316 §3.1 eq. 5.

Two penalties keep the classifier in place. A squared distance to the source weights, which becomes plain L2 weight decay when the source weights are zero, keeps it near the source model 2502.19316 §3.2 eq. 6. A clustering term combines a conditional entropy on target images with a perturbation-based consistency term 2502.19316 §3.3 eq. 7 eq. 8 eq. 9. The paper does not state the generator's semantic-loss weight λ_s, the perturbation radius, the batch size or the epoch count 2502.19316 §4.1 eq. 4 alg. 1.

On digits, the author-reported accuracy is 99.4 for SVHN to MNIST, against 76.4 for the source-only model. For MNIST to USPS it is 97.3, against 92.4 2502.19316 tab. 1. On Office-31 with ResNet50 the average is 89.6, against 76.1 for the source-only model 2502.19316 tab. 2. On VisDA17 with ResNet101 the class-mean accuracy is 81.6, against 52.4 for the source-only model 2502.19316 tab. 3. The SVHN to MNIST result ties the DIRT-T figure of 99.4 that the table lists as the best other method [arxiv:2502.19316 tab. 1]. In the digit ablation, removing the generator's classifier loss stops training from converging 2502.19316 tab. 4 §4.4. The paper's own prose gives some figures that do not match its tables, so use the tables 2502.19316 §4.2. The paper names no code, and the card quotes none. Card: Model Adaptation: Unsupervised Domain Adaptation without Source Data.

Source-free, streaming: pseudo labels refined in a memory queue

AdaContrast adapts a source-trained ResNet to an unlabeled target set, with no source images during adaptation 2204.10377 §1. The target model starts from the source weights, and a momentum copy follows it as a moving average 2204.10377 §3.1 eq. 2. Each target mini-batch gives one weak view and two strong views of every image 2204.10377 §3. The pseudo label of the weak view is the argmax of the averaged probabilities of its N nearest entries in a queue of earlier features. The paper refines it at every step rather than once per epoch 2204.10377 §3.1 eq. 3 eq. 4. A contrastive loss pulls the two strong views together and drops negatives whose pseudo label matches the query's 2204.10377 §3.2 eq. 5 eq. 6. The total loss is a cross-entropy on the strong view, that contrastive term and a term on the mini-batch mean prediction, all with weight 1 and no tuning in any experiment 2204.10377 §3.3 eq. 10.

On VisDA-C train to validation, the author-reported per-class average is 86.8, against 83.0 for SHOT, a source-free baseline 2204.10377 tab. 1. On DomainNet-126 over seven shifts it is 67.8, against 67.1 for SHOT 2204.10377 tab. 2. The paper does not say how many runs each table averages, so the 0.7-point gap on DomainNet-126 is weak evidence 2204.10377 §4. The learning-rate sweep is more telling. At 10 times the base rate, SHOT falls to 72.1 on VisDA-C, while AdaContrast stays at 86.6 2204.10377 tab. 3. Removing the contrastive term at that rate drops VisDA-C to 9.9, and removing the diversity term gives 84.3 against 86.6 2204.10377 tab. 4. The queue size matters too, with 86.3 at 2048 entries against 84.4 at 512 2204.10377 §4.3. The paper's text calls 78.7 the overall accuracy of its online mode, while table 1 lists it as a per-class average, so read that label from the table 2204.10377 §4.2 tab. 1.

The official code differs from the paper in several places. It uses 10 neighbours by default, not 11, and fills the queue from a deterministic transform rather than weak augmentations 2204.10377 §4.3 [code: configs/learn/target.yaml#L18] target.py. It updates the queue before building the negative mask, which changes which negatives are used target.py. The knowledge-distillation and online modes are not in the repository 2204.10377 §4.2. The repository has no LICENSE file, so its terms are unstated and the card cites it by path only. Implement from the paper's equations, which the main loss matches [code: target.py#L398-L402]. Card: Contrastive Test-Time Adaptation.

Continued pre-training on domain text

Continued pre-training starts from base weights and trains further on unlabeled domain text with the language-model loss. No task labels are used at this stage 2402.10373 §3.2.

BioMistral continues pre-training Mistral 7B Instruct v0.1 on 3 billion tokens of open-access medical articles from PubMed Central [arxiv:2402.10373 §3.1] §3.2. It uses AdamW at a learning rate of 2e-5 with a cosine schedule 2402.10373 §3.2. The result is mixed. On the ten English few-shot tasks, the average is 50.3 for BioMistral against 51.2 for Mistral 7B Instruct. BioMistral is higher on 8 of the 10 tasks, and the average is pulled down by PubMedQA, 37.6 against 72.2 2402.10373 tab. 2. The paper's best variant, a merge of the two models with SLERP, reaches 55.4 [arxiv:2402.10373 tab. 2]. Card: BioMistral.

ChipNeMo applies the same step to LLaMA2 at three sizes, on a mix of internal chip-design material and a sampled public share of about 9% of training tokens 2311.00176 §2.2 app. Training Data. It adds about 9K tokens to the tokenizer first 2311.00176 §2.1 §3.1, uses a learning rate of 5e-6 for about 23,200 steps at batch 256 2311.00176 §2.2 [app. DAPT Hyperparameters], and runs alignment afterwards 2311.00176 §2.3. Its token counts disagree across the paper 2311.00176 §2.2 [app. Training Data]. The authors report that DAPT on a chat-aligned model badly degraded alignment, so DAPT has to start from base weights [arxiv:2311.00176 §2.3]. Cost at 70B is 20,500 A100-hours 2311.00176 §3.5. On the authors' internal design-knowledge multiple-choice set, ChipNeMo-70B scores 76.6 against 58.4 for GPT-4. On the scripting set, GPT-4 is ahead, 77.4 against 73.9 2311.00176 app. Auto Evaluation Results. These benchmarks are internal and not released 2311.00176 §3.7. Card: ChipNeMo.

COVID-Twitter-BERT (CT-BERT) continues BERT-Large, uncased with whole-word masking, on 22.5M cleaned COVID-19 tweets, using the masked-token and next-sentence objectives 2005.07503 §2 §3.1. Sequences are at most 96 tokens, batches hold 1024 of them, and the paper stops at 500,000 steps 2005.07503 §2 §3.1. The run took 120 hours on one TPU v3-8 2005.07503 §2. Each downstream set is then fine-tuned with learning rate 2e-5 2005.07503 §2.1. The authors say that recipe was tuned on BERT-Large and that CT-BERT may do better with its own tuning 2005.07503 §2.1 §4.

Author-reported mean F1 over five classification sets is 0.833, against 0.802 for BERT-Large 2005.07503 tab. 2. The paper's relative-gain measure is largest on the COVID-19 category set, at 25.88%, and smallest on SemEval, at 8.97% 2005.07503 tab. 2 §3.2. SST-2 is not Twitter text and serves as a control, where the score moves from 0.937 to 0.944 2005.07503 §2.1 tab. 2. The released pretraining script uses warmup and linear decay, where the paper uses a constant rate, and it runs fewer steps than the paper run_pretrain.py run_pretrain.sh. Pick one setting and report which.

MEDITRON continues Llama-2 at 7B and 70B on GAP-Replay, about 48 billion tokens of clinical guidelines, PubMed abstracts and PubMed Central papers. About 1% of the mix is general text from RedPajama, to limit forgetting 2311.16079 §2 tab. 1. The 70B run uses peak learning rate 1.5e-4, global batch 512 and 2000 warmup steps, on 128 A100 80GB GPUs for 332 hours 2311.16079 §3 §4.1 app. A. The end learning rate is stated more than one way 2311.16079 app. D.1.

Each benchmark then gets its own supervised finetune, and the paper compares a top-token answer, chain-of-thought and self-consistency over five sampled answers 2311.16079 §4.2 §4.3. Few-shot averages rise from 60.8 for Llama-2-70B to 63.3 for MEDITRON-70B 2311.16079 tab. 4. After finetuning, self-consistency gives 72.0 against 69.2 for Llama-2-70B, and the top-token mode gives 69.0 against 67.2 2311.16079 tab. 5. The data-mix choices were tested only with 7B models 2311.16079 §7.2 tab. 8. The responsible-AI appendix tests truthfulness, risk and bias, and the authors advise testing in each use case before medical deployment [arxiv:2311.16079 app. E]. Card: MEDITRON-70B and COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter.

MatSciBERT continues SciBERT, not a general model, on about 285M words of materials-science full text and abstracts 2109.15290 tab. 1 §2.1. It starts from SciBERT's uncased weights and uses a RoBERTa-style recipe with whole-word masking and no next-sentence loss 2109.15290 §2.2. The authors report about 10 days on two V100 GPUs 2109.15290 §2.2. The corpus itself is not released, so only the released checkpoint can be reused 2109.15290 §2.1 run.sh.

On the authors' test sets, the BiLSTM-CRF head reaches 65.92 Macro-F1 on SOFC-Slot against 61.68 for SciBERT, and abstract classification F1 is 93.57 against 91.04 2109.15290 tab. 2 tab. 6. Gains are smaller on other tables. On Matscholar Macro-F1 the two models tie at 85.66 for the BiLSTM-CRF head, and relation classification is 91.26 against 91.04 Micro-F1 2109.15290 tab. 5 tab. 7. The baseline is SciBERT alone, and the MatSciBERT path normalises its text while the SciBERT path does not. Part of the gain may therefore come from preprocessing ner.py. The numbers are means over three seeds with no spread shown [arxiv:2109.15290 §2.4.4] tab. 2. Card: MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction.

A legal-domain survey makes the same point from the evidence side. It lists domain pretraining and retrieval as ways to adapt LLMs to legal text. It says domain pretraining gives legal vocabulary but stale knowledge, and that retrieval can miss the right source or cite the wrong authority [arxiv:2303.09136 tab. 5]. It proposes no experiments of its own. Card: A Short Survey of Viewing Large Language Models in Legal Aspect.

Retriever and generator: joint training on the target domain

In retrieval-augmented question answering, the retriever is a second model that the adaptation can also change. RAG-end2end trains the DPR question and passage encoders together with the BART generator on domain data. Every change to the passage encoder makes the stored passage vectors stale, so the authors re-encode the passages and rebuild the FAISS index in background processes while training continues. Training uses the answer loss, and an auxiliary task rebuilds a domain sentence from retrieved passages, marked with a control token 2210.02627 §3.3 §3.4 eq. 3 fig. 1.

The author-reported gains are exact match with reconstruction added (QA+R) of 3.66 to 8.32 on COVID-19, 8.62 to 14.08 on news and 14.21 to 25.95 on conversations, each against the public RAG checkpoint fine-tuned on the same QA pairs 2210.02627 tab. 1. Initialising RAG from a domain-adapted DPR, without training RAG end to end, gives part of that gain [arxiv:2210.02627 tab. 4]. The cost is the re-encoding and re-indexing, which the paper gives as several GPU hours and several CPU hours on a knowledge base of millions of passages 2210.02627 §3.3. The paper reports one run per setting, with no seeds or intervals §4.2. Card: Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering.

Head first, then fine-tune all layers (LP-FT)

Fine-tuning all parameters from a pretrained network usually beats linear probing on the training distribution. On a large shift it can do worse. The authors trace this to feature distortion: with a random head, the early gradient moves the features in the directions the training data touches, which hurts the directions out-of-distribution data uses [arxiv:2202.10054 §3.2] §3.2.1.

Linear probing then fine-tuning (LP-FT) first fits the head with the features frozen, then fine-tunes everything from that head 2202.10054 §4.2. Averaged over 10 shifts, the author-reported numbers are 85.1 ID and 59.3 OOD for fine-tuning, 82.9 and 66.2 for linear probing, and 85.7 and 68.9 for LP-FT 2202.10054 tab. 1 tab. 2. The gain depends on the features and the shift. With an earlier MoCo-v1 checkpoint, fine-tuning beats linear probing OOD on Living-17 (71% against 66%), and on CIFAR-10 against CIFAR-10.1 fine-tuning is also better OOD 2202.10054 §4.3. The paper calls LP-FT a first step, not a state-of-the-art method 2202.10054 §1. On ImageNet, LP and LP-FT run without augmentation while FT uses it, and LP-FT's ID accuracy ties FT at 81.7 2202.10054 app. B tab. 1. Card: Fine-Tuning can Distort Pretrained Features.

Federated personalisation: align the head to each user's data

FedHealth has two stages. A cloud CNN is built from encrypted updates sent by the users 1907.09173 §3.3. Each user then freezes the convolution and pooling layers and fine-tunes the fully connected layers on its own data 1907.09173 §3.4 §4.2. The user objective adds η times a CORAL term, which compares the covariance matrices of the two feature sets 1907.09173 eq. 5 eq. 6. The paper fixes η at 0.01 [arxiv:1907.09173 §4.2]. The text does not say which tensor or batch each covariance is estimated from 1907.09173 §3.4 eq. 5.

On five isolated UCI Smartphone subjects, the author-reported mean accuracy is 99.4%, against 94.1% for the cloud model with no transfer. That is a gap of 5.3 points. Per subject the gap runs from 3.9 to 7.4 points [arxiv:1907.09173 tab. 2]. The text says transfer beats no transfer by 4% on average, and that plain fine-tuning and MMD alignment also help. Their per-variant values are only in fig. 4 1907.09173 §4.4.

The encryption scheme and the server's aggregation rule are not stated, so the federated part cannot be rebuilt from the text 1907.09173 §3.3. The experiments section does not describe running the encrypted steps, so whether the reported numbers include them is not stated 1907.09173 §4.2. The result rests on five test subjects, five runs and no spread [arxiv:1907.09173 tab. 2]. Card: FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare.

Merge a model with a donor model

Fisher merging averages the parameters of models that share an initialization, with each parameter weighted by a diagonal Fisher information value rather than by a plain mean 2111.09832 §2.3 §2.4. For domain adaptation, the paper merges a domain-adapted pre-trained checkpoint with a task fine-tuned checkpoint, with RoBERTa-base 2111.09832 §3.4. On ChemProt the merge scores 83.1 ± 0.4 micro-F1, against 82.7 ± 0.3 for the unmerged model and 82.5 ± 0.1 for a model fine-tuned from the domain checkpoint 2111.09832 tab. 1. On ACL-ARC, the Fisher merge reaches 73.2 ± 1.7 and isotropic merging 72.5 ± 2.3. On SciERC, isotropic merging is the best row at 81.7 ± 0.5, and the Fisher merge is 81.3 ± 0.5 2111.09832 tab. 1. The authors estimate the merge at about 6 times fewer FLOPs than fine-tuning for an RTE checkpoint 2111.09832 §3.3. That estimate counts the Fisher computation, and it is not a wall-clock time 2111.09832 §3.3.

With an MNLI donor on RTE, a sequential gradient-trained model reaches 77.4 ± 1.6, against 73.2 ± 5.1 for the Fisher merge 2111.09832 tab. A1 [tab. A3]. So merging does not beat gradient-based transfer in every setting. Card: Merging Models with Fisher-Weighted Averaging.

Choose a model without target labels

This is model selection, not training. Deep Embedded Validation (DEV) scores candidate UDA models using labeled source validation data, reweighted to look like the target 2606.04665 §4.2 §4.3. The weights come from a domain classifier trained on source-training and unlabeled target features, computed in the model's own feature space 2606.04665 eq. 7 alg. 1. A control variate reduces the variance of the estimate 2606.04665 §4.4 eq. 9 eq. 10.

On VisDA with MCD, the mean target accuracy of the selected model is 72.42 with DEV, 66.36 with source risk and 54.72 with IWCV. Target-label selection gives 72.55 2606.04665 tab. 2. On the other three tables, DEV is within 0.13 to 0.41 points of target-label selection [arxiv:2606.04665 tab. 2–5]. These gaps come from one number per method with no run count or spread, so they may be within noise 2606.04665 §5. The IWCV rows use a classifier-based density ratio on pre-trained features, not the Gaussian fit of the original method 2606.04665 §2.2 §5. The paper gives no bound on the classifier's error, so expect bias in practice 2606.04665 §4.3. Card: Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation.

Domain generalization: no target data at all

Domain generalization (DG) trains on several source domains and tests on an unseen one, with no target data 2103.03097 §II.A. The survey groups methods into data manipulation, representation learning and learning strategies 2103.03097 §IV. Its benchmark uses DeepDG with ResNet-18. On PACS, ERM averages 81.2, and CORAL and RSC both average 82.0. On Office-Home, ERM averages 62.9 and RSC 63.7 2103.03097 §VII.C. Each cell is one run, with no seed count, so differences of 0.5 or less are within what one seed can move 2103.03097 §VII.C. Card: Generalizing to Unseen Domains.

Two-stage supervised transfer: general fact-checking, then science

The SciFact paper trains its label model in two stages. It starts from FEVER, a general fact-checking set, and then continues on the 809 labeled scientific training claims 2004.14974 tab. 2 §5. Both stages use labeled data. The paper does not frame the step as domain adaptation, and it is listed here because the card carries that concept.

On the dev set, label accuracy is 81.9 after FEVER and SciFact, against 75.7 for SciFact alone and 67.6 for FEVER alone 2004.14974 tab. 3. For rationale selection the order reverses: F1 is 69.7 after FEVER and SciFact, and 72.1 with SciFact alone 2004.14974 tab. 3. So the two-stage model helps the label step and hurts the rationale step on the authors' numbers. The FEVER stage needs the FEVER data and takes about 700 minutes on one P100 2004.14974 app. A.4. On the open test setting, a FEVER-only system that was not trained on SciFact reaches 36.4 for abstract label with rationale, and the SciFact-trained VeriSci reaches 46.5 2004.14974 tab. 4. The zero-shot FEVER threshold was tuned on SciFact dev, so that row is not free of target data 2004.14974 app. A.1. Card: Fact or Fiction: Verifying Scientific Claims.

Surveys, benchmarks and taxonomies

Plested et al. survey deep transfer learning for image classification. Their main advice is to decide how far the weights may move from the source, not whether to transfer at all 2205.09904 §7 §9. The taxonomy has two axes: source-target similarity, which is continuous, and target size, with small around 10,000 labeled examples and large a million or more [arxiv:2205.09904 §3.7] fig. 5. The survey ranks source types, from a very similar labeled source much larger than the target down to self-training on the target when no related source exists 2205.09904 §4.2. Its penalties toward the pretrained weights, such as L2-SP, and its rules for layers and learning rates are given in prose and are not quantified 2205.09904 §6.2 §7. The only experiment is a Big Transfer extension to less similar targets. On Cars, the mean moves from 81.8 to 87.6 with the less-similar learning rate, but the column meanings need checking against the Big Transfer setup 2205.09904 tab. 3. The claim that viewing transfer as a continuum can remove negative transfer is the authors' proposal and is not tested in the paper 2205.09904 §9. Card: Deep transfer learning for image classification: a survey.

A review of deep learning on UAV imagery lists domain adaptation among its open directions. It names GAN-based translation between domains and category-aware adaptation, but evaluates neither 2101.10861 §5.3. Its figures are the authors' own tallies of 232 studies, and they do not measure adaptation 2101.10861 §3. Card: A Review on Deep Learning in UAV Remote Sensing.

The aspect-based sentiment survey has no experiments. Its only domain-adaptation content is a short taxonomy of cross-domain methods for ABSA 2203.01054 §6.1. Card: A Survey on Aspect-Based Sentiment Analysis.

ToxicChat is a benchmark, not an adaptation method. It shows the shift that adaptation has to close. Public toxicity tools score poorly on real chatbot prompts: OpenAI Moderation reaches F1 20.6 and Perspective at threshold 0.7 reaches 5.3 2310.17389 tab. 1. Among RoBERTa-base models trained on other toxicity datasets, the best F1 on the ToxicChat test half is 51.6, and a model trained on the ToxicChat training half reaches 72.1 [arxiv:2310.17389 tab. 2]. That last result is in-domain supervised training, not adaptation. The paper does not test any transfer method. Card: ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation.

When to use which

Implementation sketch

Sketches follow, one for each family of variants with a distinct training loop. Learning rates, batch sizes and iteration counts are not always stated in the cards, so expect to tune them.

Translation-based adaptation, as the DTN paper describes it 1611.02200 §4.

# Data-based adaptation by translation (DTN-style)  [arxiv:1611.02200 §4]
# Inputs: labeled source xs, unlabeled target xt, frozen feature function f
# g: trainable decoder on top of f; D: three-way discriminator

def train_step(xs, xt, alpha, beta, gamma):
    # Encode both domains with frozen f  [§4]
    fs, ft = f(xs), f(xt)
    # Decode into the target domain: G = g o f  [§4]
    gen_s, gen_t = g(fs), g(ft)

    # Discriminator: classes for generated-from-source,
    # generated-from-target and real target  [eq. 3]
    update(D, discriminator_loss(gen_s, gen_t, xt))

    # Generator: both generated sets should look real  [eq. 4]
    # Feature constancy on source, identity on target  [eq. 5, eq. 6]
    # Optional smoothness term, weight gamma  [eq. 7]
    update(g, adversarial_loss + alpha * constancy + beta * identity
              + gamma * smoothness)

# After training: translate the labeled source, keep its labels,
# and fit a classifier on the translated samples  [§5.1]
translated = g(f(source_images))
classifier = fit(translated, source_labels)

Loss-based adaptation with a maximum squares target term 1909.13589 eq. 1 eq. 6.

# Source supervision plus a maximum-squares target term
# [arxiv:1909.13589 eq. 1] [eq. 6]
# Inputs: labeled source (xs, ys), unlabeled target xt; lambda_t (paper: 0.1) [§4.2]

def adapt_loss(xs, ys, xt, lambda_t):
    # Supervised cross-entropy on the source batch  [eq. 2]
    loss_src = cross_entropy(model(xs), ys)
    # Target probabilities on unlabeled pixels  [§3.2]
    p_t = softmax(model(xt))                     # (B, C, H, W)
    # Negative sum of squared probabilities, normalised per image  [eq. 6]
    loss_tgt = -sum(p_t ** 2) / (2 * num_pixels)
    return loss_src + lambda_t * loss_tgt

# Optional image-wise weights from the predicted class counts  [eq. 13]
# Each class term is divided by 2 * N_c^alpha * N^(1 - alpha), alpha = 0.2  [§4.2]

Pseudo-labelling of target pairs, as AugSBERT describes it 2010.08240 §3.2.

# Pseudo-labelling a target domain with a cross-encoder  [arxiv:2010.08240 §3.2]
# Inputs: labeled source pairs; unlabeled target candidate pairs
# Output: bi-encoder trained on the target-domain soft labels

def adapt_by_pseudo_labels(source_pairs, target_candidates):
    # Fit the accurate but slow cross-encoder on labeled source pairs  [§3.2]
    cross_encoder = train_cross_encoder(source_pairs)
    # Score each candidate; the score is the soft label  [§3.1]
    # Candidates should come from BM25 neighbours, not random draws  [§6.1]
    silver = [(a, b, cross_encoder.score(a, b)) for a, b in target_candidates]
    # Fit the fast bi-encoder on the soft-labeled target pairs  [§3.2]
    return train_bi_encoder(silver)

Minibatch transport with a label-aware cost, as JUMBOT describes it 2103.03606 eq. 9 eq. 10.

# One JUMBOT training step, k = 1  [arxiv:2103.03606 §5.2] [eq. 9] [eq. 10]
# Inputs: class-stratified source batch (xs, ys) and unlabeled target batch xt,
# both of size m; eta1, eta2, eta3 weights; tau, eps for the unbalanced plan

def jumbot_step(xs, ys, xt, eta1, eta2, eta3, tau, eps):
    # Embed both batches with the feature extractor  [§5.2]
    fs, ft = g(xs), g(xt)                          # (m, d)
    # Class probabilities for the target batch  [§5.2]
    prob_t = softmax(f(ft))                        # (m, C)
    # Pair cost: squared feature distance plus source-label cross-entropy  [eq. 9]
    cost_feat = squared_distances(fs, ft)          # (m, m)
    cost_label = -log(prob_t[:, ys]).T             # (m, m): source i, target j
    cost = eta1 * cost_feat + eta2 * cost_label
    # Unbalanced entropic plan on the pair; the plan carries no gradient  [§2] [app. B]
    plan = unbalanced_sinkhorn(uniform(m), uniform(m), cost.detach(), eps, tau)
    # Supervised loss plus the plan-weighted transfer term  [eq. 10]
    return cross_entropy(f(fs), ys) + eta3 * sum(plan * cost)

Adversarial domain critic on a shared encoder, as the moderation paper describes it 2208.03274 §3.4.

# One WDAT step; lambda = 0.01; critic weights clipped after each update
# [arxiv:2208.03274 §3.4]
# Inputs: labeled public batch (xs, ys), unlabeled production batch xt

def wdat_step(xs, ys, xt, lam=0.01):
    # Shared encoder embeds both domains  [§3.4]
    hs, ht = encoder(xs), encoder(xt)
    # Classification loss on the labeled source batch  [§3.4]
    class_loss = classification_loss(head(hs), ys)
    # Absolute gap between mean critic scores of the two domains  [§3.4]
    gap = abs(mean(critic(hs)) - mean(critic(ht)))
    # Encoder and head minimise this; the critic is trained on the same gap
    # in the opposite direction, then its weights are clipped  [§3.4]
    return class_loss + lam * gap

MMD penalty on pseudo-labelled target features, as the deep transfer network describes it 1804.07265 eq. 10 eq. 11.

# Source supervision plus a JDA penalty on pseudo-labelled target features
# [arxiv:1804.07265 alg. 1] [eq. 10] [eq. 11]
# Inputs: labeled source (xs, ys), unlabeled target xt; lambda (grid in the paper)

def jda_train(xs, ys, xt, lam, rounds):
    # Base network on labeled source data  [alg. 1 line 2]
    net = fit_cross_entropy(net, xs, ys)
    # Initial pseudo-labels for the target  [alg. 1 line 3]
    yt_hat = argmax(net(xt))
    for _ in range(rounds):   # paper: until the labels stop changing  [alg. 1]
        fs, ft = net.features(xs), net.features(xt)
        # Marginal MMD over all features  [eq. 10]
        penalty = mmd(fs, ft)
        # One class-wise MMD per class, source vs pseudo-labelled target  [eq. 9]
        # An empty class on either side is not defined in the paper  [§III-A]
        for c in classes:
            penalty += mmd(fs[ys == c], ft[yt_hat == c])
        # Cross-entropy plus lambda times the penalty  [eq. 11]
        net = sgd_step(net, cross_entropy(net(xs), ys) + lam * penalty)
        # Re-predict the target pseudo-labels  [alg. 1]
        yt_hat = argmax(net(xt))
    return net

Queue-refined pseudo labels with a contrastive term, as AdaContrast describes them 2204.10377 alg. 1.

# One AdaContrast step on an unlabeled target mini-batch  [arxiv:2204.10377 alg. 1]
# Inputs: target images x; target model from source weights; momentum copy;
# queue_w (features, probabilities, M entries); queue_s (keys, pseudo labels)

def adapt_step(x, target, momentum, queue_w, queue_s, N, tau):
    # Weak view refines the pseudo label from the queue  [§3.1] [eq. 3-4]
    x_weak = weak_augment(x)
    f_weak = target.feature(x_weak)
    nearest = top_n_cosine(f_weak, queue_w.feats, N)
    pseudo = argmax(mean(queue_w.probs[nearest], axis=0))
    # Strong query from the target model, key from the momentum model  [§3.2]
    q, logits = target.feature_and_logits(strong_augment(x))
    k = normalize(momentum.feature(strong_augment(x)))
    # Contrastive loss; same-pseudo-label negatives are masked  [eq. 5-6]
    loss_ctr = info_nce(normalize(q), k, queue_s, pseudo, tau)
    # Cross-entropy against the refined label, plus a mean-prediction term  [eq. 7-9]
    loss_ce = cross_entropy(logits, pseudo)
    p_bar = mean(softmax(logits), axis=0)
    loss_div = sum(p_bar * log(p_bar))
    # Equal weights, no tuning  [eq. 10]
    update(target, loss_ce + loss_ctr + loss_div)
    # Momentum copy follows slowly; queues take the new entries  [eq. 1-2]
    momentum.ema_update(target, m)
    queue_w.push(*momentum(x_weak))    # (feature, probabilities)
    queue_s.push(k, pseudo)

Notes on where the variants differ:

• The translation sketch has two networks to train and one frozen extractor. The loss sketch has one network and changes only the target term. The pseudo-labelling sketch has no target-side training of the teacher at all.
• The image-wise weight divides by the count of each predicted class in the image. That count can be zero, so the weight needs a guard the paper does not state 1909.13589 eq. 13.
• The maximum squares normalisation in eq. 6 is what the paper writes. Check the card's Pitfalls before matching a λ_T value from another implementation 1909.13589 §4.2.
• The JUMBOT sketch is the only one with an optimal transport plan inside the step. Its gradient flows through the cost, not the plan [arxiv:2103.03606 app. B]. The paper's uniform vectors carry the mass m_α, so the unit-mass weights in the sketch match it only when that mass is 1 2103.03606 §4.1.
• The WDAT sketch adds a critic and weight clipping to a shared encoder. The paper does not give the critic's architecture, so the critic is a placeholder 2208.03274 §3.4. Its objective is not numbered, so cite §3.4.
• The aerial translation pipeline keeps the translation step of the DTN sketch but trains the translator in both directions. It then fine-tunes the source segmenter instead of fitting a new classifier 1905.03198 §4.1 §4.3.
• The AdaContrast sketch adds to the queue after the loss, as the paper's equations imply. The released code adds the batch to the queue before it builds the negative mask, so copying the code changes which negatives are used target.py. The paper gives no pseudo-label threshold, so every target sample reaches the loss, and the diversity term is the only guard against collapse 2204.10377 §3.3.

Contradictions

Where two cards report numbers on a similar benchmark, the protocols usually differ. The following comparisons are worth knowing before you read the tables side by side.

• Segmentation on GTA5 to Cityscapes. DISE reports 45.4 mIoU and calls it the highest in its table 1903.12212 tab. 2 §4.2. The maximum squares card reports 46.4 for its full method with the same DeepLab-v2 ResNet-101 base 1909.13589 tab. 2. Neither table contains the other method, so the two numbers are not a head-to-head comparison.
• Entropy-minimization baseline. The maximum squares card reports 42.2 for its own entropy-minimization reproduction and 42.3 for ADVENT's reported value 1909.13589 tab. 2. Both are author-reported, and the card marks the first as the authors' reproduction.
• Scope of adaptation. The test-time card adapts to each input using only that input 2209.07522 §1, while the translation, loss and feature cards use the whole unlabeled target set 1611.02200 §3 1909.13589 §3.1 1903.12212 §3.1. The numbers on these settings are not comparable.
• Fisher merging vs isotropic merging. The Fisher merge beats isotropic merging on ChemProt and ACL-ARC, but not on SciERC 2111.09832 tab. 1. The paper reports both results. Do not generalise either direction.
• Fine-tuning vs probing on OOD data. Linear probing beats fine-tuning OOD on 8 of the 10 shifts 2202.10054 §4.1 tab. 2. The same paper reports fine-tuning ahead OOD on CIFAR-10 against CIFAR-10.1 and with a weaker MoCo-v1 checkpoint 2202.10054 §4.3. Both are the authors' results, and the condition differs.
• IWCV baseline. The DEV paper's IWCV rows use a classifier-based density ratio on pre-trained features, not the Gaussian fit it describes for the original method 2606.04665 §2.2 §5. Compare these numbers only with other IWCV results that use the same ratio estimator.
• Domain generalization. The DG survey's text says RSC is best on both PACS and Office-Home, but its own table has RSC and CORAL tied at 82.0 on PACS 2103.03097 §VII.C. This page follows the table.
• RAG-end2end numbers within one paper. Table 1 gives F1 12.20 and Table 4 gives 12.12 for the same model, RAG-original-QA+R on COVID-19 [arxiv:2210.02627 tab. 1] tab. 4. The text also gives an EM gain from reconstruction on news of 8.39 points, where Table 1 gives 5.69 2210.02627 §4.4. Use the table values and check any quoted difference against them.
• Classification. The Office-31 results come from a ResNet-50 classifier, while the DTN results are on digits 1909.13589 tab. 1 1611.02200 tab. 2. The two sets of gains are not on the same task.
• Reporting protocol on Office-Home. The JUMBOT main table gives the end-of-training average, 70.0 2103.03606 tab. 1. The appendix gives best-over-epochs averages instead, with JUMBOT at 70.1 and the other methods at different values, for example CDAN-E at 67.0 against 66.6 in the main table 2103.03606 app. C, Table 4. Compare numbers within one protocol.
• Epoch choice on target test data. The aerial paper reports the gain at the epoch chosen on target test labels 1905.03198 §5.2.4. The JUMBOT appendix also reports best test accuracy over the run [arxiv:2103.03606 app. C]. Numbers chosen this way are not comparable to results with a held-out target split, and neither paper describes one.
• USPS and MNIST under two protocols. The MEDA paper works on 256-feature vectors and reports USPS to MNIST at 72.1 and MNIST to USPS at 89.5 1807.07258 tab. 3. Its table lists 1,800 USPS and 2,000 MNIST samples, while its text describes the full splits 1807.07258 tab. 1 §4.1. The source-free paper works on images with a DIRT-T-style classifier and reports MNIST to USPS at 97.3 and USPS to MNIST at 99.3 [arxiv:2502.19316 tab. 1]. The inputs and the sample counts differ, so the pairs of numbers are not comparable.
• Source-only on VisDA. The source-free GAN paper reports 52.4 for the source-only model on VisDA17 with ResNet-101 2502.19316 tab. 3. The contrastive test-time paper reports 43.8 for source-only on VisDA-C with ResNet-101 2204.10377 tab. 1. Both name the VisDA family and the backbone, but the two cards do not show the same split or averaging, so the baselines and the adapted scores should not be compared across the papers.

Papers

• Unsupervised Cross-Domain Image Generation: DTN, which translates unpaired source samples into the target domain through a frozen feature function, and reports the translated set as domain adaptation for digits.
• Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval: aligns sketch and photo embeddings with a gradient-reversal domain loss, alongside a triplet loss and a semantic loss, for zero-shot retrieval.
• Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images: restyles labelled Potsdam tiles to the Vaihingen look with a cycle-consistent translator, then fine-tunes the segmenter on them.
• All about Structure: DISE, which splits images into a shared structure code and a domain-specific texture code, and swaps textures to carry source labels into the target domain.
• Domain Adaptation for Semantic Segmentation with Maximum Squares Loss: replaces the entropy target term with a maximum squares term, adds image-wise class weights and a low-level head, for segmentation and Office-31 classification.
• Augmented SBERT: pseudo-labels sampled sentence pairs with a cross-encoder to train a bi-encoder, with pair sampling as the main design choice and a domain-adaptation setting on four datasets.
• StyleGAN-NADA: shifts a StyleGAN2 generator to a text-defined domain with no target images, using a directional CLIP loss and adaptive layer selection.
• Unbalanced minibatch Optimal Transport; applications to Domain Adaptation: replaces balanced minibatch transport with unbalanced transport and proves the estimator's properties; JUMBOT uses it for domain adaptation with a label-aware cost.
• Merging Models with Fisher-Weighted Averaging: averages model parameters with a diagonal Fisher weight, and reports merges with donor and domain-adapted checkpoints on GLUE and three scientific NLP tasks.
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution: explains when full fine-tuning does worse OOD than linear probing, and proposes LP-FT as a two-step recipe.
• Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering: RAG-end2end, which trains the DPR encoders and the BART generator jointly on domain text, refreshes the passage index in the background, and adds a sentence-reconstruction task.
• Test-Time Training with Masked Autoencoders: adapts a masked autoencoder to each test image by reconstruction, then resets the weights, and reports gains on ImageNet-C.
• BioMistral: continued pre-training of a 7B language model on medical text, with evaluation on medical question answering and merges with the base model.
• ChipNeMo: domain-adaptive pre-training of LLaMA2 on chip-design text, with tokenizer extension, alignment and a fine-tuned retriever for RAG.
• A Review on Deep Learning in UAV Remote Sensing: a literature review of deep learning on UAV imagery that codes 232 studies and lists domain adaptation among its open directions.
• A Short Survey of Viewing Large Language Models in Legal Aspect: a critical survey of LLMs in legal AI that organises adaptation, retrieval and agent mechanisms by the evidence they need; it proposes no experiments.
• Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation: DEV, a label-free estimate of target error for choosing among UDA models, using a feature-space density ratio and a control variate.
• FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare: builds a cloud CNN from encrypted federated updates, then fine-tunes each user's fully connected layers with a CORAL alignment term.
• Generalizing to Unseen Domains: A Survey on Domain Generalization: survey of domain generalization with a taxonomy of methods and a DeepDG benchmark on PACS and Office-Home.
• A Survey on Aspect-Based Sentiment Analysis: a survey of ABSA with a short section on cross-domain transfer, naming feature-based, data-based and combined methods.
• Deep Transfer Network with Joint Distribution Adaptation: a CNN trained on labeled fault data, with a marginal and class-wise MMD penalty on pseudo-labelled target features, evaluated on ten fault-diagnosis transfer tasks.
• Visual Domain Adaptation with Manifold Embedded Distribution Alignment: MEDA, which adds a geodesic flow kernel and a mu-weighted marginal and class-wise MMD, with a closed-form classifier, on fixed image features.
• Fact or Fiction: Verifying Scientific Claims: the SciFact benchmark and the VeriSci baseline, whose label model is trained first on FEVER and then on scientific claims.
• Model Adaptation: Unsupervised Domain Adaptation without Source Data: 3C-GAN, which adapts a source-trained classifier with a class-conditional generator, a weight anchor and a clustering term, without source data.
• Deep transfer learning for image classification: a survey: a survey that sorts transfer by source-target similarity and target size, ranks source types, and reports one Big Transfer extension to less similar targets.
• A Holistic Approach to Undesired Content Detection in the Real World: a moderation classifier adapted from public data to production traffic with Wasserstein domain-adversarial training, alongside active learning and synthetic data.
• COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter: continued pretraining of BERT-Large on COVID-19 tweets, followed by fine-tuning on five classification sets, with a released script that differs from the paper's schedule.
• Contrastive Test-Time Adaptation: AdaContrast, a source-free method that refines pseudo labels in a memory queue and trains a contrastive loss with same-label negatives removed.
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models: continued pretraining of Llama-2 at 7B and 70B on a medical corpus with a replay share, then per-benchmark finetuning and three inference modes.
• Unsupervised Domain Adaptation through Self-Supervision: trains one shared network on the labeled source task and on rotation, flip and patch puzzles over both domains, with no adversarial term.
• ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation: a benchmark showing that public toxicity tools transfer poorly to chatbot prompts; it proposes no adaptation method.
• MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction: continued pretraining of SciBERT on materials-science text, then task fine-tuning for NER, relation and abstract classification.
• V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception: a real two-vehicle LiDAR dataset with cooperative detection, tracking and a Sim2Real domain-adversarial benchmark.