Unsupervised Domain Adaptation through Self-Supervision

In plain words

A model trained on pictures with labels usually does worse on pictures from a different source, such as digits photographed in a new style. This paper asks whether the model can adapt without any labels from the new source. The trick is to give the network extra puzzles that need no labels: guess how much a picture was rotated, guess whether it was flipped, or guess which part of the picture a crop came from. The network learns these puzzles on both the labelled source pictures and the unlabelled new pictures, while it also learns the real task on the labelled ones.

Solving the same puzzle on both sets pushes the network to represent both sets in a similar way, so the labelled knowledge carries over. The method has no adversarial game, which makes training more stable. The puzzles only help when they match the problem. For example, rotation puzzles fail on house-number images, because the network can cheat by looking at the neighbouring digits.

TL;DR

• Train one shared network on the source classification task plus several label-free puzzles (rotation, flip, patch location), using source and unlabelled target batches for the puzzles §4.
• No minimax or adversarial loss; the puzzle heads are discarded at test time §4.
• Early stopping and hyperparameter choice use an unlabelled feature-mean distance plus source error, not target labels §4.1.
• Author-reported best accuracy on 4 of 7 standard benchmarks, and a gain over source-only on GTA5 to Cityscapes segmentation tab. 2 tab. 3.

Problem

Unsupervised domain adaptation: labelled data from a source domain, only unlabelled data from a target domain, and the goal is good target accuracy §1. Most prior methods align the two domains by minimising a distribution-distance measure in feature space, often MMD or a learned domain discriminator, which yields a minimax problem that the paper reports as hard to optimise §1 §2.

Method

The method has three parts: a set of self-supervised tasks, a multi-task objective, and a validation heuristic.

Self-supervised tasks §3. Each task creates its own labels from the input.

1. Rotation: rotate by 0, 90, 180 or 270 degrees; predict the angle as a 4-way classification §3.
2. Flip: flip vertically or not; predict a binary label. Horizontal flips are not used §3.
3. Patch location: crop patches and predict where they come from. For small images the crop comes from one of four quadrants (4-way classification). For large images (segmentation) the crop position is a 2-D regression trained with squared loss §3 app. F.

Pixel-reconstruction tasks (inpainting, colourisation, denoising autoencoders) are argued to be unsuitable, because they tend to push the domains apart. The paper reports that colourisation and a denoising autoencoder were little better than source-only, and sometimes worse §3.

Shared network §4. All heads share one feature extractor φ. The main head h0 is the classifier. Each self-supervised head h_k is a single linear layer (followed by softmax or sigmoid for classification) on the shared features. The paper says the feature dimension is between 64 and 512 depending on the architecture §4.

Objective §4. The main loss uses only labelled source data:

\mathcal{L}_0(S;\phi,h_0)=\sum_{(x,y)\in S}L_0\big(h_0(\phi(x)),y\big) §4

Each self-supervised loss uses both domains, with labels created by the transform f_k. The source labels are discarded:

\mathcal{L}_k(S,T;\phi,h_k)=\sum_{(f_k(x),\tilde y)\in F(S)}L_k\big(h_k(\phi(f_k(x))),\tilde y\big)+\sum_{(f_k(x),\tilde y)\in F(T)}L_k\big(h_k(\phi(f_k(x))),\tilde y\big) eq. 1

The total objective is the plain sum, with no trade-off weights (the paper says it tried weights and found them unnecessary) §4 eq. 2:

\min_{\phi,\,h_1,\dots,h_K}\; \mathcal{L}_0(S;\phi,h_0)+\sum_{k=1}^{K}\mathcal{L}_k(S,T;\phi,h_k) eq. 2

At test time the self-supervised heads are dropped and the prediction is h0(φ(x)) §4.

Training procedure app. B. The implementation does not take one joint step on eq. 2. For each batch it:

# One pass over the labelled source data  [app. B]
def train_epoch(source_loader, target_loader, puzzle_tasks, phi, h0):
    for source_batch in source_loader:
        # One gradient step per self-supervised task  [app. B]
        for task in puzzle_tasks:
            # Half the batch from source, half from target  [app. B]
            # Source and target labels are dropped; the puzzle makes new ones
            images, puzzle_labels = task.make_batch(source_batch, target_loader)
            loss = task.loss(task.head(phi(images)), puzzle_labels)
            update(params=[phi, task.head], loss=loss)
        # Supervised step on source images with their real labels  [app. B]
        loss = cross_entropy(h0(phi(source_batch.images)), source_batch.labels)
        update(params=[phi, h0], loss=loss)
    return phi, h0

The paper states that balanced batches keep the two domains roughly balanced in each puzzle loss app. B. Its appendix says that a joint step over all K+1 losses usually changes accuracy by less than 1%, and that the loop is a memory-for-time trade-off app. B.

Early stopping and hyperparameter choice §4.1. There are no target labels to tune on, so the paper uses a heuristic on unlabelled data. Let the mean-feature distance be (on unlabelled source and target validation sets S', T'):

D(S',T';\phi)=\Big\|\frac{1}{m}\sum_{x\in S'}\phi(x)-\frac{1}{n}\sum_{x\in T'}\phi(x)\Big\|_2 eq. 3

Over T epochs, let v be the distance curve and w the source validation error curve. Each is divided by its minimum and the two are added. The epoch with the smallest sum is chosen §4.1:

# Pick the epoch by feature-mean distance plus source error  [§4.1]
def select_epoch(distances, source_errors):
    # Normalise each curve by its own minimum  [§4.1]
    combined = [
        d / min(distances) + e / min(source_errors)
        for d, e in zip(distances, source_errors)
    ]
    # Return the epoch with the smallest combined score  [§4.1]
    return argmin(combined)

The paper calls this a practical rule of thumb, not a guaranteed method §4.1.

Object recognition setup app. E. A 26-layer pre-activation ResNet is the test-time model h0(φ(x)). The self-supervised heads are linear layers on the end of φ. No data augmentation is used, to match previous work app. E.

Segmentation setup §5.2 app. F. Initialised from DeepLab-v3 pre-trained on ImageNet. Each self-supervised head is global average pooling followed by one linear layer, and location is the 2-D regression version on 400×400 crops app. F.

Hyperparameters

Object recognition, fixed across benchmarks app. E:

Per-benchmark epochs, milestones and width. The paper does not state these. The values below come from the code's run scripts and are not in the paper:

Segmentation (GTA5 → Cityscapes), paper values app. F: DeepLab-v3 backbone (ImageNet pre-trained); SGD learning rate 0.007; 15,000 iterations; batch size 48; 400×400 crops for location regression. Not stated in the paper: momentum, weight decay, learning-rate schedule, full input size, and which self-supervised tasks run. The code's defaults are different; see Paper vs code.

Dataset sizes app. D §5.2:

Results

All numbers are author-reported. Object recognition tab. 2 reports test accuracy (%) on seven benchmarks. Rows are grouped by the self-supervised tasks used: R rotation, L location, F flip. The paper says the table entries marked "R" for SVHN → MNIST use rotation and flip tab. 2 footnote. Where a baseline reports "-", the paper gives no number for that benchmark.

Source: tab. 2. Bold marks the paper's bold entries (best per benchmark, with ties). The paper does not define the "(IN)" suffix in the table. The paper reports best accuracy on four of seven benchmarks: MNIST→MNIST-M (tied with DIRT-T at 98.9), MNIST→USPS (96.5 vs 95.6 for CyCADA), CIFAR→STL (82.1 vs 80.0 for VADA), and STL→CIFAR (74.0 vs 73.5 for VADA) tab. 2 §5.1.

Segmentation, GTA5 → Cityscapes, per-class IoU (%) tab. 3. The table below is transposed from the paper's layout:

Source: tab. 3 (the paper's table lists the same per-class values). The paper says its method alone improves over source-only, and combining it with CyCADA improves over CyCADA alone §5.2 tab. 3. The paper uses "Oracle" for training on target labels §5.2.

Convergence: the paper reports that test error on both domains and the self-supervised tasks falls smoothly, and that the mean-feature distance falls along with it, although the distance is never optimised directly fig. 3.

Paper vs code

Code: the object-recognition repository at commit b256316283e74b5d1f16777f029c384ee9b6e2e7 (see code_commit) and the segmentation repository at https://github.com/erictzeng/ssa-segmentation-release (commit 273150518dae414e2444a37d350695c1d2f719d2). Both are linked from the paper's footnote in §5 and both belong to the authors. The segmentation repository has no LICENSE file, so no code from it is quoted. Paths below are relative to the object-recognition repository, except those prefixed seg/, which are relative to the segmentation repository.

• Early-stopping inputs. The paper computes the distance and the source error on unlabelled source and target validation sets §4.1. The code computes the distance on the source and target test loaders, and the source error on the labelled source test loader train.py. The score then picks the checkpoint whose target test error is reported joint_plot.py. The reported selection accuracy therefore uses the test split, and the source-test labels, to choose the checkpoint.
• Distance feature. The paper defines the distance on φ(x) eq. 3. The code takes the output of the feature extractor before global pooling, an 8×8 map for 32×32 inputs, and compares the mean maps SSHead.py get_mmd.py. The distance is therefore over spatial feature maps, not pooled vectors, which the paper does not state.
• Score form. The paper uses v/min(v) + w/min(w) §4.1. The code uses sce + mmd × min(sce)/min(mmd) get_mmd.py. This ranks epochs in the same order, since it is the paper's score multiplied by min(sce).
• Which number is reported. joint_plot.py prints two numbers: the lowest target test error over all checkpoints, which uses target test labels, and the error at the score-selected checkpoint joint_plot.py. The paper does not say which one fills Table 2.
• Evaluation frequency. The paper evaluates per epoch §4.1. The code evaluates every num_batches_per_test batches. The default is 100 main.py. CIFAR and STL scripts set 10,000, so evaluation only runs at batch 0 of each epoch cifar_stl.sh train.py. Digit scripts use the default, or 1000 for SVHN → MNIST.
• Loss scale. The paper writes sums over the datasets eq. 1 eq. 2. The code uses the mean cross-entropy over each mini-batch SSTask.py main.py. Each step therefore has a per-batch scale, not a dataset-sum scale.
• Optimiser state. The paper describes one update per task app. B. The code creates one SGD optimiser and one learning-rate schedule per self-supervised task, all sharing the extractor's parameters parse_tasks.py parse_tasks.py parse_tasks.py. Each optimiser keeps its own momentum buffer.
• Rotation for digit sources. The paper describes four rotations §3. For digit sources the code replaces the 90° and 180° rotations with a transposition and a horizontal flip DsetSSRotRand.py. The digit flag is set when the source is in a list containing 'svhn_exta', a typo. So SVHN-extra sources (as used in scripts/svhn_mnist.sh) take the natural-image rotation path parse_tasks.py svhn_mnist.sh.
• Flip task. The paper uses the vertical flip only §3. The code always uses the two-class flip parse_tasks.py DsetSSFlipRand.py. The code's three-label digit variant is never reached.
• SVHN training data. The paper's table does not say which SVHN training split is used. The code trains SVHN-source runs on svhn_extra (train plus extra) prepare_dataset.py svhn_mnist.sh, and uses it as the MNIST→SVHN target mnist_svhn.sh.
• Segmentation backbone. The paper says DeepLab-v3 app. F. The code's default is a DRN-C-26 model [seg: train_task.py#L38] [seg: train_task.py#L70-L77]. Its deeplab option builds a ResNet-101 with ASPP and loads a snapshot from a fixed source-only path [seg: train_task.py#L89-L96]. The paper does not say which one produced its numbers.
• Segmentation optimiser. The paper gives learning rate 0.007, 15,000 iterations and batch 48 app. F. The code's defaults are learning rate 0.0016, 16,000 iterations and batch 8 per GPU [seg: train_task.py#L40-L47]. It uses SGD with momentum 0.9 and weight decay 1e-4 [seg: pytorch_fcn/trainer.py#L38-L40], and a polynomial decay with power 0.9 [seg: pytorch_fcn/trainer.py#L67]. The paper states none of these three.
• Segmentation tasks. The paper describes only the 400×400 location regression app. F. The code's segmentation run uses rotation and grid regression at three crop sizes (100, 200, 400) [seg: train_task.py#L104-L115]. Vertical flip is imported but not used [seg: train_task.py#L18]. Segmentation crops are 512 [seg: train_task.py#L103].
• Hardware. The code calls .cuda() directly, so it needs a GPU main.py train.py.
• Data augmentation. The code applies resize and normalisation only prepare_dataset.py, which matches the paper app. E.

No discrepancy found for: the main-task loss, the unweighted sum in eq. 2, the balanced 64/64 batches, and SGD with momentum 0.9 and weight decay 5e-4 for object recognition.

Pitfalls

• Early stopping uses the test splits. Do not copy the paper's wording about validation sets. In the code the target test images, and the source test labels, pick the checkpoint train.py. A stricter protocol would select on held-out source validation data and leave the target test set out of the selection.
• Mixed-domain BatchNorm (inferred from code). The shared extractor, including BatchNorm, sees source-only batches in the classifier step and mixed source+target batches in each self-supervised step SSTask.py train.py. BatchNorm running statistics therefore accumulate from both. The paper does not mention BatchNorm. Test-time accuracy depends on those statistics.
• Compute cost. The code runs K+1 forward and backward passes per source batch, one per puzzle and one for the classifier train.py. The paper confirms one step per task app. B. Plan for roughly K+1 times the cost of a supervised step.
• Puzzle choice decides the outcome. The paper reports that rotation gives trivial solutions on SVHN, because the network looks at the sides of the crop, which carry neighbouring digits §5.1. Test every task on your own data before trusting it.
• Rotation on digits is not a true rotation. The digit path uses transposition and a horizontal flip (see Paper vs code). The paper does not describe this. It is a reflection, not a rotation, so it changes what the puzzle asks.
• Text vs table on the STL→CIFAR gain. The text says location and flip give "an improvement of 9%" on top of rotation §5.1. Table 2 gives 74.0 (R+L+F) against 65.6 (R), which is 8.4 points, and 65.6 against 56.1 for source-only, which is 9.5 points tab. 2. The text rounds these.
• Early-stopping heuristic is a heuristic. The paper says it has no statistical guarantee §4.1. It needs per-epoch curves for both the distance and the source error, and the figures show only two settings fig. 3.
• Unstated per-benchmark settings. Epochs, milestones and width are not in the paper. The values above come from the code scripts. Reproduce a column with its own script settings, not the paper's defaults.
• Segmentation configuration. The paper's learning rate, iterations and batch size differ from the code's defaults (see Paper vs code). Which configuration produced Table 3 is not stated, so do not assume the defaults reproduce it.

Lineage

• Builds on: rotation prediction (Gidaris et al., 2018), cited in §3 for the rotation task. Not in the wiki.
• Compared against: DANN, DRCN, DSN, kNN-Ad, PixelDA, ATT, Π-Model, ADDA, CyCADA, VADA and DIRT-T tab. 2, and CyCADA on segmentation tab. 3.
• Related but different: the paper contrasts itself with DRCN, which uses only target data for denoising reconstruction app. A, and with self-supervised robustness and few-shot papers that train without target data app. A.