The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization

In plain words

A neural network that learned on clean photos often fails when the photos look different, for example in fog, with sensor noise, or at night. Inside the network, each normalization layer stores an average and a spread of the values it saw during training, and those stored numbers are wrong for the new photos. This is the problem the paper addresses.

The fix is simple. The method keeps every learned weight fixed and only nudges those stored averages toward the new data, one incoming image at a time. To get a usable estimate from a single image, it makes a small batch of randomly cropped, flipped and rotated copies of that image. The first updates move the averages a lot and later updates move them less, so the estimate settles. No gradients are computed, so the cost stays close to ordinary inference. The method was shown to work with fewer than 1% of the unlabeled test images.

TL;DR

DUA (Dynamic Unsupervised Adaptation) updates only the running mean and variance of each BatchNorm layer while a stream of unlabeled test samples arrives §3.2. It uses a momentum that decays with each sample eq. 4 eq. 5 and a 64-copy augmented batch per sample §5.4. The paper reports strong results on corruption benchmarks and fog and rain detection, using 25 to 100 test samples per dataset, and no backpropagation §1 tab. 1 tab. 4. Check the Paper vs code section before comparing numbers: the released code reports the best score over the whole stream.

Problem

Models trained on one distribution lose accuracy under a shift, even a slight one, and the paper cites this loss for corruption and weather benchmarks §1. Unsupervised domain adaptation normally needs many labeled source images and a large unlabeled target set, and often retraining §1 §2. The paper targets dynamic settings: a pretrained model with no training data, a few unlabeled samples from a changing environment (e.g. autonomous driving in bad weather) and real-time use §1.

Related test-time methods that recompute batch statistics from scratch need large test batches, and gradient-based test-time methods need backpropagation at test time §1 §2. This card follows arXiv v2 (updated 2022-04-04), the version that fetch retrieved.

Method

Batch normalization. A BatchNorm layer normalizes each incoming activation with a training mean and variance, then applies a learned scale and shift §3.1 eq. 1. During training those statistics are running averages with momentum rho eq. 2 eq. 3. At test time the running statistics are fixed §3.1.

DUA. All weights stay fixed. Only the expected value and variance used by the BatchNorm layers are replaced by estimates from the target stream, starting from the source statistics as a prior §3.2. Each target sample updates the estimates once, sequentially §3.2. The update has the same form as eq. 2 and eq. 3, but the momentum at step k is rho_k plus a lower bound zeta, and rho_k decays geometrically eq. 4 eq. 5. The variance update mirrors the mean update eq. 6 eq. 7.

Batch per sample. For each incoming sample the method builds a batch of augmented copies. The paper uses random horizontal flips, random crops and rotations §3.2. The supplemental material states rotations of 0, 90, 180 and 270 degrees app. A. The batch size is 64 in the experiments §5.4. The paper notes that a batch of 8 still gives most of the benefit §5.4 fig. 6. Corruption types from the benchmark are not used as augmentations §3.2.

Why the schedule. With a fixed momentum the estimates are unstable or converge slowly §3.2 fig. 3. The decaying momentum starts with larger steps to move the distributions apart and later uses smaller steps to keep the estimates stable fig. 3.

Equations, as in the paper:

\hat{x} = \frac{x - \mathbb{E}[X]}{\sqrt{\mathrm{Var}[X] + \epsilon}} \cdot \gamma + \beta eq. 1

\hat\mu_k = (1-\rho)\,\hat\mu_{k-1} + \rho\,\mu_k eq. 2

\hat\sigma^2_k = (1-\rho)\,\hat\sigma^2_{k-1} + \rho\,\sigma^2_k eq. 3

\hat\mu_k = \bigl(1-(\rho_k+\zeta)\bigr)\,\hat\mu_{k-1} + (\rho_k+\zeta)\,\mu_k eq. 4

with \hat\mu_0 = \hat\mu_s, \rho_k = \rho_{k-1}\,\omega, \rho_0 = 0.1 eq. 5

\hat\sigma^2_k = \bigl(1-(\rho_k+\zeta)\bigr)\,\hat\sigma^2_{k-1} + (\rho_k+\zeta)\,\sigma^2_k eq. 6

with \hat\sigma^2_0 = \hat\sigma^2_s, \rho_k = \rho_{k-1}\,\omega, \rho_0 = 0.1 eq. 7

Here \mu_k and \sigma^2_k are the mean and variance of the incoming batch §3.1. The paper defines 0<\zeta<\rho_0 as the lower bound of the momentum §3.2.

Procedure, one sample at a time. The paper gives it in prose and equations, and it has no numbered algorithm. The pseudocode below follows the equations and the released code.

# DUA: adapt BatchNorm statistics one sample at a time  [§3.2] [eq. 4]-[eq. 7]
# Input:  trained model with BN layers; unlabeled test stream;
#         rho_0 = 0.1, omega = 0.94, zeta = 0.005  [§3.2] [§4.3]
# Output: BN running statistics adapted to the stream

def adapt_stream(model, stream, rho_0, omega, zeta):
    # Momentum starts at rho_0 and decays per sample [eq. 5] [eq. 7]
    rho = rho_0
    for sample in stream:
        # Decay before use: the first update uses rho_0 * omega
        # [code: methods/dua.py#L45] [code: methods/dua.py#L72]
        rho = rho * omega
        # Weight of the new batch statistics, floored at zeta [eq. 4]
        momentum = rho + zeta
        # Make 64 augmented copies of the one sample [§5.4]
        # [code: methods/dua.py#L126]
        batch = make_augmented_batch(sample, size=64)
        # Rotate each copy by a random 0, 90, 180 or 270 degrees [app. A]
        # [code: utils/rotation.py#L33-L45]
        batch = random_rotate(batch)
        # Only BN running statistics change; weights stay fixed [§3.2]
        set_bn_momentum(model, momentum)
        model.set_bn_train_mode()
        # Forward pass: running mean and variance are updated from the
        # statistics of the 64 copies [eq. 4] [eq. 6]
        model(batch)
        # The output of this pass is discarded [code: methods/dua.py#L79]
        model.eval()

Input normalization, the CIFAR augmentation and the rotation details come from the code, not the paper. See Paper vs code.

Hyperparameters

Results

Author-reported. Setup: corruption benchmarks at severity 5 (15 corruption types) with ResNet-26 (compared with TTT and NORM) and Wide-ResNet-40-2 (compared with TENT); ImageNet-C with ResNet-18; YOLOv3 for KITTI-Fog and KITTI-Rain §4.1 §4.2 §4.3. Baselines use the whole test set, while DUA uses under 1% of it §4.3.

CIFAR-10C, top-1 error (%), severity 5 tab. 1

Columns are the 15 corruptions in the order given in tab. 1, then the mean. Headline: DUA reaches 26.8 mean error with ResNet-26, against 31.4 for TTT and 32.9 for NORM. With Wide-ResNet-40-2, DUA (12.1) is slightly below TENT (12.3) and well below source (18.3) tab. 1. The paper states that DUA uses 80 samples on CIFAR-10C app. A; the table caption says the TENT and TTT/NORM rows use the whole test set §4.3. The paper calls the ResNet-26 result a new state of the art § Results (CIFAR-10/100C).

CIFAR-100C, top-1 error (%), severity 5 tab. 2

On CIFAR-100C, DUA beats TTT and NORM with ResNet-26 (57.3 vs 59.6 for NORM) and trails TENT with Wide-ResNet-40-2 (37.6 vs 36.0) tab. 2. The text calls this "competitive with TENT" § Results (CIFAR-10/100C).

ImageNet-C, top-1 error (%), severity 5, ResNet-18 tab. 3

On ImageNet-C the mean error of DUA (78.2) is slightly above NORM (77.3). The paper calls the result "on-par" with all baselines § Results (ImageNet-C).

KITTI-Fog and KITTI-Rain, mAP@50 for YOLOv3, most severe level tab. 4

The text reports mean gains over the source model of 17.7 mAP for fog and 7.1 mAP for rain § Results (Object Detection). From the Table 4 rows, the fog gain averages 17.3 (see Paper vs code).

Natural and cross-domain shifts and digits, supplemental app. C

Digit transfers in the digit row are, in order, SVHN→USPS, SVHN→MNIST, MNIST→USPS, MNIST→SVHN, USPS→MNIST, USPS→SVHN. Office-31 transfers are Amazon→Webcam, Amazon→DSLR, DSLR→Webcam, DSLR→Amazon, Webcam→DSLR, Webcam→Amazon app. C. The paper says DUA is not meant to beat specialized domain-adaptation methods there, but to serve as a first step before them app. C.

Lower severities. The paper reports consistent improvements at severity levels 1 to 4 for CIFAR-10C, CIFAR-100C and ImageNet-C app. B. The per-table numbers are in the supplement and are not copied here.

Ablations (CIFAR-10C, ResNet-26, author-reported).

• Sample order: over 300 shuffled runs, error after 5 samples is 36.4 ± 0.4, after 25 samples 28.3 ± 0.19, and after 100 samples 27.2 ± 0.09, from a source error of 49.2 §5.1.
• Continuous adaptation: on KITTI-Fog with day → fog → day cycles, the text quantifies the day-domain drop at iteration 100 as 5.8% and reports fast recovery §5.2 fig. 4.
• Batch-norm layers: adapting all BN layers gives the best result; the paper notes individual gains are slightly larger at later layers §5.3 fig. 5.
• Augmentation and batch size: batch 64 is used; batch 8 gives most of the benefit with a minor loss; the source error is 49.2 §5.4 fig. 6.
• Momentum decay: omega = 0.94 balances stability and speed app. B fig. 8.

Paper vs code

Code: the official repository jmiemirza/DUA at commit 5a2197154405ce947006558180f430d64b99c626, read from raw/code/2112.00463. The paper names the repository in §4.3. The repository has no LICENSE file, so its code is cited only, never quoted.

1. The reported number is the best over the stream, selected on the test set. The paper reports the value after a fixed number of samples, where results saturate app. A. The code evaluates the full test set after every sample and keeps the minimum error (or maximum mAP) over all steps. The selection uses test labels. Code: results is appended every step, then adaptation_result = min(results) for error metrics dua.py. The paper does not state a selection rule.
2. Sample count. The paper lists 25 samples for KITTI-Fog and KITTI-Rain and 100 for ImageNet-C app. A. The code uses one default of 80 for all datasets and no per-dataset override main.py. The README runs use this default README.md.
3. ImageNet-C adaptation transform. The paper says it uses an ImageNet pretrained ResNet-18 §4.3 app. A and does not describe the adaptation transform. The code's default branch, used for every non-YOLOv3 dataset, crops 32×32 with padding 4, flips, and normalizes with mean and std 0.5 dua.py. The ImageNet image getter returns the raw file, with no resize datasets.py. The ImageNet test path resizes to 256, center-crops 224 and uses ImageNet mean and std data_loader.py. If the code is right, the adaptation statistics come from a different input scale and size than evaluation. This was read from the code, not run.
4. KITTI input scaling. The YOLOv3 adaptation transform ends with a ToTensor that already maps 0–255 to 0–1 dua.py. The adaptation input then divides by 255 again dua.py. The test path divides once testing_yolov3.py. This reads as a double scaling of the adaptation inputs. It was read from the code, not run, and the paper does not mention it.
5. Momentum index. The paper defines rho_0 = 0.1 and rho_k = rho_{k-1} omega eq. 5 but does not say whether the first update uses k = 0 or k = 1. The code uses rho_1 = 0.1 × 0.94 = 0.094, so the first momentum is 0.099 dua.py dua.py dua.py.
6. Input normalization. The paper does not state a mean or std. The code uses mean 0.5 and std 0.5 for CIFAR data_loader.py.
7. Batch size flag. The adaptation batch is hard-coded to 64 copies dua.py dua.py. The --batch_size flag defaults to 8 main.py and does not set the adaptation batch.
8. Datasets not in the release. The paper reports digits, Office-31, VIS-DA, SODA10M and KITTI cross-domain results app. C. The released code's dataset choices are CIFAR-10, ImageNet, ImageNet-mini and KITTI main.py config.py. The digit, Office-31, VIS-DA and SODA10M experiments are not in the repository.
9. Matching numbers. The README reproduces the CIFAR-10C Wide-ResNet-40-2 DUA row with 80 samples and mean error 12.1 README.md, the same as tab. 1.
10. Sample order. The code takes the test images in dataset order, one per step dua.py. The shuffled runs in §5.1 are not in the release.

The rotation augmentation agrees between paper and code: each copy gets one of four angles, drawn uniformly app. A rotation.py.

Pitfalls

• Oracle selection inflates numbers. The released code's reported value is the best step on the test set (see Paper vs code, item 1). A reimplementation that reports the value after a fixed sample count will differ. Decide which one you report and say so.
• Fixed count versus stream. Paper results use fixed counts (80, 100, 25, ...). Use the same count per dataset when comparing with the paper.
• Momentum floor keeps moving the statistics. Momentum never drops below zeta = 0.005, so the statistics keep changing at that rate after the decay has flattened eq. 4 eq. 5. The paper does not discuss long streams beyond the reported counts.
• Source-statistic prior. The first steps use large momentum (about 0.099 by the code's schedule), so the first few samples dominate the final statistics. Check the first 10 steps in any reimplementation dua.py.
• Batch statistics from augmented copies. Each update uses statistics of 64 augmented versions of one image, not of a real batch §3.2. The augmentations are not the test corruptions §3.2. In practice this may shift the statistics away from the true test distribution. The paper's batch-size ablation reports a small loss at batch 8 §5.4.
• Layer coverage. Adapting all BatchNorm layers is best §5.3. A network without BatchNorm cannot use the method, as the abstract states.
• Framework convention. The code sets the framework's BatchNorm momentum to rho_k + zeta, which weights the new batch statistics as in eq. 4 dua.py. Check that a port uses the same convention. The variance update is set by the same momentum eq. 6.
• Cost. The paper claims minimal overhead and no backpropagation §1. In the code, each adaptation step is one forward pass on the 64-copy batch dua.py. The released code also runs a full test-set evaluation after each sample for reporting. That evaluation is not part of the method and is far more expensive.
• Internal arithmetic. The text gives a mean fog gain of 17.7 mAP § Results (Object Detection). The Table 4 rows give (20.5 + 14.4 + 16.9) / 3 = 17.3. The source of the 17.7 figure is not stated.
• Appendix letters are inferred from the appendices environment (A, B, C), not printed in the source. Table and figure numbers were counted from the source order of the main text.

Lineage

• Builds on: batch normalization, which the method reuses unchanged, with its running statistics as the prior §3.1 eq. 1. Ioffe and Szegedy are the citation the paper gives §3.1.
• Compared against: NORM (Schneider et al. 2020; Nado et al. 2020), which recomputes BatchNorm statistics from the whole test set §4.2. TTT (Sun et al. 2020), which uses an auxiliary task and backpropagation at test time §4.2. TENT (Wang et al. 2020), which recomputes statistics and updates scale and shift by entropy minimization §4.2. Adaptive BN (Li et al. 2016) recomputes BatchNorm statistics from scratch on the test set §2.
• Related: the released README points to DISC, a CVPR 2022 workshop extension of DUA, for domain-incremental driving README.md. The paper itself does not cite it.
• Superseded by: not stated in this card.