Model Adaptation: Unsupervised Domain Adaptation without Source Data

In plain words

A classifier trained on one set of images often performs worse on a different set, for example digits photographed in another style or synthetic images versus real photos. The usual fix reuses the original training data while adapting the model. This paper looks at the harder case where that original data is not available, because of privacy or size. Only the trained model and some unlabeled images from the new domain are on hand.

The method, called 3C-GAN, trains a small image generator alongside the classifier. The generator is asked to produce images that look like the new domain and carry a chosen label. The classifier learns from those generated, labeled images. The two parts help each other: a better classifier gives the generator better feedback about which label an image shows, and better images then improve the classifier. Two extra penalties keep training stable. One keeps the classifier close to its original weights. The other pushes the decision boundaries away from dense regions of the new-domain images.

TL;DR

• Setting: unsupervised model adaptation. The source training data is not used at any point; only the source-trained classifier and unlabeled target images are available §1 §3.
• Method: a class-conditional GAN (discriminator, generator, and the classifier itself) with a weight-anchor loss and a clustering-based regularizer §3.1 §3.2 §3.3.
• Results are author-reported on digits, traffic signs, Office-31 and VisDA17, against baselines that mostly use source data §4.2.
• No official code was found; the card is based on the paper alone.

Problem

Domain adaptation normally needs labeled source data during adaptation. The authors argue that this fails when a company only ships a trained model, or when the source set is too large to move §1. Related methods that align feature distributions or translate images either need source data or cannot be applied when the source distribution is unavailable §2. The paper assumes that source and target share the same label space §1. It also assumes the source-trained model C is available and the source data D_s is not accessible during adaptation §1.

Method

The overall procedure has three networks: the source-trained classifier C (parameters theta_C), a class-conditional generator G (conditioned on a label y and noise z, so x_g = G(y, z)), and a discriminator D (parameters theta_D). C is fixed as the reference during adaptation fig. 2. The steps below follow the paper's objectives.

1. Discriminator. D is trained to separate real target images from generated images, unlike a standard conditional GAN that separates real from fake pairs. The objective is eq. 1 §3.1.
2. Generator, adversarial part. G is trained so that D cannot tell its samples from target images. The paper writes this loss as ℓ_adv eq. 2 §3.1. The printed form has a typo; see Pitfalls.
3. Generator, semantic part. Generated samples should be classified as the conditioning label. The loss ℓ_sem is the cross-entropy between the one-hot label y and the classifier's prediction on G(y, z) eq. 3 §3.1. The inspiration is InfoGAN §3.1.
4. Generator objective. G minimizes ℓ_adv + λ_s ℓ_sem eq. 4 §3.1. The paper does not state a value for λ_s (see Hyperparameters).
5. Classifier objective. C is trained on three terms eq. 5 §3.1:
   • ℓ_gen, the cross-entropy of C on generated samples G(y, z) with label y. It is defined inline in §3.1 and has no equation number.
   • ℓ_wReg, a squared distance between the current and source weights, which keeps C close to the source model eq. 6 §3.2. If the source weights are set to zero, this becomes ordinary L2 weight decay §3.2.
   • ℓ_cluReg, a clustering-based regularizer eq. 9 §3.3. It sums the conditional entropy of C on target images eq. 7 §3.3 and a KL term between predictions on each target image and on the image shifted by an adversarial perturbation r~ eq. 8 §3.3 eq. 9.
6. Alternating updates. Each mini-batch updates D, then G, then C alg. 1 §3.4. Target images x_t and noise/label pairs (y, z) are both sampled uniformly alg. 1.

# Model adaptation with 3C-GAN  [alg. 1]
# Input:  source-trained classifier C; unlabeled target images X_t
# Output: adapted classifier parameters theta_C

def adapt(C, generator, discriminator, X_t):
    # Keep the source weights as a fixed anchor  [§3.2] [eq. 6]
    source_weights = copy_of(parameters(C))
    for epoch in range(num_epochs):  # epoch count not stated  [alg. 1]
        for x_t in batches(X_t):  # target mini-batch  [alg. 1]
            # Sample labels y and noise z, both uniform  [alg. 1]
            y, z = sample_labels_and_noise()
            x_g = generator(y, z)  # class-conditional samples  [§3.1]
            # Discriminator: real target vs generated  [§3.1] [eq. 1]
            update(discriminator, maximise_d_objective(x_t, x_g))
            # Generator: fool D and keep the label's semantics  [§3.1] [eq. 4]
            adv_loss = adversarial_loss(discriminator, x_g)
            sem_loss = cross_entropy(C(x_g), y)  # [§3.1] [eq. 3]
            update(generator, adv_loss + lambda_s * sem_loss)
            # Classifier: all three terms, once adaptation has started
            # Text says only ℓ_gen and ℓ_cluReg are delayed  [§3.4]
            if adaptation_started:
                gen_loss = cross_entropy(C(x_g), y)  # [§3.1]
                wreg_loss = weight_regulariser(C, source_weights)
                clu_loss = cluster_regulariser(C, x_t)
                total = (lambda_g * gen_loss
                         + lambda_w * wreg_loss
                         + lambda_clu * clu_loss)  # [eq. 5]
                update(C, total)

def weight_regulariser(C, source_weights):
    # Squared distance to the frozen source weights  [§3.2] [eq. 6]
    return sum((parameters(C) - source_weights) ** 2)

def cluster_regulariser(C, x_t, xi):
    # Predicted class probabilities on target images  [§3.3]
    probs = softmax(C(x_t))  # (B, K)
    # Conditional entropy, minimised  [§3.3] [eq. 7]
    entropy = mean(-sum(probs * log(probs), axis=1))
    # Adversarial direction within radius xi  [§3.3] [eq. 8]
    # The paper cites VAT for how this is computed  [§3.3]
    r_tilde = adversarial_direction(C, x_t, xi)
    # KL between prediction and perturbed prediction  [§3.3] [eq. 9]
    kl = kl_divergence(probs, softmax(C(x_t + r_tilde)))
    return entropy + kl

The network shapes are given in §4.1. Digit and sign images are resized to 32 × 32 × 3. C follows the architecture of DIRT-T §4.1. The generator uses an UpResBlock module and two dense layers. The discriminator uses two dense layers and spectral normalization §4.1. For Office-31 and VisDA17, the feature extractor is a ResNet50 or ResNet101 pre-trained on ImageNet §4.1.

Hyperparameters

Results

Author-reported. Setup: digit and sign benchmarks with 10 shared digit classes and 43 traffic-sign classes §4.1. Office-31 has three domains with 2817, 795 and 498 samples, and six adaptation tasks §4.1. VisDA17 is synthetic to real with 12 shared classes, about 152k synthetic and about 55k real validation images §4.1. The baseline "Source-Only" is the source model §4.2. Other rows that appear in the paper's tables mostly used source data during adaptation §4.2.

Digits and signs (classification accuracy, %) tab. 1

In the text, the paper also reports 99.2 (SVHN → MNIST) and 96.7 (Syn.Digits → SVHN) when source data is included in training §4.2. This is a separate setting from the card's main result.

Office-31, ResNet50 (classification accuracy, %) tab. 2

VisDA17, ResNet101 (class-mean accuracy, %) tab. 3

Per-class values are in tab. 3; only averages are copied here.

Digit ablation, small LeNet-style classifier (accuracy, %) tab. 4

Smoothness ablation (accuracy, %) tab. 5

Paper vs code

No official code.

Search: the paper and arXiv listing have no code link, no project page and no comment with a URL (the source LaTeX has none). I searched GitHub for 3C-GAN, collaborative class conditional gan and model adaptation source data free domain. The results were unrelated repositories, or third-party source-free adaptation code whose owners are not authors, so none were used.

Pitfalls

• Adversarial loss typo. ℓ_adv(G) is written as E[log D(1 − G(y, z))] eq. 2. D takes an image as input, so 1 − G(y, z) does not type-check. The discriminator objective eq. 1 uses log(1 − D(G(y, z))), which is the usual GAN form. An implementation probably intends the generator to minimise log(1 − D(G(y, z))), but this is inferred, not stated, and there is no code to confirm it.
• Gating of the classifier losses differs between text and algorithm. The text says ℓ_gen and ℓ_cluReg are not applied to C until the generator produces meaningful data after several steps §3.4. Algorithm 1 gates the whole C update, including ℓ_wReg, behind a single "starting adaptation" condition alg. 1. The paper does not say when adaptation starts or whether ℓ_wReg is delayed. Pick one and state it in your code.
• λ_s is not stated. The semantic loss weight in the generator objective eq. 4 has no value anywhere in the text or algorithm. Results may depend on it because it balances the generator's adversarial term against label consistency.
• Learning-rate decay timing is not stated. Learning rates are said to drop 10 times during training §4.1, but the epochs or steps at which they drop are not given.
• Clustering regularizer is ambiguous. In eq. 9, the expectation covers the entropy term, but the KL term is written outside it. It is unclear whether the KL is averaged over the target batch, and whether r~ is treated as a constant (detached) during the update. The paper refers to VAT for the perturbation method §3.3. The number of power-iteration steps, if any, is not stated.
• Generator-classifier interaction. C is used as the semantic supervisor for G §3.1 and is then trained on G's samples. The paper describes this as collaborative. Nothing in the text bounds the feedback loop, and the authors state that the model fails to converge without ℓ_gen §4.4 tab. 4. Expect instability if C starts poorly on the target domain.
• Ablations do not isolate the entropy term. The smoothness ablation removes the smoothness constraint and keeps the entropy term §4.4 tab. 5. The clustering regularizer is removed by setting λ_clu = 0 §4.4, which is reported only in prose. No run uses the entropy term without smoothness.
• Numerical claims in the text that do not match the tables.
  • "outperforms the baseline by around 40%" for MNIST → MNIST-M §4.2. Table 1 gives 54.2 → 98.5, which is about 44 points tab. 1.
  • "about 7 percentage points" over JDDA on MNIST → MNIST-M §4.4. Table 4 gives 97.0 vs 88.4, which is 8.6 points tab. 4.
  • "outperforms SE by 9.1% without data augmentation" §4.2. 9.1 is the difference between 83.3 (enhanced ResNet101, †) and 74.2. The standard ResNet101 result, 81.6, gives 7.4 tab. 3. Use the table values, not the prose.
• Two backbones for the same digit task. The main digit table uses the DIRT-T-style classifier §4.1. The ablation table uses a small LeNet-style classifier from JDDA tab. 4. Their numbers are not comparable.
• Run-to-run variance is unexplained. The ± values are reported but the number of runs and whether they are standard deviations or standard errors are not stated tab. 1 tab. 2 tab. 3.
• Source-only baselines differ by setting. Office-31 uses a fine-tuned ResNet50 on the source domain as the baseline §4.2. The digit baseline is the source-trained classifier with the same architecture §4.2.
• Comparisons use source data. The paper says most compared methods need source data during adaptation §4.2. Compare results with that in mind.
• Source data is used to pre-train the source model. The method does not use source data during adaptation, but the source model is trained on it §4. Whether the source model is available is a deployment question, not one the method answers.

Lineage

• Builds on: the weight-anchor idea is credited to two separate-model adaptation methods (Beyond Sharing Weights and ADDA) §3.2; the semantic loss is inspired by InfoGAN §3.1; the perturbation method is VAT §3.3; the cluster assumption and entropy term come from a semi-supervised entropy paper §3.3. None of these have cards here, so they are named but not linked.
• Compared against: DAN, DANN, DDC, MCD, SWD, GenToAdapt, MADA, CyCADA, DIRT-T, SimDA, Self-Ensembling and others in tab. 1 tab. 2 tab. 3.