Cross-Domain Gradient Discrepancy Minimization for Unsupervised Domain Adaptation

In plain words

A model learns from a labelled picture collection (the source) and must then work on a different collection (the target) that has no labels. Two classifier heads that disagree with each other is a known signal for "this target picture sits outside what the model knows", and earlier methods train the feature extractor to make the heads agree. The paper points out that agreement does not mean correctness: both heads can be confidently wrong in the same way.

The fix compares learning signals instead of answers. The method computes how much each source picture pushes the classifier weights, does the same for target pictures using guessed labels, and trains the feature extractor so that the two pushes point in the same direction. Guessed labels for the target come from grouping target features into clusters and naming each cluster after the nearest centre. The authors report that this gives better accuracy on three standard benchmarks.

TL;DR

• Starts from bi-classifier adversarial UDA (MCD-style): two classifier heads, a disagreement term, and a feature extractor that reduces it §3.1.
• Adds a cosine loss between the classifier gradients from source samples and from pseudo-labelled target samples, applied to the feature extractor §3.2.
• Pseudo labels come from weighted, cosine-distance clustering; a self-supervised classification loss on them is added to the supervised step §3.3.
• Author-reported results: VisDA-2017 82.3%, DomainNet 27.2% average, ImageCLEF 89.5% average tab. 1–3.

Problem

Bi-classifier adversarial UDA aligns the two heads' outputs. Because only the disagreement is minimised, target samples can match wrong categories while the disagreement stays small §1 fig. 1. Pseudo-label self-training is an alternative, but hard target labels from the source-only model are noisy, and training on them can spread errors §1. The paper's theory section treats the category-level misalignment as a large joint-error term in the domain adaptation bound §4, eq. 14.

Method

3.1 Bi-classifier adversarial learning (baseline)

A generator G maps inputs to features; two heads F1 and F2 produce class probabilities §3.1. Each iteration has three steps:

1. Step 1 trains G, F1, F2 to classify labelled source samples eq. 1.
2. Step 2 freezes G and trains the heads to keep source accuracy while maximising their disagreement on target samples, measured with an ℓ1 distance between the two softmax outputs eq. 2.
3. Step 3 freezes the heads and trains G to reduce that disagreement eq. 3.

3.2 Cross-domain gradient discrepancy (the new loss)

The paper argues that if one classifier classifies all samples correctly, the gradients that source and target samples give it should be similar §3.2. Two expected gradients are defined over the classifier parameters, averaged over the two heads: g_s from source samples with ordinary cross entropy eq. 4, and g_t from target samples with the weighted cross entropy below eq. 5. The loss is 1 - cos(g_s, g_t) on the flattened vectors eq. 8. It is added to step 3 with weight β eq. 13.

Weighted cross entropy on target samples eq. 6: each sample's cross entropy is multiplied by w = 1 + exp(-E), where E is the Shannon entropy of that sample's softmax output eq. 7. Confident predictions (low entropy) therefore get weights near 2, and uncertain ones near 1.

3.3 Clustering-based pseudo labels

Target labels are needed for g_t, so the paper builds them once per epoch §3.3:

1. Each target sample gets a weighted centre per class: the average of its generator features, weighted by the softmax probability of that class from both heads eq. 9.
2. Each target sample takes the label of the nearest centre under cosine distance eq. 10.

These labels also drive a self-supervised term: the same weighted cross entropy on target samples, averaged over both heads, with weight α eq. 11, eq. 12.

3.4 Objective and schedule

• Step 1 minimises L_cls(source) + α · L_cls^W(target, pseudo labels) eq. 12.
• Step 2 is unchanged from the baseline eq. 2.
• Step 3 minimises L_dis(target) + β · L_GD eq. 13.
• The three steps repeat each epoch; the pseudo labels are refreshed each epoch alg. 1. The paper does not state how many times step 3 repeats per epoch.

# Pseudo labels for every target sample, once per epoch  [eq. 9] [eq. 10]
# Paper form. The code differs; see "Paper vs code".
def pseudo_labels(x_t_all):
    raw = G(x_t_all)                                 # (N_t, D)
    # Per-class weights: softmax of each head, summed  [eq. 9]
    weights = softmax(F1(raw)) + softmax(F2(raw))    # (N_t, C)
    feats = normalise(raw)                           # (N_t, D), L2 per row
    for k in range(C):
        # Weighted class centroid  [eq. 9]
        centroid[k] = sum_i(weights[i, k] * raw[i]) / sum_i(weights[i, k])
    # Nearest centroid by cosine distance  [eq. 10]
    return argmin_k cosine_distance(feats, centroid[k])

# Weighted cross entropy on target samples  [eq. 6] [eq. 7]
# Reduction is not stated in the paper; the code uses a normalised mean.
def weighted_ce(logits, labels):
    probs = softmax(logits)                          # (B, C)
    # Shannon entropy of each prediction  (B,)
    entropy = -sum(probs * log(probs), axis=1)
    # Confident samples weigh up to 2, uncertain ones about 1  [eq. 7]
    weight = 1 + exp(-entropy)                       # (B,)
    return mean(weight * cross_entropy(logits, labels))  # [eq. 6]

# One iteration of CGDM  [alg. 1]
# Hyperparameters: alpha = 0.1, beta = 0.01  [§5.2] [§5.4]
def cgdm_iteration(x_s, y_s, x_t, y_t):
    # Step 1: classify source, learn from target pseudo labels;
    # update G, F1, F2  [eq. 1] [eq. 11] [eq. 12]
    feats = G(concat(x_s, x_t))
    l1 = F1(feats)                                   # (B_s + B_t, C)
    l2 = F2(feats)
    loss_cls = 0.5 * (ce(l1[:B_s], y_s) + ce(l2[:B_s], y_s))
    loss_pl = 0.5 * (weighted_ce(l1[B_s:], y_t) + weighted_ce(l2[B_s:], y_t))
    update(G, F1, F2, loss_cls + alpha * loss_pl)

    # Step 2: heads disagree on target, keep source accuracy;
    # update F1, F2 only  [eq. 2]
    feats = G(concat(x_s, x_t)).detach()
    l1 = F1(feats)
    l2 = F2(feats)
    loss_cls = 0.5 * (ce(l1[:B_s], y_s) + ce(l2[:B_s], y_s))
    loss_dis = l1_distance(softmax(l1[B_s:]), softmax(l2[B_s:]))
    update(F1, F2, loss_cls - loss_dis)

    # Step 3: generator reduces disagreement and the gradient gap;
    # update G only  [eq. 3] [eq. 4] [eq. 5] [eq. 8] [eq. 13]
    feats = G(concat(x_s, x_t))
    l1 = F1(feats)
    l2 = F2(feats)
    loss_dis = l1_distance(softmax(l1[B_s:]), softmax(l2[B_s:]))
    # Expected gradients over both heads, flattened  [eq. 4] [eq. 5]
    g_src = 0.5 * (grad(ce(l1[:B_s], y_s), F1) + grad(ce(l2[:B_s], y_s), F2))
    g_tgt = 0.5 * (grad(weighted_ce(l1[B_s:], y_t), F1)
                   + grad(weighted_ce(l2[B_s:], y_t), F2))
    # Keep the graph so the gradient gap can be back-propagated  [eq. 8]
    gd = 1 - cosine(flatten(g_src), flatten(g_tgt))
    update(G, loss_dis + beta * gd)

Hyperparameters

Results

Author-reported. The paper reports single-number results on three benchmarks with the backbone named in each caption. Its ImageCLEF table gives mean ± std; the number of runs is not stated.

VisDA-2017 (synthetic to real, ResNet-101, 10 epochs §5.3) tab. 2. Column key: Aero = aeroplane, Bike = bicycle, Moto = motorcycle, Board = skateboard, Plant = potted plant.

DomainNet (ResNet-50, 10 epochs §5.3): six domains, all 30 ordered pairs. The CGDM block as printed tab. 1. Rows and columns are labelled as printed; the caption says columns are sources and rows are targets. The Avg. row is the mean of each column as printed.

Average over the six-domain setting for other methods, as printed tab. 1: ResNet 20.3, MCD 20.5, BNM 25.3, CDAN 21.2, SWD 23.6.

ImageCLEF (ResNet-50, 100 epochs §5.3) tab. 3:

Model analysis (figure-only numbers; no values stated in the text):

• t-SNE of C→I features on ImageCLEF: source-only target features are disordered; DANN reduces the gap but hurts discriminability; MCD helps discriminability; CGDM aligns target classes with source classes fig. 3.
• Training curves for C→I on ImageCLEF: source classification loss falls and target accuracy rises over iterations fig. 4.
• Sensitivity: α fixed at 0.1; β from 0 to 5 on C→I. Accuracy is flat for β in [0, 1] and falls above 1 fig. 5a §5.4.
• Ablation on ImageCLEF (mean accuracy, figure only): each module improves on vanilla bi-classifier learning, and the full model is best fig. 5b §5.4.
• Per-class gradient matching (aligning source and target gradients within each category) gave no clear gain over the batch-level version the paper uses §5.4.

Paper vs code

Official repository: lijin118/CGDM at commit 012d631, linked from the abstract. It has no LICENSE file, so code may not be copied; the references below cite paths and lines only. The repository contains three training scripts (train_image_clef.py, train_visda.py, train_domainnet.py), a shared utils.py and the networks in models/basenet.py.

• Scope of each script. Each script runs one transfer, hard-coded in the file: ImageCLEF I→C train_image_clef.py; DomainNet real→sketch train_domainnet.py; VisDA synthetic→real via directory defaults train_visda.py. The paper's tables cover six ImageCLEF pairs, thirty DomainNet pairs and VisDA §5.1 tab. 1–3. The repository has no script that loops over the others.
• Loss scale. Paper eq. 1 and eq. 11 average each head's loss with a factor of 1/2 eq. 1 eq. 11. The code sums the two heads without it [code: train_image_clef.py#L192-L194]. The source-to-target ratio in step 1 is unchanged, but step 1 as a whole is twice the paper's scale, and in step 2 the source term is twice the paper's weight relative to the disagreement term eq. 2.
• Class-balance term. The code adds Entropy(p_t1) + Entropy(p_t2) to every step, with weight 0.1 (ImageCLEF, DomainNet) or 0.01 (VisDA) [code: train_image_clef.py#L182-L183] train_image_clef.py [code: train_image_clef.py#L256]. The helper is the mean per-sample entropy minus the entropy of the batch-mean prediction utils.py. The paper names only "the class balance loss" §5.2 and writes no such term in eqs. 2 or 3, so the paper does not say it is added to step 2 or 3.
• Target supervision weight in VisDA. The VisDA script uses α = 0.01 for the pseudo-label term, not 0.1 train_visda.py. The DomainNet script uses 0.1 for entropy and 0.01 for the pseudo-label term [code: train_domainnet.py#L187].
• Bottleneck not trained in VisDA and DomainNet. optimizer_g is built from G.features only train_visda.py train_domainnet.py. ResBottle keeps the 2048→256 bottleneck outside features [code: models/basenet.py#L70-L77], so in those two scripts the bottleneck is applied in every forward pass but is never updated. The ImageCLEF script optimises all of G train_image_clef.py. The paper says nothing about this split.
• Pseudo labels. The paper weights each class centre by the softmax of each head, summed over heads eq. 9 eq. 10. The code softmaxes the sum of the two heads' logits utils.py. The code appends a constant 1 to each feature (a bias term) and L2-normalises it utils.py. It then computes weighted centroids once, then runs one refinement round with hard one-hot centroids utils.py. The paper describes only the weighted assignment.
• Pseudo-label timing. ImageCLEF and DomainNet refresh labels at the start of each epoch train_image_clef.py train_domainnet.py. VisDA refreshes every pseudo_interval = 2000 iterations train_visda.py train_visda.py.
• Start epoch. The code starts the pseudo-label, self-supervised and gradient terms after epoch 0 in VisDA and DomainNet train_visda.py train_domainnet.py, but from epoch 0 in ImageCLEF [code: train_image_clef.py#L112]. The paper does not state a start epoch.
• Step 3 repeats. The code repeats the generator update num_k times per batch, with default 4 train_image_clef.py train_image_clef.py. The paper does not state a count. The code also applies the entropy term in this loop train_image_clef.py.
• Gradient cosine. The paper takes one cosine over the flattened gradients eq. 8. The code takes a cosine per parameter tensor and averages them, with rows for matrices (dim = 1) and the whole vector for biases and BatchNorm parameters utils.py. It averages the mean for each head utils.py utils.py and then the two heads [code: utils.py#L410]. The gradients are taken with respect to the heads' parameters, as in the paper, and the target term is the weighted cross entropy [code: utils.py#L348-L351].
• Per-class gradient function is unused. gradient_discrepancy_loss (per class, with gmn_N classes sampled) exists but training calls the batch-level gradient_discrepancy_loss_margin only utils.py [code: utils.py#L337-L412] train_image_clef.py. This matches §5.4, where the batch-level version is the one the paper uses.
• Disagreement scale. The paper's "ℓ1 distance" eq. 2 is implemented as a mean of |p1 − p2| over both batch and classes utils.py, which is 1/C of a per-sample ℓ1 sum (C = 12 for ImageCLEF and VisDA, 345 for DomainNet).
• Weighted cross entropy normalisation. The code divides the weights by their sum over the batch, a detached constant, and then takes the mean of the weighted losses utils.py. The paper writes w · CE with no normalisation eq. 6. The code's result is the normalised weighted mean divided by the batch size.
• Test-time prediction. The paper says predictions average the two heads' outputs §5.3. The code sums the two logit vectors and takes the argmax train_image_clef.py. Argmax of the sum of logits equals argmax of the mean of logits, not of the mean of probabilities.
• Epoch counts. The script defaults are 200 for ImageCLEF, 200 for DomainNet and 20 for VisDA train_image_clef.py train_domainnet.py train_visda.py. The paper reports 100 (ImageCLEF), 10 (DomainNet) and 10 (VisDA) §5.3. The code prints test accuracy after every epoch [code: train_image_clef.py#L282-L318]. The paper does not say which epoch it reports.
• Learning rate for the backbone. The paper says the backbone learns 10 times slower than the heads §5.2. No script does this.
• Random transforms at evaluation. The paper says it uses no ten-crop test augmentation §5.2. The code's validation transform still applies a random horizontal flip and a random crop train_image_clef.py, and the test and pseudo-label loaders both use it train_image_clef.py. Target predictions and pseudo labels therefore change between runs with the same weights.
• Source and target share one forward pass. Source and target batches are concatenated into one 64-image forward pass train_image_clef.py, so BatchNorm statistics mix both domains. The paper does not say this.

No discrepancies were found in the gradient-discrepancy coefficient β = 0.01, the momentum (0.9), the weight decay (5e-4), the classifier width (1000), the bottleneck width (256) or the learning rate of VisDA (3e-4) and DomainNet and ImageCLEF (1e-3) §5.2.

Pitfalls

• Loss scale. Because the code drops the 1/2 factor in eqs. 1 and 11 (see Paper vs code), the same α and β values do not transfer from the paper unchanged. Use the code's scale, or rescale α and the source term by 1/2 together. The ℓ1 scale also depends on the class averaging in the code.
• Pseudo labels are noisy and random. Labels come from a randomly augmented target set and one refinement round. The code logs pseudo-label accuracy against target ground truth for printing only utils.py. The paper does not report that number.
• Gradient cosine is computed through the graph. Each step 3 update needs a double backward through both heads for every parameter tensor, with create_graph=True utils.py. The cost scales with the number of repeats num_k (default 4). The paper does not state the cost.
• First epoch without pseudo-label terms in VisDA and DomainNet. Those scripts skip the pseudo-label and gradient terms in epoch 0 [code: train_visda.py#L157] train_domainnet.py, so epoch 0 trains on the source loss, the disagreement term and the entropy term only.
• Algorithm 1 cites the wrong equation numbers. The algorithm says pseudo labels come from "Eq. (7) and Eq. (8)" alg. 1. Counting the numbered equations in the LaTeX source, the pseudo-label equations are eq. 9 and eq. 10; eq. 7 is the weighting function and eq. 8 is the gradient loss. This card uses the counted numbers.
• No LICENSE in the repository. The code cannot be copied, and the paper's CC BY 4.0 licence does not cover it. Re-implement from this card and the cited lines.

Lineage

• Builds on: bi-classifier adversarial UDA with ℓ1 discrepancy (MCD) and with a sliced Wasserstein discrepancy (SWD) §2 §3.1. CGDM keeps the three-step schedule and adds the gradient term and the pseudo-label term.
• Uses: clustering-based pseudo labels as in DeepCluster and SHOT §2 §3.3.
• Compared against: DAN, RTN, DANN, MinEnt, MCD, ADR, SWD, CDAN+E, AFN, BNM, MSTN+DSBN tab. 2 tab. 3; CDAN, MCD, BNM, SWD on DomainNet tab. 1.
• Theoretical motivation: the Ben-David domain adaptation bound, with a joint-error term that category-level misalignment inflates §4, eq. 14, and a claim from §4 that the gradient gap restricts the optimisation path of the joint hypothesis.
• Superseded by: not stated in this card.