CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

In plain words

A model trained on labelled images from one setting often performs worse on images from a different setting, such as synthetic renders versus real photos. Labelling the new setting is expensive, so domain adaptation tries to carry the model over without new target labels.

Earlier methods guess a label for each unlabelled target image and then pull images with the same guessed label together across the two settings. When the guesses are wrong, the model is pulled towards wrong matches. This paper uses a transformer, whose attention lets each image part weigh the parts of the other image that look alike. The authors found this to be fairly tolerant of mismatched pairs.

The method builds pairs of source and target images, cleans up the guessed labels with a centre-based check, and trains three copies of the same transformer that share their weights: one on source images, one on target images, and one on a source-target pair, so that its attention lines the two up. The pair copy passes soft labels to the target copy. At test time only the target copy runs. The authors report their best results on VisDA-2017 and DomainNet, but not on every benchmark.

TL;DR

• Pure-transformer unsupervised domain adaptation with three weight-sharing DeiT branches. The source-target branch uses cross-attention between paired images §3.3 eq. 2.
• Pairs come from nearest neighbours in both directions, and a pair is kept only when a centre-aware pseudo label agrees with the source label §3.2 eq. 3–eq. 7.
• Author-reported, DeiT-B: 88.4 per-class accuracy on VisDA-2017 tab. 1, 92.6 on Office-31 tab. 3, 45.2 on DomainNet tab. 4. On Office-Home tab. 2 and Office-31, the ImageNet-21K-pretrained TVT is higher.
• The official code is MIT-licensed and implements the main design, but it differs from the paper in several training details (see Paper vs code). The learning rate for Office-Home, Office-31 and DomainNet is the largest gap.

Problem

A classifier trained on labelled source data loses accuracy on a target domain for which it has no labels §1. Category-level alignment methods need target pseudo labels to form source-target pairs. The authors argue that CNN-based methods do not cope well with the noisy pseudo labels this produces §1. The method assumes both domains share one label set, because the pair filter compares source labels with target pseudo labels §3.2.2. The paper does not discuss open-set or partial-label settings.

Method

The method has three parts: cross-attention as the alignment operator §3.1, the pair and pseudo-label construction §3.2, and the three-branch network with its losses §3.3.

Cross-attention §3.1. The ViT self-attention of eq. 1 is turned into a cross-attention module. Queries come from the patches of one image, and keys and values come from the patches of the other eq. 2 §3.1.1. The paper argues that each output then mixes the second image by its patches most similar to the first. The argument for robustness uses an experiment on VisDA-2017 §3.1.2 fig. 1. Random false-positive pairs replace true pairs in the training data, and accuracy is tracked against the false-positive ratio. Three curves are shown: with cross-attention, without it, and with only true pairs as an upper bound. The curve with cross-attention stays close to the upper bound and well above the one without it. Figure 1a shows a heatmap for a false-positive car–truck pair in which dissimilar patches get low weight. The text gives no numeric values for these curves.

Attn_{cross}(Q_s,K_t,V_t)=softmax\left(\frac{Q_sK_t^{T}}{\sqrt{d_k}}\right)V_t eq. 2

Pair construction §3.2.1. For each source image, take the nearest target image eq. 3. For each target image, take the nearest source image eq. 4. The training pairs are the union of the two sets, so every image of both domains appears in at least one pair. The distance d is not specified in the text. The code uses cosine similarity processor_uda.py.

Centre-aware pseudo labels §3.2.2. The source-trained classifier gives each target image a probability for each class. Class centres are the probability-weighted means of the target features eq. 5. Each target image gets the label of its nearest centre eq. 6. The centres are then recomputed from these hard labels eq. 7, using one round only. A pair is kept only if the target pseudo label equals the source label; otherwise it is dropped as noise §3.2.2. The text does not write a second assignment step after eq. 7, but the code makes one processor_uda.py.

Three weight-sharing branches §3.3 fig. 2. The source and target images of a pair go through a source branch and a target branch. Both use self-attention, and both get cross-entropy against their labels. A third, source-target branch takes its queries from the source branch in each layer, and its keys and values from the target branch in the same layer. Its output is added to the previous layer's output; the footnote says the addition is not applied in the first layer §3.3. All three branches share one classifier §3.3. The fusion branch gives soft labels that train the target branch by distillation eq. 8 §3.3.

L_{dtl}=\sum_k q_k\log p_k eq. 8

Inference uses only the target branch §3.3. The paper gives no combined loss formula. The code sums the three terms with weight 1 each processor_uda.py.

# Pair mining and pseudo labels  [§3.2.1] [§3.2.2]
# Input:  L2-normalised features, source (N_s, d) and target (N_t, d)
# Output: nearest target per source, nearest source per target

def mine_pairs(feat_source, feat_target):
    # Similarity of every source image to every target image  [§3.2.1]
    similarity = feat_source @ feat_target.T
    # Nearest target for each source image  [eq. 3]
    source_to_target = argmax(similarity, axis=1)
    # Nearest source for each target image  [eq. 4]
    target_to_source = argmax(similarity, axis=0)
    return source_to_target, target_to_source

# Centre-aware pseudo labels  [§3.2.2]
# Input:  softmax probabilities (N_t, K), target features (N_t, d)
# Output: one pseudo label per target image

def pseudo_labels(probs, feat_target):
    # Initial centres, weighted by the source-model probabilities  [eq. 5]
    centres = (probs.T @ feat_target) / probs.sum(axis=0)[:, None]
    # Label each target image by its nearest centre  [eq. 6]
    labels = argmin_distance(feat_target, centres)
    # One refinement round: centres from the hard labels  [eq. 7]
    centres = mean_per_label(feat_target, labels)
    # Final labels: nearest refined centre  [§3.2.2]
    labels = argmin_distance(feat_target, centres)
    return labels

# One CDTrans training step  [§3.3] [eq. 2] [eq. 8]
# Input:  source image x_s, target image x_t, source label y_s,
#         target pseudo label y_t
# Output: scalar loss

def training_step(x_s, x_t, y_s, y_t):
    # Source branch: self-attention, weights shared  [§3.3]
    logits_s = classifier(encode_self(x_s))
    # Target branch: self-attention, weights shared  [§3.3]
    logits_t = classifier(encode_self(x_t))
    # Fusion branch: queries from x_s, keys and values from x_t  [eq. 2]
    logits_st = classifier(encode_cross(x_s, x_t))
    # Cross-entropy on the source and target branches  [§3.3]
    loss_s = cross_entropy(logits_s, y_s)
    loss_t = cross_entropy(logits_t, y_t)
    # Distil the fusion branch's soft labels into the target branch  [eq. 8]
    loss_kd = soft_cross_entropy(teacher=logits_st, student=logits_t)
    return loss_s + loss_t + loss_kd

Hyperparameters

Values marked "code" come from the released repository, not from the paper. The paper states only the rows marked §4.1.

Results

Author-reported. Setup: DeiT-S and DeiT-B backbones at 224×224 §4.1. Starred baselines (*) were re-run by the authors with DeiT-B §4.2. The paper says most listed baselines use ResNet-50, and for VisDA-2017 the others use ResNet-101 §4.2. "Baseline" is the source-only DeiT §4.2. TVT uses a ViT pretrained on ImageNet-21K, not ImageNet-1K tab. 1 §4.2.

The DomainNet table compares only with MCD, CDAN, BNM, SWD and CGDM tab. 4. SHOT and TVT are not in it §4.2.

Pseudo-label ablation on VisDA-2017 (DeiT-B) tab. 5. "Prec" is the accuracy of the kept pairs.

Loss ablation on VisDA-2017 (DeiT-B) tab. 6. "cls" is cross-entropy, "dtl" is distillation, and the columns are the source, source-target and target branch losses.

Per class on VisDA-2017, the baseline's "person" accuracy is 10.3 and the method's is 88.6 tab. 1 §4.2. The authors take this as evidence of robustness to label noise §4.2.

Paper vs code

The repository is https://github.com/CDTrans/CDTrans, linked from the abstract and described in its README as "the official code" README.md. It carries an MIT licence LICENSE. The README says the codebase comes from TransReID README.md.

Consistent with the paper: the three branches share their projection weights and one classifier vit_pytorch_uda.py make_model.py make_model.py; the fusion query comes from the source and the keys and values from the target vit_pytorch_uda.py; the classification loss is plain cross-entropy make_loss.py; inference uses only the target branch make_model.py.

Differences:

1. Learning rate. The paper gives 3e-3 for Office-Home, Office-31 and DomainNet §4.1. None of the 13 scripts for those datasets overrides SOLVER.BASE_LR, so they run at the config value 0.008 uda.yml run_officehome_Ar.sh. Only the VisDA script sets 5e-5 run_visda.sh, which matches the paper. The card cannot tell which value produced the paper's numbers for those three datasets.
2. Schedule. The paper does not describe one. The config sets STEPS and GAMMA uda.yml, but create_scheduler reads only the epoch count, base LR and warm-up length schedular_factory.py. The effective schedule is the cosine warm-up described above.
3. Which model builds the pairs and centres. The paper says the pseudo-label centres come from the pre-trained source model §3.2.2. The code runs the current training model at every refresh, so at epochs 11, 21 and 31 the labels come from the adapted model processor_uda.py. The refresh interval is not stated in the paper.
4. Feature space. Pair retrieval uses L2-normalised features after the bottleneck processor_uda.py make_model.py. The centre step uses the CLS token before the bottleneck, because the unpacked output outputs, _, feas = probs[1] takes the pre-bottleneck feature processor_uda.py make_model.py. The two stages therefore work in different feature spaces. The paper does not say which one it uses.
5. Pair set. The code concatenates the two pair lists, and does not remove duplicates processor_uda.py processor_uda.py. The paper calls the result a union §3.2.1.
6. Distillation sign and gradients. Eq. 8 has no minus sign eq. 8. The code uses the negative sum, which is the usual soft cross-entropy processor_uda.py. The teacher is not detached, so the distillation gradient also flows into the fusion branch and the shared weights processor_uda.py.
7. First-layer residual. The paper's footnote says the cross-attention output is not added in the first layer §3.3. The code starts the fusion stream from the target tokens and adds each block's output to it, including the first block vit_pytorch_uda.py vit_pytorch_uda.py.
8. Components not in the paper. A BatchNorm1d bottleneck before the classifier make_model.py, stochastic depth of 0.1 defaults.py, gradient clipping at 1.0 processor_uda.py, mixed precision processor_uda.py, bias learning-rate factor 2 uda.yml and a fixed seed defaults.py.
9. Source pre-training. Pre-training uses label smoothing pretrain.yml; the UDA stage does not uda.yml. The paper does not describe the source pre-training.
10. Checkpoints. The README downloads the distilled DeiT checkpoints README.md. The paper cites DeiT without naming the distilled variant §4.1.
11. Target labels. The code loads target ground-truth labels into memory, but uses them only for logging accuracy, not in any loss processor_uda.py processor_uda.py processor_uda.py.
12. Checkpoint selection. Each epoch the code evaluates on the target test list. It keeps the checkpoint with the best accuracy on that list processor_uda.py. In the released scripts this list is the same target set the model trains on, and it is the only evaluation split make_dataloader.py office.py. The paper does not state its selection protocol §4.1.
13. VisDA per-class numbers. The README gives per-class values for CDTrans (DeiT-B) that differ from the Table 1 row, though both average 88.4 README.md tab. 1.

Pitfalls

• Checkpoint selection on the test split. Reported numbers may be the best epoch measured on the same target images that were used for training processor_uda.py. The paper does not say how the reported checkpoint was chosen §4.1. Use a separate protocol before comparing with other methods.
• Transductive setting. The unlabelled target images are both training data and the evaluation set. The paper says this for DomainNet §4.1, and the released scripts do the same for every dataset.
• Pseudo labels drive the result. The whole method rests on the target pseudo labels. The baseline is poor on hard classes, so the labels are noisy there, and the method's gain on those classes depends on the filter tab. 1 tab. 5. The paper reports pseudo-label precision once, for one setting tab. 5, not per refresh.
• Refresh cost. The code computes a dense similarity matrix between all source and all target images processor_uda.py. Its size grows with the product of the two set sizes. Features for every image are held in memory processor_uda.py. The pair set is rebuilt every 10 epochs, which costs a full pass over both sets.
• Gradient through the teacher. Because the teacher is not detached, the distillation term also trains the fusion branch. A stop-gradient version would behave differently, and the paper does not test one.
• Learning rate for three datasets. As above, the paper's 3e-3 and the released runs' 0.008 differ by a factor of about 2.7 for Office-Home, Office-31 and DomainNet §4.1 uda.yml. Before reproducing those numbers, decide which value to use and say so.
• Theory is a sketch under assumptions. The supplementary material (sup.tex in the source bundle) proves noise reduction by self-attention for samples drawn from well-separated Gaussian clusters [supp. §2] [supp. Thm. 1]. The constants in that theorem are only described as "universal", and the theorems do not cover this paper's branch design. Treat it as motivation. The main evidence is the experiment in fig. 1.
• Ablations have no variance. No seeds or repeat runs are reported for tab. 5 and tab. 6. The gaps there are single-run numbers (not stated as repeated).
• Backbone pre-training differs. TVT uses ImageNet-21K and is higher on Office-Home and Office-31 tab. 2 tab. 3. The authors attribute this to pre-training §4.2. Compare only with methods on the same backbone.
• Baselines use other backbones. Only starred rows were re-run on DeiT-B §4.2 tab. 1. The other rows keep their published backbones.
• Claims of "best". The abstract says the method is best on VisDA-2017 and DomainNet. On DomainNet the comparison set is the one in tab. 4, which does not include SHOT or TVT.

Lineage

• Builds on: the centre-based pseudo labels of Liang et al., "Do we really need to access the source data?" (SHOT), with extra pair filtering §2.3; the DeiT backbone §4.1; ViT §3.1.1. The code builds on TransReID README.md.
• Related: TVT, a ViT-based UDA method with ImageNet-21K pre-training (arXiv:2108.05988), which the paper reports as stronger on Office-Home and Office-31 §4.2 tab. 2 tab. 3.
• Compared against: MinEnt, DANN, CDAN+E, CDAN+BSP, CDAN+TN, rRGrad+CAT, MCD, SWD, MSTN+DSBN, SAFN+ENT, BNM, DCAN, SHOT, ATDOC-NA, CGDM and TVT §4.2.
• Superseded by: not stated in the paper.