A Free Lunch for Unsupervised Domain Adaptive Object Detection without Source Data

In plain words

A detector trained on one kind of image, say clear daytime street scenes, often performs worse on another, such as foggy streets or a different camera. Adapting it normally requires the original training images as well as the new ones. This paper asks what is possible when only the trained model and the new, unlabeled images are available.

The method lets the old model label the new images itself. Those labels are imperfect: some real objects are missed and some background is wrongly marked as objects. The authors choose how confident the model must be before a box counts as an object by checking how sure the model is about its own predictions after retraining on those labels. They also reduce the damage from missed objects by pasting four images together into one, which makes small and partly hidden objects appear more often during training.

The point is practical: a team can improve a deployed detector on new conditions without keeping or sharing the data it was originally trained on.

TL;DR

SFOD is a source-free way to adapt a Faster R-CNN detector. It labels unlabeled target images with the source model and keeps only boxes above a confidence threshold. The threshold comes from self-entropy descent, and Mosaic augmentation is applied to reduce the effect of missed objects. The authors report gains over the source-only model on four adaptation tasks, with results that are close to, but in some tasks below, source-based adaptation methods. Results are not independently verified and no code is published.

Problem

Standard unsupervised domain adaptation for detection needs labeled source data and unlabeled target data together § Introduction. The authors argue that sharing source images is often impractical because of privacy and transmission cost § Introduction. In the source-free setting only the source-trained weights are available, and the target images have no labels § Method.

With no target labels, the threshold that separates positive from negative detections cannot be tuned directly. Pseudo labels are noisy in both directions, and the authors find that missed objects (false negatives) make up most of the noise § Method (False Negatives Simulation) fig. 5.

Method

The detector is Faster R-CNN § Method (Pseudo Labels Optimization via SED). The paper's section headings are unnumbered, so this card cites them as [§ Section] (Subsection), for example [§ Method] (Self-entropy descent). Equations and figures are cited by the paper's numbering.

Step 1: Pseudo labels. Run the source model on every target image. A detection is labeled with its highest-scoring foreground class if that score is above the threshold h. Otherwise it is labeled background § Method (Self-entropy descent) eq. 2.

Step 2: Training loss. Fine-tune on the pseudo labels with the standard Faster R-CNN loss, which is the sum of the region-proposal, classification and box-regression losses, with no weights stated eq. 3. The authors use the source model's boxes as the targets for proposal and regression, and only the classification targets come from the thresholded pseudo labels. They justify this by claiming that localisation error matters less than classification error for detection § Method (Self-entropy descent).

Step 3: Self-entropy of the adapted model. Fine-tune from the source weights with the pseudo labels for threshold h, giving model θ_t eq. 4. Run θ_t over the target images and take the class probabilities of each sample. The self-entropy of each sample is the average over classes of −p log p. The mean self-entropy H(D_t) is the average of these values over the target set eq. 1 eq. 5 eq. 6. The paper does not state whether a "sample" is an image or a box.

Step 4: Threshold search (SED). Try thresholds from high confidence to low confidence. Stop at the first local minimum of H(D_t) along that scan § Method (Self-entropy descent) fig. 4. The text gives the selection as h_optimal = argmin_h H(D_t) eq. 7. The two rules differ, see Pitfalls.

Step 5: False-negative simulation with Mosaic. Mosaic joins four target images with random scaling and random cutting, and their pseudo-labeled boxes are combined with the same geometry § Method (False Negatives Simulation) eq. 8 eq. 9. The authors use it because it can create small and partly visible objects from boxes that were already detected, without removing true negatives § Method (False Negatives Simulation). The authors cite Mosaic as Bochkovskiy et al. § Method (False Negatives Simulation).

Reconstructed pseudocode for Steps 1 to 4. The paper gives no algorithm box, and the comments cite the text, not an [alg. N]:

# Self-entropy descent (SED) threshold search
# Input:  target_images; theta_s (trained on the source domain only)
# Output: threshold h, adapted model theta_t

def mean_self_entropy(model, samples):
    # H(D_t): average over samples of the class-averaged -p log p  [eq. 1]
    total = 0.0
    for sample in samples:                         # unit not stated
        probs = model(sample)                      # (n_c,) class probabilities
        total += -mean(probs * log(probs))         # mean over n_c classes
    return total / len(samples)

def search_threshold(target_images, theta_s, thresholds):
    # Scan thresholds from high confidence to low  [§ Method] (SED)
    history = []
    for h in thresholds:                           # grid step not stated
        # Keep foreground pseudo labels above h, else background  [eq. 2]
        pseudo = label_with_threshold(theta_s, target_images, h)
        # Fine-tune starting from the source weights  [eq. 4]
        theta_t = train(target_images, pseudo, init=theta_s)
        # Entropy of the fine-tuned model over the target set  [eq. 5] [eq. 6]
        history.append(mean_self_entropy(theta_t, target_images))
        # Stop at the first local minimum of the descending curve  [§ Method]
        if is_first_local_minimum(history):
            return h, theta_t
    # Behaviour when no minimum appears: not stated
    return thresholds[-1], theta_t

Mosaic (Step 5) reconstructed from the text eq. 8 eq. 9:

# Mosaic on pseudo-labeled target images  [§ Method] (False Negatives Simulation)
# Input:  four (image, boxes) pairs A, B, C, D from the target set
# Output: one mosaic image with merged boxes

def mosaic(samples, canvas_w, canvas_h):
    # Random scale s(.) for each of the four images  [eq. 8]
    scaled = [random_scale(image) for image, boxes in samples]
    # Random cut point lambda and 2D translation (u, v)  [eq. 8] [eq. 9]
    cut_x, cut_y = random_cut(canvas_w, canvas_h)   # distribution not stated
    # Binary masks M_A..M_D select the visible part of each image  [eq. 8]
    tiles = place_with_masks(scaled, cut_x, cut_y)
    # Box merge rule: not defined in the text; shift and clip boxes here
    merged_boxes = merge_boxes(samples, cut_x, cut_y)
    return tiles, merged_boxes

Training regime details are in the Hyperparameters table. Only the four adaptation tasks listed in Results are reported.

Hyperparameters

Datasets and their sizes § Experiments (Datasets):

The mask-to-box conversion for Cityscapes is not described.

Results

Author-reported. The source-only and oracle rows are defined in the paper as training on labeled source data and on labeled target data respectively [§ Experiments] (Comparison Results). Hardware and run-to-run variance are not stated. "Ideal" rows use a threshold chosen with labeled target validation data, so they are not deployable § Experiments (Adaptation to A New Sense). The baseline numbers' source (re-run or quoted) is not stated.

Car AP, KITTI to Cityscapes tab. 1 § Experiments (Adaptation to A New Sense):

Car AP, Sim10k to Cityscapes tab. 2 § Experiments (Adaptation from Synthetic to Real Images):

mAP over seven classes, Cityscapes to BDD100k daytime tab. 3 (train has no entries; mAP is the mean of the seven values) § Experiments (Adaptation to Large-Scale Dataset):

mAP over eight classes, Cityscapes to Foggy Cityscapes tab. 4 (the defogging column marks whether the target images were defogged first, as in DA-Detection) § Experiments (Adaptation from Normal to Foggy Weather):

Headline comparisons in the text, which the tables bear out: Mosaic adds 1.0 AP over SED alone on KITTI to Cityscapes tab. 1; SFOD plus Mosaic, SED threshold reaches 29.0 mAP against 26.9 for CR-DA-DET on Cityscapes to BDD100k daytime tab. 3; and SFOD plus Mosaic, SED threshold reaches 33.5 mAP on Foggy Cityscapes against 22.3 for the source-only model tab. 4. On the foggy task, DA-Detection, CR-DA-DET and AT-Faster are all higher than SFOD without defogging.

Paper vs code

No official code. The paper's LaTeX source has no repository link and the arXiv record in our metadata gives no code link. No project page is linked. A GitHub repository search for "SFOD", "source data-free domain adaptive object detection" and "self-entropy descent" returned no repository owned by the authors. Several results carry the SFOD name but belong to unrelated projects.

Pitfalls

• Stopping rule conflicts. The text describes a scan that stops at the first local minimum, but the formula writes an argmin over all thresholds § Method (Self-entropy descent) eq. 7. The two can give different thresholds. The paper's own toy example shows local minima at both ends of the curve, caused by class imbalance, and the authors say the ideal assignment gives the lowest entropy § Method (A toy example) fig. 3. The first-minimum rule can stop early. Grid step and any smoothing are not stated.
• Cost per threshold. Each candidate threshold needs a fine-tune on the pseudo labels and a re-evaluation eq. 4 eq. 5. The number of candidates and the training length per candidate are not stated, so the total compute of the search is unknown.
• Unit of the entropy average. Eq. 1 averages over samples x_t^i and divides by N_t, but the text does not say whether a sample is an image or a box. Implementers must pick one and report it.
• Mosaic dimensions. Eq. 8 writes the mosaic as a canvas of size W x H, but the masks are sized s(W) x s(H) eq. 8. The canvas size, the scale range, the cut-point distribution and the Mosaic probability are not stated.
• Box merge rule. Eq. 9 gives a weighted sum of label values, which does not say how box lists from four images are concatenated, shifted and clipped. Implementers must define it themselves.
• Duplicate rows in the tables. In tab. 3 and tab. 4 the SFOD plus Mosaic, labeled-search threshold row matches the SFOD plus Mosaic, SED threshold row in every column, and in tab. 1 the two values are both 44.6. Treat the Ideal-Mosaic numbers as unconfirmed.
• Defogging gain. The text says defogging adds "approximately 3%" § Experiments (Adaptation from Normal to Foggy Weather). The table differences are 2.8 for SED with Mosaic (33.5 to 36.3), 3.6 for SED without Mosaic (30.6 to 34.2) and 4.1 for the ideal threshold (31.0 to 35.1). The defogging method is not described in this paper, only that it is the same as DA-Detection's § Experiments (Adaptation from Normal to Foggy Weather).
• Source model dependence. The method is source-free only after the source model exists. The paper does not state how that model was trained or whether its recipe was tuned § Experiments (Implementation Details).
• Missing optimisation details. Momentum, weight decay, iteration count, LR schedule, anchors and NMS are not stated. Batch size 1 and LR 0.001 are the only optimisation values given.
• Baseline provenance. The text does not say whether the baseline numbers were re-run or copied from the cited papers.
• Cityscapes boxes. The conversion from instance masks to boxes is not described, so the box definition may differ from other Cityscapes detection setups.
• Claims beyond the tables. The discussion says the method is comparable to or better than source-based methods § Discussion and Analysis. This holds on tab. 1 and tab. 2 for the chosen rows, but not on tab. 4 without defogging.
• Scope. Four adaptation tasks, one detector (Faster R-CNN with VGG-16), and no timing or memory cost reported.

Lineage

• Builds on: Faster R-CNN, the detector used throughout § Method (Pseudo Labels Optimization via SED).
• Builds on: pseudo-label self-training for detection, with the source model as teacher § Method (Self-entropy descent).
• Builds on: Mosaic augmentation (cited as Bochkovskiy et al.), applied here to simulate false negatives § Method (False Negatives Simulation).
• Compared against: DA-Faster, SW-Faster, CR-DA-DET, AT-Faster, MAF, Noise Labeling and DA-Detection, which all use source data tab. 1 tab. 3 tab. 4.
• Superseded by: not known from content already in this wiki.