SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detection

In plain words

A detector trained on one kind of image, such as clear-weather city streets, often misses objects in another kind, such as foggy streets. Domain adaptation tries to close that gap using unlabelled images from the new kind. Earlier methods compare a single average summary per object class across the two image sets and push those averages together. That ignores how much objects of one class vary, and it breaks when a class appears in only one of the two sets.

SIGMA treats the problem as matching. It picks many small image patches that belong to objects or background, links them into a small graph for each image set, and fills in placeholder points for classes that a batch lacks in one domain. It then pairs up points across the two graphs, favouring pairs that belong to the same class and whose neighbours also line up. Pulling matched pairs together gives a finer alignment than moving one average per class.

TL;DR

• Targets domain-adaptive object detection with a single-stage (FCOS-style) detector §1 app. B.1.
• Replaces class-centre alignment with graph matching: nodes are sampled pixel features, edges link nodes within each domain §3 §4.1.
• Hallucinates nodes for classes missing from one domain, using a memory bank of source and target class statistics §4.1.
• Author-reported: 43.5 mAP (VGG-16) and 44.2 mAP (ResNet-50) on Cityscapes to Foggy Cityscapes tab. 1. Results are not reproduced here.
• No official code was found. The paper's link returns 404.

Problem

Category-level adaptation methods align one prototype per class across domains §1 §3. The paper argues this has two faults: (i) it ignores within-class variance in size and appearance §1; (ii) a class present in only one domain's batch has no target-side or source-side centre, so it is left unaligned §1 §4.1. The method assumes the two domains share a category space §4.1. The paper states that a shared category space is the usual assumption in this literature, citing the DA-Faster R-CNN line of work §4.1.

Method

Setting §3: a batch of labelled source images and unlabelled target images goes through a shared backbone φ. The SIGMA framework has three parts: Graph-embedded Semantic Completion (GSC, §4.1), Bipartite Graph Matching (BGM, §4.2), and the training objective (§4.3) fig. 2.

1. Node sampling. Source nodes are pixels inside ground-truth boxes, with a class label. Background nodes are a ratio 1/(C+1) of pixels outside boxes §4.1. Target nodes are chosen by the detector's pseudo score maps: foreground if the max score exceeds τ_fg, background if it is below τ_bg §4.1 [fn. 1 in §4.1]. At most 100 nodes are sampled per feature map per domain §5.2 tab. 4.
2. Vision-to-graph (V2G). A non-linear projection maps sampled features to raw node embeddings. The projection is Fc, LayerNorm, ReLU, Fc app. C.2 item 1 §4.1.
3. Node alignment (adversarial). A node discriminator, applied to raw node embeddings, is trained with a gradient-reversal layer and a binary cross-entropy loss §4.3 app. C.1 tab. 9.
4. Domain-guided node completion (DNC). For each class missing from one domain's batch, the method draws hallucinated nodes. It takes the standard deviation of the other domain's nodes in that class as the spread, and the memory-bank seed of the missing domain as the mean. Gaussian samples are passed through a linear projection §4.1. The paper states the spread is a "standard variance" and passes it as the second argument of the Gaussian §4.1. Hallucinated nodes are kept in the graph but excluded from memory updates §4.1.
5. Cross-image graph. Edges are the softmax of a learned projection of node pairs, after edge dropout with rate 0.1 §4.1 app. C.2 item 2. One graph-convolution layer with LayerNorm and a residual connection produces enhanced nodes §4.1.
6. Graph-guided memory bank. Each class has a memory seed. For each class in a batch, the existing (non-hallucinated) enhanced nodes plus the seed are clustered into two groups, and only the seed-containing group updates the seed through an adaptive momentum §4.1 eq. 2. Spectral clustering is used only when there are more than 5 nodes, with a 5-nearest-neighbour affinity in place of the Laplacian app. C.2 item 3.
7. Cross-graph interaction (CGI). Each domain's nodes attend to the other domain's nodes, with LayerNorm and a residual connection §4.2 eq. 3. A classifier on the output is trained with cross-entropy on source labels and on target pseudo-labels §4.2 eq. 4.
8. Semantic-aware node affinity (SNA). For every source and target node pair, the two embeddings are concatenated after a linear projection and passed through a single-output MLP. Instance normalisation and a Sinkhorn layer with k = 20 iterations give a double-stochastic affinity matrix §4.2.
9. Structure-aware matching loss (SML). The loss has three terms §4.2 eq. 5: a true-positive term that pushes up the best match for each source node within its class; a false-positive term that suppresses affinity between nodes of different classes; and a quadratic term that penalises changes in the affinity between neighbours of matched nodes.
10. Total objective. Node loss and matching loss are weighted by λ1 and λ2; the node-alignment, global-alignment and detection losses are added with weight 1 §4.3 eq. 6.
11. Training loop. alg. 1 lists these steps in order, once per iteration. Only the total loss is backpropagated.

Equations (as in the source; notation simplified where noted):

\min_{\Pi} \mathcal{F}(\Pi) = \lVert \mathcal{A}_s - \Pi\,\mathcal{A}_t\,\Pi^{T} \rVert_F^2 - \mathrm{tr}(\mathbf{X}_u^{T}\Pi) eq. 1

with Π ∈ [0,1]^{N_s × N_t}, Π 1 ≤ 1 and Πᵀ 1 ≤ 1 §3. Here A_s and A_t are the graph adjacency matrices, and X_u is the node affinity §3.

\mathbf{S}^{(\omega)} \leftarrow \mathrm{sim}(b, \mathbf{S}^{(\omega)})\,\mathbf{S}^{(\omega)} + [1 - \mathrm{sim}(b, \mathbf{S}^{(\omega)})]\,b eq. 2

with sim the cosine similarity and b = 1/(|π_seed| − 1) · Σ over π_seed of ṽ §4.1. The paper writes this as the "adaptive momentum".

\hat{\mathcal{V}}_s = \mathrm{LN}\{\mathrm{softmax}[(\tilde{\mathcal{V}}_s \mathcal{W}_q)(\tilde{\mathcal{V}}_t \mathcal{W}_k)^{T}]\,(\tilde{\mathcal{V}}_t \mathcal{W}_v)\mathcal{W}_p + \tilde{\mathcal{V}}_s\} eq. 3

and the same with s and t swapped for the target nodes eq. 3.

\mathcal{L}_{node} = -\sum_{i=1}^{N_s + N_t} y_i \log\{\mathrm{softmax}[f_{cls}(\hat{v}^i)]\} eq. 4

where y_i is the ground-truth label for source nodes and the pseudo-label from M_t for target nodes §4.2.

\begin{aligned} \mathcal{L}_{mat} = & \sum_{i} \frac{1}{N_s}\big[\max_{j}(\tilde{\mathbf{M}}_{aff} \odot \mathbf{Y}_{\Pi})_{i,j} - 1\big]^2 \\ & + \sum_{i,j} \frac{1}{\lVert \mathbf{1} - \mathbf{Y}_{\Pi} \rVert_1}\big[\tilde{\mathbf{M}}_{aff} \odot (\mathbf{1} - \mathbf{Y}_{\Pi})\big]^2_{i,j} \\ & + \sum_{i,j} \frac{1}{N_s N_t}\big(\mathcal{A}_s \tilde{\mathbf{M}}_{aff} - \tilde{\mathbf{M}}_{aff} \mathcal{A}_t\big)_{i,j} \end{aligned} eq. 5

Y_Π(i, j) = 1 when source node i and target node j share a class, else 0 §4.2.

\mathcal{L} = \lambda_1 \mathcal{L}_{node} + \lambda_2 \mathcal{L}_{mat} + \mathcal{L}_{NA} + \mathcal{L}_{GA} + \mathcal{L}_{det} eq. 6

with L_NA = −Σ D log f_dc(f_b(v_s)) − Σ (1 − D) log f_dc(f_b(v_t)) §4.3, where D is the domain label.

The training loop in pseudocode alg. 1 (names are ours; the paper gives no code):

# One SIGMA training step  [alg. 1]
# Input:  images_s, images_t (B, 3, H, W); boxes_s and labels_s per image
#         lambda_1, lambda_2 (both 0.1 in the paper)
# Output: total loss, backpropagated once per step

def sigma_step(images_s, images_t, boxes_s, labels_s, lambda_1, lambda_2):
    # Shared backbone features for both domains  [§4]
    feats_s = backbone(images_s)  # (B, D, H, W)
    feats_t = backbone(images_t)  # (B, D, H, W)

    # Image-level adversarial alignment  [§4.3] [alg. 1]
    loss_ga = global_alignment(feats_s, feats_t)

    # Supervised detection loss; pseudo score maps on target  [§4.3] [alg. 1]
    loss_det, score_maps_t = detection_head(feats_s, boxes_s, labels_s, feats_t)

    # Sample pixels: source inside GT boxes, target by score thresholds  [§4.1]
    pixels_s, classes_s = sample_source_pixels(feats_s, boxes_s, labels_s)
    pixels_t, classes_t = sample_target_pixels(feats_t, score_maps_t)

    # Vision-to-graph: project sampled pixels to raw node embeddings  [§4.1]
    nodes_raw_s = v2g_projection(pixels_s)  # (N_s, D)
    nodes_raw_t = v2g_projection(pixels_t)  # (N_t, D)

    # Node-level adversarial alignment on raw nodes  [§4.3] [alg. 1]
    loss_na = node_discriminator(nodes_raw_s, nodes_raw_t)

    # Hallucinate nodes for classes missing from one domain  [§4.1]
    nodes_s = complete_missing(nodes_raw_s, classes_s, classes_t, memory_s)
    nodes_t = complete_missing(nodes_raw_t, classes_t, classes_s, memory_t)

    # Graph convolution over an edge-dropped softmax adjacency  [§4.1]
    adj_s = edge_drop(softmax(nodes_s @ W_e @ (nodes_s @ W_e).T), rate=0.1)
    adj_t = edge_drop(softmax(nodes_t @ W_e @ (nodes_t @ W_e).T), rate=0.1)
    enhanced_s = layer_norm(adj_s @ nodes_s @ W_gcn + nodes_s)
    enhanced_t = layer_norm(adj_t @ nodes_t @ W_gcn + nodes_t)

    # Memory update uses existing nodes only, never hallucinated ones  [§4.1]
    memory_s = update_memory(memory_s, enhanced_s[is_real_s], classes_s)
    memory_t = update_memory(memory_t, enhanced_t[is_real_t], classes_t)

    # Cross-graph interaction and node classification loss  [§4.2] [eq. 3-4]
    cross_s, cross_t = cross_graph_interaction(enhanced_s, enhanced_t)
    loss_node = node_classification(cross_s, cross_t, classes_s, classes_t)

    # Semantic-aware affinity, instance norm, then Sinkhorn with k = 20  [§4.2]
    affinity = sinkhorn(instance_norm(semantic_affinity(cross_s, cross_t)), 20)

    # Structure-aware matching loss: TE, FS and QC terms  [§4.2] [eq. 5]
    same_class = classes_s[:, None] == classes_t[None, :]
    loss_mat = matching_loss(affinity, same_class, adj_s, adj_t)

    # Weighted total objective  [§4.3] [eq. 6]
    loss = lambda_1 * loss_node + lambda_2 * loss_mat
    loss = loss + loss_na + loss_ga + loss_det
    return loss

Pseudocode notes: memory_s and memory_t are the seed banks (C × D) §4.1. complete_missing and update_memory are not named in the paper; they stand for the operations described in §4.1 and are not a verbatim transcription.

Hyperparameters

Results

Author-reported. Setup: Cityscapes to Foggy Cityscapes with VGG-16 and ResNet-50 backbones §5.1 tab. 1; Sim10k and KITTI to Cityscapes, VGG-16 only §5.1 tab. 2; ablations and sensitivity on Cityscapes to Foggy Cityscapes with VGG-16 §5.4 §5.5.

Cityscapes to Foggy Cityscapes, mAP (%), with SO and GAIN (SO = source only, GAIN = improvement over SO) tab. 1. Shown: SIGMA and the rows the text compares against. Full table: tab. 1.

Sim10k to Cityscapes (S→C) and KITTI to Cityscapes (K→C), car class, VGG-16, mAP (%), with SO / GAIN tab. 2:

Ablation on Cityscapes to Foggy Cityscapes, VGG-16, mAP (%) tab. 3. The baseline is the GA model on FCOS §5.1.

Number of sampled nodes per feature map (source/target) tab. 4:

Matching strategy and loss tab. 5 (mAP0.5:0.95 / mAP0.5 / mAP0.75):

Sensitivity to λ1 and λ2 (mAP0.5:0.95 / mAP0.5 / mAP0.75) app. A.1 tab. 6: λ1 = λ2 = 0.1 gives 24.0 / 43.5 / 23.5. The best mAP0.5 in that table is 43.7 (λ1 = 0.2, λ2 = 0.1), a 0.2-point difference tab. 6.

Node-discriminator position (mAP) app. A.2 tab. 7: P1 (semantic-complete nodes, main setting) 43.5; P2 (after graph convolution) 43.0; P3 (after cross-graph interaction) 39.2.

Projection normalisation (mAP) app. A.3 tab. 8: GroupNorm 42.9; BatchNorm 41.0; LayerNorm 43.5.

Per-step cost (time, memory) is not reported.

Paper vs code

No official code. The arXiv abstract and the paper's own link point to github.com/CityU-AIM-Group/SIGMA, which returns 404 (the organisation page also returns 404). A GitHub repository search returned krumo/SIGMA (MIT, README crediting Wuyang Li, paper title match). Its owner is a GitHub user whose public profile name is not one of the paper's authors, and the paper does not link it. Under the rule for this wiki it is a third-party copy, so it was not cloned or read, and no code citations are given.

Pitfalls

• No code to check against. Every algorithmic detail below is from the paper text and alg. 1 only.
• Warm-up contradicts the end-to-end note. The main text pre-trains as a warm-up before the BGM adaptor, following EPM §5.2. The appendix says end-to-end training is possible with halved source nodes as placeholders plus 10,000 extra iterations, reaching the same result app. C.2 item 4. The warm-up length is not stated.
• σ is ambiguous. The text calls the target-node spread a "standard variance" §4.1 and passes it as the second argument of N(μ, σ). It is not clear whether σ is a standard deviation or a variance. The paper does not say which, and no code is available to settle it.
• Memory update normalisation. b averages over 1/(|π_seed| − 1), not 1/|π_seed| eq. 2 §4.1. The momentum weight is sim(b, S), a cosine similarity, which can be negative, so the weight on S can become negative or above 1. The paper does not discuss this.
• Eq. 5 is written without a square or norm on the third term, unlike the first two terms eq. 5. Whether it is squared is not stated. The first term's normalisation 1/N_s and the second term's 1/‖1 − Y_Π‖_1 are as written.
• Node-discriminator input is ambiguous. P1 is described as "semantic-complete nodes V_{s/t} (without the hallucination nodes)" app. A.2, but §4.1 defines V_{s/t} as including hallucinated nodes. Alg. 1 applies L_NA to the raw nodes, before DNC alg. 1. Which nodes the discriminator sees in the main setting is therefore unclear.
• Node count is ambiguous. "At most 100 nodes per feature map in each domain" §5.2 does not say whether this is per image, per batch, or per FPN level. Table 4 shows 200/200 gives 43.9 mAP, above the 43.5 of the main setting, and the text does not explain the choice of 100 tab. 4.
• Pseudo-labels in L_node have no stated threshold. The thresholds τ_fg and τ_bg apply to sampling §4.1. Whether target pseudo-labels in eq. 4 are filtered, and whether gradients flow through the detector's score maps, is not stated.
• Backbone mixing in the text. The comparison text gives the GPA gain as 4.7 points, which is measured against the ResNet-50 SIGMA score (44.2), while the other gains use the VGG-16 score (43.5) §5.3 tab. 1. The text does not say so.
• Text and table disagree. The SML ablation is 42.1 mAP in the text §5.4 and 42.2 in tab. 3. The multiple-matching result is cited as 43.1 mAP0.5 in the text, which is the MSE row of tab. 5, and the text refers to tab. 5 with a wrong label (eq:match) §5.5.
• Typos that could mislead an implementer. The conclusion calls the module GSM, not GSC §6. The appendix refers to "Table 5 of our manuscript" for the multiple-matching ablation, which matches tab. 5 in source order app. C.2 item 5.
• Sinkhorn cost and convergence. The affinity uses 20 Sinkhorn iterations with no convergence check; the paper calls 20 "enough" §4.2. The per-step cost of the N_s × N_t affinity and the graph operations is not reported.
• Unstated training details. Schedule, iteration count, image resolution, the node dimension D, the gradient-reversal weight and the source of target labels for the detector are all not stated.
• Known failure modes. Distant objects obscured by fog can be missed or falsely detected app. B.2 fig. 5. The paper suggests a stronger backbone (ResNet-101) and linguistic cues as remedies, without experiments app. B.2.

Lineage

• Builds on: category-level alignment methods, notably the global-alignment baseline of EPM (the GA baseline in tab. 3) and the single-stage FCOS detector §5.1. Graph matching as a quadratic assignment problem follows earlier solvers §2 §3. Sinkhorn normalisation and instance normalisation follow §4.2.
• Compared against (named in tab. 1 and tab. 2): EPM, DSS, MeGA, RPNPA, UMT, ICCR-VDD, KTNet, SSAL, CFFA, GPA, DIDN, SDA.
• Superseded by: not stated.