DETRs with Hybrid Matching

In plain words

DETR-style detectors find objects in one pass: a fixed set of output slots each propose one object. During training, every real object is paired with exactly one slot. That pairing is what lets the model skip the usual clean-up step that merges duplicate boxes. The drawback is that only a few slots ever get a strong training signal, so the model learns slowly.

H-DETR keeps the one-slot-per-object rule for the final model, but adds a second group of slots that is used only during training. In that group each real object is copied several times, so many slots are paired with it and learn from it. The two groups are blocked from seeing each other. At test time only the original group runs, so the model still gives its answer without a clean-up step. The method is a training change that can be added to several DETR-style detectors.

TL;DR

H-DETR keeps DETR's one-to-one Hungarian matching for the final model and adds a training-only one-to-many branch. That branch gets K times more queries, with each ground truth repeated K times (K=6 for COCO detection), and uses masked self-attention so the two query groups do not interact. Only the one-to-one branch runs at inference, so no NMS is needed and inference cost is unchanged. Author-reported: Deformable-DETR R50 12 epochs rises from 47.0 to 48.7 AP on COCO val tab. 1; the same recipe is applied to PETR, PETRv2 and TransTrack tab. 3 tab. 4 tab. 5.

Problem

With one-to-one matching each ground truth box supervises one query. The paper notes that Deformable-DETR selects fewer than 30 of its 300 queries as positives per COCO image, while the remaining queries get only the classification loss §1. The authors argue this limits training efficacy and motivates adding more positive queries §1, fig. 1. The paper also contrasts this with one-to-many label assignment, which is common in CNN detectors and needs NMS at inference §2.

Method

Preliminaries §3.1. DETR runs backbone, encoder and decoder, predicts a set \mathbf{P} of n outputs, and matches them one-to-one with the ground truth \mathbf{G} by bipartite matching. Matching uses a cost over class, box and GIoU terms; the loss is a classification loss, an \mathcal{L}_1 box loss and a GIoU loss §3.2. The paper follows DETR and Deformable-DETR for these losses §3.2. fig. 2 shows the DETR pipeline.

Hybrid branch scheme (the main method) §3.2. Two query groups share the same L decoder layers:

1. One-to-one group \mathbf{Q} of n queries. Predictions from each decoder layer are matched one-to-one with \mathbf{G}, giving the loss in eq. 1.
2. One-to-many group \widehat{\mathbf{Q}} of T queries. The ground truth is repeated K times, \widehat{\mathbf{G}}=\{\mathbf{G}^1,\dots, \mathbf{G}^K\} with all copies equal to \mathbf{G}, and the same matching and loss are applied per layer eq. 2.
3. Total loss: \lambda\,\mathcal{L}_{\rm one2many}+\mathcal{L}_{\rm one2one} throughout training §3.2.
4. Masked self-attention: a mask stops attention between the two groups, so they do not interact; the paper says this also lets both groups run in parallel §3.2. In the code the mask is an (n{+}T)\times(n{+}T) boolean matrix that is True on the cross-group blocks [code: models/deformable_detr.py#L212-L216].
5. Inference: only the one-to-one group is kept; no NMS is applied §3.2. The code switches the query count to num_queries_one2one for evaluation engine.py.

# H-DETR training loss, hybrid branch scheme  [§3.2] [eq. 1-2]
# Input:  images (B, 3, H, W); targets: per image a list of (class, box)
# Output: scalar loss

def hybrid_training_loss(images, targets, k_repeat, lambda_one2many):
    # One forward pass; two query groups share the decoder, with a mask
    # that blocks attention between them  [§3.2]
    # [code: models/deformable_detr.py#L212-L216]
    outputs = model(images)
    # One-to-one group: match each decoder layer's predictions  [eq. 1]
    loss = hungarian_loss(outputs.one2one, targets)
    # Repeat every ground truth K times for the second group  [§3.2]
    # [code: engine.py#L37-L40]
    repeated_targets = [repeat_boxes(t, k_repeat) for t in targets]
    # One-to-many group: same matching and loss on repeated targets  [eq. 2]
    # [code: engine.py#L48-L54]
    loss_many = hungarian_loss(outputs.one2many, repeated_targets)
    loss += lambda_one2many * loss_many
    return loss

Hungarian loss used by both groups. Matching uses a cost matrix built from classification, L1 box and GIoU terms, then the Hungarian algorithm matcher.py. The code computes the cost matrix on the GPU and moves it to the CPU for matching matcher.py; the paper says the Hungarian step runs on CPU §4.2. Each matched pair contributes the losses from §3.2 and the code's per-layer auxiliary losses deformable_detr.py.

Hybrid epoch variant §3.2. One query group of size M. For the first \rho fraction of epochs, the loss is one-to-many eq. 3; for the rest it is one-to-one on the original ground truth eq. 4. Evaluation uses the same queries without NMS §3.2. The paper reports \rho=\frac{2}{3} in its comparison §4.2. No code for this variant was found in the repository (see Paper vs code).

# Hybrid epoch variant  [§3.2] [eq. 3-4]
# Input:  epoch index, total epochs, rho (fraction of one-to-many epochs)

def epoch_variant_loss(outputs, targets, epoch, total_epochs, rho):
    # Early epochs: one-to-many, with ground truth repeated K times  [eq. 3]
    if epoch < rho * total_epochs:
        return hungarian_loss(outputs, repeat_targets(targets, k_repeat))
    # Later epochs: plain one-to-one on the original ground truth  [eq. 4]
    return hungarian_loss(outputs, targets)

Hybrid layer variant §3.2. One query group of size N. The first L_1 decoder layers use one-to-many matching with an augmented ground truth eq. 5; the remaining L_2 layers use one-to-one matching eq. 6. The paper reports L_1=4, L_2=2 and \bar K=10 in its comparison §4.2. No code for this variant was found in the repository.

Two-stage selection of query proposals (Deformable-DETR setting used for the headline numbers). The encoder scores all proposals, and the top one2one + one2many proposals are kept [code: models/deformable_transformer.py#L244-L246]. The first n of the ranked proposals feed the one-to-one group and the rest feed the one-to-many group deformable_detr.py. The paper does not describe how the proposals are split; this is read from the code (see Pitfalls).

Hyperparameters

Hyperparameters of the hybrid branch for each benchmark tab. 14. Code values are for the COCO R50 12-epoch configuration in the repository.

Code defaults differ from the paper's setting. k_one2many defaults to 5, lambda_one2many to 1.0, and num_queries_one2many to 0 main.py. The paper's COCO R50 setting must pass the flags from the config script.

Results

Author-reported. Setup: COCO 2017 val for detection and pose, LVIS v1.0 val, nuScenes val, MOT17 (val and test) app. A. The baselines are the original detector trained with the same schedule. Numbers in the "+x" column are the paper's own gains.

Object detection on COCO val tab. 1 (AP, \pm over the baseline in the original):

LVIS v1.0 detection tab. 2 (box training only, following Detic §A):

Multi-view 3D detection, nuScenes val tab. 3 (VoVNet-99; "repro." = authors' rerun):

Multi-person pose on COCO tab. 4 (100 epochs):

Multi-object tracking, MOT17 tab. 5 (20 epochs; mean over ~3 runs on val §4.1):

Panoptic segmentation (PQ, COCO val) is in the supplementary tab. 15: Mask-Deformable-DETR R50 12 ep 47.0 → 48.5; R50 50 ep 51.5 → 52.1; Swin-L 100 ep 56.9 → 57.2.

Comparison of hybrid schemes, Deformable-DETR with 1800 queries in total tab. 6 (AP at 12 / 24 / 36 epochs; train time averaged over epochs; FPS on V100):

Query count, cost and memory for the hybrid branch (R50; Swin-L in parentheses) tab. 7:

Ablation: components on two-stage Deformable-DETR, COCO val, 12 ep tab. 8 (2×FFN, DP0, MQS, LFT, HM):

Ablation: K, with T=300\times K, COCO val, 12 ep tab. 9: K = 0: 47.0; 1: 46.4; 2: 46.7; 3: 48.1; 4: 48.4; 5: 48.3; 6: 48.6; 7: 48.5; 8: 48.6.

Ablation: T, with K=6, COCO val, 12 ep tab. 10: T = 300: 47.8; 600: 48.3; 900: 48.4; 1200: 48.4; 1500: 48.7; 1800: 48.6.

System-level single-scale COCO val tab. 11 (H-Deformable-DETR row uses extra enhancement techniques from DINO, see Pitfalls):

Sharing parameters between the two branches, 12 ep, COCO val tab. 12 (trans. encoder / trans. decoder / box head / cls head):

One-to-many branch compared with other choices, COCO val tab. 13:

Loss weight \lambda, 12 ep, COCO val tab. 16: 0.1: 47.8; 0.2: 48.0; 0.5: 48.4; 1: 48.7; 2: 48.5; 5: 48.3.

Sharing parameters for DINO, 12 ep, COCO val tab. 17 (same column order as tab. 12): 49.1 / 48.9 / 49.2 / 49.0 / 48.5.

Textual results the tables do not show: on Deformable-DETR the top-300–600 proposals are poor in localisation; swapping them in for the top 0–300 drops AP from 47% to 42.5% app. D. A selection scheme that filters duplicates gives 47.9 AP with K=1 app. D. Training-loss and validation-loss curves are lower for the one-to-one branch than for the baseline app. D, fig. 8.

Paper vs code

Code: https://github.com/HDETR/H-Deformable-DETR, MIT license, at f58ceb1165da4b0cd10e9a5a30c7a77fbd33f492. The paper's abstract links github.com/HDETR; the GitHub search found HDETR/H-Deformable-DETR (described as the official implementation) and HDETR/H-Deformable-DETR-mmdet (Apache-2.0, the mmdet port, not read here). The LICENSE names the paper's authors.

• Checked and matching: the COCO R50 12-epoch configuration (300 one-to-one, 1500 one-to-many, K=6, λ=1.0, dropout 0, 2×FFN, mixed query selection, look forward twice) matches the paper's COCO column of tab. 14 r50_hybrid_branch_lambda1_group6_t1500_dp0_mqs_lft_deformable_detr_plus_iterative_bbox_refinement_plus_plus_two_stage.sh.
• Repetition of ground truth matches the paper: the boxes and labels are tiled K times engine.py §3.2.
• Mask between groups matches §3.2: the one-to-one and one-to-many query blocks cannot attend to each other deformable_detr.py.
• Inference uses only the one-to-one group engine.py.
• Loss normalisation is not stated in the paper and differs from a naive reading. The one-to-many loss is computed on the K-repeated targets, so the normaliser num_boxes counts K times as many boxes [code: models/deformable_detr.py#L481-L487] engine.py. Each one-to-many term is therefore divided by K|\mathbf{G}|, not |\mathbf{G}|.
• Two-stage split by rank is in the code but not in the paper: the top n+T encoder proposals are selected, and the first n by score go to the one-to-one group deformable_transformer.py deformable_detr.py. The paper's appendix describes a separate duplicate-filtering selection that it says gives no benefit at K=6 app. D; no such code was found in this repository.
• Hybrid epoch and hybrid layer variants are not in the repository. The configs cover only the branch scheme (configs/two_stage/deformable-detr-hybrid-branch/), and main.py and engine.py have no epoch or layer switch. Those variants can be reproduced only from the paper.
• Scope of the code: the repository covers Deformable-DETR (two-stage and one-stage baseline). The PETR, PETRv2 and TransTrack variants of tab. 3 to tab. 5 are not in this repository.
• Defaults: k_one2many, lambda_one2many and num_queries_one2many have defaults (5, 1.0, 0) that are not the paper's setting main.py.
• Not in the paper, only in code: the optimizer, learning rate, batch size, matcher and loss weights, and the focal form of the classification loss (see Hyperparameters).

Pitfalls

• Normaliser for the one-to-many loss. Each one-to-many term is normalised by the K-repeated ground-truth count, as described above. The paper does not state the normaliser. If you reimplement from eq. 2 alone you will get a different effective weight. To match the reported numbers, copy the code's behaviour.
• Headline configuration vs K sweep. The K-sweep sets T=300\times K, so K=6 means T=1800 and gives 48.6 AP tab. 9. The headline COCO configuration uses T=1500 and gives 48.7 AP tab. 10 r50_hybrid_branch_lambda1_group6_t1500_dp0_mqs_lft_deformable_detr_plus_iterative_bbox_refinement_plus_plus_two_stage.sh.
• Which proposals go to which group. The paper says the one-to-one group gets the top n predictions and the one-to-many group the rest, but does not say how the proposals are chosen. The code takes the top n+T encoder proposals, then splits them by rank deformable_transformer.py deformable_detr.py. This is the reading to implement.
• K=1 does not help with plain ranking. The paper finds the lower-ranked proposals (300–600) poorly localised, so a naive K=1 branch fails app. D. The paper reports consistent gains only for K>3 on COCO §4.2. Its alternative selection scheme for K=1 app. D is not in the code.
• Training cost grows with queries. Training time rises from 65 to 80 min and GPU memory from 5480 to 7728 MB for R50 at 1800 total queries tab. 7. Most of the extra memory is in the cross-attention of PETRv2, and a sequential attention implementation reduced it from 11175 MB to 8733 MB §4.1. The paper attributes the extra training time mainly to CPU Hungarian matching and the cost and loss computation §4.1 app. E. Matching both groups in one merged function reduces the per-epoch time from 85 to 80 min app. E, alg. 1.
• Inference gain depends on dropping the one-to-many branch. tab. 13 shows the one-to-many branch alone gives 13.5 AP without NMS. Keep only the one-to-one branch at test time.
• Headline SOTA mixes in other techniques. The 59.4 AP in tab. 11 uses "additional enhancement techniques" from DINO §4.3. It is not a pure H-DETR number and should not be compared with tab. 1 as one.
• Text and table disagree on the epoch order. §4.1 says Swin-T improves from 51.8 to 53.2 "under 12 and 36 epochs", and 49.3 to 50.6 "respectively" §4.1. tab. 1 puts 49.3 → 50.6 at 12 epochs and 51.8 → 53.2 at 36 epochs. Use the table.
• Not stated in the paper: batch size, learning rate, optimizer, LR schedule (the code uses step size 11 for 12 epochs and 30 for 36 epochs), gradient clipping, matcher costs and loss weights. Use the code values, marked as such in Hyperparameters.
• Figures are described, not reproduced. fig. 1 summarises the gains per task; fig. 3 shows the three hybrid schemes side by side; fig. 4 plots oLRP localisation and false-negative error, which the paper says the method lowers §4.1. The paper also reports PR curves and loss curves in the supplement app. D, fig. 7, fig. 8.

Lineage

• Builds on: DETR (one-to-one set matching with the Hungarian loss and object queries), plus Deformable-DETR (cited as zhu2020deformable §2), whose two-stage setting and loss this paper reuses §3.1 §3.2.
• Related one-to-many assignment: POTO (wang2021end) and ATSS (zhang2020bridging) §2; the paper contrasts its design with these.
• Compared against: DN-DETR and DINO (query denoising, which addresses a different training issue) §2 tab. 11; DE-DETR (which needs NMS) §2; Group-DETR and DINO-DETR tab. 11.
• Superseded by: not stated in the paper.