Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment

In plain words

Detectors that use transformers, such as DETR, learn to output one box per object and no duplicates. They learn this slowly, because each object is matched to just one prediction, so each training image teaches only a little. Older detectors avoid this by letting several predictions match the same object, and then they clean up duplicates afterwards.

Group DETR keeps the transformer's one-match-per-object rule, but runs it several times in parallel. The model holds several sets of learned "question slots" (object queries). Each set is matched to the objects on its own, and the sets only look at each other's answers within the same set. So several slots can learn to find the same object, and the model gets more training signal per image. When the model is used, only one set is kept, so the output looks like a normal DETR and no duplicate-removal step is added.

The authors report that this speeds up training and raises accuracy for several DETR variants on the COCO benchmark, and that the same idea helps 3D detection and instance segmentation.

TL;DR

Group DETR keeps K groups of object queries (K = 11 by default) and runs the usual one-to-one Hungarian matching inside each group §3.1 §4.3. Self-attention is computed separately per group, and the decoder and predictor weights are shared across groups §3.1 alg. 1. At inference only one group is used, with no architecture change §3.1. The authors report gains of 0.7 to 5.0 mAP over 12-epoch DETR-family baselines on COCO val2017 with ResNet-50 tab. 3. No code was available at compile time (see Paper vs code), and most training hyperparameters are not stated in the paper.

Problem

DETR uses one-to-one assignment: each ground-truth object is matched to a single prediction. This removes the need for non-maximum suppression, but it gives fewer positive training signals per object than one-to-many schemes used by Faster R-CNN and FCOS, which assign several anchors or pixels to each object §1 §2. The authors state that naive one-to-many assignment does not work for DETR, because the model then has no mechanism to demote duplicate predictions §1 §3.1.

Earlier DETR acceleration work mostly changed cross-attention (for example Deformable DETR, Conditional DETR, DAB-DETR) or stabilised one-to-one assignment with noised queries (DN-DETR, DINO) §1 §5. The paper's goal is to add supervision through the assignment scheme itself, without changing the inference path §1 §5.

Method

Background (DETR). The encoder maps an image I to features X eq. 1. The decoder takes X and object queries Q and outputs embeddings Q̃, which a predictor turns into predictions Y eq. 2. DETR matches predictions to ground truths by a bipartite matching σ, and the loss sums a per-pair term ℓ over the matched pairs eq. 3 eq. 4 §2. The paper says ℓ combines a classification loss and a box regression loss but does not give the matching cost or the loss weights §2.

Naive one-to-many (negative baseline). Replacing the matching by one-to-many assignment, with more queries, performs poorly: the model learns to emit several predictions per object and lacks a way to demote the duplicates §3.1 tab. 7 fig. 2.

Group-wise one-to-many assignment. Form K groups of N queries. The first group is the normal set of N queries, and K−1 more groups are added §3.1. Each group k gets its own prediction set Y_k and its own bipartite matching σ_k, so within each group exactly one prediction is expected to score high for each object §3.1. Across groups, one object can therefore be matched several times §3.1 fig. 2.

Separate self-attention. Because matching is one-to-one within a group, the self-attention over queries needs to see only predictions from the same group. The paper therefore computes SA(Q_1), ..., SA(Q_K) separately §3.1 eq. 5. Cross-attention and FFN have no query-to-query interaction, so they can run on all groups at once §3.1.

Training architecture. The encoder is unchanged. The decoder becomes K parallel decoders that share parameters, and the predictor is shared too. Each group's decoder reads the same image features X §3.1 eq. 6. Algorithm 1 implements this as one decoder call by stacking the K groups along the batch axis for self-attention, then splitting them back alg. 1 fig. 6.

Loss. The total loss is the mean over the K group losses, each with its own matching σ_k §3.1 eq. 7.

Inference. Only one group of N queries is used. The model is the same as a normally trained DETR, with no architecture change §3.1 alg. 1. The algorithm keeps the first N queries, Q[:N] alg. 1.

Analysis (the authors' explanations, not proofs).

• Parameter sharing: training is viewed as K DETR models that share encoder, decoder and predictor and differ only in query initialisation. The shared weights receive more gradients, which the authors link to faster convergence §3.2.
• Assignment stability: the authors report a lower instability score with Group DETR than with DAB-DETR or DN-DETR on COCO val2017, using the instability score defined by DN-DETR §3.2 fig. 5.
• Query augmentation: predictions for the same object from different groups lie close together, and the authors read the groups as learned augmentation of the queries §3.2 fig. 3. Per-group results differ by ±0.1 mAP around the median of 37.5 mAP in one experiment §3.1 fig. 4.
• Encoder: Table 1 suggests the encoder also benefits, because a fixed encoder gives less gain than a trained one §3.2 tab. 1.

Applying to variants.

• One-stage DETRs (Conditional DETR, DAB-DETR, DN-DETR, DAB-Deformable-DETR) take multiple groups of learnable queries app. A.2.
• Two-stage DINO: the first-stage top-N query initialisation depends on predicted boxes. The authors build K pairs of classification and regression heads in the first stage, one pair per group, and use only one pair at inference app. A.2.
• For DN-DETR and DINO, each group also gets the query-denoising task, to keep the original architecture per group app. A.2.

Results of extensions. For multi-view 3D detection (PETR and PETR v2) and instance segmentation (Mask2Former), the authors apply the same group-wise scheme §4.2 tab. 5 tab. 6. They do not describe other changes to these models in the main text §4.2.

# Group decoder layer, training  [alg. 1] [eq. 6]
# queries: list of K tensors, each (N, B, C); image_feats: (HW, B, C)
# Output: list of K prediction sets

def group_decoder_train(image_feats, queries):
    # Stack the K groups along the batch axis  [alg. 1]
    stacked = concat(queries, axis=batch)           # (N, K*B, C)
    # Self-attention runs inside each group only, so no mixing  [§3.1]
    attended = self_attention(stacked)              # (N, K*B, C)
    # Split back into K groups  [alg. 1]
    per_group = split(attended, num_groups=K)       # K x (N, B, C)
    outputs = []
    for group_queries in per_group:
        # Cross-attention and FFN do not mix queries  [§3.1]
        hidden = cross_attention(group_queries, image_feats)
        outputs.append(ffn(hidden))
    return outputs

# Training loss over K groups  [§3.1] [eq. 7]
# outputs[k]: predictions of group k; targets: ground-truth objects

def group_detr_loss(outputs, targets, num_groups):
    total = 0
    for group_predictions in outputs:
        # One-to-one Hungarian matching inside this group only  [§2] [eq. 3]
        matches = hungarian_match(group_predictions, targets)
        # Per-pair loss; the paper does not state the cost or weights  [§2]
        total = total + pair_loss(group_predictions, targets, matches)
    # Average over the K groups  [eq. 7]
    return total / num_groups

# Inference: one group, same as a normal DETR  [alg. 1] [§3.1]
# Keep the first N queries; the other K-1 groups are dropped

def group_detr_inference(image_feats, queries):
    queries = queries[:N]                           # (N, B, C)
    hidden = self_attention(queries)
    return ffn(cross_attention(hidden, image_feats))

Hyperparameters

Values are taken from the paper's text, tables and the algorithm. Everything else is "not stated". In the tables, "C-DETR" is Conditional DETR, "DC5" is a suffix the paper uses without defining it, and "DINO-4scale" is DINO with four scales tab. 3 tab. 4. The paper says that the training settings (learning rate, optimizer, pre-trained weights, initialisation and augmentation) are the same as those of the baseline it extends, but it does not restate the values §4 §4.1.

Results

Author-reported, all from the paper's tables. Setup: COCO train2017, eval on val2017, ResNet-50 backbone unless noted §4.1. The gain is in brackets.

Table 3: 12 epochs tab. 3

Table 4: more epochs (50 epochs; DINO-4scale 36 epochs) tab. 4

Table 5: multi-view 3D detection, nuScenes val, 24 epochs tab. 5 (VoVNetV2 backbone, 800×320)

Table 6: instance segmentation, Mask2Former, COCO val2017 (mask mAP) tab. 6

Table 7: ablation, Conditional DETR-R50, 12 epochs, COCO val2017 tab. 7

Text: group-wise o2m adds 2.2 mAP over (a); separate SA adds 2.8 more §4.3.

Table 1: encoder vs decoder training, 50 epochs, Conditional DETR-R50 tab. 1 (the encoder for the fixed rows is initialised from a trained model; asterisks mark runs the authors say are slightly lower than regular training)

Table 2: training time, 8× A100, hours, COCO val2017 tab. 2 (baseline rows in grey in the paper are the 12- or 50-epoch Group DETR schedule; the 15- and 60-epoch rows give baselines with similar wall-clock time)

The paper's table has the 12-, 50-epoch baseline rows and Group DETR rows, plus 15- and 60-epoch baselines trained for longer wall-clock time. "Epochs" below is the schedule each row was trained with. tab. 2

Training memory and time §3.2 fig. 7: with FlashAttention, the memory increase is 1.2 GB for Conditional DETR and 1.7 GB for DAB-DETR; training time rises by about 5 minutes per epoch (23 to 28 minutes, and 28 to 33 minutes).

DN-DETR comparison §3.2 fig. 8: on DAB-DETR (12 epochs, val2017), the figure shows baseline 35.2, DN-DETR* 38.8 (best denoising-query count), Group DETR 39.1, and DN-DETR plus Group DETR 40.6. The paper reports a 1.5 mAP gain from combining the two.

Number of groups §4.3 fig. 9: mAP rises from 32.6 (K=1) to 35.3 (K=3), 36.7 (K=5), 37.0 (K=7), 37.2 (K=9), 37.6 (K=11) and 37.6 (K=13), per the figure's data points. The paper chooses K=11.

System-level, COCO test-dev, ViT-Huge §4.1 app. A.1 tab. 8: the ViT-Huge + Group DETR (DINO-4scale) model reaches 64.5 mAP on test-dev, using ImageNet-1K pre-training for the backbone and Object365 pre-training for the detector (see Hyperparameters for schedules). The caption of tab. 8 says all results in that table use test-time augmentation. The leaderboard in tab. 8 includes EVA (64.7) and InternImage-H (65.4), which are higher than this result.

SAM-DETR and SAM-DETR++ app. A.4 tab. 9: Group DETR adds 3.1 mAP to SAM-DETR at 12 epochs (33.1 to 36.2) and 1.9 mAP at 50 epochs (39.8 to 41.7). For SAM-DETR++, the gains are 2.2 mAP at 12 epochs (41.1 to 43.3) and 1.3 mAP at 50 epochs (46.1 to 47.4).

Paper vs code

No official code.

The abstract links https://github.com/Atten4Vis/GroupDETR. That repository belongs to the authors' organisation, and its README says it is the official implementation of this ICCV 2023 paper. It is the only candidate, so the GitHub search was not needed. The clone at commit 00f8745a9f704c4ea76f991adf90f223590f4cf8 (depth 1) contains only README.md, .gitignore and two PNG images in .github/. It has no source files, no configs, no training or evaluation scripts, and no LICENSE. The README lists "Release code and models" as an unchecked TODO. The abstract says the code will be available. So there is no code to cite and no licence to record.

Because no code is available, every hyperparameter and algorithm step in this card comes from the paper. Pseudocode follows the paper's Algorithm 1 and equations, not an implementation.

Pitfalls

• No code to check against. The linked repository has only a README and images. Nothing in this card can be verified against an implementation. The paper's Algorithm 1 is the only code-level detail.
• Loss scale. The paper averages the K group losses (1/K) eq. 7. It does not say whether the learning rate or loss weights were changed relative to the baseline. The paper says the training settings are the same as the baseline's §4. An implementer should decide how to match the baseline loss scale, and the paper gives no guidance on this.
• Self-attention separation. Algorithm 1 implements separate self-attention by stacking groups along the batch axis, so each group attends only to itself alg. 1. The paper's reason is that one-to-one matching inside a group makes predictions from other groups unnecessary, so their influence is removed §3.1. The ablation gives +2.8 mAP for separate SA over group-wise assignment alone tab. 7. Concatenating the groups along the query axis without a mask would mix them; the paper does not describe that variant. Cross-attention and FFN do not mix queries §3.1.
• Inference group. Inference keeps the first N queries alg. 1. The paper does not say whether any other group could be used instead.
• Naive one-to-many is not the same as Group DETR. The naive variant with 3300 queries reaches 8.4 mAP tab. 7 §4.3. The gain comes from the group structure together with separate self-attention, not from more queries alone.
• Denoising queries and the DN-DETR comparison. The main DN-DETR baseline uses 100 denoising queries tab. 3 app. A.3. The DN-DETR bar in the comparison figure uses the best of several counts, 300 queries and 38.8 mAP fig. 8 fig. 10. These two numbers should not be mixed.
• Two-stage setup. For DINO, the first-stage heads are duplicated per group app. A.2. The paper does not give the exact construction beyond this sentence, and it does not say how the heads are initialised.
• Memory and time figures depend on FlashAttention. The 1.2 GB and 1.7 GB increases §3.2 fig. 7 are for the memory-efficient attention implementation. The naive implementation's numbers are only in the figure.
• Time comparison. Table 2 compares Group DETR with baselines of similar wall-clock time by adding epochs to the baseline tab. 2. Those extra epochs are a fair comparison in time, but not in epochs.
• Encoder claim. The encoder-improvement result uses a fixed encoder pre- trained for 50 epochs, and the authors mark two rows as slightly lower than regular training tab. 1. The result is only suggestive.
• Convergence curves. The "faster convergence" claim is shown only as a curve figure fig. 1. No numeric convergence speedup is given.
• Stability claim. The stability result fig. 5 uses the DN-DETR instability score and is shown as a curve. The authors call the explanation a speculation §3.2.
• System-level number. The 64.5 test-dev result uses ViT-Huge, Object365 pre-training, EMA, CDN queries, LR decay and test-time augmentation app. B. It uses a different setup from the ResNet-50 12-epoch table tab. 3.
• Batch size. The paper says the batch size may be adjusted for memory, and that baseline and Group DETR use the same value §4 footnote 1. The value itself is not stated.

Lineage

• Builds on: DETR (one-to-one set matching with object queries, which this paper keeps for inference) §1 §2.
• Builds on: Conditional DETR, the main one-stage baseline in the tables tab. 3 §4.1.
• Related, concurrent: H-DETR also adds one-to-many supervision with extra queries. The paper says H-DETR uses a hybrid of one-to-one and one-to-many groups, and that its extra decoder is not used at inference §5. DETA, which brings NMS back §5, has no card.
• Related, no card in this wiki: DN-DETR, DINO and DAB-DETR are compared in the tables tab. 3 tab. 4; Deformable DETR and FCOS are discussed in §5.