Conditional DETR for Fast Training Convergence

In plain words

DETR finds objects in an image with a transformer: a fixed set of learned "slots" each look over the whole image and output one box and class. It works, but it needs about 500 passes over the training data to get good, far more than classic detectors. The authors argue part of the reason is that each slot has to learn, from its content features alone, where in the image to look, for example where the left edge of a dog is.

Conditional DETR gives every slot an explicit "where to look" signal. After each decoder layer, the slot already has a rough guess of its box. The model turns that guess, plus a reference point, into a spatial query that is compared only against the position codes of image pixels, while the content part of the slot is compared only against image content. Each attention head then naturally focuses on a band around one side of the object or on a spot inside it.

Because the "where" job is handled separately, the content features no longer have to do double duty, and training gets much faster: the paper reports reaching DETR's 500-epoch accuracy in 50 to 75 epochs on COCO, with the same simple single-scale design and no extra hand-made attention shapes.

TL;DR

Replace DETR's additive decoder cross-attention ((c_q + p_q)·(c_k + p_k)) with concatenation, so the logit becomes c_q·c_k + p_q·p_k per head. The spatial query p_q is the sinusoidal embedding of a per-query reference point, scaled element-wise by a vector predicted by a small FFN from the previous decoder layer's output. Combined with Deformable-DETR-style focal loss, 300 queries and center-offset regression, ResNet-50 hits 40.9 AP at 50 epochs and 43.0 AP at 108 epochs on COCO val (DETR: 42.0 AP at 500 epochs). It keeps single-scale global attention; it does not address encoder cost.

Problem

DETR §1 needs 500 training epochs; at 50 epochs DETR-R50 gets 34.9 AP vs 42.0 at 500 tab. 1. The authors attribute this to the decoder cross-attention depending heavily on content embeddings: in DETR, removing positional embeddings in keys and object queries from the second decoder layer onward only costs a little AP (34.9 to 34.0 at 50 epochs, their reproduction) §1. At 50 epochs, two of the four extremity-attending spatial maps in DETR fail to highlight the box edges fig. 1. Their explanation: object queries give only a generic, image-independent spatial prior, and the content query is not yet strong enough to match both content keys and spatial keys §1 §3.4.

Deformable DETR addresses convergence by replacing dense attention with sparse deformable attention and a multi-scale high-resolution encoder; this paper instead keeps global dense attention and only changes decoder cross-attention §1.

Method

The model is DETR (CNN backbone, transformer encoder, 6-layer transformer decoder, class and box FFNs after every decoder layer with shared parameters) §3.1 §3.5 fig. 3. Changes are confined to how cross-attention queries and keys are formed, plus Deformable DETR's loss and reference-point box parameterization §3.5.

Box prediction relative to a reference point §3.1 eq. 1:

\mathbf{b} = \operatorname{sigmoid}(\operatorname{FFN}(\mathbf{f}) + [\mathbf{s}^\top~0~0]^\top) eq. 1

f is the decoder embedding, b = [cx, cy, w, h] normalized to [0,1], and s is the unnormalized 2D reference point (s = (0,0) in original DETR). Classification is e = FFN(f) §3.1.

DETR cross-attention (for contrast). Query = content query c_q (output of decoder self-attention) + spatial query p_q = object query o_q; key = content key c_k (encoder output) + spatial key p_k (sinusoidal embedding of normalized 2D pixel coordinate); value = encoder content §3.2:

(\mathbf{c}_q + \mathbf{p}_q)^\top (\mathbf{c}_k + \mathbf{p}_k) = \mathbf{c}_q^\top \mathbf{c}_k + \mathbf{c}_q^\top \mathbf{p}_k + \mathbf{o}_q^\top \mathbf{c}_k + \mathbf{o}_q^\top \mathbf{p}_k eq. 2

Conditional cross-attention. Query = concat(c_q, p_q), key = concat(c_k, p_k), so the dot product separates §3.3:

\mathbf{c}_q^\top\mathbf{c}_k + \mathbf{p}_q^\top\mathbf{p}_k eq. 3

Conditional spatial query (s, f) → p_q eq. 4 §3.3:

\mathbf{p}_s = \operatorname{sinusoidal}(\operatorname{sigmoid}(\mathbf{s})) eq. 5

p_s is 256-dimensional and built the same way as the key positional embedding. T = FFN(f) with FFN = linear + ReLU + linear. T is constrained to a diagonal matrix, i.e. a 256-d vector λ_q:

\mathbf{p}_q = \mathbf{T}\mathbf{p}_s = \boldsymbol{\lambda}_q \odot \mathbf{p}_s eq. 6

Multi-head. M = 8 heads. Spatial and content queries (and keys) are projected to each head by separate linear projections; value projections are as in DETR and content-only §3.3.

Reference points §3.5: either learnable unnormalized 2D parameters per query, or predicted from the object query, s = FFN(o_q) (linear + ReLU + linear), as in Deformable DETR. Sigmoid-normalized when building p_s.

First decoder layer. There is no decoder content embedding yet; the paper says only that it makes small changes on top of the DETR implementation, folding the object-query positional embedding into the original DETR-style query and key §3.5. See Paper vs code for what this means concretely.

Loss §3.5: Hungarian matching and loss as in Deformable DETR (same matching cost, same trade-off weights, 300 queries), focal-loss classification, L1 + GIoU box loss as in DETR. Weights are not stated in the paper; see Hyperparameters (from code).

Algorithm: Conditional DETR decoder forward  [§3.3] [§3.5] [code: models/transformer.py#L131-L175]
Input: encoder memory C_k (HW x 256), key pos-embeds P_k (HW x 256),
       object queries O (N x 256, learned), N = 300
1. s_raw = FFN_ref(O)                       # N x 2, shared across layers
   s = sigmoid(s_raw)                       # fixed for all 6 layers (not refined)
2. f = zeros(N x 256)                        # decoder target init
3. for layer l = 0..5:
     λ = 1 if l == 0 else FFN_scale(f)        # FFN_scale shared across layers
     p_q = λ ⊙ sinusoidal(s)                  # 256-d, same encoding as P_k
     # self-attention: q = k = W1 f + W2 O, v = W3 f; residual + LayerNorm
     c_q = LN(f + SelfAttn(...))
     q_c = Wqc c_q ; k_c = Wkc C_k ; v = Wv C_k ; k_p = Wkp P_k
     if l == 0: q_c += Wqp O ; k_c += k_p     # first-layer special case
     q_p = Wqs p_q
     per head h (8 heads, 32 dims each):
        q_h = concat(q_c[h], q_p[h])          # 64 dims
        k_h = concat(k_c[h], k_p[h])          # 64 dims
     f = LN(c_q + CrossAttn(q, k, v))         # attention over 64-d per head
     f = LN(f + FFN(f))
     emit f_l (after a shared final LayerNorm)
4. for each layer output f_l:
     box_l = sigmoid(MLP_box(f_l) + [s_raw, 0, 0])   # eq. 1, shared MLP
     cls_l = Linear_cls(f_l)                         # sigmoid / focal
Output: per-layer boxes and class logits (aux losses on layers 0..4)

Hyperparameters

Results

Author-reported, COCO 2017 val (≈118K train / 5K val images), standard COCO AP §4.1. Comparison to DETR and single-scale variants tab. 1:

*Deformable DETR single-scale numbers are taken from the Deformable DETR GitHub repository tab. 1. The "6.67×" (R50/R101) and "10×" (DC5) speed-ups compare epochs needed to match DETR at 500 epochs §4.2 tab. 1.

Against multi-scale / higher-resolution detectors tab. 2:

Ablations (all R50, 50 epochs unless noted):

Reference point s §4.3: (0,0) as in DETR → 36.8 AP; learned per-query parameters → 40.7; predicted from object query → 40.9.

Way of forming the spatial query tab. 3 (CSQ-C: decoder content embedding f used directly; CSQ-T: only λ_q; CSQ-P: only p_s; CSQ-I: λ predicted from the self-attention output c_q instead of f, times p_s) §4.3:

Focal loss (FL), center-offset regression (OR), conditional spatial query (CSQ) tab. 4:

Form of T fig. 5: identity 37.8, single scalar 39.0, block diagonal (32×32 per head) 40.8, full 40.7, diagonal 40.9 AP.

Paper vs code

• Reference point is not refined across layers. The code predicts s once from the object query with a shared 2-layer MLP and reuses it in all 6 layers and for all box heads transformer.py conditional_detr.py. The paper is consistent with this but never states it. The learnable-parameter variant from §4.3 is not in the code.
• First layer. In layer 0 the code sets λ = 1 (no transformation) transformer.py and, in addition to concatenating the spatial parts, adds a projected object query to the content query and the projected key position embedding to the content key, DETR-style transformer.py. The paper only says the object-query embedding is concatenated into the query/key §3.5. Layers 1–5 use content only in the content half transformer.py.
• Extra projection on the spatial query. p_q passes through a per-layer linear ca_qpos_sine_proj, then is split into 8 heads of 32 dims and concatenated head-wise with the 32-dim content part transformer.py; key position embeddings likewise get a per-layer linear ca_kpos_proj transformer.py. This matches "separately projected to each head" §3.3.
• λ FFN and reference FFN are shared across decoder layers (single modules on the decoder) transformer.py; the paper does not say. Only layer 0 keeps its ca_qpos_proj.
• Input to the λ FFN is the previous layer's output before the final shared LayerNorm transformer.py.
• Classification head is a single linear layer in code conditional_detr.py; the paper says "an FNN" §3.1.
• Self-attention uses the object query (not the conditional spatial query) as positional term, with separate content and position projections that are summed transformer.py. Not discussed in the paper.
• Only 50-epoch training scripts are provided conddetr_r50_epoch50.sh.

Pitfalls

• Attention scale. The cross-attention is built with embed dim 512 and 8 heads, so per-head dim is 64 (32 content + 32 spatial) and logits are scaled by 1/√64, not 1/√32 transformer.py attention.py. The value/output stay 256-d (vdim). Re-implementing with stock 256-d attention plus summed scores changes the scale.
• Concatenate per head, not globally. Concatenating two 256-d vectors and then splitting into heads would put content in heads 0–3 and space in heads 4–7. The code reshapes each to (8, 32) first, then concatenates on the last axis transformer.py.
• Sine embedding must match the keys. p_s uses 128 frequencies per axis, temperature 10000, scale 2π, and (y, x) ordering, the same layout as the normalized key embedding transformer.py position_encoding.py. A mismatch silently breaks p_q·p_k.
• Box parameterization. The reference point is added in logit space to the center only: sigmoid(MLP(f)[:2] + inverse_sigmoid(sigmoid(s_raw))) conditional_detr.py; width/height have no prior. The last box-MLP layer is zero-initialized so the first predicted centers equal the reference points conditional_detr.py.
• Much of the gain is not the attention change. Focal loss + offset regression alone move DETR-R50 at 50 epochs from 34.9 to 36.9 AP; CSQ adds the remaining 4.0 tab. 4. 300 queries vs DETR's 100 also adds cost (GFLOPs 85.5 → 88.1 from the loss/query change) tab. 4.
• Focal loss scaling. The classification loss is mean over classes, summed, divided by the number of boxes, then multiplied by the number of queries conditional_detr.py segmentation.py; copying a different focal-loss reduction changes the effective weight 2.
• Class count. COCO uses num_classes = 91 (max category id + 1) with sigmoid outputs and no explicit no-object logit conditional_detr.py conditional_detr.py.
• Unstated in the paper: batch size, gradient clipping 0.1, class-bias prior, matching/loss weights, top-100 evaluation. All come from the code (see Hyperparameters).
• Encoder cost unchanged. Global single-scale encoder self-attention remains; DC5 variants cost 195–262 GFLOPs tab. 1. The method does not address this §4.2.
• The caption of Table 3 refers to "Section 5.3"; the ablations are in §4.3.
• Attention-head roles (four extremities + one interior region, three duplicates) are a qualitative observation and may differ between training runs §3.4 fig. 4.

Lineage

• Builds on: DETR (architecture, matching loss, training protocol); changes decoder cross-attention to concatenated content/spatial queries with a conditional spatial query. Borrows focal loss, 300 queries and reference-point offset regression from Deformable DETR (arXiv:2010.04159).
• Compared against: DETR, Deformable DETR (single- and multi-scale), UP-DETR, TSP-FCOS / TSP-RCNN, Faster R-CNN-FPN tab. 1 tab. 2. Concurrent and closest: SMCA, which modulates cross-attention with Gaussian maps around learned centers §2.