End-to-End Object Detection with Transformers

In plain words

An object detector looks at a photo and returns a list of boxes, each with a label such as "cat" or "car". Classic detectors get there indirectly. They make thousands of guesses from a grid of preset box shapes (anchors), score each one, and then run a clean-up step called non-maximum suppression to throw away near-duplicate boxes. These steps rely on hand-tuned rules that are hard to get right.

DETR asks for the answer directly. The model has a fixed set of 100 "slots", each a learned query that asks the image "is there an object for me?". A transformer lets the slots look at the whole image and at each other, so they can agree on who reports which object. Each slot then outputs one box and a label, or "no object". During training, every real object is paired with exactly one slot by solving an assignment problem, the same kind as matching workers to jobs at lowest total cost, and only that slot is told to predict it. Because duplicates are penalized by design, no clean-up step is needed.

The result is a detector built only from a standard CNN and a standard transformer. It matches a strong, carefully tuned Faster R-CNN on the COCO benchmark, does better on large objects and worse on small ones, and needs a much longer training schedule. Adding a small mask head on top turns the same model into a panoptic segmenter, which labels every pixel of the image.

TL;DR

DETR predicts a fixed-size set of N=100 (class, box) pairs in one parallel pass of a transformer decoder over CNN features. Training uses a bipartite (Hungarian) matching between predictions and ground truth, then a cross-entropy plus \ell_1 + GIoU loss on the matched pairs; unmatched slots must predict a "no object" class. Anchors, proposal generation and NMS disappear. With a ResNet-50 backbone and a 500-epoch schedule it gets 42.0 AP on COCO val (Faster R-CNN-FPN+ with GIoU, crops and a 9x schedule: 42.0), with 61.1 vs 53.4 APL and 20.5 vs 26.6 APS (author-reported, tab. 1). Read this if you implement set-based detectors or anything built on learned object queries; budget for very long training.

Problem

Modern detectors solve detection through surrogate tasks: regression and classification on large sets of proposals, anchors or window centers §1. Their accuracy depends heavily on post-processing that removes near-duplicate predictions, on how the anchor set is designed and on heuristics that assign ground-truth boxes to anchors §1. Earlier direct set-prediction detectors either added other prior knowledge or were not competitive with strong baselines on hard benchmarks §1. Most of them decoded autoregressively with RNNs §1, §2. The goal is a detector that predicts the final set of boxes directly, end to end, from standard components only §1.

Method

Two parts: a set loss that forces a unique match between predicted and ground-truth objects, and an architecture that predicts all objects in one pass and models their relations §3, fig. 2.

Set prediction loss §3.1.

1. The model outputs N predictions \hat y = \{\hat y_i\}_{i=1}^N, with N much larger than the typical object count. The ground-truth set y is padded to size N with \varnothing (no object) §3.1.
2. Each ground-truth element is y_i = (c_i, b_i): class c_i (possibly \varnothing) and box b_i \in [0,1]^4, holding center coordinates plus height and width relative to the image size §3.1.
3. Find the lowest-cost permutation with the Hungarian algorithm: \hat\sigma = \arg\min_{\sigma \in \Sigma_N} \sum_{i}^{N} \mathcal{L}_{\rm match}(y_i, \hat y_{\sigma(i)}) eq. 1 with the pairwise matching cost \mathcal{L}_{\rm match}(y_i, \hat y_{\sigma(i)}) = -\mathbb{1}_{\{c_i \neq \varnothing\}}\,\hat p_{\sigma(i)}(c_i) + \mathbb{1}_{\{c_i \neq \varnothing\}}\,\mathcal{L}_{\rm box}(b_i, \hat b_{\sigma(i)}) §3.1 The matching cost uses probabilities, not log-probabilities, so the class term is on a scale comparable to the box term; the authors also saw better results this way §3.1. Matching an object to \varnothing costs a constant §3.1.
4. Compute the Hungarian loss over the matched pairs: \mathcal{L}_{\rm Hungarian}(y, \hat y) = \sum_{i=1}^{N}\left[-\log \hat p_{\hat\sigma(i)}(c_i) + \mathbb{1}_{\{c_i \neq \varnothing\}}\,\mathcal{L}_{\rm box}(b_i, \hat b_{\hat\sigma(i)})\right] eq. 2 The log-probability term is down-weighted by a factor of 10 when c_i = \varnothing, to offset class imbalance §3.1.
5. Box loss: boxes are predicted directly, not as offsets to anchors. Plain \ell_1 scales with box size, so it is combined with the scale-invariant generalized IoU §3.1: \mathcal{L}_{\rm box}(b_{\sigma(i)}, \hat b_i) = \lambda_{\rm iou}\,\mathcal{L}_{\rm iou}(b_{\sigma(i)}, \hat b_i) + \lambda_{\rm L1}\,\lVert b_{\sigma(i)} - \hat b_i \rVert_1 eq. 9 \mathcal{L}_{\rm iou}(b_{\sigma(i)}, \hat b_i) = 1 - \left(\frac{|b_{\sigma(i)} \cap \hat b_i|}{|b_{\sigma(i)} \cup \hat b_i|} - \frac{|B(b_{\sigma(i)}, \hat b_i) \setminus b_{\sigma(i)} \cup \hat b_i|}{|B(b_{\sigma(i)}, \hat b_i)|}\right) eq. 10 |\cdot| is area and B(\cdot,\cdot) is the smallest box enclosing both; all areas are min/max of linear functions of the coordinates app. A.2.
6. All losses are normalized by the number of objects in the batch. In distributed training, normalize by the total number of objects across all GPUs, not the local sub-batch §3.1, app. A.2.

Architecture §3.2, fig. 2, fig. 10.

1. Backbone. Image x_{\rm img} \in \mathbb{R}^{3 \times H_0 \times W_0}; images in a batch are zero-padded to the largest (H_0, W_0) §3.2 footnote. A ResNet produces f \in \mathbb{R}^{C \times H \times W}, typically C = 2048, H, W = H_0/32, W_0/32 §3.2.
2. Encoder. A 1x1 conv reduces C to d, giving z_0 \in \mathbb{R}^{d \times H \times W}; flatten to a d \times HW sequence §3.2. Each layer is multi-head self-attention plus FFN §3.2. Fixed 2D sine positional encodings are added to queries and keys at every attention layer §3.2, app. A.3: for each of the two spatial coordinates, d/2 sine and cosine channels at different frequencies, concatenated to d channels app. A.4.
3. Decoder. Takes N learned positional embeddings ("object queries"), added to the input of every attention layer; the decoder input itself starts at zero §3.2, app. A.3. Each layer runs self-attention over the N slots and cross-attention into the encoder output, and all N objects are decoded in parallel, not autoregressively §3.2. The first self-attention of the first decoder layer can be skipped app. A.3.
4. Prediction heads. A 3-layer perceptron with ReLU and hidden size d predicts the normalized box center, height and width; a linear layer predicts the class with softmax, including \varnothing §3.2. Each slot is decoded independently §3.2.
5. Auxiliary decoding losses. Prediction heads and the Hungarian loss are applied after every decoder layer. All heads share parameters, and an extra shared LayerNorm normalizes their input from each decoder layer §3.2.

Attention, as defined in the appendix (with d' = d/M for M heads) app. A.1:

\bar{\bar X}^{(q)} = [\mathrm{attn}(X_q, X_{kv}, T_1); \ldots; \mathrm{attn}(X_q, X_{kv}, T_M)] eq. 5 X'_q = \mathrm{layernorm}\big(X_q + \mathrm{dropout}(L\,\bar{\bar X}^{(q)})\big) eq. 6 [Q; K; V] = [T'_1(X_q + P_q);\; T'_2(X_{kv} + P_{kv});\; T'_3 X_{kv}] eq. 7 \alpha_{i,j} = \frac{e^{\frac{1}{\sqrt{d'}} Q_i^\top K_j}}{Z_i}, \quad Z_i = \sum_{j=1}^{N_{kv}} e^{\frac{1}{\sqrt{d'}} Q_i^\top K_j} eq. 8

Positional encodings go into queries and keys but not values eq. 7, and are shared across all attention layers for a given sequence app. A.1. The transformer FFN is two 1x1-conv layers with ReLU, followed by residual, dropout and layernorm as in eq. 6 app. A.1.

Algorithm: DETR training step  [§3.1, §3.2, app. A.2]
Input: batch of images (zero-padded to common size), ground-truth sets {(c_i, b_i)}
1. f   = ResNet(images)                      # C=2048, stride 32 (16 for DC5)
2. z0  = Conv1x1(f) -> d channels; flatten to HW tokens
3. mem = Encoder(z0, pos = 2D sine enc. added to Q and K in every layer)
4. h_1..h_L = Decoder(tgt = zeros(N, d), query_pos = learned N x d,
                      memory = mem, pos as above)   # outputs of every layer
5. for each decoder layer l:
     p_l = softmax(Linear(LN(h_l)))            # K classes + no-object
     b_l = MLP3(LN(h_l))                       # normalized (center, h, w)
     for each image: sigma = Hungarian(cost = -p(c_i) + L_box) over real objects
     loss_l = CE(all N slots, matched targets, no-object weight 0.1)
              + 1{c != no-object} * (5 * L1 + 2 * GIoU)
6. normalize box losses by total #objects across all GPUs
7. loss = sum over l of loss_l; AdamW step, grad-norm clip 0.1
Output: updated parameters

Inference §4. Each slot gives a class and a box. To optimize AP, a slot that predicts \varnothing is given its second-highest-scoring class with that class's confidence, instead of being dropped; this adds 2 AP over filtering empty slots §4. No NMS is applied §4.2.

Panoptic extension §4.4, fig. 8.

1. Train DETR to predict boxes for both stuff and thing classes; boxes are needed because matching is computed from box distances §4.4.
2. Mask head: for each decoder output embedding, compute multi-head (M heads) attention scores over the encoder output, giving M low-resolution heatmaps per object; an FPN-like network upsamples them to stride-4 masks §4.4, fig. 8.
3. Supervise each mask independently with DICE/F-1 and Focal loss §4.4: \mathcal{L}_{\rm DICE}(m, \hat m) = 1 - \frac{2\,m\,\sigma(\hat m) + 1}{\sigma(\hat m) + m + 1} eq. 11 with \hat m the mask logits, m the binary target, normalized by the number of objects app. A.2.
4. Train in two steps: train DETR for boxes, freeze all weights, train only the mask head for 25 epochs. Joint training gave similar results §4.4.
5. Inference: drop detections with confidence below 85%, take the per-pixel argmax over mask scores, merge masks of the same stuff category, drop masks under 4 pixels §4.4. The argmax guarantees non-overlapping masks §4.4.

Hyperparameters

Parameters: 41.3M total for the R50 model, 23.5M in ResNet-50 and 17.8M in the transformer §4.1.

Results

Author-reported, COCO 2017 (118k train, 5k val) §4. Faster R-CNN comparison uses val AP at the last epoch; ablations report the median of val results over the last 10 epochs §4. FLOPS/FPS measured with TorchScript models; FLOPS averaged over the first 100 val images tab. 1, app. A.3.

Comparison with Faster R-CNN (rows without R101 use ResNet-50; "+" = GIoU, random crops, 9x schedule) tab. 1:

Number of encoder layers (R50, 6 decoder layers, short schedule) tab. 2:

Positional encodings (baseline is the last row; all use learned object queries; learned embeddings shared across layers) tab. 3:

Loss components (same weights every time) tab. 4:

Other ablations:

• Removing the FFNs from the transformer layers (41.3M to 28.7M parameters, 10.8M left in the transformer) costs 2.3 AP §4.2.
• AP and AP50 improve after every decoder layer, +8.2 / +9.5 from first to last layer §4.2, fig. 4. NMS helps the first decoder layer's output, but slightly lowers AP at the last layers by removing true positives §4.2, fig. 4.
• The 500-epoch schedule adds 1.5 AP over 300 epochs §4; the R50 model gets 40.6 / 42.0 AP on short / long schedules and runs at 28 FPS §4.2.

Panoptic segmentation, COCO val (PanopticFPN++ retrained with DETR's augmentation on an 18x schedule; UPSNet uses 1x; UPSNet-M uses multiscale test-time augmentation; AP is mask AP on things before the panoptic argmax) tab. 5, §4.4:

On the COCO test set the panoptic model reaches 46 PQ §4.4. Gains come mostly from stuff classes; mask AP on things trails the baselines by up to 8 points, but PQth stays competitive §4.4.

Paper vs code

Not checked yet. The official repository is linked in the abstract p. 1, but it was not cloned when this card was compiled, so code_commit and code_license are not recorded and nothing below is verified against it.

The paper's own inline PyTorch listing app. A.6 is a simplified demo and differs from the model described in §3.2 in ways the paper states itself: it uses learned rather than fixed positional encodings in the encoder (50 row and 50 column embeddings of size d/2), and adds them only at the input rather than at every attention layer. It also has no auxiliary losses, uses a single linear layer (not a 3-layer MLP) for boxes followed by a sigmoid, supports no batching, and sets num_classes=91 app. A.6. Do not use it as the reference for training.

Pitfalls

• Probabilities in the matching cost, log-probabilities in the loss §3.1. Using log-probs in the matching cost, or forgetting the 0.1 weight on the no-object class in the loss, changes the method.
• Loss normalization across GPUs. Normalize by the total object count over all GPUs, not per GPU; sub-batches are not balanced app. A.2.
• Positional encodings go into Q and K of every attention layer, not V eq. 7, app. A.3. Adding them once at the input, as in the original transformer, costs 1.4 AP tab. 3. Encoder sine encodings and the object queries are shared across layers app. A.1.
• Decoder input starts at zero; the object queries enter only as positional encodings in each attention layer app. A.3.
• Box format. Boxes are normalized center plus height and width in [0,1] §3.1; GIoU needs corner coordinates, a conversion the paper does not spell out. The order of the four values is not stated. A sigmoid on the box output appears only in the inline listing app. A.6.
• Padding. Images are zero-padded to the batch maximum §3.2 footnote; the paper does not say whether padded positions are masked in attention or how the sine encoding treats them.
• Auxiliary heads share weights across decoder layers, plus one shared LayerNorm §3.2. Separate heads per layer would not match the paper.
• Post-norm transformer: residual, then dropout, then layernorm, as in eq. 6 app. A.1.
• The +2 AP inference trick. Reported AP relies on replacing no-object predictions with the second-best class §4. A deployment that thresholds scores will not see the same AP-to-precision trade-off.
• Training cost. 300 epochs take 3 days on 16 V100s §4; headline numbers use 500 epochs, against about 109 for the Faster R-CNN+ baselines app. A.4. The backbone LR must be about 10x lower or early training is unstable app. A.4.
• Gradient clipping at max norm 0.1 is small and easy to miss app. A.4.
• Small objects. The text states +7.8 APL and -5.5 APS versus Faster R-CNN §4.1, but tab. 1 gives 61.1 vs 53.4 APL (+7.7) and 20.5 vs 26.6 APS (-6.1) against Faster RCNN-FPN+; neither Faster R-CNN row reproduces the text's deltas exactly. DC5 helps but multiplies encoder self-attention cost by 16 §4; encoder cost is \mathcal{O}(d^2 HW + d(HW)^2), quadratic in feature-map size app. A.3.
• Hard cap of N detections. In a synthetic test that tiles one class on a 10x10 grid, DETR finds all instances up to about 50; with all 100 present it finds about 30 on average app. A.5, fig. 12.
• Results protocol. Ablation numbers are medians over the last 10 epochs, comparison numbers are last-epoch values §4; do not mix them when comparing.
• Panoptic head under-specified. §4.4 points to the supplement for mask-head details, but the appendix only gives the DICE loss; the architecture is shown only in fig. 8, and the Focal-loss weight and the number of heads M are not stated.
• Unstated values: transformer FFN width, warmup, the LR drop factor in the 500-epoch schedule, crop size range, test-time resolution.

Lineage

• Builds on: the Transformer (arXiv:1706.03762), used here with parallel instead of autoregressive decoding §3.2; Hungarian-matching losses from earlier set-prediction detectors such as Stewart et al.'s end-to-end people detection with RNNs §2; generalized IoU (arXiv:1902.09630) for the box loss §3.1.
• Compared against: Faster R-CNN (arXiv:1506.01497) with FPN and DC5 from Detectron2 tab. 1; Panoptic FPN (arXiv:1901.02446) and UPSNet for panoptic segmentation tab. 5.