Direct set prediction

Overview

Many outputs are sets: the objects in a photo, the segments of a scene. Order carries no meaning, the size changes from example to example, and the same element must not be reported twice. Classic detectors do not predict this set directly. They score a large pool of anchors, proposals or window centers, assign ground truth to them with hand-written rules, and then prune near-duplicates with non-maximum suppression (NMS); accuracy depends heavily on those steps 2005.12872 §1.

Direct set prediction asks the model for the final set. Three ingredients are shared by the approach 2005.12872 §3:

1. A fixed number of slots. The model always emits N predictions, with N chosen well above the typical set size; the target set is padded to N with a "no object" element \varnothing 2005.12872 §3.1.
2. A permutation-invariant loss. Because slot order is arbitrary, each target is paired with one prediction by a minimum-cost one-to-one assignment before the loss is computed 2005.12872 eq. 1, eq. 2. The details live on bipartite matching.
3. Interaction between slots. To avoid duplicates without post-processing, slots need to know what the others are predicting; in DETR this happens through self-attention among the decoder slots 2005.12872 §3.2. The slot embeddings are covered on object queries.

The invariant of the pure form: the training signal is unchanged under any reordering of the predictions, and each target supervises exactly one slot, so duplicates are penalized during training rather than removed afterwards 2005.12872 §1, §3.1. One variant below relaxes the one-slot rule during training only 2207.13080 §3.2.

Variants

Eleven papers with cards use this formulation. All keep a set predictor trained with a matching loss. They differ in the decoder, in the candidate slots the encoder or decoder works on, in the matching cost, in the training schedule, and in what a query carries: image content, a candidate point, a track from the previous frame, a 3D reference point, one instance followed across a short video clip, or an action segment in a long untrimmed video. The sections below describe design choices, not a survey of competing methods.

Parallel decoding with a transformer (DETR)

A CNN backbone and transformer encoder produce image features; a transformer decoder turns N=100 learned query slots into N (class, box) pairs in one pass, not one element at a time 2005.12872 §3.2, app. A.4. Card: End-to-End Object Detection with Transformers.

• Decoding order. The paper contrasts its parallel decoding with earlier set-prediction detectors, most of which decoded autoregressively with RNNs 2005.12872 §1, §2. Those earlier methods are named only as background here; no card covers them.
• Duplicate handling. No NMS. Applying NMS anyway helps the output of the first decoder layer but slightly lowers AP after the last layers, because by then it mostly removes true positives 2005.12872 §4.2, fig. 4. AP rises after every decoder layer, +8.2 from first to last 2005.12872 §4.2.
• Each slot is decoded independently by shared heads: a 3-layer MLP for the normalized box and a linear softmax over classes plus \varnothing 2005.12872 §3.2.
• Inference. For AP evaluation, a slot predicting \varnothing is given its second-best class and that class's score instead of being dropped; this adds 2 AP over filtering empty slots 2005.12872 §4.
• Reported result. ResNet-50 DETR: 42.0 AP on COCO val, the same as Faster R-CNN-FPN trained with GIoU, random crops and a 9x schedule; 61.1 vs 53.4 AP on large objects, 20.5 vs 26.6 on small ones 2005.12872 tab. 1.

Conditional cross-attention for faster convergence (Conditional DETR)

The same set predictor, with a changed decoder cross-attention. The authors observe that DETR's cross-attention depends heavily on content embeddings: removing positional embeddings from the keys and queries of the second decoder layer onward costs little (34.9 to 34.0 AP at 50 epochs, in their reproduction), and two of four extremity-attending heads fail to find box edges at 50 epochs 2108.06152 §1, fig. 1. Card: Conditional DETR for Fast Training Convergence.

• Split content and spatial parts. Query and key are concatenations of a content part and a spatial part, so the dot product separates into a content term and a spatial term 2108.06152 §3.3, eq. 3. Each slot's spatial query is a sinusoidal embedding of a reference point, scaled element-wise by a vector predicted from the previous decoder layer's output 2108.06152 §3.3, eq. 4–6.
• Reference point. Three options are compared: the fixed point (0,0) as in DETR (36.8 AP at 50 epochs), learned per-query points (40.7), and points predicted from the object query (40.9) 2108.06152 §4.3.
• Heads. Content and spatial queries and keys are projected separately for each of 8 heads 2108.06152 §3.3.
• Loss and queries change too. The paper also takes focal-loss classification and 300 queries from Deformable DETR, and uses Hungarian matching with the same cost 2108.06152 §3.5. These changes alone raise DETR-R50 at 50 epochs from 34.9 to 36.9 AP; the spatial query adds 4.0 more 2108.06152 tab. 4. The attention change is therefore not the whole gain.
• Reported result. ResNet-50 at 50 epochs reaches 40.9 AP, against 34.9 for DETR at the same length and 42.0 for DETR at 500 epochs; at 108 epochs it reaches 43.0. The DC5 variant at 50 epochs (43.8 AP) is above DETR-DC5 at 500 epochs (43.3) 2108.06152 tab. 1. The authors report 6.7x and 10x faster convergence, measured as epochs needed to match DETR at 500 epochs [arxiv:2108.06152 §4.2, tab. 1].

Restricted matching over selected features (TSP)

Keep the Hungarian set loss, but drop the decoder. A transformer encoder runs only on a short list of candidates: 700 locations picked by an FCOS-style scoring head (TSP-FCOS) or 700 RPN proposals (TSP-RCNN) 2011.10881 §4.1, §4.2, §5.2. Self-attention cost then depends on that list rather than on every feature location 2011.10881 §4.1. Card: Rethinking Transformer-based Set Prediction for Object Detection.

• Restricted matching. For TSP-FCOS a location can match an object only if it lies inside the object's box at a suitable pyramid level; for TSP-RCNN a proposal is positive only if its IoU with a ground-truth box is above 0.5 2011.10881 §4.1, §4.2. These are the hand-written assignment rules DETR removed, kept here for the candidate list. See label assignment.
• Diagnosis. The authors report that a matching distilled from a trained teacher helps only the first few epochs, and that cross-attention sparsity keeps rising after 100 epochs; they read the cross-attention as the larger factor. Encoder-only DETR matches DETR's overall AP [arxiv:2011.10881 §3.1–§3.3, fig. 1, fig. 2, fig. 4]. These are the authors' observations; the paper does not isolate each cause.
• Reported result. ResNet-50, 36 epochs: TSP-FCOS 43.1 AP and TSP-RCNN 43.8 AP, against 42.0 for DETR+ at 500 epochs. The "+" rows use random crops and a longer schedule, so the 36-epoch and 500-epoch rows are not one training recipe 2011.10881 tab. 1. TSP-RCNN reaches 28.6 AP-S against 20.5 for DETR+, but 55.7 AP-L against 61.1 2011.10881 tab. 1.
• Cost. The authors measure TSP-RCNN at 188 GFLOPs and 11 FPS, and DETR+ at 86 GFLOPs and 21 FPS 2011.10881 tab. 1.

One-to-many training branch (H-DETR)

Keep one-to-one matching for the slots used at test time, and add a training-only group in which each ground truth is repeated K times, so several slots learn from every object. A mask stops the two groups from attending to each other. At inference only the one-to-one group runs, so no NMS is applied 2207.13080 §3.2. Card: DETRs with Hybrid Matching.

• Motivation. With one-to-one matching, Deformable-DETR selects fewer than 30 of its 300 queries as positives per COCO image; the authors argue this limits training 2207.13080 §1, fig. 1.
• Where the invariant changes. During training each ground truth is matched to one slot in the one-to-one group and to K slots in the one-to-many group 2207.13080 §3.2. Only the one-to-one slots are kept at inference, so the one-slot-per-target rule holds for the output, not for the loss.
• Reported result. Deformable-DETR R50 rises from 47.0 to 48.7 AP at 12 epochs and from 49.0 to 50.0 at 36 epochs on COCO val 2207.13080 tab. 1. The paper's COCO setting uses K=6.
• Cost. At 1800 total queries the R50 model trains in 80 minutes instead of 65 and uses 7728 MB of GPU memory instead of 5480. Inference speed is unchanged at 6.7 FPS 2207.13080 tab. 6, tab. 7.
• Inference path matters. The one-to-many group alone, used at test time without NMS, gives 13.5 AP 2207.13080 tab. 13.
• Choice of K. On COCO, K=0 gives 47.0 AP and K=6 gives 48.6 2207.13080 tab. 9. The paper reports consistent gains only for K above 3 2207.13080 §4.2.

Neighbour-context matching for dense points (CLTR)

Crowd localization: the set predictor outputs the centre of every head, trained from point labels only, with no boxes, density maps or NMS 2202.13065 §1. A ResNet-50 feature map at stride 32 feeds six encoder and six decoder layers with 500 learned queries; each query gives a point and a confidence 2202.13065 §3, §4.1. Card: An End-to-End Transformer Model for Crowd Localization.

• Context-aware matching. The matching cost adds an L1 term between each head's mean distance to its k nearest neighbours, so a prediction is paired with a head that has a similar local layout 2202.13065 §3.3, eq. 2–3. The authors state the neighbour value is an output of the network 2202.13065 §3.3.
• Ablation. On UCF-QNRF the plain L1-plus-confidence cost gives F1 80.02 and the neighbour-context cost gives 80.97. These are single runs 2202.13065 tab. 8.
• Reported result. F1 80.97 on UCF-QNRF, averaged over σ = 1 to 100 px, against 80.34 for TopoCount 2202.13065 tab. 2. On ShanghaiTech Part A at σ=4, F1 43.2 against 41.1 for TopoCount 2202.13065 tab. 3. On NWPU-Crowd test, F1 68.5 against 69.1 for TopoCount, which uses box labels 2202.13065 tab. 1.
• Cost. 43.4M parameters, the most in the authors' comparison, and 157.2 GMACs at 1024×768, the fewest 2202.13065 tab. 10.

Set predictor on table structure (PubTables-1M)

The dataset paper applies DETR without change to two tasks: table detection, and table structure with functional analysis 2110.00061 §4. The structure model predicts six object types: the table, its columns, its rows, its column-header cells, its projected row headers and its spanning cells. Grid cells are their intersections 2110.00061 §4. Card: PubTables-1M.

• Query counts. 15 queries for table detection and 125 for structure and functional analysis 2110.00061 app. Training Details.
• Inference adds a step. The ground truth is conflict-free, so the authors add a conflict-resolution step at inference, similar to NMS but with overlap measured in children 2110.00061 app. Inference. This pipeline therefore does not match the plain "no NMS" description.
• Reported result. On canonical test tables, DETR reaches content accuracy 0.8138 and GriTS_Top 0.9845, against 0.1039 and 0.8616 for Faster R-CNN on the same data 2110.00061 tab. 4. The paper reports one run per cell with no variance 2110.00061 tab. 4.
• Data dependence. The labels come from a canonicalization step designed for PubMed Central markup, which the authors say may not transfer to other sources 2110.00061 §3, Limitations.

Query decoder over fused sensor features (DeepInteraction)

A 3D detector on LiDAR and six cameras. Each sensor keeps its own backbone, an encoder of two layers lets the two feature sets interact, and a decoder refines 200 object queries alternately with image and LiDAR features 2208.11112 §3, §3.1, §3.2. Card: DeepInteraction.

• Query start and matching. Queries start at the strongest heatmap peaks, as in TransFusion, and matching is Hungarian as in TransFusion 2208.11112 §3.2, §4.2. Every decoder layer is supervised 2208.11112 §3.2.
• Test-time output. The paper does not mention NMS. The card's "Paper vs code" section records that the released code applies none at test time 2208.11112 §4.2.
• Decoder layers. Replacing the interaction layers with DETR decoder layers in both branches gives 68.6 mAP on val, against 69.9 for the full model 2208.11112 tab. 3(a).
• Reported result. On nuScenes test, 70.8 mAP and 73.4 NDS for the ResNet-50 base model, against 68.9 and 71.6 for TransFusion 2208.11112 tab. 1.
• Cost and dependency. On A100 the model runs at 4.9 FPS, against 6.2 for TransFusion [arxiv:2208.11112 tab. 2]. The method needs sensor calibration to build 2D-3D correspondences 2208.11112 app. A.3.

Track queries carried across frames (TrackFormer)

The same decoder runs frame by frame, and each active track is a query: the previous frame's output embedding for that object, taken from the last decoder layer 2101.02702 §3.2. Learned object queries (500 on MOT17) still start new tracks 2101.02702 §3.2, §4.2. Track-query outputs are matched to ground-truth objects by track identity, and only new objects are matched to object queries by cost 2101.02702 §3.3. Card: TrackFormer: Multi-Object Tracking with Transformers.

• Training on frame pairs. The previous frame is sampled from a window around the current one; the window size is not stated 2101.02702 §3.3. Two augmentations add errors on purpose: track queries are dropped with probability 0.4 (false negatives), and background outputs are added as false-positive tracks with probability 0.1 per track query 2101.02702 §3.3, app. A.1.
• Duplicate handling. Tracks are removed by NMS at IoU 0.9 2101.02702 §3.2.
• Reported result. MOT17 test, private detections, CrowdHuman training only: 74.1 MOTA and 68.0 IDF1, against 69.1 and 63.9 for TraDeS 2101.02702 tab. 1. With public detections, IDF1 is 57.6 against 62.8 for TMOH (MOTA 62.3 against 62.1) 2101.02702 tab. 1. Runtime is 7.4 FPS, self-measured 2101.02702 tab. 1.
• Ablation. The row without track queries, which uses greedy center matching, is 14.1 IDF1 below the row above it. The Δ columns are cumulative, so this is not a stand-alone effect of track queries 2101.02702 tab. 3.

Reference points projected into several cameras (DETR3D)

Each query is decoded to a 3D reference point. The point is projected into every camera with that camera's 3×4 matrix, features are sampled there at every level, and the visible samples are averaged and added back to the query 2110.06922 eq. 1–4. Self-attention among queries follows 2110.06922 §3.3. Card: DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries.

• Calibration is an input. The method assumes the camera projection matrices are known 2110.06922 §3.1.
• No NMS. Inference uses the last layer only, with no NMS 2110.06922 §3.3.
• Reported result. nuScenes validation: 0.374 NDS and 0.303 mAP for the plain model, against 0.373 and 0.299 for FCOS3D with NMS 2110.06922 tab. 1. The mATE is worse, 0.860 against 0.785; the authors suspect FCOS3D's direct depth prediction supervises translation more strongly 2110.06922 tab. 1, §4.2. NDS rises from 0.380 at the first decoder layer to 0.425 at the last 2110.06922 tab. 5.
• Released code. The released code departs from the equations in several places, including how the reference point is formed and how outputs are capped; read the card's Paper vs code section before reimplementing.

One instance sequence per slot across a clip (VisTR)

The same set predictor runs on a short clip of T=36 frames instead of one image. It has N = n \cdot T learned queries, with n=10 instances per frame, and slot k of every frame forms one instance sequence. Tracking therefore comes from the slot layout, not from a separate linking step 2011.14503 §3.1, §4.1. Card: End-to-End Video Instance Segmentation with Transformers.

• Sequence-level matching. A slot's cost sums over all T frames, combining class probability and box terms. A one-to-one Hungarian assignment then pairs each sequence with one ground-truth instance 2011.14503 §3.2, eq. 2, eq. 6. The matched pair's loss is a class term, a box loss averaged over frames, and a mask loss 2011.14503 §3.2, eq. 7, eq. 8.
• Masks over time. Per-frame mask features are stacked along time for each instance. Three 3D convolution layers then produce one mask per frame, supervised with Dice and focal terms 2011.14503 §3.3, eq. 9.
• Query layout. The default of 360 prediction-level queries gives 33.3 AP. Ten instance-level embeddings shared over frames give 32.0, 36 frame-level embeddings give 13.7, and one video-level embedding gives 8.4. These are 10-epoch ResNet-50 ablations 2011.14503 tab. 1 (b).
• Clip length. In the same ablation, AP rises with clip length: 29.7 at T=18 and 33.3 at T=36 2011.14503 tab. 1 (a). The paper's own results use the longest YouTube-VIS clip as the fixed length 2011.14503 §4.1.
• Reported result. YouTube-VIS validation, ResNet-101: 40.1 AP, 64.0 AP50 and 45.0 AP75, at 27.7 FPS with data loading and 57.7 FPS without it. The ResNet-50 model reaches 36.2 AP 2011.14503 tab. 2.

Temporal deformable attention over snippets (TadTR)

The same set predictor finds action segments in an untrimmed video. Each query outputs a class, a segment (normalised centre and length) and an actionness score. The input is a sequence of pre-extracted snippet features, projected to width 256 2106.10271 §III-A, §IV-A. Card: End-to-end Temporal Action Detection with Transformer.

• Local attention. Dense attention is replaced by temporal deformable attention in the encoder and in decoder cross-attention. Each head samples K = 4 points around a reference point, with 8 heads 2106.10271 §III-A, §IV-A. The authors argue that dense attention is insensitive to small local changes between neighbouring snippets, and that it costs more on long sequences 2106.10271 §I.
• Iterative segment refinement. Each decoder layer predicts offsets to the segment from the layer before it, and the next layer's cross-attention reference is the refined centre 2106.10271 §III-A, eq. 4–5.
• Actionness head. The predicted segment is enlarged by a factor of 1.5 and pooled with 16 temporal RoIAlign bins. An MLP regresses the maximum IoU between the segment and any ground-truth action 2106.10271 §III-A, §IV-A, tab. 11. The paper puts this head at about 6.59% of the model's FLOPs 2106.10271 §III-A. Inference combines the class probability with actionness 2106.10271 §III-B.
• Duplicate handling. Queries are matched one-to-one with the Hungarian algorithm, and the paper says no NMS is needed 2106.10271 §III-B. Overlapping test windows on THUMOS14 are merged with NMS 2106.10271 §IV-A, §IV-B.
• Query count. The paper uses 40 queries on THUMOS14, 30 on HACS and 10 on ActivityNet-1.3 2106.10271 §IV-C. On HACS, 30 queries give 30.83 average mAP and 40 give 29.99 2106.10271 tab. 8.
• Ablation on HACS. The full model reaches 30.83 average mAP. Without the encoder it reaches 26.94, and without actionness 28.51. Without segment refinement it reaches 27.65, and dense attention in place of the deformable one 13.58 2106.10271 tab. 4.
• Reported result. THUMOS14 test, average mAP: 49.3 for the plain model and 56.7 for the starred row, which uses IBIF and focal loss 2106.10271 tab. 1, §IV-B. ActivityNet-1.3 with two-stream features and no external classifier: 28.21 plain and 29.90 starred 2106.10271 tab. 3.
• Limits. The model misses actions when a video has more true actions than queries, and the authors report weaker results on short actions 2106.10271 §IV-C, §IV-D. The paper trains on frozen features and names end-to-end training of the video encoder as future work 2106.10271 §III, §V.

When to use which

Implementation sketch

Common core of a set predictor, following DETR 2005.12872 §3.1, §3.2, §4:

Train:
  feats  = Encoder(Backbone(x))
  slots  = Decoder(queries = learned N x d, memory = feats)   # parallel, slots attend to each other
  preds  = for each slot: (softmax over K classes + no-object, box)
  targets padded to N with no-object
  σ      = min-cost one-to-one assignment(targets, preds)     # see bipartite-matching
  loss   = class loss on all N slots under σ (no-object down-weighted)
           + box loss on slots matched to real objects
  (optionally repeat for every decoder layer's output, shared heads)

Infer:
  preds = model(x)                       # exactly N outputs, no NMS
  keep slots by class score; DETR's AP numbers instead relabel no-object slots
  with their second-best class

Where variants differ: how slots interact (DETR uses self-attention), how decoder cross-attention forms its query and key (Conditional DETR concatenates content and spatial parts 2108.06152 §3.3), the classification loss (softmax with a no-object class in DETR 2005.12872 §3.2; focal loss in Conditional DETR 2108.06152 §3.5), the matching cost, and how many slots are used. N and the no-object weight must be chosen together, since most slots are empty 2005.12872 §3.1.

Two further differences change the sketch. TSP replaces the decoder with an encoder over a fixed list of candidates and limits matching to allowed pairs 2011.10881 §4.1, §4.2. H-DETR adds a training-only group: it repeats every target K times, runs the same matching and loss on the repeated targets, and masks attention between the two groups. Only the one-to-one slots are returned at inference 2207.13080 §3.2. The paper does not state how the repeated-target loss is normalized; check the card's "Paper vs code" section before reimplementing.

Two more changes touch the matching step. CLTR adds a neighbour-context term to the matching cost 2202.13065 §3.3. DeepInteraction starts its queries from heatmap peaks and matches as TransFusion does 2208.11112 §4.2. The table model keeps the DETR loss and adds conflict resolution after inference 2110.00061 app. Inference.

Two variants change what a query is. TrackFormer writes each track's previous-frame output embedding back in as a query, and matches those queries by identity rather than by cost 2101.02702 §3.2, §3.3. DETR3D turns each query into a 3D point, projects it into every camera and samples features at the projections, adding the average back to the query 2110.06922 eq. 1–4. In both, the Hungarian loss stays the training signal and the sketch above is otherwise unchanged.

VisTR changes the unit of matching. Slot k of each frame is one instance sequence, and the cost and losses are averaged over the clip's frames before the Hungarian step 2011.14503 §3.2. The mask head gains a time axis through 3D convolutions 2011.14503 §3.3. The sketch's targets are then instance sequences, not single boxes.

TadTR keeps the sketch's matching and loss and changes the attention and the output. Its encoder uses temporal deformable self-attention over the snippet sequence, and each decoder layer's reference point is the refined segment centre from the layer before 2106.10271 §III-A. Its score head (actionness) is regressed from RoI-pooled features and enters the inference score 2106.10271 §III-A, §III-B.

Contradictions

Papers disagree on what limits training speed, and they measure different things, so their results do not settle the question. H-DETR attributes slow training to few positive slots per image 2207.13080 §1. TSP finds that matching effects fade after about 15 epochs and points to cross-attention sparsity 2011.10881 §3.1, §3.2. Conditional DETR points to DETR's cross-attention depending on content embeddings 2108.06152 §1. The three diagnoses are not tested against one another in a shared setup.

The paper defines TSP's matching as a permutation with minimum total cost 2011.10881 eq. 2. The released code does not enforce that one-to-one assignment: for each ground-truth object it takes the lowest-cost candidate. The TSP card records this in its "Paper vs code" section, together with the matching-cost weights that the paper leaves out.

"No NMS" describes the trained slots, not every released pipeline. DETR's text says NMS is not needed 2005.12872 §1. The TSP-RCNN release runs NMS at test time unless a flag is set; the card lists the threshold. H-DETR's one-to-one slots give the same AP with and without NMS 2207.13080 tab. 13. The table model adds a conflict-resolution step after inference, which works like NMS on children 2110.00061 app. Inference. TrackFormer keeps a duplicate-removal step over tracks at IoU 0.9 2101.02702 §3.2, so a no-NMS property shown for detection does not carry over to every set-prediction pipeline. TadTR states that it needs no NMS 2106.10271 §III-B, yet it merges detections in THUMOS14's overlap regions with NMS 2106.10271 §IV-B. Its text gives a square root of the class-by-actionness product as the inference score and cross-entropy as the default matching cost. The card's "Paper vs code" section records that the released code uses the plain product and a focal-style matching term only 2106.10271 §III-B, §IV-A.

The CLTR abstract claims the best localisation on NWPU-Crowd, but on the test split TopoCount has the higher F1 2202.13065 tab. 1. The text calls CLTR competitive there. Use the table figures. The authors' own code also departs from their equations: it averages three neighbours where the paper says k=4, and it appends the neighbour value as a third coordinate 2202.13065 §4.1, eq. 3. The card's "Paper vs code" section lists the differences.

Conditional DETR's DETR-R50 baseline at 500 epochs (42.0 AP) agrees with the DETR paper's own result 2108.06152 tab. 1 2005.12872 tab. 1. Within DETR, the text states +7.8 APL and -5.5 APS against Faster R-CNN 2005.12872 §4.1, while tab. 1 gives +7.7 and -6.1 against Faster R-CNN-FPN+ 2005.12872 tab. 1.

The authors describe VisTR as the most accurate single model on YouTube-VIS. The same table lists MaskProp with ResNet-101 at 42.5 AP, above VisTR's 40.1 2011.14503 tab. 2. The paper excludes MaskProp because it combines several networks and post-processing 2011.14503 §4.3. That exclusion is the authors' argument; the table alone does not decide the comparison. The paper's clip-length study also relies on matching instances across overlapping clips 2011.14503 §4.2, which the released inference code does not implement; the card's Paper vs code section gives the details.

Papers

• End-to-End Object Detection with Transformers: detection as direct set prediction with a transformer decoder and a Hungarian matching loss, no anchors or NMS.
• Conditional DETR for Fast Training Convergence: keeps DETR's set predictor and changes decoder cross-attention to use an explicit spatial query per slot, reaching DETR's accuracy in far fewer epochs.
• Rethinking Transformer-based Set Prediction for Object Detection: replaces the decoder with an encoder over 700 FCOS or RPN candidates, with restricted matching, and reaches DETR's accuracy in 36 epochs at higher compute.
• DETRs with Hybrid Matching: adds a training-only one-to-many query group with repeated targets and masked self-attention; inference keeps the one-to-one group.
• An End-to-End Transformer Model for Crowd Localization: regresses head points with 500 learned queries, and adds a neighbour-context term to the Hungarian matching cost.
• PubTables-1M: Towards comprehensive table extraction from unstructured documents: a table dataset with canonicalized structure labels; DETR is applied unchanged to table detection and structure recognition.
• DeepInteraction: 3D Object Detection via Modality Interaction: keeps LiDAR and camera representations separate, and refines 200 queries alternately with each modality in the decoder.
• TrackFormer: Multi-Object Tracking with Transformers: a DETR-style decoder whose track queries carry each object's embedding into the next frame, so detection and association share one attention step.
• DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries: object queries decoded to 3D reference points, projected into six cameras to sample features, trained with a Hungarian set loss and no NMS.
• End-to-End Video Instance Segmentation with Transformers: VisTR; each slot yields one instance sequence over a 36-frame clip, matched and supervised as a whole, so no tracking heuristics are needed.
• End-to-end Temporal Action Detection with Transformer: TadTR; action queries predict segments in untrimmed video with temporal deformable attention, segment refinement and an actionness head, matched one-to-one, so no anchors or proposal stage are needed.