--- kind: concept slug: set-prediction title: Direct set prediction summary: >- Framing a task whose answer is an unordered, variable-size collection (e.g. objects in an image) as predicting the whole set at once: a fixed number of output slots, padded with "no object", trained with a permutation-invariant matching loss. DETR drops anchors and NMS; it costs a hard output cap and long training. Variants shorten training, add training-only supervision or change matching cost. tags: [object-detection, set-prediction, transformer, computer-vision, loss-function, training-efficiency, multi-object-tracking, 3d-object-detection, video-instance-segmentation, temporal-action-detection] papers: ["2005.12872", "2108.06152", "2011.10881", "2207.13080", "2202.13065", "2110.00061", "2208.11112", "2101.02702", "2110.06922", "2011.14503", "2106.10271"] related: [bipartite-matching, object-queries, vision-transformer, label-assignment, object-detection, bev-representation, visual-object-tracking, instance-segmentation, deformable-convolution] status: draft updated_at: 2026-11-01 --- ## Overview Many outputs are sets: the objects in a photo, the segments of a scene. Order carries no meaning, the size changes from example to example, and the same element must not be reported twice. Classic detectors do not predict this set directly. They score a large pool of anchors, proposals or window centers, assign ground truth to them with hand-written rules, and then prune near-duplicates with non-maximum suppression (NMS); accuracy depends heavily on those steps [arxiv:2005.12872 §1]. Direct set prediction asks the model for the final set. Three ingredients are shared by the approach [arxiv:2005.12872 §3]: 1. **A fixed number of slots.** The model always emits $N$ predictions, with $N$ chosen well above the typical set size; the target set is padded to $N$ with a "no object" element $\varnothing$ [arxiv:2005.12872 §3.1]. 2. **A permutation-invariant loss.** Because slot order is arbitrary, each target is paired with one prediction by a minimum-cost one-to-one assignment before the loss is computed [arxiv:2005.12872 eq. 1, eq. 2]. The details live on [bipartite matching](/concepts/bipartite-matching). 3. **Interaction between slots.** To avoid duplicates without post-processing, slots need to know what the others are predicting; in DETR this happens through self-attention among the decoder slots [arxiv:2005.12872 §3.2]. The slot embeddings are covered on [object queries](/concepts/object-queries). The invariant of the pure form: the training signal is unchanged under any reordering of the predictions, and each target supervises exactly one slot, so duplicates are penalized during training rather than removed afterwards [arxiv:2005.12872 §1, §3.1]. One variant below relaxes the one-slot rule during training only [arxiv:2207.13080 §3.2]. ## Variants Eleven papers with cards use this formulation. All keep a set predictor trained with a matching loss. They differ in the decoder, in the candidate slots the encoder or decoder works on, in the matching cost, in the training schedule, and in what a query carries: image content, a candidate point, a track from the previous frame, a 3D reference point, one instance followed across a short video clip, or an action segment in a long untrimmed video. The sections below describe design choices, not a survey of competing methods. ### Parallel decoding with a transformer (DETR) A CNN backbone and transformer encoder produce image features; a transformer decoder turns $N=100$ learned query slots into $N$ (class, box) pairs in one pass, not one element at a time [arxiv:2005.12872 §3.2, app. A.4]. Card: [End-to-End Object Detection with Transformers](/papers/2005.12872). - **Decoding order.** The paper contrasts its parallel decoding with earlier set-prediction detectors, most of which decoded autoregressively with RNNs [arxiv:2005.12872 §1, §2]. Those earlier methods are named only as background here; no card covers them. - **Duplicate handling.** No NMS. Applying NMS anyway helps the output of the first decoder layer but slightly lowers AP after the last layers, because by then it mostly removes true positives [arxiv:2005.12872 §4.2, fig. 4]. AP rises after every decoder layer, +8.2 from first to last [arxiv:2005.12872 §4.2]. - **Each slot is decoded independently** by shared heads: a 3-layer MLP for the normalized box and a linear softmax over classes plus $\varnothing$ [arxiv:2005.12872 §3.2]. - **Inference.** For AP evaluation, a slot predicting $\varnothing$ is given its second-best class and that class's score instead of being dropped; this adds 2 AP over filtering empty slots [arxiv:2005.12872 §4]. - **Reported result.** ResNet-50 DETR: 42.0 AP on COCO val, the same as Faster R-CNN-FPN trained with GIoU, random crops and a 9x schedule; 61.1 vs 53.4 AP on large objects, 20.5 vs 26.6 on small ones [arxiv:2005.12872 tab. 1]. ### Conditional cross-attention for faster convergence (Conditional DETR) The same set predictor, with a changed decoder cross-attention. The authors observe that DETR's cross-attention depends heavily on content embeddings: removing positional embeddings from the keys and queries of the second decoder layer onward costs little (34.9 to 34.0 AP at 50 epochs, in their reproduction), and two of four extremity-attending heads fail to find box edges at 50 epochs [arxiv:2108.06152 §1, fig. 1]. Card: [Conditional DETR for Fast Training Convergence](/papers/2108.06152). - **Split content and spatial parts.** Query and key are concatenations of a content part and a spatial part, so the dot product separates into a content term and a spatial term [arxiv:2108.06152 §3.3, eq. 3]. Each slot's spatial query is a sinusoidal embedding of a reference point, scaled element-wise by a vector predicted from the previous decoder layer's output [arxiv:2108.06152 §3.3, eq. 4–6]. - **Reference point.** Three options are compared: the fixed point (0,0) as in DETR (36.8 AP at 50 epochs), learned per-query points (40.7), and points predicted from the object query (40.9) [arxiv:2108.06152 §4.3]. - **Heads.** Content and spatial queries and keys are projected separately for each of 8 heads [arxiv:2108.06152 §3.3]. - **Loss and queries change too.** The paper also takes focal-loss classification and 300 queries from Deformable DETR, and uses Hungarian matching with the same cost [arxiv:2108.06152 §3.5]. These changes alone raise DETR-R50 at 50 epochs from 34.9 to 36.9 AP; the spatial query adds 4.0 more [arxiv:2108.06152 tab. 4]. The attention change is therefore not the whole gain. - **Reported result.** ResNet-50 at 50 epochs reaches 40.9 AP, against 34.9 for DETR at the same length and 42.0 for DETR at 500 epochs; at 108 epochs it reaches 43.0. The DC5 variant at 50 epochs (43.8 AP) is above DETR-DC5 at 500 epochs (43.3) [arxiv:2108.06152 tab. 1]. The authors report 6.7x and 10x faster convergence, measured as epochs needed to match DETR at 500 epochs [arxiv:2108.06152 §4.2, tab. 1]. ### Restricted matching over selected features (TSP) Keep the Hungarian set loss, but drop the decoder. A transformer encoder runs only on a short list of candidates: 700 locations picked by an FCOS-style scoring head (TSP-FCOS) or 700 RPN proposals (TSP-RCNN) [arxiv:2011.10881 §4.1, §4.2, §5.2]. Self-attention cost then depends on that list rather than on every feature location [arxiv:2011.10881 §4.1]. Card: [Rethinking Transformer-based Set Prediction for Object Detection](/papers/2011.10881). - **Restricted matching.** For TSP-FCOS a location can match an object only if it lies inside the object's box at a suitable pyramid level; for TSP-RCNN a proposal is positive only if its IoU with a ground-truth box is above 0.5 [arxiv:2011.10881 §4.1, §4.2]. These are the hand-written assignment rules DETR removed, kept here for the candidate list. See [label assignment](/concepts/label-assignment). - **Diagnosis.** The authors report that a matching distilled from a trained teacher helps only the first few epochs, and that cross-attention sparsity keeps rising after 100 epochs; they read the cross-attention as the larger factor. Encoder-only DETR matches DETR's overall AP [arxiv:2011.10881 §3.1–§3.3, fig. 1, fig. 2, fig. 4]. These are the authors' observations; the paper does not isolate each cause. - **Reported result.** ResNet-50, 36 epochs: TSP-FCOS 43.1 AP and TSP-RCNN 43.8 AP, against 42.0 for DETR+ at 500 epochs. The "+" rows use random crops and a longer schedule, so the 36-epoch and 500-epoch rows are not one training recipe [arxiv:2011.10881 tab. 1]. TSP-RCNN reaches 28.6 AP-S against 20.5 for DETR+, but 55.7 AP-L against 61.1 [arxiv:2011.10881 tab. 1]. - **Cost.** The authors measure TSP-RCNN at 188 GFLOPs and 11 FPS, and DETR+ at 86 GFLOPs and 21 FPS [arxiv:2011.10881 tab. 1]. ### One-to-many training branch (H-DETR) Keep one-to-one matching for the slots used at test time, and add a training-only group in which each ground truth is repeated $K$ times, so several slots learn from every object. A mask stops the two groups from attending to each other. At inference only the one-to-one group runs, so no NMS is applied [arxiv:2207.13080 §3.2]. Card: [DETRs with Hybrid Matching](/papers/2207.13080). - **Motivation.** With one-to-one matching, Deformable-DETR selects fewer than 30 of its 300 queries as positives per COCO image; the authors argue this limits training [arxiv:2207.13080 §1, fig. 1]. - **Where the invariant changes.** During training each ground truth is matched to one slot in the one-to-one group and to $K$ slots in the one-to-many group [arxiv:2207.13080 §3.2]. Only the one-to-one slots are kept at inference, so the one-slot-per-target rule holds for the output, not for the loss. - **Reported result.** Deformable-DETR R50 rises from 47.0 to 48.7 AP at 12 epochs and from 49.0 to 50.0 at 36 epochs on COCO val [arxiv:2207.13080 tab. 1]. The paper's COCO setting uses $K=6$. - **Cost.** At 1800 total queries the R50 model trains in 80 minutes instead of 65 and uses 7728 MB of GPU memory instead of 5480. Inference speed is unchanged at 6.7 FPS [arxiv:2207.13080 tab. 6, tab. 7]. - **Inference path matters.** The one-to-many group alone, used at test time without NMS, gives 13.5 AP [arxiv:2207.13080 tab. 13]. - **Choice of K.** On COCO, K=0 gives 47.0 AP and K=6 gives 48.6 [arxiv:2207.13080 tab. 9]. The paper reports consistent gains only for K above 3 [arxiv:2207.13080 §4.2]. ### Neighbour-context matching for dense points (CLTR) Crowd localization: the set predictor outputs the centre of every head, trained from point labels only, with no boxes, density maps or NMS [arxiv:2202.13065 §1]. A ResNet-50 feature map at stride 32 feeds six encoder and six decoder layers with 500 learned queries; each query gives a point and a confidence [arxiv:2202.13065 §3, §4.1]. Card: [An End-to-End Transformer Model for Crowd Localization](/papers/2202.13065). - **Context-aware matching.** The matching cost adds an L1 term between each head's mean distance to its k nearest neighbours, so a prediction is paired with a head that has a similar local layout [arxiv:2202.13065 §3.3, eq. 2–3]. The authors state the neighbour value is an output of the network [arxiv:2202.13065 §3.3]. - **Ablation.** On UCF-QNRF the plain L1-plus-confidence cost gives F1 80.02 and the neighbour-context cost gives 80.97. These are single runs [arxiv:2202.13065 tab. 8]. - **Reported result.** F1 80.97 on UCF-QNRF, averaged over σ = 1 to 100 px, against 80.34 for TopoCount [arxiv:2202.13065 tab. 2]. On ShanghaiTech Part A at σ=4, F1 43.2 against 41.1 for TopoCount [arxiv:2202.13065 tab. 3]. On NWPU-Crowd test, F1 68.5 against 69.1 for TopoCount, which uses box labels [arxiv:2202.13065 tab. 1]. - **Cost.** 43.4M parameters, the most in the authors' comparison, and 157.2 GMACs at 1024×768, the fewest [arxiv:2202.13065 tab. 10]. ### Set predictor on table structure (PubTables-1M) The dataset paper applies DETR without change to two tasks: table detection, and table structure with functional analysis [arxiv:2110.00061 §4]. The structure model predicts six object types: the table, its columns, its rows, its column-header cells, its projected row headers and its spanning cells. Grid cells are their intersections [arxiv:2110.00061 §4]. Card: [PubTables-1M](/papers/2110.00061). - **Query counts.** 15 queries for table detection and 125 for structure and functional analysis [arxiv:2110.00061 app. Training Details]. - **Inference adds a step.** The ground truth is conflict-free, so the authors add a conflict-resolution step at inference, similar to NMS but with overlap measured in children [arxiv:2110.00061 app. Inference]. This pipeline therefore does not match the plain "no NMS" description. - **Reported result.** On canonical test tables, DETR reaches content accuracy 0.8138 and GriTS_Top 0.9845, against 0.1039 and 0.8616 for Faster R-CNN on the same data [arxiv:2110.00061 tab. 4]. The paper reports one run per cell with no variance [arxiv:2110.00061 tab. 4]. - **Data dependence.** The labels come from a canonicalization step designed for PubMed Central markup, which the authors say may not transfer to other sources [arxiv:2110.00061 §3, Limitations]. ### Query decoder over fused sensor features (DeepInteraction) A 3D detector on LiDAR and six cameras. Each sensor keeps its own backbone, an encoder of two layers lets the two feature sets interact, and a decoder refines 200 object queries alternately with image and LiDAR features [arxiv:2208.11112 §3, §3.1, §3.2]. Card: [DeepInteraction](/papers/2208.11112). - **Query start and matching.** Queries start at the strongest heatmap peaks, as in TransFusion, and matching is Hungarian as in TransFusion [arxiv:2208.11112 §3.2, §4.2]. Every decoder layer is supervised [arxiv:2208.11112 §3.2]. - **Test-time output.** The paper does not mention NMS. The card's "Paper vs code" section records that the released code applies none at test time [arxiv:2208.11112 §4.2]. - **Decoder layers.** Replacing the interaction layers with DETR decoder layers in both branches gives 68.6 mAP on val, against 69.9 for the full model [arxiv:2208.11112 tab. 3(a)]. - **Reported result.** On nuScenes test, 70.8 mAP and 73.4 NDS for the ResNet-50 base model, against 68.9 and 71.6 for TransFusion [arxiv:2208.11112 tab. 1]. - **Cost and dependency.** On A100 the model runs at 4.9 FPS, against 6.2 for TransFusion [arxiv:2208.11112 tab. 2]. The method needs sensor calibration to build 2D-3D correspondences [arxiv:2208.11112 app. A.3]. ### Track queries carried across frames (TrackFormer) The same decoder runs frame by frame, and each active track is a query: the previous frame's output embedding for that object, taken from the last decoder layer [arxiv:2101.02702 §3.2]. Learned object queries (500 on MOT17) still start new tracks [arxiv:2101.02702 §3.2, §4.2]. Track-query outputs are matched to ground-truth objects by track identity, and only new objects are matched to object queries by cost [arxiv:2101.02702 §3.3]. Card: [TrackFormer: Multi-Object Tracking with Transformers](/papers/2101.02702). - **Training on frame pairs.** The previous frame is sampled from a window around the current one; the window size is not stated [arxiv:2101.02702 §3.3]. Two augmentations add errors on purpose: track queries are dropped with probability 0.4 (false negatives), and background outputs are added as false-positive tracks with probability 0.1 per track query [arxiv:2101.02702 §3.3, app. A.1]. - **Duplicate handling.** Tracks are removed by NMS at IoU 0.9 [arxiv:2101.02702 §3.2]. - **Reported result.** MOT17 test, private detections, CrowdHuman training only: 74.1 MOTA and 68.0 IDF1, against 69.1 and 63.9 for TraDeS [arxiv:2101.02702 tab. 1]. With public detections, IDF1 is 57.6 against 62.8 for TMOH (MOTA 62.3 against 62.1) [arxiv:2101.02702 tab. 1]. Runtime is 7.4 FPS, self-measured [arxiv:2101.02702 tab. 1]. - **Ablation.** The row without track queries, which uses greedy center matching, is 14.1 IDF1 below the row above it. The Δ columns are cumulative, so this is not a stand-alone effect of track queries [arxiv:2101.02702 tab. 3]. ### Reference points projected into several cameras (DETR3D) Each query is decoded to a 3D reference point. The point is projected into every camera with that camera's 3×4 matrix, features are sampled there at every level, and the visible samples are averaged and added back to the query [arxiv:2110.06922 eq. 1–4]. Self-attention among queries follows [arxiv:2110.06922 §3.3]. Card: [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](/papers/2110.06922). - **Calibration is an input.** The method assumes the camera projection matrices are known [arxiv:2110.06922 §3.1]. - **No NMS.** Inference uses the last layer only, with no NMS [arxiv:2110.06922 §3.3]. - **Reported result.** nuScenes validation: 0.374 NDS and 0.303 mAP for the plain model, against 0.373 and 0.299 for FCOS3D with NMS [arxiv:2110.06922 tab. 1]. The mATE is worse, 0.860 against 0.785; the authors suspect FCOS3D's direct depth prediction supervises translation more strongly [arxiv:2110.06922 tab. 1, §4.2]. NDS rises from 0.380 at the first decoder layer to 0.425 at the last [arxiv:2110.06922 tab. 5]. - **Released code.** The released code departs from the equations in several places, including how the reference point is formed and how outputs are capped; read the [card's Paper vs code section](/papers/2110.06922) before reimplementing. ### One instance sequence per slot across a clip (VisTR) The same set predictor runs on a short clip of $T=36$ frames instead of one image. It has $N = n \cdot T$ learned queries, with $n=10$ instances per frame, and slot $k$ of every frame forms one instance sequence. Tracking therefore comes from the slot layout, not from a separate linking step [arxiv:2011.14503 §3.1, §4.1]. Card: [End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503). - **Sequence-level matching.** A slot's cost sums over all $T$ frames, combining class probability and box terms. A one-to-one Hungarian assignment then pairs each sequence with one ground-truth instance [arxiv:2011.14503 §3.2, eq. 2, eq. 6]. The matched pair's loss is a class term, a box loss averaged over frames, and a mask loss [arxiv:2011.14503 §3.2, eq. 7, eq. 8]. - **Masks over time.** Per-frame mask features are stacked along time for each instance. Three 3D convolution layers then produce one mask per frame, supervised with Dice and focal terms [arxiv:2011.14503 §3.3, eq. 9]. - **Query layout.** The default of 360 prediction-level queries gives 33.3 AP. Ten instance-level embeddings shared over frames give 32.0, 36 frame-level embeddings give 13.7, and one video-level embedding gives 8.4. These are 10-epoch ResNet-50 ablations [arxiv:2011.14503 tab. 1 (b)]. - **Clip length.** In the same ablation, AP rises with clip length: 29.7 at T=18 and 33.3 at T=36 [arxiv:2011.14503 tab. 1 (a)]. The paper's own results use the longest YouTube-VIS clip as the fixed length [arxiv:2011.14503 §4.1]. - **Reported result.** YouTube-VIS validation, ResNet-101: 40.1 AP, 64.0 AP50 and 45.0 AP75, at 27.7 FPS with data loading and 57.7 FPS without it. The ResNet-50 model reaches 36.2 AP [arxiv:2011.14503 tab. 2]. ### Temporal deformable attention over snippets (TadTR) The same set predictor finds action segments in an untrimmed video. Each query outputs a class, a segment (normalised centre and length) and an actionness score. The input is a sequence of pre-extracted snippet features, projected to width 256 [arxiv:2106.10271 §III-A, §IV-A]. Card: [End-to-end Temporal Action Detection with Transformer](/papers/2106.10271). - **Local attention.** Dense attention is replaced by temporal deformable attention in the encoder and in decoder cross-attention. Each head samples K = 4 points around a reference point, with 8 heads [arxiv:2106.10271 §III-A, §IV-A]. The authors argue that dense attention is insensitive to small local changes between neighbouring snippets, and that it costs more on long sequences [arxiv:2106.10271 §I]. - **Iterative segment refinement.** Each decoder layer predicts offsets to the segment from the layer before it, and the next layer's cross-attention reference is the refined centre [arxiv:2106.10271 §III-A, eq. 4–5]. - **Actionness head.** The predicted segment is enlarged by a factor of 1.5 and pooled with 16 temporal RoIAlign bins. An MLP regresses the maximum IoU between the segment and any ground-truth action [arxiv:2106.10271 §III-A, §IV-A, tab. 11]. The paper puts this head at about 6.59% of the model's FLOPs [arxiv:2106.10271 §III-A]. Inference combines the class probability with actionness [arxiv:2106.10271 §III-B]. - **Duplicate handling.** Queries are matched one-to-one with the Hungarian algorithm, and the paper says no NMS is needed [arxiv:2106.10271 §III-B]. Overlapping test windows on THUMOS14 are merged with NMS [arxiv:2106.10271 §IV-A, §IV-B]. - **Query count.** The paper uses 40 queries on THUMOS14, 30 on HACS and 10 on ActivityNet-1.3 [arxiv:2106.10271 §IV-C]. On HACS, 30 queries give 30.83 average mAP and 40 give 29.99 [arxiv:2106.10271 tab. 8]. - **Ablation on HACS.** The full model reaches 30.83 average mAP. Without the encoder it reaches 26.94, and without actionness 28.51. Without segment refinement it reaches 27.65, and dense attention in place of the deformable one 13.58 [arxiv:2106.10271 tab. 4]. - **Reported result.** THUMOS14 test, average mAP: 49.3 for the plain model and 56.7 for the starred row, which uses IBIF and focal loss [arxiv:2106.10271 tab. 1, §IV-B]. ActivityNet-1.3 with two-stream features and no external classifier: 28.21 plain and 29.90 starred [arxiv:2106.10271 tab. 3]. - **Limits.** The model misses actions when a video has more true actions than queries, and the authors report weaker results on short actions [arxiv:2106.10271 §IV-C, §IV-D]. The paper trains on frozen features and names end-to-end training of the video encoder as future work [arxiv:2106.10271 §III, §V]. ## When to use which | Situation | Use | Why | |---|---|---| | You want a detector without anchors, assignment heuristics or NMS | Direct set prediction | Matching penalizes duplicates in training; the pipeline is a standard CNN plus transformer [arxiv:2005.12872 §1, §4.2] | | Training budget is short, around 50 epochs | Conditional cross-attention | 40.9 AP at 50 epochs against 34.9 for DETR-R50, at 90 vs 86 GFLOPs [arxiv:2108.06152 tab. 1] | | You can afford a long schedule and want the published recipe | Plain DETR | 42.0 AP at 500 epochs; Conditional DETR matches that at about 75 epochs (42.1) [arxiv:2005.12872 tab. 1] [arxiv:2108.06152 tab. 1] | | A proposal stage and higher compute are acceptable, and you need fewer epochs | Restricted matching over selected features (TSP-RCNN) | 43.8 AP at 36 epochs against 42.0 for DETR+ at 500 epochs, but at 188 vs 86 GFLOPs and 11 vs 21 FPS [arxiv:2011.10881 tab. 1] | | Few positive slots per object limit learning and training compute is available | One-to-many training branch | +1.7 AP on Deformable-DETR R50 at 12 epochs; training time and memory rise, inference cost does not [arxiv:2207.13080 tab. 1, tab. 6, tab. 7] | | Large objects or scene-level context matter | Set prediction with a transformer decoder | DETR gains +7.7 AP on large objects over Faster R-CNN-FPN+ [arxiv:2005.12872 tab. 1] | | Small objects matter most | Benchmark an anchor-based detector first | DETR trails on small-object AP (20.5 vs 26.6); a stride-16 backbone helps but multiplies encoder self-attention cost by 16 [arxiv:2005.12872 tab. 1, §4]. Conditional DETR-DC5 keeps the encoder cost (195 GFLOPs) [arxiv:2108.06152 tab. 1, §4.2]. TSP-RCNN reports 28.6 AP-S at 36 epochs [arxiv:2011.10881 tab. 1] | | Images can contain very many instances | Set $N$ well above the maximum count, or avoid the approach | $N$ caps the output; with 100 same-class instances DETR found about 30 on average [arxiv:2005.12872 app. A.5, fig. 12] | | Need panoptic output from the same model | Add a mask head on the slot embeddings | Reported competitive PQ, with a frozen detector and 25 epochs for the head [arxiv:2005.12872 §4.4, tab. 5] | | Dense crowds, point labels only, and you need head positions | Neighbour-context matching (CLTR) | F1 80.97 on UCF-QNRF against 80.34 for TopoCount [arxiv:2202.13065 tab. 2]; on NWPU test TopoCount, which uses box labels, is ahead (69.1 vs 68.5) [arxiv:2202.13065 tab. 1] | | 3D detection from LiDAR and cameras, with calibration available, and a lower frame rate is acceptable | Query decoder over fused sensor features (DeepInteraction) | 70.8 mAP against 68.9 for TransFusion on nuScenes test; 4.9 FPS against 6.2 on A100 [arxiv:2208.11112 tab. 1, tab. 2] | | Tables whose cell structure is known from markup, and you must recover it from page images | DETR applied without change, trained on canonicalized labels | Content accuracy 0.8138 against 0.1039 for Faster R-CNN on canonical test tables; one run; PMCOA-specific labels [arxiv:2110.00061 tab. 4, §3] | | Video with several objects whose identities must persist, and you can train on frame pairs | Track queries carried across frames (TrackFormer) | 74.1 MOTA against 69.1 for TraDeS on MOT17 private, CrowdHuman-only [arxiv:2101.02702 tab. 1]; with public detections IDF1 trails TMOH (57.6 vs 62.8), and runtime is 7.4 FPS self-measured [arxiv:2101.02702 tab. 1] | | Calibrated multi-camera 3D detection, and you want no depth network and no NMS step | Reference points projected into all views (DETR3D) | 0.374 NDS against 0.373 for FCOS3D with NMS, but mATE 0.860 against 0.785 [arxiv:2110.06922 tab. 1]; the authors' own pseudo-LiDAR baseline reaches 0.160 NDS [arxiv:2110.06922 tab. 4] | | Short video clips, and instance identity should come from the model rather than a linking step, with the clip fitting in memory | One instance sequence per slot (VisTR) | 40.1 AP with ResNet-101 on YouTube-VIS validation [arxiv:2011.14503 tab. 2]; 33.3 AP at T=36 against 29.7 at T=18 in the ablation [arxiv:2011.14503 tab. 1 (a)]. Videos longer than the clip need extra work the released code does not do; see [the card's Paper vs code section](/papers/2011.14503) | | Action segments in long untrimmed video, from frozen pre-extracted features, and you want no proposal or NMS stage in the model | Temporal deformable attention with segment refinement (TadTR) | 30.83 average mAP on HACS validation at 0.1 GFLOPs per video; dense attention in place of the deformable one falls to 13.58 at 5.6 times the FLOPs [arxiv:2106.10271 tab. 2, tab. 4]. The 56.7 THUMOS14 figure is a starred setting [arxiv:2106.10271 tab. 1] | ## Implementation sketch Common core of a set predictor, following DETR [arxiv:2005.12872 §3.1, §3.2, §4]: ```text Train: feats = Encoder(Backbone(x)) slots = Decoder(queries = learned N x d, memory = feats) # parallel, slots attend to each other preds = for each slot: (softmax over K classes + no-object, box) targets padded to N with no-object σ = min-cost one-to-one assignment(targets, preds) # see bipartite-matching loss = class loss on all N slots under σ (no-object down-weighted) + box loss on slots matched to real objects (optionally repeat for every decoder layer's output, shared heads) Infer: preds = model(x) # exactly N outputs, no NMS keep slots by class score; DETR's AP numbers instead relabel no-object slots with their second-best class ``` Where variants differ: how slots interact (DETR uses self-attention), how decoder cross-attention forms its query and key (Conditional DETR concatenates content and spatial parts [arxiv:2108.06152 §3.3]), the classification loss (softmax with a no-object class in DETR [arxiv:2005.12872 §3.2]; focal loss in Conditional DETR [arxiv:2108.06152 §3.5]), the matching cost, and how many slots are used. $N$ and the no-object weight must be chosen together, since most slots are empty [arxiv:2005.12872 §3.1]. Two further differences change the sketch. TSP replaces the decoder with an encoder over a fixed list of candidates and limits matching to allowed pairs [arxiv:2011.10881 §4.1, §4.2]. H-DETR adds a training-only group: it repeats every target $K$ times, runs the same matching and loss on the repeated targets, and masks attention between the two groups. Only the one-to-one slots are returned at inference [arxiv:2207.13080 §3.2]. The paper does not state how the repeated-target loss is normalized; check the card's "Paper vs code" section before reimplementing. Two more changes touch the matching step. CLTR adds a neighbour-context term to the matching cost [arxiv:2202.13065 §3.3]. DeepInteraction starts its queries from heatmap peaks and matches as TransFusion does [arxiv:2208.11112 §4.2]. The table model keeps the DETR loss and adds conflict resolution after inference [arxiv:2110.00061 app. Inference]. Two variants change what a query is. TrackFormer writes each track's previous-frame output embedding back in as a query, and matches those queries by identity rather than by cost [arxiv:2101.02702 §3.2, §3.3]. DETR3D turns each query into a 3D point, projects it into every camera and samples features at the projections, adding the average back to the query [arxiv:2110.06922 eq. 1–4]. In both, the Hungarian loss stays the training signal and the sketch above is otherwise unchanged. VisTR changes the unit of matching. Slot $k$ of each frame is one instance sequence, and the cost and losses are averaged over the clip's frames before the Hungarian step [arxiv:2011.14503 §3.2]. The mask head gains a time axis through 3D convolutions [arxiv:2011.14503 §3.3]. The sketch's `targets` are then instance sequences, not single boxes. TadTR keeps the sketch's matching and loss and changes the attention and the output. Its encoder uses temporal deformable self-attention over the snippet sequence, and each decoder layer's reference point is the refined segment centre from the layer before [arxiv:2106.10271 §III-A]. Its score head (actionness) is regressed from RoI-pooled features and enters the inference score [arxiv:2106.10271 §III-A, §III-B]. ## Contradictions Papers disagree on what limits training speed, and they measure different things, so their results do not settle the question. H-DETR attributes slow training to few positive slots per image [arxiv:2207.13080 §1]. TSP finds that matching effects fade after about 15 epochs and points to cross-attention sparsity [arxiv:2011.10881 §3.1, §3.2]. Conditional DETR points to DETR's cross-attention depending on content embeddings [arxiv:2108.06152 §1]. The three diagnoses are not tested against one another in a shared setup. The paper defines TSP's matching as a permutation with minimum total cost [arxiv:2011.10881 eq. 2]. The released code does not enforce that one-to-one assignment: for each ground-truth object it takes the lowest-cost candidate. The [TSP card](/papers/2011.10881) records this in its "Paper vs code" section, together with the matching-cost weights that the paper leaves out. "No NMS" describes the trained slots, not every released pipeline. DETR's text says NMS is not needed [arxiv:2005.12872 §1]. The TSP-RCNN release runs NMS at test time unless a flag is set; the card lists the threshold. H-DETR's one-to-one slots give the same AP with and without NMS [arxiv:2207.13080 tab. 13]. The table model adds a conflict-resolution step after inference, which works like NMS on children [arxiv:2110.00061 app. Inference]. TrackFormer keeps a duplicate-removal step over tracks at IoU 0.9 [arxiv:2101.02702 §3.2], so a no-NMS property shown for detection does not carry over to every set-prediction pipeline. TadTR states that it needs no NMS [arxiv:2106.10271 §III-B], yet it merges detections in THUMOS14's overlap regions with NMS [arxiv:2106.10271 §IV-B]. Its text gives a square root of the class-by-actionness product as the inference score and cross-entropy as the default matching cost. The card's "Paper vs code" section records that the released code uses the plain product and a focal-style matching term only [arxiv:2106.10271 §III-B, §IV-A]. The CLTR abstract claims the best localisation on NWPU-Crowd, but on the test split TopoCount has the higher F1 [arxiv:2202.13065 tab. 1]. The text calls CLTR competitive there. Use the table figures. The authors' own code also departs from their equations: it averages three neighbours where the paper says k=4, and it appends the neighbour value as a third coordinate [arxiv:2202.13065 §4.1, eq. 3]. The card's "Paper vs code" section lists the differences. Conditional DETR's DETR-R50 baseline at 500 epochs (42.0 AP) agrees with the DETR paper's own result [arxiv:2108.06152 tab. 1] [arxiv:2005.12872 tab. 1]. Within DETR, the text states +7.8 APL and -5.5 APS against Faster R-CNN [arxiv:2005.12872 §4.1], while tab. 1 gives +7.7 and -6.1 against Faster R-CNN-FPN+ [arxiv:2005.12872 tab. 1]. The authors describe VisTR as the most accurate single model on YouTube-VIS. The same table lists MaskProp with ResNet-101 at 42.5 AP, above VisTR's 40.1 [arxiv:2011.14503 tab. 2]. The paper excludes MaskProp because it combines several networks and post-processing [arxiv:2011.14503 §4.3]. That exclusion is the authors' argument; the table alone does not decide the comparison. The paper's clip-length study also relies on matching instances across overlapping clips [arxiv:2011.14503 §4.2], which the released inference code does not implement; the [card's Paper vs code section](/papers/2011.14503) gives the details. ## Papers - [End-to-End Object Detection with Transformers](/papers/2005.12872): detection as direct set prediction with a transformer decoder and a Hungarian matching loss, no anchors or NMS. - [Conditional DETR for Fast Training Convergence](/papers/2108.06152): keeps DETR's set predictor and changes decoder cross-attention to use an explicit spatial query per slot, reaching DETR's accuracy in far fewer epochs. - [Rethinking Transformer-based Set Prediction for Object Detection](/papers/2011.10881): replaces the decoder with an encoder over 700 FCOS or RPN candidates, with restricted matching, and reaches DETR's accuracy in 36 epochs at higher compute. - [DETRs with Hybrid Matching](/papers/2207.13080): adds a training-only one-to-many query group with repeated targets and masked self-attention; inference keeps the one-to-one group. - [An End-to-End Transformer Model for Crowd Localization](/papers/2202.13065): regresses head points with 500 learned queries, and adds a neighbour-context term to the Hungarian matching cost. - [PubTables-1M: Towards comprehensive table extraction from unstructured documents](/papers/2110.00061): a table dataset with canonicalized structure labels; DETR is applied unchanged to table detection and structure recognition. - [DeepInteraction: 3D Object Detection via Modality Interaction](/papers/2208.11112): keeps LiDAR and camera representations separate, and refines 200 queries alternately with each modality in the decoder. - [TrackFormer: Multi-Object Tracking with Transformers](/papers/2101.02702): a DETR-style decoder whose track queries carry each object's embedding into the next frame, so detection and association share one attention step. - [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](/papers/2110.06922): object queries decoded to 3D reference points, projected into six cameras to sample features, trained with a Hungarian set loss and no NMS. - [End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503): VisTR; each slot yields one instance sequence over a 36-frame clip, matched and supervised as a whole, so no tracking heuristics are needed. - [End-to-end Temporal Action Detection with Transformer](/papers/2106.10271): TadTR; action queries predict segments in untrimmed video with temporal deformable attention, segment refinement and an actionness head, matched one-to-one, so no anchors or proposal stage are needed.