--- kind: concept slug: object-queries title: Object queries (learned query slots in transformer decoders) summary: >- Query slots that a transformer decoder turns into outputs in parallel: boxes in DETR, 3D boxes in DETR3D, points in CLTR, class masks in Segmenter, 3D instance masks in Mask3D, depth bins in BinsFormer. Variants add a reference point or a sampled position per slot, carry slots across frames or match whole sequences. Slots meet targets by matching, a fixed class map or a pre-assigned positive. tags: [transformer, object-detection, semantic-segmentation, computer-vision, architecture, depth-estimation, oriented-object-detection, image-enhancement, table-recognition, crowd-localization, 3d-object-detection, multi-object-tracking, video-segmentation, 3d-instance-segmentation, point-clouds, end-to-end-driving, temporal-action-detection] papers: ["2005.12872", "2103.15679", "2105.05633", "2108.06152", "2211.12860", "2409.19648", "2205.14871", "2204.00987", "2110.00061", "2202.13065", "2206.15398", "2208.11112", "2101.02702", "2110.06922", "2011.14503", "2310.06753", "2511.12590", "2504.01989", "2505.17771", "2210.03105", "2512.08186", "2404.00717", "2409.02035", "2207.14024", "2106.10271"] related: [bipartite-matching, set-prediction, label-assignment, semantic-segmentation, vision-transformer, attention-interpretability, explanation-evaluation, object-detection, positional-encoding, monocular-depth-estimation, depthwise-separable-convolution, bev-representation, visual-object-tracking, instance-segmentation, bezier-curve-parameterization, multi-layer-perceptron, bird-eye-view-perception, graph-neural-network, deep-supervision, vision-language-action-model, graph-transformer, scene-graph-generation, collaborative-perception, robotic-grasp-generation, imitation-learning, multimodal-fusion, safety-filter, set-prediction, deformable-convolution] status: draft updated_at: 2026-11-01 --- ## Overview Dense vision models predict one thing per pixel, anchor or patch. Many tasks instead want a small list of outputs: one entry per object, one point per head, or one mask per class. Object queries give a transformer decoder that list directly. You allocate a set of query slots, let them read the image features through attention, and read one output off each slot with a shared head [arxiv:2005.12872 §3.2] [arxiv:2105.05633 §3.2]. Later variants keep this pattern but change where a slot's position comes from, how a slot is assigned to a training target [arxiv:2108.06152 §3.3] [arxiv:2211.12860 §3.3] [arxiv:2202.13065 §3.3], or what a slot predicts [arxiv:2204.00987 §3.2.2] [arxiv:2205.14871 §3.2] [arxiv:2110.00061 §4]. TrackFormer keeps the same decoder but feeds it the previous frame's outputs as well, so the slots persist across frames [arxiv:2101.02702 §3.2]. DETR3D keeps learned slots and gives each one a 3D reference point that is projected into every camera [arxiv:2110.06922 §3.3]. VisTR applies the same kind of decoder to a whole video clip, and groups its slots into one sequence per instance, matched as a unit [arxiv:2011.14503 §3.2]. TadTR uses learned action queries over the snippets of a video, each predicting one action's class, start and end [arxiv:2106.10271 §III-A]. A 2025 survey of lane topology describes the onboard pipeline's centerline decoder the same way: learned query vectors, each mapping one centerline instance [arxiv:2504.01989 §2.3]. Mask3D takes the non-learned route: its slots sit at farthest-point-sampled positions of a 3D scene, and each slot predicts one instance mask [arxiv:2210.03105 §3, §3.1]. Two newer uses change what a slot is for. D3G keeps DETR's object queries and treats them as the nodes of a dependency graph between objects [arxiv:2409.02035 §4.2]. UniV2X sends a roadside unit's query slots to a vehicle, which pairs them with its own slots before planning [arxiv:2404.00717 §3.3]. What the variants share: - **A fixed number of query slots.** Most variants use learned embeddings; Mask3D's positions come from sampling the input points instead [arxiv:2210.03105 §3.1]. N=100 in DETR [arxiv:2005.12872 app. A.4], N=500 in CLTR [arxiv:2202.13065 §4.1], 15 queries for table detection and 125 for table structure and functional analysis in PubTables-1M [arxiv:2110.00061 app. Training Details], N=300 in Conditional DETR [arxiv:2108.06152 §3.5], 300 or 900 in Co-DETR depending on the base detector [arxiv:2211.12860 §4.1], N=300 in OrientedFormer [arxiv:2409.19648 tab. 9], 64 bin queries plus one scene query in BinsFormer [arxiv:2204.00987 §4.2.4], K (the number of semantic classes) in Segmenter [arxiv:2105.05633 §3.2], 10 global queries in IAT [arxiv:2205.14871 §3.2], and n·T = 360 in VisTR, which is 10 instances in each of 36 frames [arxiv:2011.14503 §4.1]. Mask3D uses 100 in all of its paper experiments [arxiv:2210.03105 app. §1]. D3G does not state N; its released code uses 100 (see the [D3G card](/papers/2409.02035), Hyperparameters) [arxiv:2409.02035 §4.1]. UniV2X's roadside queries are the output of a tracker, one per tracked agent, so their count varies with the scene and is not stated as a number in the paper [arxiv:2404.00717 §3.2]. PolarFormer's learned queries are described without a count in the paper [arxiv:2206.15398 §3.4]. DeepInteraction uses 200 slots, but they are started from heatmap peaks of each scene, not from a learned table alone [arxiv:2208.11112 §4.2]. TrackFormer uses N_object=500 learned object queries on MOT17, plus one track query per active track [arxiv:2101.02702 §4.2] [arxiv:2101.02702 §3.2], so its total slot count changes from frame to frame. DETR3D uses 900 [arxiv:2110.06922 tab. 6]. TadTR uses 40 action queries on THUMOS14, 30 on HACS and 10 on ActivityNet [arxiv:2106.10271 §IV-C]. TopoMLP uses 300 lane queries, each predicting four Bezier control points [arxiv:2310.06753 §4.2, tab. 3]. TopoFG uses 200 lanes with 11 queries each, 2200 slots in total (our arithmetic) [arxiv:2511.12590 §4.2]. TopoPoint uses 200 point queries and 300 lane queries [arxiv:2505.17771 §4.2, tab. 5]. - **Slots interact.** Self-attention runs over the slots as well as between slots and image features, so each output is computed with the others in view [arxiv:2005.12872 §3.2] [arxiv:2105.05633 §3.2] [arxiv:2409.19648 §3.4] [arxiv:2206.15398 §3.4]. DETR-style decoders for tables and crowds keep this [arxiv:2110.00061 §4] [arxiv:2202.13065 §3.2]. TrackFormer runs self-attention over all of its track and object queries together [arxiv:2101.02702 §3.1] [arxiv:2101.02702 app. A.3], and DETR3D applies multi-head self-attention across its queries after each sampling step [arxiv:2110.06922 §3.3]. TopoPoint merges its point and lane queries into one self-attention [arxiv:2505.17771 §3.4, eq. 3]. TadTR's decoder runs self-attention across its action queries before each cross-attention step [arxiv:2106.10271 §III-A]. Mask3D's queries also run self-attention between queries [arxiv:2210.03105 §3]. Two variants are exceptions. Co-DETR's auxiliary query groups are isolated from one another [arxiv:2211.12860 §1]. IAT's queries have no query-to-query step at all [arxiv:2205.14871 §3.2]. In D3G, the graph layers run attention over pairs of slots after the decoder, so each pair's relation is computed with the rest of the graph in view [arxiv:2409.02035 §4.2]. - **Parallel decoding.** All slots of a group are produced in one pass, not one after another [arxiv:2005.12872 §3.2] [arxiv:2204.00987 §3.2.2]. TrackFormer decodes every query of a frame in one pass [arxiv:2101.02702 §3.1]; DETR3D decodes all its queries in each of its six layers [arxiv:2110.06922 §3.3]; VisTR decodes the queries of a whole clip in one pass [arxiv:2011.14503 §3.1]. - **Shared heads applied to each slot.** One head per output type, applied to every slot independently [arxiv:2005.12872 §3.2] [arxiv:2105.05633 eq. 4] [arxiv:2204.00987 §3.2.2] [arxiv:2202.13065 §3] [arxiv:2210.03105 §3] [arxiv:2409.02035 §4.1] [arxiv:2106.10271 §III-A]. The variants differ in two places: how a slot's input enters the decoder (a positional encoding, a spatial query, a token, a box-derived Gaussian, a reference point refined in every layer, a 1-D action centre refined in every layer [arxiv:2106.10271 §III-A], separate learned reference points for endpoint and lane queries, a heatmap peak, a learned 3-D anchor per control point, a BEV-mask prior plus a point-index embedding per lane point, a global query, the previous frame's output for the same object, or a slot's output in each frame of a clip; Mask3D starts each slot from a sampled scene position with zero features; in UniV2X, a roadside tracker's query moved into the vehicle's frame; in D3G, a node whose pair scores come from learned edge maps), and how a slot is tied to a training target (learned assignment by matching, identity matching for carried-over slots, a fixed one-to-one map to a class, a pre-assigned positive from a one-to-many head, a matched sequence of per-frame slots as in VisTR, a Hungarian match on mask costs as in Mask3D, or no per-slot tie at all because the loss acts on a dense output map, as in IAT and BinsFormer). Two papers add a third case. UniV2X pairs vehicle and roadside slots with Hungarian matching at fusion time, not as a training target [arxiv:2404.00717 §3.3]. D3G matches slots to objects and then applies a BCE loss to the relation matrix of the matched slots [arxiv:2409.02035 §4.3]. ## Variants ### Detection slots with bipartite matching (DETR) DETR feeds $N=100$ learned positional embeddings, the object queries, into a 6-layer transformer decoder of width 256 [arxiv:2005.12872 §3.2, §4.1, app. A.4]. Card: [End-to-End Object Detection with Transformers](/papers/2005.12872). - **How a slot enters.** The decoder input starts at zero. The object queries are added to the queries and keys (never the values) of every attention layer, and are shared across layers [arxiv:2005.12872 §3.2, eq. 7, app. A.1, app. A.3]. Each decoder layer runs self-attention over the $N$ slots and cross-attention into the encoder output [arxiv:2005.12872 §3.2]. - **What a slot outputs.** A 3-layer MLP gives a normalized box (center, height, width) and a linear layer with softmax gives a class among $K$ classes plus "no object" $\varnothing$ [arxiv:2005.12872 §3.2]. - **How a slot gets its target.** No slot is tied to a class or region in advance. Per image, a Hungarian matching pairs each ground-truth object with one slot; unmatched slots are trained to predict $\varnothing$ [arxiv:2005.12872 §3.1, eq. 1, eq. 2]. Details of the loss are on the [bipartite matching](/concepts/bipartite-matching) page. - **Deep supervision.** The shared heads and the matching loss are applied after every decoder layer [arxiv:2005.12872 §3.2]. Reported behaviour: - AP rises after every decoder layer, by 8.2 from the first to the last. NMS helps the first layer's output but slightly lowers AP at the last layers, where it removes true positives [arxiv:2005.12872 §4.2, fig. 4]. - Where positional information is injected matters. With learned queries and sine spatial encodings both added at every attention layer, the R50 model gets 40.6 AP; adding both only at the decoder input gives 39.2; with no spatial encodings and queries only at the input, 32.8 [arxiv:2005.12872 tab. 3]. - R50 DETR reaches 42.0 AP on COCO val, equal to a tuned Faster R-CNN-FPN, with 61.1 vs 53.4 AP on large and 20.5 vs 26.6 on small objects [arxiv:2005.12872 tab. 1]. - $N$ is a hard cap. In a synthetic image with one class tiled on a 10x10 grid, DETR finds all instances up to about 50 and about 30 on average when all 100 are present [arxiv:2005.12872 app. A.5, fig. 12]. - The slot outputs also drive masks: for panoptic segmentation, each decoder output embedding attends over the encoder output to give per-object heatmaps that an FPN-like head upsamples to masks [arxiv:2005.12872 §4.4, fig. 8]. ### Conditional spatial queries (Conditional DETR) Conditional DETR keeps DETR's learned object queries but gives each slot an explicit "where to look" query. The decoder's cross-attention then compares content with content and position with position, instead of mixing them. Card: [Conditional DETR for Fast Training Convergence](/papers/2108.06152). - **How a slot enters.** Query and key are the concatenation of a content part and a spatial part, so the dot product splits into a content term plus a spatial term [arxiv:2108.06152 §3.3, eq. 3]. The spatial query is the sinusoidal embedding of a reference point, scaled element-wise by a vector predicted from the previous decoder layer's output [arxiv:2108.06152 §3.3, eq. 5, eq. 6]. - **Where the reference point comes from.** It is predicted from the object query with a small FFN, or set as a learnable per-query parameter [arxiv:2108.06152 §3.5, §4.3]. The object query still feeds the self-attention and the first decoder layer [arxiv:2108.06152 §3.5]. - **How a slot gets its target.** Unchanged: Hungarian matching and a DETR-style loss, here with focal classification and 300 queries [arxiv:2108.06152 §3.5]. Reported behaviour (R50 unless noted, COCO val, author-reported): - At 50 epochs it reaches 40.9 AP against 34.9 for DETR; DETR reaches 42.0 only at 500 epochs [arxiv:2108.06152 tab. 1]. The 6.67x (R50, R101) and 10x (DC5) speed-ups are the epochs DETR needs to match its own 500-epoch result, divided by the epochs Conditional DETR needs [arxiv:2108.06152 §4.2]. - Part of this gain is not the attention change. Focal loss and center-offset regression alone give 36.9 AP at 50 epochs; the conditional spatial query adds the remaining 4.0 [arxiv:2108.06152 tab. 4]. - The form of the spatial query matters. Predicting the scale from the self-attention output instead of the layer's decoder embedding gives 40.2 AP, against 40.9 [arxiv:2108.06152 tab. 3]. - The reference point matters too. Fixing it at (0,0), as DETR does, gives 36.8 AP at 50 epochs; learned per-query points give 40.7; predicting it from the object query gives 40.9 [arxiv:2108.06152 §4.3]. - Encoder cost is unchanged. Global single-scale encoder attention remains, and the DC5 variants cost 195 to 262 GFLOPs [arxiv:2108.06152 tab. 1, §4.2]. ### Class embeddings as mask queries (Segmenter mask transformer) Segmenter's mask transformer uses $K$ learnable class embeddings, one per semantic class, randomly initialised [arxiv:2105.05633 §3.2, §4.2]. The authors cite DETR's object queries as an inspiration [arxiv:2105.05633 §3.2]. Card: [Segmenter: Transformer for Semantic Segmentation](/papers/2105.05633). - **How a slot enters.** The class embeddings are tokens: they are concatenated with the $N$ patch encodings from the ViT encoder, and a 2-layer transformer encoder (self-attention only) processes the joint sequence [arxiv:2105.05633 §3.2, §4.3]. - **What a slot outputs.** A mask: the scalar product between each L2-normalised output patch embedding and each output class embedding gives an $N \times K$ score map, reshaped to the patch grid and bilinearly upsampled to pixels [arxiv:2105.05633 eq. 4, §3.2]. - **How a slot gets its target.** Fixed: slot $k$ is class $k$. Training is plain per-pixel cross-entropy, with no matching [arxiv:2105.05633 §3, §4.2]. - The authors suggest replacing class embeddings with object embeddings to handle panoptic segmentation, but do not evaluate it [arxiv:2105.05633 §3.2]. Reported behaviour, ADE20K val single-scale mIoU, linear decoder vs mask transformer [arxiv:2105.05633 tab. 4]: | Backbone / patch | Linear | Mask | |---|---|---| | ViT-B/32 | 43.07 | 44.19 | | DeiT-B/16 | 47.10 | 48.70 | | ViT-B/16 | 48.06 | 48.48 | | ViT-L/16 | 50.71 | 51.30 | The gain is not uniform across object sizes: Seg-B-Mask/16 is below the linear Seg-B/16 on large objects (52.66 vs 53.28) [arxiv:2105.05633 tab. 4]. Throughput barely changes (DeiT-B/16: 77 vs 76 images/s; ViT-L/16: 33 vs 31) [arxiv:2105.05633 tab. 6]. On Cityscapes the largest mask model uses a 1-layer mask transformer because of memory [arxiv:2105.05633 §4.4]. ### Training-only query groups from one-to-many heads (Co-DETR) Co-DETR keeps a DETR-family detector unchanged at inference, but adds auxiliary detection heads and query groups during training. The heads use classic one-to-many label assignment, which gives the encoder dense supervision; their positives become extra query groups for the decoder. Card: [DETRs with Collaborative Hybrid Assignments Training](/papers/2211.12860). - **How a slot enters.** A group's queries are built per image from the positive boxes of one auxiliary head: a linear map of a box positional encoding plus a linear map of the pyramid feature at each positive [arxiv:2211.12860 §3.3, eq. 4]. These queries are not fixed learned embeddings; they change with the image. - **Groups.** Each auxiliary head forms its own group. Groups pass through the same decoder layers with shared weights, and each group is isolated from the others [arxiv:2211.12860 §1, §3.3]. - **How a slot gets its target.** Every query in an auxiliary group is a positive with a pre-assigned target, so no matching is run for it [arxiv:2211.12860 §3.3]. The main branch still uses one-to-one Hungarian matching [arxiv:2211.12860 §3.3, eq. 6]. - **Encoder supervision.** Each auxiliary head is also trained on the encoder pyramid, with positives and negatives from its own assigner. This is the dense signal the encoder receives [arxiv:2211.12860 §3.2, eq. 1, eq. 2]. - **Inference.** Only the original DETR branch runs. There are no extra parameters or compute at test time, and the base method uses no NMS [arxiv:2211.12860 §1]. Reported behaviour (Deformable-DETR R50 unless noted, COCO val, author-reported): - At 12 epochs, AP is 37.1 with no auxiliary head, 42.3 with one (ATSS) and 42.9 with two (ATSS and Faster R-CNN). At 36 epochs the same comparison is 43.3, 46.8 and 46.5, so the second head does not help there [arxiv:2211.12860 tab. 2]. - Both parts matter. At 12 / 36 epochs, the auxiliary head alone gives 41.6 / 46.2, the customized positive queries alone give 40.5 / 45.3, and both together give 42.3 / 46.8, against 37.1 / 43.3 for the baseline [arxiv:2211.12860 tab. 9]. - Training cost rises with the number of heads. For Deformable-DETR++ R50 at 12 epochs, GPU hours go 70 (no head), 86 (one head), 120 (two heads), and memory goes from 12808 MB to 14387 MB at two heads [arxiv:2211.12860 tab. 7]. - Extra training with a different budget: a 36-epoch Co-Deformable-DETR run (K=1, 288 GPU hours, 46.8 AP) is above the plain baseline at 150 epochs (1000 GPU hours, 45.9 AP) [arxiv:2211.12860 tab. 10]. - More heads is not better. Six different heads score 48.9 AP, below ATSS plus Faster R-CNN (49.5); the authors attribute this to conflicting assignments [arxiv:2211.12860 tab. 7, §4.4]. - The strongest DINO-based rows also use QFL and NMS, so their gains over DINO mix the training scheme with post-processing [arxiv:2211.12860 §4.3, tab. 4]. ### Table objects from an unchanged DETR (PubTables-1M) PubTables-1M applies DETR to three table tasks without architectural changes: table detection, table structure recognition and functional analysis. The contribution is the dataset; the detector is a reference point. Card: [PubTables-1M: Towards comprehensive table extraction from unstructured documents](/papers/2110.00061). - **How a slot enters.** Learned object queries, as in DETR [arxiv:2110.00061 §4]. The appendix sets 15 queries for table detection and 125 for structure and functional analysis [arxiv:2110.00061 app. Training Details]. - **What a slot outputs.** For structure, one detector predicts six object types: the whole table, its columns, its rows, its column-header cells, its projected row headers and its spanning cells. The grid cell is the intersection of a column and a row and is not predicted as its own type [arxiv:2110.00061 §4]. - **How a slot gets its target.** DETR's one-to-one matching. The paper does not state the matching weights [arxiv:2110.00061 app. Training Details]. - **Training targets are dilated.** Each shared boundary between neighbouring rows, and between neighbouring columns, moves to the midpoint of the gap, so the boxes fill the gaps. Scoring then tightens the boxes to the text they contain [arxiv:2110.00061 §4, app. Scoring]. This is a change to the targets, not to the slots. Reported behaviour (ResNet-18 backbone, PubTables-1M test split, author-reported): - Table detection: AP 0.966 for DETR against 0.825 for Faster R-CNN [arxiv:2110.00061 tab. 3]. For table structure and functional analysis the same comparison is 0.912 against 0.722 [arxiv:2110.00061 tab. 3]. - Structure recognition on canonical test data, all tables: content accuracy 0.8138 and GriTS_Top 0.9845 for DETR, against 0.1039 and 0.8616 for Faster R-CNN [arxiv:2110.00061 tab. 4]. - The DETR and Faster R-CNN runs share the paper's own data, which the paper canonicalises. The gap therefore does not isolate the detector from the data changes [arxiv:2110.00061 §3, §5]. ### Head points with a kNN-context matching cost (CLTR) CLTR predicts each head as a point, not a box, so a slot outputs a point and a confidence. Training pairs slots with heads in a way that also looks at each head's neighbours. Card: [An End-to-End Transformer Model for Crowd Localization](/papers/2202.13065). - **How a slot enters.** The 500 learned queries are the decoder's object queries [arxiv:2202.13065 §4.1]. The cross-attention uses the conditional form, with the learned embedding in the query, following Conditional DETR [arxiv:2202.13065 §3.2]. - **What a slot outputs.** A point, from a regression head, and a confidence, from a classification head [arxiv:2202.13065 §3]. - **How a slot gets its target.** One-to-one Hungarian matching. The cost adds the L1 distance between points and the L1 distance between kNN-context values, then subtracts the predicted confidence. The kNN value of each prediction is an output of the network [arxiv:2202.13065 §3.3, eq. 2, eq. 3]. The loss is focal over all slots plus L1 on the matched points, with weight λ=2.5 [arxiv:2202.13065 §3.4, eq. 5]. - **No boxes, density maps or NMS.** Inference keeps slots with confidence at least 0.35 and counts them [arxiv:2202.13065 §1, §4.1]. Reported behaviour (ResNet-50 at stride 32, point labels only, author-reported): - UCF-QNRF localisation F1, averaged over σ from 1 to 100 px: 80.97 for CLTR against 80.34 for TopoCount [arxiv:2202.13065 tab. 2]. - ShanghaiTech Part A at σ=4: F1 43.2 for CLTR against 41.1 for TopoCount [arxiv:2202.13065 tab. 3]. - NWPU-Crowd test: F1 68.5 for CLTR against 69.1 for TopoCount, which uses box labels in training [arxiv:2202.13065 tab. 1]. - Slot count on UCF-QNRF: F1 79.89 with 300 queries, 80.97 with 500, 80.34 with 700 [arxiv:2202.13065 tab. 7]. Each setting is one run [arxiv:2202.13065 tab. 7]. - Matching cost on UCF-QNRF: F1 80.02 with plain L1 plus confidence, 80.97 with the kNN term. One run [arxiv:2202.13065 tab. 8]. ### Carried-over track slots (TrackFormer) TrackFormer tracks objects with a DETR-style decoder that runs once per frame. Each frame it gets two kinds of slot: one track query per active track, which is the object's output from the previous frame, and a fixed set of learned object queries that can start new tracks [arxiv:2101.02702 §3.2]. Card: [TrackFormer: Multi-Object Tracking with Transformers](/papers/2101.02702). - **How a slot enters.** A track query is the previous frame's output embedding for that object, taken from the last decoder layer before the prediction heads [arxiv:2101.02702 §3.2] [arxiv:2101.02702 app. A.3]. The object queries are learned embeddings, 500 for the MOT17 model [arxiv:2101.02702 §3.2] [arxiv:2101.02702 §4.2]. For the deformable-attention variant, a track query's reference point is the centre of its previous box [arxiv:2101.02702 §4.2]. - **What a slot outputs.** A box and a class from small MLP heads, as in DETR [arxiv:2101.02702 §3.1]. A track starts when an object query scores above 0.4 and ends when its score falls below 0.4 [arxiv:2101.02702 §3.2] [arxiv:2101.02702 app. A.4]. Re-identification from an inactive pool is described in the paper [arxiv:2101.02702 app. A.4]; see the card's Paper vs code for its default setting in the released code. - **How a slot gets its target.** Track-query outputs are matched to the ground-truth object with the same identity. Tracks that have disappeared are trained towards background. New objects are matched to object queries with the cost in eq. 2 and eq. 3 [arxiv:2101.02702 §3.3] [arxiv:2101.02702 eq. 2] [arxiv:2101.02702 eq. 3]. The loss sums over all predictions, object and track [arxiv:2101.02702 §3.3] [arxiv:2101.02702 eq. 4]. - **Frame pairs and augmentations.** Training uses a previous frame sampled from a window around frame t. The paper drops each track query with probability 0.4 (false negatives) and adds false-positive track queries with probability 0.1 [arxiv:2101.02702 §3.3] [arxiv:2101.02702 app. A.1]. The window size is not stated. The released code's augmentation and gradient choices differ from the text; see the [TrackFormer card](/papers/2101.02702), Paper vs code. Reported behaviour (author-reported, MOT17 unless noted): - MOT17 test, private detections, CrowdHuman-only training data: 74.1 MOTA and 68.0 IDF1, against 69.1 and 63.9 for TraDeS [arxiv:2101.02702 tab. 1]. - MOT17 test, public detections with the IoU filter: 62.3 MOTA and 57.6 IDF1, against 62.1 and 62.8 for TMOH, which is higher on IDF1 [arxiv:2101.02702 tab. 1]. The paper treats the public-detection filter as a choice that changes the numbers [arxiv:2101.02702 app. B.1]. - MOTS20 test: 54.9 sMOTSA, against 40.6 for Track R-CNN [arxiv:2101.02702 tab. 2]. - Ablation on the MOT17 training data, 50/50 frame split. Each row removes one component on top of the rows above it. The full model reaches 71.3 MOTA and 73.4 IDF1; the last row, without track queries and with greedy centre matching, reaches 61.0 MOTA and 45.1 IDF1 [arxiv:2101.02702 tab. 3]. Read the row differences as cumulative, not as the effect of one component alone. ### 3D reference points projected into camera images (DETR3D) DETR3D predicts 3D boxes from six cameras with no depth network and no NMS. It keeps learned object queries, but each query first gives a 3D reference point. That point is projected into every camera image, and the image features found there update the query [arxiv:2110.06922 §3.3]. Card: [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](/papers/2110.06922). - **How a slot enters.** 900 learned queries of width 256 [arxiv:2110.06922 tab. 6] [arxiv:2110.06922 §4.1]. Each decoder layer decodes a 3D reference point from the query [arxiv:2110.06922 eq. 1]. The point is appended with 1 and projected with each camera's 3×4 matrix [arxiv:2110.06922 eq. 2]. Bilinear samples are taken from each of the four FPN levels and each camera [arxiv:2110.06922 eq. 3]. The samples that fall inside the image are averaged and added back to the query as a residual [arxiv:2110.06922 eq. 4] [§3.3]. - **What a slot outputs.** A box and a class, predicted from the query after every layer. At inference only the last layer is used and no NMS is applied [arxiv:2110.06922 eq. 5] [arxiv:2110.06922 §3.3] [arxiv:2110.06922 §4.1]. - **How a slot gets its target.** Hungarian matching on a cost that combines class probability and an L1 box term. A focal loss covers all predictions, and an L1 loss covers only the matched boxes. The loss is applied to every decoder layer [arxiv:2110.06922 §3.4] [arxiv:2110.06922 §3.3]. The paper gives no weights for these terms. - **Calibration is required.** The method assumes the camera projection matrices are known [arxiv:2110.06922 §3.1]. Reported behaviour (nuScenes, author-reported): - Validation, all cameras: 0.374 NDS and 0.303 mAP for the plain model, against 0.373 NDS and 0.299 mAP for FCOS3D, which uses NMS [arxiv:2110.06922 tab. 1]. The CBGS variant reaches 0.434 NDS and 0.349 mAP [arxiv:2110.06922 tab. 1]. - Translation error is worse than FCOS3D: mATE 0.860 against 0.785. The paper suspects FCOS3D's direct depth prediction supervises translation more strongly [arxiv:2110.06922 §4.2] [arxiv:2110.06922 tab. 1]. - Test set: 0.479 NDS and 0.412 mAP, a leaderboard result as of 13 October 2021 [arxiv:2110.06922 tab. 2] [arxiv:2110.06922 §4.2]. - Query count: 30 queries give 0.201 mAP, 100 give 0.313 and 900 give 0.346. The mAP peak is at 600 (0.347) [arxiv:2110.06922 tab. 6]. - Layers: NDS rises from 0.380 after the first decoder layer to 0.425 after the sixth [arxiv:2110.06922 tab. 5]. - Backbone: ResNet-50 gives 0.373 NDS and ResNet-101 gives 0.425 [arxiv:2110.06922 tab. 7]. - Against the authors' pseudo-LiDAR baseline, a supervised depth network followed by CenterPoint, DETR3D reaches 0.374 NDS against 0.160 [arxiv:2110.06922 tab. 4] [arxiv:2110.06922 §4.4]. ### Bezier-curve lane queries with pairwise topology heads (TopoMLP) TopoMLP detects lanes and traffic elements from several cameras, then predicts which lanes connect to each other and which lanes connect to which traffic element. Each lane query is a curve, not a box: it outputs a few Bezier control points, and the curve is evaluated from them. Card: [TopoMLP: A Simple yet Strong Pipeline for Driving Topology Reasoning](/papers/2310.06753). - **How a slot enters.** Each lane query starts from a learnable 3-D anchor point per control point. The anchors go through a linear layer and a transformer decoder [arxiv:2310.06753 §3.1, eq. 1]. The decoded features then give offsets to the anchors [arxiv:2310.06753 §3.1]. - **What a slot outputs.** Control points, one set per lane, with M = 4 [arxiv:2310.06753 §4.2, tab. 3]. A second MLP predicts the class score of the lane [arxiv:2310.06753 §3.1]. - **Topology from the slots.** The lane features are paired with each other (N_L x N_L) and with the traffic queries (N_L x N_T) by repeating and concatenating them. An MLP scores each pair [arxiv:2310.06753 §3.3, eq. 3] [arxiv:2310.06753 §3.4, eq. 4]. The camera view matrix is added to the lane features before the lane-traffic pairing [arxiv:2310.06753 §3.4]. - **How a slot gets its target.** Bipartite matching assigns slots to lanes and traffic elements for the detection losses. The topology losses reuse that same assignment, so an unmatched query gets no topology supervision [arxiv:2310.06753 §3.5]. - **Design claim.** The authors argue that the topology step is simple once detection is good. In their motivating test, giving the topology heads ground-truth detections raised TOP_ll only from 9.5 to 10.0 and TOP_lt from 28.3 to 30.9 on Swin-B [arxiv:2310.06753 §1, fig. 1]. Reported behaviour (OpenLane-V2, author-reported, 24 epochs unless stated): - Subset A, ResNet-50: OLS 38.2 against 35.6 for TopoNet; TOP_ll 7.2 against 4.1 [arxiv:2310.06753 tab. 1]. With extra YOLOv8 proposals, OLS is 41.2 [arxiv:2310.06753 tab. 1]. - Subset B, ResNet-50: OLS 38.7 against 33.2 for TopoNet [arxiv:2310.06753 tab. 2]. - Lane queries: 200, 300 and 500 give OLS 36.9, 38.2 and 38.0 [arxiv:2310.06753 tab. 3]. - Metric: raising confidences of the same TopoMLP predictions moves OLS from 38.2 to 42.2. The adjusted metric TOP† gives 36.3 for the original and 34.5 for the raised-confidence version [arxiv:2310.06753 tab. 4]. The paper's own ablations use one training run each, with no variance reported [arxiv:2310.06753 tab. 3]. - The paper describes 300 lanes and no extra training branch [arxiv:2310.06753 §3.1, §4.2]. The official code adds a one-to-many lane branch in training, and its topology heads use only the one-to-one lanes. See the card's Paper vs code before reproducing numbers. ### Fine-grained per-point queries for lane topology (TopoFG) TopoFG gives each lane 11 ordered queries, one per point, where TopoMLP and TopoLogic use one query per lane. Only the first and last query of each lane feed a pairwise connectivity head. Card: [Fine-Grained Representation for Lane Topology Reasoning](/papers/2511.12590). - **How a slot enters.** A query is the sum of a lane-level spatial prior and a point-level sequential prior [arxiv:2511.12590 §3.3, eq. 4]. The spatial prior is a learned lane query refined from a predicted BEV lane mask, which is turned into weights and encoded over the BEV grid [arxiv:2511.12590 §3.2, eq. 1, eq. 2]. The sequential prior is a learned point query plus an encoding of the point's index, from 1 to 11 [arxiv:2511.12590 §3.2, eq. 3]. - **Where the sampling points come from.** Deformable cross-attention reads the BEV features at reference points taken from the predicted lane mask, not from a learned table alone [arxiv:2511.12590 §3.3, eq. 5]. - **Slots interact along and across lanes.** Each of the six decoder layers attends across lanes at the same point index, then along the points of each lane, then reads the BEV features [arxiv:2511.12590 §3.3]. - **What a slot outputs.** One key point of the lane per query, following TopoLogic [arxiv:2511.12590 §3.5]. Lane connections come from the first and last query of each lane, through a shared MLP over ordered pairs [arxiv:2511.12590 §3.4, eq. 6, eq. 7]. At inference the learned similarity is summed with a geometric term built from the distance between boundary points [arxiv:2511.12590 §3.4]. - **How a slot gets its target.** One-to-one Hungarian matching for the lanes [arxiv:2511.12590 §3.4]. For the topology, the paper adds G = 5 denoising groups, each a noisy copy of every ground-truth lane. Their adjacency is the fixed ground-truth matrix, so the labels do not shift with the matching step. The groups are used in training only [arxiv:2511.12590 §3.4, §4.2]. Reported behaviour (OpenLane-V2 subset_A, ResNet-50, author-reported; the paper gives no seeds or variance): - OLS 48.0 against 44.1 for TopoLogic, and TOP_ll 30.8 against 23.9 [arxiv:2511.12590 tab. 1]. On subset_B, OLS is 45.4 against 42.3 [arxiv:2511.12590 tab. 1]. - Ablation, cumulative: baseline 44.1; plus the priors 45.4; plus the decoder 45.8; plus the boundary-point topology 48.0 [arxiv:2511.12590 tab. 3]. - Denoising groups alone add 1.5 OLS over the same model without them (47.3 against 45.8). Boundary-point topology alone adds 0.8 (46.6 against 45.8) [arxiv:2511.12590 tab. 6]. - The text says the global spatial prior improves all metrics. In Table 4 the full prior gives lower TOP_ll (26.1 against 26.7) and TOP_lt (26.6 against 27.3) than the sequential prior alone [arxiv:2511.12590 §4.4, tab. 4]. - The released code differs from the paper in several places, including the distance metric and the inference geometry. See the card's Paper vs code before reproducing numbers. ### Endpoint queries merged with lane queries (TopoPoint) TopoPoint adds one query per lane endpoint, next to the lane queries. The two sets share one self-attention and a point-lane graph, so detected endpoints can pull connected lane ends toward one shared point. Card: [TopoPoint: Enhance Topology Reasoning via Endpoint Detection in Autonomous Driving](/papers/2505.17771). - **How a slot enters.** The decoder has 200 point queries and 300 lane queries, each set with its own learnable reference point [arxiv:2505.17771 §4.2, §3.4]. Both sets go into one self-attention. The paper adds geometric terms, built from the previous layer's predictions, to the attention logits [arxiv:2505.17771 §3.4, eq. 3, eq. 4-8]. - **What a slot outputs.** A point query gives one 3D endpoint and a score. A lane query gives 11 3D points and a score [arxiv:2505.17771 §3.4, eq. 17]. Pairwise MLP embeddings give the point-lane, lane-lane and lane-traffic topology matrices [arxiv:2505.17771 §3.5, eq. 18-20]. - **Graph updates.** A point-lane graph convolution passes messages both ways between points and lanes. It runs before and after the lane-lane and lane-traffic branches [arxiv:2505.17771 §3.4, eq. 11-16]. - **How a slot gets its target.** The point and lane losses are focal classification plus L1 regression, and the three topology losses are focal [arxiv:2505.17771 §3.6, eq. 21]. The paper does not state the matching costs [arxiv:2505.17771 §3.6]. - **Inference step.** Points and lanes scoring above 0.3 are kept. Each kept point is averaged with the lane endpoints within 1.5 m, and each matched lane takes that average as its endpoint [arxiv:2505.17771 §4.2, alg. 1]. The released code does not run this as written; see the [TopoPoint card](/papers/2505.17771), Paper vs code. Reported behaviour (OpenLane-V2 subset_A and subset_B, ResNet-50, 24 epochs, author-reported): - subset_A: OLS 48.8 and DET_p 52.6 [arxiv:2505.17771 tab. 1]. The TopoLogic row gives OLS 44.1, copied from an earlier paper, and DET_p 45.2, computed by the authors [arxiv:2505.17771 tab. 1, §4.1]. - subset_B: OLS 49.2 against 42.3 for TopoLogic, and DET_p 45.1 against 39.2 [arxiv:2505.17771 tab. 1]. - Ablation, from the authors' re-run of the TopoLogic code at OLS 43.4 and DET_p 44.5: native-scale front view adds 2.6 OLS, merged self-attention 1.6, the point-lane graph 0.7 and the inference step 0.5 [arxiv:2505.17771 tab. 2]. The first step is an input-resolution change, not a query change. - Query counts: 200 point and 300 lane queries give the best OLS, 48.8. 300 point queries with 300 lanes give 48.3, and 100 point queries with 200 lanes give 46.5 [arxiv:2505.17771 tab. 5]. ### Relevance of a slot to image regions (post-hoc explanation) This variant does not change the slots. It explains a trained DETR-style decoder: for each detected object, which image regions its query relied on. Card: [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](/papers/2103.15679). - **How it works.** Each attention map is weighted by the gradient of the target score, its positive part is kept, and the heads are averaged. Relevance is then propagated layer by layer. In the DETR decoder, the query self-attention rule runs first and the query-to-image rule second; the row of the detected object gives one score per image region [arxiv:2103.15679 §3, §3.2, eq. 5, eq. 10]. - **Target.** The paper uses the model's output for the class of the detected object, but does not say whether that output is a logit or a probability [arxiv:2103.15679 §3]. - **No training.** DETR is used as trained, and the method adds no parameters [arxiv:2103.15679 §5]. - **Segmentation from relevance.** Queries whose class probability is above 50% are kept. For each, the relevance row is thresholded with Otsu's method and upsampled to a mask. The procedure runs once per kept query [arxiv:2103.15679 app. C, alg. 1]. Reported behaviour (author-reported, COCO val, weakly supervised DETR segmentation): - AP 13.1 against 7.2 for the best baseline, Transformer attribution. Rollout and raw attention reach 0.1 and 5.6 [arxiv:2103.15679 tab. 1]. - Both the accumulation over layers and the normalisation step matter for this result. Without the accumulation, AP falls to 0.1; without normalisation, to 11.7 [arxiv:2103.15679 tab. 3]. - The paper's LRP variant is close to the raw-attention version on LXMERT, so the paper does not need LRP [arxiv:2103.15679 app. E, tab. 5]. ### Oriented-box positional queries (OrientedFormer) OrientedFormer is a DETR-style decoder for rotated boxes in remote-sensing images. It uses no encoder and 300 queries. Each query carries a content part and a positional part, which is the oriented box (x, y, w, h, θ) and is updated from the content in every layer [arxiv:2409.19648 §3.1, §3.2] [arxiv:2409.19648 alg. 1]. Card: [OrientedFormer](/papers/2409.19648). - **How a slot enters.** The box becomes a 2-D Gaussian, and its sinusoidal encoding is replaced by the expectation of sine and cosine under that Gaussian [arxiv:2409.19648 §3.3, eq. 2, eq. 6]. Cross-attention samples features at points offset from the box centre and rotated by θ [arxiv:2409.19648 §3.5, eq. 13, eq. 14]. - **Self-attention bias.** The squared 2-Wasserstein distance between two slots' Gaussians is turned into an additive bias on the attention logits. The bias has no parameters [arxiv:2409.19648 §3.4, eq. 8, eq. 9]. - **How a slot gets its target.** One-to-one Hungarian matching with a focal, an L1 and a rotated-IoU loss [arxiv:2409.19648 §3.6, eq. 19]. The paper claims no hand-designed post-processing [arxiv:2409.19648 §1]. The official code selects its initial queries with an NMS step, which the card's Paper vs code section describes. Reported behaviour (ResNet-50, DIOR-R AP50, author-reported): - Positional encoding: none 66.85, learnable 64.27, deformable 65.97, DAB 66.03, Gaussian 67.28. The Gaussian encoding is 0.4 AP50 above no encoding [arxiv:2409.19648 tab. 10]. - Query count: 100 gives 65.16, 300 gives 67.28, 500 gives 66.93 [arxiv:2409.19648 tab. 9]. - DOTA-v1.0 single-scale AP50 is 75.37 against 74.16 for ARS-DETR. Oriented R-CNN is higher at 75.87 [arxiv:2409.19648 tab. 4]. - Cost: 529 GFLOPs at 11.4 FPS, against 186 GFLOPs for ARS-DETR and Deformable DETR-O [arxiv:2409.19648 tab. 14]. The authors report 12 training epochs for their model against 36 for the two baselines [arxiv:2409.19648 §4.8]. ### Reference-point queries on a polar bird's-eye-view map (PolarFormer) PolarFormer detects 3D objects from six cameras on a polar grid of radius and azimuth. Its decoder uses learned queries that carry a reference point, refined in every layer. Card: [PolarFormer: Multi-camera 3D Object Detection with Polar Transformer](/papers/2206.15398). - **How a slot enters.** One learned embedding is split into a content part and a position part. The position part gives a 3-D reference point (azimuth, radius, height), through a sigmoid of a linear map [arxiv:2206.15398 §3.4]. Deformable cross-attention samples the polar map around the reference point [arxiv:2206.15398 §3.3, §3.4]. - **Refinement.** Each layer adds offsets to the reference point, so the slot moves across the map [arxiv:2206.15398 §3.4]. - **What a slot outputs.** A centre, a log size, and sine and cosine of the heading and velocity, both measured relative to the azimuth of the object [arxiv:2206.15398 §3.4]. - **How a slot gets its target.** The paper names focal and L1 losses but not a matching rule [arxiv:2206.15398 §3.4]. The released code uses Hungarian matching; see the card's Paper vs code. - **Not object slots.** The cross-plane encoder also has "ray queries", one per polar ray and image column. These are not object slots and are not matched to objects [arxiv:2206.15398 §3.1]. Reported behaviour (author-reported, nuScenes, six cameras, single frame): - Val, ResNet-101 with an FCOS3D initialisation: 39.6 mAP and 45.8 NDS, against 37.5 and 44.8 for BEVFormer-S [arxiv:2206.15398 tab. 3]. - Test, same setting: 41.5 mAP and 47.0 NDS, against 40.9 and 46.2 for BEVFormer-S [arxiv:2206.15398 tab. 1]. - The temporal variant's numbers are not reproducible from the released code, and the paper's improved setting uses extra pretraining data [arxiv:2206.15398 tab. 1, §4]. ### Heatmap-initialised slots with modality interaction (DeepInteraction) DeepInteraction keeps one representation per sensor and lets LiDAR and camera features exchange information. Its decoder refines 200 slots that start at the strongest peaks of a heatmap built from the LiDAR features. Card: [DeepInteraction: 3D Object Detection via Modality Interaction](/papers/2208.11112). - **How a slot enters.** Each slot starts at a peak of the BEV heatmap, as in TransFusion (no card; named only) [arxiv:2208.11112 §4.2]. The start is scene-dependent, not a learned embedding alone. - **How a slot is refined.** The slot's parameters are generated from the slot itself and applied to a 7 x 7 RoI pooled from the other modality. Odd layers pool from the image, even layers from the LiDAR BEV map [arxiv:2208.11112 §3.2]. - **What a slot outputs.** Centre, height, size, rotation and velocity, through heads applied to the refined slots [arxiv:2208.11112 §3.2]. - **How a slot gets its target.** Hungarian matching as in TransFusion, with the same loss applied to every layer's predictions [arxiv:2208.11112 §3.2]. The paper does not mention non-maximum suppression, and the released code applies none at test time (see the card's Paper vs code). Reported behaviour (nuScenes, author-reported): - Test: 70.8 mAP and 73.4 NDS for the base model with a ResNet-50 image backbone [arxiv:2208.11112 tab. 1]. - Decoder layer type, val, with both branches: 68.6 mAP and 71.6 NDS for DETR-style layers, 69.9 and 72.6 for the interaction layers [arxiv:2208.11112 tab. 3]. - Alternating LiDAR and image refinement against LiDAR only: 69.9 against 69.2 mAP on val [arxiv:2208.11112 tab. 3]. - Speed on a V100: 2.6 FPS against 3.8 for TransFusion [arxiv:2208.11112 tab. 2]. - The best query setting in the paper's ablation uses 200 slots in training and 300 at test: 70.1 mAP and 72.7 NDS on val [arxiv:2208.11112 tab. 3]. None of the ablations in tab. 3 varies the start of the slots, so the gain from heatmap starts is not separated from the rest of the decoder. ### Per-image depth bins from queries (BinsFormer) BinsFormer uses the queries to generate depth bins for each image rather than object outputs. A Transformer decoder reads multi-scale features with 64 bin queries and one scene query [arxiv:2204.00987 §3.2.2, §4.2.4]. Each pixel's depth is a probability-weighted sum of bin centres [arxiv:2204.00987 §3.2.3, eq. 2]. Card: [BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation](/papers/2204.00987). - **How a slot enters.** Learned query embeddings, one per bin and one for the scene. The paper says the queries start at zero and carry learned positional embeddings [arxiv:2204.00987 §3.2.5]. - **What a slot outputs.** A bin-length logit, softmaxed over the bins to give the bin widths, and a bin embedding from a 3-layer MLP. The scene query feeds a classifier over 25 indoor scene classes on NYU [arxiv:2204.00987 §3.2.2, §3.2.4, §4.2.1]. - **How a slot gets its target.** No slot is matched to an object. The loss is a scale-invariant depth loss on pixels plus a scene cross-entropy term, both on the output maps [arxiv:2204.00987 §3.2.3, §3.2.4, eq. 3]. - **Scales.** The decoder runs over three feature scales with three layers each, nine in total [arxiv:2204.00987 §4.2.4]. Reported behaviour (author-reported, NYU Depth v2 unless noted; the ablation backbone is not stated): - Adding a bin query to a regression baseline raises δ1 from 0.852 to 0.870. The full model with scene loss and multi-scale refinement reaches 0.890 [arxiv:2204.00987 tab. 5]. - Number of bins: 1 bin does not converge, 8 bins give δ1 0.889, the default 64 give 0.890, and 512 give 0.883 [arxiv:2204.00987 tab. 6]. - Feature scales: the finest scale alone gives 0.882 with three layers; the default three scales give 0.890 with nine layers [arxiv:2204.00987 tab. 7]. - With a Swin-Large backbone pre-trained on ImageNet-22K, δ1 is 0.925 against 0.903 for AdaBins [arxiv:2204.00987 tab. 3]. ### Global queries for image-level parameters (IAT) IAT is a light enhancement network for low-light and mis-exposed photos. Its global branch does not predict per-object outputs. It predicts one colour matrix and one gamma for the whole image, using 10 learned queries: nine for the matrix and one for gamma [arxiv:2205.14871 §3.2]. Card: [IAT, a 90K-parameter transformer for light and exposure correction](/papers/2205.14871). - **How a slot enters.** The queries cross-attend to keys and values from a low-resolution encoding of the image. There is no query-to-query self-attention and no residual around the attention. A feed-forward network follows [arxiv:2205.14871 §3.2]. The paper starts the queries at zero; the card's Paper vs code section notes that the released code starts them at one. - **What a slot outputs.** Gamma is one plus a linear map of its query. The colour matrix is the identity plus a linear map of the other nine queries, so the model starts as the identity [arxiv:2205.14871 §3.2]. - **How a slot gets its target.** None per slot. The whole image is supervised by an L1 loss, or by a smooth L1 plus VGG loss on LOL [arxiv:2205.14871 §4.1]. Reported behaviour (LOL-V2-real PSNR, ablation, author-reported): - With the proposed normalisation and local branch only, PSNR is 20.01. Adding the global matrix gives 21.95, adding the global gamma gives 22.76, and both give 23.50 [arxiv:2205.14871 tab. 7]. - The full model has 0.09M parameters and reports PSNR 23.50 and SSIM 0.824 on LOL-V2-real [arxiv:2205.14871 tab. 1]. ### Instance-sequence slots over a video clip (VisTR) VisTR runs the object-query decoder over a whole clip of T = 36 frames. It has N = n·T learned queries, with n = 10 instances per frame, so each slot is one instance in one frame, and the slots of one instance form a sequence over time [arxiv:2011.14503 §3.1, §4.1]. Card: [End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503). - **How a slot enters.** Each query is a separate learned embedding with the width of the pixel features. The encoder's space-time tokens carry a fixed 3D sine/cosine positional encoding, with d/3 channels per axis [arxiv:2011.14503 §3.1, eq. 1]. - **What a slot outputs.** For each frame, a box and a class from small heads, as in DETR [arxiv:2011.14503 §3.2]. Masks come from a separate stage: each slot's per-frame mask feature is stacked over the clip, and three 3D convolutions turn that stack into a mask sequence [arxiv:2011.14503 §3.3]. - **How a slot gets its target.** At the level of sequences. The matching cost of a sequence is computed over all of its frames, and a one-to-one Hungarian assignment pairs sequences with ground-truth instances [arxiv:2011.14503 §3.2, eq. 2, eq. 6]. The box loss and the mask loss are both averaged over the T frames [arxiv:2011.14503 eq. 8, eq. 9]. - **Tracking by layout.** Because slot k is the same instance in every frame, no separate linking step is needed. The paper states this as a design goal [arxiv:2011.14503 §1, §3.2]. Reported behaviour (author-reported, YouTube-VIS validation unless noted): - Main table: 40.1 AP and 64.0 AP50 with ResNet-101, and 36.2 AP with ResNet-50 [arxiv:2011.14503 tab. 2]. On the same ResNet-50 backbone, MaskTrack R-CNN has 30.3 AP in the same table, and its numbers come from re-implementations in the MaskTrack R-CNN paper [arxiv:2011.14503 tab. 2]. - Speed: 27.7 FPS with ResNet-101 including data loading, and 57.7 without [arxiv:2011.14503 tab. 2]. Images are loaded serially in the measurement, which the authors say costs much time. - Clip length, ResNet-50, 10 epochs: AP 29.7, 30.5, 31.7 and 33.3 for T = 18, 24, 30 and 36 [arxiv:2011.14503 tab. 1]. - Query layout, same setting: one query per instance and frame (360) gives 33.3 AP. Ten instance embeddings shared across frames give 32.0, and 36 frame embeddings give 13.7 [arxiv:2011.14503 tab. 1]. - Positional encoding, same setting: 28.4 AP without it and 33.3 with it [arxiv:2011.14503 tab. 1]. - 3D convolutions in the mask head: 34.4 AP with them, 33.3 without [arxiv:2011.14503 tab. 1]. - The authors call VisTR the most accurate single-model method. MaskProp with ResNet-101 reaches 42.5 AP in the same table, and the authors exclude it because it combines several networks and post-processing [arxiv:2011.14503 tab. 2, §4.3]. This is their argument, not a result of the table. ### Non-parametric instance queries on 3D scans (Mask3D) Mask3D segments object instances in a 3D point cloud. It has no learned query table. It places 100 slots at farthest-point-sampled positions of the scene, and the decoder turns each slot into one instance: a class and a binary voxel mask. Card: [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](/papers/2210.03105). - **How a slot enters.** Features start at zero. Only the position is used, Fourier-encoded from the sampled point, and it stays the same across layers [arxiv:2210.03105 §3.1, app. §1]. The paper says the slot count can change at inference without retraining [arxiv:2210.03105 §3.1]. - **Masked cross-attention.** A slot attends only to voxels inside its mask from the previous layer; the other voxels are excluded by adding minus infinity to their logits [arxiv:2210.03105 §3, eq. 3]. Attention runs over the four coarsest of five scales. The decoder repeats three times with shared weights, giving 12 refinement steps [arxiv:2210.03105 §3, app. §1]. - **What a slot outputs.** A mask from a dot product between the slot's instance feature and the per-voxel features, then a sigmoid and a 0.5 threshold [arxiv:2210.03105 eq. 1, §3]. A class from a linear layer with softmax over C+1 classes, the extra one being "no object" [arxiv:2210.03105 §3]. The mask module runs after every layer, so each layer gives an auxiliary mask. - **How a slot gets its target.** Hungarian matching on a cost of Dice, BCE and class terms, with weights 2, 5 and 2 [arxiv:2210.03105 eq. 4, §3.1]. The loss on matched pairs is 5 times BCE plus 2 times Dice [arxiv:2210.03105 eq. 5]. Unmatched slots get a "no object" class weight of 0.1, and the per-layer losses are summed [arxiv:2210.03105 §3.1, eq. 6]. Reported behaviour (ScanNet, author-reported): - ScanNet test mAP 56.6 against 50.4 for SoftGroup, the best listed baseline [arxiv:2210.03105 tab. 1]. The paper's headline gains on S3DIS and ScanNet200 mix pretrained and non-pretrained baselines, so compare table rows, not abstract figures [arxiv:2210.03105 tab. 2, tab. 3]. - Slot design, ScanNet val at 5 cm: zero features with sampled positions reach 40.6 ± 0.3 mAP, against 39.7 ± 0.7 for learned parametric queries and 38.4 ± 0.3 when the sampled points' features initialise the slots [arxiv:2210.03105 tab. 4]. - Mask loss, same setting: BCE alone 27.0, Dice alone 38.0, both 40.6 [arxiv:2210.03105 tab. 4]. - Query count: fewer queries lower accuracy, and more add less than 1 mAP. The paper reports this in a figure it does not tabulate [arxiv:2210.03105 §4.2]. ### Learned query tokens that are not object slots (DualVLN) DualVLN's planner also uses learned tokens, but they do not output one object each. Its latent tokens condition a downstream policy, and nothing is matched to them. Card: [Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation](/papers/2512.08186). - **How a token enters.** Four learnable `` tokens are appended to the planner's text after the pixel-goal text. They start random and are trained by prompt tuning [arxiv:2512.08186 §3.2, app. A]. Their final-layer hidden states form the latent goal [arxiv:2512.08186 app. A, §3.3]. - **Visual compression.** A Q-Former reads the fused features of two frames and outputs 32 visual-memory tokens [arxiv:2512.08186 §3.2]. These are inputs to the policy, not per-object outputs. - **What the tokens output.** None directly. A diffusion transformer generates 32 waypoints, conditioned on the latent goal and the visual tokens [arxiv:2512.08186 §3.2]. - **How they get their target.** None per token. The policy is trained with a flow-matching loss on the trajectory [arxiv:2512.08186 eq. 3]. The matching and slot-count rules above do not apply. Reported behaviour: the paper says that removing the latent goal makes results worse, but gives the numbers only in a figure [arxiv:2512.08186 fig. 7]. ### Object queries as dependency-graph nodes (D3G) D3G detects objects with a DETR-style decoder, then predicts for every pair of slots whether one object has to be removed before the other, for picking from a cluttered bin. The slots are not used as boxes alone: a dense graph transformer turns the N node embeddings into an N x N relation matrix. Card: [A Modern Take on Visual Relationship Reasoning for Grasp Planning](/papers/2409.02035). - **How a slot enters.** Learned object queries, as in DETR, give N node embeddings [arxiv:2409.02035 §4.1]. Three learned maps turn the nodes into an initial node state and two edge inputs. Pairwise concatenation of the two edge inputs gives an N x N x 2F tensor, which a learned map halves to F [arxiv:2409.02035 §4.2]. - **Graph layer.** Each layer has S heads. Every pair of nodes gets a score per head, which is multiplied by an edge projection and summed over channels. A softmax over the first node index gives the attention, and the node and edge features are updated from it [arxiv:2409.02035 §4.2, eq. 1-4]. The authors argue this dense form is cheaper than message passing over sparse graphs [arxiv:2409.02035 §4.2]. - **What a slot outputs.** Class and box, as in DETR. The relation output is the N x N logit matrix [arxiv:2409.02035 §4.2]. - **How a slot gets its target.** One-to-one Hungarian matching of queries to objects, with a class loss that includes the empty class and a box loss on the matched objects [arxiv:2409.02035 §4.3, eq. 5-7]. The relation loss is a binary cross-entropy over the P x P matched sub-matrix, against the ground-truth dependency matrix [arxiv:2409.02035 §4.3, eq. 8-9]. Reported behaviour (author-reported, D3GD benchmark, AP_rel unless noted): - Synth Easy: 76.27 for D3G (DETR) against 73.29 for VMRN, which uses Mask-RCNN [arxiv:2409.02035 tab. 3]. Synth Hard: 43.99 against 42.49 [tab. 3]. Synth-to-real: 24.44 against 13.36 [tab. 3]. - Detection mAP is not the best in these rows. On Synth Easy, D3G (DETR) reaches 83.22 against 86.96 for VMRN [arxiv:2409.02035 tab. 3]. The paper does not analyse this gap. - The synth-to-real OR and OP values vary a lot, because no validation set is available to tune the score threshold [arxiv:2409.02035 §6.3]. - On the standard VMRD benchmark with PascalVOC mAP, D3G reaches 96.50 and an IA of 78.33, against 96.40 and 75.33 for GGNN [arxiv:2409.02035 tab. 6]. ### Roadside query fusion across vehicles (UniV2X) UniV2X is an end-to-end planner that also uses a roadside unit's camera. The roadside does not send a dense feature map for the agents and lanes: it sends its tracked agent queries, its lane queries and one occupancy map with its flow. The vehicle aligns the roadside queries with its own, fuses the matched pairs and then plans. Card: [End-to-End Autonomous Driving through V2X Cooperation](/papers/2404.00717). - **How a slot enters.** A roadside agent query is a 256-dimensional feature with a tracking id and a reference point [arxiv:2404.00717 §3.2]. The vehicle advances it to its own timestamp with a predicted flow, which the paper gives as eq. 2, and moves its reference point into the vehicle frame with the relative pose [arxiv:2404.00717 §3.3, eq. 2-3]. - **How slots are matched.** Vehicle and roadside queries are paired by Hungarian matching on the Euclidean distance of their reference points. Each matched pair is fused into the vehicle query, and each unmatched roadside query is appended to the vehicle's set [arxiv:2404.00717 §3.3]. This pairing happens at fusion time, on the vehicle, and is not a training target. - **Other slots.** Roadside lane queries are concatenated with the vehicle's lane queries [arxiv:2404.00717 §3.4]. The roadside occupancy map is expanded over future steps with the flow, moved into the vehicle frame and combined with the vehicle's map by max [arxiv:2404.00717 §3.5, eq. 1]. - **Training.** The fusion modules are trained in a stage of their own, with the tracking and mapping losses [arxiv:2404.00717 eq. 5]. Reported behaviour (author-reported, DAIR-V2X, averaged over 2.5, 3.5 and 4.5 s for planning): - Collision rate 0.34% for UniV2X, against 0.89% with no fusion and 0.83% for BEV feature fusion [arxiv:2404.00717 tab. 2]. - The L2 error is not the best. UniV2X gives 3.48 m, against 3.44 m with no fusion and 3.30 m for BEV feature fusion [arxiv:2404.00717 tab. 2]. - Transmission: 8.09e5 BPS for UniV2X, against 8.19e7 for BEV feature fusion [arxiv:2404.00717 tab. 2]. - Car detection mAP is 0.295, against 0.268 for V2X-ViT and 0.243 for early fusion. AMOTA is 0.239, below 0.287 for V2X-ViT and 0.263 for late fusion [arxiv:2404.00717 tab. 3]. - Latency of 500 ms: mAP 25.9 without flow prediction and 28.5 with it, against 29.5 at 0 ms [arxiv:2404.00717 tab. 9]. The released code has no flow module (see the card's Paper vs code), so this row cannot be reproduced from it. ### Per-cell density queries beside waypoint queries (InterFuser) InterFuser is an end-to-end driving model with one transformer over tokens from several cameras and a LiDAR bird's-eye-view image. Its decoder has three query groups: waypoint queries, one query per cell of a 20 x 20 density map, and one traffic-rule query [arxiv:2207.14024 §3.2]. Each cell covers 1 m, and the map is in the ego-vehicle frame, so it covers a 20 x 20 m area and not the whole scene [arxiv:2207.14024 §3.1]. - **How a slot enters.** Every query gets a learned positional embedding. There is no reference point and no matching step: the cell's position in the grid fixes which target that query is trained on [arxiv:2207.14024 §3.2]. - **How a cell is read.** Each density query goes through its own three-layer MLP to seven channels: object existence, a 2-D offset from the cell centre, box size, heading and velocity [arxiv:2207.14024 §3.1, §3.2]. - **Training.** The loss has three terms: L1 on the waypoints, a density-map term and a binary cross-entropy term for the traffic outputs [arxiv:2207.14024 eq. 2, app. A]. The density term averages its error over positive and negative labels separately, so rare objects weigh as much as common ones [arxiv:2207.14024 app. A]. - **Who reads the map.** The map is not a detection list. A cell counts as an object if its existence probability exceeds 0.9, or exceeds 0.5 and is a local maximum among its neighbours; the second rule avoids counting one object several times [arxiv:2207.14024 §3.3, app. G]. A separate controller uses these objects, the traffic outputs and the waypoints to choose a speed [arxiv:2207.14024 §3.3, app. B]. Reported behaviour (author-reported, CARLA only): - Public CARLA leaderboard, accessed June 2022: driving score 76.18, route completion 88.23 and infraction score 0.84, ranked first [arxiv:2207.14024 tab. 1]. - Town05 Long: driving score 51.6 for the full model. Removing the safety controller gives 43.9, and replacing transformer fusion with concatenated inputs gives 21.5 [arxiv:2207.14024 tab. 3]. Card: [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](/papers/2207.14024). The released density head, label encoding and controller differ from the paper; see the card's Paper vs code before reusing any of them. ### Action segment queries with temporal deformable attention (TadTR) TadTR finds action instances in an untrimmed video. Its learned action queries are shared by all videos. Each one outputs a class and a segment. The encoder has 2 layers and the decoder has 4, both of width 256 [arxiv:2106.10271 §III-A, §IV-A]. Card: [End-to-end Temporal Action Detection with Transformer](/papers/2106.10271). - **How a slot enters.** A linear layer and a sigmoid predict each query's first reference centre from its embedding. Each decoder layer runs self-attention across the queries, then samples the encoder memory around the reference with temporal deformable cross-attention, using 4 sampling points for each of 8 heads [arxiv:2106.10271 §III-A, §IV-A]. - **What a slot outputs.** After each layer, an FFN predicts a class probability and a segment, given as a normalised centre and length. Layer $l$ refines the segment of layer $l-1$ with an offset, through a sigmoid and its inverse. The next layer's reference is the refined centre [arxiv:2106.10271 §III-A, eq. 4, eq. 5]. - **Actionness.** A separate head scores each segment. The segment is widened by a factor of 1.5, pooled to 16 bins by temporal RoIAlign, and regressed to the largest IoU with the video's ground-truth actions [arxiv:2106.10271 §III-A, §IV-A, tab. 11]. The paper's detection score is the square root of the class probability times this value [arxiv:2106.10271 §III-B]. - **How a slot gets its target.** Hungarian matching pairs each action query with at most one ground-truth action, and unmatched queries predict no action. No NMS is applied [arxiv:2106.10271 §III-B]. The paper's default classification cost is cross-entropy; focal loss is used in its improved setting [arxiv:2106.10271 §III-B, §IV-A]. Reported behaviour (author-reported): - HACS validation, average mAP: 30.83 for TadTR against 27.48 for G-TAD. Per video, 19 ms against 941 ms, and 0.1 GFLOPs against 45.7 [arxiv:2106.10271 tab. 2]. - HACS ablations: removing the encoder gives 26.94, replacing temporal deformable attention with dense attention gives 13.58, removing the actionness head gives 28.51 and removing segment refinement gives 27.65, against 30.83 for the full model [arxiv:2106.10271 tab. 4]. - Encoder choice on HACS, without NMS: the Transformer encoder gives 30.83 average mAP, a two-layer 1-D CNN gives 27.95 [arxiv:2106.10271 tab. 5]. ## When to use which | Situation | Use | Why | |---|---|---| | Variable number of instances per image (detection, instance or panoptic masks) | Matched object queries (DETR) | Slots are not pre-assigned; matching gives each object one slot and removes NMS [arxiv:2005.12872 §3.1, §4.2] | | Detection on a roughly 50-epoch budget | Conditional spatial queries (Conditional DETR) | 40.9 AP at 50 epochs against 34.9 for DETR [arxiv:2108.06152 tab. 1]; focal loss and offset regression alone account for 2.0 of the gain [arxiv:2108.06152 tab. 4] | | Better DETR-family accuracy with an unchanged inference path, and extra training time is acceptable | Training-only one-to-many heads (Co-DETR) | No inference change [arxiv:2211.12860 §1]; 43.3 to 46.8 AP at 36 epochs with one ATSS head [arxiv:2211.12860 tab. 8], for 86 vs 70 GPU hours at 12 epochs [arxiv:2211.12860 tab. 7] | | Training memory is tight | One auxiliary head at most, or none | Memory goes from 12808 MB (no head) to 13947 MB (one head) and 19385 MB (six heads) [arxiv:2211.12860 tab. 7]; more heads did not help [arxiv:2211.12860 §4.4] | | Fixed label set, one output per class (semantic segmentation) | Class-embedding queries (Segmenter) | Slot $k$ = class $k$, so a plain cross-entropy loss suffices, no matching [arxiv:2105.05633 §3.2, §4.2] | | Already have a ViT segmentation model with a linear head | Add a 2-layer mask transformer | +0.4 to +1.6 mIoU on ADE20K at near-equal throughput [arxiv:2105.05633 tab. 4, tab. 6] | | Many instances per image | Set $N$ well above the expected count, or avoid fixed slots | Detection recall collapses as the count approaches $N$ [arxiv:2005.12872 app. A.5, fig. 12] | | Small objects matter most | Benchmark against an anchor-based detector first | DETR trails Faster R-CNN on APS (20.5 vs 26.6) [arxiv:2005.12872 tab. 1] | | Table images with cell-level structure labels, one DETR-style detector per task | Unchanged DETR with 15 and 125 queries (PubTables-1M) | Content accuracy 0.8138 against 0.1039 for Faster R-CNN on canonical test data [arxiv:2110.00061 tab. 4]; the gap uses the paper's own data [arxiv:2110.00061 §3] | | Head counting or point localisation with point labels only | Point regression with kNN-context matching (CLTR) | UCF-QNRF F1 80.97 against 80.34 for TopoCount [arxiv:2202.13065 tab. 2]; NWPU test F1 is 0.6 below TopoCount, which uses boxes [arxiv:2202.13065 tab. 1] | | Need to know which image regions a detected object's query relied on, without retraining | Post-hoc relevance propagation through the decoder (2103.15679) | No training; author-reported weak segmentation AP 13.1 against 7.2 for the best baseline [arxiv:2103.15679 tab. 1]; one pass per kept query [arxiv:2103.15679 alg. 1] | | Rotated objects, for example in aerial or satellite images | Oriented-box positional queries (OrientedFormer) | DOTA-v1.0 AP50 75.37 against 74.16 for ARS-DETR, at 529 against 186 GFLOPs [arxiv:2409.19648 tab. 4, tab. 14]; the decoder output needs no NMS, but the official query selection uses one (see the card's Paper vs code) [arxiv:2409.19648 §1] | | Multi-camera 3D detection from a single frame, where a polar grid is acceptable | Polar reference-point queries (PolarFormer) | Val 39.6 mAP against 37.5 for BEVFormer-S [arxiv:2206.15398 tab. 3]; the matching rule comes from the code, not the paper [arxiv:2206.15398 §3.4] | | LiDAR and camera 3D detection where accuracy matters more than speed | Heatmap-started slots with modality interaction (DeepInteraction) | 70.8 mAP test for the base model [arxiv:2208.11112 tab. 1]; 2.6 FPS on a V100 against 3.8 for TransFusion [arxiv:2208.11112 tab. 2]; depends on calibration [arxiv:2208.11112 app. A.3] | | Per-pixel depth where the bins should adapt to each scene | Per-image bins from queries (BinsFormer) | δ1 0.925 against 0.903 for AdaBins with Swin-Large on NYU [arxiv:2204.00987 tab. 3]; 8 bins are within 0.001 of 64 [arxiv:2204.00987 tab. 6] | | One set of image-wide parameters (colour matrix, gamma) for a tiny model | Global queries without self-attention (IAT) | 0.09M parameters [arxiv:2205.14871 tab. 1]; LOL-V2-real PSNR 20.01 to 23.50 with the global branch [arxiv:2205.14871 tab. 7] | | Multi-object tracking in video, where identity must persist across frames and detection and association should be one trained model | Carried-over track queries (TrackFormer) | MOT17 private test 74.1 MOTA against 69.1 for TraDeS, both with CrowdHuman data [arxiv:2101.02702 tab. 1]; with public detections and the IoU filter, IDF1 is 57.6 against 62.8 for TMOH, so the advantage is not uniform [arxiv:2101.02702 tab. 1]; the released code's training path differs from the text (see the card's Paper vs code) | | Camera-only 3D detection from several calibrated cameras, where a depth network or an NMS step is unwanted | 3D reference-point queries (DETR3D) | 0.374 NDS and 0.303 mAP without NMS, against 0.373 and 0.299 for FCOS3D with NMS [arxiv:2110.06922 tab. 1]; needs the camera projection matrices [arxiv:2110.06922 §3.1]; about 18 hours on 8 GPUs at batch 1 per GPU [arxiv:2110.06922 §4.1] | | Lane topology from multi-camera driving data, where lanes are better as a few curve points than as boxes | Bezier-curve lane queries with pairwise MLP heads (TopoMLP) | OLS 38.2 against 35.6 for TopoNet on subset A, ResNet-50 [arxiv:2310.06753 tab. 1]; the pairwise tensor is 300 x 300 x 512 per image at the paper's sizes, about 46 million values before the MLP (our arithmetic) [arxiv:2310.06753 §3.3, §4.2]; the paper's metric can be raised by confidence changes alone [arxiv:2310.06753 tab. 4] | | Video instance masks on short, fixed-length clips, where per-frame linking rules are unwanted | Clip-level instance sequences (VisTR) | Same-backbone comparison: 36.2 AP for VisTR against 30.3 for MaskTrack R-CNN, both ResNet-50 [arxiv:2011.14503 tab. 2]; accuracy drops with shorter clips, 29.7 AP at T = 18 against 33.3 at T = 36 [arxiv:2011.14503 tab. 1]; the released inference reads only the first 36 frames (see the card's Paper vs code) | | Lane topology where connections depend on lane ends, and per-point lane queries are acceptable | Fine-grained per-point queries with boundary pairing (TopoFG) | OLS 48.0 against 44.1 for TopoLogic on subset_A [arxiv:2511.12590 tab. 1]; denoising groups alone add 1.5 OLS [arxiv:2511.12590 tab. 6]; the paper reports no memory, parameter count or latency [arxiv:2511.12590 §4] | | Lane topology where connected lane ends must meet at one point, and a second query set is affordable | Endpoint queries merged with lane queries (TopoPoint) | OLS 48.8 against 44.1 for TopoLogic on subset_A, DET_p 52.6 against 45.2 [arxiv:2505.17771 tab. 1]; the inference step adds 0.5 OLS over the graph layers [arxiv:2505.17771 tab. 2]; the released code does not run that step (see the card's Paper vs code) | | Instance masks on 3D point clouds, with no hand-tuned voting or clustering step | Non-parametric queries with masked cross-attention (Mask3D) | ScanNet test mAP 56.6 against 50.4 for SoftGroup [arxiv:2210.03105 tab. 1]; 39.6 M parameters, and about 78 hours on one A40 for one 2 cm run [arxiv:2210.03105 tab. 6, §3.1]; the query count of 100 is tied to memory-limited voxel resolution [arxiv:2210.03105 §4.2] | | Need to know which object must be removed before another, or any pairwise relation between detected objects in one image, without a crop per pair | Object queries as dependency-graph nodes (D3G) | Synth Hard AP_rel 43.99 against 42.49 for VMRN, and synth-to-real 24.44 against 13.36 [arxiv:2409.02035 tab. 3]; detection mAP is lower than the Mask-RCNN rows on Synth Easy (83.22 against 86.96) [arxiv:2409.02035 tab. 3] | | Vehicle planning with a roadside camera, where the link carries sparse queries and one occupancy map | Roadside queries fused by Hungarian matching (UniV2X) | Collision rate 0.34% against 0.89% without fusion [arxiv:2404.00717 tab. 2]; BPS 8.09e5 against 8.19e7 for dense BEV fusion [arxiv:2404.00717 tab. 2]; L2 is 3.48 m against 3.44 m with no fusion [arxiv:2404.00717 tab. 2]; the flow-based latency step is not in the released code (see the card's Paper vs code) | | End-to-end driving from cameras and LiDAR, where the model must expose what it sees as a grid and a safety layer must read it | Per-cell density queries beside waypoint queries (InterFuser) | Leaderboard driving score 76.18, first place when accessed in June 2022 [arxiv:2207.14024 tab. 1]; 43.9 without the safety controller against 51.6 on Town05 Long [arxiv:2207.14024 tab. 3]; CARLA only, and the map covers 20 x 20 m around the vehicle [arxiv:2207.14024 §3.1]; the released controller is a heuristic, not the paper's linear programme (see the card's Paper vs code) | | Action start and end times in untrimmed video, where anchors and NMS are unwanted and frozen, pre-extracted features are acceptable | Action segment queries with temporal deformable attention (TadTR) | HACS validation average mAP 30.83 against 27.48 for G-TAD, at 19 ms against 941 ms per video [arxiv:2106.10271 tab. 2]; the paper trains on frozen features and names end-to-end training of the video encoder as future work [arxiv:2106.10271 §III, §V]; a video with more true actions than N_q loses the extra ones [arxiv:2106.10271 §IV-D] | | Memory-bound decoder | Fewer query-decoder layers | Segmenter drops to 1 mask-transformer layer for Seg-L on Cityscapes [arxiv:2105.05633 §4.4]; for DETR, the first decoder layer's output is 8.2 AP below the last [arxiv:2005.12872 §4.2, fig. 4] | ## Implementation sketch ```text Input: image features F (S tokens x d) from a CNN+encoder or a ViT Params: Q (num_slots x d), learned DETR-style (queries as positional encodings, cross-attention): [arxiv:2005.12872 §3.2, app. A.3] h = zeros(num_slots, d) for each decoder layer: h = SelfAttn(q = h + Q, k = h + Q, v = h) # slots see each other h = CrossAttn(q = h + Q, k = F + pos_F, v = F) # slots read the image h = FFN(h) # residual + dropout + post-LayerNorm out_l = heads(LayerNorm(h)) # shared heads, every layer targets: Hungarian matching per image (see bipartite-matching) Segmenter-style (queries as tokens, joint self-attention): [arxiv:2105.05633 §3.2, eq. 4] x = concat(F_patches, Q) # Q: one embedding per class, num_slots = K for each of M layers (M = 2): x = TransformerBlock(x) # pre-norm self-attention + MLP p, c = split(x) # patches (N x d), classes (K x d) masks = normalize(p) @ c^T # N x K reshape to grid, bilinear upsample, per-pixel cross-entropy with target class k for slot k Conditional DETR cross-attention, per decoder layer: [arxiv:2108.06152 §3.3, eq. 3-6] s = sigmoid(FFN_ref(O)) # reference point, from object query O p_s = sine_embed(s) # same layout as the key pos. embedding lam = FFN_scale(f_prev) # per-channel scale from previous layer p_q = lam * p_s # conditional spatial query q = concat(c_q, p_q) per head # content and spatial halves k = concat(c_k, p_k) per head h = CrossAttn(q, k, v = c_v) # logit = c_q.c_k + p_q.p_k Co-DETR training step (Deformable-DETR-style base): [arxiv:2211.12860 §3.2, §3.3] memory = encoder(backbone(images)) pyramid = memory + [conv_s2(memory[-1])] # one extra coarse level main = decoder(Q_learned, memory) # Hungarian loss per layer for head in aux_heads: # e.g. ATSS, Faster R-CNN preds = head(pyramid) pos, boxes, targets = head.assign(preds, gt) # one-to-many, eq. 1 q = Linear(PE(boxes)) + Linear(gather(pyramid, pos)) # eq. 4 out = decoder(q, memory) # own group, no matching loss += lambda_1 * detr_loss(out, targets) # eq. 5 loss += lambda_2 * head.loss(preds, pos) # eq. 2, 3 inference: run decoder(Q_learned, memory) only CLTR matching, one image [arxiv:2202.13065 §3.3, eq. 2-3] cost = L1(gt_point, pred_point) + L1(gt_knn, pred_knn) - pred_conf match = hungarian(cost) # one-to-one, unmatched = background loss = focal(pred_conf, labels) + 2.5 * L1(pred_point[match], gt_point) TrackFormer, one frame [arxiv:2101.02702 §3.2, app. A.3, app. A.4] Q_track = [t.embedding for t in tracks] # last decoder output per track h = Decoder(concat(Q_track, Q_object), memory = Encoder(frame)) # self-attn over all for each object-query output with score > 0.4: start a new track for each track with score < 0.4: end the track DETR3D, one decoder layer [arxiv:2110.06922 §3.3, eq. 1-4] c = Phi_ref(q) # 3D reference point per query uv = project(T_m, [c, 1]) for each camera m, mapped to [-1, 1] f = mean of bilinear(F_level,m, uv) over visible samples q = q + f # residual, eq. 4 q = MultiHeadSelfAttn(q) # queries exchange information box, class = heads(q) # eq. 5, loss on every layer VisTR, one clip of T frames [arxiv:2011.14503 §3.1, §3.2] feats = CNN(frames) # T frame maps, in time order mem = Encoder(flatten(conv1x1(feats)) + PE3d(T, H, W)) out = Decoder(Q, mem) # Q: n*T learned queries, one pass seq = out.reshape(T, n, d) # slot t*n + k is instance k in frame t for k in range(n): cost[k] = sum over frames of class and box terms for sequence k pairs = hungarian(cost) # one-to-one, over sequences loss = per pair: class NLL, box loss and mask loss, each averaged over T TopoPoint, one point-lane layer [arxiv:2505.17771 §3.4, eq. 3-16] # Merged self-attention with geometric bias from last layer's predictions bias = [[0, M_pl], [M_pl.T, M_ll]] # eq. 4-7, paper only h = LayerNorm(SelfAttn(concat(P, L), bias)) # eq. 3, eq. 8 # Each set reads the BEV map at its own learned reference points P = FFN(DeformAttn(h_P, ref_P, bev)) # eq. 9-10 L = FFN(DeformAttn(h_L, ref_L, bev)) # Point-lane graph: messages in both directions P, L = PLGCN(P, L, A_pl) # eq. 11-13 # Lane-lane and lane-traffic graphs, then a linear map to lane width L = Linear(concat(GCN_ll(L) + L, GCN_lt(T) + L)) # eq. 14 P, L = PLGCN(P, L, A_pl) # eq. 15-16 Mask3D, one refinement step [arxiv:2210.03105 §3, eq. 1-3] inst = MLP(h) # instance feature per query masks = inst @ F0^T # per-voxel logits at full resolution allow = resize(sigmoid(masks_prev), scale) >= 0.5 # voxels inside the last mask h = CrossAttn(q = h, k = F_s, v = F_s, bias = where(allow, 0, -inf)) h = SelfAttn(h) # queries see each other h = FFN(h) # run for 4 scales, coarse to fine, 3 times over: 12 steps [§3] D3G relation step, one image [arxiv:2409.02035 §4.2, eq. 1-4] # N object queries from the DETR decoder, shape (N, F) nodes = decoder(Q_learned, memory) # pairwise concatenation of two learned edge maps, shape (N, N, F) e = pair_concat(M_e1(nodes), M_e2(nodes)) for layer in range(L): # L = 2 in the paper's chosen setting nodes, e = DenseGraphLayer(nodes, e) # S heads, attention over pairs rel = Linear(e) # (N, N) relation logits # train: BCE on the P x P sub-matrix of matched queries vs ground truth UniV2X agent fusion, one frame [arxiv:2404.00717 §3.3, eq. 2-3] # latency compensation by predicted flow (paper only, not in released code) inf = advance_by_flow(inf, dt) inf.ref = to_vehicle_frame(inf.ref, pose) # reference points, eq. 3 cost = cdist(veh.ref, inf.ref) # Euclidean distance veh_i, inf_i = hungarian(cost) # one-to-one pairs veh[veh_i] = fuse(veh[veh_i], inf[inf_i]) # matched pairs update vehicle veh = concat(veh, inf[unmatched(inf_i)]) # roadside queries not matched ``` Notes: - DETR's decoder layers are post-norm (residual, dropout, LayerNorm) [arxiv:2005.12872 eq. 6]; the first self-attention of the first layer can be skipped, since all slots start at zero [arxiv:2005.12872 app. A.3]. - DETR's inference keeps every slot: a slot predicting $\varnothing$ is scored with its second-best class, which adds 2 AP over dropping such slots [arxiv:2005.12872 §4]. - Segmenter's released code differs from eq. 4 [arxiv:2105.05633 eq. 4]: it projects decoder input, applies a LayerNorm after the blocks, uses separate learned projections for patches and class embeddings, L2-normalises both (so masks are cosine similarities) and applies a LayerNorm over the class dimension before upsampling, with softmax only inside the loss. The paper's own order (softmax, then layer norm) is internally inconsistent; the card's "Paper vs code" section has the code references. - Conditional DETR: concatenate content and spatial parts within each head, not across the full 256-d vector, or the two halves land in different heads [arxiv:2108.06152 §3.3]. The attention scale then follows the concatenated per-head dimension. The first decoder layer is handled differently, and the released code predicts the reference point once and reuses it in every layer; see the [Conditional DETR card](/papers/2108.06152), Paper vs code. - Co-DETR: the paper sends only positive boxes into the auxiliary groups [arxiv:2211.12860 §3.5]. The released code pads each group with negative anchors; see the [Co-DETR card](/papers/2211.12860), Paper vs code. - Co-DETR: eq. 6 puts $\lambda_2 \mathcal{L}^{enc}$ inside the sum over the $L$ decoder layers [arxiv:2211.12860 eq. 6]. Implementing the auxiliary encoder loss once, at weight 2, gives the encoder only 1/L of the weight the formula describes. - CLTR's kNN value: the paper uses k=4 neighbours [arxiv:2202.13065 §4.1, eq. 3]. The released code averages three, because it drops the point itself from its four-point query. The paper's sum does not say whether the point itself counts; see the [CLTR card](/papers/2202.13065), Paper vs code. - CLTR's training crops: the paper observes that its patches hold fewer than 500 people [arxiv:2202.13065 §5.4.1]. The released code also drops training crops whose point count reaches the query count, a rule the paper does not state. - OrientedFormer: the Gaussian encoding uses 2^(z−r) and 2^(z+r) as printed in eq. 13, which does not match the paper's own z and r definitions [arxiv:2409.19648 eq. 11, eq. 13]. Follow the definitions, and rotate about the box centre, not the image origin as eq. 14 reads literally [arxiv:2409.19648 §3.5]. - OrientedFormer's self-attention bias. The paper adds the bias to the logits as is [arxiv:2409.19648 eq. 10]. The card's Paper vs code section reports a learnable per-head scale in the released code, which the paper does not describe. Implementing the paper alone gives different attention behaviour. - PolarFormer's reference points are 3-D in the paper [arxiv:2206.15398 §3.4]. The released deformable sampling uses only azimuth and radius, and the height is refined by regression alone; see the card's Paper vs code. - DeepInteraction's decoder depth: the paper says five cascaded decoder layers [arxiv:2208.11112 §4.2]. The released code has one transformer decoder layer and four interaction layers. Follow the code if you want to reproduce the reported numbers. - BinsFormer's loss placement. The paper applies the depth and scene losses to every decoder layer, weighted by scale [arxiv:2204.00987 §3.2.5, eq. 4]. The card's Paper vs code section reports that the released Swin config supervises three layers and uses a scene-loss weight of 1e-2, not the paper's 1e-3. - IAT's query initialisation. The paper starts the global queries at zero [arxiv:2205.14871 §3.2]. The card's Paper vs code section reports that the released code starts them at one. - TrackFormer's track count changes from frame to frame, but the decoder stacks track queries per image. The paper pads the batch with false-positive queries [arxiv:2101.02702 §4.2]. The released code gives every image the same count per batch instead, so a per-image random count needs padding to batch. See the [TrackFormer card](/papers/2101.02702), Paper vs code. - TrackFormer's identity matching must be forced. Track-query outputs go to the ground truth of their own track, and false-positive track queries never match [arxiv:2101.02702 §3.3]. A plain Hungarian match over all slots breaks identity. - TrackFormer's previous-frame pass: the paper does not say whether gradients flow through it. The released code runs it without gradients by default [arxiv:2101.02702 §3.3]; see the card's Paper vs code. - DETR3D's aggregation: the paper's eq. 4 averages visible samples, with a denominator that has an unspecified ε [arxiv:2110.06922 eq. 4] [arxiv:2110.06922 §3.3]. The released code uses learned sigmoid weights per camera, level and point, and does not normalise [arxiv:2110.06922 §3.3]; see the [DETR3D card](/papers/2110.06922), Paper vs code. Following the equation can give a different feature scale. - DETR3D's reference points: the released code detaches them between layers, so no gradient flows from later layers into earlier reference points. The paper does not mention this [arxiv:2110.06922 eq. 1]; see the card. - VisTR's slot count is tied to the clip length: N = n·T, with n = 10 and T = 36 [arxiv:2011.14503 §4.1]. The released matcher hard-codes 10 slots per frame in frame-major order [code: models/matcher.py#L47-L49], so changing the query count needs a change there too. Its inference also takes only the first 36 frames of a video. See the [VisTR card](/papers/2011.14503), Paper vs code. - TopoPoint: the paper adds the geometric matrices to the self-attention logits [arxiv:2505.17771 §3.4, eq. 7]. The released code builds no such mask and uses the matrices only as graph adjacencies. Its point-to-lane refinement is commented out. Following the equations alone gives a different model from the released one; see the [TopoPoint card](/papers/2505.17771), Paper vs code. - Mask3D's query count: the paper uses 100 in every experiment [arxiv:2210.03105 app. §1]. The released ScanNet and ScanNet200 scripts use 150 and the STPLS3D script 160. Check the release scripts before reproducing a table row; see the [Mask3D card](/papers/2210.03105), Paper vs code. - Mask3D's top-k: the paper computes confidences for queries by their dominant class and describes no top-k step [arxiv:2210.03105 §3.1]. The released code keeps the top query-class pairs, so one query can yield several instances. See the [Mask3D card](/papers/2210.03105), Paper vs code. - Mask3D's empty attention masks: the paper does not say what happens when a query's mask covers no voxel. The released code unmasks all voxels for that query. See the [Mask3D card](/papers/2210.03105), Paper vs code. - D3G's relation loss scale: eq. 9 sums the BCE over all P x P pairs [arxiv:2409.02035 eq. 9]. The released code divides that sum by the number of matched objects, per image [see the D3G card, Paper vs code]. A literal implementation of eq. 9 trains with a different loss scale. - D3G's edge softmax: check the axis against the code before implementing eq. 2. The card's Paper vs code section gives the lines; the paper's text does not settle it. - D3G's loss is not only on the last layer in the code. The released code also computes the relation loss on each auxiliary decoder layer [see the D3G card, Paper vs code]; whether those terms are summed depends on the trainer, which the repository does not include. - UniV2X's latency step: eq. 2 is the only description of query flow, and the released code has no flow module [arxiv:2404.00717 eq. 2]. Plan without latency compensation unless you add the module yourself. - UniV2X's fusion gate: the paper describes a Hungarian match on Euclidean distance [arxiv:2404.00717 §3.3]. The released code adds a per-axis distance gate and a score gate for vehicle queries [see the UniV2X card, Paper vs code]. A paper-only implementation can match pairs that the code would reject. ## Contradictions Two papers diagnose slow DETR convergence differently. Neither tests the other's explanation, and this page does not resolve them. - Conditional DETR attributes it to decoder cross-attention depending heavily on content embeddings; object queries give only a generic spatial prior [arxiv:2108.06152 §1]. - Co-DETR attributes it to sparse one-to-one supervision: with few positive queries, the encoder features separate foreground from background poorly, and decoder cross-attention learns inefficiently [arxiv:2211.12860 §1, §3.4]. Their baselines also differ, so their numbers are not directly comparable. Conditional DETR reports 50, 75 and 108 epochs [arxiv:2108.06152 tab. 1]. Co-DETR's table reports Conditional DETR-C5 at 36 epochs, 39.4 AP without an auxiliary head [arxiv:2211.12860 tab. 2]. The Segmenter, BinsFormer, IAT, PubTables-1M, CLTR, PolarFormer, DeepInteraction and Mask3D results measure different tasks or use different datasets, so they are not comparable to the detection tables or to each other. Mask3D's 3D instance tables compare against bottom-up methods such as SoftGroup and HAIS [arxiv:2210.03105 tab. 1]. TrackFormer's MOT and MOTS numbers measure tracking, and DETR3D's nuScenes rows compare against camera-only baselines (CenterNet, FCOS3D, Mono3D, DHNet, PGD, DD3D); neither set of tables includes PolarFormer or DeepInteraction [arxiv:2110.06922 tab. 1] [arxiv:2110.06922 tab. 2] [arxiv:2101.02702 tab. 1]. VisTR's video table compares against tracking and mask-propagation methods, not against the detectors above [arxiv:2011.14503 tab. 2]. Learned slots (DETR, CLTR, PolarFormer) and heatmap-started slots (DeepInteraction) are not compared against each other in any of these papers, so this page cannot rank them. The lane-topology papers report TopoMLP differently. TopoMLP reports OLS 38.2 and TOP_ll 7.2 on subset A with ResNet-50 [arxiv:2310.06753 tab. 1]. The survey repeats TOP_ll 7.2 with DET_l 28.3 [arxiv:2504.01989 tab. 3]. TopoFG lists TopoMLP's official model at OLS 44.1 and TOP_ll 21.7, evaluated with the released model [arxiv:2511.12590 tab. 1]. TopoPoint lists OLS 44.1 and TOP_ll 21.6 [arxiv:2505.17771 tab. 1], but copies these rows from the TopoLogic and TopoFormer papers, except DET_p, which it computed. The survey gives TopoLogic DET_l 34.4 and TOP_ll 23.4 on subset A [arxiv:2504.01989 tab. 3], against 29.9 and 23.9 in TopoPoint [arxiv:2505.17771 tab. 1]. The survey does not say whether its numbers were re-run or copied. The settings differ, and this page does not resolve these gaps. D3G's own tables give the S=2, L=2 Easy AP_rel as 76.77 in tab. 5, and as 76.27 in tab. 3 and tab. 4 [arxiv:2409.02035 tab. 3] [arxiv:2409.02035 tab. 4] [arxiv:2409.02035 tab. 5]. The other columns agree. The numbers in this page use tab. 3, and the paper does not say which one is right. TadTR's ActivityNet headline of 36.75 mAP uses TSP features and external classifiers. Its self-contained rows use two-stream features and reach 28.21 (TadTR) and 29.90 (TadTR*) [arxiv:2106.10271 tab. 3]. On THUMOS14, the 56.7 average is the starred setting with IBIF and focal loss, and the default TadTR row is 49.3 [arxiv:2106.10271 tab. 1]. This page does not merge the two settings. TadTR's temporal-action tables measure a different task from the detection, segmentation and lane tables above, so they are not comparable to them. InterFuser's CARLA results are closed-loop driving scores from a simulator, and its ablations use the Town05 Long route. They are not comparable to the detection, segmentation, lane or depth tables above [arxiv:2207.14024 tab. 1, tab. 3]. ## Papers - [End-to-End Object Detection with Transformers](/papers/2005.12872): introduces learned object queries decoded in parallel and assigned to objects by Hungarian matching. - [Conditional DETR for Fast Training Convergence](/papers/2108.06152): adds a per-slot spatial query, predicted from a reference point, so that cross-attention separates position from content. - [DETRs with Collaborative Hybrid Assignments Training](/papers/2211.12860): trains extra one-to-many heads and query groups built from their positives; inference is the unchanged DETR. - [Segmenter: Transformer for Semantic Segmentation](/papers/2105.05633): uses one learned query per class, processed jointly with ViT patch tokens, to produce class masks by scalar product. - [PubTables-1M: Towards comprehensive table extraction from unstructured documents](/papers/2110.00061): applies unchanged DETR, with 15 and 125 learned queries, to table detection, structure and functional analysis on a dataset with canonicalised structure labels. - [An End-to-End Transformer Model for Crowd Localization](/papers/2202.13065): regresses one point and one confidence per learned query, and matches queries to heads with a kNN-context cost. - [TrackFormer: Multi-Object Tracking with Transformers](/papers/2101.02702): carries each track's previous output embedding into the next frame as a track query, next to learned object queries for new objects. - [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](/papers/2103.15679): propagates relevance through DETR's decoder to show which image regions each detected query used. - [OrientedFormer](/papers/2409.19648): gives each query an oriented box as its positional part, encoded as a Gaussian, with a Wasserstein self-attention bias and rotated cross-attention sampling. - [PolarFormer: Multi-camera 3D Object Detection with Polar Transformer](/papers/2206.15398): decodes learned queries with refined 3-D reference points on a polar BEV map built from six cameras. - [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](/papers/2110.06922): decodes 900 learned queries to 3D reference points, projects them into six cameras and samples image features there, with no NMS. - [TopoMLP: A Simple yet Strong Pipeline for Driving Topology Reasoning](/papers/2310.06753): decodes 300 lane queries into Bezier control points and pairs them with traffic queries in small MLP heads that predict topology. - [DeepInteraction: 3D Object Detection via Modality Interaction](/papers/2208.11112): starts 200 slots at BEV heatmap peaks and refines them by alternating image and LiDAR RoI pooling. - [IAT, a 90K-parameter transformer for light and exposure correction](/papers/2205.14871): uses ten global queries, without self-attention, to predict one colour matrix and one gamma per image. - [BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation](/papers/2204.00987): uses bin queries and a scene query in a Transformer decoder to generate per-image depth bins. - [End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503): decodes n·T learned queries over a 36-frame clip, and matches each per-instance sequence of slots to a ground-truth instance as a unit. - [Fine-Grained Representation for Lane Topology Reasoning](/papers/2511.12590): describes each lane with 11 ordered queries, seeded by a BEV lane mask and point-order priors, and pairs only the first and last query of each lane. - [A Concise Survey on Lane Topology Reasoning for HD Mapping](/papers/2504.01989): surveys lane-topology methods and the scores reported for them on Topo-Boundary and OpenLane-V2. It has no method of its own. - [TopoPoint: Enhance Topology Reasoning via Endpoint Detection in Autonomous Driving](/papers/2505.17771): adds endpoint queries, merged with lane queries in one self-attention, and at inference pulls connected lane ends to a shared detected point. - [Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation](/papers/2512.08186): uses four learned latent tokens and a 32-token Q-Former for a navigation policy. These are not object slots. - [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](/papers/2210.03105): predicts 100 non-parametric instance queries, placed at sampled scene points, and refines them with masked cross-attention to per-instance masks trained by Hungarian matching. - [A Modern Take on Visual Relationship Reasoning for Grasp Planning](/papers/2409.02035): uses DETR-style object queries as nodes of a dense graph transformer that predicts a dependency matrix between objects. - [End-to-End Autonomous Driving through V2X Cooperation](/papers/2404.00717): fuses the roadside unit's tracked agent queries into the vehicle's queries by Hungarian matching, before an end-to-end planner. - [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](/papers/2207.14024): decodes waypoint queries, 400 density-map cell queries and one traffic query from one transformer over camera and LiDAR tokens, and feeds the density map to a safety controller. - [End-to-end Temporal Action Detection with Transformer](/papers/2106.10271): learned action queries each predict one segment and class, refined layer by layer with temporal deformable attention and scored by an actionness head, with no anchors or NMS.