SpecDETR: A transformer-based hyperspectral point object detection network

In plain words

A hyperspectral camera records a long list of light measurements for every pixel, not just red, green and blue. Some targets are smaller than one pixel, so they show up only as a faint mix of materials inside a pixel. The usual approach labels each pixel as "target" or "background" using a known spectrum of the target. It finds where a material is, but it does not return separate objects.

SpecDETR treats the whole image cube as a set of pixel tokens and reads it with a transformer. Its output is a set of boxes, each with one of eight simulated object classes. It is built on an object detector called DINO, but it removes the image backbone that normally shrinks the picture. Keeping full resolution matters for objects that are tiny.

Training uses simulated scenes with exact labels, because hand-labelling mixed pixels is unreliable. The network learns to predict several overlapping boxes for one object and then uses a final clean-up step to remove the duplicates.

TL;DR

• SpecDETR is a DETR-style transformer detector for hyperspectral point objects, built on DINO with the backbone removed §1 §3.2.
• Every pixel is a token; a six-layer encoder uses deformable attention on the pixel map plus a global token §3.3 §3.4.
• Training assigns several predictions to each object (one-to-many) and adds noisy boxes for contrastive denoising §3.6 §3.7.
• Author-reported: SPOD mAP 0.856 versus 0.757 for DINO with Swin-L, and 16.1 M parameters versus 218.3 M tab. 2 tab. 3.
• Inference is slower than Faster R-CNN, the training data is simulated, and the official code disagrees with the paper on several settings (see Paper vs code).

Problem

Traditional hyperspectral target detection (HTD) classifies each pixel by its similarity to a prior target spectrum, so it returns per-pixel scores rather than objects §1. It cannot use object-level shape and layout, and its methods typically have one class and need only one prior spectrum §3.1 §4.1.

Visual object detectors can return instance-level boxes and classes, but ordinary backbones downsample the image. Point objects that cover a fraction of one pixel are then lost §3.4. DETR-like detectors with one-to-one matching also produce redundant boxes for point objects, which the authors' experiments found §3.5.

Assumptions: the method needs a large set of training images with exact boxes and classes §4.1. The public HTD datasets lack such a training set, so the authors simulate one from a single prior spectrum per class §4.1 [supp. Generation of Simulated Training Data for Public HTD Datasets].

Method

Overview. SpecDETR is built on DINO and removes its backbone. It contains a multi-layer transformer encoder, a multi-layer decoder, and several prediction heads. A linear layer maps the bands of each pixel to a token, and every pixel is one token §3.2 fig. 3. Each encoder output token proposes one candidate box at its pixel location, and the top candidates initialise the decoder queries §3.2 §3.5. The decoder refines the boxes and predicts classes layer by layer. Training adds a denoising branch and replaces the one-to-one matching of DETR with a hybrid one-to-many assigner. Inference keeps the top-scoring predictions and applies non-maximum suppression (NMS) §3.2.

Data tokenisation. A cube X with H × W pixels and N bands is divided by one constant V and passed through a linear layer and layer normalisation. The constant is shared by the whole dataset §3.3 eq. 1. The initial global token is the mean of all pixel tokens §3.3 eq. 2. The authors reject per-band and per-image normalisation because bands differ in spread and images differ in range §3.3.

Encoder with self-S2A. Each encoder layer applies a self-excited subpixel-scale attention module (S2A) with a residual connection and layer normalisation, then a feed-forward network with the same structure §3.4 eq. 3 eq. 4. S2A is deformable attention with two sampled maps. The first is the pixel map at full resolution, sampled at sub-pixel positions by bilinear interpolation §3.4 eq. 8 eq. 10. The second is the global token, which is treated as a one-pixel map so that it can amplify its own features ("self-excitation") §3.4 fig. 6. Offsets and weights come from the query token and its position encoding §3.4 eq. 5 eq. 6. The weights are normalised over the sampling points §3.4. Each head produces a value, and the heads are concatenated and projected §3.4 eq. 9. The authors describe the sampling as a linear mixing model: the sampled spectrum is a non-negative combination of the four nearest pixel spectra §3.4 eq. 11.

Decoder. The first candidate boxes are built from encoder outputs. Each token gives an initial box and class scores §3.5 eq. 12 eq. 13. The top Q_match candidates are kept, and each becomes a decoder query. All content queries start as a vector of ones, with no index embedding, because the authors judge distinct queries unnecessary under one-to-many matching §3.5. Each decoder layer updates the queries with a cross-attention to the encoder output and a feed-forward network §3.5 eq. 14 eq. 15. The authors remove the self-attention between queries, so the queries do not interact §3.5. Each layer then predicts a box update, a detached copy is passed on, and the loss uses a second prediction built from the previous layer's box, which DINO calls "look forward twice" §3.5 eq. 16 eq. 17 eq. 18 eq. 19. The class scores of each query form Q × N_C predictions §3.5.

Hybrid one-to-many assignment. Forced matching first runs the Hungarian algorithm once, so each ground-truth box gets one positive prediction, using a combined GIoU, L1 and classification cost §3.6. Dynamic matching then adds positives from the remaining predictions whose IoU with a ground-truth box is above a threshold. At most T extra positives are kept per ground-truth box, highest IoU first §3.6 [supp. Hybrid Label Assigner of SpecDETR].

Center-shifting contrastive denoising (CCDN). The denoising branch makes noisy copies of ground-truth boxes as positive and negative training queries §3.7. Each copy is shifted from the box centre by a random amount tied to the box size. Positive copies stay near the centre, and negative copies land between half and all of a τ₁ fraction of the size §3.7. The width and height of each copy are then resampled around the original size with a factor τ₂ §3.7. The paper states τ₁ = 0.5 and τ₂ = 1.5 §5.1.1. The original CDN of DINO uses negatives that overlap the positives; the authors say such negatives confuse the classifier of a one-to-many detector §3.7 tab. 12.

Losses and inference. The loss is DINO's: L1 and GIoU for boxes and focal loss for classes, applied to each decoder layer and to the encoder candidates, with weights 5, 2 and 1 [supp. Loss Function]. At test time the top 300 (query, class) scores are kept, then NMS with IoU 0.01 removes overlaps §5.1.1.

Algorithms

Pseudocode is in the authors' notation. Each line is a rewrite in our own words. Each step cites the paper section it follows.

# Data tokenisation  [§3.3] [eq. 1] [eq. 2]
# cube: (H, W, N) bands; V: one constant for the whole dataset
def tokenize(cube, V):
    # One token per pixel, after linear map and layer norm  [§3.3] [eq. 1]
    pixel_tokens = layer_norm(linear(cube / V))   # (H, W, C)
    # Global token: mean over all pixel tokens  [§3.3] [eq. 2]
    global_token = mean_over_pixels(pixel_tokens) # (C,)
    return pixel_tokens, global_token

# One encoder layer  [§3.4] [eq. 3] [eq. 4]
# tokens: pixel tokens and the global token, (H*W + 1, C)
def encoder_layer(tokens):
    # Self-S2A with residual, then layer norm  [§3.4] [eq. 3]
    mixed = layer_norm(self_s2a(tokens) + tokens)
    # Feed-forward with residual, then layer norm  [§3.4] [eq. 4]
    return layer_norm(feed_forward(mixed) + mixed)

# Self-S2A over all encoder tokens  [§3.4] [eq. 5-9]
# M heads, K sampling points per head; pixel map plus global token
def self_s2a(tokens):
    pixel_map, global_map = split_pixel_and_global(tokens)  # [§3.4]
    # Offsets and weights from each token and its position  [§3.4] [eq. 5-6]
    token_input = tokens + position_encoding(tokens)
    offsets = linear(token_input)     # (N, M, K, 2) per pixel and global part
    weights = linear(token_input)     # (N, M, K) per pixel and global part
    # Weights are normalised over the K points  [§3.4]
    weights = weights / sum_over_points(weights)
    # One shared linear layer gives the sampled values  [§3.4] [eq. 7]
    value_pixels, value_global = shared_linear(pixel_map, global_map)
    outputs = []
    for query in tokens:
        head_outputs = []
        for head in range(M):
            # Weighted bilinear samples of both maps  [§3.4] [eq. 8]
            head_outputs.append(sample_and_sum(head, query))
        # Concatenate heads, project, add the query as residual  [§3.4] [eq. 9]
        outputs.append(linear(query + concat(head_outputs)))
    return stack(outputs)

# Decoder forward  [§3.5] [eq. 12-19]
# encoder_out: encoder tokens; anchors: one per pixel
def decoder_forward(encoder_out, anchors):
    # Candidate box and class scores from each encoder token  [§3.5] [eq. 12-13]
    boxes = sigmoid(box_head_0(encoder_out) + inverse_sigmoid(anchors))
    scores = sigmoid(class_head_0(encoder_out))   # (H*W, C)
    # Keep the top Q_match candidates by their best class score  [§3.5]
    chosen = topk(max_over_classes(scores), Q_match)
    queries = ones(Q_match, C)            # all-one content queries  [§3.5]
    boxes = boxes[chosen]
    # Assumed: b_hat of layer 0 is b_0, not stated  [§3.5]
    previous_hat = boxes
    outputs = []
    for layer in range(D):
        # Cross-attention only, no attention between queries  [§3.5] [eq. 14-15]
        queries = decoder_layer(queries, encoder_out, boxes)
        # Box refinement, layer d  [§3.5] [eq. 16]
        hat = sigmoid(box_head_d(queries) + inverse_sigmoid(boxes))
        # Loss box: uses the previous layer's un-detached box  [§3.5] [eq. 18]
        loss_box = sigmoid(box_head_d(queries) + inverse_sigmoid(previous_hat))
        # Next layer starts from a detached copy  [§3.5] [eq. 17]
        boxes = detach(hat)
        previous_hat = hat
        # Class scores for the loss box  [§3.5] [eq. 19]
        outputs.append((loss_box, sigmoid(class_head_d(queries))))
    return outputs

# Hybrid one-to-many assignment  [§3.6] [supp. Hybrid Label Assigner]
# gts: (G, 4) boxes; preds: (Q, 4) boxes; cost: (Q, G) forced-matching cost
def hybrid_assign(gts, preds, cost, T, iou_min):
    # Forced matching: one Hungarian pair per ground truth  [§3.6]
    positives = hungarian_pairs(cost)
    for gt in gts:
        # Predictions with IoU above the threshold  [§3.6] [supp. alg.]
        candidates = [p for p in preds if iou(p, gt) > iou_min]
        # Keep at most T of them, highest IoU first  [§3.6] [§5.1.1]
        positives += top_by_iou(candidates, T)
    return positives

# Centre-shifting contrastive denoising, one ground-truth box  [§3.7]
# box = (x, y, w, h); tau1 = 0.5, tau2 = 1.5  [§5.1.1]
def ccdn_noise(box, is_positive, tau1, tau2):
    x, y, w, h = box
    if is_positive:
        # Centre shift inside +-0.5 * tau1 of the size  [§3.7]
        shift_x = uniform(-0.5 * tau1 * w, 0.5 * tau1 * w)
        shift_y = uniform(-0.5 * tau1 * h, 0.5 * tau1 * h)
    else:
        # Centre shift between 0.5 and 1 times tau1 of the size  [§3.7]
        shift_x = random_sign() * uniform(0.5 * tau1 * w, tau1 * w)
        shift_y = random_sign() * uniform(0.5 * tau1 * h, tau1 * h)
    # Width and height resampled around the size  [§3.7] (see Paper vs code)
    new_w = uniform(min(0, w - tau2 * w), w + tau2 * w)
    new_h = uniform(min(0, h - tau2 * h), h + tau2 * h)
    return (x + shift_x, y + shift_y, new_w, new_h)

Hyperparameters

The paper's own values come first. Where the paper is silent, the official code at the commit above gives a value, which is marked as code. Config paths are relative to the repository root. Values with [supp. ...] come from the supplement, cited by section title because its numbering is not checked against the compiled PDF.

Datasets

SPOD (simulated). Built from AVIRIS flight lines over California and Nevada. One flight line gives 150 training background images and another gives 500 test backgrounds, all 128 × 128 §4.2. The paper says 150 bands are kept; the listed band IDs give 151 bands (see Pitfalls). Averaging every 5 adjacent bands leaves 30 bands §4.2. Each image gets between 1 and 20 synthetic objects, and each object is placed with a template that combines endmember spectra from 12 USGS reflectance curves §4.2 tab. 1. Eight object classes are defined, with single, hybrid and combined templates §4.2 tab. 1.

Table 1. Object simulation setup on SPOD §4.2:

Mixed pixels, those next to background, get an abundance drawn at random and assigned by distance to the object centre §4.2. If every pixel of an object is mixed, the centre pixel gets a value from the maximum-abundance range §4.2. The simulation maps reflectance to radiance by a seawater-based linear scale, with a multiplicative factor M_t in [2000, 3000] and a wide-area fluctuation factor b drawn from [−0.3, 0.3] [supp. Supplements to Implementation Details]. Mixing follows the linear mixing model §4.2 [supp. Supplements to Implementation Details].

Avon (real). A 128 × 128 test crop from one flight, with 24 real objects: 12 blue and 12 brown tarps. Bands are 360, spanning 400 to 2450 nm at 5 nm resolution §4.3. Labels were drawn by hand from two detection maps, then converted to boxes by 8-connected components §4.1 §4.3. The training set is 150 simulated images from another flight, each with 10 blue and 10 brown simulated tarps §4.3. Four other flights of the same area test robustness to lighting §5.3.1.

SanDiego (real). A 128 × 128 crop from a 400 × 400 AVIRIS image of an airport with 3 aircraft, 180 bands, 3.5 m resolution §4.4. Training is 100 simulated images with 10 aircraft each, using one prior spectrum per class §4.4. The test labels come from a mask in the literature §4.4 §4.6.1.

Gulfport (real). Five 128 × 128 test crops from flight 3 of a CASI-1500 sensor, 72 bands between 367.7 and 1043.4 nm, 1 m resolution. There are 57 cloth panels in four colours §4.5. Confidence levels were given by human annotators: visible 9, probably visible 7, possibly visible 8, not visible 33 §4.5. Training uses 200 simulated images with five panels of each type, built from flight 4 §4.5 [supp. Supplements to Implementation Details].

Evaluation. On SPOD and Avon, the COCO protocol gives AP averaged over IoU 0.5–0.95, mAR, and AP at IoU 0.25 (mAP25, mRe25). Each image keeps the top 100 predictions §4.6.1. For SanDiego a prediction is correct if its IoU with the box is above 0.25. For Gulfport the correct area is a 5 × 5 inner box and a 9 × 9 outer box around the GPS point §4.6.1. For HTD methods, normalised score maps are cut at the threshold that maximises IoU, chosen per class with steps of 0.01, and then converted to boxes §4.6.2. The paper also reports mAUC and mIoU for them §4.6.2.

Results

Author-reported. Setup: the simulated SPOD benchmark at native 128 × 128, 30 bands, 100 epochs, batch 4 on an RTX 3090 §5.1.1. Visual detectors receive the cube upsampled ×4 by pixel replication §5.1.1.

Visual detectors on SPOD tab. 2

Among these rows, SpecDETR has the highest mAP, mAP25 and per-class AP for all eight classes. DINO with Swin-L has the highest mAR and mRe25 tab. 2.

Computational cost tab. 3

HTD methods on SPOD tab. 4

The HTD rows are the adapted baselines of §5.1.1. Most HTD methods take prior spectra and need no training. LSSA was changed to output all classes at once, and IRN and TSTTD were trained on the training images §5.1.1 [supp. Experimental Setup for Compared HTD Methods]. mAUC is a pixel-level score, and it does not track the object-level columns (mAP, mAP25).

Avon tab. 5

The Avon test set is a single image. Validating SpecDETR on the simulated Avon training images gives 0.979 mAP and 0.998 mAP25, against 0.885 and 0.990 on the real test image [supp. Generalization Analysis]. The gap is the only held-out evidence the paper gives for Avon.

SanDiego and Gulfport tab. 6

B., D.G., P.G. and F.V.G. are brown, dark green, pea green and faux vineyard green panels tab. 6. The training epochs are 12 for SanDiego and 24 for Gulfport, with the learning rate reduced at epoch 10 and 20 respectively §5.2.

Gulfport detections under fixed confidence thresholds, SpecDETR only tab. 7:

Num. is the number of predictions after confidence filtering, and TP is the number of true positives. Pr is TP divided by Num., and Re is TP divided by the 57 panels tab. 7.

Robustness on Avon tab. 8, tab. 9

Lighting. Four other flights of the same area, imaged at different times §5.3.1:

Additive Gaussian white noise on the test image, with no retraining. The noise level is defined by the per-band signal-to-noise ratio §5.3.2:

Ablations on SPOD tab. 10 to tab. 14

Module combinations tab. 10:

Encoder and decoder depth, at 36 epochs tab. 11:

Assignment and denoising, mAP at 36 and 100 epochs tab. 12:

S2A variants tab. 13. In the second row, the structure exists but does not do its job:

Content query initialisation tab. 14:

The supplement also tests generalisation on SPOD. Training on the 100 training images and testing on the 500 test images gives mAP 0.856; training on the test images and validating on the training images gives 0.922 [supp. Generalization Analysis]. In a single-class comparison with sparse-representation HTD methods, SpecDETR reaches AP 0.907 and AP25 0.989. The best sparse baseline, KSRBBH, reaches AP 0.073 and AP25 0.341 [supp. Sparse Representation-Based HTD Methods].

Paper vs code

Official repository: https://github.com/ZhaoxuLi123/SpecDETR at commit 5a13a63c7274422c9b30f7e82ac6e5d22fe83d4c, Apache-2.0. The abstract links this repository, so it is treated as the official code. The code is an mmdet fork, and its head docstring says it is modified from DINO specdetr_head.py.

Differences (paper first, then code):

• Learning rate. Paper: initial 0.001, "consistent with DINO" §5.1.1. Code: AdamW at 0.0001 in every dataset config SpecDETR_SPOD_100e.py SpecDETR_Avon_36e.py.
• Centre-shift and box-scale noise. Centre shifts match the paper: ±0.25 w for positives with τ₁ = 0.5 §3.7 specdetr_layers.py specdetr_layers.py. Box scaling does not: the paper gives width in [min(0, w − τ₂w), w + τ₂w] §3.7, but the code adds uniform noise on [−τ₂, τ₂] times w/2, so width moves by at most 0.75 w specdetr_layers.py.
• Global token. Paper: g₀ is the mean of the pixel tokens §3.3 eq. 2. Code: a second 1 × 1 map is built as the spatial mean of the first feature map and detached from the graph st_nobackbone.py. The gradient of the global token therefore does not flow back to the input projection through this path.
• Dynamic matching. Paper: for each ground-truth box, take the top-T predictions by IoU among those above the threshold §3.6 [supp. Hybrid Label Assigner]. Code: repeated Hungarian matching on the cost 1 − IoU, masking matched predictions each round, stopping early if a round finds nothing above the threshold dynamic_iou_hungarian_assigner.py. The code also caps all matches per image at 100 dynamic_iou_hungarian_assigner.py dynamic_iou_hungarian_assigner.py.
• Forced-matching weights. Paper's supplement algorithm sums the three costs with unit weights [supp. Hybrid Label Assigner]. Code weights them 2 (focal), 5 (L1) and 2 (GIoU) SpecDETR_SPOD_100e.py.
• Number threshold. Paper: T = 9 dynamic positives §5.1.1. Code: match_num=10, which means forced matching plus nine dynamic rounds SpecDETR_SPOD_100e.py. The two agree only if the forced match is not counted.
• Look-forward-twice. Paper: Eq. 16–18 §3.5. Code: no un-commented detach on the box update was found in the active head SpecDetrHead specdetr_head.py. The detector detaches only the top-k coordinates specdetr.py. Not confirmed; verify before relying on it.
• Top-Q selection. Paper: "highest confidences" §3.5, without saying how to reduce the class vector. Code: the maximum over classes is used specdetr.py.
• Query self-attention. Paper: removed §3.5. Code: removed; the self-attention lines are commented out specdetr_layers.py specdetr_layers.py. Matches.
• Denoising pairs. Paper: 200 pairs §5.1.1. Code: num_dn_queries=200 divided by the maximum number of targets in the batch, which gives about 200 pairs per batch SpecDETR_SPOD_100e.py specdetr_layers.py. Matches in spirit.
• Random flip. Paper: not mentioned. Code: horizontal flip with probability 0.5 in the training pipeline [code: configs/base/datasets/hsi_detection.py#L24].
• NMS. Paper: IoU 0.01 §5.1.1. Code: the same threshold, class-agnostic by default specdetr_head.py. NMS runs only when use_nms=True; the base class default is False detr_head.py. All dataset configs set it SpecDETR_SPOD_100e.py.
• Normalisation. Paper: V = 3000 (SPOD), 5000 (Avon) §3.3. Code: the same constants [code: configs/base/datasets/hsi_detection.py#L17] [code: configs/base/datasets/hsi_detection_Avon.py#L19], applied by division hsi_load.py.
• Rounding. Paper: Avon boxes are rounded to integers §5.1.1. Code: rounding is on for Avon, SanDiego and Gulfport and off for SPOD SpecDETR_Avon_36e.py SpecDETR_SPOD_100e.py.
• Epochs and milestones. Paper: 100 / 36 / 12 / 24 epochs §5.1.1 §5.2. Code matches SpecDETR_SPOD_100e.py SpecDETR_Avon_36e.py SpecDETR_Sandiego_12e.py SpecDETR_MUUFLGulfport_24e.py.
• Encoder depth. Paper: six layers §5.4.2. Code: six SpecDETR_SPOD_100e.py.
• Band counts. Paper: 30 SPOD, 360 Avon, 180 SanDiego, 72 Gulfport §4.2-§4.5. Code matches SpecDETR_SPOD_100e.py.
• Simulator. The SPOD simulator and the dataset generator are not in the repository. Searching the Python sources for endmember, USGS and simulation terms found nothing relevant. The training data path is hard-coded to a local directory [code: configs/base/datasets/hsi_detection.py#L3].
• Inherited files. The 24- and 36-epoch SPOD configs inherit specdetr_sb-2s-100e_hsi.py SpecDETR_SPOD_24e.py SpecDETR_SPOD_36e.py, which is not in the repository. They cannot be loaded as shipped.
• Self-excited initialisation. Paper: 2k × 2k square offsets, with the first sample of head k inside the active region §3.4. The code uses the attention library's default initialisation, which was not checked here.

Pitfalls

Internal contradictions in the paper. Where the paper's tables and text disagree, use the tables.

• The SanDiego text gives 0.750 for CentripetalNet and 0.776 for CornerNet. Table 6 gives the reverse tab. 6.
• The text attributes 0.626 mAP to CentripetalNet and 0.691 to CornerNet. Table 2 gives 0.695 for CentripetalNet and 0.626 for CornerNet tab. 2 §5.1.2.
• The text says the decoder change cuts parameters by 10.6%. From Table 10, 17.9 M to 16.1 M is about 10.1% tab. 10 §5.4.1.
• The band IDs listed in §4.2 cover 151 bands, not 150. Averaging groups of 5 only gives 30 bands from 150 §4.2.
• The mixed-pixel abundance range in §4.2 is 0.01–1, but the supplement gives 0.1–1 for Avon §4.2 [supp. Generation of Simulated Training Data].
• The Gulfport maximum-abundance range is 0.3–1 in the supplement text and 0.7–1 in its table [supp. Generation of Simulated Training Data] [supp. parameter table].
• The SPOD training background set has 150 images in §4.2 but 100 in the supplement's generalisation study §4.2 [supp. Generalization Analysis].
• The lighting text says "mRe25 of 1.0 and above 0.97"; Table 8 has mRe25 = 1.000 everywhere, while 0.974–1.000 is the mAP25 range §5.3.1 tab. 8.
• The Gulfport text prints "Re of 0.26.3", which is a typo for 0.263 §5.2 tab. 7.
• Table and figure numbers here follow the TeX order and are not checked against the compiled PDF tab. 2 fig. 3 fig. 6.

Hidden costs and fragile steps.

• Top-Q selection is ambiguous in the paper and clear only in the code (maximum over classes) §3.5 specdetr.py.
• The first decoder layer needs the box before refinement, b̂₀. The paper never defines b̂₀; the card takes it as b₀ §3.5 eq. 18.
• The supplement's assigner does not exclude predictions already matched in forced matching or matched to another ground truth, so one prediction can become positive for two boxes. The code masks matched rows instead (see Paper vs code) [supp. Hybrid Label Assigner of SpecDETR].
• The paper's width range is literal. With τ₂ = 1.5, min(0, w − τ₂w) = −0.5w, which is negative for any w > 0, so a literal implementation can draw negative widths §3.7. The code does not use this range (see Paper vs code). It clamps the corners to [0, 1] specdetr_layers.py, which can still give zero-width boxes.
• The number of denoising pairs follows from the largest number of objects in the batch. A batch with a few large objects gets more groups than the paper's 200 suggests specdetr_layers.py.
• NMS is applied after keeping the top 300 (query, class) pairs, so the final count can be well below 300 specdetr_head.py specdetr_head.py.
• Full resolution means many tokens. The encoder runs over H × W tokens, and FLOPs grow almost linearly with encoder depth §5.4.2 tab. 11.
• Speed is the weak point. SpecDETR runs at 24.7 FPS, against 40.3 FPS for Faster R-CNN with ResNet50, and 139.7 GFLOPs against 68.8 tab. 3 §5.1.2.

Evaluation caveats.

• The HTD baselines get a per-class threshold chosen on the test labels to maximise IoU §4.6.2. That makes the HTD numbers optimistic. In the supplement, ASD's mAP changes strongly with the number of pixels kept per object [supp. Discussion on the Conversion of Detection Score Maps].
• The SanDiego and Gulfport predictions have low confidence, mostly below 0.5, which the authors attribute to the gap between simulated training and real test images §5.2. Gulfport recall depends strongly on the confidence cut-off tab. 7.
• Avon's test set is one image with 24 objects, so small differences between methods say little §4.3.
• Visual baselines run at ×4 input and SpecDETR at ×1, so the FLOPs and FPS columns compare different input sizes §5.1.1 tab. 3.
• Robustness results use Gaussian noise added to the test image only, with no retraining §5.3.2.
• Every result is author-reported. The tables give single values without error bars.

Reproduction gaps. The official code can train SpecDETR on the SPOD training set only after a simulator is written. The 24- and 36-epoch SPOD configs need the missing base file. For the learning rate, use the code's 1e-4 unless the authors confirm the paper's 1e-3.

Lineage

• Builds on: DETR for set prediction and Hungarian matching §3.6. DINO (arXiv:2203.03605, no card yet) for the decoder, two-stage design, look-forward-twice and contrastive denoising; SpecDETR keeps these and removes the backbone §3.2 §3.5 specdetr_head.py. Deformable DETR, cited for deformable attention and its eight-head default §3.4. DN-DETR's centre-shift queries, which CCDN adapts §3.7.
• Compared against: visual detectors Faster R-CNN, TOOD, CentripetalNet, CornerNet, RepPoints, RetinaNet, Deformable DETR and DINO §5.1.1 tab. 2 tab. 3. HTD methods ASD, CEM, OSP, KOSP, SMF, KSMF, TCIMF, CR, KSR, KSRBBH, LSSA, IRN and TSTTD §5.1.1 tab. 4 tab. 5. Also DN, CDN and different assignment rules in ablation tab. 12.
• Superseded by: not known from the content of this wiki.