Label assignment

Overview

Training a detector needs a rule that says, for each candidate prediction, whether it is a positive (it should predict an object, with its class and box) or a negative (it should predict background). This rule is the label assignment. Many predictions can fit one object, and the choice shapes both what the model learns and what it outputs at inference 2211.12860 §1.

The papers on this page fall into these families:

• One-to-one: each ground-truth box supervises exactly one prediction (query), chosen by set matching. Duplicates are never trained, so the output needs no non-maximum suppression (NMS) 2211.12860 §1. A restricted variant limits which locations may match 2011.10881 §4.1.
• One-to-many: a rule marks several positions as positive for the same object. The model gets more feedback per object, but its output contains duplicates that must be filtered afterwards 2211.12860 §1, tab. 1.
• Hybrid: one-to-one stays the inference branch, and a one-to-many branch is added during training only 2207.13080 §3.2 2211.12860 §1 2405.14458 §3.1.
• Dynamic, per object: later YOLO versions use assignment rules that adapt to each object, such as SimOTA and AlignOTA 2304.00501 §12 2304.00501 §15. RTMDet also picks the number of positives per object dynamically, following SimOTA, and scores matches with the IoU as a soft label 2212.07784 §3.3.
• Scored by distance: positives are picked by a distance between Gaussian models of the feature point and the box, not by IoU 2208.08738 §3.2.
• Separate per task: on one-stage detectors, classification and regression each select their own positives by a cost that mixes class probability and IoU 2107.02963 §4.1.
• Seed selection by distance: a point-cloud detector picks positive seed points by their distance to the box centre 2104.00678 §3.1.

The DETR-family papers argue that one-to-one feedback is sparse. Deformable-DETR selects fewer than 30 of its 300 queries as positives per COCO image 2207.13080 §1. The Co-DETR paper reports that the encoder features separate foreground from background less well, and that the decoder learns slowly with few positive queries 2211.12860 §1, §3.4.

Variants

One-to-one set matching (DETR)

Each ground-truth box is paired with one query by a matching step, and only that pair is supervised. Matching is unstable during training: the ground truth assigned to a query can change between steps 2211.12860 §3.4. The matching algorithm itself is covered in Bipartite matching.

TSP restricts the matching. A location may be matched to an object only if it lies inside that object's box at a suitable pyramid level, as in FCOS. The paper then runs Hungarian matching over the allowed pairs [arxiv:2011.10881 §4.1] 2011.10881 eq. 2. TSP-FCOS selects 700 locations with a classifier trained by FCOS's rule and runs a six-layer transformer encoder over them. TSP-RCNN takes the top 700 RPN proposals instead [arxiv:2011.10881 §4.1, §4.2, §5.2]. The paper reports 43.1 and 43.8 AP at R-50 with 36 epochs, against 42.0 for DETR after 500 epochs 2011.10881 tab. 1. Removing the set loss from TSP-RCNN gives 42.7 AP against 43.8 2011.10881 tab. 2. The released code does not run Hungarian matching; see Contradictions. Card: Rethinking Transformer-based Set Prediction for Object Detection.

One-to-many rule-based assignment (ATSS, Faster R-CNN, RetinaNet, FCOS)

Classic detectors assign positives with a fixed rule per head. The Co-DETR paper tabulates the rules 2211.12860 tab. 1:

These heads have no matching step, and they produce many positives per object. The Co-DETR card reports 18.7 positives per ground-truth box on COCO for Faster R-CNN and 8.8 for ATSS 2211.12860 app. A. The original papers for these heads are not covered by a card here, so their details are not described on this page beyond the rules above.

A motion planner uses a similar rule for its anchor classifier. An anchor is positive when its best ground-truth IoU is above 0.4. Boxes that no anchor reaches fall back to their nearest neighbour, and hard negatives are mined 2101.06679 §3.3. Its planning loss is a positive-negative split too: the human-driven trajectory is the positive, and sampled trajectories are negatives 2101.06679 §3.3. Card: End-to-end Interpretable Neural Motion Planner.

Dynamic label assignment in YOLO detectors (YOLOX, YOLOv6, YOLOv7, DAMO-YOLO)

The YOLO survey names the assignment rule of several later versions. Its text gives names and one-line descriptions, not equations, so the details are not described here:

• YOLOX: an anchor-free head, with several positives taken from the central 3×3 area, and SimOTA for label assignment 2304.00501 §12.
• YOLOv6: TOOD-style task-aligned label assignment 2304.00501 §13.
• YOLOv7: a coarse auxiliary head and a fine lead head, each with its own assignment; the survey does not state the exact rule 2304.00501 §14.
• DAMO-YOLO: AlignOTA, which uses IoU as a soft label 2304.00501 §15.

The survey notes that anchors were dropped after YOLOX 2304.00501 §20. Card: A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS.

Dynamic soft-label assignment with SimOTA-style k (RTMDet)

RTMDet scores each training prior against each ground truth with a three-part cost. The classification term uses the IoU between prediction and ground truth as a soft target, the regression term is the negative log of the IoU, and a centre prior is added. The weights are 1, 3 and 1 2212.07784 §3.3 2212.07784 eq. 1-4. The number of positives per ground truth is chosen from the costs, following SimOTA 2212.07784 §3.3.

Ablation on ResNet-50, 1x: the baseline reaches 39.9 AP. Adding the soft classification cost gives 40.3, the soft centre prior 40.8, and the log-IoU regression cost 41.3 2212.07784 tab. 6. In the same table, ATSS gives 39.2, PAA 40.4, OTA 40.7 and TOOD 40.7, against 41.3 for the paper's assignment 2212.07784 tab. 6. On RTMDet-s, SimOTA gives 43.2 AP and the paper's assignment 44.5 2212.07784 tab. 6. Card: RTMDet: An Empirical Study of Designing Real-Time Object Detectors.

Hybrid training with auxiliary heads (Co-DETR)

Co-DETR keeps the one-to-one DETR branch and adds one or two one-to-many heads during training only. Their positives are fed back to the shared decoder as extra query groups, each already paired with its object, so no matching runs for them 2211.12860 §3.2, §3.3. The auxiliary parts are dropped at inference, so the deployed model is the original DETR variant 2211.12860 §1. Card: Co-DETR, DETRs with Collaborative Hybrid Assignments Training.

Reported effect on Deformable-DETR R50, 12 epochs, COCO val: AP goes from 37.1 (no auxiliary head) to 42.3 with ATSS only and to 42.9 with ATSS and Faster R-CNN 2211.12860 tab. 2. Ablations on 36 epochs show that the query groups and the auxiliary head each add accuracy 2211.12860 tab. 9.

Hybrid training with duplicated query groups (H-DETR)

H-DETR adds a second, training-only query group. Each ground truth is repeated K times, so several queries are paired with it, and masked self-attention keeps the two groups apart. Only the one-to-one group runs at inference 2207.13080 §3.2. Card: DETRs with Hybrid Matching.

COCO defaults are 300 one-to-one queries, 1500 one-to-many queries, K = 6 and loss weight λ = 1.0 2207.13080 tab. 14. Author-reported gain on Deformable-DETR R50, 12 epochs: 47.0 to 48.7 AP 2207.13080 tab. 1. The one-to-many branch alone reaches 13.5 AP without NMS, so it has to be dropped at test time 2207.13080 tab. 13. Ablations on the same setting: the number of one-to-many queries T gives 47.8 AP at T = 300 and 48.7 at T = 1500 2207.13080 tab. 10. The loss weight gives 48.4 at λ = 0.5 against 47.8 at λ = 0.1 2207.13080 tab. 16.

The one-to-many loss is normalised by the K-repeated ground-truth count in the code, so each term is divided by K times the number of objects. The paper does not state the normaliser 2207.13080 §3.2 deformable_detr.py. The paper also describes two alternatives that it only compares: alternating loss types across epochs, and splitting decoder layers between the two assignments. Neither variant is in the released repository [arxiv:2207.13080 §3.2, §4.2].

Hybrid training with a one-to-one head and a top-k one-to-many head (YOLOv10)

YOLOv10 trains two heads with the same structure on one shared backbone and neck. Both heads score prediction-instance pairs with one metric, m = s · p^α · IoU^β, where s is 1 when the anchor point lies inside the object 2405.14458 §3.1 2405.14458 eq. 1. The one-to-one head takes the top-1 positive per object, which the paper says gives the same result as Hungarian matching with less training time 2405.14458 §3.1. The one-to-many head takes several positives per object. The code uses top-10, which the paper does not state loss.py. Both branches use α = 0.5 and β = 6.0 in the reported setting 2405.14458 §4.3. At inference only the one-to-one head runs, so NMS is not needed 2405.14458 §3.1.

On the small model S, training with the one-to-one head alone gives 43.4 AP at 2.44 ms. Adding the one-to-many head gives 44.3 AP at the same latency, so the extra supervision is worth about 0.9 AP 2405.14458 tab. 3. Card: YOLOv10: Real-Time End-to-End Object Detection.

Receptive-field assignment for tiny objects (RFLA)

RFLA replaces the IoU or centre rules with a distance between Gaussian models. Each feature point gets a Gaussian whose radius is half its theoretical receptive field. Each ground-truth box gets a Gaussian centred on the box with the half-sides as standard deviations 2208.08738 §3.1 [arxiv:2208.08738 §3.2]. The distance is either Wasserstein-2 or KL divergence, mapped to a score in (0, 1] 2208.08738 §3.2. Positives are chosen in two stages 2208.08738 §3.3:

1. Each ground truth takes its top-k points by score (k = 3 in the card).
2. The receptive fields shrink by β (0.9), the scores are recomputed, and each ground truth takes its top-1 point. Stage 1 keeps priority where it assigned a point.

The paper applies RFLA to Faster R-CNN and FCOS 2208.08738 §3.4. The released code changes only the Faster R-CNN RPN; see Contradictions. Author-reported on AI-TOD, whose average object is 12.8 px: Faster R-CNN goes from 11.1 to 21.1 AP, and DetectoRS from 14.8 to 24.8 2208.08738 tab. 5. On TinyPerson the gains are smaller: Faster R-CNN goes from 48.7 to 50.1 AP50 for tiny objects, and FCOS from 23.4 to 26.5 2208.08738 tab. 6. KL and Wasserstein tie at 21.1 AP, and KL is the default 2208.08738 tab. 1. Card: RFLA: Gaussian Receptive Field based Label Assignment for Tiny Object Detection.

Separate cost-based assignment per task (DDOD)

DDOD keeps a one-stage dense detector (RetinaNet, FCOS or ATSS) and changes only the training targets. Classification and regression each run their own assignment. For each prediction and ground truth, the cost is the class probability raised to (1 − α) times the IoU raised to α. Candidates must have their centre inside the ground-truth box 2107.02963 §4.1 eq. 1. The paper uses α = 0.8 for classification and α = 0.5 for regression 2107.02963 §5.4.1. With α = 0.8 the IoU term carries the larger exponent, so the classification cost is mostly IoU, although §4.1 describes higher foreground probability as the priority; the paper's own ablation favours IoU for classification 2107.02963 §5.4.1. In the paper's grid over the two values, α = 0.8 for classification and 0.5 for regression gives the best ATSS result, 40.4 AP 2107.02963 tab. 5.

The paper adds two changes beside the assignment: a deformable first layer in each head branch, and per-level loss weights from positive counts 2107.02963 §4.2 2107.02963 §4.3. On the ATSS baseline (39.4 AP), each change alone gives 40.4 (assignment), 40.6 (deformable layer) and 40.0 (pyramid weights). All three together give 41.6 2107.02963 tab. 4. Author-reported gains are +2.0 AP on RetinaNet, +2.4 on FCOS and +2.2 on ATSS, on COCO val with ResNet-50 at 1x 2107.02963 tab. 3. Card: Disentangle Your Dense Object Detector.

Positive seeds by distance in 3D (Group-Free)

Group-Free 3D does not group points to candidates. It still needs positive seed points for its sampler loss. The paper calls a seed positive when it lies inside a ground-truth box and is among the k points closest to the box centre, with k = 4 in the reported setting 2104.00678 §3.1 [arxiv:2104.00678 tab. 4]. Card: Group-Free 3D Object Detection via Transformers.

When to use which

Dropping NMS costs little in both ablations, and the sign depends on the schedule. H-DETR's one-to-many-only training with NMS gives 49.4, 50.2 and 48.8 AP at 12, 24 and 36 epochs. Its one-to-one branch without NMS gives 48.7, 49.9 and 50.0 2207.13080 tab. 13. YOLOv10's one-to-many-only row, with NMS, gives 44.9 AP on S against 44.3 for NMS-free dual training 2405.14458 tab. 3.

Memory also grows with the number of heads: 12808 MB without auxiliary heads against 14387 MB with two 2211.12860 tab. 7. Each number comes from one paper's code base, so compare them with your own runs before you rely on them.

Implementation sketch

Hybrid training, as in the Co-DETR paper. The one-to-one loss is the usual matching loss. Each auxiliary head adds its own one-to-many loss and a set of extra queries. Lines are shortened from the paper's pseudocode.

# One-to-one branch: Hungarian matching per decoder layer  [eq. 6]
def main_loss(decoder_layers, gt):
    return sum(hungarian_loss(p, gt) for p in decoder_layers)

# One-to-many branch: rule-based positives per auxiliary head  [eq. 1-3]
def aux_loss(head, pyramid, gt, lambda_2, num_layers):
    preds = head(pyramid)
    pos, pos_boxes, pos_targets, neg = head.assign(preds, gt)
    return num_layers * lambda_2 * head.loss(preds, pos, neg)

# Extra queries from the positives, decoded without matching  [eq. 4-5]
def aux_queries_loss(decoder, memory, pyramid, pos, pos_boxes, pos_targets):
    queries = embed(pos_boxes) + project(gather(pyramid, pos))
    return sum(detr_loss(p, pos_targets) for p in decoder(queries, memory))

# Inference: only main_loss's branch runs; auxiliary heads are dropped  [§1]

Where variants differ: the positive rule in head.assign (table above), the weights lambda_1 and lambda_2 (1.0 and 2.0 in the Co-DETR paper's main setting) 2211.12860 §4.1, and how many heads are used. For YOLOv10 the variant differs in the top-k of each branch (1 for one-to-one, 10 for one-to-many in the code) and in the shared α and β 2405.14458 §3.1 loss.py.

The RFLA paper's assignment for one image, shortened and in our own wording:

# RFLA assignment for one image  [§3.2] [§3.3]
def assign_rfla(points, erf_radius, gts, k=3, beta=0.9):
    # Score each point against each gt by receptive-field distance  [eq. 8]
    scores = rfd_matrix(points, erf_radius, gts)             # (G, N)
    # Stage 1: each gt takes its top-k points  [§3.3]
    stage1 = take_top_k_per_gt(scores, k)                    # (N,)
    # Stage 2: shrink receptive fields by beta, score again, take top-1  [§3.3]
    shrunk = rfd_matrix(points, beta * erf_radius, gts)      # (G, N)
    stage2 = take_top_k_per_gt(shrunk, 1)                    # (N,)
    # Stage 1 wins where it assigned a point  [eq. 9]
    return where(stage1 > 0, stage1, stage2)                 # (N,)

Contradictions

Results across papers are not directly comparable. The baselines differ, and each paper uses its own setting.

• Deformable-DETR baselines differ. The Co-DETR paper reports 37.1 AP for plain Deformable-DETR R50 at 12 epochs 2211.12860 tab. 2. On its Deformable-DETR++ R50 baseline at 12 epochs it reports 47.1 with no auxiliary head, 48.7 with one ATSS head and 49.5 with ATSS and Faster R-CNN 2211.12860 tab. 3. The H-DETR paper reports 47.0 for its Deformable-DETR R50 baseline at 12 epochs, which uses two-stage, box-refinement and mixed query selection, and 48.7 with its one-to-many branch 2207.13080 tab. 1 r50_hybrid_branch_lambda1_group6_t1500_dp0_mqs_lft_deformable_detr_plus_iterative_bbox_refinement_plus_plus_two_stage.sh. On the ++ baseline the gains are +2.4 for Co-DETR with K = 2 and +1.7 for H-DETR. The auxiliary heads and the baseline setup differ, so these are not a like-for-like comparison.
• TSP's matching. The paper defines the matching as Hungarian matching 2011.10881 eq. 2. The released scripts instead take, for each object, the proposal with the lowest cost, and nothing enforces one proposal per object my_fcos_outputs.py my_fast_rcnn_output.py. Implement one of the two and say which.
• TSP-RCNN and NMS. The one-to-one argument above says the output needs no NMS. The released TSP-RCNN script still applies NMS at IoU 0.7 at test time tsp_rcnn.sh, and the paper frames set prediction as removing NMS 2011.10881 §2.2.
• TSP's encoder drop. The paper says 70% of encoder inputs are randomly dropped during training 2011.10881 §5.2. The released code keeps a fraction sampled between 70% and 100% my_fcos.py tsp_fcos.sh.
• Group-Free positive seeds. The paper requires a seed to lie inside a ground-truth box and ranks by plain distance to the centre [arxiv:2104.00678 §3.1]. The released code divides the offset by the box size on each axis before ranking, and gives seeds outside the box a penalty. A filler seed from another box can therefore stay positive, which breaks the paper's inside condition loss_helper.py.
• DDOD candidate rule. The paper says the top K predictions by cost are kept per FPN level 2107.02963 §4.1. The released code instead keeps the K anchors per level whose centres are nearest the ground-truth centre, then tests only those against the cost threshold [code: mmdet/core/bbox/assigners/atss_cost_assigner.py#L217-L230]. The paper gives no value of K, and the config uses 9 ddod_r50_1x.py. The two rules are not equivalent, so the reported numbers match the code.
• RFLA's Faster R-CNN stage. The paper says RFLA replaces anchor tiling and MaxIoU anchor assignment for Faster R-CNN 2208.08738 §3.4. The released code applies the assigner to the RPN only, and the R-CNN head keeps MaxIoU with positive IoU 0.5 aitod_faster_r50_rfla_kld_1x.py.
• RFLA's ERF radius. The paper sets the ERF radius to half the theoretical receptive field radius 2208.08738 §3.1. The code uses half the base size as the side of the ERF box rf_generator.py. That corresponds to a radius of TRF/4 unless TRF is read as a diameter. The paper does not say which.
• YOLOv10's detached one-to-one head. The paper says the two heads are optimised jointly so that the shared backbone and neck get the one-to-many signal 2405.14458 §3.1. The released code feeds the one-to-one head detached copies of the neck features, so the one-to-one loss does not update the backbone or neck head.py. The paper does not mention this.
• Within the Co-DETR paper. The "no NMS" claim holds for the base method only. The Co-DINO results in table 4 use QFL and NMS [arxiv:2211.12860 §4.3, tab. 4]. Table 3 reports 51.2 AP for DINO-Deformable-DETR (R50, 12 epochs, two auxiliary heads), and table 4 reports 52.1 AP for Co-DINO-Deformable-DETR (R50, 12 epochs) 2211.12860 tab. 3, tab. 4. The two rows differ by model and by post-processing, so the gap between them is not a clean measure of the training scheme.
• RTMDet's centre cost. Eq. 4 is written with a single coordinate difference and no stride 2212.07784 eq. 4. The released assigner uses the Euclidean distance divided by the stride [code: mmdet/models/task_modules/assigners/dynamic_soft_label_assigner.py#L150-L152]. The two forms differ; the equation does not show the stride.
• RTMDet's candidates and k. The released assigner admits only priors whose centre lies inside some ground-truth box; the paper does not state this [code: mmdet/models/task_modules/assigners/dynamic_soft_label_assigner.py#L116-L127]. The paper says k follows SimOTA. The code takes the sum of the top-13 IoUs, truncated to an integer with a minimum of 1 [code: mmdet/models/task_modules/assigners/dynamic_soft_label_assigner.py#L203-L206] rtmdet_l_8xb32-300e_coco.py. The paper does not state the value 13.
• RTMDet's loss weights. The matched IoU is both the classification target and the weight of the regression loss, and both losses are normalised by the summed IoU over the batch rtmdet_head.py. The paper does not describe this, so an implementation that uses hard labels or unweighted boxes will not match the reported behaviour.

Papers

• DETRs with Collaborative Hybrid Assignments Training: trains DETR variants with one-to-many auxiliary heads and extra query groups, and drops them at inference.
• DETRs with Hybrid Matching: adds a training-only one-to-many query group with each ground truth repeated K times, and keeps one-to-one matching at inference.
• YOLOv10: Real-Time End-to-End Object Detection: trains a one-to-many and a one-to-one head with a shared matching metric, and runs only the one-to-one head.
• A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS: a survey that names the assignment rules of YOLOX, YOLOv6, YOLOv7 and DAMO-YOLO, without their equations.
• Rethinking Transformer-based Set Prediction for Object Detection: restricts DETR-style matching to locations inside each object, on a short list of features of interest.
• RFLA: Gaussian Receptive Field based Label Assignment for Tiny Object Detection: scores positives by a Gaussian receptive-field distance and selects them in two stages, for tiny objects.
• End-to-end Interpretable Neural Motion Planner: uses an IoU-threshold anchor rule with a nearest-neighbour fallback, and a max-margin positive-negative split for planning.
• Group-Free 3D Object Detection via Transformers: ranks positive seed points by distance to the box centre for its sampler loss.
• Disentangle Your Dense Object Detector: gives classification and regression separate cost-based assignments on one-stage detectors, with a deformable first layer and per-level loss weights.
• RTMDet: An Empirical Study of Designing Real-Time Object Detectors: uses a dynamic soft-label assignment with IoU as the classification target, on a real-time one-stage detector.