Disentangle Your Dense Object Detector

In plain words

A dense object detector looks at every spot of an image and asks two things at each spot: is there an object here, and where exactly is its box. The authors argue that the usual training recipe ties these two jobs together in three ways nobody had tested. Both jobs learn only from the same spots, both read the same features, and every level of the feature pyramid counts the same in the loss, even though the deep, coarse levels that handle large objects get far fewer training spots.

The fix separates the jobs. Each one picks its own training spots, each one learns its own view of the feature map, and the pyramid levels are weighted by how many training spots they receive. These changes slot into existing detectors. The gains the authors report are about two points of average precision on three popular detectors.

TL;DR

DDOD is a training recipe for one-stage detectors (RetinaNet, FCOS and ATSS). It gives classification and regression separate label assignment, puts a deformable convolution at the first layer of each head branch, and weights feature-pyramid levels by their positive-sample counts. Author-reported gains on COCO val are +2.0 to +2.4 mAP across the three detectors; on ATSS the ablation reaches 41.6 mAP with ResNet-50 at 1x. The repository has code for the ATSS and FCOS variants only, and the WIDER FACE model has no config there.

Problem

The authors call three shared assumptions in dense-detector training "conjunctions", which they say are simplifications from engineering practice rather than things that were tested §1:

1. Label assignment. The regression head is trained only on the samples that the classification branch marks positive §3.1.
2. Spatial features. Both heads use the same receptive fields. The authors argue that classification should attend to semantic regions and regression to object contours, and show with Grad-CAM that the two heads' sensitive regions differ. This is a visual check, not a measured result §3.2 fig. 2.
3. Pyramid supervision. Predictions from all levels are flattened and concatenated into one loss. P3 gets about 256 times as many training samples as P7 §1, so large objects on the deep levels are under-trained §3.3.

Pilot evidence from the authors (author-reported):

• On RetinaNet, the best pair of IoU thresholds is 0.5 for classification and 0.4 for regression, giving 36.5 mAP. The matched 0.5/0.5 pair gives 36.1 tab. 1. This is a fixed-threshold test, not the cost-based assignment the method finally uses §3.1 §4.1.
• On the ATSS baseline (39.4 mAP), linearly increasing weights from 1.0 at P3 to 2.0 at P7 give 39.8. Constant weights give 39.4 and the reversed ramp 3.0-to-1.0 gives 37.6 tab. 2 §3.3.

Method

DDOD is a set of three changes to a standard one-stage FPN detector, plus an IoU-prediction branch that replaces the centerness branch §5.1 fig. 3:

• Classification and regression each get their own label assignment, based on a matching cost §4.1.
• The first convolution of each head branch is a deformable convolution, so each branch learns its own sampling positions §4.2.
• The loss of each pyramid level is scaled by a weight that depends on how many positive samples the level has collected so far §4.3.

Label assignment §4.1. For each prediction and each ground truth, the cost is the predicted probability of the ground-truth class raised to the power (1 - alpha), times the IoU between the predicted box and the ground truth, raised to the power alpha eq. 1. Candidates are restricted to predictions whose centres lie inside the ground-truth box. Per ground truth, the paper takes an adaptive threshold from the candidates' costs, and a candidate is positive if its cost reaches it. Classification uses alpha = 0.8 and regression uses alpha = 0.5 §5.4.1. The paper's candidate rule and the code's differ; see Paper vs code.

Deformable first layer §4.2. The paper writes the feature at each position as a weighted sum over sampling points whose offsets are learned eq. 2. In practice the first convolution of the classification tower and of the regression tower is replaced by a deformable convolution. The other convolutions stay regular, so the head keeps the same number of convolutions. Placing the deformable layer first was the best of four positions in the ablation §5.4.2 tab. 6.

Pyramid weights §4.3. For level i, the weight is 2 minus the level's positive count, normalised to 0 to 1 between the smallest and largest counts over all levels eq. 3 eq. 4. The classification and regression counts are kept separately. The total loss uses these weights on the classification term and the regression term eq. 5. Counts are accumulated over all iterations so far, which the paper describes as a moving average. The weight is 1.0 for the level with the most positives (P3 in practice) and 2.0 for the level with the fewest (P7 in practice) §4.3.

IoU branch §5.1. The centerness branch is replaced by a branch that predicts the IoU of each box. Its training target is the IoU between the predicted box and its matched ground truth. At inference the class score is multiplied by the predicted IoU ddod_head.py.

Pseudocode of the method as the code implements it. Comments name the paper section and, where they differ, the code location:

# Cost-based assignment for one head branch  [§4.1] [eq. 1]
# Input:  probs (P, C) class probabilities, boxes (P, 4) decoded boxes,
#         gt_boxes (N, 4), gt_labels (N,), alpha, k, level_sizes
# Output: assigned (P,) gt index per prediction, -1 for background

def assign_labels(probs, boxes, gt_boxes, gt_labels, alpha, k,
                  level_sizes):
    # Matching cost of every prediction for every ground truth  [§4.1]
    # Probability of the gt class, raised to (1 - alpha)
    class_term = probs[:, gt_labels] ** (1 - alpha)      # (P, N)
    # IoU between each predicted box and each gt, raised to alpha
    iou_term = box_iou(boxes, gt_boxes) ** alpha         # (P, N)
    cost = class_term * iou_term                         # (P, N)
    # Candidates: per FPN level, the k anchors whose centres are closest
    # to each gt centre. The paper ranks by cost instead (Paper vs code).
    # [code: mmdet/core/bbox/assigners/atss_cost_assigner.py#L217-L230]
    candidates = []
    for level_slice in split_by_level(level_sizes):
        distances = centre_distance(level_slice, gt_boxes)  # (A_l, N)
        candidates.append(smallest_k(distances, k))         # (k, N)
    candidates = concatenate(candidates)                    # (k*L, N)
    # Adaptive threshold per gt: mean + std of its candidates' costs
    # [§4.1] [code: mmdet/core/bbox/assigners/atss_cost_assigner.py#L234-L239]
    cand_cost = gather(cost, candidates)                    # (k*L, N)
    threshold = mean(cand_cost, axis=0) + std(cand_cost, axis=0)
    # Keep candidates at or above the threshold whose centre lies inside
    # the gt  [code: mmdet/core/bbox/assigners/atss_cost_assigner.py#L239-L257]
    is_positive = cand_cost >= threshold
    is_positive = is_positive & centre_inside(candidates, gt_boxes)
    # A prediction claimed by several gts keeps the one with the highest
    # cost. The code comment says "highest IoU", but the value compared
    # is the cost.
    # [code: mmdet/core/bbox/assigners/atss_cost_assigner.py#L259-L269]
    assigned = resolve_by_max_cost(candidates, is_positive, cost)
    return assigned

# Per-level weights from cumulative positive counts  [§4.3] [eq. 3]
# Fewest positives -> 2.0, most positives -> 1.0, linear in between
# [code: mmdet/models/dense_heads/ddod_head.py#L315-L321]
def level_weights(counts):
    lowest = min(counts)
    highest = max(counts)
    # 1e-10 avoids division by zero when all levels have equal counts
    interval = 1.0 / (highest - lowest + 1e-10)
    return [2.0 - (count - lowest) * interval for count in counts]

# Positives per level, summed over every iteration so far  [§4.3]
# Five levels, P3 to P7  [code: mmdet/models/dense_heads/ddod_head.py#L309-L313]
def add_positives_per_level(assigned_levels, counts):
    for level in range(5):
        counts[level] += number_of_positives(assigned_levels[level])
    return counts

# Training loss: two assignment passes, then per-level weights  [§4.1] [§4.3]
# [code: mmdet/models/dense_heads/ddod_head.py#L285-L400]
# Input:  per level: cls_logits, box_deltas, iou_logits; gt_boxes, gt_labels
#         counts_cls, counts_reg: per-level positive counts (kept across calls)
# Output: loss_cls, loss_box, loss_iou

def training_loss(levels, gt_boxes, gt_labels, counts_cls, counts_reg):
    # Pass 1: classification targets from the classification assignment
    # alpha = 0.8, top k = 9  [§5.4.1] [code: coco_cfg/ddod_r50_1x.py#L48]
    cls_assigned = assign_per_level(levels, gt_boxes, gt_labels, 0.8, 9)
    counts_cls = add_positives_per_level(cls_assigned, counts_cls)
    weights_cls = level_weights(counts_cls)                   # [eq. 3]
    loss_cls = 0
    for level in range(5):
        loss_cls += weights_cls[level] * focal_loss(levels[level],
                                                    cls_assigned[level])
    # Pass 2: box and IoU losses from the regression assignment
    # alpha = 0.5, top k = 9  [§5.4.1] [code: coco_cfg/ddod_r50_1x.py#L49]
    reg_assigned = assign_per_level(levels, gt_boxes, gt_labels, 0.5, 9)
    counts_reg = add_positives_per_level(reg_assigned, counts_reg)
    weights_reg = level_weights(counts_reg)                   # [eq. 4]
    loss_box = 0
    loss_iou = 0
    for level in range(5):
        positives = select_positives(reg_assigned[level])
        # IoU target: overlap of each decoded positive box with its gt,
        # with no gradient through the target  [code: ddod_head.py#L226-L227]
        iou_target = iou(decode(positives.boxes).detach(), positives.gts)
        # GIoU box loss with weight 2.0  [code: coco_cfg/ddod_r50_1x.py#L43]
        loss_box += weights_reg[level] * 2.0 * giou_loss(positives)
        # Cross-entropy on positives only, weight 1.0
        # [code: ddod_head.py#L195] [code: coco_cfg/ddod_r50_1x.py#L44-L45]
        loss_iou += weights_reg[level] * bce(levels[level].iou_logits,
                                             iou_target, positives)
    return loss_cls, loss_box, loss_iou

Hyperparameters

Values from the paper are cited to the paper. Values marked "code" appear only in the official repository at the commit in the frontmatter. "not stated" means neither the paper nor the code gives a value.

Results

Author-reported throughout. Setup: COCO with ResNet-50 and the 1x schedule §5.1; ablations on the ATSS baseline with minval/val-2017 §5.3 tab. 4; FPS on a single Titan V100 with batch size 1 where possible tab. 8; WIDER FACE validation subset tab. 9.

Headline numbers:

• Each detector with DDOD, same backbone, COCO val: RetinaNet 36.5 to 38.5, FCOS 39.2 to 41.6, ATSS 39.4 to 41.6 tab. 3.
• Component ablation on ATSS: 39.4 baseline, 41.6 with all three tab. 4.
• COCO test-dev, single model and single scale: DDOD with ResNet-50 45.0; with Res2Net-101-DCN, 51.2; DDOD-X with Res2Net-101-DCN, 52.5. The 55.0 row adds multi-scale testing and other techniques, which the paper describes only loosely §5.5 tab. 8.
• WIDER FACE validation (easy, medium, hard): ATSS 0.964 / 0.958 / 0.930; DDOD-Face 0.970 / 0.964 / 0.935 tab. 9.

Table 3: three detectors with and without DDOD (COCO val-2017)

Source: tab. 3. FCOS uses the improved implementation in ATSS's codebase, per the table footnote.

Table 4: component ablation on ATSS (COCO val-2017)

Source: tab. 4. Columns: the three components, then AP and its parts.

Table 5: classification and regression alpha (ATSS, COCO val)

Source: tab. 5. Rows: alpha for classification. Columns: alpha for regression.

Table 6: position of the deformable layer in the head

Source: tab. 6. Position 1 is the first conv of each branch.

Table 7: pyramid re-weighting strategies

Source: tab. 7. Sum-Reweight uses weights proportional to the positive count per level. Linear-Reweight is the linear ramp from 1.0 to 2.0 used in QueryDet. Linear-Interpolate is the paper's method. An asterisk marks the moving-average version, which counts all valid samples up to the current iteration.

Table 8: COCO test-dev comparison

Source: tab. 8. Single model and single scale, except the last row. MS_train and MS_test mark multi-scale training and testing. FPS is on a single Titan V100 with batch size 1 where possible; "n/a" means no trained model or timing was available. Backbone abbreviations: R = ResNet, X = ResNeXt, HG = Hourglass, DCN = deformable convolution, R2 = Res2Net. FCOS* uses the improved implementation in ATSS's codebase. The table mixes backbones and schedules, so only rows with the same backbone are directly comparable.

Table 9: WIDER FACE validation subset

Source: tab. 9. The source file has a commented-out MogFace row, which is omitted here; the source footnote says it uses a model ensemble.

Paper vs code

Repository: https://github.com/zehuichen123/DDOD (linked from the abstract), commit ca2852432f4d2ab17cd3d7e90ab872dab31d4a1a. It is a fork of MMDetection under Apache-2.0 LICENSE.

1. Candidate selection. The paper says "top K predictions with the largest cost value are selected from each FPN level" §4.1. The code ranks anchors per level by distance from the anchor centre to the gt centre, keeping the nearest k. Only those candidates are then tested against the cost threshold atss_cost_assigner.py. The paper gives no value of K; the config uses k = 9 for both assigners [code: coco_cfg/ddod_r50_1x.py#L48-L49]. Implement the code's rule to match the reported numbers, or the paper's rule to match the text; the two are not equivalent.
2. Threshold statistics. The paper says the threshold uses "batch statistics like ATSS" §4.1. The code computes mean plus standard deviation per ground truth, over that gt's own candidates [code: mmdet/core/bbox/assigners/atss_cost_assigner.py#L234-L239].
3. Conflicts between ground truths. The code comment says a prediction claimed by several gts keeps the one with the highest IoU, but the value compared is the matching cost [code: mmdet/core/bbox/assigners/atss_cost_assigner.py#L259-L269]. The paper does not describe this case.
4. Level weights. The paper describes the counts as accumulated up to the current iteration, in a moving-average manner §4.3. The code accumulates counts from the first iteration and never decays them [code: mmdet/models/dense_heads/ddod_head.py#L309-L313]. The code also applies the regression weight to the IoU loss as well as to the box loss [code: mmdet/models/dense_heads/ddod_head.py#L245] [code: mmdet/models/dense_heads/ddod_head.py#L377-L389]. The paper's loss equation shows only the classification and box terms.
5. IoU branch. The paper says the IoU branch replaces centerness §5.1. The paper does not give the training target or the inference score. The code trains the IoU branch only on positives, with a target equal to the IoU of each decoded positive box ddod_head.py ddod_head.py, and scores each detection as class probability times the sigmoid of the IoU prediction [code: mmdet/models/dense_heads/ddod_head.py#L540-L582] [code: mmdet/core/post_processing/bbox_nms.py#L57-L62].
6. Deformable layer. The paper says the first convolution of both branches is replaced §4.2. The code does this for the first tower layer only, with one deformable group ddod_head.py. This matches. The offset-prediction layer lives in the mmcv DCN module, which is not part of this repository.
7. Extras not in the paper. The config enables fp16 with a loss scale of 512 ddod_r50_1x.py and a learnable per-level scale on box outputs ddod_head.py. Neither is mentioned in the paper.
8. Missing code. The repository has configs only for ATSS-based DDOD (R-50) and FCOS-based DDOD (R-50) ddod_r50_1x.py [code: coco_cfg/ddod_r50_1x_fcos.py]. There is no RetinaNet variant, no ResNet-101, ResNeXt or Res2Net config for the test-dev rows, and no WIDER FACE DDOD config. Results for those settings cannot be checked against code here.

Pitfalls

• Assignment runs twice per iteration. Targets are computed once for classification and once for regression, each with its own assignment and sampling pass ddod_head.py. Budget for roughly double the assignment work. The abstract says the extra overhead is negligible, but the paper gives no matched speed comparison: the FPS column of tab. 8 compares different backbones.
• The name alpha is overloaded. Section 4.1 says samples with higher foreground probability should be prioritised for classification §4.1. The chosen classification alpha of 0.8 does the opposite in the cost: it weights the IoU term at 0.8 and the probability term at 0.2. The paper's own ablation says IoU is decisive for classification targets §5.4.1. Implement the formula with alpha = 0.8, not the stated intuition.
• Counts are cumulative and not saved. The counters grow for the whole run, so late in training the weights hardly change. They are plain Python lists on the head ddod_head.py, so they are not part of the model's state dict. Resuming from a checkpoint restarts them from zero, which the code does not record.
• Counts are per process. The increments use each process's own positives, with no synchronisation across GPUs [code: mmdet/models/dense_heads/ddod_head.py#L309-L313]. Under distributed training, each GPU may end up with different level weights. The paper does not discuss this.
• Five levels are hard-coded. The counters are created for exactly five levels ddod_head.py. A different number of pyramid levels needs changes there.
• Positives only for the IoU branch. Negatives get zero weight in the IoU loss ddod_head.py. If a batch has no positives, the IoU and box losses are zero [code: mmdet/models/dense_heads/ddod_head.py#L239-L241].
• Pilot numbers use other thresholds. The IoU-threshold grid in tab. 1 is a RetinaNet experiment with fixed thresholds. Do not read it as the final cost-based assignment.
• WIDER FACE figures are not checkable. There is no DDOD-Face config here. The text's "overall AP of 93.0" matches the hard-subset value of the ATSS row in tab. 9, not an average over the three subsets. The reported DDOD gain is about 0.5 to 0.6 points on each subset §5.6.
• The 55.0 test-dev row is not fully specified. The paper lists multi-scale testing, Soft-NMS and stronger augmentation as "techniques like" these §5.5. The exact recipe is not given, and no config for that row is in the repository.
• Equation numbers are inferred. Equations 3 to 5 were numbered from the order of the LaTeX source. The compiled PDF was not checked, so cite these numbers with care.
• Training recipe gaps. The paper gives no learning rate, batch size or GPU count for COCO, and defers to MMDetection defaults §5.1. The config supplies lr 0.02 with 4 images per GPU ddod_r50_1x.py but no GPU count.

Lineage

• Builds on: ATSS (Zhang et al., 2020), whose adaptive threshold and base detector code this head extends §4.1 [code: mmdet/models/dense_heads/ddod_head.py#L15-L24]. The cost formula is taken from DeFCN (Wang et al., 2020; arXiv:2012.03544) §4.1. Deformable convolution is from Dai et al. (2017) §4.2. The observation that classification and localisation want different features comes from TSD (Song et al., 2020) §3.2.
• The linear per-level ramp is the Linear-Reweight of QueryDet (Yang et al., 2021; arXiv:2103.09136), used as a baseline §5.4.3 tab. 7.
• The face model adds the SSH module (Najibi et al., 2017) and DIoU loss (Zheng et al., 2020) §5.6.
• Compared against: RetinaNet, FCOS and ATSS in the same detectors tab. 3; the COCO test-dev field, including GFL, PAA, RepPoints, TridentNet, CenterNet, SAPD and FreeAnchor tab. 8; WIDER FACE methods including PyramidBox, RetinaFace, TinaFace and HAMBox tab. 9. None of these works has a card on this wiki, so only their names are used here.