RFLA: Gaussian Receptive Field based Label Assignment for Tiny Object Detection

In plain words

Detectors decide which spots in the image should learn to recognise an object, and for big objects this is easy: a box that overlaps the object is a good spot. Tiny objects break that rule. A tiny object is often so small that none of the pre-set boxes or points overlap it, so it gets very few training examples.

This paper replaces the overlap test with a comparison of shapes. Each spot in the network has a soft, blob-shaped region of the image it "sees", and the object is also treated as a soft blob. The method measures how similar the two blobs are and gives the best-matching spots to each object. A second pass shrinks each blob a little and gives each object one more spot, so every object gets enough training signal regardless of its size.

The result is a drop-in change to how training examples are chosen. The paper reports no extra cost at inference time, and its largest gains are on the smallest size group.

TL;DR

• Replaces IoU or centre-based positive selection with a distance between a Gaussian receptive field and a Gaussian model of each ground-truth box §3.1 §3.2.
• Selection is two-stage: top-k points per object, then a top-1 pass with the receptive field shrunk by a factor β §3.3.
• Author-reported on AI-TOD: Faster R-CNN 11.1 → 21.1 AP, DetectoRS 14.8 → 24.8 AP with RFLA tab. 5.
• Code (MIT) is an MMDetection fork; the RPN assigner and FCOS head use it, while the Faster R-CNN second stage keeps MaxIoU.

Problem

Anchor-based detectors assign positives by IoU with pre-set boxes, and anchor-free detectors assign them by whether a point lies inside the object. For tiny objects (under 16×16 px in the AI-TOD definition §1) the object often has IoU 0 with every anchor or contains no feature point at all, so it receives few or no positives §1 fig. 4. The paper also argues that a uniform "box" or "point" prior differs from the Gaussian-shaped effective receptive field of a feature point, so the assigned point's receptive field may not match the object §1 §3.1.

The authors describe a scale-sample imbalance: tiny objects are outliers for existing anchor-based and anchor-free assignment, and RFLA makes the number of positives per object more even across scales §4.5 fig. 4.

Method

Overview. RFLA scores every (feature point, ground truth) pair with a receptive-field distance and then assigns labels with a hierarchical rule §3. It is applied to the Faster R-CNN RPN and to FCOS §3.4.

Step 1. Gaussian effective receptive field (ERF) per feature point §3.1.

1. Compute the theoretical receptive field (TRF) size of each layer with the recurrence in eq. 2, where k_n and s_n are kernel size and stride.
2. Put a 2-D Gaussian on each feature point: mean at the point location (x_n, y_n), isotropic variance er_n² eq. 3.
3. Set er_n to half the TRF radius, as an approximation of the effective receptive field (ERF) §3.1.

tr_{n} = tr_{n-1} + (k_{n} - 1) \prod_{i=1}^{n-1} s_{i} eq. 2

\boldsymbol{\mu_{e}} = (x_{n}, y_{n})^{\top}, \quad \boldsymbol{\Sigma_{e}} = \mathrm{diag}(er_{n}^{2}, er_{n}^{2}) eq. 3

Step 2. Gaussian model of each ground-truth box §3.2.

1. Mean at the box centre (x_g, y_g); covariance diag(w_g²/4, h_g²/4) eq. 4.
2. Gaussian model of the box: the box's half-side is the standard deviation.

\boldsymbol{\mu_{g}} = (x_{g}, y_{g})^{\top}, \quad \boldsymbol{\Sigma_{g}} = \mathrm{diag}\!\left(\tfrac{w_{g}^{2}}{4}, \tfrac{h_{g}^{2}}{4}\right) eq. 4

Step 3. Receptive-field distance (RFD) §3.2. The paper compares three distances between Gaussians and keeps two (Wasserstein and KL). J-S divergence is dropped because it has no closed form §3.2.

1. Wasserstein-2 distance, closed form in the paper's notation eq. 5.
2. KL divergence between the ERF and the ground truth, general form eq. 6 and simplified form eq. 7.
3. Map any distance value RFDC to a score in (0, 1] with RFD = 1 / (1 + RFDC) eq. 8.

W_{2}^{2}(n_{e}, n_{g}) = \left\| \left( [x_{n}, y_{n}, er_{n}, er_{n}]^{\top},\; [x_{g}, y_{g}, \tfrac{w_{g}}{2}, \tfrac{h_{g}}{2}]^{\top} \right) \right\|_{2}^{2} eq. 5

D_{\mathrm{KL}}(n_{e} \| n_{g}) = \tfrac{1}{2}\left( \mathrm{tr}(\boldsymbol{\Sigma_g^{-1}}\boldsymbol{\Sigma_e}) + (\boldsymbol{\mu_g}-\boldsymbol{\mu_e})^{\top}\boldsymbol{\Sigma_g^{-1}}(\boldsymbol{\mu_g}-\boldsymbol{\mu_e}) + \ln\tfrac{\lvert\boldsymbol{\Sigma_g}\rvert}{\lvert\boldsymbol{\Sigma_e}\rvert} - 2 \right) eq. 6

D_{\mathrm{KL}}(n_{e} \| n_{g}) = \frac{2er_{n}^{2}}{w_{g}^{2}} + \frac{2er_{n}^{2}}{h_{g}^{2}} + \frac{2(x_{n}-x_{g})^{2}}{w_{g}^{2}} + \frac{2(y_{n}-y_{g})^{2}}{h_{g}^{2}} + \ln\frac{w_{g}}{2er_{n}} + \ln\frac{h_{g}}{2er_{n}} - 1 eq. 7

\mathrm{RFD} = \frac{1}{1 + \mathrm{RFDC}} eq. 8

The paper states that KL is scale-invariant and is the better choice for very tiny objects, while Wasserstein is not scale-invariant §3.2 §4.3. KL is the default RFD in the experiments §4.3.

Step 4. Hierarchical label assignment (HLA) §3.3. The RFD score is not on the same scale as IoU, so HLA ranks points by score instead of using a threshold.

1. Stage 1: for each ground truth, the top-k points by RFD become positive. This gives r₁ and a mask m of the points assigned in this stage.
2. Stage 2: shrink every ERF radius by β, recompute the RFD, and give the top-1 point per ground truth a positive label. This gives r₂.
3. Combine: keep r₁ where m is set, else use r₂ eq. 9. The mask stops extra stage-2 positives from being added to objects that already have enough.

r = r_{1} m + r_{2}(1 - m) eq. 9

# RFLA label assignment for one image  [§3.2] [§3.3]
# Input:  points (N, 2): feature point locations  [§3.1]
#         erf_radius (N,): er_n per point         [§3.1]
#         gts (G, 4): boxes as (x, y, w, h)       [§3.2]
# Output: per point: 1-based gt index, 0 = background

def assign_rfla(points, erf_radius, gts, k=3, beta=0.9):
    # Stage 1: RFD of every point against every gt  [eq. 8]
    scores = rfd_matrix(points, erf_radius, gts)        # (G, N)
    # Each gt takes its top-k points by RFD  [§3.3]
    stage1 = take_top_k_per_gt(scores, k)               # (N,)
    # Stage 2: shrink every ERF radius by beta, then score again  [§3.3]
    shrunk = rfd_matrix(points, beta * erf_radius, gts)  # (G, N)
    # Each gt takes its top-1 point  [§3.3]
    stage2 = take_top_k_per_gt(shrunk, 1)               # (N,)
    # Keep stage 1 where it assigned a point, else use stage 2  [eq. 9]
    return where(stage1 > 0, stage1, stage2)            # (N,)

# RFD between one ERF and one gt, KL form  [eq. 7] [eq. 8]
# erf_center: (x_n, y_n); erf_radius: er_n
# gt_center: (x_g, y_g); gt_size: (w_g, h_g)
def rfd_kl(erf_center, erf_radius, gt_center, gt_size):
    # Offset between the two centres  [§3.2]
    offset = erf_center - gt_center                     # (2,)
    # KL divergence between the two Gaussians  [eq. 7]
    kl = (
        2 * erf_radius ** 2 / gt_size[0] ** 2
        + 2 * erf_radius ** 2 / gt_size[1] ** 2
        + 2 * offset[0] ** 2 / gt_size[0] ** 2
        + 2 * offset[1] ** 2 / gt_size[1] ** 2
        + log(gt_size[0] / (2 * erf_radius))
        + log(gt_size[1] / (2 * erf_radius))
        - 1
    )
    # Map the distance to a score in (0, 1]  [eq. 8]
    return 1 / (1 + kl)

Step 5. Application to detectors §3.4.

• Faster R-CNN: RFLA replaces the standard anchor tiling and the MaxIoU anchor assignment. The text does not say whether the second (R-CNN) stage changes; the code does not change it (see Paper vs code) §3.4.
• FCOS: the constraint that positive points lie inside the ground-truth box is removed, and point-based assignment is replaced by RFLA §3.4. The centerness target is changed to avoid gradient explosion, using eq. 10 with c = 0.01 §3.4.

\text{centerness}^{*} = \sqrt{ \frac{\varepsilon[\min(l^{*}, r^{*})]\,\min(l^{*}, r^{*}) + c}{\max(l^{*}, r^{*})} \times \frac{\varepsilon[\min(t^{*}, b^{*})]\,\min(t^{*}, b^{*}) + c}{\max(t^{*}, b^{*})} } eq. 10

Here l*, t*, r*, b* are the FCOS regression targets and ε is the step function of eq. 1, equal to 1 for positive input and 0 otherwise §1 §3.4.

Hyperparameters

Values are from the paper unless the Where column says "code". Training is on AI-TOD trainval and evaluated on the test split tab. 5 caption.

Results

Author-reported. Setup: AI-TOD (28,036 images, average object size 12.8 px §4.1), trained on trainval and evaluated on test tab. 5 caption; ResNet-50 backbone; 1× RTX 3090 §4.2. The paper does not state whether competitor rows are re-runs or copied from other sources.

Main result, AI-TOD tab. 5. Full table as reported:

The paper reports DetectoRS w/ RFLA as 4.0 AP above the best listed competitor (DetectoRS w/ NWD, 20.8) §4.4 tab. 5.

Ablations on AI-TOD, Faster R-CNN backbone §4.3.

Distance choice, all with HLA tab. 1:

Component contribution tab. 2:

Stage-2 decay β tab. 3:

Gaussian vs receptive anchor tab. 4:

The paper also reports k from 1 to 4 giving 20.7, 21.1, 21.1, and 20.9 AP §4.3 (no table).

Other datasets (author-reported). TinyPerson tab. 6:

VisDrone2019 tab. 7, trained on train, validated on val:

DOTA-v2.0 tab. 8, trained on train, validated on val:

Visualisations on AI-TOD show fewer false negatives when RFLA is added to Faster R-CNN §4.6 fig. 5. fig. 3 compares top-k with anchor-size tuning, and fig. 4 shows the scale-sample imbalance §4.3 §4.5.

Paper vs code

Compared the paper against the official repository at commit 198977b5e198005fec783e81f10f9363946d2ef8 (MIT, Copyright 2022 Chang Xu, a paper author). Repository link is the one printed in the abstract.

• Positive selection (HLA) matches. Stage-1 top-k with the KL RFD, stage-2 shrink by 0.9 with top-1, and the mask combination all follow §3.3 and eq. 9 hierarchical_assigner.py hierarchical_assigner.py.
• KL RFD matches eq. 7 and eq. 8 when the ERF box side is taken as 2·er_n metric_calculator.py.
• Wasserstein is implemented as RFD = 1/(1 + centre-distance² + ((w1−w2)² + (h1−h2)²)/4), with w2 = 2·er_n, which reads eq. 5 as a squared difference of the two 4-vectors metric_calculator.py.
• ERF radius: the paper's er_n is half the TRF "radius" §3.1. The code uses base size × 0.5 as the side of the ERF box rf_generator.py, so er_n = TRF/4. This reading is consistent with the paper if TRF is a diameter, but the paper does not say which.
• Faster R-CNN second stage: the paper says RFLA replaces "anchor tiling and MaxIoU anchor assigning" §3.4. The code applies HieAssigner to the RPN only. The R-CNN head keeps MaxIoUAssigner with positive IoU 0.5 aitod_faster_r50_rfla_kld_1x.py.
• Centerness target differs. The paper's eq. 10 adds c to the minimum of each pair before dividing by the maximum. The code clamps the minimum to at least 0.01 instead rfla_fcos_head.py. For a minimum m ≥ 0.01 the paper gives (m + 0.01)/M and the code gives m/M. For 0 < m < 0.01 the paper gives (m + 0.01)/M and the code gives 0.01/M.
• Extra hard-coded threshold: both assignment stages set points whose best RFD is below 0.8 to background, a value the paper does not mention hierarchical_assigner.py hierarchical_assigner.py.
• TRF values are fixed for ResNet-50 FPN in gen_trf, not computed from eq. 2 at run time rf_generator.py. Evaluating that arithmetic by hand gives 35, 91, 267, 427 and 555 px for P2–P6 (not run).
• Warmup is in the code and absent from the paper: linear, 5000 iterations, starting at 0.001× the learning rate aitod_faster_r50_rfla_kld_1x.py.
• Learning schedule matches: SGD, momentum 0.9, weight decay 1e-4, batch 2, 12 epochs, lr 0.005 with steps at 8 and 11 §4.2 aitod_faster_r50_rfla_kld_1x.py.
• Configs for some headline rows are for AI-TOD-v2, not AI-TOD v1. The DetectoRS RFLA config targets aitodv2 aitodv2_detectors_rfla_kld_1x.py. The only FCOS config with P2 (the FCOS* row) is aitodv2_fcos_r50_p2_rfla_kld_1x.py aitodv2_fcos_r50_p2_rfla_kld_1x.py. The v1 FCOS config uses P3–P7 aitod_fcos_r50_rfla_kld_1x.py. The repository README reports DetectoRS w/ RFLA as 24.8 AP on the v1 table and 25.7 AP on the v2 table [README.md, Table 1 and Table 2]. No v1 DetectoRS config is in the repository.
• README rows not in the paper: the README lists FCOS w/ RFLA at 13.3 AP and FCOS (reproduce) at 12.0 AP on AI-TOD. Neither row is in the paper's table.
• Tie-breaking and "smaller gt" rule: the paper says an occluded sample goes to the smaller ground truth §3.3. The HieAssigner loops over gts with the later index winning hierarchical_assigner.py. I did not find a size-based rule there and did not trace whether gts are sorted upstream.
• Repository scope: the README states the code is "unsuited for generic object detection (e.g. VOC, COCO)" because the problem is not obvious for medium and large objects.

Pitfalls

• Stage-1 and stage-2 both compute a full gt×point RFD matrix, and stage 2 uses shrunk radii for all points. Stage-1 loops over gts and k in Python hierarchical_assigner.py. The With gpu_assign_thr=512 in the config aitod_faster_r50_rfla_kld_1x.py, assignment runs on CPU when there are more than 512 gts hierarchical_assigner.py.
• Points with a best RFD of 0.8 or more that are not in any top-k are left at −1 (ignore), not 0 (negative). The paper has no ignore state for these points hierarchical_assigner.py hierarchical_assigner.py.
• Exact ties in RFD are assigned with an equality test against the top-k values, so ties can give more than k points for one gt hierarchical_assigner.py.
• The ERF is a Gaussian centred on the feature point location. The code places points at i·stride with no half-stride offset rf_generator.py. The paper does not define the coordinate origin.
• Wasserstein is not scale-invariant (paper's own warning) and was slightly worse on AP50 §3.2 tab. 1. KL is the default for good reason.
• β is a tie at 0.9 vs 0.95 on AP (21.1 both) tab. 3. The paper picks 0.9 for AP50 and APvt.
• k was swept only on AI-TOD. The text says the AP is "quite robust" to k §4.3; the tested values give 20.7–21.1 AP.
• Headline DetectoRS and FCOS numbers do not map to a single v1 config in the repo* (see Paper vs code).
• Warmup and BN settings are code-only. Re-implementing from the paper alone will not match the code's learning-rate warmup or frozen first stage.
• Author-reported gains for the headline are on one backbone (ResNet-50) and one dataset (AI-TOD). The paper reports no variance or seed information tab. 5.
• Cost: the paper says there is no extra inference cost §1. The added cost is in training-time label assignment only.

Lineage

• Builds on: the anchor-based and anchor-free label-assignment families it critiques (Faster R-CNN's MaxIoU, ATSS, FCOS centre sampling) §1 §3.4; the Gaussian effective-receptive-field model §3.1; Gaussian Wasserstein and KL distances between boxes, as cited in §3.2; and the scale-invariance argument for KL, as cited in §3.2.
• Compared against: TridentNet, Cascade RPN, Cascade R-CNN, DetectoRS, DotD, DetectoRS w/ NWD, SSD, RetinaNet, PAA, ATSS, RepPoints, OTA, AutoAssign, FCOS, M-CenterNet on AI-TOD tab. 5.
• Superseded by: not stated.