ReDet: A Rotation-equivariant Detector for Aerial Object Detection

In plain words

Objects photographed from above can point in any direction, and a detector that has only seen a car facing north has to learn each direction again. The usual fix is to show the network many rotated copies of the training images. That works, but it makes the network larger and slower to train.

ReDet builds the rotation into the network itself. Its feature maps keep one copy of the picture's features for each of a fixed set of directions, and the convolutions are shared across those directions, so a rotated object produces the same features turned by a known amount. A second step then lines up the direction copies with the object's own orientation before the object's region is read out. The result is a detector that needs fewer weights to describe rotated objects, and whose region features stay put when the object turns.

TL;DR

ReDet is an oriented-object detector for aerial images. Its backbone is a ResNet-50 with FPN in which every layer is equivariant to the discrete rotation group C8, so orientation is a feature dimension and parameters are shared across it. An RPN and a RoI Transformer step produce rotated RoIs, and RiRoI Align pools each one by warping it in space and cyclically switching and interpolating the orientation channels. Author-reported on DOTA-v1.0, DOTA-v1.5 and HRSC2016 the detector beats the listed alternatives, with a backbone of 12 Mb against 103 Mb tab. 1. Several numbers in the abstract do not match the paper's own tables; use the tables (see Pitfalls).

Problem

Aerial objects have arbitrary orientation, large aspect ratios and dense layouts, so detectors use oriented boxes (OBBs) rather than horizontal ones §1. Ordinary CNNs are not equivariant to rotation. Feeding a rotated image does not give the rotated feature map, so region features taken from the same object at different angles differ §1 fig. 1. Current methods approximate rotation invariance with more parameters, more rotation-augmented data, or RRoI warping, which aligns only the spatial dimension §1 §2.3. The paper's aim is to make the backbone and the RoI pooling both rotation-aware §1 §3.

Method

Equivariance and invariance. A feature map is equivariant to a group of transforms if transforming the input transforms the output in a known way eq. 1. For a rotation T_r applied to an oriented RoI, the paper wants the feature of the rotated RoI to equal the rotated feature of the original RoI eq. 6. The rotation-invariant feature of an RoI is then recovered by undoing the rotation of its orientation, given the RoI's angle theta eq. 7 §3.

Rotation-equivariant backbone §4.1 fig. 3. Every layer of a ResNet-50 with FPN is rebuilt as a rotation-equivariant layer over the discrete group of translations and C_N rotations, with N = 8 in the main results tab. 1 §4.1. A feature map of size (K, N, H, W) therefore carries one channel block per orientation. The authors use the e2cnn library for the layers §4.1. The paper says the backbone has about 1/N of the parameters of the baseline when the output channel count is kept the same §4.1 (c).

RPN and RoI Transformer §4 fig. 3. An RPN proposes horizontal RoIs on the FPN levels. A RoI Transformer head turns each one into a rotated RoI (RRoI) with centre, size and angle (x, y, w, h, theta) §4 §2.1.

RiRoI Align §4.2 fig. 3. It runs in two parts.

1. Spatial alignment. The RRoI is sampled from the feature maps as in RRoI Align, which is the same spatial warp used by earlier methods §4.2 (a).
2. Orientation alignment. The orientation channels are switched cyclically so that the channel nearest to theta comes first, and the two channels around theta are linearly interpolated eq. 8 eq. 9 §4.2 (b).

The paper reports that keeping every orientation response is better than max-pooling over orientations, which discards the other responses §4.2 tab. 3.

# RiRoI Align for one RRoI  [§4.2] [eq. 8] [eq. 9]
# Input:  feature maps (K, N, H, W); RRoI (x, y, w, h, theta)
# Output: region features (K, N, S, S), orientation-aligned

def riroi_align(feature_maps, roi):
    # Spatial part: sample the RRoI grid, as in RRoI Align  [§4.2 (a)]
    region = rroi_sample(feature_maps, roi)          # (K, N, S, S)

    # Index r of the orientation bin below theta, and the fraction  [eq. 8]
    position = roi.theta * N / (2 * pi)
    r = floor(position)
    alpha = position - r

    # Output channel o reads input channels (o - r) and (o - r + 1)  [eq. 9]
    # Index sign follows the released kernel (see Paper vs code)
    aligned = zeros_like(region)
    for o in range(N):
        first = (o - r) % N
        second = (o - r + 1) % N
        aligned[:, o] = ((1 - alpha) * region[:, first]
                         + alpha * region[:, second])
    return aligned

Enlarged RoIs in the released code. Before RiRoI Align, the released code multiplies the RRoI width by 1.2 and its height by 1.4 in both training and testing rbox_single_level.py ReDet.py ReDet.py. The paper does not describe this step.

Training data §5.1 §5.2. DOTA images are cut into 1024 x 1024 patches with stride 824. Training uses the train and validation sets, and testing uses the test set. HRSC2016 images are resized to 800 x 512 without changing the aspect ratio. Horizontal flips are used in all settings.

Hyperparameters

Results

Author-reported. Setup: DOTA-v1.0 and DOTA-v1.5 OBB and HBB mAP, HRSC2016 mAP under VOC2007 and VOC2012 metrics. Backbone ReR50-ReFPN with C_8 unless stated otherwise §5.4 tab. 6 tab. 7 tab. 8. Baseline is the strongest other row in the same table, not a re-run.

Ablations, DOTA-v1.5 test set, ReR50 + ReFPN unless stated:

The paper also reports the model size of the full detector as 313 Mb for the baseline and 121 Mb for ReDet, in the abstract and introduction §1. No table row gives those two numbers.

Paper vs code

Code: official repository at commit f4a8e7dadec990f8a92437024416f0fb07a868db, Apache-2.0, built on mmdetection 1.x-style code with e2cnn layers. Checked: the ReDet configs, the RiRoI Align module and CUDA kernel, the ReDet detector, the rotation augmentation, and the DOTA split script.

• Anchors per location. Paper: 15 §5.2. Code: 3 anchors, one scale and three aspect ratios ReDet_re50_refpn_1x_dota1.py anchor_head.py. The rotated RPN variant, which has an angle dimension, is not the one used by the config.
• Interpolation count. Paper: ablation over 1, 2 and 4 orientation channels tab. 3 §4.2. Code: the kernel always blends the floor channel and the next channel riroi_align_kernel.cu. The Python module takes no interpolation count riroi_align.py. The l = 1 and l = 4 variants of tab. 3 cannot be reproduced from this code.
• RoI enlargement. Not in the paper. The code scales RRoIs by 1.2 and 1.4 before RiRoI Align, in training and testing rbox_single_level.py ReDet.py ReDet.py.
• Rotation augmentation. Paper: "random rotation" §5.2. The 80.10 config uses five fixed angles, 30 to 150 degrees, with probability 0.5 and no 0-degree or 180-degree options ReDet_re50_refpn_1x_dota1_ms.py rotate_aug.py. The ablation baseline in tab. 4 uses {0, 45, ..., 315} degrees, a different protocol.
• Extra training and test settings not in the paper. Gradient clipping, warm-up and test-time NMS ReDet_re50_refpn_1x_dota1.py ReDet_re50_refpn_1x_dota1.py.
• Metric naming for HRSC2016. Paper tab. 8 gives 90.46 as mAP under VOC2007. The config comment labels 90.46 as AP50 and 70.41 as mAP under the same VOC2007 heading ReDet_re50_refpn_3x_hrsc2016.py. tab. 5 reports 90.46 as AP50 in COCO style. Only the values agree.
• Channel-index sign. Paper eq. 8 says the channels are switched so that orientation r comes first. The kernel reads input channel (o - r) for output channel o riroi_align_kernel.cu. This card's pseudocode follows the kernel.

No other discrepancies found in the checked parts.

Pitfalls

• The abstract's gains do not match the tables. The abstract gives 1.2, 3.5 and 2.6 mAP over the previous best. Against the strongest other row in the paper's own tables the gains are 0.68 (DOTA-v1.0 multi-scale, 80.10 against 79.42), 1.90 (DOTA-v1.5 OBB multi-scale, 76.80 against 74.90) and 2.62 (HRSC2016 VOC2012, 97.63 against 95.01). On HRSC2016 VOC2007 the gap is 0.29 (90.46 against 90.17). Implement from the tables, and treat the abstract's numbers as unverified.
• Equation and table numbers. They were counted from the LaTeX source in order, since the compiled PDF was not read. Nine equations are numbered in the Preliminaries and Method, and no appendix is in the source. If the PDF numbering differs, cite by section.
• Group size is fixed in code. The RiRoI module defaults to 8 orientations and the config does not override it riroi_align.py. Using the C_4 or C_16 backbones of tab. 1 needs an explicit orientation count, and the kernel's two-channel interpolation still applies.
• Angle wrap-around. Angles are in radians. The channel index is floor(theta N / 2 pi), corrected modulo N, so negative angles are wrapped riroi_align_kernel.cu.
• Approximate invariance. The backbone is equivariant only to the discrete group C_N, and RiRoI Align interpolates between two neighbouring orientations §4.1 §4.2. The paper itself says that too many interpolated channels hurt the orientation relations §5.3. Expect invariance only at grid angles and approximate invariance in between.
• RoI enlargement changes the RoI features. Without the 1.2 and 1.4 factors, the RiRoI features will differ from the released results.
• Cost is not fully stated. The paper says the equivariant backbone has the same computation as the baseline when channel counts match §4.1 (c), but it gives no FLOPs, memory or latency table for the detector. tab. 4 reports training hours on hardware that is not named. The e2cnn version is not stated.
• Rotation augmentation is part of the 80.10 number. The multi-scale configuration uses rotation augmentation and test-time rotation, so the 80.10 result is not a pure equivariance result §5.4 ReDet_re50_refpn_1x_dota1_ms.py.
• Pretraining is not in this repository. The config loads a C8 ReResNet-50 checkpoint by path, and the ImageNet pretraining setup is in the paper only §5.2.

Lineage

• Builds on: RoI Transformer (cited as ding2018transformer) for rotated RoIs from horizontal ones §2.1 §4; rotated RoI warping, RRoI Align and RRPN (ma2018arbitrary, ding2018transformer) §2.3; group convolutions for rotation-equivariant CNNs (cohen2016gcnn) §2.2; the e2cnn library (weiler2019e2cnn) §4.1. Citation keys are as written in the paper; no arXiv links are given because the ids were not in the source.
• Compared against: RetinaNet-O, FR-O, Mask R-CNN, HTC, S2A-Net, R3Det, CenterMap, CSL and OWSR on DOTA and HRSC2016 tab. 6 tab. 7 tab. 8.
• Superseded by: not stated in this wiki.