Rotation-equivariant CNN

Overview

A standard CNN is not equivariant to rotation. Rotating the input does not turn the feature map in any simple way, so the same object at two angles gives two different region features 2103.07733 §1. A map is equivariant to a group of transforms when transforming the input transforms the output in a known way 2103.07733 eq. 1.

The rotation-equivariant design builds that property into the layers for a discrete set of rotations. Orientation becomes an extra feature dimension, and the weights are shared across it. An orientation-invariant region feature can then be recovered by undoing the object's rotation, given its angle 2103.07733 eq. 7 2103.07733 §3.

The alternatives the paper names are more parameters, more rotation-augmented training data, and RRoI warping, which aligns only the spatial dimension 2103.07733 §1 2103.07733 §2.3. Every variant on this page keeps a feature map shaped (channels, orientations, height, width) 2103.07733 §4.1.

Variants

Discrete C_N backbone with orientation-aware RoI pooling

Each layer of a ResNet-50 with FPN is rebuilt as a layer equivariant to translations and the cyclic rotation group C_N. The main results use N = 8 2103.07733 §4.1 2103.07733 tab. 1. The authors use the e2cnn library for these layers 2103.07733 §4.1.

The detector is oriented: an RPN proposes horizontal RoIs, and a RoI Transformer head turns them into rotated RoIs 2103.07733 §4. The pooling step, RiRoI Align, first samples the rotated RoI in space as earlier RRoI methods do. It then shifts the orientation channels so the channel nearest the RoI angle comes first, and linearly blends the two channels around that angle 2103.07733 §4.2 2103.07733 eq. 8 2103.07733 eq. 9. Keeping all orientation responses beats max-pooling over orientations 2103.07733 tab. 3.

Card: ReDet: A Rotation-equivariant Detector for Aerial Object Detection.

Other variants

No other rotation-equivariant design has a paper card yet, so none is listed.

When to use which

Constraints below come from the paper's own tables. The paper reports no latency, FLOPs or memory measurements for the detector 2103.07733 §4.1.

Invariance is approximate. The backbone is equivariant only to the discrete group, and RiRoI Align blends neighbouring orientations. Expect invariance at the grid angles and approximate invariance between them 2103.07733 §4.1 2103.07733 §4.2 2103.07733 §5.3.

Evaluation covers DOTA-v1.0, DOTA-v1.5 and HRSC2016 only 2103.07733 tab. 6 2103.07733 tab. 7 2103.07733 tab. 8. Results on other domains are not stated.

Implementation sketch

Pseudocode for the RiRoI Align pooling step. Shapes are noted on each line.

# RiRoI Align for one rotated RoI  [arxiv:2103.07733 §4.2]
# Input:  feature maps (K, N, H, W); RRoI (x, y, w, h, theta)
# Output: region features (K, N, S, S), orientation-aligned

def riroi_align(feature_maps, roi):
    # Spatial part: sample the RRoI grid, as in RRoI Align  [§4.2 (a)]
    region = rroi_sample(feature_maps, roi)          # (K, N, S, S)

    # Index r of the orientation bin below theta, and the fraction  [eq. 8]
    position = roi.theta * N / (2 * pi)
    r = floor(position)
    alpha = position - r

    # Output channel o blends input channels (o - r) and (o - r + 1)  [eq. 9]
    # The sign convention is a known difference; see Pitfalls
    aligned = zeros_like(region)
    for o in range(N):
        first = (o - r) % N
        second = (o - r + 1) % N
        aligned[:, o] = ((1 - alpha) * region[:, first]
                         + alpha * region[:, second])
    return aligned

Contradictions

Only one paper is in the wiki, so no disagreement between papers is recorded. Inside the ReDet paper, the abstract's gains and the table values do not agree. See Pitfalls.

Pitfalls

• Abstract gains do not match the tables. The card records the abstract's gains as 1.2, 3.5 and 2.6 mAP over the previous best. Against the strongest other row in the tables, the gains are 0.68 on DOTA-v1.0 multi-scale 2103.07733 tab. 6, 1.90 on DOTA-v1.5 OBB multi-scale 2103.07733 tab. 7 and 2.62 on HRSC2016 under the VOC2012 metric 2103.07733 tab. 8. Implement from the tables.
• Interpolation count. The paper ablates 1, 2 and 4 blended orientation channels 2103.07733 tab. 3. The card's review of the released code found that the kernel always blends two channels, so the 1 and 4 variants cannot be reproduced from that code. The card's Paper vs code section has the details.
• Sign convention. The card reports that the paper's equation and the released kernel index the channels in opposite directions. The sketch above follows the kernel. Check both before reimplementing.
• Extra RoI scaling. The released code enlarges each RRoI, with width times 1.2 and height times 1.4, before RiRoI Align. The paper does not describe this step, so results may differ without it [see card, Paper vs code].
• Group size is a setting, not a constant. The RiRoI module defaults to 8 orientations. Other group sizes need an explicit setting, and the two-channel blend still applies [see card, Pitfalls].
• Rotation augmentation in the top result. The DOTA-v1.0 80.10 result uses rotation augmentation and test-time rotation, so it is not a pure equivariance result 2103.07733 §5.4.
• Pretraining is outside the repository. The ImageNet pretraining setup is given in the paper only 2103.07733 §5.2.

Papers

• ReDet: A Rotation-equivariant Detector for Aerial Object Detection: a C_8-equivariant ResNet-50 with FPN backbone and RiRoI Align pooling for oriented aerial object detection.