Oriented object detection

Overview

An oriented detector outputs, for each object, a class, a score and an oriented box (x, y, w, h, θ). The angle lets the box follow an object that is tilted in the image, which matters for aerial photos where objects have no preferred direction and are often packed closely 2103.07733 §1. The rotated box is the one thing every variant shares: its centre, size and angle are the state the detector predicts and the loss compares [arxiv:2409.19648 §3.2] 2103.07733 §4.

The variants differ in where rotation is handled. One keeps the image backbone rotation-aware, so an object seen at a new angle yields a predictably rotated feature map 2103.07733 §3 2103.07733 §4.1. The other keeps a standard backbone and makes the transformer decoder aware of box angle, through the positional encoding, the attention bias and the sampling points 2409.19648 §3.3 2409.19648 §3.5.

Variants

Rotation-equivariant backbone with rotated RoI pooling (ReDet)

The backbone is a ResNet-50 with FPN whose layers are equivariant to the discrete rotation group C8, so orientation becomes a channel axis and filters are shared across the eight directions 2103.07733 §4.1. The RPN and a RoI Transformer head produce rotated RoIs, and RiRoI Align pools each one in two steps: a spatial warp of the rotated box, then a cyclic shift of the orientation channels by the box angle with linear interpolation between the two nearest bins 2103.07733 §4.2 2103.07733 eq. 8 2103.07733 eq. 9. Card: Rotation-equivariant detector for aerial objects.

The reported backbone size is 12 Mb against 103 Mb for the R50-FPN baseline, at a DOTA-v1.5 score of 66.86 against 65.03 2103.07733 tab. 1. The paper says the equivariant backbone has about one N-th of the baseline's parameters when channel counts match 2103.07733 §4.1. The released RiRoI kernel is a CUDA extension, so the method needs that build step.

End-to-end transformer with rotation-aware decoder (OrientedFormer)

A DETR-style decoder with two decoder layers and 300 queries, with no encoder 2409.19648 §3.1 2409.19648 alg. 1 2409.19648 tab. 9. Three changes adapt it to oriented boxes. The box is encoded as a 2-D Gaussian, so position, size and angle enter the positional encoding together 2409.19648 eq. 2 2409.19648 eq. 6. Self-attention gets a bias from the Wasserstein distance between box Gaussians 2409.19648 eq. 8 [arxiv:2409.19648 eq. 9]. Cross-attention samples features at points that are rotated with the box and weighted across feature levels 2409.19648 §3.5 2409.19648 eq. 14 2409.19648 eq. 16. Matching is one-to-one with the Hungarian algorithm and the loss is a focal classification term, an L1 term and a rotated-IoU term 2409.19648 §3.6 [arxiv:2409.19648 eq. 19]. Card: OrientedFormer.

Author-reported ResNet-50 results are 67.28 AP50 on DIOR-R against 66.12 for ARS-DETR, and 75.37 AP50 on DOTA-v1.0 single-scale against 74.16 for ARS-DETR 2409.19648 tab. 3 2409.19648 tab. 4. The authors also report convergence in 12 epochs against 36 for two end-to-end baselines 2409.19648 §4.8. The cost is higher: 529 GFLOPs against 186 for ARS-DETR and Deformable DETR-O, at 11.4 FPS on hardware the table does not name 2409.19648 tab. 14.

When to use which

Both cards report results on DOTA and related sets. Neither reports results on a non-aerial oriented task, so transfer outside remote sensing is not supported by the cards.

Implementation sketch

The shared core is a rotated region read-out: sample features on a grid that lies in the box's own frame, then rotate that grid to the box angle. ReDet adds an orientation shift for its equivariant channels. OrientedFormer rotates the sampling offsets in its attention layer instead.

# Oriented box: centre, size, angle  [arxiv:2103.07733 §4] [arxiv:2409.19648 §3.2]
# Input:  feature maps; candidate (x, y, w, h, theta); grid size S
# Output: region feature of shape (S*S, C)

def oriented_region_feature(features, box, S):
    # Regular grid inside the box, in the box's own frame
    local = grid_in_box(box.w, box.h, S)                       # (S*S, 2)
    # Rotate the grid by the box angle and move it to the box centre
    #   [arxiv:2409.19648 eq. 14] (rotation about the centre, as in the code)
    c, s = cos(box.theta), sin(box.theta)
    px = box.x + c * local[:, 0] - s * local[:, 1]             # (S*S,)
    py = box.y + s * local[:, 0] + c * local[:, 1]             # (S*S,)
    # Bilinear sample at the rotated points  [arxiv:2103.07733 §4.2 (a)]
    region = bilinear(features, px, py)                        # (S*S, C)
    # Equivariant backbones only: cyclic shift and interpolation of the
    # orientation channels by theta  [arxiv:2103.07733 eq. 8] [eq. 9]
    if features.has_orientation_axis:
        region = orientation_align(region, box.theta)
    return region

The transformer variant trains with a rotated IoU term on matched boxes 2409.19648 eq. 19, using one-to-one Hungarian matching.

Contradictions

• ReDet's abstract and tables disagree. The abstract gives gains of 1.2, 3.5 and 2.6 mAP over the previous best. The tables give 0.68, 1.90 and 2.62 against the strongest other row 2103.07733 tab. 6 [arxiv:2103.07733 tab. 7] 2103.07733 tab. 8. Implement from the tables.
• ReDet labels one number two ways. The single-scale DOTA-v1.0 score of 76.25 is labelled AP50 in one table and mAP in another 2103.07733 tab. 5 2103.07733 tab. 6. OrientedFormer reports AP50 throughout 2409.19648 tab. 4. Scores from the two cards are therefore not guaranteed to share a metric. This is not resolved here.
• The 80.10 ReDet result is not a pure equivariance result. It uses rotation augmentation and test-time rotation 2103.07733 §5.4. Compare it with the single-scale rows only.
• ReDet's paper and code differ. The paper's ablation over 1, 2 and 4 orientation channels 2103.07733 tab. 3 cannot be reproduced from the released kernel, which always blends two channels. The paper also states 15 anchors per location 2103.07733 §5.2; the config uses 3. The code enlarges RoIs by 1.2 and 1.4 before RiRoI Align, which the paper does not describe.
• OrientedFormer's paper and code differ. The code multiplies the self-attention bias by a learnable scale per head and divides the Wasserstein distance by an extra normalisation term. Neither is in the equations as printed 2409.19648 eq. 9. Implementing from the equations alone gives different attention behaviour.
• OrientedFormer's sampling-point text disagrees with its table. The text says the count goes "from 4 to 32", while the table starts at 8 2409.19648 tab. 9. The sampling exponent in the equation also does not match the paper's own definitions 2409.19648 eq. 11 2409.19648 eq. 13. Follow the definitions.

Papers

• Rotation-equivariant detector for aerial objects: ReDet, a C8 rotation-equivariant ResNet-50 with FPN and RiRoI Align for rotated RoIs.
• OrientedFormer: end-to-end transformer detector for oriented boxes, with Gaussian positional encoding, Wasserstein attention bias and oriented cross-attention.