OrientedFormer: An End-to-End Transformer-Based Oriented Object Detector in Remote Sensing Images

In plain words

Photos taken from satellites and aircraft show objects such as planes, ships and vehicles. These objects are often tilted at arbitrary angles and packed closely together. A standard detector draws upright rectangles around objects, which fit tilted objects poorly.

This paper builds on a detector that uses a transformer to predict all objects in one pass, with no hand-written step to remove duplicate boxes afterwards. The authors adapt that design to tilted rectangles in three places. First, each tilted rectangle is turned into a soft blob, so its position, size and angle are described in one shared way. Second, the attention that lets predictions interact is given a bias based on how far apart the blobs are, which helps the model avoid duplicate predictions. Third, when the model reads image features for an object, it looks at points arranged around the object and rotates that arrangement to match the object's tilt.

The authors report better accuracy than earlier detectors of this kind on several remote-sensing benchmarks, while training for fewer epochs. All numbers are theirs, and the detector is built on existing detection code, so the card's pitfalls section matters as much as the headline results.

TL;DR

OrientedFormer is an end-to-end transformer detector for rotated boxes in remote-sensing images. It adds three things to a DETR-style decoder: a Gaussian positional encoding of (x, y, w, h, θ), a bias from Gaussian Wasserstein distances in decoder self-attention, and a cross-attention whose sampling points rotate with each box angle §3.3 §3.4 §3.5. Author-reported ResNet-50 results are 67.28 AP50 on DIOR-R and 75.37 on DOTA-v1.0 single-scale, against 66.12 and 74.16 for ARS-DETR tab. 3 tab. 4. The official code is Apache-2.0 and differs from the paper in several places, including a learnable bias scale and an NMS step in its query initialisation (see Paper vs code). Training takes 12 epochs in the authors' runs, against 36 for two end-to-end baselines §4.8, but the model is about 2.8 times the FLOPs of those baselines tab. 14.

Problem

Oriented objects rotate freely, and transformer detectors with positional encodings for upright boxes miss this in three ways §1.

1. The encoding covers position and size but not angle, so the model cannot tell box orientation from the query itself §1 §2.3.
2. Self-attention sees only the content of each query. It does not see the geometric relation between the boxes that the queries describe §1 §2.2.
3. Cross-attention samples image features at points that do not line up with a tilted box, so the features it reads are misaligned with the object §1 §2.2.

The method assumes a one-to-one assignment between predictions and objects, trained with a Hungarian match and no hand-designed post-processing §3.6. The code adds a DDQ-style query initialisation that does use NMS (see Paper vs code).

Method

Overview. The encoder is not used. Multi-scale backbone features are mapped to a common width D = 256 §3.1. Queries are initialised by an enhancement method §3.1 fig. 2. Each object query has a content part Q_c (N × D) and a positional part Q_p (N × 5), where N is the number of queries and Q_p holds the oriented box §3.2. Each decoder layer runs self-attention, oriented cross-attention and a feed-forward network. It then predicts classes and boxes and updates Q_p from Q_c §3.1 alg. 1. The loss is applied to every decoder layer §3.6.

Gaussian positional encoding §3.3 fig. 3. An oriented box (x, y, w, h, θ) is a 2-D Gaussian with mean (x, y) and covariance R · diag(w²/4, h²/4) · Rᵀ, where R is the rotation by θ eq. 2. The standard sinusoidal encoding eq. 1 is rewritten as a sine and cosine of a scaled coordinate eq. 3. Its value under a Gaussian input is the expectation of the sine or cosine, which equals the sine or cosine of the mean damped by exp(−σ²/2) eq. 4 eq. 5. The Gaussian positional encoding is that expectation for each coordinate, computed with the scaled covariance, so only the diagonal of the covariance is needed eq. 6 eq. 7.

Wasserstein self-attention §3.4 fig. 4. For two boxes with Gaussians N(μ₁, Σ₁) and N(μ₂, Σ₂), the squared 2-Wasserstein distance is ‖μ₁ − μ₂‖² plus the trace of Σ₁ + Σ₂ − 2(Σ₁^½ Σ₂ Σ₁^½)^½ eq. 8. The paper rescales this into a bias G_ij = log(1/(τ + d_ij) + ε), with τ = 1 and ε = 10⁻⁷, where d_ij is the distance itself eq. 9. Coinciding queries get G = 0, and far-apart queries get about log ε §3.4. The attention logits are (Q_c + φ)(Q_c + φ)ᵀ / √d_q + G, and the values are Q_c eq. 10. The bias adds no parameters, and the paper gives the cost as O(D²N + DN²) §3.4.

Oriented cross-attention §3.5 fig. 5. A positional query (x, y, w, h, θ) is converted to (x, y, z, r, θ) with z = log₂√(wh) and r = log₂(h/w) eq. 11. Offsets (Δx, Δy, Δz) come from a linear layer on Q_c eq. 12. The offsets are scaled by the box size, and the in-plane pair is rotated by θ eq. 13 eq. 14. Values are bilinear samples of every feature level at the rotated points divided by the level stride eq. 15. Each level is weighted by a sigmoid of the squared gap between the sampling depth z̃ and log₂ of the stride, with softness η = 2 eq. 16. A channel gate eq. 17 and then a spatial gate eq. 18 are applied, both computed from Q_c. The heads are flattened, projected to width D, and added back to Q_c §3.5. The paper gives g = 64 heads and O = 32 points per head as the default tab. 9. The paper lists four ways this differs from deformable attention: it rotates points, uses three attention types, learns its positional queries, and samples in a virtual 3-D space §3.5.

Matching and loss §3.6 alg. 1 eq. 19. Predictions are matched to objects one-to-one with the Hungarian algorithm. The loss is 2 times a focal classification loss, plus 2 times an L1 box loss, plus 5 times a rotated-IoU loss eq. 19 §4.2.

The pseudocode below is written from the equations and the algorithm. Paper vs code lists where the official code differs.

# Gaussian positional encoding of one oriented box  [§3.3] [eq. 2] [eq. 6]
# Input:  box (x, y, w, h, theta); temperature T; num_feats (64)
# Output: encoding of length 4 * num_feats (256)

def gaussian_pe(box, temperature, num_feats):
    # Mean is the centre of the box  [eq. 2]
    mean = (box.x, box.y)                                   # (2,)
    # Covariance from size and angle: R diag(w^2/4, h^2/4) R^T  [eq. 2]
    rot = rotation_matrix(box.theta)                        # (2, 2)
    cov = rot @ diag(box.w ** 2 / 4, box.h ** 2 / 4) @ rot.T  # (2, 2)
    var = diagonal(cov)                                     # (2,)
    encoding = []
    for coord in (0, 1):                                    # x, then y
        for k in range(num_feats):
            # Frequency for index k, shared by the sine and cosine  [eq. 1]
            freq = temperature ** (2 * (k // 2) / num_feats)
            # Expectation of sin and cos under the scaled Gaussian  [eq. 6]
            damp = exp(-0.5 * var[coord] / freq ** 2)
            encoding.append(damp * sin(mean[coord] / freq))
            encoding.append(damp * cos(mean[coord] / freq))
    return encoding

# Squared 2-Wasserstein distance between two boxes' Gaussians  [eq. 8]
# Input:  means (2,), covariances (2, 2) for boxes a and b
# Output: scalar

def squared_wasserstein(mean_a, cov_a, mean_b, cov_b):
    # Squared distance between the centres  [eq. 8]
    centre = sum((mean_a - mean_b) ** 2)
    # Covariance term: Tr(A + B - 2 (A^1/2 B A^1/2)^1/2)  [eq. 8]
    root = sqrtm(sqrtm(cov_a) @ cov_b @ sqrtm(cov_a))
    return centre + trace(cov_a + cov_b - 2 * root)

# Gaussian Wasserstein bias for all pairs of N queries  [§3.4] [eq. 9]
# Input:  means (N, 2), covs (N, 2, 2)
# Output: bias (N, N), added to the attention logits

def wasserstein_bias(means, covs, tau=1.0, eps=1e-7):
    bias = zeros(N, N)                                      # (N, N)
    for i in range(N):
        for j in range(N):
            sq = squared_wasserstein(means[i], covs[i], means[j], covs[j])
            dist = sqrt(sq)
            # Zero for coinciding queries, about log(eps) for distant ones
            bias[i, j] = log(1.0 / (tau + dist) + eps)
    return bias

# Wasserstein self-attention, one head  [§3.4] [eq. 10]
# Input:  content (N, D), pe (N, D), bias (N, N), head width d_q
# Output: updated content (N, D)

def wasserstein_self_attention(content, pe, bias, d_q):
    # Positional encoding enters the scores only  [eq. 10]
    keys_input = content + pe                               # (N, D)
    scores = keys_input @ keys_input.T / sqrt(d_q) + bias   # (N, N)
    # Values are the plain content queries  [eq. 10]
    return softmax(scores, axis=-1) @ content               # (N, D)

# Oriented cross-attention for one query  [§3.5] [eq. 11]-[eq. 18]
# Input:  features f^l with strides s^l; content (D,); box (x, y, w, h, t)
# Output: content update (D,)

def oriented_cross_attention(features, strides, content, box):
    # Virtual coordinates of the box  [eq. 11]
    z = log2(sqrt(box.w * box.h))
    r = log2(box.h / box.w)
    # Offsets per head and point, from the content query  [eq. 12]
    offsets = linear(content).reshape(g, O, 3)              # (g, O, 3)
    # Offsets scaled by box size; exponent z - r/2 is what [eq. 11] implies
    dx = offsets[..., 0] * 2 ** (z - r / 2)                 # (g, O)
    dy = offsets[..., 1] * 2 ** (z + r / 2)                 # (g, O)
    # Rotate the offsets about the box centre by its angle  [eq. 14]
    cos_t, sin_t = cos(box.t), sin(box.t)
    points_x = box.x + cos_t * dx - sin_t * dy              # (g, O)
    points_y = box.y + sin_t * dx + cos_t * dy              # (g, O)
    # Sampling depth of each point  [eq. 13]
    z_points = z + offsets[..., 2]                          # (g, O)
    values = zeros(g, O, D // g)
    for level, stride in zip(features, strides):
        # Scale weight of this level for each point  [eq. 16], eta = 2
        weight = sigmoid(-(z_points - log2(stride)) ** 2 / 2.0)
        # Bilinear sample of this level at the points / stride  [eq. 15]
        sampled = bilinear(level, stack(points_x, points_y) / stride)
        values += weight * sampled
    # Channel gate from the content, then spatial gate  [eq. 17] [eq. 18]
    values = channel_gate(content) * values
    values = spatial_gate(content) * values
    # Flatten heads and points, project to D, add back  [§3.5]
    return linear_out(values.flatten()) + content

# Training loss over all decoder layers  [§3.6] [eq. 19] [alg. 1]
# Input:  per-layer class scores and boxes; ground-truth boxes and classes
# Output: scalar loss

def training_loss(layers, gt_boxes, gt_classes):
    total = 0.0
    for layer in layers:
        # One-to-one Hungarian matching of predictions to objects  [§3.6]
        pred_idx, gt_idx = hungarian(layer.scores, layer.boxes,
                                     gt_boxes, gt_classes)
        # Weights 2, 2 and 5 for focal, L1 and rotated IoU  [§4.2] [alg. 1]
        total += 2.0 * focal_loss(layer.scores, pred_idx, gt_classes)
        total += 2.0 * l1_loss(layer.boxes[pred_idx], gt_boxes[gt_idx])
        total += 5.0 * rotated_iou_loss(layer.boxes[pred_idx],
                                        gt_boxes[gt_idx])
    return total

Hyperparameters

Values marked "code" were read from the official repository at code_commit. Where the paper and the config differ, both are shown in Paper vs code.

Results

All numbers are author-reported. Baseline numbers are copied from the baseline papers, except ICDAR2015, where the authors re-implemented baselines in MMRotate §4.3. Setup: 2 × RTX 2080 Ti, MMRotate-based code §4.2. Dataset sizes: DIOR-R 23,463 images and 20 classes; DOTA-v1.0 2,806 images and 15 classes; DOTA-v2.0 11,268 images and 18 classes §4.1.

Convergence §4.8 fig. 8 fig. 9. On DIOR-R with 12 epochs and 300 queries, the ResNet-50 model reaches 67.3 AP50, against 31.2 for Deformable DETR-O with CSL and 38.9 for ARS-DETR (text values). On DOTA-v1.0, it reaches 75.37 in 12 epochs, against 74.16 for ARS-DETR and 69.48 for Deformable DETR-O, both at 36 epochs §4.8.

Ablations (AP50, DIOR-R, ResNet-50 unless stated).

Speed and size (DOTA-v1.0, tab. 14). OrientedFormer with ResNet-50 runs at 11.4 FPS with 44M parameters and 529 GFLOPs, and reaches 75.37 AP50. ARS-DETR runs at 10.1 FPS with 42M parameters and 186 GFLOPs (74.16 AP50). Deformable DETR-O runs at 10.8 FPS with 41M parameters and 186 GFLOPs (69.48 AP50). Two-stage and one-stage CNN detectors in the same table run between 6.1 and 17.7 FPS.

Paper vs code

Repository: https://github.com/wokaikaixinxin/OrientedFormer at cb760790ccf1d5f9d9504e10c2c25cc2ee410fcb (Apache-2.0). It is linked from the paper's abstract, and it contains MMRotate code alongside projects/OrientedFormer, where the method lives. The repository was read at this commit; the multi-scale DOTA config (orientedformer_le90_r50_q300_layer2_head64_point32_1x_dotav1.0-ms.py) was not inspected.

• Learnable bias scale. The paper's self-attention bias is G added to the logits as is eq. 10. The code multiplies G by a learnable scalar per head (8 heads), initialised uniformly in [0, 4], which the paper does not describe orientedformer_decoder_layer.py orientedformer_decoder_layer.py orientedformer_decoder_layer.py. The code's constant τ = 1 orientedformer_decoder_layer.py shares the name of the learnable parameter, which is a different quantity.
• Distance normalisation. The code divides the Wasserstein distance by 2 · (det Σ_i · det Σ_j)^(1/8) by default, a step not in the paper orientedformer_decoder_layer.py. The paper's only rescaling is the log(1/(τ + d) + ε) step eq. 9.
• Value tensor in self-attention. The code passes content + positional encoding as the one argument to the self-attention module orientedformer_decoder_layer.py. The paper uses plain Q_c as the value eq. 10. Whether the value is the same tensor depends on the mmcv MultiheadAttention default, which is not in this repository and was not checked.
• Sampling-point exponent. The paper's sampling formula uses 2^(z−r) and 2^(z+r) eq. 13. The code uses 2^(z∓r/2) oriented_attention.py, which matches the paper's own definitions of z and r eq. 11.
• Rotation centre. The paper rotates the sampled point (x̃, ỹ) eq. 14. The code rotates the offset around the box centre and then adds the centre oriented_attention.py. The code matches the stated aim of rotating around the positional query §1.
• Gate layers. The paper writes γ1 twice in the channel gate eq. 17 and "Linear and ReLU" for the spatial gate eq. 18. The code uses two distinct bias-free linear layers per gate, with hidden width D/4 oriented_attention.py oriented_attention.py.
• Offset initialisation. The code zeros the offset weights and sets their biases: x and y uniform in ±0.5 (the number of points is not a square), and Δz to −1 oriented_attention.py. The paper does not state this.
• Losses and matching in the query initialiser. The paper says focal loss and one-to-one Hungarian matching §3.6. The DDQ-style query-initialisation head uses QualityFocalLoss with a top-k Hungarian assigner (k = 8) orientedformer_le90_r50_q300_layer2_head64_point32_1x_dior.py orientedformer_le90_r50_q300_layer2_head64_point32_1x_dior.py. The decoder layers use the focal loss (γ = 2, α = 0.25, weight 2) with one-to-one matching orientedformer_le90_r50_q300_layer2_head64_point32_1x_dior.py.
• NMS. The paper's claim is that no post-processing is needed §1 §3.6. The query-initialisation head applies rotated NMS at IoU 0.7 to the top 1000 candidates before selecting queries orientedformer_le90_r50_q300_layer2_head64_point32_1x_dior.py oriented_ddq_fcn.py.
• Feature levels. The algorithm and the default config use 5 levels with strides 4 to 64 alg. 1 orientedformer_le90_r50_q300_layer2_head64_point32_1x_dior.py. The text picks "4 layers" as the best setting §4.10 tab. 15. The table rows are backbone layers, so the two are not the same count.
• Warm-up. Table 2 lists 500 warm-up iterations with factor 0.333 tab. 2. Neither the OrientedFormer configs nor configs/_base_/schedules/schedule_1x.py contain a warm-up setting.
• Augmentation. The paper says random flips §4.2. The DOTA config uses flips with probability 0.75 in horizontal, vertical and diagonal directions [code: configs/base/datasets/dota.py#L10-L16]. The DIOR config uses probability 0.75 [code: configs/base/datasets/dior.py#L10-L15].
• Cropping. The paper crops DOTA into 1024 × 1024 patches with 200-pixel overlap §4.2. The training pipeline reads a pre-split directory and resizes to 1024 [code: configs/base/datasets/dota.py#L5] [code: configs/base/datasets/dota.py#L10]. The split script was not inspected.
• Clipping labels. Table 2 labels the values "clip gradient value 1.0" and "clip gradient norm 2.0". The config sets max_norm 1 and norm_type 2 orientedformer_le90_r50_q300_layer2_head64_point32_1x_dior.py. These are the same settings under different labels.

Pitfalls

• Two different τ. The code's learnable per-head τ scales the bias, and the constant τ = 1 sits inside the distance term. Implementing from the paper alone gives the bias without the learnable scale, which changes the attention behaviour.
• Distance normalisation. Following the code means dividing the distance by the extra scale term, so the bias values differ from Eq. 9 as printed orientedformer_decoder_layer.py. Far pairs approach log(1e-7) ≈ −16.1, because the bias is log(1/(τ + d) + ε) and the square roots are clamped at 1e-7 orientedformer_decoder_layer.py.
• Sampling exponent. Eq. 13 as printed (2^(z−r)) does not match the paper's own definitions of z and r (Eq. 11). Follow the definitions, which the code does.
• Rotation about the origin. Read literally, Eq. 14 rotates coordinates about the image origin, which would move every sample far from its box. The intended meaning, and the code's behaviour, is rotation about the box centre.
• Hidden NMS. The end-to-end claim holds for the decoder output, but query selection in the code uses NMS. A re-implementation that drops it will get different queries.
• Cost hidden in the abstract. The paper states the parameter count is slightly larger than the baselines §4.9. The table shows 529 GFLOPs against 186 for ARS-DETR and Deformable DETR-O tab. 14, about 2.8 times the compute.
• Ablation text vs table. §4.4 says the sampling-point count goes "from 4 to 32", but the table starts at 8 with 65.60 tab. 9 (b).
• DOTA-v1.5 caption and AP. Table 5 is captioned as small-object results, but its AP50 column is not the mean of the six classes shown (for OrientedFormer that mean is 64.58, against 67.06). The AP50 is probably over all 16 classes, but the paper does not say so.
• Not uniformly best. Oriented R-CNN (ResNet-50) has higher AP50 than OrientedFormer on DOTA-v1.0 single-scale (75.87 against 75.37) and higher mAP on HRSC2016 (90.40 against 90.17 at VOC07). The text calls the HRSC2016 result competitive. Read the comparisons from the tables.
• ICDAR baselines. These were re-implemented by the authors in MMRotate with the same settings §4.3, so they are not the baselines' own published numbers.
• Hardware for training. The paper trains on two RTX 2080 Ti GPUs with a batch of 4 §4.2. Speed numbers in Table 14 do not state the hardware they were measured on.
• Version. The fetched source is the version recorded in the metadata (v2). Equation and table numbers follow that source.

Lineage

• Builds on: DETR for the set-prediction decoder and the Hungarian matching §2.1 §3.6. Deformable DETR, the deformable-attention baseline it is compared with §2.2. DAB-DETR for box-based positional encoding §2.3. DDQ for the query-initialisation method, which the code uses in its query-selection head §3.1. Mip-NeRF is cited for the Gaussian expectation identity §3.3.
• Compared against: ARS-DETR (closest end-to-end baseline), AO2-DETR and Deformable DETR-O with CSL as end-to-end baselines §4.3 tab. 4. CNN detectors including DCFL, Oriented R-CNN and RoI Transformer tab. 3 tab. 4 tab. 8.
• Superseded by: not known from the content of this wiki.