Remote Sensing Image Super-resolution and Object Detection: Benchmark and State of the Art

In plain words

Spotting cars, boats and planes in satellite and aerial pictures is hard when the objects cover only a few pixels. This paper argues that the field lacks a good dataset for that case, so it builds one: 1,759 high-resolution image tiles in which people drew boxes around five kinds of objects.

It also proposes a two-step pipeline. First, a generative network makes a low-resolution picture sharper. Then a standard object detector looks for objects in the sharpened picture. The network is trained in both directions, from low to high resolution and back, so that the sharpened picture also reproduces the original low-resolution one. The authors report that the sharpened pictures look better and that detection scores are close to the best baseline, but the detector does not beat the baseline on the full-resolution test images.

TL;DR

• RSSOD: 1,759 hand-annotated tiles with 22,091 horizontal boxes, five classes (vehicle, tree, airplane, ship, low vegetation), about 0.05 m resolution §3.
• MCGR: a cyclic GAN built from residual blocks with feature aggregation, followed by a YOLOv5 detector, trained with one weighted sum of three losses §4.1 eq. 5.
• Author-reported: best x4 super-resolution PSNR (34.68 dB) on the test set tab. 4; detection at x2 and x4 beats the other tested detectors at IoU 0.10 tab. 5.
• The paper's own test table shows YOLOv5 on full-resolution images ahead of MCGR (0.76 vs 0.746) tab. 5.
• No official code was found.

Problem

Object detectors degrade badly on objects that cover only a few percent of the image §2.2.2.2. Remote-sensing detection datasets are either built for scene classification or contain too few small objects, and detection pipelines that upsample first have not been benchmarked on a dataset built for that purpose §1 §2.1 tab. 1.

Method

Dataset (RSSOD)

1. Source tiles: 1000×1000 patches were cut from 38 ISPRS Potsdam tiles with a 100-pixel overlap, 36 patches per tile, giving 1,368 images §3.1.
2. Further images come from four other sources (UC Merced, NWPU-RESISC45, a Draper chronology set and ship images) to a total of 1,759 images, average size 856×853 §3.1 tab. 2.
3. Classes: vehicle, tree, airplane, ship and low vegetation. Buildings were left out because they fill most of the urban tiles §3.2. Most objects are small §3.1 fig. 3.
4. Boxes are horizontal and stored in YOLO form (class, centre x, centre y, width, height). Labels were reviewed in three rounds §3.3.
5. Split: 70:20:10 by image, giving 1,232 train, 351 validation and 176 test images §3.2 fig. 4.
6. Subsets for one, two, four and five classes were defined for training §3.3 §5.1.

MCGR: super-resolution with residual feature aggregation

7. The generator replaces the plain residual blocks of SRResNet and EDSR with a residual feature aggregation (RFA) block. Its features are combined through a 1×1 convolution §4.1 fig. 6.
8. The HR generator is an RFA network of 48 blocks with 3×3 kernels; the paper also gives a "block size" of 64×64 §4.1.
9. A second generator (LR generator) maps HR images back to LR. Training enforces consistency in both directions, which is the cyclic part §4.1 fig. 7.
10. The critic is a WGAN-style discriminator trained with a gradient penalty weighted by λ = 10 §4.1 eq. 3.
11. A YOLOv5 detector takes the HR generator output and predicts boxes and classes §4.1 fig. 8.
12. Training uses one weighted total loss over the generator, critic and detector eq. 5 §4.3.

# Cycle consistency between the two generators  [eq. 1]
# Input:  lr (B, 3, h, w); hr (B, 3, H, W)
# Output: scalar cycle loss

def cycle_loss(lr, hr, hr_generator, lr_generator):
    # Super-resolve the LR input and compare with the HR target  [eq. 1]
    sr = hr_generator(lr)                         # (B, 3, H, W)
    loss = l1(sr, hr)
    # Degrade the HR target and compare with the LR input  [eq. 1]
    lr_fake = lr_generator(hr)                    # (B, 3, h, w)
    loss = loss + l1(lr_fake, lr)
    # HR -> LR -> HR cycle, mean squared error to the HR target  [eq. 1]
    hr_cycle = hr_generator(lr_fake)
    loss = loss + mse(hr_cycle, hr)
    # LR -> HR -> LR cycle, mean squared error to the LR input  [eq. 1]
    lr_cycle = lr_generator(sr)
    loss = loss + mse(lr_cycle, lr)
    return loss

# Weighted training objective  [eq. 2] [eq. 3] [eq. 4] [eq. 5]
# Input:  sr (B, 3, H, W) from the HR generator; hr (B, 3, H, W)
#         detector_boxes, target_boxes: box coordinates for the detector
# Output: scalar total loss

def total_loss(sr, hr, critic, detector_boxes, target_boxes):
    # Generator term: L1 on the SR output  [eq. 2]
    generator_term = mean(abs(sr - hr))
    # Critic term with gradient penalty, lambda = 10  [eq. 3] [§4.1]
    # The paper samples the penalty points from {HR, SR} images; it does
    # not say whether they are interpolated between the two  [§4.1]
    critic_term = critic_loss(critic, hr, sr, lambda_gp=10.0)
    # Detector term: squared error on centre and size  [eq. 4]
    # Matching of predictions to targets is not stated  [§4.1]
    box_term = sum_squared_error(detector_boxes, target_boxes)
    # Weighted sum: mu1 = 0.90, mu2 = 10, mu3 = 0.10  [eq. 5] [§4.3]
    return 0.90 * generator_term + 10.0 * critic_term + 0.10 * box_term

Key equations

Equation 1, cyclic objective for the two generators. Written as printed in eq. 1:

\mathcal{L}(SRCGAN) = \mathcal{L}_{L1}(HR_{GEN}(I_{LR}), I_{HR}) + \mathcal{L}_{MSE}(HR_{GEN}(LR_{GEN}(I_{HR})), I_{HR}) + \mathcal{L}_{L1}(LR_{GEN}(I_{HR}), I_{LR}) + \mathcal{L}_{MSE}(LR_{GEN}(HR_{GEN}(I_{LR})), I_{LR}) eq. 1

Equation 2, generator L1 loss over N samples:

\mathcal{L}(HR_{GEN}) = \frac{1}{N} \sum_{i=1}^{N} \left\lVert HR_{GEN}(I^{i}_{LR}) - I^{i}_{HR} \right\rVert_{1} eq. 2

Equation 3, critic (discriminator) loss. The paper writes sums over the distributions. The gradient is printed as a Δ symbol, which we read as the gradient:

\mathcal{L}(Dis) = \sum_{I_{HR}} Dis(I_{HR}) - \sum_{I_{SR}} Dis(I_{SR}) + \lambda \sum_{I_{ran}} \left( \lVert \nabla Dis(I_{ran}) \rVert_{2} - 1 \right)^{2}, \quad \lambda = 10 eq. 3

Equation 4, detector box term. The text defines (x, y, w, h) and (x', y', w', h') inconsistently (see Pitfalls):

\mathcal{L}(Y5) = \sum_{i=0}^{Det} \sum_{j=0}^{Anc} \left[ (x - x')^{2} + (y - y')^{2} + (w - w')^{2} + (h - h')^{2} \right] eq. 4

Equation 5, total loss with weights 0.90, 10 and 0.10:

\mathcal{L}(T) = \mu_{1} \mathcal{L}(HR_{GEN}) + \mu_{2} \mathcal{L}(Dis) + \mu_{3} \mathcal{L}(Y5) eq. 5

Hyperparameters

Results

Author-reported. Setup: RSSOD test set; super-resolution metrics at scale factor 4 tab. 4; detection at IoU 0.10 on the five-class test set, with detectors trained on HR training images §5.2 tab. 5.

Super-resolution at ×4 on the test set tab. 4:

Five-class detection mAP on the test set tab. 5:

Training-run mAP, as reported in the text and the abstract. §5.1 fig. 12 These are not the test-set numbers above:

Other results:

• Single-image SR figures show MCGR recovering texture that Real-ESRGAN, SwinIR-L, BSRGAN and DRN smooth out fig. 9 fig. 10 fig. 11.
• Transfer to DroneDeploy §5.3 fig. 14: qualitative only, no metric.
• Transfer to the Aerial Dataset of Floating Objects (AFO) §5.3 fig. 15: MCGR mAP 0.845 at IoU 0.50, against 0.8216 for an ensemble by Gąsienica-Józkowy et al. The paper calls this state of the art; the training setup for this transfer is not stated.
• Inference time is 16.61 ms per image tab. 5 §5.2. The text states about four times YOLOv5 §5.2; the table does not state the hardware or whether the time includes super-resolution.

Paper vs code

No official code.

We searched the following. The paper names no project page, and its only footnote is a Mendeley Data DOI for the dataset (10.17632/b268jv86tf.1), with no code. A GitHub repository search for "MCGR" returned wangren09/MCGR: a personal account, no description and no license. Its README and file list concern adversarial defence (curve-finding and PGD scripts) and do not mention super-resolution, remote sensing or RSSOD, so we do not use it. A search for "RSSOD" returned only unrelated projects, and a search for "residual feature aggregation super-resolution detection" returned nothing.

Pitfalls

• Abstract mAP numbers are training-run values. The abstract's 0.758, 0.881, 0.841 and 0.983 are described in §5.1 as training mAP for the four class subsets §5.1 fig. 12. The test-set numbers are in tab. 5. The abstract's claim that MCGR surpasses YOLOv5 holds at ×2 and ×4 only; at full resolution tab. 5 YOLOv5 scores 0.76 against MCGR's 0.746.
• Detection metric is inconsistent. §4.2 says mAP@0.5 with a low IoU and confidence threshold, tab. 5 says IoU 0.10, and the training figures use mAP@0.5 and mAP@0.95 fig. 12. The confidence threshold is not given.
• Single figures are attributed to different runs. §5.1 says four-class training reaches 0.983 for vehicle, but the abstract gives 0.983 for the one-class run, and the one-class panel of fig. 12 labels its class as vehicle. §5.1 also gives "0.265 at 0.5 IoU" for five classes, which conflicts with the 0.758 reported in the same section. Treat these as unreliable without the code.
• Garbled PSNR sentence. §5.1 says NLSN, EDSR and DRN reach PSNR of 31.17, 37.64 and 44.89 dB. Those are MSE values from tab. 4; the PSNR values are 33.48, 33.13 and 32.69 dB. The SSIM for DRN is printed as "085" §5.1.
• Eq. 4 is ambiguous. The text says (x, y, w, h) are the HR and SR-predicted boxes, and then that they are the ground truth and the prediction eq. 4. The equation has no objectness or class term. Whether it is the actual box loss used in training cannot be checked without code.
• Eq. 5 weight label. The text names the weights μ1, μ2 and μ5 §4.3, while eq. 5 uses μ3 for the detector term. The values agree.
• Critic formulation. Eq. 3 is written with sums over distributions rather than expectations. The gradient-penalty points are said to come from a uniform random choice of {HR, SR} images, not from interpolation §4.1. The paper cites the WGAN (Arjovsky et al.) but not the gradient-penalty paper.
• Generator adversarial term missing. The generator term in eq. 2 is only L1, and the generator's adversarial loss does not appear in eq. 1 or eq. 5. How eq. 1 and eq. 5 relate is not stated.
• Detector gradient path. Whether the detector's gradients reach the generator, and how boxes are matched to anchors, is not stated §4.1.
• Detector training data. §5.2 says the detectors were trained on HR images to avoid bias. It does not say which input the MCGR detector was trained on, SR images or HR images.
• Dataset size and resolution are inconsistent. Table 1 gives average image size 1000, while Table 2 and the text give 856×853. Source tiles differ in size (1000×1000, 256×256, about 421×389 for ships), and the detector input is 640×640. How they are resized is not stated.
• Annotation formats. The page-one abstract says YOLO and COCO formats; §3.3 describes only YOLO. We could not confirm a COCO export from the paper.
• Class subsets. §3.3 says the data is grouped into four, two and one class sets but does not list the classes in the four- and two-class sets.
• Dataset licence. The footnote says the dataset "will be publicly available" under a Mendeley DOI. Its licence is not stated, and it combines images from ISPRS Potsdam, UC Merced, NWPU-RESISC45, a Draper set and ship images from Google, each with its own terms. Check before any reuse.
• Timing and hardware. The 16.61 ms figure has no stated hardware for inference, batch size, or whether SR is included tab. 5.
• Single-scale SR evaluation. Table 4 reports SR metrics only at ×4 on the test set. ×2 SR quality is not reported.
• Convergence. The paper says four-class training converges by about 40 epochs and one- or two-class training by about 25 §5.1. Training is 100 epochs in all cases §4.3.

Lineage

• Builds on: SRResNet and EDSR residual blocks §4.1 fig. 6; the Wasserstein GAN (Arjovsky et al.) for the cyclic network §4.1; and the 1×1 feature-aggregation step of Bashir & Wang 2021b (Remote Sensing 13(9), 1854) §4.1.
• Compared against: Real-ESRGAN, SwinIR-L, BSRGAN, DRN, EDSR, NLSN for super resolution tab. 4; YOLOv5, RetinaNet, SSD, Faster R-CNN and EfficientDet-D5 for detection tab. 5.
• Superseded by: not known from this card.