Transformers in Remote Sensing: A Survey

In plain words

Transformers are a kind of neural network first built for language. Their core tool, self-attention, lets every part of an input look at every other part and decide how much each one matters. For images, the picture is cut into small squares that are handled as a sequence, much like words in a sentence.

Satellite and aerial images are a special case. They come from different sensors: very high-resolution optical cameras, hyperspectral scanners that record many narrow wavelength bands, and radar. Researchers have tried transformers on all three.

This paper is a survey. It gathers more than 60 recent papers that apply transformers to these kinds of images, groups them by task, and puts their reported results side by side. It also notes a pattern in this work: most of it mixes transformers with convolutions and starts from weights pre-trained on ordinary photos. It closes with open questions, such as self-supervised pre-training on unlabelled satellite data and lighter models for large images.

TL;DR

• A survey of transformer-based methods in remote sensing, covering very high-resolution (VHR) optical, hyperspectral and SAR imagery §1.
• It discusses more than 60 methods §1. The three overview tables list 64 entries: 37 for VHR, 16 for hyperspectral and 11 for SAR tab. 5 tab. 7 tab. 8.
• It reproduces the benchmark numbers from the original papers for scene classification, detection, change detection, segmentation and hyperspectral classification. It does not describe re-running any of them §5 §6.
• Most reviewed methods are hybrids of CNNs and transformers, and most start from ImageNet pre-training. Open issues include self-supervised pre-training and computational cost §8.
• Read it for a map of the field and its reported numbers, not for an implementable method. The repository linked from the abstract is a paper list, not model code (see Paper vs code).

Problem

Remote-sensing research on transformers is growing quickly, and the authors say the influx of methods makes it hard to keep up §1. The reviews cited in the related-work section cover machine learning, active learning and deep learning for remote sensing, not transformers §2. The survey therefore sets out to collect and compare recent transformer work for the three imaging types above §1.

The background argument is that convolutions capture local structure and share weights across positions, while self-attention models long-range relations with content-dependent weights §1 §4.2. The survey notes that vision transformers are pre-trained on large datasets and use more training data than CNNs §4.2 §8.

Method

This is a literature survey, so there is no algorithm of its own. The card follows the survey's structure and gives the attention computation it builds on.

Structure of the survey

1. Imaging types: VHR, hyperspectral and SAR data, with their resolutions and typical uses §3.
2. Background: CNN backbones (AlexNet, VGG, ResNet) §4.1; the vision transformer, Swin and PVT backbones §4.2.
3. Task sections for VHR §5, hyperspectral §6 and SAR §7, each with a summary table.
4. Discussion of open directions §8.

Background mechanism (the survey's own equations, rewritten as pseudocode; the survey gives no code)

# Multi-head self-attention over patch tokens  [§4.2] [eq. 1] [eq. 3]
# Input:  tokens (M, E), M patches with embedding size E
# Output: updated tokens (M, E), output width as printed: see Pitfalls

def multi_head_self_attention(tokens, heads):
    # Each head has its own query, key and value projections  [§4.2]
    head_outputs = []
    for head in heads:
        queries = tokens @ head.W_Q       # (M, E_q)
        keys = tokens @ head.W_K          # (M, E_k), with E_k = E_q
        values = tokens @ head.W_V        # (M, E_v)
        # Scaled dot-product scores, softmax over keys, weighted sum  [eq. 1]
        scores = queries @ keys.T / sqrt(E_q)
        head_outputs.append(softmax(scores) @ values)
    # Concatenate the heads channel-wise and apply the output matrix  [eq. 3]
    return concatenate(head_outputs) @ W_O

Attention for the decoder is written with a causal mask eq. 2; see Pitfalls for how the printed form differs from a working implementation.

Architectures named as building blocks §4.2

• ViT: the image is split into patches that are flattened and projected, then passed through a standard transformer encoder. No convolutional inductive bias is built in; pre-training uses ImageNet-21k or JFT-300M §4.2.
• Swin: attention is computed inside non-overlapping local windows, with cross-window connections; the survey describes its cost as linear in image size §4.2.
• PVT: a pyramid that shrinks progressively, with spatial-reduction attention, producing multi-scale features for dense prediction §4.2.

Methods per task (names as listed in the overview tables; details in the cited sections)

• Scene classification §5.1 tab. 1 tab. 5: V16-21K (ViT with augmentation), TRS (ResNet with its last three bottlenecks replaced by attention encoders), TSTNet (two-stream Swin with an edge stream), CTNet (ViT stream plus CNN stream), HHTL (heterogeneous and superpixel-based homogeneous patches), RSP (backbones pre-trained on MillionAID).
• Object detection §5.2 tab. 2 tab. 5: SAIEC, T-TRD-DA, GANsformer, ADT-Det, PointRCNN, HybridNetwork22, Oriented RepPoints, O2DETR, AO2DETR, RBox, Rodformer. Most are hybrid CNN-transformer detectors; a few adapt DETR §5.2.
• Change detection §5.3 tab. 3 tab. 5: CD-Trans, MSPSNet, SwinSUNet, TransUNetCD, Hybrid-TransCD. UVACD is discussed in the text and listed in tab. 3 but not in the overview table.
• Semantic segmentation §5.4 tab. 4 tab. 5: CCTNet, STransFuse, Trans-CNN, SwinTF, Efficient-T.
• Building extraction, image captioning, super-resolution §5.5 tab. 5: STT, STEB-UNet, BuildFormer, T-Trans; TRL, MLAT, Ren et al. (topic-token masked transformer); TR-MISR, MSE-Net, SRT.
• Hyperspectral classification §6.1 tab. 6 tab. 7: pure transformers (HSI-BERT, SpectralFormer, the spectral-spatial transformer SSTN, DSS-TRM); hybrids with convolutions (CTN, HiT, MCT, SSFTT); multi-modal fusion with LiDAR or SAR (MFT, DHViT).
• Hyperspectral pansharpening §6.2 tab. 7: CTIN, HyperTransformer, PMACNet, CPT-noRef, MSIT, Su et al.
• SAR interpretation §7.1 tab. 8: classification (ViT-PolSAR, GLNS, ST-PN), segmentation (GCBANet), detection (CRTransSar, Geospatial Transformers, SFRE-Net, 3DET-ViT).
• Other SAR tasks §7.2 tab. 8: despeckling (ID-ViT), change detection (CLT), registration (CF-ViT).

Hyperparameters

The survey trains nothing and gives no hyperparameters of its own. The settings below are the evaluation protocols it states for the benchmark tables.

Results

Author-reported. The survey does not re-run these methods. Each row comes from the paper named in the source column, under that paper's own setup, backbone and pre-training. Compare the rows only with care.

Scene classification on AID, 20:80 split tab. 1

Oriented object detection on DOTA, mAP tab. 2

Change detection, F1 score tab. 3

Semantic segmentation, overall accuracy tab. 4

Hyperspectral classification, overall accuracy, 200 training samples per class tab. 6

SAR methods and the pansharpening and VHR overview entries have no numeric results in the survey. Their numbers are in the original papers.

Paper vs code

The abstract links https://github.com/VIROBO-15/Transformer-in-Remote-Sensing. The GitHub API returns that path as amandpkr/Transformer-in-Remote-Sensing, which is the repository we cloned at code_commit. Its owner is Amandeep Kumar, a listed author, and the latest commit is by the same name. That is the official repository.

• It contains a README and one figure. It is a curated paper list that mirrors the survey, not implementation code. It has no LICENSE file, so code_license is recorded as none, and no code is quoted.
• The survey does not implement any method, so there are no algorithms or hyperparameters to compare.
• The list differs from the survey's tables. Two papers in the repository's change-detection and building-extraction sections, "Remote sensing image change detection with transformers" (arXiv 2103.00208) and "Building extraction with vision transformer" (arXiv 2111.15637), do not appear in the overview table tab. 5. The survey's UVACD row is commented out of its overview table in the LaTeX source, though the text discusses it §5.3.
• No discrepancies in the method descriptions were checked, because the survey has no method code to check them against.

Pitfalls

• No re-runs. Every number is taken from a different paper with its own split, backbone, pre-training and sensor. A higher number in one row does not show a better method under a common protocol. The tables differ in backbone and pre-training, so the survey's ranking is only a summary of reported results.
• Preprints mixed with venues. Three DOTA rows (Oriented RepPoints, O2DETR, AO2-DETR) are listed with venue "arXiv" in tab. 2, so they are not peer-reviewed venues in that table.
• Masked attention as printed eq. 2. The survey writes the causal mask as an element-wise product on the scaled scores before softmax, and says masked scores "are set to zero". A zero score is not zero weight after softmax (exp of 0 is 1), so a working implementation sets masked entries to negative infinity before softmax. This is our reading of the printed equation, not a statement by the survey.
• Notation inconsistencies. The text uses M for the number of patches but also writes X in R^{N×E}, and eq. 2 uses n for the mask size and d_q where eq. 1 uses E_q. The output projection is printed as "W^O × R^{h·E_v×M}", where the intended product and output width are not stated. The text defines an attention matrix A that the equations never use.
• Table and figure numbers were derived from LaTeX source order (captions in source order: AID table is 1, DOTA 2, change 3, segmentation 4, VHR overview 5, hyperspectral results 6, hyperspectral overview 7, SAR overview 8). They were not checked against the compiled PDF; verify before citing a number to readers.
• Computational cost is named as a weakness of global self-attention, and the survey points to efficient variants without measuring them itself §8. Its tables give no speed, memory or parameter counts.
• Data scale. Several methods rely on ImageNet pre-training or on the large remote-sensing dataset MillionAID; the survey notes that all approaches in the AID table except RSP use ImageNet models §5.1.

Lineage

• Builds on: the transformer and ViT work cited in §4 (vaswani2017attention, dosovitskiy2021an), Swin and PVT. Not linked here because no cards exist for them.
• Compared against: earlier remote-sensing reviews named in §2 (active learning, hyperspectral learning and deep learning for remote sensing) and transformer surveys for general vision, medical imaging and video §2.
• Superseded by: not stated.