SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding

In plain words

Satellite and aerial photos show roads, buildings, wind turbines, floods and much more. Most computer-vision work on them handles one kind of object at a time, and labelled examples are scarce for many of those kinds. This paper builds a large labelled collection of such images, called SatlasPretrain. It holds hundreds of millions of labels across many kinds of objects and land features, and each label is tied to a place and a date.

The paper also presents SatlasNet, one model that learns all these label types at once. A single shared image encoder feeds a separate output for each label type. The collection is meant to be used as a starting point: a model trained on it and then fine-tuned on a small task does better than one that starts from general-purpose photos. The authors report these gains themselves, on a set of downstream tasks they chose.

TL;DR

Use it when you need a pre-trained remote-sensing backbone or a training corpus that spans many label types. The dataset is large (856K tiles, 302M labels) and needs care with its label validity and recall. The body gives the architecture and most training settings, but leaves some details to a supplement that is not in its source. The reported gains are author-reported.

Problem

Remote-sensing benchmarks are small and split by task. Object-detection and segmentation datasets such as DOTA, iSAID and DeepGlobe each contain fewer than 10K images, far below what architectures such as ViT and CLIP need to reach peak accuracy §1. Existing benchmarks also cover one label type each, which blocks transfer across tasks and categories §1 tab. 1. Image time series from mixed sensors, and long-range spatial context, are open problems that existing datasets do not test well §1 §3.

Method

Dataset (SatlasPretrain)

• Tiles. The data is organised as 856K Web-Mercator tiles at zoom level 13, each covering up to 25 km² §3.1. Of these, 828K are for training and 28K for testing §3.1.
• Low-resolution mode. All tiles have 512×512 Sentinel-2 images at 10 m per pixel. Each tile has 8 to 12 images from 2022, plus historical images from 2016 to 2021 §3.1. Only 122 of the 137 categories are evaluated in this mode because some are not visible at this resolution §3.1.
• High-resolution mode. 46K tiles (45.5K train, 512 test) have 8192×8192 NAIP images at 1 m per pixel: 3 to 5 public-domain aerial images per tile from 2011 to 2020. These tiles are US-only §3.1.
• Labels. There are 137 categories under seven label types: segmentation, regression, points, polygons, polylines, properties of objects, and classification §3.2. Labels come from new expert annotation (12 categories), Amazon Mechanical Turk (9 categories), OpenStreetMap (101 categories extracted), NOAA lidar topobathy maps (5,123 tiles), WorldCover (11 categories), Microsoft Buildings (70 GeoJSON files) and C2S flood and cloud labels §3.2.
• Space and time. Labels are tied to geographic coordinates and time ranges, not to a single image, so a model can use several images of one place and labels can refer to a specific date §3.
• Validity. A category is valid in only some tiles. Outside the valid tiles, its label is unknown and must be masked from the loss, not treated as empty §3.2. Categories are split into slow-changing ones, labelled against the newest image at a tile, and dynamic ones such as vessels and floods, which carry the timestamp of their image §3.2.
• Recall and precision. OSM objects have high precision but variable recall. Heuristics prune tiles that probably miss objects §3.2. For 13 categories with low recall, the test set was relabelled by hand after a model proposed candidates §3.2. Precision per category is above 99% for 116 of 137 categories §3.2.

Model (SatlasNet)

1. Each input is a time series of spatially aligned images of one window, each with several bands §4.
2. A Swin-Base backbone runs on each image separately and returns feature maps at four scales §4.
3. At each scale, max pooling over the time axis gives one set of features §4.
4. Seven output heads read these features, one per label type §4. Polylines use a segmentation head, and the mask is turned into lines by thresholding, morphological thinning, and line following with simplification §4.

The code adds a feature pyramid and an upsampling decoder between the backbone and the heads, and uses one head per task rather than one per label type; see Paper vs code.

# SatlasNet forward pass  [§4] [fig. 4]
# Input:  images (T, C, H, W), one window; T = 8 Sentinel-2 or 4 NAIP
# Output: one prediction per label type

def satlasnet_forward(images, valid_heads):
    # Run the same backbone on every image of the time series  [§4]
    per_image_maps = [swin_backbone(image) for image in images]
    # Each image gives four maps, at 1/4, 1/8, 1/16 and 1/32 scale  [§4]
    scale_stacks = stack_by_scale(per_image_maps)
    # Take the max over the time axis, separately at each scale  [§4]
    fused_maps = [max_over_time(stack) for stack in scale_stacks]
    # Code adds a feature pyramid over the four scales
    # [code: satlas/model/model.py#L324-L347]
    pyramid_maps = feature_pyramid(fused_maps)
    # Every head reads the shared maps and predicts its own label type  [§4]
    outputs = {}
    for head in valid_heads:
        outputs[head.name] = head(pyramid_maps)
    return outputs

Training and sampling

# One training step  [§5.1]
# Optimizer: Adam; batch size 32 (paper)
# [code: satlas/cmd/model/train.py#L195-L196]

def training_step(optimizer, batch):
    # Each head's loss covers only the examples where its category is valid
    # [§3.2]
    per_head_losses = model(batch.images, batch.targets)
    # Code averages the per-head losses with equal weight
    # [code: satlas/cmd/model/train.py#L297]
    loss = per_head_losses.mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss

# Tile sampling  [§5.1]
# Weight each tile by the rarest class it contains, capped (code)
# [code: satlas/cmd/model/compute_bal_weights.py#L117-L128]

def tile_weight(tile_classes, class_counts, peak_count):
    # Inverse frequency per category, capped at 20 in the code
    # [code: satlas/cmd/model/compute_bal_weights.py#L14, #L117-L120]
    class_weight = {
        name: min(peak_count / count, 20)
        for name, count in class_counts.items()
    }
    # A tile takes the largest weight of any category in it  [§5.1]
    return max([class_weight[name] for name in tile_classes], default=0)

Training details: the backbone starts from ImageNet weights, inputs are 512×512 windows, and augmentation is random crop, horizontal and vertical flips, and random resize §5.1. High-resolution inference splits each 8K tile into 256 windows of 512×512 and merges the outputs §5.1.

Evaluation

• SatlasPretrain test split. Compared with eight baselines, each fine-tuned on a label type they handle §5.1. Three SatlasNet variants are trained: per-type, joint over all categories, and fine-tuned from the joint model §5.1. Each is run in single-image and multi-image mode §5.1.
• Downstream tasks. Seven tasks, each evaluated with 50 examples and with the full training set §5.2. Pre-training on SatlasPretrain high-resolution images is compared with pre-training on BigEarthNet, MillionAID, DOTA, iSAID, and two self-supervised methods, MoCo and SeCo §5.2.

Hyperparameters

Values in the paper are stated in the body; the supplement that the paper points to is not in its source and is not used here.

Results

Author-reported. Setup: SatlasPretrain test split, 512×512 inputs, Sentinel-2 and NAIP, metrics as listed in §5.1.

Metrics as given in the table caption: accuracy for segmentation, regression, points, polygons, polylines and properties, with absolute error for regression (lower is better) and accuracy for classification tab. 2. The text in §5.1 names different metrics (see Pitfalls).

SatlasPretrain test results tab. 2

Headline statements from the text, not recomputed here: the single-image model matches or beats the per-type baselines when trained per type §5.1. Fine-tuning from the joint model gives an average 7.1% relative gain over per-type training across label types and image modes §5.1. Multi-image input gives a further 5.6% relative gain §5.1.

Downstream tasks, pre-training comparison tab. 3

Each cell is the metric with 50 training examples (left) and with the full training set (right). Accuracy for UCM, RESISC45 and AID; F1 for FMoW, Mass Roads, Mass Buildings and Airbus Ships tab. 3.

Author-reported setup §5.2: the same SatlasNet architecture is used for every pre-training source, with the Swin backbone fine-tuned on each downstream dataset. The text reports the 50-example gains as 18% over ImageNet and 6% over the next best baseline §5.2.

Qualitative results: wind turbines and water towers are detected well; oil wells are under-detected; polylines such as roads and railways come out as short noisy segments §5.1 fig. 5.

Paper vs code

Official code: allenai/satlas at commit c8b9aa5d4acdd3e4f58eb7cbb28ac18bb12c985f. The paper's abstract and its supplementary-material pointer (Appendix) both lead to this repository. Code license Apache-2.0 LICENSE. The dataset is documented under a separate ODC-By file DataLicense; the paper does not state a dataset license.

• Backbone family. Paper: Swin Transformer, Swin-Base §4. Code: torchvision swin_v2_b, i.e. Swin V2 swinb_si_rgb.txt model.py. ModelArchitecture.md names Swin-v2-Base for the deployed models ModelArchitecture.md.
• Backbone initialisation. Paper: ImageNet-pretrained §5.1. Code: loads ImageNet weights, then replaces the patch-embedding convolution with a freshly initialised layer that takes the number of input bands model.py model.py. The first layer is therefore not pre-trained, for RGB too.
• Intermediate layers. Paper: the pooled four-scale features go straight to the heads §4. Code: a feature pyramid with 128 channels model.py, then an upsampling decoder with transposed convolutions back to full resolution model.py. The code's ModelArchitecture.md describes the same stack.
• Heads. Paper: seven heads, one per label type §4. Code: one head per task. The Sentinel-2 single-image RGB config defines 19 tasks (polygon, point, land cover, polyline, DEM, crop type, tree cover, park and power-plant types, road type, wildfire, vessel, smoke, snow, flood, cloud) swinb_si_rgb.txt. Points and polygons use a Faster R-CNN head; polygons add a mask head model.py. Classification heads average-pool the lowest-resolution map and apply a linear layer model.py.
• Learning rate. Paper: 1e-4 §5.1. Code pretraining configs: 1e-5 swinb_si_rgb.txt swinb_mi_rgb.txt swinb_si.txt. The deployed explorer configs use 1e-4 satlas_explorer_wind_turbine.txt. Which value reproduces the paper's numbers is not stated.
• Training length. Paper: 100K batches of 32 §5.1. Code: NumEpochs 10000, no step count swinb_si_rgb.txt. The loop runs NumEpochs epochs train.py, and each epoch draws as many weighted samples as there are options dataset.py. The total is therefore not fixed by the config.
• Warmup. Paper: not stated. Code: 8192 examples of linear warmup, which is 256 steps at batch 32 swinb_si_rgb.txt train.py train.py.
• Scheduler. Matches the paper: halve on plateau, minimum 1e-6 swinb_si_rgb.txt. It is stepped on the mean training loss at each summary point, after warmup, not on validation loss train.py train.py.
• Sampling cap. Paper: maximum inverse frequency §5.1. Code: inverse frequency capped at 20 compute_bal_weights.py compute_bal_weights.py.
• Augmentation. Paper: random crop, flips, random resize §5.1. Code: a crop side drawn from 416–512 augment.py, flips with probability 0.5 augment.py, then resize to 512 swinb_si_rgb.txt.
• Multi-task loss. Paper: not stated. Code: the mean of the per-head losses, with equal weights train.py. A head with no valid example in the batch contributes a zero, but still counts in the mean model.py model.py model.py.
• Metric names. Paper text: F1 for segmentation, mAP for points and polygons, GEO accuracy for polylines §5.1. Code: the detection and instance evaluators report F1 with a grid of per-class score thresholds evaluate.py evaluate.py. The polyline task in the Sentinel-2 config uses F1 swinb_si_rgb.txt. No GEO-accuracy code was found in satlas/metrics or the evaluator registry.
• Point matching. The code has two matchers: a 20-pixel tolerance with IoU 0.5 evaluate.py and a 128-pixel distance threshold point.py. The paper states neither.
• Multi-band input scaling. Not in the paper body. Code: NAIP and RGB divided by 255 Normalization.md; non-TCI Sentinel-2 bands divided by 8160 and clipped Normalization.md; Landsat (N−4000)/16320 Normalization.md; Sentinel-1 divided by 255 Normalization.md.
• Mixed precision. Code uses half precision with gradient scaling swinb_si_rgb.txt train.py. The paper does not state this.
• Downstream fine-tuning. Paper: freeze the backbone for 32K examples, then fine-tune everything §5.2. Code: an Unfreeze option exists, set to 65536 examples in the deployed explorer configs train.py satlas_explorer_wind_turbine.txt. The downstream experiment scripts for the paper were not located in the code.
• Dynamic labels and high-resolution inference. Paper: sampled image placed last for dynamic labels §5.1; 256-window inference merge §5.1. The code was not checked for dynamic-label ordering, and the 256-window merge was not located in satlas/cmd/model/infer.py or satlas/model/evaluate.py.

Pitfalls

• Hyperparameters are not complete in the paper. Warmup, mixed precision, batch count, hardware and the downstream batch size are missing, and the supplement that holds some of them is not in the source. Use the code configs for these, and note which config you followed.
• Learning rate differs by a factor of ten. Paper 1e-4; pretraining configs 1e-5; deployed configs 1e-4. Start with the value in the config that produced the checkpoint you use.
• Backbone stem is re-initialised. The paper says the backbone starts from ImageNet weights, but the code replaces the first convolution with a new random layer model.py. Expect slower early training than "ImageNet init" suggests.
• Equal-weight mean over tasks dilutes losses. A task with no valid example in a batch adds a zero to the mean model.py train.py. With many sparse categories, the effective learning signal per task is smaller than the task count suggests.
• Unknown is not empty. A category is valid in only some tiles. Mask its loss where it is invalid, otherwise you teach the model that absence is a negative §3.2. The code masks per head using a validity flag model.py.
• Recall is uneven. OSM-derived categories have high precision but variable recall. The paper's estimate of at least 80% recall comes from 1300 tiles, for 13 categories §3.2. Training on these labels is noisier than test-set labels, so evaluate on the hand-labelled test split.
• Reported 18% and 6% are absolute. They are the differences in the 50-example Average column of Table 3 (0.56 vs 0.38 and 0.50), in points of an average that mixes accuracy and F1 across seven tasks tab. 3 §5.2.
• Dataset size claims need a reference point. Table 1 gives about 39× the pixels of FMoW (17003B vs 437B) and about 173× the labels of BigEarthNet (302222K vs 1750K). The paper states 40× and 150× without naming the reference datasets tab. 1 §3.
• Internal inconsistencies. Related Work says DOTA has 18 categories §2; Table 1 lists 19 tab. 1. The Table 2 caption describes every column as accuracy or error tab. 2, while §5.1 names F1, mAP and GEO accuracy.
• Two point-matching thresholds. A 20-pixel and a 128-pixel tolerance both exist in the code evaluate.py point.py. Report which one you used.
• Time series are only sometimes used. The single-image models ignore the time axis; multi-image models need 8 Sentinel-2 or 4 NAIP images per window, and fewer images are handled by skipping missing indices model.py. Check the number of images per tile before comparing against the table.
• Dataset license is unstated in the paper. The repository carries an ODC-By file for the data DataLicense. Check it before redistributing the dataset.
• Compute cost is high. Swin-Base at 512×512 with up to 8 images per sample, for 100K batches, is a large job. The paper does not state the hardware in the body.

Lineage

• Builds on: Swin Transformer for the backbone, Feature Pyramid Network and Mask R-CNN as code components; the paper cites Swin, FPN and Mask R-CNN by name only, with no card in this wiki.
• Compared against: ImageNet initialisation; pre-training on BigEarthNet, MillionAID, DOTA and iSAID; MoCo v2 and SeCo for self-supervised pre-training; per-task baselines PSPNet, LinkNet, DeepLabv3, ResNet-50, ViT-Large, Swin-Base, Mask R-CNN and ISTR §5.1 §5.2.
• Superseded by: not known from wiki content.