SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery

In plain words

Satellite photos are plentiful, but labelling them is slow and expensive. A common way to learn from unlabelled pictures is to hide most of each picture and train a network to fill in what is missing. SatMAE does this for satellite data, which differs from ordinary photos in two ways: the same place is photographed again and again over the years, and each photo can carry more colour channels than the red, green and blue of a normal picture.

For the repeated photos, the method hides parts of each photo separately, so the network cannot just copy a visible part from another date. It also tells the network when each photo was taken, using the year, month and hour, so it can learn that a season or a year changes how a place looks. For the extra channels, the method splits them into a few groups of related wavelengths and gives each group its own input layer and a label saying which group it is.

After this unlabelled pre-training, the network is fine-tuned on a labelled task such as classifying what kind of site a satellite image shows. The authors report that this beats earlier self-supervised methods for satellite images on the benchmarks they tested.

TL;DR

SatMAE is a masked-autoencoder pre-training recipe for satellite data with a time dimension or many spectral bands. It adds a temporal encoding, uses independent masking across the images of a sequence, and groups spectral bands with a group encoding. Results are author-reported on fMoW (RGB and Sentinel-2 versions), and on NAIP, EuroSAT, BigEarthNet and SpaceNet v1. The gains are clearest on fMoW; on some transfer tasks the method is not the best reported result.

Problem

Labelled satellite data is expensive, while unlabelled imagery is plentiful §1. Self-supervised pre-training methods of the time were designed for natural RGB images, and existing satellite methods could not handle temporal and multi-spectral data together §1 §2.

Concatenating a time series along the channel axis and applying MAE unchanged has four stated problems: the model may not generalise to a different temporal order, it cannot reason about the gap between acquisitions, it loses fine temporal detail in deeper layers, and it is not shift-invariant in time §4.1.

Method

The base model is MAE: an encoder sees only the visible patch tokens, and a lighter decoder reconstructs the image. The loss is mean squared error on the masked patches only §3. SatMAE keeps this structure and changes three things: how tokens are encoded, how masking is applied, and how bands are embedded.

Temporal SatMAE §4.1. A sequence of images is patchified separately, and all images share one patch embedding. The patch size in time is 1, so a sequence of length T gives L·T tokens §4.1. Each token receives a positional code for its place in the image and a temporal code built from the year, month and hour of its image (the day, minute and second are dropped) §4.1.1 eq. 2 fig. 2. The two codes are concatenated to the embedding width D.

Masking across time §4.1.2 fig. 3. Three variants are described. In consistent masking the same patch locations are hidden in every image of the sequence, as in VideoMAE. In independent masking the hidden locations differ per image. The third variant combines independent masking with cropping each image of the sequence at a different place. The appendix states that the temporal experiments use independent masking with a fixed fraction of masked patches for each image app. A.3.2.

Multi-spectral SatMAE §4.2. Sentinel-2 images have 13 bands at 10, 20 and 60 m resolution. The authors compare three ways of feeding them in. Stack treats all bands as input channels of one patch embedding. Group splits the bands into G groups, each with its own patch embedding sized to that group's band count, so the result is G·L tokens. The group encoding is a sinusoidal code of the group index, concatenated to the positional code eq. 3. Masking is either consistent (the same location hidden in every group) or independent (the hidden set is drawn over all group-location tokens) §4.2 §4.1.2.

Band groups used in the experiments §5.4. Bands B1, B9 and B10 are dropped, and the other ten are grouped as RGB+NIR (B2, B3, B4, B8), Red Edge (B5, B6, B7, B8A) and SWIR (B11, B12). The grouping is chosen so that each group has one spatial resolution and similar wavelengths. app. A.6 describes how the dropped bands were chosen, using a Stack model and mean-value masking.

Pseudocode for the temporal path, following the text above:

# Temporal SatMAE pre-training step  [§4.1]
# Input:  images (B, T=3, C, H, W); timestamps (B, T, 3) as (year, month, hour)
# Output: scalar reconstruction loss on masked tokens

def temporal_satmae_loss(images, timestamps, mask_ratio=0.75):
    # Embed each image of the sequence with one shared patch embedding  [§4.1]
    patch_tokens = [patch_embed(images[:, t]) for t in range(T)]
    tokens = concatenate(patch_tokens, axis=sequence)  # (B, T*L, D)

    # Add the positional code and the temporal code, which together fill
    # width D  [§4.1.1] [eq. 2] [app. A.3.2]
    tokens = tokens + concatenate(positional, temporal(timestamps))

    # Hide p_m of the L patch locations in each image  [§4.1.2] [app. A.3.2]
    visible, mask, restore = mask_per_image(tokens, mask_ratio)

    # Encode the visible tokens after a CLS token, then decode all tokens  [§3]
    latent = encoder(append_cls(visible))
    prediction = decoder(latent, restore, timestamps)

    # Mean squared error on the hidden patches only  [§3]
    return mean_squared_error(prediction[mask], patches[mask])

Pseudocode for the grouped band embedding:

# Group-channel tokenisation for multi-spectral input  [§4.2] [eq. 3]
# Input:  image (B, C=10, H, W) after dropping B1, B9, B10
# Output: tokens (B, G*L, D), G = 3 band groups

def group_tokens(image, groups, group_embeddings):
    # One patch embedding per band group, sized to that group's band count  [§4.2]
    per_group = []
    for group, embed in zip(groups, group_embeddings):
        per_group.append(embed(image[:, group]))  # (B, L, D)
    tokens = stack(per_group, axis=group)  # (B, G, L, D)

    # Add the location code and the group code, concatenated to width D  [eq. 3]
    tokens = tokens + concatenate(location_code, group_code)
    return flatten(tokens)  # (B, G*L, D)

Hyperparameters

Values are as stated in the paper unless marked as code. Code-derived values are in the README's commands and are cited to the code.

Results

All numbers are author-reported. Each row names its setup.

fMoW RGB, non-temporal (top-1 %, linear probe / fine-tune) §5.2

fMoW RGB, temporal (top-1 / top-5 %) §5.3

Ablation on the temporal model (top-1) §5.3: no temporal encoding, independent masking, consistent cropping: 78.07. Temporal encoding with consistent masking, consistent cropping: 78.45. Temporal encoding, independent masking, no consistent cropping: 79.90. Full model without test augmentation: 79.69. Full model with test augmentation: 81.49.

fMoW-Sentinel, validation (top-1 / top-5 %) §5.4

Ablation (top-1) §5.4: the chosen grouping with independent masking and group code (X) 59.30; the same without the group code 58.87; consistent masking with the group code 57.76; two halves (H) 57.78; random three groups (R) 58.76. ViT-B with the chosen setup: 59.11.

Transfer learning (top-1 or mIoU or mAP) §5.5

Longer pre-training on Sentinel, top-1 after fine-tuning app. A.3.3: ViT-B for 200 epochs 62.65; ViT-L for 50 epochs 61.48; ViT-L for 200 epochs 63.84.

Paper vs code

Compared against the official repository at commit 0b210aceb37a14bbbd897110db5b104b3271d818 (GitHub owner sustainlab-group, linked from the project page). The repository is licensed CC BY-NC 4.0, so only paraphrased pseudocode and citations are used here. The non-temporal fMoW RGB setup was not compared in detail; the vanilla MAE model file was not read.

• Sentinel pre-train length: the paper says 50 epochs app. A.3.3. The README's Sentinel command uses 200 epochs and 20 warmup epochs README.md. The 200-epoch run appears in the longer pre-training table app. A.3.3.
• Sentinel fine-tune: the paper gives base lr 1e-3 at effective batch 4096 app. A.3.3. The README uses blr 2e-4 at batch 8 x 16 x 8 = 1024 README.md. Under the code's scaling rule (lr = blr x batch / 256), the paper's 1e-3 at batch 4096 would be 1.6e-2 absolute, and the README gives 8e-4 absolute. The paper's "base" is ambiguous here; the README command is the closest concrete setting.
• Sentinel best config drop path: the paper gives 0.1 app. A.3.3. The README uses 0.2 README.md.
• Temporal pre-train: the paper says 100 epochs app. A.3.2; the README default is 100 README.md, but the published temporal checkpoint is listed as 50 epochs README.md. The checkpoint and the command do not match.
• Temporal masking: the appendix describes per-image masking at a fixed ratio app. A.3.2. The code randomly shuffles all 3L tokens of the sequence together and keeps a fixed fraction of that joint set models_mae_temporal.py (length rule at L129, joint noise at L131), so the fraction is fixed over the whole sequence, not per image. Consistent masking is a separate path selected only by the model name mae_vit_large_patch16_samemask models_mae_temporal.py models_mae_temporal.py.
• Temporal encoding offset: the paper gives year, month and hour eq. 2 but not the origin of the year. The code subtracts 2002 for fMoW, and uses zero-indexed months datasets.py datasets.py.
• Temporal decoder encoding: the code also encodes time in the decoder, with 64 dims per field (192 total) models_mae_temporal.py. The paper does not state this.
• Temporal loss normalisation: the README temporal command enables --norm_pix_loss README.md. The paper does not mention per-patch normalised targets. The code default is off main_pretrain.py.
• Test-time augmentation: the paper averages prediction scores over 9 sequences app. A.3.2. The code has the test path switched off (tta = False) engine_finetune.py. When enabled, it takes a one-hot argmax per sample and averages those, which is majority voting, and it hard-codes 1000 classes engine_finetune.py.
• Group encodings and widths: the code uses a 256-dim group code and a 768-dim positional code models_mae_group_channels.py models_mae_group_channels.py, matching the paper. The spectral decoder adds a 128-dim group code, which the paper does not state models_mae_group_channels.py.
• Spectral band groups: the default groups are channel indices (0, 1, 2, 6), (3, 4, 5, 7), (8, 9) models_mae_group_channels.py. The README drops indices 0, 9, 10 README.md. Together they reproduce the paper's three groups over bands B2 to B12 minus B1, B9 and B10, but the index-to-band mapping is only implied, not written in the code.
• Spectral masking default: consistent masking is off by default, so independent masking is the default models_mae_group_channels.py main_pretrain.py. This matches the paper's best setting.
• Spectral loss: the code averages the squared error over the channels of each group, weights each group by its mask, and divides by the number of hidden group-location pairs models_mae_group_channels.py. The paper does not describe this normalisation.
• Learning-rate rule: the code derives the absolute learning rate as blr x batch / 256 main_pretrain.py and uses AdamW with betas (0.9, 0.95) main_pretrain.py. The paper gives base rates, and the code confirms the same convention for the Sentinel and temporal pre-train commands.
• Warmup: the paper refers to "the same warmup" as MAE, without a count app. A.3.3. The code default is 40 epochs main_pretrain.py; the Sentinel README command sets 20 README.md.

Pitfalls

• Masking unit decides the result. The large gain from independent masking comes from the temporal ablation and the spectral ablation §5.3 §5.4. The appendix describes per-image masking at a fixed ratio, while the code masks the whole sequence jointly (see Paper vs code). Implement one, and say which.
• Temporal encoding needs timestamps. The temporal path needs a year, month and hour for every image; the paper gives no fallback for missing timestamps §4.1.1. The origin of the year (2002 in the code) must be kept the same between pre-training and fine-tuning datasets.py.
• Token count and memory. A time series of length 3 gives 3L tokens. Group bands give G*L tokens, so 3 groups give 3L. The authors' own note: the per-channel grouping (10 groups) gave 10L = 1440 tokens, a very large memory footprint, and unstable training that often produced NaN loss app. A.3.3. The README warns that the temporal runs are memory-heavy README.md.
• Test-time augmentation is not the same in paper and code. The paper's reported 81.49 uses 9 random sequences §5.3 app. A.3.2. The code's switch for it is off, and its enabled path votes rather than averaging scores. Reproducing the headline number needs the paper's averaging.
• Numbers in the text do not match the tables. Section 5.3 says the temporal model beats the previous state of the art by 4.48% and improves on the non-temporal result by 2.06%. The table values give different differences, for example 81.49 - 78.90 = 2.59 over MAE with test augmentation. The card reports the table values.
• The abstract and the experiments disagree. The abstract says supervised gains are "up to 7%"; the section 5 introduction says "up to 6%".
• Sentinel "6.27%" compares with a row labelled SatMAE+Group*. The gain over scratch is computed against the ViT-L from-scratch row, which the table labels SatMAE+Group* §5.4. The ResNet152 scratch row is lower.
• Not the best on every transfer task. SatMAE is below GASSL on SpaceNet mIoU (78.07 vs 78.51) and below SeCo on BigEarthNet mAP (82.13 vs 82.62) §5.5. The text describes these as comparable, not as wins.
• Band dropping is approximate. The band importance study masks bands with their mean values in a single model trained on all bands app. A.6. The authors say a true per-band ablation was too costly. Treat B1, B9 and B10 as the paper's choice, not as a verified result.
• Dataset terms. The paper's licence covers the paper only. The fMoW and Sentinel-2 data terms are footnoted app. A.1 app. A.2, and the fMoW-Sentinel dataset is hosted by the authors. Check those terms before reuse of the data.

Lineage

• Builds on: the masked autoencoder and its decoder design, which this card follows §3 (MAE, arXiv:2111.06377); the Vision Transformer backbone §1; VideoMAE for consistent masking over video §4.1.2.
• Compared against: GASSL and SeCo for remote-sensing self-supervision §5.2 §5.3 §5.5; MoCo-v3 on Sentinel data §5.4; UTAE as a time-series model §5.3.
• Superseded by: not stated in the paper.