Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications

In plain words

Satellite pictures of the same spot taken months apart show how land changes with the seasons and over the years. Prithvi-EO-2.0 is a large neural network that learns from millions of such short image sequences. It learns by hiding parts of the pictures and trying to fill them back in, so it picks up general knowledge about land and how it changes before anyone asks it a specific question.

Unlike many earlier models built for single images, this one treats the sequence as a whole. It can also be given the date and place a sample was taken, though it still works when that information is missing.

After this general training, the model is adapted to specific jobs such as flood mapping, burn-scar mapping, landslide detection, crop mapping, biomass estimation and carbon-flux estimation. The authors report that it does well on these tasks, often with fewer labelled examples than the alternatives.

TL;DR

• A ViT masked autoencoder (MAE) in two sizes, 300M and 600M, pretrained on 4.2M samples of four Harmonized Landsat-Sentinel-2 images (30 m) §3.1 §3.2.
• Adds 3D patch embeddings and learned-weight bias terms for date and location so time and place are used when present §3.2.
• Author-reported: 8% better than Prithvi-EO-1.0 across GEO-Bench tasks, and SME-led downstream results §4 [abstract]. Weights are on Hugging Face. The code is MIT-licensed LICENSE. The weights' licence is not named in the paper.

Problem

Most geospatial foundation models of the time did not model multi-temporal inputs. Those that did were limited to point data, pixel time series or small patches §1. The authors also note that validation across task types was limited, and that adapting these models often needed AI expertise and tooling §1 §2.

Method

The method has three parts: how the pretraining samples are built §3.1, the pretraining model §3.2, and how the model is evaluated and fine-tuned §3.3 and §4.

1. Dataset. Pretraining uses NASA's HLS product, which harmonises Landsat 8/9 and Sentinel-2A/2B at 30 m. Only six bands are used, common to both sensors: Blue, Green, Red, NIR, SWIR1 and SWIR2 §3.1.
2. Tile selection. Tiles of 3,660 × 3,660 px are chosen to cover land-cover classes, ecoregions, urban areas and high-entropy landscapes. The procedure yields 3,028 training tiles and 163 validation tiles §3.1 fig. 1.
3. Sequences and patches. For each tile, sequences of four HLS timestamps are sampled with gaps of one to six months between consecutive timestamps, spanning 2014 to 2023 §3.1. Each sequence is cut into non-overlapping 256 × 256 patches. Patches with more than 1% missing pixels in any band or more than 20% cloud (Fmask) are discarded §3.1. Missing pixels are filled with nearest-neighbour interpolation §3.1. The final set has 4.2M training and 46k validation samples §3.1 fig. 2.
4. Patch embedding. A 3D convolution splits the sequence into cubes of size (t, h, w) with t = 1, because satellite revisits are irregular §3.2. Each image therefore becomes its own temporal slice.
5. Positional encoding. A 1D sin/cos code is built for time, height and width, then the three are combined into one 3D encoding §3.2 eq. 1.
6. Metadata. Year and day-of-year of each image, and the centre latitude and longitude of the sample, are each encoded with 1D sin/cos and concatenated. The time code and the location code are each added to the tokens with a learned weight, so each has its own weight §3.2. The metadata is added as a bias, not as extra input channels §3.2.
7. Metadata dropout. During pretraining, the location and/or time code is dropped at random, so the model learns to work without it §3.2.
8. Masked autoencoding. A subset of patches is masked. The encoder sees only the visible tokens. The decoder receives the encoded visible tokens plus learned mask tokens, and reconstructs the masked tokens. The loss is MSE in pixel space on the masked tokens §3.2. The method follows MAE §3.2.
9. Fine-tuning. Downstream heads are attached to the encoder. Decoders vary by task: UPerNet for most dense tasks, a small convolutional decoder for US crops, and a deconvolution decoder for landslides. Recipes are in the table below §3.3 §4.

# Build pretraining samples from one selected HLS tile  [§3.1]
# Input:  tile, a time series of 6-band HLS images (3660 x 3660 px)
# Output: list of samples, each 4 timestamps x 256 x 256 px
def sample_tile(tile, per_tile_limit):
    # Scan candidate sequences of 4 timestamps, gaps of 1 to 6 months  [§3.1]
    kept = []
    for sequence in candidate_sequences(tile, length=4):
        # Stop after 1,500 samples per tile, 250 for validation  [§3.1]
        if len(kept) >= per_tile_limit:
            break
        # Non-overlapping 256 x 256 patches of the same 4 timestamps  [§3.1]
        for patch in non_overlapping_patches(sequence, size=256):
            # Drop patches with > 1 % missing or > 20 % cloud  [§3.1]
            if missing_fraction(patch) > 0.01:
                continue
            if cloud_fraction(patch) > 0.20:
                continue
            # Fill missing pixels with nearest-neighbour values  [§3.1]
            kept.append(fill_missing_nearest(patch))
    return kept

# One pretraining forward pass for a batch of samples  [§3.2]
# Input:  images (B, T=4, C=6, 224, 224) after random crop  [§3.2]
#         dates (B, T, 2) year and day of year; centre (B, 2) lat, lon
# Output: reconstruction loss on masked tokens
def pretrain_step(images, dates, centre, mask_ratio):
    # 3D patch embedding with t = 1  [§3.2]
    tokens = conv3d_patch_embed(images)           # (B, N, D)
    # 3D sin/cos positional encoding  [§3.2] [eq. 1]
    tokens = tokens + positional_encoding_3d(tokens)
    # Learned-weight metadata bias; time per image, location per sample  [§3.2]
    tokens = tokens + metadata_bias(dates, centre, drop_probability=0.1)
    # Mask a random subset of tokens; the ratio is not stated  [§3.2]
    visible, masked_ids = random_mask(tokens, mask_ratio)
    # Encoder processes visible tokens only  [§3.2]
    latent = vit_encoder(visible)
    # Decoder gets encoded tokens plus learned mask tokens  [§3.2]
    prediction = vit_decoder(latent, masked_ids)
    # MSE in pixel space on masked tokens  [§3.2]
    return mse(prediction[masked_ids], pixels[masked_ids])

Key equations:

\mathrm{PE}_{pos,2i} = \sin\left(\frac{pos}{10000^{2i/D}}\right),\quad \mathrm{PE}_{pos,2i+1} = \cos\left(\frac{pos}{10000^{2i/D}}\right) eq. 1

\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} eq. 2

\text{mIoU} = \frac{1}{C}\sum_{c=1}^{C}\frac{\lvert prediction_c \cap truth_c \rvert}{\lvert prediction_c \cup truth_c \rvert} eq. 3

Precision, recall and F1 are defined in eq. 4, eq. 5 and eq. 6. The F1 form used is 2TP / (2TP + FP + FN) eq. 6.

Hyperparameters

Pretraining §3.2 unless noted

Benchmarking, GEO-Bench §3.3.1

Downstream fine-tuning

Results

Author-reported throughout. Setups are given with each table. The GEO-Bench per-run results are in the paper's "GEO-Bench detailed results" section, which sits after the bibliography tab. A1–tab. A4.

GEO-Bench, mean score per dataset

Mean over 10 seeds. Values are the paper's per-dataset means tab. A1–tab. A4. The 100M variant of Prithvi-EO-2.0 is in those tables but not released (see Pitfalls).

Per the text, the top classification models are DOFA and Prithvi-EO-2.0-600M-TL and -600M. The top segmentation and overall models are Prithvi-EO-2.0-600M-TL and -600M §4.1 fig. 6 fig. 7. Other models are in tab. A1–tab. A4. The aggregate scores in fig. 6 are shown only as bars; the paper gives no numbers for them.

Flood mapping, Sen1Floods11 §4.2.1 tab. 4

Wildfire scar mapping §4.2.2 tab. 5

Burn intensity §4.2.3 tab. 6

Test set, five classes (0 = no burn to 4 = high severity).

Landslide4Sense §4.2.4 tab. 7 tab. 8 fig. 8

Full training set (100%, 3,799 images):

Small subset (about 1%, 50 images):

US multi-temporal crop segmentation §4.3.1 tab. 9

Europe, Sen4Map land cover and crops §4.3.2 tab. 10 tab. 11

Weighted F1 (std) by fraction of training data.

PASTIS crop segmentation §4.3.3 tab. 12

mIoU, models trained on 10% and 100% of the training data.

Above-ground biomass, BioMassters §4.4.1 tab. 13 tab. 14 fig. 9 fig. 10

RMSE on the test set, lower is better. Winning-challenge baseline: RMSE 27.49.

Subset training (12 Sentinel-2 timestamps, LoRA): 50% → 34.31; 20% → 34.89; 10% → 36.77; 5% → 38.06 tab. 14.

The baseline beats every Prithvi configuration on this task. The authors say optical-only Prithvi approaches multimodal models, cite prior work that optical plus SAR beats optical alone for biomass, and report that adding SAR bands to Prithvi did not help §4.4.1.

GPP at flux towers §4.4.2 tab. 15 fig. 11

Leave-one-year-out R², test year shown.

Paper vs code

The official repository is NASA-IMPACT/Prithvi-EO-2.0, linked in the paper §1 README.md. It is MIT-licensed LICENSE. It contains fine-tuning configs and notebooks built on TerraTorch README.md. The pretraining code, the GEO-Bench harness and the PASTIS code are not in this repository.

• Metadata encoding. The paper says each of latitude, longitude, year and day-of-year is encoded with 1D sin/cos, and that the time and location codes have learned weights §3.2. The README says the metadata is encoded with 2D sin/cos, and that the encoder and the decoder each receive it with separate learned weights README.md. The two descriptions differ.
• Burn intensity. The paper uses LoRA, LR 5e-4 for the encoder and 5e-5 for the decoder, cosine annealing §4.2.3. The config has no LoRA setting, a single LR of 1e-5, and a StepLR scheduler burnintensity.yaml.
• Landslide. The paper uses Adam, LR 1e-5, 100 epochs, batch 8, with weighted cross-entropy or Lovász loss §4.2.4. The config uses AdamW with LR 5e-5 and weight decay 0.1, focal loss, 50 epochs and patience 10 landslide.yaml landslide.yaml.
• BioMassters. The paper's best setting uses 12 Sentinel-2 timestamps §4.4.1. The config defaults to use_four_frames: True and backbone_num_frames: 4 biomassters.yaml biomassters.yaml. The config also uses LR 1e-4 and StepLR, while the paper gives AdamW at LR 6e-5 §4.4.1.
• US crop segmentation. The paper says 80 epochs §4.3.1. The config allows 125 multicrop.yaml. Its data path points to the Prithvi-1.0 dataset multicrop.yaml.
• Neck layer indices. The paper does not say which encoder blocks feed the decoder. The landslide, wildfire and BioMassters configs use indices 7, 15, 23, 31 for the 600M backbone landslide.yaml firescars.yaml. The 300M indices (5, 11, 17, 23) are commented out there. The Sen1Floods11 config is active with the Prithvi-1.0 indices 2, 5, 8, 11 sen1floods11.yaml, so its decoder taps are not the 2.0 ones.
• GPP backbone. The GPP trainer loads Prithvi_EO_V2_300M_TL.pt main_flux_finetune_baselines_trainer.py. The paper's best GPP result is 600M-TL tab. 15. The paper says it tried all variants. The trainer also reads fluxconfig_trainer_custom.yaml main_flux_finetune_baselines_trainer.py, which is not in the repository. The repository has fluxconfig_trainer.yaml.
• GPP model. The code matches the paper's head: a frozen encoder main_flux_finetune_baselines_trainer.py, a 1024-to-256 linear layer followed by a flattened linear layer to 64 flux_regress.py, a 1×1 convolutional MERRA branch flux_regress.py, and a final linear layer on the concatenation flux_regress.py. The MERRA convolutions use kernel size 1; the paper does not state a kernel size.
• Sen4Map. The code uses 15 × 15 crops resized to 224, 12 frames, and a 300M backbone with patch 16 sen4map_crops.yaml sen4map_crops.yaml sen4map_crops.yaml. This matches §4.3.2. The paper gives no optimiser or epoch count for this task; the config has 20 epochs and AdamW at LR 6e-5 sen4map_crops.yaml.
• Consistent items. Flood and wildfire epochs, patience and loss match the configs §4.2.1 §4.2.2 sen1floods11.yaml firescars.yaml.

Pitfalls

• Mask ratio is not stated in the paper or the repository. An implementer must choose it. The MAE default is usually assumed, but this is not confirmed here.
• Optimizer for pretraining is not named. Weight decay 0.05, cosine schedule and LR 5e-4 are given §3.2, so the optimiser family is a guess.
• Metadata dropout is underspecified. The paper gives probability 0.1 for TL models §3.2. It does not say whether the drop is per sample or per term, whether the two terms are dropped independently, or what replaces a dropped term.
• Metadata encoding has two versions. Use the paper's version §3.2 unless you are matching the released weights. Check the released checkpoint's architecture before reusing this code README.md.
• Patch size for the 300M model is not stated. The paper gives 14 × 14 for the 600M model §4.2.1. The repository's Sen4Map configs use 16 sen4map_crops.yaml. Confirm against the checkpoint.
• Sampling is not fully specified. The paper says sequences are sampled "iteratively" until all candidates are processed or 1,500 samples per tile are reached §3.1. It does not say whether the 1,500 counts candidates or kept samples, or how the four timestamps are picked from the candidates.
• Headline claim unclear. The 8% gain over Prithvi-EO-1.0 [abstract] §5 does not name a metric. The paper's own comparison of 100M-to-100M models gives 3% on the overall GEO-Bench score §4.1. Use the per-dataset tables above.
• 100M variant is not released. Prithvi-EO-2.0-100M appears in the GEO-Bench appendix tables and text §4.1 tab. A1. It is absent from the model table tab. 2 and from the released models README.md.
• Batch size for GEO-Bench is not stated. The paper says it was "fixed to a reasonable value" per dataset §3.3.1. Results may not reproduce without it.
• Seeds and trials differ by task. GEO-Bench and the flood and wildfire tasks use 10 HPO trials and 10 seeds §3.3.1 §4.2.1 §4.2.2. Most other tasks give no seed count. The GPP results are single leave-one-year-out splits, so they have no spread §4.4.2 tab. 15.
• Compute is large for pretraining, small for fine-tuning. Pretraining uses ~21,000 GPU-h (300M) and ~58,000 GPU-h (600M) on 80 and 240 A100s §3.2. The paper says all fine-tuning runs fit on one GPU §3.3.
• Figures carry numbers that the text does not. The GEO-Bench aggregates in fig. 6 and the per-run distributions in fig. 7 are not tabulated in the text. Use tab. A1–tab. A4 for numbers.
• Licensing of the model and data is not named. The conclusion calls the release "permissive" but does not name a licence §5. The downstream datasets have their own licences, which this card does not check.

Lineage

• Builds on: the masked autoencoder approach §3.2; ViT backbones §3.2; 3D patch embedding and positional encodings from video and multispectral MAE work §3.2. Its predecessor is Prithvi-EO-1.0, a US-only model trained on HLS data with 250k samples and three timestamps §1 tab. 2.
• Compared against: SatMAE, Scale-MAE, SSL4EO-S12 (MoCo, DINO), DeCUR, DOFA, Presto, U-BARN §2; Satlas tab. 2; Prithvi-EO-1.0 §4.1.
• Superseded by: not known from this paper's content.