TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis

In plain words

Satellites watch the same patch of land again and again, but clouds hide some of those views, and the clear ones fall on irregular days. Many earth-observation models first clean this up by compositing the views into one cloud-free picture. That step also removes the timing of plant growth, which many later tasks need.

TESSERA keeps the individual views. For each small patch of ground it looks at the visible observations from two satellite types, radar and optical, and turns them into one fixed list of 128 numbers for that year. The model is trained so that two random subsets of the same visible views give the same list. The list therefore does not depend on which clear days happened to be available.

The authors publish these 128-number maps for the whole globe, along with the code and a Python library to download them. Someone who wants to classify crops or estimate biomass can train a small model on top of the map instead of processing satellite data themselves.

TL;DR

TESSERA is a pixel-wise foundation model: one self-supervised encoder for Sentinel-2 and Sentinel-1 time series, trained with Barlow Twins, random sparse temporal sampling, global shuffling and mixup regularisation. Its frozen output is an annual 128-d embedding per 10 m pixel, released as int8 together with code and a Python library §1. On six benchmarks, small heads on these embeddings are best in most reported settings, most clearly with few labels; AlphaEarth is higher on PASTIS-R with all labels and at 30% tab. 1. Pretraining code is not in the public repository, so training details cannot be checked against code.

Problem

Optical satellite series are irregular because of orbit timing and cloud cover. Compositing removes clouds but also removes the phenology signal that many downstream tasks need §1. Remote-sensing foundation models are often trained on cloud-filtered or composited inputs, so their embeddings can be biased toward clear-sky conditions §1. The paper wants embeddings that stay stable when the set of valid observations changes, and that can be used with only a small head and little labelled data §1 §4.1.

Method

The method has three parts: a per-pixel input representation (d-pixel), a dual encoder with a fusion layer, and a self-supervised objective. Appendix citations ([app. ...]) name the subsection heading, because the source does not expose appendix letters.

Input (d-pixel). For one ground location, each date gives a value per band for Sentinel-2 (10 bands) and per polarisation for Sentinel-1 (VV, VH), with a validity mask. A date is valid when the pixel is not cloudy or missing §3.1 app. Model Details. Each date also carries a day-of-year (DOY) value.

Encoder. Each sensor has its own encoder. Valid dates are embedded per date and given a DOY encoding §3.2. A sequence of L = 40 valid dates is used; when a pixel has fewer valid dates, dates are drawn with replacement §3.2 §3.3. A Transformer encoder runs over the sequence, and a temporal pooling step produces one vector per sensor §3.2. The two sensor vectors are concatenated and passed through an MLP to give the 128-d fused embedding z §3.2.

Quantisation. z is trained with quantisation-aware training to 8-bit integers, which the paper reports as about a 4x storage reduction with negligible loss §3.2 §5.

Pretraining objective. For each d-pixel, two views (Y_A, Y_B) are built by sampling 40 valid dates independently for each sensor §3.3 [app. Self-Supervised Training]. The views pass through the dual encoder and a large projector (16,384 width). Barlow Twins loss L_BT is computed on the batch-normalised outputs Z_A and Z_B §3.3. A mixup term L_MIX mixes view A with a shuffled view B and penalises departures from linear behaviour of the cross-correlation matrices §3.3. The total loss is L_BT + λ_mix L_MIX §3.3. The projector is discarded after training §3.2.

Global shuffling. Before batching, d-pixels from all tiles are pooled and shuffled globally rather than tile by tile §3.3 app. Pre-training Details alg. 2.

Inference. The frozen dual encoder and fusion layer (without the projector) are run on 40 sampled dates per sensor for every land pixel, giving a 128-d embedding per pixel per year §3.4 app. Global Embedding Map Generation.

Pretraining pseudocode, following the paper's equations:

# Embed one 10 m pixel for one year  [§3.2] [§3.4]
# Each sensor input: per-date bands, DOY and validity mask
# Output: 128-d float embedding; int8 quantisation happens afterwards  [§3.2]
def embed_pixel(s2, s1, encoder_s2, encoder_s1, fuse):
    # Draw L = 40 valid dates per sensor, with replacement if fewer  [§3.3]
    seq_s2 = sample_valid_dates(s2, length=40)
    seq_s1 = sample_valid_dates(s1, length=40)
    # Per-date linear embedding of the bands plus a DOY encoding  [§3.2]
    tokens_s2 = encoder_s2.embed(seq_s2.bands) + encoder_s2.doy(seq_s2.doys)
    tokens_s1 = encoder_s1.embed(seq_s1.bands) + encoder_s1.doy(seq_s1.doys)
    # Transformer over the dates, then temporal pooling to one vector  [§3.2]
    pooled_s2 = pool_over_time(transformer(tokens_s2))
    pooled_s1 = pool_over_time(transformer(tokens_s1))
    # Concatenate the two sensors and fuse with a 2-layer MLP  [§3.2]
    return fuse(concat(pooled_s2, pooled_s1))  # (128,)

# Pretraining loss: Barlow Twins plus mixup consistency  [§3.3] [alg. 1]
# view_a, view_b: two independent 40-date samples for the same d-pixel batch
def pretraining_loss(view_a, view_b, model, lambda_bt, lambda_mix):
    # Dual-encoder output passed through the projector  [§3.2]
    z_a = model.projector(model.encode(view_a))  # (B, 16384)
    z_b = model.projector(model.encode(view_b))  # (B, 16384)
    # Batch-normalise each dimension, then cross-correlate the views  [§3.3]
    c = cross_correlation(batch_norm(z_a), batch_norm(z_b))  # (D, D)
    # Push the diagonal to 1 and the off-diagonal to 0  [§3.3]
    invariance = sum((1 - diagonal(c)) ** 2)
    redundancy = sum(off_diagonal(c) ** 2)
    loss_bt = invariance + lambda_bt * redundancy
    # Shuffle view B across the batch, then mix it with view A  [§3.3]
    perm = random_permutation(batch_size)
    view_s = view_b[perm]
    # Mixing weight from U(0, 1); alg. 1 uses one per batch  [§3.3] [alg. 1]
    alpha = uniform(0, 1)
    view_m = alpha * view_a + (1 - alpha) * view_s
    z_m = model.projector(model.encode(view_m))
    z_s = model.projector(model.encode(view_s))
    # Mixed cross-correlations should match the mixed targets  [§3.3]
    corr_ma = cross_correlation(z_m, z_a)
    corr_ma_target = (
        alpha * cross_correlation(z_a, z_a)
        + (1 - alpha) * cross_correlation(z_s, z_a)
    )
    corr_ms = cross_correlation(z_m, z_s)
    corr_ms_target = (
        alpha * cross_correlation(z_a, z_s)
        + (1 - alpha) * cross_correlation(z_s, z_s)
    )
    loss_mix = 0.5 * (
        frobenius(corr_ma - corr_ma_target) ** 2
        + frobenius(corr_ms - corr_ms_target) ** 2
    )
    # Total loss; the mixup weight is 1.0 in the paper  [§3.3]
    return loss_bt + lambda_mix * loss_mix

Key equations are written out in §3.3 of the paper and are not numbered in the source, so they are cited by section.

Hyperparameters

Results

Author-reported. Setup: frozen TESSERA embeddings with a small head (2-layer MLP for pixel-wise tasks, U-Net for patch-level tasks) §4.1 [app. Downstream Task Application Methodology]. Baselines are eleven remote-sensing foundation models plus UNet and ViT references §4.1 tab. 2. Patch-based baselines are fine-tuned with UPerNet heads; Presto and AlphaEarth are used as frozen embeddings with their own heads [app. Downstream Task Application Methodology]. Where a baseline has both, the table shows "original/finetuned" tab. 1. AlphaEarth is taken from its public V1 dataset §4.1. Datasets: TreeSatAI-TS, PASTIS-R, Austrian Crop (classification and segmentation), BioMassters and Borneo canopy height §4.1.

Ablation on Austrian Crop, validation F1 and RankMe tab. 3:

The paper also reports a few-label Austrian Crop setting with 1 to 20 samples per class, a robustness plot against valid-observation count, and scaling plots for the projector §4.2 §5 fig. 5 fig. 6.

Paper vs code

Code: ucam-eo/tessera at commit e368539f7c4711e5f16fdd09801573830e34baf9, MIT licence LICENSE. The README says embeddings and model weights are released under CC0 README.md; that was read from the README, not from a separate licence file. The repository contains inference, preprocessing and visualisation code and config files. It does not contain the pretraining script, the Barlow Twins or mixup loss, the global-shuffle pipeline, or the downstream benchmark scripts. Training-side claims below are therefore only checked against configs, not code.

• Learning rate. Paper: 0.002 §3.3. Repo config: learning_rate 0.01 ssl_config.py. The config's role for the released weights is not stated.
• Batch size. Paper: global 32,768 §3.3. Repo config: batch_size 1024 ssl_config.py. Dataset size: paper about 800 million d-pixels §3.3; repo total_samples 8,500,000 ssl_config.py.
• Mixup. Paper: mixup term on by default, λ_mix = 1.0 §3.3. Repo config: apply_mixup False ssl_config.py.
• Warmup. Paper: linear warmup over 10% of steps app. Pre-training Details. Repo config: warmup_ratio 0.2 ssl_config.py.
• Projector width. Paper: 16,384 §3.2. Repo training config: 2,048 ssl_config.py. The v1.0 inference config sets 8192 x 2 = 16,384 multi_tile_infer_config.py. The projector is not used at inference, so this only matters for matching checkpoints.
• Encoder depth and heads. Paper: 4 Transformer layers, 4 heads [app. Scaling with data and network size]. Repo v1.0 builder: 8 layers, 8 heads, feed-forward 4096 builder.py. The repo's QAT (v1.1) builder uses 4 layers builder.py.
• Encoder width. Paper: 128-d per sensor before fusion §3.2. Repo: each sensor encoder outputs latent_dim x 4 = 512 modules.py and the fusion concatenates 2 x 512 = 1,024 before one linear layer to 128 ssl_model.py.
• Band embedding. Paper: a linear projection φ §3.2. Repo: a two-layer MLP with ReLU modules.py.
• Fusion layer. Paper: a 2-layer MLP §3.2. Repo: a single linear layer ssl_model.py.
• Temporal pooling. Paper main text: the GRU output is the sensor vector §3.2; appendix: attention pooling app. Model Details. Repo: a GRU provides attention weights, but the weighted sum is over the Transformer output, not the GRU output modules.py.
• DOY encoding. Paper main text: learnable DOY positional encoding ψ §3.2; appendix: sine and cosine of the normalised DOY app. Pre-training Details. Repo: fixed sinusoids of the raw integer DOY modules.py, used by the encoder modules.py. A learnable encoding class exists but is not used by the encoder modules.py.
• Projector layers. Paper: six linear layers, batch norm and ReLU [app. Projector Network]. Repo: six linear layers with batch norm and ReLU after the first five modules.py. Structure matches; width differs.
• Inference sampling. Paper: a single draw, or repeated draws averaged, is not stated §3.4; the appendix says several draws averaged can stabilise results app. Downstream Task Application Methodology. Repo: repeat_times 1 in the v1.0 inference config multi_tile_infer_config.py; the dataset docstring says 10 repeats ssl_dataset.py. Averaging is implemented qat_engine.py.
• Pixel filter. Paper: not stated. Repo: min_valid_timesteps 0 in the v1.0 inference config multi_tile_infer_config.py, 20 in the training config ssl_config.py, 10 as the dataset default ssl_dataset.py.
• Empty pixels. Paper: not stated. Repo: if a pixel has no valid dates, all dates are used data_processing.py.
• Sampling with replacement. Matches the paper: replacement when fewer than 40 valid dates data_processing.py.
• Band standardisation. Paper: "global statistics", no values [app. Pre-training Details]. Repo: per-band constants for S2 (10 bands) and S1 (2 bands), applied with epsilon 1e-9 ssl_dataset.py data_processing.py.
• Sentinel-1 ascending and descending passes. Paper: not discussed. Repo (QAT path): ascending and descending dates are concatenated into one sequence data_processing.py.
• Quantisation. Paper: int8 embeddings, QAT, in TESSERA v1.0 §3.2 §5. Repo: the int8 path is in tessera_infer_QAT, labelled "Tessera v1.1 (QAT)" v1_1_infer_config.py. Each vector is quantised with one symmetric scale per pixel, scale = max|x| / 127 qat_engine.py quantization.py. The scale must be kept to dequantise visualize_embedding.py.
• v1.1 width. The repo's v1.1 config uses latent_dim 192, so d_model = 768 v1_1_infer_config.py. The paper's encoder uses 128 at the representation level.
• Second model. The repo also ships tessera_infer_v2, described as a distilled 128-d student from a TESSERA teacher with Matryoshka ordering model.py. The paper does not describe it.
• Global shuffle. The paper cites a custom Rust binary for global shuffling app. Pre-training Details. No shuffle code exists in the repo's Rust crates (the two rust_for_s*_stacking crates).

Pitfalls

• No training code. Barlow Twins, the mixup loss, the optimiser schedule and the global-shuffle pipeline cannot be checked against code. Several configs conflict with the paper (see Paper vs code). Implement the paper's values and report the conflict rather than picking one silently.
• Algorithm 1 and the equations disagree. The mixup term in the paper's equations is weighted only by λ_mix §3.3. Algorithm 1 also multiplies it by λ_BT alg. 1. Choose one and say which.
• Mixing weight granularity. §3.3 says α is drawn from U(0, 1) but not whether it is per sample or per batch; alg. 1 uses one value per batch alg. 1.
• Positional encoding is ambiguous. The main text says learnable, the appendix says sine/cosine of normalised DOY §3.2 app. Pre-training Details. The repo uses fixed sinusoids of the raw integer DOY. Pick one and test it.
• Pooling is ambiguous. GRU output (main text) versus attention pooling (appendix) §3.2 app. Model Details. The repo applies GRU-derived attention weights to the Transformer output.
• Conflicting parameter counts. The TESSERA encoder size is 45.7M in the text §4.2 footnote and in the appendix table app. Model Details, but 30.2M in the model table tab. 2. The footnote gives AlphaEarth as 480M, while the same table gives 30.1M. Do not rely on these counts for memory planning.
• Cloud-robustness threshold. §5 says performance drops when valid Sentinel-2 observations fall to 10 or fewer, while the figure caption says about 20 §5 fig. 5. Check against the plot before relying on it.
• Baseline F1 differs between tables. The ablation baseline is 77.3 tab. 3, while the seasonal ablation's full model is 78.94 app. Impact of Seasonal Windows. The paper does not reconcile them.
• F1 averaging is unclear. The Austrian Crop figure says weighted F1 fig. 3; the discussion says macro-F1 §5. The table says only F1 tab. 1.
• BioMassters numbers. The table gives 27.43 t/ha tab. 1; §4.4 gives 26.61 after clipping predictions above 500 t/ha, and quotes 25.90 for the competition winner. These are different setups; do not mix them.
• Presto on PASTIS-R has no reported value tab. 1.
• Unequal heads. TESSERA uses a small head on frozen embeddings, while several baselines fine-tune encoders with UPerNet [app. Downstream Task Application Methodology]. Compare results with that in mind.
• Pretraining cost. One epoch over about 800 million d-pixels with batch 32,768 is roughly 24,400 optimiser steps. This is derived from the stated numbers, not stated in the paper.
• Storage. int8 embeddings need a per-pixel float scale to dequantise visualize_embedding.py. Storing only the int8 values loses the magnitude information.
• Licence of the released data. The paper is CC BY 4.0 (card licence field). The released embeddings and weights are described as CC0 in the README README.md, which was not checked against a separate licence file. Check the data licence at the source before redistributing.

Lineage

• Builds on: Barlow Twins for self-supervised learning §3.3; Lisaius et al. (2024), whose Barlow Twins work on optical Earth-observation streams this paper extends (Lisaius is a co-author) §1. The mixup term is inspired by Bandara et al. (2023) §3.3.
• Compared against: CROMA, DOFA, Prithvi, RemoteCLIP, SatlasNet, Scale-MAE, SpectralGPT, Galileo, SkySense, Presto and AlphaEarth §4.1.
• Superseded by: not stated in this paper. The repo's v1.1 (QAT) and v2 (student) are later releases of the same project v1_1_infer_config.py model.py.