SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery

In plain words

A model that predicts something about a place, such as temperature, house prices or which animals live there, usually needs the place described somehow. Raw latitude and longitude work for nearby places but fail for places far from the training data. Downloading satellite photos for every query point is slow.

SatCLIP trains a small network that turns any latitude and longitude into a short list of numbers. It learns this by matching each location with a satellite photo taken there. Locations whose photos look alike, such as two rainforests on different continents, end up with similar numbers, so the network learns something about the ground without needing a photo at prediction time.

Once trained, the location network is small enough to run without a GPU. Its output can be fed to any downstream model in place of raw coordinates, and the authors report gains both inside the training regions and in regions held out from training.

TL;DR

SatCLIP is a CLIP-style contrastive pre-training of a location encoder against Sentinel-2 image tiles. The location side uses spherical harmonics followed by a SIREN network; the image side is a frozen, MoCo-pretrained vision encoder whose last projection is trained. The 256-dimensional location embeddings are used as inputs to small MLPs on nine tasks. Results are author-reported. They are best on seven of nine tasks in the in-distribution setting, and the L=10 model wins most held-out-continent comparisons. Two gaps matter for reproduction: the default training config differs from the paper's setup, and its contrastive negatives come from 512-sample micro-batches rather than the paper's 8k batch.

Problem

Location is informative for geospatial tasks, but raw coordinates generalise poorly. Models that use them tend to work only where the evaluation area overlaps the training area, and labelled data is often sparse and patchy §1. Image-based alternatives need a download and a vision model for every query point, which is costly in low-resource settings §2. Earlier pretrained location encoders (GPS2Vec, CSP, GeoCLIP) were trained on data with strong geographic bias towards Europe and North America, or were built for a single kind of task §1 §3.1 app. B.

Method

The method has two encoders that map into a shared space of 256 dimensions §2:

1. Location encoder. Input is a coordinate pair (longitude, latitude) §2.1. It is passed through a spherical-harmonics basis with a Legendre parameter L, then through a SIREN network with two hidden layers of 512 units and a 256-dimensional output §2.2 app. C.1. L sets the spatial resolution: the authors release models with L=10 and L=40, and L=40 is finer §2.2 app. C.2.
2. Image encoder. ResNet18, ResNet50 or ViT16 pretrained with MoCo on Sentinel-2 imagery by Wang et al. (2022). All layers are frozen except the last linear projection to 256 dimensions §2.2 app. C.1. The ViT16 has about 22 million parameters §2.2.
3. Objective. Within a batch of N coordinate–tile pairs, each coordinate is matched against all N tiles, and each tile against all N coordinates, with a temperature τ §2.1 eq. 1 eq. 2. The paper names the second direction (L_img) but does not write its formula out in the text §2.1. The code averages the two cross-entropy terms loss.py. The location encoder has about 1 million parameters, which keeps coordinate encoding fast and GPU-free §2 §2.2.
4. Training. 90% of S2-100K for training, 10% for validation, batch size 8k, 500 epochs, one A100 for about two days; the checkpoint with the lowest validation loss is kept §2.3 app. C.1.
5. Use downstream. The location embedding (or raw coordinates, for the "Identity" baseline) is the input to an MLP trained with MSE for regression and cross-entropy for classification. Hyperparameters are tuned by random search on a validation set §3.4 app. E.3. For iNaturalist, InceptionV3 image features are concatenated to the embedding §3.2.

Pseudocode for the objective, following §2.1 and eq. 1:

# SatCLIP pretraining loss on one batch  [§2.1] [eq. 1]
# Input:  images (N, C, 256, 256), C = 12 bands [§3.1];
#         coords (N, 2) as (lon, lat) in degrees
# Output: scalar loss
# log_scale is learned in the code, starting at log(1 / 0.07)
#   [code: satclip/model.py#L330]

def satclip_loss(images, coords, log_scale):
    # Image side: frozen MoCo encoder, trained last projection  [§2.2]
    image_emb = image_encoder(images)                 # (N, 256)
    # Location side: spherical harmonics, then SIREN  [§2.2] [app. C.1]
    location_emb = location_encoder(coords)           # (N, 256)

    # Scale both sides to unit length, so the dot product is a cosine
    # [code: satclip/model.py#L366-L367]
    image_emb = image_emb / norm(image_emb)
    location_emb = location_emb / norm(location_emb)

    # Pairwise cosine similarities, divided by tau  [§2.1] [eq. 2]
    logits = exp(log_scale) * image_emb @ location_emb.T    # (N, N)

    # The true pair of row i is column i  [§2.1]
    targets = arange(N)
    # Each tile against all coordinates, and each coordinate against all tiles
    # [§2.1] [eq. 1]
    loss_per_image = cross_entropy(logits, targets)
    loss_per_coord = cross_entropy(logits.T, targets)
    # Average of the two directions, equal to the (1 / 2N) form of [eq. 1]
    return (loss_per_image + loss_per_coord) / 2

Pseudocode for the location encoder. The widths and frequencies are from the code, where the paper does not state them:

# Location encoder: spherical harmonics, then a SIREN MLP  [§2.2] [app. C.1]
# Input:  coords (N, 2) as (lon, lat) in degrees
# Output: embedding (N, 256)

def location_encoder(coords, L):
    # Shift to the angles the basis uses
    # [code: satclip/positional_encoding/spherical_harmonics.py#L31-L32]
    phi = deg2rad(coords[:, 0] + 180)
    theta = deg2rad(coords[:, 1] + 90)
    # Real spherical harmonics for every degree l < L and order m in -l..l
    # [§2.2] [code: satclip/positional_encoding/spherical_harmonics.py#L34-L40]
    # This gives L * L features, so L=10 gives 100 and L=40 gives 1600
    basis = stack([sph_harm(m, l, phi, theta)
                   for l in range(L)
                   for m in range(-l, l + 1)], axis=-1)    # (N, L * L)

    # SIREN layer 1: linear, then dropout, then sin(30 x)
    # [code: satclip/location_encoder.py#L83]
    # Dropout p = 0.5, training only
    # [code: satclip/location_encoder.py#L148-L149]
    hidden = dropout(basis @ W1 + b1)
    hidden = sin(30 * hidden)

    # SIREN layer 2: linear, then dropout, then sin(1 x)
    # [code: satclip/location_encoder.py#L83]
    hidden = dropout(hidden @ W2 + b2)
    hidden = sin(hidden)

    # Output: linear map to the embedding width, no activation
    # [code: satclip/location_encoder.py#L96]
    return hidden @ W3 + b3               # (N, 256)

Hyperparameters

Values are as stated in the paper unless marked as code. "Not stated" means the paper does not give the value.

Results

All numbers are author-reported. Table 2 uses the ResNet50 image encoder. Table 3 uses the ViT16 image encoder. Both are cited from the paper.

Table 2. In-distribution downstream performance (R² for regression, accuracy in % for classification; mean ± SD over 10 runs) tab. 2 §4.1

The authors state SatCLIP is best on seven of nine tasks. On Cali. housing and median income, GeoCLIP is equal or better §4.1. L=40 beats L=10 on the regional tasks §4.1.

Table 3. Geographic adaptation to a held-out continent (R² for regression, accuracy in % for classification; * zero-shot, † few-shot with 1% of test-continent points; ViT16 encoder) tab. 3 §4.2

The authors say the L=10 model generalises better and that L=40 is better for interpolation §4.2 §4.4. Several location encoders do worse than raw coordinates on held-out continents §4.2.

Table 4. Vision encoder and L (accuracy in %, mean ± SD; in-distribution classification tasks) tab. 4 §4.4

The authors find the vision encoder changes results by less than 1%, while L changes them more §4.4.

Qualitative results: a PCA of the embeddings rendered as RGB and cosine similarity maps are shown in fig. 6 §4.3. The Congo Basin embedding is similar to other equatorial locations, particularly the Amazon and Indonesia; the US East Coast embedding is similar to densely populated, industrialised areas such as central Europe and northern China §4.3 fig. 6.

Baseline comparison, as described by the authors §3.3 tab. 1: CSP (iNaturalist, FMoW), GPS2Vec (YFCC100M tags and images), GeoCLIP (MP-16) and MOSAIKS (Planet Basemaps 2019 Q3, image features, no location encoder). Identity is a downstream model on raw coordinates.

Paper vs code

Compared against the official repository microsoft/satclip at commit 68c73233426d465bcdf5914abf13ebc88ba60f5e, licensed MIT (Microsoft Corporation). The paper links this repository in its appendix and conclusion app. A §6, and the authors are at Microsoft and affiliated institutions. The code covers pretraining, the positional encodings, the SIREN, the loss and the dataset loader. I did not find a downstream-training module in the tree; the downstream data is linked from the README README.md, and the repository has example notebooks. The downstream MLP settings are therefore taken from the paper only. I did not download the released checkpoints, so the hyperparameters behind the released L=10 and L=40 models are not verified.

• Batch size. Paper: 8k §2.3 app. C.1. Default config: batch_size 512 with accumulate_grad_batches 16 default.yaml main.py. The loss is computed per micro-batch of 512 main.py, so the negatives per step are 512, not 8192.
• Image encoder. Paper: MoCo-pretrained, frozen except the last projection §2.2 app. C.1. Default config: vision_layers: 4, vision_width: 128, vision_patch_size: 32, which builds a ViT from scratch with no freezing default.yaml model.py. The frozen MoCo branches exist model.py, but the default config does not select them.
• ViT16 size. Paper: about 22M parameters §2.2. Code: MoCo ViT-Small/16 model.py. README: "ViT-B/16" README.md. ViT-B is the larger model, so the README and the code disagree on which backbone the released ViT16 checkpoints use.
• SIREN width. Paper: 512 hidden units app. C.1. Default config and main.py default: 256 default.yaml main.py.
• Legendre parameter. Paper releases L=10 and L=40 §2.2 tab. 3. Default config: 32 default.yaml. README example: 10 README.md.
• Optimiser. Paper: Adam, weight decay 0.01 app. C.1. Code: AdamW, with weight decay excluded from 1-D, bias, norm and logit-scale parameters main.py main.py.
• Temperature. Paper: τ is a hyperparameter with no number given §2.1. Code: a learned log-scale starting at log(1/0.07), excluded from weight decay model.py main.py.
• Loss. Paper: (1 / 2N) times the sum over both directions eq. 1. Code: the mean of the two cross-entropy terms loss.py. The two are the same quantity.
• Channels. Paper: 12 bands §3.1. Code: 13 channels, with a zero-filled band inserted at index 10 transforms.py default.yaml. The paper's band list in app. A names B06 twice and omits B05.
• Normalisation. Paper: not stated. Code: divide by 10000 and nothing else in the default path transforms.py. The per-channel mean and standard deviation in transforms.py are not used by default s2geo_dataset.py.
• Coordinate jitter. Paper: shifts "up to about 1 km" app. C.1. Code: adds a value in [0, 0.01) degrees to both longitude and latitude transforms.py. The shift is never negative and is in degrees, not metres.
• Dropout in SIREN. Not in the paper. Code applies dropout with p = 0.5 to the SIREN hidden layers during training location_encoder.py location_encoder.py.
• Checkpoint selection and epochs. Paper: lowest validation loss, 500 epochs app. C.1. Code: ModelCheckpoint monitors val_loss, and max_epochs is 500 default.yaml default.yaml. Consistent.
• Learning rate, weight decay, split, embedding width. Paper and code agree: 1e-4, 0.01, 90/10, 256 app. C.1 default.yaml s2geo_dataset.py.
• L_img. Paper: named, with the formula left out of the text §2.1. Code: symmetric cross-entropy loss.py.

Pitfalls

• Negatives per micro-batch. The paper's 8k batch gives 8k negatives. The default config produces 512 negatives per step, with gradients accumulated over 16 micro-batches main.py main.py. To reproduce the paper, one step must compute the logits over all 8k embeddings at once. Accumulation does not do this.
• Default config is not the paper's model. A from-scratch, unfrozen ViT is not the MoCo-frozen encoder the paper trains default.yaml. Use the moco_* branches for the paper's setup model.py.
• Spherical harmonics size. Degrees run from 0 to L−1, so L=10 gives 100 features and L=40 gives 1600 spherical_harmonics.py spherical_harmonics.py. The paper's table caption uses the same convention ("maximum number of Legendre polynomial degrees") app. E.4.
• Harmonic angle convention. Input is (lon, lat) in degrees. The code adds 180 to longitude and 90 to latitude before converting to radians spherical_harmonics.py. Pass the columns in the other order and the embedding is silently wrong.
• SIREN initialisation and frequency. The first layer uses frequency 30, the hidden layer frequency 1, and the uniform init uses c = 6 location_encoder.py location_encoder.py. The paper does not give these.
• Float64 location path. The location encoder runs in double precision and the embeddings are cast to float afterwards model.py model.py model.py.
• Jitter is one-sided. Training coordinates are only ever moved east and north of the tile centre, never west or south transforms.py. Copy the jitter only with that bias in mind.
• Band order. The code relies on channel index to mean band, with B10 zero-filled at index 10 transforms.py. The paper's band list has a typo app. A. Verify the band order against your data before inference.
• Which results use which encoder. Table 2 uses ResNet50; Table 3 uses ViT16 tab. 2 tab. 3. The headline numbers are not from one model. Compare within a table.
• Results are author-reported, and two tasks are regional. Cali. housing covers California only and median income covers the continental US only §4.1 app. E.2. The held-out tests hold out Africa or Asia, one at a time app. E.3. The Identity baseline on raw coordinates is strong on some tasks tab. 2.
• Released checkpoints are not verified here. The README table lists six checkpoints README.md. I did not download or inspect them. Their hyperparameters may differ from both the paper and the default config.
• Licence of S2-100K. The paper does not state a licence for the dataset app. A. It is distributed on Hugging Face README.md. Check that licence before commercial reuse of the tiles. This is separate from the paper's CC BY 4.0.
• Version. This card follows arXiv v3, the version fetched. The repository's README cites a conference version README.md. It may differ from v3.

Lineage

• Builds on: CLIP, the image–text contrastive objective this work adapts to image–coordinate pairs §1 §2.1 (Radford et al., 2021); the spherical-harmonics location encoder of Rußwurm et al. (2023) §2.2; SIREN, the sinusoidal network of Sitzmann et al. (2020) §2.2; MoCo, the pretraining method of He et al. (2020), and its Sentinel-2 weights from Wang et al. (2022) §2.2.
• Compared against: CSP (Mai et al., 2023), GPS2Vec (Yin et al., 2019), GeoCLIP (Cepeda et al., 2023), MOSAIKS (Rolf et al., 2021), and an Identity baseline on raw coordinates §3.3 tab. 1.
• Superseded by: not stated in the paper.