SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing

In plain words

Models like CLIP learn to match pictures with text, and they become useful for new tasks when they have seen many pictures with descriptions. Satellite and aerial photos rarely come with such descriptions, and nobody can crawl them off the web at scale the way ordinary photos are crawled. This paper gets around the problem by using map data. Every picture has a place on Earth, and map databases list what sits at that place: roads, rivers, buildings, farmland and so on. The authors match each picture to the map objects it covers and turn the map labels into short sentences such as "natural water" or "building under construction".

The result is a large set of picture and sentence pairs with a much wider vocabulary than earlier remote-sensing datasets. A cleanup step keeps only the pairs where a standard CLIP model rates picture and sentence as similar. The authors then continue training CLIP on these pairs. The resulting model answers questions about scenes and roof or road types more accurately than the original CLIP, on test sets it never saw during this training.

The dataset is only as good as the map labels, which are uneven across the world. The sentences are built by simple rules, not written by people.

TL;DR

• SkyScript links open satellite and aerial images to OpenStreetMap tags by geo-coordinates, producing 5.2M image-caption pairs before filtering and 2.6M after keeping the top 50% by CLIP similarity § Dataset (Dataset overview).
• Continual pre-training of CLIP on it (SkyCLIP) raises zero-shot scene classification and cross-modal retrieval scores, per the authors § Experiments.
• Captions are rule-assembled from tags, so their wording is rigid app. A.3.
• The dataset-construction pipeline is not in the repository; only the derived CSV files and training code are README.md.

Problem

Remote-sensing vision-language models need large image-text data. Web crawling does not find enough remote-sensing images with text around them, human annotation needs domain expertise, and existing remote-sensing datasets have at most about 150 semantic classes and rarely reach a million images § Introduction. The authors report that a classifier applied to LAION-2B finds only 726K remote-sensing pairs, about 0.03% of the data § Dataset (Comparison with the remote sensing subset in LAION).

Method

Convention for citations. The compiled paper's main sections are unnumbered, so they are cited by title, [§ Dataset] (subsection). Appendix sections carry their own numbers, [app. A.1]. The paper has no numbered equations and no numbered algorithms; the procedures are given in prose and in the pseudocode below.

A. Images. Images come from Google Earth Engine (GEE) collections with open redistribution. Ten collections are used, with ground sampling distance (GSD) from 0.1 to 30 m. Only the RGB bands are kept, even where multispectral bands exist § Dataset (Source of images) tab. 1.

B. Semantics. Each OpenStreetMap (OSM) object carries tags, and each tag is a free-text key and value. The paper treats the rich, noisy tag vocabulary as an asset, following the logic of web-scale noisy captions § Dataset (Source of semantics) fig. 2a.

C. Which tags are visible, and at what resolution. A tag's text is embedded with the text encoder of CLIP ViT-L/14 app. A.1. Two logistic-regression stages then decide:

1. whether the tag can be seen in remote-sensing imagery at all (binary);
2. if so, the coarsest GSD at which it can still be seen. The options are 0.1, 0.2, 0.6, 1 and 10 m § Dataset (Connect images with appropriate semantics) fig. 3.

The first stage was trained on 2,000 manually labelled tags, the second on 514. The reported cross-validation F1 is 0.88 for stage 1 and accuracy is 0.53 for stage 2. The second stage uses PCA before the logistic regression; the number of retained components is not stated app. A.1.

D. Which objects to sample. Two sampling passes § Dataset (Data selection):

1. Random: 400K grids of 0.01° × 0.01° are drawn across the globe, and the OSM objects inside them are queried through the Overpass API.
2. Targeted: for rare tags that the random pass missed, every object with that tag is queried.

E. Which image collection and tile. For each object app. A.2.1 app. A.2.2:

• Point or polyline objects: a collection is suitable if its GSD is smaller than the maximum GSD predicted for the object. The base tile is 224 × 224 px centred on the point, or on a randomly chosen node of the polyline.
• Polygon objects: the polygon's bounding box is the tile. A collection is suitable if it can produce an image whose height and width are both between 75 and 1,000 px.
• If several collections are suitable, one is picked at random.
• Tiles are then jittered. For points and polylines the anchor sits within the middle third of the tile and the size is drawn from 168 to 300 px. For polygons the size is drawn from 150 to 1,500 px with aspect ratio between 0.5 and 2.
• An object is skipped if no collection is suitable.

F. Captions. Each tile gets two captions § Dataset (Assembling caption):

• a single-object caption, describing only the object that set the tile boundary;
• a multi-object caption, which also lists the other OSM objects that fall in the tile, found by geospatial overlay.

Tags become phrases by rules app. A.3:

• some keys and values are renamed (for example a non-major road is written as "road");
• a descriptive adjective key is joined with a space ("natural water");
• an attribute key is joined with "is" ("smoothness is good");
• the value "construction" is joined with "under" ("building under construction");
• otherwise the default joiner is "of" ("lanes of 2").

Phrases of a single object are joined with commas. In a multi-object caption, the centre object's phrases are joined with "with" and "and", and the surrounding objects' phrases are attached with "surrounded by".

G. Filtering. Each image and its caption are embedded with OpenAI's CLIP ViT-L/14, and the cosine similarity is computed. The pairs are sorted by similarity, and the top 20%, 30% or 50% are kept. The top 50% gives the main 2.6M set § Dataset (Filtering out uncorrelated image-text pairs) § Dataset (Dataset overview).

H. Continual pre-training (SkyCLIP). The web-pretrained CLIP is trained further on the filtered pairs with the standard image-text contrastive loss. The initial weights are LAION-2B for ViT-B/32 and OpenAI for ViT-L/14. Training runs on 4 A100 GPUs with batch size 512 for 20 epochs § Experiments (CLIP continual pre-training). The paper says only "image-text contrastive learning". The code uses a symmetric cross-entropy over the batch, averaged over both directions loss.py.

# Build one (image tile, caption) pair from an OSM object
# [§ Dataset] (Data selection)
# Input:  osm_object (point, polyline or polygon, with tags); image_collections
# Output: (tile, single_caption, multi_caption), or None if skipped
def build_pair(osm_object, image_collections, tag_model):
    # 1. Keep tags that can be seen in remote-sensing images at all  [app. A.1]
    visible_tags = [t for t in osm_object.tags if stage1_visible(tag_model, t)]
    if not visible_tags:
        return None
    # 2. Attach the coarsest visible GSD to each kept tag  [app. A.1]
    #    How several tags combine into one object-level GSD is not stated.
    max_gsd = object_max_gsd(tag_model, visible_tags)
    # 3. Keep collections fine enough for the object  [app. A.2.1]
    suitable = [c for c in image_collections if c.gsd < max_gsd]
    if not suitable:
        return None
    collection = random_choice(suitable)  # random pick among suitable ones
    # 4. Place and jitter the tile around the object  [app. A.2.2]
    tile = place_tile(osm_object, collection)
    # 5. Assemble a single-object and a multi-object caption  [app. A.3]
    single_caption = assemble(osm_object.tags)
    multi_caption = assemble_with_neighbours(osm_object, tile)
    return tile, single_caption, multi_caption

# Keep the most similar pairs for continual pre-training
# [§ Dataset] (Filtering)
# Input:  pairs = list of (tile, caption); openai_clip (ViT-L/14)
# Output: the top `keep_fraction` of pairs, by image-caption cosine similarity
def filter_pairs(pairs, openai_clip, keep_fraction=0.5):
    # 1. Embed each image and each caption, L2-normalised
    #    [code: src/open_clip/model.py#L255-L271]
    scores = []
    for tile, caption in pairs:
        image_vec = normalise(openai_clip.encode_image(tile))
        text_vec = normalise(openai_clip.encode_text(caption))
        # 2. Cosine similarity of the pair  [§ Dataset] (Filtering)
        scores.append(dot(image_vec, text_vec))
    # 3. Sort descending and keep the top fraction
    order = argsort(scores, descending=True)
    return [pairs[i] for i in order[: int(keep_fraction * len(pairs))]]

# One training step of continual pre-training  [§ Experiments] (CLIP continual)
# Input:  image_batch (B, 3, H, W); caption_batch (B tokens)
# Output: scalar loss; model is initialised from web-pretrained CLIP
def training_step(model, image_batch, caption_batch):
    # 1. Encode both modalities and L2-normalise
    #    [code: src/open_clip/model.py#L255-L271]
    image_vecs = normalise(model.encode_image(image_batch))      # (B, d)
    text_vecs = normalise(model.encode_text(caption_batch))      # (B, d)
    # 2. Similarity matrix scaled by the learned temperature
    #    [code: src/open_clip/model.py#L213]
    #    [code: src/open_clip/loss.py#L106-L120]
    logits = model.logit_scale * image_vecs @ text_vecs.T        # (B, B)
    labels = arange(B)  # matched pairs sit on the diagonal
    # 3. Symmetric contrastive loss, averaged over both directions
    #    [code: src/open_clip/loss.py#L124-L133]
    loss = (cross_entropy(logits, labels)
            + cross_entropy(logits.T, labels)) / 2
    return loss

I. Evaluation.

• Zero-shot scene classification. Seven public benchmarks (AID, EuroSAT, fMoW, Million-AID, PatternNet, NWPU-RESISC45, RSI-CB256) plus the in-domain 70-class SkyScript-classification set § Experiments (Benchmark datasets) app. A.4 app. A.6. "Zero-shot" here means unseen datasets, not unseen geo-locations or unseen classes § Experiments (Definition of zero-shot transfer).
• Zero-shot fine-grained classification. Three new test sets: roof shape (6 classes), road smoothness (5) and road surface material (5). Each class has 60 to 100 images, collected with Google and Bing Maps instead of GEE, and excludes objects already in SkyScript § Experiments (Test data) app. A.7.
• Cross-modal retrieval. A held-out SkyScript test set of 30,000 pairs, plus RSICD, RSITMD and UCM-Captions. Scoring is cosine similarity § Experiments (Cross-modal retrieval).

# Zero-shot classifier and top-1 accuracy
# [code: src/training/zero_shot.py#L17-L32]
# Input:  class_names; templates (two per dataset)
# [code: prompt_templates.py#L1-L47]
# Output: top-1 accuracy over the test loader
def zero_shot_top1(model, class_names, templates, test_loader):
    # 1. One text vector per class: average the normalised template embeddings,
    #    then re-normalise the mean
    classifier = []
    for name in class_names:
        texts = [template(name) for template in templates]
        vecs = normalise(model.encode_text(texts))
        classifier.append(normalise(mean(vecs, axis=0)))
    classifier = stack(classifier, axis=1)                       # (d, C)
    # 2. Score each image: logits = 100 * cosine
    #    [code: test_zero_shot_classification.py#L139-L140]
    correct = 0
    total = 0
    for images, labels in test_loader:
        image_vecs = normalise(model.encode_image(images))
        logits = 100.0 * image_vecs @ classifier
        correct += sum(argmax(logits, axis=1) == labels)
        total += len(labels)
    return correct / total

# Recall@k for text-to-image retrieval
# [code: cross_modal_retrieval.ipynb#L382-L392]
# Input:  text_vecs (T, d), image_vecs (I, d), gold_images[t] = set of image ids
# Output: recall at k, as a fraction of queries
def text_to_image_recall(text_vecs, image_vecs, gold_images, ks):
    # 1. Rank all images for each text query by similarity
    # 2. A query is a hit if any gold image is in the top k (not all of them)
    hits = {k: 0 for k in ks}
    for t in range(len(text_vecs)):
        ranking = argsort(100.0 * text_vecs[t] @ image_vecs.T, descending=True)
        for k in ks:
            if set(ranking[:k]) & gold_images[t]:
                hits[k] += 1
    return {k: hits[k] / len(text_vecs) for k in ks}

Hyperparameters

Stated values come from the paper. Values marked "code" or "README" are read from the repository and are not in the paper. "Not stated" means neither source gives it.

Results

Author-reported, all on the setup in the paper. Scene and fine-grained columns are top-1 accuracy in percent tab. 2; the "Avg." column averages the seven public scene benchmarks, not the SkyScript-classification set. Rows that name a SkyCLIP variant are the authors' models. Bold marks the Rows for ViT-B/32 and ViT-L/14 are from the same table.

Headline gains, as stated by the authors. SkyCLIP-50 (ViT-L/14) averages 59.93%, which is +6.17 over original CLIP ViT-L/14, +2.06 over CLIP-laion-RS, +4.52 over human-curated captions and +9.98 over RemoteCLIP § Experiments (Results). The abstract's "6.2%" is this +6.17 rounded.

Fine-grained. Against the matching CLIP model, SkyCLIP gains +6.37 to +24.77 points, except a drop of −5.5 for ViT-B/32 on roof shape § Experiments (Results, fine-grained) tab. 2. The authors say these gains are larger than the scene gains.

Retrieval. Mean recall is the average of recall@1, recall@5 and recall@10. SkyCLIP-30 beats CLIP and CLIP-laion-RS by +2.57 to +6.48 on the three benchmarks it never saw § Experiments (Cross-modal retrieval) tab. 3. The authors report that it matches or beats three supervised models that were trained on those benchmarks; on UCM-Captions it beats them by more than 10 points for text-to-image and more than 25 points for image-to-text. The paper's own recall numbers are reproduced below.

Values are mean recall in percent tab. 3. Values in parentheses are in-domain: the model was trained on that dataset. The three supervised models are bracketed because they were trained on RSICD, RSITMD and UCM. The appendix gives recall@1, @5 and @10 for each cell app. A.8.2.

Dataset size versus filtering app. A.8.1 tab. A2:

Filtering with a remote-sensing CLIP app. A.8.3 tab. A4: using CLIP-laion-RS for the similarity filter, SkyCLIP-30 reaches an average of 60.69 instead of 59.91.

Unseen classes app. A.8.4: with ten classes removed from SkyScript, the paper reports SkyCLIP averages of 96.1% on Million-AID and 94.0% on PatternNet, against 93.6% and 86.5% for CLIP. The ten excluded classes are listed, but the paper does not say how the two benchmarks were restricted to unseen classes, so these figures cannot be reproduced from the card.

Layer freezing fig. 5: the figure is described only in words. Freezing more layers of either encoder lowers zero-shot accuracy. Freezing the image encoder hurts more than freezing the text encoder. The paper concludes that both encoders need to adapt, and the image encoder adapts more. No numbers are given in the text.

Dataset statistics § Dataset (Dataset overview) (Geographic coverage) (Semantic diversity): 5.2M pairs and 44K tags before filtering; 2.6M pairs and 29K tags after keeping the top 50%; 100K distinct single-object captions and 1.2M distinct multi-object captions. The authors estimate 96.1% tag accuracy from a sample of 1,000 image-tag pairs. Coverage is concentrated in the U.S. and Europe, where high-resolution images are available, and 580 tags have at least 1,000 images.

Image collections, for Method A tab. 1:

Labels below are paraphrased from the paper's table; the numbers are unchanged.

Paper vs code

Searched: the paper's footnote link to GitHub (wangzhecheng/SkyScript, whose owner is the first author, Zhecheng Wang). Its LICENSE is MIT and names all five authors. It is the official code, cloned at code_commit. The README says the model code is adapted from OpenCLIP README.md.

• Dataset construction is not in the repository. There is no OSM query, GEE download, tag classifier, image-tile selection or caption-assembly code. The README offers the built image zips, metadata and CSV files for download README.md README.md. The paper says the caption algorithm is in "Supplementary Code" app. A.3, which is not in this repository.
• Batch size. The paper says 512 in total § Experiments. The training command uses 128 per GPU on 4 GPUs README.md, and the flag is "per GPU" params.py. Consistent once read per GPU, but the default in code is 64.
• Learning rate and schedule. The paper states neither. The README's values (3e-9 for ViT-L/14, 1e-9 for ViT-B/32) are the only ones available README.md. Check them against a run before use, since the paper gives nothing to compare them with. The code's cosine schedule is confirmed customized_train_and_test.py.
• Caption type for scene classification. The paper does not say. The README's named checkpoint for the 30% ViT-L/14 model is trained on the single-object column title, and its multi-object checkpoint is listed separately README.md. Retrieval uses multi-object captions for SkyCLIP-30, as the paper says tab. 3.
• Zero-shot prompts. The paper gives no prompts. The code uses two templates per dataset and averages their normalised embeddings, then re-normalises prompt_templates.py zero_shot.py. The fine-grained tests use "building" or "road" as a prefix prompt_templates.py.
• Logit scale. The code uses 100 at test time test_zero_shot_classification.py. The paper does not say.
• Retrieval metric. The paper says cosine similarity and mean recall. The notebook counts a query as a hit if any gold item appears in the top k, and it also computes R@100, which the paper does not report cross_modal_retrieval.ipynb. The notebook's saved UCM values match the appendix's SkyCLIP-30 numbers to within rounding; the checkpoint used in that notebook is not named.
• Unfiltered size. The paper says 5.2M pairs. The README file name and note say 5.1M README.md while the README's own text says 5.2M README.md.
• Zero-shot checkpoint table. The README row for the ViT-L/14 average of 59.93 is labelled "SkyCLIP_ViT_B32_top50pct" with the type "CLIP ViT-L14" README.md. The separate L-14 top-50% checkpoint is listed under downloads README.md; the mislabelled row is a documentation error.
• Later changes. The README says captions were re-written with ChatGPT in 2024 README.md. These are not in the paper, and no results use them.
• No code for the layer-freezing experiment, the fine-grained test set collection, the tag-classifier training, or the similarity-filter ablation. The training flags for locking towers exist params.py, but nothing runs that experiment.

Pitfalls

• Dataset images are not ours to redistribute. The paper says the GEE collections allow sharing and redistribution, and that each image's licence is covered by its collection § Dataset (Source of images). The licences per collection are not listed in the paper, and the dataset's own licence is not stated. Do not republish image files from this work without checking each collection's terms.
• The headline 6.2% is one comparison. It is ViT-L/14 SkyCLIP-50 against original CLIP ViT-L/14 on the seven public scene sets; the in-domain SkyScript-classification set is excluded from that average. Use the table, not the abstract, when comparing numbers.
• "Zero-shot" means unseen datasets, not unseen places. The paper admits that SkyScript and the benchmarks may cover the same geo-locations, but argues the images differ § Experiments (Definition of zero-shot transfer). Read the scene results with that in mind.
• Tag-to-GSD rule is underspecified. The paper uses one maximum GSD per object but does not say how an object with several tags combines the per-tag predictions app. A.2.1 app. A.1. The pseudocode marks this as a gap.
• Stage-2 classifier is weak. Its cross-validation accuracy is 0.53 over five GSD classes app. A.1. Errors in the GSD cut-off change which collection is used, and the paper reports no analysis of that effect.
• Filtering is a trade-off. Keeping only the top 50% cuts the set in half but raises accuracy. The unfiltered 5.2M set does worse than 2.6M on average (58.65 vs 59.93) app. A.8.1 tab. A2. The filter model also matters: the CLIP-laion-RS filter gives a different average than the OpenAI filter used in the main results (60.69 vs 59.91) app. A.8.3.
• Caption wording is rigid. Captions are built from rules and read like bag-of-words phrases; the paper expects better captions to help app. A.3. Do not expect natural-language captions from this set.
• Coverage bias. High-resolution imagery is mostly in the U.S. and Europe, and OSM coverage is thinner in developing countries § Dataset (Applications and limitations). Results may not transfer to those regions.
• Table label collision in the TeX source. The appendix classification table reuses the label of the main results table, so cite the appendix table by its number (A4), not by label.
• Figures are described, not reproduced. Figure numbers in this card follow the compiled paper; the appendix uses A-prefixed numbers. No figure is copied.

Lineage

• Builds on: CLIP, Radford et al. (2021), which gives the contrastive image-text objective and the web-pretrained starting weights § Introduction § Experiments (CLIP continual pre-training). The code adapts OpenCLIP README.md.
• Data sources: Google Earth Engine for images, OpenStreetMap for tags § Dataset (Source of images) (Source of semantics).
• Compared against: CLIP original, CLIP-laion-RS (a remote-sensing subset of LAION-2B), human-curated captions (NWPU-Captions, UCM-Captions, Sydney-Captions, RSICD), RemoteCLIP (Liu et al., 2023), and for retrieval AMFMN, LW-MCR and GaLR § Experiments (Results) (Cross-modal retrieval).