Earth AI: Unlocking Geospatial Insights with Foundation Models and Cross-Modal Reasoning

In plain words

Earth AI is a set of tools for questions about the planet. Some tools read satellite and aerial pictures and can be asked, in plain language, what is in them. Others describe how people live in each neighbourhood, and others forecast weather, floods and cyclones. Usually these sources sit in separate systems, so a person has to download each result and join them by hand.

The paper's idea is to put these tools behind one assistant. The assistant reads the question, splits it into steps, calls the right tool or model for each step, and writes up the answer. The authors argue that combining sources predicts better than any single one. They test this on a few prediction tasks and on question sets they wrote themselves. On those sets, their assistant answers more accurately than a general-purpose language model that has web search, maps and a code tool but none of the specialised data.

These results come from the authors. Only some of the parts are open to outside users, and the paper says where its evaluation is still thin.

TL;DR

• Earth AI has three model groups: Imagery (remote sensing), Population (embeddings built from aggregated human-activity and map data) and Environment (weather, flood and cyclone forecasts) §2.1.
• Combining embeddings helps. On 20 FEMA risk indices, AlphaEarth plus Population Dynamics gives mean R² 0.60, against 0.54 for either alone §3.2 tab. 9. On 21 CDC health indicators it gives 0.60, against 0.56 and 0.42 §3.2 tab. 10.
• A Gemini-based agent built with Google's Agent Development Kit scores 0.82 on a 100-question fill-in benchmark, against 0.50 for a Gemini 2.5 Pro baseline §3.3.1 tab. 11. On 10 crisis prompts it scores 0.87 against 0.38 on a Likert scale, with a rubric-based autorater §3.3.2 tab. 12.
• All results are author-reported. The only linked repository is the Population Dynamics embedding code, which contains notebooks and no training code for the other models, the agent or the benchmarks.

Problem

Geospatial data comes in many modalities, resolutions, time scales and levels of sparsity, and it is held in separate sources §1. A model built on one source sees only one view of the planet §2.2. Simple questions can be answered by retrieving data, but analytical and predictive questions need several sources joined and modelled §2.3. The paper's baselines show that a general language model with search, maps and code execution, but without specialised tools, scores 0.50 overall on its question set tab. 11. The approach assumes access to the Google Earth Engine, Data Commons and Places APIs, and to Google's weather and cyclone services app. A.3.1.

Method

Overview

The system has three layers. Foundation models for three data domains are trained or used as delivered §2.1. Their outputs are combined by simple tabular models that use embeddings as input features §2.2 app. A.2. A Gemini-based agent calls the models and data sources as tools or expert sub-agents §2.3 fig. 4.

1. Imagery: Remote Sensing Foundations §2.1.1 app. A.1.1 fig. 2

The family has three parts.

1. Vision-language model (VLM). Two existing VLM families are used, MaMMUT and SigLIP2, each with a 400M-parameter image encoder and 400M text encoders app. A.1.1. Images and text are mapped into one joint embedding space, which allows zero-shot classification and text-to-image and image-to-text retrieval §2.1.1 §3.1.1. The training data are three sets:

   • RS-Landmarks: 18M aerial and satellite images, aligned to Google Maps place footprints, captioned by Gemini 1.5 Pro app. A.1.1.
   • RS-WebLI: about 3M clean aerial and satellite images filtered from WebLI. Filter classifiers were trained on a hand-labelled set, then on a crowd-labelled set of 60K images. A larger variant was built from the 100-billion-image WebLI app. A.1.1.
   • RS-Global: 30M images at native sensor resolution (10 cm to 10 m), from 2003 to 2022, captioned by Gemini 1.5 Pro app. A.1.1.

   RS-MaMMUT is fine-tuned on RS-Landmarks and the smaller RS-WebLI. RS-SigLIP2 is trained on the larger RS-WebLI, RS-Landmarks and RS-Global app. A.1.1. The paper defers the training recipe to a separate work §3.1.1.

2. Open-vocabulary object detection (OVD). The detector is OWL-ViT v2. It has separate text and image transformers, so image embeddings can be computed once and cached app. A.1.1. Training has two stages. In the first, the RS-WebLI boxes are pseudo-labelled by OWL-ViT, using alt text or a fixed label set, and only boxes above a confidence threshold are kept. The model is fine-tuned on this for 2 epochs. In the second, it is fine-tuned for 32 more epochs on an internal set of 67,000 aerial images with over 3.5M instances in 34 classes. Both stages use a linear learning-rate cooldown app. A.1.1.

3. Few-shot adaptation with FLAME. A cascade adds a lightweight classifier on top of the zero-shot OVD model. FLAME is a one-step active-learning method app. A.1.1. It chooses which images to label in two steps:

   • Uncertainty filtering: each object embedding combines the image feature with the text query, and the dimension is reduced. Samples on the low-density margin of the distribution are kept as candidates app. A.1.1.
   • Diversity sampling: the candidates are clustered with k-means. From each cluster, the sample nearest the centre is taken app. A.1.1.

   The evaluation used 30 labelled examples per category §3.1.1.

4. Vision-transformer backbone. The encoder is pre-trained in two stages app. A.1.1 fig. 2. First, masked autoencoding (MAE) on 300M images, randomly cropped and resized so that resolution varies between 10 cm and 10 m. Second, multi-task pre-training on scene classification, semantic segmentation and object detection, with labelled data from Google's internal sets. The tasks are trained with Alternating Gradient Descent: the optimiser alternates between tasks instead of summing their losses into one step app. A.1.1.

2. Population: Population Dynamics Foundations (PDFM) §2.1.2 app. A.1.2 fig. 3

The original model, from [agarwal2024general] (arXiv:2411.07207), is a graph neural network. It fuses aggregated maps data, search trends, busyness and weather and air quality into one embedding per region, with privacy aims §2.1.2. The paper makes two changes app. A.1.2.

• Global embeddings (17 countries). The search-trend part aggregates Knowledge Graph entities instead of raw top-query counts. For each postal code, the top 500 entities searched at least 20 times are counted. Entities are ranked by how many regions in the 17 countries they appear in, and the top 1000 form the vocabulary. The least popular one appears in more than 9,000 regions. The search data are from October 2023.
• Monthly temporal embeddings. One embedding per month, from July 2023 onward, built the same way from that month's signals only. These use a shared search-trend vocabulary and not the Knowledge Graph mapping app. A.1.2.

Embeddings are defined per postal code, or a similar region where postal boundaries are missing app. A.1.2.

3. Environment: weather, flood and cyclone services §2.1.3

The paper integrates three services and does not describe training them.

• Weather: the Google Maps Platform Weather API, which uses machine-learning models such as MetNet. Hourly forecasts run to 240 hours and daily ones to 10 days §2.1.3.
• Flood: the Google Flood Forecasting API, which gives riverine flood predictions from gauges, with area, severity and probability. History is available from 1 August 2025 §2.1.3.
• Cyclones: an experimental stochastic neural model that gives 50 scenarios up to 15 days ahead, covering formation, track, intensity, size and shape. History is available from 1 January 2022 §2.1.3.

4. Combining models §2.2 app. A.2

Each source is first mapped to the same administrative units, then used as features in a task-specific model. Three uses are described.

a. Census-tract risk and health models (FEMA, CDC). Each census tract gets two feature sets app. A.2.1:

# Census-tract feature vector  [app. A.2.1]
# Input:  tract, AlphaEarth image of the tract (2023), PDFM zip embeddings
# Output: feature vector for one tract

def tract_features(tract, alpha_image, pdfm_by_zip, overlaps):
    # 1. AlphaEarth: 64 bands, 10 histogram features each  [app. A.2.1]
    alpha = []
    for band in range(64):
        alpha += histogram_features(alpha_image, tract, band, n_features=10)
    # 2. PDFM: zip embeddings averaged by land-area overlap  [app. A.2.1]
    #    (overlap weights come from 2020 census land-area data)
    pdfm = area_weighted_mean(pdfm_by_zip, overlaps[tract])
    # 3. Concatenate; one tree model per target  [app. A.2.1] [§3.2]
    return concatenate(alpha, pdfm)

The AlphaEarth band histograms give 640 features per tract. The PDFM embedding is then converted to tract level app. A.2.1. A gradient-boosted tree model is trained for each target, with 80% of the counties for training and validation and 20% held out §3.2. The paper says the trees use "a fixed set of hyperparameters" app. A.2.1, without giving them.

b. Disaster damage (hurricane wind). A model built by Bellwether, a Google X project §3.2, classifies whether each building is damaged, using a gradient-boosted tree model app. A.2.2. Its inputs are:

• a dataset of over 1.5 million property records from 22 US hurricane events in 2017 to 2022, with damage labels and property features (six events held out for validation and testing) app. A.2.2;
• HURDAT2 storm data and building heights from Open Buildings app. A.2.2;
• PDFM embeddings app. A.2.2;
• forecast tracks and wind speeds from the cyclone model, one of 50 ensemble members per run app. A.2.2 fig. 6.

For a forecast 3 days before Hurricane Ian's landfall, the expected count of damaged buildings is compared with the observed count §3.2 fig. 6.

c. Cholera forecasting in the DRC. A TimesFM forecast of weekly cases for 24 provinces is compared with variants that add dynamic weather forecasts and PDFM embeddings as static covariates §3.2. The paper describes the forecast horizon as 20 weeks at national level §3.2.

5. Orchestration: the Geospatial Reasoning agent §2.3 app. A.3 fig. 4

The agent is built with Google's Agent Development Kit and uses Gemini 2.5 Flash and Pro models app. A.3. The paper does not say which model ran the agent in each evaluation. Capabilities are grouped into domains app. A.3.1:

• Demographics (expert): picks Data Commons variables for statistical queries.
• Weather (expert): fetches historical and forecast weather, cyclone and flood data.
• Spatiotemporal model training (expert): trains small models on the fly on top of pre-computed embeddings, with PDFM and TimesFM as base models.
• Earth Engine (expert): finds the right dataset and band by searching the public catalogue.
• Search (expert): uses grounded web search when the other tools lack data.
• Location and places (toolset): turns named places into polygons with the Places API and related services.
• Remote sensing and imagery (toolset): runs the RS VLM and detector, with image embeddings cached per area.

The loop has three stages: think and plan, run data operations or model inference or training, then reflect and recover. It repeats until it can give a grounded answer fig. 4 §2.3.

# Agent loop as described  [§2.3] [fig. 4]
# Input:  user query, domain experts and tools
# Output: final answer with the reasoning shown

def answer(query, experts, tools):
    # 1. Think and plan: split the query into sub-tasks  [§2.3]
    plan = make_plan(query)
    # 2. Run each sub-task on an expert agent or a tool  [§2.3]
    results = [run_subtask(task, experts, tools) for task in plan]
    # 3. Reflect and recover, then repeat until grounded  [fig. 4]
    #    (stopping rule and retry limits are not stated)
    return synthesise(query, results)

Hyperparameters

Values as stated in the paper unless marked "code". The paper gives few numeric training settings. Where a value is missing, it is marked not stated.

Results

All results are author-reported. Each row is taken from the paper's table of the same number.

Remote sensing zero-shot classification tab. 1

Top-1 accuracy, zero-shot, on public benchmarks §3.1.1.

*SkyScript's own score is on an in-domain test set and is not comparable tab. 1 footnote.

Remote sensing retrieval tab. 2

Average of top-1, 5 and 10 recall, image-to-text (I2T) and text-to-image (T2I), on RSICD, UCM-Captions, RSITMD and NWPU. Baselines are SigLIP2 400M and MaMMUT 400M, with the paper's RS variants:

RS-MaMMUT and the other baselines are in tab. 2. RS-SigLIP2 is best in every column the paper marks in bold tab. 2.

Open-vocabulary detection tab. 3

Setup: mAP on DOTA and DIOR §3.1.1.

Backbone fine-tuning tab. 4

Results relative to ImageNet-pretrained ViT-L/16 (0 means baseline). Setup: same batch size and epoch count, with learning rate tuned per model §3.1.1.

State of the art on public splits tab. 5

Population embeddings tab. 6 tab. 7 fig. 5

Spatial interpolation, R², 80% of regions per country used for training and 20% predicted, over 17 countries §3.1.2. The table gives per-country values; the "All Countries Average" row gives 0.85 across the four variables. Variables are elevation (0.91 mean), night-time lights (0.88), population density (0.88) and tree cover (0.73) tab. 6.

Cross-country extrapolation, training on seven European countries and predicting France from NUTS 3 data tab. 7:

Temporal embeddings are only shown as a figure: the MAE reduction for monthly emergency-department extrapolation, using time-matched monthly embeddings against static July 2023 embeddings fig. 5. The text reports no numbers.

External partner validations are listed in §3.1.2 tab. 8. They report results such as R² 0.882 for CARTO's Human Activity Index, 93% top-2 accuracy for home insurance premium classes at county level, and a rise in dengue forecast R² from 0.500 to 0.686 at 6 months with TimesFM tab. 8. Several are described as in progress or preliminary, and none is reproducible from this paper.

Combined predictions §3.2

FEMA risk scores, R² per index, census tracts, 20% of counties held out tab. 9:

The combined model is best or tied on 19 of 20 indices. It ties on drought and hail, and on wildfire AlphaEarth alone is highest tab. 9. The paper reports an 11% relative gain in mean R² §3.2.

CDC health indicators, 21 targets, same setup tab. 10. The mean R² is 0.42 for AlphaEarth, 0.56 for PDFM and 0.60 for the combination. The combined model is best on all 21 targets tab. 10. The paper reports 7% over PDFM and 43% over AlphaEarth §3.2.

Cyclone damage, Hurricane Ian §3.2 fig. 6: the model predicts an average of 2,575 damaged buildings from 50 ensemble members three days ahead, against 2,496 observed, an error of 3%. Recall, precision and F1 for the damaged class on the held-out test set are 0.60, 0.57 and 0.59 app. A.2.2. The paper calls this one retrospective case.

Question answering tab. 11

Fill-in-the-blank questions, 100 in total, with 50 descriptive and 50 analytical. Scores are averaged over 5 runs, with 95% confidence intervals §3.3.1.

Baselines are Gemini 2.5 Pro and Flash with Google Search grounding, Maps grounding and code execution, and no custom tools app. A.3.2.

Crisis response tab. 12 app. A.3.4

Ten prompts, each run 10 times. Scores are Likert values mapped to 0 to 1, judged by an autorater §3.3.2 app. A.3.4. The prompts and rubrics are in app. A.3.3.

*The baseline could not find authoritative 2024 labour statistics, so it could not answer prompt 9 tab. 12 footnote.

Paper vs code

Repository checked. The paper links github.com/google-research/population-dynamics app. A.1.2. The owner is the Google Research organisation the authors work for. The clone is at commit 67b3f44a5b56cab39f7fa4e84f99c672440ac1e4, Apache-2.0. It holds a README, five notebooks, benchmark data files and two GeoJSON boundary files, but no model training code.

What the repository does not contain. It has no code for the graph neural network that builds PDFM embeddings, the remote sensing models, the Bellwether damage model, the FEMA and CDC pipelines, the agent, or either benchmark. The embeddings themselves are not in the repository. Access is by approval, through a BigQuery listing, for research use [code: raw/code/2510.18318/README.md#L43-L45]. The paper says both AlphaEarth and PDFM are "available for general use" §3.2. The repository says PDFM access needs approval and that commercial access is planned through a paid programme README.md.

Discrepancies found in the notebooks:

• Forecast model version. The paper says "TimesFM 2.0" §3.2. The notebook loads TimesFM_2p5_200M [code: raw/code/2510.18318/notebooks/pdfm_timesfm_forecasting_final.ipynb#L310] with max_horizon=256 ....
• Use of the embeddings. The paper says the embeddings enter as "static numerical covariates" §3.2. The notebook adds an error-correction model, a LightGBM regressor fitted on 330 PDFM features plus the TimesFM forecast, and it predicts the observed values, not the residuals [code: raw/code/2510.18318/notebooks/pdfm_timesfm_forecasting_final.ipynb#L368-L375] pdfm_timesfm_forecasting_final.ipynb. Training uses the last train_time_steps=3 observed steps by default [code: raw/code/2510.18318/notebooks/pdfm_timesfm_forecasting_final.ipynb#L402]. The forecast horizon default is 24 steps [code: raw/code/2510.18318/notebooks/pdfm_timesfm_forecasting_final.ipynb#L377], against 20 weeks in the paper.
• Boosted-tree settings. The paper gives no values for the FEMA, CDC or damage models app. A.2.1 app. A.2.2. The notebooks use different settings: 100 trees with learning rate 0.1 in the nowcasting notebook pdfm_nowcasting.ipynb, 400 trees with learning rate 0.04 in the super-resolution notebook [code: raw/code/2510.18318/notebooks/pdfm_superresolution_and_imputation.ipynb#L747-L748], and 400 trees with learning rate 0.05 for the forecasting notebook above pdfm_timesfm_forecasting_final.ipynb.
• The notebooks are demonstrations on public county and ZIP data, not the paper's experiments. No notebook reproduces the cholera, FEMA, CDC or damage results.

Pitfalls

• Most inputs and labels are not public. The damage dataset (1.5M records), the Bellwether model, the partner validations and the internal object-detection set are not released, and the paper does not say they will be. Results from them cannot be reproduced from this paper.
• Many training settings are not stated. OVD learning rate and pseudo-label threshold, MAE masking and schedule, the MTP schedule, FLAME's k-means cluster count, the VLM recipe (deferred to another work), and all tree-model hyperparameters. Do not copy the notebook values into the paper's results without saying so.
• Few-shot selection needs a choice of k. The paper names k-means for diversity sampling app. A.1.1 but gives no cluster count or distance function.
• AlphaEarth features are underspecified. The paper gives 64 bands, 10 histogram features per band and the 2023 image app. A.2.1, but not the bin edges, the normalisation or how pixels are pooled.
• Area-weighted averaging is assumed normalised. The paper says "area-weighted averaging" and names the 2020 census overlap file app. A.2.1. It does not write out the normalisation.
• Backbone average is not reproducible from the table. The text claims a 14.93% average gain across all task categories §3.1.1. The table's four category columns average to 24.87 for RS-Global MTP. The paper does not say how its 14.93% was computed. Classification is also the one column where DINOv2 (4.89%) is higher than RS-Global MTP (4.58%) tab. 4.
• SOTA claims are narrower than the text. The text says the RS models are state of the art on "the vast majority" of zero-shot benchmarks §3.1.1. RemoteCLIP scores 91.30 on AID against 78.26 for RS-SigLIP2 tab. 1. SkyScript's score is on an easier in-domain split tab. 1 footnote.
• Rounding in the Q&A text. The text gives 37% for the agent's gain over Gemini 2.5 Pro on descriptive questions §3.3.1. The rounded table values (0.91 and 0.67) give 36%. The 124% analytical gain and the 64% overall gain match the table. Use the table values.
• Crisis mean depends on row 9. The reported mean excludes prompt 9, where Gemini scored 0. Averaging over all 10 prompts gives 0.35 for Gemini (sum 3.48) and 0.88 for the agent (sum 8.84), computed from the table values. State which convention you use.
• The agent's model and stopping rule are not stated. The paper does not say whether Gemini 2.5 Pro or Flash ran the agent in each evaluation, nor how the loop decides it is grounded, how many retries it allows, or how it handles failed tool calls app. A.3 fig. 4.
• Autorater dependence. The crisis scores come from Gemini 2.5 Flash judging rubrics app. A.3.4. The paper itself notes that answers can be synthesised from news rather than from analysis of the data, and that an agent's final list can change when the underlying model's output changes app. A.3.3.
• The Q&A benchmark cannot be checked from the paper. It is said to use public data and APIs, and its gold answers were generated and then checked by experts app. A.3.3. The question list is not stated to be released.
• Temporal embedding gains exist only in a figure. Use fig. 5 for the shape of the result. No numbers are stated.
• The cyclone result is one case. One hurricane, one lead time, and a single snapshot of precision and recall. The paper warns against generalising §3.2 fig. 6.
• Hidden costs. The paper relies on Gemini 1.5 Pro to caption about 48M images (RS-Landmarks and RS-Global), on 300M images for MAE pre-training, and on a 100-billion-image source for RS-WebLI. Compute and cost are not stated.
• Licensing of the reused material. The paper is CC BY 4.0. Its data sources (Data Commons, Earth Engine, CDC and others) have their own terms, and the embeddings have access restrictions README.md.

Lineage

• Builds on: [agarwal2024general], arXiv:2411.07207 (PDFM). This paper adds 17-country global embeddings, monthly temporal embeddings and a Knowledge Graph vocabulary for search trends app. A.1.2.
• Uses without change: AlphaEarth Foundations, arXiv:2507.22291 §2.1.1 app. A.2.1; FLAME, arXiv:2510.17670 app. A.1.1; TimesFM, arXiv:2405.14227 §3.2; OWL-ViT v2 and MAE, as cited in app. A.1.1.
• Remote sensing VLM recipe: arXiv:2503.08722, which the paper cites for training details §3.1.1.
• Compared against: SatCLIP, GeoCLIP, RemoteCLIP, GeoRSCLIP, SkyScript, SigLIP2, MaMMUT, DINOv2, SIoU, IDW and RBF interpolation, Prophet and SARIMAX §3.1 §3.2 app. A.1.
• Superseded by: not stated.