wikillm

Paper · 2026-10-06 · cs.AI

GeoNatureAgent (GNA): A Framework and Benchmark for Pre-Production Evaluation of Tool-Using Agents on Geospatial and Environmental Tasks

GNA evaluates tool-using LLM agents on 16 geospatial tools served over MCP, scored by a 103-task suite with deterministic checks and no LLM judge. Nine models ran under temperature-1.0 seeds. Claude Sonnet 4 leads at 61.7% ± 0.7% on all tasks; DeepSeek V3.2 reaches 57.9% at 11.3× lower cost (author-reported).

LLM agentTool useLLM evaluationBenchmark validityllm-agentstool-usemcpbenchmarkgeospatialevaluation

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Before an AI agent that calls software tools is let loose on environmental or map work, its builders need evidence that it picks the right operation against the real service. This paper builds a test bench for that. It gives the agent a fixed set of sixteen geospatial tools, served through a standard tool protocol, and a set of 103 test questions. A fixed, rule-based scorer marks each run, and no second AI grades the answers. Any agent that speaks the protocol can be plugged in, so the agent under test is the only thing that changes.

The authors ran nine language models through it. The best one solved 61.7% of the tasks. A cheaper open-weight model solved 57.9% at a much lower price per query. The scoring is strict: a task counts only if every check passes, so many near-misses are scored as failures. The authors argue that much of the gap to other geospatial benchmarks comes from that strictness rather than from task difficulty alone.

TL;DR

  • GNA is a pre-production test harness for LLM agents that call geospatial tools. It is a fixed MCP tool server, a 103-task suite and a deterministic scorer with no LLM judge §1 §3 §4.
  • Nine models were run at temperature 1.0 under three seeds, with Claude Sonnet 4 under two samples §6.1.
  • Author-reported headline: Claude Sonnet 4 scores 61.7% ± 0.7% on all 103 tasks and DeepSeek V3.2 scores 57.9% ± 3.9% §6.2 tab. 6.
  • Missing tool calls cause 68.0% of failures across the nine models §6.4.2 tab. 8.
  • The code is Apache-2.0. Its committed result files match the leaderboard, but several details differ from the paper (see Paper vs code).

Problem

Teams deploying tool-using LLM agents in environmental and geospatial work need evidence that the agent chooses the right operations against real APIs §1. The authors say no reusable, self-hostable test substrate existed for that pre-production check. Existing geospatial agent benchmarks mostly target general GIS tasks or pixel-level vision models, and knowledge or code-generation benchmarks do not test structured API calls §1 §2.2 §2.4. The setting assumes an agent that calls a real API with multi-turn history and multilingual input, and that is judged on end-to-end task completion §2.4 §5.

Method

GNA has three parts: a fixed tool layer, a tool-calling agent loop that can wrap any model, and a deterministic scorer over a task suite §3. The tool layer uses the same tool definitions as the production API and as the evaluation harness §3 tools.json.

  1. Tool layer. Sixteen tools: eight domain tools, four GIS operations adapted from GeoBenchX, and four helper tools that list layers and summarise erosion results §3 tab. 2 tools.json. The eight domain tools and four GIS operations are listed in the table below.
  2. MCP exposure. The tools are published as a Model Context Protocol server that implements only the Tools capability (list and call). Resources and Prompts are deliberately not offered, so the agent must make the tool choice itself §3 mcp_server.py.
  3. Agent loop. The agent works in a ReAct-style cycle: read the query, reason about tools, call them, observe results, and answer §3. The loop is capped at ten tool rounds. When the cap is hit, the harness returns a fallback answer and records the round count as ten agent.py agent.py.
  4. Model adapters. A native Anthropic client and a LiteLLM client map responses into one schema §3. If a model returns no structured tool call, the client looks for tool calls in its text: Python-style calls wrapped in <|python_start|> tokens, and Gemma-style tool_code blocks llm_client.py llm_client.py.
  5. Tasks. The suite has 93 main tasks in 18 categories (v5) and 10 comparison tasks (v6) §4 tab. 3. Each task gives a query (optionally with prior turns), expected tools, required and forbidden substrings, a round budget, a cost budget and expert ground truth §4 benchmark_v5.json.
  6. Scoring. Eight check types §5 tab. 4. A run passes capability when every applicable check passes, except the cost budget, which is reported separately §5 compile_final_results.py. Partial credit is the share of checks passed §5.
  7. Aggregation. Accuracy per seed, then mean and standard deviation over seeds §6.1 §6.2.
  8. Failure label. Each failed run gets the label of its first failing check, in a fixed priority order §5 scoring.py.
  9. Cost and carbon. Cost per call is token counts times a per-model list price per million tokens metrics.py. Carbon is an order-of-magnitude token-based estimate §6.3.

Tools as listed in the paper tab. 2. The paper's name for the last domain tool differs from the code; see Paper vs code.

ToolPurpose
lookup_provinceResolve a province to its boundary geometry
lookup_municipalityResolve a municipality, with an optional province hint
analyze_areaZonal statistics for an indicator inside an area
analyze_multi_layerSeveral indicators in one call
compare_areasSet two areas against each other on one indicator
find_top_nOrder provinces by an indicator's value
toggle_layerShow or hide a map layer
generate_chartMake a bar or stacked-bar chart from results
create_bufferGrow a geometry outward by a distance in km
select_spatial_relKeep features that satisfy a spatial condition (named select_features_by_spatial_relationship in code)
get_centroidsReturn each feature's centre point
reject_taskRefuse a request the data cannot answer

The four auxiliary tools are list_layers, get_legend, get_layer_bounds and query_erosion_stats tools.json.

Scoring procedure, simplified. The paper's checks are in tab. 4. The chart and action checks, and the optional latency check, are omitted here.

Pseudocode
# Capability pass and partial credit for one run  [§5] [tab. 4]
# Input:  case (expected tools, keywords, round budget), run (answer, tools)
# Output: passed (bool), check_score (fraction of checks passed)

def score_run(case, run):
    # Each expected tool must be called at least once; extras are fine  [§5]
    checks = [set(case.expected_tools) <= run.tools_called]
    # Each required keyword must appear in the answer, ignoring case  [§5]
    answer = run.answer.lower()
    for keyword in case.must_contain:
        checks.append(keyword.lower() in answer)
    # Each forbidden keyword must be absent from the answer  [§5]
    for keyword in case.must_not_contain:
        checks.append(keyword.lower() not in answer)
    # Numeric checks: label, then first percentage within 120 characters  [§5]
    checks += numeric_checks(case.ground_truth, run.answer)
    # The loop must not exceed the case's round budget  [§5]
    checks.append(run.rounds <= case.max_rounds)
    # The cost budget is not part of capability; it is reported separately  [§5]
    passed = all(checks)
    check_score = sum(checks) / len(checks)
    return passed, check_score

Numeric check, as implemented §5 tab. 4, note b:

Pseudocode
# Numeric check for one ground-truth entry  [§5] [code: scoring.py#L205-L217]
# Input:  label (str), expected (float), tolerance (float, default 2.0)
# Output: passed (bool)

def numeric_check(answer, label, expected, tolerance):
    # Find the first occurrence of the label in the answer  [§5]
    start = answer.lower().find(label.lower())
    if start == -1:
        return False
    # Take the first number before a percent sign, within 120 characters  [§5]
    window = answer[start:start + 120]
    found = first_percentage(window)
    if found is None:
        return False
    # Pass when the value lies within the tolerance, in percentage points  [§5]
    return abs(found - expected) <= tolerance

Cost and carbon, as the paper and code define them:

Pseudocode
# USD cost of one model call  [code: geoagentbench/metrics.py#L40-L45]
# Input:  price (input and output USD per 1M tokens), token counts
# Output: cost in USD

def call_cost(price, input_tokens, output_tokens):
    # Input and output tokens are priced separately  [code: metrics.py#L43-L45]
    input_part = input_tokens * price.input / 1_000_000
    output_part = output_tokens * price.output / 1_000_000
    return input_part + output_part
Pseudocode
# Estimated grams of CO2 for one case  [§6.3]
# Input:  tokens (int), total input plus output tokens of the case
# Output: grams of CO2 (order-of-magnitude estimate)

def carbon_per_case(tokens):
    # Blended energy at 0.40 Wh per 1,000 tokens  [§6.3]
    energy_wh = tokens / 1_000 * 0.40
    # Global-average grid intensity of 400 gCO2 per kWh  [§6.3]
    return energy_wh / 1_000 * 400

Hyperparameters

NameValueWhereNotes
Sampling temperature1.0§6.1 exp_042_claude_sonnet4_v5_seeds5.yamlChosen to expose run-to-run variance. Same in the other v5 experiment files.
Max output tokens4096§6.1 exp_042_claude_sonnet4_v5_seeds5.yamlSame in the v5 and v6 experiment files for all models checked.
Tool rounds per query10§6.1 agent.pyThe yaml max_turns: 10 is only recorded in run metadata (see Paper vs code).
Seeds42, 1337, 2024§6.1 exp_036_deepseek_v32_v5_seeds5.yamlEight models run under three seeds. Claude Sonnet 4 has no seed and gets two samples §6.1 §6.2 sources.yaml.
System promptv3§6.1 v3.md agent.pyHeld fixed across models. The default in the agent module is also v3.
Numeric tolerance±2 percentage points§5 tab. 4 scoring.pyPer ground-truth entry; default when not set.
Numeric search window120 characters after the label§5 scoring.py
Keyword matchcase-insensitive substring§5 scoring.py
Per-case cost budgetset per case, 0.05 to 0.50 USD in v5§5 tab. 4, note c benchmark_v5.jsonThe paper says the default is 0.10 (see Pitfalls).
Per-case round budget2 to 16 in v5benchmark_v5.jsonSome budgets are above the loop cap of 10.
Bootstrap95% CI, 2,000 resamples over 93 cases§6.2Applied to the seven original v5 models.
Carbon factors0.40 Wh per 1,000 tokens; 400 gCO2 per kWh§6.3Order-of-magnitude estimate.
Price, DeepSeek V3.2 (Vertex)0.27 in, 1.10 out USD per 1M tokensmetrics.pyUsed for the cost axis.
Price, Claude Sonnet 43.00 in, 15.00 out USD per 1M tokensmetrics.py
Price fallback3.00 in, 15.00 out USD per 1M tokensmetrics.pyApplies to any model not in the price table (see Paper vs code).

Results

Author-reported. Setup: 103 tasks (93 v5 main suite plus 10 v6 comparison tasks), nine LLMs, temperature 1.0, v3 prompt, up to ten tool rounds, and three seeds for eight models or two samples for Claude Sonnet 4 §6.1. Infrastructure is shared across models, but some serving stacks are managed by providers and their quantization and region are not disclosed §6.1 tab. 5.

Leaderboard, combined and main-suite accuracy with per-seed standard deviation tab. 6:

#ModelAll 103 (%)Main 93 (%)Cost per seed (USD)USD per case
1Claude Sonnet 4 (2 samples)61.7 ± 0.760.8 ± 0.811.840.127
2DeepSeek V3.257.9 ± 3.956.3 ± 3.11.050.011
3GLM-552.4 ± 2.650.2 ± 2.23.580.038
4Gemini 2.5 Pro49.2 ± 3.948.0 ± 3.34.870.052
5Qwen3-235B44.3 ± 5.041.2 ± 4.30.890.010
6GPT-4o43.7 ± 2.641.6 ± 2.76.490.070
7GPT-OSS-120B32.7 ± 0.634.1 ± 1.28.270.089
8Llama 4 Scout25.9 ± 3.126.9 ± 2.10.320.003
9Gemma-3-27B14.6 ± 1.015.8 ± 1.65.810.062

Key findings, as stated by the authors §6.2 §6.3 fig. 2:

  • Claude Sonnet 4 leads on capability. DeepSeek V3.2 is second, 4.5 points behind, and the authors say that gap is within noise (Welch t = 2.40, df ≈ 2, p ≈ 0.13) §6.2.
  • Strict accuracy stays under 65% for every model, and just two clear 55% §6.2.
  • Under strict scoring the best model sits 24 to 36 points below the 85% to 97% reported on general GIS benchmarks. Partial credit for the top four models is comparable to those benchmarks, so the authors attribute much of the gap to scoring strictness rather than task difficulty alone §6.2 §2.4 [abstract].
  • Per-check partial credit is 71% to 90% across the nine models, and 86% to 90% for the top four §6.2.
  • Looking only at open-weight models whose size is published (five of them, 27B to 671B), accuracy climbs with size (Spearman ρ = 1.0). The authors note this is a small-sample trend, not a scaling law §6.2.
  • On cost, the frontier starts at Scout, the cheapest model, and ends at Claude Sonnet 4. Of its four members, three carry open weights. DeepSeek V3.2 delivers 93% of Claude's accuracy at 11.3× lower per-case cost §6.3 fig. 2.
  • Estimated carbon per case is about 6.4 g CO2 for both Claude Sonnet 4 and DeepSeek V3.2. The authors say footprint follows tokens rather than accuracy §6.3 tab. 7.

Breakdown of the 1,429 failed runs across the nine models §6.4.2 tab. 8:

Failure mode% of failures
Expected tool never called68.0
Required substring absent from answer19.3
Wrong number reported9.7
Round budget overrun1.3
Banned substring present1.3
No chart produced0.3

v6 expansion, ten comparison-heavy tasks over three seeds, with the 103-task combined metric §6.5 tab. 9. Capability scoring excuses round and cost exhaustion; raw scoring does not.

ModelCapabilityRawComparison cap/raw (%)103-task combined
Qwen3-235B73%37%78/2244.3%
GLM-573%37%83/3352.4%
DeepSeek V3.273%27%83/1757.9%
Claude Sonnet 473%20%83/061.7%
GPT-4o63%50%72/5043.7%
Gemini 2.5 Pro60%60%67/6749.2%
GPT-OSS-120B20%20%11/1132.7%
Llama 4 Scout17%13%11/625.9%
Gemma-3-27B3%3%0/014.6%

Other results, as stated by the authors:

  • Tool selection is the largest category (21 tasks). No model exceeds 37% on it §6.6.
  • The v6 finding is that comparison failures are mostly budget exhaustion during repeated area calls, not near-equal number discrimination. The control task with a 45-point gap was no easier §6.5 fig. 3.
  • Prompt sensitivity, single seed: switching from the harness's default prompt (v2) to v3 raised Claude Haiku 4.5 by 22.6 points and Gemini 2.5 Pro by 11.8 points §6.7.
  • A second dataset (Portuguese land cover, BigEarthNet V2) was added with the same tool, scoring and protocol. On its subset, Claude Sonnet 4 scored 100% on habitat cases and 94% on cross-indicator cases §7.

Paper vs code

Code: https://github.com/gabrielireland/GeoNatureAgent_Benchmark (Apache-2.0), commit 88daf4d299f68688d5153f1d6a1e9530ffba17bd. The repository is linked from the paper's abstract, and its owner matches the first author's name.

Checked and consistent:

  • The leaderboard values in tab. 6 and tab. 9 match the committed CSVs paper/final_results/leaderboard.csv and leaderboard_v5plus6.csv.
  • Capability excludes the cost budget; the v5 leaderboard excuses cost only, and the v6 analysis excuses rounds and cost compile_final_results.py aggregate_v6.py.
  • Tool count (16), category count (18 with 21 tool-selection tasks) and the tools-only MCP surface match the code tools.json benchmark_v5.json.
  • The Claude Sonnet 4 variance uses two samples, and the Anthropic client sends no seed llm_client.py.
  • The carbon column reproduces from the committed token totals (Claude about 40,200 tokens per case gives about 6.4 g CO2).

Differences:

  1. Cost fallback for two models. The paper says costs are list prices §6.3. The price table has no entry for openrouter/openai/gpt-4o or openrouter/google/gemma-3-27b-it, so both are priced at the default of 3.00 in and 15.00 out USD per million tokens metrics.py runner.py exp_043_gpt4o_openrouter_v5.yaml. Their per-case costs ($0.070 and $0.062) are therefore not list prices. The statements that GPT-4o and Gemma-3-27B are dominated on the cost frontier depend on those values.
  2. Claude seeds. The experiment file exp_042_claude_sonnet4_v5_seeds5.yaml lists seeds [42, 1337, 2024] and the name says five seeds exp_042_claude_sonnet4_v5_seeds5.yaml. The published runs use two (42 and 1337) sources.yaml. The paper's two-sample description matches the published numbers, but the file does not. Experiment files are in the repository's benchmark/experiments/ folder; this card gives their file names only.
  3. Cost budget default. The paper says per-case budgets default to 0.10 in v5 tab. 4, note c. The code has no default. Each case sets its own budget, from 0.05 to 0.50 USD. Twenty-two cases use 0.10, and V5_21 uses 0.15 benchmark_v5.json scoring.py.
  4. Numeric rule. Table 4 says the check uses "the nearest percentage value". Its note b and the code use the first percentage after the first occurrence of the label, within 120 characters scoring.py. The code follows note b.
  5. Round limit. The paper says max_turns is 10 §6.1. The experiment files record max_turns: 10, but the runner does not read it; the run metadata only stores it config.py run_meta.py. The loop cap is the constant MAX_TOOL_ROUNDS = 10 in the agent module agent.py. Case round budgets run up to 16, so some budgets can never be reached.
  6. Tool-call parser scope. The paper credits a dedicated AST parser to Llama 4 §3. The code applies the <|python_start|> parser to any model's text when no structured tool call is returned, including GPT-OSS-120B llm_client.py. The Gemma parser drops calls whose names do not match a known tool, and it sets a call's arguments to empty whenever they fail to parse as literals llm_client.py.
  7. Tool name. Table 2 lists select_spatial_rel. The code's tool is select_features_by_spatial_relationship tools.json. Section 4 uses the long name.
  8. MCP wrapper. The paper describes a thin wrapper of about 95 lines that reuses the dispatcher unchanged §3. The file is 136 lines and keeps per-process session state: after a lookup, it injects the last looked-up geometry into analyze_area calls with an empty area, to mirror the agent loop mcp_server.py.
  9. Failure table source. Table 8 covers nine models §6.4.2. The committed paper/final_results/per_case.csv covers seven (no GPT-4o or Gemma-3 rows). For those seven, it has 1,031 failed of 1,860 rows, and 665 of the failures (64.5%) are tool-missing. The 68.0% and 1,429 in the paper cannot be recomputed from the committed per-case file. I did not check the claim about eleven cases unsolved by every model.
  10. Per-run flag. The scorer's per-run passed includes the cost check, and latency when set scoring.py. Capability is recomputed without cost compile_final_results.py. The paper's eight-check table omits the optional latency check scoring.py.
  11. v6 numbers. For v6, the ground truth is free text, so the numeric check is skipped and those tasks are scored on tools and keywords only scoring.py.
  12. Repeated seeds. The Claude sample standard deviation reproduces from the two per-seed values (n − 1 denominator). The Welch statistics in §6.2 also recompute from the per-seed values in the leaderboard file, with t = 2.40. I did not recompute the p-value.

Pitfalls

  • Strict pass is all-or-nothing. One missing keyword fails a task. Keyword failures are about one fifth of all failures in the paper's table, so expect many near-misses that partial credit would score as mostly correct §5 tab. 8.
  • Accent-sensitive labels. V5_21 expects "Cordoba" in both the ground truth and the required keywords benchmark_v5.json. An answer that writes "Córdoba" fails both checks, because matching is by plain substring with no accent folding.
  • Numeric check reads the first match only. The label is searched at its first occurrence, and the first percentage in the next 120 characters is used. Another label, or a number in a different sentence, can be picked up.
  • Loop cap binds before some budgets. The loop stops after 10 tool rounds, but case budgets go up to 16 agent.py benchmark_v5.json. A round-budget failure is then not a loop-length signal.
  • Harness parsers change results. The paper reports that extending the Gemma parser roughly tripled its accuracy §6.2. An implementation that lacks the text-based tool-call fallback will score such models much lower.
  • MCP state. The analyze_area injection depends on earlier lookup calls in the same server process. Without it, a call with an empty area fails mcp_server.py.
  • Costs are token-price estimates. The paper says costs are list prices, but they come from a hand-maintained table metrics.py and fall back to Sonnet rates for unknown models. Recompute with current prices before any cost claim.
  • Managed serving. Vertex-hosted models run on provider stacks whose quantization and region are undisclosed, so cost and latency comparisons carry that confound §6.1 tab. 5.
  • Two samples for Claude. Its spread rests on two numbers and is a weak variance estimate. The authors say they did not draw a third sample, because the two already agree closely and that run is the most expensive tab. 6, note.
  • Five-model scale trend. The Spearman ρ = 1.0 rests on five points. The exact two-sided permutation p for five points with perfect rank order is 2/120 ≈ 0.017, as the authors state, but the sample is small and the models differ in architecture and training §6.2.
  • Prompt matters. Moving from prompt v2 to v3 changed single-seed scores by up to 22.6 points. Any reuse must hold the prompt fixed and version it §6.7.
  • Dataset scope. Indicators cover Spain and Portugal only, and the BigEarthNet layer is served by one tool, so multi-step composition is needed for comparisons there §4 §7.
  • Tool recall, not precision. Extra tool calls do not fail a task; a missing expected tool does §5 scoring.py.

Lineage

  • Builds on: the authors' earlier four-page SIGSPATIAL 2026 poster on the same benchmark. That poster is not in this wiki §1.
  • Compared against: GeoBenchX (the closest structured tool-calling benchmark, with four of its GIS operations adapted into GNA), ThinkGeo, UnivEARTH, MapEval, MapQA, GPSBench, SpatiaLab, GEO-Bench and GEO-Bench-2, plus multi-agent systems such as GeoJSON Agents and GeoLLM-Squad §2 §4. None has a card in this wiki, so no content from them is used.
  • Methods referenced: the Model Context Protocol and ReAct §3; HELM and GAIA for separating capability from cost §5.
  • Superseded by: not stated.

Source and license

Adapted from “GeoNatureAgent (GNA): A Framework and Benchmark for Pre-Production Evaluation of Tool-Using Agents on Geospatial and Environmental Tasks” by Gabriel Diaz-Ireland, Diego Prieto-Herráez, Mario García Peces, Javier Velázquez, Benjamin Zaitchik, Devika Jain (arXiv:2610.09112), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/gabrielireland/GeoNatureAgent_Benchmark at commit 88daf4d299f6 (Apache-2.0). Any quoted code is shown under that license.