OpenEarth-Agent: From Tool Calling to Tool Creation for Open-Environment Earth Observation

Sijie Zhao; Feng Liu; Xueliang Zhang; Hao Chen; Xinyu Gu; Zhe Jiang; Fenghua Ling; Ben Fei

cs.CVMar 23, 2026

OpenEarth-Agent: From Tool Calling to Tool Creation for Open-Environment Earth Observation

Authors: Sijie Zhao, Feng Liu, Xueliang Zhang, Hao Chen, Xinyu Gu, Zhe Jiang, Fenghua Ling, Ben Fei,

Organizations: Nanjing University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · The University of Tokyo

Abstract

Earth Observation (EO) is essential for perceiving dynamic land surface changes, yet deploying autonomous EO in open environments is hindered by the immense diversity of multi-source data and heterogeneous tasks. While remote sensing agents have emerged to streamline EO workflows, existing tool-calling agents are confined to closed environments. They rely on pre-defined tools and are restricted to narrow scope, limiting their generalization to the diverse data and tasks. To overcome these limitations, we introduce OpenEarth-Agent, the first tool-creation agent framework tailored for open-environment EO. Rather than calling predefined tools, OpenEarth-Agent employs adaptive workflow planning and tool creation to generalize to unseen data and tasks. This adaptability is bolstered by an open-ended integration of multi-stage tools and cross-domain knowledge bases, enabling robust execution in the entire EO pipeline across multiple application domains. To comprehensively evaluate EO agents in open environments, we propose OpenEarth-Bench, a novel benchmark comprising 596 real-world, full-pipeline cases across seven application domains, explicitly designed to assess agents' adaptive planning and tool creation capabilities. Only essential pre-trained model tools are provided in this benchmark, devoid of any other predefined task-specific tools. Extensive experiments demonstrate that OpenEarth-Agent successfully masters full-pipeline EO across multiple domains in the open environment. Notably, on the cross-benchmark Earth-Bench, our tool-creating agent equipped with 6 essential pre-trained models achieves performance comparable to tool-calling agents relying on 104 specialized tools, and significantly outperforms them when provided with the complete toolset. In several cases, the created tools exhibit superior robustness to data anomalies compared to human-engineered counterparts.

Figures & tables

Figure 1: Overview of existing limitations and proposed solutions. Existing EO agents are failing to generalize to unseen data and tasks due to calling predefined tools, and are restricted to specific EO stages and application domains. In contrast, the proposed OpenEarth-Agent generalize to diverse data and tasks by creating adaptive tools, achieving full EO workflow and cross-domain coverage through open-ended tool and knowledge integration.

Figure 2: Overview of the OpenEarth-Agent. The OpenEarth-Agent performs real-time perception of multi-source EO data, followed by adaptive workflow planning and structuring based on the task and data context. It then proceeds with adaptive tool creation and execution, incorporating real-time inspection and feedback on the execution process and outcomes to generate reasonable results.

Figure 3: Overview of the OpenEarth-Bench. This benchmark comprises 596 real-world application cases spanning seven domains. Each case encompasses the entire workflow, ranging from data preparation and feature extraction to geospatial analysis.

Benchmark	Data Preparation	Feature Extraction	Geospatial Analysis	Open Environment	Data Modalities
ThinkGeo Shabbir et al. (2025)	✗	✓	✗	✗	RGB
Earth-Bench Feng et al. (2025)	✗	✓	✓	✗	RGB, Multi-Spectral, Product
GeoPlan-Bench Li et al. (2025)	✓	✓	✓	✗	-
Cangling-Know. Chen et al. (2025)	✗	✓	✓	✗	RGB, Multi-Spectral, SAR
OpenEarth-Bench (Ours)	✓	✓	✓	✓	RGB, Multi-Spectral, SAR, NTL, Product

Table 1: Comparison between OpenEarth-Bench and existing remote sensing agent benchmarks.

Figure 4: Results of OpenEarth-Agent based on different LLMs in Stage-Wise evaluation. The closer to the upper-left corner, the better.

LLM	Stage-Wise			End-to-End
LLM	Data Preparation	Feature Extraction	Geospatial Analysis	Data Preparation	Feature Extraction	Geospatial Analysis
GPT-5	85.40	85.27	76.66	82.38	74.83	58.72
Gemini-2.5-Flash	76.17	77.52	64.76	74.16	61.58	45.47
Kimik2	77.85	79.19	66.11	75.84	64.09	47.81
DeepSeek-V3.1	75.17	78.02	63.08	72.82	60.74	43.12
Qwen3-Max	71.31	73.82	60.57	69.79	56.38	39.26

Table 2: Accuracy comparison of OpenEarth-Agent powered by different LLMs on OpenEarth-Bench.

LLM	Earth-Agent	OpenEarth-Agent
LLM	Earth-Agent	6 Tools	Full Tools
GPT-5	63.16	59.92	67.61
Gemini-2.5-Flash	55.06	53.44	57.89
Kimik2	56.68	52.63	59.11
DeepSeek-V3.1	52.23	51.02	56.68
Qwen3-Max	47.37	46.97	52.63

Table 3: Accuracy comparison on Earth-Bench across different LLM backbones.

Data Summary	Plan & Workflow	Result Check	Stage-Wise			End-to-End
Data Summary	Plan & Workflow	Result Check	Data Preparation	Feature Extraction	Geospatial Analysis	Data Preparation	Feature Extraction	Geospatial Analysis
✗	✓	✓	84.56	77.85	67.62	81.71	74.16	57.88
✓	✗	✓	83.72	82.21	71.81	81.54	69.13	50.67
✓	✓	✗	78.02	79.86	65.77	75.84	59.73	34.73
✓	✓	✓	85.40	85.27	76.66	82.38	74.83	58.72

Table 4: Ablation study on the multi-agent architecture.

Knowledge Integration	Tool Integration	Stage-Wise			End-to-End
Knowledge Integration	Tool Integration	Data Preparation	Feature Extraction	Geospatial Analysis	Data Preparation	Feature Extraction	Geospatial Analysis
✗	✗	79.69	81.21	72.48	78.02	66.78	47.65
✗	✓	80.87	84.40	73.82	78.86	69.46	51.34
✓	✗	84.56	82.04	74.16	81.20	70.80	52.68
✓	✓	85.40	85.27	76.66	82.38	74.83	58.72

Table 5: Ablation study on open knowledge and tool integration.

LLM Agent Tool Generation

See all 11 papers

Sep 11, 2026

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents

Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare data, perform domain computations, and derive conclusions from runtime evidence. Existing EO agents typically start from supplied observations, while benchmarks typically provide prepared inputs or candidate answers, leaving full-chain open-world EO execution largely untested. We present Earth-Agent-Pro, an execution-adaptive Plan-and-Execute framework using expert-authored skills to constrain planning and runtime tool use. Workflow-centered structured memory records planned steps, accepted evidence, and their dependencies, enabling repair of only the affected workflow suffix when runtime evidence invalidates a step. Separate large language model adapters use sequence-level supervised fine-tuning for planner workflow composition and node-level group relative policy optimization with locally verifiable rewards for executor tool-argument grounding. Earth-Bench-Pro instantiates 248 expert-curated task cores as 744 questions under three matched regimes. Its 248 Open-World Execution questions span RGB imagery, spectral observations, and remote sensing products, pairing high-level requests with runtime data requirements, executable trajectories, and open-ended answers grounded in execution evidence. With a shared GPT-5 backbone, Earth-Agent-Pro achieves 66.13% LLM-as-Judge accuracy, exceeding ReAct by 20.95 points in this metric and 24.44 points in Tools-In-Order. Joint adapter tuning raises Qwen3.5-9B LLM-as-Judge accuracy from 38.31% to 50.00%, an 11.69-point gain over the untuned configuration. Planning-only evaluation and execution with the reference workflow show that the adapters improve workflow composition and argument grounding, respectively. Code and datasets will be released soon.

Zhutao Lv, Chenhao Dang, Yi Feng +5

Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University +3

Remote Sensing AI Agent Benchmarks Tool-Using Agents LLM Agent Planning

Ask PDF

May 2, 2026

EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents

Earth Observation (EO) analysis is inherently interactive: resolving uncertainty often requires expanding the region of interest, retrieving historical observations, and switching across sensors such as optical and Synthetic Aperture Radar. However, most EO benchmarks collapse this process into fixed-input, single-turn tasks. To address this gap, we present EO-Gym, a controlled executable framework for multimodal, tool-using EO agents that formulates EO analysis as a Gymnasium-style local geospatial workspace backed by more than 660k multimodal files indexed by location, time, and sensor type, with 35 EO-specialized tools spanning six task families. Built on this environment, we construct EO-Gym-Data, a benchmark of 9,078 trajectories and 34,604 reasoning steps, and grounded in eight public EO datasets together with Landsat and Sentinel-2 imagery. Evaluating

10

open and closed VLMs shows that strong general-purpose models still struggle with interactive EO reasoning, especially on temporal and cross-modal workflows. As a reference baseline, EO-Gym-4B, obtained by fine-tuning Qwen3-VL-4B-Instruct on EO-Gym-Data, improves overall Pass@3 from

0.49

0.74

under the main evaluation setting. O-Gym provides a reproducible environment for interactive EO agents, operationalizing EO as an evidence-gathering problem that requires planning across geospatial, temporal, and sensing modality.

Sai Ma, Zhuang Li, Sichao Li +4

The Australian National University · Royal Melbourne Institute of Technology · University of Sydney +1

Spatiotemporal Reasoning Remote Sensing Geospatial Reasoning Remote Sensing Benchmarks Agent Benchmarks Multimodal Tool Use

Ask PDF

Feb 19, 2026

OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

Recent progress in multimodal reasoning has enabled agents that interpret imagery, connect it with language, and execute structured analytical tasks. Extending these capabilities to remote sensing remains challenging, as models must reason over spatial scale, geographic structures, and multispectral indices while maintaining coherent multi-step logic. To address this gap, we introduce OpenEarthAgent, a unified framework for tool-augmented geospatial reasoning trained on satellite imagery, natural-language queries, and structured reasoning traces. Beyond serving as a benchmark, OpenEarthAgent establishes a cohesive agentic architecture built around a unified executable tool registry and trajectory-based policy learning. The framework standardizes heterogeneous visual, spectral, GIS, and georeferenced raster operations under a consistent callable schema, enabling modular orchestration and deterministic execution. Training is performed via supervised fine-tuning on structured reasoning trajectories with deterministic replay validation to ensure executability and spatial correctness. The accompanying corpus comprises 14,538 training and 1,169 evaluation instances with over 107K reasoning steps, spanning urban, environmental, disaster, and infrastructure domains and incorporating GIS operations alongside index analyses such as NDVI, NBR, and NDBI. Grounded in explicit reasoning traces, the learned agent demonstrates structured reasoning, stable spatial understanding, and interpretable tool-driven behaviour across diverse EO scenarios. We report consistent improvements over a strong baseline and competitive performance against recent open and closed-source models. Our code, data and trained models are publicly available: https://github.com/mbzuai-oryx/OpenEarthAgent

Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir +8

Mohamed bin Zayed University of AI, United Arab Emirates · IBM Research, Europe · Linköping University, Sweden +2

Geospatial Reasoning Benchmarks Remote Sensing Geospatial Reasoning Tool-Using Vision-Language Agents Visually Grounded Reasoning Spatial Reasoning

Ask PDF

Abstract

LLM Agent Tool Generation

Explore similar work

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents

EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents

OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents