cs.CVOct 13, 2026

The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement

Authors: Sijie Zhao, Zeqi Zhou, Heli Qi, Xuanchao Fu, Zhuo Zheng, Junshi Xia, Junjue Wang, Weihao Xuan, 

Organizations: The University of Tokyo · RIKEN · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a benchmark. We show that this loop can be handed to an agent. Any verifiable task can be cast as a harness and a verifier, where the harness defines what the agent may use and the verifier defines what counts as success. Every EO task can be made verifiable, through labeled benchmarks, independent sensors, physical models or the repeated observation of the same place. Given only these two components, a general-purpose coding agent builds the EO model and improves it recursively on its own outputs. Settings long treated as separate research problems, from learning with a handful of labels to adapting across domains, become choices of harness and verifier. On six EO benchmarks spanning optical, SAR and multispectral time-series imagery, with outputs that range from land-cover maps and ship instances to canopy heights and answers about large images, agent-built models surpass the state of the art on every one, and on land-cover mapping, they do so from 11% of the training labels. This establishes a new paradigm for EO, in which people design harnesses and verifiers and agents build the models, so EO models improve as agents do.

Figures & tables

BenchmarkTaskInputClassesMetricTrainVerifierTestFew-label task
OpenEarthMapland-cover mappingRGB, 0.25–0.5 m8mIoU2,7003005006 tiles, one per continent (0.2%)
BRIGHTbuilding damageoptical + SAR pairs4mIoU2,91845087814 pairs, one per event (0.5%)
HRSIDship instancesSAR, 0.5–3 m1mask AP3,0905521,96236 images from 36 scenes (1.2%)
PASTIScrop-type mappingSentinel-2 time series19mIoU1,45548249620 series, five per tile (1.4%)
Open-Canopycanopy heightSPOT 6–7 RGB + NIR, 1.5 m–MAE (m)66,3397,36913,6751,000 tiles, spread over France (1.5%)
LRS-GROlarge-image VQA, training-freeRGB, up to 27,328 px, + question–accuracy2,6568559,734114 answers, six per question type (4.3%)
Table 1: The six benchmarks. Training data form the harness, the verifier holds a validation split, and the test split stays hidden until the final evaluation. The few-label task keeps a few labeled samples per natural stratum and moves the other training samples to the unlabeled pool. On Open-Canopy, it keeps a geographically spread subset instead. Parentheses give the share of the training labels. BRIGHT and PASTIS count the background as a class. LRS-GRO counts questions and forbids any training, so its labels serve only as examples and for local validation.
(a) BRIGHT building damage, pre-event optical + post-event SAR
MethodmIoU
SiamAttnUNet ( Adriano et al., 2021 )64.26
DeepLabV3+ ( Chen et al., 2018 )64.80
UNet ( Ronneberger et al., 2015 )64.94
SiamCRNN ( Chen et al., 2020 )65.73
ChangeOS ( Zheng et al., 2021 )65.98
Table 2: Agent-built models against published methods on the six benchmarks, each under its standard protocol. Rows are sorted from the weakest to the strongest result and the previous state of the art (SOTA) is underlined. Full-label RSI4EO rows give the gain over the SOTA, and few-label RSI4EO rows give the share of the training set in parentheses and, except for the MAE of Open-Canopy, the score as a percentage of the full-label score. Figure 2 and Table 4 give the few-label results of all six benchmarks. All agent rows come from GPT-6-Astra through Codex, within 12 hours on four GB200 GPUs. † Test-time augmentation. ‡ Ensemble of ChangeOS, DamageFormer and ChangeMamba with test-time augmentation and object-based voting. § Pretrained foundation model fine-tuned on PASTIS, as reported by Herzog et al. (2026) . ∥ Fold-I score from the released test logs of Cai (2023) , whose paper reports five-fold means of 66.8 and 67.9. PASTIS uses the official fold I split. The other baselines are those of Tarasiou et al. (2023) , who evaluate on 24×2424\times 24 crops of the same test fold. On HRSID, the four generic detectors are those of Wei et al. (2020) , and LFG-Net ⋆ is the stronger of the two configurations reported by its authors. OpenEarthMap reports 8-class mIoU on the official validation split, which serves as the test split because test labels are withheld. ∗ 8-class mean recomputed from the per-class IoUs of Wang et al. (2024) and Wang et al. (2025) . ¶ Encoder evaluated by Duc and Fukui (2026) . Open-Canopy reports the mean absolute error (MAE) of canopy height in meters on vegetated pixels below 60 m, where lower is better, with the baselines and the original LiDAR labels of Fogel et al. (2025) . LRS-GRO reports question-level accuracy on the 9,734 test questions under the official judge, with the baselines as reported by Zhang et al. (2026b) . ∘ Training-free, built on Qwen3-VL-8B.
Figure 2: Strong models from few labels. (a) The share of the training labels that the few-label task of each benchmark keeps (grey) and its score as a share of the full-label score (blue). For Open-Canopy, where a lower MAE is better, the score share is the full-label MAE divided by the few-label MAE. (b) Hidden-test mIoU on OpenEarthMap as the harness holds 6, 300, 600 or all 2,700 labeled tiles, which are 0.2, 11, 22 and 100% of the training set, on a log scale. The 300 and 600 tiles are selected without labels by k-center on frozen features and come without an unlabeled pool. In (b), the dashed line marks the previous state of the art and the dotted line a UNet, both trained on all labels. Tables 4 and 5 list the scores.
AgentInterfaceVerifierTest mIoU
Starter pipeline––34.0
Qwen3.8-27BClaude Code40.839.5 +5.5
Qwen3.8-MaxClaude Code53.351.9 +17.9
Kimi-K3Claude Code57.355.5 +21.5
Claude-Opus-5.5Claude Code58.156.7 +22.7
GPT-6-AstraCodex61.960.0 +26.0
Table 3: Five agents on six-label OpenEarthMap, each given the same harness, verifier and 12-hour budget. The verifier column gives the best verifier score of each run, and the gain over the starter is measured on the hidden test set.
Figure 3: Lineage of the six-label OpenEarthMap run. (a) In each generation, the best pipeline the verifier has seen labels the unlabeled pool, a student continues from the previous checkpoint on the six labeled tiles and these pseudo-labels, and the verifier decides whether it becomes the next teacher. In the later generations, controller scripts written by the agent select the teacher from the verifier log and start the next generation. (b) Verifier mIoU of the best pipeline after each generation. Generation 0 is the best model that the agent trains on the six tiles alone, the scripts alone run the last generation, and the star marks the hidden-test mIoU of the final three-network ensemble.
Figure 4: Inside the runs. (a) Share of the final verifier gain reached over time in the 14 runs of GPT-6-Astra in Table 6 , where the gain is measured from the first to the best verifier score of a run. A colored dot marks the end of each run, the black line is the median over runs, and the black dots mark the median times to half and 90% of the gain. (b) Hidden-test score against best verifier score, in the metric of each benchmark, for the runs of Table 6 scored in points and for the three further agents of Table 3 . The dashed line marks equal scores, the grey band a difference of at most 2 points, and the grey line joins the five agents on six-label OpenEarthMap from the weakest to the strongest. The two Open-Canopy runs, scored in meters of MAE, are listed in Table 6 . In both panels, blue marks few-label tasks and orange full-label or label-budget tasks.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

  Few labelsFull labels 
BenchmarkMetricStarterRSI4EOStarterRSI4EOPublished
OpenEarthMapmIoU34.060.0 +26.0–71.667.5
BRIGHTmIoU34.849.4 +14.661.871.6 +9.868.9
HRSIDmask AP–50.2–68.763.9
PASTISmIoU14.050.1 +36.253.069.7 +16.767.1
Open-CanopyMAE (m) ↓\downarrow–2.724.002.44 −- 1.562.52
Appendix
Table 4: Hidden-test scores on all six benchmarks. The starter is the plain pipeline that ships with the harness and is run as provided. RSI4EO is the model that GPT-6-Astra builds from the same harness within 12 hours, and its gain over the starter is shown in blue. The last column gives the best published result under the same protocol ( Table 2 ). Lower is better for the MAE of Open-Canopy. A dash marks a score that is not available. ∘ Training-free, no parameter is trained.
Labeled tilesShareUnlabeled tilesGPT-6-AstraClaude-Opus-5.5
60.2%4,19460.056.7
30011%064.367.8
60022%069.468.6
2,700100%1,50071.670.8
Previous SOTA Wang et al. [2025]67.5
Appendix
Table 5: Hidden-test mIoU of two agents on OpenEarthMap as the harness holds more labeled tiles. The 300 and 600 tiles are selected without labels by k-center on frozen features, and these two tasks provide no unlabeled pool. The previous state of the art uses all training labels.
TaskAgentLocal validationVerifierTestGap
With a local validation set
PASTIS, fullGPT-6-Astra160 series, then five folds69.969.7++ 0.2
OpenEarthMap, fullGPT-6-Astra170 tiles72.671.6++ 1.0
OpenEarthMap, fullClaude-Opus-5.5250 tiles72.170.8++ 1.3
OpenEarthMap, 600GPT-6-Astra60 tiles70.669.4++ 1.2
OpenEarthMap, 600Claude-Opus-5.598 tiles70.668.6++ 2.0
Appendix
Table 6: Verifier and hidden-test scores of 18 runs, grouped by whether the agent built a local validation set from its labels. The gap is the best verifier score minus the hidden-test score in points and is computed before rounding. For Open-Canopy, the scores are the MAE in meters and the gap is the test MAE minus the verifier MAE. On full-label BRIGHT, three quarters of the gap come from the destroyed class, which covers 1.38% of the verifier pixels and 0.93% of the test pixels. Its test IoU of 61.5 is below the 65.4 that the first, untuned submission of the run scored on the verifier, so the gap exists before any selection.
More in this topic

Remote Sensing

Remote sensing uses satellite, aerial, and ground-based observations to measure and map Earth and other planetary surfaces.

See all 527 papers

Explore similar work

CardsList
Apr 16, 2025

Towards LLM Agents for Earth Observation

Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. In this work we ask: Are AI systems ready for reliable Earth Observation? To answer this, we introduce UnivEARTH, a coding benchmark of 408 yes/no questions from NASA Earth Observatory articles across 7 various topics and over 15 satellite instruments and sources. Using Google Earth Engine API as a tool in a zero-shot setup, LLM agents achieve an accuracy of 40.0% where the code fails to run over 44% of the time. To better understand LLM agent behavior, we also analyze the impact of using the JavaScript API versus Python and the effect of providing documentation. Furthermore, we find that using a Reflexion framework significantly reduces errors: Claude-4.5-Sonnet, Gemini-2.5-Pro, and GPT-5 accuracies rise to around 60%. However, these results remain only marginally above random chance. Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward.

Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10

Cornell University · Columbia University

Code GenerationLLM Agent EvaluationRemote SensingAI Agent BenchmarksLLM AgentsEarth Science
AskPDF
May 2, 2026

EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents

Earth Observation (EO) analysis is inherently interactive: resolving uncertainty often requires expanding the region of interest, retrieving historical observations, and switching across sensors such as optical and Synthetic Aperture Radar. However, most EO benchmarks collapse this process into fixed-input, single-turn tasks. To address this gap, we present EO-Gym, a controlled executable framework for multimodal, tool-using EO agents that formulates EO analysis as a Gymnasium-style local geospatial workspace backed by more than 660k multimodal files indexed by location, time, and sensor type, with 35 EO-specialized tools spanning six task families. Built on this environment, we construct EO-Gym-Data, a benchmark of 9,078 trajectories and 34,604 reasoning steps, and grounded in eight public EO datasets together with Landsat and Sentinel-2 imagery. Evaluating 1010 open and closed VLMs shows that strong general-purpose models still struggle with interactive EO reasoning, especially on temporal and cross-modal workflows. As a reference baseline, EO-Gym-4B, obtained by fine-tuning Qwen3-VL-4B-Instruct on EO-Gym-Data, improves overall Pass@3 from 0.490.49 to 0.740.74 under the main evaluation setting. O-Gym provides a reproducible environment for interactive EO agents, operationalizing EO as an evidence-gathering problem that requires planning across geospatial, temporal, and sensing modality.

Sai Ma, Zhuang Li, Sichao Li +4

The Australian National University · Royal Melbourne Institute of Technology · University of Sydney +1

Spatiotemporal ReasoningRemote SensingGeospatial ReasoningRemote Sensing BenchmarksAgent BenchmarksMultimodal Tool Use
AskPDF
Sep 11, 2026

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents

Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare data, perform domain computations, and derive conclusions from runtime evidence. Existing EO agents typically start from supplied observations, while benchmarks typically provide prepared inputs or candidate answers, leaving full-chain open-world EO execution largely untested. We present Earth-Agent-Pro, an execution-adaptive Plan-and-Execute framework using expert-authored skills to constrain planning and runtime tool use. Workflow-centered structured memory records planned steps, accepted evidence, and their dependencies, enabling repair of only the affected workflow suffix when runtime evidence invalidates a step. Separate large language model adapters use sequence-level supervised fine-tuning for planner workflow composition and node-level group relative policy optimization with locally verifiable rewards for executor tool-argument grounding. Earth-Bench-Pro instantiates 248 expert-curated task cores as 744 questions under three matched regimes. Its 248 Open-World Execution questions span RGB imagery, spectral observations, and remote sensing products, pairing high-level requests with runtime data requirements, executable trajectories, and open-ended answers grounded in execution evidence. With a shared GPT-5 backbone, Earth-Agent-Pro achieves 66.13% LLM-as-Judge accuracy, exceeding ReAct by 20.95 points in this metric and 24.44 points in Tools-In-Order. Joint adapter tuning raises Qwen3.5-9B LLM-as-Judge accuracy from 38.31% to 50.00%, an 11.69-point gain over the untuned configuration. Planning-only evaluation and execution with the reference workflow show that the adapters improve workflow composition and argument grounding, respectively. Code and datasets will be released soon.

Zhutao Lv, Chenhao Dang, Yi Feng +5

Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University +3

Remote SensingAI Agent BenchmarksTool-Using AgentsLLM Agent Planning
AskPDF