Organizations: School of Computer Science and Technology, Tianjin University, Tianjin 300350, China · Key Laboratory of Target Cognition and Application Technology (TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China
Remote sensing agents rely on perception, measurement, and raster analysis tools to solve Earth observation tasks. We refer to their judgments and quantitative results about ground objects as tool observations. However, these observations are subject to uncertainty and may be incorrect even when tools execute successfully. When agents accept incorrect observations, the errors can propagate through subsequent reasoning and cause task failure. We analyze 1,229 execution trajectories across three remote sensing agent benchmarks. On each benchmark, at least 88.1 percent of tasks depend on tool observations. Among these tasks, at least 22.7 percent contain incorrect observations despite successful tool execution. These errors propagate to the final answer in at least 82.0 percent of affected tasks on each benchmark. To address this problem, we propose RSure-Agent, a framework for verifying tool observations and limiting error propagation. We introduce a verifiable observation protocol that requires tools to return process evidence for the agent to verify their observations. We construct a task-tool reliability prior from offline task feedback. The prior summarizes each tool configuration's past performance across task types and provides a task-specific reference for verification. Using process evidence and this prior, RSure-Agent decides whether to accept an observation, request additional evidence, or reject it. We evaluate RSure-Agent on EarthBench, ThinkGeo, TerraLogic, and CHOICE-420. Across three agent benchmarks, RSure-Agent reduces the error propagation rate by 21.3 to 25.9 percentage points relative to the base configuration with verification and the prior disabled. On CHOICE-420, it improves overall accuracy over direct answering by 5.71 percentage points on average across 11 backbone models. On EarthBench, it reduces the tool-call ratio by 25.9 percent relative to Earth-Agent.
Figures & tables
Figure 1: Illustration of RSure-Agent. (a) The tool executes without runtime errors, but the segmentation mask includes non-vehicle pixels, yielding an erroneous area of 506.626m2 . (b) Through the verifiable observation protocol, RSure-Agent inspects the source image, mask, and execution log. After deterministic checks pass, the verifier detects that the mask covers a swimming pool, refuses the faulty observation, and prompts mask refinement. Following SAM 2.1 refinement, the recomputed area is 34.170m2 (within the accepted range of 29.40 – 35.94m2 ). The task–tool reliability prior provides a historical reference during verification.
Benchmark
Agent
Backbone
Rdep
Rerr
Rprop
EarthBench
Earth-Agent
GPT-5
88.7
22.7
82.0
ThinkGeo
ThinkGeo-Agent
GPT-4o
91.7
41.3
91.5
TerraLogic
HieraPlan
GPT-4o
88.1
39.8
93.2
Table 1: Tool observation statistics (%) of the benchmark reference agents.
Figure 2: Examples of incorrect tool observations under normal execution across (a) detection, (b) segmentation, (c) counting, and (d) image quality.
Figure 3: Overall framework of RSure-Agent. The verifiable observation protocol requires tools to return execution status, observations, and process evidence (source evidence, intermediate results, and validity evidence). Online verification applies deterministic rule checks, then verifies observations that pass using the image, task, process evidence, and task–tool reliability prior. The verifier decides to accept, request evidence, or reject. Accepted observations and evidence enter the verified history. The backbone model uses feedback to plan subsequent tool calls and generates the final answer from this history after interaction ends.
Evidence type
perceive
measure
execute
Source evidence Etsrc
Detector weights Thresholds Sensor Resolution
Input layer sources Metric Unit specification
Operator parameters Product formulas Band mapping
Intermediate results Etmid
Object bounding boxes Confidence scores Number of filtered candidates
Measurement numerator Denominator Pixel counts
Per-node outputs Valid-pixel counts
Validity evidence Etval
Layer status Class completeness Geometry flags
Layer status Class coverage
Node status Quality masks List of failed nodes
Table 2: Process evidence returned by the three observation tools.
Task type
Tool
Configuration
Level
r^
Object Counting
perceive
objects
Coarse
0.926
Object Counting
measure
count|auto|objects
Fine
0.975
Object Presence
perceive
objects
Coarse
0.800
Image Caption
perceive
evidence
Coarse
0.923
Change Detection
perceive
change
Coarse
0.321
Env. Assessment
execute
raster.product
Coarse
0.810
Table 3: Selected entries of the task–tool reliability prior.
Figure 4: Representative remote sensing inputs across the benchmarks, including optical, SAR, infrared, thermal, and multi-band imagery.
Method
Backbone
Weights
ThinkGeo
EarthBench
TerraLogic
Ans.
AnsI
Acc.
AnsAcc
ThinkGeo-Agent
GPT-4o
Closed
9.8
20.4
–
–
HieraPlan
GPT-4o
Closed
–
–
–
25.7
Earth-Agent
GPT-4o
Closed
–
–
45.3
–
Earth-Agent
GPT-5
Closed
–
–
65.6
–
OpenEarth-Agent
GPT-5
Closed
–
–
67.6
–
Table 4: Task performance (%) on RS agent benchmarks. † denotes results following RS-Claw’s test split on EarthBench.
Backbone
Weights
Overall
Perception
Reasoning
Origin
RSure-Agent
Origin
RSure-Agent
Origin
RSure-Agent
Mistral Small 4
Open
75.71
84.05 (+8.34)
76.07
86.07 (+10.00)
75.00
80.00 (+5.00)
Qwen3.5-27B
74.05
83.33 (+9.28)
78.57
86.79 (+8.22)
65.00
76.43 (+11.43)
Qwen3.5-122B
69.29
82.86 (+13.57)
72.50
86.79 (+14.29)
62.86
75.00 (+12.14)
Qwen3.5-397B
83.81
85.95 (+2.14)
85.36
87.50 (+2.14)
80.71
82.86 (+2.15)
Qwen3.6-27B
79.76
85.24 (+5.48)
83.93
88.21 (+4.28)
71.43
79.29 (+7.86)
Table 5: Accuracy (%) on CHOICE-420: direct answering (Origin) versus RSure-Agent. Parentheses show gains in percentage points. Bold marks the best value.
Verification
Prior
ThinkGeo
EarthBench
TerraLogic
Ans.
AnsI
Acc.
AnsAcc
✗
✗
21.1
24.8
64.9
26.4
✓
✗
22.5
29.5
66.9
29.7
✓
✓
23.6
32.6
68.1
31.4
Table 6: Ablation study (%, GPT-5 backbone).
Method
Acc. (%)
LLM calls
Token usage
Time per question (s)
Mean
Median
RSure-Base
64.9
7.117
90,454
460.37
321.99
RSure-Base + CRITIC
66.9
8.871
148,562
610.45
505.76
RSure-Base + Tools Fail
65.3
9.935
125,322
557.30
458.28
RSure-Agent
68.1
8.367
107,860
477.08
334.46
Table 7: Performance and cost of tool verification strategies on EarthBench.
Benchmark
Tasks with a failed observation
Tasks with evidence requests
Tasks with an accepted observation after the request
Of which answered correctly
Rejected calls
EarthBench
179/248 (72.2%)
177/179 (98.9%)
175/177 (98.9%)
104/175 (59.4%)
6
ThinkGeo
185/436 (42.4%)
168/185 (90.8%)
154/168 (91.7%)
77/154 (50.0%)
32
TerraLogic
205/545 (37.6%)
200/205 (97.6%)
191/200 (95.5%)
65/191 (34.0%)
14
Overall
569/1,229 (46.3%)
545/569 (95.8%)
520/545 (95.4%)
246/520 (47.3%)
52
Table 8: Statistics of observation verification and subsequent execution. The first four columns count tasks, and each ratio is conditioned on the previous column.
Benchmark
RSure-Base
RSure-Agent
Δ
EarthBench
95.5
69.6
−25.9
ThinkGeo
93.4
71.2
−22.2
TerraLogic
96.8
75.5
−21.3
Table 9: Error propagation rate Rprop of tool observations (%). Δ is the change of RSure-Agent relative to RSure-Base in percentage points.
Figure 5: Case comparison. (A) The VLM miscounts aircraft (A1) and misjudges vehicle heading (A2). RSure-Agent verifies observations with instance polygons and vehicle front and rear cues, obtaining correct answers. (B) Earth-Agent uses single-day means for an annual trend test. RSure-Agent aggregates data by year and corrects output fields using rule-check feedback. The annual trend observation passes verification. (C) RSure-Agent accepts the mean of seven June–August images, but September data are missing. Missing inputs prevent an alternative computation. Under the evaluation rule, it selects the incorrect option B.
Method
Backbone
Spectrum
Products
RGB
Overall
Earth-Agent
GPT-4o
3.7736
1.1800
0.8779
2.1579
Earth-Agent
GPT-5
3.5510
1.7154
1.5312
2.4146
HiRS-Agent
Qwen3-4B
–
–
–
2.5059
HiRS-Agent
Qwen3-8B
–
–
–
2.5248
RSure-Agent
GPT-5
2.2697
1.6103
1.2502
1.7891
Table 10: Tool-call efficiency on EarthBench. – means not reported.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Argument
Description visible to the model (summary)
image_paths
Input image paths in their semantic order.
objective
Exact visual fact or layer required by the task.
mode
One of seven perception modes: evidence / document / scene / objects / regions / change / measurement.
label_space
Exact semantic classes for mode=scene, separate from the outer multiple-choice response labels.
target_groups
Semantic target id to one or more open-vocabulary prompts; prompts in one group are fused as one class.
landmarks
Exactly two visible landmark descriptions, required only for measurement mode.
Appendix
Table A.1: Key arguments of perceive .
Argument
Description visible to the model (summary)
sources
Explicit perception sources. Declare the needed geometry instead of asking the tool to infer it from question wording.
spec
Typed metric, subject/reference labels, spatial relation, optional distance, denominator, and output unit.
source.mode
objects / regions / change / extent; derived spatial relations go only in spec.
source.gsd
Meters per pixel for each source image. Required when this layer is used with meter or square-meter geometry.
source.backend
Use local for measurable geometry. VLM geometry is approximate and may be rejected by the measurement contract.
spec.distance_statistic
minimum is the closest pair; mean and maximum aggregate one edge distance per subject instance or component.
Appendix
Table A.2: Key arguments of measure .
Category
Task Type
Definition
Answer Form
Perception
Image Modality
Identify the sensor type or imaging mode of the image
Image modality
Image Quality
Judge or compare image quality
Quality judgment
Map Recognition
Identify the region from the outline or content of a map
Region
Scene Classification
Identify the overall scene category of the image
Scene category
Image Caption
Judge whether a candidate caption matches the overall image content
Matching caption
Landmark Recognition
Identify a specific landmark in the image
Landmark name
Appendix
Table C.1: Task types of CHOICE-420 (20 samples per type).
id
title
capability (3 cards)
cap.visual
Visual perception
cap.compute
Deterministic computation
cap.dag_contract
Deterministic DAG contract
recipe (13 cards)
recipe.normalized_difference
Normalized difference
Appendix
Table D.1: Retrieval card index.
Perception
Measurement
Execution
Task type
change
doc.
evid.
meas.
objects
regions
scene
area
boolean
count
distance
asset. catalog
raster. product
raster. summarize
Attribute Comparison
–
–
0.970
–
0.909
–
–
–
–
–
–
–
–
–
Change Detection
0.321
–
0.947
–
0.830
–
–
0.833
–
0.964
–
–
–
–
Spatial Relationship
–
–
0.988
–
0.880
0.818
–
0.857
–
0.951
0.860
–
–
–
Image Caption
–
–
0.923
–
–
–
0.909
–
–
–
–
–
–
–
Image Modality
–
–
0.966
–
–
–
–
–
–
–
–
–
–
–
Appendix
Table E.1: Coarse reliability prior grouped by tool.
Recent advances in large language models and multimodal models have pushed remote sensing (RS) processing from simple perception models to agentic systems designed to tackle complex, long-horizon RS tasks. However, existing systems often rely on monolithic decision-making frameworks, which fail to accommodate the multi-stage, interdependent nature of RS tasks. This centralized approach leads to challenges such as unstable task execution, incorrect tool usage, and error propagation across stages. To address these issues, we propose HiRS-Agent, a hierarchical multi-agent system for long-horizon RS task solving. HiRS-Agent adopts a two-level collaborative architecture: the Manager Layer handles dynamic routing, step-level verification, replanning, and termination control, while the Specialist Layer organizes domain-specific tools according to the RS workflow and is responsible for subtask reasoning and tool execution. To further enhance the system's capability, we introduce a two-stage supervised tuning strategy and a verification-guided hierarchical reinforcement learning stage to jointly optimize coordination and tool-use policies. Experiments on Earth-Agent Benchmark and ThinkGeo show that HiRS-Agent substantially improves long-horizon tool-use capability and final-task correctness, demonstrating the effectiveness of structured multi-agent collaboration for reliable RS agents. The code is publicly available at https://github.com/IntelliSensing/HiRS-Agent.
Boyang Mu, Zhiwei Wei, Mugen Peng +1
Beijing University of Posts and Telecommunications State Key Laboratory of Networking and Switching Technology Beijing, China · Hunan Normal University School of Geographic Sciences Changsha, Hunan, China
Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a benchmark. We show that this loop can be handed to an agent. Any verifiable task can be cast as a harness and a verifier, where the harness defines what the agent may use and the verifier defines what counts as success. Every EO task can be made verifiable, through labeled benchmarks, independent sensors, physical models or the repeated observation of the same place. Given only these two components, a general-purpose coding agent builds the EO model and improves it recursively on its own outputs. Settings long treated as separate research problems, from learning with a handful of labels to adapting across domains, become choices of harness and verifier. On six EO benchmarks spanning optical, SAR and multispectral time-series imagery, with outputs that range from land-cover maps and ship instances to canopy heights and answers about large images, agent-built models surpass the state of the art on every one, and on land-cover mapping, they do so from 11% of the training labels. This establishes a new paradigm for EO, in which people design harnesses and verifiers and agents build the models, so EO models improve as agents do.
Sijie Zhao, Zeqi Zhou, Heli Qi +6
The University of Tokyo · RIKEN · Stanford University
Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare data, perform domain computations, and derive conclusions from runtime evidence. Existing EO agents typically start from supplied observations, while benchmarks typically provide prepared inputs or candidate answers, leaving full-chain open-world EO execution largely untested. We present Earth-Agent-Pro, an execution-adaptive Plan-and-Execute framework using expert-authored skills to constrain planning and runtime tool use. Workflow-centered structured memory records planned steps, accepted evidence, and their dependencies, enabling repair of only the affected workflow suffix when runtime evidence invalidates a step. Separate large language model adapters use sequence-level supervised fine-tuning for planner workflow composition and node-level group relative policy optimization with locally verifiable rewards for executor tool-argument grounding. Earth-Bench-Pro instantiates 248 expert-curated task cores as 744 questions under three matched regimes. Its 248 Open-World Execution questions span RGB imagery, spectral observations, and remote sensing products, pairing high-level requests with runtime data requirements, executable trajectories, and open-ended answers grounded in execution evidence. With a shared GPT-5 backbone, Earth-Agent-Pro achieves 66.13% LLM-as-Judge accuracy, exceeding ReAct by 20.95 points in this metric and 24.44 points in Tools-In-Order. Joint adapter tuning raises Qwen3.5-9B LLM-as-Judge accuracy from 38.31% to 50.00%, an 11.69-point gain over the untuned configuration. Planning-only evaluation and execution with the reference workflow show that the adapters improve workflow composition and argument grounding, respectively. Code and datasets will be released soon.
Zhutao Lv, Chenhao Dang, Yi Feng +5
Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University +3