RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents

Fuyuan Liu; Nayu Liu; Wenhao Yu; Peijin Wang; Yingchao Feng; Fanglong Yao; Yue Zhou; Xue Yang

cs.CVOct 4, 2026

RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents

Authors: Fuyuan Liu, Nayu Liu, Wenhao Yu, Peijin Wang, Yingchao Feng, Fanglong Yao, Yue Zhou, Xue Yang,

Organizations: School of Computer Science and Technology, Tianjin University, Tianjin 300350, China · Key Laboratory of Target Cognition and Application Technology (TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

Abstract

Remote sensing agents rely on perception, measurement, and raster analysis tools to solve Earth observation tasks. We refer to their judgments and quantitative results about ground objects as tool observations. However, these observations are subject to uncertainty and may be incorrect even when tools execute successfully. When agents accept incorrect observations, the errors can propagate through subsequent reasoning and cause task failure. We analyze 1,229 execution trajectories across three remote sensing agent benchmarks. On each benchmark, at least 88.1 percent of tasks depend on tool observations. Among these tasks, at least 22.7 percent contain incorrect observations despite successful tool execution. These errors propagate to the final answer in at least 82.0 percent of affected tasks on each benchmark. To address this problem, we propose RSure-Agent, a framework for verifying tool observations and limiting error propagation. We introduce a verifiable observation protocol that requires tools to return process evidence for the agent to verify their observations. We construct a task-tool reliability prior from offline task feedback. The prior summarizes each tool configuration's past performance across task types and provides a task-specific reference for verification. Using process evidence and this prior, RSure-Agent decides whether to accept an observation, request additional evidence, or reject it. We evaluate RSure-Agent on EarthBench, ThinkGeo, TerraLogic, and CHOICE-420. Across three agent benchmarks, RSure-Agent reduces the error propagation rate by 21.3 to 25.9 percentage points relative to the base configuration with verification and the prior disabled. On CHOICE-420, it improves overall accuracy over direct answering by 5.71 percentage points on average across 11 backbone models. On EarthBench, it reduces the tool-call ratio by 25.9 percent relative to Earth-Agent.

Figures & tables

Figure 1: Illustration of RSure-Agent. (a) The tool executes without runtime errors, but the segmentation mask includes non-vehicle pixels, yielding an erroneous area of $506.626~\mathrm{m}^{2}$ . (b) Through the verifiable observation protocol, RSure-Agent inspects the source image, mask, and execution log. After deterministic checks pass, the verifier detects that the mask covers a swimming pool, refuses the faulty observation, and prompts mask refinement. Following SAM 2.1 refinement, the recomputed area is $34.170~\mathrm{m}^{2}$ (within the accepted range of $29.40$ – $35.94~\mathrm{m}^{2}$ ). The task–tool reliability prior provides a historical reference during verification.

Benchmark	Agent	Backbone	$\mathcal{R}_{\mathrm{dep}}$	$\mathcal{R}_{\mathrm{err}}$	$\mathcal{R}_{\mathrm{prop}}$
EarthBench	Earth-Agent	GPT-5	88.7	22.7	82.0
ThinkGeo	ThinkGeo-Agent	GPT-4o	91.7	41.3	91.5
TerraLogic	HieraPlan	GPT-4o	88.1	39.8	93.2

Table 1: Tool observation statistics (%) of the benchmark reference agents.

Figure 2: Examples of incorrect tool observations under normal execution across (a) detection, (b) segmentation, (c) counting, and (d) image quality.

Figure 3: Overall framework of RSure-Agent. The verifiable observation protocol requires tools to return execution status, observations, and process evidence (source evidence, intermediate results, and validity evidence). Online verification applies deterministic rule checks, then verifies observations that pass using the image, task, process evidence, and task–tool reliability prior. The verifier decides to accept, request evidence, or reject. Accepted observations and evidence enter the verified history. The backbone model uses feedback to plan subsequent tool calls and generates the final answer from this history after interaction ends.

Evidence type	perceive	measure	execute
Source evidence $\mathcal{E}^{\mathrm{src}}_{t}$	Detector weights Thresholds Sensor Resolution	Input layer sources Metric Unit specification	Operator parameters Product formulas Band mapping
Intermediate results $\mathcal{E}^{\mathrm{mid}}_{t}$	Object bounding boxes Confidence scores Number of filtered candidates	Measurement numerator Denominator Pixel counts	Per-node outputs Valid-pixel counts
Validity evidence $\mathcal{E}^{\mathrm{val}}_{t}$	Layer status Class completeness Geometry flags	Layer status Class coverage	Node status Quality masks List of failed nodes

Table 2: Process evidence returned by the three observation tools.

Task type	Tool	Configuration	Level	$\hat{r}$
Object Counting	perceive	objects	Coarse	0.926
Object Counting	measure	count\|auto\|objects	Fine	0.975
Object Presence	perceive	objects	Coarse	0.800
Image Caption	perceive	evidence	Coarse	0.923
Change Detection	perceive	change	Coarse	0.321
Env. Assessment	execute	raster.product	Coarse	0.810

Table 3: Selected entries of the task–tool reliability prior.

Figure 4: Representative remote sensing inputs across the benchmarks, including optical, SAR, infrared, thermal, and multi-band imagery.

Method	Backbone	Weights	ThinkGeo		EarthBench	TerraLogic
Method	Backbone	Weights	Ans.	$\mathrm{Ans}_{I}$	Acc.	AnsAcc
ThinkGeo-Agent	GPT-4o	Closed	9.8	20.4	–	–
HieraPlan	GPT-4o	Closed	–	–	–	25.7
Earth-Agent	GPT-4o	Closed	–	–	45.3	–
Earth-Agent	GPT-5	Closed	–	–	65.6	–
OpenEarth-Agent	GPT-5	Closed	–	–	67.6	–

Table 4: Task performance (%) on RS agent benchmarks. † denotes results following RS-Claw’s test split on EarthBench.

Backbone	Weights	Overall		Perception		Reasoning
Backbone	Weights	Origin	RSure-Agent	Origin	RSure-Agent	Origin	RSure-Agent
Mistral Small 4	Open	75.71	84.05 (+8.34)	76.07	86.07 (+10.00)	75.00	80.00 (+5.00)
Qwen3.5-27B		74.05	83.33 (+9.28)	78.57	86.79 (+8.22)	65.00	76.43 (+11.43)
Qwen3.5-122B		69.29	82.86 (+13.57)	72.50	86.79 (+14.29)	62.86	75.00 (+12.14)
Qwen3.5-397B		83.81	85.95 (+2.14)	85.36	87.50 (+2.14)	80.71	82.86 (+2.15)
Qwen3.6-27B		79.76	85.24 (+5.48)	83.93	88.21 (+4.28)	71.43	79.29 (+7.86)

Table 5: Accuracy (%) on CHOICE-420: direct answering (Origin) versus RSure-Agent. Parentheses show gains in percentage points. Bold marks the best value.

Verification	Prior	ThinkGeo		EarthBench	TerraLogic
Verification	Prior	Ans.	$\mathrm{Ans}_{I}$	Acc.	AnsAcc
✗	✗	21.1	24.8	64.9	26.4
✓	✗	22.5	29.5	66.9	29.7
✓	✓	23.6	32.6	68.1	31.4

Table 6: Ablation study (%, GPT-5 backbone).

Method	Acc. (%)	LLM calls	Token usage	Time per question (s)
Method	Acc. (%)	LLM calls	Token usage	Mean	Median
RSure-Base	64.9	7.117	90,454	460.37	321.99
RSure-Base + CRITIC	66.9	8.871	148,562	610.45	505.76
RSure-Base + Tools Fail	65.3	9.935	125,322	557.30	458.28
RSure-Agent	68.1	8.367	107,860	477.08	334.46

Table 7: Performance and cost of tool verification strategies on EarthBench.

Benchmark	Tasks with a failed observation	Tasks with evidence requests	Tasks with an accepted observation after the request	Of which answered correctly	Rejected calls
EarthBench	179/248 (72.2%)	177/179 (98.9%)	175/177 (98.9%)	104/175 (59.4%)	6
ThinkGeo	185/436 (42.4%)	168/185 (90.8%)	154/168 (91.7%)	77/154 (50.0%)	32
TerraLogic	205/545 (37.6%)	200/205 (97.6%)	191/200 (95.5%)	65/191 (34.0%)	14
Overall	569/1,229 (46.3%)	545/569 (95.8%)	520/545 (95.4%)	246/520 (47.3%)	52

Table 8: Statistics of observation verification and subsequent execution. The first four columns count tasks, and each ratio is conditioned on the previous column.

Benchmark	RSure-Base	RSure-Agent	$\Delta$
EarthBench	95.5	69.6	$-25.9$
ThinkGeo	93.4	71.2	$-22.2$
TerraLogic	96.8	75.5	$-21.3$

Table 9: Error propagation rate

\mathcal{R}_{\mathrm{prop}}

of tool observations (%).

\Delta

is the change of RSure-Agent relative to RSure-Base in percentage points.

Figure 5: Case comparison. (A) The VLM miscounts aircraft (A1) and misjudges vehicle heading (A2). RSure-Agent verifies observations with instance polygons and vehicle front and rear cues, obtaining correct answers. (B) Earth-Agent uses single-day means for an annual trend test. RSure-Agent aggregates data by year and corrects output fields using rule-check feedback. The annual trend observation passes verification. (C) RSure-Agent accepts the mean of seven June–August images, but September data are missing. Missing inputs prevent an alternative computation. Under the evaluation rule, it selects the incorrect option B.

Method	Backbone	Spectrum	Products	RGB	Overall
Earth-Agent	GPT-4o	3.7736	1.1800	0.8779	2.1579
Earth-Agent	GPT-5	3.5510	1.7154	1.5312	2.4146
HiRS-Agent	Qwen3-4B	–	–	–	2.5059
HiRS-Agent	Qwen3-8B	–	–	–	2.5248
RSure-Agent	GPT-5	2.2697	1.6103	1.2502	1.7891

Table 10: Tool-call efficiency on EarthBench. – means not reported.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Argument	Description visible to the model (summary)
image_paths	Input image paths in their semantic order.
objective	Exact visual fact or layer required by the task.
mode	One of seven perception modes: evidence / document / scene / objects / regions / change / measurement.
label_space	Exact semantic classes for mode=scene, separate from the outer multiple-choice response labels.
target_groups	Semantic target id to one or more open-vocabulary prompts; prompts in one group are fused as one class.
landmarks	Exactly two visible landmark descriptions, required only for measurement mode.

Appendix

Table A.1: Key arguments of perceive .

Argument	Description visible to the model (summary)
sources	Explicit perception sources. Declare the needed geometry instead of asking the tool to infer it from question wording.
spec	Typed metric, subject/reference labels, spatial relation, optional distance, denominator, and output unit.
source.mode	objects / regions / change / extent; derived spatial relations go only in spec.
source.gsd	Meters per pixel for each source image. Required when this layer is used with meter or square-meter geometry.
source.backend	Use local for measurable geometry. VLM geometry is approximate and may be rejected by the measurement contract.
spec.distance_statistic	minimum is the closest pair; mean and maximum aggregate one edge distance per subject instance or component.

Appendix

Table A.2: Key arguments of measure .

Category	Task Type	Definition	Answer Form
Perception	Image Modality	Identify the sensor type or imaging mode of the image	Image modality
	Image Quality	Judge or compare image quality	Quality judgment
	Map Recognition	Identify the region from the outline or content of a map	Region
	Scene Classification	Identify the overall scene category of the image	Scene category
	Image Caption	Judge whether a candidate caption matches the overall image content	Matching caption
	Landmark Recognition	Identify a specific landmark in the image	Landmark name

Appendix

Table C.1: Task types of CHOICE-420 (20 samples per type).

id	title
capability (3 cards)
cap.visual	Visual perception
cap.compute	Deterministic computation
cap.dag_contract	Deterministic DAG contract
recipe (13 cards)
recipe.normalized_difference	Normalized difference

Appendix

Table D.1: Retrieval card index.

Perception

Measurement

Execution

Task type

change

doc.

evid.

meas.

objects

regions

scene

area

boolean

count

distance

asset. catalog

raster. product

raster. summarize

Attribute Comparison

–

0.970

–

0.909

–

Change Detection

0.321

–

0.947

–

0.830

–

0.833

–

0.964

–

Spatial Relationship

–

0.988

–

0.880

0.818

–

0.857

–

0.951

0.860

–

Image Caption

–

0.923

–

0.909

–

Image Modality

–

0.966

–

Appendix

Table E.1: Coarse reliability prior grouped by tool.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 142 papers

HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving
Aug 31, 2026Boyang Mu, Zhiwei Wei, Mugen Peng +1Multi-Agent Collaboration Remote Sensing
Beijing University of Posts and Telecommunications State Key Laboratory of Networking and Switching Technology Beijing, China · Hunan Normal University School of Geographic Sciences Changsha, Hunan, China
The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement
Oct 13, 2026Sijie Zhao, Zeqi Zhou, Heli Qi +6Remote Sensing Recursive Self-Improvement
The University of Tokyo · RIKEN · Stanford University
Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents
Sep 11, 2026Zhutao Lv, Chenhao Dang, Yi Feng +5Remote Sensing AI Agent Benchmarks
Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University +3

Aug 31, 2026J/K move · Enter open · S save

HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving

Boyang Mu, Zhiwei Wei, Mugen Peng, Wenjia Xu

Beijing University of Posts and Telecommunications State Key Laboratory of Networking and Switching Technology Beijing, China · Hunan Normal University School of Geographic Sciences Changsha, Hunan, China

Multi-Agent Collaboration Remote Sensing Multi-Agent Systems Long-Horizon Agent Tasks Long-Horizon LLM Agents

Open paper AskPDF arXiv

Recent advances in large language models and multimodal models have pushed remote sensing (RS) processing from simple perception models to agentic systems designed to tackle complex, long-horizon RS tasks. However, existing systems often rely on monolithic decision-making frameworks, which fail to accommodate the multi-stage, interdependent nature of RS tasks. This centralized approach leads to challenges such as unstable task execution, incorrect tool usage, and error propagation across stages. To address these issues, we propose HiRS-Agent, a hierarchical multi-agent system for long-horizon RS task solving. HiRS-Agent adopts a two-level collaborative architecture: the Manager Layer handles dynamic routing, step-level verification, replanning, and termination control, while the Specialist Layer organizes domain-specific tools according to the RS workflow and is responsible for subtask reasoning and tool execution. To further enhance the system's capability, we introduce a two-stage supervised tuning strategy and a verification-guided hierarchical reinforcement learning stage to jointly optimize coordination and tool-use policies. Experiments on Earth-Agent Benchmark and ThinkGeo show that HiRS-Agent substantially improves long-horizon tool-use capability and final-task correctness, demonstrating the effectiveness of structured multi-agent collaboration for reliable RS agents. The code is publicly available at https://github.com/IntelliSensing/HiRS-Agent.

Abstract

LLM Agent Verification

Explore similar work

HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving

The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents