Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Response

Georgios Angelidis; Aggelos Psiris; Spyridon Evangelatos; Thomas Lagkas; Panagiotis Sarigiannidis; Iraklis Varlamis; Georgios Th. Papadopoulos

cs.CVOct 13, 2026

Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Response

Authors: Georgios Angelidis, Aggelos Psiris, Spyridon Evangelatos, Thomas Lagkas, Panagiotis Sarigiannidis, Iraklis Varlamis, Georgios Th. Papadopoulos

Organizations: Harokopio University of Athens, Athens, Greece · Netcompany-Intrasoft S.A., Luxembourg, Luxembourg · Dimocritus University of Thrace, Kavala, Greece · University of Western Macedonia, Kozani, Greece

Abstract

Agentic remote sensing is evaluated on how many questions it can answer, whereas crisis response depends on whether its answers can be relied on and how much of the decision can safely be handed to the agent. This paper argues that autonomy should be earned, not assumed: An Earth-Observation (EO) agent should be granted autonomy in proportion to demonstrated reliability, epistemic honesty and verifiability. The position is supported by an evidential meta-analysis of twelve agentic RS/EO/GIS benchmark studies from peer-reviewed venues. Only one repeats runs, three contain unanswerable tasks and none measures calibration. Their own tables show that completed workflows are often not correct (task success exceeds tool-call correctness by up to 54pp on identical queries), that the gap to an expert trajectory widens overall with trajectory length (from 7 to 56pp), and that capability does not predict abstention (Spearman r from -0.25 to 0.50, all n.s.). The paper then contributes three EO-specific trust properties, an autonomy ladder for crisis EO tasks, and a readiness protocol that replays archived emergency activations before any live use. In this context, current evidence justifies assistive roles only.

Figures & tables

Benchmark	Venue	Tasks	Data	Crisis tasks	Rep. runs	Unans.	Output scoring	Cost	Calib.
ThinkGeo [ 27 ]	CVPR’26 WS †	486	RGB, SAR	$\leq$ 148/436 opt. ¶	✗	✗	LLM judge	Latency	✗
Earth-Bench [ 8 ]	ICLR’26	248	MS, products, RGB	24 (+13) ¶	✗	✗	Exact match (multiple choice)	Latency	✗
OpenEarthAgent [ 28 ]	ECCV’26	1,169	RGB, SAR, indices, vector	$\approx$ 17%	✗	✗	LLM judge	Latency	✗
DORA [ 32 ]	NeurIPS’26 ∗	515	8 incl. SAR, DEM	all (45 events)	✗	✗	Deterministic, tolerances	Latency	✗
UnivEARTH [ 12 ]	ACL’26 Findings	408	MS, geophysical	fire $\approx$ 11%	✗	✗	LLM answer extraction	–	✗
GeoBenchX [ 15 ]	SIGSPATIAL’25 WS	202	vector, raster, tables	n.r.	✗	✓ (79)	3-LLM judge panel	Tokens	✗

Table 1: Trust-coverage audit of the twelve benchmarks. † Non-archival workshop; cited from arXiv. ∗ Accepted, to appear. ‡ LLM-generated (GPT-4/4o) reference trajectories, partly human-vetted. § Map-API part only; Spatial-Agent: Minimum inferred from its tables. ¶ ThinkGeo: Upper bound (includes generic change analysis); Earth-Bench: 24 disaster + 13 change detection. n.r.: Present, not counted; MS: Multispectral; products: Derived layers ( e.g . MODIS, GHSL); DORA latency for one model only.

Figure 1: The capability–trust gap in the benchmarks’ own numbers. (a) Geo-OLM, identical 2k queries, GPT-4o: Task success vs . tool-call correctness per framework; labels: Gap in pp [ 30 ] . (b) DORA: Accuracy gap (pp) between the gold trajectory and the best of five plotted agents, by gold length [ 32 ] . (c) UnivEARTH: Correct and wrong answers at zero-shot (0) and after two reflexion rounds (R2); labels: Precision among answered questions [ 12 ] . (d) Accuracy on answerable tasks vs . correct rejection of unanswerable ones in GeoBenchX [ 15 ] , GeoNatureAgent [ 5 ] (error handling) and MapEval-API [ 6 ] ; Spearman

\rho=-0.25

(

p=0.55

n=8

0.33

(

p=0.38

n=9

) and

0.50

(

p=0.12

n=11

Panel of Figure 2 — Figure 2: Autonomy ladder for crisis EO tasks. Rungs follow the user roles of [ 7 ] ; above each rung an example task, below it the gate ( Tab. 2 ) required to reach it. Shading: Evidence of Sec. 2 .

Stage	Setting	Measured	Gate to next
R1 Benchmark	Curated tasks with unanswerable variants	Accuracy, pass^ $k$ , abstention F1, risk–coverage, calibration error, $ per correct answer	Stable pass^ $k$ and abstention above user threshold
R2 Replay	Archived activations, reference = delivered products	Agreement with reference, evidence validity, time to product	Within the service’s QC tolerance
R3 Shadow	Live events, nothing released	Agreement with analysts, useful-abstention rate, analyst time saved	Sustained across event types
R4 Supervised	Outputs enter the normal QC chain	Rejection rate at QC, correction effort	Low rejection over a set period
R5 Conditional	Low-consequence sub-tasks only	Continuous monitoring, drift	Revoked on any drift

Table 2: Staged readiness protocol. Thresholds are set by the operational user; the protocol fixes the structure, not the numbers.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Inst	Tool	Arg	Ans	Ans I
GPT-4o	82.33	67.73	34.75	9.78	20.40
GPT-4-1106	86.49	74.05	36.96	5.16	14.69
Claude-Sonnet	22.31	27.31	0.00	8.97	7.57
Qwen1.5-7B	26.92	11.47	3.06	5.43	4.59
Qwen2.5-7B	64.88	51.04	20.08	7.34	6.40
InternLM3-8B	50.53	45.37	21.38	8.15	9.15

Appendix

Table A1 : ThinkGeo, optical split (T.3 of [ 27 ] ).

Model	Tool	ArgV	GIS F1	Ans
GPT-5	82.69	37.28	91.50	43.88
GPT-4o	93.88	45.80	95.80	39.22
o4-mini	68.33	37.95	86.51	35.18
Qwen2.5-7B	85.51	38.00	75.46	15.85
Llama-3.1-8B	39.30	17.49	64.64	12.70
InternLM3-8B	38.46	13.25	31.47	0.24

Appendix

Table A2 : OpenEarthAgent (T.3 and T.4 of [ 28 ] ).

Agent	Final	T-Any-O	T-Exact	ParAcc
Gold trajectory	80.48	100	100	100
Gemini-3.0-Flash	53.74	75.17	35.55	42.63
Qwen3.5-397B-A17B	53.45	75.44	32.13	36.50
MiMo-V2-Pro	52.89	74.05	34.26	38.45
Grok-4.1-Fast	52.10	59.15	28.09	31.96
Claude-Sonnet-4.6	52.01	76.29	31.43	39.48

Appendix

Table A3 : DORA (T.4 of [ 32 ] ), sorted by final accuracy.

Model	Correct	Wrong	No valid answer	Precision
Claude-4.5-Sonnet	40.0	15.7	44.2	0.72
Gemini-2.5-Pro	36.9	9.6	53.5	0.79
GPT-5	33.1	12.0	54.9	0.73
DeepSeek-Reasoner	28.5	13.0	58.6	0.69
DeepSeek-Chat	27.0	13.0	60.0	0.68
Kimi-k2	17.0	8.4	74.7	0.67

Appendix

Table A4 : UnivEARTH zero-shot (T.3 of [ 12 ] ); precision

=

correct/(correct

+

wrong) is derived.

Model	Solvable	Rejection	In (M)	Out (k)
Claude Sonnet 4	0.55	0.17	26.9	291.6
o4-mini	0.51	0.90	7.9	956.6
Claude Sonnet 3.5	0.47	0.47	12.2	151.2
GPT-4.1	0.42	0.43	12.7	121.3
Claude Haiku 3.5	0.40	0.22	14.9	196.0
Gemini 2.0 Flash	0.39	0.40	7.5	40.7

Appendix

Table A5 : GeoBenchX match rates and token use (T.1, T.2, T.8 of [ 15 ] ).

Model	Accuracy	Error handling	$ per run
Claude Sonnet 4	60.8 $\pm$ 0.8	50	11.84
DeepSeek V3.2	56.3 $\pm$ 3.1	44	1.05
GLM-5	50.2 $\pm$ 2.2	44	3.58
Gemini 2.5 Pro	48.0 $\pm$ 3.3	72	4.87
GPT-4o	41.6 $\pm$ 2.7	44	6.49
Qwen3-235B	41.2 $\pm$ 4.3	44	0.89

Appendix

Table A6 : GeoNatureAgent (T.1 and Fig. 4 of [ 5 ] ).

Framework (GPT-4o)	Success	Correct.	Total $	$/corr. unit
GeoLLM-QA	62.3	52.49	68.23	0.065
GeoLLM-Engine	82.6	74.11	64.72	0.044
Chameleon	55.1	21.87	115.08	0.263
Magentic	79.7	25.40	447.55	0.881
GeoLLM-Squad	96.2	85.32	219.67	0.129
Geo-OLM	95.1	86.03	77.02	0.045

Appendix

Table A7 : Geo-OLM benchmark, GPT-4o and o1 rows (T.1 of [ 30 ] ); last column derived.

Method	Backbone	MapEval-API	MapQA
Direct (no tools)	GPT-4o-mini	32.23	13.55
MapEval ReAct agent	GPT-4o-mini	23.00	–
ReAct	GPT-4o-mini	32.98	43.79
Reflexion	GPT-4o-mini	38.29	53.79
Spatial-Agent	GPT-4o-mini	45.15	61.45
Spatial-Agent	GPT-3.5-Turbo	34.61	–

Appendix

Table A8 : Spatial-Agent (T.1 and T.2 of [ 2 ] ); reported overall accuracy, except the MapEval row (per question over 300 questions).

Model	API overall	API unans.	Textual overall
Claude-3.5-Sonnet	64.00	55	66.33
GPT-4-Turbo	53.67	25	62.33
GPT-4o	48.67	15	63.33
Gemini-1.5-Pro	43.33	65	66.33
Gemini-1.5-Flash	41.67	30	58.67
Llama-3.2-90B	39.67	15	58.33

Appendix

Table A9 : MapEval-API (T.4) and MapEval-Textual (T.3) of [ 6 ] .

Agent Reliability

Agent reliability studies how to make AI agents behave consistently, recover safely from failures, and remain dependable under real-world uncertainty.

See all 45 papers

RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents
Oct 4, 2026Fuyuan Liu, Nayu Liu, Wenhao Yu +8LLM Agent Verification Remote Sensing
School of Computer Science and Technology, Tianjin University, Tianjin 300350, China · Key Laboratory of Target Cognition and Application Technology (TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China
Agentic AI for Remote Sensing: Technical Challenges and Research Directions
Apr 27, 2026Muhammad Akhtar Munir, Muhammad Umer Sheikh, Akashah Shabbir +5Agentic Workflows Remote Sensing
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Technical University of Munich, Munich, Germany · BIFOLD and Technische Universität Berlin, Berlin, Germany
The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement
Oct 13, 2026Sijie Zhao, Zeqi Zhou, Heli Qi +6Remote Sensing Recursive Self-Improvement
The University of Tokyo · RIKEN · Stanford University

Oct 4, 2026J/K move · Enter open · S save

RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents

Fuyuan Liu, Nayu Liu, Wenhao Yu, Peijin Wang+7

School of Computer Science and Technology, Tianjin University, Tianjin 300350, China · Key Laboratory of Target Cognition and Application Technology (TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

LLM Agent Verification Remote Sensing

Open paper AskPDF arXiv

Remote sensing agents rely on perception, measurement, and raster analysis tools to solve Earth observation tasks. We refer to their judgments and quantitative results about ground objects as tool observations. However, these observations are subject to uncertainty and may be incorrect even when tools execute successfully. When agents accept incorrect observations, the errors can propagate through subsequent reasoning and cause task failure. We analyze 1,229 execution trajectories across three remote sensing agent benchmarks. On each benchmark, at least 88.1 percent of tasks depend on tool observations. Among these tasks, at least 22.7 percent contain incorrect observations despite successful tool execution. These errors propagate to the final answer in at least 82.0 percent of affected tasks on each benchmark. To address this problem, we propose RSure-Agent, a framework for verifying tool observations and limiting error propagation. We introduce a verifiable observation protocol that requires tools to return process evidence for the agent to verify their observations. We construct a task-tool reliability prior from offline task feedback. The prior summarizes each tool configuration's past performance across task types and provides a task-specific reference for verification. Using process evidence and this prior, RSure-Agent decides whether to accept an observation, request additional evidence, or reject it. We evaluate RSure-Agent on EarthBench, ThinkGeo, TerraLogic, and CHOICE-420. Across three agent benchmarks, RSure-Agent reduces the error propagation rate by 21.3 to 25.9 percentage points relative to the base configuration with verification and the prior disabled. On CHOICE-420, it improves overall accuracy over direct answering by 5.71 percentage points on average across 11 backbone models. On EarthBench, it reduces the tool-call ratio by 25.9 percent relative to Earth-Agent.

Abstract

Agent Reliability

Explore similar work

RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents

Agentic AI for Remote Sensing: Technical Challenges and Research Directions

The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement