RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs

Liang Yao; Shengxiang Xu; Fan Liu; Chuanyi Zhang; Bishun Yao; Rui Min; Yongjun Li; Chaoqian Ouyang

cs.CVApr 9, 2026

RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs

Authors: Liang Yao, Shengxiang Xu, Fan Liu, Chuanyi Zhang, Bishun Yao, Rui Min, Yongjun Li, Chaoqian Ouyang,

Organizations: Hohai University · Southeast University · Sun Yat-sen University

Abstract

Earth Observation (EO) systems are essentially designed to support domain experts who often express their requirements through vague natural language rather than precise, machine-friendly instructions. Depending on the specific application scenario, these vague queries can demand vastly different levels of visual precision. Consequently, a practical EO AI system must bridge the gap between ambiguous human queries and the appropriate multi-granularity visual analysis tasks, ranging from holistic image interpretation to fine-grained pixel-wise predictions. While Multi-modal Large Language Models (MLLMs) demonstrate strong semantic understanding, their text-based output format is inherently ill-suited for dense, precision-critical spatial predictions. Existing agentic frameworks address this limitation by delegating tasks to external tools, but indiscriminate tool invocation is computationally inefficient and underutilizes the MLLM's native capabilities. To this end, we propose RemoteAgent, an agentic framework that strategically respects the intrinsic capability boundaries of MLLMs. To empower this framework to understand real user intents, we construct VagueEO, a human-centric instruction dataset pairing EO tasks with simulated vague natural-language queries. By leveraging VagueEO for reinforcement fine-tuning, we align an MLLM into a robust cognitive core that directly resolves image- and sparse region-level tasks. Consequently, RemoteAgent processes suitable tasks internally while intelligently orchestrating specialized tools via the Model Context Protocol exclusively for dense predictions. Extensive experiments demonstrate that RemoteAgent achieves robust intent recognition capabilities while delivering highly competitive performance across diverse EO tasks.

Figures & tables

Figure 1: (a) The usability gap between vague user intents and rigid system requirements. (b) Existing MLLMs struggle with dense output tasks, whereas tool-augmented agents suffer from indiscriminate tool overuse. (c) RemoteAgent bridges this gap by internally resolving macroscopic queries while orchestrating specialized tools strictly for dense predictions.

Figure 2: VagueEO Benchmark Overview. We construct ten diverse Earth Observation tasks that pair vague, human-centric queries with standardized structural annotations.

Figure 3: Overview of RemoteAgent. During training, the model is aligned via GRPO, guided by a unified multi-task reward that evaluates coordinate, numerical, and textual outputs. During inference, the agent dynamically plans and routes queries, directly resolving macroscopic tasks internally while delegating dense predictions to a specialized external toolkit. Task abbreviations: Visual Question Answering (VQA), Visual Grounding (VG), Classification (CLS), Detection (DET), Segmentation (SEG), Referring Expression Segmentation (RES), Change Detection (CD), and Contour Extraction (CE).

Figure 4: Intent recognition performance across diverse EO tasks on our VagueEO. RemoteAgent eclipses all baselines.

Methods	Publication	AID [ 62 ]	WHU-RS19 [ 3 ]
Methods	Publication	$Acc$	$Acc$
InternVL3.5 [ 56 ]	arXiv’25	73.80	91.50
Qwen2.5-VL [ 2 ]	arXiv’25	63.07	76.60
Phi3.5-Vision [ 1 ]	arXiv’24	56.57	68.90
GeoChat [ 20 ]	CVPR’24	73.17	84.80
EarthDial [ 50 ]	CVPR’25	87.57	95.80

Table 1: Comparison of scene classification results.

Methods	Publication	DIOR-RSVG [ 74 ]
Methods	Publication	$[email protected]$	$IoU$
SkyEyeGPT [ 75 ]	NIPS’22	70.5	-
GeoChat [ 20 ]	CVPR’24	31.4	14.7
SkySenseGPT [ 36 ]	arXiv’24	60.8	35.5
LHRS-Bot [ 38 ]	ECCV’24	73.5	-
Falcon [ 19 ]	arXiv’25	56.9	-

Table 2: Comparison of visual grounding results.

Methods	Test		Val
Methods	$[email protected]$	$[email protected]$	$gIoU$	$gIoU$
DeepSeek-VL2-tiny [ 60 ]	12.08	12.67	17.51	18.62
GeoChat [ 20 ]	10.10	8.89	12.57	11.44
Qwen2.5-VL-7B [ 2 ]	41.21	45.82	38.77	41.80
RemoteReasoner [ 70 ]	66.51	68.11	67.04	69.29
RemoteAgent	57.81	54.22	55.60	52.22

Table 3: Comparison of geospatial region reasoning results with various MLLMs on EarthReason [ 24 ] .

Methods	Publication	HRRSD [ 80 ]	DOTAv2 [ 61 ]
Methods	Publication	$Acc$	$Acc$
GeoChat [ 20 ]	CVPR’24	57.6	16.9
VHM [ 40 ]	AAAI’25	46.7	18.0
RSUniVLM [ 35 ]	arXiv’24	54.2	19.0
LLaVA-1.5 [ 31 ]	NIPS’24	-	22.1
LHRS-Bot [ 38 ]	ECCV’24	-	24.4

Table 4: Comparison of object counting results with various MLLMs on two datasets.

Methods	Publication	DIOR [ 25 ]	DIOR-R [ 7 ]
Methods	Publication	$AP50$	$AP50$
Specialized Models
GFM [ 51 ]	ICCV’23	72.84	67.67
Scale-MAE [ 42 ]	ICCV’23	73.81	66.47
SkySense [ 13 ]	CVPR’24	78.73	74.27
MLLMs

Table 5: Comparison of object detection results with various specialized models and MLLMs.

Methods	Publication	iSAID [ 57 ]	Potsdam [ 49 ]
Methods	Publication	$mIoU$	$mF1$
Scale-MAE [ 42 ]	ICCV’23	65.77	91.54
MA3E [ 30 ]	ECCV’24	64.06	91.50
SkySense [ 13 ]	CVPR’24	70.91	93.99
RS-vHeat [ 15 ]	ICCV’25	68.72	92.82
RemoteSAM [ 69 ]	MM’25	64.72	91.80

Table 6: Comparison of semantic segmentation results with various specialized models.

Methods	Publication	RRSIS-D [ 34 ]
Methods	Publication	$[email protected]$	$oIoU$	$mIoU$
Specialized Models
LAVT [ 68 ]	CVPR’22	69.52	77.19	61.04
LGCE [ 73 ]	TGRS’24	67.65	76.34	59.37
RMSIN [ 34 ]	CVPR’24	74.26	77.79	64.20
CroBIM [ 10 ]	TGRS’24	74.58	75.99	64.46

Table 7: Comparison of referring expression segmentation results with various specialized models and MLLMs.

Methods	Publication	xBD
Methods	Publication	$F1_{loc}$	$F1_{cls}$	$F1_{overall}$
ChangeOS [ 84 ]	RSE’21	85.69	71.14	75.5
DamFormer [ 5 ]	IGARSS’22	86.86	72.81	77.02
PCDASNet [ 54 ]	TGRS’24	85.48	73.83	77.33
RemoteAgent	-	80.12	73.03	77.16

Table 8: Comparison of building damage assessment results with various specialized models.

Method	VG ( $[email protected]$ )	RES ( $mIoU$ )	Intent ( $Acc$ )	Time ( $s$ )
Zero-shot	43.6	71.13	49	0.84
SFT	54.2	52.19	67	0.71
RL	68.9	71.64	95	0.83

Table 9: Ablation on different training strategies.

Method	LLM (s)	Tool (s)	Total (s)
Earth-Agent (GPT) [ 11 ]	158	42	200
Earth-Agent (DeepSeek-V3.1) [ 11 ]	51	28	79
Earth-Agent (KimiK2) [ 11 ]	105	27	132
Ours	0.84	0.34	1.18

Table 10: Comparison of inference time efficiency.

Figure 5: Qualitative results of RemoteAgent. The agent accurately interprets free-form queries and dynamically routes them to specialized tools, seamlessly bridging vague intents with precision-critical execution.

Tool-Using Vision-Language Agents

Tool-using vision-language agents combine visual and language models with external tools to gather evidence, act, and solve multi-step tasks.

See all 139 papers

OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents
Feb 19, 2026Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir +8Geospatial Reasoning Benchmarks Remote Sensing
Mohamed bin Zayed University of AI, United Arab Emirates · IBM Research, Europe · Linköping University, Sweden +2
RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent
Jun 11, 2024Wenjia Xu, Zijian Yu, Boyang Mu +3Multimodal RAG Multimodal Large Language Models
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China · City University of HongKong, Hong Kong 999077, China · School of Geographic Sciences, Hunan Normal University, Changsha 410081, China
Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents
Sep 11, 2026Zhutao Lv, Chenhao Dang, Yi Feng +5Remote Sensing AI Agent Benchmarks
Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-Sen University · Shanghai Jiao Tong University +3

Feb 19, 2026J/K move · Enter open · S save

OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir, Hiyam Debary+7

Mohamed bin Zayed University of AI, United Arab Emirates · IBM Research, Europe · Linköping University, Sweden+2

Geospatial Reasoning Benchmarks Remote Sensing Geospatial Reasoning Tool-Using Vision-Language Agents Visually Grounded Reasoning Spatial Reasoning

Open paper AskPDF arXiv

Recent progress in multimodal reasoning has enabled agents that interpret imagery, connect it with language, and execute structured analytical tasks. Extending these capabilities to remote sensing remains challenging, as models must reason over spatial scale, geographic structures, and multispectral indices while maintaining coherent multi-step logic. To address this gap, we introduce OpenEarthAgent, a unified framework for tool-augmented geospatial reasoning trained on satellite imagery, natural-language queries, and structured reasoning traces. Beyond serving as a benchmark, OpenEarthAgent establishes a cohesive agentic architecture built around a unified executable tool registry and trajectory-based policy learning. The framework standardizes heterogeneous visual, spectral, GIS, and georeferenced raster operations under a consistent callable schema, enabling modular orchestration and deterministic execution. Training is performed via supervised fine-tuning on structured reasoning trajectories with deterministic replay validation to ensure executability and spatial correctness. The accompanying corpus comprises 14,538 training and 1,169 evaluation instances with over 107K reasoning steps, spanning urban, environmental, disaster, and infrastructure domains and incorporating GIS operations alongside index analyses such as NDVI, NBR, and NDBI. Grounded in explicit reasoning traces, the learned agent demonstrates structured reasoning, stable spatial understanding, and interpretable tool-driven behaviour across diverse EO scenarios. We report consistent improvements over a strong baseline and competitive performance against recent open and closed-source models. Our code, data and trained models are publicly available: https://github.com/mbzuai-oryx/OpenEarthAgent

Abstract

Tool-Using Vision-Language Agents

Explore similar work

OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent

Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agents