TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering

Jingwen Huang; Yiming Fang; Hanxiao Wu; Boyuan He; Keyi Luo

cs.CVOct 7, 2026

TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering

Authors: Jingwen Huang, Yiming Fang, Hanxiao Wu, Boyuan He, Keyi Luo

Organizations: Yunnan University of Finance and Economics, Kunming, China

Abstract

Remote sensing visual question answering (RS-VQA) requires answers that truly depend on image evidence, yet current remote sensing multimodal large language models (RS MLLMs) are still evaluated mainly by accuracy, which cannot reveal whether a correct answer was produced without using the image. This setting introduces three challenges: demonstrations retrieved across different ground sample distances (GSD), sensors, and city textures may be poorly matched; real queries often involve long visual contexts from multi-scale tiles, multi-temporal frames, and retrieved knowledge, where evidence can be diluted and errors can propagate through multi-step reasoning; and prior-only answering remains invisible to standard metrics. We present AVA, a training-free framework that wraps a frozen RS MLLM with three modules. GDR performs GSD-bucketed, scene-graph-based demonstration retrieval and filters demonstrations whose answers are unsupported by their images. MACE decomposes each query into an evidence-typed sub-question DAG, routes sub-questions to suitable perception tools, and records verified provenance in a shared evidence ledger with bounded re-perception. CVA audits the final answer through counterfactual perturbations of cited evidence, including ablation and swap, and computes a visual-dependency score for confidence calibration and evidence-forcing re-query. On RSVQA-LR/HR, VRSBench, and LHRS-Bench, AVA improves average accuracy by up to 2.40 points over the strongest same-setting baselines, raises the visual-dependency score from 0.28 to 0.49, reduces the prior-only answer rate from 46.2 percent to 21.7 percent, and lowers the hallucination rate from 17.1 percent to 9.8 percent, with 2.6x the cost of a single backbone forward pass and no fine-tuning.

Figures & tables

Figure 1: Overview of TerraAgent. A frozen remote sensing multimodal LLM is wrapped by three training-free modules: GDR builds a scale-balanced, evidence-gated long context from retrieved demonstrations; MACE decomposes the question, routes sub-questions to perception tools, records typed evidence with provenance in a shared ledger and verifies every answer span; CVA perturbs the cited evidence (ablation and swap) to compute the visual-dependency score, which calibrates confidence and decides between acceptance, evidence-forcing re-query and abstention.

Method	Backbone	RSVQA-LR	RSVQA-HR
LLaVA-1.5 Liu et al. (2024b)	Vicuna-7B	65.86	53.99
MiniGPT-v2 Chen et al. (2023)	Vicuna-7B	59.77	66.53
InstructBLIP Dai et al. (2023)	Vicuna-7B	59.12	62.77
mPLUG-Owl2 Ye et al. (2023)	Llama-2-7B	65.24	53.04
Qwen-VL-Chat Bai et al. (2023)	Qwen-7B	58.73	63.87
InternLM-XComposer Zhang et al. (2023)	InternLM-7B	59.55	67.21

Table 1: Zero-shot RS-VQA accuracy on RSVQA-LR and RSVQA-HR (average over question types, %). Counting questions are omitted on LR and counting plus area questions on HR, following prior practice. Upper block: numbers as reported by the LHRS-Bot evaluation; lower block: TerraAgent on the same protocol.

Method	Backbone	All	Presence	Quantity	Reasoning
GeoChat (w/o ft) Kuckreja et al. (2023)	Vicuna-7B	40.8	48.5	85.9	57.4
MiniGPT-v2 Chen et al. (2023)	Vicuna-7B	38.2	61.3	26.0	21.9
GPT-4V Yang et al. (2023)	– (closed)	65.6	67.0	87.6	72.4
LLaVA-1.5 Liu et al. (2024b)	Vicuna-7B	76.4	86.9	91.8	73.4
GeoChat Kuckreja et al. (2023)	Vicuna-7B	76.0	86.5	92.1	73.5
Mini-Gemini Li et al. (2024b)	Vicuna-7B / Llama-3	77.8	87.8	92.1	74.4

Table 2: VRSBench visual question answering (official GPT-based open-set evaluation, %). “All” is the overall score. Presence, quantity and reasoning are shown because they exhibit the strongest disagreement across models.

Method	Identity	Color	Orientation	Shape	Area	Resolution	Modality	Location	Distance	Quantity	Reasoning	Avg
LLaVA-1.5 Liu et al. (2024b)	42.90	30.97	15.38	43.24	44.00	23.80	21.74	35.78	27.27	21.89	54.35	37.45
MiniGPT-v2 Chen et al. (2023)	20.19	15.93	2.56	8.11	13.33	14.29	4.35	15.20	13.64	9.49	36.96	16.88
InstructBLIP Dai et al. (2023)	20.35	22.12	5.13	24.32	22.67	38.10	0.00	13.73	22.85	13.14	30.43	18.87
mPLUG-Owl2 Ye et al. (2023)	41.48	45.13	12.82	40.54	32.00	33.33	0.00	30.88	13.64	32.12	54.34	37.00
Qwen-VL-Chat Bai et al. (2023)	31.38	32.74	7.69	32.43	29.33	19.05	4.35	24.02	21.45	21.17	56.52	28.65
InternLM-XComposer Zhang et al. (2023)	34.38	30.97	7.69	37.84	26.67	0.00	8.70	33.33	22.73	18.25	65.22	31.09

Table 3: LHRS-Bench single-choice accuracy (%) over eleven capability dimensions. Best per column in bold. The last column is the average. TerraAgent uses LHRS-Bot-7B as the frozen backbone here.

Configuration	VRSBench All	RSVQA-LR	LHRS-Bench Avg
Baseline (frozen RS MLLM, 4-shot random demos)	76.0	91.81	39.38
+GDR	78.1	92.60	41.20
+MACE	77.6	92.90	43.10
+CVA	76.9	92.20	40.50
+GDR+MACE	79.2	93.15	44.90
+GDR+CVA	78.6	92.75	41.85

Table 4: Module ablation. VRSBench and RSVQA use the GeoChat-7B backbone, LHRS-Bench uses LHRS-Bot-7B; the baseline row is the corresponding published zero-shot number. Each single module and each pair improves over the baseline, and the full configuration is best on all three suites.

Panel of Figure 2 — Figure 2: Left: TerraAgent improves every benchmark over the strongest comparable baseline. Right: longer demonstration contexts raise accuracy up to $K=6$ and then degrade it, consistent with visual evidence being diluted in long contexts.

Retrieval weight $\lambda$ (VRSBench All)	0.0	0.2	0.4	0.5	0.6	0.8	1.0
	77.2	78.6	79.5	80.2	79.8	78.4	77.6
Re-perception rounds $r$ (LHRS-Bench Avg)	0	1	2	3	4
	42.75	44.90	46.30	46.12	45.51
Gate threshold $\tau_{vd}$ (RSVQA-LR)	0.0	0.2	0.35	0.5	0.7
	92.10	92.95	93.40	93.02	92.25
Prior-only rate (%)	38.4	29.6	21.7	17.2	15.8

Table 5: Hyper-parameter sensitivity. Defaults are marked in bold and are optimal or near-optimal in every row.

Panel of Figure 3 — Figure 3: Left: after removing the cited evidence, single-pass models lose only a small part of their accuracy (VDS $0.11$ – $0.28$ , ESS $0.02$ – $0.04$ ), i.e. their correct answers are largely prior-driven; TerraAgent loses $46.2$ points (VDS $0.49$ , ESS $0.34$ ). Right: tightening the gate reduces the prior-only answer rate at a modest accuracy cost, with the default $0.35$ at the accuracy peak.

Method	Hallucination rate (%)	Unsupported spans (%)
GPT-4V Yang et al. (2023)	18.2	–
LLaVA-1.5 Liu et al. (2024b)	16.4	–
GeoChat-7B Kuckreja et al. (2023)	17.1	–
LHRS-Bot-7B Muhtar et al. (2024)	21.3	–
TerraAgent	9.8	11.4

Table 7: Faithfulness on a

2{,}000

-question VRSBench subset judged by an LLM. The unsupported-span ratio counts factual spans that the evidence ledger does not support, which is only defined for systems that expose provenance.

Method	Backbone	Latency (s)	Input tokens	Tool calls	Cost
GeoChat-7B Kuckreja et al. (2023)	Vicuna-7B	1.9	1.2k	0	$1.0\times$
LHRS-Bot-7B Muhtar et al. (2024)	Vicuna-7B	2.1	1.3k	0	$1.1\times$
GPT-4V Yang et al. (2023)	closed (API)	8.5	1.6k	0	–
TerraAgent (GDR+MACE+CVA)	Vicuna-7B (frozen)	4.6	3.4k	2.3	$2.6\times$

Table 8: Efficiency. Cost is measured in units of a single local backbone forward pass. TerraAgent is slower than a single pass but remains well below a closed-model API call, and it requires no fine-tuning.

Panel of Figure 4 — Figure 4: Left: TerraAgent leads on the perceptual dimensions (identity, shape, location, modality, reasoning) but trails closed models on resolution and distance, which are knowledge- rather than perception-bound. Right: the accuracy–cost frontier; the bubble area is proportional to input tokens.

Remote Sensing VQA

Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.

See all 35 papers

Mar 3, 2026

Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing

Multimodal large language models (MLLMs) suffer from pronounced hallucinations in remote sensing visual question-answering (RS-VQA), primarily caused by visual grounding failures in large-scale scenes or misinterpretation of fine-grained small targets. To systematically analyze these issues, we introduce RSHBench, a protocol-based benchmark for fine-grained diagnosis of factual and logical hallucinations. To mitigate grounding-induced factual hallucinations, we further propose Relative Attention-Driven Actively Reasoning (RADAR), a training-free inference method that leverages intrinsic attention in MLLMs to guide progressive localization and fine-grained local reasoning at test time. Extensive experiments across diverse MLLMs demonstrate that RADAR consistently improves RS-VQA performance and reduces both factual and logical hallucinations. Code and data will be publicly available at: https://github.com/MiliLab/RADAR

Yi Liu, Jing Zhang, Di Wang +3

School of Computer Science, Wuhan University, China · Zhongguancun Academy, China · School of Computer Science, Chongqing University, China

Remote Sensing VQA VLM Hallucination Mitigation Multimodal Large Language Models Vision-Language Grounding Hallucination Detection in VLMs Remote Sensing Benchmarks

Ask PDF

Oct 9, 2026

Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering

In grounded remote sensing visual question answering, a model produces a textual answer and identifies relevant objects in the image as grounding evidence. Existing evaluations measure answer correctness and localization quality, but do not directly assess answer-grounding consistency. We propose the Answer-Evidence Support Protocol (AESP) to assess this consistency through evidence support: whether the grounding evidence supports the model's own answer. AESP applies task-specific rules to existence, counting, and comparison questions. We further construct the Grounded Remote Sensing Visual Question Answering (G-RSVQA) benchmark, containing 15,027 samples with questions, reference answers, and all related object-level annotations. Experiments on seven grounded vision--language models reveal failures of evidence support across all three question types. Up to 50.82% of correct answers in the evidence evaluation subset are not supported by the provided grounding evidence.

Ping Guo, Zhiqi Huang, Xinran Li +3

Graduate School of Information, Production and Systems, Waseda University, Kitakyushu, Japan · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Software, Dalian University of Technology, Dalian, China +1

Remote Sensing VQA VLM Evaluation Remote Sensing Benchmarks Remote Sensing Visual Grounding

Ask PDF

Sep 28, 2026

ReVA: A Scene-Centric Dataset Beyond Repetition for Remote Sensing Video Question Answering

Multimodal Large Language Models (MLLMs) have demonstrated remarkable advances in remote sensing. However, existing remote sensing multimodal reasoning benchmarks exhibit two critical limitations: they rely on (i) template-driven questions, which causes repetitive questions; and (ii) static images that fail to capture the inherent temporal nature of drone/UAV videos. This leaves systematic evaluation of remote sensing video reasoning largely unexplored. To address this gap, we introduce ReVA, a new dataset for remote sensing video question answering, designed to assess spatiotemporal, scene-centric, and reasoning-oriented capabilities of MLLMs. ReVA comprises 2,438 drone videos spanning 18 cities worldwide (580K frames) and 22K high-quality question-answer pairs across 11 challenging QA tasks. We develop a semi-automatic annotation pipeline that leverages Text LLMs and MLLMs for question-answer generation with human verification. We evaluate 23 proprietary and open-source Video LLMs on ReVA, exposing fundamental limitations of current models. These findings position ReVA as a critical benchmark toward better remote sensing video understanding and temporal reasoning capabilities for real-world deployments. Our code and dataset are available at: https://github.com/zyaocoder/ReVA

Zhen Yao, Likai Wang, Yuming Yang +9

Lehigh University · University of Southern California · Qualcomm AI Research

Remote Sensing VQA Video QA VLM Reasoning Remote Sensing Benchmarks UAV Remote Sensing Temporal Video Understanding

Ask PDF

Abstract

Remote Sensing VQA

Explore similar work

Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing

Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering

ReVA: A Scene-Centric Dataset Beyond Repetition for Remote Sensing Video Question Answering