Selective Tool Use for Agentic Change Visual Question Answering in Remote Sensing
Authors: Yakoub Bazi, Mohamad M. Al Rahhal, Mohamed A. Mekhtiche, Mansour Zuair
Organizations: Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia · Applied Computer Science Department, College of Applied Computer Science, King Saud University, Riyadh 11543, Saudi Arabia
Change visual question answering (Change VQA) requires understanding semantic changes across bi-temporal remote sensing images. Although vision language models (VLMs) have shown promising performance on this task, they remain unreliable when answering questions that require explicit transition statistics, area measurements, or spatial information. To address this limitation, we propose a selective tool use framework in which a single VLM either answers directly or invokes a deterministic change analysis tool to obtain question specific evidence. Specifically, the selected tool operates on bi-temporal semantic maps and returns a structured observation, which the same VLM uses to generate its final answer. To support this framework, we construct a tool augmented extension of CDVQA covering eight question families and three tools for transition, spatial, and temporal analysis. Tool use supervision and observations are derived automatically from the original semantic annotations, without additional manual labeling. We then adapt Qwen3.5-4B using Low Rank Adaptation (LoRA) to jointly learn direct answering, tool invocation, and evidence conditioned answering. Experiments on 7,164 test questions show that selective tool use with reference semantic maps improves overall accuracy from 73.77% to 88.79% and average family accuracy from 69.11% to 89.65%. When the semantic maps are predicted automatically, the framework achieves 77.47% overall accuracy and 75.06% average family accuracy. These results demonstrate the benefit of question-specific semantic evidence for Change VQA, while highlighting the influence of semantic prediction quality on the resulting performance. Code and tool-augmented annotations will be made publicly available at https://github.com/yakoubbazi/ToolChangeVQA.
Figures & tables
Fig. 1: Overview of the proposed framework. A single VLM either answers a question directly or invokes a deterministic analysis tool. The tool computes question-specific evidence from bi-temporal semantic maps and returns a structured observation for final-answer generation. The semantic maps are not provided to the VLM as visual inputs.
Fig. 2: Representative examples from the tool-augmented CDVQA benchmark.
Family
Question objective
Tool
F1
change existence
direct
F2
increase/decrease
direct
F3
destination class
transition
F4
changed/stable proportion
transition
F5
largest/smallest change
transition
F6
location of new regions
spatial
TABLE I: Question families and tool-use supervision policy.
Split
Pairs
Questions
Direct
Tool
Q/pair
Train
2,077
33,299
14,621
18,678
16.0
Val
445
7,123
3,132
3,991
16.0
Test
446
7,164
3,194
3,970
16.1
Total
2,968
47,586
20,947
26,639
16.0
TABLE II: Statistics of the tool-augmented CDVQA benchmark. Direct and Tool denote direct-answer (F1–F2) and tool-dependent (F3–F8) questions, respectively.
Method
Evidence
F1
F2
F3
F4
F5
F6
F7
F8
OA
AA
Direct answer
–
88.60
84.74
69.06
50.34
47.20
52.42
80.17
80.38
73.77
69.11
Selective tool use
Pred.
88.36
82.06
74.50
63.40
48.69
66.46
83.51
93.46
77.47
75.06
Selective tool use
Ref.
88.36
82.06
97.77
99.83
59.51
99.76
89.87
100.00
88.79
89.65
TABLE III: Change VQA accuracy (%) on the test questions. Ref. and Pred. denote reference and predicted semantic maps, respectively. Bold values indicate the highest accuracy in each column.
Model
mIoU
SeK
Fscd
OA
CD Segmentation model [ 9 ]
73.08
25.42
65.29
87.38
TABLE IV: Semantic change detection performance on the test set (%). mIoU: mean intersection over union; SeK: separated kappa; Fscd : semantic change detection F1 score; OA: pixel-level overall accuracy.
Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.
Change visual question answering (Change VQA) addresses the problem of answering natural-language questions about semantic changes between bi-temporal remote sensing (RS) images. Although vision-language models (VLMs) have recently been studied for temporal RS image understanding, Change VQA remains underexplored in the context of modern multimodal models. In this letter, we revisit the CDVQA benchmark using recent Qwen models under a unified low-rank adaptation (LoRA) setting. We compare Qwen3-VL, which follows a structured vision-language pipeline with multi-depth visual conditioning and a full-attention decoder, with Qwen3.5, a native multimodal model that combines a single-stage alignment with a hybrid decoder backbone. Experimental results on the official CDVQA test splits show that recent VLMs improve over earlier specialized baselines. They further show that performance does not scale monotonically with model size, and that native multimodal models are more effective than structured vision-language pipelines for this task. These findings indicate that tightly integrated multimodal backbones contribute more to performance than scale or explicit multi-depth visual conditioning for language-driven semantic change reasoning in RS imagery.
Yakoub Bazi, Mohamad M. Al Rahhal, Mansour Zuair +1
Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia · Applied Computer Science Department, College of Applied Computer Science, King Saud University, Riyadh 11543, Saudi Arabia
Traditional change detection identifies where changes occur, but does not explain what changed in natural language. Existing remote sensing change captioning datasets typically describe overall image-level differences, leaving fine-grained localized semantic reasoning largely unexplored. To close this gap, we present RSRCC, a new benchmark for remote sensing change question-answering containing 126k questions, split into 87k training, 17.1k validation, and 22k test instances. Unlike prior datasets, RSRCC is built around localized, change-specific questions that require reasoning about a particular semantic change. To the best of our knowledge, this is the first remote sensing change question-answering benchmark designed explicitly for such fine-grained reasoning-based supervision. To construct RSRCC, we introduce a hierarchical semi-supervised curation pipeline that uses Best-of-N ranking as a critical final ambiguity-resolution stage. First, candidate change regions are extracted from semantic segmentation masks, then initially screened using an image-text embedding model, and finally validated through retrieval-augmented vision-language curation with Best-of-N ranking. This process enables scalable filtering of noisy and ambiguous candidates while preserving semantically meaningful changes. The dataset is available at https://huggingface.co/datasets/google/RSRCC.
While Multimodal Large Language Models (MLLMs) excel in general vision-language tasks, their application to remote sensing change understanding is hindered by a fundamental "temporal blindness". Existing architectures lack intrinsic mechanisms for multi-temporal contrastive reasoning and struggle with precise spatial grounding. To address this, we first introduce Delta-QA, a comprehensive benchmark comprising 180k visual question-answering samples. Delta-QA unifies pixel-level segmentation and visual question answering across bi- and tri-temporal scenarios, structuring change interpretation into four progressive cognitive dimensions. Methodologically, we propose Delta-LLaVA, a novel MLLM framework explicitly tailored for multi-temporal remote sensing interpretation. It overcomes the limitations of naive feature concatenation through three core innovations: a Change-Enhanced Attention module that systematically isolates and amplifies visual differences, a Change-SEG module utilizing Change Prior Embedding to extract differentiable difference features as input for the LLM, and Local Causal Attention to prevent cross-temporal contextual leakage. Extensive experiments demonstrate that Delta-LLaVA decisively outperforms leading generalist MLLMs and specialized segmentation models in complex change deduction and high-precision boundary localization, establishing a unified framework for earth observation intelligence.
Xiaohe Li, Jiahao Li, Kaixin Zhang +5
Beijing, China Aerospace Information Research Institute, CAS · Beijing, China Space Engineering University · Beijing, China Capital Normal University