GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
Authors: Qigan Sun, Chaoning Zhang, Jianwei Zhang, Xudong Wang, Jiehui Xie, Pengcheng Zheng, Haoyu Wang, Sungyoung Lee, Chi-lok Andy Tai, Yang Yang, Heng Tao Shen
Organizations: School of Computing, Kyung Hee University, Yongin-si, South Korea · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, China · College of Computer Science and Information Engineering, Harbin Normal University, Harbin, 150025, China · College of Professional and Continuing Education, The Hong Kong Polytechnic University, Hong Kong, China · School of Computer Science and Technology, Tongji University, Shanghai, China
In recent years, Multimodal Large Language Models (MLLMs) have made significant progress in visual question answering tasks. However, directly applying existing fine-tuning methods to remote sensing (RS) images often leads to issues such as overfitting on background noise or neglecting target details. This is primarily due to the large-scale variations, sparse target distributions, and complex regional semantic features inherent in RS images. These challenges limit the effectiveness of MLLMs in RS tasks. To address these challenges, we propose a parameter-efficient fine-tuning (PEFT) strategy called Guided Region-Aware Sparse Prompting (GRASP). GRASP introduces spatially structured soft prompts associated with spatial blocks extracted from a frozen visual token grid. Through a question-guided sparse fusion mechanism, GRASP dynamically aggregates task-specific context into a compact global prompt, enabling the model to focus on relevant regions while filtering out background noise. Extensive experiments on multiple RSVQA benchmarks show that GRASP achieves competitive performance compared to existing fine-tuning and prompt-based methods while maintaining high parameter efficiency.
Figures & tables
Fig. 1: Parameter efficiency vs. performance on Qwen2.5-VL-7B. The y-axis denotes the overall accuracy (mean of the AA scores on RSVQA-LR, RSVQA-HR, and RSIVQA), plotted against trainable parameters (log scale). Our method, GRASP, is represented by a red star. Note that proximity to the top-left corner corresponds to higher accuracy with lower parameter costs.
Fig. 2: Conceptual comparison between Conventional soft prompting and GRASP. (a) Conventional soft prompting: Relies on static, uniform prompts. It struggles to suppress background interference (e.g., E1 ), as it lacks region-aware adaptation. (b) GRASP: Utilizes question-guided sparse weighting to suppress irrelevant backgrounds ( E1 weight=0) and emphasize targets ( E4 weight=0.5). The resulting global prompt Pglobal is computed by weighted fusion over all regions, where regions with zero weights are ignored.
Fig. 3: Architecture of the GRASP framework. (a) Spatial Block-wise Soft Prompt Construction: The frozen vision encoder generates a visual token grid from the image. Average pooling partitions this grid into spatial blocks (e.g., E1 – E4 ), where each block Ei links to a learnable local soft prompt (SP). (b) Question-guided Sparse Soft Prompt Fusion: The frozen LLM derives a semantic vector q from the question. The system projects q and block-level features Ei into a shared space to compute relevance scores. Entmax activation converts these scores into sparse weights {wi} , which the model uses to aggregate local SPs into a task-specific global soft prompt Pglobal . (c) LLM-based Inference and Gradient Backpropagation: The framework injects Pglobal into the multimodal token sequence to guide the frozen LLM in answer generation. Training updates only the soft prompts and linear projection parameters.
Method
Params
RSVQA-LR
RSVQA-HR
RSIVQA
Count
R/U
Pres.
Comp.
AA
Count
Area
Pres.
Comp.
AA
Yes/No
Num.
Other
AA
LLaVA-Next (Base)
–
23.8
71.0
55.4
66.5
54.2
56.0
49.1
66.4
75.6
61.8
85.5
45.1
40.2
56.9
Additive parameter tuning methods
+ Adapter
16.7
28.0 (±1.0)
81.9 (±1.1)
74.6 (±1.3)
77.1 (±1.2)
65.4 (±1.0)
63.1 (±1.2)
68.8 (±1.4)
78.4 (±1.5)
80.9 (±1.3)
72.8 (±1.2)
90.3 (±1.1)
60.7 (±1.6)
76.3 (±1.8)
75.8 (±1.4)
+ LoRA
50.1
29.5 (±0.9)
84.0 (±1.0)
77.3 (±1.1)
79.0 (±1.0)
67.5 (±0.9)
64.2 (±1.0)
68.6 (±1.2)
78.6 (±1.3)
81.8 (±1.1)
73.3 (±1.0)
91.2 (±0.9)
62.0 (±1.4)
78.4 (±1.6)
77.2 (±1.2)
+ DoRA
50.2
30.1 (±0.8)
84.3 (±0.9)
77.4 (±1.0)
78.7 (±0.9)
67.6 (±0.8)
64.8 (±0.9)
68.5 (±1.1)
79.0 (±1.2)
82.1 (±1.0)
73.6 (±0.9)
91.2 (±0.8)
62.7 (±1.3)
78.7 (±1.5)
77.5 (±1.1)
TABLE I: Unified results of LLaVA-Next on three benchmarks. Params (M) denotes trainable parameters (millions). Count, R/U, Pres., and Comp. correspond to different question categories, namely Counting, Rural/Urban, Presence, and Comparison. AA denotes Average Accuracy. Green values in parentheses denote the standard deviation over 3 independent runs. Bold and underline mark the best and second-best results.
Method
Params
RSVQA-LR
RSVQA-HR
RSIVQA
Count
R/U
Pres.
Comp.
AA
Count
Area
Pres.
Comp.
AA
Yes/No
Num.
Other
AA
Qwen2.5-VL (Base)
–
20.9
66.0
62.3
71.4
55.1
52.6
61.4
65.6
76.4
64.0
87.3
43.8
38.7
56.6
Additive parameter tuning methods
+ Adapter
16.7
27.6 (±1.1)
81.4 (±1.1)
79.0 (±1.2)
80.5 (±1.2)
67.1 (±1.0)
60.3 (±1.3)
69.7 (±1.4)
66.4 (±1.5)
80.3 (±1.2)
69.2 (±1.3)
90.5 (±1.1)
58.0 (±1.5)
82.8 (±1.7)
77.1 (±1.4)
+ LoRA
50.1
29.5 (±0.9)
83.0 (±1.0)
80.8 (±1.1)
81.3 (±1.0)
68.7 (±0.9)
62.1 (±1.1)
70.6 (±1.2)
70.3 (±1.3)
81.9 (±1.1)
71.2 (±1.0)
92.0 (±0.9)
60.9 (±1.4)
89.2 (±1.6)
80.7 (±1.2)
+ DoRA
50.2
30.0 (±0.8)
82.9 (±0.9)
81.2 (±1.0)
81.1 (±0.9)
68.8 (±0.8)
62.4 (±1.0)
70.8 (±1.1)
71.0 (±1.2)
82.1 (±1.0)
71.6 (±0.9)
91.7 (±0.8)
61.1 (±1.3)
89.0 (±1.5)
80.6 (±1.1)
TABLE II: Unified results of Qwen2.5-VL-7B on three benchmarks. Params (M) denotes trainable parameters (millions). The abbreviations follow the same definitions as in Table I . Green values in parentheses denote the standard deviation over 3 independent runs. Bold and underline mark the best and second-best results.
Method
Objects
Features
Spatial
Quantity
Presence
Avg.
Qwen2.5-VL (Base)
58.6
74.2
57.9
28.3
46.3
53.1
Additive parameter tuning methods
+ Adapter
63.2 (±1.1)
77.3 (±1.2)
62.4 (±1.4)
34.2 (±1.5)
54.1 (±1.2)
58.2 (±0.9)
+ LoRA
71.4 (±0.8)
80.1 (±1.0)
65.8 (±1.1)
36.1 (±1.3)
68.4 (±0.9)
64.4 (±0.7)
+ DoRA
72.2 (±0.7)
79.3 (±0.9)
67.9 (±1.0)
37.2 (±1.2)
67.8 (±1.0)
64.9 (±0.6)
Prompt-based tuning methods
TABLE III: VQA performance on RSVLM-QA. All answer correctness is judged automatically by GPT-4.1. Avg. denotes the average accuracy over the five VQA categories. Green values in parentheses denote the standard deviation over 3 independent runs. Bold and underline mark the best and second-best results.
Fig. 4: Parameter efficiency vs. performance on LLaVA-Next. The y-axis denotes the overall accuracy (mean of the AA scores on RSVQA-LR, RSVQA-HR, and RSIVQA), plotted against trainable parameters (log scale). Our method, GRASP, is represented by a red star. Note that proximity to the top-left corner corresponds to higher accuracy with lower parameter costs.
Model
RSVQA-LR
RSVQA-HR
R/U
Pres.
Comp.
AA (%)
Pres.
Comp.
AA (%)
GeoChat
91.0
90.3
94.0
91.7
58.4
83.19
72.3
EarthGPT
—
—
—
—
62.7
79.53
72.0
SkyEyeGPT
88.9
88.6
75.0
84.1
80.0
80.1
80.0
GRASP (Ours)
85.7
77.5
78.8
80.6
79.2
82.3
80.8
TABLE IV: Reference comparison with representative RS-MLLMs using publicly reported RSVQA results. The AA values in this table are calculated only over the question categories shared by the compared models and are therefore not directly equivalent to the AA values reported in the main experimental tables. GRASP results are obtained with the LLaVA-NeXT-7B backbone. Gray-colored numbers denote zero-shot results; “—” indicates that the metric is not provided. This table is presented as a reference comparison rather than a strictly controlled head-to-head evaluation.
Fig. 5: Visualization of Question-guided Sparse Fusion Weights across different input resolutions. The top row shows the 4×4 grid weights for the 512×512 input, while the bottom row displays the 2×2 grid for the 224×224 input. These weights represent the contribution of each spatial block to the final representation, with answers highlighted in green denoting correct predictions. The displayed answers are raw free-form model outputs.
Fig. 6: Impact of spatial block count ( N ) on performance with the Qwen2.5-VL-7B backbone across RSVQA-LR, RSVQA-HR, and RSIVQA datasets. The ’Token’ marker ( ⋆ ) denotes the original patch-level baseline, while the solid lines track the performance trend under our proposed spatial blocking strategy.
Fig. 7: Inference latency versus the number of spatial blocks N on RSVQA-LR, RSVQA-HR, and RSIVQA. Latency is reported as the average inference time per sample.
Fig. 8: Visualization of sparse weight distributions under different input resolutions. The two rows present the token-level baseline and our GRASP across 512×512 and 224×224 scales, respectively. For visual clarity, weight magnitudes are independently normalized to their respective local maximums.
α
RSVQA-LR
RSVQA-HR
RSIVQA
Count
R/U
Pres.
Comp.
AA
Count
Area
Pres.
Comp.
AA
Yes/No
Num.
Other
AA
1.0
28.4
79.2
78.0
78.1
65.9
59.5
68.0
70.5
78.8
69.2
91.0
58.7
82.6
77.4
1.2
29.6
81.0
79.7
79.9
67.6
61.1
69.4
72.0
80.5
70.8
91.8
60.0
86.2
79.3
1.5
31.1
82.6
81.6
81.5
69.2
62.7
71.0
73.9
82.5
72.5
93.1
61.5
89.0
81.2
1.8
30.5
81.5
80.5
80.6
68.3
61.8
70.1
72.5
81.2
71.4
92.6
60.7
88.1
80.5
2.0
29.3
80.1
78.9
79.3
66.9
60.5
68.7
71.4
79.9
70.1
92.2
60.3
86.8
79.8
TABLE V: Unified ablation on Entmax sparsity parameter α for GRASP on Qwen2.5-VL-7B across RSVQA-LR, RSVQA-HR, and RSIVQA datasets. α=1.0 and α=2.0 correspond to Softmax and Sparsemax, respectively. Results are reported as accuracy (%). The block number N is fixed to the best setting from the block-number ablation (RSVQA-LR/RSIVQA: N=4 ; RSVQA-HR: N=16 ).
Metric
GRASP (Baseline)
Uniform Prompts
RSVQA-LR
Count
31.1
26.4 (-4.7)
R/U
82.6
78.3 (-4.3)
Pres.
81.6
79.0 (-2.6)
Comp.
81.5
77.9 (-3.6)
AA
69.2
65.4 (-3.8)
TABLE VI: Ablation results comparing original GRASP (baseline) and the uniform soft prompt variant (without question guidance) on Qwen2.5-VL-7B. Red values indicate performance drops relative to GRASP. Bold marks the best results.
Method
EuroSAT (G/O)
UCMerced (G/O)
AID (G/O)
Qwen2.5-VL (Base)
48.2 / 19.5
65.8 / 14.3
58.5 / 13.1
Prompt Tuning
55.1 / 22.4
66.5 / 18.0
60.3 / 19.4
APrompt
59.5 / 28.7
70.4 / 20.6
62.1 / 21.5
VPT
58.4 / 30.3
69.2 / 26.4
65.7 / 32.8
LoRA
64.1 / 35.2
76.5 / 31.4
67.0 / 38.7
DoRA
65.6 / 36.3
77.2 / 32.0
67.4 / 38.2
TABLE VII: Scene Classification Performance (Accuracy %) under Guided / Open-ended (G/O) Prompts. Guided: with candidate options; Open-ended: without options. Bold indicates the best performance.
TABLE VIII: Performance comparison on Visual Grounding task across VRSBench and DIOR-RSVG datasets. Bold indicates the best performance.
Model
RSVQA-LR
RSVQA-HR
R/U
Pres.
Comp.
AA (%)
Pres.
Comp.
AA (%)
Mini-InternVL-4B
64.7
63.1
71.8
66.5
61.0
78.0
69.5
Prompt Tuning
66.2
64.5
73.0
67.9
62.5
79.4
71.0
APrompt
67.5
65.8
74.2
69.2
63.8
80.7
72.3
VPT
69.4
66.5
75.1
70.0
65.2
81.5
73.4
LoRA
73.8
72.5
82.4
76.2
75.6
88.3
82.0
TABLE IX: Scalability analysis on Mini-InternVL-4B. Bold indicates the best performance for each dataset. Count and Area tasks are excluded following the official evaluation protocol.
Method
RSVQA-LR → RSIVQA
RSIVQA → RSVQA-LR
In-domain
Cross-dataset
Δ
In-domain
Cross-dataset
Δ
Prompt Tuning
61.1
48.3
-12.8
71.5
55.8
-15.7
VPT
67.5
57.2
-10.3
77.9
65.2
-12.7
LoRA
68.7
60.9
-7.8
80.7
72.1
-8.6
GRASP
69.2
62.3
-6.9
81.2
74.4
-6.8
TABLE X: Cross-dataset generalization results. Δ is defined as cross-dataset accuracy minus in-domain accuracy. Negative values indicate performance degradation after cross-dataset transfer; values closer to zero indicate better robustness.
RSVQA-LR
Strategy
Count.
R/U
Pres.
Comp.
AA (%)
Lat. (ms)
Fixed Grid
31.1
82.6
81.6
81.5
69.2
112
Adaptive
32.0
83.0
81.9
82.1
69.7
188
RSVQA-HR
Strategy
Count.
Area
Pres.
Comp.
AA (%)
Lat. (ms)
Fixed Grid
62.7
71.0
73.9
82.5
72.5
194
TABLE XI: Systematic analysis of spatial partition configurations on RSVQA-LR and RSVQA-HR datasets. Bold indicates the best performance for each dataset. Latency is reported as the average inference time per sample.
Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.
The rapid development of multimodal large language models (MLLMs) has introduced a flexible paradigm for remote sensing image scene understanding (RSISU), enabling natural-language interaction with remote sensing imagery. However, a systematic understanding of the capability boundaries, cross-task generalization, and task-specific limitations of existing remote sensing MLLMs (RS-MLLMs) is still lacking. This paper presents a systematic survey and diagnostic evaluation of MLLMs for RSISU. We review the technical evolution of RS-MLLMs, focusing on model design, multimodal learning, training data, and downstream capabilities. We further compare RS-MLLMs with general-purpose computer vision MLLMs (CV-MLLMs) across diverse RSISU tasks and benchmarks. RS-MLLMs remain competitive in domain-specific settings, particularly remote sensing visual grounding and high-resolution visual question answering. More notably, general-purpose CV-MLLMs can match or even outperform these specialized models on several RSISU tasks without remote sensing-specific fine-tuning. These findings demonstrate the strong transferability of general-purpose CV-MLLMs and show that current RS-MLLMs do not consistently outperform them across diverse RSISU tasks. Current MLLMs also face limitations in spatial and relational reasoning, fine-grained visual understanding, instruction diversity, and generalization across heterogeneous task formats. Based on these findings, we outline future directions toward reliable evaluation, multimodal and high-resolution reasoning, efficient deployment, and tool-augmented remote sensing agents. This survey provides a systematic reference for developing robust, generalizable, and practical MLLMs for RSISU.
Qiwei Ma, Chunping Qiu, Xinjun Cheng +5
School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, China · Intelligent Game and Decision Lab (IGDL), Beijing, 100091, China · cYuelushan Center for Industrial Innovation, Changsha, 410082, China
Recent advances in Multimodal Large Language Models (MLLMs) have shown promise for remote sensing tasks such as visual question answering and scene understanding. However, existing models remain limited to basic instruction-following and struggle with real-world scenarios that require multi-source data integration, fine-grained spatial reasoning, and domain expertise. To address this gap, we propose RS-Agent, a domain-adapted intelligent agent that connects user intent with professional remote sensing workflows through structured task planning and tool orchestration. RS-Agent consists of four components aligned with typical remote sensing workflows: a Central Controller for intent understanding and process planning, a dynamic toolkit for tool execution, a Solution Space for task-specific expert guidance, and a Knowledge Space for domain knowledge support. We further introduce Task-Aware Retrieval, which improves planning by identifying task types and retrieving expert-defined solutions, and DualRAG, a weighted dual-path retrieval-augmented generation method that enhances the relevance and completeness of retrieved knowledge. RS-Agent natively supports multiple imaging modalities, including optical and SAR imagery, and can automatically organize dedicated SAR processing tools into executable workflows. Experiments on 9 datasets and 18 remote sensing tasks show that RS-Agent significantly outperforms state-of-the-art MLLMs, achieving over 95% task planning accuracy and strong results in scene classification, object counting, and remote sensing visual question answering. These results demonstrate the value of combining LLM reasoning with remote sensing expertise for intelligent geospatial analysis.
Wenjia Xu, Zijian Yu, Boyang Mu +3
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China · City University of HongKong, Hong Kong 999077, China · School of Geographic Sciences, Hunan Normal University, Changsha 410081, China
The core objective of image captioning is to achieve lossless semantic compression from visual signals into textual modalities. However, the reliance on manually curated reference texts for evaluation essentially forces models to mimic specific human annotation styles, thereby masking the true descriptive capabilities of advanced foundation models. This systemic misalignment prompts a critical question: Is task-specific fine-tuning truly necessary for Remote Sensing Image Captioning, or is the perceived performance gap merely an artifact of flawed evaluation criteria? To investigate this discrepancy, we propose ReconScore, a novel reference-free evaluation metric. Rather than computing textual similarities, we assess caption quality by its capability to reconstruct the original visual elements solely from the generated text, effectively neutralizing human annotation biases. Applying this metric, we uncover a profound, counterintuitive truth: inherently powerful, unfine-tuned MLLMs surpass their fine-tuned counterparts in authentic zero-shot RSIC tasks. Driven by this structural discovery, we introduce RemoteDescriber, a completely training-free generation methodology. By employing ReconScore as a self-correction mechanism, we iteratively refine the semantic precision of MLLM outputs without any computational fine-tuning overhead. Comprehensive experiments demonstrate that RemoteDescriber achieves state-of-the-art performance on three datasets. Furthermore, we validate ReconScore's reliability and analyze the flaws of traditional metrics. Our code is available at https://github.com/hhu-czy/RemoteDescriber.
Ziyun Chen, Fan Liu, Liang Yao +3
Hohai University Nanjing, China · Cardiff University Cardiff, UK