GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
Authors: Qigan Sun, Chaoning Zhang, Jianwei Zhang, Xudong Wang, Jiehui Xie, Pengcheng Zheng, Haoyu Wang, Sungyoung Lee, Chi-lok Andy Tai, Yang Yang, Heng Tao Shen
Organizations: School of Computing, Kyung Hee University, Yongin-si, South Korea · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, China · College of Computer Science and Information Engineering, Harbin Normal University, Harbin, 150025, China · College of Professional and Continuing Education, The Hong Kong Polytechnic University, Hong Kong, China · School of Computer Science and Technology, Tongji University, Shanghai, China
In recent years, Multimodal Large Language Models (MLLMs) have made significant progress in visual question answering tasks. However, directly applying existing fine-tuning methods to remote sensing (RS) images often leads to issues such as overfitting on background noise or neglecting target details. This is primarily due to the large-scale variations, sparse target distributions, and complex regional semantic features inherent in RS images. These challenges limit the effectiveness of MLLMs in RS tasks. To address these challenges, we propose a parameter-efficient fine-tuning (PEFT) strategy called Guided Region-Aware Sparse Prompting (GRASP). GRASP introduces spatially structured soft prompts associated with spatial blocks extracted from a frozen visual token grid. Through a question-guided sparse fusion mechanism, GRASP dynamically aggregates task-specific context into a compact global prompt, enabling the model to focus on relevant regions while filtering out background noise. Extensive experiments on multiple RSVQA benchmarks show that GRASP achieves competitive performance compared to existing fine-tuning and prompt-based methods while maintaining high parameter efficiency.
Figures & tables
Fig. 1: Parameter efficiency vs. performance on Qwen2.5-VL-7B. The y-axis denotes the overall accuracy (mean of the AA scores on RSVQA-LR, RSVQA-HR, and RSIVQA), plotted against trainable parameters (log scale). Our method, GRASP, is represented by a red star. Note that proximity to the top-left corner corresponds to higher accuracy with lower parameter costs.
Fig. 2: Conceptual comparison between Conventional soft prompting and GRASP. (a) Conventional soft prompting: Relies on static, uniform prompts. It struggles to suppress background interference (e.g., E1 ), as it lacks region-aware adaptation. (b) GRASP: Utilizes question-guided sparse weighting to suppress irrelevant backgrounds ( E1 weight=0) and emphasize targets ( E4 weight=0.5). The resulting global prompt Pglobal is computed by weighted fusion over all regions, where regions with zero weights are ignored.
Fig. 3: Architecture of the GRASP framework. (a) Spatial Block-wise Soft Prompt Construction: The frozen vision encoder generates a visual token grid from the image. Average pooling partitions this grid into spatial blocks (e.g., E1 – E4 ), where each block Ei links to a learnable local soft prompt (SP). (b) Question-guided Sparse Soft Prompt Fusion: The frozen LLM derives a semantic vector q from the question. The system projects q and block-level features Ei into a shared space to compute relevance scores. Entmax activation converts these scores into sparse weights {wi} , which the model uses to aggregate local SPs into a task-specific global soft prompt Pglobal . (c) LLM-based Inference and Gradient Backpropagation: The framework injects Pglobal into the multimodal token sequence to guide the frozen LLM in answer generation. Training updates only the soft prompts and linear projection parameters.
Method
Params
RSVQA-LR
RSVQA-HR
RSIVQA
Count
R/U
Pres.
Comp.
AA
Count
Area
Pres.
Comp.
AA
Yes/No
Num.
Other
AA
LLaVA-Next (Base)
–
23.8
71.0
55.4
66.5
54.2
56.0
49.1
66.4
75.6
61.8
85.5
45.1
40.2
56.9
Additive parameter tuning methods
+ Adapter
16.7
28.0 (±1.0)
81.9 (±1.1)
74.6 (±1.3)
77.1 (±1.2)
65.4 (±1.0)
63.1 (±1.2)
68.8 (±1.4)
78.4 (±1.5)
80.9 (±1.3)
72.8 (±1.2)
90.3 (±1.1)
60.7 (±1.6)
76.3 (±1.8)
75.8 (±1.4)
+ LoRA
50.1
29.5 (±0.9)
84.0 (±1.0)
77.3 (±1.1)
79.0 (±1.0)
67.5 (±0.9)
64.2 (±1.0)
68.6 (±1.2)
78.6 (±1.3)
81.8 (±1.1)
73.3 (±1.0)
91.2 (±0.9)
62.0 (±1.4)
78.4 (±1.6)
77.2 (±1.2)
+ DoRA
50.2
30.1 (±0.8)
84.3 (±0.9)
77.4 (±1.0)
78.7 (±0.9)
67.6 (±0.8)
64.8 (±0.9)
68.5 (±1.1)
79.0 (±1.2)
82.1 (±1.0)
73.6 (±0.9)
91.2 (±0.8)
62.7 (±1.3)
78.7 (±1.5)
77.5 (±1.1)
TABLE I: Unified results of LLaVA-Next on three benchmarks. Params (M) denotes trainable parameters (millions). Count, R/U, Pres., and Comp. correspond to different question categories, namely Counting, Rural/Urban, Presence, and Comparison. AA denotes Average Accuracy. Green values in parentheses denote the standard deviation over 3 independent runs. Bold and underline mark the best and second-best results.
Method
Params
RSVQA-LR
RSVQA-HR
RSIVQA
Count
R/U
Pres.
Comp.
AA
Count
Area
Pres.
Comp.
AA
Yes/No
Num.
Other
AA
Qwen2.5-VL (Base)
–
20.9
66.0
62.3
71.4
55.1
52.6
61.4
65.6
76.4
64.0
87.3
43.8
38.7
56.6
Additive parameter tuning methods
+ Adapter
16.7
27.6 (±1.1)
81.4 (±1.1)
79.0 (±1.2)
80.5 (±1.2)
67.1 (±1.0)
60.3 (±1.3)
69.7 (±1.4)
66.4 (±1.5)
80.3 (±1.2)
69.2 (±1.3)
90.5 (±1.1)
58.0 (±1.5)
82.8 (±1.7)
77.1 (±1.4)
+ LoRA
50.1
29.5 (±0.9)
83.0 (±1.0)
80.8 (±1.1)
81.3 (±1.0)
68.7 (±0.9)
62.1 (±1.1)
70.6 (±1.2)
70.3 (±1.3)
81.9 (±1.1)
71.2 (±1.0)
92.0 (±0.9)
60.9 (±1.4)
89.2 (±1.6)
80.7 (±1.2)
+ DoRA
50.2
30.0 (±0.8)
82.9 (±0.9)
81.2 (±1.0)
81.1 (±0.9)
68.8 (±0.8)
62.4 (±1.0)
70.8 (±1.1)
71.0 (±1.2)
82.1 (±1.0)
71.6 (±0.9)
91.7 (±0.8)
61.1 (±1.3)
89.0 (±1.5)
80.6 (±1.1)
TABLE II: Unified results of Qwen2.5-VL-7B on three benchmarks. Params (M) denotes trainable parameters (millions). The abbreviations follow the same definitions as in Table I . Green values in parentheses denote the standard deviation over 3 independent runs. Bold and underline mark the best and second-best results.
Method
Objects
Features
Spatial
Quantity
Presence
Avg.
Qwen2.5-VL (Base)
58.6
74.2
57.9
28.3
46.3
53.1
Additive parameter tuning methods
+ Adapter
63.2 (±1.1)
77.3 (±1.2)
62.4 (±1.4)
34.2 (±1.5)
54.1 (±1.2)
58.2 (±0.9)
+ LoRA
71.4 (±0.8)
80.1 (±1.0)
65.8 (±1.1)
36.1 (±1.3)
68.4 (±0.9)
64.4 (±0.7)
+ DoRA
72.2 (±0.7)
79.3 (±0.9)
67.9 (±1.0)
37.2 (±1.2)
67.8 (±1.0)
64.9 (±0.6)
Prompt-based tuning methods
TABLE III: VQA performance on RSVLM-QA. All answer correctness is judged automatically by GPT-4.1. Avg. denotes the average accuracy over the five VQA categories. Green values in parentheses denote the standard deviation over 3 independent runs. Bold and underline mark the best and second-best results.
Fig. 4: Parameter efficiency vs. performance on LLaVA-Next. The y-axis denotes the overall accuracy (mean of the AA scores on RSVQA-LR, RSVQA-HR, and RSIVQA), plotted against trainable parameters (log scale). Our method, GRASP, is represented by a red star. Note that proximity to the top-left corner corresponds to higher accuracy with lower parameter costs.
Model
RSVQA-LR
RSVQA-HR
R/U
Pres.
Comp.
AA (%)
Pres.
Comp.
AA (%)
GeoChat
91.0
90.3
94.0
91.7
58.4
83.19
72.3
EarthGPT
—
—
—
—
62.7
79.53
72.0
SkyEyeGPT
88.9
88.6
75.0
84.1
80.0
80.1
80.0
GRASP (Ours)
85.7
77.5
78.8
80.6
79.2
82.3
80.8
TABLE IV: Reference comparison with representative RS-MLLMs using publicly reported RSVQA results. The AA values in this table are calculated only over the question categories shared by the compared models and are therefore not directly equivalent to the AA values reported in the main experimental tables. GRASP results are obtained with the LLaVA-NeXT-7B backbone. Gray-colored numbers denote zero-shot results; “—” indicates that the metric is not provided. This table is presented as a reference comparison rather than a strictly controlled head-to-head evaluation.
Fig. 5: Visualization of Question-guided Sparse Fusion Weights across different input resolutions. The top row shows the 4×4 grid weights for the 512×512 input, while the bottom row displays the 2×2 grid for the 224×224 input. These weights represent the contribution of each spatial block to the final representation, with answers highlighted in green denoting correct predictions. The displayed answers are raw free-form model outputs.
Fig. 6: Impact of spatial block count ( N ) on performance with the Qwen2.5-VL-7B backbone across RSVQA-LR, RSVQA-HR, and RSIVQA datasets. The ’Token’ marker ( ⋆ ) denotes the original patch-level baseline, while the solid lines track the performance trend under our proposed spatial blocking strategy.
Fig. 7: Inference latency versus the number of spatial blocks N on RSVQA-LR, RSVQA-HR, and RSIVQA. Latency is reported as the average inference time per sample.
Fig. 8: Visualization of sparse weight distributions under different input resolutions. The two rows present the token-level baseline and our GRASP across 512×512 and 224×224 scales, respectively. For visual clarity, weight magnitudes are independently normalized to their respective local maximums.
α
RSVQA-LR
RSVQA-HR
RSIVQA
Count
R/U
Pres.
Comp.
AA
Count
Area
Pres.
Comp.
AA
Yes/No
Num.
Other
AA
1.0
28.4
79.2
78.0
78.1
65.9
59.5
68.0
70.5
78.8
69.2
91.0
58.7
82.6
77.4
1.2
29.6
81.0
79.7
79.9
67.6
61.1
69.4
72.0
80.5
70.8
91.8
60.0
86.2
79.3
1.5
31.1
82.6
81.6
81.5
69.2
62.7
71.0
73.9
82.5
72.5
93.1
61.5
89.0
81.2
1.8
30.5
81.5
80.5
80.6
68.3
61.8
70.1
72.5
81.2
71.4
92.6
60.7
88.1
80.5
2.0
29.3
80.1
78.9
79.3
66.9
60.5
68.7
71.4
79.9
70.1
92.2
60.3
86.8
79.8
TABLE V: Unified ablation on Entmax sparsity parameter α for GRASP on Qwen2.5-VL-7B across RSVQA-LR, RSVQA-HR, and RSIVQA datasets. α=1.0 and α=2.0 correspond to Softmax and Sparsemax, respectively. Results are reported as accuracy (%). The block number N is fixed to the best setting from the block-number ablation (RSVQA-LR/RSIVQA: N=4 ; RSVQA-HR: N=16 ).
Metric
GRASP (Baseline)
Uniform Prompts
RSVQA-LR
Count
31.1
26.4 (-4.7)
R/U
82.6
78.3 (-4.3)
Pres.
81.6
79.0 (-2.6)
Comp.
81.5
77.9 (-3.6)
AA
69.2
65.4 (-3.8)
TABLE VI: Ablation results comparing original GRASP (baseline) and the uniform soft prompt variant (without question guidance) on Qwen2.5-VL-7B. Red values indicate performance drops relative to GRASP. Bold marks the best results.
Method
EuroSAT (G/O)
UCMerced (G/O)
AID (G/O)
Qwen2.5-VL (Base)
48.2 / 19.5
65.8 / 14.3
58.5 / 13.1
Prompt Tuning
55.1 / 22.4
66.5 / 18.0
60.3 / 19.4
APrompt
59.5 / 28.7
70.4 / 20.6
62.1 / 21.5
VPT
58.4 / 30.3
69.2 / 26.4
65.7 / 32.8
LoRA
64.1 / 35.2
76.5 / 31.4
67.0 / 38.7
DoRA
65.6 / 36.3
77.2 / 32.0
67.4 / 38.2
TABLE VII: Scene Classification Performance (Accuracy %) under Guided / Open-ended (G/O) Prompts. Guided: with candidate options; Open-ended: without options. Bold indicates the best performance.
TABLE VIII: Performance comparison on Visual Grounding task across VRSBench and DIOR-RSVG datasets. Bold indicates the best performance.
Model
RSVQA-LR
RSVQA-HR
R/U
Pres.
Comp.
AA (%)
Pres.
Comp.
AA (%)
Mini-InternVL-4B
64.7
63.1
71.8
66.5
61.0
78.0
69.5
Prompt Tuning
66.2
64.5
73.0
67.9
62.5
79.4
71.0
APrompt
67.5
65.8
74.2
69.2
63.8
80.7
72.3
VPT
69.4
66.5
75.1
70.0
65.2
81.5
73.4
LoRA
73.8
72.5
82.4
76.2
75.6
88.3
82.0
TABLE IX: Scalability analysis on Mini-InternVL-4B. Bold indicates the best performance for each dataset. Count and Area tasks are excluded following the official evaluation protocol.
Method
RSVQA-LR → RSIVQA
RSIVQA → RSVQA-LR
In-domain
Cross-dataset
Δ
In-domain
Cross-dataset
Δ
Prompt Tuning
61.1
48.3
-12.8
71.5
55.8
-15.7
VPT
67.5
57.2
-10.3
77.9
65.2
-12.7
LoRA
68.7
60.9
-7.8
80.7
72.1
-8.6
GRASP
69.2
62.3
-6.9
81.2
74.4
-6.8
TABLE X: Cross-dataset generalization results. Δ is defined as cross-dataset accuracy minus in-domain accuracy. Negative values indicate performance degradation after cross-dataset transfer; values closer to zero indicate better robustness.
RSVQA-LR
Strategy
Count.
R/U
Pres.
Comp.
AA (%)
Lat. (ms)
Fixed Grid
31.1
82.6
81.6
81.5
69.2
112
Adaptive
32.0
83.0
81.9
82.1
69.7
188
RSVQA-HR
Strategy
Count.
Area
Pres.
Comp.
AA (%)
Lat. (ms)
Fixed Grid
62.7
71.0
73.9
82.5
72.5
194
TABLE XI: Systematic analysis of spatial partition configurations on RSVQA-LR and RSVQA-HR datasets. Bold indicates the best performance for each dataset. Latency is reported as the average inference time per sample.
Remote sensing visual question answering (VQA) studies how AI answers natural-language questions about satellite, aerial, and other Earth-observation imagery.
School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, China · Intelligent Game and Decision Lab (IGDL), Beijing, 100091, China · cYuelushan Center for Industrial Innovation, Changsha, 410082, China
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China · City University of HongKong, Hong Kong 999077, China · School of Geographic Sciences, Hunan Normal University, Changsha 410081, China