Gradient-Guided Decoupled Adaptation for Geospatial Vision-Language Models

Dongdong Wang; Deepak Balakrishnan; Ravi Srinivasan; Shenhao Wang

cs.CVSep 26, 2026

Gradient-Guided Decoupled Adaptation for Geospatial Vision-Language Models

Authors: Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan, Shenhao Wang

Organizations: University of Florida Gainesville, FL, USA

Abstract

Existing geospatial vision-language models (Geo-VLMs) typically optimize diverse geospatial tasks through a unified multi-task adaptation paradigm without explicitly accounting for the heterogeneous optimization characteristics. Our empirical observations reveal heterogeneous gradient characteristics across tasks, including vision-language differences, intra-branch gradient relationships, and task interference, which hinder effective multi-task optimization. Motivated by these observations, we propose Gradient-Guided Decoupled Adaptation (G2DA), a gradient-aware optimization framework for multi-task Geo-VLM learning. G2DA first partitions tasks into vision- and language-centric groups through gradient-guided cross-modal decoupling. It then constructs modality-specific curricula based on task gradient similarity and employs bidirectional rehearsal to mitigate the recency effects introduced by sequential optimization. We evaluate G2DA on three Geo-VLM benchmarks using six InternVL3 and Qwen3.5-VL variants, along with GeoChat and GeoLLaVA. Across all 24 benchmark-model combinations, G2DA consistently outperforms representative baselines, improving over the strongest competitor by 3.08, 4.30, and 2.81 percentage points on UrBench-MCQ, XLRS-Bench-Lite, and VRS-Bench-VQA, respectively. These results demonstrate the effectiveness of gradient-guided task organization for Geo-VLM adaptation.

Figures & tables

Figure 1: Task-gradient analysis on UrBench-MCQ. (a) Pairwise task-gradient cosine similarity matrix. Positive (red) and negative (blue) values indicate aligned and conflicting optimization directions. (b) Normalized proportions of the total gradient norm contributed by the vision (green) and language (pink) encoders for each task.

Figure 2: Task-level gradient cosine similarity after cross-modal decoupling and curriculum ordering on UrBench-MCQ.

Figure 3: Overview of the proposed G 2 DA framework, consisting of gradient-guided cross-modal decoupling, gradient-aware curriculum scheduling, and bidirectional rehearsal.

Benchmark	Model	Optimization Method
Benchmark	Model	Joint	PCGrad	GradNorm	STGU	CAGrad	FAMO	G 2 DA
UrBench-MCQ	GeoChat-7B	$51.1_{\pm 0.2}$	$48.5_{\pm 0.2}$	$49.9_{\pm 0.2}$	$51.2_{\pm 0.3}$	$48.3_{\pm 0.2}$	$50.2_{\pm 0.3}$	$\mathbf{54.1_{\pm 0.2}}$
	GeoLLaVA-8B	$51.7_{\pm 0.2}$	$48.5_{\pm 0.2}$	$50.4_{\pm 0.3}$	$50.3_{\pm 0.2}$	$48.6_{\pm 0.2}$	$50.3_{\pm 0.2}$	$\mathbf{54.8_{\pm 0.2}}$
	InternVL3-2B	$44.2_{\pm 0.2}$	$39.2_{\pm 0.1}$	$43.4_{\pm 0.3}$	$45.8_{\pm 0.2}$	$39.9_{\pm 0.2}$	$43.2_{\pm 0.3}$	$\mathbf{48.8_{\pm 0.3}}$
	InternVL3-8B	$51.9_{\pm 0.1}$	$48.1_{\pm 0.4}$	$51.9_{\pm 0.4}$	$54.9_{\pm 0.3}$	$48.1_{\pm 0.2}$	$51.8_{\pm 0.2}$	$\mathbf{57.8_{\pm 0.1}}$
	InternVL3-14B	$61.1_{\pm 0.2}$	$53.8_{\pm 0.2}$	$61.1_{\pm 0.2}$	$63.8_{\pm 0.2}$	$54.1_{\pm 0.1}$	$61.4_{\pm 0.2}$	$\mathbf{66.8_{\pm 0.2}}$

Table 1: Performance (%) over three independent runs. Best results are shown in bold , and second-best are underlined .

Model	Vision-oriented (V) Tasks					Language-oriented (L) Tasks									Average
Model	Cam.	Traffic	Road	Ori.	Match	City	Count	Img.	Attr.	Ground	Role	Comp.	Scene	VPR	V	L	All
Qwen3.5-VL-27B (Joint)	41.2	68.1	66.7	30.1	59.8	70.2	78.1	45.3	49.9	70.4	71.1	85.7	76.1	85.2	53.2	70.2	65.1
Qwen3.5-VL-27B (G 2 DA)	45.0	72.0	72.5	32.5	62.6	70.0	83.8	45.0	52.5	72.5	75.0	90.0	85.0	88.9	56.9	73.6	68.7
GeoLLaVA-8B (Joint)	38.2	73.1	50.1	24.9	44.3	54.8	49.1	40.2	46.2	62.4	63.3	54.5	53.2	70.2	46.1	54.9	51.7
GeoLLaVA-8B (G 2 DA)	40.0	75.0	56.7	25.0	48.2	55.0	54.1	41.0	49.0	65.0	67.5	58.7	62.5	74.0	49.0	58.5	54.8

Table 2: Per-task accuracy (

\%

) on UrBench-MCQ with tasks grouped into vision and language branches.

	UrBench-MCQ		XLRS-Bench-Lite		VRS-Bench-VQA
Method	Cos. $\uparrow$	Conf. $\downarrow$	Cos. $\uparrow$	Conf. $\downarrow$	Cos. $\uparrow$	Conf. $\downarrow$
Joint	0.0191	47.3	0.0221	43.0	0.0236	41.2
G 2 DA	0.0216	41.3	0.0240	41.1	0.0248	40.5

Table 3: Gradient interaction statistics after training. Higher cosine similarity and lower conflict rate (

\%

) indicate less gradient interference.

Method	Components			InternVL3-8B		GeoLLaVA-8B
Method	D	C	R	UrBench	XLRS	UrBench	XLRS
Joint				51.9	38.1	50.7	43.8
+ Decoupling	✓			56.0	39.9	53.3	45.7
+ Curriculum	✓	✓		54.9	40.7	52.1	46.2
G 2 DA	✓	✓	✓	57.8	42.2	54.8	47.8

Table 4: Component ablation of G 2 DA (accuracy,

\%

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Locatability-Guided Adaptive Reasoning for Image Geo-Localization with Vision-Language Models
Mar 13, 2026Bo Yu, Fengze Yang, Yiming Liu +6VLM Reasoning Cross-View Geo-Localization
The University of Utah · The University of Oklahoma · Worcester Polytechnic Institute +2
GeoPID: Decomposing and Steering Visual Information in Vision-Language Models
Oct 6, 2026Seulgi Kim, Zhixiong Zhang, Xinwei Zhang +2Vision-Language Grounding Representation Geometry in VLMs
Georgia Institute of Technology · Work done as an intern at Amazon. · Amazon
GeoWorld-VLM: Geometry from World Models for Vision-Language Models
May 15, 2026Renjie Gu, Kaichen Zhou, Yan Luo +1Visual Spatial Reasoning VLM Adaptation
Harvard AI and Robotics Lab · Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University

Mar 13, 2026J/K move · Enter open · S save

Locatability-Guided Adaptive Reasoning for Image Geo-Localization with Vision-Language Models

Bo Yu, Fengze Yang, Yiming Liu, Chao Wang+5

The University of Utah · The University of Oklahoma · Worcester Polytechnic Institute+2

VLM Reasoning Cross-View Geo-Localization RL for VLMs

Open paper AskPDF arXiv

The emergence of Vision-Language Models (VLMs) has introduced new paradigms for global image geo-localization through retrieval-augmented generation (RAG) and reasoning-driven inference. However, RAG methods are constrained by retrieval database quality, while reasoning-driven approaches fail to internalize image locatability, relying on inefficient, fixed-depth reasoning paths that increase hallucinations and degrade accuracy. To overcome these limitations, we introduce an Optimized Locatability Score that quantifies an image's suitability for deep reasoning in geo-localization. Using this metric, we curate Geo-ADAPT-51K, a locatability-stratified reasoning dataset enriched with augmented reasoning trajectories for complex visual scenes. Building on this foundation, we propose a two-stage Group Relative Policy Optimization (GRPO) curriculum with customized reward functions that regulate adaptive reasoning depth, visual grounding, and hierarchical geographical accuracy. Our framework, Geo-ADAPT, learns an adaptive reasoning policy, achieves state-of-the-art performance across multiple geo-localization benchmarks, and substantially reduces hallucinations by reasoning both adaptively and efficiently.

Abstract

VLM Adaptation

Explore similar work

Locatability-Guided Adaptive Reasoning for Image Geo-Localization with Vision-Language Models

GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

GeoWorld-VLM: Geometry from World Models for Vision-Language Models