GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning

Xiao Yang; Ronghao Fu; Zhuoran Duan; Zhiwen Lin; Xueyan Liu; Bo Yang

cs.CVMar 10, 2026

GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning

Authors: Xiao Yang, Ronghao Fu, Zhuoran Duan, Zhiwen Lin, Xueyan Liu, Bo Yang

Organizations: College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University

Abstract

Vision-language pretraining models have made significant progress in bridging remote sensing imagery with natural language. However, existing approaches often fail to effectively integrate multi-granular visual and textual information, relying primarily on global image-text alignment. This limitation hinders the model's ability to accurately capture fine-grained details in images, thus restricting its performance in complex, fine-grained tasks. To address this, we propose GeoAlignCLIP, a unified framework that achieves fine-grained alignment in remote sensing tasks by learning multi-granular semantic alignments and incorporating intra-modal consistency, enabling more precise visual-semantic alignment between image regions and text concepts. Additionally, we construct RSFG-100k, a fine-granular remote sensing dataset containing scene descriptions, region-level annotations, and challenging hard-negative samples, providing hierarchical supervision for model training. Extensive experiments conducted on multiple public remote-sensing benchmarks demonstrate that GeoAlignCLIP consistently outperforms existing RS-specific methods across diverse tasks, exhibiting more robust and accurate fine-grained vision-language alignment.

Figures & tables

Figure 1: (a) Comparison of feature-map RoI cropping and pixel-space RoI cropping methods for vision-language alignment. (b) Comparison of attention heatmaps for models trained with different text granularities (brief, enriched, and multi-granular) in capturing global and local semantic relationships.

Figure 2: Overall architecture and training pipeline of GeoAlignCLIP . (a) Stage I performs global contrastive learning. (b) Stage II conducts Multi-Granularity Contrastive Learning and Multi-View Consistency Learning.

Method	Version	Fine-grained Understanding					Region-level Classification
		RRSIS-HR [ 20 ]		CHOICE-ins. [ 1 ]		CHOICE-img. [ 1 ]	NWPU-VHR-10 [ 5 ]		RRSIS-D [ 22 ]
		Acc@1	Acc@5	Acc@1	Acc@5	Acc.	Acc@1	Acc@5	Acc@1	Acc@5
CLIPs based on Natural Images
CLIP [ 26 ]	ViT-B/16	19.41	60.66	50.00	91.60	72.00	12.22	54.45	41.68	73.63
CLIP [ 26 ]	ViT-L/14	22.18	64.47	32.80	86.20	88.00	11.87	56.90	47.47	82.65
LongCLIP [ 40 ]	ViT-B/16	13.00	58.23	27.80	90.80	82.50	56.94	83.44	40.79	69.21

Table 2: Experimental results on fine-grained understanding and region-level Classification tasks. Acc@1 and Acc@5 denote Top-1 and Top-5 accuracy, respectively.

Parameter	Value	Parameter	Value	Parameter	Value
Backbone Network	ResNet-50	Pseudo-Label Threshold	0.5	Max Training Iterations	10,000
Input Image Size	1024 $\times$ 1024	RPN Filtering Threshold	0.9	Learning Rate (LR)	0.01
Supervised Loss Weight	1.0	Dynamic Queue Start	2,000	LR Warm-up Iterations	500
Unsupervised Loss Weight	2.0	Neck Out Channels	256	Momentum	0.9
FC Out Channels	768	RPN In Channels	256	Weight Decay	0.0001
BBox Head In Channels	256	Pad Size Divisor	32	Image Format	RGB

Table 4: Experimental configurations for open-vocabulary detection.

Method	Backbone	RSICD [ 23 ]						RSITMD [ 38 ]						UCM-Caption [ 23 ]
		I2T			T2I			I2T			T2I			I2T			T2I
		R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
CLIPs based on Natural Images
CLIP [ 26 ]	ViT-B/16	5.67	14.82	22.6	5.29	16.96	26.77	7.96	21.46	31.42	8.36	26.11	40.84	8.57	31.9	55.71	9.52	36.0	64.1
CLIP [ 26 ]	ViT-L/14	6.59	17.75	28.09	4.96	18.72	29.66	10.62	27.43	36.73	10.04	31.33	46.15	12.38	41.43	70.95	10.57	43.81	71.24
LongCLIP [ 40 ]	ViT-B/16	7.69	20.13	29.19	6.68	20.68	32.17	8.85	23.67	32.96	8.72	33.41	49.07	12.86	41.9	65.24	11.81	45.81	76.19

Table 5: Comparisons of image-text retrieval task on the RSICD, RSITMD, and UCM-Caption. I2T and T2I denote image-to-text retrieval and text-to-image retrieval, respectively. R@k represent Recall at top- k .

Loss —				RRSIS-HR	DOTAv1.0	UCM-Caption		SIRI-WHU
$\mathcal{L}_{\text{RPA}}$	$\mathcal{L}_{\text{VIC}}$	$\mathcal{L}_{\text{NHA}}$	$\mathcal{L}_{\text{HTC}}$	Acc@1	mAP ${}_{\text{n}}$	I2T R@1	T2I R@1	Acc@1
-	-	-	-	25.48	14.50	16.67	12.90	51.92
$\checkmark$	-	-	-	33.28	14.80	20.00	16.57	59.21
$\checkmark$	$\checkmark$	-	-	34.66	20.20	20.00	16.67	61.96
$\checkmark$	$\checkmark$	$\checkmark$	-	35.01	21.50	18.10	16.48	63.04
$\checkmark$	$\checkmark$	-	$\checkmark$	35.36	16.50	20.14	16.19	63.42

Table 6: Ablation study of different tasks on the VRSBench, DOTAv1.0, UCM-Caption, and SIRI-WHU datasets.

Model	Param.	Lat.	FLOPs	Acc.
	(M)	(ms/token)	(T)	(%)
RemoteCLIP	427.62	0.1218	5.40	8.32
SkyCLIP	427.62	0.1216	5.40	6.41
GeoRSCLIP	427.62	0.1215	5.40	11.44
LRSCLIP	427.62	0.1215	5.40	26.69
Ours	428.92	0.1327	13.65	33.45

Table 7: Efficiency comparison. Latency is the per-token inference time measured.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 355 papers

Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
Sep 3, 2026Xiangyang Miao, Kelu Yao, Yekai Huang +7Remote Sensing Image Understanding Remote Sensing VLMs
School of Computer Science and Technology, Zhejiang University · Space-based Computing System Research Center, Zhejiang Lab
ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery
Apr 2, 2026Ke Li, Ting Wang, Di Wang +4Remote Sensing Visual Grounding Remote Sensing VLMs
Xidian University, China · UC San Diego, USA · SUST, China
Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysis
Apr 1, 2026Xingxing Weng, Ruifeng Ni, Chao Pang +5Remote Sensing VLMs Continual Learning for VLMs
School of Computer Science, Wuhan University · College of Resources and Environmental Sciences, Shanxi Agricultural University · School of Artificial Intelligence, Wuhan University +2

Sep 3, 2026J/K move · Enter open · S save

Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data

Xiangyang Miao, Kelu Yao, Yekai Huang, Xiaogang Xu+6

School of Computer Science and Technology, Zhejiang University · Space-based Computing System Research Center, Zhejiang Lab

Remote Sensing Image Understanding Remote Sensing VLMs Multimodal Pretraining Remote Sensing Remote Sensing Benchmarks Multimodal Contrastive Learning+1

Open paper AskPDF arXiv

Contrastive language-image learning (CLIP) has become a key paradigm for remote sensing vision-language understanding. However, existing remote sensing contrastive learning methods are mostly built on RGB-oriented CLIP architectures, making it difficult to exploit heterogeneous sensors such as SAR, multi-spectral imaging (MSI), and hyperspectral imaging (HSI). To address this limitation, we propose OmniRSCLIP, an end-to-end contrastive learning framework that supports multi-source sensor inputs for remote sensing vision-language modeling. The key idea is to extend CLIP beyond its fixed RGB input interface without breaking the pretrained visual knowledge. To this end, OmniRSCLIP introduces Spectral-Spatial Basis Decomposition (SSBD), which formulates arbitrary-channel adaptation as a basis recomposition problem: pretrained CLIP patch embeddings provide transferable spatial bases, while wavelength-conditioned coefficients span sensor-specific embedding kernels within a constrained visual prior space. This design avoids forcing heterogeneous sensors into a fixed-channel input space, while aligning them in a unified image-text semantic space. We further introduce a spectral-context-aware mask-based contrastive learning scheme to suppress modality-specific redundant features and enhance fine-grained image-text alignment. Finally, to support multi-modal training, we construct OmniRS5M, the first large-scale remote sensing image-text corpus covering RGB, SAR, MSI, and HSI. Experiments on retrieval, zero-shot classification, and semantic localization show that OmniRSCLIP preserves strong RGB-domain performance while effectively extending CLIP to heterogeneous remote sensing modalities.

Abstract

Vision-Language Alignment

Explore similar work

Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data

ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery

Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysis