AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models

Md Aminur Hossain; Omkumar Vaghasiya; Rajeev Ranjan Dwivedi; Vinod Kurmi; Biplab Banerjee

cs.CVAug 16, 2026

AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models

Authors: Md Aminur Hossain, Omkumar Vaghasiya, Rajeev Ranjan Dwivedi, Vinod Kurmi, Biplab Banerjee

Organizations: Space Applications Centre, ISRO, Ahmedabad, India · Indian Institute of Science Education and Research (IISER) Bhopal, India · CSRE, Indian Institute of Technology Bombay, India

Abstract

Remote sensing (RS) foundation models provide transferable Earth observation representations across sensors, resolutions, and geographies, yet most remain weakly aligned with natural language, limiting natural-language archive search, image-text retrieval, and question-conditioned analysis. We propose AlignJEPA, a JEPA-inspired predictive vision-language alignment framework for remote sensing foundation models. AlignJEPA uses a pretrained AnySat visual encoder and a RemoteCLIP text encoder while training only a lightweight predictive alignment network. Instead of relying on global image--text contrastive alignment alone, the framework predicts remote-sensing text embeddings from masked visual foundation-model tokens. Its mask-aware multi-scale predictive aligner aggregates visible tokens at fine, regional, and global scales, jointly models them with a cross-scale Transformer, and projects the resulting representation into the text space using learned query pooling. Training combines semantic prediction with bidirectional contrastive retrieval. We train and evaluate AlignJEPA on BigEarthNet.txt for natural-language Sentinel retrieval, evaluate cross-dataset adaptation on RSICD, and use RSVQA only as a closed-set representation probe. AlignJEPA provides a parameter-efficient route for aligning Earth observation foundation models with language.

Figures & tables

Figure 1: Overview of AlignJEPA. A pretrained EO visual encoder extracts spatial tokens from remote-sensing imagery, and a RemoteCLIP text encoder provides the target language-semantic embedding. The trainable predictive alignment network maps partially visible visual tokens into the text embedding space using semantic prediction and bidirectional contrastive retrieval objectives.

Figure 2: Toy $4{\times}4$ example of AlignJEPA: partially visible EO context is mapped into the RemoteCLIP text-embedding space and aligned with the caption embedding $z_{t}$ via $\mathcal{L}_{\text{pred}}$ .

Method	T2I R@1	T2I R@5	T2I R@10	I2T R@1	I2T R@5	I2T R@10
RemoteCLIP zero-shot	12.42	33.80	46.90	13.68	35.16	48.60
GeoRSCLIP zero-shot	13.61	35.42	49.20	14.52	37.23	50.40
AnySat + Linear Projection	10.79	30.52	43.74	11.67	31.83	44.59
AnySat + Contrastive Alignment	16.90	40.60	55.35	17.80	42.42	57.13
AlignJEPA	20.72	47.81	62.09	21.89	49.31	63.64

Table 1: Image–text retrieval results on BigEarthNet.txt. T2I denotes text-to-image retrieval and I2T denotes image-to-text retrieval.

Method	Text-to-Image (T2I)			Image-to-Text (I2T)
Method	R@1	R@5	R@10	R@1	R@5	R@10
VSE++ [ 11 ]	2.82	11.32	18.10	3.38	9.51	17.46
SCAN [ 18 ]	3.91	16.20	26.49	4.39	10.90	17.64
CAMP [ 36 ]	4.15	15.23	27.81	5.12	12.89	21.12
MTFN [ 35 ]	4.90	17.17	29.49	5.02	12.52	19.74
KCR [ 26 ]	4.76	18.59	27.20	5.84	22.31	36.12

Table 2: Image–text retrieval on RSICD. Published RSICD results are shown only for context because their training protocols differ. The bottom block reports our controlled cross-dataset adaptation setting, where the BigEarthNet.txt-trained alignment network is fine-tuned on the RSICD training split while AnySat and RemoteCLIP remain frozen.

Specialist Models (Supervised / Fine-tuned)
Method	RSVQA-LR				RSVQA-HR (Test Set 2)
Method	Rur/Urb	Pres	Comp	Avg. Acc	Pres	Comp	Avg. Acc
RSVQA [ 21 ]	90.00	87.46	81.50	86.32	86.26	85.94	86.10
RSGPT [ 15 ]	94.00	91.17	91.70	92.29	89.87	89.68	89.78
Generalist / Instruction-Tuned Models
Qwen-VL-Chat [ 4 ]	62.00	66.54	47.65	58.73	61.75	65.98	63.87

Table 3: RSVQA results under different evaluation settings. Scores are category-specific and average accuracies (

\%

). The bottom section reports our representation probes with a lightweight answer head trained. Rur/Urb, Pres, and Comp denote Rural/Urban, Presence, and Comparison, respectively.

Factor	Variant	R@1	R@5	R@10
Masking policy	No masking	18.75	44.00	58.60
	Random, ratio $0.25$	19.60	45.20	60.10
	Random, ratio $0.50$	21.31	48.56	62.87
	Random, ratio $0.75$	18.90	44.10	58.75
	Block, ratio $0.50$	20.60	47.40	61.80
Loss design	Contrastive only	17.75	42.60	57.20

Table 4: Ablation analysis on BigEarthNet.txt. Scores are mean Recall@K averaged over text-to-image and image-to-text retrieval. Bold indicates the best result.

Remote Sensing VLMs

See all 41 papers

GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning
Mar 10, 2026Xiao Yang, Ronghao Fu, Zhuoran Duan +3Vision-Language Alignment Remote Sensing Image Understanding
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University
Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Multimodal Understanding Multimodal Grounding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5
OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
Mar 10, 2026Ronghao Fu, Haoran Liu, Weijie Zhang +4VLM Evaluation Remote Sensing Benchmarks
Jilin University, Changchun, China · Chang Guang Satellite Technology, Changchun, China

Mar 10, 2026J/K move · Enter open · S save

GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning

Xiao Yang, Ronghao Fu, Zhuoran Duan, Zhiwen Lin+2

College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University

Vision-Language Alignment Remote Sensing Image Understanding

Open paper AskPDF arXiv

Vision-language pretraining models have made significant progress in bridging remote sensing imagery with natural language. However, existing approaches often fail to effectively integrate multi-granular visual and textual information, relying primarily on global image-text alignment. This limitation hinders the model's ability to accurately capture fine-grained details in images, thus restricting its performance in complex, fine-grained tasks. To address this, we propose GeoAlignCLIP, a unified framework that achieves fine-grained alignment in remote sensing tasks by learning multi-granular semantic alignments and incorporating intra-modal consistency, enabling more precise visual-semantic alignment between image regions and text concepts. Additionally, we construct RSFG-100k, a fine-granular remote sensing dataset containing scene descriptions, region-level annotations, and challenging hard-negative samples, providing hierarchical supervision for model training. Extensive experiments conducted on multiple public remote-sensing benchmarks demonstrate that GeoAlignCLIP consistently outperforms existing RS-specific methods across diverse tasks, exhibiting more robust and accurate fine-grained vision-language alignment.

Abstract

Remote Sensing VLMs

Explore similar work

GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks