Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysis

Xingxing Weng; Ruifeng Ni; Chao Pang; XiangYu Hao; Yishan Wang; Xiaokang Zhang; Wei Xu; Gui-Song Xia

cs.CVApr 1, 2026

Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysis

Authors: Xingxing Weng, Ruifeng Ni, Chao Pang, XiangYu Hao, Yishan Wang, Xiaokang Zhang, Wei Xu, Gui-Song Xia

Organizations: School of Computer Science, Wuhan University · College of Resources and Environmental Sciences, Shanxi Agricultural University · School of Artificial Intelligence, Wuhan University · School of Electronic Information, Wuhan University ·

Abstract

Current remote sensing vision-language models (RS VLMs) demonstrate impressive performance in image interpretation but rely on static training data, limiting their ability to accommodate continuously emerging sensing modalities and downstream tasks. This exposes a fundamental challenge: enabling RS VLMs to continually adapt without catastrophic forgetting. Despite its practical importance, the continual learning capability of RS VLMs remains underexplored, and no dedicated benchmark currently exists. In this work, we present CLeaRS, a comprehensive benchmark for continual vision-language learning in remote sensing. CLeaRS comprises 10 curated subsets with over 207k image-text pairs, spanning diverse interpretation tasks, sensing modalities, and application scenarios. We further define three evaluation protocols: long-horizon, modality-incremental, and task-incremental settings, to systematically assess continual adaptation. Extensive benchmarking of diverse vision-language models reveals catastrophic forgetting across all settings. Moreover, representative continual learning methods, when adapted to RS VLMs, exhibit limited effectiveness in handling task, instruction, and modality transitions. Our findings underscore the need for developing continual learning methods tailored to RS VLMs.

Figures & tables

Dataset	Image Modality	Interpretation Task	#Train	#Test	Data Source
AID	Optical	Scene Classification	8,000 (8,000)	2,000 (2,000)	AID
VRS-Cap	Optical	Image Captioning	7,833 (7,833)	2,340 (2,340)	VRSBench
VRS-VG	Optical	Visual Grounding	14,569 (7,812)	4,416 (2,324)	VRSBench
VRS-VQA	Optical	Visual Question Answering	16,258 (4,612)	17,018 (4,680)	VRSBench
SAR-VG	SAR	Visual Grounding	9,976 (8,000)	2,730 (2,157)	SARDet-100K
SAR-VQA	SAR	Visual Question Answering	20,000 (8,000)	5,000 (2,000)	SARLANG-1M

Table 1: Dataset statistics of the CLeaRS benchmark. The image modality includes SAR (Synthetic Aperture Radar), optical, and IR (infrared). For the #Train and #Test columns, a (b) indicates the number of image-text pairs and images, respectively.

Figure 1: The CLeaRS benchmark comprises 10 subsets that progressively cover diverse interpretation tasks, sensing modalities, and application scenarios, facilitating systematic investigation of continual vision-language learning behaviors in RS VLMs.

Figure 2: Statistics of the newly constructed subsets in CLeaRS. The remaining subsets are adapted from existing image-text datasets and detailed in Appendix.

VLMs	Qwen2.5-VL [ 45 ]			MiniGPT-v2 [ 46 ]			LLaVA-1.5 [ 47 ]			GeoChat [ 3 ]			VHM [ 7 ]
VLMs	Joint	Inst.	Final	Joint	Inst.	Final	Joint	Inst.	Final	Joint	Inst.	Final	Joint	Inst.	Final
Long-Horizon Setting
AID	84.0	82.1	80.9	97.4	97.6	46.0	94.9	95.3	76.6	95.3	95.1	66.9	95.6	98.8	16.2
VRS-Cap	20.6	12.0	0.1	13.1	12.1	0.2	22.6	17.4	0.4	22.9	16.1	0.0	23.4	17.8	0.4
VRS-VG	35.4	10.9	15.2	25.8	6.2	0.9	21.0	8.5	1.8	33.7	25.2	12.5	16.3	6.7	0.2
VRS-VQA	66.9	68.7	66.8	69.1	70.1	53.1	71.8	71.3	59.6	72.6	72.6	57.9	72.8	72.4	43.1

Table 2: Benchmarking VLMs on CLeaRS. Joint corresponds to joint training on all subsets. Inst. reports performance immediately after each subset, while Final denotes performance after the last subset, both under sequential fine-tuning.

Figure 3: Comparison of VHM with and without supervised fine-tuning in the task-incremental setting.

Figure 4: Analysis of factors contributing to forgetting in RS VLMs.

Methods	Long-Horizon				Modality-Incremental				Task-Incremental
Methods	MFT	MFN	MAA	BWT	MFT	MFN	MAA	BWT	MFT	MFN	MAA	BWT
Sequential FT	51.9	27.8	35.3	-26.8	41.8	24.2	15.7	-21.2	57.6	35.3	45.0	-29.8
MoELoRA [ 25 ]	53.2	29.1	37.2	-26.8	41.8	31.6	17.1	-12.2	56.7	34.7	46.1	-29.4
HiDe-LLaVA [ 26 ]	49.4	27.7	32.8	-24.1	40.9	31.2	16.0	-11.6	56.1	31.4	43.9	-32.9
SEFE [ 29 ]	52.9	27.5	37.2	-28.3	43.8	33.1	18.2	-12.8	59.1	37.2	49.7	-29.1

Table 3: Performance of continual learning methods on CLeaRS. All methods use VHM as the backbone model for fair comparison.

Panel of Table 4 — Table 4: Performance of continual learning methods under different learning orders on CLeaRS (long-horizon). Numbered markers denote learning sequences (see Appendix).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	#Samples	Resolution (m)	License
AID [ 31 ]	Scene Classification	10,000	0.5 $\sim$ 8	-
VRSBench [ 32 ]	Image Captioning Visual Grounding Visual Question Answering	205,307	0.1 $\sim$ 30	CC BY 4.0
SARDet-100K [ 33 ]	Object Detection	116,598	0.1 $\sim$ 25	CC BY-NC 4.0
SARLANG-1M [ 34 ]	Image Captioning Visual Question Answering	118,331	0.1 $\sim$ 25	-
DroneVehicle [ 35 ]	Vehicle Detection	56,878	-	-
FireRisk [ 36 ]	Fire Risk Assessment	91,872	-	-

Appendix

Table 5: Statistics of the source datasets used to construct CLeaRS.

Figure 5: Prompts for referring expression generation in SAR and infrared imagery.

Question Type	Instruction Template	Example Answers
Object Identification	Is there at least one <specific category> present in the scene?	"Yes" or "No"
Instance Counting	How many <specific category> can you find in this image?	Number
Object Positioning	Where is the <specific category> located in this image?	"Left", "Middle", "Right"
Region Referring	Classify the object located at the bounding box <detailed coordinates>?	"Car", "Bus", "Van", "Truck", "Freight Car"

Appendix

Table 6: Instruction templates and example answers in IR-VQA.

Figure 6: Statistics of additional subsets in CLeaRS.

Figure 7: Examples from the SAR-VG (left) and IR-VG (right) subsets in CLeaRS.

ID	Learning Order
1	AID $\rightarrow$ VRS-Cap $\rightarrow$ VRS-VG $\rightarrow$ VRS-VQA $\rightarrow$ SAR-VG $\rightarrow$ SAR-VQA $\rightarrow$ IR-VG $\rightarrow$ IR-VQA $\rightarrow$ FireRisk $\rightarrow$ RescueNet
2	VRS-VQA $\rightarrow$ AID $\rightarrow$ VRS-VG $\rightarrow$ VRS-Cap $\rightarrow$ IR-VG $\rightarrow$ SAR-VG $\rightarrow$ SAR-VQA $\rightarrow$ IR-VQA $\rightarrow$ RescueNet $\rightarrow$ FireRisk
3	VRS-Cap $\rightarrow$ VRS-VQA $\rightarrow$ AID $\rightarrow$ VRS-VG $\rightarrow$ IR-VG $\rightarrow$ IR-VQA $\rightarrow$ SAR-VQA $\rightarrow$ SAR-VG $\rightarrow$ FireRisk $\rightarrow$ RescueNet

Appendix

Table 7: Different learning orders in CLeaRS (long-horizon).

VLMs	AID	VRS-Cap	VRS-VG	VRS-VQA	SAR-VG	SAR-VQA	IR-VG	IR-VQA	FireRisk	RescueNet
Qwen2.5-VL [ 45 ]	63.3	0.3	40.8	62.5	36.6	44.9	53.4	47.7	14.5	43.3
MiniGPT-v2 [ 46 ]	24.5	2.4	0.3	40.5	0.8	32.8	0.1	27.0	14.8	29.3
LLaVA-1.5 [ 47 ]	48.0	1.5	1.9	57.3	1.0	39.8	0.2	38.1	16.1	38.1
GeoChat [ 3 ]	73.5	3.2	6.2	54.1	4.4	39.7	0.1	33.9	14.3	37.3
VHM w/o SFT [ 7 ]	34.6	0.0	0.2	46.9	0.2	49.5	0.0	25.5	14.3	23.6

Appendix

Table 8: Zero-Shot Performance of VLMs on the CLeaRS benchmark. VHM w/o SFT denotes the model without supervised fine-tuning. Bounding box coordinates in VRS-VG, SAR-VG, and IR-VG are converted to match the formats used by each model.

VLMs		VRS-VG	SAR-VG	IR-VG	VRS-VQA	SAR-VQA	IR-VQA	MFT	MFN	MAA	BWT
VHM	Inst.	5.7	12.7	12.6	72.1	91.9	56.0	41.8	24.2	15.7	-21.2
VHM	Final	0.7	1.2	0.3	44.3	42.5	56.0	41.8	24.2	15.7	-21.2
VHM †	Inst.	8.7	33.6	37.3	72.7	92.4	58.8	50.6	26.3	19.1	-29.1
VHM †	Final	0.1	0.0	0.0	32.1	66.8	58.8	50.6	26.3	19.1	-29.1

Appendix

Table 9: Performance of VHM on CLeaRS (modality-incremental) with frozen vs. unfrozen vision encoder. † indicates an unfrozen vision encoder. Inst. reports performance immediately after each subset, while Final denotes performance after the last subset, both under sequential fine-tuning.

Remote Sensing VLMs

See all 41 papers

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
Jul 17, 2026Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +2Remote Sensing Image Understanding Remote Sensing VQA
INSAIT, Sofia University "St. Kliment Ohridski"
RSVideo: Are Your Vision-Language Models Ready for Remote Sensing Videos?
Aug 3, 2026Hongjie Zhou, Shiqin Wang, Haoyang Chen +5Video-Language Model Evaluation Remote Sensing
Wuhan University · Zhongguancun Academy
OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
Mar 10, 2026Ronghao Fu, Haoran Liu, Weijie Zhang +4VLM Evaluation Remote Sensing Benchmarks
Jilin University, Changchun, China · Chang Guang Satellite Technology, Changchun, China

Jul 17, 2026J/K move · Enter open · S save

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Luc Van Gool+1

INSAIT, Sofia University "St. Kliment Ohridski"

Remote Sensing Image Understanding Remote Sensing VQA Remote Sensing VLMs Large Vision-Language Models

Open paper AskPDF arXiv

Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is sufficient even without architectural novelty.

Abstract

Remote Sensing VLMs

Explore similar work

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

RSVideo: Are Your Vision-Language Models Ready for Remote Sensing Videos?

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks