OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

Stefan Maria Ailuro; Mario Markov; Mohammad Mahdi; Delyan Boychev; Luc Van Gool; Danda Pani Paudel

cs.CVMar 12, 2026

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

Authors: Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Delyan Boychev, Luc Van Gool, Danda Pani Paudel

Organizations: INSAIT, Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski”

Abstract

Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision, yet high-quality annotations for satellite and aerial imagery remain scarce and expensive to produce. Prevailing pseudo-labeling pipelines address this gap by distilling knowledge from large frontier models, but this dependence on large teachers is costly, limits scalability, and caps achievable performance at the ceiling of the teacher. We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency. Our key insight is that a capable base VLM can serve as its own annotation engine: by pairing aerial images with rendered OpenStreetMap (OSM) tiles, we leverage optical character recognition and chart comprehension capabilities of the model to generate captions enriched by OSM's vast auxiliary metadata. The model is then fine-tuned on the resulting corpus with satellite imagery alone, yielding OSMDA-VLM, a domain-adapted VLM that requires no manual labeling and no stronger external VLM teacher. We conduct exhaustive evaluations spanning six zero-shot and five in-distribution benchmarks across vision-language tasks, where OSMDA leads to substantial improvement. We further compare against nine competitive baselines, demonstrating that our method achieves superior overall performance, while being substantially cheaper to train than teacher-dependent alternatives. These results suggest that, given a strong foundation model, alignment with crowd-sourced geographic data is a practical and scalable path towards remote sensing domain adaptation. Dataset and model weights will be made publicly available upon acceptance.

Figures & tables

Figure 1: Estimated data generation costs based on API pricing and measured self-hosting costs (a). Performance per benchmark for settings sharing base model (b) and prior work (c). Datasets used for fine-tuning are highlighted in purple color , zero-shot benchmarks highlighted in teal color . Proposed OSMDA-VLM achieves favorable tradeoff of costs and performance.

Method	Student model	Teacher model	$N_{\text{gen.}}$	$C_{\text{apr.}}$
LRS-VQA	LLaVA-Next-7B	rule-based	-	-
SkyEyeGPT	MiniGPT-v2-7B	rule-based	-	-
GeoChat	LLaVA-1.5-7B	Vicuna-v1.5	320k	$200
GeoPix	LLaVA-1.5-7B	GPT-4o	140k	$600
SkySenseGPT	LLaVA-1.5-7B	GPT-3.5 & GPT-4	180k	$600
VHM	LLaVA-1.5-7B	Gemini-Vision	1.4M	$2500

Table 1: Remote Sensing VLMs: student model architecture, teacher model used for pseudo-labelling, number of generated samples

N_{\text{gen.}}

, and approximate cost

C_{\text{apr.}}

of generation based on teachers’ API cost or measured self-hosting cost. Methods further differ by architectural modifications and varying sources of weak annotations.

Panel of Figure 2 — Figure 2: The creation of OSMDA-Captions and OSMDA-VLM via the OSMDA method. We collect images of various areas and resolutions, and fetch their OSM object tags. We filter out visible objects using the image resolution and a set of heuristics, then send each object’s OSM tags through an LLM to produce a short anonymised label. We overlay the labels onto OSM map tiles and feed the resulting overlays, alongside the matching satellite images, to the base VLM to generate OSMDA-Captions – a detailed captioning dataset incorporating OSM metadata. We then mix this dataset with existing remote-sensing data and fine-tune the base model, producing OSMDA-VLM – a model specialized for the remote-sensing domain.

Figure 3: A comparison of four captions generated by different methods for the same area: Base (top left), OSMDA-Captions (top right), Base-FT (bottom left), and OSMDA-VLM (bottom right). Methods are ranked by benchmarking performance. OSMDA stabilizes the model, resulting in less hallucination and better descriptions of spatial and visual layout. Both Base and Base-FT sometimes hallucinate: e.g., incorrectly placed dome-shaped structure in the center and a nonexistent pond.

Model	EuroSAT	mAID	SkyS	XLRS cap	XLRS vqa	GEO cap	GEO vqa	GEO chdet	DM3	Avg
Model	f1	f1	f1	g_eval	l2 acc	g_eval	acc	acc	l2 acc	Avg
SkyEyeGPT	2.9	0.0	0.2	14.1	0.0	2.7	10.1	1.8	10.7	4.7
GeoPix	5.8	0.0	0.6	0.0	19.1	0.1	18.2	-	-	6.2
GeoChat	36.9	7.9	31.6	3.4	23.3	18.6	29.0	-	-	21.5
EarthDial	35.7	1.4	42.2	11.0	40.0	16.2	32.1	35.4	15.3	25.5
SkySenseGPT	44.5	14.5	39.4	12.9	25.4	23.7	30.1	-	-	27.2

Table 2: Zero-shot performance of OSMDA-VLM across six benchmarks and their tasks, compared to prior remote-sensing VLMs and fixed-backbone alternatives. Per-column maxima are highlighted in bold , the top-3 is underlined . Rank within fixed-backbone alternatives is shown in cell color.

VLM	OSM	Ann.	XLRS		GEO			DM3
VLM	repr.	engine	cap	vqa	cap	vqa	chdet	chdet
Base	-	-	44.9	50.9	38.7	50.1	47.8	28.4
SkyScript	Short	Base	27.0	49.7	40.1	52.8	50.9	28.4
OSM-Text	Dense	Base	39.5	50.0	40.9	52.5	50.2	29.1
Teacher	-	Teacher	29.7	48.5	40.0	54.6	50.5	30.1
OSMDA	Carto.	Base	44.5	51.4	41.4	52.9	49.5	29.1

Table 3: Pseudo-labelling ablation. OSM representation: Short text (from SkyScript), Dense text (list of objects and their counts), Cartographic (proposed). Annotation engine: Base (InternVL3.5-8B), or bigger Teacher (Gemma3-27B).

Figure 4: Base-OSMDA-FT vs Base-FT. Pretraining on OSMDA-Captions improves in-distribution performance.

Figure 5: Per-category accuracy for classification (left) and VQA (right). In the VQA panel, rural/urban denotes scene discrimination, while other slash-separated labels (e.g., building/scrub ) indicate quantitative object comparison.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		RSVQA	RSVQA	RSVQA	METER	Avg
Model	Input	LR	HR	BEN	ML	Avg
InternVL	RGB + OSMDA map	78.1	53.0	36.0	21.6	47.2
	RGB + raw OSM map	80.7	46.3	35.9	17.4	45.1
	RGB + OSM-Overlay	59.0	49.8	39.7	29.5	44.5
	RGB + OSM-Text	74.6	43.8	35.9	20.6	43.7
	RGB only	82.0	45.8	34.6	16.4	44.7

Appendix

Table 4: Teachers zero-shot performance on geo-referenced benchmarks. The evaluation is small-scale.

Model	EuroSAT	mAID	SkyS	XLRS cap	XLRS vqa	GEO cap	GEO vqa	GEO chdet	DM3	Avg
Model	f1	f1	f1	g_eval	l2 acc	g_eval	acc	acc	l2 acc	Avg
Base	41.6	42.7	37.5	44.9	50.9	38.7	47.8	50.1	28.4	42.5
Base-FT	40.6	40.2	37.5	38.3	50.2	40.7	47.3	50.3	27.9	41.4
Base-Teacher	38.3	56.4	49.8	32.7	48.9	35.2	46.9	44.9	24.3	41.9
Teacher-VLM	37.3	52.4	49.7	29.7	48.5	40.0	50.5	54.6	30.1	43.6
Base-OSMDA	44.4	47.7	47.2	43.9	51.6	36.9	48.9	50.9	26.1	44.1

Appendix

Table 5: Zero-shot performance under training on RS-Instruct, OSMDA-Captions, Teacher-Captions, and joint versions. Per-column maxima are highlighted in bold , the top-3 is underlined .

Figure 6: Two examples of VQAs from XLRS-Bench, with each model’s corresponding answers. Wrong format answers are colored grey, correct format wrong answers are colored red, and right answers are in green.

Figure 7: A captioning example from XLRS-Bench. GeoPix is omitted as it produces empty output. Wrong format answers are colored grey, correct format wrong answers are colored red, and right answers are in green. Ellipses indicate trimmed content. VHM, SkyEyeGPT, EarthDial and GeoChat refuse to follow the official XLRS-Bench captioning prompt, which asks to split the image into nine quadrants. Many models hallucinate an airport.

Panel of Figure 8 — Figure 8: Overview of map generation examples and data artifacts.

Benchmark	Task	Image sizes	Res, m/pixel	Note
EuroSAT ( Helber et al. 2019 )	cls	64x64	10-60	10 classes
SkyScript ( Wang et al. 2024 )	cls	77x77 - 1015x1084	0.1 - 30	70 classes, constructed from OSM tags
Million-AID ( Long et al. 2021 )	cls	256x256, 512x512	0.5 - 153	hierarchical structure of 8/28/51 classes
UCM-Captions ( Qu et al. 2016 )	cap	256x256	0.3	short rigid-form captions
NWPU-Captions ( Cheng et al. 2022 )	cap	256x256	0.2 - 30	short rigid-form captions
RSVQA LR ( Lobry et al. 2020 )	vqa	256x256	10	rigid-form vqa, counting, area measurement

Appendix

Table 6: Description of benchmarks used in the study. VRSBench and XLRS-Bench also contain visual grounding task, which was omitted in the study. Images in RSVQA are georeferenced, which also allows to evaluate base with maps, i.e. teacher in OSMDA. Resolutions for VRSBench and XLRS-Bench were not reported before, we estimate it visually.

Paper	Euro	Million	SkyScript	RSVQA	RSVQA	UCM	NWPU	VRS	XLRS	GEO	Disaster
Paper	SAT	AID	bench	LR	HR	Captions	Captions	Bench	Bench	Bench	M3
GeoPix	x	x	x	x	x	x	x	FT	x	x	x
SkyEyeGPT	x	x	x	FT	FT	FT	FT	x	x	x	x
GeoChat	x	x	x	FT	x	x	x	x	x	x	x
SkySenseGPT	x	x	x	FT	ZS	x	x	x	x	x	x
LRS-VQA	x	x	x	x	FT	x	x	x	x	x	x

Appendix

Table 7: The map of previous evaluations. FT marks whether the model was fine-tuned on the dataset, ZS stated whether the model was evaluated in zero-shot manner, x states that model was not evaluated before. Note that training data of Intern-S1-mini is not public, but it is also likely trained on some of the benchmarks.

Parameter	Range
	none, short text (SkyScript), dense text (OSM-Text),
OSM representation	overlay, overlay + counts,
	raw OSM map, OSMDA maps
Resolution info	none, image size, GSD , image area
map size / rgb image size	x0.5, x1, x2.3
annotation engine	LlaVA1.5-7B, InternVL3.5-8B , Gemma3-27B,

Appendix

Table 8: Range of hyperparameters explored. The final selection for OSMDA-VLM is highlighted in bold .

Model	EuroSAT	mAID	SkyS	XLRS cap	XLRS vqa	GEO cap	GEO vqa	GEO chdet	DM3	Avg
Model	f1	f1	f1	g_eval	l2 acc	g_eval	acc	acc	l2 acc	Avg
Base	41.56 $\pm 0.19$	42.72 $\pm 0.65$	37.53 $\pm 0.49$	44.92 $\pm 0.45$	50.90 $\pm 0.88$	38.68 $\pm 0.13$	47.77 $\pm 0.40$	50.09 $\pm 0.54$	28.42 $\pm 0.37$	42.51 $\pm 0.16$
Teacher-VLM	37.25 $\pm 0.20$	52.36 $\pm 0.65$	49.67 $\pm 0.51$	29.71 $\pm 0.60$	48.46 $\pm 0.88$	40.01 $\pm 0.13$	50.51 $\pm 0.39$	54.62 $\pm 0.54$	30.07 $\pm 0.37$	43.63 $\pm 0.16$
OSMDA-VLM	44.92 $\pm 0.21$	50.36 $\pm 0.63$	50.72 $\pm 0.50$	44.54 $\pm 0.44$	51.35 $\pm 0.88$	41.40 $\pm 0.13$	49.55 $\pm 0.40$	52.90 $\pm 0.54$	29.10 $\pm 0.37$	46.10 $\pm 0.16$

Appendix

Table 9: Zero-shot performance and standard deviation, accounting for per-sample variation, of OSMDA-VLM across six benchmarks and their tasks, compared to prior fixed-backbone alternatives. Per-column maxima are highlighted in bold .

Remote Sensing VLMs

See all 41 papers

Apr 10, 2026

Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift

Adapting vision-language models to remote sensing imagery presents a fundamental challenge: both the visual and linguistic distributions of satellite data lie far outside natural image pretraining corpora. Despite this, prompting remains the dominant deployment paradigm, driven by the assumption that domain-specific language can guide frozen model representations toward specialized tasks. We test this assumption directly on a domain where the mismatch is prominent: cloud segmentation for satellite imagery. Using CLIPSeg on the CloudSEN12+ cloud segmentation benchmark, we evaluate 60 prompt variants spanning simple labels, domain terminology, appearance descriptors, and contextual cues, finding that every variant underperforms the zero-shot baseline (0.255 mIoU), with engineered prompts scoring as low as 0.07 mIoU. No amount of linguistic refinement bridges the gap between CLIP's natural image representations and satellite spectral imagery. In contrast, supervised fine-tuning with just 0.1% labeled data (~8 images) surpasses zero-shot performance overall, and 5-10% data recovers ~85% of maximum achievable mIoU. Full fine-tuning consistently outperforms low-rank adaptation by 0.03-0.09 mIoU, with the largest gaps for spectrally ambiguous classes, and at 0.5 to 1% labeled data, fine-tuning temporarily degrades performance on these classes before recovering, a supervision dip that aggregate mIoU can mask. For practitioners adapting vision-language models to specialized imagery, our results deliver a clear message: labeled data is not the expensive alternative to prompting; it is the worthwhile path.

Harshith Kethavath, Weiming Hu

University of Georgia, USA

VLM Adaptation Remote Sensing VLMs Fine-Tuning Image Segmentation Remote Sensing Image Segmentation

Ask PDF

Jul 17, 2026

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is sufficient even without architectural novelty.

Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +2

INSAIT, Sofia University "St. Kliment Ohridski"

Remote Sensing Image Understanding Remote Sensing VQA Remote Sensing VLMs Large Vision-Language Models

Ask PDF

Mar 10, 2026

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks

Vision-Language Models (VLMs) have demonstrated effective perception and reasoning capabilities on general-domain tasks, leading to growing interest in their application to Earth observation. However, a systematic benchmark for comprehensively evaluating remote sensing vision-language models (RSVLMs) remains lacking. To address this gap, we introduce OmniEarth, a benchmark for evaluating RSVLMs under realistic Earth observation scenarios. OmniEarth organizes tasks along three capability dimensions: perception, reasoning, and robustness. It defines 28 fine-grained tasks covering multi-source sensing data and diverse geospatial contexts. The benchmark supports two task formulations: multiple-choice VQA and open-ended VQA. The latter includes pure text outputs for captioning tasks, bounding box outputs for visual grounding tasks, and mask outputs for segmentation tasks. To reduce linguistic bias and examine whether model predictions rely on visual evidence, OmniEarth adopts a blind test protocol and a quintuple semantic consistency requirement. OmniEarth includes 9,275 carefully quality-controlled images, including proprietary satellite imagery from Jilin-1 (JL-1), along with 44,210 manually verified instructions. We conduct a systematic evaluation of contrastive learning-based models, general closed-source and open-source VLMs, as well as RSVLMs. Results show that existing VLMs still struggle with geospatially complex tasks, revealing clear gaps that need to be addressed for remote sensing applications. OmniEarth is publicly available at https://huggingface.co/datasets/sjeeudd/OmniEarth.

Ronghao Fu, Haoran Liu, Weijie Zhang +4

Jilin University, Changchun, China · Chang Guang Satellite Technology, Changchun, China

VLM Evaluation Remote Sensing Benchmarks Remote Sensing VLMs Remote Sensing VQA Remote Sensing Visual Grounding

Ask PDF

Abstract

Remote Sensing VLMs

Explore similar work

Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks