OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

Stefan Maria Ailuro; Mario Markov; Mohammad Mahdi; Delyan Boychev; Luc Van Gool; Danda Pani Paudel

cs.CVMar 12, 2026

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

Authors: Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Delyan Boychev, Luc Van Gool, Danda Pani Paudel

Organizations: INSAIT, Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski”

Abstract

Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision, yet high-quality annotations for satellite and aerial imagery remain scarce and expensive to produce. Prevailing pseudo-labeling pipelines address this gap by distilling knowledge from large frontier models, but this dependence on large teachers is costly, limits scalability, and caps achievable performance at the ceiling of the teacher. We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency. Our key insight is that a capable base VLM can serve as its own annotation engine: by pairing aerial images with rendered OpenStreetMap (OSM) tiles, we leverage optical character recognition and chart comprehension capabilities of the model to generate captions enriched by OSM's vast auxiliary metadata. The model is then fine-tuned on the resulting corpus with satellite imagery alone, yielding OSMDA-VLM, a domain-adapted VLM that requires no manual labeling and no stronger external VLM teacher. We conduct exhaustive evaluations spanning six zero-shot and five in-distribution benchmarks across vision-language tasks, where OSMDA leads to substantial improvement. We further compare against nine competitive baselines, demonstrating that our method achieves superior overall performance, while being substantially cheaper to train than teacher-dependent alternatives. These results suggest that, given a strong foundation model, alignment with crowd-sourced geographic data is a practical and scalable path towards remote sensing domain adaptation. Dataset and model weights will be made publicly available upon acceptance.

Figures & tables

Figure 1: Estimated data generation costs based on API pricing and measured self-hosting costs (a). Performance per benchmark for settings sharing base model (b) and prior work (c). Datasets used for fine-tuning are highlighted in purple color , zero-shot benchmarks highlighted in teal color . Proposed OSMDA-VLM achieves favorable tradeoff of costs and performance.

Method	Student model	Teacher model	$N_{\text{gen.}}$	$C_{\text{apr.}}$
LRS-VQA	LLaVA-Next-7B	rule-based	-	-
SkyEyeGPT	MiniGPT-v2-7B	rule-based	-	-
GeoChat	LLaVA-1.5-7B	Vicuna-v1.5	320k	$200
GeoPix	LLaVA-1.5-7B	GPT-4o	140k	$600
SkySenseGPT	LLaVA-1.5-7B	GPT-3.5 & GPT-4	180k	$600
VHM	LLaVA-1.5-7B	Gemini-Vision	1.4M	$2500

Table 1: Remote Sensing VLMs: student model architecture, teacher model used for pseudo-labelling, number of generated samples

N_{\text{gen.}}

, and approximate cost

C_{\text{apr.}}

of generation based on teachers’ API cost or measured self-hosting cost. Methods further differ by architectural modifications and varying sources of weak annotations.

Panel of Figure 2 — Figure 2: The creation of OSMDA-Captions and OSMDA-VLM via the OSMDA method. We collect images of various areas and resolutions, and fetch their OSM object tags. We filter out visible objects using the image resolution and a set of heuristics, then send each object’s OSM tags through an LLM to produce a short anonymised label. We overlay the labels onto OSM map tiles and feed the resulting overlays, alongside the matching satellite images, to the base VLM to generate OSMDA-Captions – a detailed captioning dataset incorporating OSM metadata. We then mix this dataset with existing remote-sensing data and fine-tune the base model, producing OSMDA-VLM – a model specialized for the remote-sensing domain.

Figure 3: A comparison of four captions generated by different methods for the same area: Base (top left), OSMDA-Captions (top right), Base-FT (bottom left), and OSMDA-VLM (bottom right). Methods are ranked by benchmarking performance. OSMDA stabilizes the model, resulting in less hallucination and better descriptions of spatial and visual layout. Both Base and Base-FT sometimes hallucinate: e.g., incorrectly placed dome-shaped structure in the center and a nonexistent pond.

Model	EuroSAT	mAID	SkyS	XLRS cap	XLRS vqa	GEO cap	GEO vqa	GEO chdet	DM3	Avg
Model	f1	f1	f1	g_eval	l2 acc	g_eval	acc	acc	l2 acc	Avg
SkyEyeGPT	2.9	0.0	0.2	14.1	0.0	2.7	10.1	1.8	10.7	4.7
GeoPix	5.8	0.0	0.6	0.0	19.1	0.1	18.2	-	-	6.2
GeoChat	36.9	7.9	31.6	3.4	23.3	18.6	29.0	-	-	21.5
EarthDial	35.7	1.4	42.2	11.0	40.0	16.2	32.1	35.4	15.3	25.5
SkySenseGPT	44.5	14.5	39.4	12.9	25.4	23.7	30.1	-	-	27.2

Table 2: Zero-shot performance of OSMDA-VLM across six benchmarks and their tasks, compared to prior remote-sensing VLMs and fixed-backbone alternatives. Per-column maxima are highlighted in bold , the top-3 is underlined . Rank within fixed-backbone alternatives is shown in cell color.

VLM	OSM	Ann.	XLRS		GEO			DM3
VLM	repr.	engine	cap	vqa	cap	vqa	chdet	chdet
Base	-	-	44.9	50.9	38.7	50.1	47.8	28.4
SkyScript	Short	Base	27.0	49.7	40.1	52.8	50.9	28.4
OSM-Text	Dense	Base	39.5	50.0	40.9	52.5	50.2	29.1
Teacher	-	Teacher	29.7	48.5	40.0	54.6	50.5	30.1
OSMDA	Carto.	Base	44.5	51.4	41.4	52.9	49.5	29.1

Table 3: Pseudo-labelling ablation. OSM representation: Short text (from SkyScript), Dense text (list of objects and their counts), Cartographic (proposed). Annotation engine: Base (InternVL3.5-8B), or bigger Teacher (Gemma3-27B).

Figure 4: Base-OSMDA-FT vs Base-FT. Pretraining on OSMDA-Captions improves in-distribution performance.

Figure 5: Per-category accuracy for classification (left) and VQA (right). In the VQA panel, rural/urban denotes scene discrimination, while other slash-separated labels (e.g., building/scrub ) indicate quantitative object comparison.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		RSVQA	RSVQA	RSVQA	METER	Avg
Model	Input	LR	HR	BEN	ML	Avg
InternVL	RGB + OSMDA map	78.1	53.0	36.0	21.6	47.2
	RGB + raw OSM map	80.7	46.3	35.9	17.4	45.1
	RGB + OSM-Overlay	59.0	49.8	39.7	29.5	44.5
	RGB + OSM-Text	74.6	43.8	35.9	20.6	43.7
	RGB only	82.0	45.8	34.6	16.4	44.7

Appendix

Table 4: Teachers zero-shot performance on geo-referenced benchmarks. The evaluation is small-scale.

Model	EuroSAT	mAID	SkyS	XLRS cap	XLRS vqa	GEO cap	GEO vqa	GEO chdet	DM3	Avg
Model	f1	f1	f1	g_eval	l2 acc	g_eval	acc	acc	l2 acc	Avg
Base	41.6	42.7	37.5	44.9	50.9	38.7	47.8	50.1	28.4	42.5
Base-FT	40.6	40.2	37.5	38.3	50.2	40.7	47.3	50.3	27.9	41.4
Base-Teacher	38.3	56.4	49.8	32.7	48.9	35.2	46.9	44.9	24.3	41.9
Teacher-VLM	37.3	52.4	49.7	29.7	48.5	40.0	50.5	54.6	30.1	43.6
Base-OSMDA	44.4	47.7	47.2	43.9	51.6	36.9	48.9	50.9	26.1	44.1

Appendix

Table 5: Zero-shot performance under training on RS-Instruct, OSMDA-Captions, Teacher-Captions, and joint versions. Per-column maxima are highlighted in bold , the top-3 is underlined .

Figure 6: Two examples of VQAs from XLRS-Bench, with each model’s corresponding answers. Wrong format answers are colored grey, correct format wrong answers are colored red, and right answers are in green.

Figure 7: A captioning example from XLRS-Bench. GeoPix is omitted as it produces empty output. Wrong format answers are colored grey, correct format wrong answers are colored red, and right answers are in green. Ellipses indicate trimmed content. VHM, SkyEyeGPT, EarthDial and GeoChat refuse to follow the official XLRS-Bench captioning prompt, which asks to split the image into nine quadrants. Many models hallucinate an airport.

Panel of Figure 8 — Figure 8: Overview of map generation examples and data artifacts.

Benchmark	Task	Image sizes	Res, m/pixel	Note
EuroSAT ( Helber et al. 2019 )	cls	64x64	10-60	10 classes
SkyScript ( Wang et al. 2024 )	cls	77x77 - 1015x1084	0.1 - 30	70 classes, constructed from OSM tags
Million-AID ( Long et al. 2021 )	cls	256x256, 512x512	0.5 - 153	hierarchical structure of 8/28/51 classes
UCM-Captions ( Qu et al. 2016 )	cap	256x256	0.3	short rigid-form captions
NWPU-Captions ( Cheng et al. 2022 )	cap	256x256	0.2 - 30	short rigid-form captions
RSVQA LR ( Lobry et al. 2020 )	vqa	256x256	10	rigid-form vqa, counting, area measurement

Appendix

Table 6: Description of benchmarks used in the study. VRSBench and XLRS-Bench also contain visual grounding task, which was omitted in the study. Images in RSVQA are georeferenced, which also allows to evaluate base with maps, i.e. teacher in OSMDA. Resolutions for VRSBench and XLRS-Bench were not reported before, we estimate it visually.

Paper	Euro	Million	SkyScript	RSVQA	RSVQA	UCM	NWPU	VRS	XLRS	GEO	Disaster
Paper	SAT	AID	bench	LR	HR	Captions	Captions	Bench	Bench	Bench	M3
GeoPix	x	x	x	x	x	x	x	FT	x	x	x
SkyEyeGPT	x	x	x	FT	FT	FT	FT	x	x	x	x
GeoChat	x	x	x	FT	x	x	x	x	x	x	x
SkySenseGPT	x	x	x	FT	ZS	x	x	x	x	x	x
LRS-VQA	x	x	x	x	FT	x	x	x	x	x	x

Appendix

Table 7: The map of previous evaluations. FT marks whether the model was fine-tuned on the dataset, ZS stated whether the model was evaluated in zero-shot manner, x states that model was not evaluated before. Note that training data of Intern-S1-mini is not public, but it is also likely trained on some of the benchmarks.

Parameter	Range
	none, short text (SkyScript), dense text (OSM-Text),
OSM representation	overlay, overlay + counts,
	raw OSM map, OSMDA maps
Resolution info	none, image size, GSD , image area
map size / rgb image size	x0.5, x1, x2.3
annotation engine	LlaVA1.5-7B, InternVL3.5-8B , Gemma3-27B,

Appendix

Table 8: Range of hyperparameters explored. The final selection for OSMDA-VLM is highlighted in bold .

Model	EuroSAT	mAID	SkyS	XLRS cap	XLRS vqa	GEO cap	GEO vqa	GEO chdet	DM3	Avg
Model	f1	f1	f1	g_eval	l2 acc	g_eval	acc	acc	l2 acc	Avg
Base	41.56 $\pm 0.19$	42.72 $\pm 0.65$	37.53 $\pm 0.49$	44.92 $\pm 0.45$	50.90 $\pm 0.88$	38.68 $\pm 0.13$	47.77 $\pm 0.40$	50.09 $\pm 0.54$	28.42 $\pm 0.37$	42.51 $\pm 0.16$
Teacher-VLM	37.25 $\pm 0.20$	52.36 $\pm 0.65$	49.67 $\pm 0.51$	29.71 $\pm 0.60$	48.46 $\pm 0.88$	40.01 $\pm 0.13$	50.51 $\pm 0.39$	54.62 $\pm 0.54$	30.07 $\pm 0.37$	43.63 $\pm 0.16$
OSMDA-VLM	44.92 $\pm 0.21$	50.36 $\pm 0.63$	50.72 $\pm 0.50$	44.54 $\pm 0.44$	51.35 $\pm 0.88$	41.40 $\pm 0.13$	49.55 $\pm 0.40$	52.90 $\pm 0.54$	29.10 $\pm 0.37$	46.10 $\pm 0.16$

Appendix

Table 9: Zero-shot performance and standard deviation, accounting for per-sample variation, of OSMDA-VLM across six benchmarks and their tasks, compared to prior fixed-backbone alternatives. Per-column maxima are highlighted in bold .

Remote Sensing VLMs

See all 41 papers

Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift
Apr 10, 2026Harshith Kethavath, Weiming HuVLM Adaptation Remote Sensing VLMs
University of Georgia, USA
More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
Jul 17, 2026Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +2Remote Sensing Image Understanding Remote Sensing VQA
INSAIT, Sofia University "St. Kliment Ohridski"
OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
Mar 10, 2026Ronghao Fu, Haoran Liu, Weijie Zhang +4VLM Evaluation Remote Sensing Benchmarks
Jilin University, Changchun, China · Chang Guang Satellite Technology, Changchun, China

Apr 10, 2026J/K move · Enter open · S save

Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift

Harshith Kethavath, Weiming Hu

University of Georgia, USA

VLM Adaptation Remote Sensing VLMs Fine-Tuning Image Segmentation Remote Sensing Image Segmentation

Open paper AskPDF arXiv

Adapting vision-language models to remote sensing imagery presents a fundamental challenge: both the visual and linguistic distributions of satellite data lie far outside natural image pretraining corpora. Despite this, prompting remains the dominant deployment paradigm, driven by the assumption that domain-specific language can guide frozen model representations toward specialized tasks. We test this assumption directly on a domain where the mismatch is prominent: cloud segmentation for satellite imagery. Using CLIPSeg on the CloudSEN12+ cloud segmentation benchmark, we evaluate 60 prompt variants spanning simple labels, domain terminology, appearance descriptors, and contextual cues, finding that every variant underperforms the zero-shot baseline (0.255 mIoU), with engineered prompts scoring as low as 0.07 mIoU. No amount of linguistic refinement bridges the gap between CLIP's natural image representations and satellite spectral imagery. In contrast, supervised fine-tuning with just 0.1% labeled data (~8 images) surpasses zero-shot performance overall, and 5-10% data recovers ~85% of maximum achievable mIoU. Full fine-tuning consistently outperforms low-rank adaptation by 0.03-0.09 mIoU, with the largest gaps for spectrally ambiguous classes, and at 0.5 to 1% labeled data, fine-tuning temporarily degrades performance on these classes before recovering, a supervision dip that aggregate mIoU can mask. For practitioners adapting vision-language models to specialized imagery, our results deliver a clear message: labeled data is not the expensive alternative to prompting; it is the worthwhile path.

Abstract

Remote Sensing VLMs

Explore similar work

Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks