RemoteVLPG: How to Obtain a Specific VLM for Language-Guided Path Generation on Remote Sensing Images?
Authors: Junzhe Dang, Mengmeng Zhang, Yuxiang Zhang, Wei Li
Organizations: School of Information and Electronics, Beijing Institute of Technology, and the National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing 100081, China · Beijing Institute of Technology, Zhuhai, Guangdong 519088, China · Jockey Club STEM Laboratory of Quantitative Remote Sensing, Department of Geography, the University of Hong Kong, Hong Kong, China
We study how to obtain a task-specific Vision-Language Model (VLM) for language-guided path generation between start and end points on remote sensing images. We explore this question from three aspects. First, we implement three widely used architectural approaches and compare their performance. Second, we design a scalable method for constructing pre-training data and build an effective pre-training dataset with a degree of diversity. Methods trained on this dataset achieve promising zero-shot performance. Third, we construct three benchmarks that support stratified evaluation and design three metrics for path generation. We investigate this problem through extensive experiments. The results show that the zero-shot performance of methods trained on the constructed pre-training dataset can exceed that of some advanced commercial VLMs. Code and Dataset can be available at https://github.com/jzdang/RemoteVLPG
Figures & tables
Fig. 1: Architectures of the three methods: (a) NTP-based, (b) Head-based, and (c) Denoising-based.
Fig. 2: (a) Attention masks of the NTP-based and head-based methods. (b) Attention mask of the denoising-based method. Here, I, T, S-E, and P denote image, text, start and end coordinates, and path (ground truth, learnable tokens, or noise), respectively.
Fig. 3: Construction workflow for the pre-training dataset and the three benchmarks: LPBench, PPBench, and ToSh-Data.
Fig. 4: Image–path alignment in the synthetic and real pre-training datasets. (a) The trained ControlNet generates images from segmentation masks obtained by widening randomly sampled paths. (b) Binary morphological skeletonization extracts paths from real road segmentation masks.
Dataset
Samples
Tasks
Stratified evaluation
Average text length
Datasets for pre-training a
RemoteSAM [ 50 ]
270k
multi-modal segmentation
—
6.9 words
PreGRES [ 5 ]
1204k
VQA
—
11.12 words
RemotePath-175k
175k
Path generation & Path CoT VQA & VQA
—
5.79 words for class nouns; 31.73 words for coarse ; 44.80 words for full .
Benchmarks b
RefSegRS [ 51 ]
4.4k
multi-modal segmentation
No
3.09 words
TABLE I: Comparison of the constructed pre-training dataset and benchmarks with existing datasets in tasks and text length. Our datasets focus on path generation and provide longer text.
Fig. 5: Ground-truth path annotations in (a) LPBench, (b) PPBench, and (c) ToSh-Data.
Dataset
Split
Path samples number
Trend count
Path pixel length
1
2
3
4
5
<600
600–1000
>1000
Benchmarks
LPBench
train
1024
54
661
246
59
4
770
239
15
test
2047
97
1423
423
88
16
1509
510
28
PPBench
train
744
35
525
158
26
0
0
294
450
test
1483
73
1066
296
48
0
0
612
871
TABLE II: Numbers of path samples in each stratum of the three benchmarks under the two stratification strategies.
coarse input
Model
Test mode
LPBench
PPBench
ToSh-Data
NADE
NCD
PLE
NADE
NCD
PLE
NADE
NCD
PLE
Qwen3.7-Plus
zero-shot
2.62
1.88
9.97
6.33
4.57
12.60
6.15
4.58
13.27
Qwen3.8-Flash
zero-shot
2.29
1.74
9.32
5.27
3.94
11.21
5.82
4.27
11.65
GLM-5.3-Flash
zero-shot
1.35
1.09
5.53
3.40
2.56
7.31
3.92
2.90
6.39
Gemini 3.6 Flash
zero-shot
1.10
0.83
4.74
2.85
2.00
6.36
2.02
1.44
4.31
TABLE III: Overall performance after RemotePath-175k pre-training: zero-shot evaluation and fine-tuning w/o GK or w/ GK-v2, compared with seven recent commercial VLMs. All metrics are percentages, and lower is better.
Model
Test mode
LPBench
PPBench
ToSh-Data
coarse
full
coarse
full
coarse
full
NTP-based
zero-shot
1910/2047(93.3%)
1901/2047(92.9%)
1451/1483(97.8%)
1439/1483(97.0%)
33/34(97.1%)
34/34(100%)
w/o GK
1613/2047(78.8%)
1563/2047(76.4%)
1206/1483(81.3%)
1205/1483(81.3%)
—
—
w/ GK-v2
1825/2047(89.2%)
1787/2047(87.3%)
1317/1483(88.8%)
1305/1483(88.0%)
—
—
TABLE IV: Numbers of format-valid predictions of the NTP-based method after pre-training, under zero-shot evaluation and fine-tuning w/o GK or w/ GK-v2 on LPBench nand PPBench.
Fig. 6: Stratified evaluation of all metrics under coarse and full on LPBench and PPBench for the three methods: zero-shot evaluation, fine-tuning w/o GK, and fine-tuning w/ GK-v2.
Fig. 7: Predicted paths from the three methods on (a) LPBench and (b) PPBench. Architectural differences result in different path-shape characteristics.
Fig. 8: Ablation of coarse → full samples in pre-training across three methods and three benchmarks. Each bar is the metric with coarse → full samples minus the metric without coarse → full samples. Negative values mean lower metrics, hence better performance, with coarse → full samples.
Fig. 9: VLM outputs for coarse inputs after pre-training with and without coarse → full samples. Including coarse → full samples improves scene understanding.
Fig. 10: Ablation of the pre-training loss weight for coarse inputs across three methods and three benchmarks. Each bar is the metric at weight 1.0 minus the metric at weight 0.6. Negative values mean lower metrics, and hence better performance, at weight 1.0.
Model
GK setting
Training steps (LPBench/PPBench)
coarse input
full input
LPBench
PPBench
LPBench
PPBench
NADE
NCD
PLE
NADE
NCD
PLE
NADE
NCD
PLE
NADE
NCD
PLE
Head-based (ours)
w/o GK
9000/6750
4.64
2.92
28.55
7.42
5.28
16.35
5.54
3.56
30.76
8.03
5.61
17.27
w/ GK-v2
9000/6750
4.35
2.72
26.80
7.05
5.14
13.62
5.32
3.52
27.12
7.82
5.58
14.47
w/ GK-v1
15,750/13,500
4.14
2.76
21.51
6.73
4.99
11.51
5.08
3.51
23.00
7.43
5.46
11.57
Denoising-based (ours)
w/o GK
9000/6750
14.40
6.25
79.36
21.78
7.83
80.26
14.30
6.12
79.33
21.67
7.79
80.53
TABLE V: Head-based and Denoising-based performance after direct fine-tuning without RemotePath-175k task-specific pre-training, w/o GK, w/ GK-v2, or w/ GK-v1.
Model
GK setting
Training steps (LPBench/PPBench)
LPBench
PPBench
coarse
full
coarse
full
NTP-based
w/o GK
9000/6750
227/2047(11.1%)
211/2047(10.3%)
58/1483(3.9%)
40/1483(2.7%)
w/ GK-v2
9000/6750
181/2047(8.8%)
211/2047(10.3%)
42/1483(2.8%)
51/1483(3.4%)
w/ GK-v1
15,750/13,500
324/2047(15.8%)
281/2047(13.7%)
69/1483(4.7%)
74/1483(5.0%)
TABLE VI: Numbers of format-valid predictions of the NTP-based method after direct fine-tuning without RemotePath-175k task pre-training, w/o GK, w/ GK-v2, or w/ GK-v1.
Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is sufficient even without architectural novelty.
Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +2
Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision, yet high-quality annotations for satellite and aerial imagery remain scarce and expensive to produce. Prevailing pseudo-labeling pipelines address this gap by distilling knowledge from large frontier models, but this dependence on large teachers is costly, limits scalability, and caps achievable performance at the ceiling of the teacher. We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency. Our key insight is that a capable base VLM can serve as its own annotation engine: by pairing aerial images with rendered OpenStreetMap (OSM) tiles, we leverage optical character recognition and chart comprehension capabilities of the model to generate captions enriched by OSM's vast auxiliary metadata. The model is then fine-tuned on the resulting corpus with satellite imagery alone, yielding OSMDA-VLM, a domain-adapted VLM that requires no manual labeling and no stronger external VLM teacher. We conduct exhaustive evaluations spanning six zero-shot and five in-distribution benchmarks across vision-language tasks, where OSMDA leads to substantial improvement. We further compare against nine competitive baselines, demonstrating that our method achieves superior overall performance, while being substantially cheaper to train than teacher-dependent alternatives. These results suggest that, given a strong foundation model, alignment with crowd-sourced geographic data is a practical and scalable path towards remote sensing domain adaptation. Dataset and model weights will be made publicly available upon acceptance.
Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +3
INSAIT, Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski”
Recent advances in Vision Language Models (VLMs) and Vision Foundation Models (VFMs) have opened new opportunities for zero-shot text-guided segmentation of remote sensing imagery. However, most existing approaches still rely on additional trainable components, limiting their generalisation and practical applicability. In this work, we investigate to what extent text-based remote sensing segmentation can be achieved without additional training, by relying solely on existing foundation models. We propose a simple yet effective approach that integrates contrastive and generative VLMs with the Segment Anything Model (SAM), enabling a fully training-free or lightweight LoRA-tuned pipeline. Our contrastive approach employs CLIP as mask selector for SAM's grid-based proposals, achieving state-of-the-art open-vocabulary semantic segmentation (OVSS) in a completely zero-shot setting. In parallel, our generative approach enables reasoning and referring segmentation by generating click prompts for SAM using GPT-5 in a zero-shot setting and a LoRA-tuned Qwen-VL model, with the latter yielding the best results. Extensive experiments across 19 remote sensing benchmarks, including open-vocabulary, referring, and reasoning-based tasks, demonstrate the strong capabilities of our approach. Code will be released at https://github.com/josesosajs/trainfree-rs-segmentation.