RemoteVLPG: How to Obtain a Specific VLM for Language-Guided Path Generation on Remote Sensing Images?
Authors: Junzhe Dang, Mengmeng Zhang, Yuxiang Zhang, Wei Li
Organizations: School of Information and Electronics, Beijing Institute of Technology, and the National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing 100081, China · Beijing Institute of Technology, Zhuhai, Guangdong 519088, China · Jockey Club STEM Laboratory of Quantitative Remote Sensing, Department of Geography, the University of Hong Kong, Hong Kong, China
We study how to obtain a task-specific Vision-Language Model (VLM) for language-guided path generation between start and end points on remote sensing images. We explore this question from three aspects. First, we implement three widely used architectural approaches and compare their performance. Second, we design a scalable method for constructing pre-training data and build an effective pre-training dataset with a degree of diversity. Methods trained on this dataset achieve promising zero-shot performance. Third, we construct three benchmarks that support stratified evaluation and design three metrics for path generation. We investigate this problem through extensive experiments. The results show that the zero-shot performance of methods trained on the constructed pre-training dataset can exceed that of some advanced commercial VLMs. Code and Dataset can be available at https://github.com/jzdang/RemoteVLPG
Figures & tables
Fig. 1: Architectures of the three methods: (a) NTP-based, (b) Head-based, and (c) Denoising-based.
Fig. 2: (a) Attention masks of the NTP-based and head-based methods. (b) Attention mask of the denoising-based method. Here, I, T, S-E, and P denote image, text, start and end coordinates, and path (ground truth, learnable tokens, or noise), respectively.
Fig. 3: Construction workflow for the pre-training dataset and the three benchmarks: LPBench, PPBench, and ToSh-Data.
Fig. 4: Image–path alignment in the synthetic and real pre-training datasets. (a) The trained ControlNet generates images from segmentation masks obtained by widening randomly sampled paths. (b) Binary morphological skeletonization extracts paths from real road segmentation masks.
Dataset
Samples
Tasks
Stratified evaluation
Average text length
Datasets for pre-training a
RemoteSAM [ 50 ]
270k
multi-modal segmentation
—
6.9 words
PreGRES [ 5 ]
1204k
VQA
—
11.12 words
RemotePath-175k
175k
Path generation & Path CoT VQA & VQA
—
5.79 words for class nouns; 31.73 words for coarse ; 44.80 words for full .
Benchmarks b
RefSegRS [ 51 ]
4.4k
multi-modal segmentation
No
3.09 words
TABLE I: Comparison of the constructed pre-training dataset and benchmarks with existing datasets in tasks and text length. Our datasets focus on path generation and provide longer text.
Fig. 5: Ground-truth path annotations in (a) LPBench, (b) PPBench, and (c) ToSh-Data.
Dataset
Split
Path samples number
Trend count
Path pixel length
1
2
3
4
5
<600
600–1000
>1000
Benchmarks
LPBench
train
1024
54
661
246
59
4
770
239
15
test
2047
97
1423
423
88
16
1509
510
28
PPBench
train
744
35
525
158
26
0
0
294
450
test
1483
73
1066
296
48
0
0
612
871
TABLE II: Numbers of path samples in each stratum of the three benchmarks under the two stratification strategies.
coarse input
Model
Test mode
LPBench
PPBench
ToSh-Data
NADE
NCD
PLE
NADE
NCD
PLE
NADE
NCD
PLE
Qwen3.7-Plus
zero-shot
2.62
1.88
9.97
6.33
4.57
12.60
6.15
4.58
13.27
Qwen3.8-Flash
zero-shot
2.29
1.74
9.32
5.27
3.94
11.21
5.82
4.27
11.65
GLM-5.3-Flash
zero-shot
1.35
1.09
5.53
3.40
2.56
7.31
3.92
2.90
6.39
Gemini 3.6 Flash
zero-shot
1.10
0.83
4.74
2.85
2.00
6.36
2.02
1.44
4.31
TABLE III: Overall performance after RemotePath-175k pre-training: zero-shot evaluation and fine-tuning w/o GK or w/ GK-v2, compared with seven recent commercial VLMs. All metrics are percentages, and lower is better.
Model
Test mode
LPBench
PPBench
ToSh-Data
coarse
full
coarse
full
coarse
full
NTP-based
zero-shot
1910/2047(93.3%)
1901/2047(92.9%)
1451/1483(97.8%)
1439/1483(97.0%)
33/34(97.1%)
34/34(100%)
w/o GK
1613/2047(78.8%)
1563/2047(76.4%)
1206/1483(81.3%)
1205/1483(81.3%)
—
—
w/ GK-v2
1825/2047(89.2%)
1787/2047(87.3%)
1317/1483(88.8%)
1305/1483(88.0%)
—
—
TABLE IV: Numbers of format-valid predictions of the NTP-based method after pre-training, under zero-shot evaluation and fine-tuning w/o GK or w/ GK-v2 on LPBench nand PPBench.
Fig. 6: Stratified evaluation of all metrics under coarse and full on LPBench and PPBench for the three methods: zero-shot evaluation, fine-tuning w/o GK, and fine-tuning w/ GK-v2.
Fig. 7: Predicted paths from the three methods on (a) LPBench and (b) PPBench. Architectural differences result in different path-shape characteristics.
Fig. 8: Ablation of coarse → full samples in pre-training across three methods and three benchmarks. Each bar is the metric with coarse → full samples minus the metric without coarse → full samples. Negative values mean lower metrics, hence better performance, with coarse → full samples.
Fig. 9: VLM outputs for coarse inputs after pre-training with and without coarse → full samples. Including coarse → full samples improves scene understanding.
Fig. 10: Ablation of the pre-training loss weight for coarse inputs across three methods and three benchmarks. Each bar is the metric at weight 1.0 minus the metric at weight 0.6. Negative values mean lower metrics, and hence better performance, at weight 1.0.
Model
GK setting
Training steps (LPBench/PPBench)
coarse input
full input
LPBench
PPBench
LPBench
PPBench
NADE
NCD
PLE
NADE
NCD
PLE
NADE
NCD
PLE
NADE
NCD
PLE
Head-based (ours)
w/o GK
9000/6750
4.64
2.92
28.55
7.42
5.28
16.35
5.54
3.56
30.76
8.03
5.61
17.27
w/ GK-v2
9000/6750
4.35
2.72
26.80
7.05
5.14
13.62
5.32
3.52
27.12
7.82
5.58
14.47
w/ GK-v1
15,750/13,500
4.14
2.76
21.51
6.73
4.99
11.51
5.08
3.51
23.00
7.43
5.46
11.57
Denoising-based (ours)
w/o GK
9000/6750
14.40
6.25
79.36
21.78
7.83
80.26
14.30
6.12
79.33
21.67
7.79
80.53
TABLE V: Head-based and Denoising-based performance after direct fine-tuning without RemotePath-175k task-specific pre-training, w/o GK, w/ GK-v2, or w/ GK-v1.
Model
GK setting
Training steps (LPBench/PPBench)
LPBench
PPBench
coarse
full
coarse
full
NTP-based
w/o GK
9000/6750
227/2047(11.1%)
211/2047(10.3%)
58/1483(3.9%)
40/1483(2.7%)
w/ GK-v2
9000/6750
181/2047(8.8%)
211/2047(10.3%)
42/1483(2.8%)
51/1483(3.4%)
w/ GK-v1
15,750/13,500
324/2047(15.8%)
281/2047(13.7%)
69/1483(4.7%)
74/1483(5.0%)
TABLE VI: Numbers of format-valid predictions of the NTP-based method after direct fine-tuning without RemotePath-175k task pre-training, w/o GK, w/ GK-v2, or w/ GK-v1.