VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments

Haolin Yang; Yuxing Long; Zihan Yang; Hao Dong

cs.ROAug 15, 2026

VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments

Authors: Haolin Yang, Yuxing Long, Zihan Yang, Hao Dong

Organizations: CFCS, School of Computer Science, Peking University Beijing, China · PrimeBot Beijing, China

Abstract

Navigation instruction generation from ego-centric RGB video in continuous environments is an important yet challenging task for human-robot interaction and scalable dataset construction. Prior instruction generators assume discrete viewpoint graphs with panoramic observations, where trajectory structure is explicit; in continuous environments, however, the agent receives only a dense RGB stream, making trajectory cues difficult to recover. We propose VTInstructor, the first VLN instruction generation framework for continuous environments. Our key idea is to convert implicit trajectory geometry into explicit visual trajectory prompts: EDTC condenses long RGB trajectories into navigation-critical keyframes, VTP overlays path, turn, and goal cues onto these anchors, VTMod injects the resulting trajectory signals into the visual encoder, and VT-GRPO further calibrates this spatial injection during training, all without requiring a navigation graph, pre-built map, or scene reconstruction. On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art across all standard NLG metrics, surpassing the strongest baseline by +0.357 CIDEr and +0.109 CIDEr, respectively. Beyond automatic metrics, VTInstructor-generated instructions raise a frozen follower's success rate to 63.3%, a +14.7 percentage-point gain over the best competing instruction source, and provide consistent data augmentation gains of +3 SR points on downstream navigation tasks.

Figures & tables

Figure 1. Evolution of navigation instruction generation paradigms. (a)–(c) Prior speakers rely on discrete viewpoint graphs with panoramic images, optionally augmented by external knowledge or 3D representations. (d) VTInstructor generates instructions from ego-centric RGB video in continuous environments, requiring no navigation graph or 3D reconstruction at inference.

Figure 2. Overview of VTInstructor. Event-driven compression turns a raw ego-centric RGB trajectory into a compact keyframe set; each keyframe carries a VTP overlay (path ribbon / rotation arrow / endpoint marker) encoded as a 3-channel binary mask. The VTP Encoder maps the mask to patch-aligned features that are injected into ViT layer 7 by per-token spatial modulation. Training proceeds in two stages: SFT, then VT-GRPO.

Proprietary models
	R2R-CE Val Unseen						RxR-CE Val Unseen
Method	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
Qwen3.5-Plus	0.557	0.131	0.223	0.387	0.137	0.180	0.527	0.080	0.157	0.254	0.033	0.156
GPT-5.4	0.482	0.075	0.197	0.348	0.078	0.144	0.493	0.056	0.146	0.241	0.029	0.143
Gemini-3.1-Pro-Preview	0.631	0.166	0.209	0.392	0.203	0.167	0.393	0.073	0.128	0.242	0.029	0.135
xAI Grok 4	0.486	0.084	0.207	0.354	0.073	0.160	0.430	0.049	0.181	0.249	0.014	0.144

Table 1. Navigation instruction generation results on R2R-CE and RxR-CE Val Unseen .

Prior methods on R2R / RxR (discretized viewpoint-graph environments)
	R2R / R2R-CE Val Unseen						RxR / RxR-CE Val Unseen
Method	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
BT-speaker ( Fried et al., 2018 )	0.658	0.250	0.209	0.440	0.391	0.178	0.311	0.064	0.160	0.243	0.022	–
EDrop-speaker ( Tan et al., 2019 )	0.660	0.260	0.215	0.455	0.413	0.184	0.294	0.056	0.143	0.253	0.027	–
CCC-speaker ( Wang et al., 2022a )	0.679	0.254	0.226	0.456	0.401	0.183	0.266	0.042	0.113	0.206	0.016	–
Lana ( Wang et al., 2023 )	0.689	0.260	0.219	0.463	0.419	0.194	0.314	0.111	0.126	0.267	0.045	–

Table 2. Cross-setting comparison with prior instruction generation methods. Prior methods use discretized viewpoint graphs with privileged 36-view observations; VTInstructor uses only ego-centric RGB in continuous environments. We present this as a reference point for instruction quality, not a strictly controlled comparison.

Figure 3. Qualitative comparison on R2R-CE and RxR-CE Val Unseen and a real-world room-tour clip. Each case shows the compressed keyframes with VTP overlays (top) and instructions from the ground truth, the strongest baseline, and VTInstructor (bottom). Yellow highlights mark factual errors in the baseline output (e.g. wrong turn direction, hallucinated landmarks).

#	VTMod	EDTC	QF	BLEU-4	METEOR	ROUGE-L	SPICE
1				0.282	0.232	0.485	0.201
2			$\checkmark$	0.286	0.241	0.492	0.220
3		$\checkmark$	$\checkmark$	0.288	0.242	0.493	0.223
4	$\checkmark$	$\checkmark$		0.302	0.250	0.500	0.232
5	$\checkmark$	$\checkmark$	$\checkmark$	0.308	0.256	0.504	0.238

Table 3. Component-wise ablation on R2R-CE Val Unseen .

(a) EDTC: frame sampling
Configuration	BLEU-4	METEOR	ROUGE-L	SPICE
Stride-4 (avg. 13.0 fr.)	0.738 †	0.240	0.490	0.220
EDTC (avg. 10.5 fr.)	0.743 †	0.242	0.493	0.223
(b) VTMod: injection layer
Layer 7	0.299	0.248	0.497	0.224
Layers 7, 15, 23	0.295	0.244	0.491	0.221

Table 4. Design ablations on R2R-CE Val Unseen .

Human Annotations
Instruction Source	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$
Official instruction	61.6	67.2	53.3	4.53
Proprietary models
Qwen3.5-Plus	48.6	61.8	39.0	5.70
GPT-5.4	35.9	51.4	28.6	6.14
Gemini-3.1-Pro-Preview	48.5	56.4	39.9	5.59

Table 5. Downstream navigation performance on R2R-CE Val Unseen with a frozen CorrectNav follower.

Set.	Training Data	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$
A	R2R-CE + RxR-CE (human)	45.1	52.3	44.6	6.20
B	Setting A + VTInstructor-generated	48.4	54.2	46.8	5.85

Table 6. CorrectNav (LLaVA-Video-7B backbone) trained under two augmentation settings on R2R-CE Val Unseen .

Set.	Training Data	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$
A	R2R-CE + RxR-CE (human)	41.2	51.1	39.6	8.34
B	Setting A + VTInstructor-generated	44.4	53.2	41.3	7.74

Table 7. CorrectNav (LLaVA-Video-7B backbone) trained under two augmentation settings on RxR-CE Val Unseen .

Method	Action	Landmark	Direction	Follow.
GPT-5.4	2.65 ± 0.08	3.55 ± 0.06	2.53 ± 0.10	2.29 ± 0.06
Qwen3-VL-8B	2.63 ± 0.12	3.65 ± 0.09	2.67 ± 0.05	2.49 ± 0.08
VTInstructor	4.38 ± 0.05	4.21 ± 0.05	4.25 ± 0.12	4.35 ± 0.04

Table 8. Human evaluation on real-world navigation videos (1–5 scale, mean

\pm

std over 3 annotators).

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

Goal-oriented Navigation Instruction Generation with Tour Video Priors
Aug 9, 2026Fangdi Li, Juncheng Liao, Changxu Cheng +4Vision-Language Navigation
Uni-Ubi AI, Hangzhou, China · Zhejiang University, Hangzhou, China · Tongji University, Shanghai, China
Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress
Oct 9, 2026Jiagan Huang, Zikun Zhou, Hongpeng Wang +3Vision-Language Navigation
Harbin Institute of Technology, Shenzhen
DA-Nav: Direction-Aware City-Scale Vision-Language Navigation
Jul 13, 2026Ye Yuan, Kehan Chen, Xinqiang Yu +7Embodied Navigation Vision-Language Navigation
School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences, Beijing, China +3

Aug 9, 2026J/K move · Enter open · S save

Goal-oriented Navigation Instruction Generation with Tour Video Priors

Fangdi Li, Juncheng Liao, Changxu Cheng, Jiazhi Wang+3

Uni-Ubi AI, Hangzhou, China · Zhejiang University, Hangzhou, China · Tongji University, Shanghai, China

Vision-Language Navigation

Open paper AskPDF arXiv

Navigation Instruction Generation (NIG) aims to produce step-by-step natural language instructions for navigation guidance. Existing studies primarily treat NIG as an auxiliary task for vision-andlanguage navigation (VLN), focusing on data augmentation or multi-task learning. However, generating navigation instructions from compact environmental priors requires meticulous spatial reasoning, especially when the target route does not simply follow the demonstrated tour, and remains challenging for current multimodal models. In this work, we introduce VideoNIG, a goal-oriented video-grounded NIG task that generates navigation instructions from ego-centric tour videos, an initial observation, and a textual or visual goal, without relying on intermediate representations such as graphs and maps. We instantiate VideoNIG in a controlled simulator benchmark with 60K tour videos across continuous indoor environments and 37K multimodal prompts with progressive difficulty levels. We further introduce a diagnostic evaluation protocol that combines text similarity, choice-based spatial consistency tests, and downstream navigation execution. To address this task, we propose a two-stage Curriculum Learning framework that decomposes the learning into foundational motion perception and long-horizon navigation reasoning. Specifically, we first employ Action Warmup for spatial action-view alignment, followed by Complexity Progression using trajectories with increasing exploratory difficulty. Extensive experiments show that existing MLLMs struggle with VideoNIG, while our approach significantly improves instruction quality across complementary diagnostic metrics. Finally, integrating VideoNIG-generated instructions with a VLN agent demonstrates the executability of this task formulation for end-to-end navigation.

Abstract

Vision-Language Navigation

Explore similar work

Goal-oriented Navigation Instruction Generation with Tour Video Priors

Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress

DA-Nav: Direction-Aware City-Scale Vision-Language Navigation