Training Generalist Navigation Policies via Action Self-Distillation

Bowen Feng; Keqin Wang; Yulong Yang; Amogh Joshi; Christine Allen-Blanchette; Dhruv Shah; Felix Heide

cs.ROOct 13, 2026

Training Generalist Navigation Policies via Action Self-Distillation

Authors: Bowen Feng, Keqin Wang, Yulong Yang, Amogh Joshi, Christine Allen-Blanchette, Dhruv Shah, Felix Heide

Organizations: Princeton University, USA

Abstract

Language provides a natural interface for humans to specify and reason about goals, but training language-conditioned robotic policies and reasoning typically requires expensive manual annotations. We propose Action Self-Distillation (ASD), a method for learning language-conditioned reasoning for robotic navigation without human-annotated language or reasoning labels. ASD uses an open-source VLM, which contains broad world knowledge and reasoning capabilities, as both a teacher network and a student policy. During training, the self-teacher receives privileged trajectory feedback and infers the underlying navigation goal together with goal-oriented reasoning. The student is then trained to condition on the inferred navigation goal, predict future ego-frame trajectory, and provide a description of the scene semantics, plan, and reasoning, from onboard observations alone. Across 13 challenging indoor and outdoor real-world navigation tasks, our method improves success rate over the strongest baseline by $30\%$ in indoor settings, $30\%$ on multi-stage tasks, and achieves $95\%$ success rate on outdoor tasks, validating that privileged trajectory feedback can enable self-distillation for scalable generalist navigation policies without human annotations.

Figures & tables

Figure 1: Action self-distillation. The self-teacher uses privileged ground-truth trajectory feedback to extract scene semantics, infer a language goal, and generate reasoning traces explaining the demonstrated behavior. Blue arrows denote the feedback-conditioned information flow to the self-teacher, while pink arrows denote the student pathway. The student observes only deployable inputs and learns to align its reasoning and trajectory tokens with the self-teacher, removing the need for external goal or reasoning annotations.

Figure 2: Action Self-Distillation. A single pre-trained VLM is used as both the self-teacher $\pi_{\phi}$ and student $\pi_{\theta}$ . During training, the self-teacher receives privileged information by projecting the ground-truth future trajectory $y$ onto the front-view observation as $o^{*}_{0}$ , enabling it to infer a language goal $\hat{g}$ and scene-grounded reasoning trace $R$ . The student observes only deployable inputs $o_{0}$ and learns to match the teacher’s reasoning tokens with a KL loss, while predicting future trajectory tokens $\tilde{\tau}_{0:H}$ with a cross-entropy loss against the ground-truth trajectory $y$ . The teacher weights are updated as an EMA of the student.

Method	Directions			Objects				Multi-stage		Avg.
Method	Straight Door - (1 direction)	Left chair + (2 directions)	Right chair ∗ (3 directions)	Blue - chair	Trash - bin	Chair + (2 objects)	Monitor ∗ (3 objects)	Doorway +	Hallway ∗	Avg.
OmniVLA	0.8	0.9	0.2	0.7	1.0	0.4	0.2	0.7	0.0	0.54
NaVILA	0.3	0.2	0.1	0.4	0.1	0.3	0.2	0.3	0.5	0.27
Qwen2.5-VL- ASD	0.9	1.0	0.8	1.0	0.9	0.9	0.7	0.8	0.6	0.84
Qwen3.5-VL- ASD	1.0	1.0	0.9	1.0	1.0	1.0	0.9	0.9	0.8	0.94

Table 1: Indoor Language-Conditioned Navigation. We evaluate tasks across three categories: directional, object-centric, and multi-stage navigation. Difficulty level increases with the number of possible directions for directional tasks, and the number of objects in the scene for object-centric tasks. In the multi-stage category, Doorway requires the robot to pass through a door, while Hallway further requires it to turn right after the doorway and move along the hallway. We use

-

+

, and

*

to denote easy, medium, and hard tasks. More detailed task descriptions and visualizations are provided in Section C.1 of the supplementary material.

Method	Sidewalk	Statue	Tree	Car	Avg.
OmniVLA	1.0	0.6	1.0	1.0	0.90
NaVILA	0.9	0.1	0.0	0.3	0.33
Qwen2.5-VL- ASD	1.0	0.8	1.0	1.0	0.95
Qwen3.5-VL- ASD	1.0	0.9	1.0	1.0	0.98

Table 2: Outdoor Language-Conditioned Navigation. We evaluate on 4 different tasks, including navigate along the sidewalk, navigate to the statue/tree/car . More detailed task descriptions and visualizations are provided in Section C.2 of the supplementary material.

Figure 3: Steerability Demonstration . We demonstrate ASD ’s steerability under color (left), direction (middle), and object (right) prompts. Within the same setting, our method can be directed to different targets.

Figure 4: Single-Stage Navigation Across Diverse Indoor and Outdoor Environments. We demonstrate ASD ’s capability to execute single-stage navigation goals specified in natural language. The examples are ordered from left to right as two indoor tasks ( right chair and monitor ) and three outdoor tasks ( car , statue , and tree ). Full videos are provided in the Supplementary Material.

Figure 5: Multi-Stage Reasoning Capabilities for Outdoor and Indoor Navigation Tasks. We demonstrate ASD ’s capability to execute complex, multi-stage navigation goals. The first row contains an indoor navigation demonstration, while the bottom row contains an outdoor navigation demonstration, both with their respective tasks described on the figure. Full videos are provided in the Supplementary Material.

Figure 6: Reasoning Legibility. We present representative findings from our real-world experiments, showing predicted trajectories alongside the corresponding highlighted reasoning traces. These examples demonstrate that the generated reasoning is interpretable and well aligned with the predicted robot behavior.

Method	SR $\uparrow$	Time (s) $\downarrow$
OmniVLA	0.9	25.3
NaVILA	0.2	28.9
Qwen2.5-VL	0.0	–
Claude Opus 4.7	0.7	846.1
Qwen2.5-VL- ASD	1.0	15.3

Table 3: Task success rates and completion times of foundation models on indoor navigation task . We extend the evaluation of Left chair in Table 1 by including Qwen2.5-VL and Claude Opus 4.7, and report the task completion time for all methods. We omit the completion time for Qwen2.5-VL because its predicted actions are generally physically infeasible.

Method	Directions	Objects		Multi-stage	Outdoor	Avg.
Method	Right chair	Chair	Monitor	Hallway	Car	Avg.
Qwen2.5-VL- ASD (goal-only supervision)	0.4	0.2	0.2	0.1	0.4	0.26
Qwen2.5-VL- ASD (reasoning off at inference)	0.1	0.2	0.2	0.0	0.2	0.14
Claude Opus 5 labels + SFT	0.3	0.5	0.3	0.3	0.4	0.36
Qwen2.5-VL labels + SFT	0.4	0.3	0.3	0.0	0.1	0.22
NaVILA (RECON + FrodoBots-2K SFT)	0.0	0.1	0.2	0.3	0.2	0.16

Table 4: Frozen-teacher baselines and reasoning ablations. Success rate on representative tasks from Table 1 and Table 2 requiring directional, object-level, multi-stage, and outdoor reasoning. Goal-only supervision trains on self-teacher-inferred goals without reasoning traces. Reasoning off at inference disables reasoning at test time. Labels + SFT baselines fine-tune on offline annotations from a frozen Claude Opus 5 or Qwen2.5-VL teacher. Best results are in bold , second best are underlined .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value	Hyperparameter	Value
Backbone VLM	Qwen2.5-VL-3B-Instruct	Image resolution	dataset-native
Max output tokens	512	Past waypoints	8
Future waypoints	16	Prediction horizon	6.4 s
Waypoint frequency	2.5 Hz	Train/validation split	9:1
Optimizer	adamw_torch_fused	Learning rate	$1\times 10^{-5}$
Batch size per GPU	1	Gradient accumulation	2

Appendix

Table 5: Training hyperparameters. Main hyperparameters used for model training and inference in real-world experiments. Hyperparameters not listed here use the HuggingFace Transformers defaults.

	Task	Difficulty	Prompt
Directions	Door	$\star$	"Navigate straight to the door"
	left chair	$\star\star$	"Navigate to the chair on the left"
	right chair	$\star\star\star$	"Navigate to the chair on the right"
Objects	Blue chair	$\star$	"Navigate to the blue chair"
	Trash Bin	$\star$	"Navigate to the trash bin"
	Chair	$\star\star$	"Navigate to the chair"

Appendix

Table 6: Prompt for language-conditioned navigation benchmarks.

Figure 7: Summary of Benchmark Tasks. Panels (a)–(h) show the indoor tasks reported in Table 1, covering directional, object-centric, and multi-stage navigation scenarios. Panels (i)–(l) show the outdoor tasks reported in Table 2. The target object or goal region for each task is highlighted with a cyan bounding box.

Method	RECON				FrodoBots-2K
Method	ADE $\downarrow$	ATE $\downarrow$	MAE $\downarrow$	MSE $\downarrow$	ADE $\downarrow$	ATE $\downarrow$	MAE $\downarrow$	MSE $\downarrow$
Gaussian Policy	0.226	0.321	0.144	0.059	0.034	0.046	0.019	0.002
Gaussian Mixture Model	0.260	0.364	0.166	0.084	0.036	0.050	0.020	0.002
Diffusion Policy [ 36 ]	0.180	0.238	0.108	0.045	0.034	0.045	0.020	0.003
Qwen2.5-VL+SFT [ 32 ]	0.192	0.247	0.115	0.042	0.064	0.061	0.027	0.007
Alpamayo-1+SFT [ 37 ]	0.303	0.362	0.188	0.091	0.093	0.106	0.055	0.041

Appendix

Table 7: Open-loop evaluation on RECON and FrodoBots-2K. ASD achieves the best performance across both datasets under the open-loop evaluation protocol. Lower is better for all metrics.

Figure 8: Evaluation on Multi-object Tasks. We visualize the Chair and Monitor tasks for OmniVLA (left) and ASD (right). (a) OmniVLA navigates the robot toward a distractor object, the net, instead of the target chair. (b) OmniVLA again navigates toward the net instead of the target monitor. (c) ASD correctly directs the robot to the chair. (d) ASD correctly directs the robot to the monitor.

Figure 9: Evaluation on Doorway tasks. We visualize ASD ’s performance on the Doorway task compared with OmniVLA and NaVILA. (a) OmniVLA drives the robot straight forward and collides with the door frame on the left corner. (b) NaVILA also drives the robot straight forward and collides with the table on the right. (c) ASD produces a left-turning trajectory that allows the robot to pass through the door successfully.

Figure 10: Evaluation on Hallway tasks. We visualize ASD ’s performance on the Hallway task compared with OmniVLA and NaVILA. (a) OmniVLA navigates the robot through the door but fails to complete the subsequent right turn and hallway-following behavior. (b) NaVILA successfully completes the full multi-stage instruction. (c) ASD also successfully completes the full multi-stage instruction.

Figure 11: Generalization to Bronze Statue. We evaluate ASD , OmniVLA, and NaVILA on the Statue task and visualize representative examples. (a) ASD successfully reaches the target statue. (b) ASD produces a correct plan but fails to reach the target due to execution error on uneven terrain. (c) OmniVLA fails to generate a suitable trajectory toward the statue. (d) NaVILA moves mostly straight and fails to reach the target, due to a mismatch between the prompt style and the model’s expected instruction format.

Explore similar work

Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrapping
Oct 11, 2026Songze Li, Haoming Gu, Ziyang Gong +11Vision-Language Navigation Vision-Language-Action Models
Fudan University · Shanghai AI Laboratory · Harbin Institute of Technology +6
Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation
May 15, 2026Jin Shi, Brady Zhang, Yishun LuVLM Distillation Efficient VLA Model Inference
Department of Mechanical Engineering University College London London, United Kingdom · Department of Engineering Science University of Oxford Oxford, United Kingdom
Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation
May 26, 2026Hongyu Ding, Sizhuo Zhang, Ziming Xu +13Large Language Model-Based Robot Planning Robot Navigation
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beihang University, Beijing, China +2

Oct 11, 2026J/K move · Enter open · S save

Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrapping

Songze Li, Haoming Gu, Ziyang Gong, Zhitong Wang+10

Fudan University · Shanghai AI Laboratory · Harbin Institute of Technology+6

Vision-Language Navigation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language navigation requires agents to follow natural-language instructions to reach target locations, making high-quality language supervision essential. Since human annotation is costly, prior work has explored scalable data synthesis methods. However, existing methods typically either use separate models for navigation and instruction generation or couple the two tasks through task-specific components; most also annotate the full trajectory corpus with a fixed model before updating, which can cause generated supervision to lag behind the evolving policy. Therefore, we introduce RSI-Nav, a data-centric recursive self-improvement framework for navigation via policy-synchronized language-action dual bootstrapping, which unifies vision-language-to-action instruction following and vision-action-to-language instruction generation within a shared model. Specifically, after supervised fine-tuning establishes fundamental capabilities, RSI-Nav progressively processes unlabeled trajectories in small shards. The current model generates instructions for these trajectories, executes them to evaluate goal completion and path efficiency, retains qualified samples, and immediately updates itself before processing the next shard. This shard-wise update keeps newly generated supervision aligned with the evolving policy. We also incorporate spatial understanding data, since stronger spatial reasoning benefits navigation. Extensive experiments show that RSI-Nav reaches 68.6% SR on R2R-CE and 72.4% SR on RxR-CE, while providing stronger instruction-generation utility and modestly improved spatial understanding after recursive evolution.

Abstract

Explore similar work

Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrapping

Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation

Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation