Training Generalist Navigation Policies via Action Self-Distillation

Bowen Feng; Keqin Wang; Yulong Yang; Amogh Joshi; Christine Allen-Blanchette; Dhruv Shah; Felix Heide

cs.ROOct 13, 2026

Training Generalist Navigation Policies via Action Self-Distillation

Authors: Bowen Feng, Keqin Wang, Yulong Yang, Amogh Joshi, Christine Allen-Blanchette, Dhruv Shah, Felix Heide

Organizations: Princeton University, USA

Abstract

Language provides a natural interface for humans to specify and reason about goals, but training language-conditioned robotic policies and reasoning typically requires expensive manual annotations. We propose Action Self-Distillation (ASD), a method for learning language-conditioned reasoning for robotic navigation without human-annotated language or reasoning labels. ASD uses an open-source VLM, which contains broad world knowledge and reasoning capabilities, as both a teacher network and a student policy. During training, the self-teacher receives privileged trajectory feedback and infers the underlying navigation goal together with goal-oriented reasoning. The student is then trained to condition on the inferred navigation goal, predict future ego-frame trajectory, and provide a description of the scene semantics, plan, and reasoning, from onboard observations alone. Across 13 challenging indoor and outdoor real-world navigation tasks, our method improves success rate over the strongest baseline by $30\%$ in indoor settings, $30\%$ on multi-stage tasks, and achieves $95\%$ success rate on outdoor tasks, validating that privileged trajectory feedback can enable self-distillation for scalable generalist navigation policies without human annotations.

Figures & tables

Figure 1: Action self-distillation. The self-teacher uses privileged ground-truth trajectory feedback to extract scene semantics, infer a language goal, and generate reasoning traces explaining the demonstrated behavior. Blue arrows denote the feedback-conditioned information flow to the self-teacher, while pink arrows denote the student pathway. The student observes only deployable inputs and learns to align its reasoning and trajectory tokens with the self-teacher, removing the need for external goal or reasoning annotations.

Figure 2: Action Self-Distillation. A single pre-trained VLM is used as both the self-teacher $\pi_{\phi}$ and student $\pi_{\theta}$ . During training, the self-teacher receives privileged information by projecting the ground-truth future trajectory $y$ onto the front-view observation as $o^{*}_{0}$ , enabling it to infer a language goal $\hat{g}$ and scene-grounded reasoning trace $R$ . The student observes only deployable inputs $o_{0}$ and learns to match the teacher’s reasoning tokens with a KL loss, while predicting future trajectory tokens $\tilde{\tau}_{0:H}$ with a cross-entropy loss against the ground-truth trajectory $y$ . The teacher weights are updated as an EMA of the student.

Method	Directions			Objects				Multi-stage		Avg.
Method	Straight Door - (1 direction)	Left chair + (2 directions)	Right chair ∗ (3 directions)	Blue - chair	Trash - bin	Chair + (2 objects)	Monitor ∗ (3 objects)	Doorway +	Hallway ∗	Avg.
OmniVLA	0.8	0.9	0.2	0.7	1.0	0.4	0.2	0.7	0.0	0.54
NaVILA	0.3	0.2	0.1	0.4	0.1	0.3	0.2	0.3	0.5	0.27
Qwen2.5-VL- ASD	0.9	1.0	0.8	1.0	0.9	0.9	0.7	0.8	0.6	0.84
Qwen3.5-VL- ASD	1.0	1.0	0.9	1.0	1.0	1.0	0.9	0.9	0.8	0.94

Table 1: Indoor Language-Conditioned Navigation. We evaluate tasks across three categories: directional, object-centric, and multi-stage navigation. Difficulty level increases with the number of possible directions for directional tasks, and the number of objects in the scene for object-centric tasks. In the multi-stage category, Doorway requires the robot to pass through a door, while Hallway further requires it to turn right after the doorway and move along the hallway. We use

-

+

, and

*

to denote easy, medium, and hard tasks. More detailed task descriptions and visualizations are provided in Section C.1 of the supplementary material.

Method	Sidewalk	Statue	Tree	Car	Avg.
OmniVLA	1.0	0.6	1.0	1.0	0.90
NaVILA	0.9	0.1	0.0	0.3	0.33
Qwen2.5-VL- ASD	1.0	0.8	1.0	1.0	0.95
Qwen3.5-VL- ASD	1.0	0.9	1.0	1.0	0.98

Table 2: Outdoor Language-Conditioned Navigation. We evaluate on 4 different tasks, including navigate along the sidewalk, navigate to the statue/tree/car . More detailed task descriptions and visualizations are provided in Section C.2 of the supplementary material.

Figure 3: Steerability Demonstration . We demonstrate ASD ’s steerability under color (left), direction (middle), and object (right) prompts. Within the same setting, our method can be directed to different targets.

Figure 4: Single-Stage Navigation Across Diverse Indoor and Outdoor Environments. We demonstrate ASD ’s capability to execute single-stage navigation goals specified in natural language. The examples are ordered from left to right as two indoor tasks ( right chair and monitor ) and three outdoor tasks ( car , statue , and tree ). Full videos are provided in the Supplementary Material.

Figure 5: Multi-Stage Reasoning Capabilities for Outdoor and Indoor Navigation Tasks. We demonstrate ASD ’s capability to execute complex, multi-stage navigation goals. The first row contains an indoor navigation demonstration, while the bottom row contains an outdoor navigation demonstration, both with their respective tasks described on the figure. Full videos are provided in the Supplementary Material.

Figure 6: Reasoning Legibility. We present representative findings from our real-world experiments, showing predicted trajectories alongside the corresponding highlighted reasoning traces. These examples demonstrate that the generated reasoning is interpretable and well aligned with the predicted robot behavior.

Method	SR $\uparrow$	Time (s) $\downarrow$
OmniVLA	0.9	25.3
NaVILA	0.2	28.9
Qwen2.5-VL	0.0	–
Claude Opus 4.7	0.7	846.1
Qwen2.5-VL- ASD	1.0	15.3

Table 3: Task success rates and completion times of foundation models on indoor navigation task . We extend the evaluation of Left chair in Table 1 by including Qwen2.5-VL and Claude Opus 4.7, and report the task completion time for all methods. We omit the completion time for Qwen2.5-VL because its predicted actions are generally physically infeasible.

Method	Directions	Objects		Multi-stage	Outdoor	Avg.
Method	Right chair	Chair	Monitor	Hallway	Car	Avg.
Qwen2.5-VL- ASD (goal-only supervision)	0.4	0.2	0.2	0.1	0.4	0.26
Qwen2.5-VL- ASD (reasoning off at inference)	0.1	0.2	0.2	0.0	0.2	0.14
Claude Opus 5 labels + SFT	0.3	0.5	0.3	0.3	0.4	0.36
Qwen2.5-VL labels + SFT	0.4	0.3	0.3	0.0	0.1	0.22
NaVILA (RECON + FrodoBots-2K SFT)	0.0	0.1	0.2	0.3	0.2	0.16

Table 4: Frozen-teacher baselines and reasoning ablations. Success rate on representative tasks from Table 1 and Table 2 requiring directional, object-level, multi-stage, and outdoor reasoning. Goal-only supervision trains on self-teacher-inferred goals without reasoning traces. Reasoning off at inference disables reasoning at test time. Labels + SFT baselines fine-tune on offline annotations from a frozen Claude Opus 5 or Qwen2.5-VL teacher. Best results are in bold , second best are underlined .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value	Hyperparameter	Value
Backbone VLM	Qwen2.5-VL-3B-Instruct	Image resolution	dataset-native
Max output tokens	512	Past waypoints	8
Future waypoints	16	Prediction horizon	6.4 s
Waypoint frequency	2.5 Hz	Train/validation split	9:1
Optimizer	adamw_torch_fused	Learning rate	$1\times 10^{-5}$
Batch size per GPU	1	Gradient accumulation	2

Appendix

Table 5: Training hyperparameters. Main hyperparameters used for model training and inference in real-world experiments. Hyperparameters not listed here use the HuggingFace Transformers defaults.

	Task	Difficulty	Prompt
Directions	Door	$\star$	"Navigate straight to the door"
	left chair	$\star\star$	"Navigate to the chair on the left"
	right chair	$\star\star\star$	"Navigate to the chair on the right"
Objects	Blue chair	$\star$	"Navigate to the blue chair"
	Trash Bin	$\star$	"Navigate to the trash bin"
	Chair	$\star\star$	"Navigate to the chair"

Appendix

Table 6: Prompt for language-conditioned navigation benchmarks.

Figure 7: Summary of Benchmark Tasks. Panels (a)–(h) show the indoor tasks reported in Table 1, covering directional, object-centric, and multi-stage navigation scenarios. Panels (i)–(l) show the outdoor tasks reported in Table 2. The target object or goal region for each task is highlighted with a cyan bounding box.

Method	RECON				FrodoBots-2K
Method	ADE $\downarrow$	ATE $\downarrow$	MAE $\downarrow$	MSE $\downarrow$	ADE $\downarrow$	ATE $\downarrow$	MAE $\downarrow$	MSE $\downarrow$
Gaussian Policy	0.226	0.321	0.144	0.059	0.034	0.046	0.019	0.002
Gaussian Mixture Model	0.260	0.364	0.166	0.084	0.036	0.050	0.020	0.002
Diffusion Policy [ 36 ]	0.180	0.238	0.108	0.045	0.034	0.045	0.020	0.003
Qwen2.5-VL+SFT [ 32 ]	0.192	0.247	0.115	0.042	0.064	0.061	0.027	0.007
Alpamayo-1+SFT [ 37 ]	0.303	0.362	0.188	0.091	0.093	0.106	0.055	0.041

Appendix

Table 7: Open-loop evaluation on RECON and FrodoBots-2K. ASD achieves the best performance across both datasets under the open-loop evaluation protocol. Lower is better for all metrics.

Figure 8: Evaluation on Multi-object Tasks. We visualize the Chair and Monitor tasks for OmniVLA (left) and ASD (right). (a) OmniVLA navigates the robot toward a distractor object, the net, instead of the target chair. (b) OmniVLA again navigates toward the net instead of the target monitor. (c) ASD correctly directs the robot to the chair. (d) ASD correctly directs the robot to the monitor.

Figure 9: Evaluation on Doorway tasks. We visualize ASD ’s performance on the Doorway task compared with OmniVLA and NaVILA. (a) OmniVLA drives the robot straight forward and collides with the door frame on the left corner. (b) NaVILA also drives the robot straight forward and collides with the table on the right. (c) ASD produces a left-turning trajectory that allows the robot to pass through the door successfully.

Figure 10: Evaluation on Hallway tasks. We visualize ASD ’s performance on the Hallway task compared with OmniVLA and NaVILA. (a) OmniVLA navigates the robot through the door but fails to complete the subsequent right turn and hallway-following behavior. (b) NaVILA successfully completes the full multi-stage instruction. (c) ASD also successfully completes the full multi-stage instruction.

Figure 11: Generalization to Bronze Statue. We evaluate ASD , OmniVLA, and NaVILA on the Statue task and visualize representative examples. (a) ASD successfully reaches the target statue. (b) ASD produces a correct plan but fails to reach the target due to execution error on uneven terrain. (c) OmniVLA fails to generate a suitable trajectory toward the statue. (d) NaVILA moves mostly straight and fails to reach the target, due to a mismatch between the prompt style and the model’s expected instruction format.

Explore similar work

Oct 11, 2026

Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrapping

Vision-language navigation requires agents to follow natural-language instructions to reach target locations, making high-quality language supervision essential. Since human annotation is costly, prior work has explored scalable data synthesis methods. However, existing methods typically either use separate models for navigation and instruction generation or couple the two tasks through task-specific components; most also annotate the full trajectory corpus with a fixed model before updating, which can cause generated supervision to lag behind the evolving policy. Therefore, we introduce RSI-Nav, a data-centric recursive self-improvement framework for navigation via policy-synchronized language-action dual bootstrapping, which unifies vision-language-to-action instruction following and vision-action-to-language instruction generation within a shared model. Specifically, after supervised fine-tuning establishes fundamental capabilities, RSI-Nav progressively processes unlabeled trajectories in small shards. The current model generates instructions for these trajectories, executes them to evaluate goal completion and path efficiency, retains qualified samples, and immediately updates itself before processing the next shard. This shard-wise update keeps newly generated supervision aligned with the evolving policy. We also incorporate spatial understanding data, since stronger spatial reasoning benefits navigation. Extensive experiments show that RSI-Nav reaches 68.6% SR on R2R-CE and 72.4% SR on RxR-CE, while providing stronger instruction-generation utility and modestly improved spatial understanding after recursive evolution.

Songze Li, Haoming Gu, Ziyang Gong +11

Fudan University · Shanghai AI Laboratory · Harbin Institute of Technology +6

Vision-Language Navigation Vision-Language-Action Models

Ask PDF

May 15, 2026

Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation

Billion-parameter Vision-Language-Action (VLA) policies have recently shown impressive performance in robotic manipulation, yet their size and inference cost remain major obstacles for real-time closed-loop control. We introduce VLA-AD, a distillation framework that uses a Vision-Language Model as an offline semantic supervisor to transfer large VLA teachers into lightweight student policies. Instead of relying only on low-level action imitation, VLA-AD augments teacher-provided 7-DoF action targets with high-level semantic guidance, including task phase anchors and multi-frame operating-direction descriptions. These auxiliary signals are used only during training: at test time, the student policy runs independently, with neither the VLA teacher nor the VLM required. We evaluate VLA-AD on three LIBERO benchmark suites. Using OpenVLA-7B as the teacher, our method produces a 158M-parameter student, yielding a

44\times

reduction in model size while matching the teacher with only a

0.27\%

average relative gap. The resulting policy runs at 12.5 Hz on an RTX 4090, achieving a

3.28\times

inference speedup over OpenVLA-7B. We further show that the same semantic distillation pipeline generalizes to a different

π_{0.5}

-4B teacher, where the student outperforms the teacher on two suites and remains within

0.53\%

on libero\_goal. Additional analysis indicates that phase-level supervision and multi-frame directional cues make the student less sensitive to noisy teacher actions, such as erroneous high-frequency gripper changes. Overall, VLA-AD demonstrates that offline semantic guidance from VLMs can substantially improve the efficiency, robustness, and deployability of VLA policy distillation.

Jin Shi, Brady Zhang, Yishun Lu

Department of Mechanical Engineering University College London London, United Kingdom · Department of Engineering Science University of Oxford Oxford, United Kingdom

VLM Distillation Efficient VLA Model Inference Visuomotor Policy Learning Efficient VLA Models Robotic Manipulation Vision-Language-Action Models+1

Ask PDF

May 26, 2026

Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation

Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen. The dominant approach has been to scale vision-language-action (VLA) foundation models on ever-larger collections of robot trajectories. This paper argues that, for navigation specifically, generality can be obtained structurally, not only through data scale. The underlying decision structure of navigation reduces to a single Language-Vision-Robot Actions Translation. The language action emits semantic-level directional command and the vision action emits a pixel-level visual target. Both outputs lie inside the natural output manifold of pretrained multimodal large language models (MLLMs), so the task can be reasoned about by an agent rather than learned from robot data. Therefore, we present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families (VLN-CE, ObjectNav, EQA, and Aerial-VLN) and to four heterogeneous real robots (Wheeled, Quadruped, Humanoid robot, and a self-built UAV) in a zero-shot manner. Two agent-loop mechanisms make this unification practical. TODO List Memory (TDM) rewrites a structured checklist of pending sub-goals at every step, reciting the unfinished items back into the agent's most recent attention window. Second Chance Backtrack (SCB) rolls the robot back to the pre-error state and conditions the agent's next plan on the failed sub-trajectory, turning single-pass navigation into a self-correcting process. With zero training effort, Uni-LaViRA reaches 60.7% SR on VLN-CE R2R, 51.3% on VLN-CE RxR, 77.7% on HM3D-v2, 60.0% on HM3D-OVON, 54.7% on MP3D-EQA, and 40.0% on OpenUAV, matching or even surpassing recent training navigation foundation models that consume millions of samples and thousands of GPU-hours.

Hongyu Ding, Sizhuo Zhang, Ziming Xu +13

Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beihang University, Beijing, China +2

Large Language Model-Based Robot Planning Robot Navigation Vision-Language Navigation Multimodal Agents

Ask PDF