cs.RONov 12, 2026

What Can a VLA Learn from the Agent That Calls It? Task-Level Experience Distillation for Embodied Navigation

Authors: Jinze Tao, Yuxiao Han, Kangkang Dong, Yancong Wei, Xiaojun Zhu, Bokui Chen, Houde Liu

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University

Abstract

Vision-language-action (VLA) models drive the robot directly from camera views, yet a VLA alone shows three typical failures. It walks past the goal, stops before reaching it, and rarely recovers after going off track. Pairing a VLA with a large language model (LLM) planner fixes these failures, but queries the LLM throughout every episode, adding token costs and latency. We introduce Harness-to-Executor (H2E), a task-level distillation method that pairs an LLM planner and a frozen VLA as a heterogeneous training agent, keeps only sessions that reach the goal, and then trains a VLA alone on them so no LLM is used at test time. In these sessions, the planner gives sub-instructions and moves the robot with its own tools, detours are common, and the planner decides when to stop. H2E therefore labels every motion as a VLA action for the original task, whoever made it. It masks the loss on steps moving away from the goal and assigns end labels to states near it, so the VLA learns where the task ends. H2E sets a new state of the art on REVERIE-CE and NavRAG-CE. It raises an 8B VLA's success rate by 15.6% and 11.1% respectively, surpassing a method trained on 10M samples. The distilled VLA exceeds its agent on unseen episodes with zero LLM tokens and runs 3.7 times faster.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation

    May 26, 2026Hongyu Ding, Sizhuo Zhang, Ziming Xu +13Large Language Model-Based Robot PlanningRobot Navigation

  2. Agentic-Teacher: An Agentic Data Engine for Failure-Driven VLA Post-Training

    Nov 12, 2026Gongyue Teng, Yuqing Xie, Yixian Zhang +10VLA Model TrainingVision-Language-Action Models

  3. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

    Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Long-Horizon Robotic ManipulationLanguage-Conditioned Robot Manipulation