Learning and Verifying Progress: Closed-Loop Agentic Planning with Progress-Enhanced Execution for Long-Horizon Vision-Language Navigation
Organizations: AgiBot. · School of Advanced Manufacturing and Robotics, Peking University.
Abstract
Long-horizon vision-language navigation (VLN) requires agents to track instruction progress and recover from local errors before they propagate across subtasks. However, executors trained on expert trajectories encounter policy-induced state shifts during deployment, and their stop signals do not reliably indicate subtask completion. We introduce Learning and Verifying Progress for VLN (LeaP-VLN), an agentic VLN system that couples progress-aware execution with independent progress verification through plan--execute--verify--recover cycles. LeaP-VLN leverages progress both explicitly, through a progress verifier that verifies subtask completion before the agent advances and triggers dual-phase recovery upon route deviation, and implicitly, by co-training the executor on progress reasoning and action prediction to enhance the executor's progress awareness and corrective control. To adapt the executor to this closed-loop interface, we propose a two-stage progress-enhanced curriculum: behavior cloning on expert demonstrations, followed by joint training on demonstrations, expert action corrections, and progress-reasoning supervision. Policy-generated trajectories expose the executor to both intra-subtask drift and inter-subtask state shifts, aligning training with the deviations encountered during closed-loop execution. Extensive experiments on R2R-CE and RxR-CE demonstrate state-of-the-art success rates among monocular methods trained without additional navigation datasets, while real-world experiments further validate the effectiveness of LeaP-VLN in long-horizon navigation.
Figures & tables
| Method | Observation | R2R-CE Val-Unseen | RxR-CE Val-Unseen | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| S.RGB | Pano. | Depth | Odom. | NE | OSR | SR | SPL | NE | SR | SPL | |
| CMA [ 2 ] | ✓ | ✓ | ✓ | 6.20 | 52.0 | 41.0 | 36.0 | 8.76 | 26.5 | 22.1 | |
| InstructNav [ 8 ] | ✓ | ✓ | ✓ | 6.89 | – | 31.0 | 24.0 | – | – | – | |
| ETPNav [ 24 ] | ✓ | ✓ | ✓ | 4.71 | 65.0 | 57.0 | 49.0 | 5.64 | 54.7 | 44.8 | |
| HNR [ 25 ] | ✓ | ✓ | ✓ | 4.42 | 67.0 | 61.0 | 51.0 | 5.50 | 56.3 | 46.7 | |
| NavFoM [ 15 ] | ✓ | 4.61 | 72.1 | 61.7 | 55.3 | 4.74 | 64.4 | 56.2 | |||
| Components | R2R-CE Val-Unseen | RxR-CE Val-Unseen | ||||||||
| A | B | C | D | NE | OSR | SR | SPL | NE | SR | SPL |
| ✓ | 4.96 | 64.8 | 55.5 | 49.7 | 5.78 | 58.6 | 42.3 | |||
| ✓ | ✓ | 4.46 | 69.3 | 61.7 | 52.4 | 4.33 | 65.2 | 51.5 | ||
| ✓ | ✓ | 3.78 | 73.0 | 67.0 | 59.0 | 4.86 | 62.5 | 50.1 | ||
| ✓ | ✓ | ✓ | 3.91 | 71.7 | 66.7 | 57.6 | 4.10 | 67.9 | 54.4 | |
| ✓ | ✓ | ✓ | ✓ | 3.71 | 73.9 | 67.5 | 59.1 | 3.89 | 69.6 | 52.8 |
Vision-Language Navigation
Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.
See all 334 papers