We study rigid-body motion planning through multiple sequential narrow openings, which requires long-horizon geometric reasoning because the configuration used to traverse an early opening constrains the set of reachable configurations for subsequent ones. To achieve this, we propose a geometry-aligned large language model (LLM) fine-tuning framework that generates fixed-length, machine-readable waypoint sequences that are both geometrically feasible and coordinated across openings. Our approach uses a bi-level training pipeline. First, we perform failure-driven LoRA supervised fine-tuning (SFT) on human demonstrations, which incorporates structured failure feedback to teach the model common failure modes and enforce the output format. Second, we refine the same LoRA adapters using Group Relative Policy Optimization (GRPO) with geometric verification: each sampled waypoint sequence is densified by a model-based planner and scored with a deterministic geometry-derived reward to achieve continuous-motion feasibility. To validate the effectiveness of our proposed method, we provide both quantitative and qualitative results from simulations. Our method achieves the highest success rate in both in-distribution and out-of-distribution environments and qualitatively exhibits long-horizon geometric reasoning by selecting exit poses that facilitate entry into subsequent openings.
Figures & tables
Fig. 1: Geometry-aligned LLM for Sequential Narrow-Opening Planning and a Motivating Example.
Fig. 2: Bi-level fine-tuning stage. Level-1: Failure-driven LoRA SFT with human-demonstrated waypoints. Level-2: Refinement of LoRA adapters using GRPO with geometric verification.
Method
ID
OOD
Avg. Geom. Reward ⇑
Parse ⇑
Success ⇑
C1 fail ⇓
C2 fail ⇓
C3 fail ⇓
C4 fail ⇓
Success ⇑
Llama-3.1-8B-Instruct
70.4
5.2
8.0
19.6
56.0
11.2
2.1
0.09 ± 0.16
+ LoRA SFT
100
54.6
3.2
14.4
22.8
5.0
10.4
0.43 ± 0.27
+ Failure Feedback
100
71.0
1.2
10.0
16.0
1.8
31.8
0.66 ± 0.32
+ GRPO
100
92.6
0.0
2.0
5.4
0.0
77.3
0.94 ± 0.18
TABLE I: Quantitative comparison of Failure-driven LoRA SFT + GRPO with geometric verification against baselines on in-distribution (ID) and out-of-distribution (OOD) environments (all values in %).
Fig. 3: Qualitative comparison of methods in terms of Geometric Correctness. Gray rectangles are obstacles forming sequential openings. (i-A and ii-A) Failure-driven LoRA SFT + GRPO with geometric verification : produces exit poses that are already oriented toward the heading needed for the next opening ( red ). (i-B and ii-B) sub-goal decomposed PRM : plans each opening independently using subgoal (SG, shaded) regions that yield locally feasible waypoints but leave insufficient space to align for the next opening in low-gap scenes. (iii-A, iii-B) A failure example : Failure-driven LoRA SFT alone can sometimes produce infeasible waypoints (iii-B) , while adding GRPO yields feasible waypoints for the same scene (iii-A) .
Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility. To successfully execute a task, a robot must decompose goals, select task-relevant objects, and sequence actions, while ensuring that plans satisfy spatial constraints such as limited free space and object collisions. In this work, we propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning. APIVOT learns to leverage language for semantic reasoning, while using visual thoughts as imagined future states for internal verification of geometric feasibility. On long-horizon kitchen tasks, APIVOT outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings. We find that APIVOT learns meaningful modality selection behavior, demonstrating that adaptive interleaving of vision-language thoughts improves both planning success and reasoning efficiency.
Generative motion planners typically use learned trajectory priors for initial generation, while leaving test-time repair to local continuous refinement. We introduce Masked Generative Motion Planning (MGMP), which extends the learned prior from efficient parallel generation to structural repair. A masked generative transformer generates discrete trajectory candidates in parallel, and Geometry-Guided Token Search (GGTS) uses scene geometry to target where to edit and which prior-supported alternatives to evaluate. This turns refinement into an efficient search over discrete motion alternatives, enabling route-level restructuring beyond local trajectory deformation. MGMP achieves 96% success on Ring Maze and 82% repair success on Controlled Route Invalidation on Kuka, exceeding the strongest external baselines by 23 and 25 percentage points, respectively. It further generalizes to unseen layouts, additional obstacles, unseen geometries, single- and dual-arm planning, and real-world Baxter tasks.
Solving complex long-horizon robotic tasks requires joint reasoning over abstract task structure and low-level physical interaction. While combining Vision-Language Models (VLMs) and video generation models offers a promising path for zero-shot planning, their individual tendencies to hallucinate physics or violate geometric consistency often compound over time, preventing reliable real-world execution. We introduce NovaPlan, a hierarchical framework that enables robust, zero-shot long-horizon manipulation by systematically proposing, verifying, and repairing visual plans. At the high level, a VLM planner decomposes tasks and filters out dynamically inconsistent futures by verifying multiple candidate video rollouts. To translate these imagined futures into reliable physical actions, NovaPlan utilizes a hybrid geometric representation that adaptively switches between object-centric flow and human hand flow. Finally, NovaPlan closes the loop by continuously monitoring execution to verify outcomes and synthesize local, non-prehensile corrective behaviors, such as fingertip poking, when failures occur. Across diverse multi-stage tasks, NovaPlan substantially outperforms prior zero-shot systems, achieving complex assembly and dexterous error recovery entirely without task-specific training or demonstrations. Please visit our project website for additional results: https://nova-plan.github.io/
Jiahui Fu, Junyu Nan, Lingfeng Sun +7
Robotics and AI Institute · Carnegie Mellon University · Brown University +1