cs.AIOct 13, 2026

Write Once, Plan Many: Building Reusable Visual Planning Systems from One Scene

Authors: Ruixiao Yang, Yilun Hao, Mingxin Yu, Yang Zhang, Chuchu Fan

Organizations: MIT · MIT-IBM Watson AI Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language models (VLMs) can reason about visual planning tasks, but querying them for every instance incurs repeated inference cost and latency. Generating reusable planning programs offers an alternative, but constructing reliable planning systems for new problem domains without extensive supervision remains challenging. We present a training-free framework that constructs reusable perception and planning programs from a domain specification and a single practice scene without a paired solution. The central challenge is obtaining useful feedback from the limited input. Our framework addresses it by leveraging VLM's ability to provide a renderer and a dynamic model based on its scene understanding. Rendering detected states enables comparison with the practice scene, while checking simulated action outcomes against the domain specification helps expose errors in rule implementation. Together, these checks guide refinement of the grounding program and planner without requiring solution labels. After refinement and component selection, the programs are frozen and solve new instances without further VLM calls. The same framework constructs systems across domains without manual pipeline redesign. Across fifteen domain variants spanning 2D and 3D scenes with discrete and continuous state, our systems achieve 99.3% average planning success, compared with 80.9% for the strongest domain-level program-generation baseline and 90.7% for the strongest instance-level baseline. Cost analyses show a favorable success-cost trade-off, and ablations confirm the contributions of the executable domain model, execution feedback, and component selection.

Figures & tables

Figure 1: Reusable planning is both capable and efficient. Across diverse visual planning domains, our method achieves the strongest overall success–cost trade-off, outperforming domain-level program-authoring baselines while exceeding costly per-instance inference methods.
Figure 2: Illustration of RVPA framework . (A) Given a task description and unlabeled examples, the VLM constructs and selects reusable state-reading and planning programs using label-free feedback. (B) These programs are then frozen and applied directly to each new instance, producing plans through local execution without further VLM calls.
Figure 3: Visual-planning task suite. Representative examples summarize the 15 evaluated domain variants. The 2D domains are shown above the 3D domains, with discrete- and continuous-state domains grouped within each row. Polyomino includes original, affine, and polar variants (polar shown), while tangram includes piece and silhouette variants (silhouette shown).
TaskDirect Code WritingPer-domain Coding AgentOurs
LunaTerraSolAstraBof NNRefineParselCodeTreeOpenHands
2D, discrete state
Maze0.600.760.800.800.960.680.600.760.801.00
Sokoban0.040.760.931.000.200.390.250.340.831.00
Frozen Lake0.200.420.801.000.370.510.420.681.001.00
Polyomino0.000.570.801.000.400.410.401.001.001.00
Table 1: Domain-level program construction. Mean success rate (SR; higher is better) across 15 visual-planning domains averaged over five independent runs. Each run constructs a fresh reusable system from the same fixed practice instance; the resulting code is frozen and evaluated on the same held-out instances. Our framework and coding agents use GPT-5.6-Luna with high reasoning effort and a 30k output-token limit, while direct code writing uses the indicated backbone among GPT-5.6-Luna/Terra/Sol and GPT-6-Astra .
 2D Disc.2D Cont.3D Disc.3D Cont.Avg.
VariantSRΔ\DeltaSRΔ\DeltaSRΔ\DeltaSRΔ\DeltaSRΔ\Delta
Full system100.0–97.3–100.0–100.0–99.3–
w/o Renderer100.00.062.5−- 34.866.7−- 33.333.0−- 67.074.4−- 24.9
w/o Grounding refinement100.00.073.8−- 23.566.7−- 33.336.0−- 64.077.8−- 21.5
Random grounding selection94.3−- 5.778.8−- 18.566.3−- 33.757.0−- 43.079.6−- 19.7
w/o planner refinement100.00.017.5−- 79.844.4−- 55.668.0−- 32.062.6−- 36.7
Table 3: Component ablations across domain types. Each cell reports mean success rate (SR; %) and its change from the full system ( Δ\Delta ; percentage points). Results are averaged within each domain type. Each ablation changes one component while keeping the remaining system fixed.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

TaskDirect Code WritingPer-domain Coding AgentOurs
LunaTerraSolAstraBof NNRefineParselCodeTreeOpenHands
2D, discrete state
Maze0.550.430.450.450.090.440.550.430.450.00
Sokoban0.090.430.160.000.300.530.350.410.220.00
Frozen Lake0.450.530.450.000.490.470.530.440.000.00
Polyomino0.000.520.450.000.550.540.550.000.000.00
Appendix
Table 4: Domain-level program construction: standard deviations. Sample standard deviation of success rate across five independent runs.
MethodTasks$/task construction$/instance API costSuccess
Per-task program construction
One-shot Luna160.00670 ∗0.10
One-shot Terra160.07810 ∗0.36
One-shot Sol160.19170 ∗0.50
One-shot Astra160.52040 ∗0.77
Best-of-N160.05640 ∗0.26
Appendix
Table 5: API cost and planning success. Costs (USD) and success rates are averaged equally across the reported tasks. Per-instance costs reflect inference for baselines and amortized construction for Ours. Accounting and protocol details are provided in the text.
More in this topic

Program Synthesis

Program synthesis automatically constructs executable programs from examples, specifications, demonstrations, or natural-language descriptions.

See all 175 papers

Explore similar work

CardsList
Dec 11, 2025

CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates

Vision Language Models (VLMs) have shown promising planning capabilities, yet their success remains confined to the text domain, leaving visual decision-making relatively underexplored. Addressing this gap, we introduce Corrective Sequence Planning (CoSPlan) benchmark, where VLMs must plan a sequence of visual actions from an initial scene to a target scene. CoSPlan evaluates models on their ability to imagine and execute a coherent set of visual steps required to reach the goal (Step Completion). To prevent any shortcuts that simply describe the final scene, we introduce an erroneous action in decision-making, which must be detected (Error Detection) and corrected to reach the goal, enabling a deeper understanding of the task. CoSPlan spans across 4 tasks: maze navigation, block re-arrangement, image reconstruction, and object re-organization. Despite using advanced reasoning strategies such as Chain-of-Thought and Scene Graphs, VLMs struggle on CoSPlan, while still showing promising performance in the text domain. Addressing this, we propose Scene Graph Incremental updates (SGI), a novel training-free method to transform images into `textual' scene graphs, enabling step-by-step reasoning through iterative scene graph refinement. SGI yields an average of ~4.4% improvement on CoSPlan w/ generalization on PlanBench and VQA. Link for solving puzzles on the project page.

Shresth Grover, Priyank Pathak, Akash Kumar +1

UCF Institute of Artificial Intelligence, University of Central Florida (UCF)

VLM ReasoningVisually Grounded ReasoningGraph Reasoning
AskPDF
May 19, 2025

ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans, with recent works extending this idea to visual domains using Vision-Language Models (VLMs). However, an open-source benchmark for comparing these approaches under matched conditions is missing, due to a lack of visual benchmarks that support symbolic planning. We present ViPlan, the first open-source benchmark for comparing VLM-grounded symbolic approaches (VLM-as-grounder) with direct VLM planning methods (VLM-as-planner). ViPlan introduces a series of increasingly challenging tasks in two visual domains: a visual variant of the classic Blocksworld planning problem and a simulated household robotics environment. Averaged across methods, we find VLM-as-grounders to outperform direct VLM planning in Blocksworld (solving 46% of the tasks against 9%), where image grounding is both crucial and accurate. However, in the household robotics tasks, where linguistic knowledge helps, VLM-as-planner methods are greatly superior to VLM-as-grounder approaches (solving 34% of the tasks against 5%), which are hindered by partial observability. Thus, ViPlan domains capture fundamental shortcomings of both planning approaches, which we further diagnose with a qualitative failure analysis. Finally, across methods, we observe no consistent benefit from Chain-of-Thought prompting, suggesting persistent limitations in current VLMs' visual reasoning abilities.

Matteo Merler, Nicola Dainese, Minttu Alakuijala +5

Fondazione Bruno Kessler · Department of Computer Science, Aalto University · Department of Mathematics, Università degli Studi di Padova

Large Language Model-Based Robot PlanningSymbolic PlanningVisually Grounded Reasoning
AskPDF
Apr 13, 2026

Grounded World Model: Latent Planning with Language Goals

World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while pi0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and-place sub-tasks, comparable to the modular planner's 52/60, with the full system running locally on a single consumer GPU. Project website: https://quanyili.github.io/gwm-wiser/.

Quanyi Li, Lan Feng, Haonan Zhang +4

EPFL · NUS

Language-Conditioned Robot PlanningWorld Models for RoboticsWorld Model-Based PlanningAction-Conditioned World ModelsGeneralization in Robotic ManipulationVision-Language Grounding+7
AskPDF