Beyond Dense Futures: World Models as Structured Planners for Robotic Manipulation

Minghao Jin; Mozheng Liao; Mingfei Han; Zhihui Li; Xiaojun Chang

cs.ROMar 13, 2026

Beyond Dense Futures: World Models as Structured Planners for Robotic Manipulation

Authors: Minghao Jin, Mozheng Liao, Mingfei Han, Zhihui Li, Xiaojun Chang

Organizations: University of Science and Technology of China · Department of CV, MBZUAI

Abstract

Recent world-model-based Vision-Language-Action (VLA) architectures have improved robotic manipulation through predictive visual foresight. However, dense future prediction introduces visual redundancy and accumulates errors, causing long-horizon plan drift. Meanwhile, recent sparse methods typically represent visual foresight using high-level semantic subtasks or implicit latent states. These representations often lack explicit kinematic grounding, weakening the alignment between planning and low-level execution. To address this, we propose StructVLA, which reformulates a generative world model into an explicit structured planner for reliable control. Instead of dense rollouts or semantic goals, StructVLA predicts sparse, physically meaningful structured frames. Derived from intrinsic kinematic cues (e.g., gripper transitions and kinematic turning points), these frames capture spatiotemporal milestones closely aligned with task progress. We implement this approach through a two-stage training paradigm with a unified discrete token vocabulary: the world model is first trained to predict structured frames and subsequently optimized to map the structured foresight into low-level actions. This approach provides clear physical guidance and bridges visual planning and motion control. In our experiments, StructVLA achieves strong average success rates of 75.0% on SimplerEnv-WidowX and 94.8% on LIBERO. Real-world deployments further demonstrate reliable task completion and robust generalization across both basic pick-and-place and complex long-horizon tasks.

Figures & tables

Figure 1: Illustration of StructVLA (Structured Planner for Vision-Language-Action). Existing generative VLA methods either predict dense future observations (bottom left) , which accumulate errors and cause long-horizon plan drift, or rely on semantic or latent planning (bottom right) that lacks explicit geometric grounding and leads to physical misalignment. Our StructVLA (top) learns a physically grounded structured planner by training a world model to predict sparse structured frames, and transfers this representation to action control. This design enables stable long-horizon manipulation and demonstrates strong generalization and robustness across simulation and real-world benchmarks.

Figure 2: Overview of StructVLA. StructVLA is trained in two stages. Stage 1 (Structured Planner): an autoregressive world model predicts sparse structured frames that capture physically grounded progress anchors, conditioned on the instruction and visual context. Stage 2 (Action Policy): we fine-tune the structured planner for control by conditioning on the instruction together with interleaved observation and actions, transferring structured planning into low-level control.

Table 3

Figure 5: Quantitative Results on Real-World Deployments. Success rates on diverse physical manipulation tasks (10 trials per task). StructVLA matches or exceeds prior baselines on the foundation tasks and remains strong on the challenging long-horizon tidy-up task.

Figure 8: Qualitative comparison of real-world deployments. (a) StructVLA (Ours): Robust long-horizon execution with zero-shot generalization to OOD objects ( e.g . , pink bowl, Coke can). (b) UniVLA (Baseline): Prone to error accumulation and grasp failures, consistent with dense-prediction drift and limited generalization.

Table 8

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Qualitative analysis of history context length on task execution. We compare policy behavior under different history context lengths ( $L$ ) on two representative tasks: “Stack green cube on yellow cube” (top) and “Put spoon on towel” (bottom). With a short history ( $L=2$ ), the policy lacks sufficient spatio-temporal context to accurately infer task progress and interaction geometry. Top: this leads to imprecise placement, causing the cube to slip during stacking (red dashed box). Bottom: the policy exhibits temporal ambiguity and incorrectly re-grasps the object after it has already been placed. In contrast, a longer history ( $L=4$ ) provides richer temporal evidence and motion context, enabling accurate execution in both cases.

Figure S2 : Extended visual foresight predictions of the structured planner. These examples show that our planner predicts sparse yet informative visual subgoals that remain temporally coherent and physically grounded across diverse manipulation tasks. The predicted structured frames align with key interaction stages, such as approach, contact, transfer, and placement, supporting the central design of StructVLA: replacing dense future rollouts with compact structured foresight for control. In (e), we show a more challenging case involving complex kinematics, particularly for articulated or highly constrained object motions.

World Model-Based Planning

World model-based planning uses models of how environments change to compare imagined actions and choose goal-directed behavior.

See all 261 papers

Jul 1, 2026

Structured 4D Latent Predictive Model for Robot Planning

Video predictive models are emerging as a powerful paradigm in robotics, offering a promising path toward task generalization, long-horizon planning, and flexible decision-making. However, prevailing approaches often operate on 2D video sequences, inherently lacking the 3D geometric understanding necessary for precise spatial reasoning and physical consistency. We introduce a Structured 4D Latent Predictive Model, which predicts the evolution of a scene's 3D structure in a structured latent space conditioned on observations and textual instructions. Our representation encodes the scene holistically and can be decoded into diverse 3D formats, enabling a more complete and 3D consistent scene understanding. This structured 4D latent predictive model serves as a planner, generating future scenes that are translated into executable actions by a goal-conditioned inverse dynamics module. Experiments demonstrate that our model generates futures with strong visual quality, substantially better 3D consistency and multi-view coherence compared to state-of-the-art video-based planners. Consequently, our full planning pipeline achieves superior performance on complex manipulation tasks, exhibits robust generalization to novel visual conditions, and proves effective on real-world robotic platforms. Our website is available at https://structured-4d-model.github.io/.

Zhiyi Li, Peilin Wu, Xiaoshen Han +2

1MIT · 2Harvard University

Long-Horizon Robotic Manipulation World Models for Robotics Robot Task Planning 3D Scene Understanding

Ask PDF

Oct 13, 2026

WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction

Vision-language-action (VLA) policies excel at generalist robotic manipulation but generate actions without explicitly modeling their environmental consequences. World action models (WAMs) incorporate future prediction, yet most rely on costly video generation, while latent alternatives either use predicted futures only as auxiliary targets or infer latent actions from suboptimal visual transitions through costly large-scale video pretraining. We introduce WAVE-0 (World-model-Augmented VLA Execution), a framework that bridges VLAs and WAMs through a lightweight latent world model. By connecting the semantic understanding of pretrained VLAs with explicit future prediction, WAVE-0 allows anticipated action consequences to directly guide action generation. Built on a pretrained VLA, its 257M world model translates VLM-generated latent action plans into predicted future representations that condition the action expert, unifying latent planning, future prediction, and action generation. The world model is trained directly on ground-truth robot actions with a counterfactual ranking objective, requiring neither pixel reconstruction nor large-scale video pretraining. WAVE-0 achieves success rates of 91.47% on RoboTwin 2.0 and 98.7% on LIBERO, along with 88.0%, 61.4%, and 75.0% on LIBERO-Plus, LIBERO-Pro, and LIBERO-Para, respectively. On real-world tasks, it improves the success rate of

π_{0.5}

from 38.3% to 73.3%. These results demonstrate strong manipulation performance and generalization across diverse distribution shifts.

Haomin Wang, Anqiong Ji, Ke Wang +4

AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu AI Cloud · Awaken Robotics +1

World Models for Robotics World Action Models Vision-Language-Action Models Latent World Models

Ask PDF

Aug 27, 2026

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

Vision-language-action models (VLAs) have shown strong promise for general-purpose robotic manipulation by mapping language instructions and vision observations directly to actions. However, most VLAs primarily condition action prediction on current observations and lack an explicit mechanism for reasoning over future task dynamics, which is particularly important for fine-grained, contact-rich manipulation. We present PHR-VLA, a framework that enables planning-horizon reasoning in VLAs through privileged latent representations of future dynamics. PHR-VLA introduces a lightweight auxiliary future head that, during training, aligns the VLA's internal representations with latent dynamics extracted from future observations. Evaluation results demonstrate that local, contact-centric, patch-level latent dynamics supervision from the wrist camera improves success rate on LIBERO from 84.1% to 88.4% and on real-world disassembly tasks from 63.3% to 82.5%. Patch-level supervision from a third-person camera also improves performance on Meta-World from 56.70% to 57.8%. These results demonstrate that privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies. Project website: https://davoodsz.github.io/PHR-VLA.github.io/

Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4

J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA

Vision-Language-Action Models Reasoning for VLA Models Robotic Manipulation Language-Conditioned Robot Manipulation

Ask PDF

Abstract

World Model-Based Planning

Explore similar work

Structured 4D Latent Predictive Model for Robot Planning

WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models