H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model

Jinbang Huang; Wenyuan Chen; Zhiyuan Li; Oscar Pang; Xiao Hu; Lingfeng Zhang; Yuanzhao Hu; Mark Coates

cs.ROFeb 11, 2026

H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model

Authors: Jinbang Huang, Wenyuan Chen, Zhiyuan Li, Oscar Pang, Xiao Hu, Lingfeng Zhang, Yuanzhao Hu, Mark Coates, +4 more

Organizations: Huawei Noah’s Ark Lab · University of Toronto · University of British Columbia · McGill University

Abstract

World models are becoming central to robotic planning and control by predicting future state transitions. Existing approaches mainly rely on visual, latent, or language prediction, which can be difficult to ground in executable robot actions and prone to compounding errors over long horizons. In contrast, traditional robotic task and motion planning enables structured long-horizon reasoning through compact symbolic representations of world transitions, but typically lacks synchronized visual prediction. We propose Hierarchical World Model (H-WM), which jointly predicts logical and visual state transitions by combining a high-level logical world model with a low-level visual world model. The predicted logical actions and latent visual state transitions are jointly incorporated into Vision-Language-Action (VLA) models as intermediate state guidance for long-horizon task execution. Experiments on three long-horizon benchmarks and real robots show that H-WM consistently improves VLA's performance by stabilizing long-horizon execution and mitigating error accumulation. We also construct LIBERO-Logic, a frame-level aligned dataset that pairs visual observations and continuous robot states with logical actions and predicate-based logical states.

Figures & tables

Fig. 1: Overview of H-WM and guided VLA execution. H-WM jointly models logical and visual transitions at the subtask level $m$ , while the VLA performs low-level control at the higher-frequency time step $t$ . The logical world model searches and evaluates candidate transitions to predict the next logical action $a_{m+1}$ and logical state $\mathcal{X}_{m+1}$ . Conditioned on the current observation $obs_{m}$ , robot configuration $q_{m}$ , $a_{m+1}$ , and $\mathcal{X}_{m+1}$ , the visual world model uses an understanding expert and a prediction expert to generate the latent visual subgoal $f_{\mathrm{pred}}^{m+1}$ . During execution, the understanding expert encodes $obs_{t}$ and $a_{m+1}$ , while the goal expert processes $f_{\mathrm{pred}}^{m+1}$ . The action expert takes $q_{t}$ and noisy action tokens, attends to both representations, and generates the low-level action chunk $\hat{\alpha}_{m}^{t:t+k}$ , grounding long-horizon logical guidance into visually conditioned robot execution.

Methods	Task 1	Task 2	Task 3	Task 4	Task 5	Average
Methods	Tasks Q-score / Task Success Rate
H-WM- $\pi_{0.5}$ (ours)	98.0/94.0	86.7/60.0	74.0/46.0	70.7 / 42.0	95.0/82.0	84.9/64.8
H-WM-Stable-Diffusion- $\pi_{0.5}$	92.7/82.0	80.0/46.0	65.3/ 30.0	74.0 / 34.0	93.5/80.0	81.1/54.4
Logic-guided $\pi_{0.5}$	95.3/86.0	84.7/58.0	54.7/16.0	39.3/4.0	92.0/78.0	73.2/48.4
LLM-guided $\pi_{0.5}$	84.7/54.0	80.7/42.0	68.0 /24.0	41.3/4.0	59.5/10.0	66.8/26.8
$\pi_{0.5}$	66.0/4.0	73.3/24.0	54.7/4.0	44.7/0.0	38.0/0.0	55.3/6.4

TABLE I: Performance on the LIBERO-LoHo benchmark. For each task, we report Q-Score and Success Rate (in %). The best results are shown in bold , and the second-best results are underlined .

Fig. 2: (a) Evaluation results of H-WM-guided

\pi_{0.5}

and various VLA baselines on the LIBERO-10 benchmark. (b) Evaluation results of H-WM-guided

\pi_{0.5}

and baseline methods on the RoboCerebra benchmark.

Fig. 3: (a) Real-world experiment with UR5e robot evaluating H-WM guided $\pi_{0.5}$ on long-horizon manipulation task. (b) Step-wise success rates in real-robot experiments.

Explore similar work

World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis
Jun 4, 2026Yi Yang, Zhihong Liu, Siqi Kou +9Efficient World-Action Model Vision-Language-Action Framework
SJTU · SII · HUST +4
Beyond a single latent space: a dual-latent world model for long-horizon planning
Sep 29, 2026Delin Zhao, Zhengrong Yue, Shaobin Zhuang +6Latent World Models Long-Horizon Planning
Nanjing University · Shenzhen University of Advanced Technology · Shanghai Jiao Tong University +1
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent World Models Action Prediction
Jilin University · Zhongguancun Academy · Nankai University +4

Jun 4, 2026, cs.ROJ/K move · Enter open · S save

World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis

Yi Yang, Zhihong Liu, Siqi Kou, Yiyang Chen+8

SJTU · SII · HUST+4

Efficient World-Action Model Vision-Language-Action Framework World Models Long-Horizon Task Planning Diffusion Transformers

Open paper AskPDF arXiv

We propose world-language-action (WLA) models as a new class of embodied foundation models. WLA takes textual instructions, images, and robot states as inputs to jointly predict textual subtasks, subgoal images, and robot actions, conjoining the \emph{world modeling interface} to learn from extensive egocentric videos as in the world-action model (WAM) and the \emph{language reasoning} capacities to solve complex long-horizon tasks as in vision-language-action (VLA) models. At the core of WLA lies an \emph{autoregressive (AR)} Transformer backbone, instead of a bidirectional diffusion Transformer as in WAMs, to predict the \emph{next state}, comprising the \emph{semantic-level} textual intention and complementary \emph{fine-grained} physical dynamics. The physical dynamics are supervised by the world modeling objective based on a dedicated World Expert, and are leveraged to ease the characterization of the state-action correlation for the Action Expert. WLA leverages meta-queries to make the world prediction \emph{implicitly} impact the action generation so that the former can be disabled during inference. The world prediction can also be activated to enable test-time scaling for improved robot control. Our WLA-0 prototype, with 2B active parameters, achieves 40 ms per inference on an NVIDIA RTX 5090. Evaluations across simulated and real-world environments demonstrate that WLA-0 achieves state-of-the-art multi-task and long-horizon learning abilities, e.g., 92.94% success rate on RoboTwin2.0 Clean and 56.5% success rate on RMBench. WLA-0 also holds the promise to learn novel tasks directly from \emph{cross-embodiment robot videos} without action annotations.

Abstract

Explore similar work

World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis

Beyond a single latent space: a dual-latent world model for long-horizon planning

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies