cs.ROOct 1, 2026

Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation

Authors: Shukai Gong, Xuanran Zhai, Yintianrun Zhang, Ruopeng Cui, Ye Huang, Yiyang Fu, Dexuan Lyu, Chaojie Li, 

Organizations: Peking University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Long-horizon compositional manipulation has become increasingly important for real-world robot deployment, where a single task involves multiple coordinated subtasks. Existing world-action models (WAMs) jointly predict short-horizon visual futures and actions, but typically lack explicit subtask-level reasoning. We propose Visual Goal-conditioned Action Reasoning (ViGAR), a hierarchical framework that factorizes manipulation into a visual subgoal planner and a subgoal executor. Given the current observation and global instruction, the subgoal planner predicts a visual subgoal for the next subtask. The subgoal executor then jointly generates future visual trajectories and actions conditioned on the predicted subgoal. Both components share a pretrained world-model representation, enabling task-level planning and action generation to benefit from common physical knowledge. Moreover, our framework naturally supports in-context learning: using a global goal image as context can induce different subtask decompositions and behaviors without parameter updates. On the RoboTwin Clean2Random benchmark, ViGAR achieves 82.00% and 67.02% success rates under the Clean and Random settings, respectively, surpassing the strongest baseline by 12.86 percentage points in average success rate. Real-world robot experiments on five compositional and two in-context learning tasks further confirm the effectiveness of ViGAR.

Figures & tables

Figure 1: Overview of ViGAR. Our framework factorizes long-horizon compositional manipulation tasks into visual subgoal planning and subgoal execution, with both modules trained on task-diverse, long-horizon robot data. ViGAR achieves strong compositional manipulation performance in both simulated and real-world settings, and generalizes to unseen tasks through in-context steering.
Figure 2: Model architecture of ViGAR: (1) the subgoal planner Gθ\mathcal{G}_{\theta} generates the next subgoal image from the task instruction and current observation, optionally conditioned on a global goal image for in-context learning (ICL); (2) the policy πϕWAM\pi^{\text{WAM}}_{\phi} takes in the predicted subgoal and jointly predicts the future frames and the action chunk.
Figure 3: Implementation design in subgoal supervision: (a) temporal boundary handling via look-ahead redirection, and (b) spatial weighting via end-effector region-of-interest masks.
Table 1: Simulation manipulation performance and visual subgoal prediction quality of ViGAR. Bold : best, Underline : second best.
Figure 4: Real-world deployment results. We evaluate ViGAR on five long-horizon compositional manipulation tasks. Our method outperforms all baselines on the averaged task progress score.
Figure 5: Results of two in-context learning tasks in real-world robot deployment: (a) qualitative execution sequences and (b) task progress scores on in-domain and out-of-distribution settings.
Table 2: Ablation studies on the implementation of ViGAR. Success rates (%) over 50 RoboTwin tasks; Bold : best.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

SettingSubgoal plannerPolicy
Pretrained backboneCosmos3-NanoCosmos3-Nano
Camera viewsHead + two wristsHead + two wrists
Image size384×320384\times 320352×288352\times 288
OptimizerAdamWFusedAdam
Adam (β1,β2)(\beta_{1},\beta_{2})(0.9,0.95)(0.9,0.95)(0.9,0.99)(0.9,0.99)
Base LR2×10−52\times 10^{-5}1×10−51\times 10^{-5}
Appendix
Table 3: Training configurations and hyperparameters of ViGAR.
Figure 6: Visualization of the five real-world compositional manipulation tasks. The figure displays the task progress of Pile Paper Cups, Stack Colored Bowls, Stamp over Letters, Tidy-up Desktop , and Take-out Steam Buns .
TaskStageπ0.5\bm{\pi}_{0.5}Cosmos3-Nano-PolicyViGAR
Pile Paper CupsS10.950.850.90
S20.850.650.85
S30.650.450.80
Progress0.820.650.85
Stack Colored BowlsS10.700.750.85
S20.500.550.70
Appendix
Table 4: Stage-wise success rates and task progress scores of ViGAR on real-robot long-horizon manipulation tasks. Each task is evaluated over 20 rollouts. Bold : best task progress.
TaskStageIn-domainOOD
Setting ASetting B
Fruit ArrangementS10.950.700.60
S20.700.300.40
S30.50–0.30
S4––0.10
Progress0.720.500.35
Appendix
Table 5: Stage-wise success rates and task progress scores of ViGAR on in-context real-robot tasks. Results are computed over 20 rollouts.
Figure 8: Qualitative results of five real-world compositional manipulation tasks. The figure displays the task progress of Pile Paper Cups, Stack Colored Bowls, Stamp Over Letter, Tidy-up Desktop , and Take-out Steam Buns .
Figure 9: Empirical evidence for the subgoal look-ahead rule. Red borders indicate problematic subgoal images and the resulting manipulation behavior. ViGAR trained without subgoal look-ahead tends to regress to subtasks that have already been completed, whereas training with subgoal look-ahead alleviates this issue.
More in this topic

Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.

See all 431 papers

Explore similar work

CardsList

  1. AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation

    Sep 20, 2026Yicheng Jiang, Zesen Gan, Xiaobo Wang +8Robotic ControlLong-Horizon Robotic Manipulation

    The Hong Kong University of Science and Technology · Shenzhen University of Advanced Technology · Sangfor Technologies Inc. +4

  2. Making Foresight Actionable: Repurposing Representation Alignment in World Action Models

    Jun 10, 2026Lu Qiu, Yizhuo Li, Yi Chen +3Action-Conditioned Video GenerationRepresentation Learning

    The University of Hong Kong · XPENG Robotics

  3. World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

    Jul 30, 2026Xiangcheng Zhang, Yilun DuWorld Models for RoboticsWorld Model-Based Planning

    Harvard University