Long-horizon compositional manipulation has become increasingly important for real-world robot deployment, where a single task involves multiple coordinated subtasks. Existing world-action models (WAMs) jointly predict short-horizon visual futures and actions, but typically lack explicit subtask-level reasoning. We propose Visual Goal-conditioned Action Reasoning (ViGAR), a hierarchical framework that factorizes manipulation into a visual subgoal planner and a subgoal executor. Given the current observation and global instruction, the subgoal planner predicts a visual subgoal for the next subtask. The subgoal executor then jointly generates future visual trajectories and actions conditioned on the predicted subgoal. Both components share a pretrained world-model representation, enabling task-level planning and action generation to benefit from common physical knowledge. Moreover, our framework naturally supports in-context learning: using a global goal image as context can induce different subtask decompositions and behaviors without parameter updates. On the RoboTwin Clean2Random benchmark, ViGAR achieves 82.00% and 67.02% success rates under the Clean and Random settings, respectively, surpassing the strongest baseline by 12.86 percentage points in average success rate. Real-world robot experiments on five compositional and two in-context learning tasks further confirm the effectiveness of ViGAR.
Figures & tables
Figure 1: Overview of ViGAR. Our framework factorizes long-horizon compositional manipulation tasks into visual subgoal planning and subgoal execution, with both modules trained on task-diverse, long-horizon robot data. ViGAR achieves strong compositional manipulation performance in both simulated and real-world settings, and generalizes to unseen tasks through in-context steering.
Figure 2: Model architecture of ViGAR: (1) the subgoal planner Gθ generates the next subgoal image from the task instruction and current observation, optionally conditioned on a global goal image for in-context learning (ICL); (2) the policy πϕWAM takes in the predicted subgoal and jointly predicts the future frames and the action chunk.
Figure 3: Implementation design in subgoal supervision: (a) temporal boundary handling via look-ahead redirection, and (b) spatial weighting via end-effector region-of-interest masks.
Table 1: Simulation manipulation performance and visual subgoal prediction quality of ViGAR. Bold : best, Underline : second best.
Figure 4: Real-world deployment results. We evaluate ViGAR on five long-horizon compositional manipulation tasks. Our method outperforms all baselines on the averaged task progress score.
Figure 5: Results of two in-context learning tasks in real-world robot deployment: (a) qualitative execution sequences and (b) task progress scores on in-domain and out-of-distribution settings.
Table 2: Ablation studies on the implementation of ViGAR. Success rates (%) over 50 RoboTwin tasks; Bold : best.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Subgoal planner
Policy
Pretrained backbone
Cosmos3-Nano
Cosmos3-Nano
Camera views
Head + two wrists
Head + two wrists
Image size
384×320
352×288
Optimizer
AdamW
FusedAdam
Adam (β1,β2)
(0.9,0.95)
(0.9,0.99)
Base LR
2×10−5
1×10−5
Appendix
Table 3: Training configurations and hyperparameters of ViGAR.
Figure 6: Visualization of the five real-world compositional manipulation tasks. The figure displays the task progress of Pile Paper Cups, Stack Colored Bowls, Stamp over Letters, Tidy-up Desktop , and Take-out Steam Buns .
Task
Stage
π0.5
Cosmos3-Nano-Policy
ViGAR
Pile Paper Cups
S1
0.95
0.85
0.90
S2
0.85
0.65
0.85
S3
0.65
0.45
0.80
Progress
0.82
0.65
0.85
Stack Colored Bowls
S1
0.70
0.75
0.85
S2
0.50
0.55
0.70
Appendix
Table 4: Stage-wise success rates and task progress scores of ViGAR on real-robot long-horizon manipulation tasks. Each task is evaluated over 20 rollouts. Bold : best task progress.
Task
Stage
In-domain
OOD
Setting A
Setting B
Fruit Arrangement
S1
0.95
0.70
0.60
S2
0.70
0.30
0.40
S3
0.50
–
0.30
S4
–
–
0.10
Progress
0.72
0.50
0.35
Appendix
Table 5: Stage-wise success rates and task progress scores of ViGAR on in-context real-robot tasks. Results are computed over 20 rollouts.
Figure 8: Qualitative results of five real-world compositional manipulation tasks. The figure displays the task progress of Pile Paper Cups, Stack Colored Bowls, Stamp Over Letter, Tidy-up Desktop , and Take-out Steam Buns .
Figure 9: Empirical evidence for the subgoal look-ahead rule. Red borders indicate problematic subgoal images and the resulting manipulation behavior. ViGAR trained without subgoal look-ahead tends to regress to subtasks that have already been completed, whereas training with subgoal look-ahead alleviates this issue.
As AI agents become increasingly capable, agent-driven robotic control is emerging as a compelling paradigm. However, prevailing vision-language-action (VLA) models and world action models (WAMs) still rely on natural-language instructions to specify manipulation tasks, an ill-suited interface for agent-driven control: referentially ambiguous, spatially imprecise, redundant with the agent's inherent language understanding, and entangling intent with execution. We present AR-WAM, a visual-conditioned, agent-ready world action model that replaces language with two complementary conditions: a visual grounding prompt (a bounding box of the target) denoting the interaction object and location, and a learnable operation token dictating the atomic skill to execute. Our compact 0.5B-parameter model, with a frozen pretrained visual encoder and no language encoder, predicts scene evolution within compact latent states while decoding actions, exposing the policy's intent through explicit, supervisable reasoning signals. A model-agnostic compatibility layer provides three primitives (detect, execute, and query) so that local VLMs or online agent APIs can drive the policy directly, with long-horizon memory and closed-loop error recovery delegated to the agent side. On RoboTwin 2.0, RMBench, and a real Astribot S1 dual-arm platform, AR-WAM attains the highest average success on standard manipulation (85.7% over the clean and randomized settings) and outperforms all baselines on memory-dependent and real-robot long-horizon tasks, improving success rates by 5.9% and 36.7%, respectively, while maintaining the lowest inference latency (14.1 ms). Project page is at https://ar-wam.github.io/.
Yicheng Jiang, Zesen Gan, Xiaobo Wang +8
The Hong Kong University of Science and Technology · Shenzhen University of Advanced Technology · Sangfor Technologies Inc. +4
World Action Models (WAMs) offer a promising route for robot manipulation by using video generation models to model future scene evolution before producing control actions. However, our empirical observations reveal a phenomenon: generating plausible visual futures does not always guarantee the extraction of accurate actions. To diagnose this failure, we conduct action-head attention analysis and causal interventions. We find that the action decoder fails to focus on task-relevant interaction regions and remains sensitive to perturbations in task-irrelevant areas. This reveals a representation mismatch: hidden states optimized for visual reconstruction are not inherently organized in a form useful for low-level action control. In this paper, we propose AGRA, an Action-Grounded Representation Alignment objective that regularizes the world-action interface by aligning intermediate video diffusion features with spatially coherent semantic representations from a foundation visual encoder. We evaluate AGRA on real-world manipulation tasks. Experiments show that AGRA makes world model representations more action-grounded: by focusing the action decoder on the correct interaction regions, it improves object localization accuracy and affordance understanding, and makes the policy more robust to perturbations in task-irrelevant regions. As a result, AGRA consistently improves both in-distribution performance and out-of-distribution generalization over the baseline world action model.
Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io