cs.ROApr 13, 2026

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps

Authors: Liaoyuan Fan, Zetian Xu, Chen Cao, Wenyao Zhang, Mingqi Yuan, Jiayu Chen

Organizations: INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China · The University of Hong Kong, Hong Kong SAR, China · Shanghai Jiao Tong University, Shanghai, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Pretrained video generation models provide strong priors for robot control, but existing unified world action models still struggle to decode reliable actions without substantial robot-specific training. We attribute this limitation to a structural mismatch: while video models capture how scenes evolve, action generation requires explicit reasoning about where to interact and the underlying manipulation intent. We introduce AIM, an intent-aware unified world action model that bridges this gap via an explicit spatial interface. Instead of decoding actions directly from future visual representations, AIM predicts an aligned spatial value map that encodes task-relevant interaction structure, enabling a control-oriented abstraction of future dynamics. Built on a pretrained video generation model, AIM jointly models future observations and value maps within a shared mixture-of-transformers architecture. It employs intent-causal attention to route future information to the action branch exclusively through the value representation. We further propose a self-distillation reinforcement learning stage that freezes the video and value branches and optimizes only the action head using dense rewards derived from projected value-map responses together with sparse task-level signals. To support training and evaluation, we construct a simulation dataset of 30K manipulation trajectories with synchronized multi-view observations, actions, and value-map annotations. Experiments on RoboTwin 2.0 benchmark show that AIM achieves a 94.0% average success rate, significantly outperforming prior unified world action baselines. Notably, the improvement is more pronounced in long-horizon and contact-sensitive manipulation tasks, demonstrating the effectiveness of explicit spatial-intent modeling as a bridge between visual world modeling and robot control.

Figures & tables

Figure 1: (a) A typical unified world action model decodes actions directly from shared future visual representations. (b) AIM (Ours) introduces a spatial value-map interface for action decoding.
Figure 2: Framework of AIM. Given multi-view RGB videos and a language instruction, AIM jointly learns future frame generation, action prediction, and spatial value map estimation in Stage I, where intent-causal attention transfers task-relevant intention from visual prediction to action generation; in Stage II, the policy is further optimized with GRPO using both sparse and dense rewards.
Simulation Taskπ0.5\pi_{0.5}X-VLAMotusStage1AIM (Ours)
 EasyHardEasyHardEasyHardEasyHardEasyHard
Adjust Bottle100%99%100%99%89%93%98%99%100%100%
Beat Block Hammer96%93%92%88%95%88%98%100%100%100%
Blocks Ranking RGB92%85%83%83%99%97%91%77%92%77%
Blocks Ranking Size49%26%67%74%75%63%47%44%47%43%
Click Alarmclock98%89%99%99%100%100%98%99%100%100%
Table 1: Per-task success rates SR on 50 RoboTwin simulation tasks under Easy and Hard settings.
Settingπ0\pi_{0}π0.5\pi_{0.5}X-VLAMotusFast-WAMGiga-WorldLingBot-VAStage1AIM (Ours)
Easy65.9%82.7%72.8%88.7%91.9%87.0%92.9%93.0%94.0%
Hard58.4%76.8%72.8%87.0%91.8%85.0%91.6%92.0%92.1%
Average62.2%79.8%72.8%87.8%91.8%86.0%92.2%92.5%93.1%
Table 2: RoboTwin 2.0 average success rate (SR) under Easy and Hard settings.
Figure 3: Representative task execution processes in Robotwin 2.0 benchmark [ 2 ] , including place mouse pad , press stapler , scan object , turn switch , and open laptop . The left column shows examples under Easy setting, while the right column shows examples under Hard setting.
More in this topic

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 211 papers

Explore similar work

CardsList
Aug 22, 2026

DELE-w0.5: Inferring Action from Future Latent State for Robotic Manipulation

World-Action Models (WAMs) build robot control on video-generation backbones, which jointly predict dense future visual trajectories and robot actions. We argue that video generation is an unnecessary intermediate objective for world-action modeling. For robotic manipulation, the goal of a world model is not to reproduce how the world looks at every intermediate moment, but to predict the state that the world will reach after an action is executed. The intermediate frames only describe the visual transition between physical states, which consumes substantial model capacity and computation, but do not directly specify the physical outcome that the robot action is intended to produce. In this paper, we propose DELE-w0.5, which infers robot actions from predicted future states without relying on video generation. Concretely, DELE-w0.5 infers the action sequence from its corresponding compact future latent state. The future latent state captures the action-relevant physical outcome of robot interaction and serves as an explicit bridge between world modeling and action generation. The core design principle of DELE-w0.5 is to model how the physical world changes under robot actions, rather than how its visual appearance evolves frame by frame. This formulation removes the high-dimensional visual redundancy introduced by dense video representations, and it therefore enables cheaper training and low-latency inference. Across 640 real-robot trials on four long-horizon manipulation tasks, our DELE-w0.5 achieves the best performance among all compared policies, attaining 62.5% overall full-task success and 81.3% macro ordered-stage progress. It outperforms the strongest baseline by 32.5 percentage points in full-task success and 20.1 percentage points in macro progress.

Fenghao Lei, Zhixiong Huang, Long Yang +5

DeepLeap Research, Shenzhen, China

Long-Horizon Robotic ManipulationWorld Models for RoboticsRobotic ManipulationWorld Action Models
AskPDF
May 31, 2026

τ0τ_0-WM: A Unified Video-Action World Model for Robotic Manipulation

Robotic manipulation requires models that generate executable actions while anticipating and evaluating their future consequences before physical execution. We present τ0τ_0-World Model (τ0τ_0-WM), a unified video-action world model that integrates policy learning, video prediction, and action evaluation within a single future-predictive framework. Built on a shared video diffusion backbone, τ0τ_0-WM provides two complementary interfaces. First, a video action model jointly predicts future visual latents and continuous action chunks from multi-view observations, language instructions, and robot state. Second, an action-conditioned video simulator rolls out candidate action chunks into multi-view futures and predicts dense task-progress scores. The model is trained on approximately 27,30027{,}300 hours of real-robot teleoperation, UMI-style interaction, egocentric human videos, and rollout or failure trajectories using modality-specific supervision masks. At inference time, τ0τ_0-WM uses test-time computation to sample action candidates, rank them with re-denoising consistency, and invoke simulator-based rectification for low-quality candidates. On challenging long-horizon and fine-grained robotic manipulation tasks, τ0τ_0-WM shows superior performance over other relevant baselines.

Pengfei Zhou, Shengcong Chen, Di Chen +17

1Shanghai Innovation Institute · 2AGIBOT Finch

Action-Conditioned Video GenerationLong-Horizon Robotic ManipulationRobot Skill LearningWorld Models for RoboticsWorld ModelsWorld Model-Based Planning+2
AskPDF
Jun 11, 2026

RepWAM: World Action Modeling with Representation Visual-Action Tokenizers

This work presents RepWAM, a representation-centric world action model (WAM) built on representation visual-action tokenizers. Existing WAMs typically inherit reconstruction-oriented video tokenizers from pretrained video generation models. Although these tokenizers preserve visual fidelity, pixel reconstruction alone provides limited guidance for learning instruction-following dynamics that connect future prediction with robot control. To address this, we explore a semantic visual-action latent space for representation-centric world action modeling. Specifically, we train a representation visual-action tokenizer that maps visual inputs into aligned visual and latent action tokens. We then pretrain our WAM to jointly model future visual states and the latent actions that connect them under language instructions, followed by adaptation to real robot trajectories for closed-loop manipulation. Experiments on real-world manipulation tasks and simulation benchmarks show that RepWAM delivers strong performance across diverse manipulation settings, while ablations highlight the value of semantic visual-action tokenization over reconstruction-oriented alternatives. These results establish representation visual-action tokenization as a promising foundation for world action models and a step toward generalist robot policies. Code and weights will be available at https://github.com/wdrink/RepWAM.

Junke Wang, Qihang Zhang, Shuai Yang +5

Institute of Trustworthy Embodied AI, Fudan University · 2Robbyant, Ant Group · 3Hongkong University of Science and Technology

Latent Action LearningWorld Models for RoboticsLanguage-Conditioned Robot ManipulationVisual TokenizationRobotic ManipulationVision-Language-Action Models+1
AskPDF