Action Images: End-to-End Policy Learning via Multiview Video Generation

Haoyu Zhen; Zixian Gao; Qiao Sun; Yilin Zhao; Yuncong Yang; Yilun Du; Pengsheng Guo; Tsun-Hsuan Wang

cs.CVApr 7, 2026

Action Images: End-to-End Policy Learning via Multiview Video Generation

Authors: Haoyu Zhen, Zixian Gao, Qiao Sun, Yilin Zhao, Yuncong Yang, Yilun Du, Pengsheng Guo, Tsun-Hsuan Wang,

Organizations: UMass Amherst · NVIDIA · Harvard University · Genesis AI

Abstract

World action models (WAMs) have emerged as a promising direction for robot policy learning, as they can leverage powerful video backbones to model the future states. However, existing approaches often rely on separate action modules, or use action representations that are not pixel-grounded, making it difficult to fully exploit the pretrained knowledge of video models and limiting transfer across viewpoints and environments. In this work, we present Action Images, a unified world action model that formulates policy learning as multiview video generation. Instead of encoding control as low-dimensional tokens, we translate 7-DoF robot actions into interpretable action images: multi-view action videos that are grounded in 2D pixels and explicitly track robot-arm motion. This pixel-grounded action representation allows the video backbone itself to act as a zero-shot policy, without a separate policy head or action module. Beyond control, the same unified model supports video-action joint generation, action-conditioned video generation, and action labeling under a shared representation. On RLBench and real-world evaluations, our model achieves the strongest zero-shot success rates and improves video-action joint generation quality over prior video-space world models, suggesting that interpretable action images are a promising route to policy learning.

Figures & tables

Figure 1: Action Images turns policy learning as multiview video generation: 7-DoF actions are translated into pixel-grounded action images that explicitly track robot-arm motion, enabling a zero-shot policy directly from a unified video backbone

Figure 2: Action as image. We convert each 7-DoF robot action into three semantic 3D points (position, normal, and up), project them into image space, and render them as RGB Gaussian heatmaps. The blue channel further encodes gripper openness in the low-response background, producing a pixel-grounded action representation.

Figure 3: Action images decoding. A 2D heatmap point is selected in the main view, lifted to 3D by ray casting and side-view matching, and repeated for all semantic points to recover the original 7-DoF action.

Figure 4: Unified world-action model training. Multi-view video and action latents are packed with text and camera conditions, and trained under diverse mask strategies.

Dataset	#Traj.	#Views	Real	Action Ann.	Cam. Calib.	Cam. Motion
DROID	80k	2	✓	✓	✓	Static
RLBench	180k	4	✗	✓	✓	Diverse
BridgeV2	30k	1-4	✓	✓	✗	Static

Table 1: Summary of dataset for unified world action model training.

Methods	RLBench				Real
Methods	pick cup	reach target	close drawer	close laptop	Place Cup	Pick Unseen Toy	Pick Tissue	Close Drawer	Close Box
MV-Policy	0	0	0	0	0	0	0	0	0
${\pi_{0.5}}$	0	5	35	20	5	0	0	0	0
MolmoAct	20	5	10	0	10	5	5	5	0
TesserAct	0	0	0	0	0	0	0	0	0
Cosmos-Policy	0	5	20	0	0	0	0	0	0

Table 2: Zero-shot evaluation results on RLBench and Real-world settings.

Methods	close box	close door	open door	phone base	open bottle	close drawer	open oven	open jar	wipe desk	Avg.
MV-Diffusion Policy	20	40	15	20	5	50	10	0	0	17.8
MomolAct (zeroshot)	5	10	0	0	5	10	0	0	0	3.3
$\pi_{0.5}$	10	0	5	5	45	65	0	0	0	14.4
TesserAct	40	25	5	15	20	70	5	5	0	20.6
Cosmos-Policy	40	15	0	15	30	80	0	0	0	20.0
Ours	55	60	0	0	5	60	5	0	0	20.6

Table 3: RLBench in-domain tasks evaluation

Models	Video				Action
Models	PSNR $\uparrow$	SSIM (%) $\uparrow$	FVD $\downarrow$	LPIPS $\downarrow$	2DErr $\downarrow$	3DErr $\times 10^{3}$ $\downarrow$
Cosmos-Predict2.5-14B †	17.92	50.77	208.65	0.409	-	-
Cosmos-Policy	18.29	53.41	192.58	0.418	2.11	19.4
TesserAct	20.83	59.20	154.38	0.351	1.84	19.0
TesserAct-RGB	20.31	60.19	147.83	0.372	1.55	14.2
Ours	23.48	78.62	143.74	0.209	1.61	12.2

Table 4: Video-and-Action Joint Generation Quality. (

{\dagger}

denotes a zero-shot model.).

Table 9

Figure 5: Real-world zero-shot rollouts on xArm robot. From left to right, we show the input observation, generated future video frames with predicted action-image trajectories, and the reconstructed 3D visualization. The results demonstrate that our model can generalize to unseen real-world objects and environments, while producing executable action predictions that are consistent with the generated visual outcomes.

Figure 6: Zero-shot video and action-image generation on FR3M [ 50 ] rooms. This example illustrates a challenging setting with an unseen object, unseen task, and unseen environment ( pick up the mouse ). The predicted action trajectories stay aligned with the scene geometry.

Models	Size	GPU	Steps	#Frames	Image Res.	Inference Time (s)
TesserAct	5B	1 H100	50	49	(480, 640)	137.5
DreamZero	14B	1 H100	16	48	(176, 320)	5.7
DreamZero-Flash	14B	2 GB200	1	48	(176, 320)	0.15
Ours	5B	1 H100	50	164	(512, 512)	49.1
+ Parallelism	5B	8 H100	50	164	(512, 512)	11.8
+ Caching	5B	8 H100	16	164	(512, 512)	2.3

Table 7: Inference efficiency.

Figure 8: Additional robot manipulation results, mainly on grasping tasks.

Figure 9: Camera control results in complex scenes from the $\pi_{0}$ website.

Figure 10: Action-conditioned generation results.

Action Representation Learning

See all 11 papers

GigaWorld-Policy: An Efficient Action-Centered World--Action Model
Mar 18, 2026Angen Ye, Boyuan Wang, Chaojun Ni +21World Action Models Visuomotor Policy Learning
ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?
Jun 17, 2026Yuyang Zhang, Wenyao Zhang, Zekun Qi +7Action-Conditioned World Models World Action Models
Shanghai Jiao Tong University · Tsinghua University · Tencent Robotics X +1
AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps
Apr 13, 2026Liaoyuan Fan, Zetian Xu, Chen Cao +3World Action Models Robotic Manipulation
INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China · The University of Hong Kong, Hong Kong SAR, China · Shanghai Jiao Tong University, Shanghai, China

Mar 18, 2026J/K move · Enter open · S save

GigaWorld-Policy: An Efficient Action-Centered World--Action Model

Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang+20

World Action Models Visuomotor Policy Learning Efficient Inference for World Action Models Action-Conditioned Video Generation

Open paper AskPDF arXiv

World-Action Models (WAM) initialized from pre-trained video generation backbones have demonstrated remarkable potential for robot policy learning. However, existing approaches face two critical bottlenecks that hinder performance and deployment. First, jointly reasoning over future visual dynamics and corresponding actions incurs substantial inference overhead. Second, joint modeling often entangles visual and motion representations, making motion prediction accuracy heavily dependent on the quality of future video forecasts. To address these issues, we introduce GigaWorld-Policy, an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation. Specifically, we formulate policy training into two coupled components: the model predicts future action sequences conditioned on the current observation, and simultaneously generates future videos conditioned on the predicted actions and the same observation. The policy is supervised by both action prediction and video generation, providing richer learning signals and encouraging physically plausible actions through visual-dynamics constraints. With a causal design that prevents future-video tokens from influencing action tokens, explicit future-video generation is optional at inference time, allowing faster action prediction during deployment. To support this paradigm, we curate a diverse, large-scale robot dataset to pre-train an action-centered video generation model, which is then adapted as the backbone for robot policy learning. Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus, while improving task success rates by 7%. Moreover, compared with pi-0.5, GigaWorld-Policy improves performance by 95% on RoboTwin 2.0.

Abstract

Action Representation Learning

Explore similar work

GigaWorld-Policy: An Efficient Action-Centered World--Action Model

ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps