Organizations: GigaAI · Zhejiang University · Peking University · Institute of Automation, Chinese Academy of Sciences · Robotics Department, Mohamed bin Zayed University of Artificial Intelligence
Recent advances in robot foundation models trained on large-scale human teleoperation data have enabled robots to perform increasingly complex real-world tasks. However, scaling these systems remains difficult because collecting task-specific demonstrations is expensive and labor-intensive. Synthetic data, especially generated videos, offer a promising direction, but existing World Models (WMs) are not directly suitable for policy learning since they do not provide paired action trajectories. World-Action (WA) models partially address this by predicting actions with visual outputs, yet often lack strong video-action alignment, while two-stage pipelines that generate video first and then infer actions introduce inefficiency and error accumulation. To address these limitations, we propose VAG, a unified flow-matching-based dual-stream framework that jointly generates video and action under visual and language conditioning. By synchronizing denoising in both branches and using an adaptive 3D pooling mechanism to transfer compact global video context to the action branch, VAG improves cross-modal consistency during generation. Across both simulated and real-world settings, VAG produces aligned video-action pairs with competitive prediction quality, supports executable trajectory replay, and provides useful synthetic pretraining data that improves downstream policy generalization, indicating its potential as a practical world-action model for embodied data synthesis.
Figures & tables
Figure 1: Illustration and capabilities of VAG. (a) We train our dual-stream video-action generation model using teleoperated robot trajectories. (b) Given an initial frame and a language instruction, the model can synchronously generate aligned video–action data pairs. (c) The generated data can be used to train robot policies for improved generalization. (d) The actions generated by the model can also be applied for real-world robot replay. The successful task execution demonstrates that the model holds the potential to function as a policy.
Figure 2: Architecture comparison of embodied models. (a) Vision-Language-Action (VLA) models iteratively predict and execute actions, serving as a policy [ 6 ] . (b) World models (WMs) generate rich visual rollouts spanning diverse scenarios but lack aligned action trajectories for direct policy learning [ 1 ] . (c) World-Action (WA) models serve as a policy and enhance action prediction by incorporating video generation as an Action-Auxiliary signal [ 9 ] . (d) World-Action (WA) models synthesize robot training data by combining world models and IDM [ 29 ] . (e) Our framework VAG specializes in joint and aligned video-action generation for embodied data synthesis.
Figure 3: Inference and training pipeline of VAG. Both the video branch and the action branch are based on flow matching.
Figure 4: Training loss curve of VAG over time on the AgiBot dataset for 40,000 iterations.
Figure 5: Qualitative video generation results across different methods. Prompt: “Use the right hand to pour the water from the gray teapot into the cup.” CP2 stands for Cosmos-Predict2.
Methods
FVD ↓
FID ↓
LPIPS ↓
SSIM ↑
PSNR ↑
SVD
1311
150
0.421
0.339
12.7
Wan2.2
1152
129
0.325
0.612
14.5
CP2
988
135
0.352
0.427
14.2
VAG
965
130
0.320
0.512
15.1
Table 1: Quantitative video generation results on the AgiBot dataset. CP2 stands for Cosmos-Predict2.
Methods
AgiBot
LIBERO
ED ↓
SR ↑
ED ↓
SR ↑
VAG-Video + ResNet
1.54
8%
0.87
37%
VAG-Video + AnyPos
0.98
29%
0.55
66%
VAG
0.81
45%
0.38
79%
Table 2: Quantitative action generation results on the AgiBot dataset and LIBERO dataset. ED stands for Euclidean Distance and SR stands for Success Rate.
Figure 6: Action curve of all 16 dimensions (predicted vs GT) over time on a grasping task of the AgiBot dataset. Prompt: “Use the right hand to pick up the apple from the table”.
Figure 7: Visualization of the generated videos and actions in the simulation environment of LIBERO. Prompt: “Pick up the black bowl from table center and place it on the plate.” Top to bottom: generated head-view video; synchronized generated hand-view video; trajectory replay visualization from the head-view; trajectory replay visualization from the hand-view.
Figure 8: Training loss curve of VAG over time on the LIBERO dataset for 20,000 iterations.
Figure 9: Action curves of all 7 dimensions (predicted vs. ground truth) over time on the spatial task of LIBERO dataset. Prompt: “Pick up the black bowl between the plate and the ramekin and place it on the plate.” The blue curves denote ground-truth actions, while the yellow ones denote predicted actions.
Methods
Success Rate (%) — Replay
Spatial
Object
Goal
Long
Avg
VAG-Video + ResNet
33
34
23
10
25
VAG-Video + AnyPos
59
62
56
39
54
VAG
70
72
64
42
62
Table 3: Quantitative action generation results on the LIBERO benchmark. The best results are marked in bold .
Figure 10: Real-world demonstrations of VLAs. In the face of changes in the position or color of the grasped object, π0.5 -w-VAG-pretrain outperforms π0.5 , verifying VAG’s effectiveness.
Figure 11: Success rate of VLAs in the real world . Powered by the synthesized data from VAG, π0.5 -w-VAG-pretrain showcases better generalization than π0.5 (a 20% increase in success rate).
Figure 12: Representative examples of VAG-generated video and action trajectories for left-arm, right-arm, and bimanual manipulation. Each example includes (1) the initial VAG input frame and generated video, (2) the textual prompt and action trajectory, and (3) the action trajectory executed on the Agilex robot.
We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.
World-Action Models (WAM) initialized from pre-trained video generation backbones have demonstrated remarkable potential for robot policy learning. However, existing approaches face two critical bottlenecks that hinder performance and deployment. First, jointly reasoning over future visual dynamics and corresponding actions incurs substantial inference overhead. Second, joint modeling often entangles visual and motion representations, making motion prediction accuracy heavily dependent on the quality of future video forecasts. To address these issues, we introduce GigaWorld-Policy, an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation. Specifically, we formulate policy training into two coupled components: the model predicts future action sequences conditioned on the current observation, and simultaneously generates future videos conditioned on the predicted actions and the same observation. The policy is supervised by both action prediction and video generation, providing richer learning signals and encouraging physically plausible actions through visual-dynamics constraints. With a causal design that prevents future-video tokens from influencing action tokens, explicit future-video generation is optional at inference time, allowing faster action prediction during deployment. To support this paradigm, we curate a diverse, large-scale robot dataset to pre-train an action-centered video generation model, which is then adapted as the backbone for robot policy learning. Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus, while improving task success rates by 7%. Moreover, compared with pi-0.5, GigaWorld-Policy improves performance by 95% on RoboTwin 2.0.
Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present \method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. \method{} builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 \method-generated trajectories improves out-of-distribution real-robot success from 77% to 84% on single-arm tasks and from 53% to 70% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization.
Zexuan Yan, Yuzhou Wu, Yue Ma +9
1Shanghai Jiao Tong University · 2Alibaba Group · 3Tianji KernalMind Co., Ltd. +4