Action-Conditioned Video Generation

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 74

All topics

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
Jul 6, 2026Mengmeng Liu, Diankun Zhang, Jiuming Liu +7Action-Conditioned Video Generation Video Diffusion Models
University of Twente · Xiaomi EV · University of Cambridge +1
MemLearner: Learning to Query Context memory for Video World Models
Jun 30, 2026Jiwen Yu, Jianxiong Gao, Jianhong Bai +7Action-Conditioned Video Generation Long-Horizon Video Generation
The University of Hong Kong · ‡Work done during an internship at Kling Team, Kuaishou Technology. · Fudan University +2
Learning Transferable Dynamics Priors from Action to World Modeling
Jun 28, 2026Ze Huang, Jiahui Zhang, Hairuo Liu +3Action-Conditioned Video Generation Robot Policy Learning
School of Data Science, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China +1
ViPSim: Collaborating Visual and Parameter Spaces for Consistent Long-Horizon Embodied World Models
Jun 27, 2026Longyu Chen, Heng Li, Wei Yang +2Action-Conditioned Video Generation World Models for Robotics
Huawei Technologies Co., Ltd. · Huazhong University of Science and Technology
Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model
Jun 25, 2026Zizhao Yuan, Zhengtu Liang, Taowen Wang +7Action-Conditioned Video Generation Dexterous Manipulation
The Hong Kong University of Science and Technology (Guangzhou) · Shenzhen University · Beijing University of Technology +1
Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models
Jun 24, 2026Kaiwen Zheng, Guande He, Min Zhao +7Action-Conditioned Video Generation Autoregressive Diffusion
Tsinghua University · UT Austin · NVIDIA
IOI: Decoupling Kinematics and Physics for Interactive World Models
Jun 22, 2026Chengyu Bai, Peidong Jia, Tiecheng Guo +11Action-Conditioned Video Generation World Models for Robotics
1Beijing Innovation Center of Humanoid Robotics · 2Peking University
World Action Models: A Survey
Jun 18, 2026Qiuhong Shen, Shihua Zhang, Yue Liao +5Action-Conditioned Video Generation Action-Conditioned World Models
National University of Singapore
SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation
Jun 17, 2026Wei-Cheng Tseng, Gashon Hussein, Yuzhu Dong +9Action-Conditioned Video Generation Physical Consistency in Video Generation
University of Toronto · Vector Institute · NVIDIA +3
Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
Jun 15, 2026Jie Zhang, Xiaoyue Chen, Anzhe Chen +36Action-Conditioned Video Generation Video Diffusion Models
Qwen Team
CausalDrive: Real-time Causal World Models for Autonomous Driving
Jun 13, 2026Tianyi Yan, Huan Zheng, Dubing Chen +10Action-Conditioned Video Generation Autonomous Driving
SKL-IOTSC, CIS, University of Macau · Xiaomi EV · CASIA
Diffusion Transformer World-Action Model for AV Scene Prediction
Jun 11, 2026Ruslan Sharifullin, Benjamin Jiang, Kai Xi ChewAction-Conditioned Video Generation Diffusion Transformer
Stanford University
Making Foresight Actionable: Repurposing Representation Alignment in World Action Models
Jun 10, 2026Lu Qiu, Yizhuo Li, Yi Chen +3Action-Conditioned Video Generation Representation Learning
The University of Hong Kong · XPENG Robotics
BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression
Jun 8, 2026Shaohao Rui, Xiaofeng Mao, Zhanyu Zhang +5Action-Conditioned Video Generation Camera-Controlled Video Generation
LynnReal AI · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
Echo-Memory: A Controlled Study of Memory in Action World Models
Jun 8, 2026Wayne King, Zeyue Xue, Yuxuan Bian +13Action-Conditioned Video Generation Video Diffusion Models
The University of Hong Kong · Joy Future Academy, JD · The Chinese University of Hong Kong +5
DisCo: World Models with Discrete Camera Motion Control
Jun 6, 2026Hongrui Huang, Junke Wang, Quanhao Li +2Action-Conditioned Video Generation Camera-Controlled Video Generation
1Fudan University
OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics
Jun 3, 2026Zhuoyuan Wu, Jun GaoAction-Conditioned Video Generation Action-Conditioned World Models
Peking University · University of Michigan · NVIDIA
Unified Video-Action Joint Denoising for Dexterous Action and Data Generation
Jun 2, 2026Dingrui Wang, YuAn Wang, Jinkun Liu +4Action-Conditioned Video Generation Video Diffusion Models
1Technical University of Munich · 2ByteDance · 3Tsinghua University
NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation
Jun 2, 2026Aarti Basant, Amlan Kar, Despoina Paschalidou +30Action-Conditioned Video Generation Video Diffusion Models
NVIDIA
Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization
Jun 1, 2026Jingyun Liang, Min Wei, Shikai Li +5Action-Conditioned Video Generation Video Diffusion Models
DAMO Academy, Alibaba Group · Hupan Lab · INSAIT +1
PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion
May 31, 2026Heyuan Gao, Bangxun Tang, Yiren Song +4Action-Conditioned Video Generation Video Diffusion Models
Utopai Studios · Nanyang Technological University · University of California, Irvine +1
Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
May 31, 2026Jiuming Liu, Chaojun Ni, Mengmeng Liu +7Action-Conditioned Video Generation Long-Horizon Video Generation
Department of Engineering, University of Cambridge, U.K. · Peking University, China · University of Twente, Netherlands +4
$τ_0$ -WM: A Unified Video-Action World Model for Robotic Manipulation
May 31, 2026Pengfei Zhou, Shengcong Chen, Di Chen +17Action-Conditioned Video Generation Long-Horizon Robotic Manipulation
1Shanghai Innovation Institute · 2AGIBOT Finch
Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation
May 29, 2026Hanlin Chen, Jiaxin Wei, Xibin Song +5Action-Conditioned Video Generation Long-Horizon Video Generation
School of Computing, National University of Singapore · Technische Universität München · Vertex Lab +1
GE-Sim 2.0: A Roadmap Towards Comprehensive Closed-loop Video World Simulators for Robotic Manipulation
May 26, 2026Boxiang Qiu, Liliang Chen, Yue Liao +12Action-Conditioned Video Generation World Models for Robotics
AgiBot · BUAA · LV-NUS Lab +1
SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models
May 22, 2026Zizhao Tong, Yeying Jin, Hongfeng Lai +11Action-Conditioned Video Generation Video Diffusion Models
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences · Tencent · National University of Singapore +6
Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls
May 19, 2026Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang +2Action-Conditioned Video Generation Aerial Robotics
Institute of Artificial Intelligence, University of Central Florida
Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models
May 18, 2026Shangwen Zhu, Qianyu Peng, Zhao Pu +12Action-Conditioned Video Generation Long-Horizon Video Generation
1SJTU · 8HKU · 3Matrix Team +6
Nano World Models: A Minimalist Implementation of Future Video Prediction
May 17, 2026Siqiao Huang, Partha Kaushik, Michael Chen +5Action-Conditioned Video Generation Long-Horizon Video Generation
Tsinghua University · Carnegie Mellon University · University of Bristol +2
DiLA: Disentangled Latent Action World Models
May 15, 2026Tianqiu Zhang, Muyang Lyu, Yufan Zhang +2Action-Conditioned Video Generation Latent Action Learning
Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, IDG/McGovern Institute for Brain Research, Peking University. · Center of Quantitative Biology, Peking University. · School of Psychological and Cognitive Sciences, Key Laboratory of Machine Perception (Ministry of Education), Peking University.

Jul 6, 2026, cs.CVJ/K move · Enter open · S save

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui+6

University of Twente · Xiaomi EV · University of Cambridge+1

Action-Conditioned Video Generation Video Diffusion Models End-to-End Autonomous Driving World Action Models World Models for Autonomous Driving Autonomous Driving Planning

Open paper AskPDF arXiv

World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. However, it remains unclear which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents future-target leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a

4.3\times

speedup over joint video-action rollout while maintaining comparable planning accuracy. The same model also supports video-only and joint video-action rollouts. Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs. 89.6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.

Momentum

Related topics

Latest papers 74