Action-Conditioned World Models

Follow this topic Ask about this topic

Action-conditioned world models learn how observations or compact states evolve when an agent acts, helping agents anticipate consequences before acting in the real world. Approaches range from diffusion-based video simulators to latent JEPA dynamics models, often paired with model-predictive control, policy learning, or action generation. Research spans robot manipulation and navigation, autonomous driving, games, and other interactive settings, with methods using object-centric states, geometry, memory, and learned action representations. A central challenge is making imagined futures faithfully reflect commanded actions over long horizons: visual realism alone does not guarantee reliable planning or policy evaluation.

Latest papers

345 papers111 papers in the last four weeks, up 147% on the four weeks before. 0.6% of all new papers.

BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression
Jun 8, 2026Shaohao Rui, Xiaofeng Mao, Zhanyu Zhang +5Action-Conditioned Video Generation Camera-Controlled Video Generation
LynnReal AI · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing
Jun 8, 2026Jisong Cai, Long Ling, Shiwei Chu +10Long-Horizon Robotic Manipulation World Models for Robotics
Shanghai Jiao Tong University · Shanghai AI Laboratory · Baidu AI Cloud +1
Echo-Memory: A Controlled Study of Memory in Action World Models
Jun 8, 2026Wayne King, Zeyue Xue, Yuxuan Bian +13Action-Conditioned Video Generation Video Diffusion Models
The University of Hong Kong · Joy Future Academy, JD · The Chinese University of Hong Kong +5
$ω$ -EVA: Envision, Verify, and Act with Latent Interactive World Models
Jun 8, 2026Zhenguo Sun, Yu Sun, Hande Huang +1Robot Policy Learning World Models for Robotics
C $^3$ ache: Accelerating World Action Models with Cross Inference Chunk Cache
Jun 8, 2026Weisen Zhao, Lam Nguyen, Zhicong Lu +1World Model Learning Action-Conditioned World Models
George Mason University · University of Central Florida
Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
Jun 7, 2026Yunfan Lou, Yifan Ye, Yankai Fu +7World Models for Robotics Contact-Rich Robotic Manipulation
Peking University Beijing, China · Nanjing University Nanjing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Beijing, China
DisCo: World Models with Discrete Camera Motion Control
Jun 6, 2026Hongrui Huang, Junke Wang, Quanhao Li +2Action-Conditioned Video Generation Camera-Controlled Video Generation
1Fudan University
PiL-World: A Chunk-Wise World Model for VLA Policy-in-the-Loop Evaluation
Jun 4, 2026Chong Ma, Taiyi Su, Jian Zhu +4Action-Conditioned World Models Robotic Policy Evaluation
1Tongji University · 2AIRC, Midea Group
Discrete-WAM: Unified Discrete Vision-Action Token Editing for World-Policy Learning
Jun 4, 2026Ziyang Yao, Haochen Liu, Yuncheng Jiang +10Hierarchical Planning World Model Learning
Xiaomi EV
Flash-WAM: Modality-Aware Distillation for World Action Models
Jun 3, 2026Arman Akbari, Ci Zhang, Arash Akbari +6Video Diffusion Models Action-Conditioned World Models
1Northeastern University · University of Georgia · 3EmbodyX Inc.
OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics
Jun 3, 2026Zhuoyuan Wu, Jun GaoAction-Conditioned Video Generation Action-Conditioned World Models
Peking University · University of Michigan · NVIDIA
A 3D Isovist World Model -- Revealing a City's Unseen Geometry and Its Emergent Cross-City Signature
Jun 2, 2026Xuhui Lin, Stephen Law, Nanjiang Chen +2Representation Learning World Model Learning
Department of Geography University College London, UK · School of Project Management, Faculty of Engineering The University of Sydney, AU · School of Engineering Cardiff University, UK +1
Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects
Jun 1, 2026Jens U. Kreber, Lukas Mack, Joerg Stueckler3D Gaussian Splatting Dynamical Systems
Intelligent Perception in Technical Systems Group, University of Augsburg, Augsburg, Germany
Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
May 31, 2026Jiuming Liu, Chaojun Ni, Mengmeng Liu +7Action-Conditioned Video Generation Long-Horizon Video Generation
Department of Engineering, University of Cambridge, U.K. · Peking University, China · University of Twente, Netherlands +4
$τ_0$ -WM: A Unified Video-Action World Model for Robotic Manipulation
May 31, 2026Pengfei Zhou, Shengcong Chen, Di Chen +17Action-Conditioned Video Generation Long-Horizon Robotic Manipulation
1Shanghai Innovation Institute · 2AGIBOT Finch
StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement
May 29, 2026Junwon Seo, Sushant Veer, Ran Tian +6Video Diffusion Models Action-Conditioned World Models
Carnegie Mellon University · NVIDIA Research · University of Washington +1
MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models
May 28, 2026Tianzhuo Yang, Zihan Shen, Zirui Mi +7World Models for Robotics Action-Conditioned World Models
Institute for Artiﬁcial Intelligence, Peking University · 2Physis Lab
World Models for Robotic Manipulation: A Survey
May 27, 2026Fangyuan Wang, Ziyuan Wang, Guorui Pei +15World Model Learning World Models for Robotics
Department of Mechanical Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China · Department of Mechanical Engineering and Automation, Harbin Institute of Technology, Shenzhen, China · School of Advanced Engineering, Great Bay University, Dongguan, Guangdong, China +9
SANTS: A State-Adaptive Scheduler for World Action Models
May 27, 2026Yirui Sun, Guangyu Zhuge, Keliang Liu +4Action-Conditioned World Models Visuomotor Policy Learning
Fudan University · Deep Computing Era Technology Co., Ltd · Harbin Institute of Technology
GE-Sim 2.0: A Roadmap Towards Comprehensive Closed-loop Video World Simulators for Robotic Manipulation
May 26, 2026Boxiang Qiu, Liliang Chen, Yue Liao +12Action-Conditioned Video Generation World Models for Robotics
AgiBot · BUAA · LV-NUS Lab +1
Back to Parsimonious Latents: Learning Task-Centric World Models from Visual Foundations
May 25, 2026Minghao Fu, Fan Feng, Nicklas Hansen +1World Model Learning World Models
University of California, San Diego
UWM-JEPA: Predictive World Models That Imagine in Belief Space
May 25, 2026Santosh Kumar Radha, Oktay GoktasWorld Model Learning Latent Dynamics Modeling
AgentField AI, Toronto, Canada
World Models as Group Actions
May 23, 2026Zijie Wang, Wei Zhang, Weiming Zhang +4World Model Learning Action-Conditioned World Models
Sun Yat-sen University · Shenzhen Loop Area Institute · Baidu Inc. +2
Point Tracking Improves World Action Models
May 22, 2026Jiarui Guan, Wenshuai Zhao, Yue Pei +3Point Tracking Robot Policy Learning
Aalto University · ELLIS Institute Finland · Beihang University +3
SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models
May 22, 2026Zizhao Tong, Yeying Jin, Hongfeng Lai +11Action-Conditioned Video Generation Video Diffusion Models
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences · Tencent · National University of Singapore +6
WorldKV: Efficient World Memory with World Retrieval and Compression
May 21, 2026Jung Yi, Minjae Kim, Paul Hyunbin Cho +3KV Caching Memory-Augmented Video Generation
1KAIST AI · 2Naver AI Lab
ChronoMedicalWorld: A Medical World Model for Learning Patient Trajectories from Longitudinal Care Data
May 21, 2026Jiangyuan Wang, Xuyong Chen, Junwei He +3Action-Conditioned World Models Latent Dynamics Modeling
Beijing KidneyTec Medical Technology Co., Ltd., Beijing, China
Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models
May 18, 2026Shangwen Zhu, Qianyu Peng, Zhao Pu +12Action-Conditioned Video Generation Long-Horizon Video Generation
1SJTU · 8HKU · 3Matrix Team +6
WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform
May 18, 2026Zhuohang Li, Yiding Ma, Yu Shang +26World Models for Robotics World Models
Tsinghua University · Shanghai Jiao Tong University · Zhejiang University +7
DiLA: Disentangled Latent Action World Models
May 15, 2026Tianqiu Zhang, Muyang Lyu, Yufan Zhang +2Action-Conditioned Video Generation Latent Action Learning
Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, IDG/McGovern Institute for Brain Research, Peking University. · Center of Quantitative Biology, Peking University. · School of Psychological and Cognitive Sciences, Key Laboratory of Machine Perception (Ministry of Education), Peking University.

Jun 8, 2026J/K move · Enter open · S save

BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression

Shaohao Rui, Xiaofeng Mao, Zhanyu Zhang, Peijia Lin+4

LynnReal AI · Shanghai Innovation Institute · Shanghai Jiao Tong University+1

Action-Conditioned Video Generation Camera-Controlled Video Generation Long-Horizon Video Generation Action-Conditioned World Models Autoregressive Video Generation Interactive World Models+1

Open paper AskPDF arXiv

Transitioning bidirectional video diffusion models into an autoregressive paradigm improves the interactivity of video world models, but existing causal pipelines need many stages (control fine-tuning, autoregressive training, causal initialization, few-step distillation) and still trail bidirectional models in quality due to error accumulation. Recent world models such as Yume-1.5 and Matrix-Game-3.0 instead adopt a bidirectional autoregressive approach, gaining fidelity and stable long-horizon rollout from self-correcting error propagation, yet open-source frameworks (e.g., minWM) support only causal models. We present BiWM, the first full-stack framework for interactive video world models under the bidirectional autoregressive paradigm, jointly optimizing generation quality and inference speed. From a pretrained video backbone, BiWM injects camera control by fine-tuning, then runs a few-step Distribution Matching Distillation (DMD) stage that turns the backbone into an action/camera-controllable world model: just two training stages instead of four in minWM, converging in a few hundred steps on 8xH200 GPUs. A single recipe spans Wan2.1-1.3B, Wan2.2-5B, HunyuanVideo-1.5-8B, and LTX-2.3-22B, and also supports secondary fine-tuning of existing bidirectional models. BiWM enables real-world camera control where minWM loses controllability, integrates pluggable history compression (FramePack-style and PackForcing-style) for long rollouts, and offers an optional NVFP4 4-bit training/inference pipeline. To counter DMD's mode-seeking degradation, we add GAN and mass-covering forward-KL objectives that preserve scene dynamics. We open-source BiWM for resource-constrained research and high-fidelity environment simulation.