Action-Conditioned World Models

Follow this topic Ask about this topic

Action-conditioned world models learn how observations or compact states evolve when an agent acts, helping agents anticipate consequences before acting in the real world. Approaches range from diffusion-based video simulators to latent JEPA dynamics models, often paired with model-predictive control, policy learning, or action generation. Research spans robot manipulation and navigation, autonomous driving, games, and other interactive settings, with methods using object-centric states, geometry, memory, and learned action representations. A central challenge is making imagined futures faithfully reflect commanded actions over long horizons: visual realism alone does not guarantee reliable planning or policy evaluation.

Latest papers

345 papers111 papers in the last four weeks, up 147% on the four weeks before. 0.6% of all new papers.

WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation
Mar 17, 2026Jisu Nam, Yicong Hong, Chun-Hao Paul Huang +9Interactive World Models Action-Conditioned World Models
KAIST · Adobe Research · MAUM AI
NavThinker: Action-Conditioned World Models for Coupled Prediction and Planning in Social Navigation
Mar 16, 2026Tianshuai Hu, Zeying Gong, Lingdong Kong +7Social Robot Navigation World Models for Robotics
The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · National University of Singapore +1
RAE-NWM: Navigation World Model in Dense Visual Representation Space
Mar 10, 2026Mingkun Zhang, Wangtian Shen, Fan Zhang +3Visual Navigation World Model Learning
Department of Precision Instrument, Tsinghua University, Beijing, China · University of Rochester, Rochester, NY, USA · Beijing Information Science and Technology University, Beijing, China
PlayWorld: Learning Robot World Models from Autonomous Play
Mar 9, 2026Tenny Yin, Zhiting Mei, Zhonghe Zheng +8World Models for Robotics Video World Models
Princeton University
Interactive World Simulator for Robot Policy Training and Evaluation
Mar 9, 2026Yixuan Wang, Rhythm Syed, Fangyu Wu +7Interactive World Models World Model Learning
Columbia University · Toyota Research Institute · Amazon +1
MWM: Mobile World Models for Action-Conditioned Consistent Prediction
Mar 8, 2026Han Yan, Zishang Xiang, Zeyu Zhang +1World Models for Robotics Action-Conditioned World Models
School of Computer Science, Peking University
Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model
Mar 5, 2026Dongwon Kim, Gawon Seo, Jinsung Lee +2World Model-Based Planning Efficient Inference for World Action Models
KAIST · POSTECH · RLWRLD
Discrete World Models via Regularization
Mar 2, 2026Davide Bizzaro, Luciano SerafiniWorld Model Learning World Models
University of Padua, Padua, Italy · Fonadazione Bruno Kessler, Trento, Italy
COMBAT: Conditional World Models for Behavioral Agent Training
Feb 28, 2026Anmol Agarwal, Pranay Meshram, Sumer Singh +5Action-Conditioned World Models World Model Learning
Overworld AI · Indian Institute of Science Education and Research Bhopal
Geometric Priors for Generalizable World Models via Vector Symbolic Architecture
Feb 25, 2026William Youngwoo Chung, Calvin Yeung, Hansen Jin Lillemark +3World Model Learning World Models
University of California, Irvine, United States · University of California, San Diego, United States
Neural Fields as World Models
Feb 21, 2026Joshua NunleyWorld Model Learning Action-Conditioned World Models
Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington · Cognitive Science Program, Indiana University, Bloomington
World Models for Policy Refinement in StarCraft II
Feb 16, 2026Yixin Zhang, Ziyi Wang, Yiming Rong +6Game-Playing Agents Action-Conditioned World Models
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
Feb 15, 2026Zhennan Jiang, Shangqing Zhou, Yutong Jiang +11World Models for Robotics World Models
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +2
Learning Robust Visuomotor Policies via Dynamics Alignment
Oct 31, 2025Dohyeok Lee, Jung Min Lee, Munkyung Kim +7Visuomotor Policy Learning Flow-Based Policy Learning
Department of Electrical and Computer Engineering, Seoul National University, Seoul 08826, Korea · Department of Statistics, Korea University, Seoul 02841, Korea
Matrix-game 2.0: An open-source, real-time, and streaming interactive world model
Aug 18, 2025Xianglong He, Chunli Peng, Zexiang Liu +16Action-Conditioned Video Generation Real-Time Interactive Video Generation
Skywork AI

Mar 17, 2026J/K move · Enter open · S save

WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation

Jisu Nam, Yicong Hong, Chun-Hao Paul Huang, Feng Liu+8

KAIST · Adobe Research · MAUM AI

Interactive World Models Action-Conditioned World Models Video World Models Camera-Controlled Video Generation

Open paper AskPDF arXiv

Recent advances in video diffusion transformers have enabled interactive gaming world models that allow users to explore generated environments over extended horizons. However, existing approaches struggle with precise action control and long-horizon 3D consistency. Most prior works treat user actions as abstract conditioning signals, overlooking the fundamental geometric coupling between actions and the 3D world, whereby actions induce relative camera motions that accumulate into a global camera pose within a 3D world. In this paper, we establish camera pose as a unifying geometric representation to jointly ground immediate action control and long-term 3D consistency. First, we define a physics-based continuous action space and represent user inputs in the Lie algebra to derive precise 6-DoF camera poses, which are injected into the generative model via a camera embedder to ensure accurate action alignment. Second, we use global camera poses as spatial indices to retrieve relevant past observations, enabling geometrically consistent revisiting of locations during long-horizon navigation. To support this research, we introduce a large-scale dataset comprising 3,000 minutes of authentic human gameplay annotated with camera trajectories and textual descriptions. Extensive experiments show that our approach substantially outperforms state-of-the-art interactive gaming world models in action controllability, long-horizon visual quality, and 3D spatial consistency.