Action-Conditioned World Models

Follow this topic Ask about this topic

Action-conditioned world models learn how observations or compact states evolve when an agent acts, helping agents anticipate consequences before acting in the real world. Approaches range from diffusion-based video simulators to latent JEPA dynamics models, often paired with model-predictive control, policy learning, or action generation. Research spans robot manipulation and navigation, autonomous driving, games, and other interactive settings, with methods using object-centric states, geometry, memory, and learned action representations. A central challenge is making imagined futures faithfully reflect commanded actions over long horizons: visual realism alone does not guarantee reliable planning or policy evaluation.

Latest papers

343 papers111 papers in the last four weeks, up 147% on the four weeks before. 0.6% of all new papers.

Understanding the limits of latent world models
Oct 27, 2026Rodrigo Gonzalez Laiz, Steffen SchneiderWorld Models Latent World Models
Helmholtz Munich Munich Center for Machine Learning (MCML)
How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop
Oct 27, 2026Haojia Gao, Yue Cao, Yumin Liu +6World Model-Based Planning Action-Conditioned World Models
Tsinghua University · Everwise-Tech Co., Ltd. · Fudan University +2
Rollback: Experience Backtracking in World Models for Policy Improvement
Oct 26, 2026Jiayi Luo, Yang Zhang, Liliang Chen +13Robot Policy Learning Robotic RL
AgiBot · BUAA · NUS
TimeWorld: Towards Action-Conditioned Multimodal World Models for Time Series Forecasting
Oct 26, 2026Hang Ni, Weijia Zhang, Hao LiuTime Series Forecasting Action-Conditioned World Models
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Outcome-Aware Agents: When Does Causal Structure Help a Forward Model Foresee the Consequences of Actions?
Oct 25, 2026Mallesh MadapathiCausal World Models Action-Conditioned World Models
ThinkingDBx Pvt. Ltd., Hyderabad, India
TurboWAM: Learning What Matters for Fast World Action Models
Oct 24, 2026Guile Wu, Shuangzhi Li, Xingxin Chen +5Efficient Inference for World Action Models World Action Models
Huawei Noah’s Ark Lab · University of Alberta · Department of Foundation Model, Huawei
UniAction: Unified Action Conditioning for World Models
Oct 23, 2026Bowen Xue, Zizhang Li, Hong-Xing Yu +1Action-Conditioned World Models Action-Conditioned Video Generation
Stanford University
When Does Dynamics Randomization Test Context Inference?
Oct 23, 2026Oleg Smirnov, Sofiane Ennadir, John Pertoft +2Contextual RL Reinforcement Learning
King AI Labs, Microsoft Gaming · King AI Labs, Microsoft Gaming & KTH Royal Institute of Technology
Does LeWorldModel Preserve Local Action Geometry?
Oct 22, 2026Vasil Vasilev, Siddharth NayakWorld Models World Model Evaluation
Technological School Electronic Systems (TUES) · Massachusetts Institute of Technology
Anticipating Social Consequences with Distributional World Models
Oct 22, 2026Viet-Thanh Pham, Thuy-Trang Vu, Zhuang Li +2World Model-Based Planning Action-Conditioned World Models
Department of Data Science & AI, Monash University, Australia · Royal Melbourne Institute of Technology, Australia
Rethinking Video Autoencoders for Interactive World Models
Oct 21, 2026Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour +2Video World Models Autoencoders
Kyutai · LIGM, ENPC, IP Paris, CNRS, UGE · UC Berkeley
PhysiCalWorld: In-Context Physics Calibration for World Models
Oct 21, 2026Yunhao Luo, Wentao He, Yinpei Dai +5World Models Action-Conditioned World Models
University of Michigan
Quantum-JEPA: Quantum Born Priors for Latent-Action JEPA World Models
Oct 20, 2026Zhiyuan Liu, Sicheng Liu, Haiyu WuQuantum Circuit Born Machines Action-Conditioned World Models
Qinous Technology Co., Ltd., Beijing, China · Tsinghua University, Beijing, China · University of Notre Dame, Notre Dame, IN, USA
World Models with Configurable Perception
Oct 20, 2026Yijun Shen, Delong Chen, Xianming Hu +2World Model Learning World Model-Based Planning
East China Normal University · HKUST
Training Controllable World Models via Action Influence Maximization
Oct 19, 2026Samuel M. Bateman, Tenny Yin, Chongyi Zheng +5Action-Conditioned World Models Action-Conditioned Video Generation
Princeton University · Columbia University
State World Models: Execute the Rules, Generate the Pixels
Oct 19, 2026Chan Yan Ki, Yanran Zhang, Wenzhao Zheng +5World Models Action-Conditioned World Models
Department of Automation, Tsinghua University, China · GigaAI
PrismWAM: Decoupling State Representations in World Action Models
Oct 19, 2026Tianyu Xin, Wanqi Zhong, Bowen Hu +8World Models for Robotics World Models
Tsinghua University · NVIDIA
ActTime: Action-Conditioned World Modeling for Time Series
Oct 19, 2026Xinxing Zhou, Xiaojie Yuan, Yanlong Wen +1Action-Conditioned World Models Model-Based Planning
Nankai University · Griffith University
Frechet Forcing: Training Long-Horizon Streaming World Models from Short Clips
Oct 19, 2026Yian Zhao, Yu Yan, Hongcan Guo +3World Model Learning Video World Models
From Prediction to Action: Population Health World Model for Digital Medicine
Oct 17, 2026Xike Xie, Yu Li, Weizhong Yang +2Action-Conditioned World Models Clinical Decision-Making
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui 230026, China · Yangtze River Delta Research Institute of Group Health Technology - MIRACLE Lab, USTC Suzhou Institute for Advanced Research, Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou, Jiangsu 215123, China +6
Structured Latent Dynamics for Long-Horizon Embodied World Models
Oct 17, 2026Kevin Chen, Yu-Wei Zhan, Haoran Liang +2World Models for Robotics Embodied World Models
Tsinghua University
Beyond Next-Screen Prediction: The Decision Value of GUI World Models
Oct 17, 2026Sizhe Tang, Hanhan Zhou, Jeffrey Svajlenko +1World Model Evaluation Computer-Use Agent Evaluation
George Washington University · Amazon Web Services
Beckmann World Models: Direct Terminal-Map Learning for One-Step Action-Conditioned Prediction
Oct 16, 2026Adam Lee, Shobhit Agarwal, June Pyo SuhWorld Model Learning Action-Conditioned World Models
UC Berkeley · Stanford University · Independent Researcher
World Action Critic (WAC) Model: Joint Foresight and Criticism for Policy Learning
Oct 16, 2026Xiangteng Zhang, Haoyuan Liu, Yang Guan +7Action-Conditioned World Models World Models for Autonomous Driving
School of Vehicle and Mobility, Tsinghua University · SunRisingAI Lab · Didi Voyager Labs, Didi Autonomous Driving +1
GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations
Oct 15, 2026Nicholas Saban, Charvi Bannur, Mariya Vasileva +1World Model Evaluation LLM World Models
Patronus AI
T2-z: Test-Time Optimization of Environment Latents for Robotic World Models
Oct 15, 2026Chenyuan Zhou, Yining Hong, Huang Huang +9World Models for Robotics Test-Time Adaptation
Stanford University · CMU · Toyota Research Institute +1
Learning Intrinsic Dynamics in Latent World Models
Oct 15, 2026Jiamin Shi, Yuanhong Zhang, Zhiyang Liu +2Latent Dynamics Modeling Latent World Models
Xi’an Jiaotong University · National University of Singapore · Beijing Zhongguancun Academy
Can World and Action Modeling Benefit Each Other under Unified Causal Training?
Oct 15, 2026Yue Su, Bokai Lin, Xinkai Wang +9Action-Conditioned World Models World Model Learning
IAAI Team · SII · THU +1
PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents
Oct 14, 2026Siqi Yang, Yimu Wang, Ziqi Cai +6World Model Learning Action-Conditioned World Models
Alaya Lab · Peking University · Institute of Science Tokyo
OneLA: Learning Many Worlds from Latent Actions
Oct 14, 2026Ziyu Zhu, Zhaoxu Sun, Fan Jiang +3World Model Learning Action-Conditioned World Models
AMAP, Alibaba Group · Tsinghua University

Oct 27, 2026J/K move · Enter open · S save

Understanding the limits of latent world models

Rodrigo Gonzalez Laiz, Steffen Schneider

Helmholtz Munich Munich Center for Machine Learning (MCML)

World Models Latent World Models Latent Dynamics Modeling World Model Evaluation Action-Conditioned World Models

Open paper AskPDF arXiv

World models are widely proposed as core components for enabling agents to reason over long temporal and spatial horizons. Yet current evaluations often emphasize task performance while providing limited insight into whether learned world models acquire predictive latent dynamics that reflect the underlying structure of the environment. Here we introduce the concept of faithful latent world models as an equivariance condition between the environment's transition map and the learned latent dynamics. We propose latent rollout consistency (LRC) as a metric to evaluate action-conditioned latent prediction. Strikingly, we find that across the studied state of the art latent world models, none solves the task by actually faithfully modeling the dynamics of the environment.