Latent World Models

Follow this topic Ask about this topic

Latent world models encode observations and histories into compact states, then predict how those states change over time or under actions, avoiding the need to reconstruct every pixel. Common approaches include recurrent state-space models such as Dreamer, joint-embedding predictive architectures (JEPAs), and diffusion or flow predictors, often shaped by inverse dynamics, temporal objectives, physical structure, or causal supervision. These models support imagined rollouts and planning in visual reinforcement learning, robot manipulation and navigation, autonomous driving, and other forecasting settings. A central challenge is that accurate short-term predictions or interpretable latents do not necessarily preserve action effects and goal-relevant structure over long horizons, leaving planning evaluation, uncertainty calibration, and robust transfer open.

Latest papers

218 papers40 papers in the last four weeks, up 11% on the four weeks before. 0.3% of all new papers.

Roz: A Taste Test for World Model Latents
Oct 31, 2026Kenneth Ge, Dhruv Sheth, Kunvar ThamanWorld Model Evaluation World Models
Pantheon
Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel
Oct 30, 2026Kunal Bhosikar, Nicole Meng, Charu Sharma +1Adversarial Attacks on World Models Latent World Models
IIIT Hyderabad · Miko · Tufts University
SPT-Flow: One Integration, Many Futures in Continuous-Time World Models
Oct 30, 2026Mingyang Lyu, Yinqian Sun, Gongce Cao +5World Model Learning World Models
Institute of Automation, Chinese Academy of Sciences · University of the Chinese Academy of Sciences · Zhongguancun Academy +2
Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning
Oct 29, 2026Shashank Gupta, Pilhwa LeeModel-Based RL Latent World Models
Department of EECS, University of Michigan, Ann Arbor, MI 48109, USA · Department of Mathematics, Morgan State University, Baltimore, MD 21251, USA
Generating Critic-Guided Synthetic Experience in Latent World Models
Oct 29, 2026Jorge de Freitas, Christos Ziakas, Alessandra RussoModel-Based RL Offline RL
Imperial College London
Structural-Test Geometry as an Inductive Bias for Partially Observable World Models
Oct 28, 2026Todd Y. ZhouWorld Model Learning Latent World Models
Harvard University (Undergraduate Student), Cambridge, MA, USA
Understanding the limits of latent world models
Oct 27, 2026Rodrigo Gonzalez Laiz, Steffen SchneiderWorld Models Latent World Models
Helmholtz Munich Munich Center for Machine Learning (MCML)
GAWM: A Compact Latent World-Action Model for Multi-Task Robot Manipulation
Oct 27, 2026Yang Liu, Xiaoli Liu, Liuzhao Li +3World Action Models Latent World Models
Genisom AI, Beijing, China
Checkpoint Reload Explains Most of the Measured State Gap in Two State Discovery Reproductions
Oct 26, 2026Weijia Han, Lisha QuTraining-Inference Mismatch Latent World Models
University of Washington
Video World Models Need Predictable Latents
Oct 26, 2026Mattia Scardecchia, Sherry YangVideo World Models World Model Learning
New York University
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Oct 22, 2026Hossein Resani, Javen Qinfeng ShiJoint-Embedding Predictive Architecture Latent World Models
Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre
Exploring Temporal Context-Aware Physical Grounding in Latent World Models
Oct 20, 2026Yilei Man, Yang Chen, Jingcai GuoWorld Models Latent World Models
Department of Computing, The Hong Kong Polytechnic University · Department of Land Surveying and Geospatial Science, The Hong Kong Polytechnic University
EndoWorld-JEPA: Motion-Guided Global-Local Latent World Modeling for Endoscopic Video Representation Learning
Oct 20, 2026Jie Wang, Shilong Yang, Tao Zhou +5Video Representation Learning Latent World Models
University of Nottingham · Shenzhen University · Tsinghua University +2
LaLa-World: Scaling Latent World Models with Natural Language Actions
Oct 20, 2026Junha Song, Geonmo Gu, Jeongeun Park +5World Model Learning Latent World Models
NAVER AI Lab · Georgia Institute of Technology · Brown University +1
World Model Interpretability: From Decoding Representations to Understanding Latent Worlds
Oct 18, 2026Jize Wang, Yongcheng Jing, Xinyi Le +3World Models Latent World Models
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · College of Computing and Data Science, Nanyang Technological University
Latent World Models Should Plan with Distances Along the Latent Manifold
Oct 18, 2026Zunzhi You, Linwei Tao, Siyu Xu +2Latent World Models Latent-Space Planning
The University of Sydney · Machine Learning Research, Apple
Beyond Action Prediction: Learning Search-Free Planning in Latent Space
Oct 16, 2026Guanchen Du, Jinhao Du, Yanpeng Sun +3Latent-Space Planning Model-Based Planning
Nanjing University of Science and Technology · University of Oxford · Singapore University of Technology and Design +1
T2-z: Test-Time Optimization of Environment Latents for Robotic World Models
Oct 15, 2026Chenyuan Zhou, Yining Hong, Huang Huang +9World Models for Robotics Test-Time Adaptation
Stanford University · CMU · Toyota Research Institute +1
Learning Intrinsic Dynamics in Latent World Models
Oct 15, 2026Jiamin Shi, Yuanhong Zhang, Zhiyang Liu +2Latent Dynamics Modeling Latent World Models
Xi’an Jiaotong University · National University of Singapore · Beijing Zhongguancun Academy
Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning
Oct 14, 2026Yanbo Wang, Nan Lin, Wendelin Böhmer +1World Model-Based Planning Latent World Models
Delft University of Technology (TU Delft)
MemWM: Associative Memory-Enhanced Latent World Models for Long-Horizon Planning
Oct 14, 2026Yuwei Sun, Siyu ZhuWorld Model Learning World Models for Robotics
Shanghai Academy of AI for Science · Fudan University
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Oct 14, 2026Joshua Cheung, Zhang HeLatent World Models World Model Evaluation
Order-Consistent Latent Geometry for Reliable Short-Horizon Planning in World Models
Oct 13, 2026Junlin Wu, Wen Fan, Dandan ZhangLatent Space Geometry Latent-Space Planning
Imperial College London, London, United Kingdom
WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
Oct 13, 2026Haomin Wang, Anqiong Ji, Ke Wang +4World Models for Robotics World Action Models
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu AI Cloud · Awaken Robotics +1
PRB-WM: Physics-Regularized Beta World Models for Predictive Link Adaptation on Real Time-Varying Underwater Acoustic Channels
Oct 12, 2026Shengjie Zhong, Zhongliang Zhao, Jingxuan Chen +5Wireless Communications Latent World Models
School of Electronic and Information Engineering, Beihang University, Beijing 100191, China · Department of Strategic and Advanced Interdisciplinary Research, Peng Cheng Laboratory · Hangzhou International Innovation Institute, Beihang University, Hangzhou +8
World Modeling through Spectral Alignment
Oct 11, 2026Holger Molin, William Peng, Marco Bagatella +1World Model Learning Latent World Models
Stanford University · Brown University
LeHami: Symplectic Latent Dynamics for World Models
Oct 11, 2026Romina Yalovetzky, Randall Balestriero, Yann LeCunLatent Dynamics Modeling Latent World Models
Courant Institute, New York University · Brown University · AMI Labs
Latent World Action Model for GUI Agents
Oct 9, 2026Seungjun Yu, Hojun Choi, Seojeong Park +2World Models Action-Conditioned World Models
Korea Advanced Institute of Science and Technology (KAIST)
SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
Oct 9, 2026Timi Owolabi, Hikmah Olawore, Frances Adelakun +1World Models for Robotics World Action Models
ML Collective
When Generative Models Act as Simulators
Oct 9, 2026Sevan Brodjian, Michael Hobley, Pietro PeronaGenerative Modeling Latent World Models
California Institute of Technology

Oct 31, 2026J/K move · Enter open · S save

Roz: A Taste Test for World Model Latents

Kenneth Ge, Dhruv Sheth, Kunvar Thaman

Pantheon

World Model Evaluation World Models Latent World Models

Open paper AskPDF arXiv

State of the art latent dynamics world models achieve impressive performance for robot planning. But do they actually encode the physics of the world? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key details about objects in the scene like mass, deformation, friction, and bounciness. We achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance roughly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract latents, and a distance function between them, and it can be run in one pass. We hope future evaluations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across models.