Latent World Models

Follow this topic Ask about this topic

Latest papers 196

50 papers in the last four weeks, up 194% on the four weeks before. 0.3% of all new papers.

All topics

Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Models
Jun 19, 2026Tianyou Wang, Anson Lei, Joe Watson +1Latent Action Learning Visuomotor Policy Learning
University of Oxford
Sensorimotor World Models: Perception for Action via Inverse Dynamics
Jun 18, 2026Petr Ivashkov, Randall Balestriero, Bernhard SchölkopfRepresentation Learning World Model-Based Planning
Max Planck Institute for Intelligent Systems, Tübingen, Germany · Department of Computer Science, Brown University, Providence, RI, USA · ELLIS Institute, Tübingen, Germany +1
Looped World Models
Jun 16, 2026Hongyuan Adam Lu, Z. L. Victor Wei, Qun Zhang +28World Model Learning Recurrent Transformers
FaceMind Research Asia
NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama
Jun 16, 2026Logan Mann, Abdur Rahman, Mohammad Saifullah +2LLM World Models Multilingual Language Model Evaluation
University of California, Santa Barbara · Pocket FM
BRICKS-WM: Building Reusability via Interface Composition Kinetics for Structured World Models
Jun 15, 2026Shaowei Zhang, Jiahan Cao, Xunlan Zhou +2World Models Latent World Models
1National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent Action Learning Action-Conditioned World Models
Jilin University · Zhongguancun Academy · Nankai University +4
Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC
Jun 14, 2026Devesh Nath, Anutam Srinivasan, Haoran Yin +3Constrained Motion Planning World Model-Based Planning
Georgia Institute of Technology
DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
Jun 13, 2026David Huang, Lianlei ShanEfficient Multimodal Inference Multimodal Reasoning
University of Toronto · Tsinghua University
FlowMo-WM: A World Model with Object Momentum and Hidden Ambient Drift
Jun 11, 2026Yitao Jiang, Luyang Zhao, Muhao Chen +1World Model Learning World Models for Robotics
Department of Computer Science Dartmouth College, Hanover, NH · Department of Electrical and Computer Engineering Clemson University, Clemson, SC · Department of Mechanical and Aerospace Engineering University of Houston, Houston, TX
Certified World Models: Predictability Across Configuration, Horizon, and Resolution
Jun 11, 2026Hongbo WangEquivariant Neural Networks Latent World Models
Department of Mathematics, Stony Brook University, Stony Brook, NY 11794, USA
ReflectiChain: Epistemic Grounding in LLM-Driven World Models for Supply Chain Resilience
Jun 9, 2026Jia LuoConstrained RL LLM-Guided RL
School of Foreign Languages, Huazhong University of Science and Technology, Wuhan 430074, China.
Toward Compiler World Models: Learning Latent Dynamics for Efficient Tensor Program Search
Jun 8, 2026Haolin Pan, Lianghong Huang, Xvlin Zhou +2GPU Kernel Optimization Latent World Models
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
FF-JEPA: Long-Horizon Planning in World Models with Latent Planners
Jun 8, 2026Sergi Masip, Jonathan Swinnen, Yutong Hu +2Hierarchical Planning Long-Horizon Robotic Manipulation
PSI, ESAT, KU Leuven · RAM, MECH, KU Leuven
ATM: Why Latent World Models Can Fail to Plan
Jun 8, 2026Jiaheng Chen, Tinghe Zhang, Yucheng Xiao +5World Model-Based Planning Model-Based Planning
School of Software, Northeastern University
PRISM: PRior-guided Imagination Sampling in world Models
Jun 6, 2026Yuhai Wang, Jiawei Xia, Rongxuan Zhou +4World Models for Robotics World Model-Based Planning
1Northeastern University · University of California, Berkeley · 3Qiyuan Lab +1
PLAN-S: Bridging Planning with Latent Style Dynamics for Autonomous Driving World Models
Jun 4, 2026Xiaoyun Qiu, Jingtao He, Yijie Chen +4Roadmap-Based Motion Planning World Model-Based Planning
Intelligent Transportation Thrust, Systems Hub, and Center of Seamless Connectivity & Connected Intelligence, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China.
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
Jun 4, 2026Kaixuan Liu, Guojun Xiong, Weinan Zhang +1World Model Learning Off-Policy Evaluation
Department of Computer Science Emory University Atlanta, GA, USA · School of Computer Science Shanghai Jiao Tong University Shanghai, China
Generalization of World Models under Environmental Variability for Vision-based Quadrotor Navigation
Jun 3, 2026Luca Zanatta, Grzegorz Malczyk, Kostas AlexisWorld Models for Robotics World Models
Norwegian University of Science and Technology
CLAW: Learning Continuous Latent Action World Models via Adversarial Latent Regularization
Jun 2, 2026Tewodros Ayalew, Matthew Jeung, Samuel Wheeler +5Latent Action Learning Imitation Learning
University of Chicago · 3Argonne National Laboratory · 2Toyota Technological Institute at Chicago
Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group
Jun 2, 2026Hongbo WangWorld Models for Robotics World Models
Department of Mathematics, Stony Brook University, Stony Brook, NY 11794, USA
IMWM: Intuition Models Complement World Models for Latent Planning
Jun 1, 2026Baoqi Gao, Ruize Han, Miao Wang +1World Model-Based Planning AI Control
Beihang University · Shenzhen University of Advanced Technology
BRo-JEPA: Learning Modular Arithmetic in Latent Space
May 31, 2026Divyansh Jha, Yuanfang Xie, Varan Mehra +1Zero-Shot Learning Joint-Embedding Predictive Architecture
1Georgia Institute of Technology · 2NYU Langone Health
Dreaming Of Others: Latent Teammate Modeling In World Models For Multi-Agent Reinforcement Learning
May 29, 2026Tomas Leroy-StoneMulti-Agent Coordination Theory of Mind
Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models
May 29, 2026Lucas Thil, Jesse Read, Rim Kaddah +1Disentangled Representation Learning Representation Learning
LIX, École Polytechnique IRT SystemX Palaiseau, France · IRT SystemX Palaiseau, France · Safran Tech Chateaufort, France
World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications
May 28, 2026Arif Hassan Zidan, Yi Pan, Hanqi Jiang +23World Model Learning World Model-Based Planning
School of Computer and Cyber Sciences, Augusta University, Augusta, GA, USA · School of Computing, University of Georgia, Athens, GA, USA · Tandon School of Engineering, New York University, Brooklyn, NY, USA +9
Latent Goal Prediction from Language for Model-Based Planning
May 26, 2026Samuel Barbeau, Simon Roy, Giovanni Beltrame +2World Model-Based Planning Long-Horizon Planning
1École de Technologie Supérieure, Montréal · 2International Laboratory on Learning Systems (ILLS) · 3Polytechnique Montréal +4
Back to Parsimonious Latents: Learning Task-Centric World Models from Visual Foundations
May 25, 2026Minghao Fu, Fan Feng, Nicklas Hansen +1World Model Learning World Models
University of California, San Diego
SliceWorld: A Predictive and Controllable World-State Model for CT Report Generation
May 23, 2026Yuanhe Tian, Yan SongRadiology Report Generation World Model Learning
Zhongguancun Academy · University of Science and Technology of China
Emergent Semantic Representations in World Models through Physical Interaction without Linguistic Supervision
May 22, 2026Jiayi FangRepresentation Learning World Model Learning
1Shanghai University of Finance and Economics
Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamics
May 21, 2026Romil V. Sonigra, P. R. KumarLatent Dynamics Modeling Jacobian Regularization
Texas A&M University · Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843

Jun 19, 2026J/K move · Enter open · S save

Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Models

Tianyou Wang, Anson Lei, Joe Watson, Ingmar Posner

University of Oxford

Latent Action Learning Visuomotor Policy Learning Imitation Learning Robot Imitation Learning Latent World Models

Open paper AskPDF arXiv

Imitation learning has emerged as a powerful paradigm for learning visuomotor policies, but its generalisation and stability are limited by the scale and quality of demonstration data needed. A promising direction is to leverage more abundant but heterogeneous data sources, which differ in action space and often lack action labels altogether. Existing co-training approaches that combine heterogeneous data sources rely on heuristic and hand-engineered alignment techniques. In contrast, we argue that action representations should be grounded in prediction: actions that produce the same effect on the environment should share the same representation, regardless of their sources. To this end, we instantiate this principle by using a grounded latent-action world model (GLAM), a pair of generative models with a shared latent action space across data sources that is grounded by predicting future observations consistently across sources. This latent action space is used to train downstream behavioural cloning (BC) policies which map observations to latent actions and decode them back to robot actions, providing a paradigm for learning from heterogeneous data. Empirically, we demonstrate that GLAM successfully learns an aligned latent action space that facilitates action transfer across data sources with and without action labels. Across five manipulation tasks in simulation and in the real world, GLAM-aligned policies significantly outperform BC baselines and prior latent-action methods, achieving an average of +48% improvement in task success rate with the same data-scarce setting. Videos and code are available at https://viccccciv.github.io/glam/.