Action-Conditioned World Models

Follow this topic Ask about this topic

Action-conditioned world models learn how observations or compact states evolve when an agent acts, helping agents anticipate consequences before acting in the real world. Approaches range from diffusion-based video simulators to latent JEPA dynamics models, often paired with model-predictive control, policy learning, or action generation. Research spans robot manipulation and navigation, autonomous driving, games, and other interactive settings, with methods using object-centric states, geometry, memory, and learned action representations. A central challenge is making imagined futures faithfully reflect commanded actions over long horizons: visual realism alone does not guarantee reliable planning or policy evaluation.

Latest papers

345 papers111 papers in the last four weeks, up 147% on the four weeks before. 0.6% of all new papers.

Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations
May 14, 2026Jonathan Spieler, Angel Villar-Corrales, Sven BehnkeWorld Models for Robotics World Model-Based Planning
Autonomous Intelligent Systems, Computer Science Institute VI - Intelligent Systems and Robotics, Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany
SCAR: Self-Supervised Continuous Action Representation Learning
May 13, 2026Hongjia Liu, Fan Feng, Minghao Fu +3Latent Action Learning Representation Learning
University of California, San Diego · KTH Royal Institute of Technology
HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models
May 11, 2026Qiuxuan Feng, Jiale Yu, Jiaming Liu +8World Models for Robotics Action-Conditioned World Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
PROWL: Prioritized Regret-Driven Optimization for World Model Learning
May 11, 2026Ahmet H. Güzel, Jenny Seidenschwarz, Benjamin Graham +3Adversarial Training World Model Learning
University College London AI Centre · Odyssey · University of Basel
ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models
May 9, 2026Haotian Xue, Yipu Chen, Liqian Ma +4Action-Conditioned Video Generation World Models
Georgia Institute of Technology
One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy
May 8, 2026Zuojin Tang, Shengchao Yuan, Xiaoxin Bai +4Visual Token Compression Efficient VLA Models
Zhejiang University · Central South University · Harbin Institute of Technology +2
NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models
May 8, 2026Wen Huang, Haoran Sun, Yongjian Guo +8Action-Conditioned Video Generation Action-Conditioned World Models
Tsinghua University · Peking University · JDT AI Infra +1
Learning Visual Feature-Based World Models via Residual Latent Action
May 8, 2026Xinyu Zhang, Zhengtong Xu, Yutian Tao +3Latent Action Learning World Model Learning
Rutgers University · Purdue University · University of Wisconsin-Madison
OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulation
May 7, 2026Yushan Liu, Peibo Sun, Shoujie Li +7World Models for Robotics Action-Conditioned World Models
Tsinghua University · Shanghai Jiao Tong University · Nanyang Technological University
Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
May 7, 2026Nilaksh, Saurav Jha, Artem Zholus +1World Model Learning World Models for Robotics
Chandar Research Lab · Mila – Quebec AI Institute · Polytechnique Montréal +1
When to Trust Imagination: Adaptive Action Execution for World Action Models
May 7, 2026Rui Wang, Yue Zhang, Canyang Chen +3Long-Horizon Robotic Manipulation World Models for Robotics
Southern University of Science and Technology, Shenzhen, China · The University of Hong Kong, Hong Kong, China
EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields
May 7, 2026Zhaoyang Yang, Yurun Jin, Lizhe Qi +2Action-Conditioned Video Generation Video Diffusion Models
Fudan University · Zhongguancun Institute of Artificial Intelligence · University of Science and Technology of China +2
Motubrain: An Advanced World Action Model for Robot Control
Apr 30, 2026Motubrain Team, Chendong Xiang, Fan Bao +17Robotic Control World Models for Robotics
Motubrain Team
Lifting Embodied World Models for Planning and Control
Apr 28, 2026Alex N. Wang, Trevor Darrell, Pavel Izmailov +2Robotic Control World Model-Based Planning
Computer Science, New York University · BAIR, UC Berkeley
Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models
Apr 28, 2026Pengcheng Fang, Hongli Chen, Xiaohao CaiWorld Model Learning Action-Conditioned World Models
The University of Southampton · The University of Queensland
Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond
Apr 24, 2026Meng Chu, Xuan Billy Zhang, Kevin Qinghong Lin +47World Model Learning Action-Conditioned World Models
Beyond Patient Invariance: Learning Cardiac Dynamics via Action-Conditioned JEPAs
Apr 24, 2026Jose Geraldo Fernandes, Luiz Facury, Pedro Robles Dutenhefner +1Predictive Representation Learning Healthcare
Department of Computer Science Universidade Federal de Minas Gerais Belo Horizonte, Brazil
Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training
Apr 23, 2026Yaxuan Li, Zhongyi Zhou, Yefei Chen +5Robot Policy Learning Robot Policy Adaptation
Current Robotics · Tsinghua University · Peking University +1
WorldMark: A Unified Benchmark Suite for Interactive Video World Models
Apr 23, 2026Xiaojie Xu, Zhengyuan Lin, Kang He +5Action-Conditioned World Models World Model Evaluation
Alaya Studio, Shanda AI Research Tokyo · The University of Tokyo · Shanghai Innovation Institute
MultiWorld: Scalable Multi-Agent Multi-View Video World Models
Apr 20, 2026Haoyu Wu, Jiwen Yu, Yingtian Zou +1Multi-View Consistency Action-Conditioned World Models
The University of Hong Kong · Sreal AI
Active World-Model with 4D-informed Retrieval for Exploration and Awareness
Apr 17, 2026Elaheh Vaezpour, Amirhosein Javadi, Tara JavidiActive Perception World Model Learning
KavAI · UCSD
The Global Neural World Model: Spatially Grounded Discrete Topologies for Action-Conditioned Planning
Apr 17, 2026Noureddine KermicheWorld Model Learning World Model-Based Planning
Western Digital Corporation, Irvine, CA, USA
Grounded World Model: Latent Planning with Language Goals
Apr 13, 2026Quanyi Li, Lan Feng, Haonan Zhang +4Language-Conditioned Robot Planning World Models for Robotics
EPFL · NUS
Genie 4D: Semantic-Prior-Guided 4D Dynamic Scene Reconstruction
Apr 10, 2026Yiru Yang, Zhuojie Wu, Nishant Kumar Singh +14D Reconstruction Dynamic Scene Reconstruction
University of Zurich Zurich, Switzerland · ETH Zurich Zurich, Switzerland
Telecom World Models: Unifying Digital Twins, Foundation Models, and Predictive Planning for 6G
Apr 8, 2026Hang Zou, Yuzhi Yang, Lina Bariah +15Network Slicing World Model-Based Planning
Research Institute for Digital Future, Khalifa University, 127788 Abu Dhabi, UAE · College of Information Science and Electronic Engineering, Zhejiang University, 310027, Hangzhou, China · University of Oulu and Aalto University +2
World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry
Apr 2, 2026Yuejiang Liu, Fan Feng, Lingjing Kong +6World Models for Robotics Action-Conditioned World Models
Stanford University · UC San Diego · Carnegie Mellon University +2
Enhancing Policy Learning with World-Action Model
Mar 30, 2026Yuci Han, Alper YilmazRobot Policy Learning World Action Models
Photogrammetry and Computer Vision Lab, The Ohio State University, Columbus, OH 43210, USA.
World Reasoning Arena
Mar 26, 2026PAN Team, Qiyue Gao, Kun Zhou +15World Model Evaluation World Model-Based Planning
Institute of Foundation Models, Mohamed bin Zayed University of Artificial Intelligence
Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning
Mar 26, 2026Jai Bardhan, Patrik Drozdik, Josef Sivic +1World Models for Robotics World Models
Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague
WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG
Mar 24, 2026Zhen Li, Zian Meng, Shuwei Shi +5World Model Learning World Models
Alaya Studio, Shanda AI Research Tokyo · Beijing Institute of Technology · Shenzhen MSU-BIT University +2

May 14, 2026J/K move · Enter open · S save

Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations

Jonathan Spieler, Angel Villar-Corrales, Sven Behnke

Autonomous Intelligent Systems, Computer Science Institute VI - Intelligent Systems and Robotics, Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany

World Models for Robotics World Model-Based Planning Action-Conditioned World Models Object-Centric Representation Learning Model Predictive Control Robotic Manipulation

Open paper AskPDF arXiv

Predictive world models enable agents to model scene dynamics and reason about the consequences of their actions. Inspired by human perception, object-centric world models capture scene dynamics using object-level representations, which can be used for downstream applications such as action planning. However, most object-centric world models and reinforcement learning (RL) approaches learn reactive policies that are fixed at inference time, limiting generalization to novel situations. We propose Slot-MPC, an object-centric world modeling framework that enables planning through Model Predictive Control (MPC). Slot-MPC leverages vision encoders to learn slot-based representations, which encode individual objects in the scene, and uses these structured representations to learn an action-conditioned object-centric dynamics model. At inference time, the learned dynamics model enables action planning via MPC, allowing agents to adapt to previously unseen situations. Since the learned world model is differentiable, we can use gradient-based MPC to directly optimize actions, which is computationally more efficient than relying on gradient-free, sampling-based MPC methods. Experiments on simulated robotic manipulation tasks show that Slot-MPC improves both task performance and planning efficiency compared to non-object-centric world model baselines. In the considered offline setting with limited state-action coverage, we find that gradient-based MPC performs better than gradient-free, sampling-based MPC. Our results demonstrate that explicitly structured, object-centric representations provide a strong inductive bias for controllable and generalizable decision-making. Code and additional results are available at https://slot-mpc.github.io.