Model-Based RL

RL: Reinforcement Learning

Model-based reinforcement learning studies how agents can learn or use a model of how actions change the environment, then plan with that model to improve decisions and reduce costly interaction. Common approaches include latent world models that predict future states, Dyna-style learning from imagined rollouts, and model-predictive control or tree search; Bayesian models and ensembles help represent uncertainty, while control barriers and other safety methods constrain actions. Methods are tested on suites such as DeepMind Control, MuJoCo, Atari, and D4RL, as well as robotic manipulation, autonomous driving, and other physical systems, alongside theoretical work on regret and sample complexity. The central tension is that imagined experience can make learning more efficient, but model errors may accumulate over long horizons or be exploited by the policy, so reliable uncertainty estimates, safe planning, and transfer to real systems remain open challenges.

Latest papers

245 papers32 papers in the last four weeks, up 10% on the four weeks before. 0.2% of all new papers.

Imperfect World Models are Exploitable
May 15, 2026Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel +2Reward Hacking World Model-Based Planning
University of Edinburgh · Stanford University
Offline Reinforcement Learning with Universal Horizon Models
May 15, 2026Hojun Chung, Junseo Lee, Songhwai OhOffline RL Model-Based RL
Interdisciplinary Program in Artificial Intelligence and ASRI, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University
JEDI: Joint Embedding Diffusion World Model for Online Model-Based Reinforcement Learning
May 13, 2026Jing Yu Lim, Rushi Shah, Zarif Ikram +4World Model Learning Joint-Embedding Predictive Architecture
National University of Singapore · usc
Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling
May 12, 2026Chenran Zhao, Dianxi Shi, Yaowen Zhang +2Causal Representation Learning Reinforcement Learning
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · Institute of Military Transportation, Tianjin, China · Intelligent Game and Decision Lab (IGDL), Beijing, China
PriorZero: Bridging Language Priors and World Models for Decision Making
May 12, 2026Junyu Xiong, Yuan Pu, Jia Tang +1LLM-Guided RL Latent World Models
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Nanjing University of Aeronautics and Astronautics +1
OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence
May 12, 2026Jiajian Li, Jingyuan Huang, Junru Gong +3Robotic Control Robotics Simulation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science Shanghai Jiao Tong University
Debiased Model-based Representations for Sample-efficient Continuous Control
May 12, 2026Jiafei Lyu, Zichuan Lin, Scott Fujimoto +5Representation Learning Continuous Control
1Tencent Hunyuan · 2McGill University · School of Computer
Offline Policy Optimization with Posterior Sampling
May 8, 2026Hongqiang Lin, Dongxu Zhang, Yiding Sun +3OOD Generalization Posterior Sampling
Zhejiang University · Xi’an Jiaotong University · Institute of Automation, Chinese Academy of Sciences +1
Improved Model-based Reinforcement Learning with Smooth Kernels
May 8, 2026Kun Long, Yuqiang Li, Xianyi WuReinforcement Learning Continuous-Time RL
School of Statistics, KLATASDS-MOE, East China Normal University, North Zhongshan Road, Shanghai, 200062, PR China.
Sequential Design of Genetic Circuits Under Uncertainty With Reinforcement Learning
May 7, 2026Michal Kobiela, Diego A. Oyarzún, Michael U. GutmannReinforcement Learning Sequential Experimental Design
School of Informatics, The University of Edinburgh, UK · School of Informatics, School of Biological Sciences, The University of Edinburgh, UK · School of Informatics, The University of Edinburgh,, UK
On Training in Imagination
May 7, 2026Nadav Timor, Ravid Shwartz-Ziv, Micah Goldblum +2Reward Modeling Reinforcement Learning
Weizmann Institute of Science · New York University · Columbia University
Offline Reinforcement Learning for Rotation Profile Control in Tokamaks
May 7, 2026Rohit Sonker, Hiro Josep Farre Kaga, Jiayu Chen +5RL Control Offline RL
1Robotics Institute, Carnegie Mellon University · 2Princeton University · 4The University of Hong Kong +2
Unified Framework of Distributional Regret in Multi-Armed Bandits and Reinforcement Learning
May 6, 2026Harin Lee, Min-hwan OhMulti-Armed Bandits Regret Minimization in RL
University of Washington · Seoul National University
ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC
May 6, 2026Yurui Du, Pinhao Song, Yutong Hu +1Visual SSMs Model Predictive Control
KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images, B-3000 Leuven, Belgium · Flanders Make at KU Leuven, B-3000 Leuven, Belgium · KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics, B-3000 Leuven, Belgium
Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination
May 6, 2026Jonathan Spieler, Sven BehnkeModel Predictive Control Optimal Control
Autonomous Intelligent Systems, Computer Science Institute VI - Intelligent Systems and Robotics, Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany.
Counter-Dyna: Data-Efficient RL-Based HVAC Control using Counterfactual Building Models
May 6, 2026Jan Marco Ruiz de Vargas, Fabian Raisch, Zoltan Nagy +2Neural Surrogate Modeling Building Energy Management
Technical University of Munich Munich, Germany · Technical University of Applied Sciences Rosenheim Rosenheim, Germany · Eindhoven University of Technology Eindhoven, Netherlands +1
Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity
May 5, 2026Riddhiman Bhattacharyya, Sayak Chakrabarty, Imon BanerjeeNon-Stationary RL Markov Decision Processes
Department of Statistics, University of California, Santa Cruz Santa Cruz, California, USA · Department of Computer Science Northwestern University Evanston, IL, USA · Department of Industrial Engineering and Management Sciences Northwestern University Evanston, IL, USA
Closed-Loop CO2 Storage Control With History-Based Reinforcement Learning and Latent Model-Based Adaptation
May 4, 2026Sofianos Panagiotis Fotias, Vassilis GaganisRL Control Latent Dynamics Modeling
School of Mining and Metallurgical Engineering, National Technical University of Athens, Athens 157 73, Greece · Institute of Geoenergy, Foundation for Research and Technology, Chania, 73100, Greece
TRAP: Tail-aware Ranking Attack for World-Model Planning
May 3, 2026Siyuan Duan, Ke Zhang, Xizhao LuoBackdoor Attacks World Model-Based Planning
Department of Computer Science and Technology, Soochow University Suzhou, China
Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data
May 2, 2026Ruiqi Xue, Lei Yuan, Kainuo Cheng +2LLM-Guided RL Offline RL
National Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Polixir Technologies +2
Learning to Race in Minutes: Infoprop Dyna on the Mini Wheelbot
May 1, 2026Devdutt Subhasish, Henrik Hose, Sebastian TrimpeRobot Skill Learning Autonomous Racing
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, 52062 Aachen
Model-Based Reinforcement Learning with Double Oracle Efficiency in Policy Optimization and Offline Estimation
May 1, 2026Haichen Hu, Jian Qian, David Simchi-LeviRegret Minimization in RL Markov Decision Processes
Laboratory for Information and Decision Systems · Department of AI and Data Science
Interpretable experiential learning based on state history and global feedback
May 1, 2026Anton KoloninReinforcement Learning Game-Playing Agents
The Artificial Intelligence Research Center, Novosibirsk State University, Russia · Aigents, Russia
RAY-TOLD: Ray-Based Latent Dynamics for Dense Dynamic Obstacle Avoidance with TDMPC
Apr 30, 2026Seungho Han, Seokju Lee, Jeonguk KangRobot Navigation MPPI Control
School of Electrical Engineering, Hanyang University, Ansan 15588, South Korea. · Mechatronics, Systems and Control Lab (MSC Lab), Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Yuseong-gu, Daejeon 34141, South Korea. · Samsung Research, Samsung Electronics, Seoul, South Korea.
Detecting is Easy, Adapting is Hard: Local Expert Growth for Visual Model-Based Reinforcement Learning under Distribution Shift
Apr 30, 2026Haiyang ZhaoDistribution Shift Robustness OOD Generalization
University of Georgia United States
Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamics
Apr 29, 2026Bernd Frauenknecht, Lukas Kesper, Daniel Mayfrank +2Uncertainty Quantification Model Predictive Control
Institute for Data Science in Mechanical Engineering (DSME), RWTH Aachen University · Institute of Climate and Energy Systems (ICE), Energy Systems Engineering (ICE-1), Forschungszentrum Jülich GmbH
Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty
Apr 28, 2026Artur Eisele, Bernd Frauenknecht, Friedrich Solowjow +1Robot Safety Safety Filtering
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany
Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Space Models
Apr 28, 2026Julia Berger, Bernd Frauenknecht, Sebastian Trimpe +1Uncertainty Quantification Latent Dynamics Modeling
1RWTH Aachen University
Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planning
Apr 27, 2026Dong Liu, Yanxuan Yu, Ying Nian WuCoT Reasoning Large Language Model-Guided Optimization
University of California, Los Angeles · Columbia University
CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning
Apr 26, 2026Rahul Narava, Siddharth Verma, Ojas Jain +2Control Barrier Functions Constrained RL
Indian Institute of Technology Ropar · CRAN, Université de Lorraine

May 15, 2026J/K move · Enter open · S save

Imperfect World Models are Exploitable

Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel, Mykel J. Kochenderfer+1

University of Edinburgh · Stanford University

Reward Hacking World Model-Based Planning Model-Based Planning Safe RL Model-Based RL

Open paper AskPDF arXiv

We propose a novel definition of model exploitation in reinforcement learning. Informally, a world model is exploitable if it implies that one policy should be strictly preferred over another while the environment's true transition model implies the reverse. We analogize our definition with a prior characterization of reward hacking but show that the associated proof of inevitability does not transfer to exploitation. To overcome this obstruction, we develop a general theory of reward hacking and model exploitation that proves that exploitation is essentially unavoidable on large policy sets and yields the corresponding claim for hacking as a special case. Unfortunately, we also find that the conditions that guarantee unhackability in finite policy sets have no counterpart that precludes exploitation. Consequently, we introduce a relaxed notion of exploitation and derive a safe horizon within which it can be avoided. Taken together, our results establish a formal bridge between reward hacking and model exploitation and elucidate the limits of safe planning in world models.