Model-Based RL

RL: Reinforcement Learning

Overview

Model-based reinforcement learning studies how agents can learn or use a model of how actions change the environment, then plan with that model to improve decisions and reduce costly interaction. Common approaches include latent world models that predict future states, Dyna-style learning from imagined rollouts, and model-predictive control or tree search; Bayesian models and ensembles help represent uncertainty, while control barriers and other safety methods constrain actions. Methods are tested on suites such as DeepMind Control, MuJoCo, Atari, and D4RL, as well as robotic manipulation, autonomous driving, and other physical systems, alongside theoretical work on regret and sample complexity. The central tension is that imagined experience can make learning more efficient, but model errors may accumulate over long horizons or be exploited by the policy, so reliable uncertainty estimates, safe planning, and transfer to real systems remain open challenges.

Latest papers

273 papers29 papers in the last four weeks, down 12% on the four weeks before. 0.2% of all new papers.

Jan 26, 2025

Safe Learning Control with Optimality and Stability Guarantees

Merely pursuing performance may adversely affect safety, while a conservative policy for safe exploration will degrade the performance. How to guarantee both safety and performance in learning-based control problems is an interesting yet challenging issue. This paper aims to enhance system performance with a safety guarantee by solving reinforcement learning (RL)-based optimal control problems for nonlinear systems subject to high-relative-degree state constraints and unknown time-varying disturbance/actuator faults. A new type of control barrier functions (CBFs), termed high-order reciprocal-based control barrier function, is proposed to handle high-relative-degree constraints, which extends the design of CBFs to enforce robust safety without knowing the disturbance bound. The concept of gradient similarity is proposed to quantify the relationship between safety and performance. Finally, gradient manipulation and adaptive mechanisms are introduced in the model-based safe RL framework to enhance the performance with a safety guarantee. Two simulation examples illustrate the efficacy of the proposed algorithms.
Jul 31, 2024

ProSpec RL: Plan Ahead, then Execute

Imagining potential outcomes of actions before execution helps agents make more informed decisions, a prospective thinking ability fundamental to human cognition. However, mainstream model-free Reinforcement Learning (RL) methods lack the ability to proactively envision future scenarios, plan, and guide strategies. These methods typically rely on trial and error to adjust policy functions, aiming to maximize cumulative rewards or long-term value, even if such high-reward decisions place the environment in extremely dangerous states. To address this, we propose the Prospective (ProSpec) RL method, which makes higher-value, lower-risk optimal decisions by imagining future n-stream trajectories. Specifically, ProSpec employs a dynamic model to predict future states (termed "imagined states") based on the current state and a series of sampled actions. Furthermore, we integrate the concept of Model Predictive Control and introduce a cycle consistency constraint that allows the agent to evaluate and select the optimal actions from these trajectories. Moreover, ProSpec employs cycle consistency to mitigate two fundamental issues in RL: augmenting state reversibility to avoid irreversible events (low risk) and augmenting actions to generate numerous virtual trajectories, thereby improving data efficiency. We validated the effectiveness of our method on the DMControl benchmarks, where our approach achieved significant performance improvements. Code will be open-sourced upon acceptance.
Apr 2, 2024

Active Exploration in Bayesian Model-based Reinforcement Learning for Robot Manipulation

Efficiently tackling multiple tasks within complex environments, such as those found in robot manipulation, remains an ongoing challenge in robotics and an opportunity for data-driven solutions such as reinforcement learning (RL). The high cost of data gathering in robotics encourages data-efficient approaches such as model-based RL, which builds a dynamic model of the robot to conduct most policy learning on cheaper simulations, while enabling data reuse and transfer between tasks. In this work, we employ Bayesian deep learning (BDL) with a double purpose. First, we increase the model quality by improving its calibration. Second, we improve data efficiency by performing active learning of the dynamic model during a preliminary exploration phase, actively estimating the information gain of each transition and using it as the exploration reward. We compare several BDL methods in a realistic robot manipulation setup. Our experiments show that our approach matches the quality of relevant alternatives with much lower robot execution requirements and, unlike previous studies validated only on standard RL environments, we tackle robotic arm end-tasks on a complete manipulator simulator such as CoppeliaSim. Code is available at https://github.com/ropertunizar/BayesianActiveExploration.