Model-Based Reinforcement Learning for Control under Time-Varying Dynamics

Klemens Iten; Bruce Lee; Chenhao Li; Lenart Treven; Andreas Krause; Bhavya Sukhija

cs.LGApr 2, 2026

Model-Based Reinforcement Learning for Control under Time-Varying Dynamics

Authors: Klemens Iten, Bruce Lee, Chenhao Li, Lenart Treven, Andreas Krause, Bhavya Sukhija

Organizations: ETH Zürich, Rämistrasse 101, 8092 Zürich, Switzerland · ETH AI Center at ETH Zürich

Abstract

Learning-based control methods typically assume stationary system dynamics, an assumption often violated in real-world systems due to drift, wear, or changing operating conditions. We study reinforcement learning for control under time-varying dynamics. We consider a continual model-based reinforcement learning setting in which an agent repeatedly learns and controls a dynamical system whose transition dynamics evolve across episodes. We analyze the problem using Gaussian process dynamics models under frequentist variation-budget assumptions. Our analysis shows that persistent non-stationarity requires explicitly limiting the influence of outdated data to maintain calibrated uncertainty and meaningful dynamic regret guarantees. Motivated by these insights, we propose a practical optimistic model-based reinforcement learning algorithm with adaptive data buffer mechanisms and demonstrate improved performance on continuous control benchmarks with non-stationary dynamics.

Figures & tables

Fig. 1: Learning curves for the setting with GP dynamics on the Pendulum environment. We report the mean cumulative regret $R_{N}$ over 5 random seeds. At episode $N=10$ , we induce a change in dynamics by limiting the maximum applicable action ${\bm{u}}_{N,t}$ to half its original value over time. This leads to a linear regret for the stationary SOMBRL baseline, while R-OMBRL and SW-OMBRL adapt to the change in dynamics.

Fig. 2: Dynamic regret under time-varying dynamics for different decay rates. The stationary baseline SOMBRL accumulates large regret due to stale data, while R-OMBRL and SW-OMBRL improve tracking by restricting the data buffer. We report the mean regret compared to an estimate of the optimal performance over five seeds with standard error.

Fig. 3: Dynamic regret across multiple environments. The top row shows the evolution of the maximum admissible torque $\bar{u}_{n}$ and its decay over environment training steps, while the bottom rows report the cumulative regret $R_{n}$ averaged over five seeds with standard error. Under stationary dynamics, all methods perform similarly. After the onset of non-stationarity, R-OMBRL and SW-OMBRL significantly reduce regret compared to the stationary baseline.

Fig. 4: Hardware experiments on a real RC car. The task is a parking maneuver that transitions from drift-based behavior to standard parking as the maximum throttle decays at $N=14$ episodes. The top row shows rollouts after $N=30$ , where R-OMBRL successfully completes the task while SOMBRL fails to adapt. The bottom row shows the mean return and error bands along trajectories ${\bm{\tau}}_{N}$ on the real system over episodes, averaged over 6 random seeds. Restricting the data buffer enables adaptation to changing dynamics and improves performance compared to the stationary baseline.

Fig. 5: Ablation of the soft reset parameter $\alpha_{1}=\alpha_{2}=\alpha$ from ( 15 ) for R-OMBRL under non-stationary dynamics. We vary the strength of the parameter perturbations applied to the dynamics model and policy after replay-buffer resets. Moderate perturbations yield the best trade-off between stability and plasticity, while aggressive policy perturbations are particularly detrimental.

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 272 papers

Safe Continual Reinforcement Learning in Non-stationary Environments
Apr 21, 2026Austin Coursey, Abel Diaz-Gonzalez, Marcos Quinones-Grueiro +1RL Control Non-Stationary RL
Institute for Software Integrated Systems, Vanderbilt University
A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment
Oct 21, 2026Angela P. Schoellig, Matthias A. Müller, Sebastian Trimpe +8RL Control
Chair of Safety, Performance, and Reliability for Learning Systems, Technical University of Munich · Institute of Automatic Control, Leibniz University Hannover · Institute for Data Science in Mechanical Engineering, RWTH Aachen University +2
Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning
Sep 16, 2026Everest Yang, Skye Thompson, George D. KonidarisNon-Stationary RL Continual Robot Learning
Brown University, Providence, RI, USA

Apr 21, 2026J/K move · Enter open · S save

Safe Continual Reinforcement Learning in Non-stationary Environments

Austin Coursey, Abel Diaz-Gonzalez, Marcos Quinones-Grueiro, Gautam Biswas

Institute for Software Integrated Systems, Vanderbilt University

RL Control Non-Stationary RL Safe RL

Open paper AskPDF arXiv

Reinforcement learning (RL) offers a compelling data-driven paradigm for synthesizing controllers for complex systems when accurate physical models are unavailable; however, most existing control-oriented RL methods assume stationarity and, therefore, struggle in real-world non-stationary deployments where system dynamics and operating conditions can change unexpectedly. Moreover, RL controllers acting in physical environments must satisfy safety constraints throughout their learning and execution phases, rendering transient violations during adaptation unacceptable. Although continual RL and safe RL have each addressed non-stationarity and safety, respectively, their intersection remains comparatively unexplored, motivating the study of safe continual RL algorithms that can adapt over the system's lifetime while preserving safety. In this work, we systematically investigate safe continual reinforcement learning by introducing three benchmark environments that capture safety-critical continual adaptation and by evaluating representative approaches from safe RL, continual RL, and their combinations. Our empirical results reveal a fundamental tension between maintaining safety constraints and preventing catastrophic forgetting under non-stationary dynamics, with existing methods generally failing to achieve both objectives simultaneously. To address this shortcoming, we examine regularization-based strategies that partially mitigate this trade-off and characterize their benefits and limitations. Finally, we outline key open challenges and research directions toward developing safe, resilient learning-based controllers capable of sustained autonomous operation in changing environments.

Abstract

RL Control

Explore similar work

Safe Continual Reinforcement Learning in Non-stationary Environments

A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment

Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning