Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
Authors: Yilie Huang, Xun Yu Zhou
Organizations: Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.
We study reinforcement learning (RL) for a class of continuous-time stochastic linear-quadratic (LQ) control problems, where volatilities depend on both states and controls while states are scalar-valued and there are no running control rewards, the last feature also leading to the so-called indefinite LQ controls. We propose a model-free, data-driven exploration mechanism that adaptively adjusts entropy regularization by the critic and policy variance by the actor. Unlike the constant or deterministic exploration schedules employed in the existing literature, which require extensive tuning for implementations and ignore learning progress during iterations, our adaptive exploratory approach boosts learning efficiency with minimal tuning. Despite its flexibility, our method achieves a sublinear regret bound that matches the best-known model-free results for this class of LQ problems, which were previously derived only with fixed exploration schedules. Numerical experiments demonstrate that adaptive explorations accelerate convergence and improve regret performance compared to the non-adaptive model-free and model-based counterparts.
Figures & tables
Figure 1: Log-log plot of Algorithm 1 .
Figure 2: Log-log plot of model-based LQ-RL algorithm with fixed exploration schedule.
Method
MSE slope of ϕn
Regret slope
Model-free (this paper)
−0.52
0.73
Model-based benchmark
−0.25
0.84
Table 1: Empirical slopes (log–log) of the MSE of ϕn and the regret for the model-free and model-based algorithms.
Figure 3: Comparison of exploration level and regret under excessive initial exploration.
Figure 4: Comparison of exploration level and regret under insufficient initial exploration.
Figure 5: Comparison of regrets under randomized environments.
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
School of Economics and Management, Fuzhou University, Fuzhou, 350108, Fujian, China · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China