cs.LGJul 1, 2025

Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems

Authors: Yilie Huang, Xun Yu Zhou

Organizations: Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.

Abstract

We study reinforcement learning (RL) for a class of continuous-time stochastic linear-quadratic (LQ) control problems, where volatilities depend on both states and controls while states are scalar-valued and there are no running control rewards, the last feature also leading to the so-called indefinite LQ controls. We propose a model-free, data-driven exploration mechanism that adaptively adjusts entropy regularization by the critic and policy variance by the actor. Unlike the constant or deterministic exploration schedules employed in the existing literature, which require extensive tuning for implementations and ignore learning progress during iterations, our adaptive exploratory approach boosts learning efficiency with minimal tuning. Despite its flexibility, our method achieves a sublinear regret bound that matches the best-known model-free results for this class of LQ problems, which were previously derived only with fixed exploration schedules. Numerical experiments demonstrate that adaptive explorations accelerate convergence and improve regret performance compared to the non-adaptive model-free and model-based counterparts.

Figures & tables

Explore similar work

CardsList
  1. Reinforcement learning for constrained stochastic linear--quadratic control in continuous time

    Oct 26, 2026Weiping Wu, Qitong Pan, Xun Li +1Stochastic Optimal ControlLinear Quadratic Regulator

  2. Optimistic Online LQR via Intrinsic Rewards

    Mar 30, 2026Marcell Bartos, Bruce D. Lee, Lenart Treven +3Linear Quadratic RegulatorRegret Minimization in RL

  3. Rate-Optimal Regret for the Safe Learning-based Control of the Constrained Linear Quadratic Regulator

    Apr 24, 2026Spencer Hutchinson, Nanfei Jiang, Mahnoosh AlizadehStochastic Linear Dynamical SystemsStochastic Optimal Control