Reinforcement learning for constrained stochastic linear--quadratic control in continuous time
Authors: Weiping Wu, Qitong Pan, Xun Li, Peng Wu
Organizations: School of Economics and Management, Fuzhou University, Fuzhou, 350108, Fujian, China · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China
We study continuous-time stochastic linear--quadratic control with a scalar state and nonnegative control constraints. With quadratic running costs, randomized exploration can change the optimal mean feedback. We introduce an auxiliary problem without entropy in which the conditional policy mean is nonnegative and the covariance is prescribed in proportion to the squared state. Moment reduction and state separation yield two regional Riccati equations and characterize the finite-horizon value function and optimal conditional mean. We give a sufficient control--cost compatibility condition under which this mean coincides with the optimal feedback of the original constrained problem. For time-invariant coefficients, we characterize the undiscounted infinite-horizon optimum and establish mean square asymptotic stability under the stated assumptions. For unknown dynamics, we develop model-free actor--critic schemes for finite and infinite horizons. Numerical experiments compare the learned policies with their analytical targets and examine the effect of exploration on the optimal mean.
Figures & tables
Case
Method
Critic error
EΨ0
E0
Terminal
Full
0.0138±0.0039
0.0035±0.0004
0.0035±0.0004
Running
Full
0.0263±0.0136
0.0071±0.0006
0.0070±0.0006
Running
Ablation
1.0000±6.9×10−8
0.0683±0.0001
0.0678±0.0001
Compatible
Full
0.1004±0.0626
0.0049±0.0002
0.0049±0.0002
Incompatible
Full
0.2855±0.1225
0.0098±0.0007
0.0115±0.0003
Table 1: Finite-horizon errors after 4000 actor iterations using the unnormalized gradients in ( 82 ) and ( 88 ). Entries are means ± sample standard deviations over three independent seeds.
Figure 1: Critic coefficient error during learning. The full method is shown by solid lines and the terminal-only ablation by dashed lines. Curves are three-seed means, shaded bands are one sample standard deviation, and the vertical scale is logarithmic.
Figure 2: Full-method gain errors relative to the exploratory and original oracles. The error to K+Ψ0 is shown by solid lines and the error to K+0 by dashed lines. Curves are three-seed means, shaded bands are one sample standard deviation, and the vertical scale is logarithmic.
Region
Learned K±
K±Ψ,∗
Learned θ±
P±Ψ,∗
Current-policy critic error
Stable runs
+
0.9837±0.0097
1.0000
0.5529±0.0589
0.5789
0.0517
20/20
−
0.0000±0.0000
0.0000
2.3526±0.0282
2.3333
0.0291
20/20
Table 2: Batch infinite-horizon actor–critic results after 400 rounds. Learned quantities are means ± sample standard deviations over 20 independent seeds. The current-policy critic error is the mean of ∣θ±+L±Ψ(K±)/N±Ψ(K±)∣ ; “stable runs” counts runs with N±Ψ(K±,k)<0 at every recorded iterate. Rates and oracle values are retrospective model-based diagnostics, not training gates.
Figure 3: Batch infinite-horizon learning across 20 independent seeds. Solid curves show means, shaded bands one sample standard deviation, and dashed lines the model-based algebraic targets. The positive critic has intermittent excursions even though the final learned gains are near their targets; the curves do not establish monotone or general convergence.
Delft Center for Systems and Control, Delft University of Technology, 2628 CD, Delft, The Netherlands · Department of Engineering Science, University of Oxford, OX1 3PJ Oxford, U.K.