Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling
Organizations: School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China.
Abstract
Actor--critic methods rely on critic gradients to improve continuous-action policies, yet finite-sample guarantees must control both value and gradient errors and their propagation through the actor. We develop a neural actor-critic procedure that combines a regularized minimax Bellman estimator over sigmoid networks with a finite reflected Langevin sampler driven by the fitted critic gradient. Under uniform Bellman regularity and covered reset sampling, a positive high-order action-derivative penalty yields value and action-gradient risk bounds with classical nonparametric minimax exponents up to logarithmic factors. We propagate estimation, mixing, discretization, temperature, and policy-iteration errors to obtain return guarantees, with refinements for bounded multimodal potentials and strongly concave regimes. Empirical studies across controlled simulations, logged continuous-control data, and a simulated decision task show favorable performance relative to the considered baselines.
Figures & tables
| Method | Uniform | sigmoid DNN | linear–REM | quadratic–REM |
|---|---|---|---|---|
| Return (MC SE) | 1.55 (0.27%) | 2.68 (2.80%) | 1.52 (0.29%) | 1.25 (2.93%) |
| Method | Uniform | sigmoid DNN | linear–REM | quadratic–REM |
|---|---|---|---|---|
| Return (MC SE) | 1.62 (0.05%) | 1.95 (0.48%) | 1.60 (0.05%) | 1.58 (0.22%) |
| Dataset | sigmoid DNN | Linear | Quadratic |
|---|---|---|---|
| RMSE | |||
| HalfCheetah Replay | |||
| Hopper Replay | |||
| Walker2d Replay | |||
| HalfCheetah Medium | |||
| Hopper Medium | |||
| Dataset | sigmoid DNN | Linear | Quadratic |
|---|---|---|---|
| HalfCheetah Replay | |||
| Hopper Replay | |||
| Walker2d Replay | |||
| HalfCheetah Medium | |||
| Hopper Medium | |||
| Walker2d Medium |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Work | Critic information | Sampling and numerical layer | RL guarantees |
|---|---|---|---|
| Finite-time actor–critic ( Wu et al., 2020 ; Chen and Maguluri, 2025 ; Chen et al., 2026 ) | Data-derived value error; no separate action-gradient risk bound | No iterative action-space sampling layer | Finite-sample actor–critic convergence |
| Nonparametric policy evaluation ( Chen and Qi, 2022 ) | Data-derived value and derivative rates | No conditional action sampler | Fixed-policy off-policy evaluation |
| LQL and noise-conditioned LQL ( Ki et al., 2026 ) | Neural temporal-difference critic; learned action gradients | Direct Langevin actions; noise-conditioned smoothing | Algorithmic construction and empirical policy performance |
| WPG ( Zhu et al., 2026a ) | Exact soft critic and its action gradient | Langevin evolution with discretization control | Entropy-regularized convergence with an exact critic |
| WPPG ( Zhu et al., 2026b ) | Soft-critic value and gradient errors assumed | Wasserstein proximal update; heat-step implementation | Proximal-update convergence under prescribed critic errors |
| Projected Langevin ( Lamperski, 2021 ; Zheng and Lamperski, 2022 ) | Prescribed potential; exact or stochastic gradients | Nonconvex constrained sampling; time-uniform control in the latter work | Sampling and optimization, not neural Bellman policy iteration |
| Method | Heating load | Reward | Oracle regret | Efficient-quartile probability |
|---|---|---|---|---|
| sigmoid DNN | ||||
| Linear | ||||
| Uniform | ||||
| Oracle |
Actor-Critic Methods
Actor-critic methods train a policy alongside a value estimator to guide decisions in complex control and decision-making tasks.
See all 119 papers