Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning

Saunak Kumar Panda; Tong Li; Yisha Xiang; Ruiqi Liu

cs.LGSep 13, 2026

Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning

Authors: Saunak Kumar Panda, Tong Li, Yisha Xiang, Ruiqi Liu

Organizations: Human-Centered AI Institute, C.T. Bauer College of Business, University of Houston, Houston, TX 77204 USA · Department of Industrial and Systems Engineering, University of Houston, Houston, TX 77204 USA · Department of Mathematics and Statistics, Texas Tech University, Lubbock, TX 79409 USA

Abstract

Variance penalization is a principled approach to risk-sensitive reinforcement learning (RL) that explicitly trades expected return for policy stability. Existing methods require a dedicated second critic to estimate return variance online, adding architectural complexity and compounding estimation error during learning. We propose a nonparametric variance-penalized actor-critic (VPAC) framework that replaces the variance critic with statistically grounded online estimators based on bootstrapping and random scaling, techniques drawn from the statistical inference literature for stochastic approximation. These estimators require no auxiliary network, maintain a single-critic architecture, and produce variance penalties that are bounded by construction, enabling clean convergence analysis. We establish almost-sure convergence for both a variance-penalized Q-learning algorithm and a two-timescale actor-critic variant via the ordinary differential equation (ODE) method, requiring only that variance estimates remain bounded rather than consistent. Empirically, we evaluate across discrete and continuous stochastic environments, demonstrating that the proposed methods match or exceed the variance reduction achieved by the existing dual-critic VPAC baseline while eliminating the overhead of a second critic. We further validate on a high-temperature superconductor (HTS) manufacturing case study, where VPAC-RS (Random Scaling) achieves a 74% reduction in steady-state critical current variability and a 63% reduction in episode return standard deviation, translating directly to improved yield consistency. Our results establish nonparametric statistical inference as a practical and theoretically sound alternative to auxiliary critics for risk-sensitive RL.

Figures & tables

Method	Learned components	Timescales	Extra hyperparameters
VPAC	$\hat{Q}$ , $\hat{\sigma}$ , $\pi_{\theta}$	3	$\alpha_{z}$ , $z$ -architecture
VPAC-RS (ours)	$\hat{Q}$ , $\pi_{\theta}$	2	$\alpha_{\sigma}$ , $c_{\max}$
VPAC-BS (ours)	$\hat{Q}$ , $\{Q^{(k)}\}$ , $\pi_{\theta}$	2	$K$ , $c_{\max}$

Table 1: Architectural comparison: VPAC [ 1 ] versus the proposed nonparametric variants.

Figure 1: Q-learning with random scaling variance penalties (VPAC-RS) in the discrete puddle world. Left: mean episodic return (smoothed) across penalty coefficients $\beta\in\{0,0.01,0.1,0.3\}$ . Right: return variance over training. The baseline ( $\beta=0$ ) reaches a steady-state variance of $\sim\!3{,}000$ ; $\beta=0.1$ achieves $\sim\!500$ (83% reduction) while maintaining comparable mean return. Error bars show $\pm 1$ standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.

Figure 2: Q-learning with bootstrap variance penalties (VPAC-BS) in the discrete puddle world. Higher $\beta$ values suppress early-training policy excursions into the frozen region and produce more stable convergence. Error bars show $\pm 1$ standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.

Figure 3: Actor-critic comparison in the discrete puddle world. Left: mean episodic return. Right: return variance. VPAC-RS and VPAC-BS match or exceed the variance reduction of the dual-critic VPAC baseline while requiring no learned variance critic. Error bars show $\pm 1$ standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.

Panel of Figure 4 — Figure 4: Representative trajectories from a common initial state in the discrete puddle world (left to right: AC, VPAC, VPAC-RS, VPAC-BS). The risk-neutral AC policy traverses the frozen region (grey center block); all three variance-penalized policies route around it.

Figure 5: State-visitation frequency heatmaps in the discrete puddle world, averaged over 10 seeds and 98 trajectory rollouts per seed per method (left to right: AC, VPAC, VPAC-RS, VPAC-BS). Darker shading indicates higher visitation frequency; the red rectangle marks the boundary of the frozen zone. The goal state is excluded from the colour normalisation. AC exhibits frequent frozen zone traversal. VPAC adopts a bottom routing strategy that largely avoids the frozen zone. VPAC-RS and VPAC-BS both achieve near-complete frozen zone avoidance, with visitation concentrated on an upper-right route to the goal.

Figure 6: Performance in the continuous-state puddle world. Left: mean episodic return. Right: return variance over training. VPAC-RS and VPAC-BS achieve 40–50% variance reduction relative to the AC baseline while maintaining comparable mean return. Error bars show $\pm 1$ standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.

Figure 7: Representative trajectories in the continuous puddle world. The shaded region indicates the high-variance puddle. The risk-neutral AC policy (left) passes through the puddle; VPAC-RS and VPAC-BS (right) route consistently around it.

$\beta$	Mean $I_{c}$ [A]	Ret mean	Ret std	SS IC std [A]	CVaR-10%
0.000 (baseline)	339.7	58.84	6.78	55.4	50.77
0.001	332.7	57.95	5.11	28.2	50.77
0.010	318.3	56.59	2.52	14.1	52.55
0.100	322.6	57.03	2.52	9.6	52.20
1.000	329.3	58.04	4.16	21.3	53.43

Table 2: VPAC-RS evaluation metrics at

\kappa=1.5

, averaged over 10 seeds

\times

40 rollouts. SS IC std: steady-state

I_{c}

standard deviation (steps 140–200). CVaR-10%: mean of worst 10% of episode returns (higher is better). Best non-baseline entry per column in bold .

Figure 8: Mean $\pm$ IQR of the $I_{c}(A)$ trajectory across 400 greedy rollouts (10 seeds $\times$ 40 rollouts) under the VPAC-RS policy at $\kappa=1.5$ . The solid green line is the risk-neutral baseline ( $\beta=0$ ); the dashed blue line is the best risk-adjusted policy ( $\beta=0.01$ ); thin grey lines show the remaining $\beta$ values. The dashed horizontal line marks the dataset maximum (590 A), which is the physical upper bound of the surrogate model and not a tracking setpoint. The shaded region indicates the steady-state evaluation window (steps 140–200), over which SS IC std is computed. The $\beta=0.01$ policy exhibits a markedly tighter IQR band throughout, with the separation widening in the steady-state window (SS IC std: 14.1 A versus 55.4 A for the baseline).

Figure 9: Risk-return Pareto scatter for VPAC-RS at $\kappa=1.5$ . Each marker represents one $\beta$ value evaluated over 400 greedy rollouts; axes report mean $I_{c}(A)$ across all rollout steps and episode return standard deviation (Ret std). The starred marker identifies the best risk-adjusted policy ( $\beta=0.01$ , Ret std $=2.52$ , 63% below the baseline). The dashed line connects the Pareto-non-dominated operating points. The $\beta=1.0$ point falls inside the frontier on both axes, confirming that over-penalisation degrades both objectives simultaneously.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Common parameters
Parameter	Discrete	Continuous
Discount $\gamma$	0.99	0.99
Max episodes	1,000	2,000
Max steps	500	5,000
Seeds	10	10
Smoothing ( $\alpha_{\mathrm{smooth}}$ , returns)	0.03	0.03

Appendix

Table 3: Hyperparameters used across discrete and continuous puddle world settings.

lr_{c}

: critic learning rate.

lr_{a}

: actor learning rate.

lr_{\sigma}

: variance critic learning rate (VPAC only).

$\beta$	Mean $I_{c}$ [A]	Ret mean	Ret std	SS IC std [A]	CVaR-10%
0.000 (baseline)	338.3	59.06	4.48	–	51.27
0.001	331.3	57.97	4.58	–	52.32
0.010	340.7	59.78	5.55	–	50.18
0.100	320.5	56.74	2.52	9.6	51.76
1.000	332.0	58.62	4.13	–	52.93

Appendix

Table 4: VPAC-BS evaluation metrics at

\kappa=1.5

, averaged over 10 seeds

\times

40 rollouts. SS IC std: steady-state

I_{c}

standard deviation (steps 140–200). CVaR-10%: mean of worst 10% of episode returns (higher is better). Best non-baseline entry per column in bold .

Figure 10: Mean $\pm$ IQR of the $I_{c}(A)$ trajectory across 400 greedy rollouts under the VPAC-BS policy at $\kappa=1.5$ . The solid green line is the risk-neutral baseline ( $\beta=0$ ); the dashed blue line is the best risk-adjusted policy ( $\beta=0.1$ ). The $\beta=0.1$ IQR band is wider than the baseline in the transient phase (steps 50–130) before narrowing in the steady-state window, indicating that VPAC-BS achieves variance reduction primarily in the settled phase of the episode rather than throughout.

Figure 11: Risk-return Pareto scatter for VPAC-BS at $\kappa=1.5$ . Axes and marker conventions follow Figure 9 . The starred marker identifies the best risk-adjusted policy ( $\beta=0.1$ , Ret std $=2.52$ , 44% below the baseline). Four operating points are Pareto-non-dominated; the $\beta=0.01$ point is non-dominated on the mean $I_{c}$ axis (highest mean $I_{c}$ of all configurations) despite exhibiting the highest Ret std, reflecting the irregular variance signal produced by the bootstrapping estimator at small penalty values in this environment.

Figure 12: Supporting evaluation panels for VPAC-RS at $\kappa=1.5$ (400 greedy rollouts, stochastic environment). (B) Rolling $I_{c}(A)$ standard deviation across rollouts at each episode step, smoothed with a uniform window of 15 steps. Annotated values show the mean steady-state IC std (steps 140–200) for $\beta=0$ (55.4 A) and $\beta=0.01$ (14.1 A). The shaded region is the steady-state evaluation window. (C) Episode return distributions across all rollouts; boxes span the interquartile range, with the median as the horizontal bar and whiskers extending to 1.5 times the IQR. The $\beta=0.01$ and $\beta=0.1$ policies produce the tightest return distributions, consistent with the Ret std values in Table 2 .

Figure 13: Supporting evaluation panels for VPAC-BS at $\kappa=1.5$ (400 greedy rollouts, stochastic environment). Panel and axis conventions follow Figure 12 . (B) The $\beta=0.01$ curve substantially exceeds the baseline IC std throughout the episode, confirming the pathological result reported in Table 4 . The $\beta=0.1$ policy reaches near-zero IC std in the steady-state window (annotated: 9.6 A). (C) Episode return distributions are wider and less ordered across $\beta$ values than for VPAC-RS (cf. Figure 12 C), consistent with the weaker and less consistent variance-reduction signal produced by the bootstrapping estimator in this environment.

Figure 14: Training curves for VPAC-RS at $\kappa=1.5$ (10 seeds; error bars show $\pm 1$ standard error of the mean, plotted at sampled intervals for clarity). Left: mean episode return over training. All $\beta$ values converge to comparable return levels, confirming that variance penalisation does not significantly degrade mean performance during learning. Right: variance of normalised discounted returns estimated during training. The y-axis is truncated to show relative ordering across $\beta$ values; absolute variance range is 1.45–1.90.

Figure 15: Training curves for VPAC-BS at $\kappa=1.5$ (10 seeds; error bars show $\pm 1$ standard error of the mean, plotted at sampled intervals for clarity). Left: mean episode return over training. All $\beta$ values converge to comparable return levels. Right: variance of normalised discounted returns estimated during training. Notably, $\beta=0.01$ (green) sits above the risk-neutral baseline ( $\beta=0$ , blue) throughout training, consistent with the pathological evaluation result for this configuration (Table 4 ), and in contrast to VPAC-RS where $\beta=0.01$ produces the lowest training variance (Figure 14 ). The y-axis is truncated to show relative ordering; absolute variance range is 1.45–1.90.

Risk-Sensitive RL

Risk-sensitive reinforcement learning optimizes policies for downside risk, tail outcomes, uncertainty, or safety constraints beyond expected reward.

See all 85 papers

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
Sep 28, 2026Hsiao-Ru Pan, Florent Draye, Bernhard SchölkopfReinforcement Learning Actor-Critic Methods
MPI-IS · ELLIS Institute Tübingen
Actor-Critic with Active Importance Sampling
May 8, 2026Majid Molaei, Gabor Paczolay, Matteo Papini +2Reinforcement Learning Importance Sampling for RL
Politecnico di Milano Milan, Italy
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
Oct 11, 2026Yibo Wang, Yuanxin Liu, Zhaoran WangReinforcement Learning with Verifiable Rewards Policy Gradient Methods

Sep 28, 2026J/K move · Enter open · S save

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

Hsiao-Ru Pan, Florent Draye, Bernhard Schölkopf

MPI-IS · ELLIS Institute Tübingen

Reinforcement Learning Actor-Critic Methods Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.

Abstract

Risk-Sensitive RL

Explore similar work

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

Actor-Critic with Active Importance Sampling

Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards