Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
Authors: Saunak Kumar Panda, Tong Li, Yisha Xiang, Ruiqi Liu
Organizations: Human-Centered AI Institute, C.T. Bauer College of Business, University of Houston, Houston, TX 77204 USA · Department of Industrial and Systems Engineering, University of Houston, Houston, TX 77204 USA · Department of Mathematics and Statistics, Texas Tech University, Lubbock, TX 79409 USA
Variance penalization is a principled approach to risk-sensitive reinforcement learning (RL) that explicitly trades expected return for policy stability. Existing methods require a dedicated second critic to estimate return variance online, adding architectural complexity and compounding estimation error during learning. We propose a nonparametric variance-penalized actor-critic (VPAC) framework that replaces the variance critic with statistically grounded online estimators based on bootstrapping and random scaling, techniques drawn from the statistical inference literature for stochastic approximation. These estimators require no auxiliary network, maintain a single-critic architecture, and produce variance penalties that are bounded by construction, enabling clean convergence analysis. We establish almost-sure convergence for both a variance-penalized Q-learning algorithm and a two-timescale actor-critic variant via the ordinary differential equation (ODE) method, requiring only that variance estimates remain bounded rather than consistent. Empirically, we evaluate across discrete and continuous stochastic environments, demonstrating that the proposed methods match or exceed the variance reduction achieved by the existing dual-critic VPAC baseline while eliminating the overhead of a second critic. We further validate on a high-temperature superconductor (HTS) manufacturing case study, where VPAC-RS (Random Scaling) achieves a 74% reduction in steady-state critical current variability and a 63% reduction in episode return standard deviation, translating directly to improved yield consistency. Our results establish nonparametric statistical inference as a practical and theoretically sound alternative to auxiliary critics for risk-sensitive RL.
Figures & tables
Method
Learned components
Timescales
Extra hyperparameters
VPAC
Q^ , σ^ , πθ
3
αz , z -architecture
VPAC-RS (ours)
Q^ , πθ
2
ασ , cmax
VPAC-BS (ours)
Q^ , {Q(k)} , πθ
2
K , cmax
Table 1: Architectural comparison: VPAC [ 1 ] versus the proposed nonparametric variants.
Figure 1: Q-learning with random scaling variance penalties (VPAC-RS) in the discrete puddle world. Left: mean episodic return (smoothed) across penalty coefficients β∈{0,0.01,0.1,0.3} . Right: return variance over training. The baseline ( β=0 ) reaches a steady-state variance of ∼3,000 ; β=0.1 achieves ∼500 (83% reduction) while maintaining comparable mean return. Error bars show ±1 standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.
Figure 2: Q-learning with bootstrap variance penalties (VPAC-BS) in the discrete puddle world. Higher β values suppress early-training policy excursions into the frozen region and produce more stable convergence. Error bars show ±1 standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.
Figure 3: Actor-critic comparison in the discrete puddle world. Left: mean episodic return. Right: return variance. VPAC-RS and VPAC-BS match or exceed the variance reduction of the dual-critic VPAC baseline while requiring no learned variance critic. Error bars show ±1 standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.
Figure 4: Representative trajectories from a common initial state in the discrete puddle world (left to right: AC, VPAC, VPAC-RS, VPAC-BS). The risk-neutral AC policy traverses the frozen region (grey center block); all three variance-penalized policies route around it.
Figure 5: State-visitation frequency heatmaps in the discrete puddle world, averaged over 10 seeds and 98 trajectory rollouts per seed per method (left to right: AC, VPAC, VPAC-RS, VPAC-BS). Darker shading indicates higher visitation frequency; the red rectangle marks the boundary of the frozen zone. The goal state is excluded from the colour normalisation. AC exhibits frequent frozen zone traversal. VPAC adopts a bottom routing strategy that largely avoids the frozen zone. VPAC-RS and VPAC-BS both achieve near-complete frozen zone avoidance, with visitation concentrated on an upper-right route to the goal.
Figure 6: Performance in the continuous-state puddle world. Left: mean episodic return. Right: return variance over training. VPAC-RS and VPAC-BS achieve 40–50% variance reduction relative to the AC baseline while maintaining comparable mean return. Error bars show ±1 standard error of the mean across 10 seeds, plotted at sampled intervals for clarity.
Figure 7: Representative trajectories in the continuous puddle world. The shaded region indicates the high-variance puddle. The risk-neutral AC policy (left) passes through the puddle; VPAC-RS and VPAC-BS (right) route consistently around it.
β
Mean Ic [A]
Ret mean
Ret std
SS IC std [A]
CVaR-10%
0.000 (baseline)
339.7
58.84
6.78
55.4
50.77
0.001
332.7
57.95
5.11
28.2
50.77
0.010
318.3
56.59
2.52
14.1
52.55
0.100
322.6
57.03
2.52
9.6
52.20
1.000
329.3
58.04
4.16
21.3
53.43
Table 2: VPAC-RS evaluation metrics at κ=1.5 , averaged over 10 seeds × 40 rollouts. SS IC std: steady-state Ic standard deviation (steps 140–200). CVaR-10%: mean of worst 10% of episode returns (higher is better). Best non-baseline entry per column in bold .
Figure 8: Mean ± IQR of the Ic(A) trajectory across 400 greedy rollouts (10 seeds × 40 rollouts) under the VPAC-RS policy at κ=1.5 . The solid green line is the risk-neutral baseline ( β=0 ); the dashed blue line is the best risk-adjusted policy ( β=0.01 ); thin grey lines show the remaining β values. The dashed horizontal line marks the dataset maximum (590 A), which is the physical upper bound of the surrogate model and not a tracking setpoint. The shaded region indicates the steady-state evaluation window (steps 140–200), over which SS IC std is computed. The β=0.01 policy exhibits a markedly tighter IQR band throughout, with the separation widening in the steady-state window (SS IC std: 14.1 A versus 55.4 A for the baseline).
Figure 9: Risk-return Pareto scatter for VPAC-RS at κ=1.5 . Each marker represents one β value evaluated over 400 greedy rollouts; axes report mean Ic(A) across all rollout steps and episode return standard deviation (Ret std). The starred marker identifies the best risk-adjusted policy ( β=0.01 , Ret std =2.52 , 63% below the baseline). The dashed line connects the Pareto-non-dominated operating points. The β=1.0 point falls inside the frontier on both axes, confirming that over-penalisation degrades both objectives simultaneously.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Discrete
Continuous
Common parameters
Discount γ
0.99
0.99
Max episodes
1,000
2,000
Max steps
500
5,000
Seeds
10
10
Smoothing ( αsmooth , returns)
0.03
0.03
Appendix
Table 3: Hyperparameters used across discrete and continuous puddle world settings. lrc : critic learning rate. lra : actor learning rate. lrσ : variance critic learning rate (VPAC only).
β
Mean Ic [A]
Ret mean
Ret std
SS IC std [A]
CVaR-10%
0.000 (baseline)
338.3
59.06
4.48
–
51.27
0.001
331.3
57.97
4.58
–
52.32
0.010
340.7
59.78
5.55
–
50.18
0.100
320.5
56.74
2.52
9.6
51.76
1.000
332.0
58.62
4.13
–
52.93
Appendix
Table 4: VPAC-BS evaluation metrics at κ=1.5 , averaged over 10 seeds × 40 rollouts. SS IC std: steady-state Ic standard deviation (steps 140–200). CVaR-10%: mean of worst 10% of episode returns (higher is better). Best non-baseline entry per column in bold .
Figure 10: Mean ± IQR of the Ic(A) trajectory across 400 greedy rollouts under the VPAC-BS policy at κ=1.5 . The solid green line is the risk-neutral baseline ( β=0 ); the dashed blue line is the best risk-adjusted policy ( β=0.1 ). The β=0.1 IQR band is wider than the baseline in the transient phase (steps 50–130) before narrowing in the steady-state window, indicating that VPAC-BS achieves variance reduction primarily in the settled phase of the episode rather than throughout.
Figure 11: Risk-return Pareto scatter for VPAC-BS at κ=1.5 . Axes and marker conventions follow Figure 9 . The starred marker identifies the best risk-adjusted policy ( β=0.1 , Ret std =2.52 , 44% below the baseline). Four operating points are Pareto-non-dominated; the β=0.01 point is non-dominated on the mean Ic axis (highest mean Ic of all configurations) despite exhibiting the highest Ret std, reflecting the irregular variance signal produced by the bootstrapping estimator at small penalty values in this environment.
Figure 12: Supporting evaluation panels for VPAC-RS at κ=1.5 (400 greedy rollouts, stochastic environment). (B) Rolling Ic(A) standard deviation across rollouts at each episode step, smoothed with a uniform window of 15 steps. Annotated values show the mean steady-state IC std (steps 140–200) for β=0 (55.4 A) and β=0.01 (14.1 A). The shaded region is the steady-state evaluation window. (C) Episode return distributions across all rollouts; boxes span the interquartile range, with the median as the horizontal bar and whiskers extending to 1.5 times the IQR. The β=0.01 and β=0.1 policies produce the tightest return distributions, consistent with the Ret std values in Table 2 .
Figure 13: Supporting evaluation panels for VPAC-BS at κ=1.5 (400 greedy rollouts, stochastic environment). Panel and axis conventions follow Figure 12 . (B) The β=0.01 curve substantially exceeds the baseline IC std throughout the episode, confirming the pathological result reported in Table 4 . The β=0.1 policy reaches near-zero IC std in the steady-state window (annotated: 9.6 A). (C) Episode return distributions are wider and less ordered across β values than for VPAC-RS (cf. Figure 12 C), consistent with the weaker and less consistent variance-reduction signal produced by the bootstrapping estimator in this environment.
Figure 14: Training curves for VPAC-RS at κ=1.5 (10 seeds; error bars show ±1 standard error of the mean, plotted at sampled intervals for clarity). Left: mean episode return over training. All β values converge to comparable return levels, confirming that variance penalisation does not significantly degrade mean performance during learning. Right: variance of normalised discounted returns estimated during training. The y-axis is truncated to show relative ordering across β values; absolute variance range is 1.45–1.90.
Figure 15: Training curves for VPAC-BS at κ=1.5 (10 seeds; error bars show ±1 standard error of the mean, plotted at sampled intervals for clarity). Left: mean episode return over training. All β values converge to comparable return levels. Right: variance of normalised discounted returns estimated during training. Notably, β=0.01 (green) sits above the risk-neutral baseline ( β=0 , blue) throughout training, consistent with the pathological evaluation result for this configuration (Table 4 ), and in contrast to VPAC-RS where β=0.01 produces the lowest training variance (Figure 14 ). The y-axis is truncated to show relative ordering; absolute variance range is 1.45–1.90.
Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.
This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm, an extension of the Actor-Critic framework for reducing variance in policy gradient estimation. AISAC optimizes the behavior policy to minimize gradient variance while preserving unbiased gradient estimates. Using importance sampling principles, the algorithm adapts the behavior policy toward efficient data collection distributions aligned with target policy gradients. For continuous action spaces, AISAC employs Gaussian behavior policies optimized through cross-entropy minimization. We provide theoretical analysis demonstrating variance reduction and unbiasedness. Experiments on Inverted Pendulum and Half Cheetah tasks show improved learning speed, sample efficiency, and training stability compared to standard Actor-Critic methods. Results indicate that optimizing the behavior policy improves both target policy updates and critic estimation accuracy across different hyperparameter settings. AISAC accelerates convergence and stabilizes reinforcement learning training, making it promising for real-world applications. Future work includes integration with advanced algorithms such as Soft Actor-Critic and TD3 for more complex environments.
Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, μvarπ(s)∝dπ(s)σπ(s), which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.