RE-SAC: Disentangling aleatoric and epistemic risks in bus fleet control: A stable and robust ensemble DRL approach

Yifan Zhang; Liang Zheng; Qifan Zhang

cs.LGMar 19, 2026

RE-SAC: Disentangling aleatoric and epistemic risks in bus fleet control: A stable and robust ensemble DRL approach

Authors: Yifan Zhang, Liang Zheng, Qifan Zhang

Organizations: Central South University

Abstract

Bus holding control is challenging due to stochastic traffic and passenger demand. While deep reinforcement learning (DRL) shows promise, standard actor-critic algorithms suffer from Q-value instability in volatile environments. A key source of this instability is the conflation of two distinct uncertainties: aleatoric uncertainty (irreducible noise) and epistemic uncertainty (data insufficiency). Treating these as a single risk leads to value underestimation in noisy states, causing catastrophic policy collapse. We propose a robust ensemble soft actor-critic (RE-SAC) framework to explicitly disentangle these uncertainties. RE-SAC applies Integral Probability Metric (IPM)-based weight regularization to the critic network to hedge against aleatoric risk, providing a smooth analytical lower bound for the robust Bellman operator without expensive inner-loop perturbations. To address epistemic risk, a diversified Q-ensemble penalizes overconfident value estimates in sparsely covered regions. This dual mechanism prevents the ensemble variance from misidentifying noise as a data gap, a failure mode identified in our ablation study. Experiments in a realistic bidirectional bus corridor simulation demonstrate that RE-SAC achieves the highest cumulative reward (approx. -0.4e6) compared to vanilla SAC (-0.55e6). Mahalanobis rareness analysis confirms that RE-SAC reduces Oracle Q-value estimation error by up to 62% in rare out-of-distribution states (MAE of 1647 vs. 4343), demonstrating superior robustness under high traffic variability.

Figures & tables

Contrast	Late return difference $\uparrow$	Fresh return difference $\uparrow$	Native task-Q evidence
Mean target $-$ min target ( $K{=}10$ )	$+24{,}369\pm 896$	$+32{,}830\pm 4{,}177$	$-673.493\to+16.295$ ; all 3 seeds improve
Action-path gradient control	$+5{,}401\pm 2{,}821$	$+11{,}216\pm 2{,}467$	—
Full $-$ bundle, repaired gradient	$-261{,}545\pm 173{,}796$	—	—

Table 1: Frozen controlled evidence for the noisy-bus value-estimation claim. Entries are mean

\pm

seed SE in native cumulative reward; “late” averages days 425/450/475/500 and “fresh” averages the prespecified fresh evaluation window. The task-Q column reports the corresponding native task-Q bias comparison where available; negative values indicate underestimation. These are separate controls and are not pooled with the headline seed-extension table.

Algorithm	Hop.	HCheetah	Walk.	Ant
SAC	15.5	59.0	10.1	81.7
DSAC	16.6	45.9	10.6	107.7
TD3	6.8	47.6	10.7	61.4
BAC	13.4	42.2	7.5	86.5
REDQ	–	–	–	–
SAC-N	–	–	–	–

Table 2: Oracle MAE (seed 8, single-seed Q-error analysis) and final evaluation returns (5-seed mean

\pm

std over the last 25% of evaluations, seeds

\{8,16,24,32,40\}

). Bold : best value within the comparable block (Oracle MAE columns for seed-8 Q passes; return columns for 5-seed rows only). RE-SAC v1 (shared min targets, seed 8 only) achieves the lowest Q-estimation error on 3/4 envs; RE-SAC (B0,

\alpha_{\mathrm{ind}}{=}0.75

corrected from sensitivity analysis, 5 seeds) trades per-head accuracy for ensemble diversity. Variants v2, v5 are retained from earlier drafts as single-seed reference points; the headline five-seed RE-SAC number for this paper is the B0 row.

Algorithm	Oracle	Mean-head	LCB aggregator	Min-head
SAC	41.6	43.0	43.5	43.3
DSAC	45.2	46.4	46.8	46.6
TD3	31.6	33.8	34.9	34.2
BAC	37.4	38.6	39.1	38.9
RE-SAC (B0)	45.0	49.1	50.0	49.8
RE-SAC v1 †	23.3	25.0	25.8	25.7

Table 3: Non-oracle Q-error aggregators averaged over the four MuJoCo environments (seed 8 Q-data, MAE

\downarrow

). Oracle selects the best head post hoc and reports the best available aligned head prediction. Mean, LCB, and Min are deployable aggregators, but the errors here still use per-algorithm alignment. For RE-SAC, the LCB form is

\bar{Q}+\beta_{\mathrm{lcb}}\sigma_{\mathrm{JAX}}

Figure 1: Oracle Q-Error (MAE) binned by Mahalanobis rareness on four MuJoCo environments. Legend entries show overall Oracle MAE. RE-SAC v1 maintains the lowest oracle Q-estimation error across rareness bins on three of four environments, with BAC the closest pessimistic-ensemble competitor on Ant. The figure focuses on the seed-8 Q-error diagnostic algorithms from Table 2 ; older exploratory variants (v2/v5/v6b) remain available in the analysis pipeline but are excluded for legend readability.

Figure 2: Five-seed evaluation returns on MuJoCo benchmarks (8M steps). Curves show the mean over seeds $\{8,16,24,32,40\}$ ; shaded bands show one standard error. The archived RE-SAC (B0) launch family is plotted alongside the baseline configurations; B0 uses environment-specific settings.

Sweep	Setting	Ant-v2	HalfCheetah-v2
$K$ (ensemble size)	2	diverged	diverged
	5	2621 $\pm$ 5262	16356 $\pm$ 3995
	10	6829 $\pm$ 8586	16171 $\pm$ 4436
	20	7012 $\pm$ 8920	16487 $\pm$ 2803
$\lambda_{\mathrm{anc}}$ (anchor)	0	5006 $\pm$ 10036	15387 $\pm$ 2312
	0.001	3871 $\pm$ 7765	15534 $\pm$ 3098

Table 4: Sensitivity sweep results (final return, mean

\pm

std over the last 25% of evaluations and 5 seeds). Each sweep changes the listed parameter relative to its archived base launch recipe; these sweep recipes are distinct from the headline B0 family.

Controller	Reward ( $10^{6}$ )	Wait (min)	Travel (min)	$\|h_{f}-h_{b}\|$ (s)	Bunching rate
No holding	$-0.841$	3.76	10.93	127.1	13.4%
Headway-equalising	$-0.437$	3.62	12.19	59.2	0.7%

Table 8: Classical bus-holding sanity baselines on the bidirectional corridor simulator (one seeded rollout, seed 8). Reward is cumulative environment reward; operational metrics are computed from passenger and headway traces.

Figure 3: Learning curves of cumulative reward for the bus-control experiments. SAC, RE-SAC variants, and the channel ablations aggregate the available training seeds (8/32/40); shaded bands show one standard deviation across seeds. DSAC-v1 and BAC are shown as seed-8 reference curves.

Figure 4: Aligned Oracle Q-error by Mahalanobis rareness. Per-algorithm z-score alignment and ideal head selection make this a representation diagnostic, not native task-Q calibration.

Uncertainty Estimation in RL

See all 24 papers

Epistemic Robust Offline Reinforcement Learning
Apr 8, 2026Abhilash Reddy Chenreddy, Erick DelageUncertainty Estimation in RL Offline RL
GERAD, Department of Decision Sciences, HEC Montréal, Montréal, Québec H3T 2A7, Canada
Mitigating Bus Bunching with Reinforcement Learning Enhanced by Semantic Stop Embedding
Aug 10, 2026Xin Dong, Vikash V. GayahRL Control LLM-Guided RL
Department of Civil and Environmental Engineering, The Pennsylvania State University University Park, PA, 16803
Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization
Sep 23, 2026Jiaxi Wu, Tiantian Zhang, Yuxing Wang +2Reinforcement Learning Risk-Sensitive RL
Shenzhen International Graduate School, Tsinghua University Shenzhen, China

Apr 8, 2026J/K move · Enter open · S save

Epistemic Robust Offline Reinforcement Learning

Abhilash Reddy Chenreddy, Erick Delage

GERAD, Department of Decision Sciences, HEC Montréal, Montréal, Québec H3T 2A7, Canada

Uncertainty Estimation in RL Offline RL Robust RL Epistemic Uncertainty Risk-Sensitive RL

Open paper AskPDF arXiv

Offline reinforcement learning learns policies from fixed datasets without further environment interaction. A key challenge in this setting is epistemic uncertainty, arising from limited or biased data coverage, particularly when the behavior policy systematically avoids certain actions. This can lead to inaccurate value estimates and unreliable generalization. Ensemble-based methods like SAC-N mitigate this by conservatively estimating Q-values using the ensemble minimum, but they require large ensembles and often conflate epistemic with aleatoric uncertainty. To address these limitations, we propose a unified and generalizable framework that replaces discrete ensembles with compact uncertainty sets over Q-values. %We further introduce an Epinet based model that directly shapes the uncertainty sets to optimize the cumulative reward under the robust Bellman objective without relying on ensembles. We also introduce a benchmark for evaluating offline RL algorithms under risk-sensitive behavior policies, and demonstrate that our method achieves improved robustness and generalization over ensemble-based baselines across both tabular and continuous state domains.

Abstract

Uncertainty Estimation in RL

Explore similar work

Epistemic Robust Offline Reinforcement Learning

Mitigating Bus Bunching with Reinforcement Learning Enhanced by Semantic Stop Embedding

Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization