Epistemic Robust Offline Reinforcement Learning

Abhilash Reddy Chenreddy; Erick Delage

cs.LGApr 8, 2026

Epistemic Robust Offline Reinforcement Learning

Authors: Abhilash Reddy Chenreddy, Erick Delage

Organizations: GERAD, Department of Decision Sciences, HEC Montréal, Montréal, Québec H3T 2A7, Canada

Abstract

Offline reinforcement learning learns policies from fixed datasets without further environment interaction. A key challenge in this setting is epistemic uncertainty, arising from limited or biased data coverage, particularly when the behavior policy systematically avoids certain actions. This can lead to inaccurate value estimates and unreliable generalization. Ensemble-based methods like SAC-N mitigate this by conservatively estimating Q-values using the ensemble minimum, but they require large ensembles and often conflate epistemic with aleatoric uncertainty. To address these limitations, we propose a unified and generalizable framework that replaces discrete ensembles with compact uncertainty sets over Q-values. %We further introduce an Epinet based model that directly shapes the uncertainty sets to optimize the cumulative reward under the robust Bellman objective without relying on ensembles. We also introduce a benchmark for evaluating offline RL algorithms under risk-sensitive behavior policies, and demonstrate that our method achieves improved robustness and generalization over ensemble-based baselines across both tabular and continuous state domains.

Figures & tables

Table 1: Returns aggregated across

\tau\in\{0.1,0.5,0.9\}

for each dataset size. Bold indicates best method, underline the worst, when mean differences

\geq 1

Env	$\tau{=}0.1$	$\tau{=}0.5$	$\tau{=}0.9$
CartPole	$95\pm 8$ (1)	$93\pm 14$ (2)	$92\pm 14$ (3)
LunarLander	$103\pm 7$ (1)	$99\pm 5$ (2)	$99\pm 5$ (2)
MR	$93\pm 1$ (3)	$95\pm 4$ (1)	$94\pm 3$ (2)
RS	$87\pm 15$ (2)	$87\pm 17$ (1)	$84\pm 22$ (3)

Table 2: Agg. performance of Ell_0.9-N across environments with mean

\pm

std and within-environment rank (1 = best).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$\tau$	6	7	8	9	10
0.1	0	0	0	1	1
0.5	0	1	1	1	1
0.9	1	1	1	1	1

Appendix

Table 3: Optimal actions for each state under different expectile levels

\tau

. Action 0 corresponds to progressing forward; Action 1 corresponds to jumping to state 1 with -100 reward.

Panel of Figure 1 — Figure 1: State visitation frequency distributions under different expectile policies.

Panel of Figure 2 — Figure 2: (a)–(c): Uncertainty sets and worst-case policy evaluations for states 0, 5, and 10 in the machine replacement example at epoch 1. Each subplot illustrates the distribution of ensemble Q-values along with the corresponding box, convex hull, and ellipsoidal uncertainty sets. Markers X indicate the worst-case Q-value $q^{*}$ under different policies $\pi$ .

\displaystyle y(r,s^{\prime})\leftarrow

\displaystyle r+\gamma\bigg(\mathop{\rm min}_{q\in\mathcal{U}_{\theta^{\prime}}(s^{\prime})}\langle\pi_{\phi}(\cdot\mid s^{\prime}),\,q\rangle-\alpha\,\mathbb{E}_{a^{\prime}\sim\pi_{\phi}}[\log\pi_{\phi}(a^{\prime}\mid s^{\prime})]\bigg)

Appendix

Algorithm 1 Epistemic Robust SAC Training

\mathcal{Q}(s)\leftarrow\{\mathfrak{q}_{\theta}(s,\cdot,\tilde{z}_{i})\}_{i=1}^{N},\qquad\mathcal{Q}(s^{\prime})\leftarrow\{\mathfrak{q}_{\theta^{\prime}}(s^{\prime},\cdot,\tilde{z}_{i})\}_{i=1}^{N}.

Appendix

Algorithm 2 Sample-based Epistemic Robust SAC with Box (ERSAC-B) and Convex Hull (ERSAC-CH) Sets

\hat{\mu}(s)\leftarrow\frac{1}{N}\sum_{i=1}^{N}\mathfrak{q}_{\theta}(s,\cdot,\tilde{z}_{i})

Appendix

Algorithm 3 Sample-based Epistemic Robust SAC with Ellipsoidal Uncertainty (ERSAC-E)

\hat{\mu}(s^{\prime})\leftarrow\mu_{\theta^{\prime}_{\mu}}(s^{\prime})

Appendix

Algorithm 4 Sample-based ERSAC with Ellipsoidal Uncertainty using Epinet (ERSAC-E-Epi)

\displaystyle y\leftarrow

\displaystyle\mathop{\rm sup}\bigg\{z:\ \mathbb{E}_{s^{\prime}\sim\hat{p}_{{N_{s}}}(\cdot\mid s,a)}\bigg[\left|\tau-\mathbb{I}\left(z<r+\gamma\mathop{\rm max}_{a^{\prime}}Q_{\theta^{\prime}}(s^{\prime},a^{\prime})\right)\right|\cdot\left(z-r-\gamma\mathop{\rm max}_{a^{\prime}}Q_{\theta^{\prime}}(s^{\prime},a^{\prime})\right)\bigg]\leq 0\bigg\}

Appendix

Algorithm 5 Offline Data Generation via Dynamic Expectile Risk Policies

Env	DS	$\tau$	SAC-N	CH-N	Ell-N	Ell_0.9-N	Beh. Policy
Machine Replacement	10 $\times$	0.1	$80\pm 3$	$85\pm 2$	$87\pm 1$	$\mathbf{88\pm 2}$	$86\pm 3$
	100 $\times$	0.1	$97\pm 1$	$97\pm 1$	$95\pm 2$	$96\pm 2$	$86\pm 3$
	1000 $\times$	0.1	$98\pm 2$	$98\pm 2$	$96\pm 2$	$96\pm 1$	$86\pm 3$
	10 $\times$	0.5	$87\pm 2$	$88\pm 2$	$90\pm 2$	$\mathbf{91\pm 2}$	$100\pm 0$
	100 $\times$	0.5	$97\pm 1$	$\mathbf{98\pm 1}$	$92\pm 2$	$94\pm 2$	$100\pm 0$
	1000 $\times$	0.5	$98\pm 2$	$98\pm 2$	$98\pm 2$	$\mathbf{99\pm 0}$	$100\pm 0$

Appendix

Table 4: Normalized returns with 90% confidence interval achieved by SAC-N, CH-N, Ell-N, and Ell_0.9-N across dataset sizes

\{10\times,100\times,1000\times\}

and behavior policy risk levels

\tau\in\{0.1,0.5,0.9\}

in the Machine Replacement and RiverSwim environments. Scores are computed over 10 evaluation seeds and normalized relative to the random and optimal policy baselines. Bold and underline highlight respectively the best and worst performing method when the margin is larger or equal to one. The final column reports the return of the behavior policy used to generate the offline data.

Env	DS	$\tau$	SAC-N	CH-N	Ell_0.9-N	Ell-Epi	Ell-Epi ∗	Beh. Policy
CartPole	1k	0.1	$84\pm 3$	$81\pm 2$	$\mathbf{86\pm 1}$	$84\pm 1$	$85\pm 2$	$86\pm 2$
	10k	0.1	$92\pm 2$	$94\pm 2$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$86\pm 2$
	100k	0.1	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$86\pm 2$
	1k	0.5	$70\pm 2$	$72\pm 1$	$\mathbf{73\pm 3}$	$72\pm 2$	$71\pm 2$	$100\pm 0$
	10k	0.5	$97\pm 2$	$99\pm 1$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$
	100k	0.5	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$

Appendix

Table 5: Normalized returns with 90% confidence intervals achieved by the five algorithms across dataset sizes

\{1\text{k},10\text{k},100\text{k}\}

and behavior-policy risk levels

\tau\in\{0.1,0.5,0.9\}

in CartPole and LunarLander. Scores are averaged over 10 evaluation seeds and normalized against random and optimal baselines. Bold and underline highlight respectively the best and worst performing method when the margin is larger or equal to one.

Model	SAC-N	CH-N	Ell_0.9-N	Ell-Epi	Ell-Epi ∗
Runtime (s/epoch)	0.35	0.42	0.56	0.60	0.10

Appendix

Table 6: Runtime per training epoch for each model in LunarLander with 100,000 offline transitions and

\tau=0.5

, averaged over 10 seeds.

Panel of Figure 3 — Figure 3: Policy entropy during training across B_N, CH_N, Ell_0.9, and Ell_Epi models in the CartPole and LunarLander environments. Entropy is computed per epoch and averaged over 10 evaluation seeds. Lower entropy indicates more confident, deterministic policies, while higher entropy reflects greater stochasticity in policies.

Env	SAC-N	CQL	IQL	BRAC-BCQ	ERSAC-CH-N	ERSAC-Ell-N	ERSAC-Ell-Epi ∗
Breakout	$58\pm 6$	$\mathbf{71\pm 5}$	$68\pm 4$	$35\pm 7$	$62\pm 5$	$64\pm 5$	$66\pm 4$
Pong	$78\pm 8$	$\mathbf{86\pm 6}$	$84\pm 7$	$55\pm 9$	$80\pm 6$	$82\pm 6$	$83\pm 5$
Q*bert	$54\pm 7$	$63\pm 6$	$\mathbf{66\pm 6}$	$29\pm 8$	$60\pm 6$	$62\pm 5$	$64\pm 5$
Seaquest	$33\pm 6$	$47\pm 5$	$45\pm 5$	$18\pm 6$	$42\pm 6$	$46\pm 5$	$\mathbf{49\pm 5}$
Hero	$38\pm 7$	$52\pm 6$	$56\pm 6$	$22\pm 7$	$50\pm 6$	$55\pm 5$	$\mathbf{58\pm 5}$

Appendix

Table 7: Average episodic returns on Atari 2600 environments using fixed offline datasets from Minari. Results are reported as mean

\pm

standard deviation over three random seeds. ERSAC variants are compared against standard offline RL baselines in high-dimensional visual domains.

Explore similar work

Robust Regularized Policy Iteration under Transition Uncertainty
Mar 10, 2026Hongqiang Lin, Zhenghui Fu, Weihao Tang +4Offline RL Robust RL
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an, China · School of Software Technology, Zhejiang University, Hangzhou, China +2
Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
Jul 21, 2026Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun HuangReinforcement Learning Offline RL
Nanjing University of Aeronautics and Astronautics, Nanjing, China
Pessimistic Auxiliary Policy for Offline Reinforcement Learning
Feb 27, 2026Fan Zhang, Baoru Huang, Xin ZhangOffline RL Policy Learning
Department of Mathematics, The Hong Kong University of Science and Technology, Hong Kong, China. · University of Liverpool, UK · Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology, Hong Kong, China.

Mar 10, 2026J/K move · Enter open · S save

Robust Regularized Policy Iteration under Transition Uncertainty

Hongqiang Lin, Zhenghui Fu, Weihao Tang, Pengfei Wang+3

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an, China · School of Software Technology, Zhejiang University, Hangzhou, China+2

Offline RL Robust RL Distributionally Robust RL

Open paper AskPDF arXiv

Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distribution state-action pairs where value estimates and learned dynamics are unreliable. To address policy-induced extrapolation and transition uncertainty in a unified framework, we formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set and optimizing the policy against the worst-case dynamics. We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate and derives an efficient policy iteration procedure based on a robust regularized Bellman operator. We provide theoretical guarantees by showing that the proposed operator is a

γ

-contraction and that iteratively updating the surrogate yields monotonic improvement of the original robust objective with convergence. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines including percentile-based methods on the majority of environments while remaining competitive on the rest. Moreover, RRPI exhibits robust performance by aligning lower

Q

-values with high epistemic uncertainty, which prevents the policy from executing unreliable out-of-distribution actions.

Abstract

Explore similar work

Robust Regularized Policy Iteration under Transition Uncertainty

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation

Pessimistic Auxiliary Policy for Offline Reinforcement Learning