Epistemic Robust Offline Reinforcement Learning

Abhilash Reddy Chenreddy; Erick Delage

cs.LGApr 8, 2026

Epistemic Robust Offline Reinforcement Learning

Authors: Abhilash Reddy Chenreddy, Erick Delage

Organizations: GERAD, Department of Decision Sciences, HEC Montréal, Montréal, Québec H3T 2A7, Canada

Abstract

Offline reinforcement learning learns policies from fixed datasets without further environment interaction. A key challenge in this setting is epistemic uncertainty, arising from limited or biased data coverage, particularly when the behavior policy systematically avoids certain actions. This can lead to inaccurate value estimates and unreliable generalization. Ensemble-based methods like SAC-N mitigate this by conservatively estimating Q-values using the ensemble minimum, but they require large ensembles and often conflate epistemic with aleatoric uncertainty. To address these limitations, we propose a unified and generalizable framework that replaces discrete ensembles with compact uncertainty sets over Q-values. %We further introduce an Epinet based model that directly shapes the uncertainty sets to optimize the cumulative reward under the robust Bellman objective without relying on ensembles. We also introduce a benchmark for evaluating offline RL algorithms under risk-sensitive behavior policies, and demonstrate that our method achieves improved robustness and generalization over ensemble-based baselines across both tabular and continuous state domains.

Figures & tables

Table 1: Returns aggregated across

\tau\in\{0.1,0.5,0.9\}

for each dataset size. Bold indicates best method, underline the worst, when mean differences

\geq 1

Env	$\tau{=}0.1$	$\tau{=}0.5$	$\tau{=}0.9$
CartPole	$95\pm 8$ (1)	$93\pm 14$ (2)	$92\pm 14$ (3)
LunarLander	$103\pm 7$ (1)	$99\pm 5$ (2)	$99\pm 5$ (2)
MR	$93\pm 1$ (3)	$95\pm 4$ (1)	$94\pm 3$ (2)
RS	$87\pm 15$ (2)	$87\pm 17$ (1)	$84\pm 22$ (3)

Table 2: Agg. performance of Ell_0.9-N across environments with mean

\pm

std and within-environment rank (1 = best).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$\tau$	6	7	8	9	10
0.1	0	0	0	1	1
0.5	0	1	1	1	1
0.9	1	1	1	1	1

Appendix

Table 3: Optimal actions for each state under different expectile levels

\tau

. Action 0 corresponds to progressing forward; Action 1 corresponds to jumping to state 1 with -100 reward.

Panel of Figure 1 — Figure 1: State visitation frequency distributions under different expectile policies.

Panel of Figure 2 — Figure 2: (a)–(c): Uncertainty sets and worst-case policy evaluations for states 0, 5, and 10 in the machine replacement example at epoch 1. Each subplot illustrates the distribution of ensemble Q-values along with the corresponding box, convex hull, and ellipsoidal uncertainty sets. Markers X indicate the worst-case Q-value $q^{*}$ under different policies $\pi$ .

\displaystyle y(r,s^{\prime})\leftarrow

\displaystyle r+\gamma\bigg(\mathop{\rm min}_{q\in\mathcal{U}_{\theta^{\prime}}(s^{\prime})}\langle\pi_{\phi}(\cdot\mid s^{\prime}),\,q\rangle-\alpha\,\mathbb{E}_{a^{\prime}\sim\pi_{\phi}}[\log\pi_{\phi}(a^{\prime}\mid s^{\prime})]\bigg)

Appendix

Algorithm 1 Epistemic Robust SAC Training

\mathcal{Q}(s)\leftarrow\{\mathfrak{q}_{\theta}(s,\cdot,\tilde{z}_{i})\}_{i=1}^{N},\qquad\mathcal{Q}(s^{\prime})\leftarrow\{\mathfrak{q}_{\theta^{\prime}}(s^{\prime},\cdot,\tilde{z}_{i})\}_{i=1}^{N}.

Appendix

Algorithm 2 Sample-based Epistemic Robust SAC with Box (ERSAC-B) and Convex Hull (ERSAC-CH) Sets

\hat{\mu}(s)\leftarrow\frac{1}{N}\sum_{i=1}^{N}\mathfrak{q}_{\theta}(s,\cdot,\tilde{z}_{i})

Appendix

Algorithm 3 Sample-based Epistemic Robust SAC with Ellipsoidal Uncertainty (ERSAC-E)

\hat{\mu}(s^{\prime})\leftarrow\mu_{\theta^{\prime}_{\mu}}(s^{\prime})

Appendix

Algorithm 4 Sample-based ERSAC with Ellipsoidal Uncertainty using Epinet (ERSAC-E-Epi)

\displaystyle y\leftarrow

\displaystyle\mathop{\rm sup}\bigg\{z:\ \mathbb{E}_{s^{\prime}\sim\hat{p}_{{N_{s}}}(\cdot\mid s,a)}\bigg[\left|\tau-\mathbb{I}\left(z<r+\gamma\mathop{\rm max}_{a^{\prime}}Q_{\theta^{\prime}}(s^{\prime},a^{\prime})\right)\right|\cdot\left(z-r-\gamma\mathop{\rm max}_{a^{\prime}}Q_{\theta^{\prime}}(s^{\prime},a^{\prime})\right)\bigg]\leq 0\bigg\}

Appendix

Algorithm 5 Offline Data Generation via Dynamic Expectile Risk Policies

Env	DS	$\tau$	SAC-N	CH-N	Ell-N	Ell_0.9-N	Beh. Policy
Machine Replacement	10 $\times$	0.1	$80\pm 3$	$85\pm 2$	$87\pm 1$	$\mathbf{88\pm 2}$	$86\pm 3$
	100 $\times$	0.1	$97\pm 1$	$97\pm 1$	$95\pm 2$	$96\pm 2$	$86\pm 3$
	1000 $\times$	0.1	$98\pm 2$	$98\pm 2$	$96\pm 2$	$96\pm 1$	$86\pm 3$
	10 $\times$	0.5	$87\pm 2$	$88\pm 2$	$90\pm 2$	$\mathbf{91\pm 2}$	$100\pm 0$
	100 $\times$	0.5	$97\pm 1$	$\mathbf{98\pm 1}$	$92\pm 2$	$94\pm 2$	$100\pm 0$
	1000 $\times$	0.5	$98\pm 2$	$98\pm 2$	$98\pm 2$	$\mathbf{99\pm 0}$	$100\pm 0$

Appendix

Table 4: Normalized returns with 90% confidence interval achieved by SAC-N, CH-N, Ell-N, and Ell_0.9-N across dataset sizes

\{10\times,100\times,1000\times\}

and behavior policy risk levels

\tau\in\{0.1,0.5,0.9\}

in the Machine Replacement and RiverSwim environments. Scores are computed over 10 evaluation seeds and normalized relative to the random and optimal policy baselines. Bold and underline highlight respectively the best and worst performing method when the margin is larger or equal to one. The final column reports the return of the behavior policy used to generate the offline data.

Env	DS	$\tau$	SAC-N	CH-N	Ell_0.9-N	Ell-Epi	Ell-Epi ∗	Beh. Policy
CartPole	1k	0.1	$84\pm 3$	$81\pm 2$	$\mathbf{86\pm 1}$	$84\pm 1$	$85\pm 2$	$86\pm 2$
	10k	0.1	$92\pm 2$	$94\pm 2$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$86\pm 2$
	100k	0.1	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$86\pm 2$
	1k	0.5	$70\pm 2$	$72\pm 1$	$\mathbf{73\pm 3}$	$72\pm 2$	$71\pm 2$	$100\pm 0$
	10k	0.5	$97\pm 2$	$99\pm 1$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$
	100k	0.5	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$	$100\pm 0$

Appendix

Table 5: Normalized returns with 90% confidence intervals achieved by the five algorithms across dataset sizes

\{1\text{k},10\text{k},100\text{k}\}

and behavior-policy risk levels

\tau\in\{0.1,0.5,0.9\}

in CartPole and LunarLander. Scores are averaged over 10 evaluation seeds and normalized against random and optimal baselines. Bold and underline highlight respectively the best and worst performing method when the margin is larger or equal to one.

Model	SAC-N	CH-N	Ell_0.9-N	Ell-Epi	Ell-Epi ∗
Runtime (s/epoch)	0.35	0.42	0.56	0.60	0.10

Appendix

Table 6: Runtime per training epoch for each model in LunarLander with 100,000 offline transitions and

\tau=0.5

, averaged over 10 seeds.

Panel of Figure 3 — Figure 3: Policy entropy during training across B_N, CH_N, Ell_0.9, and Ell_Epi models in the CartPole and LunarLander environments. Entropy is computed per epoch and averaged over 10 evaluation seeds. Lower entropy indicates more confident, deterministic policies, while higher entropy reflects greater stochasticity in policies.

Env	SAC-N	CQL	IQL	BRAC-BCQ	ERSAC-CH-N	ERSAC-Ell-N	ERSAC-Ell-Epi ∗
Breakout	$58\pm 6$	$\mathbf{71\pm 5}$	$68\pm 4$	$35\pm 7$	$62\pm 5$	$64\pm 5$	$66\pm 4$
Pong	$78\pm 8$	$\mathbf{86\pm 6}$	$84\pm 7$	$55\pm 9$	$80\pm 6$	$82\pm 6$	$83\pm 5$
Q*bert	$54\pm 7$	$63\pm 6$	$\mathbf{66\pm 6}$	$29\pm 8$	$60\pm 6$	$62\pm 5$	$64\pm 5$
Seaquest	$33\pm 6$	$47\pm 5$	$45\pm 5$	$18\pm 6$	$42\pm 6$	$46\pm 5$	$\mathbf{49\pm 5}$
Hero	$38\pm 7$	$52\pm 6$	$56\pm 6$	$22\pm 7$	$50\pm 6$	$55\pm 5$	$\mathbf{58\pm 5}$

Appendix

Table 7: Average episodic returns on Atari 2600 environments using fixed offline datasets from Minari. Results are reported as mean

\pm

standard deviation over three random seeds. ERSAC variants are compared against standard offline RL baselines in high-dimensional visual domains.

Explore similar work

Mar 10, 2026

Robust Regularized Policy Iteration under Transition Uncertainty

Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distribution state-action pairs where value estimates and learned dynamics are unreliable. To address policy-induced extrapolation and transition uncertainty in a unified framework, we formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set and optimizing the policy against the worst-case dynamics. We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate and derives an efficient policy iteration procedure based on a robust regularized Bellman operator. We provide theoretical guarantees by showing that the proposed operator is a

γ

-contraction and that iteratively updating the surrogate yields monotonic improvement of the original robust objective with convergence. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines including percentile-based methods on the majority of environments while remaining competitive on the rest. Moreover, RRPI exhibits robust performance by aligning lower

Q

-values with high epistemic uncertainty, which prevents the policy from executing unreliable out-of-distribution actions.

Hongqiang Lin, Zhenghui Fu, Weihao Tang +4

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an, China · School of Software Technology, Zhejiang University, Hangzhou, China +2

Offline RL Robust RL Distributionally Robust RL

Ask PDF

Jul 21, 2026

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation

Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset, but its performance is fundamentally limited by dataset coverage. Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training. However, existing methods still face two key challenges: selecting informative preference queries and effectively exploiting the collected feedback. Current approaches typically rely only on the distance between policy actions and dataset actions for query selection, while enforcing fixed constraints that keep the policy close to queried preferences. Such strategies often lead to unstable policy updates and integrate poorly with value regularization. To address these limitations, we propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation. Specifically, we employ a Morse network to estimate the uncertainty of policy actions with respect to the offline dataset. Based on this uncertainty, we introduce a conservative query strategy that selectively queries actions near the dataset to preserve Bellman-update stability, together with an uncertainty-aware adaptive regularization scheme that dynamically adjusts data-level constraints during policy optimization. We integrate our framework with CQL and evaluate it extensively on the D4RL benchmark. Experimental results demonstrate superior or competitive performance across a wide range of tasks.

Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun Huang

Nanjing University of Aeronautics and Astronautics, Nanjing, China

Reinforcement Learning Offline RL Preference-Based RL

Ask PDF

Feb 27, 2026

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning process introduces approximation errors, causing the error accumulation and considerable overestimation. In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions. Specifically, we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function. The pessimistic auxiliary strategy exhibits a relatively high value and low uncertainty in the vicinity of the learned policy, avoiding the learned policy sampling high-value actions with potentially high errors during the learning process. Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improve the efficacy of other offline RL approaches.

Fan Zhang, Baoru Huang, Xin Zhang

Department of Mathematics, The Hong Kong University of Science and Technology, Hong Kong, China. · University of Liverpool, UK · Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology, Hong Kong, China.

Offline RL Policy Learning

Ask PDF