Learning Optimal and Sample-Efficient Decision Policies with Guarantees

Daqian Shao

cs.LGFeb 20, 2026

Learning Optimal and Sample-Efficient Decision Policies with Guarantees

Authors: Daqian Shao

Organizations: Wolfson College University of Oxford

Abstract

The paradigm of decision-making has been revolutionised by reinforcement learning and deep learning. Although this has led to significant progress in domains such as robotics, healthcare, and finance, the use of RL in practice is challenging, particularly when learning decision policies in high-stakes applications that may require guarantees. Traditional RL algorithms rely on a large number of online interactions with the environment, which is problematic in scenarios where online interactions are costly, dangerous, or infeasible. However, learning from offline datasets is hindered by the presence of hidden confounders. Such confounders can cause spurious correlations in the dataset and can mislead the agent into taking suboptimal or adversarial actions. Firstly, we address the problem of learning from offline datasets in the presence of hidden confounders. We work with instrumental variables (IVs) to identify the causal effect, which is an instance of a conditional moment restrictions (CMR) problem. Inspired by double/debiased machine learning, we derive a sample-efficient algorithm for solving CMR problems with convergence and optimality guarantees, which outperforms state-of-the-art algorithms. Secondly, we relax the conditions on the hidden confounders in the setting of (offline) imitation learning, and adapt our CMR estimator to derive an algorithm that can learn effective imitator policies with convergence rate guarantees. Finally, we consider the problem of learning high-level objectives expressed in linear temporal logic (LTL) and develop a provably optimal learning algorithm that improves sample efficiency over existing methods. Through evaluation on reinforcement learning benchmarks and synthetic and semi-synthetic datasets, we demonstrate the usefulness of the methods developed in this thesis in real-world decision making.

Figures & tables

Figure 2.1 : An overview of the types of problems considered in policy learning, categorised by the learning objective and the data source from which the policy learns. Problems considered in this thesis are highlighted with the corresponding chapters.

Figure 2.2 : The causal graph of outcome $Y$ , treatment $A$ and hidden confounder $U$ .

Figure 2.3 : The causal graph of outcome $Y$ , treatment $A$ , hidden confounder $U$ and an instrumental variable $Z$ .

Figure 2.4 : The causal graph of outcome $Y$ , treatment $A$ , hidden confounder $U$ and proxies $V$ and $W$ .

Figure 4.1 : The causal graph of the contextual IV setting, where $R=f_{r}(C,A)+\epsilon$ and $Z$ is an instrumental variable that affects $R$ only through $A$ .

Panel of Figure 4 — Figure 4.2 : Results on the ticket demand dataset with low-dimensional context.

Figure 5.1 : A causal graph of MDPs with hidden confounders $u_{t}=(u^{o}_{t},u^{\epsilon}_{t})$ . The black dotted lines represent the causal effect of the expert-observable confounder $u^{o}_{t}$ , which directly affects the expert action $a_{t}$ . It also directly affects $s_{t+1}$ and $r_{t}$ . The red dotted lines represent the causal effect of the expert-unobservable $u^{\epsilon}_{t}$ , which acts as confounding noise and directly affects the states and actions. $u^{\epsilon}_{t}$ does not directly affect $r_{t}$ (following Swamy et al. (2022a) ) because the expert policy does not take $u^{\epsilon}_{t}$ into account, and letting $u^{\epsilon}_{t}$ directly affect $r_{t}$ would only add noise to the expected return.

Panel of Figure 5 — Figure 5.2 : Plane Ticket Environment ( Example 5.1 ): On the left, the MSE between the learnt policy and the expert. On the right, the average reward of our approach and baselines.

Panel of Figure 6 — Figure 6.1 : An LDBA for “ FG a & G !c” (left), and a probabilistic gate MDP (right) motivating the $K$ counter (see Example 6.5 ).

Figure 6.3 : Sensitivity analysis on key hyperparameters.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : A graph of the nonlinear function $\psi(t)$ in the ticket demand dataset for IV regression.

Figure A.2 : An example of dSprites image, which is used as the treatment $A$ in PCL experiments. Its scale, rotation, $x$ position and $y$ position are randomly generated.

Table A.1 : Network architecture for DML-CMR and CE-DML-CMR for the ticket demand low-dimensional dataset for IV regression. For the input layer, we provide the input variables. For mixture of Gaussians output, we report the number of components. The dropout rate is given in the main text.

Layer Type	Configuration
Input	$28\times 28$
Conv + ReLU	$3\times 3\times 32$ , s:1, p:0
Max Pooling	$2\times 2$ , s:2
Dropout	-
Conv + ReLU	$3\times 3\times 64$ , s:1, p:0
Max Pooling	$2\times 2$ , s:2

Appendix

Table A.2 : Network architecture of the feature extractor used for the ticket demand dataset with MNIST for IV regression. For each convolution layer, we list the kernel size, input dimension and output dimension, where s stands for stride and p stands for padding. For max-pooling, we provide the size of the kernel. The dropout rate here is set to 0.3. We denote this feature extractor as ImageFeature .

Table A.3 : Network architecture for DML-CMR and CE-DML-CMR for the ticket demand dataset with MNIST for IV regression. For the input layer, we provide the input variables. For a mixture of Gaussians output, we report the number of components. The dropout rate is given in the main text.

Table A.4 : Network architecture for DML-CMR and CE-DML-CMR for the ticket demand dataset for PCL. For the input layer, we provide the input variables. For mixture of Gaussians output, we report the number of components. The dropout rate is given in the main text.

Layer Type	Configuration
Input	$64\times 64$
Conv + ReLU	$5\times 5\times 64$ , s:1, p:0
Max Pooling	$2\times 2$ , s:2
Dropout	-
Conv + ReLU	$5\times 5\times 128$ , s:1, p:0
Max Pooling	$2\times 2$ , s:2

Appendix

Table A.5 : Network architecture of the feature extractor used for the dSprites image dataset for PCL. For each convolution layer, we list the kernel size, input dimension and output dimension, where s stands for stride and p stands for padding. For max-pooling, we provide the size of the kernel. The dropout rate here is set to 0.2.

Figure A.5 : Comparing the average reward obtained by policies learnt using offline bandit algorithms that do not take IVs into account with a random policy on the ticket demand dataset with low-dimensional context.

IV Strength	1.0	0.8	0.6	0.4	0.2	0.01
DML-CMR	0.0676(0.0116)	0.0984(0.0161)	0.1295(0.0168)	0.1859(0.0376)	0.2899(0.0494)	0.4872(0.1295)
CE-DML-CMR	0.0765(0.0119)	0.1064(0.0120)	0.1514(0.0203)	0.2070(0.0329)	0.3194(0.0572)	0.5302(0.1625)
DeepIV	0.1213(0.0209)	0.2039(0.0269)	0.3051(0.0415)	0.4476(0.0656)	0.6891(0.1210)	0.9293(0.2382)
DFIV	0.1124(0.0481)	0.1586(0.0320)	0.3080(0.1907)	0.8117(0.2779)	0.9622(0.3892)	1.6503(0.6845)
DeepGMM	0.2699(0.0522)	0.3330(0.1171)	0.4762(0.1056)	0.8666(0.2248)	1.0056(0.4334)	2.0218(0.6555)
KIV	0.2312(0.0272)	0.3149(0.0218)	0.4275(0.0368)	0.6646(0.0538)	0.8099(0.0657)	1.226(0.1014)

Appendix

Table A.6 : Results for the low-dimensional ticket demand dataset when the IV is weakly correlated with the action.

IV Strength	Dataset Size	DNN (results in the paper)	Random Forests	Gradient Boosting
DML-CMR	2000	0.1308(0.0206)	0.1689(0.0172)	0.1301(0.0112)
CE-DML-CMR	2000	0.1410(0.0246)	0.1733(0.0198)	0.1329(0.0125)
DeepIV	2000	0.2388(0.0438)	0.2642(0.0261)	0.2052(0.0232)
DML-CMR	5000	0.0676(0.0129)	0.1067(0.0131)	0.0632(0.0107)
CE-DML-CMR	5000	0.0765(0.0119)	0.1154(0.0138)	0.0699(0.0069)
DeepIV	5000	0.1213(0.0209)	0.1626(0.0128)	0.1020(0.0091)

Appendix

Table A.7 : Results for the low-dimensional ticket demand dataset using tree-based estimators compared to DNN estimators.

Learning Rate	Weight Decay	Dropout	DNN Width	DML-CMR	CE-DML-CMR
0.0002	0.001	0.1	128	0.0676(0.0129)	0.0765(0.0119)
0.0005				0.0752(0.0122)	0.0897(0.0196)
0.0001				0.0703(0.0195)	0.0794(0.0201)
	0.0005			0.0794(0.0185)	0.0823(0.0149)
	0.005			0.0765(0.0135)	0.0809(0.0159)
	0.01			0.0820(0.0162)	0.0865(0.0174)

Appendix

Table A.8 : Results for the low-dimensional ticket demand dataset for a range of hyperparameter values. The default hyperparameters in this case are: learning rate=0.0002, weight decay=0.001, dropout=0.1 and DNN width 128. The bold results are the best performing hyperparameters.

Learning Rate	Weight Decay	Dropout	CNN Channels	DML-CMR	CE-DML-CMR
0.001	0.05	0.2	64	0.3513(0.0125)	0.3808(0.0150)
0.0005				0.4063(0.0129)	0.5008(0.0369)
0.002				0.3659(0.0219)	0.4133(0.0267)
0.005				0.3377(0.0218)	0.3555(0.0202)
	0.01			0.3935(0.0176)	0.4461(0.0478)
	0.02			0.3595(0.03013)	0.3851(0.0293)

Appendix

Table A.9 : Results for the high-dimensional ticket demand dataset for a range of hyperparameter values. The default hyperparameters in this case are: learning rate 0.001, weight decay=0.05, dropout=0.2 and 64 CNN channels. The bold results are the best performing hyperparameters.

Table B.1 : Network architecture for DML-IL. For the mixture of Gaussians output, we report the number of components. No dropout is used.

Figure C.0 : The optimal policy for the example probabilistic gate MDP task in Figure 6.1

Panel of Figure 33 — Figure C.1 : MDP environments used in the experiments.

Explore similar work

Jul 28, 2026

Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.

Zeyu Bian, Ying Zhou, Yifan Cui

Department of Statistics, Florida State University · Department of Statistics, University of Connecticut · Center for Data Science, Zhejiang University

Causal Effect Estimation Off-Policy Evaluation Offline RL

Ask PDF

Mar 31, 2026

Offline Constrained RLHF with Multiple Preference Oracles

We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.

Brenden Latham, Mehrdad Moharrami

Computer Science Department, University of Iowa, Iowa City, IA, USA

Constrained RL Reinforcement Learning RL from Human Feedback Preference-Based RL Offline RL

Ask PDF

Jul 21, 2025

RAD: Retrieval High-quality Demonstrations to Enhance Decision-making

Offline reinforcement learning (RL) learns policies from fixed datasets, thereby avoiding costly or unsafe environment interactions. However, its reliance on finite static datasets inherently restricts the ability to generalize beyond the training distribution. Prior solutions based on synthetic data augmentation often fail to generalize to unseen scenarios in the (augmented) dataset. To address these challenges, we propose Retrieval High-quAlity Demonstrations (RAD) for decision-making, which innovatively introduces a retrieval mechanism into offline RL. Specifically, RAD retrieves high-return and reachable states from the offline dataset as target states, and leverages a generative model to generate sub-trajectories conditioned on these targets for planning. Since the targets are high-return states, once the agent reaches such a target, it can continue to obtain high returns by following the associated high-return actions, thereby improving policy generalization. Extensive experiments confirm that RAD achieves competitive or superior performance compared to baselines across diverse benchmarks, validating its effectiveness. Our code is available at https://anonymous.4open.science/r/RAD_0925_1-690E.

Lu Guo, Yixiang Shan, Zhengbang Zhu +5

School of Artificial Intelligence, Jilin University, Changchun, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China

Offline RL Generalization in RL

Ask PDF