Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation

Raphael Kim; Michele Santacatterina; Ramin Zabih; Keith Barnatchez

stat.MLOct 29, 2026

Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation

Authors: Raphael Kim, Michele Santacatterina, Ramin Zabih, Keith Barnatchez

Organizations: Cornell Tech, Cornell University · Division of Biostatistics, NYU Grossman School of Medicine · Department of Biostatistics, Johns Hopkins University

Abstract

Reinforcement learning (RL) is increasingly used to train robust, adaptable systems across healthcare, e-commerce, and robotics. In these settings, legal, ethical, or logistical concerns may preclude certain key variables that drive decisions from being collected during deployment time. This gives rise to a setting known as runtime confounding: the logged data hold full observability, but deployment time lacks knowledge of certain variables, known as runtime confounders. Unlike single timepoint and non-adaptive settings, runtime confounding under reinforcement learning has the potential to propagate over time, as earlier decisions affect future states, observations, and action assignments, leading to biased learning. We address these problems with three contributions: we (1) develop policy learning algorithms under runtime-confounding, (2) develop semi-parametric efficient off-policy estimation procedures, and (3) provide diagnostics for understanding the price of runtime confounding.

Figures & tables

Panel of Figure 1 — Figure 1: Problem setting. $U_{t}$ is observed in the historical data, but unmeasured during deployment and in turn unavailable to the target policy.

Off-Policy Evaluation

Off-policy evaluation estimates a policy’s value from data collected by another policy, using weighting, models, and uncertainty-aware methods.

See all 75 papers

Learning Optimal and Sample-Efficient Decision Policies with Guarantees
Feb 20, 2026Daqian ShaoReinforcement Learning Sample-Efficient RL
Wolfson College University of Oxford
Transportable Offline Reinforcement Learning in Presence of Unmeasured Confounding
Oct 28, 2026Junyi Liao, Hangqi Ren, Zhuoran YangOffline RL Causal Inference with Unobserved Confounding
Department of Electrical and Computer Engineering, Duke University · Department of Electrical and Computer Engineering, Vanderbilt University · Department of Statistics and Data Science, Yale University
PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data
Jul 26, 2025Aishwarya Mandyam, Jason Meng, Ge Gao +4Off-Policy Evaluation Healthcare
Stanford University · Gladstone Institutes · Stanford University[cs.LG]

Feb 20, 2026J/K move · Enter open · S save

Learning Optimal and Sample-Efficient Decision Policies with Guarantees

Daqian Shao

Wolfson College University of Oxford

Reinforcement Learning Sample-Efficient RL Offline RL Causal RL

Open paper AskPDF arXiv

The paradigm of decision-making has been revolutionised by reinforcement learning and deep learning. Although this has led to significant progress in domains such as robotics, healthcare, and finance, the use of RL in practice is challenging, particularly when learning decision policies in high-stakes applications that may require guarantees. Traditional RL algorithms rely on a large number of online interactions with the environment, which is problematic in scenarios where online interactions are costly, dangerous, or infeasible. However, learning from offline datasets is hindered by the presence of hidden confounders. Such confounders can cause spurious correlations in the dataset and can mislead the agent into taking suboptimal or adversarial actions. Firstly, we address the problem of learning from offline datasets in the presence of hidden confounders. We work with instrumental variables (IVs) to identify the causal effect, which is an instance of a conditional moment restrictions (CMR) problem. Inspired by double/debiased machine learning, we derive a sample-efficient algorithm for solving CMR problems with convergence and optimality guarantees, which outperforms state-of-the-art algorithms. Secondly, we relax the conditions on the hidden confounders in the setting of (offline) imitation learning, and adapt our CMR estimator to derive an algorithm that can learn effective imitator policies with convergence rate guarantees. Finally, we consider the problem of learning high-level objectives expressed in linear temporal logic (LTL) and develop a provably optimal learning algorithm that improves sample efficiency over existing methods. Through evaluation on reinforcement learning benchmarks and synthetic and semi-synthetic datasets, we demonstrate the usefulness of the methods developed in this thesis in real-world decision making.

Abstract

Off-Policy Evaluation

Explore similar work

Learning Optimal and Sample-Efficient Decision Policies with Guarantees

Transportable Offline Reinforcement Learning in Presence of Unmeasured Confounding

PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data