Near-Equivalent Q-learning Policies for Dynamic Treatment Regimes

Sophia Yazzourh; Erica E. M. Moodie

stat.MLMar 19, 2026

Near-Equivalent Q-learning Policies for Dynamic Treatment Regimes

Authors: Sophia Yazzourh, Erica E. M. Moodie

Organizations: Department of Epidemiology and Biostatistics McGill University Montréal, Québec, Canada

Abstract

Precision medicine aims to tailor therapeutic decisions to individual patient characteristics. This objective is commonly formalized through dynamic treatment regimes, which use statistical and machine learning methods to derive sequential decision rules adapted to evolving clinical information. In most existing formulations, these approaches produce a single optimal treatment at each stage, leading to a unique decision sequence. However, in many clinical settings, several treatment options may yield similar expected outcomes, and focusing on a single optimal policy may conceal meaningful alternatives. We extend the Q-learning framework for retrospective data by introducing a worst-value tolerance criterion controlled by a hyperparameter $\varepsilon$ , which specifies the maximum acceptable deviation from the optimal expected value. Rather than identifying a single optimal policy, the proposed approach constructs sets of $\varepsilon$ -optimal policies whose performance remains within a controlled neighborhood of the optimum. This formulation shifts Q-learning from a vector-valued representation to a matrix-valued one, allowing multiple admissible value functions to coexist during backward recursion. The approach yields families of near-equivalent treatment strategies and explicitly identifies regions of treatment indifference where several decisions achieve comparable outcomes. We illustrate the framework in two settings: a single-stage problem highlighting indifference regions around the decision boundary, and a multi-stage decision process based on a simulated oncology model describing tumor size and treatment toxicity dynamics.

Figures & tables

Figure 1: Estimated blip surface $\widehat{\Delta}(X)=\widehat{Q}(X,1)-\widehat{Q}(X,-1)$ projected onto the $(X_{0},X_{1})$ plane for increasing values of $\varepsilon$ . The dashed line represents the true decision boundary $X_{0}+X_{1}=0$ . White contours indicate the $\varepsilon$ -selection region defined by $|\widehat{\Delta}(X)|\leq\varepsilon$ .

Figure 2: Combined average tumor size and toxicity over six stages for 5000 simulated patients under identical initial conditions ( $n_{\text{train}}=500$ , $n_{\text{test}}=5000$ ). Each panel corresponds to a different value of $\varepsilon$ , illustrating how the admissible set of policies expands as the tolerance increases. Dashed lines correspond to constant treatment regimes ranging from 0.1 to 1.0. The red solid line (“opt”) represents the policy learned via classical backward Q-learning. Bold dashed lines denote the $\varepsilon$ -equivalent policies obtained with the proposed method. The shaded red band indicates the $\varepsilon$ -tolerance region around the optimal policy.

Q-Learning

Q-learning estimates long-term action values from experience so agents can improve decisions without a learned transition model.

See all 98 papers

May 19, 2026

Set-Valued Policy Learning

Conventional treatment policies map patient covariates to a single recommended intervention in order to maximize expected clinical outcomes. However, when multiple treatments yield statistically indistinguishable outcomes or when treatment has no effect, recommending a single intervention may result in somewhat arbitrary interventions, undermining clinical adoption and trust. To address this, we propose a set-valued policy learning paradigm. By outputting sets of valuable treatments whose cardinality reflects the recommendation's ambiguity, our approach better supports clinical decision-making. Evaluating a set-valued policy proves subtle due to the range of possible downstream decisions. To do so, we define the set-policy value using a choice function to model clinical decision-making, and we develop doubly robust estimators thereof. Despite its practical importance, set-valued policy learning for categorical treatments remains largely unexplored. In this context, we introduce two complementary approaches: the Greatest Lower Bound method, which extends the learning-to-defer framework to multiple treatments, and conformal set-valued policy learning, which bridges the gap between unobserved ground-truth optimal treatments and estimated optimal treatment rules. Through experiments on synthetic data and real-world applications to trauma care and in-vitro fertilization (IVF), we demonstrate that our methods produce robust and actionable policies that naturally incorporate clinical considerations while effectively balancing performance and reliability.

Laura Fuentes-Vicente, Mathieu Even, Gaëlle Dormion +3

Inria PreMeDICaL, Inserm, Montpellier, France · Elixir Health, Paris, France · Université Paris Cité, CNRS, MAP5, F-75006 Paris, France +1

Healthcare Learning to Defer Clinical Decision-Making

Ask PDF

Apr 13, 2026

Cost-optimal Sequential Testing via Doubly Robust Q-learning

Clinical decision-making often involves selecting tests that are costly, invasive, or time-consuming, motivating individualized, sequential strategies for what to measure and when to stop ascertaining. We study the problem of learning cost-optimal sequential decision policies from retrospective data, where test availability depends on prior results, inducing informative missingness. Under a sequential missing-at-random mechanism, we develop a doubly robust Q-learning framework for estimating optimal policies. The method introduces path-specific inverse probability weights that account for heterogeneous test trajectories and satisfy a normalization property conditional on the observed history. By combining these weights with auxiliary contrast models, we construct orthogonal pseudo-outcomes that enable unbiased policy learning when either the acquisition model or the contrast model is correctly specified. We establish oracle inequalities for the stage-wise contrast estimators, along with convergence rates, regret bounds, and misclassification rates for the learned policy. Simulations demonstrate improved cost-adjusted performance over weighted and complete-case baselines, and an application to a prostate cancer cohort study illustrates how the method reduces testing cost without compromising predictive accuracy.

Doudou Zhou, Yiran Zhang, Dian Jin +3

Department of Statistics and Data Science, National University of Singapore · Public Health Sciences Division, Fred Hutch · Department of Biomedical Data Science, Stanford University School of Medicine +1

Q-Learning Cost-Sensitive Learning Learning with Missing Data Doubly Robust Estimation

Ask PDF

May 7, 2026

Dynamic Treatment on Networks

In networks, effective dynamic treatment allocation requires deciding both whom to treat and also when, so as to amplify policy impact through spillovers. An early intervention at a well-connected node can trigger cascades that change which nodes are worth targeting in the next period. Existing treatment strategies under network interference are largely static while dynamic treatment frameworks typically ignore network structure altogether. We integrate these perspectives and propose Q-Ising, a three-stage pipeline that (i) estimates network adoption dynamics via a Bayesian dynamic Ising model from a single observed panel, (ii) augments treatment adoption histories with continuous posterior latent states, and (iii) learns a dynamic policy via offline reinforcement learning. The Bayesian mechanism enables uncertainty quantification over dynamic decisions, yielding posterior ensemble policies with interpretable spillover estimates. We provide a finite-sample regret upper bound that decomposes into standard offline-RL uncertainty, network abstraction error, and first stage error in Ising state estimation. We apply our method to data from Indian village microfinance networks and synthetic stochastic block models under simulated heterogeneous susceptible-infected-susceptible (SIS) dynamics and demonstrate that adaptive targeting outperforms static centrality benchmarks.

Bengusu Nar, Jiguang Li, Veronika Ročková +1

Econometrics and Statistics at the Booth School of Business of the University of Chicago · Bruce Lindsay Professor of Econometrics and Statistics in the Wallman Society of Fellows at the Booth School of Business of the University of Chicago · Professor of Econometrics and Statistics, and John E. Jeuck Faculty Fellow at the Booth School of Business of the University of Chicago

Sequential Decision Making Ising Model

Ask PDF

Abstract

Q-Learning

Explore similar work

Set-Valued Policy Learning

Cost-optimal Sequential Testing via Doubly Robust Q-learning

Dynamic Treatment on Networks