Boosted Distributional Reinforcement Learning: Analysis and Healthcare Applications

Zequn Chen; Wesley J. Marrero

cs.LGApr 6, 2026

Boosted Distributional Reinforcement Learning: Analysis and Healthcare Applications

Authors: Zequn Chen, Wesley J. Marrero

Organizations: Thayer School of Engineering, Dartmouth College

Abstract

Researchers and practitioners are increasingly considering reinforcement learning to optimize decisions in complex domains like robotics and healthcare. To date, these efforts have largely utilized expectation-based learning. However, relying on expectation-focused objectives may be insufficient for making consistent decisions in highly uncertain situations involving multiple heterogeneous groups. While distributional reinforcement learning algorithms have been introduced to model the full distributions of outcomes, they can yield large discrepancies in realized benefits among comparable agents. This challenge is particularly acute in healthcare settings, where physicians (controllers) must manage multiple patients (subordinate agents) with uncertain disease progression and heterogeneous treatment responses. We propose a Boosted Distributional Reinforcement Learning (BDRL) algorithm that optimizes agent-specific outcome distributions while enforcing comparability among similar agents and analyze its convergence. To further stabilize learning, we incorporate a post-update projection step formulated as a constrained convex optimization problem, which efficiently aligns individual outcomes with a high-performing reference within a specified tolerance. We apply our algorithm to manage hypertension in a large subset of the US adult population by categorizing individuals into cardiovascular disease risk groups. Our approach modifies treatment plans for median and vulnerable patients by mimicking the behavior of high-performing references in each risk group. Furthermore, we find that BDRL improves the number and consistency of quality-adjusted life years compared with reinforcement learning baselines.

Figures & tables

Figure 1: Overview of the BDRL algorithm.

Parameter	Value	Source
BP reduction: standard dose (half dose)
Systolic BP	5.5 (3.7) mm Hg	( Blood Pressure Lowering Treatment Trialists’ Collaboration, 2014 ) , ( Sussman et al., 2013 )
Diastolic BP	3.3 (2.2) mm Hg	( Blood Pressure Lowering Treatment Trialists’ Collaboration, 2014 ) , ( Sussman et al., 2013 )
Risk for ASCVD events	Varies by patient	( Yadlowsky et al., 2018 )
ASCVD risk reduction: standard dose (half dose)
Myocardial infarction	13% (7%)	( Blood Pressure Lowering Treatment Trialists’ Collaboration, 2014 ) , ( Sussman et al., 2013 )

Table 1: Base case parameters.

Figure 2: Overview of the simulation framework.

Figure 3: Probability of action selection during training for the most vulnerable patient and the most resilient patient across risk clusters before and after boosting. The treatment choice ranging from 0 to 20 represents the index of each of the 21 actions considered in the MDPs .

Figure 4: Learned QALYs distributions of the two most different patients in each group.

Algorithm	Risk Group	Most Resilient	Median	Most Vulnerable
BDRL (Base)	Low	32.35	32.19	31.82
	Intermediate	26.63	26.35	26.26
	High	19.78	19.63	19.37
DRL ( $\lambda=0$ )	Low	32.35	31.62	29.75
	Intermediate	26.63	26.08	24.46
	High	19.78	18.64	17.06

Table 2: Comparison of the most resilient, median, and most vulnerable patients across risk groups.

Panel of Figure 5 — Figure 5: Comparison of QALYs across different penalty weights.

		Low risk			Intermediate			High risk
Scenario		Resi.	Med.	Vulne.	Resi.	Med.	Vulne.	Resi.	Med.	Vulne.
Base case		32.35	32.19	31.82	26.63	26.35	26.26	19.78	19.63	19.37
$\epsilon$	0.005	32.08	31.52	31.03	26.46	26.12	25.91	19.25	18.78	18.57
$\epsilon$	0.050	32.19	31.68	31.25	26.41	26.03	25.76	19.34	19.08	18.87
$\rho$	0.80	31.95	31.43	31.11	26.37	26.09	25.79	19.19	18.86	18.64
$\rho$	0.95	31.86	31.54	31.13	25.83	25.38	25.05	18.84	18.39	18.12

Table 3: Mean discounted QALYs. The base case uses

W_{2}

-regularization with

\lambda=0.1

\epsilon=0.010

\rho=0.90

\alpha=0.05

. Sensitivity rows vary one parameter at a time. The KL and JS rows show results across regularization strengths

\lambda\in\{0,0.01,0.10,0.50\}

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Supplementary Figure 1 : Selecting the number of centroids ( $k$ ).

Supplementary Figure 2 : Finding the optimal number of batches ( $B$ ).

Supplementary Figure 3 : Convergence of $W_{2}$ distance within subgroups, showing the reduction in disparity over training episodes.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management
May 10, 2026Prabhjot Singh, Abhishek Gupta, Chris Betz +4Reinforcement Learning Type 1 Diabetes
Altitude
Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms
Apr 4, 2024Miao Lu, Han Zhong, Tong Zhang +1Distributionally Robust RL Robust RL
†Department of Management Science and Engineering, Stanford University. · ‡Center for Data Science, Peking University. · §Department of Computer Science, University of Illinois Urbana-Champaign.
Distributionally Robust Average-Reward Reinforcement Learning: Finite-Sample Guarantees under Weak Communication
Sep 26, 2026Chenyu Lu, Zijun Chen, Nian SiDistributionally Robust RL Average-Reward RL
The Hong Kong University of Science and Technology

May 10, 2026J/K move · Enter open · S save

Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management

Prabhjot Singh, Abhishek Gupta, Chris Betz, Abe Flansburg+3

Altitude

Reinforcement Learning Type 1 Diabetes Healthcare Offline RL

Open paper AskPDF arXiv

Reinforcement learning (RL) in healthcare has had mixed results, with reward sparsity, unreliable off-policy evaluation, and deployment-simulation gap as recurring failure modes. We argue that chronic disease management is structurally a more tractable RL setting than the acute-care problems the field has primarily studied, but only if the problem is formalized to exploit chronic care's properties. We propose such a formalization. The agent's objective is to compress time-to-control (TTC) under a tiered reward calibrated to the CMS ACCESS Model. Two quantities from our companion preference-learning paper [Singh et al. 2026] enter as load-bearing structural elements: the execution intensity εbounds action availability under a constrained Markov Decision Process, and the clinician capability κweights offline-data transitions during RL training. Together they couple preference learning and RL into a two-loop architecture. We present simulation results on synthetic state machines for hypertension and type 2 diabetes. Capability-weighted offline RL outperforms uniform-weighted offline RL and the behavior policy by 15 percentage points on T2D TTC; the uniform-weighted formulation (the standard in existing healthcare RL) underperforms even the heterogeneous behavior policy. \Epsilon-aware policies generalize across deployment regimes while ε-naive policies do not.

Abstract

Reinforcement Learning

Explore similar work

Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management

Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms

Distributionally Robust Average-Reward Reinforcement Learning: Finite-Sample Guarantees under Weak Communication