Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling

Nafiseh Payani; Soham Das; G. Anthony Wilson; Anahita Khojandi

cs.LGSep 30, 2026

Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling

Authors: Nafiseh Payani, Soham Das, G. Anthony Wilson, Anahita Khojandi

Organizations: Department of Industrial and Systems Engineering at the University of Tennessee, Knoxville, TN, USA · Department of Industrial Engineering at the University of Arkansas, Fayetteville, AR, USA

Abstract

Patient preference, defined as a patient's demonstrated willingness and capacity to adhere to clinical recommendations, is a primary determinant of therapeutic effect yet remains structurally absent from existing computational treatment planning models. We address this gap by presenting patient-centered factored-action hierarchical option-critic (FAHOC), a hierarchical reinforcement learning (HRL) framework that jointly learns high-level options corresponding to therapeutic strategies and factored intra-option policies that decompose the joint action space into disease- and intervention-specific subcomponents, while imposing a cooperation-aware action masking mechanism. This enables structured exploration, improved credit assignment across hierarchy levels, and more interpretable decision pathways, while enforcing patients' preferences. Formal guarantees establish that cooperative patients achieve higher optimal expected health outcomes than non-cooperative patients, and that the factored Q-function approximation error is provably bounded. The framework is evaluated using longitudinal data collected from approximately 50,000 comorbid hypertension and type 2 diabetes mellitus patients from five hospitals in the Southeast U.S. FAHOC achieves a quality-adjusted life year expectancy equivalent improvement of 0.669 (vs -0.133 observed clinician practice), correctly identifies cooperative patients in 95.9% of cases and never violates a patient's preference in held-out test, demonstrating that HRL with explicit preference constraints can support preference-consistent, clinically safe decision-making in multimorbidity management.

Figures & tables

Figure 1: The proposed patient–preference–centered factored hierarchical offline RL framework

Type	Features	Description
Continuous	$SBP_{t}$ ; $A1C_{t}$ ; $BMI_{t}$ ; $eGFR_{t}$ ; $Age_{t}$	Systolic blood pressure (mmHg); glycated hemoglobin (%); body mass index (kg/m 2 ); kidney function (mL/min/1.73 m 2 ); age in years (adjusted based on day of the visit)
Ordinal	$b(s)_{t}$ ; $I^{\mathrm{T2DM}}_{t-1}$ ; $I^{\mathrm{HTN}}_{t-1}$	BMI category (0: Normal; 1: Overweight; 2: Obese); T2DM medication intensity at $t{-}1$ ; HTN medication intensity at $t{-}1$
Binary	$c$ ; no-visit flag at visit $t$	BMI cooperation type ( $c=0$ : non-cooperative; $c=1$ : cooperative), gap-visit indicator flags missed encounters
OHE	G., R., E.	Gender; Race; Ethnicity with 2, 2, and 3 binary indicators respectively, time independent features

Table 1: State features in encounter

t

(

|\mathcal{S}|=18

)

Figure 2: Overview of the FAHOC model. The shared encoder $\phi(s_{t})$ maps the patient state to a latent representation; the high-level critic $Q_{\Omega}(s_{t},\omega)$ induces a greedy policy $\omega^{*}=\arg\max_{\omega}Q_{\Omega}(s_{t},\omega)$ over two clinical options ( $\omega_{0}$ : Single-Target, $\omega_{1}$ : Multi-Target); per-option factored Q-heads produce the weighted composition $Q(s_{t},a_{t}\mid\omega)$ (Eq. ( 21 )) over three treatment dimensions (T2DM $\in\mathbb{R}^{3}$ , HTN $\in\mathbb{R}^{3}$ , BMI $\in\mathbb{R}^{2}$ ); and action masking enforces patient-preference and clinical safety constraints. Per-option termination heads $\beta_{\omega}(s_{t})\in(0,1)$ govern option switching via the option-critic gradient (see Appendix H ).

Dataset Partition
Characteristic	Train	Validation	Test
Patients	33,108	7,095	7,095
Transitions	414,880	90,231	89,556
Transitions/patient (mean $\pm$ std)	$12.5\pm 9.6$	$12.7\pm 9.5$	$12.6\pm 9.6$
BMI Cooperation Status
Cooperative	264,587 (63.8%)	58,567 (64.9%)	56,534 (63.1%)

Table 2: Cohort and Transition Dataset Statistics

Metric	Test	Random (%)
Overall action accuracy	69.8 %	5.6
T2DM component accuracy	82.3 %	33.3
HTN component accuracy	80.7 %	33.3
BMI $F_{1}$ (eligible patients) †	95.9 %	50.0
Option accuracy (Single- vs. Multi-Target)	53.8 %	50.0
Mean termination prob. $\bar{\beta}$	0.201	—

Table 3: Clinician Action Agreement: HRL Policy.

	Test Set ( $n=89{,}556$ )
Estimator	Estimate	95 % CI
Clinician (baseline)	$-0.133$	—
FQE ‡	$3.373$	$[3.342,\ 3.404]$
WIS	$-0.485$	$[-0.972,\ -0.102]$
DR	$0.669$	$[0.638,\ 0.697]$

Table 4: Off-Policy Evaluation: Estimated Policy Value (

\gamma=0.97

B=1{,}000

bootstrap resamples)

Figure 3: Per-patient treatment trajectories (4 patients, test set). Columns: T2DM, HTN, and BMI recommendations and per-encounter obsered reward (green $>0$ , red $<0$ ). Rows: Cooperative Patient I (Single-Target), Cooperative Patient II (Multi-Target), Non-Cooperative Patients III and IV (Single-Target).

Figure 4: Left: Concordance advantage surface ( $\Delta=\text{agent}-\text{clinician}$ ) over SBP (y-axis) and A1C (x-axis) intensification thresholds. Green ( $\Delta>0$ ) and red ( $\Delta<0$ ) denote agent and clinician advantage, respectively. Dashed white lines mark the ACC/AHA 2017 SBP threshold (130) and ADA 2025 A1C target (7.0); above both, the agent is uniformly more concordant ( $\Delta\in[{+}0.03,{+}0.14]$ ), with the clinician advantage confined to the sub-guideline region. Right: Concordance vs. SBP threshold at fixed A1C $=$ 7.0; the agent (blue) surpasses the clinician (red) at SBP $\geq$ 130 with a widening advantage through 170.

Accuracy
Metric	Uncontrolled	Controlled
Patients ( $n$ )	4,692	2,403
Transitions ( $n$ )	57,710	31,846
Overall action	68.1 %	72.8 %
T2DM component	80.7 %	85.2 %
HTN component	80.0 %	81.9 %

Table 5: Stratified Evaluation on the Test Set: Uncontrolled vs. Controlled Trajectories

Metric	HRL (Test)	baseline DDQN (Val)	$\Delta$ (HRL $-$ DDQN)
Overall action accuracy	69.8 %	69.2 %	$+$ 0.6%
T2DM component accuracy	82.3 %	81.4 %	$+$ 0.9%
HTN component accuracy	80.7 %	79.1 %	$+$ 1.6%
BMI accuracy (all)	96.6 %	96.3 %	$+$ 0.3%
BMI discriminative †	95.9 %	94.0 %	$+$ 1.9%
Option accuracy	53.8 %	N/A	—

Table 6: Clinician Action Agreement: HRL Policy vs. DDQN Baseline.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Age	25–34	35–44	45–54	55–64	65–74	$\geq$ 75
A1C	0.65	0.46	0.25	0.13	0.05	0.01
SBP	0.69	0.62	0.53	0.44	0.36	0.23

Appendix

Table 7: Maximum QALY utilities by age group.

HTN Medications	T2DM Medications
ACE inhibitors : benazepril, lisinopril, enalapril, captopril, fosinopril, perindopril, quinapril, ramipril, trandolapril, moexipril	Biguanides : metformin
ARB combinations : sacubitril	Sulfonylureas : chlorpropamide, diabinese, glipizide, glucotrol, glyburide, diabeta, glimepiride, glibenclamide
ARBs : valsartan, losartan, candesartan, azilsartan, eprosartan, irbesartan, olmesartan, telmisartan	DPP-4 inhibitors : sitagliptin, vildagliptin, linagliptin, saxagliptin, alogliptin
Calcium channel blockers : amlodipine, felodipine, diltiazem, isradipine, nicardipine, nifedipine, nisoldipine, verapamil	SGLT-2 inhibitors : ertugliflozin, canagliflozin, empagliflozin, dapagliflozin, bexagliflozin
Centrally acting : clonidine, methyldopa, guanfacine, reserpine	Thiazolidinediones : pioglitazone, rosiglitazone
Alpha blockers : doxazosin, prazosin	GLP-1 receptor agonists : liraglutide, albiglutide, semaglutide, exenatide, dulaglutide, lixisenatide

Appendix

Table 8: Medications for HTN and T2DM by class encoded in the model.

Parameter	Value	Parameter	Value
Discount $\gamma$	0.97	Hidden dimension	256
Soft-update $\tau$	0.001	PER $\alpha$	0.6
Buffer capacity	$500{,}000$	PER $\beta_{\mathrm{IS}}$	$0.4\to 1.0$
Batch size	256	Target $\bar{\beta}$	0.30
Encoder lr	$5\times 10^{-5}$	TD clamp	$[-10,10]$
High-level lr	$2\times 10^{-5}$	Grad clip (low/high)	1.0

Appendix

Table 9: Hyperparameters.

Figure 5: Training dynamics over 150 epochs. From left to right: low-level intra-option Q-loss; high-level option-value loss; termination loss; validation accuracy by disease component (Action, T2DM, HTN); and average termination probability $\bar{\beta}$ (dashed red line = target $\leq 0.5$ ).

Figure 6: Factored action hierarchical option-critic (FAHOC) framework network architecture. The shared encoder maps state $s\in\mathbb{R}^{18}$ to a 256-dimensional embedding $\mathbf{h}$ . Four parameter groups branch from $\mathbf{h}$ : the high-level head outputs $Q_{\Omega}(s,\omega)$ over the two options ( $\omega_{0}$ : Single-Target, $\omega_{1}$ : Multi-Target); per-option factored heads decompose the intra-option Q-value as $Q(s,\omega,a)=0.4\,Q_{\mathrm{t2dm}}+0.4\,Q_{\mathrm{htn}}+0.2\,Q_{\mathrm{bmi}}$ (Eq. 21 ); an action mask sets $Q(s,\omega,a){=}{-}\infty$ for $a_{\mathrm{bmi}}{=}1$ when the patient is non-cooperative or has normal BMI, preventing infeasible treatment selections; and per-option termination heads output $\beta_{\omega}(s)\in(0,1)$ via sigmoid. LN = Layer Normalization; $\sigma$ = Sigmoid.

Domain Pair	Global	Binned	Bootstrap mean	95% CI
T2DM $\times$ HTN	0.2629	0.6948	0.2758	[0.246, 0.311]
T2DM $\times$ BMI	0.3351	1.2150	0.3739	[0.304, 0.477]
HTN $\times$ BMI	0.4275	1.0531	0.4380	[0.361, 0.510]
$\hat{\Phi}_{\mathrm{all}}$	1.0255	2.9630	1.0877	[0.911, 1.298]

Appendix

Table 10: Empirical

\hat{\Phi}_{k\ell}

from 414,880 training transitions (

B=500

bootstrap resamples, 80% subsample).

	Train	Val	Test	All
Transitions / Patients	414,880 / 33,108	90,231 / 7,095	89,556 / 7,095	594,667 / 47,298
C1 cooperative	264,587 (63.8%)	58,567 (64.9%)	56,534 (63.1%)	379,688 (63.9%)
C2 overweight/obese	334,992 (80.7%)	72,986 (80.9%)	71,612 (80.0%)	479,590 (80.7%)
C3 uncontrolled	298,369 (71.9%)	64,674 (71.7%)	64,726 (72.3%)	427,769 (71.9%)
C1 $\cap$ C2	184,765 (44.5%)	41,368 (45.8%)	38,628 (43.1%)	264,761 (44.5%)
C1 $\cap$ C2 $\cap$ C3	133,139 (32.1%)	29,520 (32.7%)	28,245 (31.5%)	190,904 (32.1%)

Appendix

Table 11: BMI-beneficial state counts. C1: cooperative (

c=1

); C2: overweight/obese (BMI

\geq\!25

kg/m 2 ); C3: uncontrolled (A1C

>\!7.0

% or SBP

>\!130

mmHg).

Explore similar work

Apr 30, 2026cs.LG

Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care

We reframe clinician overrides of clinical AI recommendations as implicit preference data - the same signal structure exploited by reinforcement learning from human feedback (RLHF), but richer: the annotator is a domain expert, the alternatives carry real consequences, and downstream outcomes are observable. We present a formal framework extending standard preference learning with three contributions: a five-category override taxonomy mapping override types to distinct model update targets; a preference formulation conditioned on patient state s, organizational context c, and clinician capability kappa, where kappa decomposes into execution capability kappa-exec and alignment capability kappa-align; and a dual learning architecture that jointly trains a reward model and a capability model via alternating optimization, preventing a failure mode we term suppression bias-the systematic suppression of correct-but-difficult recommendations when clinician capability falls below the execution threshold. We argue that chronic disease management under outcome-based payment contracts produces override data with uniquely favorable properties-longitudinal density, concentrated decision space, outcome labels, and natural capability variation-and that training environments combining longitudinal outcome measurement with aligned financial incentives are a necessary condition for learning a reward model aligned with patient trajectory rather than with encounter economics. This framework emerged from operational work to improve clinician capability in a live value-based care deployment.

Prabhjot Singh, Abhishek Gupta, Chris Betz +4

Altitude

Clinician Trust Disagreement

Ask PDF

May 10, 2026cs.LG

Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management

Reinforcement learning (RL) in healthcare has had mixed results, with reward sparsity, unreliable off-policy evaluation, and deployment-simulation gap as recurring failure modes. We argue that chronic disease management is structurally a more tractable RL setting than the acute-care problems the field has primarily studied, but only if the problem is formalized to exploit chronic care's properties. We propose such a formalization. The agent's objective is to compress time-to-control (TTC) under a tiered reward calibrated to the CMS ACCESS Model. Two quantities from our companion preference-learning paper [Singh et al. 2026] enter as load-bearing structural elements: the execution intensity εbounds action availability under a constrained Markov Decision Process, and the clinician capability κweights offline-data transitions during RL training. Together they couple preference learning and RL into a two-loop architecture. We present simulation results on synthetic state machines for hypertension and type 2 diabetes. Capability-weighted offline RL outperforms uniform-weighted offline RL and the behavior policy by 15 percentage points on T2D TTC; the uniform-weighted formulation (the standard in existing healthcare RL) underperforms even the heterogeneous behavior policy. \Epsilon-aware policies generalize across deployment regimes while ε-naive policies do not.

Prabhjot Singh, Abhishek Gupta, Chris Betz +4

Altitude

Offline Reinforcement Learning

Ask PDF

Jun 12, 2026cs.MA

Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning

Cooperative multi-objective multi-agent reinforcement learning (MOMARL) models team decision making under multiple, potentially conflicting objectives. In this setting, conflicts arise not only across objectives but also across agents with different observations, roles, and contributions. We propose Preference Coordinated Multi-agent Policy Optimization (PCMA), which learns coordinated agent-specific preferences to enable complementary trade-offs among agents. Theoretically, we formulate cooperative MOMARL as a team-optimal equilibrium problem, and show that, under suitable conditions, preference diversity yields a first-order improvement in the team objective. Experiments on multiple cooperative MOMA environments and a practical traffic-control scenario show that PCMA improves both performance and trade-off coordination.

Pengxin Wang, Lihao Guo, Yi Xie +3

Department of Electrical and Computer Engineering, University of Arizona

Multi-Agent Reinforcement Learning

Ask PDF