Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling

Nafiseh Payani; Soham Das; G. Anthony Wilson; Anahita Khojandi

cs.LGSep 30, 2026

Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling

Authors: Nafiseh Payani, Soham Das, G. Anthony Wilson, Anahita Khojandi

Organizations: Department of Industrial and Systems Engineering at the University of Tennessee, Knoxville, TN, USA · Department of Industrial Engineering at the University of Arkansas, Fayetteville, AR, USA

Abstract

Patient preference, defined as a patient's demonstrated willingness and capacity to adhere to clinical recommendations, is a primary determinant of therapeutic effect yet remains structurally absent from existing computational treatment planning models. We address this gap by presenting patient-centered factored-action hierarchical option-critic (FAHOC), a hierarchical reinforcement learning (HRL) framework that jointly learns high-level options corresponding to therapeutic strategies and factored intra-option policies that decompose the joint action space into disease- and intervention-specific subcomponents, while imposing a cooperation-aware action masking mechanism. This enables structured exploration, improved credit assignment across hierarchy levels, and more interpretable decision pathways, while enforcing patients' preferences. Formal guarantees establish that cooperative patients achieve higher optimal expected health outcomes than non-cooperative patients, and that the factored Q-function approximation error is provably bounded. The framework is evaluated using longitudinal data collected from approximately 50,000 comorbid hypertension and type 2 diabetes mellitus patients from five hospitals in the Southeast U.S. FAHOC achieves a quality-adjusted life year expectancy equivalent improvement of 0.669 (vs -0.133 observed clinician practice), correctly identifies cooperative patients in 95.9% of cases and never violates a patient's preference in held-out test, demonstrating that HRL with explicit preference constraints can support preference-consistent, clinically safe decision-making in multimorbidity management.

Figures & tables

Figure 1: The proposed patient–preference–centered factored hierarchical offline RL framework

Type	Features	Description
Continuous	$SBP_{t}$ ; $A1C_{t}$ ; $BMI_{t}$ ; $eGFR_{t}$ ; $Age_{t}$	Systolic blood pressure (mmHg); glycated hemoglobin (%); body mass index (kg/m 2 ); kidney function (mL/min/1.73 m 2 ); age in years (adjusted based on day of the visit)
Ordinal	$b(s)_{t}$ ; $I^{\mathrm{T2DM}}_{t-1}$ ; $I^{\mathrm{HTN}}_{t-1}$	BMI category (0: Normal; 1: Overweight; 2: Obese); T2DM medication intensity at $t{-}1$ ; HTN medication intensity at $t{-}1$
Binary	$c$ ; no-visit flag at visit $t$	BMI cooperation type ( $c=0$ : non-cooperative; $c=1$ : cooperative), gap-visit indicator flags missed encounters
OHE	G., R., E.	Gender; Race; Ethnicity with 2, 2, and 3 binary indicators respectively, time independent features

Table 1: State features in encounter

t

(

|\mathcal{S}|=18

)

Figure 2: Overview of the FAHOC model. The shared encoder $\phi(s_{t})$ maps the patient state to a latent representation; the high-level critic $Q_{\Omega}(s_{t},\omega)$ induces a greedy policy $\omega^{*}=\arg\max_{\omega}Q_{\Omega}(s_{t},\omega)$ over two clinical options ( $\omega_{0}$ : Single-Target, $\omega_{1}$ : Multi-Target); per-option factored Q-heads produce the weighted composition $Q(s_{t},a_{t}\mid\omega)$ (Eq. ( 21 )) over three treatment dimensions (T2DM $\in\mathbb{R}^{3}$ , HTN $\in\mathbb{R}^{3}$ , BMI $\in\mathbb{R}^{2}$ ); and action masking enforces patient-preference and clinical safety constraints. Per-option termination heads $\beta_{\omega}(s_{t})\in(0,1)$ govern option switching via the option-critic gradient (see Appendix H ).

Dataset Partition
Characteristic	Train	Validation	Test
Patients	33,108	7,095	7,095
Transitions	414,880	90,231	89,556
Transitions/patient (mean $\pm$ std)	$12.5\pm 9.6$	$12.7\pm 9.5$	$12.6\pm 9.6$
BMI Cooperation Status
Cooperative	264,587 (63.8%)	58,567 (64.9%)	56,534 (63.1%)

Table 2: Cohort and Transition Dataset Statistics

Metric	Test	Random (%)
Overall action accuracy	69.8 %	5.6
T2DM component accuracy	82.3 %	33.3
HTN component accuracy	80.7 %	33.3
BMI $F_{1}$ (eligible patients) †	95.9 %	50.0
Option accuracy (Single- vs. Multi-Target)	53.8 %	50.0
Mean termination prob. $\bar{\beta}$	0.201	—

Table 3: Clinician Action Agreement: HRL Policy.

	Test Set ( $n=89{,}556$ )
Estimator	Estimate	95 % CI
Clinician (baseline)	$-0.133$	—
FQE ‡	$3.373$	$[3.342,\ 3.404]$
WIS	$-0.485$	$[-0.972,\ -0.102]$
DR	$0.669$	$[0.638,\ 0.697]$

Table 4: Off-Policy Evaluation: Estimated Policy Value (

\gamma=0.97

B=1{,}000

bootstrap resamples)

Figure 3: Per-patient treatment trajectories (4 patients, test set). Columns: T2DM, HTN, and BMI recommendations and per-encounter obsered reward (green $>0$ , red $<0$ ). Rows: Cooperative Patient I (Single-Target), Cooperative Patient II (Multi-Target), Non-Cooperative Patients III and IV (Single-Target).

Figure 4: Left: Concordance advantage surface ( $\Delta=\text{agent}-\text{clinician}$ ) over SBP (y-axis) and A1C (x-axis) intensification thresholds. Green ( $\Delta>0$ ) and red ( $\Delta<0$ ) denote agent and clinician advantage, respectively. Dashed white lines mark the ACC/AHA 2017 SBP threshold (130) and ADA 2025 A1C target (7.0); above both, the agent is uniformly more concordant ( $\Delta\in[{+}0.03,{+}0.14]$ ), with the clinician advantage confined to the sub-guideline region. Right: Concordance vs. SBP threshold at fixed A1C $=$ 7.0; the agent (blue) surpasses the clinician (red) at SBP $\geq$ 130 with a widening advantage through 170.

Accuracy
Metric	Uncontrolled	Controlled
Patients ( $n$ )	4,692	2,403
Transitions ( $n$ )	57,710	31,846
Overall action	68.1 %	72.8 %
T2DM component	80.7 %	85.2 %
HTN component	80.0 %	81.9 %

Table 5: Stratified Evaluation on the Test Set: Uncontrolled vs. Controlled Trajectories

Metric	HRL (Test)	baseline DDQN (Val)	$\Delta$ (HRL $-$ DDQN)
Overall action accuracy	69.8 %	69.2 %	$+$ 0.6%
T2DM component accuracy	82.3 %	81.4 %	$+$ 0.9%
HTN component accuracy	80.7 %	79.1 %	$+$ 1.6%
BMI accuracy (all)	96.6 %	96.3 %	$+$ 0.3%
BMI discriminative †	95.9 %	94.0 %	$+$ 1.9%
Option accuracy	53.8 %	N/A	—

Table 6: Clinician Action Agreement: HRL Policy vs. DDQN Baseline.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Age	25–34	35–44	45–54	55–64	65–74	$\geq$ 75
A1C	0.65	0.46	0.25	0.13	0.05	0.01
SBP	0.69	0.62	0.53	0.44	0.36	0.23

Appendix

Table 7: Maximum QALY utilities by age group.

HTN Medications	T2DM Medications
ACE inhibitors : benazepril, lisinopril, enalapril, captopril, fosinopril, perindopril, quinapril, ramipril, trandolapril, moexipril	Biguanides : metformin
ARB combinations : sacubitril	Sulfonylureas : chlorpropamide, diabinese, glipizide, glucotrol, glyburide, diabeta, glimepiride, glibenclamide
ARBs : valsartan, losartan, candesartan, azilsartan, eprosartan, irbesartan, olmesartan, telmisartan	DPP-4 inhibitors : sitagliptin, vildagliptin, linagliptin, saxagliptin, alogliptin
Calcium channel blockers : amlodipine, felodipine, diltiazem, isradipine, nicardipine, nifedipine, nisoldipine, verapamil	SGLT-2 inhibitors : ertugliflozin, canagliflozin, empagliflozin, dapagliflozin, bexagliflozin
Centrally acting : clonidine, methyldopa, guanfacine, reserpine	Thiazolidinediones : pioglitazone, rosiglitazone
Alpha blockers : doxazosin, prazosin	GLP-1 receptor agonists : liraglutide, albiglutide, semaglutide, exenatide, dulaglutide, lixisenatide

Appendix

Table 8: Medications for HTN and T2DM by class encoded in the model.

Parameter	Value	Parameter	Value
Discount $\gamma$	0.97	Hidden dimension	256
Soft-update $\tau$	0.001	PER $\alpha$	0.6
Buffer capacity	$500{,}000$	PER $\beta_{\mathrm{IS}}$	$0.4\to 1.0$
Batch size	256	Target $\bar{\beta}$	0.30
Encoder lr	$5\times 10^{-5}$	TD clamp	$[-10,10]$
High-level lr	$2\times 10^{-5}$	Grad clip (low/high)	1.0

Appendix

Table 9: Hyperparameters.

Figure 5: Training dynamics over 150 epochs. From left to right: low-level intra-option Q-loss; high-level option-value loss; termination loss; validation accuracy by disease component (Action, T2DM, HTN); and average termination probability $\bar{\beta}$ (dashed red line = target $\leq 0.5$ ).

Figure 6: Factored action hierarchical option-critic (FAHOC) framework network architecture. The shared encoder maps state $s\in\mathbb{R}^{18}$ to a 256-dimensional embedding $\mathbf{h}$ . Four parameter groups branch from $\mathbf{h}$ : the high-level head outputs $Q_{\Omega}(s,\omega)$ over the two options ( $\omega_{0}$ : Single-Target, $\omega_{1}$ : Multi-Target); per-option factored heads decompose the intra-option Q-value as $Q(s,\omega,a)=0.4\,Q_{\mathrm{t2dm}}+0.4\,Q_{\mathrm{htn}}+0.2\,Q_{\mathrm{bmi}}$ (Eq. 21 ); an action mask sets $Q(s,\omega,a){=}{-}\infty$ for $a_{\mathrm{bmi}}{=}1$ when the patient is non-cooperative or has normal BMI, preventing infeasible treatment selections; and per-option termination heads output $\beta_{\omega}(s)\in(0,1)$ via sigmoid. LN = Layer Normalization; $\sigma$ = Sigmoid.

Domain Pair	Global	Binned	Bootstrap mean	95% CI
T2DM $\times$ HTN	0.2629	0.6948	0.2758	[0.246, 0.311]
T2DM $\times$ BMI	0.3351	1.2150	0.3739	[0.304, 0.477]
HTN $\times$ BMI	0.4275	1.0531	0.4380	[0.361, 0.510]
$\hat{\Phi}_{\mathrm{all}}$	1.0255	2.9630	1.0877	[0.911, 1.298]

Appendix

Table 10: Empirical

\hat{\Phi}_{k\ell}

from 414,880 training transitions (

B=500

bootstrap resamples, 80% subsample).

	Train	Val	Test	All
Transitions / Patients	414,880 / 33,108	90,231 / 7,095	89,556 / 7,095	594,667 / 47,298
C1 cooperative	264,587 (63.8%)	58,567 (64.9%)	56,534 (63.1%)	379,688 (63.9%)
C2 overweight/obese	334,992 (80.7%)	72,986 (80.9%)	71,612 (80.0%)	479,590 (80.7%)
C3 uncontrolled	298,369 (71.9%)	64,674 (71.7%)	64,726 (72.3%)	427,769 (71.9%)
C1 $\cap$ C2	184,765 (44.5%)	41,368 (45.8%)	38,628 (43.1%)	264,761 (44.5%)
C1 $\cap$ C2 $\cap$ C3	133,139 (32.1%)	29,520 (32.7%)	28,245 (31.5%)	190,904 (32.1%)

Appendix

Table 11: BMI-beneficial state counts. C1: cooperative (

c=1

); C2: overweight/obese (BMI

\geq\!25

kg/m 2 ); C3: uncontrolled (A1C

>\!7.0

% or SBP

>\!130

mmHg).

Explore similar work

Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care
Apr 30, 2026Prabhjot Singh, Abhishek Gupta, Chris Betz +4Clinician Trust Disagreement
Altitude
Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management
May 10, 2026Prabhjot Singh, Abhishek Gupta, Chris Betz +4Offline Reinforcement Learning
Altitude
Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning
Jun 12, 2026Pengxin Wang, Lihao Guo, Yi Xie +3Multi-Agent Reinforcement Learning
Department of Electrical and Computer Engineering, University of Arizona

Apr 30, 2026, cs.LGJ/K move · Enter open · S save

Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care

Prabhjot Singh, Abhishek Gupta, Chris Betz, Abe Flansburg+3

Altitude

Clinician Trust Disagreement

Open paper AskPDF arXiv

We reframe clinician overrides of clinical AI recommendations as implicit preference data - the same signal structure exploited by reinforcement learning from human feedback (RLHF), but richer: the annotator is a domain expert, the alternatives carry real consequences, and downstream outcomes are observable. We present a formal framework extending standard preference learning with three contributions: a five-category override taxonomy mapping override types to distinct model update targets; a preference formulation conditioned on patient state s, organizational context c, and clinician capability kappa, where kappa decomposes into execution capability kappa-exec and alignment capability kappa-align; and a dual learning architecture that jointly trains a reward model and a capability model via alternating optimization, preventing a failure mode we term suppression bias-the systematic suppression of correct-but-difficult recommendations when clinician capability falls below the execution threshold. We argue that chronic disease management under outcome-based payment contracts produces override data with uniquely favorable properties-longitudinal density, concentrated decision space, outcome labels, and natural capability variation-and that training environments combining longitudinal outcome measurement with aligned financial incentives are a necessary condition for learning a reward model aligned with patient trajectory rather than with encounter economics. This framework emerged from operational work to improve clinician capability in a live value-based care deployment.

Abstract

Explore similar work

Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care

Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management

Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning