ALPACA: A Reinforcement Learning Environment for Medication Repurposing and Treatment Optimization in Alzheimer's Disease

Nolan Brady; Tom Yeh

cs.AIFeb 22, 2026

ALPACA: A Reinforcement Learning Environment for Medication Repurposing and Treatment Optimization in Alzheimer's Disease

Authors: Nolan Brady, Tom Yeh

Organizations: University of Colorado Boulder, United States

Abstract

Evaluating personalized, sequential treatment strategies for Alzheimer's disease (AD) using clinical trials is often impractical due to long disease horizons and substantial inter-patient heterogeneity. To address these constraints, we present the Alzheimer's Learning Platform for Adaptive Care Agents (ALPACA), an open-source, Gym-compatible reinforcement learning (RL) environment for systematically exploring personalized treatment strategies using existing therapies. ALPACA is powered by the Continuous Action-conditioned State Transitions (CAST) model trained on longitudinal trajectories from the Alzheimer's Disease Neuroimaging Initiative (ADNI), enabling medication-conditioned simulation of disease progression under alternative treatment decisions. We show that CAST autoregressively generates realistic medication-conditioned trajectories and that RL policies trained in ALPACA outperform no-treatment and behavior-cloned clinician baselines on memory-related outcomes. Interpretability analyses further indicated that the learned policies relied on clinically meaningful patient features when selecting actions. Overall, ALPACA provides a reusable in silico testbed for studying individualized sequential treatment decision-making for AD.

Figures & tables

Figure 1: The ALPACA environment wraps the CAST model. Given a current state $s_{t}$ and medical action $a_{t}$ , the CAST model predicts the future state $s_{t+1}$ ( $t_{\Delta}$ months ahead), with $r_{t+1}$ representing the resulting reward.

Overview
Characteristic	Full Dataset
Visits	12,984
Unique subjects	1,905
Demographics
Age (mean $\pm$ SD)	76.2 $\pm$ 7.4
Range (yrs)	55.0–103.1

Table 1: Demographic and Trajectory Summary

Figure 2: Policy performance over 1000 simulated rollouts (mean and 95% CI across patients).

Policy	Cumulative Reward	Per-step Reward	Final ADNI-Mem
No Medication (baseline)	$-17.81$	$-0.95$	$-1.53$
Clinician	$-13.72$	$-0.73$	$-1.36$
Heuristic	$-15.10$	$-0.79$	$-1.42$
A2C	$-0.66$	$-0.06$	$-0.59$
PPO	$\mathbf{3.38}$	$\mathbf{0.27}$	$\mathbf{-0.46}$
SAC	$0.18$	$0.01$	$-0.76$

Table 2: Performance of reinforcement learning agents, no-treatment and behavior cloned clinician baselines evaluated over 1000 simulated patient rollouts. Reported values are means of reward values calculated across rollouts.

Figure 3: Aggregated SHAP values from all models (PPO, SAC, A2C, and BDQ) for the No Medication and AD Treatment actions. The red values indicate higher feature values, and higher SHAP scores indicate a greater propensity to act.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Drug	Mapped Class
Aricept, Donepezil, Namenda, Exelon	AD Treatment
Lipitor, Simvastatin, Crestor, Zocor, Atorvastatin	Statin
Lisinopril, Atenolol, Amlodipine, Metoprolol, Norvasc, Losartan	Antihypertensive
Levothyroxine, Synthroid	Thyroid Hormone
Aspirin, Ibuprofen, Aleve, ASA	NSAID
Tylenol, Acetaminophen	Analgesic

Appendix

Table 3: Drug–class mapping used in ALPACA.

Alzheimer's Disease

Alzheimer’s disease research studies its biology, detection, progression, and care using clinical, molecular, and brain-imaging evidence.

See all 51 papers

A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning
May 31, 2026Yuepeng Wang, Ken Kawano, Yoshihiko Fujisawa +12RL Benchmarks Reinforcement Learning
Tokyo University of Agriculture and Technology · Institute of Science Tokyo · National University of Singapore +5
Learning Clinical-Trial Strategy: Offline Policy Training for Decision Agents
Aug 4, 2026William Bolton, Philip TorrClinical Decision-Making Imitation Learning
Department of Engineering Science, University of Oxford, Oxford, United Kingdom.
MEMOR-E: In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics
Apr 28, 2026Maissa Abir Smaili, Eren Sadikoglu, Ransalu SenanayakeAlzheimer's Disease LLM Personalization
Istanbul Medipol University Istanbul, Türkiye · Arizona State University Tempe, USA

May 31, 2026J/K move · Enter open · S save

A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning

Yuepeng Wang, Ken Kawano, Yoshihiko Fujisawa, Yongqi Zhou+11

Tokyo University of Agriculture and Technology · Institute of Science Tokyo · National University of Singapore+5

RL Benchmarks Reinforcement Learning Clinical Decision Support Continuous-Time RL Safe RL

Open paper AskPDF arXiv

Medical treatment recommendation poses several challenges to reinforcement learning (RL): patient physiology evolves in continuous time, measurements and interventions are performed at irregular intervals, and treatment effects vary substantially across individuals. Existing RL formulations and simulated environments, however, are based on discrete-time MDPs with fixed decision intervals. Thus, it remains difficult to evaluate whether RL methods can handle time-interval-dependent disease progression, personalized treatment response, and safety between consecutive measurement points. To address this gap, we introduce MedGym, a benchmark environment for dynamic treatment recommendation. MedGym models longitudinal patient evolution in a continuous-time framework and constructs a configurable medical RL benchmark from clinical data by using Physics-Informed Neural Networks. The resulting benchmark enables direct comparison between discrete-time and continuous-time methods under irregular treatment timing and patient-specific dynamics. Furthermore, MedGym supports evaluation from clinically important perspectives, such as personalization and trajectory-level safety. By providing a standardized and configurable benchmark for continuous-time dynamic treatment, MedGym enables more realistic and informative evaluation of medical RL methods.

Abstract

Alzheimer's Disease

Explore similar work

A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning

Learning Clinical-Trial Strategy: Offline Policy Training for Decision Agents

MEMOR-E: In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics