Optimizing Resource-Constrained Non-Pharmaceutical Interventions for Multi-Cluster Outbreak Control Using Hierarchical Reinforcement Learning

Xueqiao Peng; Andrew Perrault

cs.LGMar 19, 2026

Optimizing Resource-Constrained Non-Pharmaceutical Interventions for Multi-Cluster Outbreak Control Using Hierarchical Reinforcement Learning

Authors: Xueqiao Peng, Andrew Perrault

Organizations: The Ohio State University

Abstract

Non-pharmaceutical interventions (NPIs), such as diagnostic testing and quarantine, are crucial for controlling infectious disease outbreaks but are often constrained by limited resources, particularly in early outbreak stages. In real-world public health settings, resources must be allocated across multiple outbreak clusters that emerge asynchronously, vary in size and risk, and compete for a shared resource budget. Here, a cluster corresponds to a group of close contacts generated by a single infected index case. Thus, decisions must be made under uncertainty and heterogeneous demands, while respecting operational constraints. We formulate this problem as a constrained restless multi-armed bandit and propose a hierarchical reinforcement learning framework. A global controller learns a continuous action cost multiplier that adjusts global resource demand, while a generalized local policy estimates the marginal value of allocating resources to individuals within each cluster. We evaluate the proposed framework in a realistic agent-based simulator of SARS-CoV-2 with dynamically arriving clusters. Across a wide range of system scales and testing budgets, our method consistently outperforms RMAB-inspired and heuristic baselines, improving outbreak control effectiveness by 20%-30%. Experiments on up to 40 concurrently active clusters further demonstrate that the hierarchical framework is highly scalable and enables faster decision-making than the RMAB-inspired method.

Figures & tables

Figure 1: Overview of the proposed hierarchical RL framework for multi-cluster outbreak control. A global PPO controller adjusts a shared testing cost multiplier, which modulates the perceived test cost used by a pretrained generalized DQN applied to each active cluster. Based on these cost-conditioned local value estimates, a global Q-ranking policy selects testing actions across clusters while strictly enforcing the hard global budget constraint.

	$\alpha_{3}{=}0.01$	$\alpha_{3}{=}0.05$	$\alpha_{3}{=}0.08$	$\alpha_{3}{=}0.10$
Fixed-DQN	$-0.23{\pm}0.02$	$-0.33{\pm}0.02$	$-0.42{\pm}0.02$	$-0.50{\pm}0.04$
Gen-DQN	$-0.25{\pm}0.02$	$-0.34{\pm}0.02$	$-0.41{\pm}0.02$	$-0.47{\pm}0.03$

Table 1: Generalized vs. Fixed DQN performance under varying test penalty cost (

\alpha_{3}

). Results indicate that the generalized policy achieves comparable performance, demonstrating adaptability to different test penalties.

Method	$(\#C,\#B)$	S1	S2	S3
Symp-AvgRand	$(20,40)$	$1.64{\pm}0.04$	$0.44{\pm}0.02$	$12.54{\pm}0.09$
Thres-AvgRand		$0.33{\pm}0.01$	$1.59{\pm}0.03$	$12.54{\pm}0.09$
Thres-SizeRand		$0.39{\pm}0.01$	$1.58{\pm}0.03$	$9.33{\pm}0.06$
Fixed- $M$ -QR		$0.35{\pm}0.01$	$1.02{\pm}0.02$	$3.45{\pm}0.05$
Bin- $M$ -QR		$0.36{\pm}0.01$	$0.99{\pm}0.02$	$3.35{\pm}0.04$
Hier-PPO		$0.30{\pm}0.04$	$0.95{\pm}0.02$	$3.41{\pm}0.06$

Table 3: Comparison of different methods under asynchronous settings for

\#C{=}20

with two global budgets. Results are reported for metrics

S_{1}

S_{2}

, and

S_{3}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Incubation period	Lognormal (mean=1.57 days, std=0.65 days)
Infectious period	7 days (from 2 days before to 5 days after symptom onset)
Baseline transmission probability	0.03 per contact
Probability symptomatic given infected	0.8
Probability symptomatic without infection	0.01 per day
Probability asymptomatic infection	0.2

Appendix

Table A.1 : Key epidemiological parameters used in the agent-based SARS-CoV-2 simulation. Unless otherwise stated, all parameters follow the configuration reported in Peng 2023

Asynchronous activation
$(\#C,\#B)$	$m=0.5$	$m=1.0$	$m=1.5$	$m=2.0$
$(20,40)$	$-0.62{\pm}0.01$	$-0.65{\pm}0.02$	$-0.69{\pm}0.01$	$-0.70{\pm}0.01$
$(20,400)$	$-0.64{\pm}0.01$	$-0.68{\pm}0.01$	$-0.71{\pm}0.01$	$-0.73{\pm}0.01$
Synchronous activation
$(\#C,\#B)$	$m=0.5$	$m=1.0$	$m=1.5$	$m=2.0$
$(20,40)$	$-0.33{\pm}0.01$	$-0.37{\pm}0.01$	$-0.35{\pm}0.01$	$-0.38{\pm}0.01$

Appendix

Table A.2 : Ablation of fixed-

m

Q-ranking under asynchronous and synchronous activation.

Panel of Figure 6 — Figure A.1 : Sensitivity of testing behavior to the test-cost coefficient $\alpha_{3}$ . We report the average number of tests per timestep across different cluster sizes. The gradient-penalty variant exhibits more stable and coherent monotonic responses to increasing $\alpha_{3}$ than the variant trained without the penalty.

#C	#B	Bin-M-QR (ms)	Hier-PPO (ms)	Speedup
10	20	$7.18{\pm}0.47$	$1.03{\pm}0.12$	$6.97\times$
10	100	$3.16{\pm}0.56$	$0.61{\pm}0.08$	$5.18\times$
20	40	$12.39{\pm}0.52$	$2.55{\pm}0.11$	$4.86\times$
20	200	$4.44{\pm}0.48$	$0.78{\pm}0.15$	$5.69\times$
40	80	$16.21{\pm}0.59$	$2.64{\pm}0.32$	$6.14\times$
40	400	$5.87{\pm}0.64$	$0.82{\pm}0.23$	$7.16\times$

Appendix

Table A.3: Decision-time runtime comparison under different numbers of clusters. To isolate the effect of cluster count on computational cost, we fix the cluster size to 20 and scale the global testing budget linearly with the number of clusters. Reported values correspond to the average wall-clock decision time per timestep, excluding environment simulation, and include all policy evaluations required for feasibility enforcement.

	$\alpha_{2}{=}0.01$	$\alpha_{2}{=}0.1$	$\alpha_{2}{=}0.3$	$\alpha_{2}{=}0.5$
Fixed-DQN	$-0.39{\pm}0.01$	$-0.57{\pm}0.02$	$-0.67{\pm}0.02$	$-0.83{\pm}0.03$
Gen-DQN	$-0.41{\pm}0.03$	$-0.57{\pm}0.03$	$-0.66{\pm}0.04$	$-0.86{\pm}0.06$

Appendix

Table A.4: Generalized vs. Fixed DQN performance under varying quarantine cost (

\alpha_{2}

), with fixed test cost (

\alpha_{3}

= 0.05). Results indicate that the generalized policy achieves comparable performance.

Table A.5 : Performance comparison of various test allocation methods under synchronous and asynchronous cluster activation environments. Results highlight the advantage of the proposed hierarchical allocation PPO (Hier-PPO) method across different cluster and budget settings. (#C: number of clusters, #B: daily testing budget)

Healthcare

Healthcare AI develops and evaluates models and systems for clinical care, medical research, public health, and healthcare operations.

See all 608 papers

Empowering Epidemic Response: The Role of Reinforcement Learning in Infectious Disease Control
Mar 26, 2026Mutong Liu, Yang Liu, Jiming LiuReinforcement Learning Healthcare
Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China
Neetyabhas: A Framework for Uncertainty-Aware Public Policy Optimization in Rational Agent-Based Models
Jun 3, 2026Janani Venugopalan, Gaurav Deshkar, Rishabh Gaur +2Policy Optimization Agent-Based Modeling
Engineering for Research (e4r), Thoughtworks Technologies, Binarius Building, Deepak Complex, Shastrinagar, Yerawada, Pune, Maharashtra, India - 411006
Optimizing Interventions for Agent-Based Infectious Disease Simulations
Apr 2, 2026Anja Wolpers, Johannes Ponge, Adelinde M. UhrmacherAgent-Based Modeling
University of Rostock, Rostock, Germany · University of Münster, Münster, Germany*

Mar 26, 2026J/K move · Enter open · S save

Empowering Epidemic Response: The Role of Reinforcement Learning in Infectious Disease Control

Mutong Liu, Yang Liu, Jiming Liu

Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China

Reinforcement Learning Healthcare RL Control

Open paper AskPDF arXiv

Reinforcement learning (RL), owing to its adaptability to various dynamic systems in many real-world scenarios and the capability of maximizing long-term outcomes under different constraints, has been used in infectious disease control to optimize the intervention strategies for controlling infectious disease spread and responding to outbreaks in recent years. The potential of RL for assisting public health sectors in preventing and controlling infectious diseases is gradually emerging and being explored by rapidly increasing publications relevant to COVID-19 and other infectious diseases. However, few surveys exclusively discuss this topic, that is, the development and application of RL approaches for optimizing strategies of non-pharmaceutical and pharmaceutical interventions of public health. Therefore, this paper aims to provide a concise review and discussion of the latest literature on how RL approaches have been used to assist in controlling the spread and outbreaks of infectious diseases, covering several critical topics addressing public health demands: resource allocation, balancing between lives and livelihoods, mixed policy of multiple interventions, and inter-regional coordinated control. Finally, we conclude the paper with a discussion of several potential directions for future research.

Abstract

Healthcare

Explore similar work

Empowering Epidemic Response: The Role of Reinforcement Learning in Infectious Disease Control

Neetyabhas: A Framework for Uncertainty-Aware Public Policy Optimization in Rational Agent-Based Models

Optimizing Interventions for Agent-Based Infectious Disease Simulations