Optimizing Resource-Constrained Non-Pharmaceutical Interventions for Multi-Cluster Outbreak Control Using Hierarchical Reinforcement Learning

Xueqiao Peng; Andrew Perrault

cs.LGMar 19, 2026

Optimizing Resource-Constrained Non-Pharmaceutical Interventions for Multi-Cluster Outbreak Control Using Hierarchical Reinforcement Learning

Authors: Xueqiao Peng, Andrew Perrault

Organizations: The Ohio State University

Abstract

Non-pharmaceutical interventions (NPIs), such as diagnostic testing and quarantine, are crucial for controlling infectious disease outbreaks but are often constrained by limited resources, particularly in early outbreak stages. In real-world public health settings, resources must be allocated across multiple outbreak clusters that emerge asynchronously, vary in size and risk, and compete for a shared resource budget. Here, a cluster corresponds to a group of close contacts generated by a single infected index case. Thus, decisions must be made under uncertainty and heterogeneous demands, while respecting operational constraints. We formulate this problem as a constrained restless multi-armed bandit and propose a hierarchical reinforcement learning framework. A global controller learns a continuous action cost multiplier that adjusts global resource demand, while a generalized local policy estimates the marginal value of allocating resources to individuals within each cluster. We evaluate the proposed framework in a realistic agent-based simulator of SARS-CoV-2 with dynamically arriving clusters. Across a wide range of system scales and testing budgets, our method consistently outperforms RMAB-inspired and heuristic baselines, improving outbreak control effectiveness by 20%-30%. Experiments on up to 40 concurrently active clusters further demonstrate that the hierarchical framework is highly scalable and enables faster decision-making than the RMAB-inspired method.

Figures & tables

Figure 1: Overview of the proposed hierarchical RL framework for multi-cluster outbreak control. A global PPO controller adjusts a shared testing cost multiplier, which modulates the perceived test cost used by a pretrained generalized DQN applied to each active cluster. Based on these cost-conditioned local value estimates, a global Q-ranking policy selects testing actions across clusters while strictly enforcing the hard global budget constraint.

	$\alpha_{3}{=}0.01$	$\alpha_{3}{=}0.05$	$\alpha_{3}{=}0.08$	$\alpha_{3}{=}0.10$
Fixed-DQN	$-0.23{\pm}0.02$	$-0.33{\pm}0.02$	$-0.42{\pm}0.02$	$-0.50{\pm}0.04$
Gen-DQN	$-0.25{\pm}0.02$	$-0.34{\pm}0.02$	$-0.41{\pm}0.02$	$-0.47{\pm}0.03$

Table 1: Generalized vs. Fixed DQN performance under varying test penalty cost (

\alpha_{3}

). Results indicate that the generalized policy achieves comparable performance, demonstrating adaptability to different test penalties.

Method	$(\#C,\#B)$	S1	S2	S3
Symp-AvgRand	$(20,40)$	$1.64{\pm}0.04$	$0.44{\pm}0.02$	$12.54{\pm}0.09$
Thres-AvgRand		$0.33{\pm}0.01$	$1.59{\pm}0.03$	$12.54{\pm}0.09$
Thres-SizeRand		$0.39{\pm}0.01$	$1.58{\pm}0.03$	$9.33{\pm}0.06$
Fixed- $M$ -QR		$0.35{\pm}0.01$	$1.02{\pm}0.02$	$3.45{\pm}0.05$
Bin- $M$ -QR		$0.36{\pm}0.01$	$0.99{\pm}0.02$	$3.35{\pm}0.04$
Hier-PPO		$0.30{\pm}0.04$	$0.95{\pm}0.02$	$3.41{\pm}0.06$

Table 3: Comparison of different methods under asynchronous settings for

\#C{=}20

with two global budgets. Results are reported for metrics

S_{1}

S_{2}

, and

S_{3}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Incubation period	Lognormal (mean=1.57 days, std=0.65 days)
Infectious period	7 days (from 2 days before to 5 days after symptom onset)
Baseline transmission probability	0.03 per contact
Probability symptomatic given infected	0.8
Probability symptomatic without infection	0.01 per day
Probability asymptomatic infection	0.2

Appendix

Table A.1 : Key epidemiological parameters used in the agent-based SARS-CoV-2 simulation. Unless otherwise stated, all parameters follow the configuration reported in Peng 2023

Asynchronous activation
$(\#C,\#B)$	$m=0.5$	$m=1.0$	$m=1.5$	$m=2.0$
$(20,40)$	$-0.62{\pm}0.01$	$-0.65{\pm}0.02$	$-0.69{\pm}0.01$	$-0.70{\pm}0.01$
$(20,400)$	$-0.64{\pm}0.01$	$-0.68{\pm}0.01$	$-0.71{\pm}0.01$	$-0.73{\pm}0.01$
Synchronous activation
$(\#C,\#B)$	$m=0.5$	$m=1.0$	$m=1.5$	$m=2.0$
$(20,40)$	$-0.33{\pm}0.01$	$-0.37{\pm}0.01$	$-0.35{\pm}0.01$	$-0.38{\pm}0.01$

Appendix

Table A.2 : Ablation of fixed-

m

Q-ranking under asynchronous and synchronous activation.

Panel of Figure 6 — Figure A.1 : Sensitivity of testing behavior to the test-cost coefficient $\alpha_{3}$ . We report the average number of tests per timestep across different cluster sizes. The gradient-penalty variant exhibits more stable and coherent monotonic responses to increasing $\alpha_{3}$ than the variant trained without the penalty.

#C	#B	Bin-M-QR (ms)	Hier-PPO (ms)	Speedup
10	20	$7.18{\pm}0.47$	$1.03{\pm}0.12$	$6.97\times$
10	100	$3.16{\pm}0.56$	$0.61{\pm}0.08$	$5.18\times$
20	40	$12.39{\pm}0.52$	$2.55{\pm}0.11$	$4.86\times$
20	200	$4.44{\pm}0.48$	$0.78{\pm}0.15$	$5.69\times$
40	80	$16.21{\pm}0.59$	$2.64{\pm}0.32$	$6.14\times$
40	400	$5.87{\pm}0.64$	$0.82{\pm}0.23$	$7.16\times$

Appendix

Table A.3: Decision-time runtime comparison under different numbers of clusters. To isolate the effect of cluster count on computational cost, we fix the cluster size to 20 and scale the global testing budget linearly with the number of clusters. Reported values correspond to the average wall-clock decision time per timestep, excluding environment simulation, and include all policy evaluations required for feasibility enforcement.

	$\alpha_{2}{=}0.01$	$\alpha_{2}{=}0.1$	$\alpha_{2}{=}0.3$	$\alpha_{2}{=}0.5$
Fixed-DQN	$-0.39{\pm}0.01$	$-0.57{\pm}0.02$	$-0.67{\pm}0.02$	$-0.83{\pm}0.03$
Gen-DQN	$-0.41{\pm}0.03$	$-0.57{\pm}0.03$	$-0.66{\pm}0.04$	$-0.86{\pm}0.06$

Appendix

Table A.4: Generalized vs. Fixed DQN performance under varying quarantine cost (

\alpha_{2}

), with fixed test cost (

\alpha_{3}

= 0.05). Results indicate that the generalized policy achieves comparable performance.

Table A.5 : Performance comparison of various test allocation methods under synchronous and asynchronous cluster activation environments. Results highlight the advantage of the proposed hierarchical allocation PPO (Hier-PPO) method across different cluster and budget settings. (#C: number of clusters, #B: daily testing budget)

Healthcare

Healthcare AI develops and evaluates models and systems for clinical care, medical research, public health, and healthcare operations.

See all 608 papers

Mar 26, 2026

Empowering Epidemic Response: The Role of Reinforcement Learning in Infectious Disease Control

Reinforcement learning (RL), owing to its adaptability to various dynamic systems in many real-world scenarios and the capability of maximizing long-term outcomes under different constraints, has been used in infectious disease control to optimize the intervention strategies for controlling infectious disease spread and responding to outbreaks in recent years. The potential of RL for assisting public health sectors in preventing and controlling infectious diseases is gradually emerging and being explored by rapidly increasing publications relevant to COVID-19 and other infectious diseases. However, few surveys exclusively discuss this topic, that is, the development and application of RL approaches for optimizing strategies of non-pharmaceutical and pharmaceutical interventions of public health. Therefore, this paper aims to provide a concise review and discussion of the latest literature on how RL approaches have been used to assist in controlling the spread and outbreaks of infectious diseases, covering several critical topics addressing public health demands: resource allocation, balancing between lives and livelihoods, mixed policy of multiple interventions, and inter-regional coordinated control. Finally, we conclude the paper with a discussion of several potential directions for future research.

Mutong Liu, Yang Liu, Jiming Liu

Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China

Reinforcement Learning Healthcare RL Control

Ask PDF

Jun 3, 2026

Neetyabhas: A Framework for Uncertainty-Aware Public Policy Optimization in Rational Agent-Based Models

Purpose The WHO's COVID-19 non-pharmaceutical interventions (e.g., lockdowns, vaccinations) effectively curb transmission but impose heavy economic strains. Existing research often neglects individual behaviors and falsely assumes perfect infection tracking and flawless policy execution, failing to account for real-world uncertainties and errors. Methods We propose an integrative approach incorporating uncertainties in both epidemic measurement (infections/hospitalizations) and policy implementation. We built a simulation model of 1,000 individuals making real-time choices regarding mask-wearing, vaccination, and shopping. Concurrently, policymakers deploy interventions (lockdowns, mandates) based on health and economic observations. This framework is driven by hierarchical reinforcement learning agents, utilizing deep Q-networks alongside uncertainty-aware policy gradient variants (DDPG and TD3). Results The simulations effectively managed the epidemic's progression. Masking and vaccinations proved highly effective, significantly reducing both the outbreak's peak height and duration. By integrating individual behaviors, policy uncertainties, and multifaceted interventions, our dynamic control approach successfully mitigated the epidemic's impact. Conclusions Our model overcomes previous research limitations by embedding uncertainty and human behavior into public health policy frameworks. The simulation demonstrates that accounting for individual choices and imperfect data is crucial for designing effective interventions during complex pandemics, with masks and vaccines serving as pivotal tools.

Janani Venugopalan, Gaurav Deshkar, Rishabh Gaur +2

Engineering for Research (e4r), Thoughtworks Technologies, Binarius Building, Deepak Complex, Shastrinagar, Yerawada, Pune, Maharashtra, India - 411006

Policy Optimization Agent-Based Modeling

Ask PDF

Apr 2, 2026

Optimizing Interventions for Agent-Based Infectious Disease Simulations

Non-pharmaceutical interventions (NPIs) are commonly used tools for controlling infectious disease transmission when pharmaceutical options are unavailable. Yet, identifying effective interventions that minimize societal disruption remains challenging. Agent-based simulation is a popular tool for analyzing the impact of possible interventions in epidemiology. However, automatically optimizing NPIs using agent-based simulations poses a complex problem because, in agent-based epidemiological models, interventions can target individuals based on multiple attributes, affect hierarchical group structures (e.g., schools, workplaces, and families), and be combined arbitrarily, resulting in a very large or even infinite search space. We aim to support decision-makers with our Agent-based Infectious Disease Intervention Optimization System (ADIOS) that optimizes NPIs for infectious disease simulations using Grammar-Guided Genetic Programming (GGGP). The core of ADIOS is a domain-specific language for expressing NPIs in agent-based simulations that structures the intervention search space through a context-free grammar. To make optimization more efficient, the search space can be further reduced by defining constraints that prevent the generation of semantically invalid intervention patterns. Using this constrained language and an interface that enables coupling with agent-based simulations, ADIOS adopts the GGGP approach for simulation-based optimization. Using the German Epidemic Micro-Simulation System (GEMS) as a case study, we demonstrate the potential of our approach to generate optimal interventions for realistic epidemiological models

Anja Wolpers, Johannes Ponge, Adelinde M. Uhrmacher

University of Rostock, Rostock, Germany · University of Münster, Münster, Germany*

Agent-Based Modeling

Ask PDF

Abstract

Healthcare

Explore similar work

Empowering Epidemic Response: The Role of Reinforcement Learning in Infectious Disease Control

Neetyabhas: A Framework for Uncertainty-Aware Public Policy Optimization in Rational Agent-Based Models

Optimizing Interventions for Agent-Based Infectious Disease Simulations