Evaluating Type-1 Diabetes Digital Twins by Decision Quality

Victor Li; Owen Tucker; Michael S. Hughes; Temiloluwa Prioleau; Shengpu Tang

cs.LGOct 22, 2026

Evaluating Type-1 Diabetes Digital Twins by Decision Quality

Authors: Victor Li, Owen Tucker, Michael S. Hughes, Temiloluwa Prioleau, Shengpu Tang

Organizations: Department of Computer Science, Emory University · School of Medicine, Emory University

Abstract

A digital twin (DT) for type 1 diabetes (T1D) can predict how an individual's blood glucose will respond to different insulin treatments without actually trying them. For each treatment, downstream statistics of predicted glucose trajectories, such as glycemic risk, can be computed to guide clinical decisions. However, the quality of DTs is usually measured by the accuracy of their predicted glucose trajectories, which may not reflect the quality of the downstream treatment decisions they inform. In this work, we directly evaluate DTs by this decision quality on downstream treatment ranking, i.e., whether a DT predicts glucose trajectories that can be used to rank candidate treatments as the real patient would. We present an evaluation protocol over 30 virtual patients from the FDA-accepted UVA/Padova model, each simulated under 21 different basal-bolus treatments. For each patient, a DT is fitted under one treatment and then produces glucose trajectories for the remaining 20 treatments. Our results show that prediction accuracy and decision quality disagree. A baseline that models the average patient, never seeing the individual's glucose data, has the lowest prediction accuracy of all six methods (RMSE: 63.6 mg/dL) but ranks treatments with the second-highest mean Spearman (0.766), while a black-box neural network that predicts glucose nearly twice as accurately (RMSE: 32.9 mg/dL) ranks them no better than a random ordering of the 20 treatments (0.049 Spearman). Therefore, selecting a DT based on prediction accuracy alone can prefer one that recommends treatments at chance, and decision quality must be evaluated directly rather than assumed to follow from prediction accuracy.

Figures & tables

Prediction accuracy	Decision quality
RMSE $\dfrac{1}{\|\Pi\|}\sum\limits_{\pi\in\Pi}\sqrt{\dfrac{1}{T}\sum\limits_{k=1}^{T}\Big(\widehat{\mathrm{IG}}^{p,\pi}_{k}-\mathrm{IG}^{p,\pi}_{k}\Big)^{2}}$ MARD $\dfrac{100}{\|\Pi\|}\sum\limits_{\pi\in\Pi}\dfrac{1}{T}\sum\limits_{k=1}^{T}\dfrac{\big\|\widehat{\mathrm{IG}}^{p,\pi}_{k}-\mathrm{IG}^{p,\pi}_{k}\big\|}{\mathrm{IG}^{p,\pi}_{k}}$	Spearman $\mathrm{corr}\big(\mathrm{rank}(R(\mathrm{IG}^{p,\pi})),\mathrm{rank}(R(\widehat{\mathrm{IG}}^{p,\pi}))\big)_{\pi\in\Pi}$ Regret $\max\limits_{\pi\in\Pi}R(\mathrm{IG}^{p,\pi})-R(\mathrm{IG}^{p,\widehat{\pi}})$ Regret Norm $\text{Regret}\big/\big(\max\limits_{\pi}R(\mathrm{IG}^{p,\pi})-\min\limits_{\pi}R(\mathrm{IG}^{p,\pi})\big)$

Table 1: Metric definitions.

T

is the number of samples and

\widehat{\pi}=\arg\max_{\pi\in\Pi}R(\widehat{\text{IG}}^{p,\pi})

Component	Choice	Source
Simulator	simglucose (UVA/Padova), 1 min dopri5 integration	simglucose default [ 22 ]
CGM	Breton–Kovatchev noise model, 5 min interval $\Rightarrow$ 288 samples per 24 h day	simglucose GuardianRT sensor [ 22 ]
Subjects	30 built-in virtual patients: 10 adults, 10 adolescents, 10 children	simglucose default [ 22 ]
Meals	RandomScenario , one independent draw per day (Appendix A.1 )	simglucose default [ 22 ]
Bolus formula	BBController meal bolus with a correction above 150 mg/dL (Equation 1 )	simglucose default [ 22 ]
Correction bolus	computed from noise-free IG rather than CGM, so delivered insulin is deterministic given the patient, meals and treatment	ours; ReplayBG [ 5 ] instead replays a fixed insulin record

Table 2: Benchmark summary. The last column names the source of each choice, either a simglucose default, a published protocol, or our own design.

	Prediction Accuracy		Decision Quality
Method	RMSE (mg/dL)	MARD (%)	Spearman	Regret	Regret Norm	$\Delta$ rank
MCMC	$23.0\pm 9.1$ (1)	$21.0\pm 16.2$	$0.861\pm 0.111$ (1)	$370\pm 470$	$0.083\pm 0.113$	$0$
SBI	$29.2\pm 11.3$ (2)	$32.4\pm 31.5$	$0.728\pm 0.368$ (3)	$584\pm 858$	$0.137\pm 0.208$	$-1$
Neural	$32.9\pm 9.0$ (3)	$31.5\pm 17.6$	$0.049\pm 0.466$ (5)	$3222\pm 2888$	$0.541\pm 0.327$	$\mathbf{-2}$
UKF	$35.6\pm 20.6$ (4)	$42.5\pm 55.3$	$0.492\pm 0.614$ (4)	$1297\pm 1979$	$0.215\pm 0.278$	$0$
Linear	$36.6\pm 8.7$ (5)	$36.8\pm 23.5$	$-0.207\pm 0.524$ (6)	$4068\pm 3801$	$0.654\pm 0.342$	$-1$

Table 3: Results over the full cohort, ordered by RMSE. Parenthesized numbers are ranks on each family’s primary metric, and

\Delta

rank is the move from the accuracy rank to the decision rank, positive meaning better on decisions than on accuracy.

Metric	MCMC	SBI	UKF	Linear	Neural	Prior	Friedman $p$
RMSE	1.47	2.67	3.37	4.60	3.37	5.53	$<10^{-4}$
MARD	1.57	2.83	3.27	4.63	3.33	5.37	$<10^{-4}$
Spearman	1.87	2.57	3.37	5.27	4.83	3.10	$<10^{-4}$
Regret	2.10	2.77	3.03	4.88	4.88	3.33	$<10^{-4}$
Regret Norm	2.07	2.77	3.00	4.92	4.92	3.33	$<10^{-4}$

Table 4: Omnibus Friedman test. Mean within-patient rank (

n=30

patients,

k=6

methods, rank 1 = best) and the omnibus Friedman

p

per metric.

Figure 1: Critical difference diagrams for RMSE and Spearman. Each method sits at its mean rank over the 30 patients (1 is best). A bar joins methods that the Holm-corrected pairwise Wilcoxon tests of Table 7 cannot separate at

\alpha=0.05

. A filled circle marks a member of the group, and an open circle marks a method the bar passes over but does not include.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Horizon	24 h starting at 5am after a steady-state start at midnight
CGM Interval	5 min $\Rightarrow$ 288 samples per test day
Simglucose Default RandomScenario
Main Meals	breakfast / lunch / dinner at $p=0.95$ , mealtimes at $\mathcal{N}$ (07:00, 12:00, 18:00; $\sigma=60$ min) truncated to $[05,09]$ , $[10,14]$ , $[16,20]$ , amounts of $\mathcal{N}$ (45g, 70g, 80g; $\sigma=10$ g)
Snacks	three at $p=0.30$ , mealtimes at $\mathcal{N}$ (09:30, 15:00, 21:30; $\sigma=30$ min), amounts of $\mathcal{N}$ (10g; $\sigma=5$ g)
Measured	1–6 eating occasions per day, $196\pm 30$ g/day

Appendix

Table 5: Scenario Table.

p

is the probability of a given meal or snack occurring.

	MCMC	SBI	UKF	Prior	Neural	Linear
Cost per patient	8 h 18 m	245 s	181 s	175 s	2.4 s	0.1 s

Appendix

Table 6: Per-patient fit-and-score cost on one core, median over the 30 subjects.

Metric	A vs B	$\Delta$ (A $-$ B)	raw $p$	Holm $p$	significant at $0.05$
RMSE	MCMC vs SBI	$-6.25$	$<10^{-4}$	$0.0002$	yes
	MCMC vs UKF	$-12.61$	$<10^{-4}$	$<10^{-4}$	yes
	MCMC vs Linear	$-13.58$	$<10^{-4}$	$<10^{-4}$	yes
	MCMC vs Neural	$-9.97$	$<10^{-4}$	$<10^{-4}$	yes
	MCMC vs Prior	$-40.62$	$<10^{-4}$	$<10^{-4}$	yes
	SBI vs UKF	$-6.36$	$0.0262$	$0.0787$	no

Appendix

Table 7: Pairwise Wilcoxon signed-rank tests with Holm correction within each metric.

\Delta

is the mean within-patient difference.

Digital Twins

Digital twins are virtual models synchronized with real systems to support monitoring, prediction, simulation, and decision-making.

See all 253 papers

$\text{DT}^2$ : Decision-Targeted Digital Twins
Jun 24, 2026Harry Amad, Mihaela van der SchaarDecision-Focused Learning Digital Twins
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom.
MetaboNet-Bench: A Multi-modal Benchmark for Glucose Forecasting in Type 1 Diabetes
Jun 17, 2026Nathaniel Jeffries, Miriam Wolff, Sam Royston +5Multivariate Time Series Forecasting Type 1 Diabetes
Department of Genetics, Stanford University School of Medicine, Stanford, CA, USA · Replica Health, New York, NY, USA · Boston Children’s Hospital, Harvard Medical School, Boston, MA, USA +1
PhysioSeq2Seq: A Hybrid Physiological Digital Twin and Sequence-to-Sequence LSTM for Long-Horizon Glucose Forecasting in Type 1 Diabetes
May 16, 2026Phat Tran, Neville Mehta, Clara Mosquera-Lopez +3Long-Term Time Series Forecasting Type 1 Diabetes
Oregon State University · Oregon Health & Science University

Jun 24, 2026J/K move · Enter open · S save

$\text{DT}^2$ : Decision-Targeted Digital Twins

Harry Amad, Mihaela van der Schaar

Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom.

Decision-Focused Learning Digital Twins

Open paper AskPDF arXiv

A digital twin (DT) is a virtual model of a real-world system that can assist decision-making by simulating scenarios induced by different policies. However, typical machine learning-based DTs do not optimise for this use case. We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function. We further show that this holds empirically, even with expressive model classes. To address this, we introduce

\text{DT}^2

, a decision-targeted DT training paradigm. Firstly,

\text{DT}^2

uses fitted Q-evaluation to estimate values of candidate policies from offline data. A DT is then trained to generate rollouts that preserve pairwise policy rankings derived from these proxy ground-truth values with an architecture-agnostic loss function. We empirically demonstrate the efficacy of our method across a range of settings and architectures.

\text{DT}^2

consistently improves policy ranking and reduces decision regret during policy selection relative to conventional DT training, both for policies used during training and for unseen policies, while maintaining a good level of raw simulation fidelity.

Abstract

Digital Twins

Explore similar work

DT2\text{DT}^2DT2: Decision-Targeted Digital Twins

MetaboNet-Bench: A Multi-modal Benchmark for Glucose Forecasting in Type 1 Diabetes

PhysioSeq2Seq: A Hybrid Physiological Digital Twin and Sequence-to-Sequence LSTM for Long-Horizon Glucose Forecasting in Type 1 Diabetes

$\text{DT}^2$ : Decision-Targeted Digital Twins