FairGlucose: A CGM Fairness Benchmark Reveals Subgroup Disparities Hidden in Population-Level Validation

Junjie Luo; Xuzhe Zhi; Rui Han; Abhimanyu Kumbara; Anand K. Iyer; Mansur E. Shomali; Ritu Agarwal; Guodong Gordon Gao

cs.CYAug 18, 2026

FairGlucose: A CGM Fairness Benchmark Reveals Subgroup Disparities Hidden in Population-Level Validation

Authors: Junjie Luo, Xuzhe Zhi, Rui Han, Abhimanyu Kumbara, Anand K. Iyer, Mansur E. Shomali, Ritu Agarwal, Guodong Gordon Gao

Organizations: Carey Business School Johns Hopkins University Baltimore, MD, USA · Welldoc, Inc. Columbia, MD, USA

Abstract

As CGM-based AI tools approach clinical deployment, whether their accuracy is equitable across patient demographics remains insufficiently tested. To enable this evaluation, we constructed FairGlucose, a 300-patient CGM cohort balanced across 12 demographic strata (age x gender x type 1/type 2 diabetes), with 132,480 forecasting samples and 3,945 unique behavioral events (meals, exercise, medication) logged by 81 patients. Benchmarking 33 models across four families on 2-hour glucose forecasting, we find that population-level external validation can conceal substantial subgroup disparities. Aggregate out-of-distribution metrics appear stable (approximately 1.0), yet subgroup-level ratios range from 0.8 to 1.4, with T1D patients showing 6 mg/dL higher prediction error than T2D (p < 0.001). This disparity persists across all 33 models, suggesting a property of the prediction task rather than any single architecture. Further analysis shows that subgroup performance gaps align with the proportion of clinically hard cases, and that input-length sensitivity varies across demographics, motivating personalized configurations. Frontier LLMs underperform specialized neural models by 1-6 mg/dL; behavioral events contribute negligibly (approximately 0.1 mg/dL) even under oracle event access. These findings establish that population-level validation alone is insufficient for equity assessment of digital health AI, motivating subgroup-disaggregated reporting as a default standard.

Figures & tables

Figure 1: FairGlucose Benchmark Curation Process. 300 patients stratified across 12 balanced subgroups (age: 18–39, 40–64, $\geq$ 65 $\times$ gender: M/F $\times$ diabetes type: T1D/T2D), 25 per subgroup. Each subgroup has 20 held-in patients (train/valid/test-id: 15/2/3 days) and 5 held-out patients (test-od: 12 days). Each day yields 24 prediction pairs (24h input + 8h output). Includes 3,945 unique event annotations (meals, exercise, medication) logged by 81 patients across 132,480 prediction pairs.

Overall and marginal subgroups
Stratum	N	Mean glucose	CV	TIR (%)	MAGE
Overall	300	162.7	0.26	67.1	4.06
Age 18–39	100	163.8	0.26	65.9	4.40
Age 40–64	100	165.9	0.26	64.9	4.23
Age $\geq$ 65	100	158.5	0.25	70.6	3.57
Female	150	162.9	0.25	67.9	4.11

Table 1: Glycemic characteristics of the FairGlucose cohort by demographic stratum (all 25 patients per stratum, held-in and held-out pooled; split-wise breakdowns in Supplementary Note A). Each stratum contains 25 patients (20 held-in for training/validation/test-ID; 5 held-out for test-OD). Glucose values in mg/dL; CV is the coefficient of variation; TIR is time-in-range (70–180 mg/dL); MAGE is the mean amplitude of glycemic excursions.

Split		Patients	Patient-Days	I/O Pairs
Held-in	Train	20 (240)	300 (3,600)	7,200 (86,400)
	Valid	20 (240)	40 (480)	960 (11,520)
	Test-ID	20 (240)	60 (720)	1,440 (17,280)
Held-out	Test-OD	5 (60)	60 (720)	1,440 (17,280)
Total		25 (300)	460 (5,520)	11,040 (132,480)

Table 2: Dataset partition overview. Each cell shows the per-subgroup value (total across 12 subgroups in parentheses).

Model	rMSE (mg/dL)								Clarke A (%)
Model	All	T1D	T2D	Female	Male	18–39	40–64	65+	All	T1D	T2D	Female	Male	18–39	40–64	65+
ARIMA	27.44	30.78	24.10	27.89	26.99	27.76	28.05	26.51	74.67	70.26	79.07	74.14	75.19	74.11	74.55	75.38
Naive	29.03	32.42	25.63	29.54	28.51	29.41	29.78	27.85	72.86	68.61	77.11	72.45	73.27	71.98	72.90	73.76
AutoETS	29.72	33.35	26.09	30.44	29.00	30.27	30.53	28.31	72.96	68.63	77.29	72.41	73.51	71.53	73.03	74.46
LightGBM	25.96	29.07	22.84	26.34	25.58	26.59	26.58	24.63	76.77*	72.45	81.10*	76.30*	77.24*	75.25*	77.16*	78.08*
Linear	26.04	28.92	23.15	26.51	25.56	26.80	26.50	24.75	75.57	71.64	79.50	74.97	76.17	73.87	76.22	76.74

Table 3: Model performance on rMSE (mg/dL) and Clarke Error Grid Zone A (%) at the 2-hour prediction horizon, averaged across test-ID and test-OD splits, across all subgroups defined by diabetes type (T1D, T2D), gender (Female, Male), and age (18–39, 40–64, 65+). Clarke A represents clinically accurate predictions. Higher Clarke A percentage indicates better clinical accuracy. Underlined values indicate the best performance within each model family. Bolded values highlight the top-2 overall performers per subgroup: * denotes the best performance and bold without asterisk denotes the second best.

Figure 2: Generalization performance across models. Top: 2-hour rMSE on test-ID and test-OD with 95% CIs. Bottom: OD/ID ratio. Figure created by the authors using Matplotlib.

Figure 3: Intersectional subgroup accuracy and generalization across 12 demographic strata. Figure created by the authors using Matplotlib.

Figure 4: Model accuracy vs. subgroup disparity. Models near bottom-left (low error, low disparity) are preferred. PatchTST and TFT achieve strong performance on both axes. Figure created by the authors using Matplotlib.

Panel of Figure 5 — Figure 5: Input-length sensitivity analysis. (a) Subgroup-level temporal ratio (TR) across 12 demographic strata: shorter input windows degrade performance more for older T2D patients. (b) Model-level 2-hour rMSE across input lengths: neural architectures benefit most from extended input windows. Figure created by the authors using Matplotlib.

Panel of Figure 6 — Figure 6: Instance-level difficulty analysis. (a) Difficulty distribution across 12 demographic strata: hard cases concentrate in T1D subgroups. (b) Hard vs. easy case performance by model (normalized by Naive): top models concentrate gains on easy cases with limited improvement on hard cases. Figure created by the authors using Matplotlib.

Subgroup	Event rMSE (mg/dL)				NoEvent rMSE (mg/dL)				$\Delta$ (Event - NoEvent)
Subgroup	PatchTST	TFT	Autoformer	VanillaTrans.	PatchTST	TFT	Autoformer	VanillaTrans.	PatchTST	TFT	Autoformer	VanillaTrans.
All	25.37	26.00	37.95	47.52	25.48	26.08	38.71	47.72	-0.11	-0.08	-0.77	-0.20
T1D	28.21	28.94	42.30	48.66	28.36	28.97	43.04	48.90	-0.15	-0.04	-0.74	-0.25
T2D	22.53	23.06	33.59	46.37	22.60	23.18	34.39	46.53	-0.07	-0.12	-0.80	-0.16
Female	25.78	26.39	38.55	49.07	25.91	26.51	39.26	49.24	-0.13	-0.13	-0.71	-0.16
Male	24.96	25.61	37.34	45.96	25.05	25.64	38.17	46.20	-0.09	-0.03	-0.83	-0.24

Table 4: Comparison of event vs noevent model performance at 2-hour prediction horizon across subgroups. Event features include both historical (pre-observation) and future (post-observation) events; these results represent an upper-bound estimate of prospective event-aware forecasting.

\Delta

= Event - NoEvent: Negative values (dark green,

\Delta<-0.1

) indicate event model is better, positive values (dark red,

\Delta\geq 0.1

) indicate noevent model is better. Differences with

|\Delta|<0.1

mg/dL are shown in black (negligible). Note that the models in this table are the Nixtla/ neuralforecast implementations used for the event ablation, whereas Table 3 reports the TSLib implementations; VanillaTrans. here is therefore not the same code path as Transformer there, and the two should not be read across. Only the Event vs. NoEvent contrast within a column is meaningful.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: t-SNE embeddings of 24-hour CGM patient-day traces (288 5-minute steps), colored by (top) composite stratum and (bottom) univariate attributes. Stratum-coloring produces the most separable clusters; univariate colorings overlap substantially. Star markers denote subgroup centroids.

Panel of Figure 8 — Figure 8: Input length sensitivity across prediction horizons and metrics. Each panel shows model performance (y-axis) at varying input lengths (x-axis) for a specific horizon-metric combination.

Panel of Figure 9 — Figure 9: Input length sensitivity for TIR Absolute Error across prediction horizons.

Panel of Figure 10 — Figure 10: Instance-level difficulty analysis: subgroup-level hard/easy case distribution and rMSE/MAE spread across prediction horizons.

Panel of Figure 11 — Figure 11: Instance-level difficulty analysis for TIR Absolute Error across prediction horizons.

Statistical
Model	All	T1D	T2D	Female	Male	18-39	40-64	65+
ARIMA	27.39	30.84	23.94	27.36	27.42	28.48	26.93	26.70
Naive	29.03	32.35	25.70	28.96	29.10	29.96	28.84	28.19
AutoETS	29.81	33.61	26.01	29.87	29.75	30.79	29.81	28.69
ML
Linear	25.87	29.15	22.60	25.79	25.96	27.13	25.70	24.65

Appendix

Table 5: 2-hour rMSE (mg/dL) across subgroups – Test-ID. Lower values indicate better prediction accuracy. T1D patients consistently show higher rMSE than T2D across all models, reflecting greater glucose variability.

Statistical
Model	All	T1D	T2D	Female	Male	18-39	40-64	65+
ARIMA	27.50	30.73	24.26	28.43	26.57	27.04	29.16	26.33
Naive	29.02	32.49	25.55	30.11	27.93	28.86	30.73	27.51
AutoETS	29.63	33.08	26.18	31.00	28.26	29.76	31.24	27.92
ML
LightGBM	26.02	28.76	23.29	27.02	25.03	25.94	27.44	24.72

Appendix

Table 6: 2-hour rMSE (mg/dL) across subgroups – Test-OD. Out-of-distribution evaluation on held-out patients. OD/ID ratios near 1.0 indicate robust generalization; subgroup-level variation reveals hidden heterogeneity.

Statistical
Model	All	T1D	T2D	Female	Male	18-39	40-64	65+
ARIMA	23.35	26.22	20.49	23.31	23.39	24.27	22.95	22.78
Naive	24.78	27.55	22.01	24.67	24.90	25.54	24.61	24.13
AutoETS	25.24	28.35	22.12	25.24	25.23	26.05	25.26	24.28
ML
Linear	22.03	24.77	19.29	21.99	22.07	23.12	21.85	20.99

Appendix

Table 7: 2-hour MAE (mg/dL) across subgroups – Test-ID. Mean absolute error provides a complementary view to rMSE, being less sensitive to large outlier predictions.

Statistical
Model	All	T1D	T2D	Female	Male	18-39	40-64	65+
ARIMA	23.51	26.25	20.76	24.27	22.75	23.00	24.98	22.58
Naive	24.85	27.80	21.89	25.75	23.95	24.59	26.30	23.69
AutoETS	25.16	28.09	22.24	26.36	23.97	25.29	26.48	23.75
ML
LightGBM	22.10	24.42	19.78	22.95	21.24	22.00	23.26	21.06

Appendix

Table 8: 2-hour MAE (mg/dL) across subgroups – Test-OD. The same subgroup disparity patterns observed in rMSE persist in MAE, confirming the robustness of the findings across metrics.

Figure 12: 30-minute prediction accuracy: Model comparison between Test-ID and Test-OD splits. Top panel in each subfigure shows metric values with 95% CI; bottom panel shows OD/ID generalization ratio.

Figure 13: 1-hour prediction accuracy: Model comparison between Test-ID and Test-OD splits.

Figure 14: 2-hour prediction accuracy: Model comparison between Test-ID and Test-OD splits.

Figure 15: Fairness comparison across demographic subgroups (age, gender, diabetes type) for all prediction horizons. Top panel in each subfigure shows Test-ID performance; bottom panel shows OD/ID generalization ratio.

Panel of Figure 16 — Figure 16: Accuracy vs. fairness tradeoff across prediction horizons (Spread measure). Columns show rMSE and MAE metrics; rows show 30-minute, 1-hour, and 2-hour predictions. Each point represents a model, colored by family. Lower values on both axes indicate better performance.

Panel of Figure 17 — Figure 17: Accuracy vs. fairness tradeoff across prediction horizons (Gini coefficient measure). Columns show rMSE and MAE metrics; rows show 30-minute, 1-hour, and 2-hour predictions. Each point represents a model, colored by family. Lower values on both axes indicate better performance.

Continuous Glucose Monitoring

See all 16 papers

Glucose-ML: A collection of longitudinal diabetes datasets for development of robust AI solutions
Jul 18, 2025Temiloluwa Prioleau, Baiying Lu, Yanjun CuiForecasting Benchmarks Blood Glucose Forecasting
Department of Computer Science Emory University Atlanta, GA 30322 · Department of Computer Science Dartmouth College Hanover, NH 03755
GlucoFM-Bench: Benchmarking Time-Series Foundation Models for Blood Glucose Forecasting
Jun 5, 2026Baiying Lu, Zhaohui Liang, Ryan Pontius +2Zero-Shot Time Series Forecasting Time Series Forecasting
Department of Computer Science Dartmouth College Hanover, NH 03784 · Department of Computer Science Emory University Atlanta, GA 30322 · Quantitative Biomedical Sciences Dartmouth College Hanover, NH 03784
GlyRAG: Context-Aware Retrieval-Augmented Framework for Blood Glucose Forecasting
Jan 8, 2026Shovito Barua Soumma, Hassan GhasemzadehTime Series Forecasting Blood Glucose Forecasting
College of Health Solutions, Arizona State University, Phoenix, AZ 85004, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA

Jul 18, 2025J/K move · Enter open · S save

Glucose-ML: A collection of longitudinal diabetes datasets for development of robust AI solutions

Temiloluwa Prioleau, Baiying Lu, Yanjun Cui

Department of Computer Science Emory University Atlanta, GA 30322 · Department of Computer Science Dartmouth College Hanover, NH 03755

Forecasting Benchmarks Blood Glucose Forecasting Clinical Prediction ML Reproducibility Wearable Health Monitoring Continuous Glucose Monitoring+1

Open paper AskPDF arXiv

Artificial intelligence (AI) algorithms are a critical part of state-of-the-art digital health technology for diabetes management. Yet, access to large high-quality datasets is creating barriers that impede development of robust AI solutions. To accelerate development of transparent, reproducible, and robust AI solutions, we present Glucose-ML, a collection of 10 publicly available diabetes datasets, released within the last 7 years (i.e., 2018 - 2025). The Glucose-ML collection comprises over 300,000 days of continuous glucose monitor (CGM) data with a total of 38 million glucose samples collected from 2500+ people across 4 countries. Participants include persons living with type 1 diabetes, type 2 diabetes, prediabetes, and no diabetes. To support researchers and innovators with using this rich collection of diabetes datasets, we present a comparative analysis to guide algorithm developers with data selection. Additionally, we conduct a case study for the task of blood glucose prediction - one of the most common AI tasks within the field. Through this case study, we provide a benchmark for short-term blood glucose prediction across all 10 publicly available diabetes datasets within the Glucose-ML collection. We show that the same algorithm can have significantly different prediction results when developed/evaluated with different datasets. Findings from this study are then used to inform recommendations for developing robust AI solutions within the diabetes or broader health domain. We provide direct links to each longitudinal diabetes dataset and openly provide our code.

Abstract

Continuous Glucose Monitoring

Explore similar work

Glucose-ML: A collection of longitudinal diabetes datasets for development of robust AI solutions

GlucoFM-Bench: Benchmarking Time-Series Foundation Models for Blood Glucose Forecasting

GlyRAG: Context-Aware Retrieval-Augmented Framework for Blood Glucose Forecasting