Organizations: Gary W. Rollins College of Business, The University of Tennessee at Chattanooga, Chattanooga, Tennessee, USA. · Department of Biomedical Engineering and Chemical Engineering, The University of Texas at San Antonio, San Antonio, Texas, USA. · Department of Biostatistics and Data Science, School of Public Health, The University of Texas Health Science Center at Houston, Houston, Texas, USA. · Center for Spatial-Temporal Modeling for Applications in Population Sciences, School of Public Health, The University of Texas Health Science Center at Houston, Houston, Texas, USA. · School of Engineering Medicine, Texas A&M University, Houston, Texas, USA. · Department of Information Systems and Cybersecurity, The University of Texas at San Antonio, San Antonio, Texas, USA.
Persistent low retention and completion rates in medications for opioid use disorder (MOUD) have driven the use of machine learning (ML) models to predict retention and identify patients at risk of premature discontinuation. However, the fairness of these models across patient populations remains largely unexplored, raising concerns about their application in treatment decision support. This study systematically assesses algorithmic fairness in ML models for predicting MOUD retention and premature discontinuation and investigates the effectiveness of bias mitigation techniques. Using the cross-sectional Treatment Episode Data Set-Discharges (TEDS-D), which includes treatment episodes for individuals in the U.S. discharged between 2015 and 2019, we trained four ML models to predict premature treatment discontinuation and retention beyond 180 days among individuals receiving outpatient MOUD. We evaluated overall performance and subgroup-level error rates across patient subgroups defined by race, ethnicity, age, and sex, complemented by model explanation analyses. We further assessed bias mitigation techniques and their effects on both fairness and predictive performance. Our findings demonstrate that ML models for MOUD outcome prediction can exhibit subgroup-level performance gaps even when overall predictive performance appears acceptable and that bias mitigation can reduce, but not fully eliminate, these gaps without trade-offs. By demonstrating the importance of fairness-aware evaluation and transparent reporting of subgroup performance, this study provides practical insights for the responsible and context-sensitive use of ML models for risk stratification and care prioritization in MOUD treatment settings.
Figures & tables
Figure 1: Overall pipeline.
Predicted - Positive
Predicted - Negative
Actual - Positive
True Positive (TP)
False Negative (FN)
TPR=TP+FNTP
FNR=1−TPR
Actual - Negative
False Positive (FP)
True Negative (TN)
FPR=FP+TNFP
TNR=1−FPR
Table 1: Confusion matrix and derived metrics
Premature Discontinuation
Treatment Retention
Discontinuation
Completion
> 180 days
≤ 180 days
Sex
Male
115,613 (31.52%)
97,178 (26.49%)
81,857 (22.32%)
130,934 (35.69%)
Female
83,008 (22.63%)
71,026 (19.36%)
59,910 (16.33%)
94,124 (25.66%)
Race
White
163,842 (44.66%)
145,169 (39.48%)
116,872 (31.86%)
192,139 (52.28%)
Black
34,779 (9.48%)
23,035 (6.28%)
24,895 (6.79%)
32,919 (8.97%)
Ethnicity
Non-Hispanic
182,285 (49.69%)
157,103 (42.83%)
130,078 (35.46%)
209,310 (57.06%)
Table 2: Characteristics of cohorts ( N=366,825 ) presented as the number of episodes, with percentages indicating their proportion within the entire cohort (not within each outcome category).
Figure 2: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) as the baseline models for predicting premature treatment discontinuation. The subfigures presents results stratified by (a) race, (b) ethnicity, (c) age, and (d) sex. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Premature Discontinuation
Retention > 180 days
LR
0.6562 (0.0011)
0.6559 (0.0011)
RF
0.6863 (0.0013)
0.6797 (0.0011)
GBDT
0.6892 (0.0013)
0.6836 (0.0010)
MLP
0.6647 (0.0025)
0.6657 (0.0014)
Table 3: Overall predictive performance, measured by area under the receiver operating characteristic curve (AUROC), for ML models predicting premature treatment discontinuation and treatment retention exceeding 180 days.
Figure 3: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) as the baseline models for predicting treatment retention > 180 days. The subfigures presents results stratified by (a) race, (b) ethnicity, (c) age, and (d) sex. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Figure 4: Figures (a) and (b) present feature importance plots of the GBDT model for two prediction tasks: (a) premature treatment discontinuation and (b) treatment retention > 180 days. The x-axis represents feature attribution values, where positive values indicate that a feature increases the likelihood of the positive class, while negative values indicate a decrease. Features are ranked by their average importance, with the most influential ones appearing at the top.
Figure 5: Figures (a) and (b) display histograms of feature attribution values for sensitive attributes (race, ethnicity, and age) for each task: (a) premature treatment discontinuation and (b) treatment retention > 180 days, to visualize how these attributes contribute to model predictions.
Premature Discontinuation
Retention > 180 days
False Positive White patients (%)
False Positive Black patients (%)
False Positive White patients (%)
False Positive Black patients (%)
Age
18-24
11.3
2.8
7.1
0.5
25-54
82.6
67.6
72.6
54.5
> 55
6.1
29.6
20.3
45.0
Education
Table 4: Sample characteristics of false positive subgroup from GBDT model for premature treatment discontinuation and retention > 180 days, comparing Black and White patients.
Figure 6: FPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting premature treatment discontinuation, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 independent runs, with error bars indicating the standard deviation across these runs. Higher values indicate better results.
Figure 7: FPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting treatment retention > 180 days, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 independent runs, with error bars indicating the standard deviation across these runs. Higher values indicate better results.
Figure 8: FPRs for cross-tabulated subgroups across all bias mitigation methods in predicting two tasks (a) premature discontinuation and (b) treatment retention > 180 days with GBDT model. Lower FPR values indicate better results.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Cohort and variable selection.
Figure 10: TPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting premature treatment discontinuation, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean TPR values over 10 independent runs, with error bars indicating the standard deviation across these runs.
Figure 11: TPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting treatment retention > 180 days, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean TPR values over 10 independent runs, with error bars indicating the standard deviation across these runs.
LR
RF
GBDT
MLP
Baseline
Race
White
0.3545 (0.0031)
0.2842 (0.0047)
0.3011 (0.0064)
0.2878 (0.0666)
Black
0.6512 (0.0067)
0.5368 (0.0108)
0.5692 (0.0085)
0.5705 (0.0702)
Ethnicity
Non-Hispanic
0.3839 (0.0033)
0.3116 (0.0045)
0.3277 (0.0067)
0.3160 (0.0653)
Hispanic
0.5565 (0.0085)
0.4225 (0.0143)
0.4824 (0.0168)
0.4775 (0.0731)
Appendix
Table 13: All FPR results for predicting premature treatment discontinuation.
LR
RF
GBDT
MLP
Baseline
Race
White
0.5718 (0.0025)
0.5352 (0.0047)
0.5559 (0.0057)
0.5069 (0.0700)
Black
0.8209 (0.0027)
0.7564 (0.0078)
0.7863 (0.0063)
0.7532 (0.0588)
Ethnicity
Non-Hispanic
0.5998 (0.0025)
0.5623 (0.0048)
0.5821 (0.0064)
0.5352 (0.0673)
Hispanic
0.7893 (0.0064)
0.7029 (0.0080)
0.7530 (0.0118)
0.7139 (0.0594)
Appendix
Table 14: All TPR results for predicting premature treatment discontinuation.
LR
RF
GBDT
MLP
Baseline
Race
White
0.1107 (0.0021)
0.1114 (0.0016)
0.1155 (0.0020)
0.1288 (0.0397)
Black
0.2459 (0.0050)
0.2495 (0.0053)
0.2656 (0.0068)
0.2701 (0.0465)
Ethnicity
Non-Hispanic
0.1233 (0.0021)
0.1264 (0.0016)
0.1319 (0.0018)
0.1471 (0.0401)
Hispanic
0.2264 (0.0092)
0.2023 (0.0102)
0.2124 (0.0139)
0.2422 (0.0505)
Appendix
Table 15: All FPR results for predicting treatment retention > 180 days.
LR
RF
GBDT
MLP
Baseline
Race
White
0.2480 (0.0022)
0.2771 (0.0028)
0.2940 (0.0039)
0.2856 (0.0646)
Black
0.4561 (0.0075)
0.5026 (0.0041)
0.5294 (0.0071)
0.4991 (0.0623)
Ethnicity
Non-Hispanic
0.2727 (0.0023)
0.3073 (0.0023)
0.3262 (0.0032)
0.3204 (0.0626)
Hispanic
0.4165 (0.0108)
0.4207 (0.0109)
0.4363 (0.0168)
0.4430 (0.0630)
Appendix
Table 16: All TPR results for predicting treatment retention > 180 days.
Model
Premature Discontinuation
Retention > 180 days
AUROC
LR
0.7153 (0.0009)
0.6427 (0.0015)
RF
0.7367 (0.0011)
0.6770 (0.0017)
GBDT
0.7378 (0.0012)
0.6781 (0.0018)
MLP
0.7173 (0.0022)
0.6555 (0.0039)
EOD - Race
LR
0.1633 (0.0110)
0.1486 (0.0081)
RF
0.1425 (0.0094)
0.1390 (0.0100)
Appendix
Table 17: Overall predictive performance, measured by area under the receiver operating characteristic curve (AUROC), as well as Equalized Odds Difference (EOD) across subgroups for ML models predicting premature treatment discontinuation and treatment retention exceeding 180 days using 2020-2023 TEDS-D data.
Figure 12: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) for predicting premature treatment discontinuation with 2020-2023 data. The subfigures presents results stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Figure 13: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) for predicting treatment retention > 180 days with 2020-2023 data. The subfigures presents results stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
Algorithmic fairness methods are increasingly used to identify and mitigate bias in machine learning models, yet most approaches are evaluated in isolation and along single demographic axes. This limits practical guidance for selecting fairness strategies, where disparities may arise across intersectional subgroups and across multiple stages of the modeling lifecycle. This work presents FairSelect, a toolkit for systematically evaluating fairness mitigation strategies applied individually and in combination across preprocessing, inprocessing, and postprocessing stages. FairSelect supports multiple model architectures, intersectional subgroup evaluation, and comparison of fairness utility tradeoffs across baseline, single method, and multi level configurations. The framework was validated using synthetic clinical datasets designed to represent specific bias mechanisms and a real-world replication of two-year stroke risk prediction among patients with atrial fibrillation. Synthetic experiments showed that targeted fairness methods generally reduced intended subgroup disparities, while combined strategies produced larger average fairness improvements with modest utility tradeoffs. In the clinical prediction task, mitigation effects were highly variable, with some combinations improving both fairness and predictive performance while others were ineffective or counterproductive. These findings demonstrate that fairness interventions interact in nonadditive and context dependent ways. FairSelect provides a practical framework for systematically identifying fairness strategies that improve subgroup equity while preserving model performance in clinical machine learning.
Nick Souligne, Isabella Mixton-Garcia, Vignesh Subbian
College of Engineering, University of Arizona, Tucson, AZ
Electronic health records (EHRs) may incompletely capture patient-reported factors associated with opioid use disorder (OUD). We evaluated whether survey data improve prediction of a first recorded OUD diagnosis among 267,747 All of Us participants with documented opioid exposure, including 15,287 OUD cases. We compared EHR-only and EHR+survey models across 6-, 12-, and 24-month look-back windows using logistic regression, random forest, XGBoost, LightGBM, multilayer perceptron, LSTM, GRU, and Transformer. Survey augmentation improved PR-AUC across all 24 model-window combinations by 0.0087-0.0505; the best 24-month LightGBM model improved from 0.6219 to 0.6603. Survey coverage increased with longer windows and differed by OUD status (24 months: 21.7% OUD-positive vs. 60.7% OUD-negative). Permutation analysis ranked survey features as the second most important information domain at 24 months in both evaluated models. Patient-reported data provide complementary predictive signals beyond structured EHRs while highlighting the importance of survey availability.
Xiyue Jiang, Zihan Ding, Grace Han +3
Department of Applied Mathematics & Statistics, Stony Brook University, Stony Brook, NY, USA · Department of Computer Science, Stony Brook University, Stony Brook, NY, USA · Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA +1
Intersectional biases in healthcare data can produce compound disparities in clinical machine learning models, yet most fairness evaluations assess demographic attributes independently. FairLogue, a toolkit for intersectional fairness auditing, was applied across multiple clinical prediction tasks to evaluate disparities across combined demographic groups. Using the All of Us dataset, two published models were selected for replication and evaluation: (A) prediction of selective serotonin reuptake inhibitor associated bleeding events and (B) two-year stroke risk in patients with atrial fibrillation. Observational fairness metrics were computed across race, gender, and intersectional subgroups, followed by counterfactual analysis to evaluate whether disparities were attributable to group membership. Intersectional evaluation revealed larger disparities than single-axis analyses; however, counterfactual diagnostics indicated that most observed disparities were comparable to those expected under randomized group membership. These results highlight the importance of intersectional fairness auditing and demonstrate how FairLogue provides deeper insight into bias in clinical machine learning systems.
Nick Souligne, Vignesh Subbian
College of Engineering, The University of Arizona, Tucson, AZ, United States