Organizations: Gary W. Rollins College of Business, The University of Tennessee at Chattanooga, Chattanooga, Tennessee, USA. · Department of Biomedical Engineering and Chemical Engineering, The University of Texas at San Antonio, San Antonio, Texas, USA. · Department of Biostatistics and Data Science, School of Public Health, The University of Texas Health Science Center at Houston, Houston, Texas, USA. · Center for Spatial-Temporal Modeling for Applications in Population Sciences, School of Public Health, The University of Texas Health Science Center at Houston, Houston, Texas, USA. · School of Engineering Medicine, Texas A&M University, Houston, Texas, USA. · Department of Information Systems and Cybersecurity, The University of Texas at San Antonio, San Antonio, Texas, USA.
Persistent low retention and completion rates in medications for opioid use disorder (MOUD) have driven the use of machine learning (ML) models to predict retention and identify patients at risk of premature discontinuation. However, the fairness of these models across patient populations remains largely unexplored, raising concerns about their application in treatment decision support. This study systematically assesses algorithmic fairness in ML models for predicting MOUD retention and premature discontinuation and investigates the effectiveness of bias mitigation techniques. Using the cross-sectional Treatment Episode Data Set-Discharges (TEDS-D), which includes treatment episodes for individuals in the U.S. discharged between 2015 and 2019, we trained four ML models to predict premature treatment discontinuation and retention beyond 180 days among individuals receiving outpatient MOUD. We evaluated overall performance and subgroup-level error rates across patient subgroups defined by race, ethnicity, age, and sex, complemented by model explanation analyses. We further assessed bias mitigation techniques and their effects on both fairness and predictive performance. Our findings demonstrate that ML models for MOUD outcome prediction can exhibit subgroup-level performance gaps even when overall predictive performance appears acceptable and that bias mitigation can reduce, but not fully eliminate, these gaps without trade-offs. By demonstrating the importance of fairness-aware evaluation and transparent reporting of subgroup performance, this study provides practical insights for the responsible and context-sensitive use of ML models for risk stratification and care prioritization in MOUD treatment settings.
Figures & tables
Figure 1: Overall pipeline.
Predicted - Positive
Predicted - Negative
Actual - Positive
True Positive (TP)
False Negative (FN)
TPR=TP+FNTP
FNR=1−TPR
Actual - Negative
False Positive (FP)
True Negative (TN)
FPR=FP+TNFP
TNR=1−FPR
Table 1: Confusion matrix and derived metrics
Premature Discontinuation
Treatment Retention
Discontinuation
Completion
> 180 days
≤ 180 days
Sex
Male
115,613 (31.52%)
97,178 (26.49%)
81,857 (22.32%)
130,934 (35.69%)
Female
83,008 (22.63%)
71,026 (19.36%)
59,910 (16.33%)
94,124 (25.66%)
Race
White
163,842 (44.66%)
145,169 (39.48%)
116,872 (31.86%)
192,139 (52.28%)
Black
34,779 (9.48%)
23,035 (6.28%)
24,895 (6.79%)
32,919 (8.97%)
Ethnicity
Non-Hispanic
182,285 (49.69%)
157,103 (42.83%)
130,078 (35.46%)
209,310 (57.06%)
Table 2: Characteristics of cohorts ( N=366,825 ) presented as the number of episodes, with percentages indicating their proportion within the entire cohort (not within each outcome category).
Figure 2: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) as the baseline models for predicting premature treatment discontinuation. The subfigures presents results stratified by (a) race, (b) ethnicity, (c) age, and (d) sex. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Premature Discontinuation
Retention > 180 days
LR
0.6562 (0.0011)
0.6559 (0.0011)
RF
0.6863 (0.0013)
0.6797 (0.0011)
GBDT
0.6892 (0.0013)
0.6836 (0.0010)
MLP
0.6647 (0.0025)
0.6657 (0.0014)
Table 3: Overall predictive performance, measured by area under the receiver operating characteristic curve (AUROC), for ML models predicting premature treatment discontinuation and treatment retention exceeding 180 days.
Figure 3: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) as the baseline models for predicting treatment retention > 180 days. The subfigures presents results stratified by (a) race, (b) ethnicity, (c) age, and (d) sex. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Figure 4: Figures (a) and (b) present feature importance plots of the GBDT model for two prediction tasks: (a) premature treatment discontinuation and (b) treatment retention > 180 days. The x-axis represents feature attribution values, where positive values indicate that a feature increases the likelihood of the positive class, while negative values indicate a decrease. Features are ranked by their average importance, with the most influential ones appearing at the top.
Figure 5: Figures (a) and (b) display histograms of feature attribution values for sensitive attributes (race, ethnicity, and age) for each task: (a) premature treatment discontinuation and (b) treatment retention > 180 days, to visualize how these attributes contribute to model predictions.
Premature Discontinuation
Retention > 180 days
False Positive White patients (%)
False Positive Black patients (%)
False Positive White patients (%)
False Positive Black patients (%)
Age
18-24
11.3
2.8
7.1
0.5
25-54
82.6
67.6
72.6
54.5
> 55
6.1
29.6
20.3
45.0
Education
Table 4: Sample characteristics of false positive subgroup from GBDT model for premature treatment discontinuation and retention > 180 days, comparing Black and White patients.
Figure 6: FPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting premature treatment discontinuation, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 independent runs, with error bars indicating the standard deviation across these runs. Higher values indicate better results.
Figure 7: FPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting treatment retention > 180 days, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 independent runs, with error bars indicating the standard deviation across these runs. Higher values indicate better results.
Figure 8: FPRs for cross-tabulated subgroups across all bias mitigation methods in predicting two tasks (a) premature discontinuation and (b) treatment retention > 180 days with GBDT model. Lower FPR values indicate better results.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Cohort and variable selection.
Figure 10: TPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting premature treatment discontinuation, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean TPR values over 10 independent runs, with error bars indicating the standard deviation across these runs.
Figure 11: TPRs across different subgroups are shown for three bias mitigation methods—Reweighing (pre-processing), EGR (in-processing), and Threshold (post-processing)—in predicting treatment retention > 180 days, compared to a baseline without fairness considerations. Results are stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean TPR values over 10 independent runs, with error bars indicating the standard deviation across these runs.
LR
RF
GBDT
MLP
Baseline
Race
White
0.3545 (0.0031)
0.2842 (0.0047)
0.3011 (0.0064)
0.2878 (0.0666)
Black
0.6512 (0.0067)
0.5368 (0.0108)
0.5692 (0.0085)
0.5705 (0.0702)
Ethnicity
Non-Hispanic
0.3839 (0.0033)
0.3116 (0.0045)
0.3277 (0.0067)
0.3160 (0.0653)
Hispanic
0.5565 (0.0085)
0.4225 (0.0143)
0.4824 (0.0168)
0.4775 (0.0731)
Appendix
Table 13: All FPR results for predicting premature treatment discontinuation.
LR
RF
GBDT
MLP
Baseline
Race
White
0.5718 (0.0025)
0.5352 (0.0047)
0.5559 (0.0057)
0.5069 (0.0700)
Black
0.8209 (0.0027)
0.7564 (0.0078)
0.7863 (0.0063)
0.7532 (0.0588)
Ethnicity
Non-Hispanic
0.5998 (0.0025)
0.5623 (0.0048)
0.5821 (0.0064)
0.5352 (0.0673)
Hispanic
0.7893 (0.0064)
0.7029 (0.0080)
0.7530 (0.0118)
0.7139 (0.0594)
Appendix
Table 14: All TPR results for predicting premature treatment discontinuation.
LR
RF
GBDT
MLP
Baseline
Race
White
0.1107 (0.0021)
0.1114 (0.0016)
0.1155 (0.0020)
0.1288 (0.0397)
Black
0.2459 (0.0050)
0.2495 (0.0053)
0.2656 (0.0068)
0.2701 (0.0465)
Ethnicity
Non-Hispanic
0.1233 (0.0021)
0.1264 (0.0016)
0.1319 (0.0018)
0.1471 (0.0401)
Hispanic
0.2264 (0.0092)
0.2023 (0.0102)
0.2124 (0.0139)
0.2422 (0.0505)
Appendix
Table 15: All FPR results for predicting treatment retention > 180 days.
LR
RF
GBDT
MLP
Baseline
Race
White
0.2480 (0.0022)
0.2771 (0.0028)
0.2940 (0.0039)
0.2856 (0.0646)
Black
0.4561 (0.0075)
0.5026 (0.0041)
0.5294 (0.0071)
0.4991 (0.0623)
Ethnicity
Non-Hispanic
0.2727 (0.0023)
0.3073 (0.0023)
0.3262 (0.0032)
0.3204 (0.0626)
Hispanic
0.4165 (0.0108)
0.4207 (0.0109)
0.4363 (0.0168)
0.4430 (0.0630)
Appendix
Table 16: All TPR results for predicting treatment retention > 180 days.
Model
Premature Discontinuation
Retention > 180 days
AUROC
LR
0.7153 (0.0009)
0.6427 (0.0015)
RF
0.7367 (0.0011)
0.6770 (0.0017)
GBDT
0.7378 (0.0012)
0.6781 (0.0018)
MLP
0.7173 (0.0022)
0.6555 (0.0039)
EOD - Race
LR
0.1633 (0.0110)
0.1486 (0.0081)
RF
0.1425 (0.0094)
0.1390 (0.0100)
Appendix
Table 17: Overall predictive performance, measured by area under the receiver operating characteristic curve (AUROC), as well as Equalized Odds Difference (EOD) across subgroups for ML models predicting premature treatment discontinuation and treatment retention exceeding 180 days using 2020-2023 TEDS-D data.
Figure 12: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) for predicting premature treatment discontinuation with 2020-2023 data. The subfigures presents results stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Figure 13: FPRs and TPRs across different subgroups are evaluated for four ML models (LR, RF, GBDT, MLP) for predicting treatment retention > 180 days with 2020-2023 data. The subfigures presents results stratified by (a) race, (b) ethnicity, and (c) age. Bars represent the mean values over 10 runs, while error bars indicate the standard deviation computed from these runs. Higher values indicate better results.
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
Department of Applied Mathematics & Statistics, Stony Brook University, Stony Brook, NY, USA · Department of Computer Science, Stony Brook University, Stony Brook, NY, USA · Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA +1