Reweighting is a widely used pre-processing method for promoting discrimination-free model behavior by assigning different weights to data points during model training. While it is often credited with improving fairness, particularly by mitigating systematic underestimation for underrepresented groups, it can also introduce unintended harms under certain data conditions. Using controlled simulations that vary group size and baseline event prevalence, we show how reweighting can yield misleading improvements in fairness by reducing underestimation for one group while introducing it for another. Across 3,000 simulated datasets with event prevalence of 7-14%, reweighting improved parity in positive classification and false negative rates (FNR) between groups, but did so partly by decreasing positive classifications and increasing FNR for the group with higher baseline prevalence. These effects were most pronounced when that group was a minority: its FNR rose by 6 percentage points, approximately 82 additional missed cases per 10,000 group members, and roughly three-quarters of the improvement in FNR parity came from this increase rather than from gains for the other group. The pattern held across six classifiers, from logistic regression to neural networks, while overall accuracy was essentially unchanged, leaving the shift invisible in aggregate performance metrics. These findings mirror patterns reported in real-world applications and underscore the critical role of data characteristics in shaping the impact of this fairness intervention. While reweighting may create desirable outcomes in some applications, practitioners should critically evaluate it relative to their problem area and goals. Specifically, when the goal is to reduce underestimation in one group without introducing underestimation in another, reweighting may not be the appropriate approach.
Figures & tables
#
Proportion Group A
Relative Event Likelihood in Group A
Prevalence A
Prevalence B
1
0.2 (Minority)
Less Likely ( ba=−0.7 )
7.0%
10.0%
2
0.5 (Equal)
Less Likely ( ba=−0.7 )
7.0%
10.0%
3
0.2 (Minority)
Equally Likely ( ba=0 )
10.0%
10.0%
4
0.5 (Equal)
Equally Likely ( ba=0 )
10.0%
10.0%
5
0.2 (Minority)
More Likely ( ba=+0.7 )
13.5%
10.0%
6
0.5 (Equal)
More Likely ( ba=+0.7 )
13.5%
10.0%
Table 1. Experimental conditions varying two data characteristics: dataset diversity and relative event likelihood.
#
Proportion Group A
Relative Event Likelihood in Group A
Substantive Example
1
0.2 (Minority)
Less Likely
Mortgage Approval ( Bhutta et al., 2021 )
2
0.5 (Equal)
Less Likely
Resume Selection ( Wilson and Caliskan, 2024 )
3
0.2 (Minority)
Equally Likely
Control Group
4
0.5 (Equal)
Equally Likely
Control Group
5
0.2 (Minority)
More Likely
Postpartum Depression a ( Wong et al., 2024 )
6
0.5 (Equal)
More Likely
Breast Cancer ( Anderson et al., 2010 )
Table 2. Experimental conditions and their corresponding substantive examples.
Group
Group
Relative Event
Two-sided
Label
Size
Likelihood
Before
After
ΔP(Y^=1∣⋅)
95% CI
p -value
A
Minority
More Likely
0.1263
0.1109
−0.0154
[ −0.0156 , −0.0152 ]
<.001
A
Equal
More Likely
0.1264
0.1172
−0.0092
[ −0.0094 , −0.0091 ]
<.001
B
Equal
More Likely
0.0849
0.0748
−0.0101
[ −0.0102 , −0.0100 ]
<.001
B
Majority
More Likely
0.0853
0.0813
−0.0040
[ −0.0040 , −0.0039 ]
<.001
A
Minority
Less Likely
0.0472
0.0640
+0.0167
[ +0.0165 , +0.0170 ]
<.001
Table 3. The effect of reweighting on the probability of positive classification by group size and relative event likelihood (logistic regression, held-out test set). Before and after are mean rates; Δ is the mean paired difference (reweighted minus unweighted) across 3,000 repetitions, with 95% confidence intervals (CI) and two-sided p -values from Wilcoxon signed-rank tests. Values correspond to Figure 2 .
Figure 1. The effect of reweighting on Demographic Parity (DP) (logistic regression; held-out test set).
Figure 2. The effect of reweighting on model predictions (logistic regression; held-out test set).
Figure 3. The effect of reweighting on false negative rates (FNR) parity (logistic regression; held-out test set.
Figure 4. The effect of reweighting on false negative rates (FNR) (logistic regression, held-out test set).
Group
Group
Relative Event
Two-sided
Additional missed
Label
Size
Likelihood
Δ FNR
95% CI
p -value
cases per 10,000
A
Minority
More Likely
+0.0608
[ 0.0597 , 0.0618 ]
<.001
82
A
Equal
More Likely
+0.0356
[ 0.0351 , 0.0362 ]
<.001
48
B
Equal
More Likely
+0.0531
[ 0.0524 , 0.0539 ]
<.001
53
B
Majority
More Likely
+0.0202
[ 0.0198 , 0.0206 ]
<.001
20
Table 4. The effect of reweighting on FNR for groups with higher baseline likelihoods across group sizes (logistic regression, held-out test set). Values are mean paired differences (reweighted minus unweighted) across 3,000 repetitions, with 95% confidence intervals (CI) and two-sided p -values from Wilcoxon signed-rank tests. Additional missed cases are Δ FNR × group prevalence × 10,000, using training-set prevalence. Values correspond to Figure 4 .
Figure 5. Change after reweighting (reweighted minus unweighted) in the positive-prediction rate of Group A (top), the FNR of Group A (middle), and the FNR of Group B (bottom), for six classifiers. Left: Group A is the minority (20%); right: groups are of equal size. Points are means across 3,000 repetitions on the held-out test set; error bars are 95% confidence intervals.
Test
Higher-baseline group
Lower-baseline group
Classifier
AUROC
Minority
Equal
Majority
Minority
Logistic regression
0.960
+0.0608
+0.0356
+0.0202
−0.1117
[0.0597, 0.0618]
[0.0351, 0.0362]
[0.0198, 0.0206]
[ −0.1138 , −0.1097 ]
Logistic regression (L2)
0.960
+0.0624
+0.0365
+0.0208
−0.1147
[0.0613, 0.0635]
[0.0360, 0.0371]
[0.0204, 0.0211]
[ −0.1168 , −0.1126 ]
Multilayer perceptron
0.959
+0.0581
+0.0352
+0.0190
−0.1090
Table 5. Change in FNR after reweighting by classifier (held-out test set). Values are mean paired differences (reweighted minus unweighted) across 3,000 repetitions, with 95% confidence intervals in brackets; all p<.001 (Wilcoxon signed-rank). Higher-baseline groups: Group A as minority (condition 5) and as equal-sized group (condition 6), and Group B as majority (condition 1). Lower-baseline group: Group A as minority (condition 1).
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
As machine learning (ML) systems increasingly shape access to credit, jobs, and other opportunities, the fairness of algorithmic decisions has become a central concern. Yet it remains unclear when enforcing fairness constraints in these systems genuinely improves outcomes for affected groups or instead leads to "leveling down," making one or both groups worse off. We address this question in a unified, population-level (Bayes) framework for binary classification under prevalent group fairness notions. Our Bayes approach is distribution-free and algorithm-agnostic, isolating the intrinsic effect of fairness requirements from finite-sample noise and from training and intervention specifics. We analyze two deployment regimes for ML classifiers under common legal and governance constraints: attribute-aware decision-making (sensitive attributes available at decision time) and attribute-blind decision-making (sensitive attributes excluded from prediction). We show that, in the attribute-aware regime, fair ML necessarily (weakly) improves outcomes for the disadvantaged group and (weakly) worsens outcomes for the advantaged group. In contrast, in the attribute-blind regime, the impact of fairness is distribution-dependent: fairness can benefit or harm either group and may shift both groups' outcomes in the same direction, leading to either leveling up or leveling down. We characterize the conditions under which these patterns arise and highlight the role of "masked" candidates in driving them. Overall, our results provide structural guidance on when pursuing algorithmic fairness is likely to improve group outcomes and when it risks systemic leveling down, informing fair ML design and deployment choices.
Yi Yang, Xiangyu Chang, Pei-yu Chen
Department of Information Systems, Arizona State University · Department of Information Systems and Intelligent Business, Xi’an Jiaotong University
Exact demographic parity requires the same prediction distribution across sensitive groups. In regression, this can be too restrictive and reduce predictive accuracy. We instead bound every pairwise Wasserstein distance between group-specific prediction distributions by a chosen tolerance, allowing the groups to differ while keeping fairness tunable. On the real line, the prediction and constrained transport problems have the same infimum. If the transport problem attains this value and suitable Monge maps exist, a deterministic transport-form predictor realizes it. Under a standard one-dimensional condition, the problem becomes a bounded quantile optimization that is convex when the conditional expected loss is convex in prediction. Under squared loss with two positively weighted groups, the W1, W2, and W∞ constraints yield soft-thresholding, shrinkage, and clipping. Under the stated conditions, we establish rates for the known-cost plug-in estimator. When the cost is unavailable, we describe an observed-loss computational proxy. Synthetic and California 2018 ACS studies illustrate the tunable fairness-accuracy trade-off and distinguish calibration feasibility from held-out fairness and accuracy.
Public institutions increasingly procure AI systems whose design they cannot inspect or change. In higher education, proprietary Early Warning Systems (EWS) leave colleges with few options beyond adjusting model outputs to address inequity. This raises the question of how fairness work is coordinated among vendors, institutions, advisors, and students with unequal power to change these systems? Using student records from a public college in Ontario, Canada, we evaluate six post-hoc fairness interventions on a research EWS under simulated procurement constraints. We compare fairness, accuracy, and demographic disparities, introducing error-type profiling to trace how interventions redistribute false positives and false negatives. Interventions redistributed disparities without consistently reducing them. Two implementations favored already-advantaged groups because they used group size to define disadvantage; small, marginalized groups remained poorly served. These findings show how procurement constraints and implementation choices shape the possibilities for fairness work. We call the resulting condition fairness theatre; dashboard metrics converge while groups' error burdens persist or worsen.
Kelly McConvey, Angelina Zhai, Rebecca Li +1
University of Toronto, Toronto, Ontario, Canada · Georgia Institute of Technology, Atlanta, Georgia, USA