No evaluation without fair representation : Impact of label and selection bias on the evaluation, performance and mitigation of classification models
Authors: Magali Legast, Toon Calders, François Fouss
Organizations: Université catholique de Louvain, Institute of Information and Communication Technologies, Electronics and Applied Mathematics, Louvain-la-Neuve, Belgium · Universiteit Antwerpen, Adrem Data Lab, Antwerpen, Belgium · Université catholique de Louvain, Louvain Research Institute in Management and Organizations, Institute of Information and Communication Technologies, Electronics and Applied Mathematics, Mons, Belgium
Bias can be introduced in diverse ways in machine learning datasets, for example via selection or label bias. Although these bias types in themselves have an influence on important aspects of fair machine learning, their different impact has been understudied. In this work, we empirically analyze the effect of label bias and several subtypes of selection bias on the evaluation of classification models, on their performance, and on the effectiveness of bias mitigation methods. We also introduce a biasing and evaluation framework that allows to model fair worlds and their biased counterparts through the introduction of controlled bias in real-life datasets with low discrimination. Using our framework, we empirically analyze the impact of each bias type independently, while obtaining a more representative evaluation of models and mitigation methods than with the traditional use of a subset of biased data as test set. Our results highlight different factors that influence how impactful bias is on model performance. They also show an absence of trade-off between fairness and accuracy, and between individual and group fairness, when models are evaluated on a test set that does not exhibit unwanted bias. They furthermore indicate that the performance of bias mitigation methods is influenced by the type of bias present in the data. Our findings call for future work to develop more accurate evaluations of prediction models and fairness interventions, but also to better understand other types of bias, more complex scenarios involving the combination of different bias types, and other factors that impact the efficiency of the mitigation methods, such as dataset characteristics.
Figures & tables
Article
Dual dataset
Main findings based on fair test set
Known bias
Avoid criticized datasets
Friedler et al. (2019)
✗
✗
✗
✗
Biswas and Rajan (2020)
✗
✗
✗
✗
Jones et al. (2020)
✗
✗
✓
✱
Fairea ( Hort et al., 2021 )
✗
✗
✗
✗
Ganesh et al. (2025)
✗
✗
✗
✱
Chen et al. (2023)
✗
✗
✗
✗
Table 1. Overview of the evaluation settings in related work compared to ours. A ✱ indicates a more nuanced response, like the use of both criticized datasets and more than two other alternatives.
Figure 1. Fair World Framework. The fair world represents an ideal situation in which the desired fairness criteria hold. The available observable data is a distorted representation of that world, typically obtained through an unknown biasing process. Such data is traditionally used to both train prediction models and evaluate their performance. 3 3 3 All symbols composing the diagrams in Figure 3 and 3 have been designed by OpenMoji , the open-source emoji and icon project, and were adapted for this document. License: CC BY-SA 4.0
Figure 2. Comparison of metric results evaluated on a biased test set (Biased evaluation) or a fair test set (Fair evaluation) for values in (−1,1) . Points that fall outside of the x=y diagonal indicate that the metric measurement is skewed by the bias present in the unfair test set. Are included the results for 4 datasets with bias intensities ranging from 0 to 0.9, 3 fairness-agnostic training algorithms, 8 bias mitigation procedures, and unmitigated models.
Figure 3. Biasing and evaluation framework. We assume that a given dataset with high fairness level, the fair data , accurately represents the fair world. We introduce a specific bias type in this fair data to obtain datasets with known bias. This biased data can be used to train prediction models with or without bias mitigation. The performance and fairness of these models can then be evaluated using the fair data. They can also be compared to baseline models trained on that fair data.
Figure 4. Experiment pipeline representing data biasing, model training (with or without bias mitigation), and evaluation of models on fair data. Thin dashed arrows indicate where each type of data is used. Bold dashed arrows indicate an optional path. We do not cumulate the use of pre- and pot-processing methods.
Dataset
Size
Target
Sens. attr.
Unpriv. group
Unpriv. prop.
SPD
BCC
P(y=1)
Student
649
G3 (final grade)
sex
Boys
40.9%
-0.057
0.721
0.846
StudentBalanced
532
G3 (final grade)
sex
Boys
50%
0.0
0.680
0.812
OULADstem
7040
final_result
gender
Men
81.7%
-0.025
0.662
0.482
OULADsocial
7632
final_result
gender
Men
11.6%
-0.026
0.596
0.487
Table 2. Overview of datasets characteristics
Figure 5. Fair and biased evaluation of Accuracy and EqOd for increasing levels of label bias injected in OULADstem training sets
Figure 6. Fair and biased evaluation of Accuracy and SPD for increasing levels of malicious selection injected in OULADstem training sets.
Figure 7. Evolution of RF models performance when trained on datasets presenting increasing levels of label bias , as evaluated on unbiased data.
Figure 8. Evolution of RF models performance when trained on datasets presenting increasing levels of random selection , as evaluated on unbiased data. pu=1 corresponds to the complete removal of the unprivileged group from the training data.
Figure 9. Evolution of RF models performance when trained on datasets presenting increasing levels of self-selection , as evaluated on unbiased data. pu=1 corresponds to the complete removal of the unprivileged group from the training data.
Figure 10. Evolution of RF models performance when trained on datasets presenting increasing levels of malicious selection , as evaluated on unbiased data. When pu=1 , the members of the privileged (resp. unprivileged) group all receive positive (resp. negative) outcomes.
Figure 11. Evolution of RF models performance trained on datasets from which the unprivileged group is excluded and presenting increasing levels of malicious selection , as evaluated on unbiased data. Models at pu=0 have been trained with the original dataset containing the two groups.
Bias
Reweighing
Massaging
FTU
EOP
CEO
ROC-SPD
ROC-EqOp
ROC-AvOd
Label
✓
✓
✓
✗
✗
✓✓
✱
✱
Malicious select.
✓✓
✗✗
✓✓
✗
✗✗
✗✗
✱
✱
Self-select.
✓✓
✗✗
✓✓
✗
✗✗
✗✗
✗✗
✗
Table 3. Overview of methods ability to mitigate each bias type. ✓ indicates an overall positive impact (excluding results for the highest and less realistic bias levels). The symbol is doubled when the mitigated results are close to the fair baseline. ✗ indicate a mostly negative or neutral effect. The symbol is doubled when the mitigated models introduce significant unfairness. ✱ indicates results that may be positive or negative depending on the dataset and bias level.
Figure 12. Evolution of accuracy and fairness metrics for RF models trained on data with increasing levels of label bias and evaluated on unbiased data. The fair baseline is indicated by a dashed horizontal black line.
Figure 13. Evolution of accuracy and fairness metrics for RF models trained on data with increasing levels of self-selection and evaluated on unbiased data. The fair baseline is indicated by a dashed horizontal black line.
Figure 14. Evolution of accuracy and fairness metrics for RF models trained on data with increasing levels of malicious selection and evaluated on unbiased data. The fair baseline is indicated by a dashed horizontal black line.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 15. Comparison of metric results evaluated on a biased test set (Biased evaluation) or a fair test set (Fair evaluation). Points that fall outside of the x=y diagonal indicate that the metric measurement is skewed by the bias present in the unfair test set. Are included the results for all four datasets with bias intensities ranging from 0 to 0.9, the three fairness-agnostic training algorithms, the eight bias mitigation procedures, and unmitigated models.
Figure 16. Fair and biased evaluation of Accuracy and EqOd for increasing levels of label bias injected in OULADsocial training sets.
Figure 17. Fair and biased evaluation of Accuracy and EqOd for increasing levels of label bias in StudentBalanced training sets.
Figure 18. Fair and biased evaluation of Accuracy and SPD for increasing levels of self-selection injected in OULADstem training sets.
Figure 19. Fair and biased evaluation of Accuracy and SPD for increasing levels of self-selection in OULADsocial training sets.
Figure 20. Fair and biased evaluation of Accuracy and SPD for increasing levels of self-selection in StudentBalanced training sets.
Figure 21. Fair and biased evaluation of Accuracy and SPD for increasing levels of malicious selection in OULADsocial training sets.
Figure 22. Fair and biased evaluation of Accuracy and SPD for increasing levels of malicious selection in StudentBalanced training sets.
Figure 23. Effect of randomly undersampling the whole training set on RF models. When pu=1 , the number of individuals removed corresponds to the size of the unprivileged group.
Figure 24. Evolution of accuracy and fairness metrics for RF models trained OULADsocial with increasing bias levels and evaluated on unbiased data. The fair baseline is indicated by a dashed horizontal black line. For label bias and self-selection, aggregated metrics (Accuracy and GEI) often have higher values than for OULADstem because the privileged group is a minority in OULADsocial and a majority in OULADstem.
Figure 25. Evolution of Balanced Conditioned Consistency (BCC) for RF models trained on data with increasing bias levels and evaluated on unbiased data. The fair baseline is indicated by a dashed horizontal black line.
Algorithmic fairness methods are increasingly used to identify and mitigate bias in machine learning models, yet most approaches are evaluated in isolation and along single demographic axes. This limits practical guidance for selecting fairness strategies, where disparities may arise across intersectional subgroups and across multiple stages of the modeling lifecycle. This work presents FairSelect, a toolkit for systematically evaluating fairness mitigation strategies applied individually and in combination across preprocessing, inprocessing, and postprocessing stages. FairSelect supports multiple model architectures, intersectional subgroup evaluation, and comparison of fairness utility tradeoffs across baseline, single method, and multi level configurations. The framework was validated using synthetic clinical datasets designed to represent specific bias mechanisms and a real-world replication of two-year stroke risk prediction among patients with atrial fibrillation. Synthetic experiments showed that targeted fairness methods generally reduced intended subgroup disparities, while combined strategies produced larger average fairness improvements with modest utility tradeoffs. In the clinical prediction task, mitigation effects were highly variable, with some combinations improving both fairness and predictive performance while others were ineffective or counterproductive. These findings demonstrate that fairness interventions interact in nonadditive and context dependent ways. FairSelect provides a practical framework for systematically identifying fairness strategies that improve subgroup equity while preserving model performance in clinical machine learning.
Nick Souligne, Isabella Mixton-Garcia, Vignesh Subbian
College of Engineering, University of Arizona, Tucson, AZ
Machine learning models have been shown to exhibit discriminatory outcomes or degraded performance for individuals at the intersection of multiple sensitive attributes, such as race and gender. This stems in part from two interrelated challenges: the lack of principled measures for quantifying bias (potentially intersectional), and insufficient representation of intersectional subgroups in training data. We extend a recent bias mitigation framework to incorporate coverage constraints that enforce sufficient representation across groups, including intersectional subgroups. Since achieving exactly zero bias for all groups may not be data efficient (meaning it may require large amounts of data), our solution trades small approximation errors in bias for greater data efficiency while satisfying coverage constraints. We also formulate bias mitigation as an integer linear program that optimizes over all mitigation strategies, and characterize the price of fairness, the minimum data modification cost, as a function of fairness tolerance. This is essential both for legal compliance, where regulations may mandate specific fairness thresholds, and for data governance, enabling practitioners to make informed trade-offs between bias reduction and data modification (particularly, data purchasing) costs. We evaluate our techniques on publicly available datasets, demonstrating that bias mitigation via our framework preserves predictive accuracy across multiple classifiers, and that coverage constraints, while motivated by statistical considerations, are essential for preserving downstream ML performance.
Bruno Scarone, Alfredo Viola, Renée J. Miller
Khoury College of Computer Sciences, Northeastern University, USA · Casa de Investigadores Científicos La Comarca, Uruguay · Cheriton School of Computer Science, University of Waterloo, Canada
The evaluation of fairness in machine learning systems has become a central concern in high-stakes applications, including biometric recognition, healthcare decision-making, and automated risk assessment. Existing approaches typically rely on a small number of fairness metrics to assess model behaviour across group partitions, implicitly assuming that these metrics provide consistent and reliable conclusions. However, different fairness metrics capture distinct statistical properties of model performance and may therefore produce conflicting assessments when applied to the same system. In this work, we investigate the consistency of fairness evaluation by conducting a systematic multi-metric analysis of demographic bias in machine learning models. Using face recognition as a controlled experimental setting, we evaluate model performance across multiple group partitions under a range of commonly used fairness metrics, including error-rate disparities and performance-based measures. Our results demonstrate that fairness assessments can vary significantly depending on the choice of metrics, leading to contradictory conclusions regarding model bias. To quantify this phenomenon, we introduce the Fairness Disagreement Index (FDI), a measure designed to capture the degree of inconsistency across fairness metrics. We further show that disagreement remains high across thresholds and model configurations. These findings highlight a critical limitation in current fairness evaluation practices and suggest that single-metric reporting is insufficient for reliable bias assessment.
Khalid Adnan Alsayed
School of Computing, Engineering & Digital Technologies, Teesside University, UK