cs.LGNov 11, 2026

Characterizing Predictive Multiplicity in Ranking and Its Fairness Implications

Authors: Ainhize Barrainkua, Mikel Malagón

Organizations: University of the Basque Country (EHU)

Abstract

Prediction problems often admit competing models that achieve comparable performance yet produce conflicting outputs, a phenomenon known as predictive multiplicity. While this challenge has been extensively studied in classification tasks, its implications remain largely unexplored in ranking frameworks, which are critical for high-stakes societal decisions such as the allocation of welfare resources and social assistance. This paper investigates the phenomenon of predictive multiplicity in score-based ranking problems. We introduce formal definitions to capture multiplicity in ranking settings, supported by foundational theoretical guarantees. Furthermore, we examine the dual relationship between predictive multiplicity and algorithmic fairness in ranking problems: we analyze how model multiplicity can act as a potential source of unfairness, while demonstrating how exploring this decision space provides a principled avenue for discovering less discriminatory rankings. Finally, we complement our theoretical contributions with insightful empirical evaluations on both synthetic and real-world datasets, validating our analytical findings and illustrating their practical implications. Taken together, this work highlights the real-world relevance of predictive multiplicity in ranking and sheds light on the potential of leveraging this effect to achieve fairer outcomes.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Mar 12, 2026

Trustworthy Calibration or Vulnerability? Unveiling the Over-Calibration Paradox and Attack Surface in Rashomon Sets

Post-hoc calibration is standard practice for aligning classifier confidence scores with empirical frequencies in high-stakes decision systems. However, its downstream effects on decision stability across competing, equally accurate models remain largely unexamined. In this paper, we formalize the over-calibration paradox within the framework of predictive multiplicity, reflecting the Rashomon effect where competing models with near-identical loss assign conflicting predictions to the same individual. Evaluating nine credit risk portfolios, we examine the impact of Platt Scaling, Isotonic Regression, and Beta Calibration on individual prediction agreement and expected calibration error. We identify two distinct operating regimes. In systematically miscalibrated models, post-hoc calibration consistently aligns probability estimates and reduces prediction disagreement. Conversely, when baseline models are already well-calibrated, post-hoc calibration, particularly non-parametric estimators constrained by finite sample sizes, overfits to hold-out noise. This distortion shifts decision thresholds near the classification boundary, paradoxically increasing both expected calibration error and predictive multiplicity. From an auditing and security standpoint, this paradox exposes a practical vulnerability to soft adversarial manipulation, whereby an interested party or automated model search can exploit calibration choices to steer specific individual outcomes while preserving aggregate performance and passing black-box evaluations.
Sep 30, 2026

A Rank Graduation metric for Algorithmic fairness

Fairness assessment in algorithmic decisions that affect individuals, such as credit scoring, often relies on parity measures calculated at the aggregate group level. Such measures may not reveal which individuals experience unfairness or which explanatory factors contribute to it. In this paper, we propose a rank-based framework that evaluates fairness through the distribution of model prediction errors, thereby linking fairness assessment with predictive accuracy and explainability. The framework combines Rank Graduation Fairness (RGF), its integrated measure AURGF, a centered Cramer--von Mises permutation test, and a feature removal procedure for fairness explainability. We evaluate the methodology using logistic regression, random forest, gradient boosting, and a multilayer perceptron. The simulation study shows that protected-group imbalance can reverse descriptive fairness comparisons, whereas the proposed inferential procedure correctly distinguishes fair from unfair mechanisms. Its application to HMDA mortgage data produces model rankings that differ from those obtained with classical fairness criteria. Tree-based models, rather than logistic regression, provide the strongest combination of predictive accuracy and rank-based fairness, while the fairness null hypothesis is rejected for all four models. The persistence of disparity across statistical, bagging, boosting, and neural network specifications, together with the feature removal results, indicates that the observed unfairness is not specific to a single algorithm or predictor, but is associated with group differences embedded in the characteristics of the lending data. These findings support a broader approach to trustworthy artificial intelligence that combines predictive accuracy, fairness measurement, statistical inference, and explainability.
Jun 16, 2026

A fairness-aware extension of Stochastic Multicriteria Acceptability Analysis for ranking

Fairness has become a central concern in ranking problems involving individuals or social groups, particularly under the Responsible Artificial Intelligence agenda. In Multi-Criteria Decision Analysis, Stochastic Multicriteria Acceptability Analysis (SMAA) provides a robust framework for handling uncertainty and incomplete preference information, but it does not explicitly address fairness in the resulting rankings. This paper proposes SMAA-Fair, a fairness-aware extension of SMAA for ranking problems. The approach reweights the simulated rankings generated by SMAA according to their level of group fairness, so that fairer rankings contribute more strongly to the acceptability indices and central weights vector. The framework is independent of the aggregation model and can incorporate different fairness metrics. In this study, Statistical Parity, normalized discounted Kullback--Leibler divergence (rKL) and normalized discounted cumulative Kullback--Leibler divergence (nDKL) are adopted. Rankings are derived from the fairness-adjusted acceptability matrix using expected ranking and maximum acceptability ranking. We also derive the central weight according to the degree of fairness in the obtained rankings. Numerical experiments with synthetic and real data show that SMAA-Fair improves the representation of protected groups among favourable ranking positions, while preserving robustness to preference uncertainty.