Operationalizing Fairness: Post-Hoc Threshold Optimization Under Hard Resource Limits
Authors: Moirangthem Tiken Singh, Amit Kalita, Sapam Jitu Singh
Organizations: Department of Computer Science and Engineering, DUIET, Dibrugarh University, Assam, India · Department of Computer Science and Engineering, MIT, Manipur University, 795003, India
The deployment of machine learning in high-stakes domains requires a balance between predictive safety and algorithmic fairness. However, existing fairness interventions often as- sume unconstrained resources and employ group-specific decision thresholds that violate anti- discrimination regulations. We introduce a post-hoc, model-agnostic threshold optimization framework that jointly balances safety, efficiency, and equity under strict and hard capacity constraints. To ensure legal compliance, the framework enforces a single, global decision thresh- old. We formulated a parameterized ethical loss function coupled with a bounded decision rule that mathematically prevents intervention volumes from exceeding the available resources. An- alytically, we prove the key properties of the deployed threshold, including local monotonicity with respect to ethical weighting and the formal identification of critical capacity regimes. We conducted extensive experimental evaluations on diverse high-stakes datasets. The principal re- sults demonstrate that capacity constraints dominate ethical priorities; the strict resource limit determines the final deployed threshold in over 80% of the tested configurations. Furthermore, under a restrictive 25% capacity limit, the proposed framework successfully maintains high risk identification (recall ranging from 0.409 to 0.702), whereas standard unconstrained fairness heuristics collapse to a near-zero utility. We conclude that theoretical fairness objectives must be explicitly subordinated to operational capacity limits to remain in deployment. By decou- pling predictive scoring from policy evaluation and strictly bounding intervention rates, this framework provides a practical and legally compliant mechanism for stakeholders to navigate unavoidable ethical trade-offs in resource-constrained environments.
Figures & tables
Figure 1: Overview of the risk assessment system architecture.
Baseline Policy
Operational Interpretation
Relationship to Framework
Performance-Optimal
Maximizes aggregate accuracy
Ignores C and γ
Risk-Averse
Prioritizes safety (minimizes FN)
Extreme α weighting, ignores C
Inclusion-Oriented
Pure capacity enforcement
Defaults strictly to τ(C)
Fairness-Aware
Minimizes disparity ( Δ ) constraint
High γ weighting, ignores C
Table 1: Illustrative ethical threshold policies serving as operational baselines.
Figure 2: Comprehensive analysis of fair decision-making under capacity constraints. (A) Safety-capacity trade-off showing a near-linear increase in recall with the available intervention capacity across datasets. (B) Pareto frontier in the efficiency-equity plane (false-positive rate versus disparity), illustrating inherent trade-offs under constrained resources. (C) Proportion of configurations in which the capacity constraint is binding ( τ∗=τ(C) ) across models and datasets, consistently exceeding 70–85%. (D) Distribution of optimized decision thresholds τ∗ per dataset and model, with many solutions clustering near the capacity-imposed bound (the dashed red line denotes an example threshold at C=0.2 ).
Dataset
Strategy
Recall
Disparity ( Δ TPR)
Intervention Rate
Feasibility
ACS Income
Proposed Framework
0.702
0.254
0.250
Feasible
Unconstrained Optimization
0.993
0.000
0.990
Infeasible
Demographic Parity
0.181
0.000
0.172
Feasible (low utility)
Equalized Odds
0.260
0.000
0.250
Feasible
Random Allocation
0.250
0.000
0.250
Feasible (no signal)
COMPAS
Proposed Framework
0.409
0.317
0.250
Feasible
Table 2: Operational feasibility and performance at fixed capacity C=0.25 . Unconstrained optimization consistently violates the resource limit by nearly four × , rendering it undeployable. Heuristic fairness constraints deliver severely degraded utility, with Equalized Odds performing no better than the expected recall of Random Allocation ( ≈C ). Only the Proposed Framework achieves high risk identification while strictly respecting the budget.
Figure 3: Focused analysis of sensitivity to individual ethical weighting under fixed capacity C=0.25 . (A) Safety Saturation: As the safety weight α increases, the constraint activation rate (red dashed line) rises steeply and attains 100% for α≥3.0 . For larger values of α , further increases do not induce any additional operational changes, indicating the saturation regime. (B) The Cost of Fairness: Increasing the equity weight γ monotonically decreases demographic disparity (purple dashed line), but simultaneously necessitates a reduction in Recall (green solid line). This behavior exemplifies the inherent trade-off between fairness and predictive performance under a fixed capacity constraint.
Figure 4: Comprehensive parameter sensitivity analysis across the full factorial design. (A) Recall as a function of the safety weight α . (B) Disparity Δ as a function of equity weight γ . (C) Constraint activation rate as a function of capacity constraint C . (D) Joint parameter space of α and γ colored by the mean recall. The shaded regions indicate variability across the models and bootstrap resamples. All values averaged over configurations.
C
Mean Recall
Mean Δ TPR
Constraint Activation (%)
Feasible Region (%)
0.10
0.255
0.142
98.7
1.3
0.15
0.312
0.138
94.2
5.8
0.20
0.368
0.131
87.6
12.4
0.25
0.425
0.127
80.8
19.2
0.30
0.525
0.119
68.4
31.6
0.40
0.682
0.108
42.1
57.9
Table 3: Ablation study on the capacity constraint C . Values are averaged across all models, weight combinations, and bootstrap resamples (N=1,000 per configuration).
Dataset
Model Architecture
Recall (95% CI)
Disparity Δ (95% CI)
Constraint Active
ACS Income
Gradient Boosting
0.613 [0.603, 0.623]
0.068 [0.066, 0.071]
86.9%
Logistic Regression
0.556 [0.546, 0.566]
0.094 [0.092, 0.097]
82.6%
Random Forest
0.542 [0.528, 0.555]
0.196 [0.192, 0.200]
90.0%
COMPAS
Gradient Boosting
0.319 [0.310, 0.329]
0.153 [0.149, 0.158]
83.4%
Logistic Regression
0.281 [0.271, 0.290]
0.167 [0.162, 0.173]
79.6%
Random Forest
0.275 [0.264, 0.286]
0.204 [0.197, 0.211]
77.3%
Table 4: Comprehensive performance analysis across all model architectures under capacity constraint C=0.25 . The high rate of constraint activation (73%–90%) across diverse algorithms confirms that resource limits, not model choice, drive the decision boundary.
Method
Stage
Model-agnostic
Constraint Handling
Multi-obj.
Budget
Hard limit
Hardt et al. [ 15 ]
Post
✓
✗
✗
△
Pleiss et al. [ 16 ]
Post
✓
✗
✗
✗
Zafar et al. [ 21 ]
In
✗
✗
✗
△
Corbett-Davies et al. [ 28 ]
Post
✓
△
△
✗
Xian et al. [ 17 ]
Post
✓
✗
✗
△
Table 5: Comparison of representative fairness-aware classification approaches. Symbols: ✓= supported, △ = partially supported, ✗= not supported.
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
Fairness audits in production ML typically occur once, at deployment, on a single domain. Both fail in practice: fairness can shift after retraining or a changing user base, and interventions validated on one dataset are rarely tested across the heterogeneous domains an organization deploys. We present FAPE (Fairness Auditing for Production Environments), a four-stage framework evaluating a single post-processing intervention, Fairlearn's ThresholdOptimizer, across eight domain evaluations: criminal justice, income prediction, legal admissions, credit lending, agricultural lending, a multi-domain benchmark corpus, healthcare, and education. Each is scored on demographic parity and equalized odds difference, plus disparate impact ratio and accuracy cost where computable. Intervention effectiveness tracks baseline disparity magnitude: across model-domain pairs the constraint improved disparity in 9 of 14 high-disparity cases and worsened it in 3 of 4 near-fair ones. Each of the five high-disparity exceptions reverses under one of two measurement checks, a minimum group size or thresholds fit on held-out data. A CUSUM monitor started at deployment, tested on a simulated shift, separates constrained models that never met a 0.1 parity convention from those that met it and later regressed. A single deployment-time audit is therefore an unreliable guide, which argues for baseline-disparity screening and continuous monitoring
Nithin Raghava Ramachandra Narla
Independent Researcher, Dallas-Fort Worth, Texas, United States
Designing fair algorithmic decision systems requires balancing model performance with fairness toward affected individuals: More fairness might require sacrificing some performance and vice versa, yet the space of possible trade-offs is still poorly understood. We investigate fairness in binary prediction-based decision problems by conceptualizing decision making as a multi-objective optimization problem that simultaneously considers decision-maker utility and group fairness. We investigate the set of Pareto-optimal decision rules for arbitrary utility functions for decision maker, arbitrary population distributions, and a wide range of group fairness metrics. We find that the Pareto frontier consists of deterministic, group-specific threshold rules applied to individuals' success probability. This complements existing optimality theorems from literature which, for specific fairness constraints, posit lower-bound threshold rules only. However we also show that, depending on the used fairness metric, the Pareto frontier may include upper-bound threshold rules, thus preferring individuals with lower success probabilities. We show that the location of the Pareto frontier depends only on population characteristics, utility functions and fairness score, but not on the technical design of the algorithm - our findings hold for pre-, in-, and post-processing approaches alike. Our results generalize existing optimality theorems for fairness-constrained classification and extend them to generalized fairness metrics and fairness principles, and to partial fairness regimes. This paper connects formal fairness research with legal and ethical requirements to search for less discriminatory alternatives, offering a principled foundation for evaluating and comparing algorithmic decision systems.
Mieke Wilms, Christoph Heitz
Zurich University of Applied Sciences / University of Zurich, Switzerland · Zurich University of Applied Sciences, Switzerland
As artificial intelligence is increasingly deployed, algorithmic unfairness has raised growing concerns and intensified demands for transparent fairness auditing. In practice, the tolerable degree of algorithmic unfairness depends on the specific legal, ethical, or application context. Given a prespecified tolerance threshold, an important statistical question is how to determine whether a group disparity exceeds the allowable tolerance across different auditing objectives. To address this problem, we develop a unified tolerance-based fairness auditing framework for two complementary auditing objectives: violation certification, which prioritizes control of false violation declarations, and sensitivity screening, which prioritizes reducing missed violations. For the first objective, we develop a constrained empirical likelihood test for formal settings that uses least-favorable-point calibration and can be combined with false flagging rate control for simultaneous subgroup auditing. For the second objective, we develop split empirical likelihood and adjusted split empirical likelihood tests using an adaptive boundary-proxy principle for early-warning settings. Numerical experiments show the distinct error-control--sensitivity trade-offs of these procedures. A COMPAS analysis illustrates the framework in predictive fairness auditing.
Jie Tang, Chuanlong Xie, Lixing Zhu
School of Statistics, Beijing Normal University, Zhuhai, Guangdong 519087, China