Explanation-Aware Fairness Beyond Output Parity: Joint Output-Attribution Regularization for Medical Image Classification
Organizations: Idiap Research Institute, Martigny, Switzerland · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Universidade Federal de São Paulo (UNIFESP), São Paulo, Brazil
Abstract
Output-level fairness does not constrain whether demographic groups rely on similar predictive contributions in representation space. For a nonlinear encoder with a linear head, we show that outcome-conditioned output score disparity equals the signed sum of feature attribution differences across subgroups by allowing small prediction gaps to arise through cancellation effect. We test whether an attribution disparity penalty complements output-level equalized odds based fairness surrogates. Across 960 DINOv2-LoRA models on HAM10000 and Harvard-GF datasets for gender and race fairness, attribution augmentation increases utility-fairness trade-off performance for hypervolume in six different settings with paired 95% confidence intervals and relative gains of 1.49-10.57%. Under a predefined utility constraint, augmented model families also achieve 9.0-38.7% lower equalized odds difference, lower attribution disparity, and reduced cancellation ratio while preserving predictive utility. These results support attribution fairness as a complementary explanation-aware constraint on output fairness.
Figures & tables
| Pair | HV: out joint. | HV (+rel.) [95% CI] |
|---|---|---|
| HAM10000 | ||
| MeanEodd Joint 1 | (+7.02%) [0.0072,0.0306] | |
| Fairret Joint 2 | (+1.49%) [0.0006,0.0255] | |
| DiffEodd Joint 3 | (+10.57%) [0.0175,0.0432] | |
| Harvard-GF | ||
| MeanEodd Joint 1 | (+4.14%) [0.0040,0.0293] | |
| Dataset | Pair | EOD ( ) | BAcc ( ) | AUROC ( ) | ||
|---|---|---|---|---|---|---|
| HAM10000 | MeanEodd Joint 1 | (23.9%) | ( ) | ( ) | (58.2%) | ( ) |
| Fairret Joint 2 | (35.8%) | ( ) | ( ) | (42.5%) | ( ) | |
| DiffEodd Joint 3 | (38.7%) | ( ) | ( ) | (42.7%) | ( ) | |
| Harvard-GF | MeanEodd Joint 1 | (26.5%) | ( ) | ( ) | (82.0%) | ( ) |
| Fairret Joint 2 | (14.0%) | ( ) | ( ) | (66.5%) | ( ) | |
| DiffEodd Joint 3 | (9.0%) | ( ) | ( ) | (81.7%) | ( ) |
Group Fairness
Group fairness measures and reduces disparities in how algorithms predict, rank, allocate resources, and make decisions for different groups.
See all 157 papers