Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

Aditya Parikh; Eike Petersen; Stella Frank; Enzo Ferrante; Melanie Ganz; Aasa Feragen

cs.LGSep 25, 2026

Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

Authors: Aditya Parikh, Eike Petersen, Stella Frank, Enzo Ferrante, Melanie Ganz, Aasa Feragen

Organizations: Technical University of Denmark, Kgs. Lyngby, Denmark · Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany · CONICET – Universidad de Buenos Aires, ICC, Buenos Aires, Argentina · University of Copenhagen, Copenhagen, Denmark

Abstract

Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness.

Figures & tables

Figure 1: Two forms of representation invariance and the effects of enforcing those in a case study predicting disease for two groups ( $a_{1}$ and $a_{2}$ ) with different prevalence. Top: $Z\perp A$ gives identical marginal risk distributions across groups. Under unequal prevalence, the class-conditional risk distributions must differ, giving different error rates across groups. Bottom: $Z\perp A\mid Y$ gives equal error rates, but still has drawbacks: it is incompatible with calibration by groups and requires equalizing away potentially important differences between unknown disease subtypes.

Figure 2: Effects of enforcing Class-conditional representation invariance in the presence of intra-class variation. When the within-class distribution of a clinically relevant factor (here disease severity among

Y{=}1

patients) differs between groups, enforcing

Z\perp A\mid Y

requires mapping patients with differing disease severities to the same latent representation, depending on which group a patient belongs to (center). The aligned representation then hides a real difference in disease presentation, and the same

z

no longer means the same severity in both groups (see also Lohaus et al. (2022) ).

Figure 3: Fig. 1 estimated from data (Dutch, shift-50, seed 42). Rows: ERM,

Z\perp A

Z\perp A\mid Y

. Left: marginal risk densities

p(r\mid a)

A\in\{0,1\}

, with

\Delta_{\mathrm{DP}}

and

D_{A}

. Middle and right: class-conditional densities

p(r\mid y,a)

within each group, with

\Delta_{\mathrm{EO}}

and

D_{A\mid Y}

. Dotted: Youden threshold. Marginal alignment makes the left column coincide by pulling the within-class curves apart across groups (

\mathrm{P}_{1}

); class-conditional alignment makes the middle and right columns coincide and leaves the left column at the base-rate gap (

\mathrm{P}_{3}

		Utility	Decodability of $A$		Indep.	Separation	Sufficiency	Group calibration
	Method	AUROC	$D_{A}$	$D_{A\mid Y}$	$\Delta_{\mathrm{DP}}$	$\Delta_{\mathrm{EO}}$	$\Delta_{\mathrm{PPV}}\,/\,\Delta_{\mathrm{NPV}}$	$\Delta_{\mathrm{bias}}\,/\,\Delta_{\mathrm{cal}}$
Adult	ERM	$0.89$	$0.93$	$0.99$	$0.40$	$0.28$	$0.18\,/\,0.05$	$0.02\,/\,0.32$
	$Z\perp A$	$0.79$	$0.58$	$0.85$	$\mathbf{0.04}\,\checkmark$	$0.40$	$0.49\,/\,0.04$	$0.23\,/\,2.12$
	$Z\perp A\mid Y$	$0.86$	$0.62$	$0.55$	$0.17$	$\mathbf{0.08}\,\checkmark$	$0.40\,/\,0.07$	$0.15\,/\,1.80$
Dutch	ERM	$0.90$	$0.77$	$0.77$	$0.30$	$0.05$	$0.27\,/\,0.28$	$0.22\,/\,2.00$
	$Z\perp A$	$0.71$	$0.52$	$0.79$	$\mathbf{0.01}\,\checkmark$	$0.31$	$0.63\,/\,0.21$	$0.44\,/\,2.32$

Table 1: Fairness criteria under the shift-50 on test split, mean over

3

seeds.

Z\perp A

is marginal FNF at

\gamma=1

(CheXpert:

\gamma=0.95

; strongest setting),

Z\perp A\mid Y

class-conditional FNF at the same

\gamma

, both with the feature set and width selected by the lowest probe AUROC.

D_{A}

and

D_{A\mid Y}

are the maximum probe AUROC for recovering

A

from

Z

marginally and within class.

\checkmark

denotes the criterion each invariance target guarantees (

\mathrm{P}_{1,3}

); every other

\Delta

is not-stable relative to ERM.

\pm

std are omitted for space and reported, together with the full

\gamma

and feature-set sweep and the remaining CheXpert configurations, in Appendix A.4 and A.3 .

Figure 4: Guarantees under a change of evaluation population (

\mathrm{P}_{4}

). Trained on the equalized condition (shaded) and evaluated on increasing base-rate gap:

P(Y\mid A)

changes,

P(X\mid Y,A)

fixed.

Z\perp A

loses parity as the gap grows;

Z\perp A\mid Y

keeps

\Delta_{\mathrm{EO}}

flat and loses parity by exactly the base-rate gap.

\Delta_{\text{bias}}

grows. Mean

\pm

std over three seeds.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Fairness gaps increase with the base-rate gap. Top:

\Delta_{\mathrm{DP}}

; Middle:

\Delta_{\mathrm{EO}}

; Bottom:

\Delta_{\mathrm{bias}}

against the training base-rate gap

|\pi_{1}-\pi_{0}|

over the four prevalence conditions, per dataset and method. Marginal representation invariance harms separation for parity at a rate set by the gap (

\mathrm{P}_{1}

); class-conditional representation invariance holds

\Delta_{\mathrm{EO}}

(

\mathrm{P}_{3}

) but not

\Delta_{\mathrm{bias}}

(

\mathrm{P}_{3}

). Mean

\pm

std over three seeds reported.

Figure 6: Every metric against the training base-rate gap

|\pi_{1}-\pi_{0}|

(equalized, natural, shift-50, shift-75; matched test). Dotted line in the

\Delta_{\mathrm{DP}}

row:

\Delta_{\mathrm{DP}}=|\pi_{1}-\pi_{0}|

. Mean

\pm

sd over seeds.

Figure 7: Summed group error

e_{0}+e_{1}

against the bound

\max\{0,|\pi_{0}-\pi_{1}|-\Delta_{\mathrm{DP}}\}

(mentioned in

\mathrm{P}_{2}

) for every model

\times

prevalence condition.

Dataset	Cond.	Method	$\bm{\Delta\pi}$	AUROC	$\bm{D_{A}}$	$\bm{D_{A\mid Y}}$	$\bm{\Delta_{\mathrm{DP}}}$	$\bm{\Delta_{\mathrm{EO}}}$	$\bm{\Delta_{\mathrm{PPV}}}$	$\bm{\Delta_{\mathrm{bias}}}$	$\bm{e_{0}+e_{1}}$
Adult	Eq	ERM	0.00	0.87 $\pm$ 0.00	0.92 $\pm$ 0.00	0.98 $\pm$ 0.00	0.13 $\pm$ 0.01	0.14 $\pm$ 0.01	0.14 $\pm$ 0.01	0.01 $\pm$ 0.00	0.42 $\pm$ 0.02
	Eq	M	0.00	0.72 $\pm$ 0.01	0.56 $\pm$ 0.00	0.88 $\pm$ 0.02	0.03 $\pm$ 0.00	0.22 $\pm$ 0.05	0.06 $\pm$ 0.03	0.01 $\pm$ 0.00	0.57 $\pm$ 0.06
	Eq	C	0.00	0.84 $\pm$ 0.01	0.55 $\pm$ 0.00	0.61 $\pm$ 0.02	0.05 $\pm$ 0.01	0.08 $\pm$ 0.00	0.01 $\pm$ 0.00	0.01 $\pm$ 0.00	0.54 $\pm$ 0.05
	Nat	ERM	0.19	0.88 $\pm$ 0.00	0.93 $\pm$ 0.00	0.99 $\pm$ 0.00	0.32 $\pm$ 0.01	0.22 $\pm$ 0.01	0.06 $\pm$ 0.01	0.00 $\pm$ 0.00	0.38 $\pm$ 0.01
	Nat	M	0.19	0.77 $\pm$ 0.00	0.57 $\pm$ 0.01	0.85 $\pm$ 0.01	0.04 $\pm$ 0.00	0.31 $\pm$ 0.02	0.42 $\pm$ 0.03	0.17 $\pm$ 0.00	0.65 $\pm$ 0.07
	Nat	C	0.19	0.85 $\pm$ 0.00	0.60 $\pm$ 0.00	0.59 $\pm$ 0.01	0.15 $\pm$ 0.01	0.09 $\pm$ 0.02	0.28 $\pm$ 0.00	0.10 $\pm$ 0.00	0.49 $\pm$ 0.04

Appendix

Table 2: Evaluation metrics by training condition across datasets (reported as mean

\pm

std over all seeds).

\Delta\pi=|\pi_{1}-\pi_{0}|

is the base-rate gap. Conditions : Eq (equalized), Nat (natural), S50 (shift-50), S75 (shift-75). Methods : ERM, M (marginal alignment,

Z\perp A

), and C (class-conditional alignment,

Z\perp A\mid Y

). Datasets: Adult, Dutch census (Dutch), Communities & Crime (Crime), Taiwan credit (Credit), and COMPAS.

Figure 8: Probe

\mathrm{AUROC}

to decode for

A

from

Z

under six probe families ( shift-50 , reported mean

\pm

std over all seeds). Dotted:

(1+\mathrm{TV})/2

for the marginal flow.

Dataset	Method	LR	GB	MLP-0	MLP-1	MLP-2	LR $-A$	Adv BA	Bound	Max $W$
		Probe AUC ( $A$ from $Z$ )
Adult	ERM	0.93 $\pm$ 0.00	0.93 $\pm$ 0.00	0.93 $\pm$ 0.00	0.93 $\pm$ 0.00	0.93 $\pm$ 0.00	0.51 $\pm$ 0.02	0.86 $\pm$ 0.00	–	0.99 $\pm$ 0.00
	M	0.56 $\pm$ 0.00	0.57 $\pm$ 0.01	0.57 $\pm$ 0.01	0.57 $\pm$ 0.01	0.58 $\pm$ 0.01	0.51 $\pm$ 0.01	0.55 $\pm$ 0.01	0.55 $\pm$ 0.00	0.85 $\pm$ 0.02
	C	0.61 $\pm$ 0.01	0.62 $\pm$ 0.00	0.62 $\pm$ 0.01	0.62 $\pm$ 0.00	0.62 $\pm$ 0.00	0.51 $\pm$ 0.01	0.58 $\pm$ 0.01	0.53 $\pm$ 0.00	0.56 $\pm$ 0.00
Dutch census	ERM	0.75 $\pm$ 0.00	0.77 $\pm$ 0.00	0.77 $\pm$ 0.00	0.77 $\pm$ 0.00	0.77 $\pm$ 0.00	0.51 $\pm$ 0.04	0.68 $\pm$ 0.00	–	0.77 $\pm$ 0.00
	M	0.51 $\pm$ 0.00	0.51 $\pm$ 0.00	0.51 $\pm$ 0.01	0.51 $\pm$ 0.00	0.51 $\pm$ 0.00	0.49 $\pm$ 0.00	0.50 $\pm$ 0.00	0.51 $\pm$ 0.00	0.79 $\pm$ 0.01

Appendix

Table 3: Probe ablation at shift 50. Methods: ERM, Marginal ( M :

Z\perp A

), and Conditional ( C :

Z\perp A\mid Y

). Columns 3–8 report AUC of probe families predicting

A

from

Z

: LR (logistic regression), GB (gradient-boosted trees), MLP-

k

(

k

hidden layers), and LR

-A

(logistic control baseline excluding

A

). Right columns report the original FNF adversary balanced accuracy ( Adv BA ), the certified balanced accuracy bound

(1+\mathrm{TV})/2

( Bound ), and the maximum within-class probe AUC ( Max

W

). Reported as mean

\pm

sd over three seeds.

Figure 9: Marginal alignment method as a function of the invariance strength

\gamma

( shift-50 , selected feature set, MADE width 50). Reported mean

\pm

std over seeds.

Dataset	Feature set	MADE	n fits	AUROC	$D_{A}$	$D_{A\mid Y}$	$\Delta_{\mathrm{DP}}$	$\Delta_{\mathrm{EO}}$	exact TV
Adult	paper6	50	3	0.79 $\pm$ 0.00	0.58 $\pm$ 0.01	0.89 $\pm$ 0.01	0.08 $\pm$ 0.00	0.38 $\pm$ 0.03	0.103 $\pm$ 0.003
Adult	paper6	100	3	0.79 $\pm$ 0.00	0.59 $\pm$ 0.01	0.87 $\pm$ 0.02	0.09 $\pm$ 0.01	0.36 $\pm$ 0.04	0.102 $\pm$ 0.005
Adult	paper6_age (selected)	50	3	0.79 $\pm$ 0.00	0.58 $\pm$ 0.01	0.85 $\pm$ 0.02	0.04 $\pm$ 0.00	0.40 $\pm$ 0.03	0.096 $\pm$ 0.007
Adult	paper6_age	100	3	0.80 $\pm$ 0.00	0.58 $\pm$ 0.00	0.84 $\pm$ 0.02	0.05 $\pm$ 0.01	0.38 $\pm$ 0.02	0.093 $\pm$ 0.024
Dutch census	core6	50	3	0.72 $\pm$ 0.00	0.52 $\pm$ 0.01	0.81 $\pm$ 0.01	0.01 $\pm$ 0.01	0.27 $\pm$ 0.02	0.022 $\pm$ 0.003
Dutch census	core6	100	3	0.72 $\pm$ 0.01	0.52 $\pm$ 0.00	0.81 $\pm$ 0.00	0.01 $\pm$ 0.01	0.30 $\pm$ 0.01	0.022 $\pm$ 0.003

Appendix

Table 4: Marginal alignment method at

\gamma=1

for every feature set and MADE width in the sweep ( shift 50 , mean

\pm

std over all seeds).

Dataset	Method	Youden $\tau$	Acc ( $\tau$ )	$\Delta_{\mathrm{DP}}$ ( $\tau$ )	$\Delta_{\mathrm{EO}}$ ( $\tau$ )	Acc (0.5)	$\Delta_{\mathrm{DP}}$ (0.5)	$\Delta_{\mathrm{EO}}$ (0.5)
Adult	ERM	0.22 $\pm$ 0.02	0.78 $\pm$ 0.02	0.40 $\pm$ 0.03	0.28 $\pm$ 0.02	0.84 $\pm$ 0.00	0.24 $\pm$ 0.01	0.39 $\pm$ 0.03
Adult	$Z\perp A$	0.17 $\pm$ 0.02	0.69 $\pm$ 0.02	0.04 $\pm$ 0.00	0.40 $\pm$ 0.03	0.78 $\pm$ 0.00	0.01 $\pm$ 0.01	0.06 $\pm$ 0.03
Adult	$Z\perp A\mid Y$	0.17 $\pm$ 0.00	0.75 $\pm$ 0.02	0.17 $\pm$ 0.01	0.08 $\pm$ 0.04	0.82 $\pm$ 0.00	0.11 $\pm$ 0.01	0.06 $\pm$ 0.01
Dutch census	ERM	0.43 $\pm$ 0.02	0.82 $\pm$ 0.00	0.30 $\pm$ 0.00	0.05 $\pm$ 0.00	0.83 $\pm$ 0.00	0.29 $\pm$ 0.00	0.06 $\pm$ 0.00
Dutch census	$Z\perp A$	0.40 $\pm$ 0.04	0.64 $\pm$ 0.00	0.01 $\pm$ 0.01	0.31 $\pm$ 0.03	0.65 $\pm$ 0.01	0.01 $\pm$ 0.01	0.22 $\pm$ 0.03
Dutch census	$Z\perp A\mid Y$	0.41 $\pm$ 0.04	0.81 $\pm$ 0.00	0.27 $\pm$ 0.01	0.03 $\pm$ 0.01	0.81 $\pm$ 0.00	0.28 $\pm$ 0.01	0.02 $\pm$ 0.01

Appendix

Table 5: Optimal Youden threshold

\tau

(for validation) vs. a default

0.5

( shift 50 , reported mean

\pm

std over all seeds).

			Training Prevalences ( $\pi_{0}$ / $\pi_{1}$ )
Dataset	Samples	Natural $\pi_{0}/\pi_{1}$	Equalized	Natural	Shift-50	Shift-75
Adult income	48,842	0.11 / 0.30	0.11 / 0.11	0.11 / 0.31	0.06 / 0.31	0.03 / 0.31
	(29,312 / 4,880 / 4,885 / 9,765)
Dutch census	60,420	0.33 / 0.63	0.33 / 0.33	0.33 / 0.62	0.20 / 0.62	0.11 / 0.62
	(36,252 / 6,042 / 6,042 / 12,084)
Comm. & Crime	1,994	0.27 / 0.71	0.26 / 0.26	0.26 / 0.69	0.15 / 0.69	0.08 / 0.69

Appendix

Table 6: Dataset statistics and prevalence shift configurations. Binary groups (

A{=}0

vs.

A{=}1

) and prediction targets (

Y{=}1

) are: Adult : female vs. male, income

>

50K; Dutch : female vs. male, high-level occupation; Crime : Black population share below vs. at or above median, violent crime rate above median; Taiwan credit : female vs. male, default next month; COMPAS : Caucasian vs. African-American, two-year recidivism. Rightmost columns report the training-split base rates (

\pi_{0}

\pi_{1}

) across synthetic shift regimes. Selected feature sets: Adult paper6_age (workclass, education, marital status, occupation, relationship, race, binned age), Dutch core6_hh (age, education, economic status, current activity, marital status, household position, household size), Crime socio4 (median income, poverty, unemployment, two-parent families; quantile-binned), Credit core5 (education, marriage, binned age, last repayment status, binned limit), COMPAS paper7 (sex, age category, charge degree, binned priors, three juvenile counts).

Dataset	Method	Seed	AUROC	$D_{A}$	$D_{A\mid Y}$	$\Delta_{\mathrm{DP}}$	$\Delta_{\mathrm{EO}}$	$\Delta_{\mathrm{PPV}}$	$\Delta_{\mathrm{bias}}$
Adult	ERM	42	0.89	0.93	0.99	0.42	0.30	0.17	0.00
Adult	ERM	123	0.89	0.93	0.99	0.36	0.27	0.19	0.02
Adult	ERM	456	0.89	0.93	0.99	0.40	0.27	0.18	0.02
Adult	M-FNF ( $Z\perp A$ )	42	0.79	0.58	0.86	0.04	0.37	0.48	0.23
Adult	M-FNF ( $Z\perp A$ )	123	0.79	0.57	0.86	0.05	0.43	0.47	0.23
Adult	M-FNF ( $Z\perp A$ )	456	0.79	0.58	0.83	0.04	0.41	0.51	0.23

Appendix

Table 7: Detailed results for the shift-50 prevalence condition per selected seeds.

Dataset	Method	$\mathbf{ECE}_{0}$	$\mathbf{ECE}_{1}$	$\bm{b}_{0}$	$\bm{b}_{1}$	$\bm{a}_{0}$	$\bm{a}_{1}$	$\bm{\bar{y}_{0}-\bar{p}_{0}}$	$\bm{\bar{y}_{1}-\bar{p}_{1}}$	$\bm{\Delta_{\mathrm{bias}}}$	$\bm{\Delta_{\mathrm{cal}}}$
Adult	ERM	0.02 $\pm$ 0.00	0.02 $\pm$ 0.00	0.96 $\pm$ 0.03	0.85 $\pm$ 0.01	$-0.29$ $\pm$ 0.08	0.03 $\pm$ 0.12	$-0.013$ $\pm$ 0.004	0.004 $\pm$ 0.016	0.02 $\pm$ 0.01	0.32 $\pm$ 0.06
	M	0.13 $\pm$ 0.01	0.11 $\pm$ 0.01	0.06 $\pm$ 0.04	1.34 $\pm$ 0.05	$-1.36$ $\pm$ 0.06	0.75 $\pm$ 0.06	$-0.116$ $\pm$ 0.007	0.112 $\pm$ 0.009	0.23 $\pm$ 0.00	2.12 $\pm$ 0.01
	C	0.08 $\pm$ 0.01	0.08 $\pm$ 0.01	0.79 $\pm$ 0.03	0.88 $\pm$ 0.02	$-1.20$ $\pm$ 0.14	0.60 $\pm$ 0.09	$-0.077$ $\pm$ 0.013	0.075 $\pm$ 0.010	0.15 $\pm$ 0.00	1.80 $\pm$ 0.07
Dutch	ERM	0.12 $\pm$ 0.00	0.10 $\pm$ 0.00	1.01 $\pm$ 0.02	0.90 $\pm$ 0.01	$-1.24$ $\pm$ 0.05	0.75 $\pm$ 0.02	$-0.120$ $\pm$ 0.004	0.095 $\pm$ 0.003	0.22 $\pm$ 0.00	2.00 $\pm$ 0.03
	M	0.23 $\pm$ 0.00	0.22 $\pm$ 0.01	0.51 $\pm$ 0.08	1.65 $\pm$ 0.06	$-1.25$ $\pm$ 0.01	1.07 $\pm$ 0.04	$-0.224$ $\pm$ 0.003	0.219 $\pm$ 0.007	0.44 $\pm$ 0.00	2.32 $\pm$ 0.03
	C	0.12 $\pm$ 0.01	0.11 $\pm$ 0.00	0.98 $\pm$ 0.03	0.91 $\pm$ 0.02	$-1.16$ $\pm$ 0.05	0.89 $\pm$ 0.04	$-0.121$ $\pm$ 0.006	0.114 $\pm$ 0.005	0.23 $\pm$ 0.01	2.05 $\pm$ 0.07

Appendix

Table 8: Calibration by groups at shift 50 : 15-bin ECE, Cox recalibration slope ( Cox, 1958 )

b_{g}

and fixed-slope intercept

a_{g}

(logits), signed mean bias

\bar{y}_{g}-\bar{p}_{g}

, and the two between-group gaps

\Delta_{\mathrm{bias}}

and

\Delta_{\mathrm{cal}}

. Methods: ERM, M (marginal alignment,

Z\perp A

), and C (class-conditional alignment,

Z\perp A\mid Y

). Datasets: Adult, Dutch census (Dutch), Communities & Crime (Crime), Taiwan credit (Credit), and COMPAS. Reported as mean

\pm

std over all three seeds.

Method	AUROC	$\bm{D_{A}}$	$\bm{D_{A\mid Y}}$	$\bm{\Delta_{\mathrm{DP}}}$	$\bm{\Delta_{\mathrm{EO}}}$	$\bm{\Delta_{\mathrm{PPV}}}$	$\bm{\Delta_{\mathrm{NPV}}}$	$\bm{\Delta_{\mathrm{bias}}}$	$\bm{\Delta_{\mathrm{cal}}}$
ERM	0.896	0.70	0.64	0.296	0.055	0.27	0.28	0.22	2.00
Score $R\perp A$	0.813	0.54	0.77	0.006	0.258	0.55	0.35	0.44	3.68
Score $R\perp A\mid Y$ (within-class map)	0.881	0.67	0.59	0.257	0.024	0.36	0.28	0.24	2.11
Mean matching, $\lambda=1$	0.896	0.70	0.63	0.291	0.051	0.29	0.28	0.22	1.95
Mean matching, $\lambda=10$	0.889	0.69	0.66	0.270	0.082	0.27	0.30	0.28	1.79
Mean matching, $\lambda=100$	0.883	0.69	0.66	0.266	0.081	0.30	0.30	0.41	1.87

Appendix

Table 9: Score-level alignment and class-conditional mean matching on Dutch, shift-50 , test split, mean over three seeds.

Task	Partition	Images	Patients	$\pi_{0}$	$\pi_{1}$
Edema	Train	107,688	37,580	0.198	0.313
	Checkpoint	9,050	3,134	0.195	0.325
	Development	8,872	3,138	0.202	0.333
	Probe val.	17,186	6,271	0.206	0.310
	Test	35,078	12,551	0.205	0.309
Cardiomegaly	Train	111,980	38,082	0.081	0.159

Appendix

Table 10: shift50 cohorts. Patient partitions are fixed across seeds; image-level deletion changes the retained patient counts.

Figure 10: Invariance strength changes the fairness trade-off. Lines and bands show means

\pm

std. Conditional alignment lowers

\mathrm{EO}

gaps at strong alignment, while marginal alignment produces smaller

\mathrm{DP}

gaps.

Task	Method	$\Delta_{\rm DP}$	$\Delta_{\rm EO}$	Max ECE	$\Delta_{\rm bias}$	$\Delta_{\rm cal}$
Edema	ERM	$0.147\pm 0.002$	$0.102\pm 0.007$	$0.020\pm 0.009$	$0.013\pm 0.006$	$0.092\pm 0.037$
	M .05	$0.132\pm 0.002$	$0.088\pm 0.005$	$0.037\pm 0.003$	$0.018\pm 0.005$	$0.160\pm 0.043$
	M .50	$0.070\pm 0.005$	$0.019\pm 0.005$	$0.037\pm 0.002$	$0.062\pm 0.004$	$0.504\pm 0.012$
	M .95	$0.009\pm 0.009$	$0.048\pm 0.014$	$0.069\pm 0.006$	$0.101\pm 0.006$	$0.797\pm 0.071$
	C .05	$0.134\pm 0.005$	$0.094\pm 0.005$	$0.034\pm 0.009$	$0.020\pm 0.009$	$0.165\pm 0.064$
	C .50	$0.092\pm 0.009$	$0.042\pm 0.006$	$0.039\pm 0.004$	$0.046\pm 0.008$	$0.387\pm 0.055$

Appendix

Table 11: Fairness and calibration for every invariance method strength (mean

\pm

seed reported).

\Delta_{\rm bias}

is the absolute between-sex difference in mean residual

Y-R

;

\Delta_{\rm cal}

is the absolute difference in fixed-slope calibration intercepts.

Task	Method	AUROC	AUPRC	$D_{A}(Z)$	$D_{A\mid Y}(Z)$	$D_{A\mid Y}(R)$
Edema	ERM	$0.839\pm 0.003$	$0.646\pm 0.007$	$0.978\pm 0.002$	$0.978\pm 0.002$	$0.613\pm 0.019$
	M .05	$0.835\pm 0.005$	$0.641\pm 0.011$	$0.693\pm 0.022$	$0.678\pm 0.022$	$0.583\pm 0.011$
	M .50	$0.835\pm 0.002$	$0.640\pm 0.006$	$0.631\pm 0.034$	$0.624\pm 0.029$	$0.519\pm 0.008$
	M .95	$0.830\pm 0.002$	$0.630\pm 0.005$	$0.623\pm 0.042$	$0.629\pm 0.043$	$0.555\pm 0.008$
	C .05	$0.829\pm 0.003$	$0.632\pm 0.008$	$0.675\pm 0.013$	$0.664\pm 0.024$	$0.582\pm 0.010$
	C .50	$0.837\pm 0.003$	$0.642\pm 0.007$	$0.613\pm 0.004$	$0.599\pm 0.013$	$0.537\pm 0.010$

Appendix

Table 12: All settings: utility and strongest demographic probe AUROC (mean

\pm

sample std over all training seeds). M/C denote marginal/conditional objectives, followed by

\gamma

Figure 11: Conditional probe-family audit at

\gamma=.95

. Top: representations; bottom: scores.

Task	Seed	Test $D_{A\mid Y}(R)$	Test upper	Dev. upper	Test pass
Edema	42	0.515	0.540	0.591	Yes
	123	0.510	0.541	0.563	Yes
	456	0.514	0.545	0.572	Yes
Cardiomegaly	42	0.514	0.557	0.637	No
	123	0.516	0.559	0.618	No
	456	0.515	0.558	0.606	No

Appendix

Table 13: Conditional score audit. Upper bounds use the within-scope Bonferroni correction; passing upper bound

\leq 0.55

. All three Edema test runs also pass the balanced-accuracy and log-loss interval checks.

Task	Method	$A$	ECE	Slope $b_{a}$	Intercept $\alpha_{a}$	$\bar{y}_{a}-\bar{r}_{a}$
Edema	ERM	0	$0.010\pm 0.005$	$0.978\pm 0.054$	$0.001\pm 0.034$	$-0.000\pm 0.004$
		1	$0.020\pm 0.009$	$0.926\pm 0.044$	$0.093\pm 0.071$	$0.013\pm 0.010$
	M .95	0	$0.069\pm 0.006$	$0.875\pm 0.037$	$-0.561\pm 0.057$	$-0.069\pm 0.006$
		1	$0.036\pm 0.004$	$0.858\pm 0.056$	$0.236\pm 0.030$	$0.032\pm 0.003$
	C .95	0	$0.051\pm 0.006$	$0.867\pm 0.079$	$-0.416\pm 0.036$	$-0.050\pm 0.007$
		1	$0.029\pm 0.007$	$0.861\pm 0.076$	$0.166\pm 0.043$	$0.023\pm 0.005$

Appendix

Table 14: Sex-calibration at

\gamma=.95

(mean

\pm

std reported).

\alpha_{a}

is fitted separately with slope fixed at one. Negative intercepts and negative mean residuals

\implies

overprediction. Ideal slope is 1; ideal intercept and mean residual are 0.

			Sex Probe AUROC $\downarrow$			Fairness Gaps $\downarrow$
Pathology	Method	Task AUROC $\uparrow$	Marginal	$\bm{Y=0}$	$\bm{Y=1}$	DP Gap	EO Gap
Edema	DANN	0.842 $\pm$ 0.003	0.950 $\pm$ 0.005	0.950 $\pm$ 0.006	0.938 $\pm$ 0.008	0.069 $\pm$ 0.023	0.033 $\pm$ 0.025
Edema	CDANN	0.845 $\pm$ 0.003	0.952 $\pm$ 0.006	0.955 $\pm$ 0.007	0.940 $\pm$ 0.012	0.111 $\pm$ 0.015	0.074 $\pm$ 0.025
Cardiomegaly	DANN	0.842 $\pm$ 0.004	0.947 $\pm$ 0.006	0.951 $\pm$ 0.006	0.928 $\pm$ 0.018	0.054 $\pm$ 0.031	0.037 $\pm$ 0.015
Cardiomegaly	CDANN	0.843 $\pm$ 0.006	0.944 $\pm$ 0.030	0.957 $\pm$ 0.008	0.924 $\pm$ 0.027	0.105 $\pm$ 0.028	0.078 $\pm$ 0.042

Appendix

Table 15: Evaluation across pathologies and objectives. Task AUROC, marginal and class-conditional sex probe AUROCs, demographic parity (

\mathrm{DP}

) gap, and equalized odds (

\mathrm{EO}

) gap. Results are reported as mean

\pm

std.

Panel of Figure 14 — Figure 14: Evaluating demographic erasure on CheXpert. (a) Comparison of linear and nonlinear probes in the single-seed exploratory Edema study. (b–c) Summary of DANN and CDANN experiments across adversarial strengths and seeds. Chance probe AUROC does not establish independence, and marginal probes alone do not assess conditional invariance.

Method	Configs.	Task AUROC $\uparrow$	Probe AUROC $\downarrow$	Pass	Main Finding / Limitation
Baselines and Controls
ERM	1	0.840	0.961	0/1	Strong demographic leakage.
ERM adapters	3	0.833	0.554–0.803	0/3	Compression reduced leakage but did not meet the criterion.
PCA + ERM	2	0.838	0.813–0.886	0/2	Substantial residual leakage.
Constant output	1	0.500	0.502	1/1	Trivial removal through complete task collapse.
Learned Representations

Appendix

Table 16: Exploratory demographic-erasure experiments on CheXpert Edema at natural prevalence (one training seed; 51 configurations). Results do not reflect general limitations of the evaluated methods.

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 420 papers

Feb 6, 2025

Targeted Learning for Inference on Data Fairness

As algorithmic decision making has become increasingly ubiquitous even in high-stakes settings, significant effort has been invested in defining, detecting, and eliminating unfair model-driven outcomes. Prior work has largely focused on estimating the fairness properties of a given predictive model, and quantifying uncertainty based on variability in a fixed model's performance across evaluation datasets. Alternatively, we can also ask about uncertainty that the model inherits from its training data; here we evaluate data fairness - fairness in the data generating process used for training. We perform inference on data fairness using targeted learning, a flexible framework for nonparametric inference, and derive estimators for demographic parity, equal opportunity, and conditional mutual information. Simulations show that our data fairness estimators improve confidence-interval coverage and reduce false detections of population unfairness relative to fixed-model inference. We also demonstrate their application to real data.

Joseph H. Rudoler, Alexander Asemota, Giles Hooker

University of California Berkeley, USA · University of Pennsylvania, USA

Algorithmic Fairness Group Fairness

Ask PDF

Mar 25, 2026

Exploring How Fair Model Representations Relate to Fair Recommendations

One of the many fairness definitions pursued in recent recommender system research targets mitigating demographic information encoded in model representations. Models optimized for this definition are typically evaluated on how well demographic attributes can be classified given model representations, with the (implicit) assumption that this measure accurately reflects recommendation parity, i.e., how similar recommendations given to different users are. We challenge this assumption by comparing the amount of demographic information encoded in representations with various measures of how the recommendations differ. We propose two new approaches for measuring how well demographic information can be classified given ranked recommendations. Our results from extensive testing of multiple models on one real and multiple synthetically generated datasets indicate that optimizing for fair representations positively affects recommendation parity, but also that evaluation at the representation level is not a good proxy for measuring this effect when comparing models. We also provide extensive insight into how recommendation-level fairness metrics behave for various models by evaluating their performances on numerous generated datasets with different properties.

Bjørnar Vassøy, Benjamin Kille, Helge Langseth

Norwegian University of Science and Technology, Trondheim, Norway

Fairness in Recommender Systems Algorithmic Fairness Recommender Systems Group Fairness Representation Learning

Ask PDF

Oct 5, 2026

Representation Disentanglement for Fair Chest X-Ray Diagnosis

Deep learning has advanced chest X-ray (CXR) diagnosis, yet demographic biases in learned representations may contribute to performance disparities across intersectional groups. We propose a single-encoder framework combining dual-level decorrelation with prototype-guided cross-group contrastive learning to reduce demographic dependence while accounting for within-class variation. We further propose Demographic Representation Alignment Reduction (DRAR), a new metric that quantifies the reduction in demographic structure within disease representations. The framework is evaluated on four classification tasks using 34,809 CheXpert test images across eight intersectional groups, defined by age, sex and ethnicity. Compared with empirical risk minimization (ERM), our method reduces the mean equalized-odds gap from 15.41% to 10.86% and the AUC gap from 5.95% to 5.01%. Our method achieves a DRAR of 59.04% relative to ERM, with only a slight decrease in mean AUC. These results demonstrate that representation disentanglement can reduce demographic bias and improve intersectional fairness. Code is available at https://github.com/06Yujie/Fair-Medical-Imaging.

Yujie Sun, Ruizhe Li, Xiaowu Sun

Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University, China · Nottingham Biomedical Research Centre, School of Medicine, University of Nottingham, UK

Chest X-Ray Classification Representation Disentanglement Algorithmic Bias Intersectional Fairness Fairness in Medical Image Classification

Ask PDF

Abstract

Algorithmic Fairness

Explore similar work

Targeted Learning for Inference on Data Fairness

Exploring How Fair Model Representations Relate to Fair Recommendations

Representation Disentanglement for Fair Chest X-Ray Diagnosis