Beyond Procedure: Substantive Fairness in Conformal Prediction

Pengqi Liu; Zijun Yu; Mouloud Belbahri; Arthur Charpentier; Masoud Asgharian; Jesse C. Cresswell

stat.MLFeb 18, 2026

Beyond Procedure: Substantive Fairness in Conformal Prediction

Authors: Pengqi Liu, Zijun Yu, Mouloud Belbahri, Arthur Charpentier, Masoud Asgharian, Jesse C. Cresswell

Organizations: McGill University, Montreal, Canada · TD Insurance, Montreal, Canada · UQAM, Montreal, Canada · Layer 6 AI, Toronto, Canada

Abstract

Conformal prediction (CP) offers distribution-free uncertainty quantification for machine learning models, yet its interplay with fairness in downstream decision-making remains underexplored. Moving beyond CP as a standalone operation (procedural fairness), we analyze the holistic decision-making pipeline to evaluate substantive fairness-the equity of downstream outcomes. Theoretically, we derive an upper bound that decomposes prediction-set size disparity into interpretable components, clarifying how label-clustered CP helps control method-driven contributions to unfairness. To facilitate scalable empirical analysis, we introduce an LLM-in-the-loop evaluator that approximates human assessment of substantive fairness across diverse modalities. Our experiments show that label-clustered CP often provides a favorable balance between utility and substantive fairness, while reducing set-size disparities in line with our theory. Finally, we empirically show that equalized set sizes, rather than coverage, strongly correlate with improved substantive fairness, enabling practitioners to design more fair CP systems. Our code is available at https://github.com/layer6ai-labs/llm-in-the-loop-conformal-fairness.

Figures & tables

Task	Acc	$\Delta_{\text{Acc}}$	CP Method	Cvg	Size
FACET	70.0	22.2	Marginal	89.9	2.62
			Mondrian	89.9	2.66
			Label-Clustered	89.1	2.92
			Group-Clustered	89.3	2.51
			Backward	90.1	3.50
BiosBias	78.9	2.70	Marginal	89.5	1.68

Table 1: Base Model and CP Method Metrics on

\mathcal{D}_{\text{test}}

	Human-in-the-loop		LLM-in-the-loop
Dataset	Marginal	Mondrian	Marginal	Mondrian
FACET	26	51	9.0	38
BiosBias	12	33	6.9	8.1
RAVDESS	1.0	28	11	79

Table 2: Human vs. LLM Evaluator Comparison (maxROR %).

Figure 1: maxROR (%) of each CP method across four tasks. Lower is more substantively fair.

Figure 2: Accuracy improvement (%) relative to Control of each CP method, across four tasks. Higher is better.

Panel of Figure 3 — Figure 3: Coverage gap (blue dots, left axis) and set size gap (red squares, right axis) across CP methods. The two procedural fairness metrics are in direct tension. Corresponding plots for FACET and ACSIncome are in Appendix F.2 .

Figure 4: maxROR (%) compared to the coverage gap (Left) and set size gap (Right) between groups, across CP methods and datasets. Regression lines are fitted for each dataset individually to show trends.

Panel of Figure 5 — Figure 5: Average prediction set size gap between Female and Male on the BiosBias and RAVDESS datasets over 10 random splits. The maximum standard error of the average set size gap is 0.016 in (a) and 0.010 in (b).

Panel of Figure 6 — Figure 6: Numerical computation of the three terms in 14 vs. number of clusters $K$ on RAVDESS.

Figure 7: Numerical computation of

\Delta_{a,b}

vs. the upper bound from 14 on RAVDESS with Label-Clustered CP. The bound is reasonably tight and faithfully reflects the shape of

\Delta_{a,b}

K

is varied.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{C}_{\hat{q}_{\alpha}}(x)=\{y\in\mathcal{Y}:s(x,y)\leq\hat{q}_{\alpha}\}

Appendix

Algorithm 1 Marginal (Split) Conformal Prediction

\hat{q}_{\alpha}^{(a)}\leftarrow\operatorname{Quantile}_{\lceil(|\mathcal{I}_{a}|+1)(1-\alpha)\rceil/|\mathcal{I}_{a}|}\left(\{S_{i}\}_{i\in\mathcal{I}_{a}}\right)

Appendix

Algorithm 2 Mondrian (Group-Conditional) Conformal Prediction

h:\mathcal{Y}\to[K]\cup\{\mathrm{null}\}

Appendix

Algorithm 3 Label-Clustered Conformal Prediction

\tilde{h}:\mathcal{A}\to[K]\cup\{\mathrm{null}\}

Appendix

Algorithm 4 Group-Clustered Conformal Prediction

E_{\mathrm{test}}(y)\leftarrow\frac{s(x_{\mathrm{test}},y)}{\frac{1}{n+1}\Bigl(\sum_{i=1}^{n}S_{i}+s(x_{\mathrm{test}},y)\Bigr)}

Appendix

Algorithm 5 Backward Conformal Prediction

Table 15

Group	$\mathcal{D}_{\text{calval}}$	$\mathcal{D}_{\text{cal}}$	$\mathcal{D}_{\text{test}}$
White	6655	13487	6637
Black or African American	842	1560	819
Asian	662	1362	663
Two or More Races	1134	2164	1117
All Other Races	707	1427	764
Total	10000	20000	10000

Appendix

Table 6: ACSIncome Group Counts.

Dataset	Score Function	Marginal			Mondrian			Label-Clustered			Group-Clustered
Dataset	Score Function	$T$	$\lambda$	$k_{\mathrm{reg}}$	$T$	$\lambda$	$k_{\mathrm{reg}}$	$T$	$\lambda$	$k_{\mathrm{reg}}$	$T$	$\lambda$	$k_{\mathrm{reg}}$
FACET	RAPS	0.53	0.07	4	0.30	0.16	4	0.51	1.38	4	0.30	0.16	4
BiosBias	SAPS	0.56	0.20	–	0.40	0.20	–	0.74	0.28	–	0.47	0.19	–
RAVDESS	RAPS	0.16	1.61	3	0.15	0.69	3	0.17	0.35	3	0.16	0.51	3
ACSIncome	RAPS	0.09	0.05	4	0.10	0.05	4	0.09	0.05	4	0.09	0.05	4

Appendix

Table 7: Hyperparameter Settings for Each Dataset After Tuning

Dataset	LLM-in-the-loop	$M$	Total predictions	Cost/wall-clock time
FACET	Qwen2.5-VL-7B-Instruct	8	67200	30 h 56 min
BiosBias	GPT-4o-mini	5	60000	$0.82
BiosBias	Llama-3.2-3B	8	96000	1 h 52 min
RAVDESS	GPT-4o-audio-preview	5	10800	$16
ACSIncome	Qwen2.5-7B	4	240000	22 h 50 min

Appendix

Table 8: Computational cost of the LLM-in-the-loop evaluation. The column “Total predictions” counts the total number of LLM predictions across task instances and treatments, including the

M

repeated predictions per instance-treatment pair. For API-based models, cost is reported in USD; for open-source models run locally, we report elapsed wall-clock runtime on a Nvidia RTX 5060Ti 16GB GPU.

Treatment	Group	Human adoption%	LLM adoption%
Marginal	Middle	92.82	96.52
	Older	91.66	90.66
	Unknown	96.76	99.01
	Younger	94.79	98.23
Mondrian	Middle	93.22	96.42
	Older	92.91	92.17

Appendix

Table 9: Adoption rate of human vs. LLM (Qwen2.5-VL-7B-Instruct) for FACET

Treatment	Group	Human adoption%	LLM adoption%
Marginal	Female	92.31	95.22
Marginal	Male	93.87	98.20
Mondrian	Female	92.30	92.07
Mondrian	Male	94.48	97.36

Appendix

Table 10: Adoption rate of human vs. LLM (GPT-4o-mini) for BiosBias

Treatment	Group	Human adoption%	LLM adoption%
Marginal	Female	90.67	91.89
Marginal	Male	87.97	92.44
Mondrian	Female	91.95	90.00
Mondrian	Male	92.20	94.11

Appendix

Table 11: Adoption rate of human vs. LLM (GPT-4o-audio-preview) for RAVDESS

	FACET	BiosBias	RAVDESS
$\mathbb{P}(\text{correct}\mid\text{adoption}=1)$	78.40%	80.64%	46.58%
$\mathbb{P}(\text{correct}\mid\text{adoption}=0)$	37.99%	76.74%	0.00%
$\mathbb{P}(\,\,\,\text{invalid}\mid\text{adoption}=0)$	25.14%	9.30%	41.75%

Appendix

Table 12: Empirical probabilities of correct response and invalid response conditioning on adoption status for FACET, BiosBias, and RAVDESS datasets. As we can see, the

\mathbb{P}(\text{correct}|\text{adoption}=1)

is higher than

\mathbb{P}(\text{correct}|\text{adoption}=0)

, especially for the challenging FACET and RAVDESS tasks.

Group	$\textbf{OR}_{\text{marginal}}$	$\textbf{OR}_{\text{Mondrian}}$	$\textbf{maxROR}_{\text{marginal}}(\%)$	$\textbf{maxROR}_{\text{Mondrian}}(\%)$
Female	1.026	1.219	27.4	9.9
Male	1.308	1.110	27.4	9.9

Appendix

Table 13: The OR and maxROR computed from fitting a GEE without the “adoption” covariate for BiosBias. As shown on the table,

\textbf{OR}_{\text{Marginal},\text{Male}}

has a relatively large value from the relatively high adoption rate, resulting in larger

\textbf{maxROR}_{\text{Marginal}}

than

\textbf{maxROR}_{\text{Mondrian}}

Dataset	Interaction term	$p$ -value
FACET	adoption $\times$ Older	.896
	adoption $\times$ Unknown	.454
	adoption $\times$ Younger	.812
BiosBias	adoption $\times$ Male	.425
ACSIncome	adoption $\times$ Asian alone	.128
	adoption $\times$ Black or African American alone	.824 †

Appendix

Table 14: Wald-test

p

-values for adoption-by-group interaction terms in Model 2. The null hypothesis is that the corresponding adoption-by-group interaction coefficient equals zero. Statistical significance is assessed at level

0.05

. Baseline group of FACET, BiosBias, and ACSIncome is Middle, Female, and All Other Races (Aggregated), respectively.

Dataset	Treatment	Model 1 maxROR (%)	Model 2 maxROR (%)
FACET	Marginal	9.0	9.1
	Mondrian	37.7	37.4
	Label-Clustered	13.5	14.0
	Group-Clustered	14.5	14.3
	Backward	8.5	8.4
BiosBias	Marginal	6.9	6.9

Appendix

Table 15: Comparison of maxROR (%) computed from Model 1 and Model 2. Model 1 is the GEE used in the main experiments, with adoption included as a main effect. Model 2 additionally includes an adoption-by-group interaction. The maxROR values are nearly unchanged, indicating that our main substantive-fairness conclusions are not driven by collider-related distortion from the adoption specification.

Treatment	Cvg (Gap)%	Size (Gap)	Accuracy (Gap)%	maxROR%
Control			78.96 (2.86)
Marginal	89.5 (2.75)	1.68 (.050)	80.71 (3.75)	6.9
Mondrian	90.0 (.220)	1.80 (.474)	80.96 (3.90)	8.1
Label-Clustered	90.2 (2.80)	1.81 (.033)	79.72 ( 2.55 )	1.6
Group-Clustered	90.2 (.193)	1.75 (.419)	81.05 (4.53)	12.5
Backward	91.5 (1.87)	2.50 (.025)	79.41 ( 2.87 )	0.3

Appendix

Table 16: Accuracy and fairness result on BiosBias experiment with GPT-4o-mini as the LLM-in-the-loop.

Treatment	Cvg (Gap)%	Size (Gap)	Accuracy (Gap)%	maxROR%
Control			67.54 (4.97)
Marginal	89.7 (2.76)	1.69 (.027)	76.03 (3.28)	50.0
Mondrian	89.1 (0.00)	1.70 (.435)	75.98 (5.03)	66.3
Label-Clustered	90.1 (.019)	1.78 (.067)	74.83 ( 1.55 )	36.0
Group-Clustered	90.3 (.010)	1.74 (.343)	75.74 (4.30)	59.3
Backward	91.5 (.018)	2.49 (.016)	71.19 ( 1.38 )	36.2

Appendix

Table 17: Accuracy and fairness result on BiosBias experiment with Llama-3.2-3B as the LLM-in-the-loop.

Treatment	Cvg (Gap)%	Size (Gap)	Accuracy (Gap)%	maxROR%
Control			21.11 (0.67)
Marginal	88.33 (5.56)	1.89 (.039)	44.28 ( 1.00 )	10.5
Mondrian	87.50 (.556)	1.86 (.578)	44.56 (12.0)	79.2
Label-Clustered	87.78 (2.22)	1.92 (.011)	46.22 ( 1.33 )	12.1
Group-Clustered	87.50 (.556)	1.90 (.594)	42.94 (15.44)	110.3
Backward	91.94 (8.33)	2.48 (.011)	39.22 (2.22)	17.0

Appendix

Table 18: Accuracy and fairness result on RAVDESS experiment with GPT-4o-audio-preview as the LLM-in-the-loop.

Treatment	Cvg (Gap)%	Size (Gap)	Accuracy (Gap)%	maxROR%
Control			74.04 (18.00)
Marginal	89.9 (10.7)	2.62 (.738)	76.91 ( 16.27 )	9.0
Mondrian	89.9 (3.02)	2.69 (2.68)	77.06 (21.71)	37.7
Label-Clustered	89.1 (7.46)	2.92 (.356)	78.81 ( 14.47 )	13.5
Group-Clustered	89.1 (8.14)	2.50 (1.28)	77.22 (18.05)	14.5
Backward	90.3 (8.85)	3.50 (.053)	75.54 (18.39)	8.5

Appendix

Table 19: Accuracy and fairness result on FACET experiment with Qwen2.5-VL-7B-Instruct as the LLM-in-the-loop.

Treatment	Cvg (Gap)%	Size (Gap)	Accuracy (Gap)%	maxROR%
Control			14.70 (2.22)
Marginal	89.8 (3.18)	5.35 (.256)	20.41 (3.15)	19.4
Mondrian	89.5 (3.87)	7.16 (1.08)	14.91 ( 1.59 )	17.5
Label-Clustered	89.9 (3.67)	5.33 (.270)	18.68 ( 2.88 )	7.2
Group-Clustered	89.8 (2.88)	5.37 (.410)	19.02 (5.05)	23.1
Backward	92.3 (3.03)	6.50 (.010)	15.02 (4.37)	19.7

Appendix

Table 20: Accuracy and fairness result on ACSIncome experiment with Qwen2.5-7B as the LLM-in-the-loop.

Treatment	Accuracy (Gap)%	maxROR%
Control	79.01 (2.84)
Marginal	80.79 (3.69)	9.0 $\pm$ .22
Mondrian	81.06 (3.87)	9.7 $\pm$ .22
Label-Clustered	79.78 ( 2.54 )	6.1 $\pm$ .16
Group-Clustered	81.15 (4.49)	13.1 $\pm$ .26
Backward	79.49 ( 2.82 )	6.5 $\pm$ .16

Appendix

Table 21: Mean accuracy (gap) and mean

\pm

one standard error for maxROR over 1,000 resamples for BiosBias experiment on Table 16

Treatment	Accuracy (Gap)%	maxROR%
Control	21.07 (.840)
Marginal	44.26 ( .820 )	24.6 $\pm$ 1.1
Mondrian	44.56 (11.79)	70.0 $\pm$ 1.2
Label-Clustered	46.09 ( 1.29 )	23.4 $\pm$ .78
Group-Clustered	42.89 (15.25)	92.5 $\pm$ 1.5
Backward	39.28 (2.17)	18.4 $\pm$ .49

Appendix

Table 22: Mean accuracy (gap) and mean

\pm

one standard error for maxROR over 1,000 resamples for RAVDESS experiment on Table 18

Treatment	Accuracy (Gap)%	maxROR%
Control	73.64 (18.17)
Marginal	76.69 ( 16.33 )	23.4 $\pm$ .38
Mondrian	76.26 (21.76)	42.3 $\pm$ .60
Label-Clustered	78.86 ( 14.64 )	28.5 $\pm$ .44
Group-Clustered	77.04 (18.07)	30.8 $\pm$ .47
Backward	75.07 (18.43)	21.6 $\pm$ .35

Appendix

Table 23: Mean accuracy (gap) and mean

\pm

one standard error for maxROR over 1,000 resamples for FACET experiment on Table 19

Treatment	Accuracy (Gap)%	maxROR%
Control	14.55 (2.27)
Marginal	20.29 (3.22)	32.6 $\pm$ .45
Mondrian	14.67 ( 1.55 )	30.9 $\pm$ .45
Label-Clustered	18.38 ( 2.90 )	23.4 $\pm$ .34
Group-Clustered	18.19 (5.06)	32.2 $\pm$ .43
Backward	15.10 (4.35)	33.2 $\pm$ .45

Appendix

Table 24: Mean accuracy (gap) and mean

\pm

one standard error for maxROR over 1,000 resamples for ACSIncome experiment on Table 20

Treatment	Group	Cvg%	Size	Singleton%	Accuracy%
Control	Female				80.43
Control	Male				77.58
Marginal	Female	90.92	1.65	62.33	82.64
Marginal	Male	88.17	1.70	60.14	78.89
Mondrian	Female	89.89	1.56	65.33	82.97
Mondrian	Male	90.11	2.03	29.00	79.07

Appendix

Table 25: Continuation of Table 16 . Results on BiosBias experiment for each group.

Panel of Figure 8 — Figure 8: Experiment results of BiosBias with $K=3$ in label-clustered CP and $K=2$ in group-clustered CP. LLM-in-the-loop: GPT-4o-mini.

Treatment	Group	Cvg%	Size	Singleton%	Accuracy%
Control	Female				20.78
Control	Male				21.44
Marginal	Female	91.11	1.91	37.78	44.78
Marginal	Male	85.56	1.87	40.56	43.78
Mondrian	Female	87.78	1.57	57.78	50.56
Mondrian	Male	87.22	2.15	22.22	38.56

Appendix

Table 26: Continuation of Table 18 . Results on RAVDESS experiment for each group.

Panel of Figure 9 — Figure 9: Experiment results of RAVDESS with $K=2$ in both label-clustered and group-clustered CP. LLM-in-the-loop: GPT-4o-audio-preview.

Treatment	Group	Cvg%	Size	Singleton%	Accuracy%
Control	Middle				71.40
	Older				65.25
	Unknown				74.63
	Younger				83.24
Marginal	Middle	88.61	2.72	29.22	74.49
	Older	84.62	2.91	18.68	69.51

Appendix

Table 27: Continuation of Table 19 . Result on FACET experiment for each group.

Panel of Figure 10 — Figure 10: Experiment results of FACET with $K=2$ in both label-clustered and group-clustered CP. LLM-in-the-loop: Qwen2.5-VL-7B-Instruct.

Treatment	Group	Cvg%	Size	Singleton%	Accuracy%
Control	All Other Races				15.12
	Asian				12.90
	Black or African American				15.08
	Two or More Races				15.02
	White				14.72
Marginal	All Other Races	87.83	5.44	.654	19.08

Appendix

Table 28: Continuation of Table 20 . Result on ACSIncome experiment for each group.

Panel of Figure 11 — Figure 11: Experiment results of ACSIncome with $K=2$ in both label-clustered and group-clustered CP. LLM-in-the-loop: Qwen2.5-7B.

Panel of Figure 12 — Figure 12: Coverage gap (blue dots, left axis) and set size gap (red squares, right axis) across CP methods on FACET, BiosBias, RAVDESS, and ACSIncome.

$K$	$1$	$2$	$3$	$4$	$5$	$6$	$7$	$8$	$9$	$10$
Set-size gap	.050	.024	.033	.045	.027	.034	.053	.054	.055	.057
maxROR (%)	6.9	4.4	1.6	2.7	2.0	0.5	1.1	1.8	2.7	3.1

Appendix

Table 29: Sensitivity of set-size gap and downstream fairness ( maxROR ) to the number of label clusters

K

for Label-Clustered CP on BiosBias. The case

K=1

corresponds to Marginal CP. Set-size gap denotes the absolute difference in average prediction set size between female and male groups.

Explore similar work

May 12, 2026

Fair Conformal Classification via Learning Representation-Based Groups

Conformal prediction methods provide statistically rigorous marginal coverage guarantees for machine learning models, but such guarantees fail to account for algorithmic biases, thereby undermining fairness and trust. This paper introduces a fair conformal inference framework for classification tasks. The proposed method constructs prediction sets that guarantee conditional coverage on adaptively identified subgroups, which can be implicitly defined through nonlinear feature combinations. By balancing effectiveness and efficiency in producing compact, informative prediction sets and ensuring adaptive equalized coverage across unfairly treated subgroups, our approach paves a practical pathway toward trustworthy machine learning. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of the framework.

Senrong Xu, Yanke Zhou, Yuhao Tan +5

State Key Lab of Novel Software Technology, Nanjing University, China · ETH Zürich · Birkbeck, University of London

Group-Conditional Conformal Prediction Conformal Prediction Group Fairness

Ask PDF

Apr 8, 2026

CAFP: A Post-Processing Framework for Group Fairness via Counterfactual Model Averaging

Ensuring fairness in machine learning predictions is a critical challenge, especially when models are deployed in sensitive domains such as credit scoring, healthcare, and criminal justice. While many fairness interventions rely on data preprocessing or algorithmic constraints during training, these approaches often require full control over the model architecture and access to protected attribute information, which may not be feasible in real-world systems. In this paper, we propose Counterfactual Averaging for Fair Predictions (CAFP), a model-agnostic post-processing method that mitigates unfair influence from protected attributes without retraining or modifying the original classifier. CAFP operates by generating counterfactual versions of each input in which the sensitive attribute is flipped, and then averaging the model's predictions across factual and counterfactual instances. We provide a theoretical analysis of CAFP, showing that it eliminates direct dependence on the protected attribute, reduces mutual information between predictions and sensitive attributes, and provably bounds the distortion introduced relative to the original model. Under mild assumptions, we further show that CAFP achieves perfect demographic parity and reduces the equalized odds gap by at least half the average counterfactual bias.

Irina Arévalo, Marcos Oliva

School of Civil Engineering, Universidad Politecnica de Madrid, Profesor Aranguren 3, Madrid, 28040, Spain. · Bain&Company, Velazquez, Madrid, 28006, Spain.

Algorithmic Fairness Group Fairness Counterfactual Fairness

Ask PDF

May 14, 2026

On the Burden of Achieving Fairness in Conformal Prediction

Conformal prediction is often calibrated with a single pooled threshold, but this can hide cross-group heterogeneity in score distributions and distort group-wise coverage. We study this phenomenon through the population score distributions underlying split conformal calibration. First, we derive a conservation law and lower bound showing that pooled calibration incurs irreducible group-wise coverage distortion at a scale set by cross-group quantile heterogeneity. Second, we demonstrate that the two leading fairness definitions for conformal prediction, Equalized Coverage and Equalized Set Size, are fundamentally in tension. Third, we quantify the cost of moving between policies which treat groups separately or pool them. Experiments on synthetic and real data confirm the same bidirectional trade-off after finite-sample calibration. Our results show that, for the policy families studied here, calibration choice does not remove cross-group heterogeneity; it determines whether the resulting distortion appears in the coverage or size dimension, providing a principled lens for analyzing fairness-oriented calibration choices in practice.

Ziang Gao, Pengqi Liu, Archer Yi Yang +3

McGill University Montreal, Canada · TD Insurance Montreal, Canada · Layer 6 AI Toronto, Canada

Conformal Prediction Group Fairness

Ask PDF