Softmax is not Enough (for Adaptive Conformal Classification)

Navid Akhavan Attar; Hesam Asadollahzadeh; Ling Luo; Uwe Aickelin

cs.LGFeb 23, 2026

Softmax is not Enough (for Adaptive Conformal Classification)

Authors: Navid Akhavan Attar, Hesam Asadollahzadeh, Ling Luo, Uwe Aickelin

Organizations: School of Computing and Information Systems, Faculty of Engineering and IT The University of Melbourne, Victoria, Australia

Abstract

The merit of Conformal Prediction (CP), as a distribution-free framework for uncertainty quantification, depends on generating prediction sets that are efficient, reflected in small average set sizes, while adaptive, meaning they signal uncertainty by varying in size according to input difficulty. A central limitation for deep conformal classifiers is that the nonconformity scores are derived from softmax outputs, which can be unreliable indicators of how certain the model truly is about a given input, sometimes leading to overconfident misclassifications or undue hesitation. In this work, we argue that this unreliability can be inherited by the prediction sets generated by CP, limiting their capacity for adaptiveness. We propose a new approach that leverages information from the pre-softmax logit space, using the Helmholtz Free Energy as a measure of model uncertainty and sample difficulty. By reweighting nonconformity scores with a monotonic transformation of the energy score of each sample, we improve their sensitivity to input difficulty. Our experiments with four state-of-the-art score functions on multiple datasets and deep architectures show that this energy-based enhancement improves the adaptiveness of the prediction sets, leading to a notable increase in both efficiency and adaptiveness compared to baseline nonconformity scores, without introducing any post-hoc complexity.

Figures & tables

Figure 1: Prediction sets from a standard method ( APS Romano et al. (2020) ) versus our energy-based variant, demonstrating improved adaptiveness on ImageNet. (i) For an easy input like the image of a Macaw , whose clear visual cues (vivid colors, long tail) make it simple to classify, our energy-based method produces a smaller, more efficient set. (ii) For a hard input, a bird image labeled as Hummingbird , its appearance deviates from typical hummingbirds (e.g., a thicker, less tapered beak) and shares features with other bird classes, making the image difficult for the model. In this case, the energy-based method returns a larger prediction set, signaling higher uncertainty. (iii) Finally, for an out-of-distribution (OOD) input like a brain MRI that the model wasn’t trained on, our method generates a much larger set, warning the user that the prediction is unreliable. This improvement in adaptive behavior is guided by the Helmholtz free energy, which captures the model’s uncertainty about an input.

Figure 2: (a) Softmax probability distributions and (b) raw logit outputs of two CIFAR-100 samples computed by a trained ResNet-56. Both samples receive similarly high softmax confidence scores, despite differing significantly in difficulty (1 vs. 27). In contrast, their negative energy scores more clearly reflect this difference.

Figure 3: Distribution of negative energy scores (

-F(x)

), stratified by sample difficulty. As difficulty increases, the distribution shifts toward lower energy values, indicating reduced model confidence.

CIFAR-100 (ResNet-56)
		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$		$\bm{\alpha=0.025}$		$\bm{\alpha=0.01}$
Method		Coverage	Set Size	Coverage	Set Size	Coverage	Set Size	Coverage	Set Size
	w/o Energy	0.90 $\pm$ 0.01	3.17 $\pm$ 0.09	0.95 $\pm$ 0.00	6.91 $\pm$ 0.24	0.975 $\pm$ 0.002	13.29 $\pm$ 0.44	0.99 $\pm$ 0.00	25.79 $\pm$ 1.20
APS	w/ Energy	0.90 $\pm$ 0.01	3.16 $\pm$ 0.08	0.95 $\pm$ 0.00	6.49 $\pm$ 0.24	0.974 $\pm$ 0.001	11.48 $\pm$ 0.25	0.99 $\pm$ 0.00	22.90 $\pm$ 0.82
	w/o Energy	0.90 $\pm$ 0.00	3.13 $\pm$ 0.07	0.95 $\pm$ 0.01	8.17 $\pm$ 0.47	0.974 $\pm$ 0.002	16.38 $\pm$ 0.81	0.99 $\pm$ 0.00	30.88 $\pm$ 1.90
RAPS	w/ Energy	0.90 $\pm$ 0.01	3.13 $\pm$ 0.08	0.95 $\pm$ 0.00	6.18 $\pm$ 0.25	0.974 $\pm$ 0.002	11.34 $\pm$ 0.32	0.99 $\pm$ 0.00	23.63 $\pm$ 0.86

Table 1: Performance comparison of APS , RAPS , and SAPS nonconformity score functions and their energy-based variants on CIFAR-100, ImageNet, and Places365 at miscoverage levels

{\alpha\in\{0.01,0.025,0.05,0.1\}}

. Results are averaged over 10 trials. For the Set Size column, lower is better. Bold values indicate the best performance within each method family (e.g., APS with and without Energy ).

Panel of Figure 4 — Figure 4: Distributions of negative energy scores across various class bins under balanced and imbalanced training. Results are for CIFAR-100. (a) Balanced model: scores are consistent across class bins. (b) Imbalanced model ( $\lambda=0.03$ ): minority classes exhibit lower negative energy scores, reflecting reduced confidence.

CIFAR-100-LT ( $\bm{\lambda=0.005}$ ) (ResNet-56)
		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$		$\bm{\alpha=0.025}$		$\bm{\alpha=0.01}$
Method		Coverage	Set Size	Coverage	Set Size	Coverage	Set Size	Coverage	Set Size
	w/o Energy	0.90 $\pm$ 0.01	8.44 $\pm$ 0.30	0.95 $\pm$ 0.00	17.22 $\pm$ 0.55	0.973 $\pm$ 0.003	28.32 $\pm$ 0.86	0.99 $\pm$ 0.00	45.10 $\pm$ 0.75
APS	w/ Energy	0.90 $\pm$ 0.01	7.41 $\pm$ 0.22	0.95 $\pm$ 0.01	13.30 $\pm$ 0.65	0.973 $\pm$ 0.004	21.72 $\pm$ 1.27	0.99 $\pm$ 0.00	34.78 $\pm$ 1.72
	w/o Energy	0.90 $\pm$ 0.01	8.88 $\pm$ 0.30	0.95 $\pm$ 0.00	18.54 $\pm$ 0.73	0.972 $\pm$ 0.003	30.09 $\pm$ 0.94	0.99 $\pm$ 0.00	51.65 $\pm$ 1.75
RAPS	w/ Energy	0.90 $\pm$ 0.01	7.59 $\pm$ 0.25	0.95 $\pm$ 0.01	13.26 $\pm$ 0.69	0.973 $\pm$ 0.004	22.27 $\pm$ 1.24	0.99 $\pm$ 0.00	35.43 $\pm$ 1.88

Table 2: Performance comparison of different nonconformity scores and their energy-based variants on imbalanced CIFAR-100 with an imbalance factor of

\lambda=0.005

and at miscoverage levels

{\alpha\in\{0.01,0.025,0.05,0.1\}}

. Results are averaged over 10 trials with a ResNet-56 model. For the Set Size column, lower is better. Bold values indicate the best performance within each method family (e.g., APS with and without Energy ). Results for additional

\lambda

values are provided in Appendix K .

Panel of Figure 5 — Figure 5: Prediction set size distributions for the SAPS score and its energy-based variant with $\alpha=0.05$ , on (a) in-distribution CIFAR-100 and (b) out-of-distribution Places365 data. The energy-based variant produces larger prediction sets on OOD data. Here, $\mu$ represents the overall set size.

		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$
Method		Set Size ID (in distribution)	Set Size OOD (out of distribution)	Set Size ID (in distribution)	Set Size OOD (out of distribution)
	w/o Energy	3.17 $\pm$ 0.09	6.18 $\pm$ 0.25	6.91 $\pm$ 0.24	14.91 $\pm$ 0.81
APS	w/ Energy	3.16 $\pm$ 0.08	86.76 $\pm$ 0.94	6.49 $\pm$ 0.24	93.40 $\pm$ 0.53
	w/o Energy	3.13 $\pm$ 0.07	3.70 $\pm$ 0.04	8.17 $\pm$ 0.47	8.95 $\pm$ 0.47
RAPS	w/ Energy	3.13 $\pm$ 0.08	5.53 $\pm$ 0.07	6.18 $\pm$ 0.25	9.05 $\pm$ 0.49
	w/o Energy	2.87 $\pm$ 0.09	3.78 $\pm$ 0.05	7.47 $\pm$ 0.43	8.82 $\pm$ 0.46

Table 3: Comparison of average prediction set sizes for a ResNet-56 model trained on CIFAR-100. The model is evaluated on both in-distribution (CIFAR-100) and out-of-distribution (Places365) data. Energy-based variants demonstrate adaptiveness to the distributional shift by maintaining small sets on ID data while producing significantly larger sets for OOD inputs. Bold values indicate the preferred result: the smallest average set size for ID (efficiency) and the largest for OOD (uncertainty awareness).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$x\!\in\!\mathbb{R}^{D}$	Input (feature) vector
$K$	Total number of classes
$y\!\in\!\{1,\dots,K\}$	Class label
$\mathbf{f}(x)\!=\!(f_{1},\dots,f_{K})$	Pre-softmax logit vector produced by the classifier
$\hat{\pi}(y\mid x)$	Model’s softmax probability for class $y$ , Equation 1
$f_{\max}(x)$	$\max_{k}f_{k}(x)$

Appendix

Table 4: Notation used in this work.

C(x_{\text{new}})=\{y\in\{1,\dots,K\}:S(x_{\text{new}},y)\leq\hat{q}\}

Appendix

Algorithm 1 Split Conformal Prediction

		Method	Nonconformity Score
		LAC / THR ( Sadinle et al., 2019 )	$S_{\texttt{LAC}}(x,y)\;=\;1-\hat{\pi}(y\mid x)\equiv-\hat{\pi}(y\mid x)$
Adaptive Scores	$\left.\vbox{\vrule height=0.0pt,width=0.0pt}\textnormal{\hbox{}}\right\{$	APS ( Romano et al., 2020 )	$S_{\texttt{APS}}(x,y)\;=\;{\sum_{k=1}^{K}\hat{\pi}(k\mid x)\,\mathbb{I}\bigl\{\hat{\pi}(k\mid x)>\hat{\pi}(y\mid x)\bigr\}}\;+\;u\,\cdot\hat{\pi}(y\mid x)$
		RAPS ( Angelopoulos et al., 2021 )	$S_{\texttt{RAPS}}(x,y)\;=\;S_{\texttt{APS}}(x,y)\;+\;\lambda\,\bigl(o_{x}(y)-k_{\text{reg}}\bigr)^{+}$
		SAPS ( Huang et al., 2024b )	$S_{\texttt{SAPS}}(x,y)\;=\;\begin{cases}u\,\cdot\hat{\pi}_{\max}(x),&o_{x}(y)=1,\\[4.0pt] \hat{\pi}_{\max}(x)+\bigl(o_{x}(y)-2+u\bigr)\,\lambda,&\text{otherwise},\end{cases}$

Appendix

Table 5: Nonconformity scores considered in this work. All scores are functions of

\hat{\pi}(y\mid x)

where

u\!\sim\!U[0,1]

; The function

o_{x}(y)

returns the rank position of label

y

among all possible labels, ordered by the model’s predicted probabilities (with rank 1 being the most likely);

\hat{\pi}_{\max}(x)=\max_{k}\hat{\pi}(k\mid x)

;

(\cdot)^{+}

denotes the positive part;

\lambda

and

k_{\text{reg}}

are hyperparameters.

Panel of Figure 6 — Figure 6: Evolution of uncertainty landscapes on a 2D toy dataset throughout the training process. Columns represent progressive checkpoints from the early training phase (left) to full convergence (right). The red line indicates the decision boundary. (a) Softmax probabilities saturate rapidly. The yellow region (confidence $\approx 1.0$ ) is flat, making points near the boundary indistinguishable from points far away. (b) Entropy exhibits similar saturation (dark blue region), vanishing to zero for most of the domain. (c) Negative Free Energy retains gradients throughout the domain. Note the continuous color transition scaling with the distance from the decision boundary, identifying “easier” points with higher values even when softmax is saturated.

		APS				RAPS				SAPS
		w/o Energy		w/ Energy		w/o Energy		w/ Energy		w/o Energy		w/ Energy
Difficulty	Count	Cov.	Set Size	Cov.	Set Size	Cov.	Set Size	Cov.	Set Size	Cov.	Set Size	Cov.	Set Size
1 to 1	15990	1.00	39.08	1.00	32.31	1.00	36.97	1.00	30.53	1.00	35.69	1.00	29.11
2 to 3	2547	1.00	38.38	1.00	34.96	1.00	37.17	1.00	34.10	1.00	36.98	1.00	33.69
4 to 6	638	1.00	37.86	1.00	36.46	1.00	37.32	1.00	36.29	1.00	37.35	1.00	36.02
7 to 10	306	1.00	37.66	1.00	37.09	1.00	37.40	1.00	37.20	1.00	37.47	1.00	36.96

Appendix

Table 6: Coverage and average set size on ImageNet, stratified by sample difficulty. Results are for a ResNet-50 model at

\alpha=0.01

and averaged over 10 trials. The table compares baseline adaptive scores with their energy-based variants, which generally produce smaller sets for easier samples while maintaining coverage.

		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$		$\bm{\alpha=0.025}$		$\bm{\alpha=0.01}$
Method		Coverage	Set Size	Coverage	Set Size	Coverage	Set Size	Coverage	Set Size
CIFAR-100 (ResNet-56)
	w/o Energy	0.90 $\pm$ 0.01	2.54 $\pm$ 0.06	0.95 $\pm$ 0.00	5.26 $\pm$ 0.17	0.974 $\pm$ 0.001	9.50 $\pm$ 0.21	0.99 $\pm$ 0.00	21.14 $\pm$ 0.74
LAC	w/ Energy	0.90 $\pm$ 0.01	2.52 $\pm$ 0.04	0.95 $\pm$ 0.00	5.21 $\pm$ 0.15	0.974 $\pm$ 0.002	9.09 $\pm$ 0.16	0.99 $\pm$ 0.00	20.65 $\pm$ 0.83
ImageNet (ResNet-50)
	w/o Energy	0.90 $\pm$ 0.00	1.49 $\pm$ 0.01	0.95 $\pm$ 0.00	2.68 $\pm$ 0.05	0.975 $\pm$ 0.001	5.48 $\pm$ 0.18	0.99 $\pm$ 0.00	14.79 $\pm$ 0.80

Appendix

Table 7: Performance of the LAC nonconformity score function and its energy-based variant on CIFAR-100, ImageNet, and Places365 at miscoverage levels

{\alpha\in\{0.01,0.025,0.05,0.1\}}

. Results are averaged over 10 trials. For the Set Size column, lower is better. Bold values indicate the best performance within the method family (with and without Energy ).

		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$		$\bm{\alpha=0.025}$		$\bm{\alpha=0.01}$
Method		Coverage	Set Size	Coverage	Set Size	Coverage	Set Size	Coverage	Set Size
CIFAR-100-LT ( $\bm{\lambda=0.005}$ , mild imbalance) (ResNet-56)
	w/o Energy	0.897 $\pm$ 0.007	7.04 $\pm$ 0.24	0.947 $\pm$ 0.004	12.98 $\pm$ 0.49	0.973 $\pm$ 0.003	21.31 $\pm$ 0.84	0.988 $\pm$ 0.002	33.97 $\pm$ 1.37
LAC	w/ Energy	0.897 $\pm$ 0.006	6.91 $\pm$ 0.19	0.947 $\pm$ 0.005	12.68 $\pm$ 0.51	0.973 $\pm$ 0.004	20.56 $\pm$ 1.01	0.989 $\pm$ 0.002	32.99 $\pm$ 0.87
CIFAR-100-LT ( $\bm{\lambda=0.01}$ ) (ResNet-56)
	w/o Energy	0.900 $\pm$ 0.007	11.92 $\pm$ 0.41	0.951 $\pm$ 0.003	20.75 $\pm$ 0.51	0.975 $\pm$ 0.003	30.58 $\pm$ 0.68	0.990 $\pm$ 0.001	45.93 $\pm$ 1.02

Appendix

Table 8: Performance of LAC and its energy-based variant on imbalanced CIFAR-100 with varying imbalance factors (

\lambda\in\{0.005,0.01,0.02,0.03\}

) and at miscoverage levels

{\alpha\in\{0.01,0.025,0.05,0.1\}}

. Results are averaged over 10 trials with a ResNet-56 model. For the average set size, lower is better. Bold values indicate the best performance.

		LAC
		$\alpha=0.01$				$\alpha=0.025$
		w/o Energy		w/ Energy		w/o Energy		w/ Energy
Difficulty Level	Count	Cov.	Set Size	Cov.	Set Size	Cov.	Set Size	Cov.	Set Size
1 to 1	15990	1.00	10.77	1.00	9.24	1.00	4.34	1.00	4.25
2 to 3	2547	1.00	21.47	1.00	20.61	0.99	8.86	0.99	8.78
4 to 6	638	0.99	32.07	0.98	32.96	0.93	13.29	0.93	13.36

Appendix

Table 9: Coverage and average set size for the LAC method on ImageNet, stratified by sample difficulty. Results are shown for

\alpha=0.01

and

\alpha=0.025

. The table compares the baseline LAC with its energy-based variant.

Figure 7: Class distributions under varying imbalance levels. The number of samples per class follows an exponential decay pattern proportional to

\exp(-\lambda\cdot j)

, where larger

\lambda

values induce stronger imbalance.

CIFAR-100-LT ( $\bm{\lambda=0.01}$ , mild imbalance) (ResNet-56)
		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$		$\bm{\alpha=0.025}$		$\bm{\alpha=0.01}$
Method		Coverage	Set Size	Coverage	Set Size	Coverage	Set Size	Coverage	Set Size
	w/o Energy	0.90 $\pm$ 0.01	14.56 $\pm$ 0.46	0.95 $\pm$ 0.00	25.59 $\pm$ 0.25	0.975 $\pm$ 0.003	37.56 $\pm$ 0.98	0.99 $\pm$ 0.00	55.26 $\pm$ 2.49
APS	w/ Energy	0.90 $\pm$ 0.01	11.86 $\pm$ 0.37	0.95 $\pm$ 0.00	21.44 $\pm$ 0.52	0.975 $\pm$ 0.003	31.70 $\pm$ 0.97	0.99 $\pm$ 0.00	46.40 $\pm$ 0.72
	w/o Energy	0.90 $\pm$ 0.01	15.48 $\pm$ 0.47	0.95 $\pm$ 0.00	27.62 $\pm$ 0.33	0.974 $\pm$ 0.003	40.87 $\pm$ 1.00	0.99 $\pm$ 0.00	60.13 $\pm$ 1.86
RAPS	w/ Energy	0.90 $\pm$ 0.01	11.77 $\pm$ 0.42	0.95 $\pm$ 0.00	21.57 $\pm$ 0.56	0.975 $\pm$ 0.003	31.93 $\pm$ 0.98	0.99 $\pm$ 0.00	47.30 $\pm$ 1.01

Appendix

Table 10: Performance comparison of APS , RAPS , SAPS , and their Energy-based variants on imbalanced CIFAR-100 with varying imbalance factors (

\lambda\in\{0.01,0.02,0.03\}

) and at miscoverage levels

{\alpha\in\{0.01,0.025,0.05,0.1\}}

. Results are averaged over 10 trials with a ResNet-56 model. For the Set Size column, lower is better. Bold values indicate the best performance within each method family.

CIFAR-100-LT ( $\lambda=0.005$ , mild imbalance)
		$\alpha=0.025$			$\alpha=0.01$
Method	Variant	Cov	Size	MacroCov	Cov	Size	MacroCov
LAC	Standard	0.97 $\pm$ 0.00	21.48 $\pm$ 0.86	0.97 $\pm$ 0.00	0.99 $\pm$ 0.00	34.60 $\pm$ 1.77	0.99 $\pm$ 0.00
	Prevalence-Adj. (PAS)	0.98 $\pm$ 0.00	22.84 $\pm$ 0.99	0.98 $\pm$ 0.00	0.99 $\pm$ 0.00	36.23 $\pm$ 0.98	0.99 $\pm$ 0.00
	Energy-based	0.97 $\pm$ 0.00	21.11 $\pm$ 0.79	0.97 $\pm$ 0.00	0.99 $\pm$ 0.00	33.18 $\pm$ 1.44	0.99 $\pm$ 0.00
APS	Standard	0.97 $\pm$ 0.00	28.61 $\pm$ 1.23	0.97 $\pm$ 0.00	0.99 $\pm$ 0.00	46.37 $\pm$ 1.39	0.99 $\pm$ 0.00

Appendix

Table 11: Performance on fully imbalanced CIFAR-100-LT for high confidence levels (

\alpha\in\{0.025,0.01\}

). For each method, we compare the Standard baseline, the Prevalence-Adj. variant, and our Energy-based variant. Lower Set Size is better.

CIFAR-100-LT ( $\lambda=0.005$ , mild imbalance)
		$\alpha=0.1$			$\alpha=0.05$
Method	Variant	Cov	Size	MacroCov	Cov	Size	MacroCov
LAC	Standard	0.90 $\pm$ 0.01	7.12 $\pm$ 0.26	0.90 $\pm$ 0.01	0.95 $\pm$ 0.01	13.08 $\pm$ 0.55	0.95 $\pm$ 0.01
	Prevalence-Adj. (PAS)	0.90 $\pm$ 0.01	7.26 $\pm$ 0.30	0.90 $\pm$ 0.01	0.95 $\pm$ 0.01	13.39 $\pm$ 0.60	0.95 $\pm$ 0.01
	Energy-based	0.90 $\pm$ 0.01	6.96 $\pm$ 0.23	0.90 $\pm$ 0.01	0.95 $\pm$ 0.01	12.74 $\pm$ 0.54	0.95 $\pm$ 0.01
APS	Standard	0.90 $\pm$ 0.01	8.48 $\pm$ 0.42	0.90 $\pm$ 0.01	0.95 $\pm$ 0.01	17.55 $\pm$ 1.03	0.95 $\pm$ 0.01

Appendix

Table 12: Performance on fully imbalanced CIFAR-100-LT for lower confidence levels (

\alpha\in\{0.1,0.05\}

). For each method, we compare the Standard baseline, the Prevalence-Adj. variant, and our Energy-based variant. Lower Set Size is better.

		$\bm{\alpha=0.1}$				$\bm{\alpha=0.05}$
Method		Coverage	Set Size $\downarrow$	CovGap $\downarrow$	SSCV $\downarrow$	Coverage	Set Size $\downarrow$	CovGap $\downarrow$	SSCV $\downarrow$
	w/o Energy	0.89 $\pm$ 0.00	9.13 $\pm$ 0.14	5.03 $\pm$ 0.14	0.119 $\pm$ 0.015	0.95 $\pm$ 0.00	21.50 $\pm$ 0.59	3.25 $\pm$ 0.12	0.124 $\pm$ 0.039
APS	w/ Energy	0.89 $\pm$ 0.00	8.65 $\pm$ 0.17	5.09 $\pm$ 0.18	0.100 $\pm$ 0.00	0.95 $\pm$ 0.00	19.20 $\pm$ 0.49	3.24 $\pm$ 0.10	0.087 $\pm$ 0.027
	w/o Energy	0.89 $\pm$ 0.00	9.11 $\pm$ 0.18	4.98 $\pm$ 0.15	0.100 $\pm$ 0.00	0.95 $\pm$ 0.00	22.03 $\pm$ 0.59	3.24 $\pm$ 0.13	0.090 $\pm$ 0.024
RAPS	w/ Energy	0.89 $\pm$ 0.00	8.59 $\pm$ 0.17	4.95 $\pm$ 0.19	0.112 $\pm$ 0.011	0.95 $\pm$ 0.00	19.11 $\pm$ 0.47	3.22 $\pm$ 0.11	0.119 $\pm$ 0.020
	w/o Energy	0.89 $\pm$ 0.00	8.92 $\pm$ 0.20	4.98 $\pm$ 0.20	0.100 $\pm$ 0.00	0.95 $\pm$ 0.00	21.68 $\pm$ 0.58	3.26 $\pm$ 0.14	0.060 $\pm$ 0.019

Appendix

Table 13: Class-conditional performance comparison of different nonconformity score functions and their Energy-based variants on the Places365 dataset at miscoverage levels

{\alpha=0.05}

and

{\alpha=0.1}

. For Set Size, CovGap, and SSCV, lower values indicate better performance. Bold values denote the best result within each method family. Results are averaged over 10 trials with a ResNet-50 model.

	Average Set Size (w/o Energy $\rightarrow$ w/ Energy ) $\downarrow$
	Regularization Penalty ( $\lambda$ )
$k_{reg}$	0	0.05	0.1	0.2	0.5	0.7	1.0
1	14.08 $\rightarrow$ 12.88	13.11 $\rightarrow$ 12.88	13.60 $\rightarrow$ 12.24	14.07 $\rightarrow$ 12.61	14.30 $\rightarrow$ 12.90	14.30 $\rightarrow$ 12.95	14.30 $\rightarrow$ 13.00
2	14.09 $\rightarrow$ 12.86	13.13 $\rightarrow$ 12.83	13.58 $\rightarrow$ 12.34	14.05 $\rightarrow$ 12.63	14.30 $\rightarrow$ 12.97	14.30 $\rightarrow$ 13.05	14.30 $\rightarrow$ 13.11
5	14.01 $\rightarrow$ 12.85	13.11 $\rightarrow$ 13.06	13.61 $\rightarrow$ 12.67	14.07 $\rightarrow$ 12.86	14.30 $\rightarrow$ 13.22	14.30 $\rightarrow$ 13.29	14.30 $\rightarrow$ 13.34
10	14.05 $\rightarrow$ 12.90	13.13 $\rightarrow$ 13.07	13.58 $\rightarrow$ 13.57	14.07 $\rightarrow$ 13.46	14.29 $\rightarrow$ 13.62	14.30 $\rightarrow$ 13.74	14.30 $\rightarrow$ 13.82

Appendix

Table 14: Comparison of average prediction set sizes for varying regularization parameters (

k_{reg}

and

\lambda

). Lower set size is better. Values are reported as: Size ( RAPS

\rightarrow

Energy-based RAPS ). The bold value highlights the superior (smaller) set size.

Panel of Figure 8 — Figure 8: Average Set Size heatmap for different hyperparameter settings across Energy-based variants of RAPS and SAPS .

Panel of Figure 9 — Figure 9: Ablation study of the parameter $\beta$ on CIFAR-100 with ResNet-56. The plots show the average prediction set size (shaded regions indicate standard deviation) as a function of $\beta$ . Performance stabilizes for sufficiently large values of $\beta$ , indicating that the method does not require specific tuning of this parameter.

Figure 10: Conceptual diagram of a reliable conformal classifier facing an OOD input. The desired behaviors are to produce an empty or large set, both signaling uncertainty, and to avoid producing a small set that implies false confidence.

		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$
Family		Coverage	Set Size $\downarrow$	Coverage	Set Size $\downarrow$
LAC	baseline	0.898 $\pm$ 0.002	1.487 $\pm$ 0.013	0.950 $\pm$ 0.002	2.682 $\pm$ 0.039
	w/ Energy	0.898 $\pm$ 0.002	1.485 $\pm$ 0.012	0.949 $\pm$ 0.002	2.680 $\pm$ 0.043
	w/ Entropy	0.898 $\pm$ 0.002	1.496 $\pm$ 0.014	0.949 $\pm$ 0.002	2.696 $\pm$ 0.043
APS	baseline	0.899 $\pm$ 0.002	1.605 $\pm$ 0.022	0.950 $\pm$ 0.002	4.007 $\pm$ 0.164
	w/ Energy	0.899 $\pm$ 0.002	1.599 $\pm$ 0.022	0.950 $\pm$ 0.002	3.842 $\pm$ 0.159

Appendix

Table 15: Performance comparison of different nonconformity score functions and their Energy-based and Entropy-based variants on ImageNet using a ResNet-50 classifier at miscoverage levels

{\alpha=0.05}

and

{\alpha=0.1}

. Results are averaged over 10 trials and reported as empirical coverage and average prediction set size. Bold values indicate the best performance within each group. The Entropy-based variants of adaptive method, APS , RAPS , and SAPS are defined as

{S_{\mathbb{H}}(x,y)=\frac{S(x,y)}{\mathbb{H}(x)}}

, and for LAC it is defined as

{S_{\mathbb{H}}(x,y)=-\hat{\pi}(y|x)\cdot\mathbb{H}(x)}

		$\bm{\alpha=0.1}$		$\bm{\alpha=0.05}$
Method	Variant	Set Size $\downarrow$	WSC $\uparrow$	Set Size $\downarrow$	WSC $\uparrow$
APS	w/o Energy	9.02 $\pm$ 0.11	0.892 $\pm$ 0.00	20.98 $\pm$ 0.28	0.949 $\pm$ 0.00
APS	w/ Energy	8.54 $\pm$ 0.18	0.893 $\pm$ 0.00	18.77 $\pm$ 0.31	0.948 $\pm$ 0.00
RAPS	w/o Energy	8.99 $\pm$ 0.15	0.892 $\pm$ 0.00	21.46 $\pm$ 0.27	0.950 $\pm$ 0.00
RAPS	w/ Energy	8.47 $\pm$ 0.18	0.891 $\pm$ 0.00	18.65 $\pm$ 0.31	0.948 $\pm$ 0.00
SAPS	w/o Energy	8.80 $\pm$ 0.15	0.891 $\pm$ 0.00	21.13 $\pm$ 0.29	0.949 $\pm$ 0.00

Appendix

Table 16: X-conditional performance comparison on the Places365 dataset (ResNet-50) using Worst-Slab Coverage (WSC). We compare Set Size and WSC at miscoverage levels

{\alpha=0.1}

and

{\alpha=0.05}

. Lower values are better for Set Size; higher values (closer to

1-\alpha

) are better for WSC. Bold values denote the best Set Size within each method family. The Energy-based variants reduce set size without degrading conditional coverage.

CIFAR-100 (ResNet-56)
		$\bm{\alpha=0.1}$	$\bm{\alpha=0.05}$	$\bm{\alpha=0.025}$	$\bm{\alpha=0.01}$
Method	Comparison	$p$ -value	$p$ -value	$p$ -value	$p$ -value
APS	w/o vs. w/ Energy	0.398	0.001	$<$ 0.0001	$<$ 0.0001
RAPS	w/o vs. w/ Energy	0.500	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001
SAPS	w/o vs. w/ Energy	0.500	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001
ImageNet (ResNet-50)

Appendix

Table 17: Statistical significance (

p

-values from one-tailed Welch’s t-test) for the experiments on balanced datasets presented in Table 1 . Bold values indicate statistical significance (

p<0.05

		$\bm{\alpha=0.1}$	$\bm{\alpha=0.05}$	$\bm{\alpha=0.025}$	$\bm{\alpha=0.01}$
Method	Comparison	$p$ -value	$p$ -value	$p$ -value	$p$ -value
CIFAR-100-LT ( $\lambda=0.005$ )
APS	w/o vs. w/ Energy	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001
RAPS	w/o vs. w/ Energy	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001
SAPS	w/o vs. w/ Energy	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001
CIFAR-100-LT ( $\lambda=0.02$ )

Appendix

Table 18: Statistical significance (

p

-values) for the experiments on imbalanced data presented in Table 2 (

\lambda=0.005

) and Table 10 (

\lambda=0.02

). Bold values indicate statistical significance (

p<0.05

	Accuracy		Average Set Size (w/o Energy $\rightarrow$ w/ Energy ) $\downarrow$
Model	Top-1	Top-5	APS	RAPS	SAPS
ResNet152	$0.783$	$0.940$	3.82 $\rightarrow$ 3.48	3.25 $\rightarrow$ 3.17	2.87 $\rightarrow$ 2.83
ResNet101	$0.774$	$0.936$	3.97 $\rightarrow$ 3.70	3.60 $\rightarrow$ 3.32	3.15 $\rightarrow$ 3.03
ResNet50	$0.761$	$0.929$	4.09 $\rightarrow$ 3.97	4.16 $\rightarrow$ 3.84	3.70 $\rightarrow$ 3.63
ResNet34	$0.733$	$0.914$	9.86 $\rightarrow$ 8.29	9.94 $\rightarrow$ 7.73	9.57 $\rightarrow$ 7.41
ResNet18	$0.698$	$0.891$	14.23 $\rightarrow$ 12.06	15.16 $\rightarrow$ 11.43	14.56 $\rightarrow$ 10.93

Appendix

Table 19: Comparison of average prediction set sizes and accuracy for conformal methods on ImageNet at confidence level of

95\%

(\alpha=0.05)

. Lower average set size is better. Set sizes are shown as baseline (w/o Energy )

\rightarrow

Energy-based (w/ Energy ). The bold value highlights the superior (smaller) set size within each pair. The results are reported as the median of means over 10 trials.

Explore similar work

Feb 24, 2026

DANCE: Doubly Adaptive Neighborhood Conformal Estimation

The recent developments of complex deep learning models have led to unprecedented ability to accurately predict across multiple data representation types. Conformal prediction for uncertainty quantification of these models has risen in popularity, providing adaptive, statistically-valid prediction sets. For classification tasks, conformal methods have typically focused on utilizing logit scores. For pre-trained models, however, this can result in inefficient, overly conservative set sizes when not calibrated towards the target task. We propose DANCE, a doubly locally adaptive nearest-neighbor based conformal algorithm combining two novel nonconformity scores directly using the data's embedded representation. DANCE first fits a task-adaptive kernel regression model from the embedding layer before using the learned kernel space to produce the final prediction sets for uncertainty quantification. We test against state-of-the-art local, task-adapted and zero-shot conformal baselines, demonstrating DANCE's superior blend of set size efficiency and robustness across various datasets.

Brandon R. Feng, Brian J. Reich, Daniel Beaglehole +7

Department of Statistics, North Carolina State University, Raleigh, NC, USA · Amazon.com, Seattle, WA, USA · Department of Computer Science, University of California San Diego, San Diego, CA, USA +2

Conformal Prediction Classification k-Nearest Neighbors

Ask PDF

May 24, 2026

Benchmarking non-conformity score functions in conformal prediction

Conformal prediction is a useful and versatile alternative to model calibration in machine learning classification. It replaces single-class prediction with prediction sets, guaranteeing that the a priori probability of the prediction sets containing the true class is larger than or equal to a pre-specified rate. The size and usefulness of the prediction sets relies heavily on the choice of the non-conformity score function. The scientific literature contains many examples of non-conformity score functions but there is an absence of studies examining their properties and effectiveness. In this paper, we give an overview of properties of non-conformity score functions. We give examples of non-conformity score functions in the existing literature and introduce original modifications. We introduce an original method of evaluating the prediction set sizes of conformal predictors and use it to provide a comparison between non-conformity score functions. We also examine efficacy of different non-conformity score functions for class-conditional conformal prediction in a setting with imbalanced classes.

Sol Erika Boman

Department of Medical Epidemiology and Biostatistics, Karolinska Institutet · Department of Molecular Medicine and Surgery, Karolinska Institutet

Set Prediction Conformal Prediction

Ask PDF

Jun 12, 2026

Audited Conformal Prediction for Classification under Unknown Distribution Shift

We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift. We propose Audited Conformal Prediction (ACP), a method that leverages a small labeled dataset from the target population to train an auxiliary audit model identifying inputs where the legacy model is likely to fail. By integrating the audit model's outputs into the conformal prediction framework, ACP produces prediction sets that guarantee marginal coverage while achieving substantially higher conditional coverage in practice than existing approaches. We develop and analyze two complementary integration strategies -- one targeting marginal coverage with improved conditional performance, the other providing explicit group-conditional coverage guarantees -- and establish theoretical guarantees for both. Experiments on synthetic and real-world datasets validate the method and illustrate trade-offs between prediction set size and conditional coverage.

Yanfei Zhou, Rizal Fathony, Nam H. Nguyen +1

Department of Data Sciences and Operations University of Southern California Los Angeles, CA 90089 · AI Foundations, Capital One New York, NY 10017 · Department of Data Sciences and Operations Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90089

Distribution Shift Robustness Uncertainty Quantification Group-Conditional Conformal Prediction Conformal Prediction

Ask PDF