Scaling Full Conformal Image Classifiers

Julio Silva-Rodríguez; Ender Konukoglu

cs.LGSep 29, 2026

Scaling Full Conformal Image Classifiers

Authors: Julio Silva-Rodríguez, Ender Konukoglu

Organizations: Computer Vision Lab, ETH Zürich

Abstract

Conformal prediction provides set-valued predictions with distribution-free coverage guarantees, making it attractive for high-stakes image classification. However, split conformal prediction is data-inefficient, while full conformal prediction (FCP), despite its stronger statistical efficiency, is computationally prohibitive at scale because it requires candidate-specific model refits at test time. We address this limitation by leveraging zero-shot vision-language models (VLMs) to guide scalable FCP in large label spaces. We introduce Targeted Full Conformal Prediction (T-FCP), which uses a lightweight inductive conformal predictor to prune unlikely labels and applies FCP only to the remaining candidates, reducing computation while retaining the formal guarantee of the combined conformal procedure. We further propose Stabilized Online LDA (SO-LDA), an efficient VLM adaptation solver based on rank-one inverse-covariance updates. Across multiple benchmarks, including ImageNet, T-FCP enables practical full-conformal image classification with modest test-time overhead, yielding efficient prediction sets and more stable empirical coverage than split conformal alternatives.

Figures & tables

			Cov. ( $\alpha=0.10$ )			Set size			Cov. ( $\alpha=0.05$ )			Set size
		Acc.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.
ICP	ZS	65.7	90.1	2.0	77.8	4.8	4.5	37.3	95.2	1.5	84.0	7.9	7.3	28.2
ICP-T	OT Silva-Rodríguez et al. (2025b)	68.6	90.6	2.0	82.3	3.9	3.7	42.4	95.4	1.4	86.8	6.1	5.6	32.6
ICP-T	TIM Silva-Rodríguez et al. (2025a)	71.5	90.6	2.0	80.5	3.6	3.5	45.2	95.4	1.4	87.2	5.6	5.3	34.3
SCP	GD	79.9	90.7	2.7	77.1	2.2	2.1	64.0	95.5	1.9	79.5	3.3	3.0	51.9
FCP	SO-LDA	80.3	90.1	2.2	77.6	2.2	1.8	69.3	95.2	1.6	80.5	3.3	2.6	57.4

Table 1: Comparison of conformal procedures atop CLIP ViT-B/16 using

N=C\times 16

. Results are averaged across 11 datasets, and per-dataset results are in Appendix F.2 .

Panel of Figure 1 — Figure 1: Analysis on empirical coverage and set size distribution on representative datasets. Results using CLIP ViT-B/16 as backbone, $N=C\times 16$ samples for calibration.

Figure 2: Computational efficiency analysis. Results using ViT-B/16, $N=C\times 16$ and $\alpha=0.05$ . All our experiments were conducted on a single NVIDIA A100-PCIE-40GB.

Panel of Figure 3 — Figure 3: Ablation studies on T-FCP. Results using CLIP ViT-B/16 as backbone, $N=C\times 16$ samples for calibration, and conformal sets created at $\alpha=0.1$ error rate.

Panel of Figure 4 — Figure 4: Studies at lower data regimes: (a-d) conformal procedures, and (e) few-shot solvers. Results using CLIP ViT-B/16 as backbone, and conformal sets at $\alpha=0.1$ , averaged for 11 datasets.

Panel of Figure 5 — Figure 5: Generalization atop other CLIP Radford et al. (2021) and MetaCLIP Xu et al. (2024) backbones of conformal procedures. Results using $N=C\times 16$ samples and $\alpha=0.1$ , averaged across 11 datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Cov. ( $\alpha=0.10$ ) Set size Acc. Avg. $2\sigma$ %Valid Mean %Sing. SO-LDA 81.0 90.8 2.1 80.0 2.3 70.3 Non-online 81.0 90.8 2.1 80.0 2.3 70.2 Ridge 78.8 90.7 2.3 76.0 2.4 64.6 Unstable 78.8 90.4 2.3 67.5 2.6 62.3	Cov. ( $\alpha=0.10$ ) Set size Acc. Avg. $2\sigma$ %Valid Mean %Sing. SO-LDA 76.4 88.2 4.2 27.0 3.1 61.6 Non-online 76.4 88.3 4.2 27.5 3.2 61.3 Ridge 74.1 88.0 4.3 26.0 2.9 58.4 Unstable 70.1 86.0 5.3 13.0 4.6 45.9
(a) $N=C\times 16$	(b) $N=C\times 4$

Appendix

Table 2: Study of the SO-LDA solver configurations under T-FCP. Average results for 10 datasets. In contrast to the main manuscript, these studies use

20

random seeds and exclude ImageNet.

Dataset	Classes	Splits			$b$ / $i$	Task description
		Train	Val	Test
EuroSAT [ 19 ]	10	13,500	5,400	8,100	$i$	Satellite image classification.
OxfordPets [ 38 ]	37	2,944	736	3,669	$i$	Pets classification.
DTD [ 7 ]	47	2,820	1,128	1,692	$b$	Textures classification.
FGVCAircraft [ 33 ]	100	3,334	3,333	3,333	$i$	Aircraft classification.
Caltech101 [ 12 ]	100	4,128	1,649	2,465	$i$	Natural objects classification.

Appendix

Table 3: Datasets overview. “

b

"/“

i

" denotes balanced/imbalanced.

			Cov. ( $\alpha=0.10$ )			Set size			Cov. ( $\alpha=0.05$ )			Set size
		Acc.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.
SCP	SO-LDA	79.5	90.8	3.5	71.3	2.7	2.3	69.4	95.6	2.4	77.0	4.1	3.3	58.4

Appendix

Table 4: Comparison of conformal procedures atop CLIP ViT-B/16 using

N=C\times 16

. This table reports the results using configurations not included in Table 1 .

				Cov. ( $\alpha=0.10$ )			Set size			Cov. ( $\alpha=0.05$ )			Set size
			Acc.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.
EuroSAT	ICP	ZS	48.4	90.4	6.1	62.0	4.2	4.4	9.4	95.6	4.8	66.0	5.2	5.3	6.6
	SCP	GD	83.4	89.8	6.5	50.0	1.4	1.1	73.4	96.1	4.7	76.0	2.2	1.8	43.8
	T-FCP	SO-LDA	83.0	90.2	5.3	58.0	1.4	1.0	71.4	95.9	3.9	90.0	2.0	1.9	42.0
Pets	ICP	ZS	88.8	89.9	3.2	54.0	1.0	1.0	92.9	95.1	2.0	68.0	1.2	1.0	80.4
	SCP	GD	92.3	90.0	4.9	65.0	1.0	1.0	94.8	95.2	3.3	66.0	1.1	1.0	91.0

Appendix

Table 5: Per-dataset conformal prediction results of average performances of the main conformal procedures reported in Table 1 , using CLIP ViT-B/16.

	Cov. ( $\alpha=0.10$ )				Set size			Cov. ( $\alpha=0.05$ )				Set size
	Avg.	$2\sigma$	%Valid	CCV	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	CCV	Mean	Med.	%Sing.
ICP	90.4	2.2	81.5	8.2	6.3	4.9	34.4	95.3	1.5	85.0	5.0	10.1	8.0	27.1
SCP	90.1	3.1	67.5	6.3	3.0	2.2	54.0	95.1	2.4	70.0	5.0	4.3	3.1	46.9
T-FCP	90.4	1.7	84.0	6.4	3.3	2.3	52.6	95.3	1.4	89.0	4.9	4.5	2.9	46.9

Appendix

Table 6: Results using APS. SO-LDA is used as solver, with

N=C\times 16

. Average results for 10 datasets. These studies use 20 random seeds and exclude ImageNet.

	Cov. ( $\alpha=0.05$ )			Set size
	Avg.	$2\sigma$	%Valid	Mean
ICP	95.2	1.3	80.0	12.3
SCP	95.0	1.2	70.0	11.7
T-FCP	95.1	1.2	80.0	10.6

Appendix

Table 7: Conformal prediction results on ImageNet-V2 using ResNet-50 pre-trained on ImageNet, LAC, and 8 labeled samples per class as calibration data. Results across 20 seeds.

Explore similar work

Jun 30, 2026cs.CV

Localized Conformal Prediction for Image Classification with Vision-Language Models

Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.

Clément Fuchs, Tim Bary, Benoît Macq

ICTEAM, UCLouvain Louvain-la-Neuve, Belgium

Online Conformal Prediction Image Classification

Ask PDF

May 24, 2026cs.LG

Cluster Frequency Conformal Prediction for Local Coverage

Conformal prediction provides distribution-free coverage guarantees, but in many-class classification it may still under-cover specific classes or subpopulations, preventing safe deployment in high-stakes applications. We propose Cluster Frequency Conformal Prediction (CFCP), a plug-in framework that adapts conformal prediction to local structure in a learned representation space. CFCP clusters learned embeddings, estimates cluster-level label-frequency distributions from calibration data, and for each test point constructs a sample-specific probability vector by softly mixing nearby cluster distributions regularized with global-prior and reliability-aware shrinkage. This vector is then conformalized using standard set constructors. In the disjoint-split regime, CFCP inherits standard finite-sample marginal validity. Under additional assumptions, CFCP further admits a local-validity interpretation. Since representation clusters aggregate locally similar samples, their empirical class frequencies provide a stable estimate of local label ambiguity. Across image and text benchmarks, CFCP achieves the best class coverage in 15/16 dataset/score-family comparisons and a competitive prediction set size efficiency, with several settings substantially more efficient. Overall, our results show that cluster-frequency information provides an effective localized signal for improving classwise reliability in many-class conformal prediction.

Tomer Lavi, Bracha Shapira, Nadav Rappoport

Institute of Applied AI Research (AAIR) Faculty of Computer and Information Science Ben-Gurion University of the Negev, Beer-Sheva 84105, ISRAEL · Institute for Interdisciplinary Computational Science (ICS) Faculty of Computer and Information Science Ben-Gurion University of the Negev, Beer-Sheva 84105, ISRAEL

Online Conformal Prediction Clustering

Ask PDF

May 22, 2026cs.LG

Empirical Bayes Conformal Prediction for Vision and Language Models

Conformal prediction (CP) gives distribution-free coverage for modern vision and language models, but it is often forced to make a ranking decision from a single unstable nonconformity score. Standard CP uses one realization, while average-then-calibrate variants smooth multiple realizations into a point estimate. Both options discard the inconsistency that can help identify whether a candidate is indeed stable. A weak answer can enter the conformal set even if the evidence is not strong, simply because one posterior sample or prompt phrasing made it look strong. But variability can help distinguish a stable signal from noise-driven fluctuations. We describe an empirical Bayes conformal prediction framework that uses

r

-values to convert score variability into an uncertainty informed nonconformity score. The resulting

r

-value estimates how likely a candidate's latent score belongs to the top-ranked group after accounting for both its mean score and its uncertainty. It admits both a closed-form Normal-Normal empirical Bayes estimator and a nonparametric posterior-sampling estimator. Using the

r

-value as the nonconformity score preserves the target conformal coverage while provably reducing the inclusion of high variance false candidates under mild regularity conditions. Across image classification, CLIP-based VLM benchmarks, and LLMs, we show that

r

-value conformal prediction preserves target coverage while improving ranking stability and reducing set size when variability is informative, and reverting to CP-like behavior when variability vanishes.

Jiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng +2

University of Wisconsin–Madison · University of California San Diego · Xiamen University

Nonconformity Scores Empirical Bayes Posterior Sampling

Ask PDF