Scaling Full Conformal Image Classifiers

Julio Silva-Rodríguez; Ender Konukoglu

cs.LGSep 29, 2026

Scaling Full Conformal Image Classifiers

Authors: Julio Silva-Rodríguez, Ender Konukoglu

Organizations: Computer Vision Lab, ETH Zürich

Abstract

Conformal prediction provides set-valued predictions with distribution-free coverage guarantees, making it attractive for high-stakes image classification. However, split conformal prediction is data-inefficient, while full conformal prediction (FCP), despite its stronger statistical efficiency, is computationally prohibitive at scale because it requires candidate-specific model refits at test time. We address this limitation by leveraging zero-shot vision-language models (VLMs) to guide scalable FCP in large label spaces. We introduce Targeted Full Conformal Prediction (T-FCP), which uses a lightweight inductive conformal predictor to prune unlikely labels and applies FCP only to the remaining candidates, reducing computation while retaining the formal guarantee of the combined conformal procedure. We further propose Stabilized Online LDA (SO-LDA), an efficient VLM adaptation solver based on rank-one inverse-covariance updates. Across multiple benchmarks, including ImageNet, T-FCP enables practical full-conformal image classification with modest test-time overhead, yielding efficient prediction sets and more stable empirical coverage than split conformal alternatives.

Figures & tables

			Cov. ( $\alpha=0.10$ )			Set size			Cov. ( $\alpha=0.05$ )			Set size
		Acc.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.
ICP	ZS	65.7	90.1	2.0	77.8	4.8	4.5	37.3	95.2	1.5	84.0	7.9	7.3	28.2
ICP-T	OT Silva-Rodríguez et al. (2025b)	68.6	90.6	2.0	82.3	3.9	3.7	42.4	95.4	1.4	86.8	6.1	5.6	32.6
ICP-T	TIM Silva-Rodríguez et al. (2025a)	71.5	90.6	2.0	80.5	3.6	3.5	45.2	95.4	1.4	87.2	5.6	5.3	34.3
SCP	GD	79.9	90.7	2.7	77.1	2.2	2.1	64.0	95.5	1.9	79.5	3.3	3.0	51.9
FCP	SO-LDA	80.3	90.1	2.2	77.6	2.2	1.8	69.3	95.2	1.6	80.5	3.3	2.6	57.4

Table 1: Comparison of conformal procedures atop CLIP ViT-B/16 using

N=C\times 16

. Results are averaged across 11 datasets, and per-dataset results are in Appendix F.2 .

Panel of Figure 1 — Figure 1: Analysis on empirical coverage and set size distribution on representative datasets. Results using CLIP ViT-B/16 as backbone, $N=C\times 16$ samples for calibration.

Figure 2: Computational efficiency analysis. Results using ViT-B/16, $N=C\times 16$ and $\alpha=0.05$ . All our experiments were conducted on a single NVIDIA A100-PCIE-40GB.

Panel of Figure 3 — Figure 3: Ablation studies on T-FCP. Results using CLIP ViT-B/16 as backbone, $N=C\times 16$ samples for calibration, and conformal sets created at $\alpha=0.1$ error rate.

Panel of Figure 4 — Figure 4: Studies at lower data regimes: (a-d) conformal procedures, and (e) few-shot solvers. Results using CLIP ViT-B/16 as backbone, and conformal sets at $\alpha=0.1$ , averaged for 11 datasets.

Panel of Figure 5 — Figure 5: Generalization atop other CLIP Radford et al. (2021) and MetaCLIP Xu et al. (2024) backbones of conformal procedures. Results using $N=C\times 16$ samples and $\alpha=0.1$ , averaged across 11 datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Cov. ( $\alpha=0.10$ ) Set size Acc. Avg. $2\sigma$ %Valid Mean %Sing. SO-LDA 81.0 90.8 2.1 80.0 2.3 70.3 Non-online 81.0 90.8 2.1 80.0 2.3 70.2 Ridge 78.8 90.7 2.3 76.0 2.4 64.6 Unstable 78.8 90.4 2.3 67.5 2.6 62.3	Cov. ( $\alpha=0.10$ ) Set size Acc. Avg. $2\sigma$ %Valid Mean %Sing. SO-LDA 76.4 88.2 4.2 27.0 3.1 61.6 Non-online 76.4 88.3 4.2 27.5 3.2 61.3 Ridge 74.1 88.0 4.3 26.0 2.9 58.4 Unstable 70.1 86.0 5.3 13.0 4.6 45.9
(a) $N=C\times 16$	(b) $N=C\times 4$

Appendix

Table 2: Study of the SO-LDA solver configurations under T-FCP. Average results for 10 datasets. In contrast to the main manuscript, these studies use

20

random seeds and exclude ImageNet.

Dataset	Classes	Splits			$b$ / $i$	Task description
		Train	Val	Test
EuroSAT [ 19 ]	10	13,500	5,400	8,100	$i$	Satellite image classification.
OxfordPets [ 38 ]	37	2,944	736	3,669	$i$	Pets classification.
DTD [ 7 ]	47	2,820	1,128	1,692	$b$	Textures classification.
FGVCAircraft [ 33 ]	100	3,334	3,333	3,333	$i$	Aircraft classification.
Caltech101 [ 12 ]	100	4,128	1,649	2,465	$i$	Natural objects classification.

Appendix

Table 3: Datasets overview. “

b

"/“

i

" denotes balanced/imbalanced.

			Cov. ( $\alpha=0.10$ )			Set size			Cov. ( $\alpha=0.05$ )			Set size
		Acc.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.
SCP	SO-LDA	79.5	90.8	3.5	71.3	2.7	2.3	69.4	95.6	2.4	77.0	4.1	3.3	58.4

Appendix

Table 4: Comparison of conformal procedures atop CLIP ViT-B/16 using

N=C\times 16

. This table reports the results using configurations not included in Table 1 .

				Cov. ( $\alpha=0.10$ )			Set size			Cov. ( $\alpha=0.05$ )			Set size
			Acc.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	Mean	Med.	%Sing.
EuroSAT	ICP	ZS	48.4	90.4	6.1	62.0	4.2	4.4	9.4	95.6	4.8	66.0	5.2	5.3	6.6
	SCP	GD	83.4	89.8	6.5	50.0	1.4	1.1	73.4	96.1	4.7	76.0	2.2	1.8	43.8
	T-FCP	SO-LDA	83.0	90.2	5.3	58.0	1.4	1.0	71.4	95.9	3.9	90.0	2.0	1.9	42.0
Pets	ICP	ZS	88.8	89.9	3.2	54.0	1.0	1.0	92.9	95.1	2.0	68.0	1.2	1.0	80.4
	SCP	GD	92.3	90.0	4.9	65.0	1.0	1.0	94.8	95.2	3.3	66.0	1.1	1.0	91.0

Appendix

Table 5: Per-dataset conformal prediction results of average performances of the main conformal procedures reported in Table 1 , using CLIP ViT-B/16.

	Cov. ( $\alpha=0.10$ )				Set size			Cov. ( $\alpha=0.05$ )				Set size
	Avg.	$2\sigma$	%Valid	CCV	Mean	Med.	%Sing.	Avg.	$2\sigma$	%Valid	CCV	Mean	Med.	%Sing.
ICP	90.4	2.2	81.5	8.2	6.3	4.9	34.4	95.3	1.5	85.0	5.0	10.1	8.0	27.1
SCP	90.1	3.1	67.5	6.3	3.0	2.2	54.0	95.1	2.4	70.0	5.0	4.3	3.1	46.9
T-FCP	90.4	1.7	84.0	6.4	3.3	2.3	52.6	95.3	1.4	89.0	4.9	4.5	2.9	46.9

Appendix

Table 6: Results using APS. SO-LDA is used as solver, with

N=C\times 16

. Average results for 10 datasets. These studies use 20 random seeds and exclude ImageNet.

	Cov. ( $\alpha=0.05$ )			Set size
	Avg.	$2\sigma$	%Valid	Mean
ICP	95.2	1.3	80.0	12.3
SCP	95.0	1.2	70.0	11.7
T-FCP	95.1	1.2	80.0	10.6

Appendix

Table 7: Conformal prediction results on ImageNet-V2 using ResNet-50 pre-trained on ImageNet, LAC, and 8 labeled samples per class as calibration data. Results across 20 seeds.

Explore similar work

Localized Conformal Prediction for Image Classification with Vision-Language Models
Jun 30, 2026Clément Fuchs, Tim Bary, Benoît MacqOnline Conformal Prediction Image Classification
ICTEAM, UCLouvain Louvain-la-Neuve, Belgium
Cluster Frequency Conformal Prediction for Local Coverage
May 24, 2026Tomer Lavi, Bracha Shapira, Nadav RappoportOnline Conformal Prediction Clustering
Institute of Applied AI Research (AAIR) Faculty of Computer and Information Science Ben-Gurion University of the Negev, Beer-Sheva 84105, ISRAEL · Institute for Interdisciplinary Computational Science (ICS) Faculty of Computer and Information Science Ben-Gurion University of the Negev, Beer-Sheva 84105, ISRAEL
Empirical Bayes Conformal Prediction for Vision and Language Models
May 22, 2026Jiapeng Zeng, Yogesh Prabhu, Zhanpeng Zeng +2Nonconformity Scores Empirical Bayes
University of Wisconsin–Madison · University of California San Diego · Xiamen University

Jun 30, 2026, cs.CVJ/K move · Enter open · S save

Localized Conformal Prediction for Image Classification with Vision-Language Models

Clément Fuchs, Tim Bary, Benoît Macq

ICTEAM, UCLouvain Louvain-la-Neuve, Belgium

Online Conformal Prediction Image Classification

Open paper AskPDF arXiv

Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.

Abstract

Explore similar work

Localized Conformal Prediction for Image Classification with Vision-Language Models

Cluster Frequency Conformal Prediction for Local Coverage

Empirical Bayes Conformal Prediction for Vision and Language Models