Efficient Credal Prediction through Decalibration

Paul Hofman; Timo Löhr; Maximilian Muschalik; Yusuf Sale; Eyke Hüllermeier

cs.LGMar 9, 2026

Efficient Credal Prediction through Decalibration

Authors: Paul Hofman, Timo Löhr, Maximilian Muschalik, Yusuf Sale, Eyke Hüllermeier

Organizations: LMU Munich · Munich Center for Machine Learning (MCML) · DFKI (DSA)

Abstract

A reliable representation of uncertainty is essential for the application of modern machine learning methods in safety-critical settings. In this regard, the use of credal sets (i.e., convex sets of probability distributions) has recently been proposed as a suitable approach to representing epistemic uncertainty. However, as with other approaches to epistemic uncertainty, training credal predictors is computationally complex and usually involves (re-)training an ensemble of models. The resulting computational complexity prevents their adoption for complex models such as foundation models and multi-modal systems. To address this problem, we propose an efficient method for credal prediction that is grounded in the notion of relative likelihood and inspired by techniques for the calibration of probabilistic classifiers. For each class label, our method predicts a range of plausible probabilities in the form of an interval. To produce the lower and upper bounds of these intervals, we propose a technique that we refer to as decalibration. Extensive experiments show that our method yields credal sets with strong performance across diverse tasks, including coverage-efficiency evaluation, out-of-distribution detection, and in-context learning. Notably, we demonstrate credal prediction on models such as TabPFN and CLIP -- architectures for which the construction of credal sets was previously infeasible.

Figures & tables

Figure 1: Overview of Efficient Credal Prediction through Decalibration. Given a probabilistic classifier ( maximum likelihood estimate ), our method decalibrates the predicted distributions by their logits. The resulting credal set contains the ground-truth distribution , as visualized in the credal spider plot (see Appendix C for an explanation). Note that we only show the decalibration of three classes for visualization purposes—in practice, all classes are decalibrated.

Figure 4: EffCre used with TabPFN . Top: Coverage versus efficiency performance all multi-class tabarena datasets. Bottom: Active In-Context Learning performance versus the random baseline.

Figure 5: Credal Prediction with CLIP . Examples from cifar-10h with high epistemic uncertainty ( left ) and high aleatoric uncertainty ( right ) as predicted by applying our method on CLIP .

Figure 6: EffCre used with CLIP -based models. We demonstrate the performance of EffCre by applying it to CLIP , SigLIP , and SigLIP-2 without retraining—something that can’t be done by the baselines.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Example credal spider plots for CLIP , SigLIP , and SigLIP-2 . used for illustration purposes. The credal spider plots includes the maximum likelihood estimate , the plausible intervals , and the ground-truth distribution.

Hyperparameter	ChaosNLI	CIFAR-10	QualityMRI
Model	FCNet	ResNet18	ResNet18
Epochs	300	200	200
Learning rate	0.01	0.1	0.01
Weight decay	0.0	0.0005	0.0005
Optimizer	Adam	SGD	SGD
Ensemble members	20	20	20

Appendix

Table 1: Hyperparameters used for each dataset.

Component	Specification
CPU	AMD EPYC MILAN 7413 Processor, 24C/48T 2.65GHz 128MB L3 Cache
GPU	2 × NVIDIA A40 (48 GB GDDR each)
RAM	128 GB (4x 32GB) DDR4-3200MHz ECC DIMM
Storage	2 × 480GB Samsung Datacenter SSD PM893

Appendix

Table 2: Specifications of Computing Resources

Model	CIFAR-10				ImageNet	DermaMNIST
	EN	SW	FR	ZH
CLIP	88.97%	9.11%	85.97%	33.73%	57.14%	24.74%
SigLIP	92.17%	15.33%	84.05%	91.28%	72.83%	8.28%
SigLIP2	93.91%	10.21%	92.21%	93.85%	69.87%	11.67%
BiomedCLIP	–	–	–	–	–	45.89%

Appendix

Table 3: Zero-shot classification accuracy (%) of CLIP -based models on CIFAR-10 (EN = English, SW = Swahili, FR = French, ZH = Chinese), ImageNet, and DermaMNIST.

Panel of Figure 8 — Figure 8: Coverage versus Efficiency. Our method, EffCre, is compared to baselines on qualitymri .

Method	Training time	Inference time
EffCre	$2136.33\pm 1.70$	$1.50\pm 0.02$
CreRL	$12675.84\pm 412.68$	$11.46\pm 0.12$
CreWra	$21363.34\pm 33.99$	$11.44\pm 0.19$
CreEns	$21363.34\pm 33.99$	$11.44\pm 0.23$
CreNet	$24996.65\pm 180.12$	$11.41\pm 0.18$
CreBNN	$29796.67\pm 12.94$	$12.74\pm 1.1$

Appendix

Table 4: Training and inference time in seconds for models trained on CIFAR10. Mean with standard deviation over three runs. Computed based on ensembles with 10 members.

Method	SVHN	Places365	CIFAR-100	FMNIST	ImageNet
$\text{EffCre}_{0.0}$	$0.478\scriptstyle{\pm 0.006}$	$0.478\scriptstyle{\pm 0.005}$	$0.480\scriptstyle{\pm 0.005}$	$0.486\scriptstyle{\pm 0.002}$	$0.481\scriptstyle{\pm 0.002}$
$\text{EffCre}_{0.2}$	$0.474\scriptstyle{\pm 0.003}$	$0.474\scriptstyle{\pm 0.001}$	$0.473\scriptstyle{\pm 0.002}$	$0.474\scriptstyle{\pm 0.002}$	$0.473\scriptstyle{\pm 0.003}$
$\text{EffCre}_{0.4}$	$0.303\scriptstyle{\pm 0.100}$	$0.389\scriptstyle{\pm 0.072}$	$0.335\scriptstyle{\pm 0.040}$	$0.325\scriptstyle{\pm 0.057}$	$0.338\scriptstyle{\pm 0.035}$
$\text{EffCre}_{0.6}$	$0.415\scriptstyle{\pm 0.010}$	$0.428\scriptstyle{\pm 0.007}$	$0.440\scriptstyle{\pm 0.017}$	$0.404\scriptstyle{\pm 0.024}$	$0.435\scriptstyle{\pm 0.008}$
$\text{EffCre}_{0.8}$	$0.744\scriptstyle{\pm 0.009}$	$0.721\scriptstyle{\pm 0.009}$	$0.720\scriptstyle{\pm 0.007}$	$0.733\scriptstyle{\pm 0.012}$	$0.700\scriptstyle{\pm 0.008}$
$\text{EffCre}_{0.9}$	$0.854\scriptstyle{\pm 0.005}$	$0.827\scriptstyle{\pm 0.006}$	$0.822\scriptstyle{\pm 0.004}$	$0.860\scriptstyle{\pm 0.005}$	$0.796\scriptstyle{\pm 0.005}$

Appendix

Table 5: Out-of-Distribution Detection.

Method	SVHN	Places365	CIFAR-100	FMNIST	ImageNet
EDL	$0.938\scriptstyle{\pm 0.010}$	$0.889\scriptstyle{\pm 0.001}$	$0.887\scriptstyle{\pm 0.001}$	$0.940\scriptstyle{\pm 0.005}$	$0.866\scriptstyle{\pm 0.001}$
DDU	$0.973\scriptstyle{\pm 0.001}$	$0.969\scriptstyle{\pm 0.001}$	$0.873\scriptstyle{\pm 0.002}$	$0.892\scriptstyle{\pm 0.010}$	$0.969\scriptstyle{\pm 0.000}$

Appendix

Table 6: Out-of-Distribution Detection.

Panel of Figure 9 — Figure 9: Active In-Context Learning with TabPFN . Performance on TabArena datasets 46963, and 46930 versus the random baseline.

Panel of Figure 10 — Figure 10: Credal spider plots for an image of a bird with CLIP , SigLIP , and SigLIP-2 across different languages. In English, all models confidently predict the image as bird . In French, SigLIP-2 maintains the correct maximum likelihood prediction but shows increased uncertainty toward cat . In Chinese, CLIP exhibits high uncertainty across all classes, indicating difficulties in this language, whereas SigLIP and SigLIP-2 remain as confident as in English. In Swahili, all models struggle and display high uncertainty across all classes; notably, bird and airplane share the same word in Swahili, complicating the prediction. These examples align well with models’ performances across the different languages (see Table 3 ).

Panel of Figure 11 — Figure 11: Comparison of credal sets for CLIP , SigLIP , and SigLIP-2 on observations with high uncertainty with CLIP . Observation (a) shows a swimming deer, where the MLE is ship . High uncertainty is spread across ship , two sky-related classes ( airplane , bird ), the amphibious frog , and the correct class deer . Both SigLIP models exhibit similar patterns with even greater uncertainty. Observation (b) depicts a dark image of a frog misclassified as ship , with high uncertainty again on that class; both SigLIP models additionally assign probability to cat . Observation (c) is a challenging deer image, where annotators themselves showed high disagreement. CLIP is confident it is either deer or horse , while SigLIP favors bird or frog , and SigLIP-2 remains certain it is an animal but not which. Observation (d) illustrates a case where human annotators are nearly evenly split between cat and dog , and the uncertainties of all three models capture this ambiguity.

Panel of Figure 12 — Figure 12: Comparison of credal sets for BiomedCLIP and CLIP on dermamnist for a melanocytic nevi (a) and a melanoma (b) . While BiomedCLIP demonstrates higher overall performance than CLIP (see Table 3 ), it misclassifies the melanoma with high confidence and low uncertainty, which could be dangerous if applied in medical contexts. Interestingly, CLIP classifies the melanoma correctly, albeit with greater uncertainty. Both models predict the melanocytic nevi correctly, though BiomedCLIP shows increased uncertainty toward the related melanoma class, suggesting a more challenging case.

Method	Members	SVHN	Places365	CIFAR-100	FMNIST	ImageNet
$\text{EffCre}_{0.95}$	1	$0.885\scriptstyle{\pm 0.003}$	$0.862\scriptstyle{\pm 0.005}$	$0.854\scriptstyle{\pm 0.003}$	$0.907\scriptstyle{\pm 0.002}$	$0.826\scriptstyle{\pm 0.004}$
$\text{CreRL}_{0.95}$	5	$0.917\scriptstyle{\pm 0.012}$	$0.894\scriptstyle{\pm 0.002}$	$0.885\scriptstyle{\pm 0.002}$	$0.928\scriptstyle{\pm 0.004}$	$0.863\scriptstyle{\pm 0.002}$
CreWra	5	$0.943\scriptstyle{\pm 0.006}$	$0.904\scriptstyle{\pm 0.001}$	$0.905\scriptstyle{\pm 0.001}$	$0.939\scriptstyle{\pm 0.001}$	$0.879\scriptstyle{\pm 0.001}$
$\text{CreEns}_{0.0}$	5	$0.938\scriptstyle{\pm 0.007}$	$0.898\scriptstyle{\pm 0.001}$	$0.900\scriptstyle{\pm 0.001}$	$0.929\scriptstyle{\pm 0.001}$	$0.874\scriptstyle{\pm 0.001}$
CreBNN	5	$0.843\scriptstyle{\pm 0.006}$	$0.829\scriptstyle{\pm 0.006}$	$0.831\scriptstyle{\pm 0.007}$	$0.851\scriptstyle{\pm 0.007}$	$0.809\scriptstyle{\pm 0.007}$
CreNet	5	$0.938\scriptstyle{\pm 0.003}$	$0.908\scriptstyle{\pm 0.001}$	$0.900\scriptstyle{\pm 0.001}$	$0.941\scriptstyle{\pm 0.003}$	$0.871\scriptstyle{\pm 0.002}$

Appendix

Table 7: Ablation of different numbers of trained ensemble members for Out-of-Distribution Detection.

Panel of Figure 13 — Figure 13: Out-of-distribution detection performance (based on AUROC score) as a function of ensemble size. CIFAR-10 is the in-distribution data while various datasets are used as OOD data.

Panel of Figure 14 — Figure 14: Active In-Context Learning with TabPFN . Performance on TabArena datasets 46941, 46963, 46930 for different values of $\alpha$ .

Method	Model	CIFAR10		ChaosNLI		QualityMRI
		ECE	Acc	ECE	Acc	ECE	Acc
EffCre	1	$0.04\pm 0.00$	$0.93\pm 0.00$	$0.04\pm 0.01$	$0.61\pm 0.01$	$0.30\pm 0.05$	$0.49\pm 0.05$
CreRL 0.8	1	$0.04\pm 0.00$	$0.94\pm 0.00$	$0.09\pm 0.03$	$0.60\pm 0.02$	$0.33\pm 0.09$	$0.48\pm 0.04$
	2	$0.06\pm 0.01$	$0.87\pm 0.01$	$0.13\pm 0.01$	$0.57\pm 0.01$	$0.35\pm 0.08$	$0.50\pm 0.05$
	3	$0.06\pm 0.00$	$0.87\pm 0.00$	$0.10\pm 0.01$	$0.56\pm 0.01$	$0.37\pm 0.10$	$0.51\pm 0.02$
	4	$0.06\pm 0.00$	$0.88\pm 0.00$	$0.09\pm 0.01$	$0.58\pm 0.01$	$0.30\pm 0.09$	$0.49\pm 0.03$

Appendix

Table 8: Comparison of ECE and accuracy of single models per method across datasets.

Epistemic Uncertainty

Epistemic uncertainty is uncertainty arising from limited knowledge or model support, rather than irreducible randomness in data.

See all 113 papers

Mar 28, 2026

Quantification of Credal Uncertainty: A Distance-Based Approach

Credal sets, i.e., closed convex sets of probability measures, provide a natural framework to represent aleatoric and epistemic uncertainty in machine learning. Yet how to quantify these two types of uncertainty for a given credal set, particularly in multiclass classification, remains underexplored. In this paper, we propose a distance-based approach to quantify total, aleatoric, and epistemic uncertainty for credal sets. Concretely, we introduce a family of such measures within the framework of Integral Probability Metrics (IPMs). The resulting quantities admit clear semantic interpretations, satisfy natural theoretical desiderata, and remain computationally tractable for common choices of IPMs. We instantiate the framework with the total variation distance and obtain simple, efficient uncertainty measures for multiclass classification. In the binary case, this choice recovers established uncertainty measures, for which a principled multiclass generalization has so far been missing. Empirical results confirm practical usefulness, with favorable performance at low computational cost.

Xabier Gonzalez-Garcia, Siu Lun Chau, Julian Rodemann +6

Dept. of Statistics, CS & Mathematics, Public Univ. of Navarre · EPIC Lab, Nanyang Technological Univ. Singapore · Rational Intelligence Lab, CISPA Helmholtz Center for Information Security +7

Uncertainty Quantification Aleatoric Uncertainty Integral Probability Metrics Uncertainty Decomposition Epistemic Uncertainty

Ask PDF

May 25, 2025

Optimal Conformal Prediction under Epistemic Uncertainty

Conformal prediction (CP) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees. In practice, CP is typically applied on top of probabilistic classifiers, which are able to express aleatoric but not epistemic uncertainty. In this paper, we consider the question of how to optimally employ CP on top of a more expressive formalism, namely credal sets, which can express both aleatoric and epistemic uncertainty. More specifically, we propose probabilistic Bernoulli prediction sets (BPS) and derive a variant that achieves conditional coverage for valid credal sets while remaining minimal in expected size. We then address the more realistic scenario in which the validity of the credal sets is not guaranteed. Assuming access to calibration data with ground-truth distributions over labels, we apply conformal risk control to BPS and derive a PAC-style guarantee: with high probability over the data, the achieved conditional coverage is at least the desired level. We validate our theoretical findings empirically over various datasets.

Alireza Javanmardi, Soroush H. Zargarbashi, Santo M. A. R. Thies +3

1LMU Munich · 2MCML · 3CISPA +3

Conformal Prediction Conformal Risk Control Epistemic Uncertainty

Ask PDF

Oct 8, 2026

Credal Machine Learning for Risk-Averse Decision Making

In many machine learning applications, it is necessary to guard against worst-case scenarios and predictions that could result in substantial losses. In principle, this can be achieved by training risk-averse predictive models that minimize loss functions such as conditional value-at-risk (CVaR), rather than relying on models that perform well on average. In practice, however, the effectiveness of this approach to risk aversion is undermined by the learner's uncertainty regarding the true loss distribution and, consequently, the true CVaR. To achieve reliable risk-aversion, we propose a method in which this (epistemic) uncertainty is represented in terms of credal sets, i.e., sets of probability distributions. More specifically, we develop an efficient yet reliable learner that produces predictions in the form of credal sets and combine it with a novel decision rule that maps each credal set to a single predictive distribution for CVaR minimization. Across classification, under distribution shift, and in reinforcement learning, our approach reliably avoids catastrophic decisions, while sacrificing little in expected performance.

Timo Löhr, Paul Hofman, Maximilian Muschalik +1

LMU Munich, MCML · LMU Munich, MCML, DFKI

Decision-Making under Uncertainty Conditional Value-at-Risk Risk-Sensitive RL

Ask PDF

Abstract

Epistemic Uncertainty

Explore similar work

Quantification of Credal Uncertainty: A Distance-Based Approach

Optimal Conformal Prediction under Epistemic Uncertainty

Credal Machine Learning for Risk-Averse Decision Making