VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learning

Rahul D Ray; Utkarsh Srivastava

cs.LGApr 9, 2026

VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learning

Authors: Rahul D Ray, Utkarsh Srivastava

Organizations: Department of Electronics & Electrical Engineering BITS Pilani, Hyderabad Campus. · Department of Economics & Finance BITS Pilani, Hyderabad Campus

Abstract

Uncertainty quantification (UQ) is essential for deploying deep learning models in safety critical applications, yet no consensus exists on which UQ method performs best across different data modalities and distribution shifts. This paper presents a comprehensive benchmark of ten widely used UQ baselines including MC Dropout, SWAG, ensemble methods, temperature scaling, energy based OOD, Mahalanobis, hyperbolic classifiers, ENN, Taylor Sensus, and split conformal prediction against a simplified yet highly effective variant of VOLTA that retains only a deep encoder, learnable prototypes, cross entropy loss, and post hoc temperature scaling. We evaluate all methods on CIFAR 10 (in distribution), CIFAR 100, SVHN, uniform noise (out of distribution), CIFAR 10 C (corruptions), and Tiny ImageNet features (tabular). VOLTA achieves competitive or superior accuracy (up to 0.864 on CIFAR 10), significantly lower expected calibration error (0.010 vs. 0.044 to 0.102 for baselines), and strong OOD detection (AUROC 0.802). Statistical testing over three random seeds shows that VOLTA matches or outperforms most baselines, with ablation studies confirming the importance of adaptive temperature and deep encoders. Our results establish VOLTA as a lightweight, deterministic, and well calibrated alternative to more complex UQ approaches.

Figures & tables

Dataset	Role	Norm. (mean/std)	Splits (tr/val/te)
CIFAR-10	ID	(0.5,0.5,0.5)/(0.5,0.5,0.5)	40k/10k/10k
CIFAR-100	OOD (for C10); ID	(0.507,0.487,0.441)/(0.268,0.257,0.276)	45k/5k/10k
SVHN	OOD	same as CIFAR-10	–/–/26k
Uniform Noise	OOD	same as CIFAR-10	–/–/5k
CIFAR-10-C	corruption	same as CIFAR-10	–/–/10k per corr.
TinyImageNetFeat	ID+OOD	none	64k/16k/5k (ID); 5k OOD

Table 1: Summary of preprocessing steps for all datasets. Norm.: normalisation; OOD: out-of-distribution; ID: in-distribution. Important thing to note that SVHN, Uniform Noise and CIFAR-10-C do not have any training or validation split. SVHN was used exclusively as out - of - distribution (OOD) test data. No training or validation split, CIFAR-10-C was used as corrupted test set. No training or validation split..

Figure 1: VOLTA Architecture Overview. The model consists of a frozen feature extractor (ResNet-18 for vision or raw feature input for tabular data), followed by a deep encoder that maps inputs into a normalized latent space. Learnable class prototypes are embedded in the same space, and classification is performed via cosine similarity between features and prototypes. A learnable temperature parameter scales the logits, followed by post-hoc temperature calibration. The resulting framework enables efficient, deterministic uncertainty estimation without stochastic sampling.

Dataset	Role	Test	Normalisation (mean, std)	Notes
CIFAR-10	ID	10k	$(0.5,0.5,0.5)$ , $(0.5,0.5,0.5)$	Random crop, horizontal flip
CIFAR-100	OOD / ID	10k	$(0.507,0.487,0.441)$ , $(0.268,0.257,0.276)$	None
SVHN	OOD	26k	Same as CIFAR-10	–
Uniform Noise	OOD	10k	Same as CIFAR-10	Synthetic $\mathcal{U}[0,1]$
CIFAR-10-C	Corruption	10k/type	Same as CIFAR-10	19 corruptions (severity 1)
Tiny ImageNet Feat.	ID	5k (ID)	None	ID: 0–99; OOD: 100–199 (5k)

Table 2: Dataset summary. OOD: out-of-distribution, ID: in-distribution.

Figure 2: Accuracy vs. Expected Calibration Error (ECE) across different methods. Each point represents a model, where the x-axis denotes classification accuracy (%), and the y-axis denotes calibration error (lower is better).

Method	Acc $\uparrow$	ECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	FPR@95 $\downarrow$
MC Dropout	$0.872\pm 0.006$	$0.0185\pm 0.003$	$0.421\pm 0.012$	$0.201\pm 0.008$	$0.8419\pm 0.010$	$0.245$
SWAG	$0.884\pm 0.004$	$0.0935\pm 0.010$	$0.395\pm 0.009$	$0.189\pm 0.006$	$0.7982\pm 0.012$	$0.310$
Temperature Scaling	$0.870\pm 0.005$	$0.0268\pm 0.004$	$0.410\pm 0.010$	$0.198\pm 0.007$	$0.7765\pm 0.015$	$0.335$
Energy-based OOD	$0.868\pm 0.006$	$0.0412\pm 0.006$	$0.438\pm 0.014$	$0.210\pm 0.009$	$0.8015\pm 0.013$	$0.290$
Mahalanobis	$0.865\pm 0.007$	$0.0450\pm 0.008$	$0.452\pm 0.016$	$0.215\pm 0.010$	$0.8560\pm 0.009$	$0.210$
Hyperbolic	$0.871\pm 0.005$	$0.0920\pm 0.011$	$0.430\pm 0.013$	$0.205\pm 0.009$	$0.8611\pm 0.008$	$0.205$

Table 3: Comprehensive comparison of uncertainty quantification methods on CIFAR-10 (in-distribution) and CIFAR-100 (OOD). Results are reported as mean

\pm

standard deviation over three random seeds. Lower is better for ECE and NLL; higher is better for Accuracy and AUROC.

Figure 3: Performance density landscape showing the distribution of methods in the accuracy–ECE space. Contours represent the density of baseline methods, while the position of VOLTA highlights its placement in the optimal region of high accuracy and low calibration error.

Figure 4: Parallel Coordinate Visualization of Method Performance Across Accuracy, Calibration, and OOD Metrics.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 649 papers

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration
Oct 8, 2026Yutong Xie, Jiawei Tang, Zhenglin Hua +6Model Calibration Uncertainty Calibration
School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2
Uncertainty quantification for trustworthy deep learning: Methods and measures
Jul 30, 2026H. Martin Gillis, Thomas TrappenbergBayesian Neural Networks Uncertainty Quantification
Faculty of Computer Science, Dalhousie University, Halifax, Nova Scotia, Canada
VNDUQE: Information-Theoretic Novelty Detection using Deep Variational Information Bottleneck
May 12, 2026Aryan Gondkar, Hayder Radha, Yiming DengUncertainty Quantification Information Bottleneck
Nondestructive Evaluation Lab, Department of Electrical and Computer Engineering Michigan State University East Lansing, MI · Department of Electrical and Computer Engineering Michigan State University East Lansing, MI

Oct 8, 2026J/K move · Enter open · S save

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Yutong Xie, Jiawei Tang, Zhenglin Hua, Yuxiang Ma+5

School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China+2

Model Calibration Uncertainty Calibration Language Model Calibration

Open paper AskPDF arXiv

Deep neural networks, including large language models, have achieved remarkable performance across various tasks. However, they are prone to overconfidence during training or fine-tuning. In this work, we observe a consistent phenomenon across different models that the early model is better calibrated, while later training or fine-tuning yields marginal accuracy gains but substantially increases calibration errors. Our analysis suggests that the early model retains uncertainty awareness in both its predictions and features, which is gradually lost with continued training. To avoid training away this uncertainty awareness, we propose EUA-Cal, a novel method that exploits the Early model as an Uncertainty Anchor for Calibration. EUA-Cal introduces early prediction regularization to preserve early predictive uncertainty and prototype structure regularization to exploit uncertainty reflected in the early feature space, jointly mitigating overconfidence. Extensive experiments on image classification and multiple-choice question answering across eight diverse models demonstrate that EUA-Cal outperforms state-of-the-art calibration methods.

Abstract

Uncertainty Quantification

Explore similar work

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Uncertainty quantification for trustworthy deep learning: Methods and measures

VNDUQE: Information-Theoretic Novelty Detection using Deep Variational Information Bottleneck