cs.LGNov 6, 2026

Calibrating your confidence requires thinking of all the ways you could be wrong

Authors: Marco Miani

Organizations: DTU Compute, Department of Applied Mathematics and Computer Science

Abstract

A primary limitation of modern deep learning is the miscalibration of model confidence; traditional architectures fundamentally lack a mechanism to quantify their own uncertainty. Larger models and larger datasets mitigate this issue, but they do not resolve the underlying structural deficiency. Bayesian methods offer a principled uncertainty-aware learning framework through stochastic parameters. Yet, despite the established benefits, the inherent computational overhead often compromises either scalability or quality. This trade-off currently prevents the widespread adoption of Bayesian methods. This Thesis bridges the gap between high-quality and low-complexity by introducing a novel distribution on parameter space: a degenerate Gaussian whose covariance is a projection matrix aligned with the null space of the Fisher information matrix. We demonstrate the scalability of this Bayesian posterior through two numerically efficient algorithms: one for posterior sampling and another for the direct computation of predictive confidence. Empirically, we show State-Of-The-Art confidence calibration and out-of-distribution detection for large models. We provide theoretical grounding for this approach. We study the degeneracies inherent in Amari's information geometry and highlight its most relevant component: the split of parameter's perturbations into two orthogonal subspaces. From this perspective, our posterior naturally emerges as the simplest choice that captures this split, assigning unitary variance to one subspace and zero variance to the other. Moreover, through the tools of differential geometry, we utilize this split to identify and rectify a critical misalignment issue of the non-linearized Laplace Approximation. Finally, with a focus on PAC-Bayes generalization bounds, we pave the way for a deeper understanding of the learning capabilities of overparametrized models.

Figures & tables

Explore similar work

CardsList
  1. Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning

    Jun 15, 2026Tobias Jan Wieczorek, Leon de Andrade, Thomas Möllenhoff +1Bayesian Neural NetworksUncertainty Quantification

  2. Uncertainty Estimation and Generalization Bounds for Modern Deep Learning

    Jun 11, 2026Luis A. OrtegaBayesian Neural NetworksNeural Network Generalization

  3. Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

    Oct 8, 2026Yutong Xie, Jiawei Tang, Zhenglin Hua +6Model CalibrationUncertainty Calibration