Activation-Space Uncertainty Quantification for Pretrained Networks

Richard Bergna; Stefan Depeweg; Sergio Calvo-Ordoñez; Jonathan Plenk; Alvaro Cartea; Jose Miguel Hernández-Lobato

stat.MLFeb 16, 2026

Activation-Space Uncertainty Quantification for Pretrained Networks

Authors: Richard Bergna, Stefan Depeweg, Sergio Calvo-Ordoñez, Jonathan Plenk, Alvaro Cartea, Jose Miguel Hernández-Lobato

Organizations: Department of Engineering, University of Cambridge, Cambridge, UK · Siemens AG, Munich, Germany · Mathematical Institute and Oxford-Man Institute, University of Oxford, Oxford, UK

Abstract

Reliable uncertainty estimates are crucial for deploying pretrained models; yet, many strong methods for quantifying uncertainty require retraining, Monte Carlo sampling, or expensive second-order computations and may alter a frozen backbone's predictions. To address this, we introduce Gaussian Process Activations (GAPA), a post-hoc method that shifts Bayesian modeling from weights to activations. GAPA replaces standard nonlinearities with Gaussian-process activations whose posterior mean exactly matches the original activation, preserving the backbone's point predictions by construction while providing closed-form epistemic variances in activation space. To scale to modern architectures, we use a sparse variational inducing-point approximation over cached training activations, combined with local k-nearest-neighbor subset conditioning, enabling deterministic single-pass uncertainty propagation without sampling, backpropagation, or second-order information. Across regression, classification, image segmentation, and language modeling, GAPA matches or outperforms strong post-hoc baselines in calibration and out-of-distribution detection while remaining efficient at test time.

Figures & tables

Method	Post-hoc	Single Pass	Preserves Mean	Epistemic UQ	Foundation Ready
BNNs	✗	✗	✗	✓	✗
Ensembles	✗	✗	—	✓	✗
MC Dropout	✗	✗	✗	✓	✗
Laplace	✓	✗	✗	✓	✗
LL-Laplace	✓	✓	✗	✓	✗
Temp. Scaling	✓	✓	✗	✗	✓

Table 1: The gap in uncertainty quantification methods.

Figure 1: Comparison of uncertainty quantification methods on a toy binary classification task. Left to right : MAP (deterministic backbone), MC Dropout, Last-Layer Laplace, and GAPA (ours). Background shading indicates predictive confidence (darker = more confident); orange/yellow points show the two classes. Key observation : GAPA preserves the backbone’s decision boundary (black line) exactly while adding epistemic uncertainty that grows smoothly away from training data.

Figure 2: GAPA overview. Top: GAPA leaves the network’s point predictions unchanged (mean-preserving activations) while propagating an additional epistemic variance signal to the output. Bottom left: deterministic

\tanh

activation; orange points denote cached training activations. Bottom right: GAPA-

\tanh

, whose posterior mean matches

\tanh

exactly; the shaded region shows

\pm 2

standard deviations.

Operation	Exact GP	GAPA (Ours)
Preprocessing	$\mathcal{O}(N_{\ell}^{3})$	$\mathcal{O}(N_{\ell}I_{\text{kmeans}})+\mathcal{O}(M_{\ell}\log M_{\ell})$
Inference (per query)	$\mathcal{O}(N_{\ell}^{2})$	$\mathcal{O}(\log(M_{\ell}))+\mathcal{O}(K^{3})$
Memory	$\mathcal{O}(N_{\ell}^{2})$	$\mathcal{O}(M_{\ell})$

Table 2: Computational complexity per layer

\ell

. Exact GP refers to conditioning on all

N_{\ell}

cached activations.

I_{\text{kmeans}}

denotes the number of

k

-means iterations.

Model	Airline			Year			Taxi
Model	NLL	CRPS	CQM	NLL	CRPS	CQM	NLL	CRPS	CQM
MAP	5.121	18.695	0.148	3.673	5.023	0.134	3.775	3.755	0.211
LLA Diag	5.125	18.648	0.143	3.647	4.917	0.088	3.722	3.990	0.257
LLA KFAC	5.127	18.631	0.142	3.648	4.915	0.086	3.706	3.986	0.256
LLA*	5.127	18.631	0.141	3.648	4.915	0.086	3.726	3.985	0.256
LLA*KFAC	5.127	18.631	0.141	3.648	4.914	0.086	3.726	3.985	0.256

Table 3: Results on regression datasets. Best values are in purple , and second-best in teal . An asterisk (*) indicates a last-layer LLA variant. Results are averages over 5 random seeds; standard deviations (

<10^{-3}

in all cases) are omitted for brevity. The full table with stds can be found in Table 7 in the Appendix.

Figure 3: Predictive NLL under rotation corruption for MNIST (left panel) and FMNIST (right panel); lower is better. Results are averaged over 5 random seeds.

Figure 4: OOD detection vs inference cost on CIFAR-10. OOD AUROC is plotted against test-time inference cost (log scale) for ResNet backbones. Dashed lines indicate Pareto frontiers (higher OOD, lower cost). GAPA consistently lies on the frontier, achieving strong OOD performance at substantially lower inference cost than baselines.

Panel of Figure 5 — Figure 5: Left: Effect of number of inducing points $N_{\text{inducing}}$ and $k$ (for nearest neighbor inducing points) on OOD detection task with GAPA at layer [27]. Right: effect of layer placement of GAPA at $N_{\text{inducing}}=10^{5}$ . In both experiments results are averaged over $5$ runs with $512$ sequences each. In both panels we also show the $\ell/T_{\mathrm{opt}}$ bound (green) as an upper threshold of what can be achieved by global logits scaling.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative segmentation results with pixel-wise error and epistemic uncertainty. Columns : (1) Input image $X$ , (2) Ground-truth mask $Y$ , (3) Predicted mask $\hat{Y}$ , (4) Error map $\lvert Y-\hat{Y}\rvert$ , (5) Epistemic uncertainty (mutual information). Rows : three representative validation examples.

Figure 7: Regression predictions and uncertainty: (a) MAP backbone, (b) MC Dropout, (c) last-layer Laplace, (d) GAPA (ours).

Figure 8: Effect of the number of inducing points $N_{\text{inducing}}$ and preprocessing strategy on OOD detection task. We plot the AUC using EU (blue) with GAPA at layer [27]. Results are averaged over $5$ runs with $512$ sequences each. In both panels we also show the $\ell/T_{\mathrm{opt}}$ bound (green) as an upper threshold of what can be achieved by global logits scaling.

Model	Airline			Year			Taxi
Model	NLL	CRPS	CQM	NLL	CRPS	CQM	NLL	CRPS	CQM
MAP (backbone)	5.121 ( $\pm$ 0.5)	18.695 ( $\pm$ 0.6)	0.148 ( $\pm$ 0.4)	3.673 ( $\pm$ 0.4)	5.023 ( $\pm$ 0.5)	0.134 ( $\pm$ 0.3)	3.775 ( $\pm$ 0.5)	3.755 ( $\pm$ 0.4)	0.211 ( $\pm$ 0.4)
LLA Diag	5.125 ( $\pm$ 0.4)	18.648 ( $\pm$ 0.5)	0.143 ( $\pm$ 0.3)	3.647 ( $\pm$ 0.3)	4.917 ( $\pm$ 0.4)	0.088 ( $\pm$ 0.2)	3.722 ( $\pm$ 0.4)	3.990 ( $\pm$ 0.5)	0.257 ( $\pm$ 0.3)
LLA KFAC	5.127 ( $\pm$ 0.3)	18.631 ( $\pm$ 0.4)	0.142 ( $\pm$ 0.3)	3.648 ( $\pm$ 0.3)	4.915 ( $\pm$ 0.4)	0.086 ( $\pm$ 0.2)	3.706 ( $\pm$ 0.3)	3.986 ( $\pm$ 0.4)	0.256 ( $\pm$ 0.3)
LLA*	5.127 ( $\pm$ 0.4)	18.631 ( $\pm$ 0.5)	0.141 ( $\pm$ 0.3)	3.648 ( $\pm$ 0.3)	4.915 ( $\pm$ 0.4)	0.086 ( $\pm$ 0.2)	3.726 ( $\pm$ 0.4)	3.985 ( $\pm$ 0.5)	0.256 ( $\pm$ 0.3)
LLA* KFAC	5.127 ( $\pm$ 0.3)	18.631 ( $\pm$ 0.4)	0.141 ( $\pm$ 0.3)	3.648 ( $\pm$ 0.3)	4.914 ( $\pm$ 0.4)	0.086 ( $\pm$ 0.2)	3.726 ( $\pm$ 0.4)	3.985 ( $\pm$ 0.4)	0.256 ( $\pm$ 0.3)

Appendix

Table 7: Results on regression datasets with standard deviations (in

\times 10^{-3}

units). Best values are in purple , and second-best in teal . An asterisk (*) indicates a last-layer LLA variant. Results are averages over 5 random seeds. This is the full version of Table 3 with stds included.

Model	MNIST					FMNIST
Model	ACC	NLL	ECE	OOD	BALD	ACC	NLL	ECE	OOD	BALD
MAP	0.978 ( $\pm$ 0.4)	0.068 ( $\pm$ 0.2)	0.005 ( $\pm$ 0.3)	0.919 ( $\pm$ 0.5)	0.919 ( $\pm$ 0.4)	0.859 ( $\pm$ 0.3)	0.392 ( $\pm$ 0.6)	0.007 ( $\pm$ 0.3)	0.846 ( $\pm$ 0.5)	0.821 ( $\pm$ 0.5)
LLA Diag	0.976 ( $\pm$ 0.5)	0.177 ( $\pm$ 0.5)	0.105 ( $\pm$ 0.6)	0.932 ( $\pm$ 0.6)	0.941 ( $\pm$ 0.5)	0.856 ( $\pm$ 0.4)	0.421 ( $\pm$ 0.5)	0.057 ( $\pm$ 0.4)	0.872 ( $\pm$ 0.5)	0.873 ( $\pm$ 0.6)
LLA KFAC	0.978 ( $\pm$ 0.4)	0.102 ( $\pm$ 0.4)	0.042 ( $\pm$ 0.4)	0.971 ( $\pm$ 0.3)	0.971 ( $\pm$ 0.4)	0.858 ( $\pm$ 0.4)	0.395 ( $\pm$ 0.5)	0.020 ( $\pm$ 0.3)	0.909 ( $\pm$ 0.4)	0.970 ( $\pm$ 0.5)
LLA*	0.978 ( $\pm$ 0.4)	0.070 ( $\pm$ 0.3)	0.009 ( $\pm$ 0.3)	0.924 ( $\pm$ 0.5)	0.924 ( $\pm$ 0.5)	0.859 ( $\pm$ 0.4)	0.395 ( $\pm$ 0.5)	0.019 ( $\pm$ 0.3)	0.850 ( $\pm$ 0.5)	0.716 ( $\pm$ 0.5)
LLA* KFAC	0.979 ( $\pm$ 0.3)	0.070 ( $\pm$ 0.3)	0.009 ( $\pm$ 0.2)	0.923 ( $\pm$ 0.4)	0.928 ( $\pm$ 0.5)	0.859 ( $\pm$ 0.4)	0.394 ( $\pm$ 0.5)	0.017 ( $\pm$ 0.3)	0.849 ( $\pm$ 0.4)	0.717 ( $\pm$ 0.6)

Appendix

Table 8: Results on classification datasets with standard deviations (in

\times 10^{-3}

units). Best values are in purple , second-best in teal . Values are averages over 5 random seeds; consistent with

<10^{-3}

in all cases.

Figure 9: Comparison of metrics at different GAPA layer placements (

M=55{,}000

)

GAPA layers	MNIST			FMNIST
GAPA layers	NLL $\downarrow$	OOD-AUC $\uparrow$	OOD BALD $\uparrow$	NLL $\downarrow$	OOD-AUC $\uparrow$	OOD BALD $\uparrow$
[1]	0.072	0.915	0.916	0.326	0.870	0.900
[2]	0.070	0.921	0.920	0.321	0.884	0.907
[3]	0.068	0.933	0.933	0.309	0.901	0.911
[4]	0.117	0.951	0.957	0.353	0.973	0.969
[1, 2]	0.069	0.923	0.922	0.318	0.901	0.921

Appendix

Table 11: Comparison of metrics at different GAPA layer placements (

M=55{,}000

). Best values are bold . Lower is better (↓) for NLL; higher is better (↑) for OOD-AUC/BALD.

Figure 10: FMNIST: NLL (left) and OOD-AUC (right) for KMeans vs. FPS across

M

Figure 11: Setup time (FAISS indexing) for KMeans vs. FPS on FMNIST (left) and MNIST (right).

Figure 12: MNIST: NLL (left) and OOD-AUC (right) for KMeans vs. FPS across

M

Figure 13: BALD-based OOD detection for MNIST (left) and FMNIST (right).

$M$	FPS NLL↓	FPS OOD↑	Rand1 NLL↓	Rand1 OOD↑	Rand2 NLL↓	Rand2 OOD↑	Rand3 NLL↓	Rand3 OOD↑
5000	0.470	0.943	0.394	0.957	0.394	0.957	0.394	0.957
10000	0.442	0.952	0.380	0.960	0.380	0.960	0.380	0.960
20000	0.390	0.964	0.369	0.964	0.369	0.964	0.369	0.964
40000	0.355	0.968	0.359	0.967	0.359	0.967	0.359	0.967

Appendix

Table 13: Comparison of NLL and OOD BALD for FPS and three random baselines (FMNIST, gapa_index= last layer).

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 649 papers

Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning
Jun 15, 2026Tobias Jan Wieczorek, Leon de Andrade, Thomas Möllenhoff +1Bayesian Neural Networks Uncertainty Quantification
TU Darmstadt & hessian.AI, Darmstadt, Germany · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
Gaussian Neural Networks
Sep 28, 2026Peter Kuhn, Victoria Heusinger-Heß
Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut, EMI, Germany
Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration
Oct 8, 2026Yutong Xie, Jiawei Tang, Zhenglin Hua +6Model Calibration Uncertainty Calibration
School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2

Jun 15, 2026J/K move · Enter open · S save

Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning

Tobias Jan Wieczorek, Leon de Andrade, Thomas Möllenhoff, Marcus Rohrbach

TU Darmstadt & hessian.AI, Darmstadt, Germany · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan

Bayesian Neural Networks Uncertainty Quantification Uncertainty Propagation Uncertainty Calibration

Open paper AskPDF arXiv

Modern deep learning models remain notoriously prone to overconfidence, limiting their reliability in high-stakes applications. Bayesian methods aim to counter this by learning a distribution over model parameters, and recent advances now make this feasible for large-scale architectures at costs comparable to AdamW. However, a challenge remains at test time: predictions must be averaged across many forward passes with weights sampled from the posterior, which is prohibitively expensive. Variance propagation offers an efficient alternative, computing layer-wise analytical approximations of uncertainty in a single forward pass. While such techniques are effective for MLPs, their extension to modern architectures remains challenging, due to increased depth and diversity of layer types. To fill this gap, we propose Calibrated Variance Propagation (CVP), which introduces a new propagation method for normalization layers, combines it with recent techniques for handling activation functions, and absorbs residual error through a light calibration step. CVP yields comparably accurate uncertainty estimates to MC sampling across transformers and CNNs, at a fraction of the cost. Against prior variance propagation work, CVP improves coverage at

0.5\%

risk from

8.2\%

14.6\%

with BEiT-3 on Visual Reasoning (NLVR2) and from

2.6\%

10.8\%

with ViLT on VQAv2, with gains extending to convolutional architectures.

Abstract

Uncertainty Quantification

Explore similar work

Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning

Gaussian Neural Networks

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration