Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces

Wouter N. Edeling; Peter V. Coveney

cs.AIOct 8, 2026

Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces

Authors: Wouter N. Edeling, Peter V. Coveney

Organizations: Scientific Computing Group, CWI, Science park 123, Amsterdam, 1098XG, the Netherlands. · Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Hallenweg 15, Enschede, 7500AE, the Netherlands. · Centre for Computational Science, University College London, 20 Gordon Street, London, WC1H 0AJ, UK. · Advanced Research Computing Centre, University College London, Gordon Street, London, WC1E 6BT, UK.

Abstract

Reliable uncertainty quantification (UQ) is essential for deploying neural networks in scientific and high-stakes applications, but full Bayesian inference over the network parameters is computationally infeasible. We propose a low-rank generalized Laplace approximation for neural-network UQ based on a small number of data-informed curvature directions. Starting from a generalized Bayesian posterior defined through an empirical loss, we construct a local Gaussian approximation around a pretrained set of weights in this active curvature subspace. The posterior variances in the retained subspace are available in closed form, and the prior variance is calibrated by an empirical Bayes procedure. The generalized Bayesian formulation allows us to compare two posterior scalings: the standard Bayesian scaling associated with the summed negative log likelihood, and a mean-loss scaling in which the empirical loss is normalized by the number of data. A central finding is that the standard scaling induces a data-size dependent contraction of the posterior variance in the leading active directions. In regression problems, this can force the low-rank framework to retain additional weak-curvature directions in order to achieve nominal coverage of calibration data. When posterior samples are propagated through the non-linear network, these additional directions can degrade the coherence of the predictive intervals and shift the posterior predictive mean away from the pretrained model. In contrast, the generalized mean-loss scaling yields a more stable, lower dimensional active subspace and produces calibrated, coherent predictive confidence intervals. These results indicate that generalized Laplace active subspaces provide a practical and scalable route to calibrated uncertainty quantification in neural networks.

Figures & tables

Figure 1: The matrix-vector product $Gv\in\mathbb{R}^{D}$ with $D=2701$ , computed using the matrix-free method outlined in Section 4.1.5 , and by explicitly forming matrix $G$ via ( 28 ). The absolute error $|Gv-(Gv)_{explicit}|$ is plotted on the right vertical axis.

Panel of Figure 2 — Figure 2: The $d$ leading eigenvalues $\lambda_{j}$ and leading eigenvector $p_{1}$ of $G$ , computed using the matrix-free method outlined in Section 4.1.5 , and by explicitly forming matrix $G$ via ( 28 ). The absolute errors are plotted on the right vertical axis.

Panel of Figure 3 — Figure 3: 90% confidence intervals for the, a) generalized Bayesian scaling, and b) standard Bayesian scaling. Both CIs were generated by drawing 1000 posterior samples, and propagating these through the network.

Figure 4: The results for the standard Bayesian scaling $\beta=N\sigma^{-2}$ with $N=1000$ and $d=43$ . The CIs were generated by drawing 1000 posterior samples, and propagating these through the network.

Panel of Figure 5 — Figure 5: The results for the standard Bayesian scaling $\beta=N\sigma^{-2}$ combined with the linearized Laplace predictive posterior distribution, computed using $f(x;\theta_{0})\pm 2\sigma_{f}$ .

Panel of Figure 6 — Figure 6: The coverage of the calibration procedure as outlined in Section 4.1.7 , versus the active subspace dimension $d$ , for both the standard Bayesian scaling $\beta=N\sigma^{-2}$ and the generalized scaling $\beta=\sigma^{-2}$ .

Panel of Figure 7 — Figure 7: The parity plot of the mean compressive strength of concrete, with 90% CIs. We show the standard and generalized scaling results, both computed with sampled Laplace.

Figure 8: The cumulative (normalized) activity scores for our concrete compression strength feed-forward neural network with approximately $D=10^{6}$ weights.

Figure 9: The inner products $\left|\left<p_{1},p_{1}^{ref}\right>\right|$ as a function of $N_{sub}\in\{500,1000,1500,2000,2500,3000\}$ , replicated 25 times per $N_{sub}$ . The reference vectors $p_{1}^{ref}$ were computed using $N_{sub}=10.000$ SMILES strings.

	baseline	Laplace (std. scaling)	Laplace (gen. scaling)
$d=1$	8692	8709	6863
$d=2$	8692	8753	5279
$d=3$	8692	8732	4435

Table 1: Number of valid SMILES strings (out of 10k).

Figure 10: Kernel density estimates of the sequence length for the standard and generalized scaling with $d\in\{1,2,3\}$ . The maximum allowed sequence length is 256.

Quantity	Description
Molecular weight	Molecular size based on atomic masses, large values indicate large molecules.
SA score	Synthetic Accessibility score, higher values indicate more difficult synthesis.
SlogP	Estimate of how fat-soluble a molecule is.

Table 2: Molecular quantities of interest used to compare generated SMILES distributions within REINVENT.

Panel of Figure 11 — Figure 11: Kernel density estimates of the QoIs for the standard and generalized scaling with $d\in\{1,2,3\}$ .

Figure 12: The cumulative (normalized) activity scores for the REINVENT LSTM with approximately 1.6M weights. Computed using $d=2$ .

Panel of Figure 13 — Figure 13: The contraction ratio ( 10 ) and posterior variances ( 22 ) for the generalized (left column) and standard (right column) Bayesian scaling, for both $N=50$ and $N=1000$ . These results were computed using the analytic regression function ( 7 ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Index	Token	Description	Index	Token	Description
0	$	Stop/end token	17	Br	Bromine atom
1	^	Start token	18	C	Aliphatic carbon atom
2	#	Triple bond	19	Cl	Chlorine atom
3	%10	Two-digit ring closure	20	F	Fluorine atom
4	(	Branch opening	21	N	Aliphatic nitrogen atom
5	)	Branch closing	22	O	Oxygen atom

Appendix

Table 3: Vocabulary used by the autoregressive SMILES generator. The vocabulary contains start and stop tokens, bond symbols, branch symbols, ring-closure indices, atom tokens, and charged or aromatic atom tokens.

No.	SMILES
1	CC(C)C(NC(=O)C(NC(=O)C1OC2OC(C)(C)OC2C2OC(C)(C)OC12)C(C)C)C(O)=O
2	Fc1ccc(cc1)C(=O)NN1C(SCc2ccc(cc2)N(=O)=O)=Nc2ccccc2C1=O
3	CC(C)CN1Cc2cnnn2-c2ccc(cc2C1)-c1ccccc1
4	CCN(C(=O)CSc1nnnn1-c1ccccc1)C1=C(N)N(Cc2ccccc2)C(=O)NC1=O
5	CC(C)N1CCN(CC1)C(=O)N1c2ccccc2Sc2ccccc12
6	CC1(C)Cc2cc(OCc3ccc(cc3)-c3nn[nH]n3)c(Cl)c(Cl)c2C1=O

Appendix

Table 4: 10 SMILES strings from the ChEMBL database.

Explore similar work

An Isotropic Approach to Efficient Uncertainty Quantification with Gradient Norms
Mar 31, 2026Nils Grünefeld, Jes Frellsen, Christian HardmeierUncertainty Quantification LLM Uncertainty Estimation
1IT University of Copenhagen · 3Pioneer Centre for Artificial Intelligence · 2Technical University of Denmark
Optimality of Sub-network Laplace Approximations: New Results and Methods
May 9, 2026Swarnali Raha, Kshitij Khare, Rohit K PatraBayesian Neural Networks Uncertainty Quantification
Department of Statistics University of Florida Gainesville, FL · Linkedin Inc. New York, NY
Is the Last Layer Sufficient for Uncertainty Quantification?
May 29, 2026Joseph Wilson, Chris van der Heide, Liam Hodgkinson +1Uncertainty Quantification Bayesian Inference
School of Mathematics and Physics, University of Queensland, Australia · ARC Training Centre for Information Resilience (CIRES), Brisbane, Australia · Department of Electrical and Electronic Engineering, University of Melbourne, Australia +1

Mar 31, 2026, cs.LGJ/K move · Enter open · S save

An Isotropic Approach to Efficient Uncertainty Quantification with Gradient Norms

Nils Grünefeld, Jes Frellsen, Christian Hardmeier

1IT University of Copenhagen · 3Pioneer Centre for Artificial Intelligence · 2Technical University of Denmark

Uncertainty Quantification LLM Uncertainty Estimation Uncertainty Decomposition Epistemic Uncertainty

Open paper AskPDF arXiv

Existing methods for quantifying predictive uncertainty in neural networks are either computationally intractable for large language models or require access to training data that is typically unavailable. We derive a lightweight alternative through two approximations: a first-order Taylor expansion that expresses uncertainty in terms of the gradient of the prediction and the parameter covariance, and an isotropy assumption on the parameter covariance. Together, these yield epistemic uncertainty as the squared gradient norm and aleatoric uncertainty as the Bernoulli variance of the point prediction, from a single forward-backward pass through an unmodified pretrained model. We justify the isotropy assumption by showing that covariance estimates built from non-training data introduce structured distortions that isotropic covariance avoids, and that theoretical results on the spectral properties of large networks support the approximation at scale. Validation against reference Markov Chain Monte Carlo estimates on synthetic problems shows strong correspondence that improves with model size. We then use the estimates to investigate when each uncertainty type carries useful signal for predicting answer correctness in question answering with large language models, revealing a benchmark-dependent divergence: the combined estimate achieves the highest mean AUROC on TruthfulQA, where questions involve genuine conflict between plausible answers, but falls to near chance on TriviaQA's factual recall, suggesting that parameter-level uncertainty captures a fundamentally different signal than self-assessment methods.

Abstract

Explore similar work

An Isotropic Approach to Efficient Uncertainty Quantification with Gradient Norms

Optimality of Sub-network Laplace Approximations: New Results and Methods

Is the Last Layer Sufficient for Uncertainty Quantification?