Prior Predictive Matching for Bayesian LoRA

Tomasz Kuśmierczyk; Eliezer de Souza da Silva; Marcin Sendera

cs.LGOct 26, 2026

Prior Predictive Matching for Bayesian LoRA

Authors: Tomasz Kuśmierczyk, Eliezer de Souza da Silva, Marcin Sendera

Abstract

Bayesian variants of Low-Rank Adaptation (LoRA) provide principled uncertainty estimation when fine-tuning large foundation models, but require specifying the prior over the low-rank weight updates. We address this hyperparameter selection problem by introducing a closed-form prior-predictive estimator that operates before fine-tuning. We analyze a second-order expansion of the network and quantify how the low-rank prior drives function-space deviations from the pretrained outputs. We also show that these prior-predictive quantities depend on the prior only through the product of the adaptation rank and the fourth power of the prior scale. Because the training targets share a single draw of the latent weight update, we can leverage the covariance between example residuals to derive a moment-matching estimator. Furthermore, our theoretical analysis establishes that the optimal prior precision scales as $r^{-1/2}$ with the rank $r$ , allowing a single estimate to generalize across all ranks. Controlled experiments across regression and classification tasks validate our approximation and show that our estimator recovers the true prior scale. On RoBERTa-large, our approach improves predictive negative log-likelihood (NLL) and calibration over both MAP and marginal-likelihood selection without requiring validation data, while achieving competitive out-of-distribution detection.

Figures & tables

	validation search	marginal likelihood	ours
available before fine-tuning	$\times$	$\times$	✓
needs no held-out data	$\times$	✓	✓
one estimate for all ranks	$\times$	$\times$	✓

Table 1: Prior selection for Bayesian LoRA.

Figure 1: Ratio of moments of the network output to their second-order predictions (Equations 5 and 6 ). Shaded:

\theta\bar{s}\leq 3\cdot 10^{-3}

(the range used in Section 7.2 ).

Figure 2: Recovery of the prior scale: average

\hat{\sigma}

\pm

one standard deviation.

Figure 3: Recovery of the rank from the kurtosis (Equation 15 ). Median

\hat{K}

over 10 repetitions, with interquartile bars. The green line in the right panel ends early, as for larger

K

the estimates

\hat{K}

are undefined or outside the plotted range.

Figure 4: RoBERTa-large on SST-2, MR, CR and AG News, mean over the four tasks and three seeds. Bottom right: reliability diagram at

4{,}096

training examples, ten bins of equal size. Solid lines use no validation data. Ours + T adds a temperature fitted on the validation set. The other methods with a temperature, and each task separately, are in Figure 8 .

Panel of Figure 5 — Figure 5: Prior precision across LoRA ranks on RoBERTa-large. Left: validation NLL against the prior precision at three ranks for one seed, with its minimum $\lambda^{\ast}$ and ours, $\hat{\lambda}$ . Right: $\lambda^{\ast}$ of every run against the rank, its least-squares fit, and ours (medians).

Figure 6: AUROC for separating OOD inputs from the task’s own test inputs, with the mutual information as the score, mean over three seeds (left, also over the three tasks). The OOD inputs are AG News for the sentiment tasks and SST-2 for AG News. Per-task results are in Figure 9 and Table 7 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: The three estimators of

\theta

for binary classification on the task of Figure 2 . Average

\hat{\sigma}

\pm

one standard deviation, over the tasks with

\hat{\theta}>0

task	$r$	$\lambda^{\ast}$	$\hat{\lambda}$	at $\lambda^{\ast}$		+ T		+ T
				test NLL
					ours		validation search
AG News	4	2,512	3,340	0.308 $\pm$ 0.009	0.367 $\pm$ 0.113	0.302 $\pm$ 0.003	0.306 $\pm$ 0.003	0.305 $\pm$ 0.005
	8	1,585	2,362	0.304 $\pm$ 0.009	0.380 $\pm$ 0.129	0.303 $\pm$ 0.011	0.304 $\pm$ 0.014	0.301 $\pm$ 0.005
	16	631	1,670	0.301 $\pm$ 0.005	0.411 $\pm$ 0.128	0.300 $\pm$ 0.008	0.301 $\pm$ 0.006	0.301 $\pm$ 0.007
	32	398	1,181	0.306 $\pm$ 0.007	0.448 $\pm$ 0.134	0.305 $\pm$ 0.006	0.305 $\pm$ 0.007	0.303 $\pm$ 0.005

Appendix

Table 2: Rank study on RoBERTa-large with

1{,}024

training examples.

\lambda^{\ast}

and

\hat{\lambda}

: median over three seeds. Test NLL: mean over the seeds

\pm

their standard deviation, at

\lambda^{\ast}

and for each method. “+ T” adds a temperature fitted on the validation set.

	training examples
	64	256	$1{,}024$	$4{,}096$
test NLL
zero-shot	0.746	0.746	0.746	0.746
MAP	1.007 $\pm$ 0.149	0.875 $\pm$ 0.136	0.747 $\pm$ 0.035	0.473 $\pm$ 0.039
marginal likelihood	1.336 $\pm$ 0.018	1.250 $\pm$ 0.019	1.034 $\pm$ 0.021	0.354 $\pm$ 0.019
ours	0.458 $\pm$ 0.054	0.423 $\pm$ 0.109	0.395 $\pm$ 0.147	0.277 $\pm$ 0.012

Appendix

Table 3: Results on AG News: mean over three seeds

\pm

their standard deviation. Zero-shot does not depend on the seed. Bold: the best value in each column of a block, among the methods without validation data (upper rows) and among those with it (lower rows).

	training examples
	64	256	$1{,}024$	$4{,}096$
test NLL
zero-shot	0.461	0.461	0.461	0.461
MAP	0.936 $\pm$ 0.055	0.895 $\pm$ 0.062	0.807 $\pm$ 0.060	0.635 $\pm$ 0.048
marginal likelihood	0.644 $\pm$ 0.015	0.593 $\pm$ 0.009	0.514 $\pm$ 0.011	0.284 $\pm$ 0.004
ours	0.477 $\pm$ 0.069	0.416 $\pm$ 0.020	0.331 $\pm$ 0.020	0.296 $\pm$ 0.006

Appendix

Table 5: Results on MR, as in Table 4 .

	training examples
	64	256	$1{,}024$	$4{,}096$
AG News, OOD inputs from SST-2
marginal likelihood	0.360 $\pm$ 0.168	0.396 $\pm$ 0.038	0.745 $\pm$ 0.063	0.946 $\pm$ 0.006
ours	0.861 $\pm$ 0.017	0.829 $\pm$ 0.077	0.871 $\pm$ 0.051	0.904 $\pm$ 0.002
validation search	0.885 $\pm$ 0.024	0.838 $\pm$ 0.092	0.892 $\pm$ 0.040	0.922 $\pm$ 0.005
CR, OOD inputs from AG News

Appendix

Table 7: AUROC for separating OOD inputs from each task’s test inputs, with the mutual information as the score: mean over three seeds

\pm

their standard deviation (two seeds for CR at 256 and SST-2 at

4{,}096

examples). Bold: the best value in each column of a block.

Figure 8: Figure 4 for each task, with all methods, mean over three seeds.

Figure 9: Figure 6 for each task, mean over three seeds. The OOD inputs are SST-2 for AG News and AG News for the other tasks.

Figure 10: Each method with LoRA trained without a prior (gray) and under

\hat{\lambda}

(magenta), mean over tasks and seeds. The OOD AUROC of the MAP, which has no posterior, uses the maximum probability. Bottom row: reliability diagrams at

4{,}096

training examples.

Bayesian LoRA

Bayesian low-rank adaptation (LoRA) adds probabilistic parameter-efficient updates to estimate uncertainty while fine-tuning large models.

See all 18 papers

BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models
Apr 27, 2026Dario Coscia, Sindy Löwe, Max WellingBayesian Neural Networks Uncertainty Quantification
mathLab, SISSA · CuspAI
BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
Jun 28, 2026Zhibin Duan, Yuhong Wang, Jiahong Fu +3Bayesian Neural Networks Low-Rank Adaptation
National Key Lab of Radar Signal Processing, Xidian University, Xi’an, Shaanxi 710071, China · School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, 710049, China
Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
Jan 28, 2026Moule Lin, Shuhao Guan, Andrea Patane +2Bayesian Neural Networks Confidence Estimation in Language Models
School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

Apr 27, 2026J/K move · Enter open · S save

BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models

Dario Coscia, Sindy Löwe, Max Welling

mathLab, SISSA · CuspAI

Bayesian Neural Networks Uncertainty Quantification Low-Rank Adaptation Bayesian LoRA Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large pre-trained models at reduced computational cost. However, its low-rank point-estimate updates limit expressiveness, leave a persistent gap relative to full fine-tuning accuracy, and provide no built-in uncertainty quantification, limiting its applicability in settings where reliability matters as much as accuracy. We introduce BaLoRA, a Bayesian extension of LoRA with a novel input-adaptive Bayesian parameterization of LoRA matrices that adds minimal parameters and compute. Surprisingly, not only does the Bayesian extension yield well-calibrated uncertainty estimates, but the adaptive noise injection underlying our approach also significantly improves prediction accuracy, narrowing the gap with full fine-tuning across both natural language reasoning and vision tasks. When applied to band gap prediction in metal-organic frameworks, BaLoRA produces zero-shot test-time uncertainty estimates that correlate more strongly with model error than a trained ensemble of LoRA models, and improve monotonically with compute without sacrificing accuracy.

Abstract

Bayesian LoRA

Explore similar work

BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models

BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models