Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Moule Lin; Shuhao Guan; Andrea Patane; David Gregg; Goetz Botterweck

cs.AIJan 28, 2026

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Authors: Moule Lin, Shuhao Guan, Andrea Patane, David Gregg, Goetz Botterweck

Organizations: School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

Abstract

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ${\approx}1.2{\times}$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Figures & tables

Figure 1 : Bayesian-LoRA overview. (a) Standard LoRA decomposes the weight update into low-rank matrices

B

and

A

. (b) Bayesian-LoRA places a Sparse Gaussian Process prior over the LoRA subspace. For each LoRA factor, base samples

U_{0}

are transformed by a factor-specific normalizing flow (

T_{\phi}^{A}

T_{\phi}^{B}

) into inducing variables (

U^{A}

U^{B}

), which are projected via conditional Gaussians to produce the stochastic LoRA matrices

A

and

B

. The effective weight

W_{\text{merged}}=W_{\text{pre}}+\frac{\alpha}{r}BA

is used for the forward pass; Monte Carlo samples capture epistemic uncertainty.

	All tokens						Top 5% entropy tokens
	Validation			Test			Validation			Test
Method	NLL $\downarrow$	Brier $\downarrow$	ECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$	ECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$	ECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$	ECE $\downarrow$
LoRA (MAP)	1.76	0.52	1.68	1.75	0.52	1.48	5.16	0.97	0.82	5.16	0.97	1.60
LoRA + Temp	1.78	0.51	1.62	1.77	0.50	1.54	5.22	0.92	0.81	5.25	0.92	1.49
Dropout ( $\text{N}=4$ )	1.77	0.50	1.54	1.70	0.50	1.59	5.21	0.94	0.79	5.19	0.95	1.51
Bayesian-LoRA (N = 1)	1.73	0.51	1.46	1.71	0.51	1.51	5.14	0.95	0.80	5.14	0.92	1.31

Table 2 : Language modeling on WikiText-2 (validation/test). We report NLL

\downarrow

, Brier score

\downarrow

, and ECE

\downarrow

(15 bins) for both all tokens and the top 5% most uncertain tokens (selected by MAP predictive entropy).

Model (Zero-shot)	Method	CoT-NLL $\downarrow$	CoT-ECE $\downarrow$	Answer Acc. $\uparrow$
Qwen2.5-14B-Instruct	Baseline FT	2.165	12.2	49.8
	Dropout ( Gal and Ghahramani, 2016 )	2.103	11.9	50.0
	Temp ( Guo et al., 2017 )	1.96	10.7	49.9
	LA (post-hoc) ( Yang et al., 2024 )	0.81	7.12	49.8
	BLoB ( $\text{N}=10$ ) ( Wang et al., 2024a )	1.21	8.41	47.2
	Bayesian-LoRA ( $\text{N}=4$ )	0.513	5.81	51.1

Table 3 : Performance on the MATH dataset with large-scale models. We report Chain-of-Thought NLL (CoT-NLL), CoT Expected Calibration Error (CoT-ECE), and final answer accuracy. Dropout was evaluated only on Qwen2.5-14B-Instruct. Training hyperparameters are in Table 22 .

Method	Trainable Params	Train time ( $\times$ MAP)	Peak mem. ( $\times$ MAP)	Inference ( $\times$ MAP)	Samples (Val)
MAP (LoRA) ( Hu et al., 2022 )	4.48M	1.00	1.00	1.00	1
Dropout ( Gal and Ghahramani, 2016 )	4.48M	$\approx 4\times$	$\approx 1\times$	$\approx 4\times$	4
Ckpt Ens (3) ( Huang et al., 2017 )	$1\times$ MAP	$\approx 1\times$	$\approx 1\times$	$\approx 3\times$	3
Deep Ens (3) ( Lakshminarayanan et al., 2017 )	$3\times$ MAP	$\approx 3\times$	$\approx 3\times$	$\approx 3\times$	3
BBB ( Blundell et al., 2015 )	$\approx 2\times$ MAP	$\approx 4.19\times$	$\approx 1.05\times$	$\approx 4.6\times$	4
BLoB ( $\text{N}=4$ ) ( Wang et al., 2024a )	$\approx$ 1.5 $\times$ MAP	$\approx 1.11\times$	$\approx 0.95\times$	$\approx 6.3\times$	4

Table 4 : Efficiency comparison normalized to standard LoRA (MAP) on WinoGrande-M. All baseline results are reproduced from open-source code. Inference-cost comparisons use a comparable sample budget (

\text{N}=4

) across sampling-based methods; the accuracy and calibration tables report BLoB at its recommended

\text{N}=10

. KFAC denotes Kronecker-Factored Approximate Curvature.

Flow depth $L$	ACC $\uparrow$		ECE $\downarrow$		NLL $\downarrow$		Efficiency ( $\times$ MAP)
Flow depth $L$	value	$\Delta$ vs. $L{=}1$	value	$\Delta$ vs. $L{=}1$	value	$\Delta$ vs. $L{=}1$	train time	peak mem.
0 (pure SGP)	79.0 $\pm$ 0.21	-2.6	5.8 $\pm$ 0.13	+0.1	0.58 $\pm$ 0.08	+0.09	1.19	1.002
1	81.6 $\pm$ 0.10	0.0	5.7 $\pm$ 0.20	0.0	0.49 $\pm$ 0.02	0.00	1.23	1.003
2	80.8 $\pm$ 0.14	-0.8	5.6 $\pm$ 0.09	-0.1	0.52 $\pm$ 0.06	+0.03	1.30	1.008
4	80.9 $\pm$ 0.08	-0.7	4.9 $\pm$ 0.03	-0.8	0.48 $\pm$ 0.13	-0.01	1.38	1.010

Table 5 : Ablation on flow depth

L

in the posterior transform

T_{\phi}

(OBQA). Values are mean

\pm

std over three seeds.

\Delta

denotes the change from the

L{=}1

baseline. Efficiency is relative to standard LoRA (MAP).

Figure 2 : Ablation on inducing-point dimension $r=c$ . Increasing the dimension improves calibration (lower ECE) with diminishing returns beyond $r{=}16$ . The dashed line shows the default $r=c=9$ used in the main-paper Bayesian-LoRA experiments.

		ID	Smaller Distribution Shift		Larger Distribution Shift
Metrics	Methods	OBQA	ARC-C	ARC-E	CS	Eng	Law	Health
ACC $\uparrow$	MAP ( Hu et al., 2022 )	$77.7\pm{0.8}$	$67.9\pm{1.4}$	$77.7\pm{0.3}$	$42.0\pm{3.2}$	$41.2\pm{2.0}$	$37.4\pm{0.4}$	$48.3\pm{0.3}$
	Dropout ( Gal and Ghahramani, 2016 )	$77.7\pm{0.2}$	$67.7\pm{0.6}$	$77.2\pm{0.6}$	$41.9\pm{2.2}$	$39.6\pm{1.7}$	${37.9\pm{0.4}}$	$48.2\pm{0.9}$
	Ckpt Ens ( Huang et al., 2017 )	$78.2\pm{0.2}$	$67.9\pm{0.8}$	$77.4\pm{0.8}$	$41.1\pm{2.0}$	$38.7\pm{1.2}$	$37.7\pm{0.2}$	$48.2\pm{0.6}$
	Temp ( Guo et al., 2017 )	$77.7\pm{0.8}$	$68.0\pm{0.2}$	$76.7\pm{1.0}$	$43.5\pm{0.9}$	$44.4\pm{2.0}$	$37.4\pm{0.1}$	$47.7\pm{0.8}$
	BBB ( Blundell et al., 2015 )	${77.0\pm{0.2}}$	$67.3\pm{1.2}$	$75.8\pm{0.8}$	$40.5\pm{0.3}$	$36.7\pm{0.2}$	$36.1\pm{0.2}$	$47.6\pm{0.5}$

Table 6 : Robustness under distribution shift. We evaluate on the in-distribution OBQA test set and six out-of-distribution datasets spanning small shifts (ARC-C, ARC-E) and large shifts (CS, Eng, Law, Health from MMLU). Experimental settings follow ( Yang et al., 2024 ) .

Method	ACC $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
LoRA (MAP)	$77.70\pm{0.80}$	$9.80\pm{1.00}$	$0.71\pm{0.03}$
Degenerate Bayesian-LoRA	$77.81\pm{0.71}$	$9.77\pm{1.02}$	$0.70\pm{0.02}$
Bayesian-LoRA (full)	$81.60\pm{0.10}$	$5.70\pm{0.20}$	$0.49\pm{0.02}$

Table 7 : MAP recovery validation on OBQA. Degenerate Bayesian-LoRA uses near-zero posterior variance and no flow, approximating the MAP limit of Corollary 4.2 . Results are mean

\pm

std over three seeds.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task type	Candidates	Primary metric(s)
WinoGrande-S/M (WG-S/M)	Cloze coreference (2-way)	2	ACC, ECE, NLL
ARC-Challenge (ARC-C)	Multiple-choice science QA	3–5 (mostly 4)	ACC, ECE, NLL
ARC-Easy (ARC-E)	Multiple-choice science QA	3–5 (mostly 4)	ACC, ECE, NLL
OpenBookQA (OBQA)	Multiple-choice science QA	4	ACC, ECE, NLL
BoolQ	Yes/No reading comprehension	2	ACC, ECE, NLL

Appendix

Table 8: Task formats and primary metrics.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$5\times 10^{-4}$
Betas	$(0.9,\,0.999)$
Epsilon ( $\epsilon$ )	$1\times 10^{-5}$
Weight decay	$0.1$
Scheduler	MultiStepLR (milestones = [4, 6], $\gamma=0.1$ )

Appendix

Table 9 : Hyperparameters used in the commonsense reasoning and OoD experiments. The MATH configuration is given in Table 22 .

Hyperparameter	Value
inducing_rows	9
inducing_cols	9
whitened_u	True
q_inducing	diagonal
learn_lambda	True
init_lambda	0.001

Appendix

Table 10 : Inducing hyperparameters.

Subject	Tasks
Computer Science (CS)	college computer science, computer security,
	high school computer science, machine learning
Engineering (Eng)	electrical engineering
Law	international law, jurisprudence, professional law
Health	anatomy, clinical knowledge, college medicine, human aging,
	nutrition, professional medicine, virology

Appendix

Table 11 : MMLU subjects and tasks.

Figure 3 : Pareto analysis on the WinoGrande-M dataset (NLL vs. ECE). Each point denotes a hyperparameter pair $(\mathrm{lr},\mathrm{wd})$ . Gray regions show dominated solutions, and the cross marks the “Best choice” near the Pareto front.

Figure 4 : Pareto analysis on the ARC-Easy dataset (ACC vs. NLL and ACC vs. ECE). Each point denotes a hyperparameter pair $(\mathrm{lr},\mathrm{wd})$ . Gray regions show dominated solutions, and the cross marks the “Best choice” near the Pareto front.

Method	ACC $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
Qwen3-14B base (4-shot)	76.7	9.07	0.2856
Standard LoRA	80.0	11.26	0.2982
Bayesian-LoRA (epoch 5)	80.0	5.26	0.1144

Appendix

Table 16: Qwen3-14B on AIME 2024 (

\text{N}=4

; train: AIME 1983–2023, test: 30 problems). We report accuracy (ACC, %) together with ECE and NLL of the chain-of-thought predictive distribution. Best results are in bold .

Metric	Method	ARC-C	ARC-E
ACC $\uparrow$	BLoB-Mean + TFB	83.33 $\pm$ 0.19	91.76 $\pm$ 0.48
ACC $\uparrow$	Bayesian-LoRA	83.01	91.12
ECE $\downarrow$	BLoB-Mean + TFB	6.48 $\pm$ 0.36	2.44 $\pm$ 0.50
ECE $\downarrow$	Bayesian-LoRA	3.23	1.83
NLL $\downarrow$	BLoB-Mean + TFB	0.530 $\pm$ 0.04	0.230 $\pm$ 0.02
NLL $\downarrow$	Bayesian-LoRA	0.503	0.217

Appendix

Table 18: Comparison with TFB ( Shi et al., 2025 ) on Llama-3.1-8B. TFB numbers are taken from Table 1 of ( Shi et al., 2025 ) . Best results in each row are in bold .

Figure 5 : Accuracy (ACC,

\uparrow

) vs. Expected Calibration Error (ECE,

\downarrow

) on Llama-2-7B, averaged across six commonsense reasoning benchmarks. Each marker is a method. Bayesian-LoRA (red star) is Pareto-optimal, with the highest accuracy of all methods; among the baselines, only LA (post-hoc) attains a slightly lower average ECE, at a clear cost in accuracy. The ECE axis is reversed (low ECE on the right), so the ideal region, with high ACC and low ECE, is the upper-right corner.

Figure 6 : Accuracy (ACC,

\uparrow

) vs. Expected Calibration Error (ECE,

\downarrow

) on the WinoGrande-S (WG-S) benchmark with Llama-2-7B. Bayesian-LoRA (red star) lies on the upper-right Pareto frontier, achieving both the highest accuracy and the lowest ECE. BBB collapses to near-chance accuracy on this small binary task, and post-hoc methods (Temperature Scaling, Laplace, Last-Layer Laplace) reduce ECE relative to MAP but remain less accurate and less calibrated than Bayesian-LoRA .

Seed	ACC $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
42	94.59	0.76	0.1347
123	94.57	0.88	0.1345
456	94.57	0.88	0.1348
789	94.57	0.81	0.1349
2024	94.57	0.76	0.1347
mean $\pm$ std	94.57 $\pm$ 0.01	0.82 $\pm$ 0.06	0.1347 $\pm$ 0.0001

Appendix

Table 19: Seed stability of Bayesian-LoRA on ARC-Easy (Qwen3-14B,

\text{N}=4

). The last row reports mean

\pm

standard deviation over five seeds.

Placement	Target modules	ACC $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
A (default)	q, k, lm_head	94.59	0.76	0.1347
B (LoRA-style)	q, v, lm_head	94.57	0.74	0.1347
C (Attention)	q, k, v, o, lm_head	94.57	0.71	0.1348
D (MLP)	gate, up, down, lm_head	94.49	0.93	0.1348
E (All)	all linear + lm_head	94.53	0.74	0.1348

Appendix

Table 20: Adapter placement ablation on ARC-Easy (Qwen3-14B, seed

=42

\text{N}=4

). Rows differ only in the set of target modules to which Bayesian-LoRA adapters are attached.

Configuration	$r$	$c$	ACC $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
$r=c$ (default)	9	9	94.59	0.76	0.1347
$r<c$	4	9	94.57	0.80	0.1347
$r>c$	9	4	94.57	0.72	0.1349
$r<c$ (larger)	9	16	94.53	0.82	0.1349
$r>c$ (larger)	16	9	94.53	0.83	0.1349

Appendix

Table 21: Ablation over the inducing-matrix shape

(r,c)

on ARC-Easy (Qwen3-14B, seed

=42

\text{N}=4

). Performance is robust to the choice of

r

versus

c

and does not require

r=c

Category	Hyperparameter (value)
Model & tokenizer
Model name	Qwen/Qwen2.5-14B-Instruct , Qwen/Qwen3-30B-A3B-Instruct-2507
Max input length	1024
BF16 precision	True
FP16 precision	False
Load in 8-bit	False

Appendix

Table 22: Training hyperparameters for deterministic LoRA and Bayesian-LoRA fine-tuning on the MATH dataset.

Explore similar work

BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models
Apr 27, 2026Dario Coscia, Sindy Löwe, Max WellingModel Fine-Tuning Low-Rank Structure
mathLab, SISSA · CuspAI
Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation
Jul 2, 2026Jijie Zhang, Zhe Ren, Quan Zhang +1Large Language Model Uncertainty Training-Side Stage-Aware Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · Michigan State University
Bayesian Fine-tuning in Projected Subspaces
May 8, 2026Viktar Dubovik, Patryk Marszałek, Jacek Tabor +1Parameter-Efficient Fine-Tuning Methods Low-Rank Structure
Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland

Apr 27, 2026, cs.LGJ/K move · Enter open · S save

BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models

Dario Coscia, Sindy Löwe, Max Welling

mathLab, SISSA · CuspAI

Model Fine-Tuning Low-Rank Structure Bayesian Model Size

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large pre-trained models at reduced computational cost. However, its low-rank point-estimate updates limit expressiveness, leave a persistent gap relative to full fine-tuning accuracy, and provide no built-in uncertainty quantification, limiting its applicability in settings where reliability matters as much as accuracy. We introduce BaLoRA, a Bayesian extension of LoRA with a novel input-adaptive Bayesian parameterization of LoRA matrices that adds minimal parameters and compute. Surprisingly, not only does the Bayesian extension yield well-calibrated uncertainty estimates, but the adaptive noise injection underlying our approach also significantly improves prediction accuracy, narrowing the gap with full fine-tuning across both natural language reasoning and vision tasks. When applied to band gap prediction in metal-organic frameworks, BaLoRA produces zero-shot test-time uncertainty estimates that correlate more strongly with model error than a trained ensemble of LoRA models, and improve monotonically with compute without sacrificing accuracy.

Abstract

Explore similar work

BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models

Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation

Bayesian Fine-tuning in Projected Subspaces