Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning

Ibne Farabi Shihab; Sanjeda Akter; Anuj Sharma

cs.LGFeb 19, 2026

Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning

Authors: Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Organizations: Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University

Abstract

Parameter-efficient fine-tuning methods such as LoRA enable practical adaptation of large language models but provide no principled uncertainty estimates, leading to poorly calibrated predictions and unreliable behavior under domain shift. We introduce Stiefel-Bayes Adapters (SBA), a Bayesian PEFT framework that places a Matrix Langevin prior over orthonormal adapter factors on the Stiefel manifold $\St$ and performs approximate posterior inference via tangent space Laplace approximation with geodesic retraction. Unlike Gaussian priors in flat space projected onto orthogonality constraints, our prior on the manifold naturally encodes the inductive bias that adapter subspaces should be well conditioned and orthogonal, while the posterior provides calibrated predictive uncertainty without recalibration. We prove formally that the tangent space approximation strictly avoids the structural variance inflation inherent in projecting from ambient space, establishing a rigorous theoretical advantage for intrinsic manifold inference. Across GLUE and SuperGLUE benchmarks on RoBERTa-large, LLaMA-2-7B, LLaMA-2-13B, Mistral-7B, and Qwen2.5-7B, domain shift evaluations, selective prediction protocols, and an abstractive summarization task, SBA achieves task performance comparable to LoRA and DoRA while reducing Expected Calibration Error by 18 to 34% over deterministic baselines, improving selective prediction AUROC by 12 to 25% under domain shift, and outperforming deep ensembles of five LoRA models on OOD detection at a fraction of the parameter cost. Our results demonstrate that where you place uncertainty, on the right geometric structure, matters more than simply adding any Bayesian treatment to adapters.

Figures & tables

Figure 1: Overview of Stiefel-Bayes Adapters. The Matrix Langevin prior respects the geometry of orthonormal adapter factors, and the tangent space Laplace approximation enables efficient posterior sampling via retraction.

Layer	$\mathrm{KL}(q_{\mathrm{tang}})$	$\mathrm{KL}(q_{\mathrm{proj}})$	$\Delta\mathrm{KL}$
Layer 1 (q_proj)	5.2	9.3	4.1
Layer 8 (q_proj)	3.8	6.9	3.1
Layer 16 (v_proj)	4.1	7.3	3.2
Layer 24 (o_proj)	2.9	5.2	2.3
Layer 32 (q_proj)	3.1	5.7	2.6
Mean (all layers)	3.7	6.8	3.1

Table 1: Empirical KL gap between tangent space and projected approximations across representative adapter layers (MNLI, LLaMA-2-7B).

\Delta\mathrm{KL}=\mathrm{KL}(q_{\mathrm{proj}}\|p_{\mathrm{ML}})-\mathrm{KL}(q_{\mathrm{tang}}\|p_{\mathrm{ML}})

in nats. Posterior spread is the trace of the tangent space covariance.

		MNLI			QQP			SST-2
Base	Method	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	F1 $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$
RoBERTa	LoRA	$87.8_{\pm.2}$	$.058_{\pm.004}$	$.168_{\pm.005}$	$88.1_{\pm.2}$	$.055_{\pm.003}$	$.163_{\pm.004}$	$94.7_{\pm.2}$	$.030_{\pm.003}$	$.074_{\pm.004}$
	Laplace-LoRA	$87.6_{\pm.2}$	$.037_{\pm.003}$	$.139_{\pm.004}$	$87.9_{\pm.2}$	$.036_{\pm.003}$	$.143_{\pm.004}$	$94.5_{\pm.2}$	$.020_{\pm.002}$	$.059_{\pm.003}$
	Deep-Ens-LoRA	$88.1_{\pm.1}$	$.029_{\pm.002}$	$.125_{\pm.003}$	$88.4_{\pm.1}$	$.027_{\pm.002}$	$.130_{\pm.003}$	$94.9_{\pm.1}$	$.016_{\pm.002}$	$.050_{\pm.002}$
	Gauss+Proj	$87.5_{\pm.2}$	$.034_{\pm.003}$	$.135_{\pm.004}$	$87.8_{\pm.2}$	$.033_{\pm.003}$	$.140_{\pm.004}$	$94.6_{\pm.2}$	$.018_{\pm.002}$	$.056_{\pm.003}$
	SBA (Ours)	$\underline{87.9_{\pm.2}}$	$\mathbf{.025_{\pm.002}}$	$\mathbf{.121_{\pm.003}}$	$\underline{88.2_{\pm.2}}$	$\mathbf{.022_{\pm.002}}$	$\mathbf{.126_{\pm.003}}$	$\underline{94.8_{\pm.2}}$	$\mathbf{.013_{\pm.001}}$	$\mathbf{.047_{\pm.002}}$

Table 2: In-distribution results on GLUE benchmarks. Results are mean

\pm

std over 5 seeds. Best result in each column is bolded, second best underlined.

\downarrow

indicates lower is better.

Figure 2: Reliability diagrams on MNLI (in-distribution, top) and MNLI

\to

SNLI (domain shift, bottom). SBA maintains calibration under shift while other methods degrade.

	MNLI $\to$ SNLI		Amz $\to$ Yelp		SQD $\to$ AdvQA
	ECE $\downarrow$	Sel. $\uparrow$	ECE $\downarrow$	Sel. $\uparrow$	ECE $\downarrow$	Sel. $\uparrow$
LoRA	$.098_{\pm.006}$	$.712_{\pm.012}$	$.121_{\pm.007}$	$.684_{\pm.014}$	$.142_{\pm.008}$	$.658_{\pm.015}$
Laplace-LoRA	$.072_{\pm.005}$	$.768_{\pm.010}$	$.089_{\pm.005}$	$.742_{\pm.011}$	$.108_{\pm.006}$	$.718_{\pm.012}$
Gauss+Proj	$.068_{\pm.004}$	$.779_{\pm.009}$	$.084_{\pm.005}$	$.753_{\pm.010}$	$.102_{\pm.006}$	$.729_{\pm.011}$
Deep-Ens	$.058_{\pm.003}$	$.801_{\pm.007}$	$.071_{\pm.004}$	$.775_{\pm.008}$	$.088_{\pm.005}$	$.756_{\pm.009}$
Temp. Scal.	$.064_{\pm.004}$	—	$.079_{\pm.005}$	—	$.097_{\pm.006}$	—

Table 3: Domain shift results on LLaMA-2-7B: calibration (ECE

\downarrow

) and selective prediction (Sel. AUROC

\uparrow

). Mean

\pm

std over 5 seeds.

Figure 3: Selective prediction on MNLI

\to

SNLI. SBA’s uncertainty estimates enable more effective abstention at all coverage levels.

Method	20News (far)	IMDB (near)
LoRA	$.841_{\pm.012}$	$.623_{\pm.018}$
Laplace-LoRA	$.892_{\pm.008}$	$.694_{\pm.014}$
Gauss+Proj	$.901_{\pm.007}$	$.712_{\pm.013}$
Deep-Ens-LoRA	$.921_{\pm.005}$	$.738_{\pm.010}$
SBA (Ours)	$\mathbf{.934_{\pm.005}}$	$\mathbf{.761_{\pm.009}}$

Table 4: OOD detection AUROC (

\uparrow

) using predictive entropy. In-distribution: SST-2 on LLaMA-2-7B. Mean

\pm

std over 5 seeds.

Figure 4: Uncertainty decomposition across settings. Epistemic uncertainty (mutual information) rises sharply under domain shift and on OOD data, while aleatoric uncertainty remains stable. SBA captures genuine model ignorance that flat-space methods underestimate.

$\kappa_{0}$	Acc $\uparrow$	ECE $\downarrow$	Brier $\downarrow$
0.1	$89.2_{\pm.2}$	$.041_{\pm.003}$	$.134_{\pm.004}$
0.5	$89.3_{\pm.2}$	$.032_{\pm.002}$	$.124_{\pm.003}$
1.0	$89.4_{\pm.2}$	$.027_{\pm.002}$	$.118_{\pm.003}$
2.0	$89.3_{\pm.2}$	$.027_{\pm.002}$	$.119_{\pm.003}$
5.0	$89.2_{\pm.2}$	$.028_{\pm.002}$	$.120_{\pm.003}$

Table 5: Ablation: prior concentration

\kappa_{0}

(MNLI, LLaMA-2-7B). Mean over 5 seeds.

$S$	ECE $\downarrow$	Sel. AUROC $\uparrow$	NLL $\downarrow$	Rel. cost
1	$.038_{\pm.003}$	$.791_{\pm.009}$	$.412_{\pm.008}$	1.0 $\times$
2	$.033_{\pm.002}$	$.808_{\pm.008}$	$.398_{\pm.007}$	2.0 $\times$
5	$.029_{\pm.002}$	$.819_{\pm.007}$	$.389_{\pm.006}$	5.0 $\times$
10	$.027_{\pm.002}$	$.821_{\pm.007}$	$.385_{\pm.006}$	10.0 $\times$
20	$.026_{\pm.002}$	$.822_{\pm.007}$	$.384_{\pm.006}$	20.0 $\times$
50	$.026_{\pm.002}$	$.822_{\pm.007}$	$.383_{\pm.006}$	50.0 $\times$

Table 6: Ablation: number of posterior samples

S

(MNLI, LLaMA-2-7B). Mean over 5 seeds.

Rank $k$	Acc $\uparrow$	ECE $\downarrow$	Params	Rel. cost
4	$88.9_{\pm.3}$	$.033_{\pm.003}$	0.26M	1.02 $\times$
8	$89.2_{\pm.2}$	$.029_{\pm.002}$	0.52M	1.04 $\times$
16	$89.4_{\pm.2}$	$.027_{\pm.002}$	1.05M	1.08 $\times$
32	$89.4_{\pm.2}$	$.026_{\pm.002}$	2.10M	1.15 $\times$
64	$89.5_{\pm.2}$	$.026_{\pm.002}$	4.19M	1.28 $\times$

Table 7: Ablation: adapter rank

k

(MNLI, LLaMA-2-7B). Mean over 5 seeds.

Bayesian over	Acc $\uparrow$	ECE $\downarrow$	Rel. cost
None (OrthoLoRA)	$89.3_{\pm.2}$	$.052_{\pm.003}$	1.0 $\times$
$\Sigma$ only	$89.2_{\pm.2}$	$.043_{\pm.003}$	1.02 $\times$
$U$ only	$89.3_{\pm.2}$	$.031_{\pm.002}$	1.04 $\times$
$U+V$	$89.4_{\pm.2}$	$.028_{\pm.002}$	1.08 $\times$
$U+\Sigma+V$ (full)	$89.4_{\pm.2}$	$.027_{\pm.002}$	1.08 $\times$

Table 8: Ablation: Bayesian treatment of different components (MNLI, LLaMA-2-7B). Mean over 5 seeds.

Under shift (MNLI $\to$ SNLI):
Method	ECE $\downarrow$	Sel. $\uparrow$	NLL $\downarrow$	Cost
OrthoLoRA	$.052_{\pm.003}$	$.731_{\pm.011}$	$.452_{\pm.009}$	1.0 $\times$
SBA ( $S$ =10)	$.027_{\pm.002}$	$.821_{\pm.007}$	$.385_{\pm.006}$	10 $\times$
SBA-Distilled	$.032_{\pm.002}$	$.798_{\pm.008}$	$.401_{\pm.007}$	1.0 $\times$
OrthoLoRA	$.087_{\pm.005}$	$.731_{\pm.011}$	—	1.0 $\times$
SBA ( $S$ =10)	$.049_{\pm.003}$	$.821_{\pm.007}$	—	10 $\times$

Table 9: Predictive distillation results (MNLI, LLaMA-2-7B). The distilled model retains most of SBA’s calibration at

1\times

inference cost.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Adapter rank $k$	16
Prior concentration $\kappa_{0}$	1.0
Prior scale $\tau$ (for $\Sigma$ )	0.1
Learning rate	$2\times 10^{-4}$
Batch size	16 (per GPU)
Epochs	3

Appendix

Table 10: Hyperparameters for SBA and baselines.

Bayesian LoRA

Bayesian low-rank adaptation (LoRA) adds probabilistic parameter-efficient updates to estimate uncertainty while fine-tuning large models.

See all 17 papers

Jan 28, 2026

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and

{\approx}1.2{\times}

training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Moule Lin, Shuhao Guan, Andrea Patane +2

School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

Bayesian Neural Networks Confidence Estimation in Language Models LLM Fine-Tuning Low-Rank Adaptation Language Model Calibration LLM Reliability+2

Ask PDF

Mar 1, 2026

HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

Modern Transformer-based models frequently suffer from miscalibration, producing overconfident predictions that do not reflect true empirical frequencies. This work investigates the calibration dynamics of LoRA: Low-Rank Adaptation and a novel hyper-network-based adaptation framework as parameter-efficient alternatives to full fine-tuning for RoBERTa. Evaluating across the GLUE benchmark, we demonstrate that LoRA-based adaptation consistently achieves calibration parity with (and in specific tasks exceeds) full fine-tuning, while maintaining significantly higher parameter efficiency. We further explore a dynamic approach where a shared hyper-network generates LoRA factors (A and B matrices) to induce structural coupling across layers. This approach produced results similar to standard LoRA fine-tuning, even achieving better MCC on CoLA dataset. Our study also reveal a critical trade-off: constraining the adaptation space (e.g., freezing matrices A) acts as a powerful regularizer that enhances Expected Calibration Error (ECE), but necessitates a carefully balanced sacrifice in downstream task accuracy. To support future research, we provide a unified and reproducible implementation of contemporary calibration metrics, including ECE, MCE, and ACE. Our findings clarify the relationship between parameter efficiency and probabilistic reliability, positioning structured low-rank updates as a viable foundation for uncertainty-aware Transformer architectures. Code available at: https://github.com/btrojan-official/HypeLoRA

Bartosz Trojan, Filip Gębala

Upper-Secondary Schools of Communications in Cracow

Low-Rank Adaptation Language Model Calibration Parameter-Efficient Fine-Tuning Hypernetworks

Ask PDF

Sep 8, 2026

Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent

Several geometry-aware approaches to low-rank adaptation have emerged for parameter-efficient fine-tuning of large pre-trained models. These methods aim to take full advantage of the geometric structure of low-rank manifolds for improving the efficiency in subspace utilization and reducing redundancy by enforcing orthogonality constraints during optimization. The strong empirical results of these techniques have motivated further study into whether predictions from such geometry-based adaptation methods could be overconfident. In this paper, we build on the singular value decomposition factorization of adapters to develop a framework based on Stein variational gradient descent (SVGD). In this formulation, the low-rank matrices are transported along the Stiefel manifold to match the targeted distributions while retaining their crucial geometric structure. Since this geometry-aware SVGD approach provides multiple solutions during inference, it supports uncertainty quantification and produces better-calibrated adapters on the Stiefel manifold. Extensive experiments show that our method delivers strong model calibration and attains higher prediction accuracy than SVGD and related uncertainty estimation methods that are formulated in Euclidean space.

Quang-Duy Tran, Trung Le, Bao Duong +2

Deakin Applied Artificial Intelligence Initiative, Deakin University, Geelong, Australia · Department of Computer Science, Aalto University, Espoo, Finland · Department of Data Science & AI, Monash University, Melbourne, Australia

Stiefel Manifold Optimization Uncertainty Quantification Model Calibration Bayesian LoRA Stein Variational Gradient Descent

Ask PDF

Abstract

Bayesian LoRA

Explore similar work

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent