cs.LGFeb 19, 2026

Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning

Authors: Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Organizations: Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Parameter-efficient fine-tuning methods such as LoRA enable practical adaptation of large language models but provide no principled uncertainty estimates, leading to poorly calibrated predictions and unreliable behavior under domain shift. We introduce Stiefel-Bayes Adapters (SBA), a Bayesian PEFT framework that places a Matrix Langevin prior over orthonormal adapter factors on the Stiefel manifold \St\St and performs approximate posterior inference via tangent space Laplace approximation with geodesic retraction. Unlike Gaussian priors in flat space projected onto orthogonality constraints, our prior on the manifold naturally encodes the inductive bias that adapter subspaces should be well conditioned and orthogonal, while the posterior provides calibrated predictive uncertainty without recalibration. We prove formally that the tangent space approximation strictly avoids the structural variance inflation inherent in projecting from ambient space, establishing a rigorous theoretical advantage for intrinsic manifold inference. Across GLUE and SuperGLUE benchmarks on RoBERTa-large, LLaMA-2-7B, LLaMA-2-13B, Mistral-7B, and Qwen2.5-7B, domain shift evaluations, selective prediction protocols, and an abstractive summarization task, SBA achieves task performance comparable to LoRA and DoRA while reducing Expected Calibration Error by 18 to 34% over deterministic baselines, improving selective prediction AUROC by 12 to 25% under domain shift, and outperforming deep ensembles of five LoRA models on OOD detection at a fraction of the parameter cost. Our results demonstrate that where you place uncertainty, on the right geometric structure, matters more than simply adding any Bayesian treatment to adapters.

Figures & tables

Figure 1: Overview of Stiefel-Bayes Adapters. The Matrix Langevin prior respects the geometry of orthonormal adapter factors, and the tangent space Laplace approximation enables efficient posterior sampling via retraction.
LayerKL(qtang)\mathrm{KL}(q_{\mathrm{tang}})KL(qproj)\mathrm{KL}(q_{\mathrm{proj}})ΔKL\Delta\mathrm{KL}
Layer 1 (q_proj)5.29.34.1
Layer 8 (q_proj)3.86.93.1
Layer 16 (v_proj)4.17.33.2
Layer 24 (o_proj)2.95.22.3
Layer 32 (q_proj)3.15.72.6
Mean (all layers)3.76.83.1
Table 1: Empirical KL gap between tangent space and projected approximations across representative adapter layers (MNLI, LLaMA-2-7B). ΔKL=KL(qproj∥pML)−KL(qtang∥pML)\Delta\mathrm{KL}=\mathrm{KL}(q_{\mathrm{proj}}\|p_{\mathrm{ML}})-\mathrm{KL}(q_{\mathrm{tang}}\|p_{\mathrm{ML}}) in nats. Posterior spread is the trace of the tangent space covariance.
  MNLIQQPSST-2
BaseMethodAcc ↑\uparrowECE ↓\downarrowBrier ↓\downarrowF1 ↑\uparrowECE ↓\downarrowBrier ↓\downarrowAcc ↑\uparrowECE ↓\downarrowBrier ↓\downarrow
RoBERTaLoRA87.8±.287.8_{\pm.2}.058±.004.058_{\pm.004}.168±.005.168_{\pm.005}88.1±.288.1_{\pm.2}.055±.003.055_{\pm.003}.163±.004.163_{\pm.004}94.7±.294.7_{\pm.2}.030±.003.030_{\pm.003}.074±.004.074_{\pm.004}
Laplace-LoRA87.6±.287.6_{\pm.2}.037±.003.037_{\pm.003}.139±.004.139_{\pm.004}87.9±.287.9_{\pm.2}.036±.003.036_{\pm.003}.143±.004.143_{\pm.004}94.5±.294.5_{\pm.2}.020±.002.020_{\pm.002}.059±.003.059_{\pm.003}
Deep-Ens-LoRA88.1±.188.1_{\pm.1}.029±.002.029_{\pm.002}.125±.003.125_{\pm.003}88.4±.188.4_{\pm.1}.027±.002.027_{\pm.002}.130±.003.130_{\pm.003}94.9±.194.9_{\pm.1}.016±.002.016_{\pm.002}.050±.002.050_{\pm.002}
Gauss+Proj87.5±.287.5_{\pm.2}.034±.003.034_{\pm.003}.135±.004.135_{\pm.004}87.8±.287.8_{\pm.2}.033±.003.033_{\pm.003}.140±.004.140_{\pm.004}94.6±.294.6_{\pm.2}.018±.002.018_{\pm.002}.056±.003.056_{\pm.003}
SBA (Ours)87.9±.2‾\underline{87.9_{\pm.2}}.025±.002\mathbf{.025_{\pm.002}}.121±.003\mathbf{.121_{\pm.003}}88.2±.2‾\underline{88.2_{\pm.2}}.022±.002\mathbf{.022_{\pm.002}}.126±.003\mathbf{.126_{\pm.003}}94.8±.2‾\underline{94.8_{\pm.2}}.013±.001\mathbf{.013_{\pm.001}}.047±.002\mathbf{.047_{\pm.002}}
Table 2: In-distribution results on GLUE benchmarks. Results are mean ±\pm std over 5 seeds. Best result in each column is bolded, second best underlined. ↓\downarrow indicates lower is better.
Figure 2: Reliability diagrams on MNLI (in-distribution, top) and MNLI →\to SNLI (domain shift, bottom). SBA maintains calibration under shift while other methods degrade.
 MNLI →\to SNLIAmz →\to YelpSQD →\to AdvQA
 ECE ↓\downarrowSel. ↑\uparrowECE ↓\downarrowSel. ↑\uparrowECE ↓\downarrowSel. ↑\uparrow
LoRA.098±.006.098_{\pm.006}.712±.012.712_{\pm.012}.121±.007.121_{\pm.007}.684±.014.684_{\pm.014}.142±.008.142_{\pm.008}.658±.015.658_{\pm.015}
Laplace-LoRA.072±.005.072_{\pm.005}.768±.010.768_{\pm.010}.089±.005.089_{\pm.005}.742±.011.742_{\pm.011}.108±.006.108_{\pm.006}.718±.012.718_{\pm.012}
Gauss+Proj.068±.004.068_{\pm.004}.779±.009.779_{\pm.009}.084±.005.084_{\pm.005}.753±.010.753_{\pm.010}.102±.006.102_{\pm.006}.729±.011.729_{\pm.011}
Deep-Ens.058±.003.058_{\pm.003}.801±.007.801_{\pm.007}.071±.004.071_{\pm.004}.775±.008.775_{\pm.008}.088±.005.088_{\pm.005}.756±.009.756_{\pm.009}
Temp. Scal..064±.004.064_{\pm.004}—.079±.005.079_{\pm.005}—.097±.006.097_{\pm.006}—
Table 3: Domain shift results on LLaMA-2-7B: calibration (ECE ↓\downarrow ) and selective prediction (Sel. AUROC ↑\uparrow ). Mean ±\pm std over 5 seeds.
Figure 3: Selective prediction on MNLI →\to SNLI. SBA’s uncertainty estimates enable more effective abstention at all coverage levels.
Method20News (far)IMDB (near)
LoRA.841±.012.841_{\pm.012}.623±.018.623_{\pm.018}
Laplace-LoRA.892±.008.892_{\pm.008}.694±.014.694_{\pm.014}
Gauss+Proj.901±.007.901_{\pm.007}.712±.013.712_{\pm.013}
Deep-Ens-LoRA.921±.005.921_{\pm.005}.738±.010.738_{\pm.010}
SBA (Ours).934±.005\mathbf{.934_{\pm.005}}.761±.009\mathbf{.761_{\pm.009}}
Table 4: OOD detection AUROC ( ↑\uparrow ) using predictive entropy. In-distribution: SST-2 on LLaMA-2-7B. Mean ±\pm std over 5 seeds.
Figure 4: Uncertainty decomposition across settings. Epistemic uncertainty (mutual information) rises sharply under domain shift and on OOD data, while aleatoric uncertainty remains stable. SBA captures genuine model ignorance that flat-space methods underestimate.
κ0\kappa_{0}Acc ↑\uparrowECE ↓\downarrowBrier ↓\downarrow
0.189.2±.289.2_{\pm.2}.041±.003.041_{\pm.003}.134±.004.134_{\pm.004}
0.589.3±.289.3_{\pm.2}.032±.002.032_{\pm.002}.124±.003.124_{\pm.003}
1.089.4±.289.4_{\pm.2}.027±.002.027_{\pm.002}.118±.003.118_{\pm.003}
2.089.3±.289.3_{\pm.2}.027±.002.027_{\pm.002}.119±.003.119_{\pm.003}
5.089.2±.289.2_{\pm.2}.028±.002.028_{\pm.002}.120±.003.120_{\pm.003}
Table 5: Ablation: prior concentration κ0\kappa_{0} (MNLI, LLaMA-2-7B). Mean over 5 seeds.
SSECE ↓\downarrowSel. AUROC ↑\uparrowNLL ↓\downarrowRel. cost
1.038±.003.038_{\pm.003}.791±.009.791_{\pm.009}.412±.008.412_{\pm.008}1.0 ×\times
2.033±.002.033_{\pm.002}.808±.008.808_{\pm.008}.398±.007.398_{\pm.007}2.0 ×\times
5.029±.002.029_{\pm.002}.819±.007.819_{\pm.007}.389±.006.389_{\pm.006}5.0 ×\times
10.027±.002.027_{\pm.002}.821±.007.821_{\pm.007}.385±.006.385_{\pm.006}10.0 ×\times
20.026±.002.026_{\pm.002}.822±.007.822_{\pm.007}.384±.006.384_{\pm.006}20.0 ×\times
50.026±.002.026_{\pm.002}.822±.007.822_{\pm.007}.383±.006.383_{\pm.006}50.0 ×\times
Table 6: Ablation: number of posterior samples SS (MNLI, LLaMA-2-7B). Mean over 5 seeds.
Rank kkAcc ↑\uparrowECE ↓\downarrowParamsRel. cost
488.9±.388.9_{\pm.3}.033±.003.033_{\pm.003}0.26M1.02 ×\times
889.2±.289.2_{\pm.2}.029±.002.029_{\pm.002}0.52M1.04 ×\times
1689.4±.289.4_{\pm.2}.027±.002.027_{\pm.002}1.05M1.08 ×\times
3289.4±.289.4_{\pm.2}.026±.002.026_{\pm.002}2.10M1.15 ×\times
6489.5±.289.5_{\pm.2}.026±.002.026_{\pm.002}4.19M1.28 ×\times
Table 7: Ablation: adapter rank kk (MNLI, LLaMA-2-7B). Mean over 5 seeds.
Bayesian overAcc ↑\uparrowECE ↓\downarrowRel. cost
None (OrthoLoRA)89.3±.289.3_{\pm.2}.052±.003.052_{\pm.003}1.0 ×\times
Σ\Sigma only89.2±.289.2_{\pm.2}.043±.003.043_{\pm.003}1.02 ×\times
UU only89.3±.289.3_{\pm.2}.031±.002.031_{\pm.002}1.04 ×\times
U+VU+V89.4±.289.4_{\pm.2}.028±.002.028_{\pm.002}1.08 ×\times
U+Σ+VU+\Sigma+V (full)89.4±.289.4_{\pm.2}.027±.002.027_{\pm.002}1.08 ×\times
Table 8: Ablation: Bayesian treatment of different components (MNLI, LLaMA-2-7B). Mean over 5 seeds.
MethodECE ↓\downarrowSel. ↑\uparrowNLL ↓\downarrowCost
OrthoLoRA.052±.003.052_{\pm.003}.731±.011.731_{\pm.011}.452±.009.452_{\pm.009}1.0 ×\times
SBA ( SS =10).027±.002.027_{\pm.002}.821±.007.821_{\pm.007}.385±.006.385_{\pm.006}10 ×\times
SBA-Distilled.032±.002.032_{\pm.002}.798±.008.798_{\pm.008}.401±.007.401_{\pm.007}1.0 ×\times
Under shift (MNLI →\to SNLI):
OrthoLoRA.087±.005.087_{\pm.005}.731±.011.731_{\pm.011}—1.0 ×\times
SBA ( SS =10).049±.003.049_{\pm.003}.821±.007.821_{\pm.007}—10 ×\times
Table 9: Predictive distillation results (MNLI, LLaMA-2-7B). The distilled model retains most of SBA’s calibration at 1×1\times inference cost.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ParameterValue
Adapter rank kk16
Prior concentration κ0\kappa_{0}1.0
Prior scale τ\tau (for Σ\Sigma )0.1
Learning rate2×10−42\times 10^{-4}
Batch size16 (per GPU)
Epochs3
Appendix
Table 10: Hyperparameters for SBA and baselines.
More in this topic

Bayesian LoRA

Bayesian low-rank adaptation (LoRA) adds probabilistic parameter-efficient updates to estimate uncertainty while fine-tuning large models.

See all 17 papers

Explore similar work

CardsList

  1. Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

    Jan 28, 2026Moule Lin, Shuhao Guan, Andrea Patane +2Bayesian Neural NetworksConfidence Estimation in Language Models

    School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

  2. HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

    Mar 1, 2026Bartosz Trojan, Filip GębalaLow-Rank AdaptationLanguage Model Calibration

    Upper-Secondary Schools of Communications in Cracow

  3. Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent

    Sep 8, 2026Quang-Duy Tran, Trung Le, Bao Duong +2Stiefel Manifold OptimizationUncertainty Quantification

    Deakin Applied Artificial Intelligence Initiative, Deakin University, Geelong, Australia · Department of Computer Science, Aalto University, Espoo, Finland · Department of Data Science & AI, Monash University, Melbourne, Australia