Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancing

Joyanta Jyoti Mondal; Ibne Farabi Shihab

cs.LGSep 26, 2026

Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancing

Authors: Joyanta Jyoti Mondal, Ibne Farabi Shihab

Organizations: Department of Computer and Information Sciences, University of Delaware, USA · Department of Computer Science, Iowa State University, USA

Abstract

Post-hoc Bayesian fine-tuning places Gaussians around trained low-rank adapters, yet a calibrated posterior does not by itself yield a useful generalization certificate. Such a posterior admits an informative PAC-Bayes certificate only when both the loss of its sampled predictors and its KL divergence from an admissible prior are small. In this research, we characterize this certification frontier for Gaussian LoRA posteriors and separate three interventions: changing the prior, changing the stochastic predictor, and changing only how its complexity is counted. First, an exact isotropic KL envelope eliminates the prior scale and yields a width threshold that excludes posterior widths before sampling, while a zero-KL floor identifies targets that no complexity reduction can reach at a measured risk bound. Second, we minimize KL in closed form over the full $GL(r)$ symmetry of the low-rank factors, leaving every sampled adapter product unchanged, and derive the noncentral objective required when the prior center is trained on an independent split. On a small-pool RoBERTa audit of 567 configurations, the recorded 64-draw summaries imply a certificate floor of $0.7298$ even with zero KL and Chernoff accounting, so reducing complexity alone cannot certify these posteriors at the recorded budgets. For a stable posterior in a controlled Gaussian-factor task, Chernoff accounting certifies risk below $0.1$ on 20 of 20 datasets with 1024 draws, whereas Hoeffding certifies none. On deliberately deformed synthetic rank-four factors, matrix balancing reduces KL by $29.3\%$ on average beyond scalar balancing without changing any prediction. Numerical split-prior scenarios make the remaining risk and complexity budgets explicit.

Figures & tables

Figure 1: Overview. Stage I combines a Monte Carlo risk bound

U

for the Gaussian posterior

Q

with

\operatorname{KL}(Q\|P)+a_{k}

into the certificate

C_{k}

. Stage II locates

Q

on the risk–complexity frontier: (A) a width threshold applied before sampling; (B) at fixed

U

, the zero-KL floor and matrix balancing, which leaves every sampled

BA

unchanged. Decision : pass when the actual KL fits

J_{\tau}(U)

Draws	Configurations	Mean KL	Reported certificate	Accounting floor
8	567	12.49	$1$	$0.9320$
64	216	12.67	$0.736$ – $0.928$	$0.7298$

Table 1: Small-pool audit and accounting floors. The last column is the zero-KL floor with Chernoff accounting at the favorable

N=1024

, computed conservatively from rounded summaries; it is a lower bound, not an issued certificate.

Figure 2: Accounting cannot repair the small-pool posteriors. (a) Zero-KL certificate floor

C_{0}(U)

at the most favorable

N=1024

; the markers are the Chernoff risk floors implied by the recorded 8- and 64-draw summaries, and both floors lie above

\tau=0.5

. (b) Share of the 567 configurations for which the raw isotropic threshold excludes every width

q\in(0,1]

, before any posterior sampling; bar labels are counts.

Posterior width $q$	Draws $M$	Hoeffding certificate	Chernoff certificate
0.1	8	$0.777\,\pm\,0.005$	$0.721\,\pm\,0.008$
0.1	64	$0.326\,\pm\,0.003$	$0.215\,\pm\,0.007$
0.1	256	$0.189\,\pm\,0.003$	$0.104\,\pm\,0.006$
0.1	1024	$0.117\,\pm\,0.003$	$0.064\,\pm\,0.005$
4	8	$0.999\,\pm\,0.005$	$0.984\,\pm\,0.021$
4	64	$0.804\,\pm\,0.038$	$0.792\,\pm\,0.033$

Table 2: Controlled Gaussian-factor classification: mean

\pm

sample SD of the risk certificate across 20 independent datasets. The same posterior draws are used for both inequalities. The prior, trained center, KL, and confidence allocation are held fixed within each row.

Figure 3: Posterior behavior and Monte Carlo slack have different effects. (a) The stable posterior (

q=0.1

) benefits from more draws and from the Chernoff inequality, which crosses

\tau=0.1

M=1024

. (b) The unstable posterior (

q=4

) stays far above the target under both inequalities. Lines are means and bands

\pm 1

sample SD over 20 datasets (Table 2 ); both inequalities use the same posterior draws and simultaneous confidence family.

Rank	Raw KL	Scalar optimum	Matrix optimum	Extra reduction (%)
1	$61.75\,\pm\,16.60$	$39.04\,\pm\,1.26$	$39.04\,\pm\,1.26$	$0.0\,\pm\,0.0$
2	$120.40\,\pm\,22.29$	$117.59\,\pm\,19.87$	$68.35\,\pm\,1.31$	$40.6\,\pm\,8.4$
4	$190.76\,\pm\,18.60$	$186.95\,\pm\,16.82$	$131.21\,\pm\,1.55$	$29.3\,\pm\,5.6$
8	$332.50\,\pm\,15.37$	$329.86\,\pm\,15.22$	$256.67\,\pm\,2.33$	$22.1\,\pm\,3.1$

Table 3: Gaussian factor experiment,

m=12,n=8

. KL is in nats; entries are mean

\pm

sample SD over 20 seeds. The last column is the mean per-case percentage reduction relative to scalar balancing, not a ratio of pooled means. All three columns describe the same stochastic product.

Figure 4: Prediction-preserving balancing on deformed Gaussian factors (

m=12

n=8

; 20 seeds, error bars

\pm 1

sample SD). (a) Raw, scalar-balanced, and matrix-balanced KL describe the same stochastic product. (b) Mean per-case KL reduction of the matrix optimum beyond scalar balancing; the two optima coincide at rank one.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Data, predictor, and certificate
$S$ , $N$ , $\mathcal{D}$	certification sample, its size, and the data distribution
$\ell\in[0,1]$	bounded loss
$k\in\mathcal{K}$ , $w_{k}$	configuration and its prior weight, $\sum_{k}w_{k}=1$
$W_{0}^{(\ell)}$ , $B^{(\ell)}$ , $A^{(\ell)}$ , $s_{\ell}$	frozen matrix, LoRA factors, and scaling of adapted layer $\ell$
$r$ , $m$ , $n$	adapter rank and factor dimensions, $B\in\mathbb{R}^{m\times r}$ , $A\in\mathbb{R}^{r\times n}$

Appendix

Table 4: Notation.

Component	Setting
Backbone	RoBERTa-base and its tokenizer; fitted sequence-classification head
Training pools	Requested sizes $256,512,1024$ ; three seeds $0,1,2$ ; seven tasks
Sequence processing	Maximum 128 tokens; dynamic padding
Optimization	AdamW, base learning rate $5\times 10^{-5}$ , head multiplier 10; three epochs with a floor of 400 optimizer steps; batch size 16
Adapters	Query/value matrices; ranks $4,16,64$ ; scaling $\alpha=2r$ ; dropout zero; no adapter bias
Regularization	LoRA decay $10^{-3},10^{-2},10^{-1}$ ; other trained blocks have zero decay

Appendix

Table 5: Paired selector experiment. Requested training-pool sizes are capped by the available split. Confidence guarantees hold separately for each selection problem.

Selector on matched 21 problems	Accuracy	Effect relative to validation
Five-fold cross-validation	.7180	$+.0062\;[-.0059,.0209]$
Center-risk proxy	.7166	$+.0047\;[-.0046,.0132]$
Proxy minus cross-validation	–	$-.0014\;[-.0225,.0142]$

Appendix

Table 6: Secondary paired comparisons. Intervals are descriptive 95% task-cluster bootstrap intervals. Cross-validation rows use only their matched 21-problem grid. Native metrics re-score frozen accuracy-selected configurations; selection was not rerun with a different metric.

Selector	Center	Stochastic	Effect [95% interval]	Wins
Center-risk Gaussian-KL proxy	.7053	.5107	$-.0035\;[-.0209,.0070]$	5/7
Training risk	.7066	.5186	$-.0023\;[-.0206,.0086]$	5/7
Validation, retrained	.7088	.5152	$0$	–
BIC	.6883	.5235	$-.0205\;[-.0373,-.0002]$	1/7
Smallest rank	.6877	.5229	$-.0211\;[-.0375,-.0017]$	1/7

Appendix

Table 7: Selection on 63 problems. Accuracy is evaluated at the selected deterministic center and under its posterior. Effects are relative to validation with retraining, with descriptive 95% task-cluster bootstrap intervals. Wins count tasks, not individual configurations.

Figure 5: Selected models versus the certified predictor on the 63 small-pool problems. (a) Mean evaluation accuracy of each selector’s chosen configuration at its deterministic center and under its Gaussian posterior; the oracle uses test labels and is not an admissible selector. (b) Center-risk proxy scores on all 567 configurations and literal 64-draw certificates on the matched 216; every 8-draw certificate equals one.

Quantity	Center-risk proxy	Certificate, $M=8$	Certificate, $M=64$
Charged blocks	LoRA only	All trained	All trained
Mean KL (nats)	4.29	12.49	12.67
Risk input	Center	Upper bound $1$	Upper bound $.672$ – $.761$
Score or certificate	Mean $.1935$	Constant $1$	$.736$ – $.928$
Selected accuracy, matched 24	.7634	.7257	.7388

Appendix

Table 8: The original eight-draw certificate and a matched 64-draw rerun. The latter covers 216 configurations in 24 problems. Selected accuracies in the last row are compared only within this common subset; stochastic evaluation predictions were not recorded for the rerun.

Draws	Configurations	Chernoff risk UCB floor	Zero-KL certificate floor
8	567	$0.8941$	$0.9320$
64	216	$0.6662$	$0.7298$

Appendix

Table 9: Arithmetic lower bounds for the recorded LoRA grids. Each entry is a conservative lower bound, not a rerun mean or a certificate issued after setting KL to zero. The calculation grants the favorable

N=1024

, uses

w_{k}=1/9

and

\delta_{\rm PB}=\eta=.025

, and rounds downward.

Figure 6: Monte Carlo draws needed on the 216 recorded 64-draw configurations, with each configuration’s KL and observed mean held fixed and the Hoeffding bound at

K_{\rm MC}=9

. (a) Cumulative share certifiable at target

\tau

as a function of the required draws

M

; curves stop below 100% when no finite

M

suffices. (b) Binding constraint per target: no finite

M

suffices (observed mean not below

u_{\tau}

, or no

u_{\tau}

exists), more than 64 draws, or at most 64 draws.

Figure 7: Prior-scale-free width threshold

q_{\rm cert}(\tau)

over the 567 configurations (origin-centered prior, complete dimension). Boxes show quartiles and whiskers

1.5

interquartile ranges; values above the dashed line exclude every

q\in(0,1]

. The top row gives the excluded count; “maj.” uses each configuration’s training-pool majority error as its target.

Figure 8: Task-level selector effects relative to retrained validation, in percentage points of task-mean accuracy over the nine problems of each task (higher is better). WNLI, with 71 evaluation examples, drives the negative mean effect of the proxy and training-risk selectors.

Task	$N$	$D$	$\widetilde{r}$	$U_{\rm kl}$	$K_{I}$	$K_{R}$	$C_{R}$
SST-2	1,000	0.42	0.059	0.0973	525.00	490.35	0.5719
SST-2	4,000	0.46	0.066	0.1059	575.00	547.40	0.3321
SST-2	16,000	0.50	0.071	0.1121	625.00	601.25	0.2182
SST-2	33,000	0.54	0.076	0.1181	675.00	652.73	0.1928
QNLI	1,000	0.62	0.081	0.1241	775.00	713.77	0.7003
QNLI	4,000	0.70	0.089	0.1336	875.00	821.62	0.4314

Appendix

Table 10: Split-prior sample-size frontier.

\widetilde{r}

and

D

are assumed,

K_{R}

is an assumed attained matrix KL, and

U_{\rm kl}

and

C_{R}

are calculated. KL is in nats; risks are proportions. The same posterior losses apply before and after a valid pushforward.

Figure 9: Split-prior scenarios (assumed inputs; values are formula-derived, not measured). (a) Balanced certificate

C_{R}

and Chernoff risk bound

U_{\rm kl}

versus certification sample size at

p=q=.02

. (b) The same quantities versus posterior width at the largest

N

; the nonmonotone certificate follows the coarse prior-width grid. The dashed line is

\tau=0.2

Task	$q$	$p$	$\widetilde{r}$	$U_{\rm kl}$	$K_{I}$	$K_{R}$	$C_{R}$
SST-2	0.005	0.005	0.051	0.0873	10800.0	10568.3	0.4510
SST-2	0.010	0.010	0.058	0.0961	2700.0	2642.1	0.2543
SST-2	0.015	0.020	0.064	0.1035	10839.4	10606.9	0.4748
SST-2	0.020	0.020	0.076	0.1181	675.0	652.7	0.1928
SST-2	0.025	0.020	0.102	0.1489	9243.1	9044.9	0.5015
SST-2	0.030	0.040	0.161	0.2161	10333.2	10111.6	0.6000

Appendix

Table 11: Posterior-width curves. Nominal

N

is 33,000 for SST-2 and 70,000 for QNLI. Raw KL includes both displacement and covariance mismatch. All displayed certificate entries are recalculated from the row inputs.

Task	$K_{I}$	$K_{\rm scalar}$	$K_{R}$	Reduction (%)	$C_{I}$	$C_{R}$
SST-2	675.00	667.87	652.73	3.30	0.1942	0.1928
QNLI	1125.00	1110.96	1081.12	3.90	0.2270	0.2255

Appendix

Table 12: Noncentral-gauge comparison. Identity, scalar, and matrix KLs refer to one assumed stochastic predictor. The last columns are formula-derived certificates using the same

U_{\rm kl}

. Reduction is percent of the identity KL.

Task	$N$	$q_{\rm cert}$	$C_{0}$	$J$	$J-K_{R}$	Pass
SST-2	1,000	0.03143	0.1467	28.1	-462.3	0
SST-2	4,000	0.01611	0.1308	116.7	-430.7	0
SST-2	16,000	0.00828	0.1248	430.9	-170.3	0
SST-2	33,000	0.00591	0.1272	770.9	118.1	1
QNLI	1,000	0.03818	0.1779	9.3	-704.4	0
QNLI	4,000	0.01987	0.1609	49.1	-772.5	0

Appendix

Table 13: Frontier decisions at

\tau=.2

q_{\rm cert}

applies to the raw isotropic family,

C_{0}

is the complexity-free floor,

J

is the available KL budget, and

J-K_{R}

is the balanced margin. Pass is 1 exactly when the margin is nonnegative and

U_{\rm kl}\leq\tau

Task	$N$	Pilot mean	Projected $C$	Forecast	Final $C$	Pass
SST-2	1,000	0.056	0.5671	0	0.5719	0
SST-2	4,000	0.068	0.3356	0	0.3321	0
SST-2	16,000	0.068	0.2135	0	0.2182	0
SST-2	33,000	0.078	0.1957	1	0.1928	1
QNLI	1,000	0.078	0.6966	0	0.7003	0
QNLI	4,000	0.091	0.4345	0	0.4314	0

Appendix

Table 14: Pilot-to-final decision comparison. The pilot projection substitutes its assumed mean into the final-budget calculation and is not a pilot confidence certificate. Forecast and pass use the target

.2

Task	$N$	Reference error	Reference lower	$C_{R}$	Below lower
SST-2	1,000	0.4700	0.4221	0.5719	0
SST-2	4,000	0.4700	0.4461	0.3321	1
SST-2	16,000	0.4700	0.4581	0.2182	1
SST-2	33,000	0.4700	0.4617	0.1928	1
QNLI	1,000	0.4900	0.4419	0.7003	0
QNLI	4,000	0.4900	0.4660	0.4314	1

Appendix

Table 15: Fixed-label reference comparison. The reference error counts are assumed and their lower confidence bounds are calculated. The final indicator is 1 only when the Gaussian upper bound is below the reference lower bound.

Task	$N$	Bit depth	$L$	Decoded error	$C_{\rm code}$
SST-2	33,000	4	226,144	0.088	0.996054
SST-2	33,000	8	607,264	0.079	0.999999
SST-2	33,000	16	1,605,248	0.078	1.000000
QNLI	70,000	4	251,808	0.123	0.961728
QNLI	70,000	8	638,624	0.113	0.999462
QNLI	70,000	16	1,687,168	0.111	1.000000

Appendix

Table 16: Description-length comparator.

L

includes the complete assumed code in bits.

C_{\rm code}

is computed from the assumed decoded error and actual formula, not copied from the Gaussian column. Values rounding to 1.000000 are uninformative at displayed precision.

$q/p$	$q$	$\widetilde{r}$	KL	$C_{256}$	$C_{1024}$
$1/64$	0.15625	0.463	4324787.3	1.0000	1.0000
$1/32$	0.31250	0.474	3505952.0	1.0000	1.0000
$1/16$	0.62500	0.486	2688415.3	1.0000	1.0000
$1/8$	1.25000	0.495	1876072.8	1.0000	1.0000
$1/4$	2.50000	0.501	1084506.7	1.0000	1.0000
$1/2$	5.00000	0.504	376046.8	1.0000	1.0000

Appendix

Table 17: Small-pool replacement sweep. Every candidate width is explicitly numeric. The assumed draw means produce the displayed certificates at the indicated budgets; all-trained-coordinate Gaussian KL is computed exactly.

Task	$N$	Profiles	Draws	Final evaluations	Pilot evaluations
SST-2	33,000	23	1024	777,216,000	1,605,632
QNLI	70,000	23	1024	1,648,640,000	1,605,632

Appendix

Table 18: Calculated workload. The pilot column counts the full 49-profile pilot at 32 draws over 1024 examples. The final column counts complete-data posterior evaluations, not independent PAC-Bayes observations.

Bayesian LoRA

Bayesian low-rank adaptation (LoRA) adds probabilistic parameter-efficient updates to estimate uncertainty while fine-tuning large models.

See all 17 papers

Jan 28, 2026

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and

{\approx}1.2{\times}

training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Moule Lin, Shuhao Guan, Andrea Patane +2

School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

Bayesian Neural Networks Confidence Estimation in Language Models LLM Fine-Tuning Low-Rank Adaptation Language Model Calibration LLM Reliability+2

Ask PDF

Feb 19, 2026

Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning methods such as LoRA enable practical adaptation of large language models but provide no principled uncertainty estimates, leading to poorly calibrated predictions and unreliable behavior under domain shift. We introduce Stiefel-Bayes Adapters (SBA), a Bayesian PEFT framework that places a Matrix Langevin prior over orthonormal adapter factors on the Stiefel manifold

\St

and performs approximate posterior inference via tangent space Laplace approximation with geodesic retraction. Unlike Gaussian priors in flat space projected onto orthogonality constraints, our prior on the manifold naturally encodes the inductive bias that adapter subspaces should be well conditioned and orthogonal, while the posterior provides calibrated predictive uncertainty without recalibration. We prove formally that the tangent space approximation strictly avoids the structural variance inflation inherent in projecting from ambient space, establishing a rigorous theoretical advantage for intrinsic manifold inference. Across GLUE and SuperGLUE benchmarks on RoBERTa-large, LLaMA-2-7B, LLaMA-2-13B, Mistral-7B, and Qwen2.5-7B, domain shift evaluations, selective prediction protocols, and an abstractive summarization task, SBA achieves task performance comparable to LoRA and DoRA while reducing Expected Calibration Error by 18 to 34% over deterministic baselines, improving selective prediction AUROC by 12 to 25% under domain shift, and outperforming deep ensembles of five LoRA models on OOD detection at a fraction of the parameter cost. Our results demonstrate that where you place uncertainty, on the right geometric structure, matters more than simply adding any Bayesian treatment to adapters.

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University

Bayesian LoRA Parameter-Efficient Fine-Tuning Language Model Calibration Confidence Estimation in Language Models Bayesian Inference

Ask PDF

Aug 15, 2026

Spectral Rank Certification for Foundation Model Adapters

Nominal LoRA rank is a design parameter; calibrated spectral evidence is a separate inferential quantity. This article develops a finite-sample framework for inferring effective rank structure in public foundation-model adapters. The theoretical core is an exact chi-square divergence for the fixed-dimensional Gaussian rank-one reference experiment, with an unknown signal direction integrated under a rotation-invariant reference prior. The resulting series yields a computable finite-sample Le Cam bound at concrete layer sizes, an explicit remainder bound for numerical truncation, and the rectangular Baik-Ben Arous-Peche (BBP) limit. A compact-manifold Laplace expansion shows that finite-sample likelihood evidence also depends on leading spectral gaps through the factor

s_1^{|m-n|}\prod_{i\ge2}(s_1^2-s_i^2)

, motivating joint calibration of clustered singular values. Building on these results, we introduce an empirical-null workflow for PEFT LoRA adapters: factor reconstruction, Monte Carlo

p

-values, stagewise and block testing, and module-wise and corpus-level BH reporting. In an audit of 26 public adapters, 684 modules, six architecture families, and 31,770 public-checkpoint spectra rows, calibrated effective rank is typically much smaller than nominal rank and differs systematically from 95% energy retention. A measured RoBERTa-RTE slice on

n=24

examples illustrates the measurement path from calibrated ranks to task evaluation, without treating the slice as a utility study. The main empirical finding is that calibrated effective rank is usually far below nominal rank, and that energy retention and statistical surprise answer different questions.

Mohammed Ahnouch, Lotfi Elaachak

Université Paris 1 Paris, France · Faculty of Science and Technology of Tangier Abdelmalek Essaadi University Tangier, Morocco

Low-Rank Adaptation

Ask PDF

Abstract

Bayesian LoRA

Explore similar work

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning

Spectral Rank Certification for Foundation Model Adapters