Large Language Bayes Is Not Reparameterisation-Invariant

Jian Xu

cs.LGSep 24, 2026

Large Language Bayes Is Not Reparameterisation-Invariant

Authors: Jian Xu

Organizations: RIKEN

Abstract

Large Language Bayes (LLB) answers an informal modelling question by sampling candidate probabilistic programs from a language model, running approximate inference on each, and averaging them with weights proportional to an exponentiated evidence bound. We show that this weighting depends on how a model is written. The log marginal likelihood is invariant to reparameterisation; the evidence bound is not. On eight schools the centered and non-centered programs are the same measure to $5.7\times10^{-14}$ , yet their weights differ by $6.1\times$ ; importance weighting reduces this only to $2.2\times$ , and reproducing the inference LLB actually runs, a full-covariance Gaussian matched to the posterior moments, still leaves $1.9\times$ on eight schools and $8.9\times$ in $64$ dimensions. Across likelihood families, dimensions and funnel severities the discrepancy reaches $31.9\times$ and reverses sign, so no single writing is uniformly preferable. It inverts Bayes factors against eight natural competitors, and the induced error in the model posterior, and in any downstream target, is controlled by the spread $Δ$ of the bound shortfalls through a known sharp Hilbert-distance bound. Across $360$ programs from six language models the parameterisation written ranges from $0\%$ to $100\%$ centered and is stable within a model. Detecting equivalent programs statistically can falsely merge genuinely different models at practical sample budgets; verifying reparameterisations we generate ourselves cannot, and closes the window.

Figures & tables

Panel of Figure 1 — Figure 1: Where the inconsistency enters and where the repair goes, shown for the inference procedure LLB actually runs (Section 4.2 , $K=20$ ). Top: two programs encoding the same model, hence with identical $\log p(x)$ , receive different bounds, so a hypothesis is weighted by how it was written. Bottom: the repair generates reparameterisations from templates, verifies them against the change-of-variables identity, and keeps the tightest verified bound.

$K$	$1$	$5$	$20$	$100$	$500$	$2000$
centered $\mathcal{L}_{K}$	$-33.411$	$-32.849$	$-32.586$	$-32.372$	$-32.218$	$-32.116$
non-centered $\mathcal{L}_{K}$	$-31.605$	$-31.426$	$-31.369$	$-31.342$	$-31.330$	$-31.326$
gap (nats)	$1.806$	$1.422$	$1.217$	$1.030$	$0.888$	$0.790$
weight ratio	$6.08\times$	$4.15\times$	$3.38\times$	$2.80\times$	$2.43\times$	$2.20\times$

Table 1: Evidence bounds for two programs encoding the same model. The true

\log p(x)

is common to both; the non-centered bound has essentially converged by

K=500

, while the centered bound is still

0.81

nats short of the true value at

K=2000

. LLB weights are

\exp

of these numbers.

Panel of Figure 2 — Figure 2: Two programs encoding the same model, so the true $\log p(x)$ (dashed, by quadrature) is common to both. Left: the non-centered bound converges by $K\!=\!500$ ; the centered one is still $0.81$ nats short at $K\!=\!2000$ , and the interval between them is the reversal window. Right: the gap closes far more slowly than $O(1/K)$ , because the centered writing makes the weights heavy-tailed.

	eight schools ( $J=8$ )			synthetic ( $J=64$ )
$K$	centered	non-cent.	ratio	centered	non-cent.	ratio
$20$	$-31.983$	$-31.325$	$1.93\times$	$-242.130$	$-239.945$	$8.89\times$
$100$	$-31.751$	$-31.316$	$1.54\times$	$-241.430$	$-239.937$	$4.45\times$
$500$	$-31.633$	$-31.314$	$1.38\times$	$-241.224$	$-239.934$	$3.63\times$
$2000$	$-31.569$	$-31.313$	$1.29\times$	$-241.092$	$-239.932$	$3.19\times$

Table 2: LLB as published: bounds under a full-covariance Gaussian fitted to the posterior moments of each writing, rather than by mean-field reverse KL. The discrepancy shrinks but does not close, and grows with dimension. Three seeds; standard errors below

0.01

nats. The

J=64

problem uses synthetic data.

configuration	gap (nats)	weight ratio	$\hat{k}$ cent.	$\hat{k}$ non-cent.
Gaussian (eight schools), $J=8$	$1.806$	$6.09\times$	$0.90$	$0.64$
hierarchical logistic, $J=8$	$-0.734$	$0.48\times$	$0.74$	$0.83$
hierarchical Poisson, $J=8$	$-0.880$	$0.41\times$	$0.79$	$0.91$
Gaussian (synthetic), $J=8$	$2.026$	$7.58\times$	$0.91$	$0.72$
Gaussian (synthetic), $J=64$	$3.464$	$\mathbf{31.93\times}$	$\mathbf{1.02}$	$0.64$
Gaussian, $\sigma=4$	$-1.367$	$0.25\times$	$0.56$	$0.89$

Table 3: The bound gap across model families, dimension, funnel severity and hierarchy depth, with the Pareto shape

\hat{k}

of the importance weights. Gap is

\mathcal{L}_{1}^{\text{non-cent}}-\mathcal{L}_{1}^{\text{cent}}

, so a negative gap means the centered writing is tighter. Representative rows; the full sweep of twelve configurations is Table 7 .

Panel of Figure 3 — Figure 3: Left: the reversal window. LLB scores model $A$ at its centered bound (solid) rather than its true evidence (dashed); the eight filled competitors lie between the two and rank above $A$ despite being worse. Right: model-posterior error over random configurations never exceeds $\tanh(\Delta/4)$ (Theorem 3 ), which is attained; squares mark our measured widths.

model	provider	centered	non-centered	other	no hier.	centered share
gpt-4o-mini	OpenAI	$47$	$0$	$9$	$4$	$100.0\%$
gpt-4.1	OpenAI	$58$	$1$	$1$	$0$	$98.3\%$
gpt-5.1	OpenAI	$52$	$3$	$0$	$5$	$94.5\%$
gpt-5.5	OpenAI	$27$	$26$	$0$	$7$	$\mathbf{50.9\%}$
deepseek-v4-pro	DeepSeek	$50$	$7$	$1$	$2$	$87.7\%$
claude-opus-5	Anthropic	$0$	$57$	$0$	$3$	$\mathbf{0.0\%}$

Table 4: Parameterisation of

360

LLM-generated probabilistic programs. Which writing carries the looser bound depends on the model and data (Section 4.3 ); centered is the penalised one in the Gaussian-like, weakly identified regime where the gap is largest. “other” counts hierarchical programs matching neither pattern and “no hier.” programs with no hierarchy; both are excluded from the share.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Which parameterisation each language model writes, over $20$ completions on each of three hierarchical problems. The choice spans the full range from never to always, is stable within a model across problems, and softens but does not disappear with model recency.

competing model	$\log p(x\mid B)$	ranked above $A$ by LLB?
complete pooling	$-30.844$	(genuinely better)
independent effects, $\tau=1$	$-31.456$	yes, incorrectly
independent effects, $\tau=2$	$-31.459$	yes, incorrectly
independent effects, $\tau=3$	$-31.468$	yes, incorrectly
independent effects, $\tau=4$	$-31.487$	yes, incorrectly
independent effects, $\tau=5$	$-31.521$	yes, incorrectly

Appendix

Table 5: Competing models with analytic evidence, against the hierarchical model

A

(true

\log p(x\mid A)=-31.3113

by quadrature) as scored by LLB when written centered (

-32.116

K=2000

). Models in the window are truly worse than

A

but ranked above it.

	$J=8$			$J=32$			$J=128$
pair	$500$	$2000$	$8000$	$500$	$2000$	$8000$	$500$	$2000$	$8000$
equiv (cent./non-cent.)	$.10$	$.10$	$.05$	$.10$	$.00$	$.05$	$.15$	$.10$	$.00$
equiv (affine)	$.10$	$.10$	$.05$	$.10$	$.00$	$.05$	$.15$	$.10$	$.00$
$\tau$ scale $5$ vs $5.5$	$.20$	$.25$	$.55$	$.10$	$.25$	$.70$	$.10$	$.20$	$.95$
$\mu$ scale $5$ vs $5.5$	$.15$	$.15$	$.10$	$.10$	$.05$	$.55$	$.05$	$.25$	$.60$
Normal vs Student- $t(30)$	$.05$	$.05$	$.00$	$.10$	$.00$	$.05$	$.10$	$.10$	$.00$

Appendix

Table 6: Rejection rates at

\alpha=0.05

of a prior-predictive MMD test,

20

repetitions,

200

permutations. For equiv rows, rejecting is a harmless failure to merge; for the remaining rows, not rejecting is a harmful false merge. Normal versus Student-

t(30)

is never separated.

likelihood family ( $J=8$ )
configuration	gap (nats)	weight ratio	$\hat{k}$ cent.	$\hat{k}$ non-cent.
Gaussian (eight schools)	$1.806$	$6.09\times$	$0.90$	$0.64$
hierarchical logistic	$-0.734$	$0.48\times$	$0.74$	$0.83$
hierarchical Poisson	$-0.880$	$0.41\times$	$0.79$	$0.91$
dimension (Gaussian, synthetic data)
Gaussian (synthetic), $J=8$	$2.026$	$7.58\times$	$0.91$	$0.72$

Appendix

Table 7: The bound gap across model families, dimension, funnel severity and hierarchy depth, with the Pareto shape

\hat{k}

of the importance weights for each writing. Gap is

\mathcal{L}_{1}^{\text{non-cent}}-\mathcal{L}_{1}^{\text{cent}}

, so a negative gap means the centered writing is tighter. Two seeds per configuration. Representative rows appear as Table 3 in the body.

Bayesian Inference

Bayesian inference updates probability distributions over unknowns with evidence to quantify uncertainty and guide prediction and decisions.

See all 315 papers

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
Jan 28, 2026Moule Lin, Shuhao Guan, Andrea Patane +2Bayesian Neural Networks Confidence Estimation in Language Models
School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland
The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors
Sep 2, 2026Toni J. B. Liu, Jiajun Bao, Yizhou Liu +4Belief Updating in LLMs LLM Interpretability
Cornell University, USA · MIT, USA · Imperial College London, UK +1
Likelihood Hacking in Probabilistic Program Synthesis
Mar 25, 2026Jacek Karwowski, Younesse Kaddar, Zihuiwen Ye +2Probabilistic Programming LLM-Based Program Synthesis
University of Oxford, Department of Computer Science · University of Edinburgh, School of Informatics · CIFAR Fellow, Learning in Machines and Brains

Jan 28, 2026J/K move · Enter open · S save

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Moule Lin, Shuhao Guan, Andrea Patane, David Gregg+1

School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

Bayesian Neural Networks Confidence Estimation in Language Models LLM Fine-Tuning Low-Rank Adaptation Language Model Calibration LLM Reliability+2

Open paper AskPDF arXiv

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and

{\approx}1.2{\times}

training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Abstract

Bayesian Inference

Explore similar work

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors

Likelihood Hacking in Probabilistic Program Synthesis