The Bayesian Learning Rule Beyond KL Geometry

Sophia Sklaviadis; Wu Lin; Andre Martins; Mario Figueiredo; Thomas Moellenhoff; Emtiyaz Khan

cs.LGOct 22, 2026

The Bayesian Learning Rule Beyond KL Geometry

Authors: Sophia Sklaviadis, Wu Lin, Andre Martins, Mario Figueiredo, Thomas Moellenhoff, Emtiyaz Khan

Organizations: Instituto de Telecomunicações and Instituto Superior Técnico, Universidade de Lisboa, Portugal · University of Central Florida, USA · RIKEN Center for Advanced Intelligence Project, Japan · TU Darmstadt and hessian.AI, Germany

Abstract

The Bayesian learning rule (BLR) exploits the relation between Bayesian updating and the Kullback-Leibler divergence (KLD) to derive efficient additive updates for exponential family variational posteriors. The KL geometry is powerful but restrictive: combined with posterior families with parameter-dependent support it can lead to undefined objectives. We ask which non-KL geometries preserve the additive BLR computation. Previous works that use $f$ -divergences suggest that this is only possible with the KLD. Here, we show that this is not true. We show that additive information-geometric optimization algorithms can be derived for the class of mixed-space Bregman or Fenchel-Young discrepancies. These discrepancies induce generalized maximum-entropy families, dual coordinates, and generalized-BLR canonical gradient updates, recovering the classical BLR. The generalized BLR leads to new algorithms that cannot be derived from the KL-based BLR. For the Tsallis negentropy with quadratic statistics, the generalized BLR yields bounded-support and heavy-tailed $q$ -Gaussian posteriors and an easy Newton-like implementation. Experiments up to CIFAR-100 illustrate controllable support and tail behavior and competitive performance.

Figures & tables

Figure 1: Left: We generalize the BLR by exploiting the geometry of Fenchel-Young discrepancies. We recover as special cases the BLR and the learning algorithms derived from it, and derive new algorithms using bounded-support and heavy-tailed

q

-exponential families. Right: Unlike Gaussians, bounded-support

q

-Gaussians allow control of the support by varying

q

. Thus, we can fit posteriors that closely track the MAP. The illustrated loss is that of 2D Bayesian logistic regression [ Murphy, 2012 ] , each circle indicates the first standard deviation, and outer loss contours indicate higher loss values. The Gaussian (

q=1

) fit is farther from the MAP because its full support assigns non-negligible mass to high-loss regions; bounded-support

q

-Gaussians truncate this exploration.

Figure 2

Figure 2: Bounded-support, mean-field variational posterior (red, blue) means can approach the posterior mode of features A7_0, A13_p and A7_ff, by changing the value of

q

in Bayesian logistic regression on the UCI Credit Approval dataset. This is useful when posterior marginals (grey) are skewed, and thus the empirical mode is badly approximated by a Gaussian posterior (black).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Method	Acc. $(\uparrow)$	NLL $(\downarrow)$	ECE $(\downarrow)$	Brier $(\downarrow)$	AUROC $(\uparrow)$
CIFAR-10 ResNet-20 270k params	SGD (MAP)	91.2 $\pm$ 0.1	0.292 $\pm$ 0.001	3.7 $\pm$ 0.1	0.133 $\pm$ 0.001	91.9 $\pm$ 0.1
	VSGD	91.9 $\pm$ 0.0	0.310 $\pm$ 0.005	4.5 $\pm$ 0.1	0.127 $\pm$ 0.001	92.1 $\pm$ 0.1
	IVON	92.4 $\pm$ 0.1	0.259 $\pm$ 0.006	3.5 $\pm$ 0.2	0.117 $\pm$ 0.002	92.4 $\pm$ 0.0
	+ sampling	92.6 $\pm$ 0.1	0.221 $\pm$ 0.003	0.8 $\pm$ 0.1	0.109 $\pm$ 0.002	92.6 $\pm$ 0.1
	q-VSGD ( $q<1$ )	91.9 $\pm$ 0.0	0.309 $\pm$ 0.008	4.5 $\pm$ 0.1	0.128 $\pm$ 0.002	92.0 $\pm$ 0.1
	q-VON ( $q<1$ )	92.3 $\pm$ 0.2	0.271 $\pm$ 0.008	3.9 $\pm$ 0.2	0.119 $\pm$ 0.003	92.6 $\pm$ 0.2

Appendix

Table 1: Results on convolutional neural networks. We evaluate various

q

-VON variants using the same hyperparameters used in Shen et al. [2024] for IVON. Even without tuning, the results are comparable to IVON, with the heavy-tailed

q

-VON-MF reaching slightly higher AUROC scores.

Task	Method	Acc. $(\uparrow)$	Test Loss $(\downarrow)$	ECE $(\downarrow)$	Brier $(\downarrow)$	AUROC $(\uparrow)$
MNIST MLP 550k params	SGD (MAP)	98.69 $\pm$ 0.01	0.043 $\pm$ 0.000	0.37 $\pm$ 0.02	0.020 $\pm$ 0.000	97.87 $\pm$ 0.07
	VSGD	98.75 $\pm$ 0.03	0.048 $\pm$ 0.000	0.60 $\pm$ 0.02	0.020 $\pm$ 0.000	97.68 $\pm$ 0.05
	IVON	98.74 $\pm$ 0.02	0.048 $\pm$ 0.001	0.60 $\pm$ 0.02	0.020 $\pm$ 0.000	97.52 $\pm$ 0.10
	+ sampling	98.72 $\pm$ 0.01	0.050 $\pm$ 0.000	1.53 $\pm$ 0.02	0.022 $\pm$ 0.000	97.88 $\pm$ 0.05
	q-VSGD ( $q<1$ )	98.62 $\pm$ 0.02	0.045 $\pm$ 0.001	0.46 $\pm$ 0.01	0.021 $\pm$ 0.000	97.94 $\pm$ 0.06
	q-VON ( $q<1$ )	98.69 $\pm$ 0.02	0.042 $\pm$ 0.000	0.40 $\pm$ 0.03	0.020 $\pm$ 0.000	98.04 $\pm$ 0.05

Appendix

Table 2: Additional results on MNIST. Heavy-tailed posteriors can improve generalization performance, whereas truncated ones can improve calibration.

Figure 3: Optimizer-specific convergence on 2D Bayesian logistic regression. Each panel shows the objective associated with an optimizer and posterior family. The plots show slices of the optimizer-specific generalized objective over one mean-dispersion coordinate, with the remaining coordinate fixed at its final value. The trajectories consistently approach low-objective regions for bounded-support, Gaussian, and Student-

t

variants, for both full and mean-field implementations. The same

\nu=5

in both Student-

t

panels corresponds to different

q

’s because of the dependence on

D

Figure 4: 1D Bayesian logistic regression

q

-sweep with

q

-VON.

Task	Method	Acc. $(\uparrow)$	Test Loss $(\downarrow)$	ECE $(\downarrow)$	Brier $(\downarrow)$	AUROC $(\uparrow)$
MNIST MLP 550k params	q-VON-MF ( $q=-10$ )	81.42 $\pm$ 15.77	0.514 $\pm$ 0.412	1.35 $\pm$ 0.59	0.208 $\pm$ 0.165	88.55 $\pm$ 6.79
	q-VON-MF ( $q=-8$ )	97.67 $\pm$ 0.03	0.086 $\pm$ 0.001	0.88 $\pm$ 0.01	0.036 $\pm$ 0.000	96.26 $\pm$ 0.18
	q-VON-MF ( $q=-6$ )	98.15 $\pm$ 0.08	0.070 $\pm$ 0.002	0.86 $\pm$ 0.04	0.029 $\pm$ 0.001	96.79 $\pm$ 0.12
	q-VON-MF ( $q=-4$ )	98.66 $\pm$ 0.03	0.053 $\pm$ 0.001	0.72 $\pm$ 0.01	0.022 $\pm$ 0.000	97.67 $\pm$ 0.12
	q-VON-MF ( $q=-2$ )	98.75 $\pm$ 0.01	0.048 $\pm$ 0.001	0.61 $\pm$ 0.03	0.020 $\pm$ 0.000	97.52 $\pm$ 0.05
	q-VON-MF ( $q=-1$ )	98.66 $\pm$ 0.02	0.047 $\pm$ 0.000	0.60 $\pm$ 0.02	0.021 $\pm$ 0.000	97.93 $\pm$ 0.07

Appendix

Table 3: MNIST

q

-sweep for

q

-VON-MF. Results are mean

\pm

s.e. over 5 seeds.

Bayesian Inference

Bayesian inference updates probability distributions over unknowns with evidence to quantify uncertainty and guide prediction and decisions.

See all 315 papers

The Fenchel-Young Generalization of Bayesian Principles
Feb 14, 2025Sophia Sklaviadis, Thomas Moellenhoff, Emtiyaz Khan +2Generalized Bayesian Inference Bayesian Inference
Instituto de Telecomunicacoes, and Instituto Superior Técnico, Lisbon, Portugal · RIKEN AIP
Exponential families from a single KL identity
Apr 30, 2026Marc DymetmanInformation Geometry Exponential Family
Scientific Consultant
Surprisal-Rényi Free Energy
Mar 3, 2026Shion Matsumoto, Raul Castillo, Benjamin Prada +1Large Deviation Principles Kullback-Leibler Divergence
Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, Florida, USA

Feb 14, 2025J/K move · Enter open · S save

The Fenchel-Young Generalization of Bayesian Principles

Sophia Sklaviadis, Thomas Moellenhoff, Emtiyaz Khan, Andre Martins+1

Instituto de Telecomunicacoes, and Instituto Superior Técnico, Lisbon, Portugal · RIKEN AIP

Generalized Bayesian Inference Bayesian Inference Variational Inference Expectation-Maximization

Open paper AskPDF arXiv

Conjugate Bayes plays a key role in machine learning, ensuring computationally simple posterior updates. We generalize the variational principle behind this computation by using the Fenchel-Young (FY) inequality, which lets us define a generalized evidence lower bound beyond the KL geometry, and at equality, characterizes new generalized posteriors. A generalized notion of conjugacy follows, defined as closure of prior-potential families that are affine under translation by the loss. We show that such families not only preserve additive canonical updating, but also posterior inference with parameter-dependent supports and tuneably heavy tails. This combination is not feasible with standard conjugate Bayes. We derive a new FY generalization of expectation maximization (EM), which recovers classical and Hard EM, and amortized versions. We show examples, including generalized Bayesian linear regression with closed-form bounded support and heavy-tailed posteriors, and a Gaussian mixture with sparse cluster assignments.

Abstract

Bayesian Inference

Explore similar work

The Fenchel-Young Generalization of Bayesian Principles

Exponential families from a single KL identity

Surprisal-Rényi Free Energy