The Fenchel-Young Generalization of Bayesian Principles

Sophia Sklaviadis; Thomas Moellenhoff; Emtiyaz Khan; Andre Martins; Mario Figueiredo

cs.LGFeb 14, 2025

The Fenchel-Young Generalization of Bayesian Principles

Authors: Sophia Sklaviadis, Thomas Moellenhoff, Emtiyaz Khan, Andre Martins, Mario Figueiredo

Organizations: Instituto de Telecomunicacoes, and Instituto Superior Técnico, Lisbon, Portugal · RIKEN AIP

Abstract

Conjugate Bayes plays a key role in machine learning, ensuring computationally simple posterior updates. We generalize the variational principle behind this computation by using the Fenchel-Young (FY) inequality, which lets us define a generalized evidence lower bound beyond the KL geometry, and at equality, characterizes new generalized posteriors. A generalized notion of conjugacy follows, defined as closure of prior-potential families that are affine under translation by the loss. We show that such families not only preserve additive canonical updating, but also posterior inference with parameter-dependent supports and tuneably heavy tails. This combination is not feasible with standard conjugate Bayes. We derive a new FY generalization of expectation maximization (EM), which recovers classical and Hard EM, and amortized versions. We show examples, including generalized Bayesian linear regression with closed-form bounded support and heavy-tailed posteriors, and a Gaussian mixture with sparse cluster assignments.

Figures & tables

Figure 1: By changing the Kullback-Leibler divergence underlying Bayesian principles to a Fenchel-Young discrepancy, we retain Bayes’ conceptual elegance but get generalizations with interesting properties. FY discrepancies induced by Tsallis negentropies lead to

q

-Gaussian posteriors, which can be closer or even at the MAP estimate without collapsing to a Dirac measure.

Figure 2: We extend Murphy [2022, Fig. 11.21] ’s Bayesian linear regression, to illustrate closed-form generalized conjugate posteriors with bounded support (orange) and heavy Student-

t

tails (green). Shaded bands indicate the central

95.45\%

intervals for future observations; the dotted curves correspond to the bounded predictive support.

Figure 3: We compare standard, sparse, and hard EM on GMM-based clustering. Data points are colored by their true labels; x indicates noise points. Colored ellipses denote the data-generating covariances (the same across the three panels). Black ellipses are level curves of the estimated GMM components and illustrate the effect of E-step sparsity on the fit. Sparse EM is less influenced by the outliers while retaining soft assignments.

E-Step	Adj. MI	Adj. RI	Silhouette
Standard	$.606\pm.006$	$0.531\pm.002$	$.345\pm.016$
Hard	$.537\pm.044$	$0.348\pm.022$	$.207\pm.064$
Sparse	$\textbf{.670}\pm.008$	$\textbf{.668}\pm.012$	$\textbf{.442}\pm.007$

Table 1: Clustering quality metrics (higher is better), for GMM clustering using standard EM, hard EM, and sparse FYEM with

q=0

. The mixing distribution

\pi_{0}

, and the Gaussian component parameters are learned with the shared M-step. Values are averaged over 5 random seeds

\pm

standard errors.

$q^{\prime}$	$q$	MNIST	Fashion MNIST
1	1	$13.236\pm.004$	$34.049\pm.019$
1	0.5	$12.065\pm.021$	$32.480\pm.018$
1	0	$12.086\pm.012$	$32.537\pm.025$
0	1	$12.567\pm.017$	$32.881\pm.027$
0	0.5	$9.261\pm.009$	$25.970\pm.086$
0	0	$\mathbf{9.144}\pm.020$	$\mathbf{25.851}\pm.072$

Table 2: VAE test

l_{1}

reconstruction error: mean

\pm

std. error over 5 seeds. The latent variable index

q

selects Gaussian (

q=1

), biweight (

q=0.5

), or Epanechnikov (

q=0

) posteriors. Observation models are Bernoulli (

q^{\prime}=1

) or sparse binary (

q^{\prime}=0

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: We vary the Tsallis entropic index

(2-q)\in\{0.1,0.5,0.9,1.0,1.1,1.5,2.0,3.0\}

and plot, from left to right the resulting adjusted mutual information, adjusted Rand index, and silhouette score. In the right-most panel, we quantify E-step sparsity by counting, for each observation, the number of components assigned zero posterior probability at the final iteration and then averaging across observations. Since each assignment vector sums to one and there are four components, the maximum zero count is three. The shaded regions represent standard errors over five seeds.

Generalized Bayesian Inference

See all 8 papers

The Bayesian Learning Rule Beyond KL Geometry
Oct 22, 2026Sophia Sklaviadis, Wu Lin, Andre Martins +3Bayesian Inference Variational Inference
Instituto de Telecomunicações and Instituto Superior Técnico, Universidade de Lisboa, Portugal · University of Central Florida, USA · RIKEN Center for Advanced Intelligence Project, Japan +1
A Generalization of Amari's Bayesian Duality
Sep 8, 2026Mohammad Emtiyaz Khan, Thomas MöllenhoffInformation Geometry Bayesian Inference
RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo, 103-0027, Tokyo, Japan. · Department of Computer Science, Technische Universität Darmstadt, Hochschulstraße 10, Darmstadt, 64289, Hessen, Germany. · The Hessian Center for Artificial Intelligence, Landwehrstraße 50a, Darmstadt, 64293, Hessen, Germany.
A Generalized-Bayes Perspective on Counterfactual Explanations: Posterior-Based Decision-Making and Evaluation
Jul 31, 2026Keita KinjoGeneralized Bayesian Inference Counterfactual Evaluation
Faculty of Business, Kyoritsu Women’s University

Oct 22, 2026J/K move · Enter open · S save

The Bayesian Learning Rule Beyond KL Geometry

Sophia Sklaviadis, Wu Lin, Andre Martins, Mario Figueiredo+2

Instituto de Telecomunicações and Instituto Superior Técnico, Universidade de Lisboa, Portugal · University of Central Florida, USA · RIKEN Center for Advanced Intelligence Project, Japan+1

Bayesian Inference Variational Inference Information Geometry Functional Bregman Divergences

Open paper AskPDF arXiv

The Bayesian learning rule (BLR) exploits the relation between Bayesian updating and the Kullback-Leibler divergence (KLD) to derive efficient additive updates for exponential family variational posteriors. The KL geometry is powerful but restrictive: combined with posterior families with parameter-dependent support it can lead to undefined objectives. We ask which non-KL geometries preserve the additive BLR computation. Previous works that use

f

-divergences suggest that this is only possible with the KLD. Here, we show that this is not true. We show that additive information-geometric optimization algorithms can be derived for the class of mixed-space Bregman or Fenchel-Young discrepancies. These discrepancies induce generalized maximum-entropy families, dual coordinates, and generalized-BLR canonical gradient updates, recovering the classical BLR. The generalized BLR leads to new algorithms that cannot be derived from the KL-based BLR. For the Tsallis negentropy with quadratic statistics, the generalized BLR yields bounded-support and heavy-tailed

q

-Gaussian posteriors and an easy Newton-like implementation. Experiments up to CIFAR-100 illustrate controllable support and tail behavior and competitive performance.

Abstract

Generalized Bayesian Inference

Explore similar work

The Bayesian Learning Rule Beyond KL Geometry

A Generalization of Amari's Bayesian Duality

A Generalized-Bayes Perspective on Counterfactual Explanations: Posterior-Based Decision-Making and Evaluation