Bayesian Hierarchical Models and the Maximum Entropy Principle

Brendon J. Brewer

stat.MLMar 10, 2026

Bayesian Hierarchical Models and the Maximum Entropy Principle

Authors: Brendon J. Brewer

Organizations: Department of Statistics, The University of Auckland

Abstract

Bayesian hierarchical models are frequently used in practical data analysis contexts. One interpretation of these models is that they provide an indirect way of assigning a prior for unknown parameters, through the introduction of hyperparameters. The resulting marginal prior for the parameters (integrating over the hyperparameters) is usually dependent, so that learning one parameter provides some information about the others. In this contribution, I will demonstrate that, when the prior given the hyperparameters is a canonical distribution (a maximum entropy distribution with moment constraints), the dependent marginal prior also has a maximum entropy property, with a different constraint. This constraint is on the marginal distribution of some function of the unknown quantities. The results shed light on what information is actually being assumed when we assign a hierarchical model.

Figures & tables

Figure 1: The orange distribution is the implied prior for the log of the arithmetic mean of 100 positive quantities, using a Uniform

(0,100)

distribution. The blue distribution is the implied prior using a hierarchical model with

\log\mu\sim\textnormal{Uniform}(-5,5)

, expressing more appropriate prior uncertainty about the arithmetic mean.

Figure 2: The orange distribution is the implied prior for the sum and sum of squares of 100 quantities with independent Uniform $(-100,100)$ priors. The blue distribution is the implied prior using a hierarchical model, expressing more appropriate prior uncertainty (approximately uniform in the horizontal direction and log-uniform vertically) about the sum and the sum of squares. The U-shape arises from the prior bounds for the hyperparameters. For completeness, these priors were $\mu\sim\textnormal{Uniform}(-100,100)$ and $\ln\sigma\sim\textnormal{Uniform}(-5,5)$ independently.

Hierarchical Bayesian Modeling

Hierarchical Bayesian modeling uses shared priors across groups to estimate population patterns and group-specific effects with uncertainty.

See all 36 papers

The Fenchel-Young Generalization of Bayesian Principles
Feb 14, 2025Sophia Sklaviadis, Thomas Moellenhoff, Emtiyaz Khan +2Generalized Bayesian Inference Bayesian Inference
Instituto de Telecomunicacoes, and Instituto Superior Técnico, Lisbon, Portugal · RIKEN AIP
Theoretical and Practical Advances in Hierarchical Bayesian Deep Learning Methodology
Oct 13, 2026Neil K. Chada, Simon Cotter, Kody J. H. Law +1Bayesian Neural Networks Hierarchical Bayesian Modeling
Department of Mathematics, City University of Hong Kong, Hong Kong SAR. · School of Mathematics, University of Manchester, UK.
Predictive variational inference: Learn the predictively optimal posterior distribution
Oct 18, 2024Jinlin Lai, Antonio Linero, Yuling YaoModel Misspecification Variational Inference
College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin

Feb 14, 2025J/K move · Enter open · S save

The Fenchel-Young Generalization of Bayesian Principles

Sophia Sklaviadis, Thomas Moellenhoff, Emtiyaz Khan, Andre Martins+1

Instituto de Telecomunicacoes, and Instituto Superior Técnico, Lisbon, Portugal · RIKEN AIP

Generalized Bayesian Inference Bayesian Inference Variational Inference Expectation-Maximization

Open paper AskPDF arXiv

Conjugate Bayes plays a key role in machine learning, ensuring computationally simple posterior updates. We generalize the variational principle behind this computation by using the Fenchel-Young (FY) inequality, which lets us define a generalized evidence lower bound beyond the KL geometry, and at equality, characterizes new generalized posteriors. A generalized notion of conjugacy follows, defined as closure of prior-potential families that are affine under translation by the loss. We show that such families not only preserve additive canonical updating, but also posterior inference with parameter-dependent supports and tuneably heavy tails. This combination is not feasible with standard conjugate Bayes. We derive a new FY generalization of expectation maximization (EM), which recovers classical and Hard EM, and amortized versions. We show examples, including generalized Bayesian linear regression with closed-form bounded support and heavy-tailed posteriors, and a Gaussian mixture with sparse cluster assignments.

Abstract

Hierarchical Bayesian Modeling

Explore similar work

The Fenchel-Young Generalization of Bayesian Principles

Theoretical and Practical Advances in Hierarchical Bayesian Deep Learning Methodology

Predictive variational inference: Learn the predictively optimal posterior distribution