cs.LGNov 3, 2026

Variational Learning with Lie Group Exponential Families

Authors: Eren Mehmet Kıral, Thomas Möllenhoff, Koichi Tojo, Mohammad Emtiyaz Khan, Kenichi Bannai

Organizations: RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan · Department of Mathematical Sciences, Tokai University, 4-1-1 Kitakaname, Hiratsuka-shi, Kanagawa 259-1292, Japan · Department of Computer Science, Technische Universität Darmstadt, RIKEN Center for Advanced Intelligence Project, Hochschulstraße 10, Darmstadt, 64289, Hessen, Germany · Department of Mathematics, Keio University, RIKEN Center for Advanced Intelligence Project, 3-14-1 Hiyoshi, Kohoku-ku, Yokohama, 223-8522, Japan

Abstract

Variational Bayesian learning for deep neural networks offers many potential benefits. The difficulty in implementation arises from constraints on the parameters of the variational distributions, as well as the need for low-variance gradient estimators. Our recent work proposes Lie group methods for multiplicative or affine groups that naturally overcome such difficulties. Lie groups are essentially parameter spaces with a binary operator such as addition or multiplication, which allows for natural updates within the parameter space. In this article, we streamline and extend our Lie group framework to any general Lie group, and give an explicit relation to standard natural-gradient variational learning methods such as the Bayesian learning rule (BLR). In particular, we prove that linear approximations of the Lie group algorithm give generalizations of the BLR. The key ingredient is the harmonic exponential family, a general class of Lie group exponential families associated to representations of Lie groups. Our work provides a systematic method to construct from such representations efficiently implementable algorithms for variational learning.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. The Bayesian Learning Rule Beyond KL Geometry

    Oct 22, 2026Sophia Sklaviadis, Wu Lin, Andre Martins +3Bayesian InferenceVariational Inference

  2. LieBN: Batch Normalization over Lie Groups

    Jun 13, 2026Ziheng Chen, Yue Song, Rui Wang +2Geometric Representation LearningLie Group Methods

  3. Quantitative Approximation Rates for Group Equivariant Learning

    Feb 23, 2026Jonathan W. Siegel, Snir Hordan, Hannah Lawrence +2Permutation-Equivariant Neural NetworksEquivariant Neural Networks