Variational Learning with Lie Group Exponential Families
Authors: Eren Mehmet Kıral, Thomas Möllenhoff, Koichi Tojo, Mohammad Emtiyaz Khan, Kenichi Bannai
Organizations: RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan · Department of Mathematical Sciences, Tokai University, 4-1-1 Kitakaname, Hiratsuka-shi, Kanagawa 259-1292, Japan · Department of Computer Science, Technische Universität Darmstadt, RIKEN Center for Advanced Intelligence Project, Hochschulstraße 10, Darmstadt, 64289, Hessen, Germany · Department of Mathematics, Keio University, RIKEN Center for Advanced Intelligence Project, 3-14-1 Hiyoshi, Kohoku-ku, Yokohama, 223-8522, Japan
Variational Bayesian learning for deep neural networks offers many potential benefits. The difficulty in implementation arises from constraints on the parameters of the variational distributions, as well as the need for low-variance gradient estimators. Our recent work proposes Lie group methods for multiplicative or affine groups that naturally overcome such difficulties. Lie groups are essentially parameter spaces with a binary operator such as addition or multiplication, which allows for natural updates within the parameter space. In this article, we streamline and extend our Lie group framework to any general Lie group, and give an explicit relation to standard natural-gradient variational learning methods such as the Bayesian learning rule (BLR). In particular, we prove that linear approximations of the Lie group algorithm give generalizations of the BLR. The key ingredient is the harmonic exponential family, a general class of Lie group exponential families associated to representations of Lie groups. Our work provides a systematic method to construct from such representations efficiently implementable algorithms for variational learning.
Figures & tables
Figure 1: (Left) The distributions q1 and q2 are translates of each other and have the same entropy relative to the Lebesgue base measure dν(θ)=dθ . Therefore an algorithm optimizing for Eq. 1 will not take entropy into account in deciding between them while finding q^ . (Right) Two distributions on R>0 are related by q4(θ)=21q3(2θ) . Relative to the Lebesgue measure, q4 is spread out more and has a higher entropy, however both measures have the same entropy relative to the base measure dθ/θ . In this case the densities are given by q~j with q~j(θ)=θqj(θ) ( j=3,4 ) and q~4(θ)=q~3(2θ) . Therefore the solution to the Bayesian Learning problem in Eq. 1 depends on the base measure and it is the practitioner’s choice whether to bias the solutions q^ towards q4 ( dν(θ)=dθ ) or not ( dν(θ)=θdθ ).
Figure 2: The direction Y∈g is carried to TgG by the differential of Lg and that is carried to the tangent space TqgQ by dφg . We solve for Y=Y^ so that this tangent vector on Q is in the direction of fastest ascent according to the Fisher metric.
Figure 3: Examples of Lie groups acting on Θ and distributions on Θ . (top-left) Θ=R and G=(R,+) , with action given by g⋅θ=g+θ . The orbit of the pushforwards are various translations of q0 . (middle-left) Θ=R>0 and G=(R>0,×) with group action g⋅θ=gθ . Log-normal distributions with a fixed shape parameter, Rayleigh distributions, and exponential distributions can all be realized as Eq. 6 with this group action. (bottom-left) Θ=R and G=\Aff1+(R)={(A,b):A>0,b∈R} the affine group in one dimension, with the group action (A,b)⋅θ=Aθ+b . If q0 is the unit normal distribution then q(A,b)=N(b,A2) . (right) Θ=S2 and the special orthogonal group G=SO(3,R) acts on the sphere by rotations. von Mises–Fisher distributions with a fixed concentration parameter can be constructed as a single orbit under the pushforward action of this group.
Figure 4: The update vector is automatically modified so that the mean of the distribution takes larger steps in the direction of the uncertainty of the Gaussian, and smaller steps in direction of higher precision.
Figure 5: The orbit Ω(Q) of λ0∈Ω under the action of π∗ stays in Ω .
Figure 6: Composition of maps that turn the variational inference problem into an optimization problem near the group identity element. We optimize E(qπ∗(g)λ0) by taking a small step g↦gexp(−αY) for some Y∈g . We choose Y=Y^ as the direction of fastest ascent for E∘(ι∘N)∘Lg at e computed with respect to a bilinear form on g=TeG which is the pullback of the Fisher information metric.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The Lie group BLR can only move along Q , but inside the general exponential family \EF(α) the natural parameters can be updated in directions that have more degrees of freedom.
The Bayesian learning rule (BLR) exploits the relation between Bayesian updating and the Kullback-Leibler divergence (KLD) to derive efficient additive updates for exponential family variational posteriors. The KL geometry is powerful but restrictive: combined with posterior families with parameter-dependent support it can lead to undefined objectives. We ask which non-KL geometries preserve the additive BLR computation. Previous works that use f-divergences suggest that this is only possible with the KLD. Here, we show that this is not true. We show that additive information-geometric optimization algorithms can be derived for the class of mixed-space Bregman or Fenchel-Young discrepancies. These discrepancies induce generalized maximum-entropy families, dual coordinates, and generalized-BLR canonical gradient updates, recovering the classical BLR. The generalized BLR leads to new algorithms that cannot be derived from the KL-based BLR. For the Tsallis negentropy with quadratic statistics, the generalized BLR yields bounded-support and heavy-tailed q-Gaussian posteriors and an easy Newton-like implementation. Experiments up to CIFAR-100 illustrate controllable support and tail behavior and competitive performance.
Sophia Sklaviadis, Wu Lin, Andre Martins +3
Instituto de Telecomunicações and Instituto Superior Técnico, Universidade de Lisboa, Portugal · University of Central Florida, USA · RIKEN Center for Advanced Intelligence Project, Japan +1
Manifold-valued measurements are prevalent in various machine learning tasks. Recent advances have extended Deep Neural Networks (DNNs) to operate on manifolds, accompanied by normalization techniques tailored to different geometries, collectively referred to as Riemannian normalization. However, most existing Riemannian normalization methods are either designed for specific manifolds or fail to effectively normalize manifold-valued sample distributions. To address these limitations, we propose LieBN, a framework for Riemannian Batch Normalization (RBN) over Lie groups. Our approach leverages the theoretically convenient left- and right-invariant metrics, which naturally exist in every Lie group, and provides theoretical guarantees for controlling the Riemannian mean and variance. We instantiate LieBN across nine distinct geometries: four on the Symmetric Positive Definite (SPD) manifold, one on the group of rotation matrices, and four on the manifold of full-rank correlation matrices. Notably, among the SPD metrics, we introduce a novel right-invariant metric and extend three existing Lie group structures via matrix power deformation. Extensive experiments on different manifolds validate the effectiveness of our framework. The code is available at https://github.com/GitZH-Chen/LieBN.git.
Ziheng Chen, Yue Song, Rui Wang +2
Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Computing and Mathematical Sciences, Caltech, CA, USA · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China
The universal approximation theorem establishes that neural networks can approximate any continuous function on a compact set. Later works in approximation theory provide quantitative approximation rates for ReLU networks on the class of α-Hölder functions f:[0,1]N→R. The goal of this paper is to provide similar quantitative approximation results in the context of group equivariant learning, where the learned α-Hölder function is known to obey certain group symmetries. While there has been much interest in the literature in understanding the universal approximation properties of equivariant models, very few quantitative approximation results are known for equivariant models. In this paper, we bridge this gap by deriving quantitative approximation rates for several prominent group-equivariant and invariant architectures. The architectures that we consider include: the permutation-invariant Deep Sets architecture; the permutation-equivariant Sumformer and Transformer architectures; joint invariance to permutations and rigid motions using invariant networks based on frame averaging; and general bi-Lipschitz invariant models. Overall, we show that equally-sized ReLU MLPs and equivariant architectures are equally expressive over equivariant functions. Thus, hard-coding equivariance does not result in a loss of expressivity or approximation power in these models.
Jonathan W. Siegel, Snir Hordan, Hannah Lawrence +2
Department of Mathematics, Texas A&M University, College Station, TX, 77840, USA · Faculty of Mathematics, Technion- Israel Institute of Technology, Haifa, 3200003, Israel · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, 02139 USA