Variational Learning with Lie Group Exponential Families
Authors: Eren Mehmet Kıral, Thomas Möllenhoff, Koichi Tojo, Mohammad Emtiyaz Khan, Kenichi Bannai
Organizations: RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan · Department of Mathematical Sciences, Tokai University, 4-1-1 Kitakaname, Hiratsuka-shi, Kanagawa 259-1292, Japan · Department of Computer Science, Technische Universität Darmstadt, RIKEN Center for Advanced Intelligence Project, Hochschulstraße 10, Darmstadt, 64289, Hessen, Germany · Department of Mathematics, Keio University, RIKEN Center for Advanced Intelligence Project, 3-14-1 Hiyoshi, Kohoku-ku, Yokohama, 223-8522, Japan
Variational Bayesian learning for deep neural networks offers many potential benefits. The difficulty in implementation arises from constraints on the parameters of the variational distributions, as well as the need for low-variance gradient estimators. Our recent work proposes Lie group methods for multiplicative or affine groups that naturally overcome such difficulties. Lie groups are essentially parameter spaces with a binary operator such as addition or multiplication, which allows for natural updates within the parameter space. In this article, we streamline and extend our Lie group framework to any general Lie group, and give an explicit relation to standard natural-gradient variational learning methods such as the Bayesian learning rule (BLR). In particular, we prove that linear approximations of the Lie group algorithm give generalizations of the BLR. The key ingredient is the harmonic exponential family, a general class of Lie group exponential families associated to representations of Lie groups. Our work provides a systematic method to construct from such representations efficiently implementable algorithms for variational learning.
Figures & tables
Figure 1: (Left) The distributions q1 and q2 are translates of each other and have the same entropy relative to the Lebesgue base measure dν(θ)=dθ . Therefore an algorithm optimizing for Eq. 1 will not take entropy into account in deciding between them while finding q^ . (Right) Two distributions on R>0 are related by q4(θ)=21q3(2θ) . Relative to the Lebesgue measure, q4 is spread out more and has a higher entropy, however both measures have the same entropy relative to the base measure dθ/θ . In this case the densities are given by q~j with q~j(θ)=θqj(θ) ( j=3,4 ) and q~4(θ)=q~3(2θ) . Therefore the solution to the Bayesian Learning problem in Eq. 1 depends on the base measure and it is the practitioner’s choice whether to bias the solutions q^ towards q4 ( dν(θ)=dθ ) or not ( dν(θ)=θdθ ).
Figure 2: The direction Y∈g is carried to TgG by the differential of Lg and that is carried to the tangent space TqgQ by dφg . We solve for Y=Y^ so that this tangent vector on Q is in the direction of fastest ascent according to the Fisher metric.
Figure 3: Examples of Lie groups acting on Θ and distributions on Θ . (top-left) Θ=R and G=(R,+) , with action given by g⋅θ=g+θ . The orbit of the pushforwards are various translations of q0 . (middle-left) Θ=R>0 and G=(R>0,×) with group action g⋅θ=gθ . Log-normal distributions with a fixed shape parameter, Rayleigh distributions, and exponential distributions can all be realized as Eq. 6 with this group action. (bottom-left) Θ=R and G=\Aff1+(R)={(A,b):A>0,b∈R} the affine group in one dimension, with the group action (A,b)⋅θ=Aθ+b . If q0 is the unit normal distribution then q(A,b)=N(b,A2) . (right) Θ=S2 and the special orthogonal group G=SO(3,R) acts on the sphere by rotations. von Mises–Fisher distributions with a fixed concentration parameter can be constructed as a single orbit under the pushforward action of this group.
Figure 4: The update vector is automatically modified so that the mean of the distribution takes larger steps in the direction of the uncertainty of the Gaussian, and smaller steps in direction of higher precision.
Figure 5: The orbit Ω(Q) of λ0∈Ω under the action of π∗ stays in Ω .
Figure 6: Composition of maps that turn the variational inference problem into an optimization problem near the group identity element. We optimize E(qπ∗(g)λ0) by taking a small step g↦gexp(−αY) for some Y∈g . We choose Y=Y^ as the direction of fastest ascent for E∘(ι∘N)∘Lg at e computed with respect to a bilinear form on g=TeG which is the pullback of the Fisher information metric.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The Lie group BLR can only move along Q , but inside the general exponential family \EF(α) the natural parameters can be updated in directions that have more degrees of freedom.
Instituto de Telecomunicações and Instituto Superior Técnico, Universidade de Lisboa, Portugal · University of Central Florida, USA · RIKEN Center for Advanced Intelligence Project, Japan +1
Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Computing and Mathematical Sciences, Caltech, CA, USA · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China
Department of Mathematics, Texas A&M University, College Station, TX, 77840, USA · Faculty of Mathematics, Technion- Israel Institute of Technology, Haifa, 3200003, Israel · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, 02139 USA