We study variational inference in the presence of distributional symmetry, focusing on the interaction between two distinct group actions on a sample space: a symmetry action that leaves a reference distribution invariant, and a transport action that generates a parametric variational family by pushforward. By analyzing the algebraic structure of the total transformation group generated by both actions and the induced adjoint representation of the symmetry group on the transport Lie algebra, we establish that the Riemannian gradient of the Kullback--Leibler divergence at the true transport parameter is constrained to lie in the trivial representation of the symmetry group. Consequently, symmetry forces the gradient to vanish along all non-trivial representations. Combining this representation-theoretic constraint with parameter identifiability yields an orthogonal decomposition of the transport Lie algebra into three interpretable subspaces: an identifiability kernel (along which the objective is locally constant), a bias subspace (where the gradient may be nonzero due to model misspecification), and a symmetry-breaking subspace (where the gradient is forced to vanish by symmetry alone). Finally, we characterize the residual bias directions in terms of distributional mismatch between the base model and the standardized target, and derive an explicit Stein-type moment-matching condition that eliminates this bias, extending the location-scale results of (Margossian and Saul, 2025) to general Lie group transformation families.
Figures & tables
Example
Data Domain ( X )
Transport Group ( G )
Symmetry Group ( H )
Symmetry Action
1. Reflection
R
(R,+)
Z/2Z
Sign flip ( x↦−x )
2. Euclidean
Rd
(Rd,+)
O(d)
Rotations / reflections
3. Sphere
S2⊂R3
SO(3)
SO(2)
Axial rotations around e3
4. Covariance
Rd
GL(d)
O(d)
Orthogonal basis change
5. Square
R2
SO(2)
D4
90∘ rot. & axis flips
6. Permutation
Rn
(Rn,+)
Sn
Coordinate permutations
Table 1: Summary of sample spaces, transport groups, and symmetry groups for the six guiding examples.
Example
X
G
H
N=G∩H
Regime
gH
gbiasH
∇=0 ?
1. Reflection
R
(R,+)
Z/2Z
{e}
Separable
{0}
{0}
Yes
2. Euclidean
Rd
(Rd,+)
O(d)
{e}
Separable
{0}
{0}
Yes
3. Sphere
S2
SO(3)
SO(2)
SO(2)
Internal
span(e3)
{0}
Yes
4. Covariance
Rd
GL(d)
O(d)
O(d)
Internal
span(Id)
span(Id)
No
5. Square
R2
SO(2)
D4
C4
Partial
{0}
{0}
Yes
6. Permutation
Rn
(Rn,+)
Sn
{e}
Separable
span(1)
span(1)
No
Table 2: Algebraic invariants and Lie algebra decomposition g=n⊕gbiasH⊕g⊥ for the six guiding examples. The final column indicates whether ∇GL(g∗)=0 is guaranteed by symmetry alone ( gbiasH={0} ).
Variational inference (VI) is a central tool in modern machine learning, used to approximate an intractable target density by optimising over a tractable family of distributions. As the variational family cannot typically represent the target exactly, guarantees on the quality of the resulting approximation are crucial for understanding which of its properties VI can faithfully capture. Recent work has identified instances in which symmetries of the target and the variational family enable the recovery of certain statistics, even under model misspecification. However, these guarantees are inherently problem-specific and offer little insight into the fundamental mechanism by which symmetry forces statistic recovery. In this paper, we overcome this limitation by developing a general theory of symmetry-induced statistic recovery in variational inference. First, we characterise when variational minimisers inherit the symmetries of the target and establish conditions under which these pin down identifiable statistics. Second, we unify existing results by showing that previously known statistic recovery guarantees in location-scale families arise as special cases of our theory. Third, we apply our framework to distributions on the sphere to obtain novel guarantees for directional statistics in von Mises-Fisher families. Together, these results provide a modular blueprint for deriving new recovery guarantees for VI in a broad range of symmetry settings.
Daniel Marks, Dario Paccagnan, Mark van der Wilk
Department of Computer Science University of Oxford · Department of Computing Imperial College London
When approximating an intractable density via variational inference (VI) the variational family is typically chosen as a simple parametric family that very likely does not contain the target. This raises the question: Under which conditions can we recover characteristics of the target despite misspecification? In this work, we extend previous theoretical results on robust VI with location-scale families under target symmetries in two substantial ways: (1) We open them up to a wider range of divergences by providing sufficient conditions for exact recovery of the target mean and correlation matrix when using the forward Kullback-Leibler divergence and α-divergences. (2) By doing so, we find that we can drop the restrictive assumption of a log-concave target made in previous work, allowing us to give guarantees for a wider range of targets, including multi-modal ones. In our experiments, we show how our guarantees can serve as guidelines for the choice of the variational family and α-value and we illustrate on a diverse set of examples how and why optimization can fail in the absence of our sufficient conditions.
Otto's Wasserstein gradient flow of the inclusive (forward) Kullback--Leibler (KL) divergence offers a principled framework for analyzing statistical inference algorithms, yet algorithms targeting the exclusive (reverse) KL divergence are rarely studied with such tools. We establish a unified gradient-flow and PDF framework for inclusive KL inference. We show that maximum mean discrepancy minimization can be viewed as inclusive KL inference with an approximate gradient estimator, and we develop the Fisher--Rao and Wasserstein--Fisher--Rao gradient flows that directly target the inclusive KL divergence. Restricting these flows to the manifold of Gaussian distributions yields explicit gradient-flow ODEs, providing a foundation for Gaussian variational inference. Building on this viewpoint, we further introduce a local-estimator Wasserstein gradient flow whose velocity is obtained by local nonparametric regression, free of density-ratio evaluation or kernel gradients, improving the algorithmic performance over the MMD-based particle method.
Jia-Jie Zhu
Department of Mathematics, KTH Royal Institute of Technology, Lindstedtsvägen 25, 114 28, Stockholm, Sweden