Symmetry-quotient Flatness and Generalization

Taiki Miyagawa

cs.LGSep 10, 2026

Symmetry-quotient Flatness and Generalization

Authors: Taiki Miyagawa

Organizations: NEC Corporation, Japan

Abstract

This paper develops a theorem-level pipeline in symmetry-quotient settings: quotient linear stability implies quotient flatness, quotient flatness implies input smoothness, and input smoothness yields generalization under local covering assumptions. Flatness is often associated with generalization, and Stochastic Gradient Descent (SGD) is frequently viewed as implicitly biased toward flat solutions. However, standard flatness measures are typically defined in the raw parameter space and are therefore not invariant under function-preserving symmetries such as positive rescaling. We develop a symmetry-aware theory of quotient flatness, quotient linear stability, input smoothness, and generalization on quotient spaces of neural-network parameters. For square loss and models equipped with function-preserving group actions, we define quotient flatness as the trace of the Hessian of the empirical loss on the regular quotient manifold. We show that quotient flatness controls input smoothness through a quotient-space analogue of the flatness-to-smoothness argument. We also prove that one-step mean-square quotient linear stability of the linearized SGD dynamics implies an explicit quotient-flatness bound in terms of the batch size and learning rate, and extend this analysis to higher-order tensor moments. Finally, under local covering and boundedness assumptions, we derive population generalization bounds in terms of quotient flatness and, consequently, in terms of quotient linear stability.

Figures & tables

Figure 1: Raw versus symmetry-quotient loss landscapes for two-layer ReLU network trained on MNIST. The model has hidden width $128$ and admits the function-preserving positive-rescaling symmetry $(a_{r},w_{r})\mapsto(a_{r}/c,cw_{r})$ . The left panel shows a two-dimensional cross-entropy loss slice in the raw parameter space along random filter-normalized directions, whereas the right panel shows the corresponding quotient-space slice after removing the scale degree of freedom. The star denotes the trained solution. The figure illustrates how quotienting removes representation-dependent scaling effects from local loss-geometry visualization.

Figure 2: Flatness bound in Thm. 4.2 . A two-layer ReLU student network

f(x)=\sum_{r=1}^{m}a_{r}\operatorname{ReLU}(w_{r}^{\top}x)

with

m=10

neurons and input dimension

d=4

is trained on a teacher-generated regression dataset of

n=25

samples (teacher width

m_{\rm t}=4

). After training to near-interpolation with SGD (

\ell_{\rm train}<10^{-7}

), the quotient flatness

\mathrm{Flat}^{Q}(\theta^{*})

and the one-step mean-square quotient stability operator

\mathbb{E}_{I}[M_{I}^{2}]=I-2\eta\bar{A}+\eta^{2}\,\mathbb{E}[A_{I}^{2}]

are evaluated analytically for each combination of learning rate

\eta\in\{0.01,0.05,0.10,0.20,0.30,0.40,0.50,0.70,1.00\}

and batch size

B\in\{1,5,15\}

. Stability is assessed as

\lambda_{\max}(\mathbb{E}[M_{I}^{2}])\leq 1

, computed on the active subspace of

A^{Q}

(rows of

A^{Q}

projected onto its row space via SVD, threshold

10^{-6}\,\sigma_{1}

). Blue circles : stable

({\eta,B})

pairs, for which the theorem guarantees

\mathrm{Flat}^{Q}\leq 2B/\eta

(below the diagonal

y=x

). Red crosses : unstable pairs, for which the bound may be violated. All

11

stable pairs satisfy the bound, confirming the theorem.

Figure 3: Experimental verification of Thm. 5.1 . We test the quotient-flatness generalization bound in a teacher–student regression task with two-layer ReLU networks. In this two-layer specialization, the factor $\Lambda_{\star}^{2}C_{\mathrm{sec}}^{2}$ is evaluated by the explicit constant $\kappa_{\ast}\max(1,A_{\ast}^{2})$ . The teacher has input dimension $d=4$ and width $m_{t}=4$ ; the training set contains $n=25$ points sampled uniformly from $\mathcal{S}^{d-1}$ . Student networks of width $m\in\{6,8,10,12,15,20,28,40\}$ are trained by full-batch SGD with learning rate $0.1$ for up to $2\times 10^{5}$ steps, until $\hat{\mathcal{L}}\leq 5\times 10^{-5}$ . The test distribution $P_{X}$ is uniform on $\bigcup_{i=1}^{n}B(x_{i},\varepsilon)$ with $\varepsilon=0.15$ , so the $(\varepsilon,\delta,\nu)$ -balanced covering condition holds exactly with $\nu=1$ and $\delta=0$ . We use $4$ seeds per width; $28$ of $32$ runs reach interpolation. (a) For a representative run with $m=40$ , the training loss decreases monotonically, while the quotient flatness $\mathrm{Flat}^{Q}$ increases, indicating increasing quotient-space curvature during fitting. (b) Generalization error $\mathcal{E}_{\mathrm{gen}}$ versus $\mathrm{Flat}^{Q}$ on a log–log scale. Each point is one converged model, colored by $m$ . The solid curve is the bound ( 14 ) evaluated at median geometric constants. Specifically, $\kappa_{\ast}=\|U_{\ast}\|_{\mathrm{op}}^{2}$ , where the rows of $U_{\ast}$ are $u_{r}=w_{r}/\|w_{r}\|$ ; $A_{\ast}=\max_{r}|\alpha_{r}|$ , with $\alpha_{r}=a_{r}\|w_{r}\|$ ; $\rho=\min_{i}\|x_{i}\|=1$ ; and $M$ uniformly bounds $|F_{\theta}(x)|$ and $|f_{\ast}(x)|$ on $\operatorname{supp}(P_{X})$ . Empirically, $\kappa_{\ast}\in[3.6,14.5]$ with median $6.0$ , $A_{\ast}\in[0.26,0.79]$ with median $0.56$ , $\rho=1$ , and $M\approx 0.49$ . All points lie well below the bound. (c) Empirical input smoothness $\tfrac{1}{n}\sum_{i}\|\nabla_{x}F(x_{i})\|^{2}$ versus $\mathrm{Flat}^{Q}$ on a log–log scale. The bound $\kappa_{\ast}\max(1,A_{\ast}^{2})\rho^{-2}\mathrm{Flat}^{Q}$ holds for all models, verifying the intermediate implication from quotient flatness to input smoothness. (d) Direct verification of the theorem bound: empirical $\mathcal{E}_{\mathrm{gen}}$ versus the theorem right-hand side on a log–log scale. All $28$ converged models lie below the diagonal $y=x$ , confirming the bound.

Explore similar work

Jun 16, 2026cs.LG

A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks

We develop a mathematically explicit link between shock-wave theory and the symmetry-quotiented learning dynamics of stochastic gradient descent, drawing on differential geometry, Lie group theory, and fluid mechanics. Specifically, after quotienting parameter symmetries and applying local-entropy coarse-graining, the effective dynamics satisfy a viscous Hamilton--Jacobi equation on the quotient manifold. Moreover, under the assumption that the raw parameter dynamics can be summarized by a gradient field on the quotiented space, the gradient of the coarse-grained loss function obeys a Burgers-type equation, and shock formation can be established rigorously. We apply our theory to multilayer perceptrons, convolutional neural networks, Transformers, and mean-field networks, and show that they obey the Hamilton--Jacobi or Burgers-type equations. We conjecture that this framework also yields practical diagnostics for deep learning. In architectures such as Transformers, raw parameter norms are often distorted by symmetry redundancy and may therefore be misleading, whereas symmetry-corrected quotient observables provide a principled basis for monitoring, forecasting, and controlling training-phase transitions.

Taiki Miyagawa

NEC Corporation

Neural Network Training Dynamics Gradient Descent Dynamics Stochastic Gradient Descent

Ask PDF

Jun 18, 2026cs.LG

Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima

A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations that preserve the network function, which undermines the theoretical foundations of this narrative. In this study we resolve this issue by grounding flatness in the Riemannian geometry of the statistical manifold induced by the Fisher Information Matrix (FIM). We define Riemannian sharpness mathematically and prove that it is invariant under smooth, function-preserving reparametrizations, which directly addresses the critique of Dinh et al. in the paper ``Sharp minima can generalize for deep nets''.We note that this invariance is a property of the true FIM; the diagonal empirical estimator used in practice (and in all experiments below) inherits invariance only approximately, and exact invariance under arbitrary reparametrizations would require structured estimators such as K-FAC. We formalize the gradient noise of mini-batch SGD as having a covariance structure proportional to the FIM, derive the stationary distribution of the resulting stochastic differential equation, and then show that the probability mass is exponentially concentrated at Riemannian-flat minima. A PAC-Bayes generalization bound controlled explicitly by SR formally links this geometric bias to test performance. Our experiments on MNIST and CIFAR-10 confirm that SR reliably tracks generalization in ways that Euclidean sharpness does not, and that its scaling with

η/B

matches the theoretical predictions. Together these results provide a rigorous, reparametrization-invariant account of why flat minima generalize.

Md Sakir Ahmed, Kumaresh Sarmah, Hemen Dutta

Department of Electronics and Communication Technology Gauhati University Guwahati, Assam, India · Department of Mathematics Gauhati University Guwahati, Assam, India

Information Geometry Implicit Bias Flat Minima Loss Landscape Geometry Stochastic Gradient Descent

Ask PDF

Jun 3, 2026stat.ML

Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks

A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Schmidhuber, 1994; Keskar et al., 2017), where flatness can be measured by the trace of the Hessian of the empirical loss. However, Dinh et al. 2017) showed that, using symmetry of the network that can change flatness while keeping the population and empirical losses unchanged, any interpolator can be made sharper or flatter. This result makes the earlier heuristic statement vacuous. In this paper, we show that for learning an unknown multi-index model with

2

-layer non-convex homogeneous neural networks, there is a connection between flatness and generalization, despite the existence of symmetries. This connection pertains to the "flattest" interpolators, i.e., the interpolators that have orderwise minimum flatness among all interpolators. First, we show that there exists a natural class of non-generalizing interpolators whose flatness cannot be made closer to the flattest possible, even using symmetries. Second, we show that for data generated by a sum of single-index models, if the approximation error and label noise are low, any flattest interpolator achieves small population loss, i.e., the flattest interpolators always generalize. This establishes a direct link between flatness and generalization which applies to a large class of activations and realistic data distributions.

Harsh Vardhan, Hossein Taheri, Arya Mazumdar

Neural Network Generalization Flat Minima Statistical Learning Theory Feedforward Neural Networks Multi-Index Models

Ask PDF