Quotient Geometry, Effective Curvature, and Implicit Bias in Simple Shallow Neural Networks
Authors: Hang-Cheng Dong, Pengcheng Cheng
Organizations: School of Instrumentation Science and Engineering, Harbin Institute of Technology Harbin, 150001, China · Harbin Institute of Technology Suzhou Research Institute Suzhou, 215000, China · School of Mathematics, Jilin University Changchun, 130012, China
Overparameterized shallow neural networks admit substantial parameter redundancy: distinct parameter vectors may represent the same predictor due to hidden-unit permutations, rescalings, and related symmetries. As a result, geometric quantities computed directly in the ambient Euclidean parameter space can reflect artifacts of representation rather than intrinsic properties of the predictor. In this paper, we develop a differential-geometric framework for analyzing simple shallow networks through the quotient space obtained by modding out parameter symmetries on a regular set. We first characterize the symmetry and quotient structure of regular shallow-network parameters and show that the finite-sample realization map induces a natural metric on the quotient manifold. This leads to an effective notion of curvature that removes degeneracy along symmetry orbits and yields a symmetry-reduced Hessian capturing intrinsic local geometry. We then study gradient flows on the quotient and show that only the horizontal component of parameter motion contributes to first-order predictor evolution, while the vertical component corresponds purely to gauge variation. Finally, we formulate an implicit-bias viewpoint at the quotient level, arguing that meaningful complexity should be assigned to predictor classes rather than to individual parameter representatives. Our experiments confirm that ambient flatness is representation-dependent, that local dynamics are better organized by quotient-level curvature summaries, and that in underdetermined regimes, implicit bias is most naturally described in quotient coordinates.
Figures & tables
Figure 1: Euclidean Hessian spectra across orbit-equivalent representatives (left) and projected Hessian spectra after removing scaling-orbit directions (right). The variation in the Euclidean spectrum reflects representation dependence, while the projected spectrum is invariant.
Figure 2: Parameter-space Hessian spectra vary across orbit-equivalent representatives (left), whereas Q -space Hessian spectra are numerically identical (right), confirming the intrinsic nature of quotient-level curvature.
Figure 3: Zoomed view of the small-eigenvalue region: Euclidean Hessian (left) changes under rescaling, while the projected proxy (right) remains stable, demonstrating that apparent flatness stems from symmetry-induced redundancy.
Figure 4: Pooled scatter plots of short-run empirical log-loss decay rates versus various curvature descriptors. Quotient-level quantities (e.g., Q -space trace and Frobenius norm) exhibit stronger correlation with local dynamics than ambient parameter-space condition numbers.
Figure 5: Gauge-dependent parameter complexity varies within a single symmetry orbit (left panel), while quotient-level complexity remains invariant (right panel), illustrating that meaningful complexity resides in the quotient object Q .
Figure 6: Across random seeds, raw parameter norms and path-like complexities show no coherent structure (left panels), whereas quotient-level quantities such as ∥Q∥F , ∥Q∥∗ , stable rank, and training loss reveal organized patterns, supporting a quotient-level view of implicit bias.
Figure 7: Learned singular spectra of Q across different random initializations. Substantial variation in spectral decay and effective rank demonstrates that implicit bias operates at the level of the quotient object.
Shallow neural networks use one or a few layers to learn functions, offering a setting to study efficient approximation, training dynamics, and limits of depth.
Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as reverse engineering," or parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the neuromanifold, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.
This paper develops a theorem-level pipeline in symmetry-quotient settings: quotient linear stability implies quotient flatness, quotient flatness implies input smoothness, and input smoothness yields generalization under local covering assumptions. Flatness is often associated with generalization, and Stochastic Gradient Descent (SGD) is frequently viewed as implicitly biased toward flat solutions. However, standard flatness measures are typically defined in the raw parameter space and are therefore not invariant under function-preserving symmetries such as positive rescaling. We develop a symmetry-aware theory of quotient flatness, quotient linear stability, input smoothness, and generalization on quotient spaces of neural-network parameters. For square loss and models equipped with function-preserving group actions, we define quotient flatness as the trace of the Hessian of the empirical loss on the regular quotient manifold. We show that quotient flatness controls input smoothness through a quotient-space analogue of the flatness-to-smoothness argument. We also prove that one-step mean-square quotient linear stability of the linearized SGD dynamics implies an explicit quotient-flatness bound in terms of the batch size and learning rate, and extend this analysis to higher-order tensor moments. Finally, under local covering and boundedness assumptions, we derive population generalization bounds in terms of quotient flatness and, consequently, in terms of quotient linear stability.
Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and artificial systems. Such inferences presume a meaningful link between representational geometry and the computation being performed. For artificial neural networks, however, the extent to which function constrains representation remains unclear. One key obstacle is that these networks admit parameter symmetries: changes in parameterization that preserve function exactly while reshaping representational geometry. Here, we show that a broad class of parameter symmetries acts on representations through just three primitive feature transformations: addition, duplication, and scaling. This feature-level characterization yields a closed-form decomposition of representational geometry into essential and auxiliary components, which makes precise how degeneracy in representational geometry can grow with overparameterization even when function is held fixed. Finally, we show that implementation-level selection rules can resolve this degeneracy, yielding identifiable geometries in which features are weighted according to their contributions to the network's function. Together, our results delineate when representations can support inferences about computation, and when they cannot.
Marvin Theiss, Lukas Braun, Andrew M. Saxe +1
University of Tübingen · International Max Planck Research School for Intelligent Systems · Allen Institute for Neural Dynamics +4