Symmetries and Singularities

Vishnu Varadarajan; Mihir More; Aritra Das; Debayan Gupta

cs.LGSep 13, 2026

Symmetries and Singularities

Authors: Vishnu Varadarajan, Mihir More, Aritra Das, Debayan Gupta

Organizations: Truth Audit Labs

Abstract

Deep neural networks are highly over-parameterized, and different parameter values represent the same predictive function. This makes their effective complexity difficult to measure using only the number of parameters or the rank of the Hessian. Singular Learning Theory addresses this issue through the local learning coefficient (LLC), which characterizes the effective complexity of a model near a given solution. Existing methods for estimating the LLC often rely on posterior sampling, which can be computationally expensive for large neural networks. This makes accurate LLC estimation difficult at scale. In this work, we use known structures in the model to simplify the analysis and make LLC estimation more tractable. Specifically, we study the LLC of a graph attention model by exploiting symmetries in both the graph structure and the attention parameters. An analytic framework through a teacher--student setting, and explicit LLC estimates after considering the symmetry--induced degeneracies are developed.

Figures & tables

Symbol	Meaning
$G=(V,E)$	A finite graph
$\Gamma\leq\operatorname{Aut}(G)$	A chosen graph automorphism group, elements represented by $g$
$P_{g}$	Permutation matrix action of $g\in\Gamma$ on node-indexed arrays.
$H=E_{\eta}(G,X)$	Fixed encoder output in $\mathbb{R}^{\|V\|\times d}$ , with row $h_{i}^{\mathsf{T}}$ .
$A,B$	Notation for $W_{Q},W_{K}\in\mathbb{R}^{d\times k}$ .
$C=AB^{\mathsf{T}}$	Query–key composite in $\mathbb{R}^{d\times d}$ .

Table 1

Singular Learning Theory

Singular Learning Theory studies how non-identifiability and geometric singularities shape learning, Bayesian inference, and generalization.

See all 16 papers

Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity
Jun 19, 2026Tejas Pradeep Shirodkar, P. J. NarayananSingular Learning Theory Spectral Methods
IIIT, Hyderabad
Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks
Sep 1, 2026Osvaldo M Velarde, Lucas C Parra, Alireza Hashemi +1Neural Network Interpretability Neural Network Compression
Levich Institute and Physics Department, City College of New York, 160 Convent Ave, NY, 10031, NY, USA. · Biomedical Engineering Department, City College of New York, 160 Convent Ave, NY, 610101, NY, USA.
Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability
Jun 3, 2026Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin +1Neural Representation Geometry
Technical University of Munich; School of CIT, MCML, MDSI · MIT; Dept. of EECS, CSAIL

Jun 19, 2026J/K move · Enter open · S save

Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity

Tejas Pradeep Shirodkar, P. J. Narayanan

IIIT, Hyderabad

Singular Learning Theory Spectral Methods Statistical Learning Theory

Open paper AskPDF arXiv

Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities. The local learning coefficient (LLC), the standard estimator of Watanabe's real log canonical threshold (RLCT,

λ

), reads this geometry as an integrated Bayesian scalar through SGLD, which needs per-task calibration and

10^4

10^6

forward-backward passes per checkpoint. We introduce Dead-Direction Signatures (DDS), a family of cheap closed-form spectral readings of singular structure: each reads a network's activation matrix or per-sample-gradient Fisher-Gram at a chosen layer, replacing the SGLD posterior chain with spectral linear algebra. The readings rest on a dead-direction framework that predicts a structural correlation between activation- and Fisher-side spectra at any singular minimum, and a rank-multiplicative volume identity that single-eigenvalue monitors cannot produce: the active-volume

\log\det^{+}(G)

slope counts the dead directions, tracking the rank-deficit

r

across

r \in \{1,2,3,4\}

(slope ratios

2.0, 3.1, 4.0

r{=}2,3,4

against the predicted

2,3,4

), where the smallest eigenvalue is rank-blind. On reduced-rank regression with closed-form

λ

, calibrated LLC recovers

λ

99\%

mean and the DDS observables rank-track it at the framework-predicted sign; on a non-linear modular-addition transformer DDS separates

d_{\mathrm{model}}

across eighteen orders of magnitude where calibrated LLC at the protocol budget is rank-flat. Complementary to LLC's integrated posterior reading, DDS gives a directional, layer-local handle on a network's dead directions, read in closed form from its activation and gradient spectra.

Abstract

Singular Learning Theory

Explore similar work

Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity

Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks

Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability