Spectral Indistinguishability and Learning-Curve Identifiability in Quadratic Networks

Dekun Yang

cs.LGOct 14, 2026

Spectral Indistinguishability and Learning-Curve Identifiability in Quadratic Networks

Authors: Dekun Yang

Organizations: Zhejiang University, China

Abstract

Motivated by information-organization sculpting, we study spectral summaries and scalar learning curves in centered quadratic networks. We construct analytic positive input densities with Gaussian one- and two-coordinate marginals. Their complete fourth-moment operators are orthogonally intertwined by a map fixing the entire diagonal teacher algebra, giving equal anchored spectral measures. For selected common teachers with a nonzero cycle invariant, their factorized loss curves differ because the map does not preserve matrix multiplication. Cycle length $r$ , which can be arbitrarily large, gives $O(\varepsilon^r)$ contrast uniformly in training time for each fixed construction. For a supplied pair, noisy scalar loss queries require order $\varepsilon^{-2r}$ observations, while bounded initialization interventions with scalar gradient readouts require order $\varepsilon^{-2}$ . A shared-input score, equal to a paired-gradient difference, attains the optimal input-sample exponent without knowing the teacher. An explicit family of diverging auxiliary teacher scales instead admits a nonzero finite limiting contrast at coupling $1/\log L$ . Conversely, for Gaussian inputs and a rationally independent teacher spectrum, one exact scalar loss curve identifies the initial predictor up to sign conjugation. Within this rationally independent class, with nonzero diagonal inverse-state error in every coordinate and calibrated time and loss scales, it also identifies the unknown teacher spectrum and active dimension, modulo simultaneous rotations. For fixed dimension and distinct unordered teacher sums, dimension-minimal exact observations give generic joint local recovery. Finite exact global bounds hold for a known teacher and, on the active class, under a blind dimension cap. Exact certificates and full-weight replay verify the construction; numerical inverse checks reveal severe ill-conditioning.

Figures & tables

Ingredient	Established result	Dynamic statement proved here
Quadratic predictor dynamics	Matrix closure and isotropic Riccati solutions ( Liu Ziyin et al., 2026 )	A fixed initial factor develops different effective states under matched anchored spectra.
Signed-cycle invariants	Switching classes, cycle products and principal-minor recovery ( Urschel et al., 2017 ; Brunel and Urschel, 2024 )	One exact scalar loss trajectory determines those invariants for a rationally independent teacher; diagonal activity also permits blind spectrum recovery within the active class.
Finite observations	Nonlinear observability and quadratic principal-minor query counts ( Sontag, 2002 ; Brunel and Urschel, 2024 )	For fixed dimension and distinct unordered teacher sums, any dimension-minimal set of exact observations at distinct times gives generic joint local identification.
Testing difficulty	Gaussian divergence bounds and cycle-dependent DPP bounds ( Urschel et al., 2017 )	A loss-signal bound uniform over adaptive observation times.

Table 1: The contribution is in the learning dynamics and observation channel. The listed algebraic and statistical tools are prior results.

Symbol	Meaning and context
$A(x)$	Centered quadratic feature matrix.
$A,C$	Auxiliary and interaction coordinate sets in the paired-law construction.
$B_{\pm}$	Signed interaction adjacency matrices defining the two input laws.
$C=M_{0}^{-1}-T^{-1}$	Inverse-state error matrix, used in the Gaussian inverse problem.
$A_{\pm}$	Cycle matrices defining a supplied initial-state pair in the noisy inverse test.
$t,s$	Training time in the forward construction and Gaussian inverse problem, respectively.

Table 2: Notation across the construction and inverse problem. The coordinate set

C

and inverse-state matrix

C

occur in separate sections. Arguments and subscripts distinguish

A(x)

and

A_{\pm}

from the coordinate set

A

Figure 1: Spectral equivalence, finite training contrast and its scale dependence. a , Paired common-loss differences from complete weight flow and actual weight updates at the three fixed observation times. Lines joining the flow observations only guide the eye. b , Absolute errors between the common-loss contrasts of the original coupled dynamics and the derived limiting contrasts at four fixed times, as the auxiliary teacher scale grows. Some errors are nonmonotone. All four declared scales are shown. Both panels are deterministic checks without sampling error bars. The teacher in panel a was selected by the disclosed 108-cell exploration; panel b uses diverging teacher norms.

Figure 2: Finite identifiability does not ensure stable reconstruction. a , All six singular values of the analytic known-teacher observation Jacobian in the specified three-dimensional example. b , Relative parameter errors for every declared local inverse attempt, including the noise-free control. Each noise level uses the same seeded Gaussian direction; these are single deterministic realizations, not averaged performance estimates. Initialization is within

2\%

of the true parameter in every case.

Parameter Identifiability

Parameter identifiability studies when observations and model assumptions uniquely determine a system’s unknown parameters.

See all 103 papers

Jul 28, 2026

Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks

Positive quadratic networks admit the low-rank representation f_U(x)=x^top UU^top x, where Uinmathbb{R}^{dtimes r} is identifiable only up to right orthogonal multiplication, representing a rank-r PSD matrix Q=UU^top. We study how this quotient structure governs training dynamics, curvature, recovery, and interpolation bias. On the full-column-rank stratum, we identify mathbb{R}^{dtimes r}_*/O(r) with the rank-r PSD manifold. For smooth objectives L(U)=ell(UU^top), the Euclidean factor gradient is horizontal. Thus, factor gradient flow projects exactly to quotient Riemannian gradient flow, while finite-step gradient descent induces an exact congruence recursion for the predictor. For quadratic regression, we derive the effective Hessian at interpolators as the empirical measurement Gram form restricted to the tangent space relative to the quotient metric. Under Gaussian rank-one measurements, we compute population curvature, prove uniform deviation bounds for the empirical normal operator, construct a spectral initializer, and establish local exponential convergence for gradient flow and linear convergence for small-step descent. Recovery guarantees are explicit but conservative due to reliance on full-space second-moment control. In underdetermined commuting regimes, factor gradient flow becomes an exact entropy mirror flow in joint spectral coordinates. Strictly positive initializations converge to Bregman projections onto the interpolation set. With isotropic initialization q(0)=varepsilon^2mathbf{1}, predictors approach the minimum-trace solution set as varepsilondownarrow0, resolving nonuniqueness via weighted entropy within the invariant joint spectral algebra. Finite-step descent selects interpolants differing from continuous-time Bregman projections by O(eta). Numerical experiments verify these quotient identities, curvature predictions, recovery behaviors, and selection laws.

Pengcheng Cheng

School of Mathematics, Jilin University Changchun, 130012, China

Riemannian Optimization Implicit Regularization Gradient Descent Dynamics

Ask PDF

Sep 11, 2026

Algebraic Expressivity Certificates for Shallow Polynomial Neural Networks

We study exact representability by bias-free shallow polynomial neural networks using algebraic geometry. Over

\mathbb{C}

, a width-

r

network with activation

z\mapsto z^d

computes a sum of

r

d

-th powers of linear forms, whose Zariski closure is a Veronese secant variety. Ideal elimination therefore yields polynomial certificates of nonrepresentability. We implement this construction as a generic architecture-to-certificate pipeline. For quadratics, we recover the exact symmetric determinantal description and explain its dimension through orthogonal symmetry. In higher degree, the implementation recovers classical catalecticant and secant equations and maps the practical reach of direct elimination across a finite architecture sweep. We also derive the exact population loss floor for a rank-two quadratic network on the sphere, illustrating how an algebraic obstruction induces irreducible approximation error.

Sepehr Akbari, Shahrzad Jamshidi

Dept. of Math & Computer Science Lake Forest College

Neural Network Expressivity Shallow Neural Networks Polynomial Neural Networks Neural Network Approximation Theory

Ask PDF

Sep 9, 2026

Teacher Geometry Shapes Learnability in Teacher-Student Networks

Teacher-student systems, in which a teacher neural network generates training labels so that a student neural network can learn to implement the same function, are widely used as an abstract setting to study learning. However, the structure of the teachers is often overlooked by assuming randomly-generated, normally-distributed parameters. This hides substantial variation in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum, as a function of overparameterization, learning algorithm, student initialization distribution, and teacher geometry. We both identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show that these two distributions induce markedly different success rates across a large range of settings and for different activation functions. To explain the gap, we study the loss landscape of small neural networks that contain two distinct kinds of suboptimal local minima, out-of-bounds (OOB) minima at the edge of the data distribution and interior minima within. Assuming infinite data and a fast readout layer, we analytically reduce the loss landscape of small networks to two dimensions, showing that the region of attraction of interior minima changes as a function of teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more OOB minima. Motivated by these analyses, we show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases increases success rates. These findings provide an important step in narrowing the gap between the study of teacher-student networks and more structured functions that arise in practice.

Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2

Department of Experimental Psychology, University of Oxford · Department of Physics, RWTH Aachen

Neural Network Optimization Neural Network Training Dynamics Teacher-Student Learning Loss Landscape Geometry

Ask PDF

Abstract

Parameter Identifiability

Explore similar work

Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks

Algebraic Expressivity Certificates for Shallow Polynomial Neural Networks

Teacher Geometry Shapes Learnability in Teacher-Student Networks