Dynamics to decision: A mathematical theory of Lyapunov spectra and decision boundaries in deep classifiers

Shirin Panahi; Amirhossein Nazerian; Ali Pezeshki

cs.LGSep 30, 2026

Dynamics to decision: A mathematical theory of Lyapunov spectra and decision boundaries in deep classifiers

Authors: Shirin Panahi, Amirhossein Nazerian, Ali Pezeshki

Organizations: Department of Electrical and Computer Engineering Colorado State University Fort Collins, Colorado, USA · Department of Mechanical Engineering Colorado State University Fort Collins, Colorado, USA

Abstract

A deep classifier is defined not only by the decision it produces, but also by the sequence of transformations through which that decision is formed. Treating this evolution as a dynamical system across layers provides a natural framework for asking how decision geometry emerges through depth and how far back we can trace a boundary's dynamical signature. We model a feed-forward classifier as a finite, nonautonomous discrete dynamical system, with layers playing the role of discrete time steps. We study the Finite-Time Maximum Lyapunov Exponent (FTMLE) of the data samples' dynamical trajectory through depths of the classifier. The FTMLE measures the rate of convergence/divergence of nearby trajectories. We move the observation endpoint backward from probabilities to logits and then to hidden representations. For Gaussian classes, we prove that probability-level FTMLE carries a clear geometric signature of the decision boundary, with its dominant direction aligned with the boundary normal. Moving one step backward to the logits, we prove this relationship is no longer universal but depends critically on how the classifier is trained, particularly on the choice of loss function. Moving further backward to the hidden representation, the connection becomes more conditional: boundary-related FTMLE can persist, but only under identifiable structural conditions. We propose geometry-aware fine-tuning for restructuring the classifier's hidden FTMLE, and propose conditions for guaranteed concentration of high hidden FTMLE near the decision boundary. Through our numerical results, we show the generality and validity of our theoretical results. Understanding the evolution of data samples as traveling through the layers of classifier provides a principled foundation for identifying where boundary-relevant sensitivity emerges and for developing layer-aware regularization strategies.

Figures & tables

Figure 1 : Example of input-to-probability FTMLE of the binary classification. In all panels, the theoretical results are shown via a dashed black line, and the numerical evaluations of the learned NN model are shown via a solid red line. Panel a shows two 2D Gaussians, with the theoretical bisector and the learned NN boundary. Panel b shows the variation of the input-to-probability FTMLE $\lambda_{p}(x)$ of the learned NN over the 2D space. Panel c compares the FTMLE $\langle\lambda_{p}(x)\rangle_{r(x)}$ of the learned NN (solid red line), which has been averaged over the signed-distance $r(x)$ , with the analytically derived one (dashed black line). The vertical line at $r(x)=0$ denotes the Bayes boundary.

Figure 2 : Example of input-to-probability FTMLE of the multiclass classification. In panels a-d, the theoretical results are shown via a dashed black line, and the numerical evaluations of the learned NN model are shown via a solid red line. Panel a shows three 2D Gaussians, with the theoretical bisector and the learned NN boundary. Panel b shows the variation of the input-to-probability FTMLE $\lambda_{p}(x)$ of the learned NN over the 2D space. Panel c compares the FTMLE $\langle\lambda_{p}(x)\rangle_{r(x)}$ of the learned NN (solid red line), which has been averaged over the signed-distance $r(x)$ on normal slices through $a=0.35<a_{c}$ on the active class-1/class-2 facet, with the analytically derived one (dashed black line). The vertical line at $r(x)=0$ denotes the Bayes boundary. Panel d is similar to c, with the difference that the slice is at $a=0.4>a_{c}$ . Panel e is the zoomed black triangle shown in Panel b. Here, the geometry of the simplex model is shown as an equilateral triangle, with its vertices located at the population means. A higher-resolution contour reveals that as one traverses along the decision boundary, beginning at the + mark, $a=1/3$ , the boundary is an FTMLE minimum ridge for $1/3<a<a_{c}$ , and the boundary is an FTMLE maximum ridge for $a>a_{c}$ , with the critical point $a_{c}\approx 0.38$ (see Corollary 3.14 ).

Figure 3 : Example of geometry-aware fine-tuning. a: continued CE. b: geometry-aware fine-tuning. Darker markers identify samples with hidden expansion exceedance than their projected point on the boundary (points with $R_{H}(x)>10^{-6}$ ). The black curve denotes the decision boundary. In panel b, only 6 exceedances are observed with a very small distance to the boundary (order $10^{-4}$ ), while 924 samples in panel a have hidden expansion exceedances.

Figure 4 : Example of geometry-aware fine-tuning: Residual diagnostics, hidden and task-aligned expansion profiles, and robustness. In all panels, continued cross-entropy and geometry-aware fine-tunings are shown as solid blue and dashed red lines, respectively. a: task-visible residual $r_{T}$ and b: nuisance residual $r_{\perp}$ , across boundary-normal samples at each signed offset $t$ . The black horizontal reference on panel a is $r_{T}=1$ (Theorem 5.20 requirement). c: Only a positive value supports the rate condition of Theorem 5.20 . d and e: Curves show median values of the log-ratio between hidden and task-aligned expansion normal to the boundary (more negative values away from $t=0$ indicate a desired decrease in expansion further from the boundary). Shaded bands indicate the $10$ th– $90$ th percentiles. f: linear robustness radius is increased (as desired) through geometry-aware fine-tuning.

Explore similar work

Jun 4, 2026cs.LG

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales. We formalize this phenomenon by separating the fast decay of the classification loss from the slower simplification of the learned representation, and we call the resulting pair of stopping times two training clocks. For deep linear networks, we show that a post-margin gap-growth or one-step tail-contraction condition reduces the cross-entropy loss to level epsilon on a logarithmic time scale. In contrast, when layerwise weight decay is present, the induced regularization on the end-to-end map can be expressed as a Schatten-type penalty; under a sharp late-time Kurdyka-Lojasiewicz tail, this structural energy closes on a polynomial time scale. The two clocks, therefore, separate fitting from representation simplification. We then explain how the same mechanism can appear in ReLU MLPs. In regions where the activation patterns on the training set remain fixed, the network reduces to a linear model in the active coordinates. In a two-layer ReLU embedding model, chain-rule estimates further show that the classifier head can receive larger effective gradients than the embedding block under controlled downstream norms. This supports a two-stage mechanism in which the classifier fits first, while the representation continues to simplify later. We use modular addition as the main experimental setting. The deep linear theory provides the rigorous core of the analysis. But the ReLU results are formulated as conditional reductions that account for empirical behavior without claiming a global proof for nonlinear training dynamics.

Hu Tan, Kuo Gai, Shihua Zhang

1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1

Deep Network Rectified Linear Unit Weight Decay Grokking Clocks Cross-Entropy Losses+2

Ask PDF

Aug 6, 2026cond-mat.stat-mech

Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks

A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention. One of the corner stones of this development are analytically solvable toy models, allowing for the fully tractable analysis of the learning dynamics. Here we analytically investigate such a toy model using the regularization strength as a tunable external parameter - akin to external fields in statistical physics. In previous studies, (i) an onset of learning transition was predicted analytically and (ii) it was phenomenologically/numerically established that tuning the regularization strength can result in a cascade of phase transitions. The number of those transitions was linked to the geometry of the loss landscape determined by the model complexity. Setting up a rigorous framework underpinning the previous numerical observations, our investigation reveals a precise connection between those cascades of phase transitions, learnable features and the underlying geometry. We provide analytic predictions of these phase transitions as well as tractable order parameters related to learned features. At the level of the minimal model, we connect this macroscopic perspective (that can be condensed into an effective description) to the microscopic perspective in terms of the geometry of the loss landscape characterized by the Hessian spectrum. Thus, the presented model provides a platform to explore and sharpen advances made in the scientific theory of deep learning rooted in statistical physics concepts.

Björn Ladewig, Ibrahim Talha Ersoy, Karoline Wiesner

Institute of Physics and Astronomy, University of Potsdam

Feature Learning Anisotropic Loss Landscapes Deep Network Phase Transitions Regularization Deep Learning+3

Ask PDF

May 13, 2026cs.LG

Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity,and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.

Yatin Dandi, Matteo Vilucchio, Luca Arnaboldi +2

Information Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

Feature Learning Deep Learning

Ask PDF