Dynamics to decision: A mathematical theory of Lyapunov spectra and decision boundaries in deep classifiers

Shirin Panahi; Amirhossein Nazerian; Ali Pezeshki

cs.LGSep 30, 2026

Dynamics to decision: A mathematical theory of Lyapunov spectra and decision boundaries in deep classifiers

Authors: Shirin Panahi, Amirhossein Nazerian, Ali Pezeshki

Organizations: Department of Electrical and Computer Engineering Colorado State University Fort Collins, Colorado, USA · Department of Mechanical Engineering Colorado State University Fort Collins, Colorado, USA

Abstract

A deep classifier is defined not only by the decision it produces, but also by the sequence of transformations through which that decision is formed. Treating this evolution as a dynamical system across layers provides a natural framework for asking how decision geometry emerges through depth and how far back we can trace a boundary's dynamical signature. We model a feed-forward classifier as a finite, nonautonomous discrete dynamical system, with layers playing the role of discrete time steps. We study the Finite-Time Maximum Lyapunov Exponent (FTMLE) of the data samples' dynamical trajectory through depths of the classifier. The FTMLE measures the rate of convergence/divergence of nearby trajectories. We move the observation endpoint backward from probabilities to logits and then to hidden representations. For Gaussian classes, we prove that probability-level FTMLE carries a clear geometric signature of the decision boundary, with its dominant direction aligned with the boundary normal. Moving one step backward to the logits, we prove this relationship is no longer universal but depends critically on how the classifier is trained, particularly on the choice of loss function. Moving further backward to the hidden representation, the connection becomes more conditional: boundary-related FTMLE can persist, but only under identifiable structural conditions. We propose geometry-aware fine-tuning for restructuring the classifier's hidden FTMLE, and propose conditions for guaranteed concentration of high hidden FTMLE near the decision boundary. Through our numerical results, we show the generality and validity of our theoretical results. Understanding the evolution of data samples as traveling through the layers of classifier provides a principled foundation for identifying where boundary-relevant sensitivity emerges and for developing layer-aware regularization strategies.

Figures & tables

Figure 1 : Example of input-to-probability FTMLE of the binary classification. In all panels, the theoretical results are shown via a dashed black line, and the numerical evaluations of the learned NN model are shown via a solid red line. Panel a shows two 2D Gaussians, with the theoretical bisector and the learned NN boundary. Panel b shows the variation of the input-to-probability FTMLE $\lambda_{p}(x)$ of the learned NN over the 2D space. Panel c compares the FTMLE $\langle\lambda_{p}(x)\rangle_{r(x)}$ of the learned NN (solid red line), which has been averaged over the signed-distance $r(x)$ , with the analytically derived one (dashed black line). The vertical line at $r(x)=0$ denotes the Bayes boundary.

Figure 2 : Example of input-to-probability FTMLE of the multiclass classification. In panels a-d, the theoretical results are shown via a dashed black line, and the numerical evaluations of the learned NN model are shown via a solid red line. Panel a shows three 2D Gaussians, with the theoretical bisector and the learned NN boundary. Panel b shows the variation of the input-to-probability FTMLE $\lambda_{p}(x)$ of the learned NN over the 2D space. Panel c compares the FTMLE $\langle\lambda_{p}(x)\rangle_{r(x)}$ of the learned NN (solid red line), which has been averaged over the signed-distance $r(x)$ on normal slices through $a=0.35<a_{c}$ on the active class-1/class-2 facet, with the analytically derived one (dashed black line). The vertical line at $r(x)=0$ denotes the Bayes boundary. Panel d is similar to c, with the difference that the slice is at $a=0.4>a_{c}$ . Panel e is the zoomed black triangle shown in Panel b. Here, the geometry of the simplex model is shown as an equilateral triangle, with its vertices located at the population means. A higher-resolution contour reveals that as one traverses along the decision boundary, beginning at the + mark, $a=1/3$ , the boundary is an FTMLE minimum ridge for $1/3<a<a_{c}$ , and the boundary is an FTMLE maximum ridge for $a>a_{c}$ , with the critical point $a_{c}\approx 0.38$ (see Corollary 3.14 ).

Figure 3 : Example of geometry-aware fine-tuning. a: continued CE. b: geometry-aware fine-tuning. Darker markers identify samples with hidden expansion exceedance than their projected point on the boundary (points with $R_{H}(x)>10^{-6}$ ). The black curve denotes the decision boundary. In panel b, only 6 exceedances are observed with a very small distance to the boundary (order $10^{-4}$ ), while 924 samples in panel a have hidden expansion exceedances.

Figure 4 : Example of geometry-aware fine-tuning: Residual diagnostics, hidden and task-aligned expansion profiles, and robustness. In all panels, continued cross-entropy and geometry-aware fine-tunings are shown as solid blue and dashed red lines, respectively. a: task-visible residual $r_{T}$ and b: nuisance residual $r_{\perp}$ , across boundary-normal samples at each signed offset $t$ . The black horizontal reference on panel a is $r_{T}=1$ (Theorem 5.20 requirement). c: Only a positive value supports the rate condition of Theorem 5.20 . d and e: Curves show median values of the log-ratio between hidden and task-aligned expansion normal to the boundary (more negative values away from $t=0$ indicate a desired decrease in expansion further from the boundary). Shaded bands indicate the $10$ th– $90$ th percentiles. f: linear robustness radius is increased (as desired) through geometry-aware fine-tuning.

Explore similar work

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangDeep Network Rectified Linear Unit
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerFeature Learning Anisotropic Loss Landscapes
Institute of Physics and Astronomy, University of Potsdam
Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
May 13, 2026Yatin Dandi, Matteo Vilucchio, Luca Arnaboldi +2Feature Learning Deep Learning
Information Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

Jun 4, 2026, cs.LGJ/K move · Enter open · S save

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

Hu Tan, Kuo Gai, Shihua Zhang

1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China+1

Deep Network Rectified Linear Unit Weight Decay Grokking Clocks Cross-Entropy Losses+2

Open paper AskPDF arXiv

Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales. We formalize this phenomenon by separating the fast decay of the classification loss from the slower simplification of the learned representation, and we call the resulting pair of stopping times two training clocks. For deep linear networks, we show that a post-margin gap-growth or one-step tail-contraction condition reduces the cross-entropy loss to level epsilon on a logarithmic time scale. In contrast, when layerwise weight decay is present, the induced regularization on the end-to-end map can be expressed as a Schatten-type penalty; under a sharp late-time Kurdyka-Lojasiewicz tail, this structural energy closes on a polynomial time scale. The two clocks, therefore, separate fitting from representation simplification. We then explain how the same mechanism can appear in ReLU MLPs. In regions where the activation patterns on the training set remain fixed, the network reduces to a linear model in the active coordinates. In a two-layer ReLU embedding model, chain-rule estimates further show that the classifier head can receive larger effective gradients than the embedding block under controlled downstream norms. This supports a two-stage mechanism in which the classifier fits first, while the representation continues to simplify later. We use modular addition as the main experimental setting. The deep linear theory provides the rigorous core of the analysis. But the ReLU results are formulated as conditional reductions that account for empirical behavior without claiming a global proof for nonlinear training dynamics.

Abstract

Explore similar work

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks

Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning