Activation Saturation and Floquet Spectrum Collapse in Neural ODEs

Nikolaos M. Matzakos

math.DSApr 1, 2026

Activation Saturation and Floquet Spectrum Collapse in Neural ODEs

Authors: Nikolaos M. Matzakos

Organizations: Currently visiting the Department of Mathematics, Chair of Dynamics, Control, Machine Learning and Numerics, Friedrich-Alexander-Universität Erlangen–Nürnberg (FAU), 91058 Erlangen, Germany.

Abstract

We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs $\dot{h}=f_θ(h)$ with saturating activations ( $\tanh$ , sigmoid, etc.): if $q$ hidden layers of the MLP $f_θ$ satisfy $|σ'|\leδ$ on a region~ $U$ , the input Jacobian is attenuated as $\norm{Df_θ(x)}\le C(U)$ (for activations with $\sup_{x}|σ'(x)|\le 1$ , e.g.\ $\tanh$ and sigmoid, this reduces to $C_Wδ^q$ ), forcing every Floquet (Lyapunov) exponen along any $T$ -periodic orbit $γ\subset U$ into the interval $[-C(U),\;C(U)]$ . This is a collapse of the Floquet spectrum: as saturation deepens ( $δ\to 0$ ), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural -- it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with $σ'>0$ , a saturation-weighted spectral factorisation yields a refined bound $\widetilde{C}(U)\le C(U)$ whose improvement is amplified exponentially in~ $T$ at the flow level. All results are numerically illustrated on the Stuart--Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of $\tanh$ -NODEs on the Morris--Lecar neuron model.

Figures & tables

Figure 1: Jacobian attenuation (Theorem 3.3 ). (a) Spectral norm

\lVert Df_{\theta}\rVert_{2}

(solid blue) and upper bound

C_{W}(s)\cdot\delta(s)

(dashed red) as functions of the pre-activation scale

s

. The shaded region is the gap between the bound and the true norm; the bound is satisfied for all

s

. (b) Decomposition of the bound: the weight factor

C_{W}(s)=s\,\lVert W_{1}\rVert_{2}\,\lVert W_{2}\rVert_{2}

grows linearly in

s

, but the saturation factor

\delta(s)

decays exponentially, so their product still vanishes.

Figure 2: Phase portraits: exact Stuart–Landau vs. Neural ODE at increasing saturation. Left: exact Stuart–Landau flow (

\ln\det M=-4\pi

, strongly attracting limit cycle). Panels 2–4: bias-shifted 256-unit MLP at

s=1,4,15

. Dashed circle: Stuart–Landau limit cycle (

r=1

). Annotations show

\delta

\ln\det M

, and

\det M

. At

s=1

(

\delta\approx 0.53

\det M\approx 10^{-5}

) the MLP reproduces dissipative spirals; at

s=4

(

\delta\approx 0.005

\det M\approx 0.92

) convergence weakens visibly; at

s=15

(

\delta\approx 0

\det M=1

) trajectories are nearly rectilinear and the limit cycle no longer attracts.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Floquet–Liouville obstruction (Theorem 4.4 ). The obstruction bound

d\,C(U;\,s)\,T

|\ln\det M_{\gamma}|

as a function of pre-activation scale

s

, for four bias-offset values

c\in\{0,\,1.5c_{\min},\,3c_{\min},\,6c_{\min}\}

where

c_{\min}=\max_{j}\lVert W_{1j\cdot}\rVert_{2}

. Dotted line: the exact Stuart–Landau value

|\ln\det M_{\mathrm{SL}}|=4\pi

. The unbiased case (

c=0

, top curve) grows with

s

because zero crossings keep

\delta=1

. Each positive offset eliminates zero crossings; with

c=6c_{\min}

the bound falls below

10^{-2}

at moderate

s

Quantity	Numerical	Exact
$\mathrm{Tr}(Df_{\mathrm{SL}}(\gamma(t)))$	$-2.00000$	$-2$
$\int_{0}^{2\pi}\mathrm{Tr}(Df_{\mathrm{SL}})\,dt$	$-12.56637$	$-4\pi$
$\det M_{\gamma}=e^{\int\mathrm{Tr}}$	$3.4873\times 10^{-6}$	$e^{-4\pi}$
$\|\ln\det M_{\gamma}\|$	$12.566$	$4\pi=12.566$
Thm. 4.4 bound $d\,C(U)\,T$	$30.34$	$\geq 4\pi\;\checkmark$

Appendix

Table 1: LAJ identity for the Stuart–Landau oscillator. Numerical integration vs. exact values; all quantities computed on 1000 equispaced points of the unit-circle orbit. The last row verifies the bound of Theorem 4.4 (i):

d\,C(U)\,T=30.34\geq 4\pi=12.57=|\ln\det M_{\gamma}|

$s$	$\delta$	$C(U)$	$\|\mu_{1}\|$	$\|\mu_{2}\|$	$e^{-C(U)T}$	$e^{C(U)T}$
1	0.526	2.223	$2.30\times 10^{-5}$	$1.18$	$8.58\times 10^{-7}$	$1.17\times 10^{6}$
2	0.128	0.776	$7.86\times 10^{-2}$	$7.86\times 10^{-2}$	$7.64\times 10^{-3}$	$1.31\times 10^{2}$
4	0.005	0.024	$0.951$	$0.962$	$0.858$	$1.165$
7	${<}10^{-4}$	${<}10^{-3}$	$1.000$	$1.000$	$0.999$	$1.001$

Appendix

Table 2: Individual Floquet multiplier bounds (Theorem 4.5 ). Bias-shifted

256

-unit tanh MLP (Illustration C protocol). The transition matrix

\Psi(T)

along the reference Stuart–Landau orbit is computed via the variational equation for each

s

;

\mu_{1},\mu_{2}

are its eigenvalues (no trivial

\mu_{0}=1

is enforced since

\gamma

is not a periodic orbit of

f_{\theta}

). As

\delta\to 0

, both

|\mu_{i}|\to 1

Figure 4: Individual Floquet multiplier bounds (Theorem 4.5 ). The grey band is the allowed window

[e^{-C(U)T},\,e^{C(U)T}]

; blue and red curves are

|\mu_{1}|,|\mu_{2}|

. As

s

increases (

\delta\to 0

), the window collapses to

\{1\}

and both multipliers converge to unity.

$s$	Actual	Refined	Original	Ratio	$\delta$
1.0	1.612	2.125	2.482	1.2 $\times$	0.993
4.2	3.325	3.913	9.342	2.4 $\times$	0.885
7.4	3.210	3.636	12.935	3.6 $\times$	0.695
10.7	2.564	2.846	13.164	4.6 $\times$	0.494
13.9	1.845	2.016	11.330	5.6 $\times$	0.326
17.1	1.252	1.350	8.796	6.5 $\times$	0.206

Appendix

Table 3: Bound comparison at

h^{\star}

(Illustration E). Ratio

=

original / refined. The refined bound tracks the actual norm closely (within

{\sim}5\%

at strong saturation), while the original bound overestimates by up to

9\times

Figure 5: Refined bound verification (Illustration E). A

32

-unit

\tanh

MLP is trained at

s{=}1

; weights are frozen and

s

is swept from

1

30

. (a) Actual Jacobian norm, refined bound, and original bound on a log scale. (b) Improvement ratio (original / refined) grows monotonically to

9\times

s=30

Lyapunov Exponent Estimation

See all 8 papers

Neural ODEs Meet Concurrent Learning: Stable Online Learning with Lyapunov Guarantees
Sep 26, 2026Omkar Sudhir PatilNeural ODEs Neural Network Training Dynamics
Division of Electrical and Computer Engineering Louisiana State University Baton Rouge, LA 70803, USA
Between Amnesia and Chaos: A Memory Stability Expressivity Trilemma for Trainable Dissipative Oscillator Networks
Jun 7, 2026Caleb MunigetyDynamical Systems Lyapunov Stability
Independent Researcher
Long-Time Trajectory Approximation via SA-NODEs: Model Predictive and Floquet Strategies
Aug 11, 2026Ziqian Li, Nikolaos M. MatzakosDynamical Systems Neural ODEs
Chair for Dynamics, Control, Machine Learning and Numerics, Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany · School of Pedagogical & Technological Education (ASPETE), Marousi Attikis, 151 22 Athens, Greece

Sep 26, 2026J/K move · Enter open · S save

Neural ODEs Meet Concurrent Learning: Stable Online Learning with Lyapunov Guarantees

Omkar Sudhir Patil

Division of Electrical and Computer Engineering Louisiana State University Baton Rouge, LA 70803, USA

Neural ODEs Neural Network Training Dynamics Online System Identification Lyapunov Stability

Open paper AskPDF arXiv

Neural ODEs learn dynamics from trajectory losses, but their adjoint gradients lack the regressor-times-parameter-error structure on which Lyapunov analyses of online adaptation rest, so training on streaming data comes without stability guarantees. We show that this structure is in fact present: the adjoint gradient decomposes exactly into a positive semi-definite trajectory operator acting on the parameter error plus a nonlinear perturbation with explicit, horizon-dependent bounds. A quadratic Lyapunov function then certifies online Neural ODE training over sliding windows under computable gain and horizon conditions, and the same certificate extends to stored data: its drift branch recovers concurrent learning, and its trajectory branch yields NODE-CL, a stored-segment Gauss-Newton method built on batched forward sensitivities that needs no state-derivative estimates. On four DeepMind Control Suite domains, NODE-CL attains the lowest median prediction error on three under velocity measurement noise, where observer-based concurrent learning degrades by up to 8x; with clean measurements it is best on the pendulum and within a factor of 1.6 of the best stored-data baseline on the cartpole and reacher.

Abstract

Lyapunov Exponent Estimation

Explore similar work

Neural ODEs Meet Concurrent Learning: Stable Online Learning with Lyapunov Guarantees

Between Amnesia and Chaos: A Memory Stability Expressivity Trilemma for Trainable Dissipative Oscillator Networks

Long-Time Trajectory Approximation via SA-NODEs: Model Predictive and Floquet Strategies