Continuous-Time Piecewise-Linear Recurrent Neural Networks

Alena Brändle; Lukas Eisenmann; Florian Götz; Daniel Durstewitz

cs.LGFeb 17, 2026

Continuous-Time Piecewise-Linear Recurrent Neural Networks

Authors: Alena Brändle, Lukas Eisenmann, Florian Götz, Daniel Durstewitz

Organizations: Dept. of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Mannheim, Germany · Faculty of Physics and Astronomy, Heidelberg University · Hector Institute for Artificial Intelligence in Psychiatry, CIMH · Faculty of Mathematics and Computer Science, Heidelberg University ·

Abstract

In dynamical systems reconstruction (DSR) we aim to recover the dynamical system (DS) underlying observed time series. Specifically, we aim to learn a generative surrogate model which approximates the underlying, data-generating DS, and recreates its long-term properties (`climate statistics'). In scientific and medical areas, in particular, these models need to be mechanistically tractable -- through their mathematical analysis we would like to obtain insight into the recovered system's workings. Piecewise-linear (PL), ReLU-based RNNs (PLRNNs) have a strong track-record in this regard, representing SOTA DSR models while allowing mathematical insight by virtue of their PL design. However, all current PLRNN variants are discrete-time maps. This is in disaccord with the assumed continuous-time nature of most physical and biological processes, and makes it hard to accommodate data arriving at irregular temporal intervals. Neural ODEs are one solution, but they do not reach the DSR performance of PLRNNs and often lack their tractability. Here we develop theory for continuous-time PLRNNs (cPLRNNs): We present a novel algorithm for training and simulating such models, bypassing numerical integration by efficiently exploiting their PL structure. We further demonstrate how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models. We compare cPLRNNs to both their discrete-time cousins as well as Neural ODEs on DSR benchmarks, including systems with discontinuities which come with hard thresholds.

Figures & tables

Figure 1: Example reconstructions for the three core (ReLU-based) models: Ground truth trajectories and fixed points in black for Lorenz-63 system, and model-generated trajectories ( $M=20,\ P=10$ for all models) and fixed points found with SCYFI in red.

Model	$D_{\mathrm{stsp}}$ $\downarrow$	$D_{\text{H}}$ $\downarrow$	Prediction time $\uparrow$
Neural ODE (Euler) †	$0.42\pm 0.05$	$0.109\pm 0.01$	$0.8\pm 0.9$
Neural ODE (RK4)	$0.57\pm 0.29$	$0.12\pm 0.04$	$1.2\pm 0.8$
Neural ODE (Tsit5)	$0.30\pm 0.11$	$0.085\pm 0.019$	$\mathbf{1.4\pm 0.8}$
cPLRNN	$0.37\pm 0.14$	$0.116\pm 0.012$	$\underline{1.3\pm 0.8}$
standard PLRNN	$\underline{0.24\pm 0.06}$	$\underline{0.079\pm 0.01}$	$\mathbf{1.4\pm 0.9}$
SINDy	$\mathbf{0.17}$	$\mathbf{0.052}$	$\underline{1.3\pm 0.7}$

Table 1: Comparison of DSR performance between Neural ODE (with different ODE solvers), cPLRNN, and standard PLRNN (all for

P=10

; see Tab. 9 for other values of

P

), and for Latent ODE, Neural CDE, and SINDy, on the Lorenz-63 system (all models trained for

2000

epochs). Reported are geometrical (

D_{\mathrm{stsp}}

) and temporal (

D_{\text{H}}

) disagreement in the long-term limit, and valid prediction times, as median

\pm

median absolute deviation across

10\times 100

model trainings

\times

trajectories, except for

{}^{\dagger}\text{Neural ODEs integrated by Euler's method}

where too many runs diverged (leaving only 4-6 valid model runs). For Latent ODE & Neural CDE only the best value across all training runs is provided. Note that SINDy naturally performs best here because it already has the correct function library representing the Lorenz-63 system. Best value in bold, second-best underlined.

Model	$P=2$	$P=5$	$P=10$
Neural ODE (Euler)	$43.7\pm 2.3$	$42.4\pm 2.0$	$42.0\pm 1.9$
Neural ODE (RK4)	$156\pm 15$	$158\pm 12$	$154\pm 14$
Neural ODE (Tsit5)	$109\pm 4$	$112\pm 7$	$107\pm 5$
cPLRNN	$18.8\pm 2.9$	$25\pm 5$	$33\pm 5$
standard PLRNN	$4.65\pm 0.24$	$4.68\pm 0.25$	$4.72\pm 0.24$

Table 2: Training time comparison of Neural ODE, cPLRNN, and standard PLRNN for different numbers of PL units

P

over 1999 epochs (removing the first epoch to eliminate differences due to compile time). Shown are means

\pm

standard deviation [s]. Note that for comparability sequences in each batch were not run in parallel, but sequentially.

Figure 2: A) Top: LIF model (black) and trajectory generated by cPLRNN (red) with $M=25$ and $P=2$ . The subregion-switching times (gray) in the cPLRNN align well with the spiking times in the LIF model. Bottom: Limit cycle and fixed point found in cPLRNN trained on time series from LIF model. B) Top: Membrane potential recordings (black) and trajectory generated by cPLRNN (red) with $M=25$ and $P=6$ . Bottom: Limit cycle and fixed point found in cPLRNN trained on empirical data.

System	Regularly sampled			Irregulary sampled
Model	$D_{\text{stsp}}$ $\downarrow$	$D_{\text{H}}$ $\downarrow$	MAE $\downarrow$	$D_{\text{stsp}}$ $\downarrow$	$D_{\text{H}}$ $\downarrow$	MAE $\downarrow$
Neural ODE (Euler) †	$\mathbf{2.92\pm 0.23}$	$\mathbf{0.22\pm 0.05}$	$\mathbf{0.056\pm 0.018}$	$6.8\pm 1.7$	$\underline{0.27\pm 0.04}$	$0.08\pm 0.03$
Neural ODE (RK4)	${4.1\pm 0.8}$	${0.26\pm 0.04}$	$\underline{0.075\pm 0.024}$	$\mathbf{6.2\pm 1.4}$	$\underline{0.27\pm 0.04}$	$\underline{0.074\pm 0.014}$
Neural ODE (Tsit5)	${6\pm 3}$	${0.31\pm 0.06}$	${0.17\pm 0.08}$	${7.5\pm 0.9}$	$0.40\pm 0.09$	$0.12\pm 0.04$
cPLRNN	$3.5\pm 0.5$	$\underline{0.23\pm 0.03}$	${0.076\pm 0.014}$	$\underline{6.6\pm 1.0}$	$\mathbf{0.232\pm 0.024}$	$\mathbf{0.064\pm 0.014}$
standard PLRNN	$\underline{3.3\pm 0.6}$	$\underline{0.23\pm 0.05}$	${0.10\pm 0.04}$	$15.3\pm 0.7$	$0.50\pm 0.09$	$0.17\pm 0.05$

Table 3: Comparison of Neural ODE, cPLRNN, the standard PLRNN, SINDy, Neural CDE, and Latent ODE trained on regularly and irregularly sampled data from the LIF model, assessed after

2000

training epochs. Shown are median

\pm

MAD for geometrical (

D_{\mathrm{stsp}}

) and temporal (

D_{\text{H}}

) disagreement in limit behavior, and MAE for short-term prediction,

N=10

† except for Neural ODEs integrated by Euler where 4/10 (regular) and 2/10 (irregular) runs diverged and were excluded. For Latent ODE & Neural CDE only the best value across all runs is provided. For SINDy in the regular setting, overlap in distributions was

0

. Best value in bold, second-best underlined.

Model	$D_{\text{stsp}}$ $\downarrow$	$D_{\text{H}}$ $\downarrow$	MAE $\downarrow$
Neural ODE (RK4) †	$5.4\pm 1.5$	$0.493\pm 0.016$	$\underline{0.65\pm 0.03}$
Neural ODE (Tsit5)	$6.5\pm 2.0$	$0.514\pm 0.018$	$0.651\pm 0.010$
cPLRNN	$\underline{4.0\pm 0.6}$	$0.474\pm 0.006$	$\mathbf{0.627\pm 0.010}$
standard PLRNN †	$\mathbf{3.9\pm 0.5}$	$\underline{0.466\pm 0.017}$	$0.669\pm 0.011$
SINDy	$17.74$	$\mathbf{0.363}$	$1.618$
Neural CDE	$15.14$	$0.923$	$5.05$

Table 4: Performance comparison for Neural ODE, cPLRNN, the standard PLRNN, SINDy, Neural CDE, and Latent ODE trained on membrane potential recordings, assessed after

2000

training epochs. Reported are median

\pm

MAD for geometrical (

D_{\mathrm{stsp}}

) and temporal (

D_{\text{H}}

) disagreement in limit behavior, and MAE for short-term prediction, across

10

model trainings, † except diverging runs (2/10 for the standard PLRNN and 1/10 for Neural ODE with RK4; note that this biases results in favor of these models). For Latent ODE & Neural CDE only the best value across all training runs is provided. Explicit Euler always diverged on this stiff problem. For runs producing equilibira (thus flat power spectra),

D_{\text{H}}

was set to 1. Best values in bold, second-best underlined.

Figure 3: Test MSE for models trained on 10 subjects from PhysioNet heartbeat time series, sorted from left-to-right by number of available data points. Shown are medians

\pm

MAD across all out of 10 training runs that succeeded. Overall differences in medians across all subjects were statistically significant (

p<0.025

, Wilcoxon signed ranksum test across the 10 subjects).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Illustration of several roots in a non-bracketing interval.

Figure 5: Illustration of problem setting: for obtaining a global trajectory solution at arbitrary time points

t_{i}

we need to find the switching times

t_{s,k}

between linear subregions.

	Experiment
Hyperparameter	Lorenz-63	LIF regular	LIF irregular		Membrane potential
			continuous	standard
Latent dimension $M$	20	25	25	25	25
PL units $P$	[2, 5, 10]	2	2	2	6
Start learning rate	$1.0\times 10^{-3}$	$1.0\times 10^{-3}$	$1.0\times 10^{-3}$	$1.0\times 10^{-3}$	$4.0\times 10^{-3}$
Teacher forcing interval $\tau$	16	25	3	25	25

Appendix

Table 5: Model hyperparameters for all experiments

Hyperparameter	Neural ODE	cPLRNN	standard PLRNN
Optimizer	RAdam	RAdam	RAdam
Batch size	16	16	16
Batches per epoch	50	50	50
Epochs	2000	2000	2000
End learning rate	$1.0\times 10^{-5}$	$1.0\times 10^{-5}$	$1.0\times 10^{-5}$
Gradient clipping norm	0.0	10.0	0.0

Appendix

Table 6: Hyperparameters of training algorithm for all models

Panel of Figure 6 — Figure 6: (a) Time series from the ( Izhikevich, 2003 ) spiking neuron model (parameters $a=0.02$ , $b=0.2$ , $c=-65.0$ , $d=8.0$ , $I=10.0$ ), simulated as SDE with process noise of $\sigma=2.0$ . (b) Limit cycle and fixed point identified semi-analytically, using the methods in sect. 3.2, in a cPLRNN ( $P=2$ , $M=30$ ) trained on data irregularly sampled from this model.

Panel of Figure 7 — Figure 7: Scaling and robustness analysis for limit cycle detection. Experiments were conducted on a 22-region cycle based on the Lorenz system. Each data point represents 100 independent runs of the algorithm. Left: Crossing coordinates were perturbed additively as $x_{i}^{\prime}=x_{i}+\sigma_{x}s_{x}\xi_{i}$ , where $\xi_{i}\sim\mathcal{N}(0,I_{n-1})$ and $s_{x}$ is the global RMS magnitude of all reduced crossing coordinates. Right: Segment times of flight were perturbed multiplicatively as $\tilde{\tau}_{i}=\tau_{i}\exp(\sigma_t \eta_i)$ , with $\eta_{i}\sim\mathcal{N}(0,1)$ , yielding lognormal relative perturbations, $\log(\tilde{\tau}_i/\tau_i)\sim\mathcal{N}(0,\sigma_{t}^{2})$ .

Figure 8: Analysis of scaling behavior. Top: Mean runtime per epoch for one training run per setting. Left: Variation of

M

for fixed

P=2

; right: variation of

P

for fixed

M=40

. The visually apparent absence of any scaling with

P

for the Neural ODE integrated by forward-Euler was statistically confirmed by a lack of significant correlation (

p=0.2499

). Bottom: Runtime per epoch vs. total number of regions visited in the epoch for one training run with fixed

M=40

and varying

P

values.

Figure 9: Ratio of ill-conditioning matrices

W_{\Omega}

among the total number of encountered subregions (duplicates are counted separately) during one training run for different

P

values for fixed

M=40

. For fixed

P=2

and varying

M\in\Bqty{5, 10, 20, 30, 40}

, the ratio was always zero.

	$D_{\text{stsp}}$ $\downarrow$			$D_{\text{H}}$ $\downarrow$
Model	10% LIF	20% LIF	Izhikevich	10% LIF	20% LIF	Izhikevich
cPLRNN	$\mathbf{6.6\pm 1.0}$	$\mathbf{6.0\pm 2.0}$	$\mathbf{8.35\pm 0.11}$	$\mathbf{0.232\pm 0.024}$	$\mathbf{0.27\pm 0.05}$	$0.746\pm 0.003$
standard PLRNN	$15.3\pm 0.7$	$6.3\pm 0.8$	$10.3\pm 1.5$	$0.5\pm 0.09$	$0.34\pm 0.09$	$\mathbf{0.65\pm 0.11}$

Appendix

Table 7: Performance of cPLRNN vs. standard PLRNN for different percentages of available sampling points for the LIF example under irregular sampling, and performance comparison on another synthetic example, the Izhikevich neuron with irregular sampling from the cyclic regime (in this case

D_{\text{stsp}}

was evaluated in the true original 2d space). Values for

D_{\text{stsp}}

and

D_{H}

are given as median

\pm

MAD.

Measure	Neural ODE (Euler)	Neural ODE (RK)	Neural ODE (Tsit5)	cPLRNN	standard PLRNN
MSE (test) $\downarrow$	$4e^{3}\pm 6e^{3}$	$1.57\pm 0.15$	$1.8\pm 0.3$	$\mathbf{1.512\pm 0.018}$	$2.96\pm 0.15$

Appendix

Table 8: Performance comparison on irregularly sampled heartbeat time series from the PhysioNet/Computing in Cardiology Challenge 2012 (https://physionet.org/content/challenge-2012/1.0.0/). The subject with the most data points was chosen for this comparison, and the time series delay-embedded (

\tau=1

hour,

d=3

), normalized and split into a training (80%) and test trajectory (20%), with MSE evaluated on the test set. Values for cPLRNN vs. standard PLRNN are across 5 runs (where for the standard PLRNN linear interpolation to full hours was used, see sect. 4). Neural ODE trained by forward-Euler quickly diverged on this problem, leading to the large values.

	$P=2$		$P=5$		$P=10$
Model	$D_{\mathrm{stsp}}$	$D_{\text{H}}$	$D_{\mathrm{stsp}}$	$D_{\text{H}}$	$D_{\mathrm{stsp}}$	$D_{\text{H}}$
Neural ODE (Euler) †	$14.7\pm 0.7$	$0.66\pm 0.07$	$1.9\pm 0.29$	$0.339\pm 0.005$	$0.42\pm 0.05$	$0.109\pm 0.01$
Neural ODE (RK4)	$9.3\pm 2.6$	$0.73\pm 0.07$	$2.0\pm 0.7$	$0.32\pm 0.04$	$0.57\pm 0.29$	$0.12\pm 0.04$
Neural ODE (Tsit5)	$9.7\pm 1.3$	$0.724\pm 0.03$	$1.53\pm 0.29$	$0.26\pm 0.06$	$0.30\pm 0.11$	$0.085\pm 0.019$
cPLRNN	$5.3\pm 1.9$	$0.62\pm 0.18$	$1.6\pm 0.5$	$0.30\pm 0.04$	$0.37\pm 0.14$	$0.116\pm 0.012$
standard PLRNN	$3.84\pm 0.23$	$0.31\pm 0.07$	$1.18\pm 0.26$	$0.14\pm 0.05$	$0.24\pm 0.06$	$0.079\pm 0.01$

Appendix

Table 9: Comparison of reconstruction performance between Neural ODE (with different ODE solvers), continuous PLRNN (cPLRNN), and standard PLRNN for different latent dimensions

P

for models trained for

2000

epochs on the Lorenz-63 dataset. Reported are geometrical (

D_{\mathrm{stsp}}

) and temporal (

D_{\text{H}}

) disagreement in the limit (lower is better) as median

\pm

median absolute deviation across

10

model trainings, except for † Neural ODEs integrated by Euler’s method where too many runs diverged (leaving only 4-6 valid model runs).

Recurrent Neural Networks

Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.

See all 302 papers

Sep 29, 2026

Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating

Dynamical Systems Reconstruction (DSR) aims to infer models from observed time series that reproduce a system's qualitative long-term behavior. Continual DSR (cDSR) requires learning new systems while preserving previously learned dynamics, yet even small parameter updates in recurrent models can qualitatively alter their behavior over long autonomous rollouts. We benchmark established continual learning (CL) methods spanning parameter regularization, replay, and parameter isolation on the fully trainable and interpretable Almost-Linear RNN (AL-RNN). Parameter isolation preserves earlier dynamics most effectively, but excessive task-specific allocations can rapidly exhaust a fixed-size network. We therefore introduce Continually-Recyclable Unit-Gating (CRUG), which conserves capacity through compact allocation and forward transfer. Differentiable gates trained with an

L_0

-based penalty select task-specific units, while unused units are recycled for subsequent tasks. Directed connections allow later tasks to reuse earlier representations without affecting the dynamics of previously committed units. CRUG achieves the strongest reconstruction--capacity trade-off among the tested methods with zero forgetting and reliably learns a heterogeneous sequence of nonlinear and chaotic systems. Furthermore, we show that forward transfer is more pronounced and useful when tasks share similar underlying dynamics. Lastly, we demonstrate that CRUG's advantages extend beyond autonomous cDSR to sequential cognitive tasks.

Sima Hashemi, Daniel Durstewitz, Georgia Koppe

Faculty of Mathematics and Computer Science, Interdisciplinary Center for Scientific Computing, Heidelberg University, Heidelberg, Germany · Department of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Medical Faculty Mannheim, Heidelberg University, Heidelberg, Germany · Hector Institute for AI in Psychiatry & Department of Psychiatry and Psychotherapy, CIMH, Medical Faculty Mannheim, Heidelberg University, Heidelberg, Germany +1

Dynamical Systems Continual Learning Recurrent Neural Networks

Ask PDF

May 12, 2026

Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction

Reconstructing nonlinear dynamical systems (DS) from data (DSR) is a fundamental challenge in science and engineering, but it inherently relies on sequential models. Recent breakthroughs for sequential models have produced algorithms that parallelize computation along sequence length

T

, achieving logarithmic time complexity,

\mathcal{O}(\log T)

. Since sequence lengths have been practically limited due to the linear runtime complexity

\mathcal{O}(T)

of classical backpropagation through time, this opens new avenues for DSR. This paper studies two prominent classes of parallel-in-time algorithms for this task, both of which leverage parallel associative scans as their core computational primitive. The first class comprises models with linear yet non-autonomous dynamics and a nonlinear readout, such as modern State Space Models (SSMs), while the second consists of general nonlinear models which can be parallelized using the DEER framework. We find that the linear training-time recurrence of the first class of models imposes limitations that often hinder learning of accurate nonlinear dynamics. To address this, we augment DEER with Generalized Teacher Forcing (GTF), a novel variant within the more general nonlinear framework that ensures stable and effective learning of nonlinear dynamics across arbitrary sequence lengths. Using GTF-DEER, we investigate the benefits of training on extremely long sequences (

T>10^4

) for DSR. Our results show that access to such long trajectories significantly improves DSR if the data features long time scales. This work establishes GTF-DEER as a robust tool for data-driven discovery and underscores the largely untapped potential of long-sequence learning in modeling complex DS.

Florian Hess, Florian Götz, Daniel Durstewitz

1Dept. of Theoretical Neuroscience, Central Institute of Mental Health,2026 Mannheim, Germany · Faculty of Mathematics and Computer Science, Heidelberg University, Germany · Faculty of Physics and Astronomy, Heidelberg University, Germany +1

Nonlinear System Identification Recurrent Neural Networks State Space Models

Ask PDF

May 5, 2026

Physics-Modeled Neural Networks

We introduce Dynamical Physics-Modeled Neural Networks (DynPMNNs), a continuous-time deep learning architecture in which each hidden layer is defined as the solution of an ordinary differential equation. Unlike classical feed-forward networks, this approach replaces static activation functions with time-evolving dynamical systems, providing a biologically inspired interpretation of hidden-layer behavior and enabling the integration of physically meaningful models. The framework is rigorously grounded in Reproducing Kernel Banach Spaces (RKBSs), allowing DynPMNNs to be characterized as finite-dimensional solutions of an abstract training problem and revealing structural connections with standard neural networks. We present a concrete implementation based on the FitzHugh--Nagumo model for neuronal activation, where numerical ODE solvers are embedded into the computational graph via Euler-type schemes. Both network weights and dynamical parameters are trained jointly. Through experiments on the California Housing dataset, we compare DynPMNNs with Neural ODEs (NODEs) and Closed-form Continuous-Time Networks (CfCs). Despite using fewer trainable parameters, DynPMNNs achieve competitive performance. These results position DynPMNNs as a principled bridge between dynamical systems and deep learning, with promising directions for further research in expressivity, stability, and physics-based modeling.

Raul Felipe-Sosa, Angel Martin del Rey, Maria Flores Ceballos

aDepartamento de Matemáticas, Escuela Politénica Superior, Universidad Nebrija, 28015 Madrid, Spain · bDepartment of Applied Mathematics, IUFFyM, Universidad de Salamanca, 37008 Salamanca, Spain · cUnidad de Excelencia en Luz y Materia Estructuradas (LUMES), Universidad de Salamanca, 37008 Salamanca, Spain

Dynamic Neural Networks Neural ODEs

Ask PDF

Abstract

Recurrent Neural Networks

Explore similar work

Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating

Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction

Physics-Modeled Neural Networks