Unrolled gradients in disguise: bridging interpolation-based and Jacobian regularization for stable neural dynamics

Maya Janvier; Etienne Meunier

cs.LGNov 2, 2026

Unrolled gradients in disguise: bridging interpolation-based and Jacobian regularization for stable neural dynamics

Authors: Maya Janvier, Etienne Meunier

Organizations: LOCEAN, Sorbonne University, Paris, France · Inria, Paris, France

Abstract

Hybrid models and neural differential equations (NDEs) are becoming increasingly important for modelling physical systems; however, they often encounter stability and accuracy issues during long-term integration. Training on unrolled trajectories is known to limit these divergences but quickly becomes too computationally expensive due to the need to compute gradients over an iterative process, making this method impractical for large-scale physical systems (e.g. weather and climate forecasting). Two popular paradigms for improving the stability of learned dynamics are to train on interpolated or noisy samples or to regularize the Jacobian of the learned representation. In this work, we demonstrate that these paradigms can be understood under a common lens and that they both correspond to an approximation of the gradients obtained in an unrolled learning strategy. Building on this understanding, we derive a principled and complete approach to regularize training and demonstrate its impact on the stability of learned dynamical systems.

Figures & tables

Method	Exact form	Linearized form (offline)	Regularization form (short-online)
Offline	$\\|F(x_{t})-F_{\theta}(x_{t})\\|_{2}^{2}$	$\\|\epsilon_{\theta}(x_{t})\\|_{2}^{2}$ exact $F$	X
Lipschitz	$\\|J_{F_{\theta}}(x_{t})\\|_{F}^{2}$	X	$\frac{1}{V}\sum_{j=1}^{V}\\|J_{F_{\theta}}(x_{t})\cdot v_{j}\\|_{2}^{2}$
CFO	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\gamma})\\|_{2}^{2}$	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\gamma})\\|_{2}^{2}$ spline fit $F$ various $\gamma$ (upsampling x10)	X
JacMatch	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\gamma})\\|_{2}^{2}$	$\\|\epsilon_{\theta}(x_{t})+\gamma J_{\epsilon_{\theta}}(x_{t})\cdot F(x_{t})\\|_{2}^{2}$ exact or spline fit $F$ , $J_{F}$ single $\gamma$	$\\|J_{\epsilon_{\theta}}(x_{t})\cdot F(x_{t})\\|_{2}^{2}$ exact or FD $F$ , $J_{F}$ [ 16 ] single $\lambda$
AccMatch	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\theta,\gamma})\\|_{2}^{2}$	$\\|\epsilon_{\theta}(x_{t})+\gamma\eta_{\theta}(x_{t})\\|_{2}^{2}$ exact or spline fit $F$ , $J_{F}$ single $\gamma$	$\\|\eta_{\theta}(x_{t})\\|_{2}^{2}$ exact (or FD) $F$ , $J_{F}$ single $\lambda$

Table 1: Summary of the methods . (i) Offline : The original offline method only uses the exact

F

and CFO the fitted spline version of it. CFO differs from JacMatch by the upsampling (10 more

x_{t}^{\gamma}

data points), although they come from the same theoretical formula. (ii) Short-online: the regularizations are tested within the training loss:

\mathcal{L}(\theta)=\mathcal{L}_{\text{traj}}^{N}(\theta)+\lambda\mathcal{L}_{\text{reg}}(\theta)

. The Lipschitz method refers to the Jacobian regularization approach, in the spirit of JAWS. To build an unsupervised version of JacMatch, we use a first-order finite differences (FD) approximation (eq. 16 ). A FD version of AccMatch is possible but not tested here. Both: the

\gamma

and

\lambda

hyperparameters are tuned on a held-out validation set.

Figure 1: Losses interpretation. When $\gamma=1$ , $x_{t}^{\gamma}=x_{t+\Delta t}$ and we get that JacMatch is equivalent to the error of a delayed $F_{\theta}$ trajectory starting from $x_{t+\Delta t}$ , i.e., only accounting for the error between the Jacobians while AccMatch also accounts for the error after the first timestep.

Table 3

Panel of Figure 2 — Figure 2: Rigid Body. For each row, we have the evolution of the relative error over time (left) and the distributions of final values (right). The top row represents the relative trajectory error $R_{e}(\theta,t)$ and the bottom row represents the relative conservation error (Casimir).

Panel of Figure 3 — Figure 3: Kuramoto-Sivashinsky. For each row, we have the evolution of the relative error over time (left) and the distributions of final values (right). The top row represents the relative trajectory error $R_{e}(\theta,t)$ and the bottom row represents the relative conservation error (Mass).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameters	Rigid Body	Kuramoto-Sivashinsky	Van der Pol
$U$ dimension	3	256	2
$\Delta t/\Delta t_{\text{model}}$ (s)	0.01 / 0.1	0.2 / 0.2	0.01/ 0.1 or 0.01
$T_{\text{train}}/T_{\text{val}}/T_{\text{test}}$ (s)	15 / 15 / 800	28 / 28 / 128	10/10/10
$N_{\text{train}}/N_{\text{val}}/N_{\text{test}}$ (s)	5 / 5 / 8000	2 / 2 / 640	1/1/100 or 1000
Nb traj train/val / test	40 / 40 / 100	512 / 128 / 128	10/10/10
Nb epochs	4500	1000	10000

Appendix

Table 4: Summary of model and experiment parameters

Panel of Figure 4 — Figure 4: Mean MSE on validation set for the optimization of the $\lambda$ parameter, for the Rigid Body Problem (left) and KS (right) experiments.

Parameters	Two-Body
$U$ dimension	4
$\Delta t/\Delta t_{\text{model}}$ (s)	0.01 / 0.01
$T_{\text{train}}/T_{\text{val}}/T_{\text{test}}$ (s)	8 / 8 / 100
$N_{\text{train}}/N_{\text{val}}/N_{\text{test}}$	2 / 2 / 10000
Nb traj train/val / test	40 / 40 / 100
Nb epochs	4500

Appendix

Table 5: Summary of model and experiment parameters

Panel of Figure 5 — Figure 5: Mean MSE on validation set for the optimization of the $\lambda$ parameter, for the Two-Body Problem (Right: SNODE, Left: the other methods)

Panel of Figure 6 — Figure 6: Two Body. For each row, we have the evolution of the relative error over time (left) and the distributions of final values (right). The top row represents the relative trajectory error $R_{e}(\theta,t)$ and the bottom row represents the relative conservation error (Momentum).

Figure 7: Computational cost on Kuramoto-Sivashinsky averaged over 1000 runs. For regularized methods, the cost for $N=2$ is shown to compare with the $N=10$ baseline.

Panel of Figure 8 — Figure 8: Kuramoto-Sivashinksy trajectories

Panel of Figure 9 — Figure 9: Rigid Body trajectories

Neural Network Stability

See all 56 papers

OrthoReg: Orthogonal Regularization for Hybrid Symbolic-Neural Dynamical Systems
Jun 17, 2026Till Richter, Niki KilbertusSparse Identification of Nonlinear Dynamics Neural Network Interpretability
Technical University of Munich · Helmholtz Munich
Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamics
Aug 17, 2026Conrad Ainslie, Pedram Hassanzadeh, Michael W. Mahoney +1Neural Network Stability Neural Surrogate Modeling
Department of Applied Mathematics, University of California, Santa Cruz, Santa Cruz, 95064, CA · Department of Geophysical Sciences, University of Chicago, Chicago, IL 60637 · Committee on Computational and Applied Mathematics, University of Chicago, Chicago, IL 60637 +3
Symbolic Neural ODEs: Learning interpretable models from time-series data
Aug 22, 2026Nibodh Boddupalli, Jeff MoehlisNonlinear System Identification System Identification
Dept. of Mechanical Engineering, University of California, Santa Barbara, CA 93106

Jun 17, 2026J/K move · Enter open · S save

OrthoReg: Orthogonal Regularization for Hybrid Symbolic-Neural Dynamical Systems

Till Richter, Niki Kilbertus

Technical University of Munich · Helmholtz Munich

Sparse Identification of Nonlinear Dynamics Neural Network Interpretability Hybrid Dynamical Systems

Open paper AskPDF arXiv

Dynamical systems are fundamental to modeling the natural world, yet modeling them involves a persistent trade-off: manually prescribed mechanistic models are interpretable by design but often overly simplistic and misspecified; in contrast, flexible data-driven neural methods lack physical insight. Hybrid modeling aims for the best of both worlds by combining a prescribed or symbolic, physics-based component with a flexible neural network. A critical challenge, however, is that the neural component may relearn mechanistic parts, yielding redundant and uninterpretable models, especially when the symbolic structure itself is discovered from data. Existing methods based on standard

L^2

regularization rely on a projection argument that breaks when the symbolic component is learned through sparse discovery, allowing the neural augmentation to overlap with symbolic structure. We introduce OrthoReg (Orthogonal Regularization), which directly penalizes overlap between the symbolic and neural components, preventing symbolic structure from being absorbed by the neural residual. This yields a complementary decomposition: the symbolic part captures what the library can express, and the neural part captures what remains. On benchmark dynamical systems with partial library mismatch, OrthoReg improves symbolic recovery and out-of-distribution behavior.

Method	Exact form	Linearized form (offline)	Regularization form (short-online)
Offline	$\\|F(x_{t})-F_{\theta}(x_{t})\\|_{2}^{2}$	$\\|\epsilon_{\theta}(x_{t})\\|_{2}^{2}$ exact $F$	X
Lipschitz	$\\|J_{F_{\theta}}(x_{t})\\|_{F}^{2}$	X	$\frac{1}{V}\sum_{j=1}^{V}\\|J_{F_{\theta}}(x_{t})\cdot v_{j}\\|_{2}^{2}$
CFO	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\gamma})\\|_{2}^{2}$	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\gamma})\\|_{2}^{2}$ spline fit $F$ various $\gamma$ (upsampling x10)	X
JacMatch	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\gamma})\\|_{2}^{2}$	$\\|\epsilon_{\theta}(x_{t})+\gamma J_{\epsilon_{\theta}}(x_{t})\cdot F(x_{t})\\|_{2}^{2}$ exact or spline fit $F$ , $J_{F}$ single $\gamma$	$\\|J_{\epsilon_{\theta}}(x_{t})\cdot F(x_{t})\\|_{2}^{2}$ exact or FD $F$ , $J_{F}$ [ 16 ] single $\lambda$
AccMatch	$\\|F(x_{t}^{\gamma})-F_{\theta}(x_{t}^{\theta,\gamma})\\|_{2}^{2}$	$\\|\epsilon_{\theta}(x_{t})+\gamma\eta_{\theta}(x_{t})\\|_{2}^{2}$ exact or spline fit $F$ , $J_{F}$ single $\gamma$	$\\|\eta_{\theta}(x_{t})\\|_{2}^{2}$ exact (or FD) $F$ , $J_{F}$ single $\lambda$

Abstract

Neural Network Stability

Explore similar work

OrthoReg: Orthogonal Regularization for Hybrid Symbolic-Neural Dynamical Systems

Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamics

Symbolic Neural ODEs: Learning interpretable models from time-series data