Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equations

Hongjue Zhao; Yizhuo Chen; Yuchen Wang; Hairong Qi; Lui Sha; Tarek Abdelzaher; Huajie Shao

cs.AIMar 18, 2026

Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equations

Authors: Hongjue Zhao, Yizhuo Chen, Yuchen Wang, Hairong Qi, Lui Sha, Tarek Abdelzaher, Huajie Shao

Organizations: University of Illinois Urbana Champaign · William & Mary · University of Tennessee, Knoxville

Abstract

Deep neural networks (DNNs) have achieved remarkable empirical success, yet the absence of a principled theoretical foundation continues to hinder their systematic development. In this survey, we present differential equations as a theoretical foundation for understanding, analyzing, and improving DNNs. We organize the discussion around three guiding questions: i) how differential equations offer a principled understanding of DNN architectures, ii) how tools from differential equations can be used to improve DNN performance in a principled way, and iii) what real-world applications benefit from grounding DNNs in differential equations. We adopt a two-fold perspective spanning the model level, which interprets the whole DNN as a differential equation, and the layer level, which models individual DNN components as differential equations. From these two perspectives, we review how this framework connects model design, theoretical analysis, and performance improvement. We further discuss real-world applications, as well as key challenges and opportunities for future research.

Figures & tables

Figure 1: Overview of the survey. This survey is organized around three key questions concerning DNNs through the lens of differential equations: (i) How can differential equations offer a principled understanding of DNN architectures? (ii) How can tools from differential equations be used to improve DNN performance in a principled way? and (iii) What real-world applications benefit from grounding DNNs in differential equations? We focus on two levels of abstraction: the model level , which interprets the entire DNN as a dynamical system, and the layer level , which models individual layers of DNNs as differential equations.

Scope	Paper	Summary
Model Level	[ 22 ]	This survey paper explores ODE-based neural architecture design via discretization, along with continuous modeling and optimal control-inspired optimization.
	[ 163 ]	This work reviews machine learning from a dynamical systems perspective, covering gradient-based and non-gradient optimization methods, continuous-time formulations.
	[ 80 ]	This paper systematically reviews neural differential equations (ODEs, CDEs, SDEs, RDEs) and their underlying techniques.
	[ 34 ]	This review examines the mathematical foundations of incorporating ordinary and partial differential equations into DNNs, focusing on Neural ODEs and PINNs.
	[ 115 ]	This survey provides a comprehensive review of neural differential equations, with a focus on time series analysis.
	[ 102 ]	This work discusses how different DNN architectures can be viewed the deterministic or stochastic differential equations, and tests them on CIFAR-10 and CIFAR-100.

Table 1: Existing surveys on the differential equation foundations of DNNs

Figure 2: Summary of early approaches that design DNN architectures through the lens of differential equations. A central idea in these works is to interpret various skip-connection schemes as specific discretizations of ODEs, or introduce different ODE systems for special properties. The plus and minus symbols beside the arrows denote additive or subtractive operations, respectively, and the numbers indicate coefficients.

Figure 3: Illustration of three commonly used types of differential equations in NDEs. They share the unified formulation

\differential{\hB}=\bm{F}(\bm{h})\differential{\xB}

, where the specific type depends on the regularity of the driving signal

\bm{x}(t)

. (a) When

\bm{x}(t)=t

, the system corresponds to an ODE; (b) when

\bm{x}(t)

has bounded variation, it becomes a CDE; and (c) when

\bm{x}(t)

is a Brownian motion, it defines an SDE.

Model	Formulation and Key Insight
Neural ODE [ 21 ]	$\dot{\bm{h}}(t)=\bm{f}_{\bm{\theta}}(t,\bm{h}(t))$ Models continuous-depth dynamics by taking the deep limit of ResNets
Neural CDE [ 79 ]	$\differential{\hB(t)}=\bm{F}_{\bm{\theta}}(\bm{h}(t))\dot{\bm{x}}(t)\differential{t}$ Introduces input-dependent dynamics for irregular time series
Neural RDE [ 110 ]	$\differential{\hB(t)}=\bm{F}_{\bm{\theta}}(\bm{h}(t))\differential{\mathrm{LogSig}_{r_i, r_{i+1}}(\xB(t)}$ Incorporates log-signature transforms to model long and rough time series
Neural SDE [ 148 ]	$\differential{\hB(t)}=\bm{b}_{\bm{\theta}}(t,\bm{h}(t))\differential{t}+\bm{\sigma}_{\bm{\phi}}(t,\bm{h}(t))\differential{\wB(t)}$ Extends Deep Latent Gaussian Models as Neural SDEs in the diffusion limit
Neural Jump SDE [ 71 ]	$\differential{\hB(t)}=\bm{f}_{\bm{\theta}}(t,\bm{h}(t))\differential{t}+\bm{g}_{\bm{\phi}}(t,\bm{h}(t),\bm{k}(t))\differential{N(t)}$ Captures hybrid systems with both continuous flows and discrete jumps
Latent SDE [ 92 ]	$\differential{\hb{z}(t)}=\bm{f}_{0,\bm{\theta}}(t,\bm{z}(t))\differential{t}+\bm{\sigma}_{\bm{\phi}}(t,\bm{z}(t))\differential{\wB(t)},~\differential{\zB(t)}=\bm{f}_{1,\bm{\theta}}(t,\bm{z}(t))\differential{t}+\bm{\sigma}_{\bm{\phi}}(t,\bm{z}(t))\differential{\wB(t)}$ Introduces memory-efficient adjoint methods for Neural SDEs in the latent space

Table 2: Overview of representative Neural Differential Equation (NDE) models. Each formulation extends classical differential equations with neural network parameterizations to model continuous-time dynamics.

Figure 4: Comparison between flow-based and diffusion-based generative models. Flow models generate samples by simulating ODEs, resulting in smooth and deterministic sampling trajectories. In contrast, diffusion models rely on SDEs, producing stochastic and inherently rough trajectories due to the presence of noise during the sampling process.

Model	Formulation and Key Insight
Flow Model [ 21 , 96 ]	$\differential{\xB}_{t}=\bm{u}_{t}^{\bm{\theta}}(\bm{x}_{t})\differential{t}$ Trained via flow matching or maximum likelihood; generation performed by simulating ODEs
Diffusion Model [ 62 , 143 ]	$\differential{\xB}_{t}=\bm{u}_{t}^{\bm{\theta}}(\bm{x}_{t})\differential{t}+\sigma_{t}\differential{\wB}_{t}$ Trained via score matching; generation achieved by solving SDEs

Table 3: Overview of differential equation-based generative models. Each model defines a continuous-time stochastic or deterministic process for data generation, typically trained via score matching or flow-based objectives.

Model	Summary	Pros	Cons
LSSL [ 50 ]	First to introduce linear SSMs as learnable layers in deep networks.	Unifies the advantages of recurrent, convolutional, and continuous-time models.	May suffer from memory inefficiency and numerical instability.
S4 [ 49 ]	Introduces HiPPO for long-range sequence modeling and Diagonal Plus Low-Rank (DPLR) parameterization for the state matrix $\bm{A}$ .	HiPPO enables effective modeling of long-range dependencies, while DPLR ensures efficient training.	Hard to implement; built on LTI SSMs with fixed coefficients; limited in handling complex tasks such as language modeling.
DSS [ 53 ]	Parameterizes the state matrix $\bm{A}$ as a diagonal matrix without low-rank correction, while matching S4’s performance.	Simpler to implement, analyze, and train, with provable expressiveness equivalent to general SSMs.	The diagonal structure limits interactions between state dimensions, potentially reducing modeling flexibility.
S4D [ 48 ]	Incorporates various S4 initialization techniques while maintaining a diagonal state matrix.	Simpler to implement, more efficient, and more flexible than DSS.	May sacrifice some expressiveness compared to full or low-rank parameterizations.
Liquid S4 [ 59 ]	Combines liquid time-constant (LTC) networks with S4.	Enhances generalization and causal modeling by integrating liquid networks, achieving improved performance across tasks.	The more complex state matrix increases implementation and training difficulty compared to the original S4.
S5 [ 139 ]	Introduces multi-input, multi-output (MIMO) SSMs and parallel scan algorithms for efficient training.	Enables modeling of sequences with variable sampling rates while retaining S4-level efficiency.	Still based on LTI SSMs; limited in handling complex sequence modeling tasks.

Table 4: Summary of key development of Deep SSMs

Fields	Detailed Fields	Works
Natural Science	Climate modeling	[ 150 , 128 ]
	Hydrological systems	[ 123 , 66 ]
	Chemistry	[ 117 , 160 ]
	Astrophysics	[ 85 ]
	Turbulence forecasting	[ 122 ]
Engineering	Cyber-Physical Systems	[ 10 , 145 ]

Table 5: Applications of differential equation perspective of DNNs on time series tasks.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Abbreviation	Meaning
CDE	Controlled Differential Equation
CNN	Convolutional Neural Network
DNN	Deep Neural Networks
GAN	Generative Adversarial Network
LLM	Large Language Model
LSSL	Linear State-Space Layer

Appendix

Table 6: Summary of abbreviation used in the survey.

Neural ODEs

Neural ordinary differential equations (Neural ODEs) learn continuous-time dynamics by representing a network’s evolution as a differential equation.

See all 91 papers

Oct 9, 2025

Deep Neural Networks Inspired by Differential Equations

Deep learning has become a pivotal technology in fields such as computer vision, scientific computing, and dynamical systems, significantly advancing these disciplines. However, neural Networks persistently face challenges related to theoretical understanding, interpretability, and generalization. To address these issues, researchers are increasingly adopting a differential equations perspective to propose a unified theoretical framework and systematic design methodologies for neural networks. In this paper, we provide an extensive review of deep neural network architectures and dynamic modeling methods inspired by differential equations. We specifically examine deep neural network models and deterministic dynamical network constructs based on ordinary differential equations (ODEs), as well as regularization techniques and stochastic dynamical network models informed by stochastic differential equations (SDEs). We present numerical comparisons of these models to illustrate their characteristics and performance. Finally, we explore promising research directions in integrating differential equations with deep learning to offer new insights for developing intelligent computational methods that boast enhanced interpretability and generalization capabilities.

Yongshuai Liu, Lianfang Wang, Kuilin Qin +6

School of Mathematical Sciences, Beijing Normal University, Beijing, China · Department of Mathematics, The Chinese University of Hong Kong, Shatin, Hong Kong

Neural Network Generalization Dynamic Neural Networks Neural ODEs

Ask PDF

Jun 22, 2026

Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions

Over the past decade, deep neural networks (DNNs) have achieved remarkable success on complex machine-learning tasks, yet the theoretical foundations of their performance remain incomplete. From a statistical viewpoint, a natural question is: can DNNs attain feature-learning and prediction consistency comparable to that of classical models? While a full characterization is open, we provide positive results for a broad subclass. We establish feature-learning consistency guarantees for sublinearly structured DNNs-architectures whose input/output dimensions and number of hidden neurons grow sublinearly with the sample size-when learning hierarchically compositional target functions. Importantly, this consistency still holds even in the conventional "over-parameterized" regime where the total number of parameters exceeds the number of training samples. Empirically, sublinearly structured DNNs match or surpass wide DNNs in prediction. A structural audit further indicates that widely used convolutional neural networks (CNNs), including AlexNet, VGGNet, ResNet, GoogLeNet, are sublinearly structured on their image classification benchmarks. We further prove that the sublinearly structured DNNs achieve universal approximation for hierarchically compositional functions in the large-sample limit. Moreover, images exhibit an inherent hierarchical, compositional structure. Taken together, these results explain, through a statistical lens, why many large-scale deep learning models succeed after adequate training on massive image datasets.

Sehwan Kim, Yan Sun, Faming Liang

Ewha Womans University, Seoul 03760, Republic of Korea · New Jersey Institute of Technology, Newark, NJ 07102, USA · Purdue University, West Lafayette, IN 47907, USA

Neural Network Generalization Hierarchical Representation Learning Statistical Learning Theory Compositional Representation Learning

Ask PDF

May 5, 2026

On the Architectural Complexity of Neural Networks

We introduce a unified theoretical framework for the rigorous analysis and systematic construction of deep neural networks (DNNs). This framework addresses a gap in existing theory by explicitly modeling the structure of tensor operations -- lower level information that is often abstracted. Our framework enables two novel objectives: (1) analysis of the evolution of architectural complexity over deep learning history, and (2) automatic construction of novel architectures based on new types of tensor operations. Our study of DNNs introduced over the past 40 years reveals a connection between groundbreaking architectures and increases in different types of architectural complexity. Moreover, we identify several large classes of higher complexity architectures that have not yet been explored. We then collect a dataset of 3,000+ higher complexity architectures, which we publicly release at: https://github.com/combinatoriallabs/ArchitecturalComplexity.

Nicholas J. Cooper, François G. Meyer, Michael L. Roberts +3

Combinatorial Labs · University of Colorado Boulder · SEMF/Independent Researcher

Topic pending

Ask PDF

Abstract

Neural ODEs

Explore similar work

Deep Neural Networks Inspired by Differential Equations

Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions

On the Architectural Complexity of Neural Networks