Deep learning and the rate of approximation by flows

Jingpu Cheng; Qianxiao Li; Ting Lin; Zuowei Shen

cs.LGMar 16, 2026

Deep learning and the rate of approximation by flows

Authors: Jingpu Cheng, Qianxiao Li, Ting Lin, Zuowei Shen

Organizations: Department of Mathematics, National University of Singapore, 117543, Singapore · Department of Mathematics and Institute for Functional Intelligent Materials, National University of Singapore, 117543, Singapore · School of Mathematical Sciences, Peking University, 100871, China

Abstract

We investigate the dependence of the approximation capacity of deep residual networks on its depth in a continuous dynamical systems setting. This can be formulated as the general problem of quantifying the minimal time-horizon required to approximate a diffeomorphism by flows driven by a given family $\mathcal F$ of vector fields. We show that this minimal time can be identified as a geodesic distance on a sub-Finsler manifold of diffeomorphisms, where the local geometry is characterised by a variational principle involving $\mathcal F$ . This connects the learning efficiency of target relationships to their compatibility with the learning architectural choice. Further, the results suggest that the key approximation mechanism in deep learning, namely the approximation of functions by composition or dynamics, differs in a fundamental way from linear approximation theory, where linear spaces and norm-based rate estimates are replaced by manifolds and geodesic distances.

Figures & tables

Figure 1. Diagram illustrating the connection built in the main result between the approximation complexity by flows and sub-Finsler geometry

Panel of Figure 2 — Figure 2. Comparison of distance contours between $d_{\mathcal{F}}$ and $L^{2}$ on the convex hull of three functions in $\mathcal{M}$

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 227 papers

Layer-wise Geometric Approximation Rates for Deep Networks
Apr 22, 2026Shijun Zhang, Zuowei Shen, Yuesheng XuNeural Network Approximation Theory Neural Network Expressivity
Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics National University of Singapore · Department of Mathematics and Statistics Old Dominion University
Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space
Feb 18, 2021Kuo Gai, Shihua ZhangResidual Learning Neural Network Training Dynamics
Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS) Shanghai 200433, China · Academy of Mathematics and Systems Science Chinese Academy of Sciences Beijing 100190, China · School of Mathematical Sciences2026 University of Chinese Academy of Sciences Beijing 100049, China
On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective
Feb 24, 2026Jinshu Huang, Mingfei Sun, Chunlin WuNeural Network Generalization Generalization Bounds
School of Mathematical Sciences, Nankai University, Tianjin, China

Apr 22, 2026J/K move · Enter open · S save

Layer-wise Geometric Approximation Rates for Deep Networks

Shijun Zhang, Zuowei Shen, Yuesheng Xu

Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics National University of Singapore · Department of Mathematics and Statistics Old Dominion University

Neural Network Approximation Theory Neural Network Expressivity

Open paper AskPDF arXiv

Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantitative framework in which depth admits a precise scale-dependent interpretation. Specifically, we design a single shared mixed-activation architecture of fixed width

2dN+d+2

and any prescribed finite depth such that each intermediate readout

Φ_\ell

is itself an approximant to the target function

f

. For

f\in L^p([0,1]^d)

with

p\in [1,\infty)

, the approximation error of

Φ_\ell

is controlled by

(2d+1)

times the

L^p

modulus of continuity at the geometric scale

N^{-\ell}

for all

\ell

. The estimate reduces to the geometric rate

(2d+1)N^{-\ell}

f

1

-Lipschitz. Our network design is inspired by multigrade deep learning, where depth serves as a progressive refinement mechanism. For every prescribed terminal depth, the construction yields a finite nested family of prefix readouts whose earlier correction terms remain embedded in later readouts. Thus the approximation may be truncated within the prescribed depth range once the desired certified accuracy is reached.

Abstract

Neural Network Approximation Theory

Explore similar work

Layer-wise Geometric Approximation Rates for Deep Networks

Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space

On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective