math.DSMar 30, 2026

Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect

Authors: Christian Kuehn, Sara-Viola Kuntz, Tobias Wöhrer

Organizations: Technical University of Munich, School of Computation, Information and Technology, Department of Mathematics, Boltzmannstraße 3, 85748 Garching, Germany · Munich Data Science Institute (MDSI), Garching, Germany · Munich Center for Machine Learning (MCML), München, Germany · TU Wien, Department of Mathematics, Institute of Analysis and Scientific Computing, Vienna, Austria

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We analyze the universal approximation constraints of narrow Residual Neural Networks (ResNets) both theoretically and numerically. For deep neural networks without input space augmentation, a central constraint is the inability to represent critical points of the input-output map. We prove that this has global consequences for target function approximations and show that the manifestation of this defect is typically a shift of the critical point to infinity, which we call the ``tunnel effect'' in the context of classification tasks. While ResNets offer greater expressivity than standard multilayer perceptrons (MLPs), their capability strongly depends on the signal ratio between the skip and residual channels. We establish quantitative approximation bounds for both the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes. These estimates depend explicitly on the channel ratio and uniform network weight bounds. Low-dimensional examples further provide a detailed analysis of the different ResNet regimes and how architecture-target incompatibility influences the approximation error.

Figures & tables

Figure 1: The “tunnel effect” in non-augmented MLPs.
Figure 2: Expressivity of the non-augmented ResNets depending on the ratio α≔δε\alpha\coloneqq\frac{\delta}{\varepsilon} and the upper and lower Lipschitz constants of the residual function.
Figure 3: Structure of the update rule of a residual neural network with skip parameter ε\varepsilon and residual parameter δ\delta . Every layer hl∈Rnhidh_{l}\in\mathbb{R}^{n_{\textup{hid}}} is represented by a square.
Figure 4: Classification of ResNet architectures depending on the input and the hidden dimension. Every node of the neural network is represented as a circle.
 ResNets with 0<α≪10<\alpha\ll 1ResNets with α≫1\alpha\gg 1
   
RegimeClose to neural ODEs, as introduced in Section 3.1Close to FNNs, such as MLPs, introduced in Section 4.1
   
DistanceTheorem 3.6 and Corollary 3.8 : small approximation error between ResNets and Neural ODEsTheorem 4.12 and Corollary 4.15 : small approximation error between ResNets and FNNs / MLPs
   
Critical PointsTheorem 3.9 : no critical points for non-augmented neural ODEsTheorem 4.16 : no critical points for non-augmented FNNs / MLPs
Table 1: Summary of the main results regarding the relationship between ResNets, neural ODEs, and FNNs, and the existence of critical points in the parameter regimes 0<α≪10<\alpha\ll 1 and α≫1\alpha\gg 1 .
Figure 5: Topological restrictions induced by the absence of critical points in the one-dimensional case: the neural network Φ\Phi has limited accuracy when classifying the data defined by the level sets of the quadratic function Ψz\Psi_{z} , as it is trapped in a μ\mu -tube around a strictly monotonically increasing reference function Φ‾\overline{\Phi} .
Figure 7
Figure 8: Classification of neural ODE architectures depending on the dimension of the input data and the vector field. Figure 8 8(a) - 8(b) is adapted from [ 21 , Figure 4.1(a)-(b)] .
Figure 9: Visualization of the monotonically decreasing structure of non-augmented FNNs and MLPs.
Figure 10: Approximation of f(x)=x2f(x)=x^{2} by a one-dimensional one-layer ResNet ( 5.1 ), W~1=1\widetilde{W}_{1}=1 , b~1=0\tilde{b}_{1}=0 and the other weights and biases as plotted.
Figure 11: Trained ResNet of the form ( 5.18 ) with ε=δ=α=1\varepsilon=\delta=\alpha=1 .
Figure 12: Trained ResNet ( 5.18 ) with equal channel ratio, α=1\alpha=1 , but ε=20\varepsilon=20 and δ=20\delta=20 .
Figure 13: Trained ResNet ( 5.18 ) with small residual term, ε=1\varepsilon=1 and δ=0.3\delta=0.3 .
Figure 14: Trained ResNet ( 5.18 ) close to MLPs with ε=0.01\varepsilon=0.01 , δ=1\delta=1 .
Figure 15: Two-dimensional datasets defined as sub- and super-level sets of the prototypical functions ( 5.20 ): Points with Ψi(x)<0.5\Psi_{i}(x)<0.5 have a blue label and points with Ψi(x)>0.5\Psi_{i}(x)>0.5 have an orange label, i∈{circ,xor}i\in\{\text{circ},\text{xor}\} . Figure 15 15(a) - 15(b) is adapted from [ 3 , Figure 4(a)-(b)] .
Figure 16: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) with ε=1\varepsilon=1 and δ=0.1\delta=0.1 in the neural ODE regime. The model is unable to express critical points which leads to the blue “tunnel”.
Figure 17: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) with balanced skip and residual channels ε=δ=1\varepsilon=\delta=1 . A critical point around the origin is present.
Figure 18: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) in the MLP regime, ε=0.1\varepsilon=0.1 and δ=1\delta=1 which shows an undesired orange “tunnel” at the origin due to the absence of a critical point.
Figure 19: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) with balanced skip and residual channels ε=δ=1\varepsilon=\delta=1 . A critical point in form of a saddle point around the origin is present.
Figure 20: Prediction level sets of an augmented MLP of the form ( 5.21 ) with nhid=3n_{\textup{hid}}=3 , ε=0\varepsilon=0 and δ=1\delta=1 . The embedding of a critical point in form of a minimum around the origin is achieved.
More in this topic

Universal Approximation

Universal approximation studies when model families can represent broad function classes and what architectural resources control approximation accuracy.

See all 25 papers

Explore similar work

CardsList

  1. Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

    Jul 6, 2026Qi Zhou, Xuan Zhou, Xiao-Song YangReLU Neural NetworksResidual Learning

    School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China. · Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China.

  2. Progressive Approximation in Deep Residual Networks: Theory and Validation

    Apr 27, 2026Wei Wang, Xiao-Yong Wei, Qing LiEfficient Neural Network InferenceResidual Learning

    Department of Computing, the Hong Kong Polytechnic University.

  3. ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

    Mar 18, 2026Louis-Pierre Chaintron, Lénaïc Chizat, Javier MaassNeural Network Training DynamicsResidual Dynamics Learning

    École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland.