Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect
Authors: Christian Kuehn, Sara-Viola Kuntz, Tobias Wöhrer
Organizations: Technical University of Munich, School of Computation, Information and Technology, Department of Mathematics, Boltzmannstraße 3, 85748 Garching, Germany · Munich Data Science Institute (MDSI), Garching, Germany · Munich Center for Machine Learning (MCML), München, Germany · TU Wien, Department of Mathematics, Institute of Analysis and Scientific Computing, Vienna, Austria
We analyze the universal approximation constraints of narrow Residual Neural Networks (ResNets) both theoretically and numerically. For deep neural networks without input space augmentation, a central constraint is the inability to represent critical points of the input-output map. We prove that this has global consequences for target function approximations and show that the manifestation of this defect is typically a shift of the critical point to infinity, which we call the ``tunnel effect'' in the context of classification tasks. While ResNets offer greater expressivity than standard multilayer perceptrons (MLPs), their capability strongly depends on the signal ratio between the skip and residual channels. We establish quantitative approximation bounds for both the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes. These estimates depend explicitly on the channel ratio and uniform network weight bounds. Low-dimensional examples further provide a detailed analysis of the different ResNet regimes and how architecture-target incompatibility influences the approximation error.
Figures & tables
Figure 1: The “tunnel effect” in non-augmented MLPs.
Figure 2: Expressivity of the non-augmented ResNets depending on the ratio α:=εδ and the upper and lower Lipschitz constants of the residual function.
Figure 3: Structure of the update rule of a residual neural network with skip parameter ε and residual parameter δ . Every layer hl∈Rnhid is represented by a square.
Figure 4: Classification of ResNet architectures depending on the input and the hidden dimension. Every node of the neural network is represented as a circle.
ResNets with 0<α≪1
ResNets with α≫1
Regime
Close to neural ODEs, as introduced in Section 3.1
Close to FNNs, such as MLPs, introduced in Section 4.1
Distance
Theorem 3.6 and Corollary 3.8 : small approximation error between ResNets and Neural ODEs
Theorem 4.12 and Corollary 4.15 : small approximation error between ResNets and FNNs / MLPs
Critical Points
Theorem 3.9 : no critical points for non-augmented neural ODEs
Theorem 4.16 : no critical points for non-augmented FNNs / MLPs
Table 1: Summary of the main results regarding the relationship between ResNets, neural ODEs, and FNNs, and the existence of critical points in the parameter regimes 0<α≪1 and α≫1 .
Figure 5: Topological restrictions induced by the absence of critical points in the one-dimensional case: the neural network Φ has limited accuracy when classifying the data defined by the level sets of the quadratic function Ψz , as it is trapped in a μ -tube around a strictly monotonically increasing reference function Φ .
Figure 7
Figure 8: Classification of neural ODE architectures depending on the dimension of the input data and the vector field. Figure 8 8(a) - 8(b) is adapted from [ 21 , Figure 4.1(a)-(b)] .
Figure 9: Visualization of the monotonically decreasing structure of non-augmented FNNs and MLPs.
Figure 10: Approximation of f(x)=x2 by a one-dimensional one-layer ResNet ( 5.1 ), W1=1 , b~1=0 and the other weights and biases as plotted.
Figure 11: Trained ResNet of the form ( 5.18 ) with ε=δ=α=1 .
Figure 12: Trained ResNet ( 5.18 ) with equal channel ratio, α=1 , but ε=20 and δ=20 .
Figure 13: Trained ResNet ( 5.18 ) with small residual term, ε=1 and δ=0.3 .
Figure 14: Trained ResNet ( 5.18 ) close to MLPs with ε=0.01 , δ=1 .
Figure 15: Two-dimensional datasets defined as sub- and super-level sets of the prototypical functions ( 5.20 ): Points with Ψi(x)<0.5 have a blue label and points with Ψi(x)>0.5 have an orange label, i∈{circ,xor} . Figure 15 15(a) - 15(b) is adapted from [ 3 , Figure 4(a)-(b)] .
Figure 16: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) with ε=1 and δ=0.1 in the neural ODE regime. The model is unable to express critical points which leads to the blue “tunnel”.
Figure 17: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) with balanced skip and residual channels ε=δ=1 . A critical point around the origin is present.
Figure 18: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) in the MLP regime, ε=0.1 and δ=1 which shows an undesired orange “tunnel” at the origin due to the absence of a critical point.
Figure 19: Prediction level sets of a non-augmented ResNet of the form ( 5.21 ) with balanced skip and residual channels ε=δ=1 . A critical point in form of a saddle point around the origin is present.
Figure 20: Prediction level sets of an augmented MLP of the form ( 5.21 ) with nhid=3 , ε=0 and δ=1 . The embedding of a critical point in form of a minimum around the origin is achieved.
Universal approximation studies when model families can represent broad function classes and what architectural resources control approximation accuracy.
In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions dx and dy, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve Lp approximation (1≤p<+∞) on any compact domain, we show that the exact minimum block width is max{dx,dy} when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width min{dx+dy,max{2dx+1,dy}} can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than max{dx,dy}, both in the Lp sense and the uniform sense, regardless of inner width.
Qi Zhou, Xuan Zhou, Xiao-Song Yang
School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China. · Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China.
The Universal Approximation Theorem (UAT) guarantees universal function approximation but does not explain how residual models distribute approximation across layers. We reframe residual networks as a layer-wise approximation process that builds an approximation trajectory from input to target, and prove the existence of progressive trajectories where error decreases monotonically with depth. It reveals that residual networks can implement structured, step-by-step refinement rather than end-to-end (E2E) black-box mapping. Building on this, we propose Layer-wise Progressive Approximation (LPA), a theoretically grounded training principle that explicitly aligns each layer with its residual target to realize such trajectories. LPA is architecture-agnostic: we observe progressive behavior in residual FNNs, ResNets, and Transformers across tasks including complex surface fitting, image classification, and NLP with LLMs for generation and classification. Crucially, this enables ``train once, use N models": a single network yields useful predictions at every depth, supporting efficient shallow inference without retraining. Our work unifies approximation theory with practical deep learning, providing a new lens on representation learning and a flexible framework for multi-depth deployment. The source code will be released unpon acceptance at https://(open_upon_acceptance).
Wei Wang, Xiao-Yong Wei, Qing Li
Department of Computing, the Hong Kong Polytechnic University.
We establish convergence of the training dynamics of residual neural networks (ResNets) to their joint infinite depth L, hidden width M, and embedding dimension D limit. Specifically, we consider ResNets with two-layer perceptron blocks in the maximal local feature update (MLU) regime and prove that, after a bounded number of training steps, the error between the ResNet and its large-scale limit is O(1/L + sqrt(D/(L M)) + 1/sqrt(D)). This error rate is empirically tight when measured in embedding space. For a budget of P = Theta(L M D) parameters, this yields a convergence rate O(P^(-1/6)) for the scalings of (L, M, D) that minimize the bound. Our analysis exploits in an essential way the depth-two structure of residual blocks and applies formally to a broad class of state-of-the-art architectures, including Transformers with bounded key-query dimension. From a technical viewpoint, this work completes the program initiated in the companion paper [Chi25] where it is proved that for a fixed embedding dimension D, the training dynamics converges to a Mean ODE dynamics at rate O(1/L + sqrt(D)/sqrt(L M)). Here, we study the large-D limit of this Mean ODE model and establish convergence at rate O(1/sqrt(D)), yielding the above bound by a triangle inequality. To handle the rich probabilistic structure of the limit dynamics and obtain one of the first rigorous quantitative convergence for a DMFT-type limit, we combine the cavity method with propagation of chaos arguments at a functional level on so-called skeleton maps, which express the weight updates as functions of CLT-type sums from the past.
Louis-Pierre Chaintron, Lénaïc Chizat, Javier Maass
École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland.