ReLU Neural Networks

ReLU: Rectified Linear Unit

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 60

Jul 6, 2026stat.ML

Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as

\ell^1

-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories that are largely specialized to homogeneous activations such as the ReLU, our framework provides a meaningful notion of functional complexity for deep networks with a broad range of homogeneous and non-homogeneous activation functions commonly used in practice. This simple construction unites several seemingly disparate ideas from the literature, including norm-based complexity bounds and variational characterizations of depth, and facilitates novel analyses of what kinds of functions deep norm-constrained networks can represent. To this end, we prove a novel representer theorem for our spaces and establish novel function-space complexity bounds showing that the associated function classes remain qualitatively small at arbitrary depth. In the univariate ReLU case, we prove a "depth saturation" result: depth in this setting yields only a small constant rescaling of the function class, with no added functional diversity. As a consequence, we show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This finding reveals that some commonly cited expressivity benefits of depth disappear once network complexity is controlled by an appropriate function space norm, rather than parameter count or other representational costs that permit compounded rescaling across layers. Overall, our results illustrate how a function space perspective yields new structural insights into the relationship between depth and complexity.

Julia Nakhleh, Robert D. Nowak

Department of Computer Science · University of Wisconsin-Madison · Department of Electrical & Computer Engineering

ReLU Neural Networks Neural Network Approximation Theory Neural Network Expressivity

Jul 6, 2026cs.LG

Compressed Computation under $L^4$ Loss is likely Computation in Superposition

Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an

L^4

loss (the mean fourth power of the error), rather than the usual

L^2

, elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.

Francisco Ferreira da Silva, Stefan Heimersheim

ReLU Neural Networks Feedforward Neural Networks Neural Network Expressivity

Jul 6, 2026cs.LG

Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions

d_x

and

d_y

, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve

L^p

approximation

(1\leq p <+\infty)

on any compact domain, we show that the exact minimum block width is

\max\{d_x,d_y\}

when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width

\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}

can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than

\max\{d_x, d_y\}

, both in the

L^p

sense and the uniform sense, regardless of inner width.

Qi Zhou, Xuan Zhou, Xiao-Song Yang

School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China. · Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China.

ReLU Neural Networks Residual Learning Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

Jul 3, 2026cs.LG

Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression. In a mean-field regime, the training dynamics are approximated by a Wasserstein gradient flow that converges to a unique stationary measure. We characterize the structure of this stationary measure and the predictor it represents. We show that, despite the network being infinitely overparameterized, the learned predictor admits an effectively finite representation: the input weights and biases align along finitely many directions, leading to an effective width collapse. In particular, the solution function is continuous piecewise affine, with affine regions determined by the cells of a finite hyperplane arrangement. The number of learned directions, and hence hyperplanes, is bounded above by

2\mathcal{P}-1

, where

\mathcal{P}

denotes the number of linear dichotomies realizable on the training inputs. We further establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data. Consequently, the complexity of the learned predictor is governed by the combinatorial geometry of the training data.

Shuang Liang, Tom Jacobs, Guido Montúfar

Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences

Shallow Neural Networks ReLU Neural Networks Implicit Bias Neural Network Training Dynamics Stochastic Gradient Descent

Jun 29, 2026math.LO

Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks

We study binary classification problems whose decision sets are given by definable sets in o-minimal expansions of the real field. Motivated by cell decomposition of definable sets, we introduce traceable sets as a classical proxy for definable decision regions and analyze their approximation by ReLU neural networks. Under uniform bounds on the number of connected components and suitable

C^m

extensions for the boundary functions, we prove that characteristic functions of traceable subsets of

[-1/2,1/2]^n

can be approximated in

L^p

to accuracy

\varepsilon>0

by ReLU neural networks of size

\mathcal{O}(\varepsilon^{-p(n-1)/m})

, with depth independent of

\varepsilon

and polynomially bounded weights. This establishes quantitative approximation rates for certain definable collections in o-minimal structures using ReLU neural networks. The same approach also yields the stated approximation rates for a subclass of definable maps

[-1/2,1/2]^n \to \mathbb{R}

. We then combine the approximation capabilities with entropy estimates for ReLU neural network classes to obtain statistical learning rates for empirical risk minimization with hinge loss. For

N

uniformly distributed samples, the resulting classifiers achieve expected misclassification error of order

N^{-m/(m+pn-p)}

up to an arbitrarily small polynomial loss.

Clemens Kinn, Philipp Petersen

Faculty of Mathematics, University of Vienna, 1090 Vienna, Austria

ReLU Neural Networks Binary Classification Classification Neural Network Approximation Theory Statistical Learning Theory

Jun 29, 2026stat.ML

SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

Neural networks are known to be susceptible to over-reliance on spurious correlations. However, the precise mechanism by which models exploit shortcut features is not fully understood, and algorithms to mitigate this behavior rely on as yet unjustified assumptions about the learned representations. In this work, we provide the first end-to-end theoretical characterization of spurious feature learning for two-layer ReLU neural networks trained by online minibatch SGD on the logistic loss. We consider data drawn from the high-dimensional Boolean hypercube with a quadratic signal function (namely XOR) and a linear spurious correlation. We show that SGD learns the spurious feature first, and exponentially fast. Moreover, the optimization dynamics couple the spurious and signal features, with a stronger spurious component inhibiting signal feature learning. Our analysis reveals precise phase transitions in the learning dynamics. In the first phase, alignment between the signs of the spurious feature and second-layer weight drives rapid growth of the spurious feature. In the second phase, large majority group margin slows learning and the signal feature remains suppressed. When the spurious correlation is maximally strong, we show theoretically that the spurious feature dominates even at the sample complexity threshold where XOR would be learned in isolation (i.e., if the spurious feature was absent). In contrast, when the correlation strength is constant, we provide preliminary empirical evidence that the model can eventually learn the XOR signal, although the spurious feature is not forgotten.

Tyler LaBonte, Vidya Muthukumar

H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology · School of Electrical and Computer Engineering, Georgia Institute of Technology

Representation Learning ReLU Neural Networks Neural Network Training Dynamics Shortcut Learning Stochastic Gradient Descent

Jun 18, 2026cs.LG

Recurrent neural networks approximate continuous functions

Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [-1,1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.

Valentin Abadie, Clemens Hutter, Helmut Bölcskei

ReLU Neural Networks Recurrent Neural Networks Neural Network Approximation Theory

Jun 9, 2026cs.LG

Robust Regression of General ReLUs with Queries

We study the task of agnostically learning general (as opposed to homogeneous) ReLUs under the Gaussian distribution with respect to the squared loss. In the passive learning setting, recent work gave a computationally efficient algorithm that uses

poly(d,1/ε)

labeled examples and outputs a hypothesis with error

O(opt)+ε

, where

opt

is the squared loss of the best fit ReLU. Here we focus on the interactive setting, where the learner has some form of query access to the labels of unlabeled examples. Our main result is the first computationally efficient learner that uses

d polylog(1/ε)+\tilde{O}(\min\{1/p, 1/ε\})

black-box label queries, where

p

is the bias of the target function, and achieves error

O(opt)+ε

. We complement our algorithmic result by showing that its query complexity bound is qualitatively near-optimal, even ignoring computational constraints. Finally, we establish that query access is essentially necessary to improve on the label complexity of passive learning. Specifically, for pool-based active learning, any active learner requires

\tildeΩ(d/ε)

labels, unless it draws a super-polynomial number of unlabeled examples.

Ilias Diakonikolas, Daniel M. Kane, Mingchen Ma

University of Wisconsin-Madison · University of California, San Diego

ReLU Neural Networks Active Learning Statistical Learning Theory

Jun 8, 2026cs.LG

Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

We study feed-forward ReLU networks with fixed readout and quadratic loss, and rewrite gradient descent as a collective dynamics of activation fields and conjugate fields on the training set. Working to first order in the learning rate inside a fixed activation chamber, we derive explicitly the one-, two- and three-hidden-layer cases, and then give the arbitrary-depth recursion. For one hidden layer the activation dynamics closes directly and the residual update is governed by the product of an input Gram matrix and a co-activation/backpropagation Gram matrix. For two hidden layers a conjugate field is required, but no nontrivial pullback Gram metric has yet appeared. For three hidden layers the first weight-induced pullback Gram metric enters the conjugate-field dynamics. At arbitrary depth, activation variations propagate forward through a recursive response operator

\cU_\ell^{αβ}

, while conjugate-field variations propagate backward through an effective transport operator

\cM_\ell^{αβ}

. Their contractions reconstruct a layerwise residual kernel

K_{αβ}^{(L)}=\sum_{\ell=1}^{L}Q_{αβ}^{(\ell-1)}S_{αβ}^{(\ell)}.

The resulting description exposes a duality between push-forward and pullback transport across every layer cut, and identifies the first Gram metrics as the lowest nontrivial terms in a broader hierarchy of activation-conditioned transport operators. We deliberately stop at the level of collective fields, conjugate fields, residual kernels and cut-wise transport metrics, leaving the later tensorial geometric formulation outside the scope of this paper.

Claudio Nordio

Backpropagation ReLU Neural Networks Neural Network Training Dynamics Gradient Descent Dynamics

Jun 5, 2026cs.LG

Characterizing the Discrete Geometry of ReLU Networks

It is well established that ReLU networks define continuous piecewise-linear functions, and that their linear regions are polyhedra in the input space. These regions form a complex that fully partitions the input space. The way these regions fit together is fundamental to the behavior of the network, as nonlinearities occur only at the boundaries where these regions connect. However, relatively little is known about the geometry of these complexes beyond bounds on the total number of regions, and calculating the complex exactly is intractable for most networks. In this work, we prove new theoretical results about these complexes that hold for all fully-connected ReLU networks, specifically about their connectivity graphs in which nodes correspond to regions and edges exist between each pair of regions connected by a face. We find that the average degree of this graph is upper bounded by twice the input dimension regardless of the width and depth of the network, and that the diameter of this graph has an upper bound that does not depend on input dimension, despite the number of regions increasing exponentially with input dimension. We corroborate our findings through experiments with networks trained on both synthetic and real-world data, which provide additional insight into the geometry of ReLU networks. Code to reproduce our results can be found at https://github.com/bl-ake/ICLR-2026.

Blake B. Gaines, Jinbo Bi

Department of Computer Science University of Connecticut

Neural Representation Geometry ReLU Neural Networks Neural Network Expressivity

Jun 4, 2026stat.ML

Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks

Recent progress has been made in understanding the statistical generalization performance of gradient descent methods for overparameterized neural networks within the neural tangent kernel (NTK) regime. However, most of the existing work on regression problems is limited to shallow network architectures, leaving a notable gap in the theory of deep neural networks. This paper addresses this gap by presenting a comprehensive generalization analysis for deep ReLU networks trained using gradient descent (GD) and stochastic gradient descent (SGD). Specifically, we establish the first known minimax-optimal rates of excess population risk for both GD and SGD with deep ReLU networks, under the assumption that the network width scales polynomially with respect to the network depth and training sample size. Our results demonstrate that with sufficient width, gradient descent methods for deep ReLU networks can achieve optimal generalization rates on par with kernel methods.

Junyu Zhou, Puyu Wang, Yunwen Lei +2

Mathematical Institute for Machine Learning and Data Science, KU Eichstätt-Ingolstadt, Ingolstadt, Germany · Department of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, Germany · Department of Mathematics, University of Hong Kong, Hong Kong, China +1

Neural Network Generalization ReLU Neural Networks Gradient Descent Stochastic Gradient Descent

Jun 4, 2026cs.LG

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales. We formalize this phenomenon by separating the fast decay of the classification loss from the slower simplification of the learned representation, and we call the resulting pair of stopping times two training clocks. For deep linear networks, we show that a post-margin gap-growth or one-step tail-contraction condition reduces the cross-entropy loss to level epsilon on a logarithmic time scale. In contrast, when layerwise weight decay is present, the induced regularization on the end-to-end map can be expressed as a Schatten-type penalty; under a sharp late-time Kurdyka-Lojasiewicz tail, this structural energy closes on a polynomial time scale. The two clocks, therefore, separate fitting from representation simplification. We then explain how the same mechanism can appear in ReLU MLPs. In regions where the activation patterns on the training set remain fixed, the network reduces to a linear model in the active coordinates. In a two-layer ReLU embedding model, chain-rule estimates further show that the classifier head can receive larger effective gradients than the embedding block under controlled downstream norms. This supports a two-stage mechanism in which the classifier fits first, while the representation continues to simplify later. We use modular addition as the main experimental setting. The deep linear theory provides the rigorous core of the analysis. But the ReLU results are formulated as conditional reductions that account for empirical behavior without claiming a global proof for nonlinear training dynamics.

Hu Tan, Kuo Gai, Shihua Zhang

1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1

ReLU Neural Networks Deep Linear Networks Neural Network Training Dynamics Grokking

Jun 4, 2026cs.LG

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators. While standard ReLU networks achieve minimax-optimal rates in the

L^2(P)

norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least-squares ReLU estimators can suffer from the curse of dimensionality in their uniform convergence behavior. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst-case reliability, we address this limitation by analyzing smoothly activated DNNs (smooth DNNs), encompassing both feedforward and residual structures. We establish novel pseudo-dimension bounds, non-asymptotic approximation guarantees, and Hölder-norm bounds for the approximators of these models. Leveraging these results, we derive non-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least-squares, quantile, and logistic regression. We prove that smooth DNNs can mitigate the {curse of dimensionality} in uniform convergence by adaptively exploiting the low-dimensional hierarchical composition structure of the target function. Supported by both simulation studies and a real-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees.

Yizhe Ding, Runze Li, Jia Liu +1

Department of Statistics, The Pennsylvania State University

ReLU Neural Networks Neural Network Approximation Theory Statistical Learning Theory

Jun 3, 2026cs.LG

When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks

In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function. Concretely, we consider a realizable setting where inputs are drawn i.i.d. from a Gaussian distribution and labels follow a planted linear model. This stylized framework captures salient features of end-to-end training in inverse problems and certain auto-encoder models. Despite its apparent simplicity, the dynamics remain poorly understood, in part because the loss landscape contains multiple non-strict saddle points, making it unclear why gradient descent from random initialization reliably escapes bad stationary regions. We provide a detailed characterization of the optimization landscape and prove that gradient descent from a moderately small random initialization-simultaneously training both layers-converges to a global minimizer at a linear rate with order-wise optimal sample complexity. Our analysis tracks the trajectory through three phases: an alignment phase in which hidden weights progressively align with the planted direction while the output weights maintain the correct sign pattern; a growth phase in which the norms of both layers increase while preserving alignment; and a local refinement phase in which the aligned neurons rapidly converge to the planted direction, yielding fast local convergence. To rigorously show that GD avoids non-strict saddles, we develop trajectory-level control arguments for the end-to-end dynamics. In addition, we establish novel uniform concentration results that hold along the entire trajectory, and are essential for obtaining order-wise optimal sample complexity. We corroborate our theory with extensive experiments across a range of configurations.

Berk Tinaz, Changzhi Xie, Mahdi Soltanolkotabi

Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA

ReLU Neural Networks Neural Network Optimization Neural Network Training Dynamics Gradient Descent Dynamics

May 29, 2026stat.ML

Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks

This paper studies how efficiently deep ReLU neural networks can approximate and learn smooth functions. When the error is measured in

L^p([0,1]^d)

norm and the approximator is a network with width

W

and depth

L

, recent works have proven the supper approximation rate

\mathcal{O}((WL)^{-2s/d})

for Besov space

\mathcal{B}^s_{q,r}([0,1]^d)

under the Sobolev embedding condition

s/d>1/q-1/p

. In order to overcome the curse of dimensionality in this rate, we extent this result to anisotropic and mixed smooth function classes. We establish the approximation rate

\mathcal{O}((WL)^{-2\tilde{s}})

for anisotropic Besov space

\mathcal{B}^{\boldsymbol{s}}_{q,r}([0,1]^d)

with anisotropic smoothness

\boldsymbol{s}=(s_1,\dots,s_d)

under the embedding condition

\tilde{s} > 1/q-1/p

, where the mean smoothness

\tilde{s} = (\sum_{i=1}^d s_i^{-1})^{-1}

. For mixed smooth Besov space

\mathcal{MB}^s_{q,r}([0,1]^d)

with mixed smoothness

s>1/q-1/p

, we show that the approximation rate

\mathcal{O}((WL)^{-2s})

holds up to logarithmic factors. Using these results, we also derive approximation bounds for the composition of anisotropic Besov functions. As an application, it is shown that deep ReLU neural networks can achieve minimax optimal rates up to logarithmic factors for a wide range of smooth function classes.

Yunfei Yang, Jun Fan

School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China. · Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong, China.

ReLU Neural Networks Neural Network Approximation Theory

May 26, 2026cs.LG

Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias

The successful training of neural networks hinges on the use of first order optimization methods, yet the theoretical characterization of these methods remains incomplete. This is especially true in settings with mild overparameterization. In this work, we study the gradient flow dynamics of two-layer ReLU networks from small initialization with orthogonal training data. We prove the limiting flow converges to a saddle-to-saddle jump process as the initialization scale tends to zero, revealing an incremental learning phenomenon in which a new neuron activates at each saddle. This analysis recovers the known result of Dana et al. (2025, arXiv:2502.16977) that the network interpolates the training data with high probability as soon as

m \gtrsim \log(n)

, where

m

is the network width and

n

is the number of training samples. This incremental process characterization also allows us to derive a novel implicit bias result: the learned interpolator has a squared

\ell_2

-norm scaling as

\sqrt{n}

, which is within a constant factor of the minimal

\ell_2

-norm interpolator. More broadly, our work provides the first rigorous proof of an incremental learning process for ReLU networks, whilst suggesting mildly overparameterized networks can converge to interpolating solutions whose complexity is of the same order as that of the optimal interpolator.

James Town, Etienne Boursier, Ben Lewis +2

University of Warwick · INRIA LMO, Université Paris-Saclay

ReLU Neural Networks Neural Network Optimization Incremental Learning Neural Network Training Dynamics Implicit Regularization

May 24, 2026cs.LG

On the Epistemic Uncertainty of Overparametrized Neural Networks

Epistemic uncertainty is often viewed as a reducible uncertainty that vanishes with increasing data. This perspective implicitly assumes parameter identifiability and equates epistemic uncertainty with predictive variability. In overparametrized neural networks, however, model parameters are typically non-identifiable due to symmetries and redundant representations. As a consequence, substantial parameter uncertainty can persist even when the underlying function is fully identified. In this work, we analyze epistemic uncertainty through the lens of non-identifiability and characterize both discrete and continuous sources of residual uncertainty. Focusing on one-hidden-layer ReLU networks, we thoroughly analyze the resulting posterior structure and validate our theoretical insights through empirical studies.

David Rügamer

Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany

Bayesian Neural Networks Parameter Identifiability ReLU Neural Networks Epistemic Uncertainty

May 18, 2026cs.LG

The Symmetries of Three-Layer ReLU Networks

We develop a framework for analyzing parameter symmetries in deep ReLU networks and obtain a complete characterization of the generic parameter fibers for three-layer bottleneck architectures. Our approach provides explicit semi-algebraic descriptions of these fibers and yields a polynomial time algorithm for deciding functional equivalence of two parameters. The symmetries include discrete and continuous transformations arising from layer composition, and depend on whether deeper layers hide or preserve geometric structure from preceding layers. Finally, we show that some of these symmetries induce local conservation laws along gradient flow, while others do not.

Johanna Marie Gegenfurtner, Moritz Grillo, Guido Montúfar

Technical University of Denmark · Max Planck Institute for Mathematics in the Sciences · UCLA and MPI MiS

Parameter Identifiability ReLU Neural Networks

May 17, 2026cs.LG

Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes

The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures (MLP, ResNet, ViT, GPT-nano, MP-SENe) and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching up to 90% in GPT-nano. We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above

\sim

70% activation sparsity. While ReLU

^2

achieves a good sparsity--accuracy ratio in GPT-nano, it pathologically amplifies identified activation spikes in intermediate transformer layers. Clipping resolves this while preserving the representational benefits of squaring: clipped ReLU

^2

outperforms its unclipped version, and GELU

^2

achieves the lowest validation loss on GPT-nano. Code is available at https://github.com/On-Point-RND/BugOrFeature.

Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3

ReLU Neural Networks Activation Sparsity Activation Outliers Neural Network Activation Functions Neural Network Training Dynamics

May 6, 2026cs.LG

Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning

We compare in-context learning with fixed queries and agentic learning with adaptive queries for uniform approximation of task families. We consider two settings: an unrestricted regime, where querying and approximation are arbitrary functions, and a realizable regime, where we require these operations to be implemented by ReLU neural networks. In both settings, adaptivity never hinders approximation performance. However, this advantage can change when one passes from the unrestricted regime to the realizable regime. We identify four distinct approximation scenarios, each witnessed by an explicit task family: (a) no advantage of adaptivity; (b) an advantage in the unrestricted regime that persists under ReLU realizability; (c) an advantage that arises only under realizability; and (d) an advantage that disappears under realizability. This demonstrates that representational constraints interact profoundly with the effect of adaptivity.

Anastasis Kratsios, A. Martina Neuman, Philipp Petersen

McMaster University and the Vector Institute, Main St., Hamilton, L8S 4L8, Ontario, Canada · University of Vienna, Faculty of Mathematics, Kolingasse 14-16, 1090 Wien, Austria

ReLU Neural Networks In-Context Learning Neural Network Approximation Theory Adaptive Learning

May 5, 2026math.CA

Exact ReLU realization of tensor-product refinement iterates

We study scalar dyadic refinement operators on R^2 of the form (Vf)(x,y) = sum_{(j,k) in Z^2} c_{j,k} f(2x-j, 2y-k), where only finitely many mask coefficients c_{j,k} are nonzero. Under a fixed support-window hypothesis, we prove that for every compactly supported continuous piecewise linear seed g:R^2->R, the iterates V^n g admit exact ReLU realizations of fixed width and depth O(n). This gives a first genuinely two-dimensional extension of the exact realization theory for refinement cascades. Using the one-dimensional exact loop-controller framework, the proof transports the tensor-product residual dynamics exactly on the product of two polygonal loops and reduces the remaining seam ambiguity to a final readout and selector step. The matrix cascade is then handled by a fixed-depth recursive block, and general compactly supported continuous piecewise linear seeds are reduced to a finite decomposition together with exact clamped gluing on the support window. This identifies the tensor-product dyadic case as a natural first multivariate instance of the loop-controller method for refinement iterates.

Tsogtgerel Gantumur

McGill University, Montréal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology

ReLU Neural Networks Neural Network Approximation Theory Iterative Refinement

May 5, 2026cs.LG

Most ReLU Networks Admit Identifiable Parameters

We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.

Moritz Grillo, Guido Montúfar

Max Planck Institute for Mathematics in the Sciences, Leipzig · Departments of Mathematics and Statistics & Data Science, UCLA

Representation Identifiability Parameter Identifiability ReLU Neural Networks Neural Network Expressivity

May 3, 2026math.CA

Exact Loop Controllers for ReLU Realization of Homogeneous Curve Refinements

We study homogeneous refinement operators

(Vγ)(t)=\sum_{j\in\mathbb Z}A_jγ(Mt-j)

, acting on compactly supported continuous piecewise linear curves

γ:\mathbb R\to\mathbb R^p

, where

M\ge2

and only finitely many matrices

A_j\in\mathbb R^{p\times p}

are nonzero. We prove that the iterates

V^nγ

admit exact ReLU realizations of fixed width and depth

O(n)

. The main new ingredient is an exact loop controller for the residual dynamics. Instead of propagating scalar residual surrogates, the construction transports the residual orbit by a forward-exact state on a polygonal loop. Scalar factors and digit selectors are then recovered from this loop state by complementary CPwL readouts. The loop seam is not removed, but its remaining ambiguity is confined to the final readout/selector stage, where it is harmless because the scalar atom is supported away from the seam. This gives a homogeneous

M

-ary vector-valued extension of the scalar binary refinable-function construction with a more geometric controller architecture. We also record crude exponential bounds on the network weights and biases. Affine forcing terms are handled by expanding affine iterates into finite sums of homogeneous iterates, giving exact fixed-width realizations with depth

O(n^2)

, and anchored open curves reduce to compactly supported defects with affine anchor mismatch. We also describe homogeneous polygonal generators, including dragon-type examples and a self-intersecting Hilbert-type prototype in arbitrary dimension. The extended version includes stage-dependent forcing, finite-state stacking reductions, and further geometric constructions such as Koch-, Gosper-, Morton-, and connector-based Hilbert-type variants.

Boldsaikhan Bolorkhuu, Tsogtgerel Gantumur

1McGill University · 2National University of Mongolia · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences

ReLU Neural Networks Neural Network Approximation Theory Iterative Refinement

Apr 27, 2026cs.CC

Primitive Recursion without Composition: Dynamical Characterizations, from Neural Networks to Polynomial ODEs

What do recurrent neural networks, polynomial ODEs, and discrete polynomial maps each bring to computation, and what do they lack? All three operate over the continuum--real-valued states evolved by real-valued dynamics--even when the target functions are discrete. We study them through primitive recursion. We prove that primitive recursion admits equivalent characterizations in all three frameworks: bounded iteration of a fixed recurrent ReLU network, robust computation by a fixed polynomial ODE, and iteration of a fixed polynomial map with an externally supplied step-size parameter. In each, the time bound is itself primitive recursive, composition emerges from the dynamics rather than as a closure rule, and inputs are raw integer vectors. Every primitive recursive function is first compiled into bounded iteration of a single threshold-affine normal form, then interpreted as a ReLU computation and as a polynomial ODE. The equivalences expose a structural asymmetry: no fixed polynomial map can round uniformly to the nearest integer or realize exact phase selection--operations polynomial ODEs perform robustly via continuous-time flow. Each formalism compensates for a limitation the others lack: the ReLU gate provides exact branching, continuous time provides autonomous rounding and control, and the step-size parameter recovers both at the cost of discretization precision. This opens dynamical characterizations of subrecursive hierarchies and complexity classes by restricting time bounds, polynomial degrees, or discretization resources within one framework. More broadly, these models do not compute by composing subroutines: they shape the trajectory of a dynamical system through clocks, phase selectors, and error correction built into the dynamics. This differs structurally from symbolic programming, and our theorem gives a precise framework to study the difference.

Olivier Bournez

Institut Polytechnique de Paris, Ecole Polytechnique, LIX, Palaiseau, France

Dynamical Systems ReLU Neural Networks Recurrent Neural Networks

Apr 25, 2026stat.ML

Explicit integral representations and quantitative bounds for two-layer ReLU networks

An approach to construct explicit integral representations for two-layer ReLU networks is presented, which provides relatively simple representations for any multivariate polynomial. Quantitative bounds are provided for a particular, sharpened ReLU integral representation, which involves a harmonic extension and a projection. The bounds demonstrate that functions can be approximated with

L^{2}(\mathcal{D})

errors that do not depend explicitly on dimension or degree, but rather the coefficients of their monomial expansions and the distribution

\mathcal{D}

. We also present a connection to the RKHS of the exponential kernel

K(x,y)=\exp\left(\left\langle x,y\right\rangle \right)

, and a very simple integral representation involving additionally multiplication via a fixed function which has better quantitative bounds.

Anthony Lee

School of Mathematics, University of Bristol

ReLU Neural Networks Reproducing Kernel Hilbert Spaces Neural Network Approximation Theory

Apr 15, 2026cs.LG

A Complete Symmetry Classification of Shallow ReLU Networks

Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as reverse engineering," or parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the \textit{neuromanifold}, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.

Pranavkrishnan Ramakrishnan

Shallow Neural Networks Neural Representation Geometry Parameter Identifiability ReLU Neural Networks

Jan 30, 2026cs.AI

Complete Identification of Deep ReLU Networks through Łukasiewicz Logic

Two deep ReLU networks can have entirely different architectures and parameters, yet realize the same function. We provide a complete characterization of this nonuniqueness. This is effected by building a symbolic calculus for deep ReLU networks, equivalence and simplification of networks becoming derivation of formulae, in close parallel to Shannon's analysis of switching circuits through Boolean logic. Inspired by Shannon, who turned circuit synthesis into the manipulation of Boolean formulae by the axioms of Boolean algebra, we turn ReLU network identification into the derivation of Łukasiewicz formulae by the axioms of many-valued (MV) logic. Two non-degenerate ReLU networks realize the same function on the unit cube if and only if one is obtained from the other by finitely many applications of the MV axioms for integer weights and biases, the divisible MV axioms for rational ones, and the Riesz MV axioms for real ones. The MV logic axioms characterize all symmetries of ReLU networks, the single-layer ones, which for tanh networks are the only kind, and the deep ones, spanning three or more layers. Our framework consists of three steps, an extraction algorithm turning a network into a substitution graph, whose represented formula has the network's input-output map as its truth function, a completeness theorem, by which functionally equivalent formulae are interderivable, and a construction algorithm returning from graphs to networks. The substitution graph is layered, carrying at each node a formula in the variables of the layer feeding it, encodes the network uniquely, and induces a new normal form for MV logic, compositional rather than flat as in the literature, hence retaining the algebraic structure of the network, with three local operations--node rewrite, layer collapse, layer expansion--realizing every derivation.

Yani Zhang, Helmut Bölcskei

Chair for Mathematical Information Science, ETH Zürich

Parameter Identifiability ReLU Neural Networks Feedforward Neural Networks

Sep 17, 2025eess.SY

Circuit realization and hardware linearization of monotone operator equilibrium networks

It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural network in analog hardware. We furthermore show that the gradient of such a circuit can be computed directly in hardware, using a procedure we call hardware linearization. This allows the network to be trained in hardware, which we demonstrate with a device-level circuit simulation. We extend the results to cascades of resistor-diode networks, which can be used to implement feedforward and other asymmetric networks. We finally show that different nonlinear elements give rise to different activation functions, and introduce the novel diode ReLU which is induced by a non-ideal diode model.

Thomas Chaffey

School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia

ReLU Neural Networks Neuromorphic Computing Neural Network Activation Functions

Jun 24, 2025stat.ML

Near-optimal estimates for the $\ell^p$ -Lipschitz constants of deep random ReLU neural networks

This paper studies the

\ell^p

-Lipschitz constants of ReLU neural networks

Φ: \mathbb{R}^d \to \mathbb{R}

with random parameters for

p \in [1,\infty]

. The distribution of the weights follows a variant of the He initialization. In the case of zero-bias networks, we derive high probability upper and lower bounds for wide networks that differ at most by a factor that is logarithmic in the network's depth. Remarkably, the behavior of the

\ell^p

-Lipschitz constant varies significantly between the regimes

p \in [1,2)

and

p \in [2,\infty]

. For

p \in [2,\infty]

, the

\ell^p

-Lipschitz constant behaves similarly to

\Vert g\Vert_{p'}

, where

g \in \mathbb{R}^d

is a

d

-dimensional standard Gaussian vector and

1/p + 1/p' = 1

. In contrast, for

p \in [1,2)

, the

\ell^p

-Lipschitz constant aligns more closely to

\Vert g \Vert_{2}

. We extend our analysis to networks with possibly non-zero biases drawn from arbitrary symmetric distributions. In this case, we obtain high probability upper and lower bounds that differ at most by a factor that is logarithmic in the network's width and linear in its depth.

Sjoerd Dirksen, Patrick Finke, Paul Geuchen +2

ReLU Neural Networks Lipschitz Neural Networks

Jul 13, 2023cs.LG

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set

\mathscr{A}

is defined to encompass the majority of commonly used activation functions, such as

\mathtt{ReLU}

,

\mathtt{LeakyReLU}

,

\mathtt{ReLU}^2

,

\mathtt{ELU}

,

\mathtt{CELU}

,

\mathtt{SELU}

,

\mathtt{Softplus}

,

\mathtt{GELU}

,

\mathtt{SiLU}

,

\mathtt{Swish}

,

\mathtt{Mish}

,

\mathtt{Sigmoid}

,

\mathtt{Tanh}

,

\mathtt{Arctan}

,

\mathtt{Softsign}

,

\mathtt{dSiLU}

, and

\mathtt{SRS}

. We demonstrate that for any activation function

\varrho\in \mathscr{A}

, a

\mathtt{ReLU}

network of width

N

and depth

L

can be approximated to arbitrary precision by a

\varrho

-activated network of width

3N

and depth

2L

on any bounded set. This finding enables the extension of most approximation results achieved with

\mathtt{ReLU}

networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,

\,

depth) scaling factors can be further reduced from

(3,2)

to

(1,1)

if

\varrho

falls within a specific subset of

\mathscr{A}

. This subset includes activation functions such as

\mathtt{ELU}

,

\mathtt{CELU}

,

\mathtt{SELU}

,

\mathtt{Softplus}

,

\mathtt{GELU}

,

\mathtt{SiLU}

,

\mathtt{Swish}

, and

\mathtt{Mish}

.

Shijun Zhang, Jianfeng Lu, Hongkai Zhao

Department of Mathematics Duke University

ReLU Neural Networks Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity