ReLU Neural Networks

ReLU: Rectified Linear Unit

Ask about this topic

Overview

ReLU neural networks build piecewise-linear functions by composing affine maps with a thresholding activation, making them a useful model for studying how depth and width represent data. Approximation theory examines how efficiently they represent smooth, compositional, or irregular functions, while VC and Rademacher analyses connect model capacity to sample efficiency and generalization. Gradient-flow, neural tangent kernel, and mean-field methods study optimization and implicit bias; polyhedral geometry and formal verification help analyze linear regions, parameter symmetries, and robustness. A central tension is that depth can increase expressivity without necessarily making optimization or generalization easier, while equivalent parameterizations and data-dependent assumptions complicate claims about what training learns.

Latest papers

CardsList

113 papers24 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Apr 7, 2026

ReLU Networks for Exact Generation of Similar Graphs

Generation of graphs constrained by a specified graph edit distance from a source graph is important in applications such as cheminformatics, network anomaly synthesis, and structured data augmentation. Despite the growing demand for such constrained generative models in areas including molecule design and network perturbation analysis, the neural architectures required to provably generate graphs within a bounded graph edit distance remain largely unexplored. In addition, existing graph generative models are predominantly data-driven and depend heavily on the availability and quality of training data, which may result in generated graphs that do not satisfy the desired edit distance constraints. In this paper, we address these challenges by theoretically characterizing ReLU neural networks capable of generating graphs within a prescribed graph edit distance from a given graph. In particular, we show the existence of constant depth and O(n^2 d) size ReLU networks that deterministically generate graphs within edit distance d from a given input graph with n vertices, eliminating reliance on training data while guaranteeing validity of the generated graphs. Experimental evaluations demonstrate that the proposed network successfully generates valid graphs for instances with up to 1400 vertices and edit distance bounds up to 140, whereas baseline generative models fail to generate graphs with the desired edit distance. These results provide a theoretical foundation for constructing compact generative models with guaranteed validity.

Mamoona Ghafoor, Tatsuya Akutsu

Bioinformatics Center, Institute for Chemical Research, Department of Intelligence Sciences and Technology, Kyoto University, Uji 611-0011, Japan

Graph GenerationConstrained Generative ModelingReLU Neural Networks
AskPDF
Mar 25, 2026

Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective

Deep neural networks (DNNs), particularly those using Rectified Linear Unit (ReLU) activation functions, have achieved remarkable success across diverse machine learning tasks, including image recognition, audio processing, and language modeling. Despite this success, the non-convex nature of DNN loss functions complicates optimization and limits theoretical understanding. In this paper, we highlight how recently developed convex equivalences of ReLU NNs and their connections to sparse signal processing models can address the challenges of training and understanding NNs. Recent research has uncovered several hidden convexities in the loss landscapes of certain NN architectures, notably two-layer ReLU networks and other deeper or varied architectures. This paper seeks to provide an accessible and educational overview that bridges recent advances in the mathematics of deep learning with traditional signal processing, encouraging broader signal processing applications.

Emi Zeger, Mert Pilanci

Department of Electrical Engineering, Stanford University

ReLU Neural NetworksDeep Learning OptimizationNeural Network Optimization
AskPDF
Mar 24, 2026

Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic

Grokking-the phenomenon where validation accuracy of neural networks on modular addition of two integers rises long after training data has been memorized-has been characterized in previous works as producing sinusoidal input weight distributions in transformers and multi-layer perceptrons (MLPs). We find empirically that ReLU MLPs in our experimental setting instead learn near-binary square wave input weights, where intermediate-valued weights appear exclusively near sign-change boundaries, alongside output weight distributions whose dominant Fourier phases satisfy a phase-sum relation φout=φa+φbφ_{\mathrm{out}} = φ_a + φ_b; this relation holds even when the model is trained on noisy data and fails to grok. We extract the frequency and phase of each neuron's weights via DFT and construct an idealized MLP: Input weights are replaced by perfect binary square waves and output weights by cosines, both parametrized by the frequencies, phases, and amplitudes extracted from the dominant Fourier components of the real model weights. This idealized model achieves 95.5% accuracy when the frequencies and phases are extracted from the weights of a model trained on noisy data that itself achieves only 0.23% accuracy. This suggests that grokking does not discover the correct algorithm, but rather sharpens an algorithm substantially encoded during memorization, progressively binarizing the input weights into cleaner square waves and aligning the output weights, until generalization becomes possible.

Anand Swaroop

GrokkingNeural Network InterpretabilityReLU Neural NetworksNeural Network Memorization
AskPDF
Mar 23, 2026

λλ-GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks

Gaussian Error Linear Unit (GELU) is a widely used smooth alternative to Rectifier Linear Unit (ReLU), yet many deployment, compression, and analysis toolchains are most naturally expressed for piecewise-linear (ReLU-type) networks. We study a hardness-parameterized formulation of GELU, f(x;λ)=xΦ(λ x), where Φ is the Gaussian CDF and λ \in [1, infty) controls gate sharpness, with the goal of turning smooth gated training into a controlled path toward ReLU-compatible models. Learning λ is non-trivial: naive updates yield unstable dynamics and effective gradient attenuation, so we introduce a constrained reparameterization and an optimizer-aware update scheme. Empirically, across a diverse set of model--dataset pairs spanning MLPs, CNNs, and Transformers, we observe structured layerwise hardness profiles and assess their robustness under different initializations. We further study a deterministic ReLU-ization strategy in which the learned gates are progressively hardened toward a principled target, enabling a post-training substitution of λ-GELU by ReLU with reduced disruption. Overall, λ-GELU provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with ReLU-centric downstream pipelines.

Cristian Pérez-Corral, Alberto Fernández-Hernández, Jose I. Mestre +2

Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain

Neural Network Activation FunctionsReLU Neural Networks
AskPDF
Mar 16, 2026

Neural Networks as Local-to-Global Computations

We construct a cellular sheaf from any feedforward ReLU neural network by placing one vertex for each intermediate quantity in the forward pass and encoding each computational step - affine transformation, activation, output - as a restriction map on an edge. The restricted coboundary operator on the free coordinates is unitriangular, so its determinant is 11 and the restricted Laplacian is positive definite for every activation pattern. It follows that the relative cohomology vanishes and the forward pass output is the unique harmonic extension of the boundary data. The sheaf heat equation converges exponentially to this output despite the state-dependent switching introduced by piecewise linear activations. Unlike the forward pass, the heat equation propagates information bidirectionally across layers, enabling pinned neurons that impose constraints in both directions, training through local discrepancy minimization without a backward pass, and per-edge diagnostics that decompose network behavior by layer and operation type. We validate the framework experimentally on small synthetic tasks, confirming the convergence theorems and demonstrating that sheaf-based training, while not yet competitive with stochastic gradient descent, obeys quantitative scaling laws predicted by the theory.

Vicente Bosca, Robert Ghrist

Department of Mathematics, University of Pennsylvania

Sheaf Neural NetworksFeedforward Neural NetworksReLU Neural NetworksLocal LearningNeural Network Training Dynamics
AskPDF
Mar 11, 2026

Efficient Approximation to Analytic and LpL^p functions by Height-Augmented ReLU Networks

This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and LpL^p functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general LpL^p functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.

ZeYu Li, FengLei Fan, TieYong Zeng

Chinese University of Hong Kong, Department of Mathematics, Hong Kong, 999077, NT, Hong Kong SAR, China · City University of Hong Kong, Department of Data Science, Hong Kong, 999077, Kowloon, Hong Kong SAR, China · Guangzhou Nanfang College, School of Mathematics and Statistics, Guangzhou, 510970, Guangdong, China +1

Neural Network Approximation TheoryReLU Neural Networks
AskPDF
Mar 10, 2026

Memorization capacity of deep ReLU neural networks characterized by width and depth

This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any NN data points in the unit ball with pairwise separation distance δδ and discrete labels. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We generalize these results by constructing neural networks, whose width WW and depth LL satisfy W2L2=O(Nlog⁡(δ−1))W^2L^2= \mathcal{O}(N\log(δ^{-1})), that can memorize any NN data samples. We also prove that any such networks should also satisfy the lower bound W2L2=Ω(Nlog⁡(δ−1))W^2L^2=Ω(N \log(δ^{-1})), which implies that our construction is optimal up to logarithmic factors when δ−1δ^{-1} is polynomial in NN. Hence, we explicitly characterize the trade-off between width and depth for the memorization capacity of deep neural networks in this regime.

Xin Yang, Yunfei Yang

School of Mathematics (Zhuhai), Sun Yat-sen University, Zhuhai, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

Neural Network MemorizationReLU Neural NetworksNeural Network Expressivity
AskPDF
Mar 5, 2026

How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work (Vardi and Shamir, 2021) showed that the implicit bias does not exist in the worst-case, or corresponds exactly to the minimum-ℓ2\ell_2-norm interpolating solution under exactly orthogonal data (Boursier et al., 2022). Our work interpolates between these two extremes and shows that, for sufficiently high-dimensional random data, the implicit bias approximates the minimum-ℓ2\ell_2-norm solution with high probability with a gap on the order Θ(n/∣∣λ∣∣1)Θ(\sqrt{n/||λ||_1}), where nn is the number of training examples and λλ denotes the spectrum of the data covariance matrix. Our results are obtained through a novel primal-dual analysis that carefully tracks the evolution of predictions, data-span coefficients, as well as their interactions, and show that the ReLU activation pattern quickly stabilizes with high probability over random data.

Kuo-Wei Lai, Guanghui Wang, Molei Tao +1

School of Electrical and Computer Engineering, Georgia Institute of Technology · College of Computing, Georgia Institute of Technology · School of Mathematics, Georgia Institute of Technology +1

High-Dimensional Linear RegressionRandom Feature MethodsImplicit BiasReLU Neural NetworksGradient DescentShallow Neural Networks
AskPDF
Feb 23, 2026

Path-conditioned training: a principled way to rescale ReLU neural networks

Despite recent algorithmic advances, we still lack principled ways to leverage the well-documented rescaling symmetries in ReLU neural network parameters. While two properly rescaled weights implement the same function, the training dynamics can be dramatically different. To offer a fresh perspective on exploiting this phenomenon, we build on the recent path-lifting framework, which provides a compact factorization of ReLU networks. We introduce a geometrically motivated criterion to rescale neural network parameters which minimization leads to a conditioning strategy that aligns a kernel in the path-lifting space with a chosen reference. We derive an efficient algorithm to perform this alignment. In the context of random network initialization, we analyze how the architecture and the initialization scale jointly impact the output of the proposed method. Numerical experiments illustrate its potential to speed up training.

Arthur Lebeurrier, Titouan Vayer, Rémi Gribonval

ENS de Lyon, CNRS, Inria, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France · Inria, Rennes, France · Inria, CNRS, ENS de Lyon, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France

ReLU Neural NetworksNeural Network Training DynamicsNeural Network Initialization
AskPDF
Feb 22, 2026

Online Realizable Regression and Applications for ReLU Networks

Realizable online regression can behave very differently from online classification. Even without any margin or stochastic assumptions, realizability may enforce horizon-free (finite) cumulative loss under metric-like losses, even when the analogous classification problem has an infinite mistake bound. We study realizable online regression in the adversarial model under losses that satisfy an approximate triangle inequality (approximate pseudo-metrics). Recent work of Attias et al. shows that the minimax realizable cumulative loss is characterized by the scaled Littlestone/online dimension Donl\mathbb{D}_{\mathrm{onl}}, but this quantity can be difficult to analyze. Our main technical contribution is a generic potential method that upper bounds Donl\mathbb{D}_{\mathrm{onl}} by a concrete Dudley-type entropy integral that depends only on covering numbers of the hypothesis class under the induced sup pseudo-metric. We define an entropy potential Φ(H)=∫0diam(H)log⁡N(H,ε) dεΦ(\mathcal{H})=\int_{0}^{diam(\mathcal{H})} \log N(\mathcal{H},\varepsilon)\,d\varepsilon, where N(H,ε)N(\mathcal{H},\varepsilon) is the ε\varepsilon-covering number of H\mathcal{H}, and show that for every cc-approximate pseudo-metric loss, Donl(H)≤O(c) Φ(H)\mathbb{D}_{\mathrm{onl}}(\mathcal{H})\le O(c)\,Φ(\mathcal{H}). In particular, polynomial metric entropy implies Φ(H)<∞Φ(\mathcal{H})<\infty and hence a horizon-free realizable cumulative-loss bound with transparent dependence on effective dimension. We illustrate the method on two families. We prove a sharp qq-vs.-dd dichotomy for realizable online learning (finite and efficiently achievable Θd,q(Ld)Θ_{d,q}(L^d) total loss for LL-Lipschitz regression iff q>dq>d, otherwise infinite), and for bounded-norm kk-ReLU networks separate regression (finite loss, even O~(k2)\widetilde O(k^2), and O(1)O(1) for one ReLU) from classification (impossible already for k=2,d=1k=2,d=1).

Ilan Doron-Arad, Idan Mehalel, Elchanan Mossel

Massachusetts Institute of Technology, Cambridge, MA, USA · The Hebrew University of Jerusalem, Jerusalem, Israel

Statistical Learning TheoryReLU Neural Networks
AskPDF
Feb 22, 2026

Why ReLU? A Bit-Model Dichotomy for Deep Network Training

Theoretical analyses of Empirical Risk Minimization (ERM) are standardly framed within the Real-RAM model of computation. In this setting, training even simple neural networks is known to be ∃R\exists \mathbb{R}-complete -- a complexity class believed to be harder than NP, that characterizes the difficulty of solving systems of polynomial inequalities over the real numbers. However, this algebraic framework diverges from the reality of digital computation with finite-precision hardware. In this work, we analyze the theoretical complexity of ERM under a realistic bit-level model (ERMbit\mathsf{ERM}_{\text{bit}}), where network parameters and inputs are constrained to be rational numbers with polynomially bounded bit-lengths. Under this model, we reveal a sharp dichotomy in tractability governed by the network's activation function. We prove that for deep networks with {\em any} polynomial activations with rational coefficients and degree at least 22, the bit-complexity of training is severe: deciding ERMbit\mathsf{ERM}_{\text{bit}} is #P\#P-Hard, hence believed to be strictly harder than NP-complete problems. Furthermore, we show that determining the sign of a single partial derivative of the empirical loss function is intractable (unlikely in BPP), and deciding a specific bit in the gradient is #P\#P-Hard. This provides a complexity-theoretic perspective for the phenomenon of exploding and vanishing gradients. In contrast, we show that for piecewise-linear activations such as ReLU, the precision requirements remain manageable: ERMbit\mathsf{ERM}_{\text{bit}} is contained within NP (specifically NP-complete), and standard backpropagation runs in polynomial time. Our results demonstrate that finite-precision constraints are not merely implementation details but fundamental determinants of learnability.

Ilan Doron-Arad, Elchanan Mossel

MIT, Cambridge, MA, USA

Empirical Risk MinimizationReLU Neural NetworksPolynomial Neural Networks
AskPDF
Feb 21, 2026

Theoretical Lower Bounds on the Robustness of Deep ReLU Networks

We present a theoretical study of the robustness of parameterized neural networks to random input perturbations. Specifically, we analyze local robustness by quantifying the probability that a random L_2-perturbation of a given input results in a correct classification. For deep ReLU networks, we derive lower bounds on local robustness by combining tools from high-dimensional geometry, in particular concentration of measure, with a new characterization of the geometric structure induced by their input-output functions. We prove that each convex polyhedral region in the partition of the input space induced by a ReLU network has at most as many faces as there are network units, regardless of the network depth or architecture. This geometric property serves as the key ingredient in our robustness analysis. Finally, we analyze how local robustness scales with input dimension and characterize the sets of inputs whose neighborhoods are most likely to contain adversarial examples. We show that the width of decision-boundary neighborhoods containing vulnerable points shrinks rapidly as dimension increases and grows only logarithmically with the number of network units. We also discuss the volume of a set of vulnerable points in terms of approximately space-filling shapes of decision boundaries.

Věra Kůrková

Institute of Computer Science of the Czech Academy of Sciences, Pod Vodárenskou věží 2, 18207 Prague, Czech Republic

Neural Network RobustnessAdversarial RobustnessReLU Neural Networks
AskPDF
Feb 19, 2026

From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks

We study pathwise connectivity of sublevel sets for one-hidden-layer ReLU networks with constrained first-layer weights and an ℓ1\ell_1 penalty on the output layer. The data term is assumed convex and globally Lipschitz in the scalar logit. We first give a finite-width construction that connects any two points of a common sublevel through a path controlled by a loss-consistent compression functional and a first-order perturbation term. The proof replaces the quadratic perturbation estimate in the Freeman--Bruna mechanism by a direct Lipschitz bound. Positive homogeneity is then used in a direction that is compatible with the penalty: every active atom is moved monotonically from the unit ball to the unit sphere while its output coefficient is reduced. Sphere covering and cluster merging consequently give O(m−1/(n−1))O(m^{-1/(n-1)}) fixed-level thickening for n≥2n\ge2, while the one-dimensional two-ray dictionary gives exact connectivity for every m≥4m\ge4. We also prove internally that the regularized approximation values satisfy e(l)−e∞=O(l−1/2)e(l)-e_\infty=O(l^{-1/2}). More generally, a rate O(l−s)O(l^{-s}) transfers to a near-optimal barrier rate O(m−s/((n−1)s+1))O(m^{-s/((n-1)s+1)}); under the standing assumptions, this yields the explicit rate O(m−1/(n+1))O(m^{-1/(n+1)}). A theorem-aligned finite-distribution experiment complements the analysis. The primary Huber run yields a maximal best certified upper gap 1.66×10−51.66\times10^{-5} over 720 recorded pairs at widths m≥16m\ge16; a matched binary-cross-entropy rerun and a 720-endpoint dense-representation stress test probe loss robustness and the active cluster-merging mechanism.

Saveliy Baturin

Independent Researcher, Russia.

Shallow Neural NetworksReLU Neural NetworksMode ConnectivityNeural Network Approximation Theory
AskPDF
Feb 19, 2026

Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking

We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level, or noisy input training common in sensor networks, privacy-preserving training, and federated learning, where each user may have access to partial or corrupted features. Using a Neural Tangent Kernel (NTK) analysis, we demonstrate that training a two-layer ReLU network with Gaussian randomly masked inputs achieves linear convergence up to an error region proportional to the mask's variance. A key technical contribution is resolving the randomness within the non-linear activation, a problem of independent interest.

Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko +2

Computer Science Dept., Rice University, Houston, TX, USA

Neural Network Training DynamicsReLU Neural Networks
AskPDF
Feb 17, 2026

Continuous-Time Piecewise-Linear Recurrent Neural Networks

In dynamical systems reconstruction (DSR) we aim to recover the dynamical system (DS) underlying observed time series. Specifically, we aim to learn a generative surrogate model which approximates the underlying, data-generating DS, and recreates its long-term properties (`climate statistics'). In scientific and medical areas, in particular, these models need to be mechanistically tractable -- through their mathematical analysis we would like to obtain insight into the recovered system's workings. Piecewise-linear (PL), ReLU-based RNNs (PLRNNs) have a strong track-record in this regard, representing SOTA DSR models while allowing mathematical insight by virtue of their PL design. However, all current PLRNN variants are discrete-time maps. This is in disaccord with the assumed continuous-time nature of most physical and biological processes, and makes it hard to accommodate data arriving at irregular temporal intervals. Neural ODEs are one solution, but they do not reach the DSR performance of PLRNNs and often lack their tractability. Here we develop theory for continuous-time PLRNNs (cPLRNNs): We present a novel algorithm for training and simulating such models, bypassing numerical integration by efficiently exploiting their PL structure. We further demonstrate how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models. We compare cPLRNNs to both their discrete-time cousins as well as Neural ODEs on DSR benchmarks, including systems with discontinuities which come with hard thresholds.

Alena Brändle, Lukas Eisenmann, Florian Götz +1

Dept. of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Mannheim, Germany · Faculty of Physics and Astronomy, Heidelberg University · Hector Institute for Artificial Intelligence in Psychiatry, CIMH +2

Recurrent Neural NetworksDynamical SystemsReLU Neural Networks
AskPDF
Jan 30, 2026

Complete Identification of Deep ReLU Networks through Łukasiewicz Logic

Two deep ReLU networks can have entirely different architectures and parameters, yet realize the same function. We provide a complete characterization of this nonuniqueness. This is effected by building a symbolic calculus for deep ReLU networks, equivalence and simplification of networks becoming derivation of formulae, in close parallel to Shannon's analysis of switching circuits through Boolean logic. Inspired by Shannon, who turned circuit synthesis into the manipulation of Boolean formulae by the axioms of Boolean algebra, we turn ReLU network identification into the derivation of Łukasiewicz formulae by the axioms of many-valued (MV) logic. Two non-degenerate ReLU networks realize the same function on the unit cube if and only if one is obtained from the other by finitely many applications of the MV axioms for integer weights and biases, the divisible MV axioms for rational ones, and the Riesz MV axioms for real ones. The MV logic axioms characterize all symmetries of ReLU networks, the single-layer ones, which for tanh networks are the only kind, and the deep ones, spanning three or more layers. Our framework consists of three steps, an extraction algorithm turning a network into a substitution graph, whose represented formula has the network's input-output map as its truth function, a completeness theorem, by which functionally equivalent formulae are interderivable, and a construction algorithm returning from graphs to networks. The substitution graph is layered, carrying at each node a formula in the variables of the layer feeding it, encodes the network uniquely, and induces a new normal form for MV logic, compositional rather than flat as in the literature, hence retaining the algebraic structure of the network, with three local operations--node rewrite, layer collapse, layer expansion--realizing every derivation.

Yani Zhang, Helmut Bölcskei

Chair for Mathematical Information Science, ETH Zürich

Parameter IdentifiabilityReLU Neural NetworksFeedforward Neural Networks
AskPDF
Sep 17, 2025

Circuit realization and hardware linearization of monotone operator equilibrium networks

It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural network in analog hardware. We furthermore show that the gradient of such a circuit can be computed directly in hardware, using a procedure we call hardware linearization. This allows the network to be trained in hardware, which we demonstrate with a device-level circuit simulation. We extend the results to cascades of resistor-diode networks, which can be used to implement feedforward and other asymmetric networks. We finally show that different nonlinear elements give rise to different activation functions, and introduce the novel diode ReLU which is induced by a non-ideal diode model.

Thomas Chaffey

School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia

ReLU Neural NetworksNeuromorphic ComputingNeural Network Activation Functions
AskPDF
Aug 30, 2025

Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying ReLU and exacerbate gradient instability as network depth increases. Methods such as He, Xavier, and orthogonal initialization preserve variance or promote approximate isometry. However, they do not necessarily regulate the pre-activation mean or control activation sparsity, and their effectiveness often diminishes in very deep architectures. This work introduces a semi-orthogonal initialization specifically optimized for ReLU by solving an optimization problem on the Stiefel manifold, thereby preserving scale and calibrating the pre-activation statistics from the outset. A family of closed-form solutions and an efficient sampling scheme are derived. Theoretical analysis at initialization shows that the proposed initialization mitigates dying ReLU, slows the decay of activation variance, and alleviates gradient vanishing, thereby stabilizing signal and gradient flow in deep architectures. Empirically, across MNIST, Fashion-MNIST, multiple tabular datasets, few-shot settings, and ReLU-family activations, the proposed initialization maintains stable performance as network depth increases, with particularly clear advantages in few-shot learning and large-depth settings.

Hyungu Lee, Taehyeong Kim, Hayoung Choi

Department of Mathematics & Nonlinear Dynamics and Mathematical Application Center, Kyungpook National University, Daegu 41566, Republic of Korea · Nonlinear Dynamics and Mathematical Application Center, Kyungpook National University, Daegu 41566, Republic of Korea

Neural Network InitializationReLU Neural NetworksNeural Network Stability
AskPDF
Jun 24, 2025

Optimal estimates for the ℓp\ell^p-Lipschitz constants of deep random ReLU neural networks

This paper studies the ℓp\ell^p-Lipschitz constants of ReLU neural networks Φ:Rd→RΦ: \mathbb{R}^d \to \mathbb{R} with random parameters for p∈[1,∞]p \in [1,\infty]. The distribution of the weights follows a variant of the He initialization. In the case of zero-bias networks, we derive matching high-probability upper and lower bounds for wide networks that differ at most by an absolute constant. The bounds reveal two remarkable phenomena. First, the bounds grow root-logarithmically in terms of the depth of the net. Second, the behavior of the ℓp\ell^p-Lipschitz constants varies significantly between the regimes p∈[1,2)p \in [1,2) and p∈[2,∞]p \in [2,\infty]. For p∈[2,∞]p \in [2,\infty], the ℓp\ell^p-Lipschitz constant behaves similarly to ∥g∥p′\|g\|_{p'}, where g∈Rdg \in \mathbb{R}^d is a standard Gaussian vector and 1/p+1/p′=11/p + 1/p' = 1. In contrast, for p∈[1,2)p \in [1,2), the ℓp\ell^p-Lipschitz constant aligns more closely with ∥g∥2\|g\|_2. For networks with possibly nonzero biases, we obtain the same lower bounds without assumptions on the bias distributions. For symmetric bias distributions with bounded densities, we also derive high-probability upper bounds for the ℓp\ell^p-Lipschitz constants. For fixed sub-exponential bias distributions, these upper bounds differ from the lower bounds by a factor that has square-root dependence on the depth and logarithmic dependence on the width.

Sjoerd Dirksen, Patrick Finke, Paul Geuchen +2

ReLU Neural NetworksLipschitz Neural Networks
AskPDF
May 22, 2025

The Computational Complexity of Counting Linear Regions in ReLU Neural Networks

An established measure of the expressive power of a given ReLU neural network is the number of linear regions into which it partitions the input space. There exist many different, non-equivalent definitions of what a linear region actually is. We systematically assess which papers use which definitions and discuss how they relate to each other. We then analyze the computational complexity of counting the number of such regions for the various definitions. Generally, this turns out to be an intractable problem. We prove NP- and #P-hardness results already for networks with one hidden layer and strong hardness of approximation results for two or more hidden layers. Finally, on the algorithmic side, we demonstrate that counting linear regions can at least be achieved in polynomial space for some common definitions.

Moritz Stargalla, Christoph Hertrich, Daniel Reichman

University of Technology Nuremberg · Worcester Polytechnic Institute

ReLU Neural NetworksNeural Network Expressivity
AskPDF
Feb 4, 2025

Networks with Finite VC Dimension: Pro and Contra

Approximation and learning of classifiers of large data sets by neural networks in terms of high-dimensional geometry and statistical learning theory are investigated. The influence of the VC dimension of sets of input-output functions of networks on approximation capabilities is compared with its influence on consistency in learning from samples of data. It is shown that, whereas finite VC dimension is desirable for uniform convergence of empirical errors, it may not be desirable for approximation of functions drawn from a probability distribution modeling the likelihood that they occur in a given type of application. Based on the concentration-of-measure properties of high dimensional geometry, it is proven that both errors in approximation and empirical errors behave almost deterministically for networks implementing sets of input-output functions with finite VC dimensions in processing large data sets. Practical limitations of the universal approximation property, the trade-offs between the accuracy of approximation and consistency in learning from data, and the influence of depth of networks with ReLU units on their accuracy and consistency are discussed.

Vera Kurkova, Marcello Sanguineti

Institute of Computer Science of the Czech Academy of Sciences Pod Vodárenskou věží, 2 - 18207 Prague, Czech Republic · DIBRIS University of Genova Via Opera Pia, 13 - 16145 Genova, Italy

Neural Network GeneralizationNeural Network Approximation TheoryReLU Neural Networks
AskPDF
Nov 15, 2024

Dense ReLU Neural Networks for Temporal-spatial Model

In this paper, we focus on fully connected deep neural networks utilizing the Rectified Linear Unit (ReLU) activation function for nonparametric estimation. We derive non-asymptotic bounds that lead to convergence rates, addressing both temporal and spatial dependence in the observed measurements. By accounting for dependencies across time and space, our models better reflect the complexities of real-world data, enhancing both predictive performance and theoretical robustness. We also tackle the curse of dimensionality by modeling the data on a manifold, exploring the intrinsic dimensionality of high-dimensional data. We broaden existing theoretical findings of temporal-spatial analysis by applying them to neural networks in more general contexts and demonstrate that our proof techniques are effective for models with short-range dependence. Our empirical simulations across various synthetic response functions underscore the superior performance of our method, outperforming established approaches in the existing literature. These findings provide valuable insights into the strong capabilities of dense neural networks (Dense NN) for temporal-spatial modeling across a broad range of function classes.

Carlos Misael Madrid Padilla, Zhi Zhang, Xiaokai Luo +2

Department of Statistics and Data Science, Washington University in St. Louis. · Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Notre Dame

Nonparametric RegressionReLU Neural NetworksNeural Network Approximation Theory
AskPDF
Jul 13, 2023

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set A\mathscr{A} is defined to encompass the majority of commonly used activation functions, such as ReLU\mathtt{ReLU}, LeakyReLU\mathtt{LeakyReLU}, ReLU2\mathtt{ReLU}^2, ELU\mathtt{ELU}, CELU\mathtt{CELU}, SELU\mathtt{SELU}, Softplus\mathtt{Softplus}, GELU\mathtt{GELU}, SiLU\mathtt{SiLU}, Swish\mathtt{Swish}, Mish\mathtt{Mish}, Sigmoid\mathtt{Sigmoid}, Tanh\mathtt{Tanh}, Arctan\mathtt{Arctan}, Softsign\mathtt{Softsign}, dSiLU\mathtt{dSiLU}, and SRS\mathtt{SRS}. We demonstrate that for any activation function ϱ∈A\varrho\in \mathscr{A}, a ReLU\mathtt{ReLU} network of width NN and depth LL can be approximated to arbitrary precision by a ϱ\varrho-activated network of width 3N3N and depth 2L2L on any bounded set. This finding enables the extension of most approximation results achieved with ReLU\mathtt{ReLU} networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width, \,depth) scaling factors can be further reduced from (3,2)(3,2) to (1,1)(1,1) if ϱ\varrho falls within a specific subset of A\mathscr{A}. This subset includes activation functions such as ELU\mathtt{ELU}, CELU\mathtt{CELU}, SELU\mathtt{SELU}, Softplus\mathtt{Softplus}, GELU\mathtt{GELU}, SiLU\mathtt{SiLU}, Swish\mathtt{Swish}, and Mish\mathtt{Mish}.

Shijun Zhang, Jianfeng Lu, Hongkai Zhao

Department of Mathematics Duke University

ReLU Neural NetworksNeural Network Activation FunctionsNeural Network Approximation TheoryNeural Network Expressivity
AskPDF