ReLU Neural Networks

ReLU: Rectified Linear Unit

Overview

ReLU neural networks build piecewise-linear functions by composing affine maps with a thresholding activation, making them a useful model for studying how depth and width represent data. Approximation theory examines how efficiently they represent smooth, compositional, or irregular functions, while VC and Rademacher analyses connect model capacity to sample efficiency and generalization. Gradient-flow, neural tangent kernel, and mean-field methods study optimization and implicit bias; polyhedral geometry and formal verification help analyze linear regions, parameter symmetries, and robustness. A central tension is that depth can increase expressivity without necessarily making optimization or generalization easier, while equivalent parameterizations and data-dependent assumptions complicate claims about what training learns.

Latest papers

136 papers19 papers in the last four weeks, down 14% on the four weeks before. 0.1% of all new papers.

Jan 29, 2026

Exact Gaussian Moment Matching for Residual Networks: a Second-Order Method

We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On large-variance random networks, median KL-divergence improvements over common baselines range from tens to thousands of times. On a variational Bayes neural network, we show that our method improves upon a state-of-the-art deterministic inference method by an order of magnitude in KL divergence from Monte Carlo ground truth. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.
Jan 23, 2026

Multigrade Neural Network Approximation

We study multigrade deep learning (MGDL) as a principled framework for structured error refinement in deep neural networks. While the approximation power of neural networks is now relatively well understood, training very deep architectures remains challenging due to highly nonconvex and often ill-conditioned optimization landscapes. In contrast, for relatively shallow networks, most notably certain one-hidden-layer ReLU models, training admits convex reformulations with global guarantees under appropriate settings, motivating learning paradigms that improve stability while scaling to depth. MGDL builds on this insight by training deep networks grade by grade: previously learned grades are frozen, and each newly added grade-wise subnetwork is composed on top of the previously learned grades and trained to fit the residual left by the current approximation, yielding a structured and interpretable hierarchical refinement process. We develop an operator-theoretic foundation for MGDL and prove that, for any continuous target function defined on a hypercube, there exists a fixed-width multigrade ReLU scheme whose residuals are pointwise nonincreasing in magnitude and converge uniformly to zero, with strict LpL^p-norm decay at every nontrivial grade for p∈[1,∞)p\in [1,\infty). To the best of our knowledge, this work provides the first rigorous constructive approximation guarantee showing that a grade-wise residual refinement scheme can achieve vanishing error in a fixed-width multigrade ReLU architecture.
Jan 23, 2026

ReLU Networks for Model Predictive Control: Network Complexity and Performance Guarantees

Recent years have witnessed a resurgence in using ReLU neural networks (NNs) to represent model predictive control (MPC) policies. However, determining the sufficient network complexity to ensure closed-loop guarantees remains a fundamental open problem. This involves a critical precision-complexity trade-off: undersized networks may fail to capture the MPC policy, while oversized ones may outweigh the benefits of ReLU network approximation. In this work, we project the MPC actions onto a tightened constraint set to satisfy the hard constraints, and establish a state-dependent Lipschitz continuity property for the optimal MPC cost function, which enables sharp convergence analysis of the closed-loop system. For the first time, we derive explicit bounds on ReLU network width and depth for approximating MPC policies with closed-loop guarantees. To reduce network complexity for the convergence to user-specified target sets, we propose a non-uniform error framework with a state-aware scaling function to adaptively adjust both the input and output of the ReLU network. Our contributions provide a foundational step toward certifiable ReLU NN-based MPC.
Sep 17, 2025

Circuit realization and hardware linearization of monotone operator equilibrium networks

It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural network in analog hardware. We furthermore show that the gradient of such a circuit can be computed directly in hardware, using a procedure we call hardware linearization. This allows the network to be trained in hardware, which we demonstrate with a device-level circuit simulation. We extend the results to cascades of resistor-diode networks, which can be used to implement feedforward and other asymmetric networks. We finally show that different nonlinear elements give rise to different activation functions, and introduce the novel diode ReLU which is induced by a non-ideal diode model.
Sep 7, 2025

Toric geometry of ReLU neural networks

Given a continuous piecewise linear function f:Rn0→Rf:\mathbb{R}^{n_0} \to \mathbb{R} and a fixed architecture of feedforward ReLU neural networks, the exact function realization problem asks when a network of that architecture realizes ff. In contrast, expressivity concerns characterizing all functions exactly realizable by a given architecture. To address these questions systematically, we establish a connection between toric geometry and ReLU neural networks, allowing tools from algebraic geometry to analyze the structures. Starting with an unbiased ReLU network with rational weights, we define the ReLU fan, the ReLU toric variety, and the ReLU Cartier divisor associated with the network. This work also relates the tropical and the toric geometry: the former captures combinatorial data of ReLU networks, while the latter recovers the underlying geometric structure. As an application, we obtain a complete characterization of the functions realizable by unbiased shallow ReLU networks with rational weights. The characterization is expressed in terms of intersection numbers of ReLU Cartier divisors with torus-invariant curves and leads to a decomposition of piecewise-linear functions into shallow and non-shallow components.
Aug 30, 2025

Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying ReLU and exacerbate gradient instability as network depth increases. Methods such as He, Xavier, and orthogonal initialization preserve variance or promote approximate isometry. However, they do not necessarily regulate the pre-activation mean or control activation sparsity, and their effectiveness often diminishes in very deep architectures. This work introduces a semi-orthogonal initialization specifically optimized for ReLU by solving an optimization problem on the Stiefel manifold, thereby preserving scale and calibrating the pre-activation statistics from the outset. A family of closed-form solutions and an efficient sampling scheme are derived. Theoretical analysis at initialization shows that the proposed initialization mitigates dying ReLU, slows the decay of activation variance, and alleviates gradient vanishing, thereby stabilizing signal and gradient flow in deep architectures. Empirically, across MNIST, Fashion-MNIST, multiple tabular datasets, few-shot settings, and ReLU-family activations, the proposed initialization maintains stable performance as network depth increases, with particularly clear advantages in few-shot learning and large-depth settings.
Jun 24, 2025

Optimal estimates for the ℓp\ell^p-Lipschitz constants of deep random ReLU neural networks

This paper studies the ℓp\ell^p-Lipschitz constants of ReLU neural networks Φ:Rd→RΦ: \mathbb{R}^d \to \mathbb{R} with random parameters for p∈[1,∞]p \in [1,\infty]. The distribution of the weights follows a variant of the He initialization. In the case of zero-bias networks, we derive matching high-probability upper and lower bounds for wide networks that differ at most by an absolute constant. The bounds reveal two remarkable phenomena. First, the bounds grow root-logarithmically in terms of the depth of the net. Second, the behavior of the ℓp\ell^p-Lipschitz constants varies significantly between the regimes p∈[1,2)p \in [1,2) and p∈[2,∞]p \in [2,\infty]. For p∈[2,∞]p \in [2,\infty], the ℓp\ell^p-Lipschitz constant behaves similarly to ∥g∥p′\|g\|_{p'}, where g∈Rdg \in \mathbb{R}^d is a standard Gaussian vector and 1/p+1/p′=11/p + 1/p' = 1. In contrast, for p∈[1,2)p \in [1,2), the ℓp\ell^p-Lipschitz constant aligns more closely with ∥g∥2\|g\|_2. For networks with possibly nonzero biases, we obtain the same lower bounds without assumptions on the bias distributions. For symmetric bias distributions with bounded densities, we also derive high-probability upper bounds for the ℓp\ell^p-Lipschitz constants. For fixed sub-exponential bias distributions, these upper bounds differ from the lower bounds by a factor that has square-root dependence on the depth and logarithmic dependence on the width.
May 22, 2025

The Computational Complexity of Counting Linear Regions in ReLU Neural Networks

An established measure of the expressive power of a given ReLU neural network is the number of linear regions into which it partitions the input space. There exist many different, non-equivalent definitions of what a linear region actually is. We systematically assess which papers use which definitions and discuss how they relate to each other. We then analyze the computational complexity of counting the number of such regions for the various definitions. Generally, this turns out to be an intractable problem. We prove NP- and #P-hardness results already for networks with one hidden layer and strong hardness of approximation results for two or more hidden layers. Finally, on the algorithmic side, we demonstrate that counting linear regions can at least be achieved in polynomial space for some common definitions.
May 16, 2025

Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization

Benign overfitting occurs when a model fits noisy training data exactly and still generalizes well. We establish this behavior for finite-width two-layer networks with the standard ReLU activation, trained by gradient flow in fixed dimension. Our data-independent initialization combines Gaussian weights with neurons initially active only on shrinking spherical caps. For uniform spherical inputs and any fixed bounded conditional response law, training and excess risks vanish in probability uniformly over all sufficiently late times. The parameters also converge, with probability tending to one, to an interpolating limit whose excess risk vanishes in probability. We use one schedule depending only on sample size and dimension. Our proof connects the network to a kernel flow whose limit combines ridge regression with localized spikes that fit the training residuals. We give explicit sample-size, width and training-time bounds.
May 9, 2025

On the Depth of Monotone ReLU Neural Networks and ICNNs

We study two models of ReLU\mathsf{ReLU} neural networks: monotone networks (ReLU+\mathsf{ReLU}^+) and input convex neural networks (ICNN\mathsf{ICNN}). Our focus is on expressivity, mostly in terms of depth, and our motivation is to gain a better understanding of the depth required to exactly represent functions using neural networks with ReLU activations, a subject that has received a lot of attention lately. We prove several lower bounds on the depth required to represent functions by monotone ReLU networks and ICNN\mathsf{ICNN}. For the maximum function MAXn\mathsf{MAX}_n computing the maximum of nn real numbers, we show that ReLU+\mathsf{ReLU}^+ networks cannot compute MAXn\mathsf{MAX}_n, or even approximate it. We prove a sharp lower bound of n−1n-1 on the ICNN\mathsf{ICNN} depth complexity of MAXn\mathsf{MAX}_n. We also prove depth separations between ReLU\mathsf{ReLU} networks and ICNN\mathsf{ICNN}s; for every kk, there is a depth-22 ReLU\mathsf{ReLU} network of size O(k2)O(k^2) that cannot be simulated by a depth-kk ICNN\mathsf{ICNN}. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.
Feb 4, 2025

Networks with Finite VC Dimension: Pro and Contra

Approximation and learning of classifiers of large data sets by neural networks in terms of high-dimensional geometry and statistical learning theory are investigated. The influence of the VC dimension of sets of input-output functions of networks on approximation capabilities is compared with its influence on consistency in learning from samples of data. It is shown that, whereas finite VC dimension is desirable for uniform convergence of empirical errors, it may not be desirable for approximation of functions drawn from a probability distribution modeling the likelihood that they occur in a given type of application. Based on the concentration-of-measure properties of high dimensional geometry, it is proven that both errors in approximation and empirical errors behave almost deterministically for networks implementing sets of input-output functions with finite VC dimensions in processing large data sets. Practical limitations of the universal approximation property, the trade-offs between the accuracy of approximation and consistency in learning from data, and the influence of depth of networks with ReLU units on their accuracy and consistency are discussed.
Nov 15, 2024

Dense ReLU Neural Networks for Temporal-spatial Model

In this paper, we focus on fully connected deep neural networks utilizing the Rectified Linear Unit (ReLU) activation function for nonparametric estimation. We derive non-asymptotic bounds that lead to convergence rates, addressing both temporal and spatial dependence in the observed measurements. By accounting for dependencies across time and space, our models better reflect the complexities of real-world data, enhancing both predictive performance and theoretical robustness. We also tackle the curse of dimensionality by modeling the data on a manifold, exploring the intrinsic dimensionality of high-dimensional data. We broaden existing theoretical findings of temporal-spatial analysis by applying them to neural networks in more general contexts and demonstrate that our proof techniques are effective for models with short-range dependence. Our empirical simulations across various synthetic response functions underscore the superior performance of our method, outperforming established approaches in the existing literature. These findings provide valuable insights into the strong capabilities of dense neural networks (Dense NN) for temporal-spatial modeling across a broad range of function classes.
Oct 9, 2024

Expressiveness of Multi-Neuron Convex Relaxations in Neural Network Certification

Neural network certification methods heavily rely on convex relaxations to provide robustness guarantees. However, these relaxations are often imprecise: even the most accurate single-neuron relaxation is incomplete for general ReLU networks, a limitation known as the single-neuron convex barrier. While multi-neuron relaxations have been heuristically applied to address this issue, two central questions arise: (i) whether they overcome the convex barrier, and if not, (ii) whether they offer theoretical capabilities beyond those of single-neuron relaxations. In this work, we present the first rigorous analysis of the expressiveness of multi-neuron relaxations. Perhaps surprisingly, we show that they are inherently incomplete, even when allocated sufficient resources to capture finitely many neurons and layers optimally. This result extends the single-neuron barrier to a universal convex barrier for neural network certification. On the positive side, we show that completeness can be achieved by either (i) augmenting the network with a polynomial number of carefully designed ReLU neurons or (ii) partitioning the input domain into convex sub-polytopes, thereby distinguishing multi-neuron relaxations from single-neuron ones which are unable to realize the former and have worse partition complexity for the latter. Our findings establish a foundation for multi-neuron relaxations and point to new directions for certified robustness, including training methods tailored to multi-neuron relaxations and verification methods with multi-neuron relaxations as the main subroutine.
Oct 8, 2024

Benign Overfitting for Regression with Trained Two-Layer ReLU Networks

We study the least-square regression problem with a two-layer fully-connected neural network, with ReLU activation function, trained by gradient flow. Our first result is a generalization result, that requires no assumptions on the underlying regression function or the noise other than that they are bounded. We operate in the neural tangent kernel regime, and our generalization result is developed via a decomposition of the excess risk into estimation and approximation errors, viewing gradient flow as an implicit regularizer. This decomposition in the context of neural networks is a novel perspective of gradient descent, and helps us avoid uniform convergence traps. In this work, we also establish that under the same setting, the trained network overfits to the data. Together, these results, establishes the first result on benign overfitting for finite-width ReLU networks for arbitrary regression functions.
Jun 21, 2024

On the growth of the parameters of approximating ReLU neural networks

This work focuses on the analysis of fully connected feed forward ReLU neural networks as they approximate a given, smooth function. In contrast to conventionally studied universal approximation properties under increasing architectures, e.g., in terms of width or depth of the networks, we are concerned with the asymptotic growth of the parameters of approximating networks. Such results are of interest, e.g., for error analysis or consistency results for neural network training. The main result of our work is that, for a ReLU architecture with state of the art approximation error, the realizing parameters grow at most polynomially. The obtained rate with respect to a normalized network size is compared to existing results and is shown to be superior in most cases, in particular for high dimensional input.
Jul 13, 2023

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set A\mathscr{A} is defined to encompass the majority of commonly used activation functions, such as ReLU\mathtt{ReLU}, LeakyReLU\mathtt{LeakyReLU}, ReLU2\mathtt{ReLU}^2, ELU\mathtt{ELU}, CELU\mathtt{CELU}, SELU\mathtt{SELU}, Softplus\mathtt{Softplus}, GELU\mathtt{GELU}, SiLU\mathtt{SiLU}, Swish\mathtt{Swish}, Mish\mathtt{Mish}, Sigmoid\mathtt{Sigmoid}, Tanh\mathtt{Tanh}, Arctan\mathtt{Arctan}, Softsign\mathtt{Softsign}, dSiLU\mathtt{dSiLU}, and SRS\mathtt{SRS}. We demonstrate that for any activation function ϱ∈A\varrho\in \mathscr{A}, a ReLU\mathtt{ReLU} network of width NN and depth LL can be approximated to arbitrary precision by a ϱ\varrho-activated network of width 3N3N and depth 2L2L on any bounded set. This finding enables the extension of most approximation results achieved with ReLU\mathtt{ReLU} networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width, \,depth) scaling factors can be further reduced from (3,2)(3,2) to (1,1)(1,1) if ϱ\varrho falls within a specific subset of A\mathscr{A}. This subset includes activation functions such as ELU\mathtt{ELU}, CELU\mathtt{CELU}, SELU\mathtt{SELU}, Softplus\mathtt{Softplus}, GELU\mathtt{GELU}, SiLU\mathtt{SiLU}, Swish\mathtt{Swish}, and Mish\mathtt{Mish}.