ReLU Neural Networks

ReLU: Rectified Linear Unit

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 60

All topics

Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity
Jul 6, 2026Julia Nakhleh, Robert D. NowakReLU Neural Networks Neural Network Approximation Theory
Department of Computer Science · University of Wisconsin-Madison · Department of Electrical & Computer Engineering
Compressed Computation under $L^4$ Loss is likely Computation in Superposition
Jul 6, 2026Francisco Ferreira da Silva, Stefan HeimersheimReLU Neural Networks Feedforward Neural Networks
Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One
Jul 6, 2026Qi Zhou, Xuan Zhou, Xiao-Song YangReLU Neural Networks Residual Learning
School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China. · Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China.
Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse
Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural Networks ReLU Neural Networks
Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences
Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks
Jun 29, 2026Clemens Kinn, Philipp PetersenReLU Neural Networks Binary Classification
Faculty of Mathematics, University of Vienna, 1090 Vienna, Austria
SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
Jun 29, 2026Tyler LaBonte, Vidya MuthukumarRepresentation Learning ReLU Neural Networks
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology · School of Electrical and Computer Engineering, Georgia Institute of Technology
Recurrent neural networks approximate continuous functions
Jun 18, 2026Valentin Abadie, Clemens Hutter, Helmut BölcskeiReLU Neural Networks Recurrent Neural Networks
Robust Regression of General ReLUs with Queries
Jun 9, 2026Ilias Diakonikolas, Daniel M. Kane, Mingchen MaReLU Neural Networks Active Learning
University of Wisconsin-Madison · University of California, San Diego
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
Jun 8, 2026Claudio NordioBackpropagation ReLU Neural Networks
Characterizing the Discrete Geometry of ReLU Networks
Jun 5, 2026Blake B. Gaines, Jinbo BiNeural Representation Geometry ReLU Neural Networks
Department of Computer Science University of Connecticut
Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks
Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Neural Network Generalization ReLU Neural Networks
Mathematical Institute for Machine Learning and Data Science, KU Eichstätt-Ingolstadt, Ingolstadt, Germany · Department of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, Germany · Department of Mathematics, University of Hong Kong, Hong Kong, China +1
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangReLU Neural Networks Deep Linear Networks
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations
Jun 4, 2026Yizhe Ding, Runze Li, Jia Liu +1ReLU Neural Networks Neural Network Approximation Theory
Department of Statistics, The Pennsylvania State University
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
Jun 3, 2026Berk Tinaz, Changzhi Xie, Mahdi SoltanolkotabiReLU Neural Networks Neural Network Optimization
Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA
Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks
May 29, 2026Yunfei Yang, Jun FanReLU Neural Networks Neural Network Approximation Theory
School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China. · Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong, China.
Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias
May 26, 2026James Town, Etienne Boursier, Ben Lewis +2ReLU Neural Networks Neural Network Optimization
University of Warwick · INRIA LMO, Université Paris-Saclay
On the Epistemic Uncertainty of Overparametrized Neural Networks
May 24, 2026David RügamerBayesian Neural Networks Parameter Identifiability
Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany
The Symmetries of Three-Layer ReLU Networks
May 18, 2026Johanna Marie Gegenfurtner, Moritz Grillo, Guido MontúfarParameter Identifiability ReLU Neural Networks
Technical University of Denmark · Max Planck Institute for Mathematics in the Sciences · UCLA and MPI MiS
Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes
May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural Networks Activation Sparsity
Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning
May 6, 2026Anastasis Kratsios, A. Martina Neuman, Philipp PetersenReLU Neural Networks In-Context Learning
McMaster University and the Vector Institute, Main St., Hamilton, L8S 4L8, Ontario, Canada · University of Vienna, Faculty of Mathematics, Kolingasse 14-16, 1090 Wien, Austria
Exact ReLU realization of tensor-product refinement iterates
May 5, 2026Tsogtgerel GantumurReLU Neural Networks Neural Network Approximation Theory
McGill University, Montréal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology
Most ReLU Networks Admit Identifiable Parameters
May 5, 2026Moritz Grillo, Guido MontúfarRepresentation Identifiability Parameter Identifiability
Max Planck Institute for Mathematics in the Sciences, Leipzig · Departments of Mathematics and Statistics & Data Science, UCLA
Exact Loop Controllers for ReLU Realization of Homogeneous Curve Refinements
May 3, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurReLU Neural Networks Neural Network Approximation Theory
1McGill University · 2National University of Mongolia · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences
Primitive Recursion without Composition: Dynamical Characterizations, from Neural Networks to Polynomial ODEs
Apr 27, 2026Olivier BournezDynamical Systems ReLU Neural Networks
Institut Polytechnique de Paris, Ecole Polytechnique, LIX, Palaiseau, France
Explicit integral representations and quantitative bounds for two-layer ReLU networks
Apr 25, 2026Anthony LeeReLU Neural Networks Reproducing Kernel Hilbert Spaces
School of Mathematics, University of Bristol
A Complete Symmetry Classification of Shallow ReLU Networks
Apr 15, 2026Pranavkrishnan RamakrishnanShallow Neural Networks Neural Representation Geometry
Complete Identification of Deep ReLU Networks through Łukasiewicz Logic
Jan 30, 2026Yani Zhang, Helmut BölcskeiParameter Identifiability ReLU Neural Networks
Chair for Mathematical Information Science, ETH Zürich
Circuit realization and hardware linearization of monotone operator equilibrium networks
Sep 17, 2025Thomas ChaffeyReLU Neural Networks Neuromorphic Computing
School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia
Near-optimal estimates for the $\ell^p$ -Lipschitz constants of deep random ReLU neural networks
Jun 24, 2025Sjoerd Dirksen, Patrick Finke, Paul Geuchen +2ReLU Neural Networks Lipschitz Neural Networks
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions
Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural Networks Neural Network Activation Functions
Department of Mathematics Duke University

Jul 6, 2026, stat.MLJ/K move · Enter open · S save

Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

Julia Nakhleh, Robert D. Nowak

Department of Computer Science · University of Wisconsin-Madison · Department of Electrical & Computer Engineering

ReLU Neural Networks Neural Network Approximation Theory Neural Network Expressivity

Open paper AskPDF arXiv

We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as

\ell^1

-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories that are largely specialized to homogeneous activations such as the ReLU, our framework provides a meaningful notion of functional complexity for deep networks with a broad range of homogeneous and non-homogeneous activation functions commonly used in practice. This simple construction unites several seemingly disparate ideas from the literature, including norm-based complexity bounds and variational characterizations of depth, and facilitates novel analyses of what kinds of functions deep norm-constrained networks can represent. To this end, we prove a novel representer theorem for our spaces and establish novel function-space complexity bounds showing that the associated function classes remain qualitatively small at arbitrary depth. In the univariate ReLU case, we prove a "depth saturation" result: depth in this setting yields only a small constant rescaling of the function class, with no added functional diversity. As a consequence, we show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This finding reveals that some commonly cited expressivity benefits of depth disappear once network complexity is controlled by an appropriate function space norm, rather than parameter count or other representational costs that permit compounded rescaling across layers. Overall, our results illustrate how a function space perspective yields new structural insights into the relationship between depth and complexity.

Momentum

Related topics

Latest papers 60