ReLU Neural Networks

ReLU: Rectified Linear Unit

Ask about this topic

Overview

ReLU neural networks build piecewise-linear functions by composing affine maps with a thresholding activation, making them a useful model for studying how depth and width represent data. Approximation theory examines how efficiently they represent smooth, compositional, or irregular functions, while VC and Rademacher analyses connect model capacity to sample efficiency and generalization. Gradient-flow, neural tangent kernel, and mean-field methods study optimization and implicit bias; polyhedral geometry and formal verification help analyze linear regions, parameter symmetries, and robustness. A central tension is that depth can increase expressivity without necessarily making optimization or generalization easier, while equivalent parameterizations and data-dependent assumptions complicate claims about what training learns.

Latest papers

CardsList

113 papers24 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

  1. Para-Gaussian Approximations of Random Deep Neural Networks

    Oct 27, 2026Mihai Nica, Janosch OrtmannNeural Network Approximation TheoryReLU Neural Networks

  2. Intrinsic Information-Theoretic Analysis of ReLU Nets

    Oct 27, 2026Johan Mylius-Kroken, Elisabeth Wetzer, Ali Ramezani-Kebrya +2ReLU Neural NetworksMutual Information Estimation

    UiT The Arctic University of Norway · University of Oslo · Norwegian Computing Center +1

  3. The Geometry of Limits in ReLU Function Spaces

    Oct 24, 2026Kathryn A. LindseyReLU Neural NetworksNeural Network Approximation Theory

  4. Extraction meets Distillation - cryptanalytic techniques to recover parameters from a Deep Neural Network

    Oct 24, 2026Biprarshi Biswas, Arya Anshuman Ojha, Yubaraj Das +5Model Extraction AttacksReLU Neural Networks

    1*Jadavpur University, Kolkata, India. · 2*National Institute of Science Education and Research, Bhubaneswar, India. · 3*Institute of Mathematics and Application, Bhubaneswar, India. +3

  5. Learning Positive Linear Combinations of ReLUs in Fully Polynomial Time

    Oct 23, 2026Ilias Diakonikolas, Jelena Diakonikolas, Puqian Wang +1PAC LearningReLU Neural Networks

    UW Madison · CSAIL, MIT

  6. Deep ReLU Networks Can Fail the Best Approximation Property for Continuous Target Functions

    Oct 22, 2026Sebastian KassingNeural Network Approximation TheoryReLU Neural Networks

  7. Depth Separation for Neural Networks under Gaussian Measure

    Oct 20, 2026Cynthia Bortolotto, Francisco Ganacim, Roberto I. OliveiraNeural Network Approximation TheoryReLU Neural Networks

  8. Two ReLU⁡\operatorname{ReLU} Layers is All You Need

    Oct 20, 2026Florestan Brunck, Christoph Hertrich, Kilian Ruess +2ReLU Neural NetworksShallow Neural Networks

  9. Two hidden layer suffice: the caterpillar net

    Oct 19, 2026Jakob Paul ZimmermannReLU Neural NetworksShallow Neural Networks

    FU Berlin · TU Berlin

  10. Polynomial Learning of Positive ReLU Networks under Gaussian Inputs

    Oct 18, 2026Zhijie ZhengReLU Neural Networks

    Peking University

  11. "Deep lifts": Neural networks can generalize even with the dumbest optimizer

    Oct 16, 2026Sunil Madhow, Yu-Xiang WangNeural Network GeneralizationGeneralization Bounds

    UC San Diego

  12. A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability

    Oct 16, 2026Quan Nguyen, Robin Yadav, Param Mody +1Neural Network OptimizationReLU Neural Networks

    University of British Columbia · Canada CIFAR AI Chair (Amii)

  13. Optimal approximation of smooth functions by ReLU neural networks

    Oct 15, 2026Thomas NaglerNeural Network Approximation TheoryReLU Neural Networks

    Department of Statistics, LMU Munich Munich Center for Machine Learning (MCML)

  14. The minimal width of neural networks and immersions of real projective space

    Oct 14, 2026BaoZhi Chu, Ben WeinkoveNeural Network Approximation TheoryNeural Network Expressivity

  15. The Surprising Expressivity of a Single Neuron with Autoregressive Chain-of-Thought

    Oct 12, 2026Roey Magen, Nirmit Joshi, Gal VardiNeural Network ExpressivityCoT Reasoning

    Weizmann Institute of Science · Toyota Technological Institute at Chicago

  16. Constructive Bounds for the Topological Expressivity of Neural Networks

    Oct 9, 2026Kosio BeshkovNeural Network ExpressivityReLU Neural Networks

    Department of Physics, University of Oslo, Oslo, Norway.

  17. Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

    Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training DynamicsStochastic Gradient Descent

    City University of Hong Kong · Microsoft · Copula Lab +1

  18. NP-Hardness of Minimizing Neurons in Two-Hidden-Layer ReLU Neural Networks

    Oct 8, 2026Sangrock LeeReLU Neural NetworksNeural Network Approximation Theory

    FAMU–FSU College of Engineering, Florida State University, Tallahassee, FL, 32310

  19. Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks

    Oct 3, 2026Cheng Xu, Pengxiao Lin, Zhangchen Zhou +1Neural Network Training DynamicsWeight Decay

    School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University

  20. A Hand-Checkable Proof That Two Hidden ReLU Layers Compute the Maximum of Six Numbers

    Oct 3, 2026Dimitrios MyrisiotisReLU Neural NetworksNeural Network Expressivity

    Great Bay University China

  21. Removing spurious minima for planar features by skip connections

    Oct 1, 2026Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1Shallow Neural NetworksReLU Neural Networks

    Freie Universität Berlin · Technische Universität Berlin

  22. Approximation Property of Dropout Neural Networks: Sobolev Rates and Confidence Bounds

    Sep 30, 2026Jia-He YaoNeural Network Approximation TheoryFeedforward Neural Networks

  23. Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus

    Sep 30, 2026Akash KumarRepresentation LearningReLU Neural Networks

    Department of Computer Science & Engineering University of California-San Diego

  24. Minimax rates for learning spectral Barron functions by deep ReLU neural networks

    Sep 30, 2026Songqiu Ma, Yunfei YangReLU Neural NetworksMinimax Estimation

    School of Mathematics (Zhuhai), Sun Yat-sen University, Zhuhai, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

  25. Let the Neurons Die: Exploiting ReLU-Induced Model Degradation

    Sep 28, 2026Kexin Li, Wenjun Qiu, Joshua Abraham +2ReLU Neural NetworksAdversarial Attacks

    University of Toronto, Toronto, Ontario, Canada

  26. Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks

    Sep 28, 2026Etienne Boursier, Nicolas FlammarionRepresentation LearningFine-Tuning

    INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland

  27. Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas

    Sep 26, 2026Recep Özkan, Christian HirschReLU Neural NetworksNeural Network Training Dynamics

    Department of Mathematics, Middle East Technical University (METU), Ankara, Turkey · Department of Mathematics, Aarhus University, Aarhus, Denmark

  28. REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models

    Sep 26, 2026Xiaoran Cheng, Sen Na, Jia LiInterpretable MLNeural Surrogate Modeling

    Department of Statistics, The Pennsylvania State University, University Park, PA 16802 USA · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA

  29. Population loss in shallow ReLU networks: Bias & families of critical points

    Sep 25, 2026Michael FieldShallow Neural NetworksReLU Neural Networks

  30. Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections

    Sep 23, 2026Karolina Drabik, Ben Lewis, Antoni Puch +4ReLU Neural NetworksNeural Network Optimization

    Department of Mathematics, Informatics and Mechanics, University of Warsaw, Poland · Department of Computer Science, University of Warwick, UK · INRIA & LMO, Université Paris-Saclay, France