ReLU Neural Networks

ReLU: Rectified Linear Unit

Overview

ReLU neural networks build piecewise-linear functions by composing affine maps with a thresholding activation, making them a useful model for studying how depth and width represent data. Approximation theory examines how efficiently they represent smooth, compositional, or irregular functions, while VC and Rademacher analyses connect model capacity to sample efficiency and generalization. Gradient-flow, neural tangent kernel, and mean-field methods study optimization and implicit bias; polyhedral geometry and formal verification help analyze linear regions, parameter symmetries, and robustness. A central tension is that depth can increase expressivity without necessarily making optimization or generalization easier, while equivalent parameterizations and data-dependent assumptions complicate claims about what training learns.

Latest papers

CardsList

136 papers19 papers in the last four weeks, down 14% on the four weeks before. 0.1% of all new papers.

  1. Hyperlyve: Parallelized Enumeration and Exact Lyapunov Stability Certification for Deep ReLU Networks

    Nov 16, 2026Brian Yarbrough, Jesse Wayment, Jingbo Wang +2Neural Network StabilityLyapunov Stability

  2. Sharp Covering-Number Bounds for Feedforward ReLU Networks via Parameter Symmetries

    Nov 11, 2026Ruizhi Deng, Guohao Shen, Tangrui XiongReLU Neural NetworksNeural Network Generalization

  3. Geometry-Dependent Approximation and Stability Properties of Shallow ReLUk^k Networks on Bounded Euclidean Domains

    Nov 10, 2026Yuqing Liu, Tong Mao, Jinchao XuShallow Neural NetworksReLU Neural Networks

  4. Scale-Invariant Empirical-Bayes Laplace Approximation for ReLU Networks

    Nov 9, 2026Shivam Pal, Piyush RaiLaplace ApproximationBayesian Neural Networks

  5. Spectral Bias Reduction in Local Gated ReLU Architectures

    Nov 7, 2026Victorita DoleanSpectral BiasReLU Neural Networks

  6. Two-Layer Random ReLU Networks Can Separate Disjoint Classes

    Nov 3, 2026Yichen Liu, Jianbin YangReLU Neural NetworksTwo-Layer Neural Networks

  7. The Complexity of ReLU Network Synthesis from a Parameter Multiset

    Nov 2, 2026Dimitrios MyrisiotisReLU Neural Networks

  8. Iterated Narrow Networks: Turing Completeness at Width 4 on Convex Input Spaces and the Complexity of Verification

    Nov 2, 2026Olivier Bournez, Johanne Cohen, Adrian WurmReLU Neural NetworksNeural Network Verification

  9. Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

    Oct 30, 2026Binghua Li, Mengzhe Li, Denny Wu +1Neural Network Training DynamicsNeural Network Optimization

  10. A Witnessing Duality for ReLU Network Depth

    Oct 28, 2026Dimitrios MyrisiotisReLU Neural NetworksCircuit Complexity

  11. Intrinsic Information-Theoretic Analysis of ReLU Nets

    Oct 27, 2026Johan Mylius-Kroken, Elisabeth Wetzer, Ali Ramezani-Kebrya +2ReLU Neural NetworksMutual Information Estimation

  12. Extraction meets Distillation - cryptanalytic techniques to recover parameters from a Deep Neural Network

    Oct 24, 2026Biprarshi Biswas, Arya Anshuman Ojha, Yubaraj Das +5Model Extraction AttacksReLU Neural Networks

  13. Learning Positive Linear Combinations of ReLUs in Fully Polynomial Time

    Oct 23, 2026Ilias Diakonikolas, Jelena Diakonikolas, Puqian Wang +1PAC LearningReLU Neural Networks

  14. Two ReLU⁡\operatorname{ReLU} Layers is All You Need

    Oct 20, 2026Florestan Brunck, Christoph Hertrich, Kilian Ruess +2ReLU Neural NetworksShallow Neural Networks

  15. A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability

    Oct 16, 2026Quan Nguyen, Robin Yadav, Param Mody +1Neural Network OptimizationReLU Neural Networks

  16. Optimal approximation of smooth functions by ReLU neural networks

    Oct 15, 2026Thomas NaglerNeural Network Approximation TheoryReLU Neural Networks

  17. The Surprising Expressivity of a Single Neuron with Autoregressive Chain-of-Thought

    Oct 12, 2026Roey Magen, Nirmit Joshi, Gal VardiNeural Network ExpressivityCoT Reasoning

  18. Constructive Bounds for the Topological Expressivity of Neural Networks

    Oct 9, 2026Kosio BeshkovNeural Network ExpressivityReLU Neural Networks

  19. Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

    Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training DynamicsStochastic Gradient Descent

  20. NP-Hardness of Minimizing Neurons in Two-Hidden-Layer ReLU Neural Networks

    Oct 8, 2026Sangrock LeeReLU Neural NetworksNeural Network Approximation Theory