ReLU Neural Networks

ReLU: Rectified Linear Unit

Overview

ReLU neural networks build piecewise-linear functions by composing affine maps with a thresholding activation, making them a useful model for studying how depth and width represent data. Approximation theory examines how efficiently they represent smooth, compositional, or irregular functions, while VC and Rademacher analyses connect model capacity to sample efficiency and generalization. Gradient-flow, neural tangent kernel, and mean-field methods study optimization and implicit bias; polyhedral geometry and formal verification help analyze linear regions, parameter symmetries, and robustness. A central tension is that depth can increase expressivity without necessarily making optimization or generalization easier, while equivalent parameterizations and data-dependent assumptions complicate claims about what training learns.

Latest papers

CardsList

136 papers19 papers in the last four weeks, down 14% on the four weeks before. 0.1% of all new papers.

  1. Exact Gaussian Moment Matching for Residual Networks: a Second-Order Method

    Jan 29, 2026Simon Kuang, Xinfan LinReLU Neural NetworksBayesian Neural Networks

  2. Multigrade Neural Network Approximation

    Jan 23, 2026Shijun Zhang, Zuowei Shen, Yuesheng XuNeural Network Approximation TheoryReLU Neural Networks

  3. ReLU Networks for Model Predictive Control: Network Complexity and Performance Guarantees

    Jan 23, 2026Xingchen Li, Keyou YouModel Predictive ControlReLU Neural Networks

  4. Circuit realization and hardware linearization of monotone operator equilibrium networks

    Sep 17, 2025Thomas Chaffeyanalog neural networksReLU Neural Networks

  5. Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

    Aug 30, 2025Hyungu Lee, Taehyeong Kim, Hayoung ChoiNeural Network InitializationReLU Neural Networks

  6. The Computational Complexity of Counting Linear Regions in ReLU Neural Networks

    May 22, 2025Moritz Stargalla, Christoph Hertrich, Daniel ReichmanReLU Neural NetworksNeural Network Expressivity

  7. Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization

    May 16, 2025Junhyung Park, Patrick Bloebaum, Shiva Prasad KasiviswanathanBenign OverfittingTwo-Layer Neural Networks

  8. Networks with Finite VC Dimension: Pro and Contra

    Feb 4, 2025Vera Kurkova, Marcello SanguinetiNeural Network GeneralizationNeural Network Approximation Theory

  9. Dense ReLU Neural Networks for Temporal-spatial Model

    Nov 15, 2024Carlos Misael Madrid Padilla, Zhi Zhang, Xiaokai Luo +2Nonparametric RegressionReLU Neural Networks

  10. Expressiveness of Multi-Neuron Convex Relaxations in Neural Network Certification

    Oct 9, 2024Yuhao Mao, Yani Zhang, Martin VechevNeural Network VerificationConvex Relaxation

  11. Benign Overfitting for Regression with Trained Two-Layer ReLU Networks

    Oct 8, 2024Junhyung Park, Patrick Bloebaum, Shiva Prasad KasiviswanathanNeural Network GeneralizationBenign Overfitting

  12. On the growth of the parameters of approximating ReLU neural networks

    Jun 21, 2024Erion Morina, Martin HollerReLU Neural NetworksNeural Network Approximation Theory

  13. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

    Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural NetworksNeural Network Activation Functions