ReLU Neural Networks

ReLU: Rectified Linear Unit

Ask about this topic

Momentum

12 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 60

All topics
CardsList

  1. Removing spurious minima for planar features by skip connections

    Oct 1, 2026Jakob Paul Zimmermann, Moritz Grillo, Andrei Balakin +1Shallow Neural NetworksReLU Neural Networks

    Freie Universität Berlin · Technische Universität Berlin

  2. Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus

    Sep 30, 2026Akash KumarRepresentation LearningReLU Neural Networks

    Department of Computer Science & Engineering University of California-San Diego

  3. Minimax rates for learning spectral Barron functions by deep ReLU neural networks

    Sep 30, 2026Songqiu Ma, Yunfei YangReLU Neural NetworksMinimax Estimation

    School of Mathematics (Zhuhai), Sun Yat-sen University, Zhuhai, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

  4. Let the Neurons Die: Exploiting ReLU-Induced Model Degradation

    Sep 28, 2026Kexin Li, Wenjun Qiu, Joshua Abraham +2ReLU Neural NetworksAdversarial Attacks

    University of Toronto, Toronto, Ontario, Canada

  5. Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks

    Sep 28, 2026Etienne Boursier, Nicolas FlammarionRepresentation LearningFine-Tuning

    INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland

  6. Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections

    Sep 23, 2026Karolina Drabik, Ben Lewis, Antoni Puch +4ReLU Neural NetworksNeural Network Optimization

    Department of Mathematics, Informatics and Mechanics, University of Warsaw, Poland · Department of Computer Science, University of Warwick, UK · INRIA & LMO, Université Paris-Saclay, France

  7. Generalized Deep Regression for Repeated Measurements

    Sep 22, 2026Kexuan LiNeural Network GeneralizationReLU Neural Networks

    Bristol Myers Squibb

  8. Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks

    Sep 17, 2026Xianjun Li, Yunfei YangShallow Neural NetworksReLU Neural Networks

    School of Statistics and Data Science, Capital University of Economics and Business, Beijing, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

  9. Rank and computation of the pathlifting Jacobian of a DAG ReLU network

    Sep 16, 2026Manon VerbockhavenAutomatic DifferentiationReLU Neural Networks

    OCKHAM

  10. Approximating Smooth Functionals with ReLU Networks

    Sep 14, 2026Shuhao JiaoReLU Neural NetworksNeural Network Approximation Theory

    Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.

  11. Exact ReLU realization of binary affine refinement iterates via reflection folding and cone switching

    Sep 14, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurReLU Neural NetworksNeural Network Approximation Theory

    National University of Mongolia · McGill University · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences

  12. Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks

    Sep 8, 2026Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang +2Neural Network GeneralizationReLU Neural Networks

    University of New South Wales · University of Texas at Austin · University of Sydney

  13. Towards an Expressivity-Normalized Energy-Demand Comparison of ANNs and SNNs

    Aug 30, 2026Miriam Kranzlmüller, Pascal Esser, Gitta KutyniokEfficient Neural Network InferenceReLU Neural Networks

    Department of Mathematics LMU Munich, Germany · *Munich Center for Machine Learning (MCML) · †Department of Physics and Technology, University of Tromsø, Norway, DLR-German Aerospace Center, Germany

  14. Every Layer Counts: An Exponential L2L_2 Depth Hierarchy for ReLU Networks

    Aug 24, 2026Itay SafranShallow Neural NetworksReLU Neural Networks

    Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev

  15. The Boolean Power of ReLU

    Aug 12, 2026Pablo Barceló, Floris Geerts, Matthias Lanzinger +2ReLU Neural NetworksGNN Expressivity

    Pontifical Catholic University · IMFD & CENIA, Chile · University of Antwerp +4

  16. Hidden Gauge Controls Feature Specialization in ReLU Networks

    Aug 7, 2026Tongxi WangReLU Neural NetworksNeural Network Training Dynamics

    School of Future Technology Southeast University Nanjing, China

  17. Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples

    Aug 7, 2026Xinliang Liu, Tong Mao, Jinchao XuNeural PDE SolversReLU Neural Networks

    School of Mathematical Sciences, Ocean University of China, Qingdao 266100, China · Shenzhen Loop Area Institute, Shenzhen 518000, P. R. China

  18. A Counterexample to Fourier Alignment in Single-Neuron Modular Addition

    Aug 5, 2026Gautam Neelakantan MemanaShallow Neural NetworksReLU Neural Networks

  19. Tight Worst-Case Bounds for the Smallest Eigenvalue of ReLU NTK Gram Matrices

    Aug 4, 2026Zhao SongReLU Neural NetworksSpectral Methods

  20. Learning Lookahead Lemmas for Neural Network Verification

    Jul 31, 2026Liam Davis, Haoze WuNeural Network VerificationReLU Neural Networks

    Amherst College

  21. New Complexity-Theoretic Frontiers of Tractability for Neural Network Training

    Jul 23, 2026Cornelius Brand, Robert Ganian, Mathis RoctonReLU Neural NetworksNeural Network Optimization

    Algorithms & Complexity Group Vienna University of Technology Favoritenstraße 9-11, 1040 Vienna, Austria

  22. Shallower ReLU Network Representations via Exact Linear Algebra

    Jul 22, 2026Kilian Rueß, Gennadiy Averkov, Florestan Brunck +7Shallow Neural NetworksReLU Neural Networks

    University of Technology Nuremberg · Brandenburg University of Technology Cottbus–Senftenberg · University of Copenhagen +3

  23. Exact ReLU realization of affine one-dimensional refinement iterates via residual memory and offset frames

    Jul 22, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurReLU Neural NetworksIterative Refinement

    2National University of Mongolia · 1McGill University · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences

  24. Neural network realization of binary refinement iterates via a two-chart atlas selector

    Jul 21, 2026Tsogtgerel GantumurReLU Neural NetworksNeural Network Approximation Theory

    McGill University, Montr´eal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology

  25. Effects of width-dependent model hyperparameters and ℓ2\ell_2-regularization on the loss landscape of two-layer ReLU networks

    Jul 18, 2026Haruka Eshima, Makoto YamadaReLU Neural NetworksWeight Decay

    Okinawa Institute of Science and Technology, Okinawa, Japan

  26. Random Parameter Noise Does Not Make Exact ReLU Verification Easy

    Jul 15, 2026Mojtaba SoltanalianNeural Network VerificationReLU Neural Networks

    Department of Electrical and Computer Engineering University of Illinois Chicago

  27. Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

    Jul 12, 2026Yanming Lai, Defeng Sun, Yang WangReLU Neural NetworksNeural Network Approximation Theory

    Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China · Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong, China

  28. Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

    Jul 8, 2026Marcel Kühn, Bernd RosenowReLU Neural NetworksSymmetry Breaking

    Institut für Theoretische Physik, Universität Leipzig, 04103 Leipzig, Germany · ScaDS.AI Dresden/Leipzig, Universität Leipzig, 04105 Leipzig, Germany

  29. On the Principles of Deep Feedforward ReLU Networks

    Jul 8, 2026Changcun HuangReLU Neural NetworksNeural Network Interpretability

    Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

  30. A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

    Jul 7, 2026Arkaprabha Ganguli, Emil ConstantinescuNeural Network GeneralizationKernel Regression

    Mathematics & Computer Science Division, Argonne National Laboratory, Lemont, IL, USA

  31. Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

    Jul 6, 2026Julia Nakhleh, Robert D. NowakReLU Neural NetworksNeural Network Approximation Theory

    Department of Computer Science · University of Wisconsin-Madison · Department of Electrical & Computer Engineering

  32. Compressed Computation under L4L^4 Loss is likely Computation in Superposition

    Jul 6, 2026Francisco Ferreira da Silva, Stefan HeimersheimReLU Neural NetworksFeedforward Neural Networks

  33. Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

    Jul 6, 2026Qi Zhou, Xuan Zhou, Xiao-Song YangReLU Neural NetworksResidual Learning

    School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China. · Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China.

  34. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

    Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural NetworksReLU Neural Networks

    Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences

  35. Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks

    Jun 29, 2026Clemens Kinn, Philipp PetersenReLU Neural NetworksBinary Classification

    Faculty of Mathematics, University of Vienna, 1090 Vienna, Austria

  36. SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

    Jun 29, 2026Tyler LaBonte, Vidya MuthukumarRepresentation LearningReLU Neural Networks

    H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology · School of Electrical and Computer Engineering, Georgia Institute of Technology

  37. Recurrent neural networks approximate continuous functions

    Jun 18, 2026Valentin Abadie, Clemens Hutter, Helmut BölcskeiReLU Neural NetworksRecurrent Neural Networks

  38. Robust Regression of General ReLUs with Queries

    Jun 9, 2026Ilias Diakonikolas, Daniel M. Kane, Mingchen MaReLU Neural NetworksActive Learning

    University of Wisconsin-Madison · University of California, San Diego

  39. Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

    Jun 8, 2026Claudio NordioBackpropagationReLU Neural Networks

  40. Characterizing the Discrete Geometry of ReLU Networks

    Jun 5, 2026Blake B. Gaines, Jinbo BiNeural Representation GeometryReLU Neural Networks

    Department of Computer Science University of Connecticut

  41. Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks

    Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Neural Network GeneralizationReLU Neural Networks

    Mathematical Institute for Machine Learning and Data Science, KU Eichstätt-Ingolstadt, Ingolstadt, Germany · Department of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, Germany · Department of Mathematics, University of Hong Kong, Hong Kong, China +1

  42. Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

    Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangReLU Neural NetworksDeep Linear Networks

    1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1

  43. Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

    Jun 4, 2026Yizhe Ding, Runze Li, Jia Liu +1ReLU Neural NetworksNeural Network Approximation Theory

    Department of Statistics, The Pennsylvania State University

  44. When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks

    Jun 3, 2026Berk Tinaz, Changzhi Xie, Mahdi SoltanolkotabiReLU Neural NetworksNeural Network Optimization

    Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA

  45. Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks

    May 29, 2026Yunfei Yang, Jun FanReLU Neural NetworksNeural Network Approximation Theory

    School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China. · Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong, China.

  46. Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias

    May 26, 2026James Town, Etienne Boursier, Ben Lewis +2ReLU Neural NetworksNeural Network Optimization

    University of Warwick · INRIA LMO, Université Paris-Saclay

  47. On the Epistemic Uncertainty of Overparametrized Neural Networks

    May 24, 2026David RügamerBayesian Neural NetworksParameter Identifiability

    Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany

  48. The Symmetries of Three-Layer ReLU Networks

    May 18, 2026Johanna Marie Gegenfurtner, Moritz Grillo, Guido MontúfarParameter IdentifiabilityReLU Neural Networks

    Technical University of Denmark · Max Planck Institute for Mathematics in the Sciences · UCLA and MPI MiS

  49. Bug or Feature2^2: Weight Drift, Activation Sparsity and Spikes

    May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural NetworksActivation Sparsity

  50. Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning

    May 6, 2026Anastasis Kratsios, A. Martina Neuman, Philipp PetersenReLU Neural NetworksIn-Context Learning

    McMaster University and the Vector Institute, Main St., Hamilton, L8S 4L8, Ontario, Canada · University of Vienna, Faculty of Mathematics, Kolingasse 14-16, 1090 Wien, Austria

  51. Exact ReLU realization of tensor-product refinement iterates

    May 5, 2026Tsogtgerel GantumurReLU Neural NetworksNeural Network Approximation Theory

    McGill University, Montréal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology

  52. Most ReLU Networks Admit Identifiable Parameters

    May 5, 2026Moritz Grillo, Guido MontúfarRepresentation IdentifiabilityParameter Identifiability

    Max Planck Institute for Mathematics in the Sciences, Leipzig · Departments of Mathematics and Statistics & Data Science, UCLA

  53. Exact Loop Controllers for ReLU Realization of Homogeneous Curve Refinements

    May 3, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurReLU Neural NetworksNeural Network Approximation Theory

    1McGill University · 2National University of Mongolia · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences

  54. Primitive Recursion without Composition: Dynamical Characterizations, from Neural Networks to Polynomial ODEs

    Apr 27, 2026Olivier BournezDynamical SystemsReLU Neural Networks

    Institut Polytechnique de Paris, Ecole Polytechnique, LIX, Palaiseau, France

  55. Explicit integral representations and quantitative bounds for two-layer ReLU networks

    Apr 25, 2026Anthony LeeReLU Neural NetworksReproducing Kernel Hilbert Spaces

    School of Mathematics, University of Bristol

  56. A Complete Symmetry Classification of Shallow ReLU Networks

    Apr 15, 2026Pranavkrishnan RamakrishnanShallow Neural NetworksNeural Representation Geometry

  57. Complete Identification of Deep ReLU Networks through Łukasiewicz Logic

    Jan 30, 2026Yani Zhang, Helmut BölcskeiParameter IdentifiabilityReLU Neural Networks

    Chair for Mathematical Information Science, ETH Zürich

  58. Circuit realization and hardware linearization of monotone operator equilibrium networks

    Sep 17, 2025Thomas ChaffeyReLU Neural NetworksNeuromorphic Computing

    School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia

  59. Near-optimal estimates for the ℓp\ell^p-Lipschitz constants of deep random ReLU neural networks

    Jun 24, 2025Sjoerd Dirksen, Patrick Finke, Paul Geuchen +2ReLU Neural NetworksLipschitz Neural Networks

  60. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

    Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural NetworksNeural Network Activation Functions

    Department of Mathematics Duke University