ReLU Neural Networks

ReLU: Rectified Linear Unit

ReLU neural networks build piecewise-linear functions by composing affine maps with a thresholding activation, making them a useful model for studying how depth and width represent data. Approximation theory examines how efficiently they represent smooth, compositional, or irregular functions, while VC and Rademacher analyses connect model capacity to sample efficiency and generalization. Gradient-flow, neural tangent kernel, and mean-field methods study optimization and implicit bias; polyhedral geometry and formal verification help analyze linear regions, parameter symmetries, and robustness. A central tension is that depth can increase expressivity without necessarily making optimization or generalization easier, while equivalent parameterizations and data-dependent assumptions complicate claims about what training learns.

Latest papers

113 papers24 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

ReLU Networks for Exact Generation of Similar Graphs
Apr 7, 2026Mamoona Ghafoor, Tatsuya AkutsuGraph Generation Constrained Generative Modeling
Bioinformatics Center, Institute for Chemical Research, Department of Intelligence Sciences and Technology, Kyoto University, Uji 611-0011, Japan
Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective
Mar 25, 2026Emi Zeger, Mert PilanciReLU Neural Networks Deep Learning Optimization
Department of Electrical Engineering, Stanford University
Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic
Mar 24, 2026Anand SwaroopGrokking Neural Network Interpretability
$λ$ -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks
Mar 23, 2026Cristian Pérez-Corral, Alberto Fernández-Hernández, Jose I. Mestre +2Neural Network Activation Functions ReLU Neural Networks
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain
Neural Networks as Local-to-Global Computations
Mar 16, 2026Vicente Bosca, Robert GhristSheaf Neural Networks Feedforward Neural Networks
Department of Mathematics, University of Pennsylvania
Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks
Mar 11, 2026ZeYu Li, FengLei Fan, TieYong ZengNeural Network Approximation Theory ReLU Neural Networks
Chinese University of Hong Kong, Department of Mathematics, Hong Kong, 999077, NT, Hong Kong SAR, China · City University of Hong Kong, Department of Data Science, Hong Kong, 999077, Kowloon, Hong Kong SAR, China · Guangzhou Nanfang College, School of Mathematics and Statistics, Guangzhou, 510970, Guangdong, China +1
Memorization capacity of deep ReLU neural networks characterized by width and depth
Mar 10, 2026Xin Yang, Yunfei YangNeural Network Memorization ReLU Neural Networks
School of Mathematics (Zhuhai), Sun Yat-sen University, Zhuhai, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.
How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?
Mar 5, 2026Kuo-Wei Lai, Guanghui Wang, Molei Tao +1High-Dimensional Linear Regression Random Feature Methods
School of Electrical and Computer Engineering, Georgia Institute of Technology · College of Computing, Georgia Institute of Technology · School of Mathematics, Georgia Institute of Technology +1
Path-conditioned training: a principled way to rescale ReLU neural networks
Feb 23, 2026Arthur Lebeurrier, Titouan Vayer, Rémi GribonvalReLU Neural Networks Neural Network Training Dynamics
ENS de Lyon, CNRS, Inria, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France · Inria, Rennes, France · Inria, CNRS, ENS de Lyon, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France
Online Realizable Regression and Applications for ReLU Networks
Feb 22, 2026Ilan Doron-Arad, Idan Mehalel, Elchanan MosselStatistical Learning Theory ReLU Neural Networks
Massachusetts Institute of Technology, Cambridge, MA, USA · The Hebrew University of Jerusalem, Jerusalem, Israel
Why ReLU? A Bit-Model Dichotomy for Deep Network Training
Feb 22, 2026Ilan Doron-Arad, Elchanan MosselEmpirical Risk Minimization ReLU Neural Networks
MIT, Cambridge, MA, USA
Theoretical Lower Bounds on the Robustness of Deep ReLU Networks
Feb 21, 2026Věra KůrkováNeural Network Robustness Adversarial Robustness
Institute of Computer Science of the Czech Academy of Sciences, Pod Vodárenskou věží 2, 18207 Prague, Czech Republic
From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks
Feb 19, 2026Saveliy BaturinShallow Neural Networks ReLU Neural Networks
Independent Researcher, Russia.
Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
Feb 19, 2026Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko +2Neural Network Training Dynamics ReLU Neural Networks
Computer Science Dept., Rice University, Houston, TX, USA
Continuous-Time Piecewise-Linear Recurrent Neural Networks
Feb 17, 2026Alena Brändle, Lukas Eisenmann, Florian Götz +1Recurrent Neural Networks Dynamical Systems
Dept. of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Mannheim, Germany · Faculty of Physics and Astronomy, Heidelberg University · Hector Institute for Artificial Intelligence in Psychiatry, CIMH +2
Complete Identification of Deep ReLU Networks through Łukasiewicz Logic
Jan 30, 2026Yani Zhang, Helmut BölcskeiParameter Identifiability ReLU Neural Networks
Chair for Mathematical Information Science, ETH Zürich
Circuit realization and hardware linearization of monotone operator equilibrium networks
Sep 17, 2025Thomas ChaffeyReLU Neural Networks Neuromorphic Computing
School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia
Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks
Aug 30, 2025Hyungu Lee, Taehyeong Kim, Hayoung ChoiNeural Network Initialization ReLU Neural Networks
Department of Mathematics & Nonlinear Dynamics and Mathematical Application Center, Kyungpook National University, Daegu 41566, Republic of Korea · Nonlinear Dynamics and Mathematical Application Center, Kyungpook National University, Daegu 41566, Republic of Korea
Optimal estimates for the $\ell^p$ -Lipschitz constants of deep random ReLU neural networks
Jun 24, 2025Sjoerd Dirksen, Patrick Finke, Paul Geuchen +2ReLU Neural Networks Lipschitz Neural Networks
The Computational Complexity of Counting Linear Regions in ReLU Neural Networks
May 22, 2025Moritz Stargalla, Christoph Hertrich, Daniel ReichmanReLU Neural Networks Neural Network Expressivity
University of Technology Nuremberg · Worcester Polytechnic Institute
Networks with Finite VC Dimension: Pro and Contra
Feb 4, 2025Vera Kurkova, Marcello SanguinetiNeural Network Generalization Neural Network Approximation Theory
Institute of Computer Science of the Czech Academy of Sciences Pod Vodárenskou věží, 2 - 18207 Prague, Czech Republic · DIBRIS University of Genova Via Opera Pia, 13 - 16145 Genova, Italy
Dense ReLU Neural Networks for Temporal-spatial Model
Nov 15, 2024Carlos Misael Madrid Padilla, Zhi Zhang, Xiaokai Luo +2Nonparametric Regression ReLU Neural Networks
Department of Statistics and Data Science, Washington University in St. Louis. · Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Notre Dame
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions
Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural Networks Neural Network Activation Functions
Department of Mathematics Duke University

Apr 7, 2026J/K move · Enter open · S save

ReLU Networks for Exact Generation of Similar Graphs

Mamoona Ghafoor, Tatsuya Akutsu

Bioinformatics Center, Institute for Chemical Research, Department of Intelligence Sciences and Technology, Kyoto University, Uji 611-0011, Japan

Graph Generation Constrained Generative Modeling ReLU Neural Networks

Open paper AskPDF arXiv

Generation of graphs constrained by a specified graph edit distance from a source graph is important in applications such as cheminformatics, network anomaly synthesis, and structured data augmentation. Despite the growing demand for such constrained generative models in areas including molecule design and network perturbation analysis, the neural architectures required to provably generate graphs within a bounded graph edit distance remain largely unexplored. In addition, existing graph generative models are predominantly data-driven and depend heavily on the availability and quality of training data, which may result in generated graphs that do not satisfy the desired edit distance constraints. In this paper, we address these challenges by theoretically characterizing ReLU neural networks capable of generating graphs within a prescribed graph edit distance from a given graph. In particular, we show the existence of constant depth and O(n^2 d) size ReLU networks that deterministically generate graphs within edit distance d from a given input graph with n vertices, eliminating reliance on training data while guaranteeing validity of the generated graphs. Experimental evaluations demonstrate that the proposed network successfully generates valid graphs for instances with up to 1400 vertices and edit distance bounds up to 140, whereas baseline generative models fail to generate graphs with the desired edit distance. These results provide a theoretical foundation for constructing compact generative models with guaranteed validity.