ReLU Neural Networks

ReLU: Rectified Linear Unit

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 63

All topics

Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD
Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training Dynamics Stochastic Gradient Descent
City University of Hong Kong · Microsoft · Copula Lab +1
NP-Hardness of Minimizing Neurons in Two-Hidden-Layer ReLU Neural Networks
Oct 8, 2026Sangrock LeeReLU Neural Networks Neural Network Approximation Theory
FAMU–FSU College of Engineering, Florida State University, Tallahassee, FL, 32310
Removing spurious minima for planar features by skip connections
Oct 1, 2026Jakob Paul Zimmermann, Moritz Grillo, Andrei Balakin +1Shallow Neural Networks ReLU Neural Networks
Freie Universität Berlin · Technische Universität Berlin
Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus
Sep 30, 2026Akash KumarRepresentation Learning ReLU Neural Networks
Department of Computer Science & Engineering University of California-San Diego
Minimax rates for learning spectral Barron functions by deep ReLU neural networks
Sep 30, 2026Songqiu Ma, Yunfei YangReLU Neural Networks Minimax Estimation
School of Mathematics (Zhuhai), Sun Yat-sen University, Zhuhai, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.
Let the Neurons Die: Exploiting ReLU-Induced Model Degradation
Sep 28, 2026Kexin Li, Wenjun Qiu, Joshua Abraham +2ReLU Neural Networks Adversarial Attacks
University of Toronto, Toronto, Ontario, Canada
Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks
Sep 28, 2026Etienne Boursier, Nicolas FlammarionRepresentation Learning Fine-Tuning
INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland
Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections
Sep 23, 2026Karolina Drabik, Ben Lewis, Antoni Puch +4ReLU Neural Networks Neural Network Optimization
Department of Mathematics, Informatics and Mechanics, University of Warsaw, Poland · Department of Computer Science, University of Warwick, UK · INRIA & LMO, Université Paris-Saclay, France
Generalized Deep Regression for Repeated Measurements
Sep 22, 2026Kexuan LiNeural Network Generalization ReLU Neural Networks
Bristol Myers Squibb
Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks
Sep 17, 2026Xianjun Li, Yunfei YangShallow Neural Networks ReLU Neural Networks
School of Statistics and Data Science, Capital University of Economics and Business, Beijing, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.
Rank and computation of the pathlifting Jacobian of a DAG ReLU network
Sep 16, 2026Manon VerbockhavenAutomatic Differentiation ReLU Neural Networks
OCKHAM
Approximating Smooth Functionals with ReLU Networks
Sep 14, 2026Shuhao JiaoReLU Neural Networks Neural Network Approximation Theory
Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.
Exact ReLU realization of binary affine refinement iterates via reflection folding and cone switching
Sep 14, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurReLU Neural Networks Neural Network Approximation Theory
National University of Mongolia · McGill University · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences
Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
Sep 8, 2026Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang +2Neural Network Generalization ReLU Neural Networks
University of New South Wales · University of Texas at Austin · University of Sydney
Towards an Expressivity-Normalized Energy-Demand Comparison of ANNs and SNNs
Aug 30, 2026Miriam Kranzlmüller, Pascal Esser, Gitta KutyniokEfficient Neural Network Inference ReLU Neural Networks
Department of Mathematics LMU Munich, Germany · *Munich Center for Machine Learning (MCML) · †Department of Physics and Technology, University of Tromsø, Norway, DLR-German Aerospace Center, Germany
Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks
Aug 24, 2026Itay SafranShallow Neural Networks ReLU Neural Networks
Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev
The Boolean Power of ReLU
Aug 12, 2026Pablo Barceló, Floris Geerts, Matthias Lanzinger +2ReLU Neural Networks GNN Expressivity
Pontifical Catholic University · IMFD & CENIA, Chile · University of Antwerp +4
Hidden Gauge Controls Feature Specialization in ReLU Networks
Aug 7, 2026Tongxi WangReLU Neural Networks Neural Network Training Dynamics
School of Future Technology Southeast University Nanjing, China
Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples
Aug 7, 2026Xinliang Liu, Tong Mao, Jinchao XuNeural PDE Solvers ReLU Neural Networks
School of Mathematical Sciences, Ocean University of China, Qingdao 266100, China · Shenzhen Loop Area Institute, Shenzhen 518000, P. R. China
A Counterexample to Fourier Alignment in Single-Neuron Modular Addition
Aug 5, 2026Gautam Neelakantan MemanaShallow Neural Networks ReLU Neural Networks
Tight Worst-Case Bounds for the Smallest Eigenvalue of ReLU NTK Gram Matrices
Aug 4, 2026Zhao SongReLU Neural Networks Spectral Methods
Learning Lookahead Lemmas for Neural Network Verification
Jul 31, 2026Liam Davis, Haoze WuNeural Network Verification ReLU Neural Networks
Amherst College
New Complexity-Theoretic Frontiers of Tractability for Neural Network Training
Jul 23, 2026Cornelius Brand, Robert Ganian, Mathis RoctonReLU Neural Networks Neural Network Optimization
Algorithms & Complexity Group Vienna University of Technology Favoritenstraße 9-11, 1040 Vienna, Austria
Shallower ReLU Network Representations via Exact Linear Algebra
Jul 22, 2026Kilian Rueß, Gennadiy Averkov, Florestan Brunck +7Shallow Neural Networks ReLU Neural Networks
University of Technology Nuremberg · Brandenburg University of Technology Cottbus–Senftenberg · University of Copenhagen +3
Exact ReLU realization of affine one-dimensional refinement iterates via residual memory and offset frames
Jul 22, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurReLU Neural Networks Iterative Refinement
2National University of Mongolia · 1McGill University · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences
Neural network realization of binary refinement iterates via a two-chart atlas selector
Jul 21, 2026Tsogtgerel GantumurReLU Neural Networks Neural Network Approximation Theory
McGill University, Montr´eal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology
Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks
Jul 18, 2026Haruka Eshima, Makoto YamadaReLU Neural Networks Weight Decay
Okinawa Institute of Science and Technology, Okinawa, Japan
Random Parameter Noise Does Not Make Exact ReLU Verification Easy
Jul 15, 2026Mojtaba SoltanalianNeural Network Verification ReLU Neural Networks
Department of Electrical and Computer Engineering University of Illinois Chicago
Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width
Jul 12, 2026Yanming Lai, Defeng Sun, Yang WangReLU Neural Networks Neural Network Approximation Theory
Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China · Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong, China
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
Jul 8, 2026Marcel Kühn, Bernd RosenowReLU Neural Networks Symmetry Breaking
Institut für Theoretische Physik, Universität Leipzig, 04103 Leipzig, Germany · ScaDS.AI Dresden/Leipzig, Universität Leipzig, 04105 Leipzig, Germany

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai, Jiaqi Wu+2

City University of Hong Kong · Microsoft · Copula Lab+1

Neural Network Training Dynamics Stochastic Gradient Descent Gradient Descent Dynamics ReLU Neural Networks

Open paper AskPDF arXiv

Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism. In a two-unit ReLU regression, a source task drives the two neurons toward positive proportionality and a target task rewards separating them. After source training for time

T

, gradient flow recovers on the target in time linear in

T

. Online SGD with batch size

b

and step size

η

in both phases instead fails with high probability throughout a horizon of order

e^{c/η}

once

T \gtrsim \log(b/η)

, uniformly on an explicit set of initializations with Gaussian probability above one percent. For each fixed

T

, small-step SGD still recovers, so the failure requires the joint limit of small steps and long pretraining. At the target clone, the population instability is carried entirely by inputs on which the two ReLU gates disagree. For units at angle

δ

these inputs form a wedge of probability

δ/π

, and weight decay shrinks the angle exponentially during pretraining. On every other input both units receive the same random linear update, which contracts their separation in conditional expectation. Bounding the cumulative probability of sampling the wedge along the exact online recursion, without a diffusion approximation, shows that recovery with fixed probability from an identical source-gradient-flow checkpoint, within

e^{c/η}

updates, requires

Nb \gtrsim e^{λT}

target samples and batch size

b \gtrsim ηe^{λT}

, where

N

counts updates and

λ

is the weight decay. In simulations, recovery is approximately a function of the disagreement budget

bδ/η

and saturates in the horizon.

Momentum

Related topics

Latest papers 63