Rectified Linear Unit

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 33

All topics

From Redundancy to Minimality: Fixed-Point-Guided Hierarchical Reduction of Learned Piecewise-Linear Dynamics
Oct 1, 2026Hiroto Tamura, Gouhei TanakaNonlinear Dynamics Nonlinearity
Graduate School of Informatics, Osaka Metropolitan University, Osaka, Japan · International Research Center for Neurointelligence (IRCN), The University of Tokyo, Tokyo, Japan · Graduate School of Engineering, Nagoya Institute of Technology, Nagoya, Japan
Flow Matching under Noisy Latent Structure: Beyond Exact Low-Dimensional Support
Sep 30, 2026Lifeng Hao, Shaolin JiLatent Variable Rectified Linear Unit
Zhongtai Securities Institute for Financial Studies, Shandong University, Jinan 250100, Shandong, China.
Conformalized Quantile Regression and Minimax Limits of Fixed-Score Calibration under Known Covariate Shift
Sep 21, 2026Rustam Isaev, Anton Conrad, Denis Belomestny +2Quantile Regression Covariate Shift
Faculty of Computer Science, HSE University, Moscow, Russia · Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Moscow, Russia · Laboratoire de Recherche d’EPITA, Le Kremlin-Bicêtre, France +2
Approximating Smooth Functionals with ReLU Networks
Sep 14, 2026Shuhao JiaoNeural Approximations Rectified Linear Unit Networks
Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.
Exact ReLU realization of binary affine refinement iterates via reflection folding and cone switching
Sep 14, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurIterative Refinement Affine
National University of Mongolia · McGill University · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences
Hard-ReLU Gradient Descent Selects an Event-Free Sensitivity Limit
Aug 31, 2026Xiaoyang Li, Runni ZhouGradient Local Curvature
College of Medicine and Biological Information Engineering Northeastern University, Shenyang, China
Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks
Aug 24, 2026Itay SafranRectified Linear Unit Networks Rectified Linear Unit
Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev
Training Under Challenge: Executable Certificates and Challenge-Closed Optimality for Neural Networks
Aug 12, 2026Farhang Yeganegi, Arian Eamaz, Mojtaba SoltanalianNeural Network Verification Finite-Sample Certificates
Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, Illinois, USA
The Boolean Power of ReLU
Aug 12, 2026Pablo Barceló, Floris Geerts, Matthias Lanzinger +2Rectified Linear Unit Activation Functions
Pontifical Catholic University · IMFD & CENIA, Chile · University of Antwerp +4
A Counterexample to Fourier Alignment in Single-Neuron Modular Addition
Aug 5, 2026Gautam Neelakantan MemanaRectified Linear Unit Counterexample
Tight Worst-Case Bounds for the Smallest Eigenvalue of ReLU NTK Gram Matrices
Aug 4, 2026Zhao SongRectified Linear Unit Spectral Decomposition
Provably Learning Multi-Head Attention with Queries
Aug 4, 2026Sunyeop Kim, Insung Kim, Jian GuoMulti-Head Attention Rectified Linear Unit
Korea University, Republic of Korea · Nanyang Technological University, Singapore
Exact ReLU realization of affine one-dimensional refinement iterates via residual memory and offset frames
Jul 22, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurAffine Rectified Linear Unit
2National University of Mongolia · 1McGill University · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences
Random Parameter Noise Does Not Make Exact ReLU Verification Easy
Jul 15, 2026Mojtaba SoltanalianNeural Network Verification Rectified Linear Unit Networks
Department of Electrical and Computer Engineering University of Illinois Chicago
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
Jul 8, 2026Marcel Kühn, Bernd RosenowHessian Anisotropic Loss Landscapes
Institut für Theoretische Physik, Universität Leipzig, 04103 Leipzig, Germany · ScaDS.AI Dresden/Leipzig, Universität Leipzig, 04105 Leipzig, Germany
Compressed Computation is (probably) not Computation in Superposition
Jun 12, 2026Jai Bhagat, Sara Molas-Medina, Giorgi Giglemiani +1Compressed Model Multilayer Perceptrons
Metamorphic · Independent · UK AI Security Institute +1
Robust Regression of General ReLUs with Queries
Jun 9, 2026Ilias Diakonikolas, Daniel M. Kane, Mingchen MaLearnability Rectified Linear Unit
University of Wisconsin-Madison · University of California, San Diego
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangDeep Network Rectified Linear Unit
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference
May 21, 2026Rui Li, Wenyuan Wu, Weijie MiaoHomomorphic Encryption Rectified Linear Unit
Chongqing Key Laboratory of Secure Computing for Biology, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, 266 Fangzheng Avenue, Beibei District, Chongqing 400714, China · Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China
A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization
May 18, 2026Lei DongBatch Normalization Rectified Linear Unit
Independent Researcher
Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement
May 14, 2026Hengjie Liu, Zhenya Zhang, Jianjun ZhaoTransformer Architectures Rectified Linear Unit
Kyushu University, Fukuoka, Japan · National Institute of Informatics, Tokyo, Japan
Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning
May 6, 2026Anastasis Kratsios, A. Martina Neuman, Philipp PetersenLearnability In-Context Learning
McMaster University and the Vector Institute, Main St., Hamilton, L8S 4L8, Ontario, Canada · University of Vienna, Faculty of Mathematics, Kolingasse 14-16, 1090 Wien, Austria
Exact ReLU realization of tensor-product refinement iterates
May 5, 2026Tsogtgerel GantumurIterative Refinement Rectified Linear Unit
McGill University, Montréal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology
Exact Loop Controllers for ReLU Realization of Homogeneous Curve Refinements
May 3, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurRectified Linear Unit Iterative Refinement
1McGill University · 2National University of Mongolia · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences
Transformer Approximations from ReLUs
Apr 27, 2026Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1Gumbel-Softmax Relaxation Rectified Linear Unit
Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1
Complexity of Linear Regions in Self-supervised Deep ReLU Networks
Apr 27, 2026Mufhumudzi Muthivhi, Terence L. van ZylSelf-Supervised Learning Rectified Linear Unit
University of Johannesburg, Johannesburg, South Africa
Primitive Recursion without Composition: Dynamical Characterizations, from Neural Networks to Polynomial ODEs
Apr 27, 2026Olivier BournezRecursive Models Rectified Linear Unit
Institut Polytechnique de Paris, Ecole Polytechnique, LIX, Palaiseau, France
Can an MLP Absorb Its Own Skip Connection?
Apr 26, 2026Antonij Mijoski, Marko KarbevskiResidual Connections Residual Networks
IRMA, Universit´e de Strasbourg, France · Independent Researcher
Explicit integral representations and quantitative bounds for two-layer ReLU networks
Apr 25, 2026Anthony LeeRectified Linear Unit Networks Rectified Linear Unit
School of Mathematics, University of Bristol
Lipschitz-Based Robustness Certification Under Floating-Point Execution
Mar 6, 2026Toby MurrayFloating-Point Lipschitz Continuity
University of Melbourne, Australia
Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU
Oct 29, 2025Jingzhou LiuRectified Linear Unit Neurons
Near-optimal estimates for the $\ell^p$ -Lipschitz constants of deep random ReLU neural networks
Jun 24, 2025Sjoerd Dirksen, Patrick Finke, Paul Geuchen +2Rectified Linear Unit Networks Lipschitz Continuity
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions
Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoActivation Functions Rectified Linear Unit Networks
Department of Mathematics Duke University

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

From Redundancy to Minimality: Fixed-Point-Guided Hierarchical Reduction of Learned Piecewise-Linear Dynamics

Hiroto Tamura, Gouhei Tanaka

Graduate School of Informatics, Osaka Metropolitan University, Osaka, Japan · International Research Center for Neurointelligence (IRCN), The University of Tokyo, Tokyo, Japan · Graduate School of Engineering, Nagoya Institute of Technology, Nagoya, Japan

Nonlinear Dynamics Nonlinearity Rectified Linear Unit Recurrent Neural Networks Inference Dynamics

Open paper AskPDF arXiv

Understanding a nonlinear dynamical system from time series requires not only reproducing its trajectories, but also identifying a simple representation that preserves its essential dynamical structure. Almost-linear recurrent neural networks (AL-RNNs) are piecewise-linear RNNs in which only a subset of units use ReLU nonlinearities, so that nonlinear capacity is explicitly controlled by the number of ReLU units. Their activation patterns define linear regions, represented as symbols, whose observed transitions form a symbolic transition graph. However, directly training AL-RNNs with few ReLU units to realize minimal dynamical representations can be unreliable. We ask whether an AL-RNN with more ReLU units can instead be trained first and systematically reduced to a minimal dynamical representation. We introduce a fixed-point-guided hierarchical reduction procedure that progressively linearizes selected ReLU units, merging neighboring linear regions and graph nodes while preserving distinct symbols containing fixed points (FPs). The resulting reduction tree defines a hierarchy of progressively simpler candidates. Each reduced candidate is initialized from the parent parameters and retrained under guidance from the parent dynamics. We also prove that reproducing

Q

distinct fixed points requires at least

Q

FP-containing symbols, providing a certificate of symbol-level minimality when this bound is attained. On the 3-scroll Chua system, direct training with the theoretical minimum of three ReLU units achieves high-fidelity minimal realizations in only 20% of seeds, whereas our learn-reduce-retrain strategy increases the seed-macro success rate to approximately 71% at the same final nonlinear capacity. These results show that redundant nonlinear capacity can serve as a scaffold for discovering and realizing minimal dynamical representations.

Momentum

Related topics

Latest papers 33