Rectified Linear Unit

Rectified Linear Units (ReLUs) are a fundamental activation function in artificial neural networks, crucial for enabling efficient training and powerful representation capabilities. Current research focuses on understanding and improving ReLU's properties, including addressing its limitations in adversarial robustness and exploring its role in various architectures like deep survival models (SurvReLU) and efficient private inference networks (xMLP). This work is significant because it enhances both the theoretical understanding of neural network behavior and the practical performance of applications ranging from medical prognosis to large-scale recommendation systems, improving accuracy, efficiency, and interpretability.

Papers

June 27, 2022

On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector
Arnulf Jentzen, Timo Kröger
Rectified Linear Unit Lipschitz Bound Neural Network Parameter Parameterized Network

June 5, 2022

Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions
Peng He
LeArning Abstract High Dimension Gradient Norm Global Convergence Rectified Linear Unit Random Neural Network Global Convergence Rate

May 24, 2022

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou, Qixuan Zhou, Zhenyuan Jin, Tao Luo, Yaoyu Zhang, Zhi-Qin John Xu
Gradient Flow Deeper Network Two Layer ReLU Rectified Linear Unit Phase Diagram Three Layer Neural Network Infinite Width Dynamical Regime

May 13, 2022

Convergence of Deep Neural Networks with General Activation Functions and Pooling
Wentao Huang, Yuesheng Xu, Haizhang Zhang
Deep Learning Deep Neural Network Deep Convolutional Neural Network Early Stage Convergence Activation Function Rectified Linear Unit Leaky ReLU

February 14, 2022

Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations
Gil I. Shamir, Dong Lin
Real World Sequential Recommendation Research Reproducibility Rectified Linear Unit

February 13, 2022

Graph-adaptive Rectified Linear Unit for Graph Neural Networks
Yifei Zhang, Hao Zhu, Ziqiao Meng, Piotr Koniusz, Irwin King
Graph Neural Network Message Passing Rectified Linear Unit GNN Backbone

November 25, 2021

Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks
Tilahun M. Getu
Deep Learning Machine Learning Error Bound Machine Learning Paradigm Rectified Linear Unit Deep Neural Network Approximation

November 17, 2021

Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verification
Shaojie Xu, Joel Vaughan, Jie Chen, Aijun Zhang, Agus Sudjianto
Neural Network Native Robustness Unified Framework ReLU Activation Rectified Linear Unit ReLU Neural Network Forward Invariant Polytopes

Rectified Linear Unit

Papers

On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector

Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width

Convergence of Deep Neural Networks with General Activation Functions and Pooling

Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations

Graph-adaptive Rectified Linear Unit for Graph Neural Networks

Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks

Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verification