Lipschitz Activation

Lipschitz activation functions, characterized by a bounded slope, are crucial in neural network research due to their role in ensuring stability, generalization, and theoretical analysis. Current research focuses on understanding their impact on network approximation capabilities, particularly within deep architectures and in relation to specific activation functions like ReLU and its variants. This work is significant because Lipschitz continuity facilitates the development of rigorous theoretical guarantees for neural network behavior, including generalization bounds and improved robustness, ultimately leading to more reliable and predictable models for various applications.

Papers

October 11, 2023

Neural networks: deep, shallow, or in between?
Guergana Petrova, Przemyslaw Wojtaszczyk
Neural Network Shannon Entropy Banach Space Lipschitz Activation

June 28, 2023

Gaussian random field approximation via Stein's method with applications to wide random neural networks
Krishnakumar Balasubramanian, Larry Goldstein, Nathan Ross, Adil Salim
Financial Application RSD Difference of Gaussian Wasserstein Distance Reproducing Kernel Hilbert Space Random Neural Network Random Field Lipschitz Activation

June 18, 2023

Agnostically Learning Single-Index Models using Omnipredictors
Aravind Gollakota, Parikshit Gopalan, Adam R. Klivans, Konstantinos Stavropoulos
Logistic Regression Distributional Assumption Bregman Divergence Agnostic Learning Single Index Model Lipschitz Activation

June 2, 2023

Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths
Yuesheng Xu, Haizhang Zhang
Deep Neural Network Convergence Analysis Uniform Convergence Weak Convergence Lipschitz Activation Lipschitz Continuity Assumption

December 1, 2022

On the Compatibility between Neural Networks and Partial Differential Equations for Physics-informed Learning
Kuangdai Leng, Jeyan Thiyagalingam
Neural Network Physic Informed Neural Network Multi Layer Partial Differential Equation Physic Informed Learning Rectified Linear Unit Self Compatibility Lipschitz Activation

November 17, 2022

On the Sample Complexity of Two-Layer Networks: Lipschitz vs. Element-Wise Lipschitz Activation
Amit Daniely, Elad Granot
Sample Complexity Activation Function Two Layer Neural Network Lipschitz Operator Norm Bounded Two Layer Network Lipschitz Activation

August 8, 2022

On Rademacher Complexity-based Generalization Bounds for Deep Learning
Lan V. Truong
Deep Learning Convolutional Neural Network Generalization Error Bound Rademacher Complexity Non Vacuous Generalization Bound Lipschitz Activation

July 13, 2022

Lipschitz Continuity Retained Binary Neural Network
Yuzhang Shang, Dan Xu, Bin Duan, Ziliang Zong, Liqiang Nie, Yan Yan
Neural Network Model Robustness Binary Neural Network Lipschitz Regularization Lipschitz Activation

November 15, 2021

Reachability analysis of neural networks using mixed monotonicity
Pierre-Jean Meyer
Neural Network Reachability Analysis Partial Monotonicity Tight Approximation Monotone Neural Network Interval Bound Propagation Lipschitz Activation