Smooth Activation Function

Smooth activation functions in neural networks are being extensively studied to improve training efficiency, generalization performance, and robustness. Current research focuses on analyzing the optimization landscape of deep networks employing these functions, particularly within weight normalization frameworks and implicit gradient descent methods, and exploring their impact on convergence rates and generalization bounds across various architectures, including convolutional recurrent networks and physics-informed neural networks. These investigations are crucial for advancing theoretical understanding of deep learning and developing more reliable and efficient algorithms for diverse applications, such as image classification, control systems, and medical image registration.

Papers

September 22, 2024

Linear Independence of Generalized Neurons and Related Functions
Leyang Zhang
Neural Network Simple Function Two Layer Neural Network Linear Dependence Smooth Activation Function

September 13, 2024

Optimization and Generalization Guarantees for Weight Normalization
Pedro Cisneros-Velarde, Zhijie Chen, Sanmi Koyejo, Arindam Banerjee
Optimization Purpose Generalization Guarantee Training Convergence Smooth Activation Function Weight Normalization

July 3, 2024

Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks
Xianliang Xu, Ting Du, Wang Kong, Ye Li, Zhongyi Huang
Neural Network Physic Informed Neural Network Early Stage Convergence Two Layer Neural Network Gradient Sharing Smooth Activation Function

May 22, 2024

On the dynamics of convolutional recurrent neural networks near their critical point
Aditi Chandra, Marcelo O. Magnasco
Real World Convolution Kernel Convolutional Recurrent Neural Network Smooth Activation Function Critical Point Compressive Memory

October 16, 2023

A Non-monotonic Smooth Activation Function
Koushik Biswas, Meghana Karri, Ulaş Bağcı
Adversarial Attack Deep Learning Model Activation Function Smooth Activation Function Output Activation

May 30, 2023

Benign Overfitting in Deep Neural Networks under Lazy Training
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Francesco Locatello, Volkan Cevher
Neural Network Deep Neural Network Neural Tangent Kernel Generalization Error Benign Overfitting Smooth Activation Function Lazy Training

May 26, 2023

An Analytic End-to-End Deep Learning Algorithm based on Collaborative Learning
Sitan Li, Chien Chern Cheah
Deep Learning Convergence Analysis Collaborative Learning Smooth Activation Function

March 13, 2023

NeurEPDiff: Neural Operators to Predict Geodesics in Deformation Spaces
Nian Wu, Miaomiao Zhang
Neural Operator Generalized Geodesic Diffeomorphic Image Registration Smooth Activation Function Orientation Preserving Diffeomorphisms

January 21, 2023

Limitations of Piecewise Linearity for Efficient Robustness Certification
Klas Leino
Fundamental Limitation Robust Generalization Piecewise Linear Free Counterpart Robustness Certification Smooth Activation Function

September 29, 2022

Restricted Strong Convexity of Deep Learning Models with Smooth Activations
Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, Mikhail Belkin
Deep Learning Model Gradient Descent Neural Tangent Kernel Strong Convexity Smooth Activation Function

May 31, 2022

Provably and Practically Efficient Neural Contextual Bandits
Sudeep Salgia, Sattar Vakili, Qing Zhao
Sublinear Regret Kernel Bandit Smooth Activation Function Neural Contextual Bandit

November 8, 2021

SMU: smooth activation function for deep networks using smoothing maximum technique
Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, Ashish Kumar Pandey
Deep Network Activation Function Smooth Optimization Smooth Activation Function