Activation Function

Activation functions are crucial components of neural networks, introducing nonlinearity to enable the learning of complex patterns from data. Current research focuses on developing novel activation functions, including those with learnable parameters, and exploring their impact within various architectures like Kolmogorov-Arnold Networks and transformers. These efforts aim to improve model performance, efficiency, and interpretability across diverse applications, from image classification and generation to solving partial differential equations and formal verification tasks. The ongoing search for optimal activation functions is driving significant advancements in the field of deep learning.

Papers

June 8, 2023

LayerAct: Advanced activation mechanism utilizing layer-direction normalization for CNNs with BatchNorm
Kihyuk Yoon, Chiehyeon Lim
CNN Model Activation Function Layer Normalization Batchnorm Minus Implementation Advanced Activation Mechanism

June 2, 2023

ErfReLU: Adaptive Activation Function for Deep Neural Network
Ashish Rajanand, Pradeep Singh
Deep Neural Network Activation Function Adaptive Activation Function

May 28, 2023

ASU-CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations
Jamshaid Ul Rahman, Faiza Makhdoom, Dianchen Lu
Deep Neural Network Image Classification Activation Function CNN Network Convolutional Neural Network Model Efficient Deep Different Convolutional Neural Network Feature Visualization

May 25, 2023

Embeddings between Barron spaces with higher order activation functions
Tjeerd Jan Heeringa, Len Spek, Felix Schwenninger, Christoph Brune
Jina Embeddings Activation Function Shallow Neural Network Barron Space

May 20, 2023

GELU Activation Function in Deep Learning: A Comprehensive Mathematical Analysis and Performance
Minhyeok Lee
Deep Learning System Performance Activation Function

May 19, 2023

Complexity of Neural Network Training and ETR: Extensions with Effectively Continuous Functions
Teemu Hankala, Miika Hannula, Juha Kontinen, Jonni Virtema
Neural Network Complexity Matter Activation Function Extension Study Continuous Space Sinusoidal Activation

May 18, 2023

Learning Activation Functions for Sparse Neural Networks
Mohammad Loni, Aditya Mohan, Mehdi Asadi, Marius Lindauer
Activation Function DNN Model Sparse Network Sparse Neural Network Sparse Activation

May 16, 2023

Hardware Realization of Nonlinear Activation Functions for NN-based Optical Equalizers
Sasipim Srivallapanondh, Pedro J. Freire, Antonio Napoli, Sergei K. Turitsyn, Jaroslaw E. Prilepsky
System Performance Activation Function Hardware Implementation BiLSTM CNN CRF

May 15, 2023

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks
Chaoyue Liu, Like Hui
Optimization Purpose Activation Function Neural Tangent Kernel Wide Neural Network Linear Neural Network Rectified Linear Unit

May 12, 2023

Saturated Non-Monotonic Activation Functions
Junjia Chen, Zhibin Pan
Activation Function Non Monotonic Activation Function

May 8, 2023

TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks
Md. Mehedi Hasan, Md. Ali Hossain, Azmain Yakin Srizon, Abu Sayeed
Neural Network Deep Learning Model Activation Function Rectified Linear Unit

May 5, 2023

A technical note on bilinear layers for interpretability
Lee Sharkey
Neural Network Inherent Interpretability Activation Function Technical Report MLP Layer Bilinear Convolutional Neural Network

April 23, 2023

Improving Classification Neural Networks by using Absolute activation function (MNIST/LeNET-5 example)
Oleg I. Berngardt
Neural Network Activation Function Adam Optimizer MNIST Canadian Institute for Advanced Adam Algorithm Neural Network Activation

April 18, 2023

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently
Jamshaid Ul Rahman, Faiza Makhdoom, Dianchen Lu
Deep Neural Network Activation Function

April 10, 2023

April 6, 2023

Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initialization
Garrett Bingham
Neural Network Machine Learning Activation Function Weight Initialization Neural Network Design

March 28, 2023

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control
Tyler Lekang, Andrew Lamperski
Neural Network Activation Function Function Approximation Random Participation Neural Network Approximation Model Reference Adaptive Integral Regression

March 12, 2023

From Compass and Ruler to Convolution and Nonlinearity: On the Surprising Difficulty of Understanding a Simple CNN Solving a Simple Geometric Estimation Task
Thomas Dagès, Michael Lindenbaum, Alfred M. Bruckstein
Neural Network Activation Function Direct Convolution Nonlinear Model Difficulty Level Geometric Estimation Simple Convolutional

March 5, 2023

Swim: A General-Purpose, High-Performing, and Efficient Activation Function for Locomotion Control Tasks
Maryam Abdool, Tony Dear
Activation Function High Performance Locomotion Control Swish Activation

Activation Function

Papers

LayerAct: Advanced activation mechanism utilizing layer-direction normalization for CNNs with BatchNorm

ErfReLU: Adaptive Activation Function for Deep Neural Network

ASU-CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations

Embeddings between Barron spaces with higher order activation functions

GELU Activation Function in Deep Learning: A Comprehensive Mathematical Analysis and Performance

Complexity of Neural Network Training and ETR: Extensions with Effectively Continuous Functions

Learning Activation Functions for Sparse Neural Networks

Hardware Realization of Nonlinear Activation Functions for NN-based Optical Equalizers

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

Saturated Non-Monotonic Activation Functions

TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks

A technical note on bilinear layers for interpretability

Improving Classification Neural Networks by using Absolute activation function (MNIST/LeNET-5 example)

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently

Criticality versus uniformity in deep neural networks

Bayesian optimization for sparse neural networks with trainable activation functions

Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initialization

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

From Compass and Ruler to Convolution and Nonlinearity: On the Surprising Difficulty of Understanding a Simple CNN Solving a Simple Geometric Estimation Task

Swim: A General-Purpose, High-Performing, and Efficient Activation Function for Locomotion Control Tasks