Neural Network Activation Functions

Ask about this topic

Overview

Activation functions transform a network’s intermediate signals, determining which patterns it can represent and how gradients behave during training. Work compares standard choices such as ReLU, GELU, and SiLU with smooth, polynomial, rational, periodic, and wavelet-based alternatives, as well as learnable or input-conditioned mixtures and gates. Theory studies approximation rates and optimization dynamics, while experiments test classification, language modeling, scientific tasks, and robustness; hardware research seeks cheaper implementations through piecewise approximations and lookup methods. A central tension is that gains depend on the task, model scale, and training regime: greater flexibility or smoothness may improve representation or stability, but can add computation or alter generalization, and the best activation is not universal.

Latest papers

CardsList

78 papers10 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

  1. DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training

    Mar 19, 2026Maoyang Xiang, Bo WangNeural Network Activation FunctionsEfficient Transformer Inference

    Information Systems Technology and Design Singapore University of Technology and Design Singapore

  2. A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networks

    Mar 18, 2026Krishna MurariNeural Network Activation FunctionsPhysics-Informed ML

    †Department of Mathematics, IIT Madras, Chennai 600036, India

  3. PAKAN: Pixel Adaptive Kolmogorov-Arnold Network Modules for Pansharpening

    Mar 16, 2026Haoyu Zhang, Haojing Chen, Zhen Zhong +1Neural Network Activation FunctionsPansharpening

    University of Electronic Science and Technology of China

  4. PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks

    Mar 7, 2026Daniel Nobrega MedeirosTransformer FFNsNeural Network Activation Functions

    Independent researcher.

  5. IGLU: The Integrated Gaussian Linear Unit Activation Function

    Mar 6, 2026Mingi Kang, Zai Yang, Jeova Farias Sales Rocha NetoNeural Network Activation Functions

    Bowdoin College, 255 Maine Street, Brunswick, ME 04011, USA

  6. Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies

    Feb 26, 2026George BirdDynamic Neural NetworksNeural Network Activation Functions

    Department of Computer Science, University of Manchester, UK

  7. GRAU: A Generic Runtime-Reconfigurable Activation-Requantization Unit for Quantized Neural Network Accelerators

    Feb 25, 2026Yuhao Liu, Salim Ullah, Akash KumarNeural Network Hardware AccelerationNeural Network Activation Functions

    Ruhr University Bochum, Germany · Dresden University of Technology, Germany · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI Dresden/Leipzig), Germany

  8. Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

    Feb 23, 2026Yuhao Liu, Zilin Wang, Lei Wu +1Neural Network Approximation TheoryNeural Network Activation Functions

    Department of Mathematical Sciences, Tsinghua University · School of Mathematical Sciences, Peking University · Center for Machine Learning Research, Peking University +1

  9. Beyond ReLU: Bifurcation, Oversmoothing, and Topological Priors

    Feb 17, 2026Erkan Turan, Gaspard Abel, Maysam Behmanesh +2GNN OversmoothingGraph Neural Networks

    LIX, Ecole Polytechnique, IP Paris · Université Paris Saclay, Université Paris Cité, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli, F-91190, Gif-sur-Yvette, France · Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS, 75006 Paris, France..

  10. Rational Neural Networks have Expressivity Advantages

    Feb 12, 2026Maosen Tang, Alex TownsendNeural Network Activation FunctionsNeural Network Approximation Theory

    Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States

  11. FlexAct: Why Learn when you can Pick?

    Jan 10, 2026Ramnath Kumar, Kyle Ritscher, Junmin Judy +2Neural Network Activation Functions

    University of California Los Angeles. · University of California Los Angeles, Arena.

  12. Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

    Oct 29, 2025Jingzhou LiuShallow Neural NetworksDynamical Systems

  13. SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks

    Oct 25, 2025Amin OmidvarNeural Network OptimizationNeural Network Activation Functions

    Independent Researcher Toronto, Ontario, Canada

  14. Circuit realization and hardware linearization of monotone operator equilibrium networks

    Sep 17, 2025Thomas ChaffeyReLU Neural NetworksNeuromorphic Computing

    School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia

  15. Understanding Two-Layer Neural Networks with Smooth Activation Functions

    Jul 11, 2025Changcun HuangNeural Network Activation FunctionsNeural Network Approximation Theory

    Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

  16. SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

    May 29, 2025Gaurav Sarkar, Syed Affan Daimi, Jay Gala +1Transformer FFNsTransformer

    IIIT · Microsoft · Intel

  17. Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

    Feb 26, 2025Shijun Zhang, Hongkai Zhao, Yimin Zhong +1Fourier Feature EmbeddingsNeural Network Optimization

    Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1

  18. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

    Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural NetworksNeural Network Activation Functions

    Department of Mathematics Duke University