Neural Network Activation Functions

Ask about this topic

Overview

Activation functions transform a network’s intermediate signals, determining which patterns it can represent and how gradients behave during training. Work compares standard choices such as ReLU, GELU, and SiLU with smooth, polynomial, rational, periodic, and wavelet-based alternatives, as well as learnable or input-conditioned mixtures and gates. Theory studies approximation rates and optimization dynamics, while experiments test classification, language modeling, scientific tasks, and robustness; hardware research seeks cheaper implementations through piecewise approximations and lookup methods. A central tension is that gains depend on the task, model scale, and training regime: greater flexibility or smoothness may improve representation or stability, but can add computation or alter generalization, and the best activation is not universal.

Latest papers

CardsList

78 papers10 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

  1. Edge-Centric Adaptive Neural Activation via Parametric Tanh Functions

    Oct 25, 2026Andreas FalkenbergNeural Network Activation FunctionsKolmogorov-Arnold Networks

    QPI Laboratories, Inc., USA

  2. A Fix for Softmax and Cross-Entropy

    Oct 22, 2026Jonathan T. Barron, Dan Gnanapragasam, Michael TschannenNeural Network Activation Functions

    Google DeepMind

  3. Stochastic Nonlinearities Improve Uncertainty Estimation

    Oct 21, 2026Mohammad Sarhangzadeh, Amirreza M. Shebly, Doruk OnerUncertainty QuantificationNeural Network Activation Functions

    NeuraVision Lab, Bilkent University Ankara, Turkey

  4. Compiling Multi-Threshold Operators for Quantized Non-Monotonic and Fine-Granular Activations in Custom-Tailored DNN Accelerators

    Oct 15, 2026Christoph Berganski, Felix Jentzsch, Marco PlatznerActivation QuantizationNeural Network Activation Functions

    Computer Engineering Group, Paderborn University, Paderborn, Germany

  5. Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

    Oct 14, 2026Xiuyi YangNeural Network OptimizationNeural Network Training Dynamics

  6. Design and Analysis of Temporal Representations in Neural Networks

    Oct 14, 2026Alban Colas-Gattepaille, Alexandre MuzyNeural EncodingSpiking Neural Networks

    I3S · ILLS

  7. Rethinking Activation Functions: Reducing Bias Can Improve Learning

    Oct 10, 2026Zirui Peng, Heyi Zheng, Ziming LiuLoss LandscapeNeural Network Activation Functions

    Department of Physics, Peking University · Department of Mathematics, Peking University · College of AI, Tsinghua University +1

  8. HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification

    Oct 8, 2026Michael W. Spratling, Heiko H. SchüttNeural Network Activation FunctionsNeural Network Generalization

    University of Luxembourg, Department of Behavioural and Cognitive Sciences, L-4366 Esch-sur-Alzette, Luxembourg.

  9. Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

    Oct 1, 2026Tilen Cadez, Sanghoon Lee, Kyoung-Min KimNeural Network Activation FunctionsKolmogorov-Arnold Networks

    Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea

  10. Let the Neurons Die: Exploiting ReLU-Induced Model Degradation

    Sep 28, 2026Kexin Li, Wenjun Qiu, Joshua Abraham +2ReLU Neural NetworksAdversarial Attacks

    University of Toronto, Toronto, Ontario, Canada

  11. Activation-Flexible ANN-to-SNN Conversion with Finite-State Markov Neurons

    Sep 24, 2026Ruiyu Jia, Zhuo-Cheng XiaoNeural Network Activation FunctionsSpiking Neural Networks

    New York University Shanghai, Shanghai, China

  12. An improved periodic activation for PINNs reconstructing convective flows

    Sep 18, 2026Michael Mommert, Marie-Christine Volk, Christian BauerNeural Network Activation FunctionsPhysics-Informed ML

    German Aerospace Center (DLR), Institute of Aerodynamics and Flow Technology, Department Ground Vehicles, Bunsenstr. 10, 37073 Göttingen, Germany · Technische Universität Ilmenau, Institute of Thermodynamics and Fluid Mechanics, 98693 Ilmenau, Germany

  13. Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery

    Sep 15, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary OptimizationNeural Network Activation Functions

    University of Neuchâtel, Neuchâtel, Switzerland · University College Dublin, Ireland

  14. Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity

    Sep 10, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Neural Network Activation FunctionsNeuroevolution

    University of Neuchâtel, Switzerland · University College Dublin, Ireland

  15. Multi-Behavioral Evolved Substrates Through Neuromodulation and Activation Selection

    Sep 10, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1NeuroevolutionArtificial Life

    University of Neuchˆatel, Switzerland · University College Dublin, Ireland

  16. Diversity of EML-type operators

    Sep 10, 2026Andrzej OdrzywołekNeural Network Activation FunctionsNeural Network Expressivity

    Institute of Theoretical Physics, Jagiellonian University, Łojasiewicza 11, 30-348 Kraków, Poland

  17. Shallow neural network approximation in mixed Sobolev spaces

    Sep 4, 2026Yuwen Li, Guozhi ZhangNeural Network Approximation TheoryFunction Approximation

    School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China

  18. InKAN: B-Spline KANs via Truncated Power Form

    Sep 2, 2026Naveen MysoreEfficient Neural Network InferenceNeural Network Activation Functions

  19. Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning

    Aug 13, 2026Zeyang Zhang, Tieliang Gong, Junyan Lu +1Loss of PlasticityContinual Learning

    School of Computer Science and Technology, Xi’an Jiaotong University and Institute of Multimedia Knowledge Fusion and Engineering, Xi’an 710049, China

  20. TESLA: Taylor Expansion of Sinusoidal Learnable Activations

    Aug 12, 2026Daehwa Ko, Jaehyeon Kim, Seunghyun Ham +1Neural Network GeneralizationNeural Network Activation Functions

    Korea Aerospace University, Goyang, Republic of Korea

  21. QFCQT: A Chaotically Gated Quantformer Framework for Volatile Time-Series Forecasting

    Aug 7, 2026Junkai Lin, Siqi Hou, Raymond LeeMultivariate Time Series ForecastingTime Series Forecasting

    Faculty of Science and Technology, Beijing Normal-Hong Kong Baptist University, Zhuhai, China · Guangdong Provincial Key Lab of Interdisciplinary Research and Application for Data Science (BNBU), Zhuhai, China

  22. Local Stability and Gaussian Smoothing of Quantized Neural Networks

    Jul 22, 2026Sergey Salishev, Anton Makarov, Oleg GranichinNeural Network StabilityNeural Network Activation Functions

    St. Petersburg State University, 199034, Saint Petersburg, Russia

  23. Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions

    Jul 20, 2026Meng Hua, Itsik Bergel, Deniz GündüzNeural Network Activation FunctionsWireless Communications

    Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, U.K. · Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel

  24. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    Jul 20, 2026Damien Teney, Liangze Jiang, Hemanth Saratchandran +1Neural Network Activation FunctionsInductive Bias

    Idiap Research Institute · Adelaide University · EPFL

  25. LRX-PINN: A Layer-Resolving XNet Physics-Informed Neural Network with Integrated Cauchy Activations for Convection-Dominated Problems

    Jul 4, 2026Zihao Guo, Xin Li, Zhihong XiaNeural PDE SolversPDE Solving

    Institute for Advanced Research, Great Bay University, Dongguan, China · College of Computer Science and Technology, Dongguan University of Technology, Dongguan, China · Department of Mathematics, Northwestern University, Evanston, USA

  26. A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

    Jul 4, 2026Roberto RossiNeural Network Activation Functions

    Business School, University of Edinburgh, Edinburgh, UK

  27. Rethinking Neural Nonlinearity as Gating

    Jul 3, 2026Muhammad Sabih, Frank Hannig, Jürgen TeichNeural Network Activation FunctionsNeural Network Approximation Theory

    Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Germany

  28. SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions

    Jun 30, 2026Hoang-Thang TaNeural Surrogate ModelingNeural Network Activation Functions

    University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam

  29. It's Much Easier for Neural Networks to learn Game of Life Dynamics with the Right Activation Function: Polynomial Kolmogorov-Arnold Networks

    Jun 22, 2026Tashin Ahmed, Q. Tyrell DavisPolynomial Neural NetworksCellular Automata

    Independent Researcher · Work completed while at Cross Labs, Cross Compass Ltd., Japan.

  30. Spectral Gating via Damped Oscillations for Adaptive Implicit Neural Representations

    Jun 22, 2026Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti +1Implicit Neural RepresentationsFrequency-Domain Feature Learning

    CVLab, University of Bologna · Ca’ Foscari University of Venice