Neural Network Activation Functions

Overview

Activation functions transform a network’s intermediate signals, determining which patterns it can represent and how gradients behave during training. Work compares standard choices such as ReLU, GELU, and SiLU with smooth, polynomial, rational, periodic, and wavelet-based alternatives, as well as learnable or input-conditioned mixtures and gates. Theory studies approximation rates and optimization dynamics, while experiments test classification, language modeling, scientific tasks, and robustness; hardware research seeks cheaper implementations through piecewise approximations and lookup methods. A central tension is that gains depend on the task, model scale, and training regime: greater flexibility or smoothness may improve representation or stability, but can add computation or alter generalization, and the best activation is not universal.

Latest papers

CardsList

85 papers8 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. BeFit: Shifting Complexity from Weights to Bernstein Activations for Resource-Constrained Sensing

    Nov 17, 2026Malak Gamal El-Din, Yasser Shoukry, Salma ElmalakiLightweight Neural NetworksNeural Network Hardware Acceleration

  2. Beyond a Single Sign: BiTE for Branch-Preserving Binary Threshold Expansion

    Nov 12, 2026Siddharth Gupta, Mikail Yayla, Akash KumarBinary Neural NetworksBinary Quantization

  3. To SIMD or Not to SIMD: Comparing RVV vs. CGRA SoCs for Non-Linear Functions in AI

    Nov 9, 2026Sameer Deshmukh, Naoya Mizuki, Takumi HondaNeural Network Hardware AccelerationNeural Network Activation Functions

  4. From Fixed Defaults to Calibrated Activation Policies for Long-Term Time-Series Forecasting

    Nov 4, 2026Shijie Xu, Boyan Tang, Kai He +1Time Series ForecastingLong-Term Time Series Forecasting

  5. Language Models Can Benefit from Domain-Specific Inductive Biases

    Nov 2, 2026Damien Teney, Liangze Jiang, Zachary Shinnick +2Language Model PretrainingEfficient Language Model Training

  6. A Fix for Softmax and Cross-Entropy

    Oct 22, 2026Jonathan T. Barron, Dan Gnanapragasam, Michael TschannenNeural Network Activation Functions

  7. Stochastic Nonlinearities Improve Uncertainty Estimation

    Oct 21, 2026Mohammad Sarhangzadeh, Amirreza M. Shebly, Doruk OnerUncertainty QuantificationNeural Network Activation Functions

  8. Compiling Multi-Threshold Operators for Quantized Non-Monotonic and Fine-Granular Activations in Custom-Tailored DNN Accelerators

    Oct 15, 2026Christoph Berganski, Felix Jentzsch, Marco PlatznerActivation QuantizationNeural Network Activation Functions

  9. Rethinking Activation Functions: Reducing Bias Can Improve Learning

    Oct 10, 2026Zirui Peng, Heyi Zheng, Ziming LiuLoss LandscapeNeural Network Activation Functions

  10. HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification

    Oct 8, 2026Michael W. Spratling, Heiko H. SchüttNeural Network Activation FunctionsNeural Network Generalization

  11. Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

    Oct 1, 2026Tilen Cadez, Sanghoon Lee, Kyoung-Min KimNeural Network Activation FunctionsKolmogorov-Arnold Networks

  12. Let the Neurons Die: Exploiting ReLU-Induced Model Degradation

    Sep 28, 2026Kexin Li, Wenjun Qiu, Joshua Abraham +2ReLU Neural NetworksAdversarial Attacks

  13. An improved periodic activation for PINNs reconstructing convective flows

    Sep 18, 2026Michael Mommert, Marie-Christine Volk, Christian BauerNeural Network Activation FunctionsPhysics-Informed ML

  14. Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery

    Sep 15, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary OptimizationNeural Network Activation Functions

  15. Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity

    Sep 10, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Neural Network Activation FunctionsNeuroevolution

  16. Multi-Behavioral Evolved Substrates Through Neuromodulation and Activation Selection

    Sep 10, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1NeuroevolutionArtificial Life

  17. Diversity of EML-type operators

    Sep 10, 2026Andrzej OdrzywołekNeural Network Activation FunctionsNeural Network Expressivity

  18. Shallow neural network approximation in mixed Sobolev spaces

    Sep 4, 2026Yuwen Li, Guozhi ZhangNeural Network Approximation TheoryFunction Approximation

  19. Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning

    Aug 13, 2026Zeyang Zhang, Tieliang Gong, Junyan Lu +1Loss of PlasticityContinual Learning

  20. TESLA: Taylor Expansion of Sinusoidal Learnable Activations

    Aug 12, 2026Daehwa Ko, Jaehyeon Kim, Seunghyun Ham +1Neural Network GeneralizationNeural Network Activation Functions

  21. QFCQT: A Chaotically Gated Quantformer Framework for Volatile Time-Series Forecasting

    Aug 7, 2026Junkai Lin, Siqi Hou, Raymond LeeMultivariate Time Series ForecastingTime Series Forecasting

  22. Local Stability and Gaussian Smoothing of Quantized Neural Networks

    Jul 22, 2026Sergey Salishev, Anton Makarov, Oleg GranichinNeural Network StabilityNeural Network Activation Functions

  23. Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions

    Jul 20, 2026Meng Hua, Itsik Bergel, Deniz GündüzNeural Network Activation FunctionsWireless Communications

  24. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    Jul 20, 2026Damien Teney, Liangze Jiang, Hemanth Saratchandran +1Neural Network Activation FunctionsInductive Bias