Neural Network Activation Functions

Follow this topic Ask about this topic

Activation functions transform a network’s intermediate signals, determining which patterns it can represent and how gradients behave during training. Work compares standard choices such as ReLU, GELU, and SiLU with smooth, polynomial, rational, periodic, and wavelet-based alternatives, as well as learnable or input-conditioned mixtures and gates. Theory studies approximation rates and optimization dynamics, while experiments test classification, language modeling, scientific tasks, and robustness; hardware research seeks cheaper implementations through piecewise approximations and lookup methods. A central tension is that gains depend on the task, model scale, and training regime: greater flexibility or smoothness may improve representation or stability, but can add computation or alter generalization, and the best activation is not universal.

Latest papers

78 papers10 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

Mar 19, 2026

DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training

Non-linear activation functions play a pivotal role in on-device inference and training, as they not only consume substantial hardware resources but also impose a significant impact on system performance and energy efficiency. In this work, we propose Distribution-Aware Piecewise Activation (DAPA), a differentiable and hardware-friendly activation function for Transformer architectures by exploiting the distribution of pre-activation data. DAPA employs a non-uniform piecewise approximation that allocates finer segments to high-probability regions of the distribution, improving generalizability over prior piecewise linear methods. The resulting approximation is further quantized using Distribution-Weighted Mean Square Error to reduce latency and resource utilization for hardware deployment. Our HLS implementation demonstrates that DAPA speeds up GELU computation by 16

\times

and decreases DSP utilization by 16

\times

while maintaining comparable or better performance across vision Transformers and GPT-2 models.

Maoyang Xiang, Bo Wang

Information Systems Technology and Design Singapore University of Technology and Design Singapore

Neural Network Activation Functions Efficient Transformer Inference On-Device Inference Activation Quantization On-Device Training

Mar 18, 2026

A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networks

Physics-Informed Neural Networks(PINNs) are a powerful and flexible learning framework that has gained significant attention in recent years. It has demonstrated strong performance across a wide range of scientific and engineering problems. In parallel, wavelets have been extensively used as efficient computational tools due to their strong approximation capabilities. Motivated by the common failure modes observed in standard PINNs, this work introduces a novel family of adaptive wavelet-based activation functions. The proposed activation functions significantly improve training stability and expressive power by combining trainable wavelet functions with either trainable or fixed hyperbolic tangent and softplus functions. Five distinct activation functions are developed within the PINN framework and systematically evaluated across four representative classes of partial differential equations (PDEs). Comprehensive comparisons using bar plots demonstrate improved robustness and accuracy compared to traditional activation functions. Furthermore, the proposed approach is validated through direct comparisons with baseline PINNs, transformer-based architectures such as PINNsFormer, and other deep learning models, highlighting its effectiveness and generality.

Krishna Murari

†Department of Mathematics, IIT Madras, Chennai 600036, India

Neural Network Activation Functions Physics-Informed ML Wavelet Methods

Mar 16, 2026

PAKAN: Pixel Adaptive Kolmogorov-Arnold Network Modules for Pansharpening

Pansharpening aims to fuse high-resolution spatial details from panchromatic images with the rich spectral information of multispectral images. Existing deep neural networks for this task typically rely on static activation functions, which limit their ability to dynamically model the complex, non-linear mappings required for optimal spatial-spectral fusion. While the recently introduced Kolmogorov-Arnold Network (KAN) utilizes learnable activation functions, traditional KANs lack dynamic adaptability during inference. To address this limitation, we propose a Pixel Adaptive Kolmogorov-Arnold Network framework. Starting from KAN, we design two adaptive variants: a 2D Adaptive KAN that generates spline summation weights across spatial dimensions and a 1D Adaptive KAN that generates them across spectral channels. These two components are then assembled into PAKAN 2to1 for feature fusion and PAKAN 1to1 for feature refinement. Extensive experiments demonstrate that our proposed modules significantly enhance network performance, proving the effectiveness and superiority of pixel-adaptive activation in pansharpening tasks.

Haoyu Zhang, Haojing Chen, Zhen Zhong +1

University of Electronic Science and Technology of China

Neural Network Activation Functions Pansharpening Kolmogorov-Arnold Networks Multi-Source Image Fusion Remote Sensing

Mar 7, 2026

PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks

Biological neural systems employ diverse neurotransmitters -- glutamate, GABA, dopamine, acetylcholine -- to implement distinct signal-processing modalities within shared neural circuits. In contrast, modern transformers apply a single fixed activation function across all feed-forward neurons. We introduce PolyGLU (Polychromatic Gated Linear Unit), a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions via a differentiable mechanism combining learned static preferences with input-conditioned gating, trained end-to-end with Gumbel-Softmax. We train PolychromaticLM, a 597M-parameter transformer, on ~10B tokens using a single NVIDIA A100 GPU. Our key finding is emergent routing behavior: without any explicit sparsity loss or entropy regularization, the routing mechanism converges to near-deterministic activation selections (mean dynamic entropy = 0.030% of maximum), with a striking depth-dependent specialization pattern -- early layers prefer GELU while deep layers strongly favor Tanh. Three layers maintain elevated routing entropy, suggesting computational flexibility points. The routing architecture adds only 0.23% parameter overhead (~1.4M parameters) and proves fully robust to supervised fine-tuning: routing entropy remains constant at ln(4) throughout 13,067 SFT steps. On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens. All code, weights, and training infrastructure are released under Apache 2.0.

Daniel Nobrega Medeiros

Independent researcher.

Transformer FFNs Neural Network Activation Functions Conditional Computation

Mar 6, 2026

IGLU: The Integrated Gaussian Linear Unit Activation Function

Activation functions are fundamental to deep neural networks, governing gradient flow, optimization stability, and representational capacity. Within historic deep architectures, while ReLU has been the dominant choice for the activation function, modern transformer-based models increasingly are adopting smoother alternatives such as GELU and other self-gated alternatives. Despite their empirical success, the mathematical relationships among these functions and the principles underlying their effectiveness remains only partially understood. We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution. This derivation yields a closed-form expression whose gating component is exactly the Cauchy CDF, providing a principled one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter

σ

. Unlike GELU's Gaussian gate, IGLU's heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients. We further introduce IGLU-Approx, a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation. Through evaluations on CIFAR-10, CIFAR-100, and WikiText-103 across ResNet-20, ViT-Tiny, and GPT-2 Small, IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost. In particular, we show that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification datasets.

Mingi Kang, Zai Yang, Jeova Farias Sales Rocha Neto

Bowdoin College, 255 Maine Street, Brunswick, ME 04011, USA

Neural Network Activation Functions

Feb 26, 2026

Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies

Varying the number of neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational interdependence between units. Elementwise activation functions, particularly their limited equivariance properties, are identified as a central hurdle to achieving this. We propose activation functions designed around chosen equivariances, enabling dense neural networks to adapt via singular value decomposition of their affine maps. This diagonalises layers into one-to-one, ordered connections, making it easier to assess the impact of individual connections on function. Low-impact neurons can be removed (neurodegeneration), and a thresholded buffer of largely inactive

scaffold' neurons is maintained (neurogenesis). These symmetry-led diagonalisation and structural changes prioritise function-invariance: neurogenesis exactly preserves function, neurodegeneration is asymptotically function-preserving with the pruned singular-value threshold. Moreover, this construction enables asymptotic 50\% parameter sparsification of dense networks while preserving function analytically. Thus, the approach enables real-time restructuring of the architecture in response to task demands, task appending, removal or changes. Conceptually, this motivates broader applications of symmetry prescriptions at the small scale, even when no data-driven model-scale symmetry adherence is required. Following this, we build activations around symmetry, not from classical assumptions about individual neurons. Unlike typical elementwise forms, these alternatives feature explicit basis independence and do not individuate neurons. We leverage this to decompose and reinterpret neurons across sequential layers, simplifying structure and providing a natural route to adaptive topologies. Additional developments include a new tunable parameter, the

intrinsic length'.

George Bird

Department of Computer Science, University of Manchester, UK

Dynamic Neural Networks Neural Network Activation Functions Neural Network Pruning Neural Network Compression

Feb 25, 2026

GRAU: A Generic Runtime-Reconfigurable Activation-Requantization Unit for Quantized Neural Network Accelerators

As neural networks continue to scale, low-precision quantization is widely adopted in edge accelerators, but conventional multi-threshold activation hardware requires 2^n-1 thresholds for n-bit integer outputs, leading to rapidly increasing hardware costs. We propose GRAU, a Generic Runtime-Reconfigurable Activation-Requantization Unit that fuses batch normalization, nonlinear activation, and requantization through piecewise-linear fitting with Power-of-Two slopes. GRAU requires only basic comparators and 1-bit right shifters, while supporting mixed-precision quantization and runtime switching among post-accumulation functions, including modern nonlinear activations such as SiLU and GELU. Compared with multi-threshold activation-requantization units, GRAU reduces LUT consumption by approximately 90% or more while improving hardware efficiency, flexibility, and scalability. We further explore the trade-off between accuracy and hardware cost across different approximation settings and provide an open-source toolchain for GRAU-aware network training.

Yuhao Liu, Salim Ullah, Akash Kumar

Ruhr University Bochum, Germany · Dresden University of Technology, Germany · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI Dresden/Leipzig), Germany

Neural Network Hardware Acceleration Neural Network Activation Functions

Feb 23, 2026

Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

Smooth activation functions are ubiquitous in modern deep learning, yet their theoretical advantages over non-smooth counterparts remain poorly understood. In this work, we study both approximation and statistical properties of neural networks with smooth activations for learning functions in the Sobolev space

W^{s,\infty}([0,1]^d)

with

s>0

. We prove that constant-depth networks equipped with smooth activations achieve smoothness adaptivity: increasing width alone suffices to attain the minimax-optimal approximation and estimation error rates (up to logarithmic factors). In contrast, for non-smooth activations such as ReLU, smoothness adaptivity is fundamentally limited by depth: the attainable approximation order is bounded by depth, and higher-order smoothness requires proportional depth growth. These results identify activation smoothness as a fundamental mechanism, complementary to depth, for achieving optimal rates over Sobolev function classes. Technically, our analysis is based on a multi-scale approximation framework that yields explicit neural network approximators with controlled parameter norms and model size. This complexity control ensures statistical learnability under empirical risk minimization (ERM) and avoids the impractical

\ell^0

-sparsity constraints commonly required in prior analyses.

Yuhao Liu, Zilin Wang, Lei Wu +1

Department of Mathematical Sciences, Tsinghua University · School of Mathematical Sciences, Peking University · Center for Machine Learning Research, Peking University +1

Neural Network Approximation Theory Neural Network Activation Functions Shallow Neural Networks Statistical Learning Theory

Feb 17, 2026

Beyond ReLU: Bifurcation, Oversmoothing, and Topological Priors

Graph Neural Networks (GNNs) learn node representations through iterative network-based message-passing. While powerful, deep GNNs suffer from oversmoothing, where node features converge to a homogeneous, non-informative state. We re-frame this problem of representational collapse from a bifurcation theory perspective, characterizing oversmoothing as convergence to a stable ``homogeneous fixed point.'' Our central contribution is the theoretical discovery that this undesired stability can be broken by replacing standard monotone activations (e.g., ReLU) with a class of functions. Using Lyapunov-Schmidt reduction, we analytically prove that this substitution induces a bifurcation that destabilizes the homogeneous state and creates a new pair of stable, non-homogeneous patterns that provably resist oversmoothing. Our theory predicts a precise, nontrivial scaling law for the amplitude of these emergent patterns, which we quantitatively validate in experiments. Finally, we demonstrate the practical utility of our theory by deriving a closed-form, bifurcation-aware initialization and showing its utility in real benchmark experiments.

Erkan Turan, Gaspard Abel, Maysam Behmanesh +2

LIX, Ecole Polytechnique, IP Paris · Université Paris Saclay, Université Paris Cité, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli, F-91190, Gif-sur-Yvette, France · Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS, 75006 Paris, France..

GNN Oversmoothing Graph Neural Networks Over-Smoothing in GNNs Neural Network Activation Functions

Feb 12, 2026

Rational Neural Networks have Expressivity Advantages

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of

\varepsilon>0

, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only

\mathrm{poly}(\log\log(1/\varepsilon))

overhead in size, while the converse provably requires

Ω(\log(1/\varepsilon))

parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Maosen Tang, Alex Townsend

Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity

Jan 10, 2026

FlexAct: Why Learn when you can Pick?

Learning activation functions has emerged as a promising direction in deep learning, allowing networks to adapt activation mechanisms to task-specific demands. In this work, we introduce a novel framework that employs the Gumbel-Softmax trick to enable discrete yet differentiable selection among a predefined set of activation functions during training. Our method dynamically learns the optimal activation function independently of the input, thereby enhancing both predictive accuracy and architectural flexibility. Experiments on synthetic datasets show that our model consistently selects the most suitable activation function, underscoring its effectiveness. These results connect theoretical advances with practical utility, paving the way for more adaptive and modular neural architectures in complex learning scenarios.

Ramnath Kumar, Kyle Ritscher, Junmin Judy +2

University of California Los Angeles. · University of California Los Angeles, Arena.

Neural Network Activation Functions

Oct 29, 2025

Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k>= 4, to detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter

α

varies. Typically, our analysis reveals that a multi-mode degeneracy consistently occurs at the critical number 0, independent of k. (2) As a by-product, we further show that such bifurcations are width-independent, arise only for nonnegative

α

and that the global minimum undergoes no further symmetry-breaking instability throughout the engineering regime

α

in range (0,1). An explicit example with k=5 is presented to illustrate the framework and exhibit the resulting bifurcation together with their symmetries.

Jingzhou Liu

Shallow Neural Networks Dynamical Systems Neural Network Optimization Neural Network Activation Functions Neural Network Training Dynamics

Oct 25, 2025

SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks

The choice of activation function plays a critical role in neural networks, yet most architectures still rely on fixed, uniform activation functions across all neurons. We introduce SmartMixed, a novel two-phase training strategy that allows networks to learn optimal per-neuron activation functions while preserving computational efficiency at inference. In the first phase, neurons adaptively select from a pool of candidate activation functions (ReLU, Sigmoid, Tanh, Leaky_ReLU, ELU, SELU) using a differentiable hard mixture mechanism. In the second phase, each neuron's activation function is fixed according to the learned selection, resulting in a computationally efficient network that supports continued training with optimized vectorized operations. We evaluate SmartMixed on the MNIST dataset using feedforward neural networks of different architectures. Our analysis reveals that neurons in different layers exhibit distinct preferences for activation functions, providing insights into the functional diversity within neural architectures. We also demonstrated that SmartMixed effectively trains the network by allowing neurons to select their preferred activation functions, competing against models using a single fixed state-of-the-art activation function.

Amin Omidvar

Independent Researcher Toronto, Ontario, Canada

Neural Network Optimization Neural Network Activation Functions

Sep 17, 2025

Circuit realization and hardware linearization of monotone operator equilibrium networks

It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural network in analog hardware. We furthermore show that the gradient of such a circuit can be computed directly in hardware, using a procedure we call hardware linearization. This allows the network to be trained in hardware, which we demonstrate with a device-level circuit simulation. We extend the results to cascades of resistor-diode networks, which can be used to implement feedforward and other asymmetric networks. We finally show that different nonlinear elements give rise to different activation functions, and introduce the novel diode ReLU which is induced by a non-ideal diode model.

Thomas Chaffey

School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia

ReLU Neural Networks Neuromorphic Computing Neural Network Activation Functions

Jul 11, 2025

Understanding Two-Layer Neural Networks with Smooth Activation Functions

This paper aims to understand the training solution, which is obtained by the back-propagation algorithm, of two-layer neural networks whose hidden layer is composed of the units with smooth activation functions, including the usual sigmoid type most commonly used before the advent of ReLUs. The mechanism contains four main principles: construction of Taylor series expansions, strict partial order of knots, smooth-spline implementation and smooth-continuity restriction. The universal approximation for arbitrary input dimensionality is proved and the explanation of training solutions is given. Through the principles proposed, the mystery of ``black box'' of the solution space is largely revealed. The new proofs employed also enrich approximation theory.

Changcun Huang

Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Training Dynamics Neural Network Expressivity Two-Layer Neural Networks

May 29, 2025

SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

Prevailing activation functions such as Swish and GELU tend toward domain-specific optima, Swish was discovered via neural architecture search on vision benchmarks, while GELU dominates transformer-based language models, and neither offers any mechanism to adapt its gating shape to individual layers. This rigidity is especially consequential in transformer FFN blocks, where LayerNorm, unlike BatchNorm, does not suppress the gradient pathologies that activation choice induces across depth. We propose SG-Blend, a per layer adaptive activation that combines SSwish, a bias-corrected, parametric Swish variant we also introduce, with learnable sharpness \b{eta} and zero-centering bias γ, with GELU through a per-layer blend coefficient α, letting each layer locate its own optimum along the SSwishGELU continuum at a cost of only three additional scalars per FFN block, with \b{eta} initialized to 1.0 and learned freely via backpropagation. On BERT-style IMDB classification (5 seeds), it matches peak accuracy (81.31%) while reducing seed-to-seed variance by 42% relative to GELU. Furthermore, it generalizes to autoregressive pretraining, achieving the lowest validation perplexity (49.10) on WikiText103 among all baselines. Crucially, ablations confirm the interpolation structure itself drives these gains, delivering reliable, top-tier performance. Beyond natural language processing, we demonstrate that SG-Blend generalizes robustly to a wider variety of tasks, extending its efficacy to computer vision and other diverse domains.

Gaurav Sarkar, Syed Affan Daimi, Jay Gala +1

IIIT · Microsoft · Intel

Transformer FFNs Transformer Neural Network Activation Functions

Feb 26, 2025

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

The architecture of a neural network and the choice of its activation function are both fundamental to its performance. Equally important is ensuring that these two elements are well matched, as their alignment is key to effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a model that combines sine-type activations with the multi-component and multi-layer structure of MMNNs. In an FMMNN, each component is represented as a trainable linear combination of fixed random sine-type basis functions, while multi-layer composition generates more complex and adaptive high-frequency features. We establish that FMMNNs retain exponential expressive power for function approximation even under a low-rank architectural structure. We also analyze the optimization landscape of FMMNNs and find it to be substantially more favorable than that of standard fully connected neural networks, especially for high-frequency targets. In addition, we propose a scaled random initialization method for the first-layer weights in FMMNNs, which accelerates training and improves final performance when sufficient samples are available. Extensive numerical experiments support our theoretical insights, showing that FMMNNs achieve strong accuracy and favorable convergence behavior on oscillatory function-approximation benchmarks.

Shijun Zhang, Hongkai Zhao, Yimin Zhong +1

Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1

Fourier Feature Embeddings Neural Network Optimization Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity Modular Neural Networks

Jul 13, 2023

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set

\mathscr{A}

is defined to encompass the majority of commonly used activation functions, such as

\mathtt{ReLU}

,

\mathtt{LeakyReLU}

,

\mathtt{ReLU}^2

,

\mathtt{ELU}

,

\mathtt{CELU}

,

\mathtt{SELU}

,

\mathtt{Softplus}

,

\mathtt{GELU}

,

\mathtt{SiLU}

,

\mathtt{Swish}

,

\mathtt{Mish}

,

\mathtt{Sigmoid}

,

\mathtt{Tanh}

,

\mathtt{Arctan}

,

\mathtt{Softsign}

,

\mathtt{dSiLU}

, and

\mathtt{SRS}

. We demonstrate that for any activation function

\varrho\in \mathscr{A}

, a

\mathtt{ReLU}

network of width

N

and depth

L

can be approximated to arbitrary precision by a

\varrho

-activated network of width

3N

and depth

2L

on any bounded set. This finding enables the extension of most approximation results achieved with

\mathtt{ReLU}

networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,

\,

depth) scaling factors can be further reduced from

(3,2)

to

(1,1)

if

\varrho

falls within a specific subset of

\mathscr{A}

. This subset includes activation functions such as

\mathtt{ELU}

,

\mathtt{CELU}

,

\mathtt{SELU}

,

\mathtt{Softplus}

,

\mathtt{GELU}

,

\mathtt{SiLU}

,

\mathtt{Swish}

, and

\mathtt{Mish}

.

Shijun Zhang, Jianfeng Lu, Hongkai Zhao

Department of Mathematics Duke University

ReLU Neural Networks Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity