Neural Network Activation Functions

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

Oct 1, 2026cs.LG

Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

Kolmogorov-Arnold Networks (KANs) represent a compelling alternative to traditional Multi-Layer Perceptron (MLP)-based neural networks. By employing activation functions as learnable elements, KANs offer superior interpretability, making them suited for scientific domains. In this work, we investigate the neural scaling laws of KANs and the structural evolution of their learnable activation functions under dataset expansion. Specifically, we evaluate the scaling behavior of three KAN variants---BSRBF-KAN, Gottlieb-KAN, and Faster-KAN---across standard image classification benchmarks (MNIST and Fashion-MNIST) and a specialized scientific regression task (magnetic parameter estimation from domain images of moiré magnetic textures). Our results demonstrate that the test loss

{\cal L}

exhibits a broken neural scaling law (BNSL) behavior as a function of the dataset size

N_D

. After passing through a random-guess regime, the loss follows architecture- and task-dependent scaling behavior. The loss crosses from a faster- to a slower-scaling branch,

{\cal L}\propto N_D^{-α}

and

{\cal L}\propto N_D^{-β}

with

α>β

for image classification tasks. The exponents

α

and

β

depend strongly on both the specific network architecture and the dataset-size regime, ranging from 0.4 to 1.5 and from 0.06 to 0.6, respectively. For the magnetic parameter-regression task, the loss follows a single scaling law with its exponent ranging from 1.28 to 2.59. Additionally, we provide a structural analysis of how activation functions refine their complexity as data volume increases, finding that dataset expansion drives a transition from simple linear-like approximations toward stable, interpretable symbolic forms. These findings provide a quantitative roadmap for the efficient application of KANs while managing the trade-off between model expressivity and computational overhead.

Tilen Cadez, Sanghoon Lee, Kyoung-Min Kim

Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea

Neural Network Activation Functions Kolmogorov-Arnold Networks Neural Scaling Laws

Sep 28, 2026cs.LG

Let the Neurons Die: Exploiting ReLU-Induced Model Degradation

Rectified linear unit (ReLU) networks can suffer from dying neurons, where units with persistently negative pre-activations produce zero outputs, blocking gradients through their activations. To exploit this failure mode, we present three training-time availability attacks based on data ordering and poisoning. We begin with the basic dynamic data-ordering attack (DOA), which greedily constructs a training prefix by selecting the next example that minimizes the target layer's post-update weight sum, aiming to push ReLU units toward negative pre-activations without modifying training samples or labels. We then develop two poisoning attacks, IG-DOA and IG-SKA, which use gradient inversion to synthesize class-conditioned samples by matching reference gradients in adverse model states constructed through data ordering or soft knockout, respectively. Soft knockout rearranges weights across adjacent layers to concentrate negative contributions. On a fully connected ReLU network trained on MNIST, ordering 100 of 60,000 training examples reduces test accuracy from 96% to 95% after only five epochs. Adding 200 poisoned samples from a single class reduces test accuracy to approximately 86-88% after five epochs in most evaluated conditions, compared with approximately 96% under clean training. These results demonstrate that ReLU-targeted data ordering and poisoning can impair learning without directly modifying the victim model's parameters.

Kexin Li, Wenjun Qiu, Joshua Abraham +2

University of Toronto, Toronto, Ontario, Canada

ReLU Neural Networks Adversarial Attacks Neural Network Activation Functions Data Poisoning Attacks

Sep 24, 2026cs.NE

Activation-Flexible ANN-to-SNN Conversion with Finite-State Markov Neurons

Most ANN-to-SNN conversion methods rely on a specific correspondence between the source activation and the spiking neuron dynamics. We propose a finite-state continuous-time Markov chain (CTMC) neuron framework whose stationary spike flux can approximate every continuous nonnegative monotone activation function on a compact interval. For a generalized CTMC family with affine input-dependent transitions, we prove uniform approximation to arbitrary accuracy over this function class and derive an explicit approximation error bound. In practice, two- and three-state CTMCs fit ReLU, sigmoid, softplus, and clipped ReLU on the evaluated input ranges, and we evaluate corresponding MLP conversions for each activation with layerwise rate scaling. Moderate clipping improves the conversion cost-accuracy tradeoff on the MNIST MLP and reduces SynOps by 27% on VGG-11/MNIST at matched ANN-SNN accuracy gap criteria, whereas the trend reverses on VGG-11/CIFAR-10. Mean-field and layerwise diagnostics indicate that finite-window sampling and terminal-layer mismatch are the main residual errors. Overall, our results establish finite-state CTMC neurons as a theoretically grounded framework for activation-flexible ANN-to-SNN conversion beyond fixed activation-neuron correspondences.

Ruiyu Jia, Zhuo-Cheng Xiao

New York University Shanghai, Shanghai, China

Neural Network Activation Functions Spiking Neural Networks Neural Network Approximation Theory

Sep 15, 2026cs.NE

Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery

Indirectly encoded neural networks can assign different activation functions to individual nodes, but the right functions are rarely known in advance. When the available set contains only standard monotonic functions, problems like parity become unsolvable, yet an all-inclusive palette underperforms a curated one. How should evolution discover which functions to use? We address this as a meta-learning problem, designing 13 strategies (11 inspired by biological adaptation mechanisms, plus baseline and oracle controls) that modify the set of available activation functions during evolution. Each strategy translates a biological principle into an evolutionary operator: for example, circadian-inspired oscillatory gating cycles functions in and out of the palette on a fixed schedule, while immune-inspired Clonal Selection permanently protects functions that consistently correlate with fitness. We evaluate all strategies across more than 3,000 runs on parity and non-parity problems, first evolving the activation palette alone, then co-evolving a per-node aggregation palette on harder problems; an independent replication with new seeds confirms a stable high-reliability tier, with Circadian holding its top rank. Bio-inspired strategies match the solve rate of a tuned baseline but converge up to twice as fast, with Circadian halving total compute. Strategy rankings reverse across problem types, with no strategy dominating all domains. Strategy success is largely shaped by timescale compatibility: strategies whose characteristic timescale matches the evolutionary evaluation window consistently outperform those that operate too slowly. The practical guideline: match the mechanism's timescale to the evaluation budget. Rescaling the slowest strategy bypasses the oscillatory barrier entirely: all nine solutions solve parity with non-oscillatory activations paired with min or max aggregation.

Romain Claret, Michael O'Neill, Paul Cotofrei +1

University of Neuchâtel, Neuchâtel, Switzerland · University College Dublin, Ireland

Evolutionary Optimization Neural Network Activation Functions Neuroevolution

Sep 10, 2026cs.SC

Diversity of EML-type operators

The discovery of the EML operator, sufficient to evaluate the standard explicit purely transcendental elementary functions, has led to considerable interest and discussion across multiple scientific disciplines. However, most authors have focused on the binary EML itself, while numerous similar variants with slightly different properties are now known. This article attempts to close this gap by enumerating and classifying them. We also take this opportunity to clarify common misconceptions related to the EML operator. The principal goal, symbolic regression within an architecture as close as possible to proven neural networks which combine matrix multiplication with a single univariate non-linear activation function, remains beyond reach. Instead, we propose a Möbius layer, with rational functions replacing matrix operations, and showcase the recently discovered activation function eml(x,1/x), which allows exp(x) and ln(x) to be recovered separately, and hence all elementary functions to be evaluated within a rational generalization of the neural network.

Andrzej Odrzywołek

Institute of Theoretical Physics, Jagiellonian University, Łojasiewicza 11, 30-348 Kraków, Poland

Neural Network Activation Functions Neural Network Expressivity

Sep 4, 2026math.NA

Shallow neural network approximation in mixed Sobolev spaces

We investigate the best

L_2

approximation of mixed Sobolev spaces by shallow neural networks with

n

neurons and general activation functions. We first establish an activation-independent Fourier-block principle: if an activation has univariate approximation order

ρ

in the sense of the Fourier-block property, then the global approximation rate has algebraic order

\min\{α,ρ\}

for target functions of mixed smoothness

α

, up to explicit logarithmic factors. To verify this property for concrete activations, we introduce a structured univariate approximation condition that implies the Fourier-block property with explicit parameters. For

\mathrm{ReLU}^k

, a matching algebraic lower bound identifies

\min\{α,k+1\}

as the optimal algebraic approximation exponent in any dimension, up to logarithmic factors in the upper bound. The framework also yields the exponent

\min\{α,k+1\}

for cardinal B-splines and soft-

\mathrm{ReLU}^k

, and the full mixed-smoothness exponent

α

for ELU and cosine activations, again up to logarithmic~factors.

Yuwen Li, Guozhi Zhang

School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China

Neural Network Approximation Theory Shallow Neural Networks Neural Network Activation Functions

Sep 2, 2026cs.LG

InKAN: B-Spline KANs via Truncated Power Form

Kolmogorov-Arnold Networks (KANs) place learnable B-spline activations on network edges rather than fixed activations on nodes. The standard Cox-de Boor recursion evaluates these activations through

k

sequential passes for degree-

k

splines, consuming over 90% of forward-pass time. InKAN replaces this recursion with the truncated power form, a classical result from approximation theory that expresses each uniform cubic B-spline as five

(x)_+^3

terms at shifted knot positions. This paper makes three contributions: (1) a torch.compile-fused implementation that collapses these operations into a single GPU kernel, eliminating all recursion, span lookup, and scatter-gather operations; (2) a bounded-coordinate stabilization that clamps the normalized input to

[0, k{+}1]

, preventing the catastrophic cancellation that historically motivated the Cox-de Boor recursion; and (3) a production-ready, open-source package (pip install inkan) that serves as a drop-in replacement for existing KAN layers.

Naveen Mysore

Efficient Neural Network Inference Neural Network Activation Functions Kolmogorov-Arnold Networks

Aug 13, 2026cs.LG

Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning

Plasticity loss has emerged as a critical challenge in continual learning that significantly hinders the acquisition of sequential tasks. While optimizing activation designs offers a potential solution, current fixed-form functions suffer from an inherent spectral bias towards low-frequency variations, whereas learnable variants permit unconstrained updates that induce catastrophic forgetting. To address these limitations, we propose a novel learnable wavelet activation that decomposes the activation function into low-frequency and high-frequency components to explicitly counter spectral bias. Furthermore, we employ dynamic wavelet injection to adaptively enhance plasticity for new tasks, alongside a regularization strategy to ensure the stability of previous learned knowledge. Theoretically, we provide rigorous mathematical guarantees for the proposed framework, proving the structural necessity of the hybrid wavelet architecture for efficient

L^2

approximation and demonstrating that the decoupled learning rate mechanism successfully restores network plasticity for high-frequency information. Additionally, we provide a formal derivation of the loss-driven injection trigger mechanism to precisely guide the injection. Extensive empirical evaluations demonstrate that our approach maintains superior trainability and generalization throughout the learning process and achieves state-of-the-art performance across diverse continual learning benchmarks.

Zeyang Zhang, Tieliang Gong, Junyan Lu +1

School of Computer Science and Technology, Xi’an Jiaotong University and Institute of Multimedia Knowledge Fusion and Engineering, Xi’an 710049, China

Loss of Plasticity Continual Learning Spectral Bias Neural Network Activation Functions

Aug 12, 2026cs.LG

TESLA: Taylor Expansion of Sinusoidal Learnable Activations

The parity problem--deciding whether the number of ones in a binary vector is odd or even--remains challenging for standard neural networks due to linear inseparability and the need for global interactions. We propose TESLA, an activation defined as a learnable combination of sine and cosine terms, enabling explicit control over polynomial degree and selective amplification of high-order components. Theoretically, we show that constraining TESLA's coefficients yields Lipschitz/Rademacher complexity bounds and shapes the training dynamics to emphasize higher-frequency structure. Empirically, on parity with input length n = 32, TESLA attains strong generalization with 100K training samples (approximately 0.002% of the 2^32 input space) and remains robust under heavy corruption, retaining high accuracy with up to 30% label noise. We also compare against periodic and frequency-based baselines (SIREN, SNAKE, and Fourier feature embeddings) on parity and Forrelation. Beyond synthetic structure, TESLA delivers comparable performance on ImageNet-100, indicating that activation-level degree control transfers to more general vision workloads. Code: https://github.com/KAU-QuantumAILab/TESLA

Daehwa Ko, Jaehyeon Kim, Seunghyun Ham +1

Korea Aerospace University, Goyang, Republic of Korea

Neural Network Generalization Neural Network Activation Functions Neural Network Training Dynamics

Aug 7, 2026cs.AI

QFCQT: A Chaotically Gated Quantformer Framework for Volatile Time-Series Forecasting

Forecasting non-stationary time series remains difficult due to long-range dependencies, local volatility bursts, structural shifts, and nonlinear oscillatory behaviors. Although Transformer-based forecasters are effective for modeling long-term temporal dependencies, their feed-forward blocks typically rely on smooth static activations that are insufficiently sensitive to abrupt regime changes. Motivated by quantitative Transformer designs and oscillator-based nonlinear activations, we propose QFCQT, short for Quantum-Fractal-inspired Chaotically Gated Quantformer, for robust forecasting under complex volatile dynamics. Here, "quantum-fractal-inspired" denotes a computational analogy based on soft oscillator superposition and multi-scale nonlinear responses, rather than a formal quantum-mechanical or fractal-theoretic derivation. QFCQT consists of three main components: (1) a Quantformer-style numerical encoder that directly processes multivariate inputs via linear embedding; (2) a learnable Lee-oscillator activation module that maps scalar pre-activations to dynamic oscillatory responses and summarizes them through Max-over-Time pooling; and (3) a smooth-chaotic gated fusion mechanism that adaptively balances conventional smooth activations and chaos-sensitive responses. Furthermore, instead of using a single fixed oscillator, QFCQT employs a soft superposition of eight parameterized Lee oscillator families to adaptively capture different nonlinear response patterns across regimes. Experiments on ETTh1, ETTh2, and A-share Stock Index benchmarks show that QFCQT consistently outperforms strong baselines, including Informer, LogTrans, LSTMa, HAT, and COTN.

Junkai Lin, Siqi Hou, Raymond Lee

Faculty of Science and Technology, Beijing Normal-Hong Kong Baptist University, Zhuhai, China · Guangdong Provincial Key Lab of Interdisciplinary Research and Application for Data Science (BNBU), Zhuhai, China

Multivariate Time Series Forecasting Time Series Forecasting Neural Network Activation Functions Non-Stationary Time Series Forecasting Time Series Transformers

Jul 22, 2026cs.LG

Local Stability and Gaussian Smoothing of Quantized Neural Networks

We study Gaussian averaging as a smooth surrogate for quantized neural models. Under bounded local oscillation, we derive a local dimension-dependent bound on |f-g|, linking Gaussian smoothing to the stability analysis of discontinuous networks. We compute closed-form Gaussian averages of the rectified linear unit (ReLU) and sign activation functions, and illustrate the mechanism on a high-dimensional binary perceptron, where layer-preactivation aggregation under an explicit quantization-noise surrogate yields the Gaussian envelope used in inference-side smoothing and training-side smooth surrogate gradients.

Sergey Salishev, Anton Makarov, Oleg Granichin

St. Petersburg State University, 199034, Saint Petersburg, Russia

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Quantization

Jul 20, 2026cs.LG

Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions

Wireless physical neural networks (WPNNs) embed neural computation directly into analog hardware, offering lower energy consumption and latency than conventional digital implementations. In this paper, we propose a deep WPNN in which nonlinear activations are realized by a multi-hop multiple-input multiple-output (MIMO) relay network, in which each relay implements a trainable complex linear gain and bias, followed by the power amplifier's intrinsic nonlinearity acting as an activation function. The cascade of multiple relays therefore realizes an over-the-air fully connected network whose parameters can be trained end-to-end. We develop two transceiver designs for different channel state information (CSI) availability scenarios: a least squares (LS)-based scheme requiring only receiver-side CSI, and a singular-value-decomposition (SVD)-based scheme requiring both transmitter-side and receiver-side CSI. Simulation results show that the proposed architecture enables accurate over-the-air inference for image classification. In particular, the results highlight the advantage of exploiting hardware nonlinearity for enhanced inference capability.

Meng Hua, Itsik Bergel, Deniz Gündüz

Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, U.K. · Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel

Neural Network Activation Functions Wireless Communications

Jul 20, 2026cs.LG

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be key for pushing AI beyond merely scaling current designs. Method. We present a method to optimize a transformer architecture for a given dataset, which we use as a tool to study optimal task-specific inductive biases. This method replaces the most important non-linearities (GeLUs,;softmax) with functions learned on held-out data. We then train the resulting architectures on other datasets, as a way to evaluate the compatibility between pairs of tasks. Findings. On algorithmic toy tasks, we identify new architectures with dramatic improvements in learning speed, in- and out-of-distribution generalization, and stability across seeds. The new designs prove very task-specific however, and indicate that these tasks require inductive biases very different from those of standard transformers. On code and language modeling datasets, we also find architectures with consistent, yet smaller improvements. These designs transfer much better across datasets and domains (English & computer code). Implications. Our results show that standard transformers are rarely a local optimum in the space of architectures. Simple alternatives can perform much better but sacrifice universality. This suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.

Damien Teney, Liangze Jiang, Hemanth Saratchandran +1

Idiap Research Institute · Adelaide University · EPFL

Neural Network Activation Functions Inductive Bias Neural Architecture Search

Jul 4, 2026math.AP

LRX-PINN: A Layer-Resolving XNet Physics-Informed Neural Network with Integrated Cauchy Activations for Convection-Dominated Problems

Convection-dominated convection-diffusion problems often develop thin layers, where the solution has sharp transition profiles and its derivatives are highly localized. This creates a structural mismatch for standard physics-informed neural networks (PINNs), whose trial spaces are not designed to match the value--derivative structure of such layers. We propose a Layer-Resolving XNet Physics-Informed Neural Network (LRX-PINN) based on integrated Cauchy activations. The proposed basis is transition-type at the solution level, while its derivative recovers a localized Cauchy kernel. We show that this structure matches the scaling of convection-dominated layers, inherits the Cauchy approximation mechanism at the derivative-profile level, and identifies

d/\|w\|

as the effective physical width of a ridge neuron. For analytic layer profiles, this yields derivative-stable exponential approximation in the stretched coordinate and a layer-scaled estimate for the strong residual of the singularly perturbed operator. Numerical experiments on several convection-dominated benchmarks show that LRX-PINN achieves higher accuracy than PIKAN and Fourier-feature PINNs while using less than

30\%

of their trainable parameters. On more challenging benchmarks, embedding the proposed representation into hp-VPINN-based frameworks further improves the best results obtained by existing hp-VPINN-based baselines without changing their original loss functionals or stabilization strategies. These results show that neural representations aligned with layer structure provide a compact and effective approach for convection-dominated problems.

Zihao Guo, Xin Li, Zhihong Xia

Institute for Advanced Research, Great Bay University, Dongguan, China · College of Computer Science and Technology, Dongguan University of Technology, Dongguan, China · Department of Mathematics, Northwestern University, Evanston, USA

Neural PDE Solvers PDE Solving Neural Network Activation Functions Neural Network Approximation Theory Physics-Informed ML

Jul 4, 2026cs.LG

A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent Bernoulli gate. This work gives an alternative interpretation: GELU is the expected output of a hard linear gate with a Gaussian random threshold. This view provides a generative interpretation for the Bernoulli gate: the gate opens once the input clears a latent Gaussian threshold. This interpretation stems from a decomposition based on well-known results in stochastic inventory theory and leads to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases. By considering a latent uniform threshold, we recover a hard-swish-like piecewise-polynomial gate whose nonlinear transition is confined to a finite interval, yielding fixed- and learned-width variants. Controlled experiments on compact vision and language models show that calibrated or learned uniform-threshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, improve over them in most tested settings, and use the finite transition region nontrivially.

Roberto Rossi

Business School, University of Edinburgh, Edinburgh, UK

Neural Network Activation Functions

Jul 3, 2026cs.LG

Rethinking Neural Nonlinearity as Gating

Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.

Muhammad Sabih, Frank Hannig, Jürgen Teich

Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Germany

Neural Network Activation Functions Neural Network Approximation Theory

Jun 30, 2026cs.LG

SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions

In recent years KolmogorovArnold Networks KANs have attracted increasing attention due to their effectiveness in machine learning and scientific computing offering a new paradigm for neural network design In this paper we present SechKAN a novel KAN based on hyperbolic secant sech functions The hyperbolic secant basis is adopted for its smooth bellshaped form localized responses and wellbehaved gradients We employ a 1D linear projection to reduce the number of parameters allowing SechKAN to maintain a model size comparable to that of multilayer perceptrons MLPs Experimental results show the effectiveness of SechKAN on function fitting PDE surrogate modeling and image classification benchmarks including MNIST FashionMNIST CIFAR10 and CIFAR100 On function fitting SechKAN achieves performance comparable to both MLPs and representative KAN variants On PDE surrogate modeling it outperforms MLPs and achieves competitive or better performance than representative KAN variants On image classification benchmarks SechKAN achieves the best performance among the evaluated KAN variants while remaining competitive with MLPs using a comparable number of parameters However SechKAN still incurs higher computational cost than MLPs and some KAN variants Our source code is publicly available at https://github.com/hoangthangta/All-KAN.

Hoang-Thang Ta

University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam

Neural Surrogate Modeling Neural Network Activation Functions Kolmogorov-Arnold Networks

Jun 22, 2026cs.LG

It's Much Easier for Neural Networks to learn Game of Life Dynamics with the Right Activation Function: Polynomial Kolmogorov-Arnold Networks

Previous work has found a gap between the scale of neural networks that reliably learn Conway's Game of Life, and minimal networks capable of representing the classic cellular automaton with hard-coded parameter values. Viewing neural network learning as a search process suggests a dependence on networks large enough to contain sub-networks with lucky initializations (sometimes known as 'winning tickets') that actually learn the task. In this work, we reorient our perspective from discovering Life rules as a search problem back to a learning problem, and reason that with fitting inductive biases, the problem should be much more amenable to minimal networks. We find that network variants with several alternative activation functions meaningfully outperform the default choice of Rectified Linear Units, and in particular, that a 2nd degree polynomial activation function consistently learns Life dynamics with or without the benefit of learning neural weights. Our results provide an informative demonstration of the benefits of matching learning to the task at hand and challenge the easy default choice of scale for all problems. In particular, we advocate for the use of cellular automata as simple test domains for developing strategies that can benefit machine learning for science, physics-based deep learning, and interpretable machine learning.

Tashin Ahmed, Q. Tyrell Davis

Independent Researcher · Work completed while at Cross Labs, Cross Compass Ltd., Japan.

Polynomial Neural Networks Cellular Automata Neural Network Activation Functions Kolmogorov-Arnold Networks

Jun 18, 2026cs.LG

Shifting-based Optimizable Linear Relaxations for General Activation Functions

The use of neural networks (NNs) is rapidly increasing, including in safety- and security-critical domains. To provide formal guarantees about NN behavior, many verification methods rely on optimizable linear relaxations of activation functions. However, existing techniques depend on hand-crafted relaxations for each activation function. Extension to state-of-the-art activation functions therefore requires substantial manual effort. In contrast, our approach SLiR (Shifting-based Linear Relaxations) is broadly applicable, requiring only a Lipschitz constant or a set of critical points. SLiR parameterizes relaxations by their slope and computes the corresponding offset via a shifting procedure that ensures sound upper and lower bounds over the input domain, enabling efficient optimization while maintaining correctness. Our experiments show that SLiR produces tight relaxations across a wide range of practical activation functions and enables verification of up to 7.8x more properties compared to state-of-the-art methods.

Philipp Kern, László Antal, Erika Ábráham +1

Karlsruhe Institute of Technology · RWTH Aachen University · Karlsruhe University of Applied Sciences

Neural Network Verification Neural Network Activation Functions

Jun 15, 2026stat.ML

Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in

W^{2,\infty}((a,b)^d)

can be approximated with arbitrary accuracy, measured in the

W^{1,\infty}

-norm, by a fixed-size neural network using the Elementary Universal Activation Function (

\mathrm{EUAF}

). To extend this result to

W^{s,\infty}((a,b)^d)

for

s\in\mathbb{N}

, we introduce a smooth activation

\mathrm{DUAF}_{\infty}

from the family of Differentiable Universal Activation Functions (

\mathrm{DUAF}_n

). We prove that any function in

W^{s,\infty}((a,b)^d)

can be approximated with arbitrary accuracy in the

W^{s-1,\infty}

-norm by a fixed-size

\mathrm{DUAF}_{\infty}

-activated network. We further construct sigmoidal variants

\widetilde{\mathrm{DUAF}}_n

and show that, for every

1\leq s\leq n

, fixed-size

\widetilde{\mathrm{DUAF}}_n

-activated networks still approximate any

f\in W^{s,\infty}((a,b)^d)

with arbitrary accuracy in the

W^{s-1,\infty}

-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.

Baicheng Li, Haizhao Yang, Shijun Zhang

Department of Mathematics, University of Maryland, College Park, MD, USA · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China

Neural Network Activation Functions Universal Approximation Neural Network Approximation Theory

Jun 14, 2026cs.LG

Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm

Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) and global normalization techniques (e.g., LayerNorm) to prevent gradient instability in deep architectures. However, these mechanisms inherently cause dead neurons, discard critical directional information, and destroy the orthogonality of feature representations. Inspired by the frequency-modulation transmission of biological axons, we propose the Z-Plane Neural Network, which maps hidden states into 2D phasor bundles on a hypersphere. We introduce a novel geometric activation function, Radial Bounding(

\mathbf{x} / \max(1, \|\mathbf{x}\|_2)

), which limits the energy magnitude while preserving the phase (direction). We demonstrate mathematically that this isotropic activation maintains 1-Lipschitz continuity and prevents gradient vanishing by preserving tangential gradients. Empirically, a 100-layer Z-Plane Multi-Layer Perceptron (MLP)-entirely devoid of ReLU and LayerNorm-successfully converges on the MNIST dataset with 98.34% accuracy and absolute numerical stability, proving that bounded geometric activation alone is sufficient for stable deep learning.

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea

Neural Representation Geometry Neural Network Optimization Neural Network Activation Functions Neural Network Training Dynamics

Jun 1, 2026cs.CV

SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks

Normalization layers such as BatchNorm and LayerNorm have long been considered essential for stable training in deep networks. This work demonstrates that they can be fully replaced by a single learnable activation mechanism. We identify a plasticity suppression effect induced by standard normalization: learnable activation parameters rapidly lose adaptability when paired with normalization layers. Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit),

\operatorname{SALU}(x;a,b) = \frac{a x}{\sqrt{1 + a b x^2}},\quad a>0,\; b>0

a bounded, learnable activation that provides intrinsic signal stabilization without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a paradigm grounded in total plasticity: SALU replaces normalization layers, while SWALU and GALU replace standard activations. With ResNet-18, SaluNet-C-18 achieves 97.35% on CIFAR-10 and 83.25% on CIFAR-100 without normalization, maintaining 93.44% and 76.23% at batch size 1 where normalized architectures fail. For transformers, SaluNet-T improves over LayerNorm-GELU from 90.92% to 91.01% on CIFAR-10 and from 66.54% to 68.10% on CIFAR-100. SaluNet-C-50 reaches 78.67% Top-1 on ImageNet-1K at

224\times224

, and

79.23\%

at

288\times288

. These results suggest normalization layers suppress total plasticity, a property biological neurons inherently possess, enabling deep networks to learn effectively.

Mourad Zaied

University of Gabes, Tuisia · Department of electrical engineering National Engineering School of Gabes (ENIG) University of Gabes Avenue Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia

Neural Network Activation Functions Convolutional Neural Networks

May 27, 2026cs.LG

Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

Most existing expressivity theories for neural networks assume exact real arithmetic, whereas practical neural networks are executed under finite-precision floating-point arithmetic with implementation-dependent execution semantics. Recent works have begun studying the expressive power of floating-point neural networks, but existing results are limited to highly restricted activation functions and idealized assumptions such as fixed left-to-right reduction orders and correctly rounded activation implementations. In this work, we study the expressive power of floating-point neural networks under generalized floating-point execution semantics, including arbitrary reduction orders and inexact activation implementations with bounded ulp errors. We investigate when floating-point neural networks can represent arbitrary functions between floating-point domains exactly. To this end, we introduce a general distinguishability framework and show that the ability to distinguish every pair of distinct inputs in the first layer is necessary for universal representability. This characterization yields broad classes of activation implementations that are not universal representators, extending previous isolated counterexamples such as the correctly rounded cosine activation. We further prove that a suitable form of distinguishability is also sufficient for universal representability under mild conditions on the activation implementation. Using this framework, we establish universal representability results for a broad class of practical activation functions, including implementations of

\mathrm{Sigmoid}

,

\tanh

,

\mathrm{ReLU}

,

\mathrm{ELU}

,

\mathrm{SeLU}

,

\mathrm{GeLU}

,

\mathrm{Swish}

,

\mathrm{Mish}

, and

\sin

, under significantly more realistic floating-point execution models than previously known.

Yeachan Park, Geonho Hwang, Wonyeol Lee +1

Neural Network Activation Functions Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

May 26, 2026cs.LG

More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

Feedforward network (FFN) layers account for a large fraction of parameters and nonlinear expressivity in Transformer-based large language models (LLMs). Despite the evolution from ReLU and GELU to gated variants such as SwiGLU, most FFN designs still use a single fixed activation function, applying the same nonlinear transformation to all tokens. In this work, we propose Mixture of Activations (MoA), a token-adaptive FFN design that mixes a dictionary of activation functions using lightweight input-dependent gates while sharing the same linear projections. As an input-independent counterpart, we also introduce learnable activations (LA), which form linear combinations of activation functions for both ReLU-type and SwiGLU-type FFNs. Theoretically, we establish strict finite-width expressive separations among fixed-activation FFNs, LA, and MoA: LA strictly contains fixed-activation FFNs, while MoA strictly contains LA, with the additional expressivity arising from input-dependent nonlinear hybridization. Empirically, we evaluate MoA through extensive pre-training experiments on dense and MoE language models ranging from 0.12B to 2B parameters under different token budgets, optimizers, and learning rate schedules. MoA consistently achieves lower terminal loss and exhibits more favorable scaling behavior than well-tuned baselines, with minimal parameter and computational overhead. These results suggest that token-adaptive activation mixing is a simple and effective mechanism for improving FFN expressivity in LLMs.

Mingze Wang, Jinbo Wang, Yikuan Xia +2

ByteDance Seed · Peking University

Transformer FFNs Transformer Expressivity Transformer Neural Network Activation Functions Feedforward Neural Networks Neural Network Expressivity

May 25, 2026cs.CL

PowLU: An Activation Function for Stable Pre-Training of LLMs

In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearity. For large positive inputs, SwiGLU approximates the quadratic function

x^2

, providing strong nonlinearity and expressive capacity. However, this property also causes numerical instability as the input or model scale increases, particularly in low-precision LLM training. The main reason is its approximate quadratic amplification, which enlarges the output range and exacerbates outliers. To address this issue, we propose a stable activation function, Power Linear Unit (PowLU), for large-scale LLM pre-training. Specifically, PowLU employs a rational power function to achieve adaptive nonlinearity, thereby improving representation ability and enabling stable training in spike regions. Moreover, we provide theoretical justification for several key properties of PowLU. Scaling law experiments confirm that the performance is consistent across model sizes, and further experimental results with the Ling architecture (7.9B and 124B total parameters) demonstrate that PowLU achieves competitive results against SwiGLU and SwiGLU-Clip in large-scale training of LLMs. In addition, the experimental results also show that PowLU effectively improves the scalability of the large-scale training of LLMs.

Peijie Jiang, Yuqi Feng, Cunyin Peng +5

Ling Team, Ant Group

Language Model Pretraining Neural Network Activation Functions LLM Training

May 23, 2026cs.LG

ChainzRule: Sample-Efficient, Robust Deep Learning Across Tabular, NLP, and Vision Tasks

Production deep learning systems across enterprise domains operate under constraints that academic benchmarks routinely obscure: labeled data is expensive, inference budgets are tight, and models that cannot explain their behavior are difficult to trust and maintain. We present ChainzRule (CR), a neural architecture replacing typical activations with learnable polynomial layers governed by Differential Regularization (DREG), a layer-wise Jacobian penalty computed analytically during the forward pass at standard inference cost. The core claim is that bounding intermediate derivatives forces the network toward low-frequency, structurally stable representations, simultaneously reducing dependence on labeled data volume, improving robustness to distribution shift, and providing a measurable, gradient-based handle on model behavior. Evaluated across five domains, CR achieves

85.71\% \pm 2.01\%

on Pima Diabetes (statistically superior to SVM and XGBoost),

46.20\% \pm 0.37\%

on SST-5 sentiment classification with a frozen encoder (superior to RNTN using approximately 5% of its training data),

55.79\%

on SST-5 with a fine-tuned BERT backbone (versus BERT-base linear head at

54.9\%

),

70.17\%

on Yelp Full ordinal regression with 3.2M parameters versus a 10-model average of

66.35\%

, and

+2.32\%

mean corruption accuracy on CIFAR-10-C. All results with reported

p

-values fall below the

α= 0.05

threshold after Bonferroni correction. CR maintains a gradient tail ratio

τ

(p99/mean) of

1.01

--

1.02

against

1.07

--

1.09

for all typical activation function baselines across every data fraction, a structural invariant we propose as the mechanistic driver of sample efficiency and a deployment-time proxy for model reliability.

Rowan Martnishn

Sentivity AI

Polynomial Neural Networks Neural Network Robustness OOD Generalization Jacobian Regularization Neural Network Activation Functions Neural Network Interpretability+3

May 22, 2026cs.LG

Optimization of randomized neural networks for transfer operator approximation

RaNNDy is a randomized neural network architecture for the data-driven approximation of transfer operators associated with complex dynamical systems. The weights and biases of the hidden layers of the network are randomly initialized and kept fixed, only the output layer is trained. This has several advantages over fully optimized neural networks, notably a closed-form solution for the output layer and significantly lower training costs. Despite these advantages, RaNNDy is restricted to the initial selection of weights and biases that parametrize the basis functions required for the operator approximation. Since the basis functions are determined by the activation function, choosing an appropriate activation function for the hidden layers is crucial. In this work, we propose an algorithm that optimizes the activation function itself, while keeping the weights and biases in the randomized neural network fixed, providing a more suitable dictionary. We illustrate the efficacy of the approach using various benchmark problems, including stochastic differential equations and random walks on graphons.

Mohammad Tabish, Stefan Klus

Maxwell Institute for Mathematical Sciences, University of Edinburgh and Heriot–Watt University, Edinburgh, UK · School of Mathematical & Computer Sciences, Heriot–Watt University, Edinburgh, UK

Dynamical Systems Neural Network Optimization Neural Network Activation Functions Neural Network Approximation Theory Operator Learning

May 21, 2026cs.CR

A Constant-Time Implementation Methodology for Activation Functions on Microcontrollers

Embedded neural-network inference can leak information through timing side channels, including leakage caused by the evaluation of activation functions. This work proposes a constant-time implementation methodology for activation functions on embedded microcontrollers and validates it on ReLU, sigmoid, tanh, GELU, and Swish on an ARM Cortex-M4 platform. The proposed methodology combines branchless selection, fixed-cost Padé-based approximation, dummy arithmetic where needed, and cycle alignment to obtain timing-regular activation-function implementations. As motivation, we also evaluate a desynchronization-based countermeasure and show that it remains vulnerable to a template-based timing attack. Experimental results show that the resulting protected implementations achieve identical cycle counts for all tested inputs, including (88) cycles in the three-function setting and (108) cycles in the five-function setting. At the same time, the numerical-error analysis indicates that the approximated nonlinear functions retain high accuracy. These results suggest that the proposed methodology provides a practical basis for constructing side-channel-resistant activation functions in embedded inference.

Andrii Tyvodar, Andreas Rechberger, Dirmanto Jap +4

Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava, Bratislava, Slovakia · Temasek Laboratories, Nanyang Technological University, Singapore · Faculty of Computer Science, Albstadt-Sigmaringen University, Albstadt, Germany +2

Edge Inference Neural Network Activation Functions

May 20, 2026cs.CV

Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style vision backbones. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures.

Jeffrey Wang, Jonathan Gregory, Grigorios G. Chrysos

University of Wisconsin–Madison, Madison, WI, USA.

Polynomial Neural Networks Neural Network Activation Functions Image Classification

May 18, 2026cond-mat.dis-nn

Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines

The great success of neural networks primarily arises from the presence of the large number of weight parameters combined with nonlinearities in the input-output relationship of single neurons. In this work, we study the relationship between the statistical properties of the weights and the nonlinearity of the hidden unit in Restricted Boltzmann Machines (RBMs) on the one side, and the distribution induced on binary visible units. We do this for four commonly used activation functions: Linear, Step, ReLU, and Exponential, and make qualitative predictions about the ability of these models to learn distributions with strong higher order interactions over the visible nodes. We show that in general, in an ensemble of RBMs with Gaussian weights, these distributions are rare and hard to learn, except when the hidden unit activation function is an Exponential.

Giovanni di Sarra, Yasser Roudi

Kavli Institute for Systems Neuroscience, Norwegian University of Science and Technology, 7491 Trondheim, Norway · Department of Mathematics, King’s College London, London, Strand, WC2R 2LS, UK

Unsupervised Representation Learning Neural Network Activation Functions Restricted Boltzmann Machines