Neural Network Activation Functions

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

All topics

Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks
Oct 1, 2026Tilen Cadez, Sanghoon Lee, Kyoung-Min KimNeural Network Activation Functions Kolmogorov-Arnold Networks
Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea
Let the Neurons Die: Exploiting ReLU-Induced Model Degradation
Sep 28, 2026Kexin Li, Wenjun Qiu, Joshua Abraham +2ReLU Neural Networks Adversarial Attacks
University of Toronto, Toronto, Ontario, Canada
Activation-Flexible ANN-to-SNN Conversion with Finite-State Markov Neurons
Sep 24, 2026Ruiyu Jia, Zhuo-Cheng XiaoNeural Network Activation Functions Spiking Neural Networks
New York University Shanghai, Shanghai, China
Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery
Sep 15, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary Optimization Neural Network Activation Functions
University of Neuchâtel, Neuchâtel, Switzerland · University College Dublin, Ireland
Diversity of EML-type operators
Sep 10, 2026Andrzej OdrzywołekNeural Network Activation Functions Neural Network Expressivity
Institute of Theoretical Physics, Jagiellonian University, Łojasiewicza 11, 30-348 Kraków, Poland
Shallow neural network approximation in mixed Sobolev spaces
Sep 4, 2026Yuwen Li, Guozhi ZhangNeural Network Approximation Theory Shallow Neural Networks
School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China
InKAN: B-Spline KANs via Truncated Power Form
Sep 2, 2026Naveen MysoreEfficient Neural Network Inference Neural Network Activation Functions
Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning
Aug 13, 2026Zeyang Zhang, Tieliang Gong, Junyan Lu +1Loss of Plasticity Continual Learning
School of Computer Science and Technology, Xi’an Jiaotong University and Institute of Multimedia Knowledge Fusion and Engineering, Xi’an 710049, China
TESLA: Taylor Expansion of Sinusoidal Learnable Activations
Aug 12, 2026Daehwa Ko, Jaehyeon Kim, Seunghyun Ham +1Neural Network Generalization Neural Network Activation Functions
Korea Aerospace University, Goyang, Republic of Korea
QFCQT: A Chaotically Gated Quantformer Framework for Volatile Time-Series Forecasting
Aug 7, 2026Junkai Lin, Siqi Hou, Raymond LeeMultivariate Time Series Forecasting Time Series Forecasting
Faculty of Science and Technology, Beijing Normal-Hong Kong Baptist University, Zhuhai, China · Guangdong Provincial Key Lab of Interdisciplinary Research and Application for Data Science (BNBU), Zhuhai, China
Local Stability and Gaussian Smoothing of Quantized Neural Networks
Jul 22, 2026Sergey Salishev, Anton Makarov, Oleg GranichinNeural Network Activation Functions Neural Network Approximation Theory
St. Petersburg State University, 199034, Saint Petersburg, Russia
Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions
Jul 20, 2026Meng Hua, Itsik Bergel, Deniz GündüzNeural Network Activation Functions Wireless Communications
Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, U.K. · Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Jul 20, 2026Damien Teney, Liangze Jiang, Hemanth Saratchandran +1Neural Network Activation Functions Inductive Bias
Idiap Research Institute · Adelaide University · EPFL
LRX-PINN: A Layer-Resolving XNet Physics-Informed Neural Network with Integrated Cauchy Activations for Convection-Dominated Problems
Jul 4, 2026Zihao Guo, Xin Li, Zhihong XiaNeural PDE Solvers PDE Solving
Institute for Advanced Research, Great Bay University, Dongguan, China · College of Computer Science and Technology, Dongguan University of Technology, Dongguan, China · Department of Mathematics, Northwestern University, Evanston, USA
A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function
Jul 4, 2026Roberto RossiNeural Network Activation Functions
Business School, University of Edinburgh, Edinburgh, UK
Rethinking Neural Nonlinearity as Gating
Jul 3, 2026Muhammad Sabih, Frank Hannig, Jürgen TeichNeural Network Activation Functions Neural Network Approximation Theory
Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Germany
SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions
Jun 30, 2026Hoang-Thang TaNeural Surrogate Modeling Neural Network Activation Functions
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam
It's Much Easier for Neural Networks to learn Game of Life Dynamics with the Right Activation Function: Polynomial Kolmogorov-Arnold Networks
Jun 22, 2026Tashin Ahmed, Q. Tyrell DavisPolynomial Neural Networks Cellular Automata
Independent Researcher · Work completed while at Cross Labs, Cross Compass Ltd., Japan.
Shifting-based Optimizable Linear Relaxations for General Activation Functions
Jun 18, 2026Philipp Kern, László Antal, Erika Ábráham +1Neural Network Verification Neural Network Activation Functions
Karlsruhe Institute of Technology · RWTH Aachen University · Karlsruhe University of Applied Sciences
Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy
Jun 15, 2026Baicheng Li, Haizhao Yang, Shijun ZhangNeural Network Activation Functions Universal Approximation
Department of Mathematics, University of Maryland, College Park, MD, USA · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
Jun 14, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungNeural Representation Geometry Neural Network Optimization
College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea
SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks
Jun 1, 2026Mourad ZaiedNeural Network Activation Functions Convolutional Neural Networks
University of Gabes, Tuisia · Department of electrical engineering National Engineering School of Gabes (ENIG) University of Gabes Avenue Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia
Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations
May 27, 2026Yeachan Park, Geonho Hwang, Wonyeol Lee +1Neural Network Activation Functions Universal Approximation
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Transformer FFNs Transformer Expressivity
ByteDance Seed · Peking University
PowLU: An Activation Function for Stable Pre-Training of LLMs
May 25, 2026Peijie Jiang, Yuqi Feng, Cunyin Peng +5Language Model Pretraining Neural Network Activation Functions
Ling Team, Ant Group
ChainzRule: Sample-Efficient, Robust Deep Learning Across Tabular, NLP, and Vision Tasks
May 23, 2026Rowan MartnishnPolynomial Neural Networks Neural Network Robustness
Sentivity AI
Optimization of randomized neural networks for transfer operator approximation
May 22, 2026Mohammad Tabish, Stefan KlusDynamical Systems Neural Network Optimization
Maxwell Institute for Mathematical Sciences, University of Edinburgh and Heriot–Watt University, Edinburgh, UK · School of Mathematical & Computer Sciences, Heriot–Watt University, Edinburgh, UK
A Constant-Time Implementation Methodology for Activation Functions on Microcontrollers
May 21, 2026Andrii Tyvodar, Andreas Rechberger, Dirmanto Jap +4Edge Inference Neural Network Activation Functions
Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava, Bratislava, Slovakia · Temasek Laboratories, Nanyang Technological University, Singapore · Faculty of Computer Science, Albstadt-Sigmaringen University, Albstadt, Germany +2
Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
May 20, 2026Jeffrey Wang, Jonathan Gregory, Grigorios G. ChrysosPolynomial Neural Networks Neural Network Activation Functions
University of Wisconsin–Madison, Madison, WI, USA.
Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines
May 18, 2026Giovanni di Sarra, Yasser RoudiUnsupervised Representation Learning Neural Network Activation Functions
Kavli Institute for Systems Neuroscience, Norwegian University of Science and Technology, 7491 Trondheim, Norway · Department of Mathematics, King’s College London, London, Strand, WC2R 2LS, UK
Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes
May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural Networks Activation Sparsity
ArcGate: Adaptive Arctangent Gated Activation
May 14, 2026Avik Bhattacharya, Siddhant Dnyanesh Gole, Subhasis Chaudhuri +2Neural Network Activation Functions Noise Robustness
Microwave Remote Sensing Lab Center of Studies in Resources Engineering Indian Institute of Technology Bombay, India · Centre of Machine Intelligence and Data Science Indian Institute of Technology Bombay, India · Department of Electrical Engineering Indian Institute of Technology Bombay, India +1
A Composite Activation Function for Learning Stable Binary Representations
May 12, 2026Seokhun Park, Choeun Kim, Kwanho Lee +3Concept Bottleneck Models Binary Neural Networks
Department of Statistics, Seoul National University · Department of Applied Mathematics, University of Twente
Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks
May 7, 2026Yi Wei, Xuan Qi, Furao ShenNeural Network Activation Functions Neural Network Approximation Theory
DITEN University of Genoa, Genoa, Italy · State Key Laboratory of Novel Software Technology School of Artificial Intelligence Nanjing University, Jiangsu, China
OptiLookUp: An Optical ROM-Based Lookup Table Engine for Photonic Accelerators
May 5, 2026Ankur Singh, Akhilesh JaiswalNeural Network Activation Functions Inference Acceleration
Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA
Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions
May 4, 2026Wentao Zhang, Yutong Zhang, Yifan Zhu +1Efficient Neural Network Inference Neural Network Optimization
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · College of Computer Science, Sichuan University, Chengdu, China
KANs need curvature: penalties for compositional smoothness
May 4, 2026James BagrowNeural Network Activation Functions Neural Network Interpretability
Mathematics & Statistics, University of Vermont, Burlington, VT, United States · Vermont Complex Systems Center, University of Vermont, Burlington, VT, United States
When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer
Apr 25, 2026Lucky VermaNeural Network Activation Functions Layer Normalization
Independent Researcher
Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions
Apr 23, 2026Eylon E. KrauseNeural Network Optimization Neural Network Activation Functions
Weizmann Institute of Science1, Rehovot, Israel (7610001) · College of Management Academic Studies2, Rishon LeZion, Israel (7579806)
Rational Neural Networks have Expressivity Advantages
Feb 12, 2026Maosen Tang, Alex TownsendNeural Network Activation Functions Neural Network Approximation Theory
Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States
FlexAct: Why Learn when you can Pick?
Jan 10, 2026Ramnath Kumar, Kyle Ritscher, Junmin Judy +2Neural Network Activation Functions
University of California Los Angeles. · University of California Los Angeles, Arena.
Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU
Oct 29, 2025Jingzhou LiuShallow Neural Networks Dynamical Systems
SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks
Oct 25, 2025Amin OmidvarNeural Network Optimization Neural Network Activation Functions
Independent Researcher Toronto, Ontario, Canada
Circuit realization and hardware linearization of monotone operator equilibrium networks
Sep 17, 2025Thomas ChaffeyReLU Neural Networks Neuromorphic Computing
School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia
Understanding Two-Layer Neural Networks with Smooth Activation Functions
Jul 11, 2025Changcun HuangNeural Network Activation Functions Neural Network Approximation Theory
Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C
SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations
May 29, 2025Gaurav Sarkar, Syed Affan Daimi, Jay Gala +1Transformer FFNs Transformer
IIIT · Microsoft · Intel
Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential
Feb 26, 2025Shijun Zhang, Hongkai Zhao, Yimin Zhong +1Fourier Feature Embeddings Neural Network Optimization
Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions
Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural Networks Neural Network Activation Functions
Department of Mathematics Duke University

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

Tilen Cadez, Sanghoon Lee, Kyoung-Min Kim

Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea

Neural Network Activation Functions Kolmogorov-Arnold Networks Neural Scaling Laws

Open paper AskPDF arXiv

Kolmogorov-Arnold Networks (KANs) represent a compelling alternative to traditional Multi-Layer Perceptron (MLP)-based neural networks. By employing activation functions as learnable elements, KANs offer superior interpretability, making them suited for scientific domains. In this work, we investigate the neural scaling laws of KANs and the structural evolution of their learnable activation functions under dataset expansion. Specifically, we evaluate the scaling behavior of three KAN variants---BSRBF-KAN, Gottlieb-KAN, and Faster-KAN---across standard image classification benchmarks (MNIST and Fashion-MNIST) and a specialized scientific regression task (magnetic parameter estimation from domain images of moiré magnetic textures). Our results demonstrate that the test loss

{\cal L}

exhibits a broken neural scaling law (BNSL) behavior as a function of the dataset size

N_D

. After passing through a random-guess regime, the loss follows architecture- and task-dependent scaling behavior. The loss crosses from a faster- to a slower-scaling branch,

{\cal L}\propto N_D^{-α}

and

{\cal L}\propto N_D^{-β}

with

α>β

for image classification tasks. The exponents

α

and

β

depend strongly on both the specific network architecture and the dataset-size regime, ranging from 0.4 to 1.5 and from 0.06 to 0.6, respectively. For the magnetic parameter-regression task, the loss follows a single scaling law with its exponent ranging from 1.28 to 2.59. Additionally, we provide a structural analysis of how activation functions refine their complexity as data volume increases, finding that dataset expansion drives a transition from simple linear-like approximations toward stable, interpretable symbolic forms. These findings provide a quantitative roadmap for the efficient application of KANs while managing the trade-off between model expressivity and computational overhead.

Momentum

Related topics

Latest papers 48