Neural Network Activation Functions

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

All topics

Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks
Oct 1, 2026Tilen Cadez, Sanghoon Lee, Kyoung-Min KimNeural Network Activation Functions Kolmogorov-Arnold Networks
Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea
Let the Neurons Die: Exploiting ReLU-Induced Model Degradation
Sep 28, 2026Kexin Li, Wenjun Qiu, Joshua Abraham +2ReLU Neural Networks Adversarial Attacks
University of Toronto, Toronto, Ontario, Canada
Activation-Flexible ANN-to-SNN Conversion with Finite-State Markov Neurons
Sep 24, 2026Ruiyu Jia, Zhuo-Cheng XiaoNeural Network Activation Functions Spiking Neural Networks
New York University Shanghai, Shanghai, China
Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery
Sep 15, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary Optimization Neural Network Activation Functions
University of Neuchâtel, Neuchâtel, Switzerland · University College Dublin, Ireland
Diversity of EML-type operators
Sep 10, 2026Andrzej OdrzywołekNeural Network Activation Functions Neural Network Expressivity
Institute of Theoretical Physics, Jagiellonian University, Łojasiewicza 11, 30-348 Kraków, Poland
Shallow neural network approximation in mixed Sobolev spaces
Sep 4, 2026Yuwen Li, Guozhi ZhangNeural Network Approximation Theory Shallow Neural Networks
School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China
InKAN: B-Spline KANs via Truncated Power Form
Sep 2, 2026Naveen MysoreEfficient Neural Network Inference Neural Network Activation Functions
Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning
Aug 13, 2026Zeyang Zhang, Tieliang Gong, Junyan Lu +1Loss of Plasticity Continual Learning
School of Computer Science and Technology, Xi’an Jiaotong University and Institute of Multimedia Knowledge Fusion and Engineering, Xi’an 710049, China
TESLA: Taylor Expansion of Sinusoidal Learnable Activations
Aug 12, 2026Daehwa Ko, Jaehyeon Kim, Seunghyun Ham +1Neural Network Generalization Neural Network Activation Functions
Korea Aerospace University, Goyang, Republic of Korea
QFCQT: A Chaotically Gated Quantformer Framework for Volatile Time-Series Forecasting
Aug 7, 2026Junkai Lin, Siqi Hou, Raymond LeeMultivariate Time Series Forecasting Time Series Forecasting
Faculty of Science and Technology, Beijing Normal-Hong Kong Baptist University, Zhuhai, China · Guangdong Provincial Key Lab of Interdisciplinary Research and Application for Data Science (BNBU), Zhuhai, China
Local Stability and Gaussian Smoothing of Quantized Neural Networks
Jul 22, 2026Sergey Salishev, Anton Makarov, Oleg GranichinNeural Network Activation Functions Neural Network Approximation Theory
St. Petersburg State University, 199034, Saint Petersburg, Russia
Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions
Jul 20, 2026Meng Hua, Itsik Bergel, Deniz GündüzNeural Network Activation Functions Wireless Communications
Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, U.K. · Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Jul 20, 2026Damien Teney, Liangze Jiang, Hemanth Saratchandran +1Neural Network Activation Functions Inductive Bias
Idiap Research Institute · Adelaide University · EPFL
LRX-PINN: A Layer-Resolving XNet Physics-Informed Neural Network with Integrated Cauchy Activations for Convection-Dominated Problems
Jul 4, 2026Zihao Guo, Xin Li, Zhihong XiaNeural PDE Solvers PDE Solving
Institute for Advanced Research, Great Bay University, Dongguan, China · College of Computer Science and Technology, Dongguan University of Technology, Dongguan, China · Department of Mathematics, Northwestern University, Evanston, USA
A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function
Jul 4, 2026Roberto RossiNeural Network Activation Functions
Business School, University of Edinburgh, Edinburgh, UK
Rethinking Neural Nonlinearity as Gating
Jul 3, 2026Muhammad Sabih, Frank Hannig, Jürgen TeichNeural Network Activation Functions Neural Network Approximation Theory
Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Germany
SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions
Jun 30, 2026Hoang-Thang TaNeural Surrogate Modeling Neural Network Activation Functions
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam
It's Much Easier for Neural Networks to learn Game of Life Dynamics with the Right Activation Function: Polynomial Kolmogorov-Arnold Networks
Jun 22, 2026Tashin Ahmed, Q. Tyrell DavisPolynomial Neural Networks Cellular Automata
Independent Researcher · Work completed while at Cross Labs, Cross Compass Ltd., Japan.
Shifting-based Optimizable Linear Relaxations for General Activation Functions
Jun 18, 2026Philipp Kern, László Antal, Erika Ábráham +1Neural Network Verification Neural Network Activation Functions
Karlsruhe Institute of Technology · RWTH Aachen University · Karlsruhe University of Applied Sciences
Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy
Jun 15, 2026Baicheng Li, Haizhao Yang, Shijun ZhangNeural Network Activation Functions Universal Approximation
Department of Mathematics, University of Maryland, College Park, MD, USA · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
Jun 14, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungNeural Representation Geometry Neural Network Optimization
College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea
SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks
Jun 1, 2026Mourad ZaiedNeural Network Activation Functions Convolutional Neural Networks
University of Gabes, Tuisia · Department of electrical engineering National Engineering School of Gabes (ENIG) University of Gabes Avenue Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia
Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations
May 27, 2026Yeachan Park, Geonho Hwang, Wonyeol Lee +1Neural Network Activation Functions Universal Approximation
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Transformer FFNs Transformer Expressivity
ByteDance Seed · Peking University
PowLU: An Activation Function for Stable Pre-Training of LLMs
May 25, 2026Peijie Jiang, Yuqi Feng, Cunyin Peng +5Language Model Pretraining Neural Network Activation Functions
Ling Team, Ant Group
ChainzRule: Sample-Efficient, Robust Deep Learning Across Tabular, NLP, and Vision Tasks
May 23, 2026Rowan MartnishnPolynomial Neural Networks Neural Network Robustness
Sentivity AI
Optimization of randomized neural networks for transfer operator approximation
May 22, 2026Mohammad Tabish, Stefan KlusDynamical Systems Neural Network Optimization
Maxwell Institute for Mathematical Sciences, University of Edinburgh and Heriot–Watt University, Edinburgh, UK · School of Mathematical & Computer Sciences, Heriot–Watt University, Edinburgh, UK
A Constant-Time Implementation Methodology for Activation Functions on Microcontrollers
May 21, 2026Andrii Tyvodar, Andreas Rechberger, Dirmanto Jap +4Edge Inference Neural Network Activation Functions
Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava, Bratislava, Slovakia · Temasek Laboratories, Nanyang Technological University, Singapore · Faculty of Computer Science, Albstadt-Sigmaringen University, Albstadt, Germany +2
Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
May 20, 2026Jeffrey Wang, Jonathan Gregory, Grigorios G. ChrysosPolynomial Neural Networks Neural Network Activation Functions
University of Wisconsin–Madison, Madison, WI, USA.
Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines
May 18, 2026Giovanni di Sarra, Yasser RoudiUnsupervised Representation Learning Neural Network Activation Functions
Kavli Institute for Systems Neuroscience, Norwegian University of Science and Technology, 7491 Trondheim, Norway · Department of Mathematics, King’s College London, London, Strand, WC2R 2LS, UK

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

Tilen Cadez, Sanghoon Lee, Kyoung-Min Kim

Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea

Neural Network Activation Functions Kolmogorov-Arnold Networks Neural Scaling Laws

Open paper AskPDF arXiv

Kolmogorov-Arnold Networks (KANs) represent a compelling alternative to traditional Multi-Layer Perceptron (MLP)-based neural networks. By employing activation functions as learnable elements, KANs offer superior interpretability, making them suited for scientific domains. In this work, we investigate the neural scaling laws of KANs and the structural evolution of their learnable activation functions under dataset expansion. Specifically, we evaluate the scaling behavior of three KAN variants---BSRBF-KAN, Gottlieb-KAN, and Faster-KAN---across standard image classification benchmarks (MNIST and Fashion-MNIST) and a specialized scientific regression task (magnetic parameter estimation from domain images of moiré magnetic textures). Our results demonstrate that the test loss

{\cal L}

exhibits a broken neural scaling law (BNSL) behavior as a function of the dataset size

N_D

. After passing through a random-guess regime, the loss follows architecture- and task-dependent scaling behavior. The loss crosses from a faster- to a slower-scaling branch,

{\cal L}\propto N_D^{-α}

and

{\cal L}\propto N_D^{-β}

with

α>β

for image classification tasks. The exponents

α

and

β

depend strongly on both the specific network architecture and the dataset-size regime, ranging from 0.4 to 1.5 and from 0.06 to 0.6, respectively. For the magnetic parameter-regression task, the loss follows a single scaling law with its exponent ranging from 1.28 to 2.59. Additionally, we provide a structural analysis of how activation functions refine their complexity as data volume increases, finding that dataset expansion drives a transition from simple linear-like approximations toward stable, interpretable symbolic forms. These findings provide a quantitative roadmap for the efficient application of KANs while managing the trade-off between model expressivity and computational overhead.

Momentum

Related topics

Latest papers 48