Activation Functions

Ask about this topic

Recent momentum

+43%

10 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Also in Neural Network

Neighbouring directions

Neural NetworkNeural ArchitecturesDeep NetworkTwo-Layer Neural NetworksRectified Linear Unit NetworksNeural Tangent KernelModern Deep NetworksRectified Linear Unit

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

4 new papers

A weekly snapshot of new work published in Activation Functions.

Period ending 2026-09-14

5 new papers

A weekly snapshot of new work published in Activation Functions.

Period ending 2026-09-07

2 new papers

A weekly snapshot of new work published in Activation Functions.

109 papers

Latest in Activation Functions

Browse all topics →
CardsList

  1. Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery

    Sep 15, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary SearchBio-Inspired Framework

  2. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

    Sep 14, 2026Xingyang Li, Dongyun Zou, Shining Zhang +8Dynamic Sparse AttentionPre-Trained Video Diffusion Models

  3. SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

    Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Dynamic Sparse AttentionQuadratic Attention

  4. Hierarchical Prototype Emergence in Modern Hopfield Models

    Sep 14, 2026Aditya Cowsik, Adithya SriramHopfield NetworksDense Associative Memory

  5. Diversity of EML-type operators

    Sep 10, 2026Andrzej OdrzywołekActivation FunctionsNeural Operators

  6. Teacher Geometry Shapes Learnability in Teacher-Student Networks

    Sep 10, 2026Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2Teacher-Student Log-Probability GapTwo-Layer Neural Networks

  7. Through the Looking Glass: Directly Reading and Writing Transformers

    Sep 9, 2026Mark OskinTransformer ArchitecturesActivation Functions

  8. LinearMask-GS: Stable-Mask Importance Pruning for Compact 3D Gaussian Splatting

    Sep 9, 2026Donghun Ryu, Minhyeok Lee3D Gaussian SplatsGaussian Splatting

  9. EFQ-Softmax: Exp-Free Quantization for Softmax

    Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Mixed-Precision QuantizationGumbel-Softmax Relaxation

  10. Wave Function Backpropagation with Explicit Temporal-Interval Dynamics

    Sep 1, 2026Byunggu Yu, Justin KimBackpropagationWave

  11. The Boolean Power of ReLU

    Aug 12, 2026Pablo Barceló, Floris Geerts, Matthias Lanzinger +2Rectified Linear UnitGraph Representations

  12. TESLA: Taylor Expansion of Sinusoidal Learnable Activations

    Aug 12, 2026Daehwa Ko, Jaehyeon Kim, Seunghyun Ham +1Activation FunctionsCoefficients

  13. Motif 3: Technical Report

    Aug 10, 2026Junghwan Lim, Joon Son Chung, Sungmin Lee +24Mixture-Of-Expert Language ModelsTechnical Report

  14. QFCQT: A Chaotically Gated Quantformer Framework for Volatile Time-Series Forecasting

    Aug 7, 2026Junkai Lin, Siqi Hou, Raymond LeeChaosStochastic Volatility

  15. When Should Graph Attention Be Sparse? Learning a Per-Edge Tsallis Index

    Aug 3, 2026Kleyton da Costa, Bernardo ModenesiHeterogeneous Graph TransformersText-Attributed Graph

  16. Mining Verdict Boundaries for Neural Network Verification

    Jul 31, 2026Jiawei Ren, Guanqin Zhang, Zhenya Zhang +1Neural Network VerificationUpper Bounds

  17. Enabling Fully Integer-Only Inference for Lightweight Detection Transformers

    Jul 27, 2026Thanh Cong Le, Michal Szczepanski, Martyna PorebaSelf-Supervised Vision TransformersRt-Detr

  18. Generalised Balanced Softmax: A Finite-Data Perspective on Logit Adjustment for Long-Tailed Recognition

    Jul 24, 2026Yi-Hang Zhu, Rajeev Raman, Shiqi Su +4Extreme Class ImbalanceBalanced Learning

  19. What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation

    Jul 24, 2026Minwoo Yu, Young-guk HaTemporal GraphsArea Under The Receiver Operating Characteristic Curve

  20. New Complexity-Theoretic Frontiers of Tractability for Neural Network Training

    Jul 23, 2026Cornelius Brand, Robert Ganian, Mathis RoctonRectified Linear Unit NetworksActivation Functions

  21. Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects

    Jul 22, 2026Seonglae Cho, Zekun Wu, Kleyton Da Costa +3Sparse Autoencoder FeaturesActivation Functions

  22. Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions

    Jul 20, 2026Meng Hua, Itsik Bergel, Deniz GündüzModern Deep NetworksMultiple-Input Multiple-Output

  23. Expressivity of Shallow Neural Networks Over Finite Fields

    Jul 19, 2026Maksym Zubkov, Carol Wu, Shiwei Yang +2Two-Layer Neural NetworksChern Numbers

  24. Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations

    Jul 15, 2026Chon-Fai Kam, Xavier Cadet, Miloud Bessafi +1GrokkingTwo-Layer Neural Networks

  25. Approximation of solutions of parameter-dependent problems by residual neural networks

    Jul 15, 2026Ana CarpioNeural-Network ApproximationNeural Network

  26. The Differential Neural Tangent Kernel and Its Positivity

    Jul 11, 2026Bangti Jin, Longjun WuNeural Tangent KernelConventional Physics-Informed Neural Networks

  27. Tubular Neighbourhoods of Pfaffian Sets and Applications to Neural Networks

    Jul 9, 2026Paul Lezeau, Martin LotzNeural NetworkNew Bounds

  28. The Key to Going Linear: Analysis-Driven Transformer Linearization

    Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiTransformer ArchitecturesCausal Attention

  29. The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System

    Jul 6, 2026Thomas HofmannGradient DescentWasserstein Gradient Flows

  30. LRX-PINN: A Layer-Resolving XNet Physics-Informed Neural Network with Integrated Cauchy Activations for Convection-Dominated Problems

    Jul 4, 2026Zihao Guo, Xin Li, Zhihong XiaConventional Physics-Informed Neural NetworksNeural Network

  31. Rethinking Neural Nonlinearity as Gating

    Jul 3, 2026Muhammad Sabih, Frank Hannig, Jürgen TeichActivation FunctionsGated Recurrent Unit

  32. Influence of Radial Basis Activation Functions on Intelligent Controller for Robotic Manipulators

    Jul 2, 2026Kimmo Paldanius, Gabriel Da Silva Lima, Wallace Moreira BessaRadial Basis FunctionRobotic Arms

  33. Low-dimensional topology of deep neural networks

    Jun 30, 2026Junyu Ren, Lek-Heng LimFeed-Forward NetworkNeural Network

  34. IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients

    Jun 29, 2026Duc Anh NguyenTransformer LayersTransformer Encoder

  35. Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models

    Jun 26, 2026Yanchen Yin, Dongqi Han, Linghui LiJailbreak AttacksLarge Language Model Safety

  36. Singular Learning and Occam's Razor in Deep Monomial Networks

    Jun 26, 2026Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir +2Activation FunctionsSingular Learning Theory

  37. DnA: Denoising Attention for Visual Tasks

    Jun 25, 2026Ron Campos, Subhajit Maity, Xin Li +2Self-Supervised Vision TransformersImage Denoising

  38. Spatial Partial Functionalization of Neural Networks based on Noise Fields

    Jun 23, 2026Shuhei Ikemoto, Fabio DallaLiberaNeural FieldsNeural Network

  39. It's Much Easier for Neural Networks to learn Game of Life Dynamics with the Right Activation Function: Polynomial Kolmogorov-Arnold Networks

    Jun 22, 2026Tashin Ahmed, Q. Tyrell DavisNeural Cellular AutomataActivation Functions

  40. Gated MLPs as Symmetry-Broken Rank-1 Bilinear Attention

    Jun 20, 2026Nathan BreslowMultilayer PerceptronsActivation Functions

  41. Shifting-based Optimizable Linear Relaxations for General Activation Functions

    Jun 18, 2026Philipp Kern, László Antal, Erika Ábráham +1Activation FunctionsSemidefinite Programming

  42. Effects of sparsity and superposition on loss in simple autoencoders

    Jun 16, 2026Mriganka Basu Roy Chowdhury, Eric McLaughlin WeinerImproving Sparse AutoencodersActivation Functions

  43. What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy

    Jun 16, 2026Truong Xuan KhanhGrokkingBatch Normalization

  44. Structural Preservation and the Logical Expressiveness of Graph Neural Networks

    Jun 16, 2026Przemysław Andrzej Wałęga, Bernardo Cuenca GrauGraph Neural NetworksModal Logic

  45. Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

    Jun 15, 2026Baicheng Li, Haizhao Yang, Shijun ZhangNeural-Network ApproximationLogarithmic Sobolev Inequalities

  46. Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm

    Jun 14, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungActivation FunctionsBatch Normalization

  47. Connections Between Pairs of Filters Improve the Accuracy of Convolutional Neural Networks

    Jun 11, 2026Kathleen Anderson, Philipp Grüning, Erhardt BarthConvolutional Neural NetworksActivation Functions

  48. Weighted universal approximation of differentiable maps on infinite-dimensional manifolds

    Jun 8, 2026Philipp Schmocker, Josef TeichmannNeural-Network ApproximationApproximation

  49. Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions

    Jun 7, 2026Blanka Köver, Alexandra Butoi, Anej Svete +2Transformer ArchitecturesActivation Functions

  50. Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

    Jun 4, 2026Yizhe Ding, Runze Li, Jia Liu +1Rectified Linear Unit NetworksActivation Functions

  51. A Negative Result on Cross-Model Activation Transfer in a Pythia Multi-Hop Setting

    Jun 2, 2026Peiyan Zhang, Jason XinModel ActivationsHidden States

  52. SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks

    Jun 1, 2026Mourad ZaiedBatch NormalizationDeep Network

  53. When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE

    May 29, 2026Melihcan Erol, Suat Evren, Oktay Ozel +3Contrastive LearningLearning-Augmented Algorithms

  54. Annealed Softmax Greedy in Many-Armed Bayesian Bandits

    May 29, 2026William Overman, Mohsen BayatiEntropy Regularized Reinforcement LearningReinforcement Learning With Verifiable Reward

  55. Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

    May 27, 2026Yeachan Park, Geonho Hwang, Wonyeol Lee +1Floating-PointActivation Functions

  56. More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

    May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Model ActivationsFeed-Forward Network

  57. The Quantization Benefits of Residual-Free Transformers

    May 25, 2026Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2Transformer Residual StreamsWeight-Only Quantization

  58. PowLU: An Activation Function for Stable Pre-Training of LLMs

    May 25, 2026Peijie Jiang, Yuqi Feng, Cunyin Peng +5Activation FunctionsModel Pretraining

  59. Measuring Alignment-Induced Activation Shifts Correctly: A Template-Controlled Difference-in-Differences Protocol

    May 23, 2026Yuki NakamuraSafety AlignmentModifications

  60. Fermi-Dirac machines as quantizations of neurons

    May 23, 2026Alexander He, Nana Liu, Mark M. WildeQuantum ComputingHamiltonian