Neural Network Activation Functions

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

May 17, 2026cs.LG

Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes

The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures (MLP, ResNet, ViT, GPT-nano, MP-SENe) and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching up to 90% in GPT-nano. We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above

\sim

70% activation sparsity. While ReLU

^2

achieves a good sparsity--accuracy ratio in GPT-nano, it pathologically amplifies identified activation spikes in intermediate transformer layers. Clipping resolves this while preserving the representational benefits of squaring: clipped ReLU

^2

outperforms its unclipped version, and GELU

^2

achieves the lowest validation loss on GPT-nano. Code is available at https://github.com/On-Point-RND/BugOrFeature.

Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3

ReLU Neural Networks Activation Sparsity Activation Outliers Neural Network Activation Functions Neural Network Training Dynamics

May 14, 2026cs.CV

ArcGate: Adaptive Arctangent Gated Activation

Activation functions are central to deep networks, influencing non-linearity, feature learning, convergence, and robustness. This paper proposes the Adaptive Arctangent Gated Activation (ArcGate) function, a flexible formulation that generates a broad spectrum of activation shapes via a three-stage non-linear transformation. Unlike conventional fixed-shape activations such as ReLU, GELU, or SiLU, ArcGate uses seven learnable parameters per layer, allowing the neural network to autonomously optimize its non-linearity to the specific requirements of the feature hierarchy and data distribution. We evaluate ArcGate using ResNet-50 and Vision Transformer (ViT-B/16) architectures on three widely used remote sensing benchmarks: PatternNet, UC Merced Land Use, and the 13-band EuroSAT MSI multispectral dataset. Experimental results show that ArcGate consistently outperforms standard baselines, achieving a peak overall accuracy of 99.67% on PatternNet. Most notably, ArcGate exhibits superior structural resilience in noisy environments, maintaining a 26.65% performance lead over ReLU under moderate Gaussian noise (standard deviation 0.1). Analysis of the learned parameters reveals a depth-dependent functional evolution, where the model increases gating strength in deeper layers to enhance signal propagation. These findings suggest that ArcGate is a robust and adaptive general node activation function for high-resolution earth observation tasks.

Avik Bhattacharya, Siddhant Dnyanesh Gole, Subhasis Chaudhuri +2

Microwave Remote Sensing Lab Center of Studies in Resources Engineering Indian Institute of Technology Bombay, India · Centre of Machine Intelligence and Data Science Indian Institute of Technology Bombay, India · Department of Electrical Engineering Indian Institute of Technology Bombay, India +1

Neural Network Activation Functions Noise Robustness

May 12, 2026cs.LG

A Composite Activation Function for Learning Stable Binary Representations

Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models.

Seokhun Park, Choeun Kim, Kwanho Lee +3

Department of Statistics, Seoul National University · Department of Applied Mathematics, University of Twente

Concept Bottleneck Models Binary Neural Networks Neural Network Activation Functions Neural Network Interpretability Activation Quantization

May 7, 2026cs.LG

Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks

Deep networks with continuous piecewise affine activations induce polyhedral partitions of the input space, making the number of realized affine regions a natural measure of expressive capacity and a key determinant of how well the model can approximate nonlinear target functions. In practice, standard training realizes far fewer region refinements in data-visited neighborhoods than the architecture could in principle support, while existing region-count theory is primarily architectural and offers little guidance on how optimization shapes the realized partition near the data. Our theory provides a sufficient condition under which bringing neuron switching surfaces sufficiently close to data points ensures their intersection with local neighborhoods, which in turn implies a strict increase in the local affine-region count, yielding a principled training-time handle for seeding data-relevant partitions early in optimization. Guided by these results, we propose a plug-and-play region-seeding regularizer that encourages early partitioning while allowing task-driven refinement to dominate later in training. Experiments show that the regularizer increases the number of realized affine regions via exact enumeration and improves overall performance on toy datasets, while also improving early-stage accuracy and achieving comparable (or slightly improved) final accuracy on ImageNet-1k for classical models.

Yi Wei, Xuan Qi, Furao Shen

DITEN University of Genoa, Genoa, Italy · State Key Laboratory of Novel Software Technology School of Artificial Intelligence Nanjing University, Jiangsu, China

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Training Dynamics Neural Network Expressivity

May 5, 2026physics.optics

OptiLookUp: An Optical ROM-Based Lookup Table Engine for Photonic Accelerators

Read-only memory (ROM) provides deterministic access to predefined data mappings. Extending ROM concepts to the optical domain enables high-bandwidth, low-latency, and parallel memory access, but realizing compact and reconfigurable optical ROM remains challenging due to loss, wavelength control, and integration constraints. This work presents a high-speed, reconfigurable photonic ROM architecture implemented using integrated microring resonators (MRRs). The ROM encodes predefined input-output mappings directly in the spectral response of the photonic devices, enabling deterministic lookup-based operation without dynamic computation during readout. To improve scalability and reduce cumulative insertion loss, the architecture employs compact banked sub-arrays that are selectively addressed through an optical decoding mechanism. Reconfigurability is achieved using transistor-based optical selectors, allowing different ROM banks to be activated without physical light rerouting or interferometric structures. The proposed photonic ROM is designed and evaluated using device-level simulations based on the GlobalFoundries 45SPCLO silicon photonics platform. Simulation results demonstrate reliable operation at data rates up to 12.5 GHz, with stable light-to-current transfer characteristics obtained through integrated photodiode readout. The optical ROM can be used to implement nonlinear activation functions utilised in photonic accelerator architectures, including sigmoid, tanh, ReLU, and exponential mappings.

Ankur Singh, Akhilesh Jaiswal

Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA

Neural Network Activation Functions Inference Acceleration Photonic Computing

May 4, 2026cs.AI

Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions

The efficacy of deep neural networks is heavily reliant on the design of non-linear activation functions, yet existing approaches often struggle to balance optimization stability with computational efficiency. While piecewise linear functions offer inference speed, they suffer from optimization instability due to non-differentiability at the origin, whereas smooth counterparts typically incur significant computational overhead through their reliance on transcendental operations. To address these limitations, this paper proposes a general smoothing framework based on constructive approximation theory and introduces the Bernstein Linear Unit (BerLU). This novel activation function utilizes Bernstein polynomials to construct a differentiable quadratic transition region that effectively eliminates singularities while maintaining a piecewise linear structure. Theoretical analysis demonstrates that the proposed method guarantees strictly continuous differentiability and a non-expansive Lipschitz constant of one, which ensures stable gradient propagation and prevents the gradient explosion problems common in deep architectures. Comprehensive empirical evaluations across representative Vision Transformer and Convolutional Neural Network architectures confirm that this approach consistently outperforms state-of-the-art baselines on standard image classification benchmarks while delivering superior computational and memory efficiency.

Wentao Zhang, Yutong Zhang, Yifan Zhu +1

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · College of Computer Science, Sichuan University, Chengdu, China

Efficient Neural Network Inference Neural Network Optimization Neural Network Activation Functions

May 4, 2026cs.LG

KANs need curvature: penalties for compositional smoothness

Kolmogorov-Arnold networks (KANs) offer a potent combination of accuracy and interpretability, thanks to their compositions of learnable univariate activation functions. However, the activations of well-fitting KANs tend to exhibit pathologically high-curvature oscillations, making them difficult to interpret, and standard regularization penalties do not prevent this. Here we derive a basis-agnostic curvature penalty and show that penalized models can maintain accuracy while achieving substantially smoother activations. Accounting for how function composition shapes curvature, we prove an upper bound on the full model's curvature relative to the curvature penalty, and use this to motivate richer forms of penalties. Scientific machine learning is increasingly bottlenecked by the trade-off between accuracy and interpretability. Results such as ours that improve interpretability without sacrificing accuracy will further strengthen KANs as a practical tool for both prediction and insight.

James Bagrow

Mathematics & Statistics, University of Vermont, Burlington, VT, United States · Vermont Complex Systems Center, University of Vermont, Burlington, VT, United States

Neural Network Activation Functions Neural Network Interpretability Kolmogorov-Arnold Networks

Apr 25, 2026cs.LG

When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer

Dynamic Tanh (DyT) removes LayerNorm by bounding activations with a learned tanh(alpha x). We show that this bounding is a regime-dependent implicit regularizer, not a uniformly beneficial replacement. Across GPT-2-family models spanning 64M to 3.78B parameters and 1M to 118M tokens, with Llama and ViT cross-checks, DyT improves validation loss by 27.3% at 64M/1M but worsens it by 18.8% at 64M/118M; the 1M benefit vanishes with capacity (+1.7% at 3.78B), while the 118M penalty reaches +27.9%. The mechanism is measurable: 49% of DyT activations saturate at 1M versus 23% at 118M, and a 500-step saturation heuristic classifies DyT's sign with 75% raw in-sample accuracy on the 12-cell GPT-2 calibration set (AUC 0.75; 64% when adding Scale 5 stress cells), correctly labels 3/3 Llama checks, but only reaches 50% raw leave-one-scale-out accuracy. Three interventions support the bounding explanation: HardTanh reproduces the regime pattern, increasing alpha at 118M monotonically reduces DyT's penalty, and vanilla+dropout(p=0.5) matches DyT's data-rich loss. We also localize Llama-DyT collapse to SwiGLU gating, where saturation separates collapse from convergence in a 3-seed component ablation (r=0.94). Scope: all experiments are compute-limited (T/P < 1.84), below Chinchilla-optimal training.

Lucky Verma

Independent Researcher

Neural Network Activation Functions Layer Normalization Neural Network Training Dynamics Implicit Regularization

Apr 23, 2026cs.LG

Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions

The choice of activation function plays a crucial role in the optimization and performance of deep neural networks. While the Rectified Linear Unit (ReLU) remains the dominant choice due to its simplicity and effectiveness, its lack of smoothness may hinder gradient-based optimization in deep architectures. In this work we propose a family of

C^{2N}

-smooth activation functions whose gate follows a log-logistic CDF, achieving ReLU-like performance with purely rational arithmetic. We introduce three variants: GEM (the base family), E-GEM (an

ε

-parameterized generalization enabling arbitrary

L^p

-approximation of ReLU), and SE-GEM (a piecewise variant eliminating dead neurons with

C^{2N}

junction smoothness). An

N

-ablation study establishes

N=1

as optimal for standard-depth networks, reducing the GELU deficit on CIFAR-100 + ResNet-56 from 6.10% to 2.12%. The smoothness parameter

N

further reveals a CNN-transformer tradeoff:

N=1

is preferred for deep CNNs, while

N=2

is preferred for transformers. On MNIST, E-GEM ties the best baseline (99.23%). On CIFAR-10 + ResNet-56, SE-GEM (

ε=10^{-4}

) surpasses GELU (92.51% vs 92.44%) -- the first GEM-family activation to outperform GELU. On CIFAR-100 + ResNet-56, E-GEM reduces the GELU deficit from 6.10% (GEM

N=2

) to just 0.62%. On GPT-2 (124M), GEM achieves the lowest perplexity (72.57 vs 73.76 for GELU), with GEM

N=1

also beating GELU (73.32). On BERT-small, E-GEM (

ε=10

) achieves the best validation loss (6.656) across all activations. The

ε

-parameterization reveals a scale-dependent optimum: small

ε

(

10^{-4}

--

10^{-6}

) for deep CNNs and larger transformers, with the special case of small transformers (BERT-small) benefiting from large

ε

(

ε=10

) due to its limited depth and unconstrained gradients.

Eylon E. Krause

Weizmann Institute of Science1, Rehovot, Israel (7610001) · College of Management Academic Studies2, Rishon LeZion, Israel (7579806)

Neural Network Optimization Neural Network Activation Functions

Feb 12, 2026cs.LG

Rational Neural Networks have Expressivity Advantages

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of

\varepsilon>0

, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only

\mathrm{poly}(\log\log(1/\varepsilon))

overhead in size, while the converse provably requires

Ω(\log(1/\varepsilon))

parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Maosen Tang, Alex Townsend

Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity

Jan 10, 2026cs.LG

FlexAct: Why Learn when you can Pick?

Learning activation functions has emerged as a promising direction in deep learning, allowing networks to adapt activation mechanisms to task-specific demands. In this work, we introduce a novel framework that employs the Gumbel-Softmax trick to enable discrete yet differentiable selection among a predefined set of activation functions during training. Our method dynamically learns the optimal activation function independently of the input, thereby enhancing both predictive accuracy and architectural flexibility. Experiments on synthetic datasets show that our model consistently selects the most suitable activation function, underscoring its effectiveness. These results connect theoretical advances with practical utility, paving the way for more adaptive and modular neural architectures in complex learning scenarios.

Ramnath Kumar, Kyle Ritscher, Junmin Judy +2

University of California Los Angeles. · University of California Los Angeles, Arena.

Neural Network Activation Functions

Oct 29, 2025math.OC

Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k>= 4, to detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter

α

varies. Typically, our analysis reveals that a multi-mode degeneracy consistently occurs at the critical number 0, independent of k. (2) As a by-product, we further show that such bifurcations are width-independent, arise only for nonnegative

α

and that the global minimum undergoes no further symmetry-breaking instability throughout the engineering regime

α

in range (0,1). An explicit example with k=5 is presented to illustrate the framework and exhibit the resulting bifurcation together with their symmetries.

Jingzhou Liu

Shallow Neural Networks Dynamical Systems Neural Network Optimization Neural Network Activation Functions Neural Network Training Dynamics

Oct 25, 2025cs.LG

SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks

The choice of activation function plays a critical role in neural networks, yet most architectures still rely on fixed, uniform activation functions across all neurons. We introduce SmartMixed, a novel two-phase training strategy that allows networks to learn optimal per-neuron activation functions while preserving computational efficiency at inference. In the first phase, neurons adaptively select from a pool of candidate activation functions (ReLU, Sigmoid, Tanh, Leaky_ReLU, ELU, SELU) using a differentiable hard mixture mechanism. In the second phase, each neuron's activation function is fixed according to the learned selection, resulting in a computationally efficient network that supports continued training with optimized vectorized operations. We evaluate SmartMixed on the MNIST dataset using feedforward neural networks of different architectures. Our analysis reveals that neurons in different layers exhibit distinct preferences for activation functions, providing insights into the functional diversity within neural architectures. We also demonstrated that SmartMixed effectively trains the network by allowing neurons to select their preferred activation functions, competing against models using a single fixed state-of-the-art activation function.

Amin Omidvar

Independent Researcher Toronto, Ontario, Canada

Neural Network Optimization Neural Network Activation Functions

Sep 17, 2025eess.SY

Circuit realization and hardware linearization of monotone operator equilibrium networks

It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural network in analog hardware. We furthermore show that the gradient of such a circuit can be computed directly in hardware, using a procedure we call hardware linearization. This allows the network to be trained in hardware, which we demonstrate with a device-level circuit simulation. We extend the results to cascades of resistor-diode networks, which can be used to implement feedforward and other asymmetric networks. We finally show that different nonlinear elements give rise to different activation functions, and introduce the novel diode ReLU which is induced by a non-ideal diode model.

Thomas Chaffey

School of Electrical and Computer Engineering, University of Sydney, New South Wales, Australia

ReLU Neural Networks Neuromorphic Computing Neural Network Activation Functions

Jul 11, 2025cs.LG

Understanding Two-Layer Neural Networks with Smooth Activation Functions

This paper aims to understand the training solution, which is obtained by the back-propagation algorithm, of two-layer neural networks whose hidden layer is composed of the units with smooth activation functions, including the usual sigmoid type most commonly used before the advent of ReLUs. The mechanism contains four main principles: construction of Taylor series expansions, strict partial order of knots, smooth-spline implementation and smooth-continuity restriction. The universal approximation for arbitrary input dimensionality is proved and the explanation of training solutions is given. Through the principles proposed, the mystery of ``black box'' of the solution space is largely revealed. The new proofs employed also enrich approximation theory.

Changcun Huang

Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Training Dynamics Neural Network Expressivity

May 29, 2025cs.LG

SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

Prevailing activation functions such as Swish and GELU tend toward domain-specific optima, Swish was discovered via neural architecture search on vision benchmarks, while GELU dominates transformer-based language models, and neither offers any mechanism to adapt its gating shape to individual layers. This rigidity is especially consequential in transformer FFN blocks, where LayerNorm, unlike BatchNorm, does not suppress the gradient pathologies that activation choice induces across depth. We propose SG-Blend, a per layer adaptive activation that combines SSwish, a bias-corrected, parametric Swish variant we also introduce, with learnable sharpness \b{eta} and zero-centering bias γ, with GELU through a per-layer blend coefficient α, letting each layer locate its own optimum along the SSwishGELU continuum at a cost of only three additional scalars per FFN block, with \b{eta} initialized to 1.0 and learned freely via backpropagation. On BERT-style IMDB classification (5 seeds), it matches peak accuracy (81.31%) while reducing seed-to-seed variance by 42% relative to GELU. Furthermore, it generalizes to autoregressive pretraining, achieving the lowest validation perplexity (49.10) on WikiText103 among all baselines. Crucially, ablations confirm the interpolation structure itself drives these gains, delivering reliable, top-tier performance. Beyond natural language processing, we demonstrate that SG-Blend generalizes robustly to a wider variety of tasks, extending its efficacy to computer vision and other diverse domains.

Gaurav Sarkar, Syed Affan Daimi, Jay Gala +1

IIIT · Microsoft · Intel

Transformer FFNs Transformer Neural Network Activation Functions

Feb 26, 2025cs.LG

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

The architecture of a neural network and the choice of its activation function are both fundamental to its performance. Equally important is ensuring that these two elements are well matched, as their alignment is key to effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a model that combines sine-type activations with the multi-component and multi-layer structure of MMNNs. In an FMMNN, each component is represented as a trainable linear combination of fixed random sine-type basis functions, while multi-layer composition generates more complex and adaptive high-frequency features. We establish that FMMNNs retain exponential expressive power for function approximation even under a low-rank architectural structure. We also analyze the optimization landscape of FMMNNs and find it to be substantially more favorable than that of standard fully connected neural networks, especially for high-frequency targets. In addition, we propose a scaled random initialization method for the first-layer weights in FMMNNs, which accelerates training and improves final performance when sufficient samples are available. Extensive numerical experiments support our theoretical insights, showing that FMMNNs achieve strong accuracy and favorable convergence behavior on oscillatory function-approximation benchmarks.

Shijun Zhang, Hongkai Zhao, Yimin Zhong +1

Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1

Fourier Feature Embeddings Neural Network Optimization Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity Modular Neural Networks

Jul 13, 2023cs.LG

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set

\mathscr{A}

is defined to encompass the majority of commonly used activation functions, such as

\mathtt{ReLU}

,

\mathtt{LeakyReLU}

,

\mathtt{ReLU}^2

,

\mathtt{ELU}

,

\mathtt{CELU}

,

\mathtt{SELU}

,

\mathtt{Softplus}

,

\mathtt{GELU}

,

\mathtt{SiLU}

,

\mathtt{Swish}

,

\mathtt{Mish}

,

\mathtt{Sigmoid}

,

\mathtt{Tanh}

,

\mathtt{Arctan}

,

\mathtt{Softsign}

,

\mathtt{dSiLU}

, and

\mathtt{SRS}

. We demonstrate that for any activation function

\varrho\in \mathscr{A}

, a

\mathtt{ReLU}

network of width

N

and depth

L

can be approximated to arbitrary precision by a

\varrho

-activated network of width

3N

and depth

2L

on any bounded set. This finding enables the extension of most approximation results achieved with

\mathtt{ReLU}

networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,

\,

depth) scaling factors can be further reduced from

(3,2)

to

(1,1)

if

\varrho

falls within a specific subset of

\mathscr{A}

. This subset includes activation functions such as

\mathtt{ELU}

,

\mathtt{CELU}

,

\mathtt{SELU}

,

\mathtt{Softplus}

,

\mathtt{GELU}

,

\mathtt{SiLU}

,

\mathtt{Swish}

, and

\mathtt{Mish}

.

Shijun Zhang, Jianfeng Lu, Hongkai Zhao

Department of Mathematics Duke University

ReLU Neural Networks Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity