IGLU: The Integrated Gaussian Linear Unit Activation Function

Mingi Kang; Zai Yang; Jeova Farias Sales Rocha Neto

cs.LGMar 6, 2026

IGLU: The Integrated Gaussian Linear Unit Activation Function

Authors: Mingi Kang, Zai Yang, Jeova Farias Sales Rocha Neto

Organizations: Bowdoin College, 255 Maine Street, Brunswick, ME 04011, USA

Abstract

Activation functions are fundamental to deep neural networks, governing gradient flow, optimization stability, and representational capacity. Within historic deep architectures, while ReLU has been the dominant choice for the activation function, modern transformer-based models increasingly are adopting smoother alternatives such as GELU and other self-gated alternatives. Despite their empirical success, the mathematical relationships among these functions and the principles underlying their effectiveness remains only partially understood. We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution. This derivation yields a closed-form expression whose gating component is exactly the Cauchy CDF, providing a principled one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter $σ$ . Unlike GELU's Gaussian gate, IGLU's heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients. We further introduce IGLU-Approx, a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation. Through evaluations on CIFAR-10, CIFAR-100, and WikiText-103 across ResNet-20, ViT-Tiny, and GPT-2 Small, IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost. In particular, we show that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification datasets.

Figures & tables

Panel of Figure 1 — Figure 1: Relevant activation functions in this work. (Left) Comparison of standard activation functions including our proposed IGLU ( $\sigma=1$ ). (Right) Parameterized $\textsc{GELU}_{a}$ for various $a$ values showing convergence to ReLU as $a\to\infty$ .

	CPU		GPU (CUDA)
Activation	Forward	Backward	Forward	Backward
$Z(\cdot)$ ( $\sigma=1$ )	21.95 $\times$	3.43 $\times$	14.77 $\times$	2.80 $\times$
$Z_{\text{approx}}(\cdot)$ ( $\sigma=1$ )	10.13 $\times$	3.61 $\times$	15.56 $\times$	3.14 $\times$
IGLU ( $\sigma=1$ )	22.44 $\times$	4.39 $\times$	14.89 $\times$	2.68 $\times$
IGLU-Approx ( $\sigma=1$ )	10.17 $\times$	3.63 $\times$	15.05 $\times$	2.23 $\times$
ReLU [ 1 ]	9.22 $\times$	3.39 $\times$	14.85 $\times$	2.66 $\times$

Table 1: Computational efficiency benchmarks for various activation functions. Forward and backward pass times are normalized relatively to the Identity function speed. Results are averaged over 1,000 iterations using a 10,000-dimensional input. Gray shading denotes self-gating activations of the form

f(x)=x\cdot g(x)

. Bold and underline indicate the best and second-best performance within the self-gating category, respectively.

	ResNet-20				ViT-Tiny
	CIFAR-10		CIFAR-100		CIFAR-10		CIFAR-100
Activation	Test Loss	Acc	Test Loss	Acc	Test Loss	Acc	Test Loss	Acc
IGLU ( $\sigma=0.1$ )	0.389	88.59	1.377	62.51	1.251	82.81	3.769	53.60
IGLU ( $\sigma=0.5$ )	0.480	91.10	1.488	65.46	1.211	85.23	3.585	57.55
IGLU ( $\sigma=1$ )	0.522	90.94	1.560	64.64	1.088	86.65	3.240	61.83
IGLU ( $\sigma=5$ )	0.509	90.71	1.601	63.75	1.099	86.70	3.193	61.74

Table 2: Test loss and test accuracy of ResNet-20 and ViT-Tiny trained on CIFAR-10 and CIFAR-100. Bold and underline indicate the best and second-best performance, respectively. ViT-Tiny experiments undergo a resizing of size

224\times 224

to fit the Vision Transformer training procedure.

Activation	Test Loss	Test Perplexity
IGLU ( $\sigma=0.1$ )	2.989	16.69
IGLU ( $\sigma=0.5$ )	2.814	16.69
IGLU ( $\sigma=1$ )	2.761	15.82
IGLU ( $\sigma=5$ )	2.754	15.71
IGLU ( $\sigma=10$ )	2.755	15.72
IGLU (learnable $\sigma$ )	2.754	15.71

Table 3: Test loss and test perplexity (the lower the better) of GPT2-Small trained on WikiText-103. Bold and underline indicate the best and second-best performance, respectively.

	10:1		20:1		50:1		100:1		500:1
Activation	Loss	Acc	Loss	Acc	Loss	Acc	Loss	Acc	Loss	Acc
IGLU ( $\sigma=0.1$ )	2.078	46.69	2.456	38.60	2.982	29.43	3.426	23.05	3.957	17.99
IGLU ( $\sigma=0.5$ )	2.719	49.99	3.309	44.97	3.923	37.76	4.878	31.45	5.659	28.94
IGLU ( $\sigma=1$ )	2.833	50.02	3.514	43.95	4.482	36.24	5.352	31.64	6.192	29.03
IGLU ( $\sigma=5$ )	2.931	49.21	3.675	43.63	4.756	35.27	5.456	30.15	6.182	28.80
IGLU ( $\sigma=10$ )	2.878	49.48	3.570	43.49	4.651	35.48	5.350	30.89	6.014	28.37

Table 4: Performance of ResNet-20 on the Imbalanced CIFAR-100 dataset across various imbalance ratios (where, for example,

100{:}1

indicates

n_{\max}/n_{\min}=100

)). We report the cross-entropy test loss (Loss) and test accuracy (Acc) for our proposed and baseline activation functions. Bold and underline indicate the best and second-best performance, respectively.

Model	Norm	$\alpha$	Optimal $\sigma$
ResNet-20	BatchNorm	10.4	0.1
GPT2-Small	LayerNorm	20.7	5.0
ViT-Tiny	LayerNorm	23.2	5.0
ResNet-20	-	11.28	0.1

Table 5: Empirical Index Index

\alpha

of pre-activation measurements using the Hill Estimator with and without Normalization (Batch Normalization for ResNet, Layer Normalization for ViT and GPT-2) using IGLU activation function.

Neural Network Activation Functions

Neural network activation functions shape how models represent nonlinear patterns, learn, remain stable, and map efficiently to hardware.

See all 78 papers

Mar 23, 2026

$λ$ -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks

Gaussian Error Linear Unit (GELU) is a widely used smooth alternative to Rectifier Linear Unit (ReLU), yet many deployment, compression, and analysis toolchains are most naturally expressed for piecewise-linear (ReLU-type) networks. We study a hardness-parameterized formulation of GELU, f(x;λ)=xΦ(λ x), where Φ is the Gaussian CDF and λ \in [1, infty) controls gate sharpness, with the goal of turning smooth gated training into a controlled path toward ReLU-compatible models. Learning λ is non-trivial: naive updates yield unstable dynamics and effective gradient attenuation, so we introduce a constrained reparameterization and an optimizer-aware update scheme. Empirically, across a diverse set of model--dataset pairs spanning MLPs, CNNs, and Transformers, we observe structured layerwise hardness profiles and assess their robustness under different initializations. We further study a deterministic ReLU-ization strategy in which the learned gates are progressively hardened toward a principled target, enabling a post-training substitution of λ-GELU by ReLU with reduced disruption. Overall, λ-GELU provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with ReLU-centric downstream pipelines.

Cristian Pérez-Corral, Alberto Fernández-Hernández, Jose I. Mestre +2

Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain

Neural Network Activation Functions ReLU Neural Networks

Ask PDF

Apr 23, 2026

Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions

The choice of activation function plays a crucial role in the optimization and performance of deep neural networks. While the Rectified Linear Unit (ReLU) remains the dominant choice due to its simplicity and effectiveness, its lack of smoothness may hinder gradient-based optimization in deep architectures. In this work we propose a family of

C^{2N}

-smooth activation functions whose gate follows a log-logistic CDF, achieving ReLU-like performance with purely rational arithmetic. We introduce three variants: GEM (the base family), E-GEM (an

ε

-parameterized generalization enabling arbitrary

L^p

-approximation of ReLU), and SE-GEM (a piecewise variant eliminating dead neurons with

C^{2N}

junction smoothness). An

N

-ablation study establishes

N=1

as optimal for standard-depth networks, reducing the GELU deficit on CIFAR-100 + ResNet-56 from 6.10% to 2.12%. The smoothness parameter

N

further reveals a CNN-transformer tradeoff:

N=1

is preferred for deep CNNs, while

N=2

is preferred for transformers. On MNIST, E-GEM ties the best baseline (99.23%). On CIFAR-10 + ResNet-56, SE-GEM (

ε=10^{-4}

) surpasses GELU (92.51% vs 92.44%) -- the first GEM-family activation to outperform GELU. On CIFAR-100 + ResNet-56, E-GEM reduces the GELU deficit from 6.10% (GEM

N=2

) to just 0.62%. On GPT-2 (124M), GEM achieves the lowest perplexity (72.57 vs 73.76 for GELU), with GEM

N=1

also beating GELU (73.32). On BERT-small, E-GEM (

ε=10

) achieves the best validation loss (6.656) across all activations. The

ε

-parameterization reveals a scale-dependent optimum: small

ε

(

10^{-4}

10^{-6}

) for deep CNNs and larger transformers, with the special case of small transformers (BERT-small) benefiting from large

ε

(

ε=10

) due to its limited depth and unconstrained gradients.

Eylon E. Krause

Weizmann Institute of Science1, Rehovot, Israel (7610001) · College of Management Academic Studies2, Rishon LeZion, Israel (7579806)

Neural Network Optimization Neural Network Activation Functions

Ask PDF

Jul 4, 2026

A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent Bernoulli gate. This work gives an alternative interpretation: GELU is the expected output of a hard linear gate with a Gaussian random threshold. This view provides a generative interpretation for the Bernoulli gate: the gate opens once the input clears a latent Gaussian threshold. This interpretation stems from a decomposition based on well-known results in stochastic inventory theory and leads to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases. By considering a latent uniform threshold, we recover a hard-swish-like piecewise-polynomial gate whose nonlinear transition is confined to a finite interval, yielding fixed- and learned-width variants. Controlled experiments on compact vision and language models show that calibrated or learned uniform-threshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, improve over them in most tested settings, and use the finite transition region nontrivially.

Roberto Rossi

Business School, University of Edinburgh, Edinburgh, UK

Neural Network Activation Functions

Ask PDF

Abstract

Neural Network Activation Functions

Explore similar work

λλλ-GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks

Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions

A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

$λ$ -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks