IGLU: The Integrated Gaussian Linear Unit Activation Function

Mingi Kang; Zai Yang; Jeova Farias Sales Rocha Neto

cs.LGMar 6, 2026

IGLU: The Integrated Gaussian Linear Unit Activation Function

Authors: Mingi Kang, Zai Yang, Jeova Farias Sales Rocha Neto

Organizations: Bowdoin College, 255 Maine Street, Brunswick, ME 04011, USA

Abstract

Activation functions are fundamental to deep neural networks, governing gradient flow, optimization stability, and representational capacity. Within historic deep architectures, while ReLU has been the dominant choice for the activation function, modern transformer-based models increasingly are adopting smoother alternatives such as GELU and other self-gated alternatives. Despite their empirical success, the mathematical relationships among these functions and the principles underlying their effectiveness remains only partially understood. We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution. This derivation yields a closed-form expression whose gating component is exactly the Cauchy CDF, providing a principled one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter $σ$ . Unlike GELU's Gaussian gate, IGLU's heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients. We further introduce IGLU-Approx, a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation. Through evaluations on CIFAR-10, CIFAR-100, and WikiText-103 across ResNet-20, ViT-Tiny, and GPT-2 Small, IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost. In particular, we show that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification datasets.

Figures & tables

Panel of Figure 1 — Figure 1: Relevant activation functions in this work. (Left) Comparison of standard activation functions including our proposed IGLU ( $\sigma=1$ ). (Right) Parameterized $\textsc{GELU}_{a}$ for various $a$ values showing convergence to ReLU as $a\to\infty$ .

	CPU		GPU (CUDA)
Activation	Forward	Backward	Forward	Backward
$Z(\cdot)$ ( $\sigma=1$ )	21.95 $\times$	3.43 $\times$	14.77 $\times$	2.80 $\times$
$Z_{\text{approx}}(\cdot)$ ( $\sigma=1$ )	10.13 $\times$	3.61 $\times$	15.56 $\times$	3.14 $\times$
IGLU ( $\sigma=1$ )	22.44 $\times$	4.39 $\times$	14.89 $\times$	2.68 $\times$
IGLU-Approx ( $\sigma=1$ )	10.17 $\times$	3.63 $\times$	15.05 $\times$	2.23 $\times$
ReLU [ 1 ]	9.22 $\times$	3.39 $\times$	14.85 $\times$	2.66 $\times$

Table 1: Computational efficiency benchmarks for various activation functions. Forward and backward pass times are normalized relatively to the Identity function speed. Results are averaged over 1,000 iterations using a 10,000-dimensional input. Gray shading denotes self-gating activations of the form

f(x)=x\cdot g(x)

. Bold and underline indicate the best and second-best performance within the self-gating category, respectively.

	ResNet-20				ViT-Tiny
	CIFAR-10		CIFAR-100		CIFAR-10		CIFAR-100
Activation	Test Loss	Acc	Test Loss	Acc	Test Loss	Acc	Test Loss	Acc
IGLU ( $\sigma=0.1$ )	0.389	88.59	1.377	62.51	1.251	82.81	3.769	53.60
IGLU ( $\sigma=0.5$ )	0.480	91.10	1.488	65.46	1.211	85.23	3.585	57.55
IGLU ( $\sigma=1$ )	0.522	90.94	1.560	64.64	1.088	86.65	3.240	61.83
IGLU ( $\sigma=5$ )	0.509	90.71	1.601	63.75	1.099	86.70	3.193	61.74

Table 2: Test loss and test accuracy of ResNet-20 and ViT-Tiny trained on CIFAR-10 and CIFAR-100. Bold and underline indicate the best and second-best performance, respectively. ViT-Tiny experiments undergo a resizing of size

224\times 224

to fit the Vision Transformer training procedure.

Activation	Test Loss	Test Perplexity
IGLU ( $\sigma=0.1$ )	2.989	16.69
IGLU ( $\sigma=0.5$ )	2.814	16.69
IGLU ( $\sigma=1$ )	2.761	15.82
IGLU ( $\sigma=5$ )	2.754	15.71
IGLU ( $\sigma=10$ )	2.755	15.72
IGLU (learnable $\sigma$ )	2.754	15.71

Table 3: Test loss and test perplexity (the lower the better) of GPT2-Small trained on WikiText-103. Bold and underline indicate the best and second-best performance, respectively.

	10:1		20:1		50:1		100:1		500:1
Activation	Loss	Acc	Loss	Acc	Loss	Acc	Loss	Acc	Loss	Acc
IGLU ( $\sigma=0.1$ )	2.078	46.69	2.456	38.60	2.982	29.43	3.426	23.05	3.957	17.99
IGLU ( $\sigma=0.5$ )	2.719	49.99	3.309	44.97	3.923	37.76	4.878	31.45	5.659	28.94
IGLU ( $\sigma=1$ )	2.833	50.02	3.514	43.95	4.482	36.24	5.352	31.64	6.192	29.03
IGLU ( $\sigma=5$ )	2.931	49.21	3.675	43.63	4.756	35.27	5.456	30.15	6.182	28.80
IGLU ( $\sigma=10$ )	2.878	49.48	3.570	43.49	4.651	35.48	5.350	30.89	6.014	28.37

Table 4: Performance of ResNet-20 on the Imbalanced CIFAR-100 dataset across various imbalance ratios (where, for example,

100{:}1

indicates

n_{\max}/n_{\min}=100

)). We report the cross-entropy test loss (Loss) and test accuracy (Acc) for our proposed and baseline activation functions. Bold and underline indicate the best and second-best performance, respectively.

Model	Norm	$\alpha$	Optimal $\sigma$
ResNet-20	BatchNorm	10.4	0.1
GPT2-Small	LayerNorm	20.7	5.0
ViT-Tiny	LayerNorm	23.2	5.0
ResNet-20	-	11.28	0.1

Table 5: Empirical Index Index

\alpha

of pre-activation measurements using the Hill Estimator with and without Normalization (Batch Normalization for ResNet, Layer Normalization for ViT and GPT-2) using IGLU activation function.

Neural Network Activation Functions

Neural network activation functions shape how models represent nonlinear patterns, learn, remain stable, and map efficiently to hardware.

See all 78 papers

$λ$ -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks
Mar 23, 2026Cristian Pérez-Corral, Alberto Fernández-Hernández, Jose I. Mestre +2Neural Network Activation Functions ReLU Neural Networks
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain
Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions
Apr 23, 2026Eylon E. KrauseNeural Network Optimization Neural Network Activation Functions
Weizmann Institute of Science1, Rehovot, Israel (7610001) · College of Management Academic Studies2, Rishon LeZion, Israel (7579806)
A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function
Jul 4, 2026Roberto RossiNeural Network Activation Functions
Business School, University of Edinburgh, Edinburgh, UK

Mar 23, 2026J/K move · Enter open · S save

$λ$ -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks

Cristian Pérez-Corral, Alberto Fernández-Hernández, Jose I. Mestre, Manuel F. Dolz+1

Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain

Neural Network Activation Functions ReLU Neural Networks

Open paper AskPDF arXiv

Gaussian Error Linear Unit (GELU) is a widely used smooth alternative to Rectifier Linear Unit (ReLU), yet many deployment, compression, and analysis toolchains are most naturally expressed for piecewise-linear (ReLU-type) networks. We study a hardness-parameterized formulation of GELU, f(x;λ)=xΦ(λ x), where Φ is the Gaussian CDF and λ \in [1, infty) controls gate sharpness, with the goal of turning smooth gated training into a controlled path toward ReLU-compatible models. Learning λ is non-trivial: naive updates yield unstable dynamics and effective gradient attenuation, so we introduce a constrained reparameterization and an optimizer-aware update scheme. Empirically, across a diverse set of model--dataset pairs spanning MLPs, CNNs, and Transformers, we observe structured layerwise hardness profiles and assess their robustness under different initializations. We further study a deterministic ReLU-ization strategy in which the learned gates are progressively hardened toward a principled target, enabling a post-training substitution of λ-GELU by ReLU with reduced disruption. Overall, λ-GELU provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with ReLU-centric downstream pipelines.

Abstract

Neural Network Activation Functions

Explore similar work

λλλ-GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks

Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions

A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

$λ$ -GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks