dynActivation: A Trainable Activation Family for Adaptive Nonlinearity

Alois Bachmann

cs.LGMar 23, 2026

dynActivation: A Trainable Activation Family for Adaptive Nonlinearity

Authors: Alois Bachmann

Organizations: Ruprecht-Karls-Universität Heidelberg

Abstract

This paper proposes $\mathrm{dynActivation}$ , a per-layer trainable activation defined as $f_i(x) = \mathrm{BaseAct}(x)(α_i - β_i) + β_i x$ , where $α_i$ and $β_i$ are lightweight learned scalars that interpolate between the base nonlinearity and a linear path and $\mathrm{BaseAct}(x)$ resembles any ReLU-like function. The static and dynamic ReLU-like variants are then compared across multiple vision tasks, language modeling tasks, and ablation studies. The results suggest that dynActivation variants tend to linearize deep layers while maintaining high performance, which can improve training efficiency by up to $+54\%$ over ReLU. On CIFAR-10, dynActivation(Mish) improves over static Mish by up to $+14.02\%$ on AttentionCNN with an average improvment by $+6.00\%$ , with a $24\%$ convergence-AUC reduction relative to Mish (2120 vs. 2785). In a 1-to-75-layer MNIST depth-scaling study, dynActivation never drops below $95\%$ test accuracy ( $95.3$ -- $99.3\%$ ), while ReLU collapses below $80\%$ at 25 layers. Under FGSM at $\varepsilon{=}0.08$ , dynActivation(Mish) incurs a $55.39\%$ accuracy drop versus $62.79\%$ for ReLU ( $7.40\%$ advantage). Transferred to language modeling, a new proposed dynActGLU-variant achieves a $10.3\%$ relative perplexity reduction over SwiGLU at 5620 steps (4.047 vs. 4.514), though the gap vanishes at 34300 steps.

Figures & tables

Panel of Figure 1 — Figure 1: Motivational view of trainable activations.

Panel of Figure 2 — Figure 2: Schematic view of the dynActivation family and its shape deformation relative to the base activation.

Object	Definition / interpretation
$\mathrm{BaseAct}(x)$	Chosen base activation, e.g. Mish, GELU, Swish, or ReLU.
$\alpha_{i}$	Trainable per-layer coefficient scaling the base activation contribution.
$\beta_{i}$	Trainable per-layer coefficient defining the explicit linear path.
$f_{i}(x)$	$\mathrm{BaseAct}(x)\cdot(\alpha_{i}-\beta_{i})+\beta_{i}x$ .
$\mathrm{dynAct}(x)$	Special case using Mish as base activation.

Table 1: Core definitions of the dynActivation family.

Panel of Figure 3 — Figure 3: Gradient flow and Lipschitz behavior of dynActivation.

System	GPU (VRAM)	CPU	RAM	SSD
System 1	RTX 2080 Ti (11 GB)	Ryzen 7 5700X CPU	32 GB DDR4	1 TB
System 2 (Cloud)	RTX 5090 (32 GB)	AMD EPYC 7763	112.8 GB DDR4	16 GB

Table 2: Hardware specification of Training Systems.

Setting	Value
Framework	PyTorch
Optimizer	Adam [ 12 ]
Learning rate	0.001
Weight decay	5e-9
Scheduler	StepLR(step_size=10, gamma=0.1)
Gradient clipping	clip_grad_norm(…, max_norm=1.0)

Table 3: Experimental protocol and reproducibility settings.

Activation	Mean Acc	Std Acc	Mean Loss	Std Loss
dynActivation(Mish)	66.8571	0.5845	1.1781	0.0235
dynActivation(GELU)	66.6886	0.7935	1.1810	0.0312
dynActivation(SiLU)	65.6500	1.1292	1.2304	0.0365
dynActivation(ReLU)	60.9117	1.9969	1.4215	0.0998

Table 4: Average performance over all dataset-architecture combinations for the dynamic variants.

Dataset	Architecture	Best dynActivation	Mean Acc	Mean Loss
CIFAR10	AttentionCNN	dynActivation(Mish)	78.27	0.6390
CIFAR10	DenseNet	dynActivation(Mish)	81.92	0.5466
CIFAR10	MobileNet	dynActivation(Mish)	80.87	0.5743
CIFAR10	ResNet18	dynActivation(GELU)	84.11	0.5196
CIFAR100	AttentionCNN	dynActivation(GELU)	45.70	2.0433
CIFAR100	DenseNet	dynActivation(Mish)	52.42	1.7713

Table 5: Best dynamic variant per dataset-architecture combination.

Aktivierung	Mean Acc (%)	$\sigma$ (Acc)	Mean Loss	$\sigma$ (Loss)	$\Delta$ Acc	$\Delta$ Loss
CIFAR10 – AttentionCNN
ReLU	73.64	1.4321	0.7605	0.0497	-4.63	+0.1215
GELU	69.83	1.7864	0.8624	0.0542	-8.44	+0.2234
Mish	64.25	1.6702	1.0001	0.0508	-14.02	+0.3611
SiLU	63.95	1.2492	1.0086	0.0323	-14.32	+0.3696
dynActivation(Mish)	78.27	0.8523	0.6390	0.0259	+0.00	+0.0000

Table 6: CIFAR-Statistics: Comparisons of the activations per combination.

\Delta

Acc/

\Delta

Loss are relative to the best dynActivation-variant for the specific combination.

Activation	Accuracy (Mean $\pm$ Std)	Loss (Mean $\pm$ Std)	Trainable
Apa [ 24 ]	$0.7926\pm 0.0055$	$0.6287\pm 0.0178$	Yes
Trainable_swish [ 4 ]	$0.7911\pm 0.0047$	$0.6318\pm 0.0112$	Yes
Pdelu [ 25 ]	$0.7909\pm 0.0054$	$0.6474\pm 0.0217$	Yes
Melu [ 26 ]	$0.7904\pm 0.0075$	$0.6597\pm 0.0258$	Yes
Mish [ 1 ]	$0.7891\pm 0.0041$	$0.6348\pm 0.0108$	No
Prelu_scalar [ 6 ]	$0.7874\pm 0.0057$	$0.6740\pm 0.0136$	No

Table 7: Comparison of standard and dynAct variants, sorted descending by mean accuracy.

Activation	$p$ -Value	$t$ -Statistic	Significant vs. Best
Apa [ 24 ]	0.2598	-1.2026	No
Trainable_swish [ 4 ]	0.0581	-2.1703	No
Pdelu [ 25 ]	0.1188	-1.7238	No
Melu [ 26 ]	0.0017	-4.4033	Yes
Mish [ 1 ]	0.0375	-2.4373	Yes
Prelu_scalar [ 6 ]	0.0208	-2.7976	Yes

Table 8: Statistical significance testing results against dynActivation(Mish) , sorted ascending by p-value.

Panel of Figure 4 — Figure 4: Accuracy-versus-Loss trade-off.

Panel of Figure 5 — Figure 5: MNIST depth scaling figure pair.

Figure 6: Learned layer-wise activation shapes for a 50-layer MNIST network.

Activation	Loss	Perplexity
Baseline SwiGLU [ 13 ]	1.5072	4.5142
dynActGLU(Swish)	1.3979	4.0467

Table 9: Short-run LLM results after 5620 training steps.

Activation	Loss	Perplexity
Baseline SwiGLU	1.3374	3.8091
dynActGLU(Swish)	1.3364	3.8056

Table 10: Long-run LLM results after 34300 training steps.

Figure 7: Initialization-ablation heatmap over $\alpha_{{init}}$ and $\beta_{{init}}$ .

Figure 8: Variance across optimizers and initialization schemes.

Figure 9: Robustness across hyperparameters.

Figure 10: Trajectories of mean and variance of activation parameters across epochs.

Figure 11: Distribution-shift robustness across corruption types and severities.

Figure 12: Adversarial robustness under FGSM and PGD.

Figure 13: L1 and L2 Regularization on dynActivation Parameters.

Figure 14: AUC-based convergence comparison.

Activation	AUC Loss	Steps to 1.0	Steps to 0.5	Conv. rate
relu	2864.364252	277	781	0.000206
mish	2784.969175	204	835	0.000203
swish	2830.302696	293	835	0.000211
gelu	2605.357465	204	781	0.000209
dynActivation(ReLU)	2350.585219	257	781	0.000213
dynActivation(Mish)	2119.673199	226	781	0.000213

Table 11: Theory-convergence statistics across activations.

Figure 15: Operator-level runtime benchmark.

Figure 16: Training efficiency combining convergence and compute cost.

Neural Network Activation Functions

Neural network activation functions shape how models represent nonlinear patterns, learn, remain stable, and map efficiently to hardware.

See all 75 papers

Oct 10, 2026

Rethinking Activation Functions: Reducing Bias Can Improve Learning

Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.

Zirui Peng, Heyi Zheng, Ziming Liu

Department of Physics, Peking University · Department of Mathematics, Peking University · College of AI, Tsinghua University +1

Loss Landscape Neural Network Activation Functions Neural Network Training Dynamics

Ask PDF

Apr 25, 2026

When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer

Dynamic Tanh (DyT) removes LayerNorm by bounding activations with a learned tanh(alpha x). We show that this bounding is a regime-dependent implicit regularizer, not a uniformly beneficial replacement. Across GPT-2-family models spanning 64M to 3.78B parameters and 1M to 118M tokens, with Llama and ViT cross-checks, DyT improves validation loss by 27.3% at 64M/1M but worsens it by 18.8% at 64M/118M; the 1M benefit vanishes with capacity (+1.7% at 3.78B), while the 118M penalty reaches +27.9%. The mechanism is measurable: 49% of DyT activations saturate at 1M versus 23% at 118M, and a 500-step saturation heuristic classifies DyT's sign with 75% raw in-sample accuracy on the 12-cell GPT-2 calibration set (AUC 0.75; 64% when adding Scale 5 stress cells), correctly labels 3/3 Llama checks, but only reaches 50% raw leave-one-scale-out accuracy. Three interventions support the bounding explanation: HardTanh reproduces the regime pattern, increasing alpha at 118M monotonically reduces DyT's penalty, and vanilla+dropout(p=0.5) matches DyT's data-rich loss. We also localize Llama-DyT collapse to SwiGLU gating, where saturation separates collapse from convergence in a 3-seed component ablation (r=0.94). Scope: all experiments are compute-limited (T/P < 1.84), below Chinchilla-optimal training.

Lucky Verma

Independent Researcher

Neural Network Activation Functions Layer Normalization Neural Network Training Dynamics Implicit Regularization

Ask PDF

May 17, 2026

Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes

The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures (MLP, ResNet, ViT, GPT-nano, MP-SENe) and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching up to 90% in GPT-nano. We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above

\sim

70% activation sparsity. While ReLU

^2

achieves a good sparsity--accuracy ratio in GPT-nano, it pathologically amplifies identified activation spikes in intermediate transformer layers. Clipping resolves this while preserving the representational benefits of squaring: clipped ReLU

^2

outperforms its unclipped version, and GELU

^2

achieves the lowest validation loss on GPT-nano. Code is available at https://github.com/On-Point-RND/BugOrFeature.

Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3

ReLU Neural Networks Activation Sparsity Activation Outliers Neural Network Activation Functions Neural Network Training Dynamics

Ask PDF

Abstract

Neural Network Activation Functions

Explore similar work

Rethinking Activation Functions: Reducing Bias Can Improve Learning

When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer

Bug or Feature2^22: Weight Drift, Activation Sparsity and Spikes

Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes