LayerNorm and attention Softmax are critical operations in Transformers. However, they cannot be efficiently mapped to General Matrix Multiplication (GEMM) computations on hardware. Therefore, modern hardware accelerator designs, such as NPUs on ASIC/FPGA, consume a large amount of hardware resources or chip area to implement their specific arithmetic units. Building on previous work about LayerNorm, Softmax, and their alternatives, we introduce Learnable Normalization States (LENS), a framework that reformulates LayerNorm as tiny MLPs by abstracting a common structure from related methods and extends this method to Softmax, which moves more of the computation in Transformers to GEMM as a hardware-friendly architecture. We evaluate LENS on image classification with Vision Transformers (ViT) and autoregressive language modeling with Pythia models, including generalization tests on WikiText and LAMBADA. Across these tasks, LENS matches or exceeds baseline performance in several vision settings and retains most of the baseline performance in the remaining vision and language experiments. These findings may suggest that hand-designed normalization and attention functions are not the only choice as alternatives to LayerNorm and Softmax. LENS also provides a basis for exploring Transformer architectures for resource-limited accelerators.
Figures & tables
Kernel
Hardware Resource Consumption
Relative to Systolic Array
LUTs
FFs
DSPs
LUTs
FFs
DSPs
Systolic Array with DSP Packing
244439
79969
128
100.00%
100.00%
100.00%
Softmax
7447
4835
38
3.05%
6.05%
29.69%
LayerNorm
12746
18751
80
5.21%
23.45%
62.50%
Softmax + 2 × LayerNorm
32939
42337
198
13.48%
52.94%
154.69%
Table 1: Hardware Resource Consumption of Systolic Array, LayerNorm, and Softmax Reported by Chen et al. (2024)
Methods
Original
a(x)
b(x)
LayerNorm(x)
Var(x)+ϵx−Mean(x)⊙γ+β
Var(x)+ϵγ
Var(x)+ϵVar(x)+ϵβ−Mean(x)γ
RMSNorm(x)
Mean(x⊙x)+ϵx⊙γ
Mean(x⊙x)+ϵγ
0
DyT(x)
tanh(αx)⊙γ+β
αγ−3α3γ⊙x⊙2
β
Derf(x)
erf(αx+s)⊙γ+β
R(αx+s)⊙αγ
R(αx+s)⊙sγ+β
SeeDNorm(x)
Mean(x⊙x)+ϵx⊙[σ(xβT)α+γ]
Mean(x⊙x)+ϵσ(xβT)α+γ
0
LENSNorm(x)
a(x)x+b(x)
a(x)
b(x)
Table 2: Unified formulation of LayerNorm and its alternatives as a(x)⊙x+b(x) in LENSNorm .
Dataset
Training Path
Epochs
Peak LR
Norm.
Attn.
Top-1 Acc. ( % )
Δ Acc. (pp)
CIFAR-10
From Scratch
400
1×10−3
LayerNorm
Softmax
80.33±0.28
0.00 (Baseline)
LENSNorm
Softmax
80.18±0.80
−0.15
LayerNorm
LENSMax
78.82±0.43
−1.51
LENSNorm
LENSMax
79.97±0.79
−0.36
Only-LENS
50
1×10−3
LENSNorm
LENSMax
68.26±3.38
−12.07
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
81.41±0.17
+1.08
Table 3: Experimental Results on CIFAR-10 and CIFAR-100 using a Single-Head ViT with LayerNorm , Softmax , LENSNorm and LENSMax .
Model
Training Path
LR
Knowledge
Top-1
Top-5
Schedule
Distillation
Acc. ( % )
Δ Acc. (pp)
Acc. ( % )
Δ Acc. (pp)
DeiT-Tiny (Base.)
Official
-
-
72.21
0.00
91.10
0.00
DeiT-Tiny+LENS
From Scratch
Cos.
No
51.67
−20.54
75.29
−15.81
Official → LENS
Cos.
No
71.94
−0.27
90.53
−0.57
Official → LENS
Exp.
No
72.44±0.03
+0.23
90.95±0.05
−0.15
Official → LENS
Exp.
Yes
73.66±0.26
+1.45
91.91±0.11
+0.81
Table 4: Experimental Results on ImageNet using DeiT-Tiny and DeiT-Small with LayerNorm , Softmax , LENSNorm and LENSMax .
Validation Results
Pythia
Training
Training
Epochs
LR
PPL ↓
Δ PPL
Next-Token
Model
Dataset
Path
Schedule
Acc. ( % ) ↑
Δ Acc. (pp)
14M (Base.)
The Pile
Official
-
-
98.01
0.00
28.95
0.00
14M+LENS
The Pile Subset
Official → LENS
2
Step.
92.54
−5.47
27.45
−1.50
14M (Base.)
FineWeb-Edu-3B
From Scratch
1
Cos.
56.67
0.00
31.56
0.00
14M+LENS
From Scratch
64.70
+8.03
30.13
−1.43
Table 5: Validation and Generalization Results of Pythia-14M/31M/70M using LayerNorm , Softmax , LENSNorm , and LENSMax .
Figure 1: Representative functional curves of LayerNorm and Softmax compared with LENS.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Training Path
Epoch
Peak LR
Norm.
Attn.
Top-1 Acc. (%)
Δ Acc. (pp)
CIFAR-10
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
81.41±0.17
0.00 (Baseline)
LENSNorm+
Softmax
81.64±0.26
+0.23
LENSNorm
LENSMax
81.00±0.19
0.00 (Baseline)
LENSNorm+
LENSMax
80.61±1.71
-0.39
CIFAR-100
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
52.14±0.83
0.00 (Baseline)
LENSNorm+
Softmax
52.66±0.86
+0.52
Appendix
Table 6: Experimental Results on CIFAR-10 and CIFAR-100 using a Single-Head ViT with LayerNorm , Softmax , LENSNorm , LENSNorm+ and LENSMax .
Figure 2: Validation trajectory of training on single-head ViT and DeiT-Tiny.
Figure 3: Validation trajectory of training on Pythia-14M.
Dataset
Training Path
Epoch
Peak LR
Norm.
Attn.
Top-1 Acc. (%)
Δ Acc. (pp)
CIFAR-10
From Scratch
400
1×10−3
DyT
Softmax
76.75
0.00 (Baseline)
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
76.97
0.22
DyT
LENSMax
76.66
-0.09
LENSNorm
LENSMax
78.15
1.4
From Scratch
400
1×10−3
Derf
Softmax
77.00
0.00 (Baseline)
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
78.61
1.61
Appendix
Table 7: Experimental Results on CIFAR-10 and CIFAR-100 using a Single-Head ViT with DyT/Derf/RMSNorm , Softmax , LENSNorm and LENSMax .
Figure 4: Relative ℓ2 of inherited weights after LENS recovery training on single-head ViT using CIFAR-10.
Figure 5: Relative ℓ2 of inherited weights after LENS recovery training on DeiT-Tiny.
Figure 6: Relative ℓ2 of inherited weights after LENS recovery training on DeiT-Small.
Figure 7: Relative ℓ2 of inherited weights after LENS recovery training on Pythia-70M.
Dept. of Artificial Intelligence Semiconductor Engineering, Hanyang University, Seoul, Republic of Korea · Dept. of Electronic Engineering, Hanyang University, Seoul, Republic of Korea