LayerNorm and attention Softmax are critical operations in Transformers. However, they cannot be efficiently mapped to General Matrix Multiplication (GEMM) computations on hardware. Therefore, modern hardware accelerator designs, such as NPUs on ASIC/FPGA, consume a large amount of hardware resources or chip area to implement their specific arithmetic units. Building on previous work about LayerNorm, Softmax, and their alternatives, we introduce Learnable Normalization States (LENS), a framework that reformulates LayerNorm as tiny MLPs by abstracting a common structure from related methods and extends this method to Softmax, which moves more of the computation in Transformers to GEMM as a hardware-friendly architecture. We evaluate LENS on image classification with Vision Transformers (ViT) and autoregressive language modeling with Pythia models, including generalization tests on WikiText and LAMBADA. Across these tasks, LENS matches or exceeds baseline performance in several vision settings and retains most of the baseline performance in the remaining vision and language experiments. These findings may suggest that hand-designed normalization and attention functions are not the only choice as alternatives to LayerNorm and Softmax. LENS also provides a basis for exploring Transformer architectures for resource-limited accelerators.
Figures & tables
Kernel
Hardware Resource Consumption
Relative to Systolic Array
LUTs
FFs
DSPs
LUTs
FFs
DSPs
Systolic Array with DSP Packing
244439
79969
128
100.00%
100.00%
100.00%
Softmax
7447
4835
38
3.05%
6.05%
29.69%
LayerNorm
12746
18751
80
5.21%
23.45%
62.50%
Softmax + 2 × LayerNorm
32939
42337
198
13.48%
52.94%
154.69%
Table 1: Hardware Resource Consumption of Systolic Array, LayerNorm, and Softmax Reported by Chen et al. (2024)
Methods
Original
a(x)
b(x)
LayerNorm(x)
Var(x)+ϵx−Mean(x)⊙γ+β
Var(x)+ϵγ
Var(x)+ϵVar(x)+ϵβ−Mean(x)γ
RMSNorm(x)
Mean(x⊙x)+ϵx⊙γ
Mean(x⊙x)+ϵγ
0
DyT(x)
tanh(αx)⊙γ+β
αγ−3α3γ⊙x⊙2
β
Derf(x)
erf(αx+s)⊙γ+β
R(αx+s)⊙αγ
R(αx+s)⊙sγ+β
SeeDNorm(x)
Mean(x⊙x)+ϵx⊙[σ(xβT)α+γ]
Mean(x⊙x)+ϵσ(xβT)α+γ
0
LENSNorm(x)
a(x)x+b(x)
a(x)
b(x)
Table 2: Unified formulation of LayerNorm and its alternatives as a(x)⊙x+b(x) in LENSNorm .
Dataset
Training Path
Epochs
Peak LR
Norm.
Attn.
Top-1 Acc. ( % )
Δ Acc. (pp)
CIFAR-10
From Scratch
400
1×10−3
LayerNorm
Softmax
80.33±0.28
0.00 (Baseline)
LENSNorm
Softmax
80.18±0.80
−0.15
LayerNorm
LENSMax
78.82±0.43
−1.51
LENSNorm
LENSMax
79.97±0.79
−0.36
Only-LENS
50
1×10−3
LENSNorm
LENSMax
68.26±3.38
−12.07
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
81.41±0.17
+1.08
Table 3: Experimental Results on CIFAR-10 and CIFAR-100 using a Single-Head ViT with LayerNorm , Softmax , LENSNorm and LENSMax .
Model
Training Path
LR
Knowledge
Top-1
Top-5
Schedule
Distillation
Acc. ( % )
Δ Acc. (pp)
Acc. ( % )
Δ Acc. (pp)
DeiT-Tiny (Base.)
Official
-
-
72.21
0.00
91.10
0.00
DeiT-Tiny+LENS
From Scratch
Cos.
No
51.67
−20.54
75.29
−15.81
Official → LENS
Cos.
No
71.94
−0.27
90.53
−0.57
Official → LENS
Exp.
No
72.44±0.03
+0.23
90.95±0.05
−0.15
Official → LENS
Exp.
Yes
73.66±0.26
+1.45
91.91±0.11
+0.81
Table 4: Experimental Results on ImageNet using DeiT-Tiny and DeiT-Small with LayerNorm , Softmax , LENSNorm and LENSMax .
Validation Results
Pythia
Training
Training
Epochs
LR
PPL ↓
Δ PPL
Next-Token
Model
Dataset
Path
Schedule
Acc. ( % ) ↑
Δ Acc. (pp)
14M (Base.)
The Pile
Official
-
-
98.01
0.00
28.95
0.00
14M+LENS
The Pile Subset
Official → LENS
2
Step.
92.54
−5.47
27.45
−1.50
14M (Base.)
FineWeb-Edu-3B
From Scratch
1
Cos.
56.67
0.00
31.56
0.00
14M+LENS
From Scratch
64.70
+8.03
30.13
−1.43
Table 5: Validation and Generalization Results of Pythia-14M/31M/70M using LayerNorm , Softmax , LENSNorm , and LENSMax .
Figure 1: Representative functional curves of LayerNorm and Softmax compared with LENS.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Training Path
Epoch
Peak LR
Norm.
Attn.
Top-1 Acc. (%)
Δ Acc. (pp)
CIFAR-10
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
81.41±0.17
0.00 (Baseline)
LENSNorm+
Softmax
81.64±0.26
+0.23
LENSNorm
LENSMax
81.00±0.19
0.00 (Baseline)
LENSNorm+
LENSMax
80.61±1.71
-0.39
CIFAR-100
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
52.14±0.83
0.00 (Baseline)
LENSNorm+
Softmax
52.66±0.86
+0.52
Appendix
Table 6: Experimental Results on CIFAR-10 and CIFAR-100 using a Single-Head ViT with LayerNorm , Softmax , LENSNorm , LENSNorm+ and LENSMax .
Figure 2: Validation trajectory of training on single-head ViT and DeiT-Tiny.
Figure 3: Validation trajectory of training on Pythia-14M.
Dataset
Training Path
Epoch
Peak LR
Norm.
Attn.
Top-1 Acc. (%)
Δ Acc. (pp)
CIFAR-10
From Scratch
400
1×10−3
DyT
Softmax
76.75
0.00 (Baseline)
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
76.97
0.22
DyT
LENSMax
76.66
-0.09
LENSNorm
LENSMax
78.15
1.4
From Scratch
400
1×10−3
Derf
Softmax
77.00
0.00 (Baseline)
Scratch → LENS
50
5×10−4
LENSNorm
Softmax
78.61
1.61
Appendix
Table 7: Experimental Results on CIFAR-10 and CIFAR-100 using a Single-Head ViT with DyT/Derf/RMSNorm , Softmax , LENSNorm and LENSMax .
Figure 4: Relative ℓ2 of inherited weights after LENS recovery training on single-head ViT using CIFAR-10.
Figure 5: Relative ℓ2 of inherited weights after LENS recovery training on DeiT-Tiny.
Figure 6: Relative ℓ2 of inherited weights after LENS recovery training on DeiT-Small.
Figure 7: Relative ℓ2 of inherited weights after LENS recovery training on Pythia-70M.
In Transformer models, non-GEMM (non-General Matrix Multiplication) operations -- especially Softmax and Layer Normalization (LayerNorm) -- often dominate hardware cost due to their nonlinear nature. To address this, previous approximation studies mainly target rank-oriented tasks, which is acceptable for classification. However, edge Natural Language Processing (NLP) applications and edge generative AI are largely evaluated based on score-oriented tasks, so normalization-guaranteed non-GEMM operations are essential. We propose a hardware-efficient Softmax and LayerNorm with Guaranteed Normalization for Edge devices. Our design employs hardware-efficient approximation methods while preserving the normalization (Softmax: ∑p=1, LayerNorm: σ=1). Our architecture is described in Verilog HDL and synthesized using the Samsung 28nm CMOS process. In accuracy evaluation, we achieve high accuracy with minimal degradation: GLUE +0.07%, SQuAD -0.01%, perplexity -0.09%. Implementation results show that our architecture is small: 942μm2 for Softmax, 1199μm2 for LayerNorm. Compared to the state of the art, we achieve up to 11x and 14x reduction in area, respectively.
Dawon Choi, Hana Kim, Ji-Hoon Kim
Dept. of Artificial Intelligence Semiconductor Engineering, Hanyang University, Seoul, Republic of Korea · Dept. of Electronic Engineering, Hanyang University, Seoul, Republic of Korea
Normalization layers are ubiquitous in large language models (LLMs) yet represent a compute bottleneck: on hardware with distinct vector and matrix execution units, the RMS calculation blocks the subsequent matrix multiplication, preventing parallel execution. We present FlashNorm, an exact reformulation of RMSNorm followed by a linear layer that (i) eliminates the normalization weights by folding them into the subsequent linear layer, and (ii) defers the scalar RMS normalization to the output of the matrix multiplication, enabling the two operations to execute in parallel. Additionally, by the scale invariance of RMS, an RMSNorm followed by a linear layer followed by another RMSNorm allows the first RMSNorm to be eliminated entirely -- a mathematically identical simplification that removes the pre-attention RMSNorm in models using QKV-normalization (e.g., Gemma~4) and in MLA-models with latent normalization (e.g., DeepSeek-V2, Mistral Small 4, and OpenMythos). The same techniques extend to LayerNorm, Dynamic Tanh (DyT), feed-forward networks with GLU variants, and RoPE-based attention. On an NVIDIA T4 GPU, FlashNorm achieves 33 - 35% lower latency on the norm-then-project operation in the compute-bound (prefill) regime at SmolLM2-135M scale, and 12 - 14% at Llama-7B scale. We verify zero-loss weight folding on three models. Beyond inference speed, FlashNorm simplifies model implementations by reducing parameter tensor count. Watch our explainer video https://youtu.be/GEuJv34_XgU?si and see https://github.com/OpenMachine-ai/transformer-tricks for code.
Nils Graef, Filip Makraduli, Andrew Wasielewski +1
The rapid growth of Large Language Models (LLMs) has intensified the need for specialized hardware accelerators that can satisfy stringent inference latency and power constraints. Although matrix multiplications dominate the overall computational workload, non-linear vector normalization operations, such as LayerNorm, RMSNorm and Softmax can become critical hardware bottlenecks. Existing accelerators typically implement these functions using dedicated hardware blocks, leading to duplicated resources and inefficient silicon utilization. To address this limitation, we propose a Minimalist Integer Vector Engine (MIVE), a programmable architecture capable of executing all three operations within a unified datapath. By exploiting common computational patterns across LayerNorm, RMSNorm and Softmax the proposed vector engine maximizes hardware sharing while reducing implementation overhead. Physical ASIC implementation results show that MIVE provides comprehensive multi-function support while achieving higher area and hardware efficiency than most state-of-the-art standalone accelerators.
Kosmas Alexandridis, Giorgos Dimitrakopoulos
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece