cs.LGNov 15, 2026

LENS: Replacing LayerNorm and Softmax in Transformers with Learnable Normalization States

Authors: Yuhao Liu, Salim Ullah, Akash Kumar

Organizations: Chair of Embedded Systems, Ruhr-University Bochum, Germany

Abstract

LayerNorm and attention Softmax are critical operations in Transformers. However, they cannot be efficiently mapped to General Matrix Multiplication (GEMM) computations on hardware. Therefore, modern hardware accelerator designs, such as NPUs on ASIC/FPGA, consume a large amount of hardware resources or chip area to implement their specific arithmetic units. Building on previous work about LayerNorm, Softmax, and their alternatives, we introduce Learnable Normalization States (LENS), a framework that reformulates LayerNorm as tiny MLPs by abstracting a common structure from related methods and extends this method to Softmax, which moves more of the computation in Transformers to GEMM as a hardware-friendly architecture. We evaluate LENS on image classification with Vision Transformers (ViT) and autoregressive language modeling with Pythia models, including generalization tests on WikiText and LAMBADA. Across these tasks, LENS matches or exceeds baseline performance in several vision settings and retains most of the baseline performance in the remaining vision and language experiments. These findings may suggest that hand-designed normalization and attention functions are not the only choice as alternatives to LayerNorm and Softmax. LENS also provides a basis for exploring Transformer architectures for resource-limited accelerators.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices

    Apr 26, 2026Dawon Choi, Hana Kim, Ji-Hoon KimEfficient Neural Network InferenceLayer Normalization

  2. FlashNorm: Fast Normalization for Transformers

    Jul 12, 2024Nils Graef, Filip Makraduli, Andrew Wasielewski +1Efficient Transformer InferenceLLM Inference Acceleration

  3. MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration

    Jun 16, 2026Kosmas Alexandridis, Giorgos DimitrakopoulosAI Accelerator InferenceInference Acceleration