Normalization layers are ubiquitous in large language models (LLMs) yet represent a compute bottleneck: on hardware with distinct vector and matrix execution units, the RMS calculation blocks the subsequent matrix multiplication, preventing parallel execution. We present FlashNorm, an exact reformulation of RMSNorm followed by a linear layer that (i) eliminates the normalization weights by folding them into the subsequent linear layer, and (ii) defers the scalar RMS normalization to the output of the matrix multiplication, enabling the two operations to execute in parallel. Additionally, by the scale invariance of RMS, an RMSNorm followed by a linear layer followed by another RMSNorm allows the first RMSNorm to be eliminated entirely -- a mathematically identical simplification that removes the pre-attention RMSNorm in models using QKV-normalization (e.g., Gemma~4) and in MLA-models with latent normalization (e.g., DeepSeek-V2, Mistral Small 4, and OpenMythos). The same techniques extend to LayerNorm, Dynamic Tanh (DyT), feed-forward networks with GLU variants, and RoPE-based attention. On an NVIDIA T4 GPU, FlashNorm achieves 33 - 35% lower latency on the norm-then-project operation in the compute-bound (prefill) regime at SmolLM2-135M scale, and 12 - 14% at Llama-7B scale. We verify zero-loss weight folding on three models. Beyond inference speed, FlashNorm simplifies model implementations by reducing parameter tensor count. Watch our explainer video https://youtu.be/GEuJv34_XgU?si and see https://github.com/OpenMachine-ai/transformer-tricks for code.
Figures & tables
Figure 1: Mathematically identical implementations of RMSNorm followed by a linear layer. (a) Standard: normalization weights g are applied element-wise before W . (b) Weightless : g is folded into W∗=diag(g)W . (c) FlashNorm : normalization is deferred to the output, enabling matrix multiplication ( W∗ ) and RMS calculation to execute in parallel. ≜ denotes mathematical identity.
Figure 2: Weightless RMSNorm followed by a linear layer and a second RMSNorm: (a) unoptimized version; (b) optimized version where the first RMSNorm is eliminated by Proposition 3 .
Figure 3: Elimination of bias β : (a) Before elimination with β between normalization weights g and linear layer. (b) Optimized version with new bias term c∗=c+βW at the output.
Figure 4: Elimination of mean centering: (a) Original weight matrix V followed by mean centering. (b) Optimized version where the mean centering is merged into the modified weight matrix V∗ .
Figure 5: FFN with GLU variant and preceding FlashNorm: (a) unoptimized version; (b) optimized version with fewer scaling multipliers.
Figure 6: FlashNorm for scaled dot-product attention with RoPE: (a) unoptimized version; (b) optimized version where the normalization is fused with cos(⋅) and sin(⋅) .
Figure 8: Example timing diagrams. (a) Sequential: matrix unit idles 8 cycles. (b) Partial overlap with interleaved scaling and vector-matrix multiplication. (c) FlashNorm : vector-matrix multiply and RMS run fully in parallel.
Model
Benchmark
Source
FlashNorm
Δ
SmolLM2-135M
wikitext word_ppl
23.1262
23.1279
+0.001689
Llama-3.2-1B
wikitext word_ppl
12.8837
12.8842
+0.000497
Llama-3.1-8B
wikitext word_ppl
8.0668
8.0672
+0.000382
Llama-3.1-8B
MMLU acc (5-shot)
0.6556
0.6562
+0.0006
Llama-3.1-8B
HellaSwag acc (0-shot)
0.6071
0.6072
+0.0001
Table 1: Quality preservation across model scales and standard benchmarks, via lm-evaluation-harness with fp16 inference on an NVIDIA A100 (wikitext at 2048-token context, MMLU at 4096-token context for 5-shot prompts, HellaSwag at 2048-token context). Deltas between source and FlashNorm-folded checkpoints are within benchmark noise at every row, empirically consistent with the mathematical exactness of Propositions 1 and 2 .
Tokens
Sequential
FlashNorm
Latency reduction
Path
1
0.374 ms
0.374 ms
0.0 %
sequential (fallback)
64
0.506 ms
0.506 ms
0.0 %
sequential (fallback)
4096
0.704 ms
0.468 ms
+ 33.6%
parallel (overlap)
8192
0.929 ms
0.599 ms
+ 35.5%
parallel (overlap)
Table 2: Norm-then-project latency on T4 GPU at SmolLM2-135M scale ( n=576,k=960 ). Below the compute-bound threshold, FlashNorm falls back to sequential execution (0% overhead).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Tokens
Sequential
FlashNorm
Latency reduction
Path
1
0.641 ms
0.641 ms
0.0 %
sequential (fallback)
64
0.307 ms
0.307 ms
0.0 %
sequential (fallback)
256
0.577 ms
0.577 ms
0.0 %
sequential (fallback)
1024
1.882 ms
1.654 ms
+ 12.1%
parallel (overlap)
4096
7.628 ms
6.570 ms
+ 13.9%
parallel (overlap)
Appendix
Table 3: Norm-then-project latency on T4 GPU at Llama-7B scale ( n=4096,k=4096 ).
Tokens
PyTorch + cuBLAS
vLLM + cuBLAS
FlashNorm
vs PyTorch
vs vLLM
1
0.054 ms
0.052 ms
0.040 ms
+ 24.8%
+ 22.2%
16
0.056 ms
0.054 ms
0.039 ms
+ 29.1%
+ 26.5%
64
0.084 ms
0.080 ms
0.054 ms
+ 35.8%
+ 32.6%
4096
0.072 ms
0.066 ms
0.061 ms
+ 14.5%
+ 7.7%
Appendix
Table 4: Fused Triton RMSNorm+QKV kernel vs two production-realistic baselines on A100 at SmolLM2-135M scale ( n=576 , k=960 ), fp16 inputs. Baselines correspond to two commonly-deployed production inference paths: HuggingFace Transformers (PyTorch rms_norm plus cuBLAS) and vLLM ( _custom_ops.rms_norm plus cuBLAS).
Token perplexity
max∣Δ∣
Model
dtype
Baseline
Fold-only
Cancelled
Naive deletion
Fold-only
Cancelled
Greedy
Gemma-4-E2B
fp32
47.9350
47.9351
47.9353
= cancelled
5.6×10−4
6.8×10−4
12/12
Gemma-4-12B
fp32
10.5807
10.5807
10.5806
= cancelled
8.9×10−2
2.3×10−2
12/12
MiniCPM3-4B
fp32
8.8354
8.8354
8.8354
188.90
0
1.4×10−4
12/12
DeepSeek-V2-Lite
bf16
7.4758
7.4781
7.4796
618.41
2.38
2.31
–
Appendix
Table 5: Post-hoc norm cancellation. Token perplexity (ppl) on wikitext-2, maximum absolute logit difference max∣Δ∣ against the unmodified model, and the number of the 12 prompts whose 48-token greedy continuation is unchanged. For Gemma-4 the cancelled variant is the full removal of the pre-attention norm (naive deletion is the same operation); for the MLA models it is the corrected partial transform.
Variant
Session 1
Session 2
(A) eager weighted norm (stock)
123.1
107.3
(A ′ ) fused F.rms_norm , weighted
154.8
135.6
(B) fused F.rms_norm , weightless
154.8
136.7
Fusion, A → A ′
+25.7%
+26.4%
Weightless call, A ′ → B
−0.0%
+0.8%
Appendix
Table 6: Decode throughput (tokens/s, median of 20 trials) of one folded Llama-3.2-1B checkpoint under three normalization implementations, bf16, H100. Fusion accounts for the whole gain; the weightless call adds less than 1%.
Figure 9: FFN with ReLU and preceding FlashNorm: (a) unoptimized version; (b) optimized version where the normalization is deferred to the output of the FFN. Up and Down denote the linear layers for up and down projections.
Figure 10: FFN with ReGLU (or bilinear GLU) and preceding FlashNorm: (a) unoptimized version; (b) optimized version, which saves 2f−n multiply operations relative to (a), where two scalings are applied at the f -dimensional Gate and Up outputs.
Precision
Tokens/s
Total decode (ms)
Norm time (ms)
Norm fraction
bf16
38.5
6,650
1,640
24.62%
8-bit (BitsAndBytes)
12.5
20,455
1,675
8.19%
4-bit (NF4 + DQ)
14.5
17,720
1,650
9.31%
Appendix
Table 7: RMSNorm wall-clock fraction during 256-token greedy decode on Llama-3.2-1B-Instruct , A100, bf16 inputs, median of 20 trials. Norm time is cumulative across all per-layer RMSNorm calls.
Model
Tokens
Path
PyTorch
vLLM
FlashNorm
vs PyTorch
vs vLLM
SmolLM2-135M
1
decode
0.054
0.052
0.040
+ 24.8%
+ 22.2%
SmolLM2-135M
16
decode
0.056
0.054
0.039
+ 29.1%
+ 26.5%
SmolLM2-135M
64
decode
0.084
0.080
0.054
+ 35.8%
+ 32.6%
SmolLM2-135M
256
GEMM
0.060
0.059
0.069
− 15.2%
− 16.9%
SmolLM2-135M
1024
GEMM
0.059
0.057
0.060
− 2.7%
− 6.0%
SmolLM2-135M
4096
GEMM
0.072
0.066
0.061
+ 14.5%
+ 7.7%
Appendix
Table 8: Full A100 benchmark for the fused Triton RMSNorm+QKV prototype. Shapes (n,k) : (576, 960) for SmolLM2-135M, (2048, 2560) for Llama-3.2-1B, (4096, 6144) for Llama-3.1-8B. The Llama-3.2-1B shape covers the Q and K projections only ( 2048+8×64=2560 ); the V projection is benchmarked separately and not shown here because the value-side fold composes differently with grouped-query attention. SmolLM2-135M and Llama-3.1-8B shapes are full QKV. The vs-baseline columns report latency reduction, computed as (tbaseline−tFlashNorm)/tbaseline ; a 33% reduction corresponds to a 1.49× speedup.
Model
Tokens
realistic
fused kernel
vs realistic
SmolLM2-135M
1
0.060 ms
0.017 ms
+ 72.3%
SmolLM2-135M
16
0.054 ms
0.017 ms
+ 69.1%
Llama-3.2-1B
1
0.079 ms
0.049 ms
+ 37.9%
Llama-3.2-1B
16
0.064 ms
0.048 ms
+ 24.7%
Llama-3.1-8B
1
0.068 ms
0.074 ms
− 9.6% (parity)
Llama-3.1-8B
16
0.081 ms
0.074 ms
+ 8.5%
Appendix
Table 9: Kernel-level fused FlashNorm vs a realistic baseline (PyTorch native rms_norm followed by cuBLAS matmul) at decode token counts, on NVIDIA A100, fp16 inputs. The fused kernel uses a 3-stage cp.async pipeline with TB_K =32 to fully hide HBM load latency behind tensor-core compute. Times are per-call (operation-level); end-to-end model wall-clock is not reported here.
Model
Tokens
HF (ms)
FlashInfer (ms)
FlashNorm (ms)
vs HF
vs FlashInfer
SmolLM2-135M
1
0.096
0.038
0.019
+ 80.4%
+ 50.6%
SmolLM2-135M
16
0.096
0.039
0.019
+ 80.4%
+ 51.3%
SmolLM2-135M
64
0.097
0.038
0.019
+ 80.6%
+ 50.6%
SmolLM2-135M
256
0.097
0.039
0.019
+ 80.5%
+ 51.3%
SmolLM2-135M
1024
0.089
0.034
0.016
+ 82.0%
+ 52.2%
SmolLM2-135M
4096
0.079
0.035
0.021
+ 74.1%
+ 42.0%
Appendix
Table 10: H100 fused FlashNorm kernel (CUTLASS 3.x Sm90 CollectiveBuilder with EVT, CUDA-graph captured) vs two realistic baselines: HuggingFace ( torch.nn.functional.rms_norm plus cuBLAS) and FlashInfer ( flashinfer.norm.rmsnorm plus cuBLAS). fp16 inputs, averaged over 100 iterations after 20 warmup. Shapes (n,k) as in Table 8 (Llama-3.2-1B is Q+K only). The proposed FlashNorm kernel is CUDA-graph captured (one cudaGraphLaunch per call); the HuggingFace and FlashInfer baselines are timed in eager mode (per-call Python dispatch). The reported reductions therefore include both kernel-level acceleration and CPU-side launch-overhead amortization, the latter on the order of 10–15 μ s per call. We report it this way because production inference servers similarly use CUDA Graphs to amortize launch overhead, and the eager baselines are the realistic out-of-the-box deployment.
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China