Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

Sergey Salishev; Zeev Volkovich; Oleg Granichin

cs.LGOct 20, 2026

Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

Authors: Sergey Salishev, Zeev Volkovich, Oleg Granichin

Organizations: St. Petersburg State University, Saint Petersburg 199034, Russia · Department of Software Engineering, Braude College of Engineering, Karmiel 2161002, Israel

Abstract

Transformers are deployed as a special model class. Neural processing units (NPU) add dedicated operators for attention and normalization, and quantization relies on Transformer-specific algorithms. We show the computation does not force this. Every Transformer block, with attention, LayerNorm and a GELU MLP, compiles exactly into a generic deep neural network built from fixed affine maps, routing and the square, sigmoid and inverse square root, plus GELU or its sigmoid approximation. Without activation-activation products, every transform-wise compilation adds at least $N^2$ scalar nonlinear evaluations per head for dense scores. Keeping that one operation type removes this cost; its elementwise form, present on every surveyed NPU, suffices with the sigmoid and inverse square root. For top- $k$ attention with exchangeable tail scores, we prove that exact or conditional-mean tail mass gives an unbiased, minimum-mean-square-error estimate of the attention distribution, while tail drop is conditionally biased. When queries or agents cluster, centroids choose the keys first, reducing the number of computed scores from $N^2$ to $Nk$ per head. On DeiT-Tiny, MambaIRv2-Light and Pythia-70M the compiled models match their references with all keys, standard post-training quantization applies, and the exact tail-mass estimate beats tail dropping with half as many keys on DeiT-Tiny and, for $k\le8$ , on Pythia-70M.

Figures & tables

Symbolic
Transform	Source	Bound, dense / routed	Strict	Strict, routed	Hadamard	Extended
$\mathrm{LN}$	$N$		$2Nhd{+}2N$	$2Nhd{+}2N$	$N$	$N$
$\mathrm{softmax}$	$hN^{2}{+}hN$		$8hN^{2}{+}2hN$	$8hNk{+}2hN$	$3hN^{2}{+}hN$	$hN^{2}{+}hN$
$\mathrm{MLP}_{\mathrm{GELU}}$	$4Nhd$		$4Nhd$	$4Nhd$	$4Nhd$	$4Nhd$
$\mathrm{Score}$	$0$	$hN^{2}$ / $hNk$	$hN^{2}d{+}2hNd$	$hNkd{+}2hNd$	$0$	$0$
$\mathrm{Mix}$	$0$	$hNd$ / $hNd$	$hN^{2}d{+}hN^{2}{+}hNd$	$hNkd{+}hNk{+}hNd$	$0$	$0$

Table 1: Pointwise work per DeiT-Tiny block. Unary evaluations per instance of each transform;

N=197

tokens,

h=3

heads of dimension

d=64

, MLP width

4hd

. Source: the source graph (Definition 4 ), with GELU and dense attention. Bound: Theorem 2 for dense scores, Corollary 2 for the

k

routed scores. Strict: Theorem 1 ; routed: Proposition 5 , with

k=16

in the DeiT-Tiny rows. Hadamard: Proposition 1 . Extended: Theorem 3 . Block totals count both LayerNorms.

Panel of Figure 1 — Figure 1: Three exact compilations of a Transformer block. Each row lists the nodes of one graph under the NPU unit that runs them; $N$ tokens, $h$ heads of dimension $d$ . ∗ if used in the source graph.

Extended

Hadamard

Strict

Strict, polarization

k

Acc.

Latency

Memory

Acc.

Latency

Memory

Acc.

Latency

Memory

Acc.

Latency

Memory

Top-

k

support

86.24

3.28\times

1.62\times

83.51

4.18\times

8.72\times

84.08

5.04\times

8.65\times

83.95

68.3\times

47.1\times

89.23

3.89\times

2.32\times

87.46

4.73\times

8.72\times

87.98

5.37\times

8.65\times

87.71

76.1\times

47.1\times

90.35

4.88\times

3.70\times

89.59

5.75\times

8.72\times

89.78

7.14\times

8.65\times

89.70

93.1\times

47.1\times

197

91.36

5.41\times

2.45\times

90.35

10.5\times

17.3\times

90.73

13.0\times

17.2\times

90.57

132\times

63.1\times

Table 2: Measured cost of the compilations. DeiT-Tiny on Pet, bf16, before distillation. Accuracy (%), and latency and peak GPU memory relative to the reference, which takes

2.20

ms per batch of 32 and

62

MB (Methods). Hadamard: Proposition 1 . Strict: Theorem 1 ; polarization: polarization for every product. The extended graph uses GELU, the others

x\odot\sigma(1.702x)

. Top-

k

: the

k

largest exact scores. Routed:

k

keys chosen by 32 query centroids before the scores (Methods). At

k=197

both supports hold every key.

$k$	PV	Graph	Nodes	bf16	PTQ	$\Delta$	PTQ	$\Delta$
					Per-channel		Per-tensor
16	0.08	strict	380	84.08	82.61	$-1.47$	83.02	$-1.06$
32	0.16	strict	380	87.98	87.30	$-0.68$	87.27	$-0.71$
64	0.32	strict	380	89.78	89.72	$-0.05$	89.64	$-0.14$
197	1.00	strict	368	90.73	90.27	$-0.46$	89.75	$-0.98$
16	0.08	extended	247	88.58	86.13	$-2.45$	84.90	$-3.68$

Table 3: Standard post-training quantization of the compiled DeiT-Tiny on Pet. Top-1 accuracy (%) of the bf16 float model and after PTQ of the whole model with per-channel or per-tensor weight scales. On the calibration data the fp32 simulation holds every integer sum exactly (Methods). Nodes: quantized nodes, the output identity included. PV:

k/N

. Strict: the reduced strict graph, before distillation. Extended: the Jeffreys-distilled extended graph.

$k$	PV	Compiled	$D_{J}$	Distilled	Latency	Compiled
		Oxford-IIIT Pet				ImageNet-1k
1	0.01	15.73	6.26	$41.22\pm 0.60$	$2.45\times$	10.84
2	0.01	42.95	4.07	$65.30\pm 0.47$	$2.85\times$	27.79
4	0.02	67.10	2.20	$78.22\pm 0.18$	$2.99\times$	45.03
8	0.04	79.50	0.97	$84.91\pm 0.15$	$3.06\times$	57.20
16	0.08	86.24	0.38	$88.56\pm 0.19$	$3.28\times$	64.66

Table 4: DeiT-Tiny compiled with calibrated tail-mass top-

k

. Pet: top-1 accuracy (%) of the compiled model, Jeffreys divergence

D_{J}

to the reference, accuracy after one epoch of Jeffreys distillation (mean

\pm

s.d. over five seeds), and latency of the compiled forward pass relative to the reference (Methods). ImageNet-1k: accuracy of the compiled model without training. PV is

k/N

. References:

91.36\%

(Pet),

72.02\%

(ImageNet).

	$\times 4$						$\times 2$
	Tail drop		Calibrated		Exact		Calibrated
$k$	pre	post	pre	post	pre	post	pre	post
0	–	–	30.22	31.10	30.22	31.10	35.98	37.24
1	22.11	30.64	30.45	31.32	30.95	31.73	36.17	37.36
2	26.05	31.63	30.62	31.55	31.36	31.97	36.35	37.52
4	29.66	32.07	30.85	31.81	31.71	32.11	36.50	37.62

Table 5: MambaIRv2-Light on Set5. PSNR (dB) before (pre) and after (post) three epochs of

L_{1}

fine-tuning on DIV2K, selected on DIV2K training crops (Methods). References:

32.51

dB (

\times 4

38.24

dB (

\times 2

). At

k=0

all weight sits on the uniform tail, which tail drop cannot express; at

k=256

the modes coincide and the tail-drop cells repeat the calibrated run.

$k$	NLL	$\Delta$ NLL	PPL	NLL	$\Delta$ NLL	PPL	$\bar{\tau}$
	Exact tail mass			Tail drop
128	4.5496	0.0000	94.59	4.5496	0.0000	94.59	0.000
64	4.5497	0.0002	94.61	4.5497	0.0002	94.61	0.004
32	4.5513	0.0017	94.75	4.5567	0.0072	95.27	0.023
16	4.5584	0.0088	95.43	4.5885	0.0390	98.35	0.062
8	4.5822	0.0326	97.73	4.6811	0.1316	107.89	0.123

Table 6: Pythia-70M on WikiText-2. Token NLL (nats) and perplexity (PPL) of the compiled model without training, with exact tail mass and with tail drop, over

2.53\times 10^{5}

validation tokens in float32.

\bar{\tau}

is the mean omitted tail mass of top-

k

on the dense scores of 4 random training minibatches of 4 sequences. Reference: NLL

4.5496

, PPL

94.59

Panel of Figure 2 — Figure 2: Tail-mass estimate against tail drop. Quality of the compiled model before any training, as a function of the number of kept keys $k$ . (a) DeiT-Tiny on Oxford-IIIT Pet, top-1 accuracy. (b) Pythia-70M on WikiText-2, increase of the token negative log-likelihood over the reference ( $N=128$ ); calibrated mode does not apply to causal rows, whose numbers of valid keys differ. (c) MambaIRv2-Light $\times 4$ super-resolution on Set5, PSNR.

Accelerator	Unary maps on the NPU	Strict graph	Hadamard graph	Extended graph
Arm Ethos-U55/U65 [ 68 ]	Exp, Log, Rsqrt, GELU, Sigmoid, Tanh	Whole	Whole	Whole
Arm Ethos-U85 [ 68 ]	Exp, Log, Rsqrt, GELU, Sigmoid, Tanh	Whole	Whole	Whole
VeriSilicon VIP [ 69 ]	Exp, Log, Rsqrt, GELU, Sigmoid, Tanh	Whole	Whole	Whole
TI TDA4VM, C7x with MMA [ 70 ]	Exp, Log, Sqrt, GELU, Sigmoid, Tanh; power by a constant	Whole	Whole	Whole
Rockchip RK3588 [ 71 ]	GELU, Sigmoid, Tanh; power with exponent $\pm 0.5$ , 2 or 3	Whole	Whole	Whole
Renesas R-Car V4H/V4M, CNN-IP and DSP [ 72 ]	CNN-IP: ReLU, Sigmoid, Tanh	DSP: Rsqrt	DSP: Rsqrt	DSP: MatMul, softmax, LayerNorm, GELU

Table S1: Documented NPU support of the dense compiled graphs. Whole-graph compilation on the devices is future work.

	Time (ms/batch)						Working memory (MiB)
	Strict		Hadamard				Strict		Hadamard
Module	top- $k$	routed	top- $k$	routed	Ext.	Ref.	top- $k$	routed	top- $k$	routed	Ext.	Ref.
$\mathrm{Score}$ ( $QK^{\top}$ )	49.7	10.7	31.6	9.2	4.0	4.1	469	77	471	78	14	14
$\mathrm{Mix}$ ( $PV$ )	7.4	11.8	4.9	7.0	5.6	2.8	81	81	78	78	39	12
LayerNorm	23.0	22.2	17.9	17.9	17.9	5.5	16	17	19	19	19	9
Softmax	24.9	24.0	14.0	12.7	15.1	2.1	15	18	9	11	8	28

Table S2: Per-module GPU time and working memory at

k=16

in eager execution, summed over the 12 blocks of one batch. Routed: 32 query centroids. Ext.: the extended graph, with GELU; the other graphs use

x\odot\sigma(1.702x)

. Ref.: the reference with unfused attention. Softmax: top-

k

with the tail term, dense for Ref.

			Norm expansion
$k$	Cancellation	Polarization, per term	partial sums	per term	per term, scaled
16	65	4.3%	9.1%	5.0%	1.0%
197	543	14.1%	57.4%	14.4%	2.7%

Table S3: bf16 error of the value mix on DeiT-Tiny/Pet , relative to float64, averaged over the 12 blocks of 16 test images. Partial sums and scaled: emulated kernels with bf16 rounding of every stored value;

\lambda

is fitted on 8 other images. The cancellation factor is

\sum_{k}v_{k}^{2}/|y|

$k$	tail drop	fixed $q=10^{-3}$	calibrated	exact	calibrated, distilled
1	2.53	2.78	15.73	8.69	41.89
2	3.16	2.86	42.95	21.31	64.57
4	2.83	3.11	67.10	58.60	78.09
8	5.70	5.51	79.50	81.77	84.71
16	39.44	39.33	86.24	88.80	88.58
32	81.71	82.07	89.23	90.84	90.19

Table S4: Tail modes on DeiT-Tiny/Pet , top-1 accuracy (%). The last column is the saved one of five Jeffreys-distilled runs with trainable

q

, the run with the highest training accuracy.

$k$	calibrated mean $q$ , training	exact $\tau$ , test
1	0.7825	$0.7943\pm 0.1540$ (0.0007, 0.9925)
2	0.7139	$0.7229\pm 0.1703$ (0.0004, 0.9852)
4	0.6294	$0.6349\pm 0.1839$ (0.0004, 0.9710)
8	0.5278	$0.5300\pm 0.1885$ (0.0001, 0.9438)
16	0.4116	$0.4125\pm 0.1770$ (0.0000, 0.8915)
32	0.2826	$0.2831\pm 0.1475$ (0, 0.7920)

Table S5: Calibrated against exact tail mass on DeiT-Tiny/Pet. Calibrated: the per-block constant from the training split, averaged over blocks. Exact: per-query tail mass on the held-out Pet test split, mean

\pm

s.d. within a block averaged over blocks (min, max).

$k$	compiled	CE saved	CE mean	Jeffreys saved	Jeffreys mean
1	15.73	41.24	$40.94\pm 0.24$	41.89	$41.22\pm 0.60$
2	42.95	64.49	$63.82\pm 0.93$	64.57	$65.30\pm 0.47$
4	67.10	77.02	$77.03\pm 0.18$	78.09	$78.22\pm 0.18$
8	79.50	84.33	$84.40\pm 0.18$	84.71	$84.91\pm 0.15$
16	86.24	88.17	$88.26\pm 0.06$	88.58	$88.56\pm 0.19$
32	89.23	89.92	$90.00\pm 0.19$	90.19	$90.18\pm 0.08$

Table S6: Distillation on DeiT-Tiny/Pet , top-1 accuracy (%) of the saved run, the one with the highest training accuracy, and mean

\pm

s.d. over five seeds.

Network	Model	bf16	fp32	bf16 $-$ fp32
DeiT-Tiny, Pet	reference	91.36	91.33	$+0.03$
	extended	91.36	91.33	$+0.03$
	strict, GELU	91.06	91.33	$-0.27$
	Hadamard, GELU	91.28	91.33	$-0.05$
	strict, sigmoid GELU	90.73	90.27	$+0.46$
	Hadamard, sigmoid GELU	90.35	90.27	$+0.08$

Table S7: bf16 against fp32 with all keys. Top-1 accuracy (%) for DeiT-Tiny, PSNR (dB) on the Y channel of Set5 for MambaIRv2-Light, validation NLL (nats) for Pythia-70M. Sigmoid GELU:

x\,\sigma(1.702x)

$k$	bf16	$\Delta$ , per-channel	$\Delta$ , per-tensor	bf16	$\Delta$ , per-channel	$\Delta$ , per-tensor
	GELU			$x\,\sigma(1.702x)$
16	86.37	$-1.44$	$-0.90$	84.08	$-1.47$	$-1.06$
32	89.07	$-0.76$	$-0.68$	87.98	$-0.68$	$-0.71$
64	90.35	$-0.49$	$+0.05$	89.78	$-0.05$	$-0.14$
197	91.06	$-0.08$	$+0.19$	90.73	$-0.46$	$-0.98$

Table S8: PTQ of the strict graph with exact GELU and with

x\,\sigma(1.702x)

on DeiT-Tiny/Pet, before distillation, with the PTQ of Table 3 . bf16: top-1 accuracy (%) of the float model.

\Delta

: change after PTQ with per-channel or per-tensor weight scales. The approximation columns repeat Table 3 .

Efficient Transformer Inference

Efficient Transformer Inference studies methods and systems that reduce the latency, memory, and energy needed to run Transformer models.

See all 121 papers

Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs
Mar 4, 2026Pranav Kumar KaliaperumalActivation Outliers Post-Training Quantization
M.S. Computer Science, University of Colorado Denver, Aurora, Colorado, USA
Expressivity of Transformers: A Tropical Geometry Perspective
Apr 16, 2026Ye Su, Yong LiuTransformer Expressivity Transformer
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · University of Chinese Academy of Sciences, Beijing 101407, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
May 7, 2026Amrit NidhiEfficient Transformer Inference Self-Attention
Independent Researcher

Mar 4, 2026J/K move · Enter open · S save

Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs

Pranav Kumar Kaliaperumal

M.S. Computer Science, University of Colorado Denver, Aurora, Colorado, USA

Activation Outliers Post-Training Quantization Activation Quantization Transformer Inference Mixed-Precision Quantization

Open paper AskPDF arXiv

Post-training quantization (PTQ) of transformers is known to suffer from severe accuracy degradation due to structured activation outliers, as originally analyzed by Bondarenko et al. (EMNLP 2021) in work associated with Qualcomm AI Research. This paper provides a reproducible empirical reproduction and systems-level extension of that phenomenon in BERT-base fine-tuned on QNLI. When global W8A8 quantization is applied, validation accuracy drops sharply from 89.66% (FP32) to 54.33%, a decrease of 35.33 points. Statistical analysis of FP32 activations shows strongly heavy-tailed behavior that intensifies with model depth: kurtosis reaches 271 in the final layers and approximately 55% of activation energy is concentrated in the top 1% of channels. We evaluate several mitigation strategies. Mixed precision PTQ restores accuracy close to the FP32 baseline (89.42%). Per-embedding-group (PEG) quantization shows strong sensitivity to grouping structure, improving accuracy from 66.12% with three groups to 86.18% with four groups. In contrast, percentile-based calibration, even at thresholds between 99.0 and 99.99, fails to recover accuracy (about 50.54%), indicating that large activation channels encode structured signal rather than rare noise. Deployment profiling on an RTX 3050 GPU shows minimal differences in latency and memory usage across methods (median latency about 58-59 ms; VRAM usage about 484-486 MB), highlighting the importance of hardware-aware evaluation. Overall, the results show that PTQ failure in transformers is primarily driven by structured channel dominance amplified through residual connections. Effective mitigation therefore requires channel-aware precision allocation rather than scalar clipping alone.

Abstract

Efficient Transformer Inference

Explore similar work

Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs

Expressivity of Transformers: A Tropical Geometry Perspective

Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers