Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training

Seyed Morteza Emadi

cs.LGFeb 21, 2026

Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training

Authors: Seyed Morteza Emadi

Organizations: UNC-Chapel Hill, Kenan-Flagler Business School, Chapel Hill, NC, USA

Abstract

Attention scores in transformers are bilinear forms $S_{ij} = x_i^\top M x_j / \sqrt{d_h}$ whose maximum magnitude governs overflow risk in low-precision training. We derive a rank-aware concentration inequality: when the interaction matrix $M = W^Q W^{K\top}$ has rank $r \ll d$ , tail probabilities for $\max_{i,j}|S_{ij}|$ decay as $\exp(-d^{2}α^{2}/(γr))$ rather than $\exp(-dα^{2})$ , where $γ> 1$ is a typicality parameter. For transformer attention where $r = d_h$ , this yields $8$ -- $28\times$ tighter concentration than rank-agnostic bounds in modern architectures. We apply this result to FP8 training, deriving geometry-aware scale factors that provide principled overflow guarantees without observing activations. The method computes per-layer scales from the spectral norm $\|W^Q W^{K\top}\|_2$ via implicit power iteration, includes a grouped query attention formulation that avoids key expansion, and remains compatible with fused attention kernels. Across GPT-2 XL to Llama-2-70B, geometry-aware scaling eliminates overflows in transient scenarios where delayed scaling fails, while achieving comparable downstream MMLU accuracy.

Figures & tables

Method	Transient-Safe	Fused-Compat.
Delayed	✗	✓
Current	✓	✗
Ours	✓	✓

Table 1: The FP8 scaling dilemma.

Model	$d$	$d_{h}$	$\gamma$	Improvement
GPT-2 XL	1600	64	2.98	$8\times$
Mistral-7B	4096	128	2.26	$14\times$
Llama-2-13B	5120	128	2.28	$18\times$
Llama-2-70B	8192	128	2.32	$28\times$

Table 2: Concentration exponent improvement from rank-aware bounds:

d/(\gamma d_{h})

tighter tail probabilities, where

\gamma

is determined by Eq. ( 12 ).

Model	$d$	$d_{h}$	$N$	$\alpha_{\min}$
GPT-2 XL	1600	64	1200	0.074
Mistral-7B	4096	128	1024	0.035
Llama-2-13B	5120	128	1600	0.028
Llama-2-70B	8192	128	5120	0.018

Table 3: Minimum calibration factor

\alpha_{\min}

for

\delta^{*}=10^{-6}

and

L=1024

Model	Overfl.	Max Scaled	Overfl.	Max Scaled
	Delayed		Ours
GPT-2 XL	48/48	8761	0/48	120
Mistral-7B	32/32	7123	0/32	196
Llama-2-13B	40/40	5600	0/40	2.5
Llama-2-70B	80/80	9498	0/80	1.0

Table 4: First forward pass after loading pretrained weights. Overflow : layers exceeding 448. Max Scaled : maximum scaled logit.

Method	Loss	Overfl.	Util.	MMLU
Delayed	0.0116	68	70.7%	33.0%
Ours (conservative)	0.0119	0	0.5%	28.1%
Ours + auto- $\alpha$	0.0116	0	31.2%	33.6%

Table 5: Training metrics and MMLU STEM accuracy on Llama-2-13B. Auto-

\alpha

achieves comparable accuracy to the baseline while eliminating all overflows.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Spectral norm

\sigma_{QK}^{(\ell)}

by layer for all four models, computed on pretrained weights. Early layers consistently exhibit larger spectral norms, with layer 0 being the maximum in three of four models.

Model	Mean	Max	Min	Max Layer
GPT-2 XL	83.1	483.9	55.8	0
Mistral-7B	4.9	46.8	2.4	0
Llama-2-13B	198.4	463.5	134.4	0
Llama-2-70B	584.2	1786.1	264.6	67

Appendix

Table 6: Spectral norm statistics across layers for pretrained weights. Max Layer indicates which layer has the largest spectral norm.

	GPT-2 XL	Mistral-7B	Llama-2-13B	Llama-2-70B
Params	1.5B	7B	13B	70B
Layers	48	32	40	80
Attention	MHA	GQA	MHA	GQA
$n_{q}$ / $n_{kv}$	25/25	32/8	40/40	64/8
$d$	1600	4096	5120	8192
$d_{h}$	64	128	128	128

Appendix

Table 7: Model architectures. MHA: multi-head attention. GQA: grouped query attention.

	GPT-2 XL	Mistral-7B	Llama-2-13B	Llama-2-70B
Batch size	8	4	4	2
Seq. length	1024	1024	1024	1024
Learning rate	$10^{-4}$	$10^{-5}$	$10^{-5}$	$10^{-5}$
Optimizer	AdamW	AdamW	AdamW	AdamW
Weight decay	0.01	0.01	0.01	0.01
Grad. clip	1.0	1.0	1.0	1.0

Appendix

Table 8: Training configuration for all experiments.

Figure 2: Response to

4\times

weight spike at step 10 (GPT-2 XL). (a) Maximum scaled attention logit. (b) Scale factor over time. Delayed scaling overflows because its history buffer contains no information about the weight change. Geometry-aware scaling adapts instantaneously because the scale factor is computed from current weights.

Model	Attention	Delayed	Ours	Overhead
GPT-2 XL	MHA	136.3 ms	137.6 ms	$+1.0\%$
Mistral-7B	GQA 4:1	58.2 ms	55.1 ms	$\mathbf{-5.3\%}$
Llama-2-13B	MHA	96.2 ms	98.1 ms	$+1.9\%$
Llama-2-70B	GQA 8:1	206.3 ms	215.0 ms	$+4.3\%$

Appendix

Table 9: Computational overhead (forward pass time, averaged over 100 iterations).

Method	Median	P10	P90
Delayed	90.5%	84.0%	98.0%
Ours (conservative)	14.3%	13.6%	15.3%
Ours + auto- $\alpha$	31.2%	28.5%	33.7%

Appendix

Table 10: FP8 dynamic range utilization (%) during training (GPT-2 XL).

Figure 3: Training loss comparison for delayed scaling, geometry-aware scaling (conservative), and geometry-aware scaling with auto-

\alpha

on Llama-2-13B (MMLU STEM, 3000 steps). All methods converge to similar final loss (

\approx 0.012

), yet downstream MMLU accuracy differs substantially (Table 5 ). The vertical dashed line indicates when auto-

\alpha

calibration completes (step 100). The conservative variant shows slightly higher loss throughout training due to reduced FP8 utilization (0.5% vs. 31.2% for auto-

\alpha

Subject	Delayed	Cons.	Auto- $\alpha$
Abstract Algebra	25.0	25.0	27.0
College Math	36.0	31.0	35.0
Elementary Math	27.0	25.9	27.8
HS Math	26.7	22.6	27.0
HS Statistics	30.1	31.0	30.6
Astronomy	41.4	29.6	37.5

Appendix

Table 11: MMLU accuracy (%) by subject on Llama-2-13B after 3000 fine-tuning steps.

Explore similar work

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
Jul 25, 2026Byeong Hoon YoonSoftmax Attention Transformer Expressivity
Independent Researcher
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
Aug 26, 2026Gerard Conangla PlanesLow-Rank Adaptation Transformer Attention
Aily Labs
Numerical stability analysis of large language models
Mar 13, 2025Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1Transformer Inference Transformer Attention
Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies

Jul 25, 2026J/K move · Enter open · S save

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

Byeong Hoon Yoon

Independent Researcher

Softmax Attention Transformer Expressivity Low-Rank Approximation Attention Mechanisms

Open paper AskPDF arXiv

Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it? We study this through the Entropic Bound, a spectral notion of task-intrinsic capacity for Transformers. We first prove that, in a linear attention surrogate, the intrinsic rank

r^*

of the token-mixing operator is a tight lower bound: any rank-deficient model incurs unavoidable excess risk, and the bound is achievable at

r^*

. We further show that gradient descent recovers this rank under standard low-rank implicit-bias assumptions, confirm all three properties empirically, and show

r^*

is recoverable from data before training. We then ask whether this transfers to real attention. A naive transfer fails, and a controlled interpolation ladder localizes the cause precisely: it is not softmax and not a rank constraint, but the input-conditioned nature of attention's mixing operator, which a static weight kernel cannot summarize. Motivated by this, we introduce an attention-native intrinsic rank -- the minimum query-key kernel rank realizing the task within the attention class -- and show that under this definition the full Entropic Bound structure (deficiency, achievability, recovery) is restored for both linear and softmax attention, with the energy effective rank as the estimator robust to softmax distortion. Finally, we map the boundary of data-only predictability:

r^*

is exactly recoverable for linear QK attention, even without the value map at scale, while softmax attention admits only partial pre-training recovery due to nonlinear inversion and kernel-value identifiability effects. Our results reframe the Entropic Bound from a post-hoc descriptor into an attention-native capacity measure with a precisely characterized predictability frontier.

Abstract

Explore similar work

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

Numerical stability analysis of large language models