We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation. Our main result is the exact identity ∥Jτ(u)∥∞→1=θ(p)/τ for the Jacobian Jτ(u) of the tempered softmax u↦softmax(u/τ), where θ(p)=4maxS⊆[L]p(S)(1−p(S)) measures how evenly the attention distribution can be bisected rather than how concentrated it is. We combine this identity with a block-∞/RMS norm under which row-stochastic attention mixing is nonexpansive. This yields a distribution-aware local Jacobian bound for multi-head attention and a sequence-length-independent Lipschitz bound on bounded input sets, with explicit dependence on width, input magnitude, temperature, and projection norms. We also identify a structural distinction between normalization placements: a pre-LN residual-sublayer Jacobian contains an additive identity term, whereas a post-LN residual-sublayer Jacobian does not. A LayerNorm projection lemma gives a sufficient condition under which the LayerNorm-only term in the post-LN expansion contracts geometrically; the condition is not tested by our experiments. Across three Pre-LN early-training runs of 774M-parameter models, attention becomes substantially more concentrated while the median lower-bound certificate for θ(p) remains near one at every sampled layer and checkpoint. This certifies near-maximal exact sensitivity for at least half of the sampled rows within each layer. A minority of rows enters a dominant-atom regime with lower exact sensitivity, consistent with the deterministic relationship between pmax and θ(p).
Figures & tables
Figure 1: Early-training dynamics for 774 M-parameter transformers over the logged trajectory. (a) Cross-entropy and (b) pre-clipping gradient norm: solid lines are the per-update mean across the three seeds, shaded regions span the per-update minimum to maximum across seeds. Panel (b) uses a logarithmic y -axis. (c) Lower-bound certificate θcert(p)/τ : solid lines show the mean across sampled layers and seeds of the per-layer medians; shaded regions show the envelope from the minimum per-layer 5th percentile to the maximum per-layer 95th percentile across layers and seeds. (d) Per-block parameter-gradient norms at the last logged checkpoint, shown for a single seed (seed 123 , the non-plateaued Post-LN seed); in the two plateaued Post-LN runs these norms are at most 7×10−6 through layers 0 – 33 . The dotted vertical line in (a)–(c) marks the end of warmup at update 500 . The gradient profiles are descriptive; they do not test the sufficient LayerNorm-contraction condition of Corollary A.3 .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Numerical check of Theorem 3.2 . For L∈{8,16} , exhaustive enumeration confirms the identity ∥Jτ(u)∥∞→1=θ(p)/τ holds to machine precision ( <10−13 relative error). For larger L , the plotted quantity is the feasible-subset lower-bound certificate θcert(p) , computed across 500 random distributions per length; it is not an independent verification of the operator norm.
Architecture
Mean Loss (final 50 updates)
Spike updates
Last-checkpoint θcert/τ
Pre-LN
4.509±0.005
9.0±1.0
0.99870
Post-LN
6.771±1.551
35.3±11.7
0.99999
Appendix
Table 1: Early-training comparison across three seeds. Loss is averaged over the final 50 optimizer updates; ± is the sample standard deviation across the three seeds (denominator n−1 ). Spike counts cover the logged trajectory and follow the rule defined in Section 5 . Certificate entries are the last-checkpoint means across sampled layers and seeds of the per-layer medians. The table does not represent converged performance.
Figure 3: Attention concentration changes while the median lower-bound certificate remains high. Across 3 seeds on 774 M Pre-LN over the logged early-training trajectory: normalized entropy decreases ( 0.987→0.726 ) and mean row maximum weight increases ( 0.010→0.174 ), while the certificate stays near one at every logged checkpoint. The certificate curve is the mean across sampled layers and seeds of the per-layer medians (last checkpoint 0.9987 ); the shaded envelope runs from the minimum per-layer 5th percentile to the maximum per-layer 95th percentile across layers and seeds ( [0.26,1.00] at the last checkpoint). Entropy and pmax curves are the mean across sampled layers of the per-layer sampled-row mean, averaged over seeds; the dotted vertical line marks the end of warmup at update 500 . The widening envelope shows that a minority of rows develops substantially smaller certificates; the dominant-atom formula in Corollary 3.4 characterizes exact sensitivity for rows with pmax≥1/2 .
Figure 4: Certificate θcert(p) at layers 0 , 18 , 35 at the last logged checkpoint, for a single seed (seed 123 ). Bars show that layer’s sampled-row median; whiskers span that layer’s sampled-row 5th and 95th percentiles. Medians are near one at all three sampled depths for both architectures, so the near-one median certificate is not confined to the final layer; the Pre-LN middle layer shows the widened lower whisker produced by its concentrated minority of rows (down to ≈0.35 ). Temporal behavior over the logged trajectory is shown in Figure 3 .