Spectral Conditioning of Attention Improves Transformer Performance

Hemanth Saratchandran; Simon Lucey

cs.LGMar 7, 2026

Spectral Conditioning of Attention Improves Transformer Performance

Authors: Hemanth Saratchandran, Simon Lucey

Organizations: Australian Institute for Machine Learning, Adelaide University

Abstract

We present a theoretical analysis of the Jacobian of an attention block within a transformer, showing that it is governed by the query, key, and value projections that define the attention mechanism. Leveraging this insight, we introduce a method that systematically alters the spectral properties of each attention layer to reduce the Jacobian's condition number, thereby improving the overall conditioning of the attention layers within a transformer network. We empirically show that this improved Jacobian conditioning translates to enhanced performance in practice. Our approach is simple, broadly applicable, and can be easily integrated as a drop-in replacement for a wide range of existing attention mechanisms. We validate its effectiveness across diverse transformer architectures and tasks, demonstrating consistent improvements in performance.

Figures & tables

Figure 1: An illustration of spectrally conditioned self-attention within a transformer layer. At each layer, the self-attention weights

W_{Q}

W_{K}

, and

W_{V}

are modified by adding correction terms

C_{Q}

C_{K}

, and

C_{V}

, respectively. The correction terms

C_{Q}

C_{K}

, and

C_{V}

are initialized before training using Theorem 3.8 and remain fixed throughout training.

Figure 2: Analysis for ViT-B. Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) and their spectrally conditioned counterparts (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) throughout training. Middle: Condition numbers of

W_{Q}

W_{K}

, and

W_{V}

, and their spectrally conditioned forms during training. Right: Average condition number of the self-attention Jacobian over the course of training, before and after spectral conditioning, along with the theoretical bound from Equation 10 .

Figure 3: Analysis for XCiT-M. Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) and their spectrally conditioned counterparts (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) throughout training. Middle: Condition numbers of

W_{Q}

W_{K}

, and

W_{V}

	ViT-B	DeiT-B	Swin-B	XCiT-M	DaViT-B
Original	80.7 ( $\pm$ 0.41)	81.6 ( $\pm$ 0.30)	83.4 ( $\pm$ 0.28)	82.6 ( $\pm$ 0.39)	84.3 ( $\pm$ 0.26)
Spec. cond.	81.7 ( $\pm$ 0.38)	82.6 ( $\pm$ 0.32)	84.1 ( $\pm$ 0.25)	83.5 ( $\pm$ 0.35)	84.9 ( $\pm$ 0.21)

Table 1: Comparison of Vision Transformers with their original attention architecture versus one with spectrally conditioned attention (Spec. cond.), pre-trained on the ImageNet-1k dataset. We report Top-1% classification accuracy. In each case, spectral conditioning improves performance over the original attention mechanism.

Model	$AP^{b}$	$AP^{b}_{50}$	$AP^{b}_{75}$	$AP^{m}$	$AP^{m}_{50}$	$AP^{m}_{75}$
original	44.9 ( $\pm$ 0.33)	66.1 ( $\pm$ 0.31)	48.9 ( $\pm$ 0.35)	40.1 ( $\pm$ 0.29)	63.1 ( $\pm$ 0.31)	42.8 ( $\pm$ 0.34)
Spec. cond.	45.6 ( $\pm$ 0.31)	66.7 ( $\pm$ 0.36)	49.6 ( $\pm$ 0.32)	40.5 ( $\pm$ 0.33)	63.4 ( $\pm$ 0.28)	43.3 ( $\pm$ 0.32)

Table 2: Performance evaluation of object detection and instance segmentation on the COCO dataset. For each metric, our spectrally conditioned architecture (Spec. cond.) outperforms the original.

Figure 4: Analysis for Nyströmformer on text classification task. Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) and their spectrally conditioned counterparts (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) throughout training. Middle: Condition numbers of

W_{Q}

W_{K}

, and

W_{V}

, and their spectrally conditioned forms during training. Right: Average condition number of the attention Jacobian over the course of training, before and after spectral conditioning, along with the theoretical bound from Equation 10 .

Model	ListOps	Text	Retrieval	Image	Pathfinder
Original	37.1 ( $\pm$ 0.21)	63.8 ( $\pm$ 0.24)	79.8 ( $\pm$ 0.26)	39.9 ( $\pm$ 0.20)	72.9 ( $\pm$ 0.25)
Spec. cond.	37.8 ( $\pm$ 0.23)	64.8 ( $\pm$ 0.20)	80.6 ( $\pm$ 0.27)	40.2 ( $\pm$ 0.22)	73.7 ( $\pm$ 0.23)

Table 3: Comparison of a Nyströmformer using its original architecture (original) with a Nyströmformer using spectral conditioned attention (Spec. cond.) on the LRA benchmark. We report evaluation accuracy (%). As shown, our methodology consistently improves performance across all tasks.

	MNLI	SST-2	STSB	RTE	QNLI	QQP	MRPC	CoLA	Avg.
Original	83.8 ( $\pm$ 0.21)	92.3 ( $\pm$ 0.18)	86.3 ( $\pm$ 0.21)	55.1 ( $\pm$ 0.20)	90.1 ( $\pm$ 0.23)	87.3 ( $\pm$ 0.15)	85.0 ( $\pm$ 0.25)	48.9 ( $\pm$ 0.26)	78.6
Spec. cond.	84.3 ( $\pm$ 0.20)	92.7 ( $\pm$ 0.19)	86.6 ( $\pm$ 0.22)	55.5 ( $\pm$ 0.24)	91.0 ( $\pm$ 0.22)	87.5 ( $\pm$ 0.23)	86.1 ( $\pm$ 0.23)	51.7 ( $\pm$ 0.24)	79.4

Table 4: Evaluation of a pre-trained Crammed BERT on the GLUE benchmark using the original architecture (Original) and our spectrally conditioned variant (Spec. cond.). Spectral conditioning improves performance across all tasks.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

$\lambda$	$2$	$4$	$6$	$8$	$10$	$12$	$14$	$16$
ViT-B Acc.	80.8	81.1	81.4	81.6	81.7	81.7	81.6	81.5

Appendix

Table 5: Ablation on

\lambda

for ViT-B

	ViT-B	DeiT-B	Swin-B	XCiT-M	DaViT-B
Original (with only layer norm.)	80.7 ( $\pm$ 0.41)	81.6 ( $\pm$ 0.30)	83.4 ( $\pm$ 0.28)	82.6 ( $\pm$ 0.39)	84.3 ( $\pm$ 0.26)
Spec. cond. + layer norm.	81.7 ( $\pm$ 0.38)	82.6 ( $\pm$ 0.32)	84.1 ( $\pm$ 0.25)	83.5 ( $\pm$ 0.35)	84.9 ( $\pm$ 0.21)
Spec. cond. - layer norm.	79.8 ( $\pm$ 0.31)	80.4 ( $\pm$ 0.33)	80.5 ( $\pm$ 0.23)	80.0 ( $\pm$ 0.35)	80.3 ( $\pm$ 0.21)

Appendix

Table 6: Performance comparison of spectral conditioning with and without layer normalization.

Model	Acc.	Training time (hrs:mins)
ViT-B (original)	80.7	29:29
ViT-B spec. cond. (Thm. 3.5)	82.0	41:38
ViT-B spec. cond. (Thm. 3.8)	81.7	29:33
DeiT-B (original)	81.6	26:16
DeiT-B spec. cond. (Thm. 3.5)	82.9	37:54
DeiT-B spec. cond. (Thm. 3.8)	82.6	26:24

Appendix

Table 7: Comparison of accuracy and training time across transformer variants with spectral conditioning of attention following Theorem 3.5 and Theorem 3.8 .

Memory Aspect	Original Attention	With Spectral Conditioning
Trainable Parameters	$3\times Dd$	$3\times Dd$
Gradient Storage	$3\times Dd$	$3\times Dd$
Correction Matrices ( $C_{Q}$ , $C_{K}$ , $C_{V}$ )	–	negligible (constant scalar)
Activation Memory	$3\times Nd$	$3\times Nd$ (no extra tensors)

Appendix

Table 8: Comparison of memory overhead for spectral conditioning relative to standard attention projections.

Operation	FLOPS (Original)	FLOPS (With Spectral Conditioning)
Query projection ( $XW_{Q}$ )	$2NDd$	$2NDd+Nd$
Key projection ( $XW_{K}$ )	$2NDd$	$2NDd+Nd$
Value projection ( $XW_{V}$ )	$2NDd$	$2NDd+Nd$
Total Projection FLOPS	$6NDd$	$6NDd+3Nd$

Appendix

Table 9: FLOPS comparison for the forward pass of the attention projection layers with and without spectral conditioning.

N

denotes the sequence length,

D

the embedding dimension, and

d

the projection dimension per head.

Figure 5: Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a ViT-B during training. We plot the mean over five trials and the standard deviation. Right: Average minimum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a ViT-B during training. We plot the mean over five trials and the standard deviation.

Figure 6: Left: Average maximum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a ViT-B during training. We plot the mean over five trials and the standard deviation. Right: Average maximum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a ViT-B during training. We plot the mean over five trials and the standard deviation.

Figure 7: Left: Average condition number of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a ViT-B during training. We plot the mean over five trials and the standard deviation. Right: Average condition number of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a ViT-B during training. We plot the mean over five trials and the standard deviation.

Figure 8: Average condition number of the self-attention Jacobian of a ViT-B over the course of training, before and after spectral conditioning, along with the theoretical bound from Equation 10 . We ran five trials with five different random seeds with plots showing the mean and standard deviations.

Figure 9: Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a XCiT-M during training. We plot the mean over five trials and the standard deviation. Right: Average minimum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a XCiT-M during training. We plot the mean over five trials and the standard deviation.

Figure 10: Left: Average maximum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a XCiT-M during training. We plot the mean over five trials and the standard deviation. Right: Average maximum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a XCiT-M during training. We plot the mean over five trials and the standard deviation.

Figure 11: Left: Average condition number of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a XCiT-M during training. We plot the mean over five trials and the standard deviation. Right: Average condition number of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a XCiT-M during training. We plot the mean over five trials and the standard deviation.

Figure 12: Average condition number of the self-attention Jacobian of a XCiT-M over the course of training, before and after spectral conditioning, along with the theoretical bound from Equation 10 . We ran five trials with five different random seeds with plots showing the mean and standard deviations.

Model	$\text{AP}^{\text{b}}$	$\text{AP}^{\text{b}}_{50}$	$\text{AP}^{\text{b}}_{75}$	$\text{AP}^{\text{m}}$	$\text{AP}^{\text{m}}_{50}$	$\text{AP}^{\text{m}}_{75}$
Swin-base	45.9	67.1	50.1	41.1	63.7	43.8
Spec. cond. Swin-base	46.8	68.1	50.7	41.8	64.2	44.6

Appendix

Table 10: Comparison of Swin-base and spectrally conditioned (spec. cond.) Swin-base on COCO benchmarks.

$\lambda$	$2$	$4$	$6$	$8$	$10$	$12$	$14$	$16$
Acc.	63.7	64.1	64.3	64.6	64.8	64.8	64.7	64.7

Appendix

Table 11: Ablation on

\lambda

for Nyströmformer on text classification task.

	Acc.(%)
Original (with only layer norm.)	63.8 ( $\pm$ 0.24)
Spec. cond. + layer norm.	64.8 ( $\pm$ 0.20)
Spec. cond. - layer norm	62.3 ( $\pm$ 0.23)

Appendix

Table 12: Comparing spectral conditioning with and without layer normalization for the Nyströmformer on text classification.

Figure 13: Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a Nyströmformer on the text classification task during training. We plot the mean over five trials and the standard deviation. Right: Average minimum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a Nyströmformer on the text classification during training. We plot the mean over five trials and the standard deviation.

Figure 14: Left: Average maximum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a Nyströmformer on the text classification task during training. We plot the mean over five trials and the standard deviation. Right: Average maximum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a Nyströmformer on the text classification during training. We plot the mean over five trials and the standard deviation.

Figure 15: Left: Average condition number of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a Nyströmformer on the text classification task during training. We plot the mean over five trials and the standard deviation. Right: Average condition number of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a Nyströmformer on the text classification task during training. We plot the mean over five trials and the standard deviation.

Figure 16: Average condition number of the self-attention Jacobian of a Nyströmformer on the text classification task over the course of training, before and after spectral conditioning, along with the theoretical bound from Equation 10 . We ran five trials with five different random seeds with plots showing the mean and standard deviations.

Figure 17: Analysis for Nyströmformer on listops task. Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) and their spectrally conditioned counterparts (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) throughout training. Middle: Condition numbers of

W_{Q}

W_{K}

, and

W_{V}

Figure 18: Left: Average minimum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a Nyströmformer on the listops task during training. We plot the mean over five trials and the standard deviation. Right: Average minimum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a Nyströmformer on the listops task during training. We plot the mean over five trials and the standard deviation.

Figure 19: Left: Average maximum singular value of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a Nyströmformer on the listops task during training. We plot the mean over five trials and the standard deviation. Right: Average maximum singular value of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a Nyströmformer on the listops task during training. We plot the mean over five trials and the standard deviation.

Figure 20: Left: Average condition number of the query, key, and value projection matrices (

W_{Q}

W_{K}

W_{V}

) for a Nyströmformer on the listops task during training. We plot the mean over five trials and the standard deviation. Right: Average condition number of the corrected query, key, and value projection matrices (

W_{Q}+C_{Q}

W_{K}+C_{K}

W_{V}+C_{V}

) for a Nyströmformer on the listops task during training. We plot the mean over five trials and the standard deviation.

Figure 21: Average condition number of the self-attention Jacobian of a Nyströmformer on the listops task over the course of training, before and after spectral conditioning, along with the theoretical bound from Equation 10 . We ran five trials with five different random seeds with plots showing the mean and standard deviations.

Explore similar work

Preconditioned Attention: Enhancing Efficiency in Transformers
Mar 28, 2026Hemanth SaratchandranAttention Mechanisms Neural Network Optimization
Australian Institute for Machine Learning (AIML), Adelaide University
Conditioned Initialization for Attention
Sep 7, 2026Hemanth Saratchandran, Simon LuceyNeural Network Initialization Transformer Attention
Australian Institute for Machine Learning Adelaide University
Exact Attention Sensitivity and the Geometry of Transformer Stability
Feb 21, 2026Seyed Morteza EmadiTransformer Attention Softmax Attention
UNC-Chapel Hill, Kenan-Flagler Business School Chapel Hill, NC, USA

Mar 28, 2026J/K move · Enter open · S save

Preconditioned Attention: Enhancing Efficiency in Transformers

Hemanth Saratchandran

Australian Institute for Machine Learning (AIML), Adelaide University

Attention Mechanisms Neural Network Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Central to the success of Transformers is the attention block, which effectively models global dependencies among input tokens associated to a dataset. However, we theoretically demonstrate that standard attention mechanisms in transformers often produce ill-conditioned matrices with large condition numbers. This ill-conditioning is a well-known obstacle for gradient-based optimizers, leading to inefficient training. To address this issue, we introduce preconditioned attention, a novel approach that incorporates a conditioning matrix into each attention head. Our theoretical analysis shows that this method significantly reduces the condition number of attention matrices, resulting in better-conditioned matrices that improve optimization. Conditioned attention serves as a simple drop-in replacement for a wide variety of attention mechanisms in the literature. We validate the effectiveness of preconditioned attention across a diverse set of transformer applications, including image classification, object detection, instance segmentation, long sequence modeling and language modeling.

Abstract

Explore similar work

Preconditioned Attention: Enhancing Efficiency in Transformers

Conditioned Initialization for Attention

Exact Attention Sensitivity and the Geometry of Transformer Stability