Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression

Anjaneya Teja Sarma Kalvakolanu

cs.LGSep 6, 2026

Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression

Authors: Anjaneya Teja Sarma Kalvakolanu

Abstract

Linear algebra provides the framework of concepts (matrix rank, singular value decomposition (SVD), and eigendecomposition) that modern artificial intelligence employs to encode, compress, and propagate information through neural networks. This paper unifies fourteen separate peer-reviewed works analyzing the usage of these techniques in the context of transformer-based foundation model research, focusing on three areas of the topic: derivations and properties of self-attention matrices' output rank, compression methods that purposefully utilize this phenomenon, and the low-rank key-value (KV) cache projection and its semiseparable-matrix duality to linear attention and state-space structured models. We were motivated to conduct this work after observing an open problem in this literature: the interplay of the mentioned compression methods with natural rank collapse of the network. With this paper, we report an original finding that using SVD compression of attention projections actually has the opposite effect on the rank collapse of the network: while it strongly suppresses it at initialization, it accelerates on pretrained models (for GPT-2 124M, GPT-2 Medium 355M, and Pythia-160M) with minimal risk of object aliasing artifacts appearing (verified on all compression ratios) and is consistent across four rank estimation methods. A controlled causal decomposition of the effect in both settings showed that the reason for this behavior can be explained by the choice of the subspace SVD makes when compressing the matrix better than the reduction of the operator norm it achieves, explaining roughly 76% of the effect at initialization and 83% on the pretrained weights, providing a refinement to the calibration-aware compression viewpoint and an explanation of why it outperformed naive SVD truncation.

Figures & tables

Study	Method	Reported Effect
Hu et al. [ 7 ]	LoRA	$\sim$ 10,000 $\times$ fewer trainable params vs. full fine-tuning; no added inference latency
Saxena et al. [ 12 ]	Eigen Attention	Up to 40% KV-cache reduction; up to 60% attention-latency reduction
Kusupati et al. [ 17 ]	MRL	Up to 14 $\times$ smaller embeddings; up to 14 $\times$ retrieval speed-up
Wang et al. [ 11 ]	SVD-LLM	Improved accuracy retention vs. magnitude-based SVD at high compression
Chang et al. [ 14 ]	PALU	Reduced KV-cache memory with smaller accuracy loss than uniform-rank baselines
Dong et al. [ 4 ]	Rank-collapse bound	Doubly-exponential convergence to rank-1, $O(\gamma^{3^{l}})$

TABLE I: Quantitative effects reported in reviewed studies

Fig. 1: Effective rank vs. layer depth for GPT-2 (124M): random initialization (left) vs. pretrained weights (right), original vs. SVD-compressed ( $r{=}0.125$ ) attention projections.

Fig. 2: Causal decomposition of the collapse-rate reduction under SVD compression, random initialization (10 seeds, mean $\pm$ SD).

Fig. 3: Causal decomposition on pretrained GPT-2 124M ( $r{=}0.125$ , layer 8).

Low-Rank Compression

Low-rank compression reduces model size, memory use, and computation by representing weights, activations, or caches with compact factors.

See all 108 papers

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression
Apr 2, 2026Atul Kumar Sinha, François FleuretLLM Compression Low-Rank Compression
University of Geneva, Geneva, Switzerland · FAIR, Meta
Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
Apr 2, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +6LLM Compression Low-Rank Compression
Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2
SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers
Oct 26, 2026Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan +5Transformer Attention Attention Head Analysis
Georgia Institute of Technology · Dolby Laboratories

Apr 2, 2026J/K move · Enter open · S save

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

Atul Kumar Sinha, François Fleuret

University of Geneva, Geneva, Switzerland · FAIR, Meta

LLM Compression Low-Rank Compression Low-Rank Approximation Singular Value Decomposition

Open paper AskPDF arXiv

We introduce a fast low-rank factorization-based framework for compressing large language models that enables rapid compression of billion-parameter models without retraining. Unlike existing factorization-based approaches that optimize only on the original inputs, ignoring distribution shifts from upstream compression and thus propagating errors forward, or those that rely only on shifted inputs and risk drifting away from the original outputs, our approach accounts for both. Beyond individual layer compression, we further refine each transformer block end-to-end, minimizing block-level output distortion and allowing compressed layers to jointly compensate for accumulated errors. By anchoring each compressed layer to the original outputs while explicitly modeling input distribution shifts, our method finds a low-rank approximation that maintains functional equivalence with the original model. Experiments on large language models show that our method consistently outperforms existing SVD-based baselines across compression ratios, with the advantage becoming increasingly pronounced at aggressive compression budgets, where competing methods degrade substantially or collapse entirely, offering a practical solution for efficient, large-scale model deployment.

Abstract

Low-Rank Compression

Explore similar work

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers