cs.LGSep 6, 2026

Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression

Authors: Anjaneya Teja Sarma Kalvakolanu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Linear algebra provides the framework of concepts (matrix rank, singular value decomposition (SVD), and eigendecomposition) that modern artificial intelligence employs to encode, compress, and propagate information through neural networks. This paper unifies fourteen separate peer-reviewed works analyzing the usage of these techniques in the context of transformer-based foundation model research, focusing on three areas of the topic: derivations and properties of self-attention matrices' output rank, compression methods that purposefully utilize this phenomenon, and the low-rank key-value (KV) cache projection and its semiseparable-matrix duality to linear attention and state-space structured models. We were motivated to conduct this work after observing an open problem in this literature: the interplay of the mentioned compression methods with natural rank collapse of the network. With this paper, we report an original finding that using SVD compression of attention projections actually has the opposite effect on the rank collapse of the network: while it strongly suppresses it at initialization, it accelerates on pretrained models (for GPT-2 124M, GPT-2 Medium 355M, and Pythia-160M) with minimal risk of object aliasing artifacts appearing (verified on all compression ratios) and is consistent across four rank estimation methods. A controlled causal decomposition of the effect in both settings showed that the reason for this behavior can be explained by the choice of the subspace SVD makes when compressing the matrix better than the reduction of the operator norm it achieves, explaining roughly 76% of the effect at initialization and 83% on the pretrained weights, providing a refinement to the calibration-aware compression viewpoint and an explanation of why it outperformed naive SVD truncation.

Figures & tables

StudyMethodReported Effect
Hu et al. [ 7 ]LoRA∼\sim 10,000 ×\times fewer trainable params vs. full fine-tuning; no added inference latency
Saxena et al. [ 12 ]Eigen AttentionUp to 40% KV-cache reduction; up to 60% attention-latency reduction
Kusupati et al. [ 17 ]MRLUp to 14 ×\times smaller embeddings; up to 14 ×\times retrieval speed-up
Wang et al. [ 11 ]SVD-LLMImproved accuracy retention vs. magnitude-based SVD at high compression
Chang et al. [ 14 ]PALUReduced KV-cache memory with smaller accuracy loss than uniform-rank baselines
Dong et al. [ 4 ]Rank-collapse boundDoubly-exponential convergence to rank-1, O(γ3l)O(\gamma^{3^{l}})
TABLE I: Quantitative effects reported in reviewed studies
Fig. 1: Effective rank vs. layer depth for GPT-2 (124M): random initialization (left) vs. pretrained weights (right), original vs. SVD-compressed ( r=0.125r{=}0.125 ) attention projections.
Fig. 2: Causal decomposition of the collapse-rate reduction under SVD compression, random initialization (10 seeds, mean ±\pm SD).
Fig. 3: Causal decomposition on pretrained GPT-2 124M ( r=0.125r{=}0.125 , layer 8).
More in this topic

Low-Rank Compression

Low-rank compression reduces model size, memory use, and computation by representing weights, activations, or caches with compact factors.

See all 108 papers

Explore similar work

CardsList
Apr 2, 2026

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

We introduce a fast low-rank factorization-based framework for compressing large language models that enables rapid compression of billion-parameter models without retraining. Unlike existing factorization-based approaches that optimize only on the original inputs, ignoring distribution shifts from upstream compression and thus propagating errors forward, or those that rely only on shifted inputs and risk drifting away from the original outputs, our approach accounts for both. Beyond individual layer compression, we further refine each transformer block end-to-end, minimizing block-level output distortion and allowing compressed layers to jointly compensate for accumulated errors. By anchoring each compressed layer to the original outputs while explicitly modeling input distribution shifts, our method finds a low-rank approximation that maintains functional equivalence with the original model. Experiments on large language models show that our method consistently outperforms existing SVD-based baselines across compression ratios, with the advantage becoming increasingly pronounced at aggressive compression budgets, where competing methods degrade substantially or collapse entirely, offering a practical solution for efficient, large-scale model deployment.

Atul Kumar Sinha, François Fleuret

University of Geneva, Geneva, Switzerland · FAIR, Meta

LLM CompressionLow-Rank CompressionLow-Rank ApproximationSingular Value Decomposition
AskPDF
Apr 2, 2026

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

Ruoling Qi, Yirui Liu, Xuaner Wu +6

Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2

LLM CompressionLow-Rank CompressionNeural Network CompressionLow-Rank ApproximationSingular Value Decomposition
AskPDF
Oct 26, 2026

SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers

Pretrained transformers spend a large share of their inference compute inside self-attention, yet the per-head query-key interaction is structurally low-rank by construction. We ask a diagnostic question: across layers and architectures, which heads actually use the full rank budget on real data, and which do not? We propose SVD-Attention, a post-hoc method that decomposes the per-head interaction matrix MhM_h and scores every singular mode by both its spectral magnitude and its activation variance across tokens on a small calibration set. The score keeps the modes the head can amplify and that real data actually activates, and discards the rest; the resulting factorization is exact for the retained modes and needs no finetuning. Across seven pretrained models and five task types, the diagnostic finds a task-dependent pattern: redundancy sits in the early to middle layers for sentiment and image classification and for generation, and in the late layers for dense prediction and extractive question answering. It is also predictive: truncating the layers it flags costs at most 1.751.75 F1, while truncating the same number of layers elsewhere in the same model costs up to 5050 F1. SVD-Attention is an analysis tool rather than a compression method: acting on it reduces inference FLOPs by 0.80.8-1.5%1.5\% on its own and by 88-11%11\% when composed with weight pruning, with 00-4%4\% change in task quality.

Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan +5

Georgia Institute of Technology · Dolby Laboratories

Transformer AttentionAttention Head AnalysisLow-Rank Attention
AskPDF