Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

Ruoling Qi; Yirui Liu; Xuaner Wu; Xiangyu Wang; Ming Li; Chen Chen; Jian Chen; Yin Chen

cs.CLApr 2, 2026

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

Authors: Ruoling Qi, Yirui Liu, Xuaner Wu, Xiangyu Wang, Ming Li, Chen Chen, Jian Chen, Yin Chen,

Organizations: Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland · University at Buffalo ·

Abstract

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

Figures & tables

Figure 1: Swift-SVD for static weights and KV cache reduction.

Figure 2: Overview of Swift-SVD . a) Optimal Activation-Aware Low-Rank Compression: At each transformer layer, Swift-SVD hooks the output activation

Y=XW

and incrementally aggregates the covariance matrix

Y^{T}Y

. A single eigenvalue decomposition of this covariance yields the singular values

\Sigma

and right singular vectors

\mathcal{V}

, from which the optimal activation-aware compression matrix

W^{*}_{k}

and minimal reconstruction loss

\epsilon^{*}_{k}

are derived; b) Dynamic Compression: Swift-SVD generates a set of candidate dynamic rank allocation schemes that jointly consider local layer-wise Frobenius loss

\epsilon^{*}

and end-to-end layer importance

\beta

. A lightweight grid search is then performed over these candidates—each model is compressed using the optimal solution in a) and evaluated on a validation set—to select the configuration that yields the best end-to-end performance.

Figure 3: Layer-wise NER across distinct modules and layer importance in Mistral-7B with dataset C4.

Ratio (MEM.)	Method	PPL ( $\downarrow$ )		Accuracy ( $\uparrow$ )
		WikiText-2	C4	ARC_e	PIQA	Openb.	WinoG.	HellaS.	MathQA	Avg.
1.0(12.6GB)	Baseline	5.68	7.34	0.76	0.79	0.34	0.70	0.57	0.27	0.57
0.8 (10.1 GB)	FWSVD	1727	1511	0.11	0.10	0.09	0.05	0.08	0.05	0.08
	ASVD	11.14	15.93	0.53	0.68	0.29	0.64	0.41	0.17	0.45
	SVD-LLM(W)	7.94	15.84	0.62	0.71	0.31	0.61	0.45	0.21	0.49
	Dobi-SVD(w/o)	8.87	10.91	-	-	-	-	-	-	-

Table 1: Performance comparison of LLaMA-7B on language modeling and zero-shot tasks. Baseline results are reported from their original papers. We mark the best and second-best results. Regarding specific variants: Swift-SVD employs a uniform rank allocation strategy, while Swift-SVD* utilizes our proposed dynamic compression. SVD-LLM(W) denotes the uniform compression of SVD-LLM with truncation-aware whitening only, and Dobi-SVD(w/o) / (w) represent the method without / with dynamic rank allocation.

Model	OPT-6.7B			LLAMA 2-7B			MISTRAL-7B
Method	Perplexity $\downarrow$		Acc $\uparrow$	Perplexity $\downarrow$		Acc $\uparrow$	Perplexity $\downarrow$		Acc $\uparrow$
Method	Wiki2	C4	Avg.	Wiki2	C4	Avg.	Wiki2	C4	Avg.
Original	10.86	12.52	0.52	5.47	9.30	0.57	5.25	9.28	0.61
ASVD	82.04	102	0.32	10.10	24.02	0.36	13.72	23.34	0.32
SVD-LLM (W)	16.04	21.27	0.41	8.50	12.69	0.53	10.21	13.17	0.42
Swift-SVD	12.12	17.93	0.50	8.41	12.54	0.56	7.40	12.80	0.54

Table 2: Cross-model compression performance under 0.8 compression ratio. Baseline results are reported from their original papers. Results show PPL on WikiText-2/C4 and average accuracy on six common sense reasoning benchmarks.

nsamples = 256				nsamples = 2048
Ratio	Datasets	Original	PPL	Ratio	Datasets	Original	PPL
0.8	C4	11.42	11.42	0.8	C4	11.34	11.34
	WikiText 2	7.86	11.33		WikiText 2	7.81	11.31
	Alpaca	8.49	10.51		Alpaca	7.84	10.38
0.6	C4	23.17	23.17	0.6	C4	22.37	22.37
	WikiText 2	13.42	37.00		WikiText 2	13.37	35.02

Table 3: Cross-domain PPL of LLaMA-7B. Original uses the evaluation set for calibration; PPL uses C4-only.

Figure 4: Impact of calibration sample size

N

on model performance. We report average accuracy on zero-shot tasks (left) and PPL on C4 (right) across three compression ratios.

Samples	Method	$\rho=0.8$	$\rho=0.6$	$\rho=0.4$	Total (s)	Speedup
16	Dobi-SVD(w/o)	960	650	373	1,983	$1.0\times$
	SVD-LLM (W)	542	489	503	1,534	$1.3\times$
	Swift-SVD	342	146	133	621	3.2 $\times$
64	Dobi-SVD(w/o)	3,823	2,551	1,508	7,882	$1.0\times$
	SVD-LLM (W)	555	530	565	1,650	$4.8\times$
	Swift-SVD	346	158	150	654	12.1 $\times$

Table 4: End-to-end compression latency (in seconds) evaluated on the C4 dataset. We report the total time required to compress the complete model.

Figure 5: Throughput improvement and memory efficiency under batch size of 16. The generated sequence length is 1024.

Input $\times$ Weight	$[128\times 128]^{2}$	$[1024\times 1024]^{2}$
Minimum	126.2506	841.1812
SVD-LLM	126.7102 (+0.4596)	854.4129 (+13.2317)
Dobi-SVD	128.5441 (+2.2935)	874.7711 (+33.5899)
Swift-SVD	126.2506 (+0.0000)	841.1812 (+0.0000)
Input $\times$ Weight	$[2048\times 2048]^{2}$	$[4096\times 4096]^{2}$
Minimum	1657.4801	3308.6428

Table 5: Reconstruction loss and absolute error for randomly generated matrices of varying shapes under ratio of 0.6 (FP32).

Strategy	LLaMA-7B	LLaMA-2-7B	OPT-6.7B	Mistral-7B
Swift-SVD	11.42	12.54	17.93	12.80
Swift-SVD(C)	16.04	22.16	20.69	22.87
Swift-SVD(I)	14.88	17.30	18.94	22.50
Swift-SVD † (C)	11.78	13.74	15.66	11.67
Swift-SVD † (I)	11.73	13.27	14.74	11.48
Swift-SVD*	11.15	12.35	13.68	11.08

Table 6: PPL (

\downarrow

) of compressed LLMs of different allocation strategies across various models under the ratio of 0.8 on C4.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Impact of hyperparameters

\alpha

and

\delta

on model performance for dynamic compression. Both configurations exhibit a U-shaped PPL curve.

Model	Ratio	Aware	ARC_e	ARC_c	PIQA	Openb.	WinoG.	mathQA	HellaS.	Avg
Llama-7B	Original	Object	0.76	0.42	0.79	0.34	0.70	0.27	0.67	0.56
	0.8	C4	0.64	0.33	0.73	0.26	0.67	0.23	0.47	0.48
		All	0.70	0.38	0.76	0.29	0.68	0.26	0.49	0.51
		Each	0.71	0.38	0.76	0.30	0.68	0.26	0.49	0.51
	0.6	C4	0.49	0.25	0.66	0.21	0.62	0.22	0.37	0.40
		All	0.61	0.31	0.72	0.25	0.62	0.25	0.40	0.45

Appendix

Table B.1 : Benchmark results categorized by compression ratios and aware methods. The Original row is highlighted with gray text.

Ratio	Method	PPL $\downarrow$		Accuracy $\uparrow$
Ratio	Method	Wiki	C4	ARC-e	ARC-c	PIQA	Openb.	WinoG.	MathQA	HellaS.	Avg
1.0	Original	5.68	7.34	0.76	0.38	0.79	0.34	0.70	0.27	0.51	0.64
0.8	Basis Sharing	7.74	15.03	0.66	0.36	0.71	0.28	0.66	0.25	0.46	0.56
0.8	Swift-SVD	7.71	11.27	0.67	0.35	0.73	0.28	0.68	0.24	0.47	0.57
0.6	Basis Sharing	12.39	41.28	0.52	0.27	0.62	0.22	0.61	0.23	0.35	0.47
0.6	Swift-SVD	12.69	22.10	0.53	0.25	0.63	0.23	0.63	0.23	0.36	0.48

Appendix

Table B.2 : Perplexity and zero-shot accuracy comparison between Basis Sharing and Swift-SVD on LLaMA-7B (FP32). The Original row is highlighted with gray text.

Ratio	Method	ARC-e	PIQA	Openb.	WinoG.	HellaS.	MathQA	Avg
1.0	Original	0.81	0.75	0.30	0.66	0.52	0.40	0.57
0.8	SVD-LLM(W)	0.64	0.62	0.24	0.57	0.40	0.27	0.46
	Dobi-SVD(w/o)	0.62	0.61	0.23	0.55	0.39	0.26	0.44
	Dobi-SVD(w)	0.31	0.60	0.19	0.55	0.30	0.21	0.36
	Swift-SVD	0.69	0.70	0.26	0.61	0.42	0.29	0.50
	Swift-SVD(C)	0.69	0.69	0.26	0.60	0.41	0.27	0.49

Appendix

Table B.3 : Zero-shot accuracy on Qwen3-4B. Baseline results are reported from their original papers. The Original row is highlighted with gray text.

Ratio	Method	PPL $\downarrow$		Accuracy $\uparrow$
Ratio	Method	Wiki	C4	ARC-e	ARC-c	PIQA	Openb.	WinoG.	MathQA	HellaS.	Avg
1.0	Original	7.60	12.40	0.25	0.23	0.50	0.36	0.73	0.21	0.64	0.42
0.8	Naive SVD	—	—	0.27	0.23	0.60	0.21	0.51	0.21	0.30	0.33
0.8	Swift-SVD	9.38	15.45	0.79	0.49	0.78	0.36	0.73	0.43	0.58	0.59
0.6	Naive SVD	—	—	0.25	0.22	0.55	0.16	0.50	0.19	0.26	0.30
0.6	Swift-SVD	12.59	22.99	0.73	0.42	0.74	0.29	0.72	0.30	0.49	0.53

Appendix

Table B.4 : Perplexity and zero-shot accuracy of Swift-SVD on Qwen-32B (single H800, 80GB). Naive SVD denotes direct low-rank decomposition without activation awareness. PPL results for Naive SVD are omitted as they are too large to be informative. The Original row is highlighted with gray text.

Figure B.1 : Singular value distribution of Key (left) and Value (right) modules in Llama-7B. The y-axis is presented on a logarithmic scale to visualize the magnitude differences across layers.

Panel of Figure 18 — Figure B.2 : Throughput improvement and memory efficiency under different batch sizes. The generated sequence length is 1024.

	Uniform Compression			Dynamic Compression
Ratio	Swift-SVD	SVD-LLM(W)	Dobi-SVD(w/o)	Swift-SVD*	Swift-SVD(C)	Dobi-SVD(w)
0.8	192.9	192.7	193.3	193.7	191.4	104.4
0.6	209.2	208.1	207.8	207.3	206.3	117.5
0.4	216.8	216.5	216.8	216.0	215.4	131.6

Appendix

Table B.5 : Serving throughput (tokens/sec) on LLaMA-2-7B (H800) under uniform and dynamic rank allocation.

Ratio	Uniform rank	q_proj	k_proj	v_proj	o_proj	gate_proj	up_proj	down_proj	Throughput
0.8	1638	1627	1661	1743	1588	2365	2379	2385	101.3
0.6	1229	1213	1230	1294	1189	1773	1781	1812	113.4
0.4	819	794	775	839	773	1207	1206	1227	130.8

Appendix

Table B.6 : Per-type average ranks from Dobi-SVD(w)’s dynamic allocation on LLaMA-2-7B, alongside the uniform rank baseline. The Throughput column reports serving throughput when compressing with these per-type average ranks, confirming that the enlarged k_proj and v_proj ranks are the primary cause of throughput degradation.

	Length 1024		Length 8192
Ratio	Throughput	Gain	Throughput	Gain
1.0	438.86	—	140.39	—
0.8	457.31	$+$ 4%	144.98	$+$ 3%
0.6	473.38	$+$ 8%	169.17	$+$ 20%
0.4	497.29	$+$ 12%	181.23	$+$ 29%
0.2	551.04	$+$ 26%	187.45	$+$ 34%

Appendix

Table B.7 : Throughput (tokens/sec) and gain over uncompressed baseline on Mistral-7B (H800, bs=16, prompt length 32) at generation lengths 1024 and 8192.

Panel of Figure 22 — Figure B.3 : Layer-wise NER and Importance comparison (Part I).

Panel of Figure 23 — Figure B.4 : Layer-wise NER and Importance comparison (Part II).

RATIO	WikiText-2	C4
Original	What color is the sky on a clear day? On a clear day, the sky appears blue due to the way sunlight scatters through the atmosphere, a phenomenon known as Rayleigh scattering.
0.8	What color is the sky on a clear day? Blue, green or white. The answer to this question should be blue. Because of the atmosphere and sunlight, it’s blue. However some people might think that it’s green because they are looking at trees in front of the sky. Others may say that it’s white due to clouds. But actually it’s blue. The same way as with other colors like red or yellow, the color of the sky can change depending on weather conditions.	What color is the sky on a clear day? The sky is blue on a clear day. The sky appears blue because sunlight is scattered by the atmosphere.
0.6	What color is the sky on a clear day? Blue or white. () in what colors are to be used for the sky. The sun and moon have their own light, but it’s not that they can change into red or green, which appears as dark clouds when we observe them from high altitude. In this case of being able to see all skies, these colors would be so different with such great brightness. For example , you could imagine that some people might find something similar to the sky’s color. I think perhaps the sky is blue."	What color is the sky on a clear day? The answer is blue. But this question might be confusing because it’s not possible to see what colors are in the atmosphere when you look at it from Earth. So I need to find out what happens if there’s no clouds, and then how much light would reflect into the atmosphere? If we have some cloud cover over the earth and only sunlight comes through, that could make an effect of reflection by something like reflected sunlight or light being absorbed.
0.4	What color is the sky on a clear day? It has blue colors. The sky’s white coloring in black, with yellow hues of purple that are gray. It s colored as red, and its colors have green tones which are dark. This landscape area is brown to some areas where they are not covered by the natural vegetation but they contain many small plants such trees and other species like plant wood. With these trees having a forest structure from their tree and being surrounded by various species including woodland.	What color is the sky on a clear day? Options: - A) white, b ) green , c ) blue , d ) yellow. The answer will be blue.
Original	What is the responsibility of the AI assistant? The responsibility of an AI assistant primarily lies in providing accurate, helpful, and ethical assistance to users. This includes adhering to guidelines, maintaining privacy, avoiding harmful or biased content, and ensuring that the information provided is reliable and up-to-date. Additionally, AI assistants should respect user autonomy, offer transparency about their capabilities and limitations, and avoid any form of deception or manipulation.
0.8	What is the responsibility of the AI assistant? First, to ensure that its responses are accurate and reliable, in accordance with human standards. The second task of AI assistants is to be a helpful companion for humans. This means they can help people by providing information about their own lives, or helping them solve problems. In addition, AI assistance can serve as an aid to enhance intelligence through learning from data collected by other AI systems. And finally, AI assistants may also have responsibilities related to ethical issues such as privacy protection.	What is the responsibility of the AI assistant? The AI assistant is responsible for providing accurate information and helping users with their questions.

Appendix

Table B.8 : Comparison of AI assistant responses across different datasets and compression ratios.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 273 papers

Jul 3, 2026

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).

Zhuowen Liu, Longkun Hao, Shiyu Feng +3

Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3

Model Compression LLM Compression Low-Rank Matrix Decomposition Low-Rank Compression Low-Rank Approximation Singular Value Decomposition

Ask PDF

May 15, 2026

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

Large language models deliver strong performance across language and reasoning tasks, but their storage and compute costs remain major barriers to deployment in resource-constrained and latency-sensitive settings. SVD-based post-training compression offers a hardware-agnostic way to reduce model size and improve inference efficiency through low-rank factorization. However, existing methods often rely on input-only whitening spaces, homogeneous rank allocation, or loss-agnostic allocation heuristics, limiting their ability to preserve model quality under aggressive compression. We propose Input-Output Whitened SVD (IO-SVD), a post-training compression method that forms a KL-aware double-sided whitening space for model weights. Using a second-order expansion of the KL loss over the top-K token probabilities, IO-SVD constructs an output-side metric that captures predictive sensitivity, while input whitening captures activation statistics. We further introduce an efficient heterogeneous rank-allocation strategy that scores whitened singular components using first-order calibration loss estimates and prunes the least sensitive components under a global budget. Inspired by prior work that combines SVD truncation with quantization, we improve hybrid SVD-quantization compression through loss-aware remapping, which selects low-rank factor rows for 8-bit quantization based on the predicted loss change incurred by quantizing them. Extensive experiments across diverse LLM and VLM families, and inference-time analysis shows that IO-SVD compresses LLMs with minimal performance degradation while delivering practical inference speedups. Code is available at https://github.com/mint-vu/IO-SVD.git

Ali Abbasi, Chayne Thrash, Haoran Qin +2

Vanderbilt University · University of California, Davis

LLM Compression Low-Rank Matrix Decomposition LLM Inference Acceleration Low-Rank Compression Neural Network Compression

Ask PDF

Apr 2, 2026

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

We introduce a fast low-rank factorization-based framework for compressing large language models that enables rapid compression of billion-parameter models without retraining. Unlike existing factorization-based approaches that optimize only on the original inputs, ignoring distribution shifts from upstream compression and thus propagating errors forward, or those that rely only on shifted inputs and risk drifting away from the original outputs, our approach accounts for both. Beyond individual layer compression, we further refine each transformer block end-to-end, minimizing block-level output distortion and allowing compressed layers to jointly compensate for accumulated errors. By anchoring each compressed layer to the original outputs while explicitly modeling input distribution shifts, our method finds a low-rank approximation that maintains functional equivalence with the original model. Experiments on large language models show that our method consistently outperforms existing SVD-based baselines across compression ratios, with the advantage becoming increasingly pronounced at aggressive compression budgets, where competing methods degrade substantially or collapse entirely, offering a practical solution for efficient, large-scale model deployment.

Atul Kumar Sinha, François Fleuret

University of Geneva, Geneva, Switzerland · FAIR, Meta

LLM Compression Low-Rank Compression Low-Rank Approximation Singular Value Decomposition

Ask PDF

Abstract

LLM Compression

Explore similar work

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression