AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

Atul Kumar Sinha; François Fleuret

cs.LGApr 2, 2026

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

Authors: Atul Kumar Sinha, François Fleuret

Organizations: University of Geneva, Geneva, Switzerland · FAIR, Meta

Abstract

We introduce a fast low-rank factorization-based framework for compressing large language models that enables rapid compression of billion-parameter models without retraining. Unlike existing factorization-based approaches that optimize only on the original inputs, ignoring distribution shifts from upstream compression and thus propagating errors forward, or those that rely only on shifted inputs and risk drifting away from the original outputs, our approach accounts for both. Beyond individual layer compression, we further refine each transformer block end-to-end, minimizing block-level output distortion and allowing compressed layers to jointly compensate for accumulated errors. By anchoring each compressed layer to the original outputs while explicitly modeling input distribution shifts, our method finds a low-rank approximation that maintains functional equivalence with the original model. Experiments on large language models show that our method consistently outperforms existing SVD-based baselines across compression ratios, with the advantage becoming increasingly pronounced at aggressive compression budgets, where competing methods degrade substantially or collapse entirely, offering a practical solution for efficient, large-scale model deployment.

Figures & tables

Figure 1: Distortion (cosine distance) between intermediate features of the original and compressed model. Diagonal lines link each method’s final-layer distortion to its WikiText2 perplexity. AA-SVD suppresses compression error consistently across depth.

Panel of Figure 2 — Figure 2: Overview of the two-stage compression pipeline. Left: Four layer-wise compression objectives, differing in which inputs and outputs are compared. 1 Input-agnostic : $\|{\bm{W}}-{\bm{W}}^{\prime}\|_{F}^{2}$ — ignores activations entirely. 2 Input-aware : $\|{\bm{W}}{\bm{X}}-{\bm{W}}^{\prime}{\bm{X}}\|_{F}^{2}$ — matches outputs on original inputs ${\bm{X}}$ . 3 Shift-aware : $\|{\bm{W}}{\bm{X}}^{\prime}-{\bm{W}}^{\prime}{\bm{X}}^{\prime}\|_{F}^{2}$ — matches outputs on the shifted inputs ${\bm{X}}^{\prime}$ seen after upstream compression. 4 Anchored adaptive (ours): $\|{\bm{W}}{\bm{X}}-{\bm{W}}^{\prime}{\bm{X}}^{\prime}\|_{F}^{2}$ — anchors the target to the original output while conditioning on the shifted input, combining an uncorrupted reference with distribution-shift awareness. Right: Block-level local refinement. Stage 1 factorizes all linear layers in the block independently via any layer-wise objective. Stage 2 then jointly optimizes all factorized weights to minimize the block-output error $\ell=\|\mathcal{L}({\bm{X}})-\mathcal{L}^{\prime}({\bm{X}}^{\prime})\|_{F}^{2}$ , keeping upstream blocks frozen — the same anchored adaptive spirit as 4 but applied at block granularity. This lets the compressed layers within a block compensate for each other’s residual errors, substantially recovering block-output fidelity.

Ratio	Method	PPL ( $\downarrow$ )			Accuracy ( $\uparrow$ )
Ratio	Method	Wiki2	PTB	C4	Openb.	ARC_e	ARC_c	WinoG.	PIQA	MathQA	HellaS.	Avg.	Drop (%)
$1.0$	Dense	$5.68$	$8.34$	$7.34$	$0.34$	$0.75$	$0.42$	$0.69$	$0.79$	$0.27$	$0.57$	$0.55$	$-$
$0.8$	ASVD	$11.14$	$16.55$	$15.93$	$0.25$	$0.53$	$0.27$	$0.64$	$0.68$	$0.24$	$0.41$	$0.43$	$21.1\%$
	SVD-LLM †	$7.94$	$16.22$	$15.84$	$0.22$	$0.58$	$0.29$	$0.63$	$0.69$	$0.24$	$0.43$	$0.44$	$19.6\%$
	Dobi-SVD ‡	$8.54$	$14.83$	$\mathbf{10.01}$	$0.26$	$0.59$	$0.31$	$\mathbf{0.66}$	$0.70$	$0.23$	$0.44$	$0.46$	$16.7\%$
	Dip-SVD ‡	$7.95$	$15.60$	$14.07$	$0.27$	$0.63$	$0.33$	$0.64$	$0.71$	$0.24$	$0.45$	$0.47$	$14.6\%$

Table 1: Comparison of AA-SVD with SOTA methods for SVD-based compression of LLaMA-7B on three language modeling tasks and seven commonsense reasoning benchmarks (zero-shot evaluation) under varying compression ratios. Best performance is marked in bold.

(^{\dagger})

uses LoRA fine-tuning,

(^{\ddagger})

uses dynamic or non-uniform capacity/rank allocation, and

(^{q})

indicates results with Dobi-SVD-style remapping enabled. Results for baseline methods are taken from the original papers or prior work where available.

Ratio	Method	LLaMA-2-7B		LLaMA-2-13B		LLaMA-3-1B		LLaMA-3-8B		Qwen-2.5-7B
Ratio	Method	PPL ( $\downarrow$ )	Acc. ( $\uparrow$ )	PPL ( $\downarrow$ )	Acc. ( $\uparrow$ )	PPL ( $\downarrow$ )	Acc. ( $\uparrow$ )	PPL ( $\downarrow$ )	Acc. ( $\uparrow$ )	PPL ( $\downarrow$ )	Acc. ( $\uparrow$ )
$1.0$	Baseline	$5.47$	$0.55$	$4.88$	$0.58$	$9.75$	$0.48$	$6.24$	$0.60$	$6.84$	$0.60$
$0.8$	SVD-LLM	$8.41$	$0.43$	$6.65$	$0.48$	$45.62$	$0.32$	$14.16$	$0.44$	$10.69$	$0.47$
	AA-SVD	$\mathbf{6.84}$	$\mathbf{0.50}$	$\mathbf{5.95}$	$\mathbf{0.53}$	$\mathbf{15.12}$	$\mathbf{0.39}$	$\mathbf{9.58}$	$\mathbf{0.50}$	$\mathbf{8.53}$	$\mathbf{0.53}$
$0.6$	SVD-LLM	$16.47$	$0.35$	$10.79$	$0.38$	$402.76$	$0.30$	$76.31$	$0.32$	$28.67$	$0.33$
	AA-SVD	$\mathbf{8.55}$	$\mathbf{0.44}$	$\mathbf{7.44}$	$\mathbf{0.46}$	$\mathbf{23.74}$	$\mathbf{0.35}$	$\mathbf{13.66}$	$\mathbf{0.41}$	$\mathbf{11.00}$	$\mathbf{0.44}$

Table 2: Comparison of AA-SVD with SOTA methods across multiple models at compression ratios

0.8

and

0.6

. PPL refers to WikiText2 perplexity; Accuracy is averaged over seven commonsense reasoning benchmarks (zero-shot). Best performance is marked in bold.

Ratio	Method	Accuracy ( $\uparrow$ )
Ratio	Method	PIQA	HellaS.	WinoG.	ARC_e	ARC_c	Avg.	Drop (%)
$1.0$	Dense	$0.78$	$0.57$	$0.69$	$0.76$	$0.43$	$0.65$	$-$
$0.6$	LLM-Pruner	$0.70$	$0.41$	$0.53$	$0.53$	$0.27$	$0.48$	$24.5\%$
	SliceGPT	$0.65$	$0.57$	$0.60$	$0.43$	$0.32$	$0.51$	$20.4\%$
	Bonsai	$0.72$	$0.45$	$0.58$	$0.59$	$0.30$	$0.53$	$18.3\%$
	Wanda-sp	$0.70$	$0.42$	$0.53$	$0.57$	$0.29$	$0.50$	$22.3\%$

Table 3: Comparison of AA-SVD with structured pruning methods on compression performance of LLaMA-2-7B across five commonsense reasoning benchmarks (zero-shot evaluation). Results for baseline methods are taken from Wang et al. (2025a) .

Memory	LLM-Pruner	SliceGPT	BlockPruner	SAES-SVD	AA-SVD (Ours)
10GB	$9.88$	$8.78$	$9.40$	${7.17}$	$\mathbf{6.89}$
9GB	$12.21$	$12.73$	$12.76$	${8.22}$	$\mathbf{7.14}$
8GB	$18.94$	$16.39$	$19.78$	${8.96}$	$\mathbf{7.84}$
7GB	$21.68$	$27.41$	$43.05$	${10.15}$	$\mathbf{8.35}$

Table 4: Perplexity (WikiText2,

\downarrow

) comparison of AA-SVD and structured pruning baselines on LLaMA-7B under different memory budgets. Results for baseline methods are taken from Hu et al. (2026) .

Ratio	Objective	Refinement	PPL ( $\downarrow$ )	Acc. ( $\uparrow$ )
$1.0$	Dense	$-$	$5.68$	$0.55$
$0.8$	$\\|\bm{W}-\bm{W}^{\prime}\\|_{F}^{2}$	✗	$2e4$	$0.31$
		✓	$7.35$	$0.50$
	$\\|\bm{W}{\bm{X}}-\bm{W}^{\prime}{\bm{X}}\\|_{F}^{2}$	✗	$7.89$	$0.45$
		✓	$\mathbf{6.89}$	$\mathbf{0.50}$
	$\\|\bm{W}{\bm{X}}^{\prime}-\bm{W}^{\prime}{\bm{X}}^{\prime}\\|_{F}^{2}$	✗	$8.22$	$0.45$

Table 5: WikiText2 Perplexity and average accuracy over seven commonsense reasoning tasks for AA-SVD with different layer-wise objectives and block-level refinement. Best performance is marked in bold.

Figure 3: Impact of calibration set size on compression performance. Performance is measured by perplexity on WikiText2 (left) and C4 (middle), and average accuracy across seven zero-shot reasoning tasks (right).

Panel of Figure 4 — Figure 4: Layer-wise error evolution across LLaMA-7B at ratio $0.8$ , evaluated on WikiText2 test split samples. Top row: MSE between original and compressed outputs. Bottom row: cosine distance between original and compressed outputs. Results are shown separately for attention output projections (O-proj), MLP-down projections, and full block outputs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Ratio	Method	PPL ( $\downarrow$ )			Accuracy ( $\uparrow$ )
Ratio	Method	Wiki2	PTB	C4	Openb.	ARC_e	ARC_c	WinoG.	PIQA	MathQA	HellaS.	Avg.	Drop (%)
$1.0$	Dense	$9.75$	$15.40$	$13.82$	$0.26$	$0.65$	$0.31$	$0.61$	$0.74$	$0.29$	$0.48$	$0.48$	$-$
$0.8$	SVD-LLM	$45.62$	$158.15$	$206.18$	$0.14$	$0.37$	$0.19$	$0.51$	$0.56$	$0.21$	$0.28$	$0.32$	$32.3\%$
	AA-SVD	$\mathbf{15.12}$	$\mathbf{36.81}$	$\mathbf{37.54}$	$\mathbf{0.20}$	$\mathbf{0.51}$	$\mathbf{0.23}$	$\mathbf{0.55}$	$\mathbf{0.64}$	$\mathbf{0.23}$	$\mathbf{0.36}$	$\mathbf{0.39}$	$\mathbf{18.6\%}$
$0.6$	SVD-LLM	$402.76$	$2027.07$	$1449.82$	$0.12$	$0.27$	$0.20$	$0.51$	$0.52$	$0.20$	$0.26$	$0.30$	$37.7\%$
	AA-SVD	$\mathbf{23.74}$	$\mathbf{72.00}$	$\mathbf{91.02}$	$\mathbf{0.19}$	$\mathbf{0.42}$	$\mathbf{0.22}$	$\mathbf{0.53}$	$\mathbf{0.58}$	$\mathbf{0.23}$	$\mathbf{0.30}$	$\mathbf{0.35}$	$\mathbf{26.1\%}$

Appendix

Table 6: Comparison of AA-SVD with SOTA methods for SVD-based compression of LLaMA-3-1B on three language modeling tasks and seven commonsense reasoning benchmarks (zero-shot evaluation). Best performance is marked in bold.

Ratio	Method	PPL ( $\downarrow$ )			Accuracy ( $\uparrow$ )
Ratio	Method	Wiki2	PTB	C4	Openb.	ARC_e	ARC_c	WinoG.	PIQA	MathQA	HellaS.	Avg.	Drop (%)
$1.0$	Dense	$5.47$	$24.09$	$7.28$	$0.32$	$0.76$	$0.43$	$0.69$	$0.78$	$0.28$	$0.57$	$0.55$	$-$
$0.8$	SVD-LLM	$8.41$	$119.32$	$20.34$	$0.26$	$0.57$	$0.26$	$0.62$	$0.66$	$0.24$	$0.39$	$0.43$	$21.7\%$
	AA-SVD	$\mathbf{6.84}$	$\mathbf{1486.20}$	$\mathbf{13.19}$	$\mathbf{0.30}$	$\mathbf{0.71}$	$\mathbf{0.37}$	$\mathbf{0.64}$	$\mathbf{0.72}$	$\mathbf{0.27}$	$\mathbf{0.48}$	$\mathbf{0.50}$	$\mathbf{8.9\%}$
	Dobi-SVD ‡,q	$5.92$	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$
	AA-SVD q	$5.92$	$30.78$	$8.41$	$0.31$	$0.74$	$0.42$	$0.69$	$0.77$	$0.29$	$0.55$	$0.54$	$\mathbf{1.6\%}$

Appendix

Table 7: Comparison of AA-SVD with SOTA methods for SVD-based compression of LLaMA-2-7B on three language modeling tasks and seven commonsense reasoning benchmarks (zero-shot evaluation). Best performance is marked in bold. ‡ uses dynamic or non-uniform ratio allocation, and q represents quantized parameters.

Ratio	Method	PPL ( $\downarrow$ )			Accuracy ( $\uparrow$ )
Ratio	Method	Wiki2	PTB	C4	Openb.	ARC_e	ARC_c	WinoG.	PIQA	MathQA	HellaS.	Avg.	Drop (%)
$1.0$	Dense	$6.24$	$9.89$	$9.57$	$0.34$	$0.82$	$0.52$	$0.74$	$0.80$	$0.40$	$0.60$	$0.60$	$-$
$0.8$	SVD-LLM	$14.16$	$64.01$	$79.14$	$0.24$	$0.59$	$0.30$	$0.64$	$0.66$	$0.26$	$0.37$	$0.44$	$27.5\%$
$0.8$	SAES-SVD	$11.49$	$-$	$-$	$0.25$	$0.59$	${0.28}$	$0.66$	${0.67}$	$0.27$	$0.39$	$0.44$	$26.3\%$
	AA-SVD	$\mathbf{9.58}$	$\mathbf{28.11}$	$\mathbf{33.12}$	$\mathbf{0.26}$	$\mathbf{0.70}$	$\mathbf{0.37}$	$\mathbf{0.69}$	$\mathbf{0.72}$	$\mathbf{0.30}$	$\mathbf{0.46}$	$\mathbf{0.50}$	$\mathbf{17.1\%}$
$0.6$	SVD-LLM	$76.31$	$971.56$	$662.65$	$0.14$	$0.32$	$0.19$	$0.52$	$0.55$	$0.21$	$0.28$	$0.32$	$47.6\%$

Appendix

Table 8: Comparison of AA-SVD with SOTA methods for SVD-based compression of LLaMA-3-8B on three language modeling tasks and seven commonsense reasoning benchmarks (zero-shot evaluation). Best performance is marked in bold.

Ratio	Method	PPL ( $\downarrow$ )			Accuracy ( $\uparrow$ )
Ratio	Method	Wiki2	PTB	C4	Openb.	ARC_e	ARC_c	WinoG.	PIQA	MathQA	HellaS.	Avg.	Drop (%)
$1.0$	Dense	$6.84$	$11.37$	$11.85$	$0.34$	$0.80$	$0.50$	$0.73$	$0.79$	$0.43$	$0.60$	$0.60$	$-$
$0.8$	SVD-LLM	$10.69$	$39.10$	$38.53$	$0.25$	$0.67$	$0.31$	$0.64$	$0.68$	$0.32$	$0.41$	$0.47$	$21.7\%$
	AA-SVD	$\mathbf{8.53}$	$\mathbf{22.90}$	$\mathbf{22.05}$	$\mathbf{0.31}$	$\mathbf{0.74}$	$\mathbf{0.41}$	$\mathbf{0.69}$	$\mathbf{0.73}$	$\mathbf{0.37}$	$\mathbf{0.49}$	$\mathbf{0.53}$	$\mathbf{10.7\%}$
$0.6$	SVD-LLM	$28.67$	$193.31$	$161.22$	$0.15$	$0.36$	$0.20$	$0.53$	$0.56$	$0.22$	$0.29$	$0.33$	$44.9\%$
	AA-SVD	$\mathbf{11.00}$	$\mathbf{49.10}$	$\mathbf{40.85}$	$\mathbf{0.25}$	$\mathbf{0.59}$	$\mathbf{0.28}$	$\mathbf{0.61}$	$\mathbf{0.65}$	$\mathbf{0.28}$	$\mathbf{0.39}$	$\mathbf{0.44}$	$\mathbf{27.2\%}$

Appendix

Table 9: Comparison of AA-SVD with SOTA methods for SVD-based compression of Qwen-2.5-7B on three language modeling tasks and seven commonsense reasoning benchmarks (zero-shot evaluation). Best performance is marked in bold.

Ratio	Method	PPL ( $\downarrow$ )			Accuracy ( $\uparrow$ )
Ratio	Method	Wiki2	PTB	C4	Openb.	ARC_e	ARC_c	WinoG.	PIQA	MathQA	HellaS.	Avg.	Drop (%)
$1.0$	Dense	$4.88$	$34.40$	$6.73$	$0.35$	$0.79$	$0.48$	$0.72$	$0.79$	$0.32$	$0.60$	$0.58$	$-$
$0.8$	SVD-LLM	$6.65$	$84.17$	$14.99$	$0.29$	$0.67$	$0.33$	$0.68$	$0.71$	$0.26$	$0.44$	$0.48$	$16.5\%$
	AA-SVD	$\mathbf{5.95}$	$\mathbf{46.43}$	$\mathbf{11.6}$	$\mathbf{0.33}$	$\mathbf{0.73}$	$\mathbf{0.40}$	$\mathbf{0.69}$	$\mathbf{0.74}$	$\mathbf{0.29}$	$\mathbf{0.52}$	$\mathbf{0.53}$	$\mathbf{8.9\%}$
$0.6$	SVD-LLM	$10.79$	$243.85$	$46.47$	$0.22$	$0.47$	$0.22$	$0.61$	$0.60$	$0.23$	$0.33$	$0.38$	$33.8\%$
	AA-SVD	$\mathbf{7.44}$	$\mathbf{79.01}$	$\mathbf{19.32}$	$\mathbf{0.25}$	$\mathbf{0.64}$	$\mathbf{0.32}$	$\mathbf{0.63}$	$\mathbf{0.68}$	$\mathbf{0.26}$	$\mathbf{0.41}$	$\mathbf{0.46}$	$\mathbf{21.2\%}$

Appendix

Table 10: Comparison of AA-SVD with SOTA methods for SVD-based compression of LLaMA-2-13B on three language modeling tasks and seven commonsense reasoning benchmarks (zero-shot evaluation).

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 273 papers

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
Apr 2, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +6LLM Compression Low-Rank Compression
Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Jul 3, 2026Zhuowen Liu, Longkun Hao, Shiyu Feng +3Model Compression LLM Compression
Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Jun 22, 2026Mahmoud Safari, Frank HutterLLM Pruning LLM Compression
University of Freiburg · Prior Labs · ELLIS Institute Tübingen

Apr 2, 2026J/K move · Enter open · S save

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

Ruoling Qi, Yirui Liu, Xuaner Wu, Xiangyu Wang+5

Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland+2

LLM Compression Low-Rank Compression Neural Network Compression Low-Rank Approximation Singular Value Decomposition

Open paper AskPDF arXiv

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

Abstract

LLM Compression

Explore similar work

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression