GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression

Shengkun Tang; Cong Zeng; Zongfang Liu; Dan Alistarh; Zhiqiang Shen

cs.LGOct 11, 2026

GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression

Authors: Shengkun Tang, Cong Zeng, Zongfang Liu, Dan Alistarh, Zhiqiang Shen

Organizations: MBZUAI · Technology Innovation Institute UAE · Zhejiang Univeristy · Westlake University ·

Abstract

Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost effectively. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank allocation under a given global compression ratio. Across extensive experiments on Mistral-7B, Qwen3 series, and Llama-3.3 models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms existing SVD-based baselines and achieves state-of-the-art (SoTA) results. We further demonstrate scalability by applying our method to models with up to 70B parameters.

Figures & tables

Panel of Figure 1 — Figure 1: Existing SVD-based compression degrades generation performance. Compared with MCQ benchmarks such as MMLU, SVDLLM suffers more severe degradation on generation-oriented benchmarks such as MBPP. GA-SVD substantially mitigates this degradation across different models.

Figure 2: Visual illustration of the GA-SVD pipeline. GA-SVD first collects on-policy generation-aware statistics from a warm-start compressed student using teacher-student distillation signals. It then performs Generation-Aware Weight Whitening (GAWW), where input activations and output gradients define a bi-whitened SVD objective for each compressed linear layer. Finally, Evolutionary Global Rank Allocation searches for an effective layer-wise rank configuration under a global compression budget, producing the final GA-SVD compressed LLM.

Methods	Coding $\uparrow$			Math $\uparrow$	MCQ $\uparrow$					PPL Avg $\downarrow$
Methods	HumanEval	MBPP	Code Avg	GSM8K	PIQA	MMLU	ARC-C	HS	Avg	PPL Avg $\downarrow$
Baseline	29.3	51.6	40.4	40.4	82.1	62.6	61.3	83.3	72.3	13.6
ASVD ( Yuan et al., 2023 )	0.0	6.6	3.3	1.5	67.4	27.0	28.6	40.7	40.9	96.1
SVDLLM ( Wang et al., 2025c )	15.2	28.6	21.9	14.9	65.6	39.8	32.2	44.3	45.4	50.4
SVDLLM v2 ( Wang et al., 2025b )	18.9	28.3	23.6	17.8	68.5	40.0	35.6	48.9	48.2	41.7
Dobi-SVD ( Wang et al., 2025a )	15.2	29.1	22.1	15.6	69.1	43.3	33.6	51.7	49.4	39.0

Table 1: Performance comparison of GA-SVD and baselines on both MCQ and generative benchmarks. The results are from Mistral-7B model with 20% compression ratio. GA-SVD consistently outperforms SVD-based baselines across coding, math, MCQ benchmarks, and perplexity.

Model	Methods	Coding $\uparrow$			Math $\uparrow$	MCQ $\uparrow$					PPL Avg $\downarrow$
Model	Methods	HumanEval	MBPP	Code Avg	GSM8K	PIQA	MMLU	ARC-C	HS	Avg	PPL Avg $\downarrow$
Qwen3-8B	Baseline	86.0	70.9	78.4	88.7	77.6	74.9	67.3	77.6	74.3	13.5
	SVDLLM	0.0	0.0	0.0	1.4	62.9	34.8	30.9	45.6	43.5	46.0
	GA-SVD (Ours)	9.1	23.8	16.5	8.4	65.4	45.4	33.3	47.1	47.8	31.2
Qwen3-14B	Baseline	87.0	89.0	88.0	91.2	79.8	78.8	69.8	79.9	77.0	12.0
	SVDLLM	19.0	18.9	19.0	11.6	68.2	50.3	40.3	51.5	52.5	30.0

Table 2: Performance of GA-SVD on different model series including Qwen3 and Llama3.3. GA-SVD consistently improves over SVDLLM on Qwen3 models from 8B to 32B and scales to Llama-3.3-70B-Instruct, improving both generation and MCQ performance while reducing perplexity.

Ratio	Methods	Coding $\uparrow$			Math $\uparrow$	MCQ $\uparrow$					PPL Avg $\downarrow$
Ratio	Methods	HumanEval	MBPP	Code Avg	GSM8K	PIQA	MMLU	ARC-C	HS	Avg	PPL Avg $\downarrow$
20%	Baseline	29.3	51.6	40.4	40.4	82.1	62.6	61.3	83.3	72.3	13.6
	SVDLLM	15.2	28.6	21.9	14.9	65.6	39.8	32.2	44.3	45.4	50.4
	GA-SVD (Ours)	20.7	34.1	27.4	18.5	69.9	47.9	36.1	53.0	51.7	37.8
40%	SVDLLM	8.5	11.1	9.8	5.3	56.3	27.5	24.3	31.8	34.9	137.4
40%	GA-SVD (Ours)	10.2	16.1	13.1	6.2	58.4	28.9	25.9	34.9	37.0	97.5

Table 3: Performance on different compression ratios. GA-SVD consistently outperforms SVD-LLM at both 20% and 40% compression, showing stronger performance across coding, math, MCQ benchmarks, and perplexity.

Panel of Figure 3 — Figure 3: Layer-wise Rank Allocation. The allocated ranks vary substantially across layers and weight matrices, indicating that uniform rank assignment is suboptimal for low-rank LLM compression.

Table 7

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$M_{T}$	Full-precision teacher model.
$M_{S}$	Compressed student model.
$M_{S}^{(0)}$	Warm-start compressed student model.
$\mathcal{L}$	Set of linear layers to compress.
$W_{l}^{\star}$	Original full-precision weight matrix of layer $l$ .
$W_{c,l}$	Low-rank compressed approximation of $W_{l}^{\star}$ .

Appendix

Table 6: Summary of notation used in GA-SVD.

Method	ROUGE-1 $\uparrow$	ROUGE-2 $\uparrow$
Dense	30.07	13.60
SVD-LLM	22.62	9.64
GA-SVD	24.98	11.50

Appendix

Table 7: Performance on abstractive summarization.

Methods	PPL $\downarrow$			MCQ $\uparrow$
Methods	Wiki	C4	PTB	ARC-E	ARC-C	PIQA	HS	MMLU	Avg
Baseline	5.2	8.1	27.7	79.5	61.3	28.1	83.3	62.6	63.0
ASVD	20.5	30.8	237.1	53.4	28.6	67.4	40.7	27	43.4
SVD-LLM	12.4	24.6	114.2	55.4	32.2	65.6	44.3	39.8	47.5
SVD-LLM v2	10.6	22.8	91.7	63.5	35.6	68.5	48.9	40.0	51.3
Dobi-SVD	14.9	21.8	80.1	64.6	33.6	69.1	51.7	43.3	49.4

Appendix

Table 8: Detailed perplexity and MCQ benchmark results on Mistral-7B with 20% compression. We report perplexity on WikiText, C4, and PTB, together with additional MCQ evaluations on ARC-E, ARC-C, PIQA, HellaSwag, and MMLU. GA-SVD achieves the best MCQ average among compressed models while maintaining competitive perplexity, demonstrating a strong overall trade-off between language modeling quality and downstream task performance.

Warmup Ratio	HumanEval	MBPP	Avg	GSM8K	PIQA	MMLU	ARC-C	HS	Avg
5%	21.5	34.8	28.1	19.2	68.6	48.2	36.5	53.6	51.7
10%	21.0	34.1	27.5	18.2	70.2	47.5	37.1	52.4	51.8
20%	20.7	34.1	27.4	18.5	69.9	47.9	36.1	53.0	51.7

Appendix

Table 9: Performance with different warmup ratios.

Methods	HumanEval	MBPP	Avg	GSM8K	PIQA	MMLU	ARC-C	HS	Avg
Wanda	4.3	4.8	4.5	1.4	70.1	24.8	27.1	37.6	39.9
SparseGPT	0	0	0	3.2	76.9	38.4	32.5	43.2	47.7
GA-SVD	20.7	34.1	27.4	18.5	69.9	47.9	36.1	53.0	51.7
GA-SVD + GPTQ 4bit	19.3	32.4	25.8	17.8	68.2	46.2	35.7	52.1	50.5

Appendix

Table 10: Performance comparison of different compression methods.

Calibration Settings		HumanEval	MBPP	Avg	GSM8K	PIQA	MMLU	ARC-C	HS	Avg
Samples	Rollout Length
256	512	20.7	34.1	27.4	18.5	69.9	47.9	36.1	53.0	51.7
128	512	19.3	33.7	26.5	17.9	69.1	46.3	36.2	52.6	51.0
256	256	19.7	34.5	27.1	18.6	70.2	47.2	37.2	52.8	51.8

Appendix

Table 11: Sensitivity to the number of calibration samples and rollout length.

Method	Compression	HumanEval	MBPP	Avg	GSM8K	PIQA	MMLU	ARC-C	HS	Avg
SVD-LLM	10%	21.1	20.5	20.8	20.3	63.6	47.2	35.8	50.1	49.1
GA-SVD	10%	31.0	39.6	35.3	28.2	69.0	53.0	42.9	54.6	54.8
SVD-LLM	20%	0.0	0.0	0.0	1.4	62.9	34.8	30.9	45.6	43.5
GA-SVD	20%	9.1	23.8	16.5	8.4	65.4	45.4	33.3	47.1	47.8

Appendix

Table 12: Performance of Qwen3-8B at different compression ratios.

Panel of Figure 4 — Figure 4: Layer-wise Rank Allocation of Mistral-7B.

Panel of Figure 5 — Figure 5: Layer-wise Rank Allocation of Qwen3-8B.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 275 papers

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Jul 3, 2026Zhuowen Liu, Longkun Hao, Shiyu Feng +3Model Compression LLM Compression
Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM Compression Low-Rank Matrix Decomposition
Vanderbilt University · University of California, Davis
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Jun 22, 2026Mahmoud Safari, Frank HutterLLM Pruning LLM Compression
University of Freiburg · Prior Labs · ELLIS Institute Tübingen

Jul 3, 2026J/K move · Enter open · S save

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

Zhuowen Liu, Longkun Hao, Shiyu Feng, Xiaowen Chang+2

Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3+3

Model Compression LLM Compression Low-Rank Matrix Decomposition Low-Rank Compression Low-Rank Approximation Singular Value Decomposition

Open paper AskPDF arXiv

The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).

Abstract

LLM Compression

Explore similar work

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression