Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

Moritz Thoma; Maximilian Groezinger; Maximilian Forstenhäusler; Emad Aghajanzadeh; Manoj Rohit Vemparala; Christos Anagnostopoulos; Pierpaolo Mori; Nael Fasfous

cs.AIOct 23, 2026

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

Authors: Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Manoj Rohit Vemparala, Christos Anagnostopoulos, Pierpaolo Mori, Nael Fasfous,

Organizations: Technical University of Munich, Germany · BMW Group, Germany · University of Glasgow, United Kingdom · TU Wien, Austria

Abstract

Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.7 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 19%. Project page and code: https://lems-svd.github.io

Figures & tables

Panel of Figure 1 — Figure 1: Overview of KFAC-SVD and LEMS . (a) KFAC-SVD mitigates the rank bottleneck of prior Fisher-based SVD by leveraging token-wise statistics (N $\gg$ k). (b, c) LEMS enables effective allocation by (b) scaling raw spectral (spec.) energy to reflect true empirical importance and (c) adding a depth-dependent bias to model error depth-dependent propagation. See Supp. B.1 for empirical validation.

	Search Method	Mistral -7B				Llama 3 -8B				Qwen3 -8B
	Search Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$
	Baseline	5.25	8.10	27.72	63.95	6.14	9.47	9.90	63.34	9.71	15.52	15.43	62.03
0.8	uniform	7.08	18.58	57.22	52.67	10.61	41.94	60.91	50.84	12.25	30.63	41.70	54.18
	ASVD	6.98	19.66	64.84	51.41	12.02	59.27	65.86	47.28	13.28	36.16	46.52	50.74
	SVD-LLMv2	7.06	18.58	56.77	52.77	10.61	41.54	60.20	51.05	12.28	30.51	41.46	54.39
	MRCS	7.06	18.32	58.12	52.96	10.54	41.29	59.27	51.01	14.24	34.23	47.34	53.16

Table 1: Comparison of LEMS against state-of-the-art search baselines across modern LLM s and two compression rates.

	SVD Method	Mistral-7B				Llama 3 -8B				Qwen3 -8B
	SVD Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$
	Baseline	5.25	8.10	27.72	63.96	6.14	9.47	9.90	63.33	9.71	15.52	15.43	62.04
0.9	FWSVD	9.47	13.75	58.35	56.93	42.11	59.96	69.83	48.88	16.95	24.75	31.54	53.82
	ASVD	9.14	13.41	49.13	57.30	65.09	77.60	207.67	47.89	20.36	29.98	36.37	51.19
	SVD-LLM	6.46	15.07	66.12	56.81	10.14	39.63	49.91	52.27	12.52	27.40	36.66	55.96
	SVD-LLMv2	6.46	15.07	66.12	56.78	10.18	39.71	50.40	52.17	12.52	27.45	36.80	55.80

Table 2: Comparison of KFAC-SVD with activation- and Fisher-based SVD across modern LLM s under uniform compression and LEMS .

Panel of Figure 2 — Figure 2: WikiText2 perplexity (log scale) and average accuracy of KFAC-SVD and SVD-LLM on Qwen3 models with uniform compression and LEMS across multiple compression ratios (0.5-0.9) and a range of sizes 4B-32B.

Panel of Figure 3 — Figure 3: WikiText2 perplexity and zero-shot accuracy comparison of KL ( blue bars ) and MSE ( yellow bars ) as rescale importance metrics with different $c_{\text{ref}}$ values on Qwen3 -4B.

LEMS components	Llama 3 -8B		Qwen3 -8B
LEMS components	Wiki $\downarrow$	time $\downarrow$	Wiki $\downarrow$	time $\downarrow$
SVD (uniform)	33.97	start	18.00	start
+ ILP search (energy)	65.70	+1 min	32.57	+1 min
+ KL calibration	27.56	+17 min	15.58	+22 min
+ cross layer bias	16.08	+2 min	15.07	+2 min
search time (w/wo cache)	3 min / 20 min		3 min / 25 min

Table 3: WikiText2 performance and added execution time of individual parts of our method on Llama 3 -8B and Qwen3 -8B.

Panel of Figure 4 — Figure 4: Spectral decay of Lanczos Fisher approximation ( GFWSVD ) compared to our KFAC approximation (left) and resulting fisher error on unseen gradients (right), showing lower spectral decay and better generalization error for KFAC .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Systematic error underestimation. The yellow bars (sum of individual errors) consistently fall below the blue bars (measured total error), confirming that cross-layer interactions amplify degradation. This gap motivates the bias correction $\beta(d_{i})$ .

Panel of Figure 6 — Figure 6: Candidate bias functions $\beta(d_{i},\gamma,\kappa=1)$ . We test various decay profiles, from linear ( $\gamma=0$ ) to strictly exponential ( $\gamma=\pm 3$ ), to model the error propagation penalty.

Panel of Figure 7 — Figure 7: Grid Search for bias parameters $\kappa$ and $\gamma$ . Scatter plots show perplexity (lower is better) for varying magnitude $\kappa$ and decay $\gamma$ for Llama 3 .2-3B and Qwen3 -4B. The improvements observed across both models support the benefit of assigning higher importance to earlier layers.

Panel of Figure 8 — Figure 8: Comparison between different sensitivity assessment metrics across 9 compression ratios for all linear layers of Llama 3 .2-3B (top) and Qwen3 -4B (bottom) in block 15.

	Llama 3 .2-3B				Qwen3 -4B
$c_{\text{ref}}$	Wiki $\downarrow$		Acc $\uparrow$		Wiki $\downarrow$		Acc $\uparrow$
	KL	MSE	KL	MSE	KL	MSE	KL	MSE
0.7	10.00	9.94	50.42	50.13	14.69	14.41	55.41	55.01
0.5	9.77	9.85	50.47	49.92	14.69	14.47	55.61	54.89
0.3	9.52	9.73	51.04	49.93	14.61	14.35	55.98	54.37
0.1	9.54	9.64	50.89	51.03	14.47	14.38	55.05	53.83

Appendix

Table 4: Evaluation of different

c_{\text{ref}}

values for MSE and KL on both Llama 3 .2-3B and Qwen3 -4B.

Panel of Figure 9 — Figure 9: Average relative importance of block positions and layer types in Qwen3 -4B and Llama 3 .2-3B using KL divergence as importance metric.

Panel of Figure 10 — Figure 10: Average relative importance of block positions and block layer types in Qwen3 -4B and Llama 3 .2-3B using KL -divergence as importance metric.

Model	Proxy score	Type	MRCS search	ASVD search	Ours / ILP search
Mistral -7B	Energy (MRCS)	Dense	7.06 (0.53)	6.48 (0.56)	9.45 (0.49)
	KL (ASVD)	Sparse	n.a.	6.70 (0.53)	6.43 (0.56)
	Hybrid (ours)	Dense	7.11 (0.51)	6.63 (0.53)	5.96 (0.57)
	Hybrid + bias (ours)	Dense	7.03 (0.52)	6.62 (0.54)	5.92 (0.58)
Llama 3 -8B	Energy (MRCS)	Dense	10.54 (0.51)	8.91 (0.54)	10.77 (0.51)
	KL (ASVD)	Sparse	n.a.	11.65 (0.47)	9.14 (0.53)

Appendix

Table 5: Cross-ablation of proxy scores and search algorithms on Mistral -7B, Llama 3 -8B and Qwen3 -8B at 0.8 parameter retention. We report WikiText2 perplexity (lower is better) and average zero-shot accuracy in parentheses (higher is better). Best result for each proxy score is shown in bold, second best underlined. MRCS cannot use the sparse KL proxy because it requires dense per-rank sensitivity scores.

		Llama 3 .2-3B						Qwen3 -4B
		$\Delta F$			Eff.Rank			$\Delta F$			Eff.Rank
Idx	Layer	KFAC	GFWSVD	$\times$	KFAC	GFWSVD	$\times$	KFAC	GFWSVD	$\times$	KFAC	GFWSVD	$\times$
0	q_proj	5.4e-07	1.4e-06	0.39	7570.9	636.1	11.90	5.3e-04	5.6e-04	0.93	169327.1	2675.6	63.29
	k_proj	5.3e-07	8.0e-07	0.66	1548.4	545.3	2.84	8.8e-04	1.1e-03	0.81	67210.2	920.8	72.99
	v_proj	2.5e-05	5.2e-05	0.49	44295.1	181.1	244.62	4.6e-04	1.0e-03	0.46	93505.1	380.8	245.54
	o_proj	4.8e-05	9.5e-05	0.51	255441.8	230.7	1107.10	5.0e-03	7.6e-03	0.66	259343.8	370.3	700.37

Appendix

Table 6: Empirical Comparison of the actual Fisher information lost, when compressing with the obtained factors to rank 128, and the effective rank of each factor representation. Greater effective rank preserves more diverse directions, maintaining better generalization on the unseen fisher components.

	Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
	Original	5.68	7.34	8.35	0.31	0.75	0.41	0.70	0.56	0.78	58.50
0.9	FWSVD	43.48	72.10	77.97	0.15	0.40	0.25	0.54	0.31	0.60	37.69
	ASVD	9.96	13.70	14.69	0.25	0.63	0.33	0.64	0.47	0.71	50.66
	SVD-LLM	7.07	12.04	12.42	0.26	0.62	0.32	0.65	0.45	0.71	50.30
	SVD-LLMv2	7.07	12.05	12.43	0.26	0.62	0.32	0.65	0.45	0.71	50.29
	DOBI-SVD	7.69	12.54	12.53	0.24	0.60	0.32	0.64	0.46	0.70	49.28

Appendix

Table 7: Results of all methods on Llama -7B at uniform compression rates of 0.9 and 0.7. Bold / Underline denote best/second-best compressed results.

	Search Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
	Original	5.68	7.34	8.35	0.31	0.75	0.41	0.70	0.56	0.78	58.50
0.8	uniform	7.27	13.26	12.75	0.24	0.67	0.34	0.63	0.43	0.70	50.08
	ASVD	6.67	10.68	11.89	0.29	0.70	0.38	0.67	0.48	0.74	54.38
	SVD-LLMv2	7.27	13.20	12.75	0.24	0.67	0.34	0.63	0.43	0.69	50.23
	MRCS	7.26	13.19	12.37	0.25	0.66	0.33	0.64	0.44	0.70	50.32
	ARS	7.29	10.67	12.49	0.27	0.69	0.34	0.62	0.48	0.74	52.19

Appendix

Table 8: Comparison of searches on Llama -7B with KFAC-SVD at 0.8 and 0.6 parameter retention. Bold / Underline denote best/second-best results.

		Mistral -7B		Llama 3 -8B		Qwen3 -8B		Llama -7B
Ratio	Decomposition	Wiki $\downarrow$	Avg. $\uparrow$	Wiki $\downarrow$	Avg. $\uparrow$	Wiki $\downarrow$	Avg. $\uparrow$	Wiki $\downarrow$	Avg. $\uparrow$
0.8	SVD-LLM	8.15	51.61	14.61	44.84	14.05	52.31	7.89	44.65
	OBD-LLM	7.33	50.88	11.18	49.42	12.19	52.66	7.42	48.88
	IO-SVD	7.27	50.88	11.23	49.54	12.39	52.40	7.41	48.68
	KFAC-SVD	7.08	52.67	10.61	50.84	12.25	54.18	7.27	50.08
0.6	SVD-LLM	16.12	37.71	136.19	32.97	28.77	36.81	13.74	37.14

Appendix

Table 9: Decompositions of concurrent work at uniform ranks and 0.8 and 0.6 parameter retention: WikiText2 perplexity and average zero-shot accuracy. Bold / Underline : best/second best.

		Mistral -7B		Llama 3 -8B		Qwen3 -8B		Llama -7B
Ratio	Search	Wiki $\downarrow$	Avg. $\uparrow$	Wiki $\downarrow$	Avg. $\uparrow$	Wiki $\downarrow$	Avg. $\uparrow$	Wiki $\downarrow$	Avg. $\uparrow$
0.8	uniform	7.08	52.67	10.61	50.84	12.25	54.18	7.27	50.08
	ZS-SVD	6.30	57.18	8.72	55.54	11.20	58.46	6.49	53.85
	Swift-SVD	7.04	52.91	10.04	51.82	12.28	54.49	7.25	50.36
	LEMS	5.92	57.61	8.04	56.77	10.28	59.03	6.23	55.37
0.6	uniform	14.47	39.22	33.97	36.24	18.00	41.16	11.12	40.45

Appendix

Table 10: Rank allocations of concurrent work applied to KFAC-SVD at 0.8 and 0.6 parameter retention: WikiText2 perplexity and average zero-shot accuracy. Bold / Underline : best/second best.

	Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
	Original	5.47	7.29	26.82	0.33	0.76	0.44	0.69	0.57	0.78	59.51
0.8	FWSVD	173.51	224.54	4138.65	0.15	0.29	0.21	0.50	0.28	0.55	33.10
	ASVD	25.24	32.73	239.02	0.26	0.54	0.28	0.60	0.40	0.65	45.44
	SVD-LLM	8.42	19.97	129.95	0.26	0.57	0.27	0.63	0.40	0.67	46.66
	DOBI-SVD	9.58	20.20	102.00	0.24	0.52	0.24	0.60	0.38	0.64	43.80
	GFWSVD	63.09	121.61	881.17	0.15	0.34	0.21	0.53	0.30	0.58	34.97

Appendix

Table 11: Additional results of all methods on Llama 2 -7B across multiple uniform compression rates and SVD baselines . Bold / Underline denote best/second-best compressed results.

Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$	time [h] $\downarrow$
Original	2.81	7.65	7.22	0.37	0.60	0.85	0.82	0.67	0.83	68.95	-
uniform	7.94	21.42	21.68	0.35	0.81	0.52	0.78	0.53	0.78	62.61	-
ASVD	7.33	26.25	18.72	0.36	0.83	0.54	0.75	0.57	0.79	64.11	22.5
SVD-LLMv2	7.82	21.13	21.46	0.34	0.81	0.52	0.77	0.53	0.78	62.61	$<$ 0.1
MRCS	7.52	22.02	22.23	0.35	0.81	0.51	0.76	0.53	0.78	62.38	$<$ 0.1
ATP	7.94	21.42	21.68	0.35	0.81	0.52	0.78	0.53	0.78	62.61	0.2

Appendix

Table 12: Extended format zero-shot performance and perplexity of Llama 3 .1-70B with a 0.8 parameter retention .

Method	Weight Memory	Accuracy
BitStack	5.19 GB	59.93
KFAC-SVD	5.07 GB	60.83

Appendix

Table 13: Perplexity on WikiText2 (

\downarrow

) and zero shot accuracy of Llama 3 -8B. For BitStack we report the quantized memory, but float accuracy due to compatibility issues, favoring its results.

Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
Original	5.47	7.29	24.09	0.33	0.75	0.43	0.69	0.57	0.78	59.43
BasisSharing (Ref. data + fp32)	7.71	16.21	59.47	0.29	0.63	0.31	0.64	0.44	0.70	50.17
BasisSharing (Our data + fp16)	8.12	19.35	106.48	0.26	0.53	0.24	0.63	0.42	0.67	45.83
$+$ LEMS	6.25	9.83	297.47	0.32	0.72	0.39	0.67	0.52	0.75	56.17

Appendix

Table 14: Llama 2 -7B compressed with BasisSharing to 0.8 parameter count with the data and precision used in the original work, the data and precision of our work and with LEMS applied to it.

Method	GParams	TFLOPs	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
Original	6.48	26.53	5.47	7.29	24.09	0.33	0.75	0.43	0.69	0.57	0.78	59.43
Matched Compute Budget: 0.8 $\cdot$ Original Linear Layer Compute
BasisSharing	4.53	21.40	10.16	32.60	231.67	0.25	0.51	0.24	0.62	0.36	0.63	43.50
KFAC-SVD	5.18	21.22	7.31	15.04	53.86	0.27	0.67	0.32	0.64	0.43	0.70	50.77
Matched Memory Budget: 0.8 $\cdot$ Original Linear Layer Memory
BasisSharing	5.18	24.46	8.12	19.35	106.48	0.26	0.53	0.24	0.63	0.42	0.67	45.83

Appendix

Table 15: Comparing KFAC-SVD with BasisSharing at matched compute (TFLOPs) and memory (GParams) budgets with respect to the linear layers that are considered in the compression of Llama 2 -7B without any search applied.

		Mistral -7B				Llama 3 -8B				Qwen3 -8B
	Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Acc $\uparrow$
	Baseline	5.25	8.10	27.72	63.96	6.14	9.47	9.90	63.33	9.71	15.52	15.43	62.04
0.9	FWSVD	9.98	13.72	59.96	57.61	51.29	50.99	95.82	51.65	27.03	25.24	inf	50.02
	ASVD	9.49	13.22	47.53	56.73	90.72	91.79	295.15	46.77	20.52	29.17	37.16	50.90
	SVD-LLM	8.23	11.20	51.89	57.32	21.93	19.58	36.73	54.11	16.23	19.85	22.06	57.89
	SVD-LLMv2	8.24	11.20	51.89	57.37	21.97	19.62	36.87	54.16	16.20	19.85	22.06	57.87

Appendix

Table 16: Comparison of KFAC-SVD with activation- and Fisher-based SVD across modern LLM s under uniform compression and LEMS on the fineweb 16M dataset.

Ratio	KFAC-SVD	SVD-LLM
Original	What is the responsibility of the AI assistant? And how can it be ensured? The responsibility and accountability of AI assistants are critical issues that require careful consideration. Here’s a structured analysis of the key points and a comprehensive approach to ensuring responsibility: 1. Ethical and Moral Responsibility: AI systems should be developed and used in a way that aligns with ethical and moral principles, avoiding harm to individuals and society. 2. …
10%	What is the responsibility of the AI assistant? And how can it be ensured? The responsibility (ethical, legal, technical, etc.) of AI assistants is a critical area of concern in the development and application of artificial intelligence technology. This issue is not only related to the fairness and impartiality of the AI system itself but also involves the rights and interests of individuals and society. In order to ensure the correct development of intelligent assistants, we need to establish a comprehensive framework that involves both technical and non-technical dimensions. …	What is the responsibility of the AI assistant? - - - According to its creators, AI assistants are designed to be helpful and to support users. - = - They should not act as a replacement for human intelligence and should instead focus on assisting users through tasks they can perform. = = - - - - However, some AI systems have been designed with specific purposes, such as responding to questions and carrying out tasks that require human thinking. For example, there are AI programs that are used in hospitals that perform complex tasks such as interpreting X-rays and diagnosing diseases. …
20%	What is the responsibility of the AI assistant? According to the given prompt, the assistant is supposed to be a helpful companion who provides the best responses. It is not supposed be something of the dark side. So the AI should not be malevolent, but rather a positive and helpful entity. But, in certain situations, even with all the good intentions, perhaps the Dark Side could come into play. Also, there are some guidelines. For example, if there is a request for harm, then the system should refuse. The AI must not provide harmful information. Therefore, to avoid any harm and to maintain a beneficial environment, it is important to ensure that the responses are safe and positive. …	What is the responsibility of the AI assistant? What is the responsibility of an AI assistant? and what should they do to be ethical? 5.00 PM, 15 March 2021 Please help me to create a presentation about AI assistants in the modern age. Please share your views and opinions about what an ethical AI should be like in today’s world. I want to present about the AI as assistants to humans and how they are being developed. I am in charge of a department to which AI is being introduced for the first time. …
30%	What is the responsibility of the AI assistant? And the main point is to be on time and precise. Which of the four are the most important. Which is most beneficial? Which one is a good solution for the whole team. I want to do the assignment for each one, but I need to clarify it. I’m not the best at it, and need help from someone. So I will have to provide some clarification. To get the support of someone who is willing to help, I have the authority. It is not clear, or maybe I don’t know about the question and the meaning. …	What is the responsibility of the AI assistant? and it will give the answer. So to answer questions, the AI may be asked the question by someone, and perhaps I know something about the subject, or perhaps he can be helpful in this situation. So the person can explain to the audience that he wants to be the assistant, because of his own identity. So he has been given to an assistant by the creator and can now become an employee. This idea has to exist in a situation, but the human wants the job. He is responsible for maintaining a position to create AI.

Appendix

Table 17: Example content generated as response to ”What is the responsibility of the AI assistant?” by Qwen3 -8B models that were compressed with SVD-LLM and KFAC-SVD without any finetuning under different compression ratios.

	Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
Mistral -7B	Original	5.25	8.10	27.72	0.35	0.80	0.51	0.75	0.62	0.81	63.96
	FWSVD	9.47	13.75	58.35	0.31	0.74	0.40	0.69	0.51	0.76	56.93
	ASVD	9.14	13.41	49.13	0.33	0.73	0.41	0.69	0.52	0.77	57.30
	SVD-LLM	6.46	15.07	66.12	0.30	0.75	0.42	0.71	0.49	0.75	56.81
	SVD-LLMv2	6.46	15.07	66.12	0.30	0.75	0.42	0.71	0.49	0.75	56.78
	DOBI-SVD	7.11	15.19	63.58	0.29	0.71	0.38	0.71	0.49	0.74	55.17

Appendix

Table 18: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) with a 0.9 parameter retention compressed with a range of SVD baselines and KFAC-SVD . Bold / Underline denote best/second-best compressed results.

	Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
Mistral -7B	Original	5.25	8.10	27.72	0.35	0.80	0.51	0.75	0.62	0.81	63.96
	FWSVD	34.84	50.30	215.10	0.21	0.59	0.25	0.60	0.34	0.66	44.24
	ASVD	28.16	42.94	163.64	0.24	0.57	0.29	0.62	0.38	0.66	46.09
	SVD-LLM	10.96	42.60	169.49	0.24	0.56	0.25	0.62	0.35	0.64	44.39
	SVD-LLMv2	10.96	42.60	169.49	0.24	0.56	0.25	0.62	0.35	0.64	44.35
	DOBI-SVD	12.37	39.71	170.82	0.22	0.50	0.26	0.60	0.35	0.65	42.94

Appendix

Table 19: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) with a 0.7 parameter retention compressed with a range of SVD baselines and KFAC-SVD . Bold / Underline denote best/second-best compressed results.

	Search Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
Mistral -7B	uniform	7.08	18.58	57.22	0.27	0.70	0.37	0.68	0.43	0.71	52.67
	ASVD	6.98	19.66	64.84	0.28	0.70	0.34	0.65	0.40	0.71	51.41
	SVD-LLMv2	7.06	18.58	56.77	0.27	0.71	0.36	0.68	0.43	0.71	52.77
	MRCS	7.06	18.32	58.12	0.27	0.71	0.37	0.68	0.43	0.71	52.96
	ARS	7.32	11.76	44.65	0.27	0.68	0.36	0.68	0.50	0.74	53.68
	ATP	7.06	18.15	58.35	0.27	0.71	0.37	0.69	0.43	0.70	52.75

Appendix

Table 20: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) compressed with a range of different searches with a 0.8 parameter retention . Bold / Underline denote best/second-best compressed results.

	Search Method	Wiki $\downarrow$	C4 $\downarrow$	PTB $\downarrow$	Openb. $\uparrow$	ARC_e $\uparrow$	ARC_c $\uparrow$	WinoG. $\uparrow$	HellaS. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
Mistral -7B	uniform	14.47	67.16	181.84	0.19	0.47	0.23	0.57	0.31	0.59	39.22
	ASVD	16.33	91.43	268.74	0.17	0.40	0.20	0.55	0.29	0.57	36.35
	SVD-LLMv2	14.50	67.16	183.27	0.20	0.47	0.23	0.57	0.31	0.59	39.37
	MRCS	14.41	70.93	222.79	0.20	0.48	0.23	0.57	0.31	0.59	39.58
	ARS	18.47	33.05	176.24	0.18	0.49	0.23	0.55	0.34	0.64	40.66
	ATP	15.73	59.73	188.35	0.20	0.48	0.25	0.60	0.33	0.61	41.13

Appendix

Table 21: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) compressed with a range of different searches with a 0.6 parameter retention . Bold / Underline denote best/second-best compressed results.

		Qwen3
	Method	1.7B		4B		8B		14B		32B
		PPL $\downarrow$	Acc $\uparrow$	PPL $\downarrow$	Acc $\uparrow$	PPL $\downarrow$	Acc $\uparrow$	PPL $\downarrow$	Acc $\uparrow$	PPL $\downarrow$	Acc $\uparrow$
	Baseline	16.68	53.33	13.64	58.97	9.71	62.04	8.64	65.28	7.59	45.02
0.9	SVD-LLM	30.99	44.18	17.15	49.28	12.52	55.96	11.07	58.82	9.23	43.23
	$+$ LEMS	17.62	51.24	14.02	57.56	9.83	62.36	8.77	65.28	7.64	65.61
	KFAC-SVD	26.98	44.11	15.98	50.79	11.27	56.73	9.87	60.49	8.26	43.58

Appendix

Table 22: Detailed performance evaluation of LEMS for both KFAC-SVD and SVD-LLM across five Qwen3 model sizes (1.7B-32B). The table contrasts the baseline uniform rank allocation with our proposed LEMS across compression ratios

c\in[0.5,0.9]

. Bold indicates the best result between the uniform and search-based allocation for each method, showing the massive improvements brought about by LEMS .

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Jul 3, 2026Zhuowen Liu, Longkun Hao, Shiyu Feng +3Model Compression LLM Compression
Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM Compression Low-Rank Matrix Decomposition
Vanderbilt University · University of California, Davis
GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression
Oct 11, 2026Shengkun Tang, Cong Zeng, Zongfang Liu +2LLM Compression Low-Rank Compression
MBZUAI · Technology Innovation Institute UAE · Zhejiang Univeristy +2

Jul 3, 2026J/K move · Enter open · S save

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

Zhuowen Liu, Longkun Hao, Shiyu Feng, Xiaowen Chang+2

Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3+3

Model Compression LLM Compression Low-Rank Matrix Decomposition Low-Rank Compression Low-Rank Approximation Singular Value Decomposition

Open paper AskPDF arXiv

The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).

Abstract

LLM Compression

Explore similar work

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression