Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
Authors: Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Manoj Rohit Vemparala, Christos Anagnostopoulos, Pierpaolo Mori, Nael Fasfous, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann
Organizations: Technical University of Munich, Germany · BMW Group, Germany · University of Glasgow, United Kingdom · TU Wien, Austria
Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.7 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 19%. Project page and code: https://lems-svd.github.io
Figures & tables
Figure 1: Overview of KFAC-SVD and LEMS . (a) KFAC-SVD mitigates the rank bottleneck of prior Fisher-based SVD by leveraging token-wise statistics (N ≫ k). (b, c) LEMS enables effective allocation by (b) scaling raw spectral (spec.) energy to reflect true empirical importance and (c) adding a depth-dependent bias to model error depth-dependent propagation. See Supp. B.1 for empirical validation.
Search Method
Mistral -7B
Llama 3 -8B
Qwen3 -8B
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Baseline
5.25
8.10
27.72
63.95
6.14
9.47
9.90
63.34
9.71
15.52
15.43
62.03
0.8
uniform
7.08
18.58
57.22
52.67
10.61
41.94
60.91
50.84
12.25
30.63
41.70
54.18
ASVD
6.98
19.66
64.84
51.41
12.02
59.27
65.86
47.28
13.28
36.16
46.52
50.74
SVD-LLMv2
7.06
18.58
56.77
52.77
10.61
41.54
60.20
51.05
12.28
30.51
41.46
54.39
MRCS
7.06
18.32
58.12
52.96
10.54
41.29
59.27
51.01
14.24
34.23
47.34
53.16
Table 1: Comparison of LEMS against state-of-the-art search baselines across modern LLM s and two compression rates.
SVD Method
Mistral-7B
Llama 3 -8B
Qwen3 -8B
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Baseline
5.25
8.10
27.72
63.96
6.14
9.47
9.90
63.33
9.71
15.52
15.43
62.04
0.9
FWSVD
9.47
13.75
58.35
56.93
42.11
59.96
69.83
48.88
16.95
24.75
31.54
53.82
ASVD
9.14
13.41
49.13
57.30
65.09
77.60
207.67
47.89
20.36
29.98
36.37
51.19
SVD-LLM
6.46
15.07
66.12
56.81
10.14
39.63
49.91
52.27
12.52
27.40
36.66
55.96
SVD-LLMv2
6.46
15.07
66.12
56.78
10.18
39.71
50.40
52.17
12.52
27.45
36.80
55.80
Table 2: Comparison of KFAC-SVD with activation- and Fisher-based SVD across modern LLM s under uniform compression and LEMS .
Figure 2: WikiText2 perplexity (log scale) and average accuracy of KFAC-SVD and SVD-LLM on Qwen3 models with uniform compression and LEMS across multiple compression ratios (0.5-0.9) and a range of sizes 4B-32B.
Figure 3: WikiText2 perplexity and zero-shot accuracy comparison of KL ( blue bars ) and MSE ( yellow bars ) as rescale importance metrics with different cref values on Qwen3 -4B.
LEMS components
Llama 3 -8B
Qwen3 -8B
Wiki ↓
time ↓
Wiki ↓
time ↓
SVD (uniform)
33.97
start
18.00
start
+ ILP search (energy)
65.70
+1 min
32.57
+1 min
+ KL calibration
27.56
+17 min
15.58
+22 min
+ cross layer bias
16.08
+2 min
15.07
+2 min
search time (w/wo cache)
3 min / 20 min
3 min / 25 min
Table 3: WikiText2 performance and added execution time of individual parts of our method on Llama 3 -8B and Qwen3 -8B.
Figure 4: Spectral decay of Lanczos Fisher approximation ( GFWSVD ) compared to our KFAC approximation (left) and resulting fisher error on unseen gradients (right), showing lower spectral decay and better generalization error for KFAC .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Systematic error underestimation. The yellow bars (sum of individual errors) consistently fall below the blue bars (measured total error), confirming that cross-layer interactions amplify degradation. This gap motivates the bias correction β(di) .
Figure 6: Candidate bias functions β(di,γ,κ=1) . We test various decay profiles, from linear ( γ=0 ) to strictly exponential ( γ=±3 ), to model the error propagation penalty.
Figure 7: Grid Search for bias parameters κ and γ . Scatter plots show perplexity (lower is better) for varying magnitude κ and decay γ for Llama 3 .2-3B and Qwen3 -4B. The improvements observed across both models support the benefit of assigning higher importance to earlier layers.
Figure 8: Comparison between different sensitivity assessment metrics across 9 compression ratios for all linear layers of Llama 3 .2-3B (top) and Qwen3 -4B (bottom) in block 15.
Llama 3 .2-3B
Qwen3 -4B
cref
Wiki ↓
Acc ↑
Wiki ↓
Acc ↑
KL
MSE
KL
MSE
KL
MSE
KL
MSE
0.7
10.00
9.94
50.42
50.13
14.69
14.41
55.41
55.01
0.5
9.77
9.85
50.47
49.92
14.69
14.47
55.61
54.89
0.3
9.52
9.73
51.04
49.93
14.61
14.35
55.98
54.37
0.1
9.54
9.64
50.89
51.03
14.47
14.38
55.05
53.83
Appendix
Table 4: Evaluation of different cref values for MSE and KL on both Llama 3 .2-3B and Qwen3 -4B.
Figure 9: Average relative importance of block positions and layer types in Qwen3 -4B and Llama 3 .2-3B using KL divergence as importance metric.
Figure 10: Average relative importance of block positions and block layer types in Qwen3 -4B and Llama 3 .2-3B using KL -divergence as importance metric.
Model
Proxy score
Type
MRCS search
ASVD search
Ours / ILP search
Mistral -7B
Energy (MRCS)
Dense
7.06 (0.53)
6.48 (0.56)
9.45 (0.49)
KL (ASVD)
Sparse
n.a.
6.70 (0.53)
6.43 (0.56)
Hybrid (ours)
Dense
7.11 (0.51)
6.63 (0.53)
5.96 (0.57)
Hybrid + bias (ours)
Dense
7.03 (0.52)
6.62 (0.54)
5.92 (0.58)
Llama 3 -8B
Energy (MRCS)
Dense
10.54 (0.51)
8.91 (0.54)
10.77 (0.51)
KL (ASVD)
Sparse
n.a.
11.65 (0.47)
9.14 (0.53)
Appendix
Table 5: Cross-ablation of proxy scores and search algorithms on Mistral -7B, Llama 3 -8B and Qwen3 -8B at 0.8 parameter retention. We report WikiText2 perplexity (lower is better) and average zero-shot accuracy in parentheses (higher is better). Best result for each proxy score is shown in bold, second best underlined. MRCS cannot use the sparse KL proxy because it requires dense per-rank sensitivity scores.
Llama 3 .2-3B
Qwen3 -4B
ΔF
Eff.Rank
ΔF
Eff.Rank
Idx
Layer
KFAC
GFWSVD
×
KFAC
GFWSVD
×
KFAC
GFWSVD
×
KFAC
GFWSVD
×
0
q_proj
5.4e-07
1.4e-06
0.39
7570.9
636.1
11.90
5.3e-04
5.6e-04
0.93
169327.1
2675.6
63.29
k_proj
5.3e-07
8.0e-07
0.66
1548.4
545.3
2.84
8.8e-04
1.1e-03
0.81
67210.2
920.8
72.99
v_proj
2.5e-05
5.2e-05
0.49
44295.1
181.1
244.62
4.6e-04
1.0e-03
0.46
93505.1
380.8
245.54
o_proj
4.8e-05
9.5e-05
0.51
255441.8
230.7
1107.10
5.0e-03
7.6e-03
0.66
259343.8
370.3
700.37
Appendix
Table 6: Empirical Comparison of the actual Fisher information lost, when compressing with the obtained factors to rank 128, and the effective rank of each factor representation. Greater effective rank preserves more diverse directions, maintaining better generalization on the unseen fisher components.
Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Original
5.68
7.34
8.35
0.31
0.75
0.41
0.70
0.56
0.78
58.50
0.9
FWSVD
43.48
72.10
77.97
0.15
0.40
0.25
0.54
0.31
0.60
37.69
ASVD
9.96
13.70
14.69
0.25
0.63
0.33
0.64
0.47
0.71
50.66
SVD-LLM
7.07
12.04
12.42
0.26
0.62
0.32
0.65
0.45
0.71
50.30
SVD-LLMv2
7.07
12.05
12.43
0.26
0.62
0.32
0.65
0.45
0.71
50.29
DOBI-SVD
7.69
12.54
12.53
0.24
0.60
0.32
0.64
0.46
0.70
49.28
Appendix
Table 7: Results of all methods on Llama -7B at uniform compression rates of 0.9 and 0.7. Bold / Underline denote best/second-best compressed results.
Search Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Original
5.68
7.34
8.35
0.31
0.75
0.41
0.70
0.56
0.78
58.50
0.8
uniform
7.27
13.26
12.75
0.24
0.67
0.34
0.63
0.43
0.70
50.08
ASVD
6.67
10.68
11.89
0.29
0.70
0.38
0.67
0.48
0.74
54.38
SVD-LLMv2
7.27
13.20
12.75
0.24
0.67
0.34
0.63
0.43
0.69
50.23
MRCS
7.26
13.19
12.37
0.25
0.66
0.33
0.64
0.44
0.70
50.32
ARS
7.29
10.67
12.49
0.27
0.69
0.34
0.62
0.48
0.74
52.19
Appendix
Table 8: Comparison of searches on Llama -7B with KFAC-SVD at 0.8 and 0.6 parameter retention. Bold / Underline denote best/second-best results.
Mistral -7B
Llama 3 -8B
Qwen3 -8B
Llama -7B
Ratio
Decomposition
Wiki ↓
Avg. ↑
Wiki ↓
Avg. ↑
Wiki ↓
Avg. ↑
Wiki ↓
Avg. ↑
0.8
SVD-LLM
8.15
51.61
14.61
44.84
14.05
52.31
7.89
44.65
OBD-LLM
7.33
50.88
11.18
49.42
12.19
52.66
7.42
48.88
IO-SVD
7.27
50.88
11.23
49.54
12.39
52.40
7.41
48.68
KFAC-SVD
7.08
52.67
10.61
50.84
12.25
54.18
7.27
50.08
0.6
SVD-LLM
16.12
37.71
136.19
32.97
28.77
36.81
13.74
37.14
Appendix
Table 9: Decompositions of concurrent work at uniform ranks and 0.8 and 0.6 parameter retention: WikiText2 perplexity and average zero-shot accuracy. Bold / Underline : best/second best.
Mistral -7B
Llama 3 -8B
Qwen3 -8B
Llama -7B
Ratio
Search
Wiki ↓
Avg. ↑
Wiki ↓
Avg. ↑
Wiki ↓
Avg. ↑
Wiki ↓
Avg. ↑
0.8
uniform
7.08
52.67
10.61
50.84
12.25
54.18
7.27
50.08
ZS-SVD
6.30
57.18
8.72
55.54
11.20
58.46
6.49
53.85
Swift-SVD
7.04
52.91
10.04
51.82
12.28
54.49
7.25
50.36
LEMS
5.92
57.61
8.04
56.77
10.28
59.03
6.23
55.37
0.6
uniform
14.47
39.22
33.97
36.24
18.00
41.16
11.12
40.45
Appendix
Table 10: Rank allocations of concurrent work applied to KFAC-SVD at 0.8 and 0.6 parameter retention: WikiText2 perplexity and average zero-shot accuracy. Bold / Underline : best/second best.
Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Original
5.47
7.29
26.82
0.33
0.76
0.44
0.69
0.57
0.78
59.51
0.8
FWSVD
173.51
224.54
4138.65
0.15
0.29
0.21
0.50
0.28
0.55
33.10
ASVD
25.24
32.73
239.02
0.26
0.54
0.28
0.60
0.40
0.65
45.44
SVD-LLM
8.42
19.97
129.95
0.26
0.57
0.27
0.63
0.40
0.67
46.66
DOBI-SVD
9.58
20.20
102.00
0.24
0.52
0.24
0.60
0.38
0.64
43.80
GFWSVD
63.09
121.61
881.17
0.15
0.34
0.21
0.53
0.30
0.58
34.97
Appendix
Table 11: Additional results of all methods on Llama 2 -7B across multiple uniform compression rates and SVD baselines . Bold / Underline denote best/second-best compressed results.
Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
time [h] ↓
Original
2.81
7.65
7.22
0.37
0.60
0.85
0.82
0.67
0.83
68.95
-
uniform
7.94
21.42
21.68
0.35
0.81
0.52
0.78
0.53
0.78
62.61
-
ASVD
7.33
26.25
18.72
0.36
0.83
0.54
0.75
0.57
0.79
64.11
22.5
SVD-LLMv2
7.82
21.13
21.46
0.34
0.81
0.52
0.77
0.53
0.78
62.61
< 0.1
MRCS
7.52
22.02
22.23
0.35
0.81
0.51
0.76
0.53
0.78
62.38
< 0.1
ATP
7.94
21.42
21.68
0.35
0.81
0.52
0.78
0.53
0.78
62.61
0.2
Appendix
Table 12: Extended format zero-shot performance and perplexity of Llama 3 .1-70B with a 0.8 parameter retention .
Method
Weight Memory
Accuracy
BitStack
5.19 GB
59.93
KFAC-SVD
5.07 GB
60.83
Appendix
Table 13: Perplexity on WikiText2 ( ↓ ) and zero shot accuracy of Llama 3 -8B. For BitStack we report the quantized memory, but float accuracy due to compatibility issues, favoring its results.
Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Original
5.47
7.29
24.09
0.33
0.75
0.43
0.69
0.57
0.78
59.43
BasisSharing (Ref. data + fp32)
7.71
16.21
59.47
0.29
0.63
0.31
0.64
0.44
0.70
50.17
BasisSharing (Our data + fp16)
8.12
19.35
106.48
0.26
0.53
0.24
0.63
0.42
0.67
45.83
+ LEMS
6.25
9.83
297.47
0.32
0.72
0.39
0.67
0.52
0.75
56.17
Appendix
Table 14: Llama 2 -7B compressed with BasisSharing to 0.8 parameter count with the data and precision used in the original work, the data and precision of our work and with LEMS applied to it.
Method
GParams
TFLOPs
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Original
6.48
26.53
5.47
7.29
24.09
0.33
0.75
0.43
0.69
0.57
0.78
59.43
Matched Compute Budget: 0.8 ⋅ Original Linear Layer Compute
BasisSharing
4.53
21.40
10.16
32.60
231.67
0.25
0.51
0.24
0.62
0.36
0.63
43.50
KFAC-SVD
5.18
21.22
7.31
15.04
53.86
0.27
0.67
0.32
0.64
0.43
0.70
50.77
Matched Memory Budget: 0.8 ⋅ Original Linear Layer Memory
BasisSharing
5.18
24.46
8.12
19.35
106.48
0.26
0.53
0.24
0.63
0.42
0.67
45.83
Appendix
Table 15: Comparing KFAC-SVD with BasisSharing at matched compute (TFLOPs) and memory (GParams) budgets with respect to the linear layers that are considered in the compression of Llama 2 -7B without any search applied.
Figure 26
Mistral -7B
Llama 3 -8B
Qwen3 -8B
Method
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Wiki ↓
C4 ↓
PTB ↓
Acc ↑
Baseline
5.25
8.10
27.72
63.96
6.14
9.47
9.90
63.33
9.71
15.52
15.43
62.04
0.9
FWSVD
9.98
13.72
59.96
57.61
51.29
50.99
95.82
51.65
27.03
25.24
inf
50.02
ASVD
9.49
13.22
47.53
56.73
90.72
91.79
295.15
46.77
20.52
29.17
37.16
50.90
SVD-LLM
8.23
11.20
51.89
57.32
21.93
19.58
36.73
54.11
16.23
19.85
22.06
57.89
SVD-LLMv2
8.24
11.20
51.89
57.37
21.97
19.62
36.87
54.16
16.20
19.85
22.06
57.87
Appendix
Table 16: Comparison of KFAC-SVD with activation- and Fisher-based SVD across modern LLM s under uniform compression and LEMS on the fineweb 16M dataset.
Ratio
KFAC-SVD
SVD-LLM
Original
What is the responsibility of the AI assistant? And how can it be ensured? The responsibility and accountability of AI assistants are critical issues that require careful consideration. Here’s a structured analysis of the key points and a comprehensive approach to ensuring responsibility: 1. Ethical and Moral Responsibility: AI systems should be developed and used in a way that aligns with ethical and moral principles, avoiding harm to individuals and society. 2. …
10%
What is the responsibility of the AI assistant? And how can it be ensured? The responsibility (ethical, legal, technical, etc.) of AI assistants is a critical area of concern in the development and application of artificial intelligence technology. This issue is not only related to the fairness and impartiality of the AI system itself but also involves the rights and interests of individuals and society. In order to ensure the correct development of intelligent assistants, we need to establish a comprehensive framework that involves both technical and non-technical dimensions. …
What is the responsibility of the AI assistant? - - - According to its creators, AI assistants are designed to be helpful and to support users. - = - They should not act as a replacement for human intelligence and should instead focus on assisting users through tasks they can perform. = = - - - - However, some AI systems have been designed with specific purposes, such as responding to questions and carrying out tasks that require human thinking. For example, there are AI programs that are used in hospitals that perform complex tasks such as interpreting X-rays and diagnosing diseases. …
20%
What is the responsibility of the AI assistant? According to the given prompt, the assistant is supposed to be a helpful companion who provides the best responses. It is not supposed be something of the dark side. So the AI should not be malevolent, but rather a positive and helpful entity. But, in certain situations, even with all the good intentions, perhaps the Dark Side could come into play. Also, there are some guidelines. For example, if there is a request for harm, then the system should refuse. The AI must not provide harmful information. Therefore, to avoid any harm and to maintain a beneficial environment, it is important to ensure that the responses are safe and positive. …
What is the responsibility of the AI assistant? What is the responsibility of an AI assistant? and what should they do to be ethical? 5.00 PM, 15 March 2021 Please help me to create a presentation about AI assistants in the modern age. Please share your views and opinions about what an ethical AI should be like in today’s world. I want to present about the AI as assistants to humans and how they are being developed. I am in charge of a department to which AI is being introduced for the first time. …
30%
What is the responsibility of the AI assistant? And the main point is to be on time and precise. Which of the four are the most important. Which is most beneficial? Which one is a good solution for the whole team. I want to do the assignment for each one, but I need to clarify it. I’m not the best at it, and need help from someone. So I will have to provide some clarification. To get the support of someone who is willing to help, I have the authority. It is not clear, or maybe I don’t know about the question and the meaning. …
What is the responsibility of the AI assistant? and it will give the answer. So to answer questions, the AI may be asked the question by someone, and perhaps I know something about the subject, or perhaps he can be helpful in this situation. So the person can explain to the audience that he wants to be the assistant, because of his own identity. So he has been given to an assistant by the creator and can now become an employee. This idea has to exist in a situation, but the human wants the job. He is responsible for maintaining a position to create AI.
Appendix
Table 17: Example content generated as response to ”What is the responsibility of the AI assistant?” by Qwen3 -8B models that were compressed with SVD-LLM and KFAC-SVD without any finetuning under different compression ratios.
Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Mistral -7B
Original
5.25
8.10
27.72
0.35
0.80
0.51
0.75
0.62
0.81
63.96
FWSVD
9.47
13.75
58.35
0.31
0.74
0.40
0.69
0.51
0.76
56.93
ASVD
9.14
13.41
49.13
0.33
0.73
0.41
0.69
0.52
0.77
57.30
SVD-LLM
6.46
15.07
66.12
0.30
0.75
0.42
0.71
0.49
0.75
56.81
SVD-LLMv2
6.46
15.07
66.12
0.30
0.75
0.42
0.71
0.49
0.75
56.78
DOBI-SVD
7.11
15.19
63.58
0.29
0.71
0.38
0.71
0.49
0.74
55.17
Appendix
Table 18: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) with a 0.9 parameter retention compressed with a range of SVD baselines and KFAC-SVD . Bold / Underline denote best/second-best compressed results.
Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Mistral -7B
Original
5.25
8.10
27.72
0.35
0.80
0.51
0.75
0.62
0.81
63.96
FWSVD
34.84
50.30
215.10
0.21
0.59
0.25
0.60
0.34
0.66
44.24
ASVD
28.16
42.94
163.64
0.24
0.57
0.29
0.62
0.38
0.66
46.09
SVD-LLM
10.96
42.60
169.49
0.24
0.56
0.25
0.62
0.35
0.64
44.39
SVD-LLMv2
10.96
42.60
169.49
0.24
0.56
0.25
0.62
0.35
0.64
44.35
DOBI-SVD
12.37
39.71
170.82
0.22
0.50
0.26
0.60
0.35
0.65
42.94
Appendix
Table 19: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) with a 0.7 parameter retention compressed with a range of SVD baselines and KFAC-SVD . Bold / Underline denote best/second-best compressed results.
Search Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Mistral -7B
uniform
7.08
18.58
57.22
0.27
0.70
0.37
0.68
0.43
0.71
52.67
ASVD
6.98
19.66
64.84
0.28
0.70
0.34
0.65
0.40
0.71
51.41
SVD-LLMv2
7.06
18.58
56.77
0.27
0.71
0.36
0.68
0.43
0.71
52.77
MRCS
7.06
18.32
58.12
0.27
0.71
0.37
0.68
0.43
0.71
52.96
ARS
7.32
11.76
44.65
0.27
0.68
0.36
0.68
0.50
0.74
53.68
ATP
7.06
18.15
58.35
0.27
0.71
0.37
0.69
0.43
0.70
52.75
Appendix
Table 20: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) compressed with a range of different searches with a 0.8 parameter retention . Bold / Underline denote best/second-best compressed results.
Search Method
Wiki ↓
C4 ↓
PTB ↓
Openb. ↑
ARC_e ↑
ARC_c ↑
WinoG. ↑
HellaS. ↑
PIQA ↑
Avg. ↑
Mistral -7B
uniform
14.47
67.16
181.84
0.19
0.47
0.23
0.57
0.31
0.59
39.22
ASVD
16.33
91.43
268.74
0.17
0.40
0.20
0.55
0.29
0.57
36.35
SVD-LLMv2
14.50
67.16
183.27
0.20
0.47
0.23
0.57
0.31
0.59
39.37
MRCS
14.41
70.93
222.79
0.20
0.48
0.23
0.57
0.31
0.59
39.58
ARS
18.47
33.05
176.24
0.18
0.49
0.23
0.55
0.34
0.64
40.66
ATP
15.73
59.73
188.35
0.20
0.48
0.25
0.60
0.33
0.61
41.13
Appendix
Table 21: Extended format zero-shot performance and perplexity of LLMs ( Mistral -7B, Llama 3 -8B, Qwen3 -8B) compressed with a range of different searches with a 0.6 parameter retention . Bold / Underline denote best/second-best compressed results.
Qwen3
Method
1.7B
4B
8B
14B
32B
PPL ↓
Acc ↑
PPL ↓
Acc ↑
PPL ↓
Acc ↑
PPL ↓
Acc ↑
PPL ↓
Acc ↑
Baseline
16.68
53.33
13.64
58.97
9.71
62.04
8.64
65.28
7.59
45.02
0.9
SVD-LLM
30.99
44.18
17.15
49.28
12.52
55.96
11.07
58.82
9.23
43.23
+ LEMS
17.62
51.24
14.02
57.56
9.83
62.36
8.77
65.28
7.64
65.61
KFAC-SVD
26.98
44.11
15.98
50.79
11.27
56.73
9.87
60.49
8.26
43.58
Appendix
Table 22: Detailed performance evaluation of LEMS for both KFAC-SVD and SVD-LLM across five Qwen3 model sizes (1.7B-32B). The table contrasts the baseline uniform rank allocation with our proposed LEMS across compression ratios c∈[0.5,0.9] . Bold indicates the best result between the uniform and search-based allocation for each method, showing the massive improvements brought about by LEMS .
The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).
Large language models deliver strong performance across language and reasoning tasks, but their storage and compute costs remain major barriers to deployment in resource-constrained and latency-sensitive settings. SVD-based post-training compression offers a hardware-agnostic way to reduce model size and improve inference efficiency through low-rank factorization. However, existing methods often rely on input-only whitening spaces, homogeneous rank allocation, or loss-agnostic allocation heuristics, limiting their ability to preserve model quality under aggressive compression. We propose Input-Output Whitened SVD (IO-SVD), a post-training compression method that forms a KL-aware double-sided whitening space for model weights. Using a second-order expansion of the KL loss over the top-K token probabilities, IO-SVD constructs an output-side metric that captures predictive sensitivity, while input whitening captures activation statistics. We further introduce an efficient heterogeneous rank-allocation strategy that scores whitened singular components using first-order calibration loss estimates and prunes the least sensitive components under a global budget. Inspired by prior work that combines SVD truncation with quantization, we improve hybrid SVD-quantization compression through loss-aware remapping, which selects low-rank factor rows for 8-bit quantization based on the predicted loss change incurred by quantizing them. Extensive experiments across diverse LLM and VLM families, and inference-time analysis shows that IO-SVD compresses LLMs with minimal performance degradation while delivering practical inference speedups. Code is available at https://github.com/mint-vu/IO-SVD.git
Ali Abbasi, Chayne Thrash, Haoran Qin +2
Vanderbilt University · University of California, Davis
Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost effectively. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank allocation under a given global compression ratio. Across extensive experiments on Mistral-7B, Qwen3 series, and Llama-3.3 models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms existing SVD-based baselines and achieves state-of-the-art (SoTA) results. We further demonstrate scalability by applying our method to models with up to 70B parameters.
Shengkun Tang, Cong Zeng, Zongfang Liu +2
MBZUAI · Technology Innovation Institute UAE · Zhejiang Univeristy +2