SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping

Yu-Chen Lu; Sheng-Feng Yu; Hui-Hsien Weng; Pei-Shuo Wang; Yu-Fang Hu; Liang Hung-Chun; Hung-Yueh Chiang; Kai-Chiang Wu

cs.CLDec 15, 2025

SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping

Authors: Yu-Chen Lu, Sheng-Feng Yu, Hui-Hsien Weng, Pei-Shuo Wang, Yu-Fang Hu, Liang Hung-Chun, Hung-Yueh Chiang, Kai-Chiang Wu

Organizations: National Yang Ming Chiao Tung University · Macronix International Co., Ltd. · Skymizer Taiwan Inc. · The University of Texas at Austin

Abstract

Large language models (LLM) have achieved remarkable performance across a wide range of tasks. However, their substantial parameter sizes pose significant challenges for deployment on edge devices with limited computational and memory resources. Low-rank compression is a promising approach to address this issue, as it reduces both computational and memory costs, making LLM more suitable for resource-constrained environments. Nonetheless, naïve low-rank compression methods require a significant reduction in the retained rank to achieve meaningful memory and computation savings. For a low-rank model, the ranks need to be reduced by more than half to yield efficiency gains. Such aggressive truncation, however, typically results in substantial performance degradation. To address this trade-off, we propose SkipCat, a novel low-rank compression framework that enables the use of higher ranks while achieving the same compression rates. First, we introduce an intra-layer shared low-rank projection method, where multiple matrices that share the same input use a common projection. This reduces redundancy and improves compression efficiency. Second, we propose a block skipping technique that omits computations and memory transfers for selected sub-blocks within the low-rank decomposition. These two techniques jointly enable our compressed model to retain more effective ranks under the same compression budget. Experimental results show that, without any additional fine-tuning, our method outperforms previous low-rank compression approaches by 7% accuracy improvement on zero-shot tasks under the same compression rate. These results highlight the effectiveness of our rank-maximized compression strategy in preserving model performance under tight resource constraints.

Figures & tables

Figure 1: Computational cost (FLOPs) and memory footprint (MB) of the LLaMA2-7B attention module versus retained rank

r

. The black dotted line represents the original computational cost and memory footprint of the module. The gray dashed line corresponds to a naïve SVD compression, which only achieves effective compression when the retained rank is reduced to less than half of the full rank. In contrast, SkipCat maximizes the ranks and reduces the computational and memory cost, resulting in better trade-offs between efficiency and performance.

Figure 2: An illustration comparing our proposed method SkipCat with naïve low-rank compression. (a) The original model layer. (b) The model structure after applying naïve SVD low-rank compression. (c) Our Cat technique, which shares low-rank projection matrices across modules, enabling more ranks to be preserved without changing the compression rate. (d) Our Skip technique, applied on top of the Cat architecture, skips submatrix computations for all low-rank projections and further increases the number of preserved ranks.

Figure 3: The output activation distribution of low-rank projections in LLaMA2-7B after applying block skipping. Left: Without column permutation, activation outliers cause FP16 overflow. Right: Column permutation stabilizes the distribution, preventing overflow in low-precision inference.

Model	Comp. Rate	Method	Perplexity ↓		Zero-shot Task Accuracy (%) ↑							Avg. (%)	Drop (%)
Model	Comp. Rate	Method	Wiki2	C4	ARC-e	ARC-c	Hella	OBQA	Wino	MathQA	PIQA	↑	↓
LLaMA2-7B	-	Dense	5.47	7.26	76.30	43.34	57.14	31.40	69.14	28.17	78.07	54.79	-
	20%	ASVD	9.06	11.66	67.80	34.56	48.25	29.60	63.77	25.49	72.20	48.81	5.98
		Basis Sharing	9.39	23.30	59.18	27.82	38.22	24.20	66.54	23.89	66.54	43.77	11.02
		Dobi-SVD	9.39	19.46	54.25	23.29	38.80	22.80	58.56	22.75	65.34	40.83	13.97
		SVD-LLM	8.82	13.42	58.67	27.65	43.10	26.20	64.25	23.85	70.18	44.84	9.95

Table 1: Perplexity and zero-shot accuracy of low-rank compression methods.

Model	Comp. Rate	Method	Zero-shot Task Accuracy
Model	Comp. Rate	Method	Avg. (%) ↑	Drop (%) ↓
LLaMA2-13B	-	Dense	58.07	-
	20%	SVD-LLM	50.09	7.98
	20%	SkipCat	54.15	3.92
	30%	SVD-LLM	46.15	11.93
	30%	SkipCat	49.29	8.78

Table 2: Zero-short accuracy of compressed larger models.

Comp. Rate	Ablation Settings			Perplexity ↓
Comp. Rate	Cat	Skip	Quant.	Wiki2	C4
20%	✗	✗	✗	8.82	13.42
	✓	✗	✗	7.84	11.99
	✗	✓	✗	6.71	9.32
	✓	✓	✗	6.29	8.95
	✓	✓	✓	6.29	8.96

Table 3: Ablation study results on LLaMA2-7B.

Comp. Rate	Method	Zero-shot Task Accuracy
Comp. Rate	Method	Avg. (%) ↑	Drop (%) ↓
-	Dense	54.79	-
20%	SVD-LLM	51.02	3.78
	SkipCat	54.41	0.39
40%	SVD-LLM	46.91	7.88
	SkipCat	48.65	6.15

Table 4: Zero-shot accuracy of fine-tuned compressed models based on LLaMA2-7B.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Visualization comparing the weights after applying Block Skipping, with and without column permutation. Applying column permutation before Block Skipping reduces outliers and improves numerical stability.

Comp. Rate	Data Type	Wiki2 Perplexity ↓
10%	FP32	5.68
	FP16	13.34
	BF16	31628.76
20%	FP32	6.19
	FP16	Overflow
	BF16	11294.57

Appendix

Table 5: Comparison of perplexity across different data types for LLaMA2-7B with Block Skipping applied without prior column permutation.

Comp. Rate	Ablation Settings			Perplexity ↓
Comp. Rate	Quant	Hadamard	Scaling	Wiki2	C4
20%	✗	✗	✗	6.29	8.95
	✓	✗	✗	6.36	9.06
	✓	✓	✗	6.32	8.97
	✓	✗	✓	6.30	8.98
	✓	✓	✓	6.29	8.96

Appendix

Table 6: Ablation study on quantization stabilization.

Figure 5: Perplexity and zero-shot accuracy of compressed LLaMA2-7B using different calibration datasets for whitening.

Figure 6: Effect of calibration dataset size on zero-shot accuracy after whitening.

Model	Comp. Rate	Method	Perplexity ↓		Zero-shot Task Accuracy (%) ↑							Avg. (%)	Drop (%)
Model	Comp. Rate	Method	Wiki2	C4	ARC-e	ARC-c	Hella	OBQA	Wino	MathQA	PIQA	↑	↓
LLaMA2-13B	-	Dense	4.88	6.73	79.42	48.29	60.07	35.20	72.22	32.19	79.11	58.07	-
	20%	SVD-LLM	6.83	10.80	69.49	36.18	47.63	28.80	68.11	25.29	75.14	50.09	7.98
	20%	SkipCat	5.79	8.50	74.96	39.93	53.72	32.20	72.14	28.68	77.42	54.15	3.92
	30%	SVD-LLM	8.43	14.39	63.51	29.18	42.03	25.60	67.32	24.05	71.33	46.15	11.93
	30%	SkipCat	6.99	11.19	67.42	33.79	45.78	27.80	69.69	25.76	74.81	49.29	8.78

Appendix

Table 7: Perplexity and zero-shot accuracy of compressed larger models.

Comp. Rate	Method	Zero-shot Task Accuracy (%) ↑							Avg. (%)	Drop (%)
Comp. Rate	Method	ARC-e	ARC-c	Hella	OBQA	Wino	MathQA	PIQA	↑	↓
-	Dense	76.30	43.34	57.14	31.40	69.14	28.17	78.07	54.79	-
20%	SVD-LLM	70.03	38.23	52.88	29.60	65.19	25.29	75.90	51.02	3.78
	SkipCat	75.08	42.83	56.00	33.80	68.11	27.54	77.48	54.41	0.39
40%	SVD-LLM	63.76	33.11	47.05	26.40	61.96	23.95	72.14	46.91	7.88
	SkipCat	66.71	33.62	49.92	28.60	62.27	24.66	74.76	48.65	6.15

Appendix

Table 8: Zero-shot accuracy of fine-tuned compressed models.

Comp. Rate	Method	Zero-shot Task Accuracy (%) $\uparrow$							Avg. (%)	Drop (%)
Comp. Rate	Method	ARC-e	ARC-c	Hella	OBQA	Wino	MathQA	PIQA	$\uparrow$	$\downarrow$
-	Dense	81.48	51.19	60.01	33.40	73.64	39.60	80.25	59.94	-
20%	SVD-LLM	58.80	28.58	41.40	19.60	61.40	25.06	68.99	43.40	16.53
	SkipCat	68.90	36.60	50.13	25.60	65.67	29.82	74.86	50.23	9.71
30%	SVD-LLM	45.58	22.18	33.36	17.60	56.20	23.55	62.19	37.24	22.70
	SkipCat	51.22	23.55	38.60	19.00	56.91	24.49	65.45	39.89	20.05

Appendix

Table 9: Zero-shot accuracy of compressed LLaMA3.1-8B.

Comp. Rate	Method	Zero-shot Task Accuracy (%) $\uparrow$						Avg. (%)	Drop (%)
Comp. Rate	Method	ARC-e	Hella	OBQA	Wino	MathQA	PIQA	$\uparrow$	$\downarrow$
-	Dense	75	57	34	70	27	79	57	-
20%	SVD-LLM V2	72	52	32	70	24	75	54	3
20%	SkipCat	75	56	35	68	26	78	56	1

Appendix

Table 10: Comparison of zero-shot accuracy between SVD-LLM V2 and our method on LLaMA-7B at a 20% compression rate. The results for SVD-LLM V2 are taken from the original paper.

Method	Comp. Rate	TTFT (sec)	Speedup
Dense	-	1.14	-
SkipCat	20%	0.99	1.15x
	40%	0.79	1.44x
	60%	0.57	2.00x
	80%	0.35	3.26x

Appendix

Table 11: Time to First Token (TTFT) of LLaMA2-7B with SkipCat compression at varying compression rates.

Figure 7: Throughput of LLaMA2-7B under different compression rates with varying batch sizes and input sequence lengths. Top: Throughput results with a fixed input sequence length of 32 and varying batch sizes. Bottom: Throughput results with a fixed batch size of 64 and varying input sequence lengths.

Method	Comp. Rate	Throughput (tokens/sec)	Speedup
Dense	-	8.63	-
SkipCat	20%	16.10	1.87x
	40%	20.84	2.41x
	60%	33.99	3.94x
	80%	58.05	6.73x

Appendix

Table 12: Throughput of LLaMA2-7B under model offloading with varying SkipCat compression rates.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Jun 20, 2026Chao Han, Yongjie Du, Junjie Tan +1LLM Compression Low-Rank Matrix Decomposition
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo
Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
Oct 7, 2025Ryan Solgi, Parsa Madinei, Jiayi Tian +4Model Compression LLM Compression
University of California-Santa Barbara, USA · Amazon, USA
Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
Aug 25, 2026Mohammad MozaffariLLM Compression Neural Network Compression
Department of Computer Science University of Toronto

Jun 20, 2026J/K move · Enter open · S save

UniRank: Unified Rank Allocation for Low-Rank LLM Compression

Chao Han, Yongjie Du, Junjie Tan, Zihao Xuan

Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo

LLM Compression Low-Rank Matrix Decomposition Low-Rank Compression Low-Rank Approximation Adaptive Rank Allocation

Open paper AskPDF arXiv

Low-rank decomposition is a promising compression paradigm for large language models (LLMs), yet its effectiveness hinges on rank budget allocation across weight matrices: uniform or hand-crafted rules ignore module-wise importance, while learning-based allocation incurs substantial training overhead. We formulate rank allocation as a global sorting-and-truncation pipeline that scores every singular component by combining local singular energy with global functional importance, estimated via layer-wise input--output cosine similarity on a tiny calibration set. We show, both geometrically and empirically, that high input--output cosine similarity implies low effective rank. We further propose rank-preserving fine-tuning (RPFT), which adapts only a small subset of retained singular components so that the allocated rank stays bounded without re-decomposition. Experimental results show that UniRank cuts zero-shot perplexity by up to 50%, improves average reasoning accuracy by 3.0% over LoRAP at 25% sparsity, and boosts four SVD-based decomposition methods as a plug-and-play module.

Abstract

LLM Compression

Explore similar work

UniRank: Unified Rank Allocation for Low-Rank LLM Compression

Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression