CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

Denis Makhov; Dmitriy Shopkhoev; Magauiya Zhussip; Ammar Ali; Stamatios Lefkimmiatis

cs.CLSep 26, 2025

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

Authors: Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Ammar Ali, Stamatios Lefkimmiatis

Organizations: Fundamental Research Center MWS AI · ITMO

Abstract

Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across LLaMA and Qwen models, CoSpaDi improves accuracy-compression and perplexity-compression trade-offs over SVD-based and structured pruning baselines at 20-40% compression ratios, while naturally supporting sparse-dense computation and post-training quantization of sparse coefficients.

Figures & tables

Figure 1: Left: low-rank factorization represents $\mathbf{W}$ in a single $r$ -dimensional subspace via two dense factors. Right: CoSpaDi represents $\mathbf{W}$ as $\mathbf{D}\mathbf{S}$ where $\mathbf{D}$ is a dictionary of $k$ atoms and $\mathbf{S}$ is column-sparse (at most $s$ nonzeros per column), yielding a union-of-subspaces model. Dictionaries may be undercomplete ( $k<d_{1}$ ), complete ( $k=d_{1}$ ), or overcomplete ( $k>d_{1}$ ).

		Accuracy $\uparrow$									Perplexity $\downarrow$
Method	CR	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	RACE	MMLU	Avg.	WikiText	LAMBADA
LLaMA-3-8B	–	80.7	79.1	75.6	77.7	53.5	93.9	40.3	62.2	70.4	7.3E+00	3.1E+00
SVD-LLM		71.1	58.4	59.3	55.5	34.0	86.4	35.5	32.6	54.1	4.1E+01	1.1E+01
CoSpaDi	0.2	75.2	66.5	73.8	66.5	41.6	89.5	38.2	42.8	61.8	2.0E+01	4.3E+0 0
SVD-LLM		65.8	46.4	38.1	41.9	27.7	70.0	31.8	27.2	43.6	1.5E+02	6.1E+01
CoSpaDi	0.3	70.5	56.2	61.3	54.2	33.5	85.7	36.2	32.2	53.7	4.5E+01	9.2E+00

Table 3: Performance comparison of CoSpaDi and SVD-LLM on LLaMA-3-8B and Qwen3-8B across compression ratios and evaluation benchmarks. Best results are provided in bold .

Method	CR	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	RACE	MMLU	Avg.	WikiText	LAMBADA
		Accuracy $\uparrow$									Perplexity $\downarrow$
LLaMA-3-8B		80.7	79.1	75.6	77.7	53.5	93.9	40.3	62.2	70.4	7.3E+00	3.1E+00
ReplaceMe	0.22	73.1	65.7	42.1	65.9	43.7	86.4	35.4	51.7	58.0	3.4E+01	2.0E+01
LLM-Pruner		75.5	67.5	51.0	62.1	36.6	87.8	35.1	25.0	55.1	1.6E+01	1.1E+01
CoSpaDi	0.2	75.2	66.5	73.8	66.5	41.6	89.5	38.2	42.8	61.8	2.0E+01	4.3E+00
ReplaceMe	0.31	66.6	53.8	24.0	50.7	37.9	77.3	34.0	30.6	46.9	6.7E+01	1.3E+02

Table 4: Comparison of the proposed CoSpaDi method with state-of-the-art structured pruning methods ReplaceMe Shopkhoev et al. (2025) and LLM-Pruner Ma et al. (2023) on LLaMA-3-8B under different compression ratios. We report accuracy on different benchmarks as well as its average and perplexity. Best results are provided in bold .

Method	CR	Accuracy $\uparrow$ (% (unnorm))									Perplexity $\downarrow$
Method	CR	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	RACE	MMLU	Avg.	WikiText	LAMBADA
LLaMA-2-7B	–	78.9 (78.0)	76.1 (57.3)	73.8	74.2 (76.0)	45.8 (42.7)	91.4 (93.9)	39.7	40.8	65.1 (62.8)	8.70	3.38
SVD-LLM	0.2	73.1 (72.1)	60.3 (44.4)	63.9	55.0 (62.0)	32.0 (29.1)	79.5 (89.2)	36.6	25.5	53.2 (52.9)	20.18	6.33
CoSpaDi (per-layer)		74.5 (73.6)	64.5 (47.7)	70.0	61.5 (67.2)	35.2 (34.0)	85.4 (91.3)	39.2	28.0	57.3 (56.4)	15.97	4.58
Basis Sharing		71.1 (70.2)	60.0 (43.4)	62.8	60.5 (65.5)	37.8 (33.5)	85.0 (90.8)	34.7	25.0	54.6 (53.2)	15.17	7.03
CoSpaDi (grouped)		75.3 (74.5)	66.3 (48.8)	71.1	68.1 (71.3)	39.3 (37.3)	88.5 (92.2)	38.5	27.0	59.3 (57.6)	11.73	4.42

Table 5: Performance comparison of CoSpaDi vs Basis Sharing ( Wang et al., 2025a ) and per-layer counterparts on LLaMA-2-7B under different CRs on various benchmarks. Best results are provided in bold . Unnormalized accuracies are shown in parentheses for direct comparison with ( Wang et al., 2025a ) .

Method	Global CR	UG CR	Avg. $\uparrow$	Wiki $\downarrow$	LAMB. $\downarrow$	Tok/s $\uparrow$
Dense baseline	–	–	70.4	7.3	3.1	44.0
SVD-LLM UG	0.22	0.40	53.5	56.2	8.5	53.3
CoSpaDi UG	0.22	0.40	61.9	25.0	2.8	48.0
SVD-LLM UG	0.27	0.50	43.8	213.0	54.5	52.5
CoSpaDi UG	0.27	0.50	54.8	52.0	5.8	50.5

Table 6: Selective up / gate compression on LLaMA-3-8B. Throughput is averaged over prompt lengths

1

–

256

in synchronous generation; higher is better.

Method	1 user	4 users	8 users	16 users
Dense baseline	77	290	589	1344
SVD-LLM UG	99	386	798	1332
CoSpaDi UG	96	376	758	1515

Table 7: Asynchronous serving throughput on LLaMA-3-8B for prompt length

1

. SVD-LLM UG and CoSpaDi UG use UG CR

=0.60

; higher is better.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Perplexity $\downarrow$
Method	WikiText	C4
LLaMA-7B	5.67	7.33
SVD-LLM V2, reported	7.12	10.47
SVD-LLM V2, reproduced	8.09	16.29
CoSpaDi	8.12	10.26

Appendix

Table 8: Comparison with SVD-LLM V2 on LLaMA-7B at CR

=0.2

. “Reported” denotes the values reported in the SVD-LLM V2 paper, while “reproduced” denotes our implementation. Lower perplexity is better.

Method	Target CR	Remapping	WikiText-2 $\downarrow$	C4 $\downarrow$	Avg. Acc. $\uparrow$
LLaMA-7B	–	–	5.67	7.33	0.550
Dobi-SVD ∗ , reported	0.2	No	8.54	10.01	0.456
Dobi-SVD, reported	0.2	Yes	6.08	7.83	0.507
Dobi-SVD, reproduced	0.2	Yes	6.08	8.49	0.533
CoSpaDi	0.2	No	8.12	10.26	0.508
Dobi-SVD ∗ , reported	0.4	No	13.54	23.54	0.380

Appendix

Table 9: Comparison with Dobi-SVD on LLaMA-7B. Dobi-SVD ∗ denotes Dobi-SVD without remapping, as reported in the original paper. “Reproduced” denotes our evaluation of the released remapped checkpoints. Average accuracy is computed over OpenBookQA, ARC-easy, Winogrande, HellaSwag, ARC-challenge, PIQA, and MathQA. Dobi-SVD remapping changes the relation between factorization rank and final storage and is therefore indicated separately rather than expressed as a separate factorization compression ratio. For a square projection, the same remapped storage budget can correspond to a negative conventional factorization compression ratio; see Eq. ( 61 ).

Hyperparameter	Value
Random seed	42
K-SVD iterations $T$	60
Ratio $\rho=k/s$	2.0
Dictionary update solver	power-iteration K-SVD
Power iterations	8
OMP batch size	8192

Appendix

Table 10: Main hyperparameters used in the experiments unless otherwise specified. The dictionary size

k

and sparsity level

s

are determined from the target compression ratio and the ratio

\rho=k/s

via Eq. ( 11 ).

Calibration data	Method	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	RACE	MMLU	Avg.	WikiText	LAMBADA
		Accuracy $\uparrow$									Perplexity $\downarrow$
	SVD-LLM	62.13	36.43	24.43	36.03	25.09	64.90	29.00	22.98	37.62	1.69E+02	1.70E+02
RefinedWeb	CoSpaDi	66.05	42.88	38.39	39.94	26.02	71.60	31.67	24.83	42.67	6.37E+01	3.51E+01
	SVD-LLM	63.50	36.20	12.70	46.20	25.80	67.50	26.50	24.00	37.80	2.82E+02	9.24E+02
Alpaca	CoSpaDi	65.50	40.10	21.20	48.10	27.60	74.70	29.60	25.40	41.53	1.79E+02	2.60E+02
	SVD-LLM	58.30	35.50	23.10	40.60	25.30	69.50	24.80	23.10	37.53	1.04E+02	1.91E+02

Appendix

Table 11: Sensitivity to the calibration domain on LLaMA-3.2-1B at

\mathrm{CR}=0.2

. All calibration sets contain 256 sequences of length 1024. We report accuracy on different benchmarks as well as its average and perplexity on WikiText and LAMBADA. Best results are provided in bold .

# Calib. samples	Method	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	RACE	MMLU	Avg.	WikiText	LAMBADA
		Accuracy $\uparrow$									Perplexity $\downarrow$
	SVD-LLM	59.30	35.30	19.80	35.10	24.10	58.80	28.30	23.10	35.48	2.03E+02	3.04E+02
64	CoSpaDi	62.00	39.20	33.90	40.10	25.30	71.10	30.30	23.30	40.65	1.15E+02	6.38E+01
	SVD-LLM	60.00	35.80	19.00	35.30	24.30	60.20	27.10	23.00	35.59	1.95E+02	3.16E+02
128	CoSpaDi	62.80	39.80	31.20	39.00	25.20	68.30	30.40	23.20	39.99	1.15E+02	7.84E+01
	SVD-LLM	62.13	36.43	24.43	36.03	25.09	64.90	29.00	22.98	37.62	1.69E+02	1.70E+02

Appendix

Table 12: Sensitivity to the number of calibration samples on LLaMA-3.2-1B at

\mathrm{CR}=0.2

using RefinedWeb calibration data. All calibration sequences have length 1024. We report accuracy on different benchmarks as well as its average and perplexity on WikiText and LAMBADA. Best results are provided in bold .

CR	SVD-LLM Avg.	CoSpaDi Avg.	Gain	Gain / $\mathrm{SE}_{\Delta}$
0.2	0.3762	0.4267	+0.0505	$9.6\times$
0.3	0.3022	0.3367	+0.0346	$6.6\times$
0.4	0.2621	0.2753	+0.0132	$2.6\times$

Appendix

Table 13: Average-accuracy gains relative to the combined standard error estimated from task-level lm-evaluation-harness outputs on LLaMA-3.2-1B. The displayed average accuracies are rounded to four decimal places; gains are computed from the underlying unrounded averages.

Panel of Figure 4 — Figure 4: Average benchmark accuracy and WikiText perplexity with respect to the number of K-SVD iterations (left) and the number of power iterations (right) for LLaMA-3.2-1B with $\rho=2$

Component	Time (min)	Share (%)
Sparse coding (OMP)	178.5	27.6
Dictionary update (K-SVD)	467.6	72.3
Misc. overhead	0.7	0.1
Total	646.8	100.0

Appendix

Table 14: Runtime breakdown of CoSpaDi on a 1B-scale model for end-to-end compression of all targeted projection matrices under the current solver configuration.

Model / Setting	Applied eq.	Quant. coeff.	Dictionary	Coeff. values	Mask / indices	Full total
LLaMA-3.2-1B	–	No	–	–	–	1856
CoSpaDi, CR=20%	Eq. 10	No	990	438	54	1484
CoSpaDi, CR=20%	Eq. 11	Yes	1018	408	58	1484
CoSpaDi, CR=30%	Eq. 10	No	866	383	47	1298
CoSpaDi, CR=30%	Eq. 11	Yes	890	357	51	1298
CoSpaDi, CR=40%	Eq. 10	No	743	329	41	1113

Appendix

Table 15: Memory footprint breakdown (MiB) for storing all compressed linear layers (excluding embeddings, lm_head , and normalization layers) under CoSpaDi at different compression ratios.

Figure 5: Average benchmark accuracy and WikiText perplexity for LLaMA-3.2-1B and Qwen3-0.6B using SVD-LLM and CoSpaDi with respect to compression ratio.

		Accuracy $\uparrow$									Perplexity $\downarrow$
Method	CR	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	RACE	MMLU	Avg.	WikiText	LAMBADA
Qwen3-14B	–	79.9	78.9	67.9	82.8	60.2	96.5	43.3	77.2	73.3	1.1E+01	3.7E+00
SVD-LLM		76.2	67.6	69.1	69.8	46.8	91.0	42.9	62.5	65.8	1.8E+01	4.1E+00
CoSpaDi	0.2	77.3	71.7	72.0	73.3	51.3	91.9	43.1	65.8	68.3	1.6E+01	3.4E+00
SVD-LLM		72.0	59.7	61.5	62.6	39.3	88.9	41.5	52.0	59.7	2.3E+01	6.5E+00
CoSpaDi	0.3	74.6	64.2	69.0	65.3	43.6	90.7	42.5	58.0	63.5	2.0E+01	4.2E+00

Appendix

Table 16: Performance comparison of CoSpaDi vs SVD-LLM on Qwen3-14B at different compression levels on different benchmarks. Best results are provided in bold .

Method	CR	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	RACE	MMLU	Avg.	WikiText	LAMBADA
		Accuracy $\uparrow$									Perplexity $\downarrow$
LLaMA-3-8B	–	80.7	79.1	75.6	77.7	53.5	93.9	40.3	62.2	70.4	7.3E+00	3.1E+00
SVD-LLM	0.4	60.3	34.5	11.4	32.4	24.5	44.2	25.7	23.1	32.0	5.5E+02	1.3E+03
Wanda 2:4	0.44	62.4	39.7	0.8	41.9	25.6	74.7	26.8	23.4	36.9	2.1E+02	5.7E+03
CoSpaDi	0.4	63.7	41.4	30.3	39.1	26.6	68.5	30.5	25.4	40.7	1.8E+02	1.2E+02

Appendix

Table 17: Performance comparison on LLaMA-3-8B between SVD-LLM, CoSpaDi, and Wanda 2:4 at compression ratio 0.4. Best compressed result in each column is highlighted in bold .

Model	CR	IFEval	BBH	MATH	GPQA	MUSR	MMLU-Pro
		(%)	(%)	(%)	(%)	(%)	(%)
Qwen3-8B	—	39.2	60.9	52.6	36.2	43.1	47.7
SVD-LLM		25.5	41.0	1.1	28.4	39.8	26.3
CoSpaDi	0.2	28.9	45.3	2.0	28.6	42.1	31.5
SVD-LLM		22.9	34.4	1.0	25.6	41.4	18.8
CoSpaDi	0.3	25.2	38.2	1.0	24.8	38.4	22.8

Appendix

Table 18: Performance comparison of CoSpaDi vs SVD-LLM on Qwen3-8B and Qwen3-14B at different compression levels on different benchmarks. Best results are provided in bold .

Method	Global CR	Accuracy $\uparrow$									Perplexity $\downarrow$
Method	Global CR	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	Race	MMLU	Avg.	Wiki	LAMBADA
LLaMA-3-8B	–	80.7	79.1	75.6	77.7	53.5	93.9	40.3	62.2	70.4	7.3E+00	3.1E+00
CoSpaDi	0.20	75.2	66.5	73.8	66.5	41.6	89.5	38.2	42.8	61.8	2.0E+01	4.3E+00
CoSpaDi UG	0.22	75.3	63.2	76.7	67.8	42.1	92.5	37.3	40.2	61.9	2.5E+01	2.8E+00
SVD-LLM UG	0.22	71.9	51.9	56.7	58.8	33.5	87.7	33.8	33.9	53.5	5.6E+01	8.5E+00
CoSpaDi	0.30	70.5	56.2	61.3	54.2	33.5	85.7	36.2	32.2	53.7	4.5E+01	9.2E+00

Appendix

Table 19: Quality of selective up / gate compression on LLaMA-3-8B. Superscript

\mathrm{UG}

denotes compressing only the MLP up and gate projections.

Prompt	UG CR	SVD-LLM UG	CoSpaDi UG
1	0.3	51.2	46.8
16		50.3	46.8
32		51.3	45.9
64		50.3	45.3
128		50.0	44.8
256		49.5	44.0

Appendix

Table 20: Synchronous generation throughput on LLaMA-3-8B using an RTX 3090. Throughput is reported in tokens/s; higher is better.

Method	1 user	4 users	8 users	16 users
Dense baseline	77.0	290.0	589.0	1344.0
SVD-LLM UG	99.4	386.0	798.0	1332.0
CoSpaDi UG	95.6	375.6	758.0	1515.0

Appendix

Table 21: Asynchronous serving throughput on LLaMA-3-8B for prompt length

1

. SVD-LLM UG and CoSpaDi UG use UG CR

=0.60

and

\rho=2

; higher is better.

Users	Prompt 1	Prompt 16	Prompt 32	Prompt 64	Prompt 128
1	95.6	93.5	92.4	101.0	98.0
4	375.6	374.8	372.4	393.0	401.0
8	758.0	742.0	749.4	802.0	792.0
16	1515.0	1479.6	1472.0	1521.0	1573.0

Appendix

Table 22: Asynchronous serving throughput of CoSpaDi UG on LLaMA-3-8B with UG CR

=0.60

and

\rho=2

. Throughput is reported in tokens/s; higher is better.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 294 papers

DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels
Nov 5, 2026Jinho Park, Se Young Chun, Mingoo SeokModel Compression LLM Compression
Columbia University · Seoul National University
COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression
Feb 16, 2026Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip +3Orthogonal Procrustes Alignment Sparse Coding
Fundamental Research Center MWS AI · ITMO
SPQ: An Ensemble Technique for Large Language Model Compression
Feb 20, 2026Jiamin Yao, Eren GultepeLLM Compression LLM Quantization
Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

Nov 5, 2026J/K move · Enter open · S save

DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

Jinho Park, Se Young Chun, Mingoo Seok

Columbia University · Seoul National University

Model Compression LLM Compression Low-Rank Compression

Open paper AskPDF arXiv

Efficient deployment of large language models (LLMs) on mobile devices requires balancing performance with compute and memory constraints. Post-training compression is an effective approach to managing this trade-off, while requiring much less computation than in-training counterparts. We propose DictLLM, a dictionary-based compression scheme in which weight matrices within grouped layers are decomposed into a shared dictionary and layer-specific sparse coefficients, directly obtained from a pre-trained model without further training. Specifically, we cast DictLLM as a four-step post-training optimization. First, we obtain shared dictionaries via singular value decomposition (SVD) of input-scaled weight matrices, amortizing rank across grouped layers; second, we sparsify the layer-specific coefficients to the target compression ratio using a dictionary-aware Hessian; and third, we refine the dictionaries given the resulting sparse coefficients. Finally, we aggressively quantize both components using factor-specific Hessian information: input covariance for the dictionaries and the dictionary-aware Hessian for the coefficients. The resulting representation reduces memory footprint by storing a single shared dictionary per layer group, along with sparse layer-specific coefficients. On language modeling and zero-shot reasoning benchmarks, DictLLM consistently outperforms existing post-training low-rank compression methods and pushes the Pareto frontier between model size and performance. DictLLM compresses the original LLaMA-7B model by 5.1x with only a 2.9-point increase in WikiText-2 perplexity, achieving a 3.6x higher compression ratio than the state-of-the-art low-rank method.

Abstract

LLM Compression

Explore similar work

DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

SPQ: An Ensemble Technique for Large Language Model Compression