DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

Jinho Park; Se Young Chun; Mingoo Seok

cs.AINov 5, 2026

DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

Authors: Jinho Park, Se Young Chun, Mingoo Seok

Organizations: Columbia University · Seoul National University

Abstract

Efficient deployment of large language models (LLMs) on mobile devices requires balancing performance with compute and memory constraints. Post-training compression is an effective approach to managing this trade-off, while requiring much less computation than in-training counterparts. We propose DictLLM, a dictionary-based compression scheme in which weight matrices within grouped layers are decomposed into a shared dictionary and layer-specific sparse coefficients, directly obtained from a pre-trained model without further training. Specifically, we cast DictLLM as a four-step post-training optimization. First, we obtain shared dictionaries via singular value decomposition (SVD) of input-scaled weight matrices, amortizing rank across grouped layers; second, we sparsify the layer-specific coefficients to the target compression ratio using a dictionary-aware Hessian; and third, we refine the dictionaries given the resulting sparse coefficients. Finally, we aggressively quantize both components using factor-specific Hessian information: input covariance for the dictionaries and the dictionary-aware Hessian for the coefficients. The resulting representation reduces memory footprint by storing a single shared dictionary per layer group, along with sparse layer-specific coefficients. On language modeling and zero-shot reasoning benchmarks, DictLLM consistently outperforms existing post-training low-rank compression methods and pushes the Pareto frontier between model size and performance. DictLLM compresses the original LLaMA-7B model by 5.1x with only a 2.9-point increase in WikiText-2 perplexity, achieving a 3.6x higher compression ratio than the state-of-the-art low-rank method.

Figures & tables

Figure 1: DictLLM: Dictionary-based compression method. DictLLM resolves the weight memory bottleneck by compressing transformer weights into a shareable dictionary. It compounds four compression steps. ➊ The first step factorizes input-scaled weight matrices of grouped layers into a shared dictionary and layer-specific coefficients. ➋ We then prune the layer-specific coefficients using a dictionary-aware Hessian. ➌ We refit the shared dictionary to the resulting sparse coefficients. ➍ We quantize both the dictionary and the coefficients into NVFP4 using input statistics for the dictionary and dictionary-aware Hessians for the coefficients. The resulting compressed representation enables efficient projections by transferring a small set of shared dictionaries.

Figure 3

Comp.	Method	Perplexity $\downarrow$			Zero-shot accuracy $\uparrow$
gain	Method	WikiText-2	PTB	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	8.35	7.34	0.28	0.67	0.56	0.78	0.27	0.67	0.38	0.52
1.25×	ASVD [ 11 ]	11.14	16.55	15.93	0.25	0.64	0.41	0.68	0.24	0.53	0.27	0.43
	SVD-LLM [ 8 ]	7.94	18.05	15.93	0.22	0.63	0.43	0.69	0.24	0.58	0.29	0.44
	Basis Sharing [ 9 ]	7.74	17.35	15.03	0.28	0.66	0.46	0.71	0.25	0.66	0.36	0.48
	DictLLM (Ours)	6.89	12.27	11.75	0.33	0.67	0.50	0.74	0.26	0.69	0.36	0.51

Table 1: Benchmark on language modeling and reasoning tasks. We benchmark DictLLM against low-rank compression baselines on language modeling and commonsense reasoning tasks for LLaMA-7B at different compression ratios. We additionally report the FP4-quantized DictLLM FP4 models at the bottom of the table. We measure perplexity on WikiText-2, Penn TreeBank (PTB), and C4, and zero-shot accuracy on OpenBookQA (OB), WinoGrande (WG), HellaSwag (HS), PIQA (PQ), MathQA (MQ), ARC-Easy (AE), and ARC-Challenge (AC).

Figure 4: Perplexity-model size trade-off. WikiText-2 perplexity of compressed LLaMA-7B across factorization ratios and coefficient sparsities. DictLLM factorizes weights at rate

\rho_{\mathrm{fac}}

and then prunes layer-specific coefficients with 20%–50% sparsity. We also report their NVFP4 versions.

Table 6

Figure 5: Ablation of dictionary update. WikiText-2 perplexity of LLaMA-7B across different sparsity levels, with and without dictionary update.

Method	Dictionary $\mathbf{D}$	Coefficient $\mathbf{C}$	PPL $\downarrow$	Comp. gain $\uparrow$
Unquantized baseline	FP16	FP16	12.98	1.00 $\times$
Mixed precision 1	NVFP4	FP16	13.66	1.61 $\times$
Mixed precision 2	NVFP4	FP8	14.71	2.60 $\times$
Mixed precision 3	NVFP4	INT8	16.05	2.60 $\times$
All NVFP4 (Ours)	NVFP4	NVFP4	16.82	3.56 $\times$

Table 4: Different quantization format. We compare quantization choices for the shared dictionary

\mathbf{D}

and layer-specific coefficients

\mathbf{C}

. Perplexity is measured on WikiText-2 using LLaMA-7B, and compression gain denotes the reduction in model size. The FP16/FP16 baseline corresponds to the factorized model with a 30% factorization reduction and 50% coefficient sparsity before quantization.

Table 9

Figure 6: Effect of calibration size. WikiText-2 perplexity of LLaMA-7B as a function of calibration set size.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	7.09	11.87	32.20	72.53	59.03	79.16	55.41	81.61	53.24	61.88
1.25×	SVD-LLM [ 8 ]	11.52	32.16	29.00	66.22	44.16	71.55	31.56	67.51	38.31	49.76
	Basis Sharing [ 9 ]	11.14	29.37	31.40	67.56	45.77	71.93	32.26	67.21	38.99	50.73
	DictLLM (Ours)	9.90	22.59	31.00	70.09	50.07	74.86	35.78	70.54	41.89	53.46
1.43×	SVD-LLM [ 8 ]	14.30	51.96	26.20	64.33	37.91	65.02	26.77	55.51	28.50	43.46

Appendix

Table A3 : Benchmark on language modeling and reasoning tasks with Qwen3-8B. We compare DictLLM against low-rank compression baselines at compression gains from 1.25× to 2×. We report perplexity on WikiText-2 and C4, and zero-shot accuracy on OpenBookQA (OB), WinoGrande (WG), HellaSwag (HS), PIQA (PQ), MathQA (MQ), ARC-Easy (AE), and ARC-Challenge (AC).

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	9.51	15.34	29.80	63.54	49.29	75.73	45.19	73.74	41.04	54.05
1.25×	SVD-LLM [ 8 ]	23.20	96.37	22.80	56.83	34.98	61.64	24.15	44.15	24.66	38.46
	Basis Sharing [ 9 ]	22.93	95.07	22.60	56.20	36.66	62.02	26.16	48.82	27.13	39.94
	DictLLM (Ours)	17.66	52.80	25.80	59.91	40.63	67.85	27.84	53.37	28.75	43.45
1.43×	SVD-LLM [ 8 ]	34.60	190.49	19.00	55.01	31.08	58.32	23.75	38.13	22.95	35.46

Appendix

Table A4 : Qwen3-1.7B language modeling and reasoning results. Methods are compared at the reported compression gains.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	6.46	10.95	35.40	74.19	61.75	80.41	61.78	83.33	55.63	64.64
1.25×	SVD-LLM [ 8 ]	9.49	27.15	28.80	68.98	47.19	72.69	34.34	67.00	38.48	51.07
	Basis Sharing [ 9 ]	9.18	25.93	30.20	70.24	49.32	74.32	36.98	66.08	39.33	52.35
	DictLLM (Ours)	8.29	19.69	32.20	70.32	54.48	76.22	39.90	73.02	43.60	55.68
1.43×	SVD-LLM [ 8 ]	11.51	44.54	26.00	65.11	39.29	67.08	29.95	62.25	31.57	45.89

Appendix

Table A5 : Qwen3-14B language modeling and reasoning results. The improvements over other baselines extend to the largest model evaluated in language modeling and reasoning tasks.

Compression gain	Factorization ratio $\rho_{\text{fac}}$	Sparsity $p$
Compression gain	Factorization ratio $\rho_{\text{fac}}$	Qwen3-8B	Qwen3-1.7B	Qwen3-14B
1.25×	10%	17.25%	17.57%	17.52%
1.43×	20%	19.40%	19.79%	19.71%
1.67×	30%	22.17%	-	-
2×	30%	44.38%	-	-

Appendix

Table A6 : Factorization and coefficient sparsity settings for Qwen3 model family. We report the factorization ratio

\rho_{\text{fac}}

and Sparsity

p

required to achieve a given compression gain for Qwen3-8B, 1.7B, and 14B models.

Stage	Comp.	Components				Perplexity $\downarrow$		Zero-shot accuracy $\uparrow$
Stage	gain	F	P	R	Q	Wiki-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Avg.
Original	1×	–	–	–	–	5.68	7.34	0.28	0.67	0.56	0.78	0.27	0.67	0.38	0.52
+ Factorization	1.43×	$\checkmark$	–	–	–	9.25	22.46	0.27	0.63	0.40	0.68	0.24	0.63	0.30	0.45
+ Hessian pruning	2×	$\checkmark$	$\checkmark$	–	–	11.53	33.82	0.25	0.62	0.36	0.64	0.23	0.57	0.27	0.42
+ Dictionary refit	2×	$\checkmark$	$\checkmark$	$\checkmark$	–	10.59	29.12	0.24	0.63	0.38	0.65	0.23	0.61	0.29	0.43
+ Quantization	7×	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	11.92	33.90	0.24	0.59	0.35	0.63	0.22	0.54	0.26	0.40

Appendix

Table A7 : Componentwise ablation of compression stages. ‘F’, ‘P’, ‘R’, and ‘Q’ respectively denote factorization, Hessian pruning, dictionary refitting, and quantization. A checkmark indicates an active component. We measure language and reasoning performance on LLaMA-7B.

Figure A1 : Spectral concentration of LLaMA-7B query projection weights. We plot cumulative spectral energy for single-layer SVD (a) with and (b) without input covariance scaling and cross-layer SVD (c) with and (d) without scaling across layers

\ell

. Input-scaled spectra rise more rapidly, showing that the energy of input-scaled weights is more concentrated in low-rank components than that of the raw weight spectrum. Vertical dashed lines indicate the selected rank

k\in\{1433,1911\}

for the target factorization rate

\rho_{\text{fac}}=30\%

in single-layer and cross-layer settings.

Figure A2 : Spectral concentration across different weight matrices in LLaMA-7B. We plot cumulative spectral energy for the shared projections K, V, gate, and up using cross-layer SVD, and for the private projections O and down using single-layer SVD, following our factorization design. Vertical dashed lines indicate the selected rank

k

for the target factorization rate

\rho_{\text{fac}}=30\%

under the corresponding single-layer or cross-layer setting.

Figure A3 : Aggressive layer sharing with LoRA fine-tuning. We increase the dictionary-sharing group size to

n=4,8,16

and apply 60% coefficient sparsity while keeping the rank of the shared dictionary fixed, then use LoRA fine-tuning to restore performance.

Calibration size	Perplexity $\downarrow$
16	9.06
32	8.98
64	8.90
128	8.88

Appendix

Table A8 : Calibration size for NVFP4 quantization. Post-quantization WikiText-2 perplexity of LLaMA-7B. Using 16 calibration samples provides a strong trade-off between perplexity and runtime.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
5×	SparseGPT [ 14 ]	86.72	257.96	14.00	48.46	25.38	51.41	18.79	25.63	21.08	29.25
	Wanda [ 15 ]	2.8 × 10 3	2.8 × 10 3	11.60	48.30	25.78	53.10	21.21	26.35	20.31	29.52
	DictLLM FP4	8.48	17.04	29.20	64.64	43.67	70.18	23.99	64.02	32.76	46.92

Appendix

Table A9 : Comparison with pruning-only methods. SparseGPT and Wanda are evaluated on LLaMA-7B at 80.32% unstructured sparsity. All results are reproduced by us.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
7×	GPTQ [ 19 ] (2-bit)	17.59	36.56	13.80	48.38	30.47	57.13	20.84	32.03	21.33	32.00
	AWQ [ 34 ] (2-bit)	2.6 × 10 5	2.9 × 10 5	13.60	49.33	25.40	52.83	21.57	25.38	22.35	30.07
	DictLLM FP4	11.92	33.90	23.60	59.19	35.32	63.17	22.21	53.62	25.60	40.39

Appendix

Table A10 : Comparison with quantization-only baselines. We compare methods at 7× compression gain on LLaMA-7B. GPTQ [ 19 ] and AWQ [ 34 ] use two-bit weights, whereas DictLLM combines factorization and coefficient sparsity with four-bit quantization. All results are reproduced by us.

Method	LLaMA-7B	OPT-6.7B
ASVD [ 11 ]	11.14	82.0
SVD-LLM [ 8 ]	7.94	16.04
SVD-LLM V2 [ 56 ]	7.12	13.46
Basis Sharing [ 9 ]	7.74	11.79
DictLLM (Ours)	6.89	11.36

Appendix

Table A11: Comparison with low rank methods including SVD-LLM V2. We measure WikiText-2 perplexity at 20% compression.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	Wiki-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
5×	BS [ 9 ] + GPTQ [ 19 ] (4-bit)	13.99	34.01	24.00	56.83	34.72	57.89	20.74	46.93	23.46	37.80
	BS [ 9 ] + AWQ [ 34 ] (4-bit)	11.38	30.95	24.40	59.91	36.09	64.09	21.94	55.35	26.79	41.22
	DictLLM FP4	8.48	17.04	29.20	64.64	43.67	70.18	23.99	64.02	32.76	46.92

Appendix

Table A12 : Effect of compounding quantization on Basis Sharing. ‘BS’ denotes Basis Sharing [ 9 ] . We measure language and reasoning performances on LLaMA-7B. All results are reproduced by us.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	Wiki-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
1.43×	BS [ 9 ] + magnitude	8.53	17.28	29.60	65.35	43.67	70.40	23.28	65.32	35.07	47.53
1.43×	DictLLM	7.79	15.20	29.00	66.38	45.67	71.27	24.25	65.53	35.07	48.17
5×	BS [ 9 ] + magnitude + AWQ [ 34 ] (4-bit)	10.39	23.50	23.80	62.98	38.64	66.05	22.28	57.32	25.51	42.37
5×	DictLLM FP4	8.48	17.04	29.20	64.64	43.67	70.18	23.99	64.02	32.76	46.92

Appendix

Table A13 : Effect of compounding pruning and quantization on Basis Sharing. ‘BS’ denotes Basis Sharing [ 9 ] . Magnitude pruning acts on the coefficient matrix. We measure language and reasoning performance on LLaMA-7B. All results are reproduced by us.

Configuration	Weight memory [GB]	Peak memory [GB]	Speed [tokens/s]
Original	13.48	13.66	102.35
Basis Sharing [ 9 ] (dense $\mathbf{D}$ , dense $\mathbf{C}$ )	9.59	9.83	113.59
DictLLM (dense $\mathbf{D}$ , sparse $\mathbf{C}$ )	7.82	8.02	126.81

Appendix

Table A14 : End-to-end decoding on LLaMA-7B. DictLLM uses MACKO for sparse coefficients. The measurement excludes the prompt prefill stage.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 128 papers

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
Sep 26, 2025Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip +2LLM Compression Sparse Coding
Fundamental Research Center MWS AI · ITMO
SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Mar 12, 2026Xinhao Huang, You-Liang Huang, Zeyi WenLLM Compression Low-Rank Compression
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
SPQ: An Ensemble Technique for Large Language Model Compression
Feb 20, 2026Jiamin Yao, Eren GultepeLLM Compression LLM Quantization
Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

Sep 26, 2025J/K move · Enter open · S save

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Ammar Ali+1

Fundamental Research Center MWS AI · ITMO

LLM Compression Sparse Coding

Open paper AskPDF arXiv

Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across LLaMA and Qwen models, CoSpaDi improves accuracy-compression and perplexity-compression trade-offs over SVD-based and structured pruning baselines at 20-40% compression ratios, while naturally supporting sparse-dense computation and post-training quantization of sparse coefficients.

Abstract

Model Compression

Explore similar work

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

SPQ: An Ensemble Technique for Large Language Model Compression