DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

Jinho Park; Se Young Chun; Mingoo Seok

cs.AINov 5, 2026

DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

Authors: Jinho Park, Se Young Chun, Mingoo Seok

Organizations: Columbia University · Seoul National University

Abstract

Efficient deployment of large language models (LLMs) on mobile devices requires balancing performance with compute and memory constraints. Post-training compression is an effective approach to managing this trade-off, while requiring much less computation than in-training counterparts. We propose DictLLM, a dictionary-based compression scheme in which weight matrices within grouped layers are decomposed into a shared dictionary and layer-specific sparse coefficients, directly obtained from a pre-trained model without further training. Specifically, we cast DictLLM as a four-step post-training optimization. First, we obtain shared dictionaries via singular value decomposition (SVD) of input-scaled weight matrices, amortizing rank across grouped layers; second, we sparsify the layer-specific coefficients to the target compression ratio using a dictionary-aware Hessian; and third, we refine the dictionaries given the resulting sparse coefficients. Finally, we aggressively quantize both components using factor-specific Hessian information: input covariance for the dictionaries and the dictionary-aware Hessian for the coefficients. The resulting representation reduces memory footprint by storing a single shared dictionary per layer group, along with sparse layer-specific coefficients. On language modeling and zero-shot reasoning benchmarks, DictLLM consistently outperforms existing post-training low-rank compression methods and pushes the Pareto frontier between model size and performance. DictLLM compresses the original LLaMA-7B model by 5.1x with only a 2.9-point increase in WikiText-2 perplexity, achieving a 3.6x higher compression ratio than the state-of-the-art low-rank method.

Figures & tables

Figure 1: DictLLM: Dictionary-based compression method. DictLLM resolves the weight memory bottleneck by compressing transformer weights into a shareable dictionary. It compounds four compression steps. ➊ The first step factorizes input-scaled weight matrices of grouped layers into a shared dictionary and layer-specific coefficients. ➋ We then prune the layer-specific coefficients using a dictionary-aware Hessian. ➌ We refit the shared dictionary to the resulting sparse coefficients. ➍ We quantize both the dictionary and the coefficients into NVFP4 using input statistics for the dictionary and dictionary-aware Hessians for the coefficients. The resulting compressed representation enables efficient projections by transferring a small set of shared dictionaries.

Figure 3

Comp.	Method	Perplexity $\downarrow$			Zero-shot accuracy $\uparrow$
gain	Method	WikiText-2	PTB	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	8.35	7.34	0.28	0.67	0.56	0.78	0.27	0.67	0.38	0.52
1.25×	ASVD [ 11 ]	11.14	16.55	15.93	0.25	0.64	0.41	0.68	0.24	0.53	0.27	0.43
	SVD-LLM [ 8 ]	7.94	18.05	15.93	0.22	0.63	0.43	0.69	0.24	0.58	0.29	0.44
	Basis Sharing [ 9 ]	7.74	17.35	15.03	0.28	0.66	0.46	0.71	0.25	0.66	0.36	0.48
	DictLLM (Ours)	6.89	12.27	11.75	0.33	0.67	0.50	0.74	0.26	0.69	0.36	0.51

Table 1: Benchmark on language modeling and reasoning tasks. We benchmark DictLLM against low-rank compression baselines on language modeling and commonsense reasoning tasks for LLaMA-7B at different compression ratios. We additionally report the FP4-quantized DictLLM FP4 models at the bottom of the table. We measure perplexity on WikiText-2, Penn TreeBank (PTB), and C4, and zero-shot accuracy on OpenBookQA (OB), WinoGrande (WG), HellaSwag (HS), PIQA (PQ), MathQA (MQ), ARC-Easy (AE), and ARC-Challenge (AC).

Figure 4: Perplexity-model size trade-off. WikiText-2 perplexity of compressed LLaMA-7B across factorization ratios and coefficient sparsities. DictLLM factorizes weights at rate

\rho_{\mathrm{fac}}

and then prunes layer-specific coefficients with 20%–50% sparsity. We also report their NVFP4 versions.

Table 6

Figure 5: Ablation of dictionary update. WikiText-2 perplexity of LLaMA-7B across different sparsity levels, with and without dictionary update.

Method	Dictionary $\mathbf{D}$	Coefficient $\mathbf{C}$	PPL $\downarrow$	Comp. gain $\uparrow$
Unquantized baseline	FP16	FP16	12.98	1.00 $\times$
Mixed precision 1	NVFP4	FP16	13.66	1.61 $\times$
Mixed precision 2	NVFP4	FP8	14.71	2.60 $\times$
Mixed precision 3	NVFP4	INT8	16.05	2.60 $\times$
All NVFP4 (Ours)	NVFP4	NVFP4	16.82	3.56 $\times$

Table 4: Different quantization format. We compare quantization choices for the shared dictionary

\mathbf{D}

and layer-specific coefficients

\mathbf{C}

. Perplexity is measured on WikiText-2 using LLaMA-7B, and compression gain denotes the reduction in model size. The FP16/FP16 baseline corresponds to the factorized model with a 30% factorization reduction and 50% coefficient sparsity before quantization.

Table 9

Figure 6: Effect of calibration size. WikiText-2 perplexity of LLaMA-7B as a function of calibration set size.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	7.09	11.87	32.20	72.53	59.03	79.16	55.41	81.61	53.24	61.88
1.25×	SVD-LLM [ 8 ]	11.52	32.16	29.00	66.22	44.16	71.55	31.56	67.51	38.31	49.76
	Basis Sharing [ 9 ]	11.14	29.37	31.40	67.56	45.77	71.93	32.26	67.21	38.99	50.73
	DictLLM (Ours)	9.90	22.59	31.00	70.09	50.07	74.86	35.78	70.54	41.89	53.46
1.43×	SVD-LLM [ 8 ]	14.30	51.96	26.20	64.33	37.91	65.02	26.77	55.51	28.50	43.46

Appendix

Table A3 : Benchmark on language modeling and reasoning tasks with Qwen3-8B. We compare DictLLM against low-rank compression baselines at compression gains from 1.25× to 2×. We report perplexity on WikiText-2 and C4, and zero-shot accuracy on OpenBookQA (OB), WinoGrande (WG), HellaSwag (HS), PIQA (PQ), MathQA (MQ), ARC-Easy (AE), and ARC-Challenge (AC).

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	9.51	15.34	29.80	63.54	49.29	75.73	45.19	73.74	41.04	54.05
1.25×	SVD-LLM [ 8 ]	23.20	96.37	22.80	56.83	34.98	61.64	24.15	44.15	24.66	38.46
	Basis Sharing [ 9 ]	22.93	95.07	22.60	56.20	36.66	62.02	26.16	48.82	27.13	39.94
	DictLLM (Ours)	17.66	52.80	25.80	59.91	40.63	67.85	27.84	53.37	28.75	43.45
1.43×	SVD-LLM [ 8 ]	34.60	190.49	19.00	55.01	31.08	58.32	23.75	38.13	22.95	35.46

Appendix

Table A4 : Qwen3-1.7B language modeling and reasoning results. Methods are compared at the reported compression gains.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	6.46	10.95	35.40	74.19	61.75	80.41	61.78	83.33	55.63	64.64
1.25×	SVD-LLM [ 8 ]	9.49	27.15	28.80	68.98	47.19	72.69	34.34	67.00	38.48	51.07
	Basis Sharing [ 9 ]	9.18	25.93	30.20	70.24	49.32	74.32	36.98	66.08	39.33	52.35
	DictLLM (Ours)	8.29	19.69	32.20	70.32	54.48	76.22	39.90	73.02	43.60	55.68
1.43×	SVD-LLM [ 8 ]	11.51	44.54	26.00	65.11	39.29	67.08	29.95	62.25	31.57	45.89

Appendix

Table A5 : Qwen3-14B language modeling and reasoning results. The improvements over other baselines extend to the largest model evaluated in language modeling and reasoning tasks.

Compression gain	Factorization ratio $\rho_{\text{fac}}$	Sparsity $p$
Compression gain	Factorization ratio $\rho_{\text{fac}}$	Qwen3-8B	Qwen3-1.7B	Qwen3-14B
1.25×	10%	17.25%	17.57%	17.52%
1.43×	20%	19.40%	19.79%	19.71%
1.67×	30%	22.17%	-	-
2×	30%	44.38%	-	-

Appendix

Table A6 : Factorization and coefficient sparsity settings for Qwen3 model family. We report the factorization ratio

\rho_{\text{fac}}

and Sparsity

p

required to achieve a given compression gain for Qwen3-8B, 1.7B, and 14B models.

Stage	Comp.	Components				Perplexity $\downarrow$		Zero-shot accuracy $\uparrow$
Stage	gain	F	P	R	Q	Wiki-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Avg.
Original	1×	–	–	–	–	5.68	7.34	0.28	0.67	0.56	0.78	0.27	0.67	0.38	0.52
+ Factorization	1.43×	$\checkmark$	–	–	–	9.25	22.46	0.27	0.63	0.40	0.68	0.24	0.63	0.30	0.45
+ Hessian pruning	2×	$\checkmark$	$\checkmark$	–	–	11.53	33.82	0.25	0.62	0.36	0.64	0.23	0.57	0.27	0.42
+ Dictionary refit	2×	$\checkmark$	$\checkmark$	$\checkmark$	–	10.59	29.12	0.24	0.63	0.38	0.65	0.23	0.61	0.29	0.43
+ Quantization	7×	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	11.92	33.90	0.24	0.59	0.35	0.63	0.22	0.54	0.26	0.40

Appendix

Table A7 : Componentwise ablation of compression stages. ‘F’, ‘P’, ‘R’, and ‘Q’ respectively denote factorization, Hessian pruning, dictionary refitting, and quantization. A checkmark indicates an active component. We measure language and reasoning performance on LLaMA-7B.

Figure A1 : Spectral concentration of LLaMA-7B query projection weights. We plot cumulative spectral energy for single-layer SVD (a) with and (b) without input covariance scaling and cross-layer SVD (c) with and (d) without scaling across layers

\ell

. Input-scaled spectra rise more rapidly, showing that the energy of input-scaled weights is more concentrated in low-rank components than that of the raw weight spectrum. Vertical dashed lines indicate the selected rank

k\in\{1433,1911\}

for the target factorization rate

\rho_{\text{fac}}=30\%

in single-layer and cross-layer settings.

Figure A2 : Spectral concentration across different weight matrices in LLaMA-7B. We plot cumulative spectral energy for the shared projections K, V, gate, and up using cross-layer SVD, and for the private projections O and down using single-layer SVD, following our factorization design. Vertical dashed lines indicate the selected rank

k

for the target factorization rate

\rho_{\text{fac}}=30\%

under the corresponding single-layer or cross-layer setting.

Figure A3 : Aggressive layer sharing with LoRA fine-tuning. We increase the dictionary-sharing group size to

n=4,8,16

and apply 60% coefficient sparsity while keeping the rank of the shared dictionary fixed, then use LoRA fine-tuning to restore performance.

Calibration size	Perplexity $\downarrow$
16	9.06
32	8.98
64	8.90
128	8.88

Appendix

Table A8 : Calibration size for NVFP4 quantization. Post-quantization WikiText-2 perplexity of LLaMA-7B. Using 16 calibration samples provides a strong trade-off between perplexity and runtime.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
5×	SparseGPT [ 14 ]	86.72	257.96	14.00	48.46	25.38	51.41	18.79	25.63	21.08	29.25
	Wanda [ 15 ]	2.8 × 10 3	2.8 × 10 3	11.60	48.30	25.78	53.10	21.21	26.35	20.31	29.52
	DictLLM FP4	8.48	17.04	29.20	64.64	43.67	70.18	23.99	64.02	32.76	46.92

Appendix

Table A9 : Comparison with pruning-only methods. SparseGPT and Wanda are evaluated on LLaMA-7B at 80.32% unstructured sparsity. All results are reproduced by us.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	WikiText-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
7×	GPTQ [ 19 ] (2-bit)	17.59	36.56	13.80	48.38	30.47	57.13	20.84	32.03	21.33	32.00
	AWQ [ 34 ] (2-bit)	2.6 × 10 5	2.9 × 10 5	13.60	49.33	25.40	52.83	21.57	25.38	22.35	30.07
	DictLLM FP4	11.92	33.90	23.60	59.19	35.32	63.17	22.21	53.62	25.60	40.39

Appendix

Table A10 : Comparison with quantization-only baselines. We compare methods at 7× compression gain on LLaMA-7B. GPTQ [ 19 ] and AWQ [ 34 ] use two-bit weights, whereas DictLLM combines factorization and coefficient sparsity with four-bit quantization. All results are reproduced by us.

Method	LLaMA-7B	OPT-6.7B
ASVD [ 11 ]	11.14	82.0
SVD-LLM [ 8 ]	7.94	16.04
SVD-LLM V2 [ 56 ]	7.12	13.46
Basis Sharing [ 9 ]	7.74	11.79
DictLLM (Ours)	6.89	11.36

Appendix

Table A11: Comparison with low rank methods including SVD-LLM V2. We measure WikiText-2 perplexity at 20% compression.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	Wiki-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
5×	BS [ 9 ] + GPTQ [ 19 ] (4-bit)	13.99	34.01	24.00	56.83	34.72	57.89	20.74	46.93	23.46	37.80
	BS [ 9 ] + AWQ [ 34 ] (4-bit)	11.38	30.95	24.40	59.91	36.09	64.09	21.94	55.35	26.79	41.22
	DictLLM FP4	8.48	17.04	29.20	64.64	43.67	70.18	23.99	64.02	32.76	46.92

Appendix

Table A12 : Effect of compounding quantization on Basis Sharing. ‘BS’ denotes Basis Sharing [ 9 ] . We measure language and reasoning performances on LLaMA-7B. All results are reproduced by us.

Comp.	Method	Perplexity $\downarrow$		Zero-shot accuracy (%) $\uparrow$
gain	Method	Wiki-2	C4	OB	WG	HS	PQ	MQ	AE	AC	Average
1×	Original	5.68	7.34	30.80	69.38	56.36	78.24	26.57	74.66	41.55	53.94
1.43×	BS [ 9 ] + magnitude	8.53	17.28	29.60	65.35	43.67	70.40	23.28	65.32	35.07	47.53
1.43×	DictLLM	7.79	15.20	29.00	66.38	45.67	71.27	24.25	65.53	35.07	48.17
5×	BS [ 9 ] + magnitude + AWQ [ 34 ] (4-bit)	10.39	23.50	23.80	62.98	38.64	66.05	22.28	57.32	25.51	42.37
5×	DictLLM FP4	8.48	17.04	29.20	64.64	43.67	70.18	23.99	64.02	32.76	46.92

Appendix

Table A13 : Effect of compounding pruning and quantization on Basis Sharing. ‘BS’ denotes Basis Sharing [ 9 ] . Magnitude pruning acts on the coefficient matrix. We measure language and reasoning performance on LLaMA-7B. All results are reproduced by us.

Configuration	Weight memory [GB]	Peak memory [GB]	Speed [tokens/s]
Original	13.48	13.66	102.35
Basis Sharing [ 9 ] (dense $\mathbf{D}$ , dense $\mathbf{C}$ )	9.59	9.83	113.59
DictLLM (dense $\mathbf{D}$ , sparse $\mathbf{C}$ )	7.82	8.02	126.81

Appendix

Table A14 : End-to-end decoding on LLaMA-7B. DictLLM uses MACKO for sparse coefficients. The measurement excludes the prompt prefill stage.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 128 papers

Sep 26, 2025

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across LLaMA and Qwen models, CoSpaDi improves accuracy-compression and perplexity-compression trade-offs over SVD-based and structured pruning baselines at 20-40% compression ratios, while naturally supporting sparse-dense computation and post-training quantization of sparse coefficients.

Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip +2

Fundamental Research Center MWS AI · ITMO

LLM Compression Sparse Coding

Ask PDF

Mar 12, 2026

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but the billion-scale parameters pose deployment challenges. Although existing methods attempt to reduce the scale of LLMs, they require either special hardware support or expensive post-training to maintain model quality. To facilitate efficient and affordable model slimming, we propose a novel training-free compression method for LLMs, named "SoLA", which leverages Soft activation sparsity and Low-rAnk decomposition. SoLA can identify and retain a minority of components significantly contributing to inference, while compressing the majority through low-rank decomposition, based on our analysis of the activation pattern in the feed-forward network (FFN) of modern LLMs. To alleviate the decomposition loss, SoLA is equipped with an adaptive component-wise low-rank allocation strategy to assign appropriate truncation positions for different weight matrices. We conduct extensive experiments on LLaMA-2-7B/13B/70B and Mistral-7B models across a variety of benchmarks. SoLA exhibits remarkable improvement in both language modeling and downstream task accuracy without post-training. For example, with a 30% compression rate on the LLaMA-2-70B model, SoLA surpasses the state-of-the-art method by reducing perplexity from 6.95 to 4.44 and enhancing downstream task accuracy by 10%.

Xinhao Huang, You-Liang Huang, Zeyi Wen

The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

LLM Compression Low-Rank Compression Activation Sparsity

Ask PDF

Feb 20, 2026

SPQ: An Ensemble Technique for Large Language Model Compression

This study presents an ensemble technique, SPQ (SVD-Pruning-Quantization), for large language model (LLM) compression that combines variance-retained singular value decomposition (SVD), activation-based pruning, and post-training linear quantization. Each component targets a different source of inefficiency: i) pruning removes redundant neurons in MLP layers, ii) SVD reduces attention projections into compact low-rank factors, iii) and 8-bit quantization uniformly compresses all linear layers. At matched compression ratios, SPQ outperforms individual methods (SVD-only, pruning-only, or quantization-only) in perplexity, demonstrating the benefit of combining complementary techniques. Applied to LLaMA-2-7B, SPQ achieves up to 75% memory reduction while maintaining or improving perplexity (e.g., WikiText-2 5.47 to 4.91) and preserving accuracy on downstream benchmarks such as C4, TruthfulQA, and GSM8K. Compared to strong baselines like GPTQ and SparseGPT, SPQ offers competitive perplexity and accuracy while using less memory (6.86 GB vs. 7.16 GB for GPTQ). Moreover, SPQ improves inference throughput over GPTQ, achieving up to a 1.9x speedup, which further enhances its practicality for real-world deployment. The effectiveness of SPQ's robust compression through layer-aware and complementary compression techniques may provide practical deployment of LLMs in memory-constrained environments. Code is available at: https://github.com/JiaminYao/SPQ_LLM_Compression/

Jiamin Yao, Eren Gultepe

Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

LLM Compression LLM Quantization LLM Pruning LLM Inference Acceleration Post-Training Quantization

Ask PDF

Abstract

Model Compression

Explore similar work

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

SPQ: An Ensemble Technique for Large Language Model Compression