MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models

Maximilian Kleinegger; Elvir Crnčević; Dan Alistarh

cs.LGFeb 3, 2026

MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models

Authors: Maximilian Kleinegger, Elvir Crnčević, Dan Alistarh

Organizations: 1Vienna University of Technology, Vienna, Austria · Institute of Science & Technology Austria (ISTA), Vienna, Austria · 3Red Hat AI, Boston, USA

Abstract

Matryoshka Quantization (MatQuant), Any-Precision-LLM (AP) and AnyBCQ (AB) are recent quantization approaches showing that a single integer-quantized model can be served across multiple precisions. In this paradigm, lower-precision models are extracted from a higher-precision model by simply reading fewer bits of the weights. This enables a single checkpoint to cover a wide range of memory and latency budgets, but makes both quantization and efficient execution substantially harder. Existing methods rely on expensive quantization-aware training (QAT) or gradient-based post-training quantization (PTQ) rather than fast one-shot PTQ, and offer limited system support: dedicated kernels are either missing or restricted to single- or small-batch decoding. We address these limitations with Post-Training Matryoshka Quantization (MatGPTQ), an end-to-end pipeline for nested-model quantization and inference. MatGPTQ casts Matryoshka quantization as multi-precision error compensation, producing a single "sliceable" parent model jointly optimized for multiple target precisions in one pass over a small calibration set. We further refine the MatQuant representation so that an $r$ -bit model reads exactly $r$ bits, and introduce the first dedicated inference kernels for this format, supporting batch sizes beyond one and integrated into vLLM. Across standard LLMs and benchmarks, MatGPTQ outperforms MatQuant while remaining competitive with AP and AB at the smallest checkpoint size, and our kernels achieve end-to-end speedups of up to 3.5 $\times$ over BF16 at the low-bit regime. Overall, MatGPTQ makes nested quantized models practical to serve from a single, compact checkpoint. Code is available at https://github.com/IST-DASLab/MatGPTQ.

Figures & tables

Property	MatQuant	AP	AB	MatGPTQ
Uses gradients during quantization	✓	$\sim$	✓	$\times$
One-shot multi-precision PTQ	$\times$	$\times$	$\times$	✓
Single multi-precision checkpoint	✓	✓	✓	✓
Attention+FFN quantization	$\sim$	✓	✓	✓
Exact $r$ -bit loading	$\times$	✓	✓	✓
Inference kernel	$\times$	$\sim$	$\sim$	✓

Table 1: Comparison of multi-precision quantization methods.

		Llama3.1-8B-Instruct					Qwen3-8B
Bits	Method	CSR	GSM8K	MMLU	IFEval	Avg	CSR	GSM8K	MMLU	IFEval	Avg
16	Baseline	73.77	85.75	69.32	79.01	76.96	71.52	87.57	74.88	83.76	79.43
8	GPTQ	73.80	84.84	69.32	78.53	76.62	71.58	87.11	74.90	83.95	79.39
	AP	73.81	85.65	69.33	78.16	76.74	71.57	87.43	74.90	84.48	79.60
	AB	73.60	84.10	69.11	78.07	76.22	71.74	87.49	74.71	84.48	79.60
	MatQuant	73.08	84.23	69.03	78.68	76.26	71.34	87.49	75.10	84.76	79.68

Table 2: Unified accuracy comparison of Llama3.1-8B-Instruct and Qwen3-8B across bitwidths, for MatGPTQ against dedicated GPTQ, MatQuant, Any-Precision-LLM (AP) and AnyBCQ (AB). CSR averages ARC-c, ARC-e, HellaSwag, PIQA and WinoGrande; IFEval averages its prompt- and instance-level strict accuracies. The mean over three seeds is reported. For each format, top methods within variance are marked in bold , and methods within variance of the full-precision model (full recovery) are underlined .

Figure 1: The figure shows the kernel speedup over FP16 torch.matmul (cuBLAS). The kernels are evaluated at 2–8 bit precision for several representative matrix shapes across different batch sizes. AnyBCQ and Marlin W4A16/W8A16 are included as reference baselines. However, AnyBCQ results are only shown for batch size 1.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	Per-column function (ms)		Complete pass (s)
Shape	quant	quant_MatGPTQ	GPTQ	GPTQ $\times$ 3	MatGPTQ	Speedup
k/v (1024 $\times$ 4096)	0.0599	0.5360	1.45	4.34	3.59	1.21 $\times$
q/o (4096 $\times$ 4096)	0.0601	0.5368	1.47	4.41	3.59	1.23 $\times$
gate/up (14336 $\times$ 4096)	0.0610	1.3444	1.41	4.24	6.58	0.64 $\times$
down (4096 $\times$ 14336)	0.0601	0.5368	5.41	16.24	12.45	1.30 $\times$
Layer total	–	–	14.07	42.21	39.96	1.06 $\times$

Appendix

Table 3: Runtime comparison of per-column functions and full quantization passes. GPTQ

\times

3 is the sum of separate 3-, 4-, and 8-bit GPTQ passes. Speedup is (GPTQ

\times

3)/MatGPTQ.

Llama3.1-8B-Instruct
Bitwidth	Method	ARC-c	ARC-e	HellaSwag	PIQA	WinoG	GSM8K	MMLU	IF-p	IF-i
16		55.20	79.55	79.22	81.01	73.88	85.75	69.32	75.42	82.61
8	GPTQ	$54.95_{0.19}$	$79.80_{0.13}$	$79.21_{0.03}$	$80.85_{0.05}$	$74.19_{0.28}$	$84.84_{0.50}$	$69.32_{0.05}$	$75.05_{0.71}$	$82.01_{0.61}$
	AP	$55.15_{0.27}$	$79.67_{0.11}$	$79.22_{0.04}$	$80.96_{0.14}$	$74.06_{0.05}$	$85.65_{0.42}$	$69.33_{0.06}$	$74.43_{0.59}$	$81.89_{0.32}$
	AB	$55.83_{0.74}$	$79.66_{0.53}$	$78.39_{0.15}$	$81.16_{0.03}$	$72.98_{0.12}$	$84.10_{0.66}$	$69.11_{0.15}$	$74.49_{0.67}$	$81.65_{0.62}$
	MatQuant	$53.90_{0.13}$	$78.63_{0.11}$	$78.93_{0.02}$	$81.08_{0.03}$	$72.88_{0.24}$	$84.23_{0.08}$	$69.03_{0.04}$	$75.11_{0.11}$	$82.25_{0.24}$

Appendix

Table 4: Evaluation results for asymmetric MatGPTQ, GPTQ, MatQuant, Any-Precision-LLM (AP) and AnyBCQ (AB) on Llama3.1-8B-Instruct. Accuracy on the evaluation tasks (

\uparrow

): ARC-c (ARC-Challenge), ARC-e (ARC-Easy), HellaSwag, PIQA, WinoG (WinoGrande), GSM8K, MMLU, and IF-p and IF-i, the prompt- and instance-level strict IFEval accuracies. The mean over three calibration seeds is reported.

Qwen3-8B
Bitwidth	Method	ARC-c	ARC-e	HellaSwag	PIQA	WinoG	GSM8K	MMLU	IF-p	IF-i
16		56.48	80.64	74.93	77.69	67.88	87.57	74.88	80.59	86.93
8	GPTQ	$56.66_{0.27}$	$80.81_{0.07}$	$74.98_{0.04}$	$77.58_{0.06}$	$67.88_{0.10}$	$87.11_{0.16}$	$74.90_{0.04}$	$80.96_{0.59}$	$86.93_{0.38}$
	AP	$56.52_{0.11}$	$80.92_{0.09}$	$74.93_{0.03}$	$77.70_{0.05}$	$67.80_{0.37}$	$87.43_{0.28}$	$74.90_{0.05}$	$81.55_{0.36}$	$87.41_{0.35}$
	AB	$56.40_{0.78}$	$80.88_{0.57}$	$74.72_{0.24}$	$77.51_{0.11}$	$69.17_{0.41}$	$87.49_{1.14}$	$74.71_{0.25}$	$81.52_{1.03}$	$87.45_{0.50}$
	MatQuant	$56.26_{0.29}$	$80.56_{0.07}$	$74.75_{0.08}$	$77.27_{0.12}$	$67.86_{0.31}$	$87.49_{0.33}$	$75.10_{0.07}$	$82.07_{0.23}$	$87.46_{0.14}$

Appendix

Table 5: Evaluation results for asymmetric MatGPTQ, GPTQ, MatQuant, Any-Precision-LLM (AP) and AnyBCQ (AB) on Qwen3-8B. Accuracy on the evaluation tasks (

\uparrow

Bits	$\lambda$	ARC-c	ARC-e	HellaS	PIQA	WinoG	GSM8K	MMLU	IF-p	IF-i	All
8	(8, 1, 1)	54.27	79.17	77.31	79.71	73.01	79.30	67.28	69.50	78.54	73.32
	(4, 2, 1)	55.12	79.04	77.90	80.41	72.61	82.34	68.08	72.27	80.58	74.96
	(4, 4, 1)	54.27	80.18	78.19	80.69	73.32	83.85	68.28	74.12	80.82	75.73
	(1, 1, 1)	54.81	80.32	78.68	80.56	73.61	83.27	68.52	73.63	81.02	75.68
	(1, 2, 4)	54.93	79.83	78.87	81.02	74.11	84.69	69.07	74.23	81.73	76.37
	(1, 4, 4)	55.38	79.67	78.89	80.69	74.35	84.08	68.81	73.20	80.46	75.88

Appendix

Table 6: Evaluation results for MatGPTQ on asymmetric Llama3.1-8B-Instruct for different

\lambda

configurations, one triple applied to both the feedback and the scale-search channel. Weightings:

(x,y,z)\rightarrow(\lambda_{3},\lambda_{4},\lambda_{8})

. IF-p and IF-i are the prompt- and instance-level strict IFEval accuracies; All weights four groups equally, the five common-sense tasks, MMLU, GSM8K and IFEval (

\uparrow

). The mean over three calibration seeds is reported.

Bits	$\lambda$	ARC-c	ARC-e	HellaS	PIQA	WinoG	GSM8K	MMLU	IF-p	IF-i	All
8	(8, 1, 1)	56.83	79.12	72.81	77.26	68.51	86.66	72.94	80.41	86.21	78.45
	(4, 2, 1)	54.44	78.70	73.45	77.48	69.93	86.96	73.65	78.19	84.77	78.22
	(4, 4, 1)	56.46	80.06	73.66	77.71	69.17	86.53	74.14	80.47	86.41	78.88
	(1, 1, 1)	56.57	79.86	74.11	77.76	68.82	86.75	74.58	80.55	86.57	79.08
	(1, 2, 4)	56.03	80.13	74.68	77.53	68.14	87.36	74.73	81.76	87.53	79.51
	(1, 4, 4)	55.80	79.84	74.58	77.58	68.59	86.96	74.54	82.26	87.77	79.45

Appendix

Table 7: Evaluation results for MatGPTQ on asymmetric Qwen3-8B for different

\lambda

configurations, one triple applied to both the feedback and the scale-search channel. Weightings:

(x,y,z)\rightarrow(\lambda_{3},\lambda_{4},\lambda_{8})

. IF-p and IF-i are the prompt- and instance-level strict IFEval accuracies; All weights four groups equally, the five common-sense tasks, MMLU, GSM8K and IFEval (

\uparrow

). The mean over three calibration seeds is reported.

Figure 5: The figure shows the kernel latency compared to FP16 torch.matmul (cuBLAS). The kernels are evaluated at 2–8 bit precision for several representative matrix shapes across different batch sizes. AnyBCQ and Marlin W4A16/W8A16 are included as reference baselines. However, AnyBCQ results are only shown for batch size 1.

Figure 8: End-to-end throughput: Transformers – Llama3.1-8B-Instruct, Prompt/Decode 256/1024.

Explore similar work

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
May 13, 2026Lingchao Zheng, Yuwei Fan, Jun Li +5Quantizer Matched Fp16 Intermediate
Huawei
KronQ: LLM Quantization via Kronecker-Factored Hessian
Jul 8, 2026Donghyun Lee, Yuhang Li, Ruokai Yin +1Large Language Model Quantization Post-Training Quantization
University of Southern California · Yale University
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Large Language Model Quantization Nvidia
Intel

May 13, 2026, stat.MLJ/K move · Enter open · S save

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference

Lingchao Zheng, Yuwei Fan, Jun Li, Chengqiu Hu+4

Huawei

Quantizer Matched Fp16 Intermediate Matrix Multiplication LLM Inference Optimization

Open paper AskPDF arXiv

Quantization is essential for efficient large language model (LLM) inference, yet the dequantization step-converting low-bit weights back to high-precision for matrix multiplication has become a critical bottleneck on modern AI accelerators. On architectures with decoupled compute units (e.g., Ascend NPUs), dequantization operations can consume more cycles than the matrix multiplication itself, leaving the high-throughput tensor cores underutilized. This paper presents Multi-Scale Dequant (MSD), a quantization framework that removes weight/KV dequantization from the GEMM critical path. Instead of lifting low-bit weights to BF16 precision, MSD decomposes high-precision BF16 activations into multiple low-precision components, each of which can be multiplied directly with quantized weights via native hardware-accelerated GEMM. This approach shifts the computational paradigm from precision conversion to multi-scale approximation, avoiding INT8-to-BF16 weight conversion before GEMM. We instantiate MSD for two weight formats and derive tight error bounds for each. For INT8 weights (W4A16), two-pass INT8 decomposition achieves near 16 effective bits. For MXFP4 weights (W4A16), two-pass MXFP4 decomposition yields near 6.6 effective bits with error bound 1/64 per block surpassing single-pass MXFP8(5.24 bits) while maintaining the same effective GEMM compute time. We further derive closed-form latency and HBM traffic models showing that MSD avoids the Vector-Cube pipeline stall caused by dequantization and reduces KV cache HBM traffic by up to 2.5 times in attention. Numerical simulations on matrix multiplication and Flash Attention kernels confirm that MSD does not degrade accuracy compared to dequantization baselines, and in many settings achieves lower L2 error.

Abstract

Explore similar work

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference

KronQ: LLM Quantization via Kronecker-Factored Hessian

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs