G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Ruikang Liu; Haoli Bai; Yuxuan Sun; Qian Zhang; Wenzheng Cai; Yanqi Hao; Feiyu Wang; Weidong Zhong

cs.CLSep 25, 2026

G $^2$ PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Authors: Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai, Yanqi Hao, Feiyu Wang, Weidong Zhong, +3 more

Organizations: ZTE Corporation · The Chinese University of Hong Kong · Northwestern Polytechnical University · Peking University · Nanjing University of Aeronautics and Astronautics

Abstract

Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G $^2$ PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G $^2$ PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model families and bit-widths demonstrate that G $^2$ PTQ enables better alignment with the full-precision model, outperforming state-of-the-art baselines. Code is available at: https://github.com/G2PTQ/G2PTQ.

Figures & tables

Algorithm 1: G 2 PTQ quantization for one linear layer

Input: Weight matrix

\mathbf{W}

, gradient

\mathbf{G}

, per-group Hessians

\{\bar{\mathbf{H}}^{(i)}\}_{i=0}^{g-1}

(Equation 9 ), row-wise trust-region scaling

\bm{\beta}\in\mathbb{R}^{d_{\text{row}}}

(Theorem 3 ), block size

B

, and #channel groups

g

Output: quantized weight

\hat{\mathbf{W}}

0 1:

\hat{\mathbf{W}},\mathbf{E}\leftarrow\mathbf{0}_{d_{\text{row}}\times d_{\text{col}}},\mathbf{0}_{d_{\text{row}}\times d_{\text{col}}}

0 2:

\mathbf{D}\leftarrow\text{diag}(B-1,B-2,\ldots,0)

0 3: for each channel group

i=0,\ldots,g-1

do 0 4:

\mathcal{C}_{i}\leftarrow i\cdot(d_{\text{row}}/g):(i+1)\cdot(d_{\text{row}}/g)

0 5:

\mathbf{W}^{(i)},\hat{\mathbf{W}}^{(i)},\mathbf{G}^{(i)},\mathbf{E}^{(i)},\bm{\beta}^{(i)}\leftarrow\mathbf{W}_{\mathcal{C}_{i},:},\hat{\mathbf{W}}_{\mathcal{C}_{i},:},\mathbf{G}_{\mathcal{C}_{i},:},\mathbf{E}_{\mathcal{C}_{i},:},\bm{\beta}_{\mathcal{C}_{i}}

0 6:

\mathbf{L}\leftarrow\text{Cholesky}\big((\bar{\mathbf{H}}^{(i)})^{-1}\big)

0 7:

\mathbf{M}\leftarrow\left({\bm{\beta}^{(i)}}\right)^{\top}\circ(\mathbf{G}^{(i)}\mathbf{L})

0 8:

\mathbf{F}\leftarrow\mathbf{M}\mathbf{L}^{\top}

\triangleright

(Equation 17 ) 0 9: for

b=0,B,2B,\ldots

do 10: for

t=b,b+1,\ldots,b+B-1

do 11:

\hat{\mathbf{W}}^{(i)}_{:,t}\leftarrow\text{Quantize}(\mathbf{W}^{(i)}_{:,t})

12:

\mathbf{E}_{:,t}^{(i)}\leftarrow({\mathbf{W}^{(i)}_{:,t}-\mathbf{\hat{W}}^{(i)}_{:,t}{\color[rgb]{0,0.625,0.8789}-\mathbf{F}_{:,t}}})/{\mathbf{L}^{\top}_{t,t}}

13:

\mathbf{W}_{:,t:(b+B)}^{(i)}\leftarrow\mathbf{W}^{(i)}_{:,t:(b+B)}-\mathbf{E}_{:,t}^{(i)}\mathbf{L}^{\top}_{t,t:(b+B)}{\color[rgb]{0,0.625,0.8789}-\mathbf{F}_{:,t:(b+B)}}

\triangleright

(Equation 16 ) 14:

\mathbf{F}_{:,t:(b+B)}\leftarrow\mathbf{F}_{:,t:(b+B)}-\mathbf{M}_{:,t}\,\mathbf{L}^{\top}_{t,t:(b+B)}

\triangleright

(Equation 17 ) 15: end for 16:

\mathbf{W}_{:,(b+B):}^{(i)}\leftarrow\mathbf{W}_{:,(b+B):}^{(i)}-\mathbf{E}^{(i)}_{:,b:(b+B)}\,\mathbf{L}^{\top}_{b:(b+B),(b+B):}{\color[rgb]{0,0.625,0.8789}-B\cdot\mathbf{F}_{:,(b+B):}}

00:

{\color[rgb]{0,0.625,0.8789}+\mathbf{M}_{:,b:(b+B)}\,\mathbf{D}\,\mathbf{L}^{\top}_{b:(b+B),(b+B):}}

17:

{\color[rgb]{0,0.625,0.8789}\mathbf{F}_{:,(b+B):}\leftarrow\mathbf{F}_{:,(b+B):}-\mathbf{M}_{:,b:(b+B)}\,\mathbf{L}^{\top}_{b:(b+B),(b+B):}}

\triangleright

(Equation 19 ) 18: end for 19: end for

Algorithm 1:

Model	Method	W2A16			W3A16			W4A16
Model	Method	KL	PPL	QA	KL	PPL	QA	KL	PPL	QA
Qwen3 0.6B	BF16	0	11.28	49.28	0	11.28	49.28	0	11.28	49.28
	RTN	1.43e+01	7.56e+06	28.95	7.44e+00	1.62e+04	29.68	5.84e-01	17.61	41.36
	GPTQ	4.73e+00	1.58e+03	29.92	1.50e+00	55.60	33.09	4.58e-01	18.33	41.57
	GuidedQ	2.46e+00	143.67	30.05	4.96e-01	17.76	38.28	1.42e-01	12.88	45.41
	GPTAQ	3.33e+00	364.96	28.61	8.92e-01	30.40	34.58	3.03e-01	15.85	42.85

Table 1: Weight-only quantization results across different model families. KL, PPL, and QA metrics are reported as averages. Results for more models are provided in Appendix E.3 .

Table 3

Table 4

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm 2: G 2 PTQ quantization for the entire Transformer model

Input: Transformer blocks:

\text{block}^{(l)}

(

l=1,2,\ldots,L

), model input

\mathbf{Y}

, indicator

\mathcal{I}_{\text{autotune}}

Output: Quantized Transformer model 0 1:

\hat{\mathbf{Y}}\leftarrow\mathbf{Y}

0 2: for

l=1,2,\ldots,L

do 0 3: Load

\text{block}^{(l)}

to GPU 0 4:

\mathcal{S}\leftarrow

Set of linear layers to be quantized in

\text{block}^{(l)}

0 5:

\hat{\mathbf{Y}}^{\prime}\leftarrow\hat{\mathbf{Y}}

0 6:

\mathbf{Y}\leftarrow\text{block}^{(l)}(\mathbf{Y})

.detach() 0 7:

\hat{\mathbf{Y}}\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

0 8: if

l<L

then 0 9:

\ell_{H}\leftarrow\ell_{\text{MSE}}\big(\hat{\mathbf{Y}},(\hat{\mathbf{Y}}+\bm{\epsilon})\text{.detach()}\big)

with

\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},d\mathbf{I})

10:

\ell_{G}\leftarrow\ell_{\text{MSE}}(\hat{\mathbf{Y}},\mathbf{Y})

11: else 12:

\ell_{H}\leftarrow-\log p(y|\hat{\mathbf{Y}})

with

y\sim p(\cdot|\mathbf{Y})

13:

\ell_{G}\leftarrow D_{\text{KL}}\big(p(\cdot|\mathbf{Y})\,\|\,p(\cdot|\hat{\mathbf{Y}})\big)

14: end if 15: Backpropagate

\ell_{H}

to cache output activation gradients for each layer in

\mathcal{S}

16: Backpropagate

\ell_{G}

to cache weight gradients for each layer in

\mathcal{S}

17:

\_\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

, compute Hessian via Equation 8 for each layer in

\mathcal{S}

18: if

\mathcal{I}_{\text{autotune}}

then 19: Search for

\alpha^{(l)}

via Algorithm 3 and save it to disk 20: else 21: Load

\alpha^{(l)}

from disk 22: end if 23: Compute trust-region scaling factor

\bm{\beta}

via Equation 56 24: for each linear layer in

\mathcal{S}

do 25: Quantize the linear layer via Algorithm 1 26: end for 27:

\hat{\mathbf{Y}}\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

28: Offload

\text{block}^{(l)}

to CPU 29: end for

Appendix

Algorithm 2:

Algorithm 3: Trust-region threshold autotuning for one Transformer block

Input: Transformer block:

\text{block}^{(l)}

, search range

[\alpha_{\min},\alpha_{\max}]

, step size

\Delta\alpha

, quantized block input

\hat{\mathbf{Y}}^{\prime}

, full-precision block output

\mathbf{Y}

Output: Trust-region threshold

\alpha_{\text{best}}

0 1:

\ell_{\text{best}}\leftarrow\infty

\alpha_{\text{best}}\leftarrow 0

0 2:

\text{state\_dict}\leftarrow\text{block}^{(l)}.\text{state\_dict}()

0 3: for

\alpha\in\{\alpha_{\min},\alpha_{\min}+\Delta\alpha,\ldots,\alpha_{\max}\}

do 0 4:

\mathcal{S}\leftarrow

Set of linear layers to be quantized in

\text{block}^{(l)}

0 5: Compute trust-region scaling factor

\bm{\beta}

via Equation 56 0 6: for each linear layer in

\mathcal{S}

do 0 7: Quantize the linear layer via Algorithm 1 0 8: end for 0 9:

\hat{\mathbf{Y}}\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

10: if

l<L

then 11:

\ell_{G}\leftarrow\ell_{\text{MSE}}(\hat{\mathbf{Y}},\mathbf{Y})

12: else 13:

\ell_{G}\leftarrow D_{\text{KL}}\big(p(\cdot|\mathbf{Y})\,\|\,p(\cdot|\hat{\mathbf{Y}})\big)

14: end if 15:

\text{block}^{(l)}.\text{load\_state\_dict}(\text{state\_dict})

16: if

\ell_{G}<\ell_{\text{best}}

then 17:

\ell_{\text{best}}\leftarrow\ell_{G}

18:

\alpha_{\text{best}}\leftarrow\alpha

19: else 20: break 21: end if 22: end for

Appendix

Algorithm 3:

Model Family	Attn Arch	MLP Arch
LLaMA3 ( Grattafiori et al., 2024 )	GQA	Dense
Qwen3 ( Yang et al., 2025 )	GQA	Dense
Qwen3.5 ( Team, 2026 )	GQA / GDN	Dense
Qwen3-MoE ( Yang et al., 2025 )	GQA	MoE
Qwen3.8-Flash-Next ( Qiu et al., 2026 )	QSA / GDN	MoE

Appendix

Table 2: Comparison of Attention and MLP architectures across different model families.

Panel of Figure 5 — Figure 5: Ablation study on the calibration set size for G 2 PTQ (left) and GPTQ (right). Experiments are conducted using 3-bit weight quantization on the Qwen3-0.6B model.

$g$	KL	PPL	QA	Mem Cost	Time Cost
1	3.43e-01	16.66	40.77	1.00x	1.00x
2	3.39e-01	16.33	40.11	1.00x	1.03x
4	3.33e-01	16.12	39.83	1.01x	1.20x
8	3.31e-01	16.25	39.81	1.02x	1.47x
16	3.23e-01	15.77	40.53	1.04x	2.00x

Appendix

Table 3: Ablation study on the number of output channel groups

g

. We report the average accuracy metrics for 3-bit quantization of the Qwen3-0.6B model, alongside the relative calibration cost.

Method	WikiText2		UltraChat		NuminaMath		QA Avg.
Method	KL	PPL	KL	PPL	KL	PPL	QA Avg.
w/ Approx Grad	7.10e-01	35.29	5.69e-01	12.08	3.32e-01	5.83	37.74
w/ Exact Grad	4.97e-01	32.34	3.39e-01	10.52	1.63e-01	5.49	39.83

Appendix

Table 4: Ablation study on exact versus approximated gradient compensation. Experiments are conducted using 3-bit weight quantization on the Qwen3-0.6B model.

Model	Autotune	WikiText2		UltraChat		NuminaMath		QA Avg.
Model	Autotune	KL	PPL	KL	PPL	KL	PPL	QA Avg.
Qwen3-0.6B (W2A16)	✓	1.85e+00	125.39	1.35e+00	26.60	5.86e-01	7.76	32.93
Qwen3-0.6B (W2A16)	$\times$	1.91e+00	131.15	1.40e+00	28.14	6.02e-01	8.11	32.56
Qwen3-0.6B (W4A16)	✓	1.60e-01	23.98	1.01e-01	8.56	5.35e-02	5.03	45.80
Qwen3-0.6B (W4A16)	$\times$	1.59e-01	23.92	1.01e-01	8.56	5.33e-02	5.04	46.25
Qwen3-0.6B-Base (W3A16)	✓	4.79e-01	20.14	3.42e-01	7.45	1.70e-01	3.75	44.99

Appendix

Table 5: Comparison of setting-specific autotuning (✓) versus applying a “you only autotune once” threshold (

\times

) transferred from the 3-bit Qwen3-0.6B baseline.

Model	H-Scale	WikiText2		UltraChat		NuminaMath		QA Avg.
Model	H-Scale	KL	PPL	KL	PPL	KL	PPL	QA Avg.
Qwen3-0.6B	$\times$	4.97e-01	32.34	3.39e-01	10.52	1.63e-01	5.49	39.83
Qwen3-0.6B	✓	4.49e-01	32.15	2.91e-01	10.35	1.31e-01	5.48	41.95
Qwen3-1.7B	$\times$	4.73e-01	21.66	3.52e-01	11.76	1.51e-01	5.69	51.33
Qwen3-1.7B	✓	3.89e-01	20.28	2.94e-01	11.50	1.09e-01	5.72	53.14
Qwen3-4B	$\times$	4.10e-01	18.38	2.43e-01	7.88	1.07e-01	4.84	60.30

Appendix

Table 6: Effect of Hessian-based weight clipping (H-Scale) across different model families. Experiments are conducted under 3-bit weight quantization.

Model	Method	WikiText2		UltraChat		NuminaMath		CommonSense QA
Model	Method	KL	PPL	KL	PPL	KL	PPL	ARC-C	ARC-E	C-eval	HellaS	LAMB	PIQA	Wino	Avg.
Qwen3 8B	BF16	0	9.72	0	5.20	0	3.60	56.14	80.77	79.57	74.98	64.10	77.97	67.96	71.64
	RTN	4.16e-01	12.33	3.42e-01	5.58	2.41e-01	3.81	44.62	64.44	64.86	66.48	48.52	73.88	64.01	60.97
	GPTQ	2.49e-01	11.32	1.85e-01	5.56	1.24e-01	3.81	52.05	75.38	69.24	69.33	57.87	75.68	64.64	66.31
	GuidedQ	2.54e-01	11.44	1.87e-01	5.59	1.25e-01	3.83	49.83	75.29	67.16	69.21	58.33	74.81	63.54	65.45
	GPTAQ	2.56e-01	11.81	1.86e-01	5.82	1.23e-01	4.00	48.12	72.94	69.02	68.04	57.91	74.54	63.93	64.93

Appendix

Table 10: 4-bit weight-activation quantization results across different model families.

Model	Method	WikiText2		UltraChat		NuminaMath		CommonSense QA
Model	Method	KL	PPL	KL	PPL	KL	PPL	ARC-C	ARC-E	C-eval	HellaS	LAMB	PIQA	Wino	Avg.
Qwen3 30B-A3B (W2A16)	BF16	0	8.70	0	4.79	0	3.41	56.66	78.96	84.32	77.74	64.80	80.58	71.11	73.45
	GPTQ	9.83e-01	20.13	6.24e-01	6.84	4.42e-01	4.65	38.99	56.78	44.73	57.62	49.82	70.24	56.12	53.47
	G 2 PTQ	4.93e-01	13.89	2.86e-01	5.68	2.48e-01	4.18	45.48	67.26	59.73	66.61	53.79	76.06	63.93	61.84
	G 2 PTQ ∗	4.27e-01	12.73	2.13e-01	5.23	2.09e-01	4.04	44.45	66.08	63.08	68.35	55.31	75.90	65.11	62.61
Qwen3 30B-A3B (W3A16)	BF16	0	8.70	0	4.79	0	3.41	56.66	78.96	84.32	77.74	64.80	80.58	71.11	73.45

Appendix

Table 11: Weight-only and weight-activation quantization results on Qwen3-30B-A3B.

Model	Method	WikiText2		UltraChat		NuminaMath		CommonSense QA
Model	Method	KL	PPL	KL	PPL	KL	PPL	ARC-C	ARC-E	C-eval	HellaS	LAMB	PIQA	Wino	Avg.
Qwen3.8 Flash-Next	BF16	0	4.67	0	6.30	0	3.14	64.51	79.08	89.38	87.59	72.81	82.97	71.59	78.28
	GPTQ	1.78e-01	5.02	9.36e-02	6.76	7.61e-02	3.23	62.03	78.03	87.30	87.19	72.48	83.62	72.22	77.55
	G 2 PTQ ∗	1.52e-01	4.85	7.43e-02	5.77	6.20e-02	3.03	62.71	79.12	88.04	87.40	72.39	84.17	71.82	77.95

Appendix

Table 12: 4-bit weight-only quantization results on Qwen3.8-Flash-Next.

Model	Method	GPQA-Diamond	Live-Code-Bench	ArXiv-Math	IFBench	Avg.
Qwen3.8 Flash-Next	BF16	$91.58\pm 0.58$	$93.78\pm 0.29$	$44.01\pm 2.80$	$73.00\pm 0.88$	$75.59\pm 0.84$
	GPTQ	$90.57\pm 0.29$	$92.64\pm 0.05$	43.37 $\pm$ 0.56	$74.11\pm 0.51$	$75.17\pm 0.11$
	G 2 PTQ ∗	91.08 $\pm$ 1.27	92.92 $\pm$ 0.33	$43.36\pm 2.97$	74.22 $\pm$ 0.69	75.40 $\pm$ 0.40

Appendix

Table 13: 4-bit weight-only quantization results for the Qwen3.8-Flash-Next model, evaluated on reasoning benchmarks. We report the mean accuracy and variance across three random seeds.

Explore similar work

KronQ: LLM Quantization via Kronecker-Factored Hessian
Jul 8, 2026Donghyun Lee, Yuhang Li, Ruokai Yin +1Mixed-Precision Quantization LLM Quantization
University of Southern California · Yale University
ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization
Aug 7, 2026Yongge Ma, Guoan Wang, Feiyu Wang +5LLM Quantization Post-Training Quantization
School of Computer Science, Peking University · School of Software and Microelectronics, Peking University · School of Physics, Peking University +2
Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate
Apr 15, 2026Jaemin Kim, Sungkyun Kim, Junyeol Lee +1LLM Quantization Language Model Calibration
Seoul National University · Hanyang University

Jul 8, 2026, cs.LGJ/K move · Enter open · S save

KronQ: LLM Quantization via Kronecker-Factored Hessian

Donghyun Lee, Yuhang Li, Ruokai Yin, Priyadarshini Panda

University of Southern California · Yale University

Mixed-Precision Quantization LLM Quantization Rotation-Based Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.

Abstract

Explore similar work

KronQ: LLM Quantization via Kronecker-Factored Hessian

ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization

Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate