G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Ruikang Liu; Haoli Bai; Yuxuan Sun; Qian Zhang; Wenzheng Cai; Yanqi Hao; Feiyu Wang; Weidong Zhong

cs.CLSep 25, 2026

G $^2$ PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Authors: Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai, Yanqi Hao, Feiyu Wang, Weidong Zhong, +3 more

Organizations: ZTE Corporation · The Chinese University of Hong Kong · Northwestern Polytechnical University · Peking University · Nanjing University of Aeronautics and Astronautics

Abstract

Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G $^2$ PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G $^2$ PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model families and bit-widths demonstrate that G $^2$ PTQ enables better alignment with the full-precision model, outperforming state-of-the-art baselines. Code is available at: https://github.com/G2PTQ/G2PTQ.

Figures & tables

Algorithm 1: G 2 PTQ quantization for one linear layer

Input: Weight matrix

\mathbf{W}

, gradient

\mathbf{G}

, per-group Hessians

\{\bar{\mathbf{H}}^{(i)}\}_{i=0}^{g-1}

(Equation 9 ), row-wise trust-region scaling

\bm{\beta}\in\mathbb{R}^{d_{\text{row}}}

(Theorem 3 ), block size

B

, and #channel groups

g

Output: quantized weight

\hat{\mathbf{W}}

0 1:

\hat{\mathbf{W}},\mathbf{E}\leftarrow\mathbf{0}_{d_{\text{row}}\times d_{\text{col}}},\mathbf{0}_{d_{\text{row}}\times d_{\text{col}}}

0 2:

\mathbf{D}\leftarrow\text{diag}(B-1,B-2,\ldots,0)

0 3: for each channel group

i=0,\ldots,g-1

do 0 4:

\mathcal{C}_{i}\leftarrow i\cdot(d_{\text{row}}/g):(i+1)\cdot(d_{\text{row}}/g)

0 5:

\mathbf{W}^{(i)},\hat{\mathbf{W}}^{(i)},\mathbf{G}^{(i)},\mathbf{E}^{(i)},\bm{\beta}^{(i)}\leftarrow\mathbf{W}_{\mathcal{C}_{i},:},\hat{\mathbf{W}}_{\mathcal{C}_{i},:},\mathbf{G}_{\mathcal{C}_{i},:},\mathbf{E}_{\mathcal{C}_{i},:},\bm{\beta}_{\mathcal{C}_{i}}

0 6:

\mathbf{L}\leftarrow\text{Cholesky}\big((\bar{\mathbf{H}}^{(i)})^{-1}\big)

0 7:

\mathbf{M}\leftarrow\left({\bm{\beta}^{(i)}}\right)^{\top}\circ(\mathbf{G}^{(i)}\mathbf{L})

0 8:

\mathbf{F}\leftarrow\mathbf{M}\mathbf{L}^{\top}

\triangleright

(Equation 17 ) 0 9: for

b=0,B,2B,\ldots

do 10: for

t=b,b+1,\ldots,b+B-1

do 11:

\hat{\mathbf{W}}^{(i)}_{:,t}\leftarrow\text{Quantize}(\mathbf{W}^{(i)}_{:,t})

12:

\mathbf{E}_{:,t}^{(i)}\leftarrow({\mathbf{W}^{(i)}_{:,t}-\mathbf{\hat{W}}^{(i)}_{:,t}{\color[rgb]{0,0.625,0.8789}-\mathbf{F}_{:,t}}})/{\mathbf{L}^{\top}_{t,t}}

13:

\mathbf{W}_{:,t:(b+B)}^{(i)}\leftarrow\mathbf{W}^{(i)}_{:,t:(b+B)}-\mathbf{E}_{:,t}^{(i)}\mathbf{L}^{\top}_{t,t:(b+B)}{\color[rgb]{0,0.625,0.8789}-\mathbf{F}_{:,t:(b+B)}}

\triangleright

(Equation 16 ) 14:

\mathbf{F}_{:,t:(b+B)}\leftarrow\mathbf{F}_{:,t:(b+B)}-\mathbf{M}_{:,t}\,\mathbf{L}^{\top}_{t,t:(b+B)}

\triangleright

(Equation 17 ) 15: end for 16:

\mathbf{W}_{:,(b+B):}^{(i)}\leftarrow\mathbf{W}_{:,(b+B):}^{(i)}-\mathbf{E}^{(i)}_{:,b:(b+B)}\,\mathbf{L}^{\top}_{b:(b+B),(b+B):}{\color[rgb]{0,0.625,0.8789}-B\cdot\mathbf{F}_{:,(b+B):}}

00:

{\color[rgb]{0,0.625,0.8789}+\mathbf{M}_{:,b:(b+B)}\,\mathbf{D}\,\mathbf{L}^{\top}_{b:(b+B),(b+B):}}

17:

{\color[rgb]{0,0.625,0.8789}\mathbf{F}_{:,(b+B):}\leftarrow\mathbf{F}_{:,(b+B):}-\mathbf{M}_{:,b:(b+B)}\,\mathbf{L}^{\top}_{b:(b+B),(b+B):}}

\triangleright

(Equation 19 ) 18: end for 19: end for

Algorithm 1:

Model	Method	W2A16			W3A16			W4A16
Model	Method	KL	PPL	QA	KL	PPL	QA	KL	PPL	QA
Qwen3 0.6B	BF16	0	11.28	49.28	0	11.28	49.28	0	11.28	49.28
	RTN	1.43e+01	7.56e+06	28.95	7.44e+00	1.62e+04	29.68	5.84e-01	17.61	41.36
	GPTQ	4.73e+00	1.58e+03	29.92	1.50e+00	55.60	33.09	4.58e-01	18.33	41.57
	GuidedQ	2.46e+00	143.67	30.05	4.96e-01	17.76	38.28	1.42e-01	12.88	45.41
	GPTAQ	3.33e+00	364.96	28.61	8.92e-01	30.40	34.58	3.03e-01	15.85	42.85

Table 1: Weight-only quantization results across different model families. KL, PPL, and QA metrics are reported as averages. Results for more models are provided in Appendix E.3 .

Table 3

Table 4

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm 2: G 2 PTQ quantization for the entire Transformer model

Input: Transformer blocks:

\text{block}^{(l)}

(

l=1,2,\ldots,L

), model input

\mathbf{Y}

, indicator

\mathcal{I}_{\text{autotune}}

Output: Quantized Transformer model 0 1:

\hat{\mathbf{Y}}\leftarrow\mathbf{Y}

0 2: for

l=1,2,\ldots,L

do 0 3: Load

\text{block}^{(l)}

to GPU 0 4:

\mathcal{S}\leftarrow

Set of linear layers to be quantized in

\text{block}^{(l)}

0 5:

\hat{\mathbf{Y}}^{\prime}\leftarrow\hat{\mathbf{Y}}

0 6:

\mathbf{Y}\leftarrow\text{block}^{(l)}(\mathbf{Y})

.detach() 0 7:

\hat{\mathbf{Y}}\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

0 8: if

l<L

then 0 9:

\ell_{H}\leftarrow\ell_{\text{MSE}}\big(\hat{\mathbf{Y}},(\hat{\mathbf{Y}}+\bm{\epsilon})\text{.detach()}\big)

with

\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},d\mathbf{I})

10:

\ell_{G}\leftarrow\ell_{\text{MSE}}(\hat{\mathbf{Y}},\mathbf{Y})

11: else 12:

\ell_{H}\leftarrow-\log p(y|\hat{\mathbf{Y}})

with

y\sim p(\cdot|\mathbf{Y})

13:

\ell_{G}\leftarrow D_{\text{KL}}\big(p(\cdot|\mathbf{Y})\,\|\,p(\cdot|\hat{\mathbf{Y}})\big)

14: end if 15: Backpropagate

\ell_{H}

to cache output activation gradients for each layer in

\mathcal{S}

16: Backpropagate

\ell_{G}

to cache weight gradients for each layer in

\mathcal{S}

17:

\_\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

, compute Hessian via Equation 8 for each layer in

\mathcal{S}

18: if

\mathcal{I}_{\text{autotune}}

then 19: Search for

\alpha^{(l)}

via Algorithm 3 and save it to disk 20: else 21: Load

\alpha^{(l)}

from disk 22: end if 23: Compute trust-region scaling factor

\bm{\beta}

via Equation 56 24: for each linear layer in

\mathcal{S}

do 25: Quantize the linear layer via Algorithm 1 26: end for 27:

\hat{\mathbf{Y}}\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

28: Offload

\text{block}^{(l)}

to CPU 29: end for

Appendix

Algorithm 2:

Algorithm 3: Trust-region threshold autotuning for one Transformer block

Input: Transformer block:

\text{block}^{(l)}

, search range

[\alpha_{\min},\alpha_{\max}]

, step size

\Delta\alpha

, quantized block input

\hat{\mathbf{Y}}^{\prime}

, full-precision block output

\mathbf{Y}

Output: Trust-region threshold

\alpha_{\text{best}}

0 1:

\ell_{\text{best}}\leftarrow\infty

\alpha_{\text{best}}\leftarrow 0

0 2:

\text{state\_dict}\leftarrow\text{block}^{(l)}.\text{state\_dict}()

0 3: for

\alpha\in\{\alpha_{\min},\alpha_{\min}+\Delta\alpha,\ldots,\alpha_{\max}\}

do 0 4:

\mathcal{S}\leftarrow

Set of linear layers to be quantized in

\text{block}^{(l)}

0 5: Compute trust-region scaling factor

\bm{\beta}

via Equation 56 0 6: for each linear layer in

\mathcal{S}

do 0 7: Quantize the linear layer via Algorithm 1 0 8: end for 0 9:

\hat{\mathbf{Y}}\leftarrow\text{block}^{(l)}(\hat{\mathbf{Y}}^{\prime})

10: if

l<L

then 11:

\ell_{G}\leftarrow\ell_{\text{MSE}}(\hat{\mathbf{Y}},\mathbf{Y})

12: else 13:

\ell_{G}\leftarrow D_{\text{KL}}\big(p(\cdot|\mathbf{Y})\,\|\,p(\cdot|\hat{\mathbf{Y}})\big)

14: end if 15:

\text{block}^{(l)}.\text{load\_state\_dict}(\text{state\_dict})

16: if

\ell_{G}<\ell_{\text{best}}

then 17:

\ell_{\text{best}}\leftarrow\ell_{G}

18:

\alpha_{\text{best}}\leftarrow\alpha

19: else 20: break 21: end if 22: end for

Appendix

Algorithm 3:

Model Family	Attn Arch	MLP Arch
LLaMA3 ( Grattafiori et al., 2024 )	GQA	Dense
Qwen3 ( Yang et al., 2025 )	GQA	Dense
Qwen3.5 ( Team, 2026 )	GQA / GDN	Dense
Qwen3-MoE ( Yang et al., 2025 )	GQA	MoE
Qwen3.8-Flash-Next ( Qiu et al., 2026 )	QSA / GDN	MoE

Appendix

Table 2: Comparison of Attention and MLP architectures across different model families.

Panel of Figure 5 — Figure 5: Ablation study on the calibration set size for G 2 PTQ (left) and GPTQ (right). Experiments are conducted using 3-bit weight quantization on the Qwen3-0.6B model.

$g$	KL	PPL	QA	Mem Cost	Time Cost
1	3.43e-01	16.66	40.77	1.00x	1.00x
2	3.39e-01	16.33	40.11	1.00x	1.03x
4	3.33e-01	16.12	39.83	1.01x	1.20x
8	3.31e-01	16.25	39.81	1.02x	1.47x
16	3.23e-01	15.77	40.53	1.04x	2.00x

Appendix

Table 3: Ablation study on the number of output channel groups

g

. We report the average accuracy metrics for 3-bit quantization of the Qwen3-0.6B model, alongside the relative calibration cost.

Method	WikiText2		UltraChat		NuminaMath		QA Avg.
Method	KL	PPL	KL	PPL	KL	PPL	QA Avg.
w/ Approx Grad	7.10e-01	35.29	5.69e-01	12.08	3.32e-01	5.83	37.74
w/ Exact Grad	4.97e-01	32.34	3.39e-01	10.52	1.63e-01	5.49	39.83

Appendix

Table 4: Ablation study on exact versus approximated gradient compensation. Experiments are conducted using 3-bit weight quantization on the Qwen3-0.6B model.

Model	Autotune	WikiText2		UltraChat		NuminaMath		QA Avg.
Model	Autotune	KL	PPL	KL	PPL	KL	PPL	QA Avg.
Qwen3-0.6B (W2A16)	✓	1.85e+00	125.39	1.35e+00	26.60	5.86e-01	7.76	32.93
Qwen3-0.6B (W2A16)	$\times$	1.91e+00	131.15	1.40e+00	28.14	6.02e-01	8.11	32.56
Qwen3-0.6B (W4A16)	✓	1.60e-01	23.98	1.01e-01	8.56	5.35e-02	5.03	45.80
Qwen3-0.6B (W4A16)	$\times$	1.59e-01	23.92	1.01e-01	8.56	5.33e-02	5.04	46.25
Qwen3-0.6B-Base (W3A16)	✓	4.79e-01	20.14	3.42e-01	7.45	1.70e-01	3.75	44.99

Appendix

Table 5: Comparison of setting-specific autotuning (✓) versus applying a “you only autotune once” threshold (

\times

) transferred from the 3-bit Qwen3-0.6B baseline.

Model	H-Scale	WikiText2		UltraChat		NuminaMath		QA Avg.
Model	H-Scale	KL	PPL	KL	PPL	KL	PPL	QA Avg.
Qwen3-0.6B	$\times$	4.97e-01	32.34	3.39e-01	10.52	1.63e-01	5.49	39.83
Qwen3-0.6B	✓	4.49e-01	32.15	2.91e-01	10.35	1.31e-01	5.48	41.95
Qwen3-1.7B	$\times$	4.73e-01	21.66	3.52e-01	11.76	1.51e-01	5.69	51.33
Qwen3-1.7B	✓	3.89e-01	20.28	2.94e-01	11.50	1.09e-01	5.72	53.14
Qwen3-4B	$\times$	4.10e-01	18.38	2.43e-01	7.88	1.07e-01	4.84	60.30

Appendix

Table 6: Effect of Hessian-based weight clipping (H-Scale) across different model families. Experiments are conducted under 3-bit weight quantization.

Model	Method	WikiText2		UltraChat		NuminaMath		CommonSense QA
Model	Method	KL	PPL	KL	PPL	KL	PPL	ARC-C	ARC-E	C-eval	HellaS	LAMB	PIQA	Wino	Avg.
Qwen3 8B	BF16	0	9.72	0	5.20	0	3.60	56.14	80.77	79.57	74.98	64.10	77.97	67.96	71.64
	RTN	4.16e-01	12.33	3.42e-01	5.58	2.41e-01	3.81	44.62	64.44	64.86	66.48	48.52	73.88	64.01	60.97
	GPTQ	2.49e-01	11.32	1.85e-01	5.56	1.24e-01	3.81	52.05	75.38	69.24	69.33	57.87	75.68	64.64	66.31
	GuidedQ	2.54e-01	11.44	1.87e-01	5.59	1.25e-01	3.83	49.83	75.29	67.16	69.21	58.33	74.81	63.54	65.45
	GPTAQ	2.56e-01	11.81	1.86e-01	5.82	1.23e-01	4.00	48.12	72.94	69.02	68.04	57.91	74.54	63.93	64.93

Appendix

Table 10: 4-bit weight-activation quantization results across different model families.

Model	Method	WikiText2		UltraChat		NuminaMath		CommonSense QA
Model	Method	KL	PPL	KL	PPL	KL	PPL	ARC-C	ARC-E	C-eval	HellaS	LAMB	PIQA	Wino	Avg.
Qwen3 30B-A3B (W2A16)	BF16	0	8.70	0	4.79	0	3.41	56.66	78.96	84.32	77.74	64.80	80.58	71.11	73.45
	GPTQ	9.83e-01	20.13	6.24e-01	6.84	4.42e-01	4.65	38.99	56.78	44.73	57.62	49.82	70.24	56.12	53.47
	G 2 PTQ	4.93e-01	13.89	2.86e-01	5.68	2.48e-01	4.18	45.48	67.26	59.73	66.61	53.79	76.06	63.93	61.84
	G 2 PTQ ∗	4.27e-01	12.73	2.13e-01	5.23	2.09e-01	4.04	44.45	66.08	63.08	68.35	55.31	75.90	65.11	62.61
Qwen3 30B-A3B (W3A16)	BF16	0	8.70	0	4.79	0	3.41	56.66	78.96	84.32	77.74	64.80	80.58	71.11	73.45

Appendix

Table 11: Weight-only and weight-activation quantization results on Qwen3-30B-A3B.

Model	Method	WikiText2		UltraChat		NuminaMath		CommonSense QA
Model	Method	KL	PPL	KL	PPL	KL	PPL	ARC-C	ARC-E	C-eval	HellaS	LAMB	PIQA	Wino	Avg.
Qwen3.8 Flash-Next	BF16	0	4.67	0	6.30	0	3.14	64.51	79.08	89.38	87.59	72.81	82.97	71.59	78.28
	GPTQ	1.78e-01	5.02	9.36e-02	6.76	7.61e-02	3.23	62.03	78.03	87.30	87.19	72.48	83.62	72.22	77.55
	G 2 PTQ ∗	1.52e-01	4.85	7.43e-02	5.77	6.20e-02	3.03	62.71	79.12	88.04	87.40	72.39	84.17	71.82	77.95

Appendix

Table 12: 4-bit weight-only quantization results on Qwen3.8-Flash-Next.

Model	Method	GPQA-Diamond	Live-Code-Bench	ArXiv-Math	IFBench	Avg.
Qwen3.8 Flash-Next	BF16	$91.58\pm 0.58$	$93.78\pm 0.29$	$44.01\pm 2.80$	$73.00\pm 0.88$	$75.59\pm 0.84$
	GPTQ	$90.57\pm 0.29$	$92.64\pm 0.05$	43.37 $\pm$ 0.56	$74.11\pm 0.51$	$75.17\pm 0.11$
	G 2 PTQ ∗	91.08 $\pm$ 1.27	92.92 $\pm$ 0.33	$43.36\pm 2.97$	74.22 $\pm$ 0.69	75.40 $\pm$ 0.40

Appendix

Table 13: 4-bit weight-only quantization results for the Qwen3.8-Flash-Next model, evaluated on reasoning benchmarks. We report the mean accuracy and variance across three random seeds.

Explore similar work

Jul 8, 2026cs.LG

KronQ: LLM Quantization via Kronecker-Factored Hessian

Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.

Donghyun Lee, Yuhang Li, Ruokai Yin +1

University of Southern California · Yale University

Mixed-Precision Quantization LLM Quantization Rotation-Based Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

Aug 7, 2026cs.AI

ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization

Post-training quantization (PTQ) is widely used to reduce the memory and computational cost of large language models. Existing PTQ methods typically obtain an initial quantized model through heuristic rules or greedy optimization, and once quantization is completed the resulting integer assignments are usually treated as final. This observation motivates a complementary optimization stage within PTQ that keeps quantized weights improvable after an executable quantized model has been produced, while preserving the quantized format. We introduce ReQuant, a backpropagation-free fixed-grid refinement procedure for this stage. Agnostic to the PTQ initializer, ReQuant takes an existing quantized model as a feasible starting point and iteratively revisits its discrete weight assignments on the fixed quantization grid. Accepted updates strictly reduce the mean squared reconstruction error and remain on the original grid. In this way, ReQuant turns the initially fixed PTQ output into an iteratively optimizable discrete solution and serves as a plug-and-play post-processing stage for existing PTQ pipelines. Experiments across diverse model families, bit-widths, and downstream tasks show that ReQuant consistently improves quantized models from heterogeneous PTQ initializers, with especially large gains on simple initializers and lower bit-widths. Notably, ReQuant can refine a simple round-to-nearest initialization across multiple sweeps until it approaches or surpasses GPTAQ under the same quantization format. These results establish ReQuant as a practical complementary stage for further improving existing PTQ pipelines.

Yongge Ma, Guoan Wang, Feiyu Wang +5

School of Computer Science, Peking University · School of Software and Microelectronics, Peking University · School of Physics, Peking University +2

LLM Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

Apr 15, 2026cs.LG

Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate

Large Language Models (LLMs) are widely used across many domains, but their scale makes deployment challenging. Post-Training Quantization (PTQ) reduces memory footprint without retraining by leveraging a small calibration set. Recent Hessian-based PTQ methods compensate quantization error via cross-channel dependencies, but such approaches degrade at low bit-widths due to noisy curvature estimates from limited calibration data. We propose DASH-Q, a robust PTQ framework using diagonal Hessian approximation and iterative weighted least squares. By discarding noise-prone dependencies, DASH-Q filters sampling noise while prioritizing the preservation of salient feature power. We outperform other PTQ baselines in ultra low-bit regime, improving zero-shot accuracy by 7.01% on average and up to 14.01% over the strongest baselines across five baseline LLM models, while showing robust and stable performance with very small calibration data.

Jaemin Kim, Sungkyun Kim, Junyeol Lee +1

Seoul National University · Hanyang University

LLM Quantization Language Model Calibration Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF