cs.LGApr 9, 2026

Rethinking Residual Errors in Compensation-based LLM Quantization

Authors: Shuaiting Li, Juncan Deng, Kedong Xu, Rongtao Deng, Hong Gu, Minghan Jiang, Haibin Shen, Kejie Huang

Organizations: Zhejiang University · vivo Mobile Communication Co., Ltd

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Methods based on weight compensation, which iteratively apply quantization and weight compensation to minimize the output error, have recently demonstrated remarkable success in quantizing Large Language Models (LLMs). The representative work, GPTQ, introduces several key techniques that make such iterative methods practical for LLMs with billions of parameters. GPTAQ extends this approach by introducing an asymmetric calibration process that aligns the output of each quantized layer with its full-precision counterpart, incorporating a residual error into the weight compensation framework. In this work, we revisit the formulation of the residual error. We identify a sub-optimal calibration objective in existing methods: during the intra-layer calibration process, they align the quantized output with the output from compensated weights, rather than the true output from the original full-precision model. Therefore, we redefine the objective to precisely align the quantized model's output with the original output of the full-precision model at each step. We then reveal that the residual error originates not only from the output difference of the preceding layer but also from the discrepancy between the compensated and original weights within each layer, which we name the 'compensation-aware error'. By inheriting the neuron decomposition technique from GPTAQ, we can efficiently incorporate this compensation-aware error into the weight update process. Extensive experiments on various LLMs and quantization settings demonstrate that our proposed enhancements integrate seamlessly with both GPTQ and GPTAQ, significantly improving their quantization performance. Our code is publicly available at https://github.com/list0830/ResComp.

Figures & tables

Figure 1: Overview of compensation-based LLM quantization methods. (A) GPTQ : Minimizes the layer-wise reconstruction error based on the current quantized input flow, neglecting error accumulation from previous layers. (B) GPTAQ : Introduces asymmetric calibration to address inter-layer error accumulation, but uses the compensated weights as the calibration target during the update process. (C) GPTAQ+Ours : Strictly aligns with the original full-precision output by incorporating a novel ”Compensation-Aware Error,” which corrects the discrepancy between compensated and original weights.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

MatrixGPTQGPTAQGPTAQ+Ours
Original weight: W(0)W^{(0)}--Co×CiC_{o}\times C_{i}
Compensated weight: WWCo×CiC_{o}\times C_{i}Co×CiC_{o}\times C_{i}Co×CiC_{o}\times C_{i}
Fake quant weight: QQCo×CiC_{o}\times C_{i}Co×CiC_{o}\times C_{i}Co×CiC_{o}\times C_{i}
Cholesky factor: LLCi×CiC_{i}\times C_{i}Ci×CiC_{i}\times C_{i}Ci×CiC_{i}\times C_{i}
Precompute 1: P1P1-Ci×CiC_{i}\times C_{i}Ci×CiC_{i}\times C_{i}
Precompute 2: P2P2--Ci×CiC_{i}\times C_{i}
Appendix
Table 10: Matrix needed to perform calibration and their sizes. CoC_{o} and CiC_{i} denotes the output-channel and input-channel of weights, while bb denotes the blocksize for lazy-batch update.
More in this topic

Quantization Error Compensation

See all 17 papers

Explore similar work

CardsList
Sep 18, 2026

Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals

Post-training weight-activation quantization reduces the memory and inference costs of large language models, but aggressive W4A4 quantization remains difficult because activation outliers degrade effective quantization resolution. Although weight optimization, channel-wise scaling, and orthogonal rotation mitigate this problem, the error components they address and their relationship remain unclear. Using an exact decomposition of local weight-activation quantization error into an activation-guided weight compensation term and an orthogonal residual, we bound the residual using persistent channel-wise outlier and regular activation quantities. This decomposition clarifies which error components can be addressed by weight compensation and which require transformation design. We then use the residual bounds to derive practical guidelines for applying randomized Hadamard rotation, sign selection, and channel scaling. In particular, the analysis explains how random signs suppress constructive interference among persistent outlier channels, how sampling multiple sign patterns can improve transformation selection, and how second-moment balancing leads to an L2L_2 scaling rule while a further relaxation recovers SmoothQuant-style L∞L_\infty scaling. We evaluate these guidelines through backpropagation-free configurations across eight Llama and Mistral models, obtaining performance competitive with gradient-trained SpinQuant.

Yamato Narita, Issei Sato

Department of Computer Science, The University of Tokyo

Post-Training QuantizationNeural Network QuantizationLLM QuantizationLow-Bit QuantizationActivation QuantizationQuantization Error Compensation+1
AskPDF
Oct 25, 2026

ResCom: Activation Residual Compensation for Large Language Model Quantization

Low-bit weight--activation quantization reduces the memory and computation needed to deploy large language models, but activation rounding can still degrade accuracy after calibration. Reconstruction from rounded inputs cannot distinguish activations within the same quantization bin. We propose ResCom, a low-rank correction that uses the activation rounding residual to compensate for the resulting layer-output error. An output-aware calibration procedure selects residual directions by their contribution to output reconstruction and computes the correction in closed form, without gradient training or updates to the calibrated weights. On Llama-3-8B with 4-bit weights and activations, rank-32 attention corrections reduce WikiText-2 perplexity from 7.708 to 7.147, improve MMLU by 2.89 percentage points, and raise five-shot GSM8K accuracy from 23.96 to 36.16 percent. Evaluations on models from 1B to 70B parameters and on held-out C4 test the method beyond this setting. The evaluated FP32 implementation adds 2.25 percent to 4K prefill time; a separate compact-storage implementation retains a 27.7 percent peak-memory reduction relative to FP16.

Ning Yang, Bohan Cui, Zhuowen Liu +1

Institute of Automation, Chinese Academy of Sciences · Fudan University · The Chinese University of Hong Kong +1

LLM QuantizationQuantization Error CompensationLow-Bit QuantizationPost-Training QuantizationActivation Quantization
AskPDF
Sep 25, 2026

G2^2PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G2^2PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G2^2PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model families and bit-widths demonstrate that G2^2PTQ enables better alignment with the full-precision model, outperforming state-of-the-art baselines. Code is available at: https://github.com/G2PTQ/G2PTQ.

Ruikang Liu, Haoli Bai, Yuxuan Sun +8

ZTE Corporation · The Chinese University of Hong Kong · Northwestern Polytechnical University +2

LLM QuantizationPost-Training QuantizationLow-Bit Quantization
AskPDF