The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

Chao Li; Shigeng Wang; Anbang Yao

cs.CLOct 1, 2026

The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

Authors: Chao Li, Shigeng Wang, Anbang Yao

Organizations: Intel Labs China

Abstract

Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with one unit quantized at each stage. State-of-the-art PTQ methods are predominantly learning-based, optimizing auxiliary quantization parameters (e.g., scaling factors, rotation matrices, clipping thresholds, and adapters) via gradient descent to minimize a reconstruction loss. A common practice is to use mean squared error (MSE) as the reconstruction loss function, yet its induced optimization behavior remains largely unexplored. In this work, we take a holistic view of sequential quantization and systematically investigate how optimization evolves from the first quantization stage to the last, aiming for a deep understanding of optimization in learning-based PTQ schemes. Through extensive empirical studies spanning representative learning-based PTQ methods, LLM families, model scales, architectures, quantization settings and various tasks, we consistently uncover Optimization Imbalance: reconstruction loss magnitudes vary dramatically across stages, accompanied by highly uneven gradient magnitudes and parameter updates under MSE. We term the cross-stage range of loss magnitudes the reconstruction loss scale, and reveal that MSE translates the unexpectedly large reconstruction loss scale into highly uneven gradient magnitudes, which in turn lead to uneven optimization strength across quantization stages. This finding suggests a general principle for improving learning-based PTQ: optimization strength across stages should be decoupled from the reconstruction loss scale. Theoretically, we show that root mean squared error (RMSE) variants defined at the sample, channel, token, and element levels naturally realize this principle through implicit gradient normalization, outperforming MSE significantly as a drop-in replacement.

Figures & tables

Figure 1: Illustration of Optimization Imbalance in learning-based PTQ and its mitigation via RMSE. Learning-based PTQ methods predominantly use MSE as the reconstruction loss function for each stage in sequential quantization. By tracking optimization dynamics from the first stage to the last, we observe that reconstruction loss magnitudes vary dramatically across quantization stages. Our analysis reveals that MSE translates these differences into highly uneven gradients and parameter updates, giving rise to Optimization Imbalance. RMSE breaks this loss-dependent coupling through implicit gradient normalization, enabling more balanced optimization across quantization stages. Statistics are collected using OmniQuant on Qwen3.5-9B under W4A4 quantization.

Panel of Table 1 — Table 1: Generalization of RMSE under OmniQuant across diverse LLMs and quantization settings. Acc denotes the average accuracy over PIQA, ARC-Easy, ARC-Challenge, Winogrande, HellaSwag, and BoolQ.

#Bits	LLaMA2-7B					Qwen3.5-9B
	MSE	RMSE				MSE	RMSE
	MSE	Sample	Channel	Token	Element	MSE	Sample	Channel	Token	Element
W4A16	68.33	68.88	69.33	69.07	68.64	74.23	74.97	74.35	75.08	74.70
W3A16	64.05	64.17	65.88	66.78	66.58	71.10	73.17	72.79	72.47	72.92
W2A16	43.98	45.09	46.01	49.97	46.76	45.44	47.61	54.78	47.63	52.76
W2A16G128	51.66	56.58	57.36	57.29	57.54	53.54	61.08	60.03	59.89	58.63

Table 2: Comparison of RMSE variants across quantization settings under OmniQuant. Results are reported as average accuracy (%) on downstream benchmarks.

#Bits	Loss	Mathematical Reasoning		Coding		Avg
#Bits	Loss	GSM8K	MATH-500	HumanEval	MBPP	Avg
Original	–	96.44	97.20	85.37	70.37	87.35
W4A16	MSE	95.15	94.20	75.61	57.14	80.53
W4A16	RMSE	95.98	94.80	75.00	62.96	82.19
W3A16	MSE	78.70	51.20	32.93	24.34	46.79
W3A16	RMSE	81.65	54.60	35.98	25.40	49.41

Table 3: Generalization of RMSE to mathematical reasoning and coding tasks under OmniQuant on Qwen3.5-9B. All results are reported as accuracy (%).

Figure 5: Perplexity evolution with quantization depth under MSE and RMSE. The horizontal axis represents the quantization stages from early to late. Left: OmniQuant with W4A4 quantization across diverse LLMs. Right: representative learning-based PTQ methods on Qwen3.5-9B under W4A4 quantization.

Model	#Bits	Runtime (minutes)			Peak memory (GB)
Model	#Bits	MSE	RMSE	Overhead	MSE	RMSE	Overhead
LLaMA2-7B	W4A16	123.9	127.4	+2.82%	6.14	6.14	+0.00%
LLaMA2-7B	W4A4	181.7	181.1	-0.33%	7.28	7.28	+0.00%
Qwen3.5-9B	W4A16	184.8	182.6	-1.19%	5.63	5.63	+0.00%
Qwen3.5-9B	W4A4	233.9	236.3	+1.02%	6.90	6.90	+0.00%

Table 4: Quantization process efficiency comparison between MSE and RMSE under OmniQuant. Runtime and peak GPU memory are measured over the entire sequential quantization process on a single NVIDIA L40 GPU with a batch size of 1.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	#Bits	Loss	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	WG $\uparrow$	HS $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
LLaMA2-7B	Original	–	5.47	79.11	74.54	46.25	69.06	76.00	77.71	70.45
	W4A16	MSE	5.75	78.35	71.63	42.83	69.53	73.90	73.73	68.33
	W4A16	RMSE	5.70	78.02	73.57	43.43	68.35	74.46	76.57	69.07
	W3A16	MSE	6.62	75.90	65.87	38.57	65.43	69.70	68.87	64.05
	W3A16	RMSE	6.38	76.77	69.99	42.49	67.32	72.04	72.08	66.78
	W2A16	MSE	36.92	57.73	35.48	25.68	50.59	35.97	58.44	43.98

Appendix

Table 5: Full downstream results under OmniQuant across diverse LLMs and quantization settings. Avg denotes the average accuracy over PIQA, ARC-Easy (ARC-e), ARC-Challenge (ARC-c), Winogrande (WG), HellaSwag (HS), and BoolQ.

Model	Method	Loss	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	WG $\uparrow$	HS $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
LLaMA2-7B	Original	–	5.47	79.11	74.54	46.25	69.06	76.00	77.71	70.45
	OmniQuant	MSE	17.42	62.35	46.38	27.73	54.46	46.94	56.67	49.09
	OmniQuant	RMSE	11.48	69.42	57.41	33.70	55.01	59.86	61.96	56.23
	SliderQuant	MSE	8.48	70.46	54.92	32.85	58.09	60.60	66.30	57.20
	SliderQuant	RMSE	8.14	72.09	60.69	34.98	60.77	62.81	67.09	59.74
	DuQuant	MSE	6.07	76.71	69.11	42.49	67.88	72.01	73.24	66.91

Appendix

Table 6: Full downstream results across representative learning-based PTQ methods under W4A4 quantization. Avg denotes the average accuracy over PIQA, ARC-Easy, ARC-Challenge, Winogrande, HellaSwag, and BoolQ.

Model	Loss	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	WG $\uparrow$	HS $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
LLaMA2-7B	MSE	6.07	76.71	69.11	42.49	67.88	72.01	73.24	66.91
	RMSE-Sample	6.02	76.28	70.03	42.49	67.88	72.88	74.10	67.28
	RMSE-Channel	6.01	78.02	71.55	43.69	66.06	72.76	73.61	67.61
	RMSE-Token	6.01	77.31	70.62	43.77	67.01	72.72	74.71	67.69
	RMSE-Element	6.01	77.31	70.54	42.66	65.59	72.68	73.55	67.06
Qwen3.5-9B	MSE	10.95	76.06	73.86	47.18	66.38	71.68	83.95	69.85

Appendix

Table 7: Comparison of RMSE variants with DuQuant under W4A4 quantization. Avg denotes the average accuracy over PIQA, ARC-Easy, ARC-Challenge, Winogrande, HellaSwag, and BoolQ.

#Bits	Loss	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	WG $\uparrow$	HS $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
W4A16	MSE	5.75	78.35	71.63	42.83	69.53	73.90	73.73	68.33
	RMSE-Sample	5.70	79.05	73.23	43.52	68.51	74.19	74.77	68.88
	RMSE-Channel	5.70	78.40	73.53	44.11	69.46	74.33	76.15	69.33
	RMSE-Token	5.70	78.02	73.57	43.43	68.35	74.46	76.57	69.07
	RMSE-Element	5.71	78.29	72.90	42.06	68.35	74.68	75.54	68.64
W3A16	MSE	6.62	75.90	65.87	38.57	65.43	69.70	68.87	64.05

Appendix

Table 8: Comparison of RMSE variants with OmniQuant on LLaMA2-7B across quantization settings. Avg denotes the average accuracy over PIQA, ARC-Easy, ARC-Challenge, Winogrande, HellaSwag, and BoolQ.

#Bits	Loss	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	WG $\uparrow$	HS $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
W4A16	MSE	7.19	80.52	78.11	50.43	71.43	76.45	78.62	72.59
	RMSE-Sample	6.96	80.90	77.95	49.66	71.74	77.17	79.20	72.77
	RMSE-Channel	6.96	79.92	78.45	51.11	71.67	77.34	78.75	72.87
	RMSE-Token	6.92	80.14	78.07	50.43	71.19	77.45	80.06	72.89
	RMSE-Element	6.96	79.76	77.27	50.43	71.27	77.23	79.97	72.65
W3A16	MSE	12.38	72.47	63.17	37.12	58.33	63.62	68.59	60.55

Appendix

Table 9: Comparison of RMSE variants with OmniQuant on LLaMA3.1-8B across quantization settings. Avg denotes the average accuracy over PIQA, ARC-Easy, ARC-Challenge, Winogrande, HellaSwag, and BoolQ.

#Bits	Loss	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	WG $\uparrow$	HS $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
W4A16	MSE	9.28	78.89	76.64	54.52	71.19	76.22	87.89	74.23
	RMSE-Sample	9.14	79.82	78.62	56.23	70.48	76.46	88.20	74.97
	RMSE-Channel	9.14	80.25	75.67	55.89	71.59	76.43	86.27	74.35
	RMSE-Token	9.20	79.16	78.87	58.11	71.03	76.49	86.79	75.08
	RMSE-Element	9.20	79.05	77.40	56.57	70.88	76.45	87.83	74.70
W3A16	MSE	11.56	76.77	77.61	52.13	68.82	70.33	80.92	71.10

Appendix

Table 10: Comparison of RMSE variants with OmniQuant on Qwen3.5-9B across quantization settings. Avg denotes the average accuracy over PIQA, ARC-Easy, ARC-Challenge, Winogrande, HellaSwag, and BoolQ.

#Bits	Loss	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	WG $\uparrow$	HS $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
W2A16	MSE	37.51	68.55	71.34	44.80	60.62	53.46	63.91	60.45
	RMSE-Sample	24.28	72.96	77.90	50.60	66.46	61.73	69.36	66.50
	RMSE-Channel	22.20	74.59	78.11	52.47	68.90	67.99	83.67	70.96
	RMSE-Token	25.04	72.58	76.14	47.70	65.98	59.64	72.08	65.69
	RMSE-Element	24.53	72.85	74.62	49.83	68.11	64.07	84.28	68.96

Appendix

Table 11: Comparison of RMSE variants with OmniQuant on Qwen3.5-27B under W2A16 quantization. Avg denotes the average accuracy over PIQA, ARC-Easy, ARC-Challenge, Winogrande, HellaSwag, and BoolQ.

Figure 6: Optimization statistics under MSE and different RMSE variants with DuQuant on LLaMA2-7B and Qwen3.5-9B under W4A4 quantization. The horizontal axis denotes the quantization stages from early to late.

Figure 7: Optimization statistics under MSE and different RMSE variants with OmniQuant on LLaMA2-7B and LLaMA3.1-8B across quantization settings. The horizontal axis denotes the quantization stages from early to late.

Figure 8: Optimization statistics under MSE and different RMSE variants with OmniQuant. Results are reported across multiple quantization settings on Qwen3.5-9B and under W2A16 quantization on Qwen3.5-27B. The horizontal axis denotes the quantization stages from early to late.

Figure 9: Mean absolute magnitudes of high-precision block outputs across model depth. The horizontal axis denotes the block order from early to late, and the vertical axis reports the mean absolute output magnitude,

\|X_{i+1}\|_{1}/N

, of each block. Statistics are collected from the original high-precision models using 128 WikiText2 samples with a sequence length of 2048.

Figure 10: Distributions of group-wise reconstruction losses across blocks. For the fully quantized model, we partition the high-precision and quantized outputs of each block into groups at the sample, channel, token, and element levels, and compute the MSE loss independently within each group. The horizontal axis denotes the block order from early to late, and the vertical axis reports the group-wise MSE. The solid line denotes the median across groups, the shaded regions show different percentile intervals, and the red dotted line denotes the maximum. Results are collected from OmniQuant models trained with MSE under W4A4 quantization.

Explore similar work

Jun 9, 2026cs.LG

Optimal Post-Training Quantization Scales and Where to Find Them

Post-training quantization (PTQ) compresses large language models by mapping weights to low-bit representations. The scaling factor that defines the quantization grid is typically chosen using simple, data-free heuristics. In this work, we present PiSO (Piecewise Scale Optimization), an algorithm that leverages calibration data to compute the optimal channel-wise weight scales exactly and efficiently under round-to-nearest quantization. PiSO partitions the scale search space into finitely many intervals on which the objective admits a closed-form minimizer. We extend PiSO to group-wise quantization via principled heuristics and propose effective strategies for interleaving scale optimization with error correction. Experiments on Llama and Qwen models across multiple model sizes and target weight bit-widths demonstrate consistent improvements in perplexity and downstream zero-shot accuracy, both standalone and combined with error correction. In particular, we observe increased benefits as the target bit-width narrows and quantization becomes more challenging.

Juan Amboage, Pablo Monteagudo-Lago, Ian Colbert +2

AMD

Post-Training Quantization Quantizer Post-Training

Ask PDF

Sep 28, 2026cs.CV

Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models

Post-training quantization (PTQ) enables efficient deployment of large vision-language models (LVLMs), but is typically calibrated on a small set while expected to generalize across diverse downstream tasks. Although recent PTQ methods for LVLMs incorporate sensitivity signals, they still minimize reconstruction loss with respect to the full-precision model, potentially over-preserving FP behavior and calibration-specific bias. Rather than treating quantization solely as an error to be minimized, we observe that it can also provide beneficial regularization for certain layers and modalities. Motivated by this observation, we propose Balanced Fitting, a quantization effect-based framework that balances precision and regularization beyond reconstruction-based optimization. By measuring layer- and component-wise quantization effects for weights, vision activations, and text activations, Balanced Fitting combines fine-grained fitting for sensitive components with coarser fitting to exploit potential regularization benefits. Experiments on multiple LVLMs show that our method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance. The source code is publicly available at https://github.com/kmc3661/BFQ

Minchan Kang, Kyeonghye Park, Seungyeon Sa +3

Korea Advanced Institute of Science and Technology (KAIST) · Hanbat National University

Post-Training Quantization Recent Vision-Language Models Class Imbalance

Ask PDF

Aug 7, 2026cs.AI

ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization

Post-training quantization (PTQ) is widely used to reduce the memory and computational cost of large language models. Existing PTQ methods typically obtain an initial quantized model through heuristic rules or greedy optimization, and once quantization is completed the resulting integer assignments are usually treated as final. This observation motivates a complementary optimization stage within PTQ that keeps quantized weights improvable after an executable quantized model has been produced, while preserving the quantized format. We introduce ReQuant, a backpropagation-free fixed-grid refinement procedure for this stage. Agnostic to the PTQ initializer, ReQuant takes an existing quantized model as a feasible starting point and iteratively revisits its discrete weight assignments on the fixed quantization grid. Accepted updates strictly reduce the mean squared reconstruction error and remain on the original grid. In this way, ReQuant turns the initially fixed PTQ output into an iteratively optimizable discrete solution and serves as a plug-and-play post-processing stage for existing PTQ pipelines. Experiments across diverse model families, bit-widths, and downstream tasks show that ReQuant consistently improves quantized models from heterogeneous PTQ initializers, with especially large gains on simple initializers and lower bit-widths. Notably, ReQuant can refine a simple round-to-nearest initialization across multiple sweeps until it approaches or surpasses GPTAQ under the same quantization format. These results establish ReQuant as a practical complementary stage for further improving existing PTQ pipelines.

Yongge Ma, Guoan Wang, Feiyu Wang +5

School of Computer Science, Peking University · School of Software and Microelectronics, Peking University · School of Physics, Peking University +2

Post-Training Quantization Qantis Refinement

Ask PDF