Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Hanzhang Wang; Tianqi Shen; Zonglin Liu; Junze He; Difan Zou; Ziye Ma

cs.LGOct 4, 2026

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Authors: Hanzhang Wang, Tianqi Shen, Zonglin Liu, Junze He, Difan Zou, Ziye Ma

Organizations: City University of Hong Kong · The University of Hong Kong

Abstract

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide a simple way to improve the pretraining-to-quantization transition. But the mechanism behind weight averaging remains insufficiently explained. This leads to inconsistent and fragile performance gains, thereby preventing practitioners from applying such a technique confidently. As a response, we formulate weight averaging as a trade-off between retaining training progress and improving robustness under perturbation. We further derive a continuous family of averaging kernels that unifies conventional strategies and achieves the Pareto frontier between the two competing goals. Critically, a theoretical framework for performing weight averaging under PTQ is developed. It can be shown that coarser quantization is more susceptible to perturbations, whereas finer quantization could be less affected. Thus, our results could provide unified theoretical guidance for performing weight averaging under different PTQ conditions. Experiments validate both the predicted behavior and the proposed averaging strategy. Code is available at https://github.com/MOFA-LAB/weight-averaging-for-ptq.

Figures & tables

Figure 1: Overview of our work. (a) Different averaging strategies’ rankings reverse between 4-bit and 2-bit PTQ. (b) Less lag favors better current performance (FP), while less noise favors greater robustness to quantization (2-bit). (c) We formulate weight averaging as a lag–noise trade-off and derive its Pareto frontier, LNWA . (d) Reduced lag can offset additional noise within a finite tolerance interval; beyond it, further noise leads to degradation. (e) The preferred balance reverses with precision, with lower precision favoring less noise and higher precision favoring less lag.

Figure 2: Illustrative visualizations with a two-layer MLP on MNIST. (a) SGD training trajectories with different data orders and their estimated latent mean. (b) Weight averaging reduces spatial noise but introduces temporal lag compared with the stable training endpoint. (c) LNWA characterizes the Pareto frontier between temporal lag and spatial noise. Colors indicate validation loss.

Methods	Kernel $w_{d}$	Lag $m(w)$	Noise $R(w)$	Exact match
LAWA	$\displaystyle 1/M$	$\displaystyle(M-1)/2$	$\displaystyle 1/M$	$\checkmark$
WMA	$\displaystyle 2(M-d)/[M(M+1)]$	$\displaystyle(M-1)/3$	$\displaystyle 2(2M+1)/[3M(M+1)]$	$\checkmark$
EMA	$\displaystyle(1-\beta)\beta^{d}/(1-\beta^{M})$	$\displaystyle\frac{\beta-M\beta^{M}+(M-1)\beta^{M+1}}{(1-\beta)(1-\beta^{M})}$	$\displaystyle\frac{(1-\beta)(1+\beta^{M})}{(1+\beta)(1-\beta^{M})}$	$M=2$ : $\checkmark$ $M>2$ : $\mathcal{O}(\eta^{2}M)$
Stable	$\displaystyle\bm{e}_{0}=(1,0,\ldots,0)$	$0$	$1$	$\checkmark$

Table 1: Relation between classical averaging kernels and LNWA . LAWA and Stable are the two endpoints, and WMA is an exact interior point. EMA is exact when

M=2

. For

M>2

\lambda_{\textnormal{{EMA}}}

gives a local approximation with error

\mathcal{O}(\eta^{2}M)

, where

\eta=-\log\beta

Figure 3: Comparison of

\Delta

NLL across LNWA coefficients for OPT-125M under GPTQ and AWQ at 4-bit (W4) and 2-bit (W2).

Figure 4: Comparison of PPL across LNWA -equivalent averaging coefficients for OPT-125M under GPTQ and AWQ at 4-bit (W4) and 2-bit (W2).

Figure 5: Comparison of PPL between LR decay, Stable LR, and LNWA for OPT-1.3B under GPTQ and AWQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and their average.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Spatial-noise correlation as the interval between saved weights increases. Left: learning-rate variation at batch size 32. Right: batch-size variation at learning rate

0.03

Figure 7: Spatial-noise variance captured by the first

k

fitted directions. Left: learning-rate variation at batch size 32. Right: batch-size variation at learning rate

0.03

Figure 8: Cross-run estimates of the pairwise inner products between nine consecutive latent-mean progress increments. Left: learning-rate variation. Right: batch-size variation.

Figure 9: Cross-run estimates of the inner products between the window-start latent mean and nine subsequent progress increments. Left: learning-rate variation. Right: batch-size variation.

Figure 10: Comparison of

\Delta

NLL across LNWA coefficients for OPT-125M, OPT-350M, and OPT-1.3B under GPTQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Model size	125M	350M	1.3B
Training tokens	2B	4B	20B
Batch size	256	512	2560
Sequence length	512	512	512
Peak learning rate	$1.0{\times}10^{-3}$	$1.0{\times}10^{-3}$	$1.0{\times}10^{-3}$
LR warmup ratio	$1\%$	$1\%$	$1\%$
LR decay ratio	$10\%$	$10\%$	$10\%$

Appendix

Table 2: Pretraining settings used in the experiments. Global batch size is measured in sequences per optimizer update, and gradient clipping is by global norm.

Figure 11: Comparison of

\Delta

NLL across LNWA coefficients for OPT-125M, OPT-350M, and OPT-1.3B under AWQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Figure 12: Comparison of PPL across LNWA -equivalent averaging coefficients for OPT-125M, OPT-350M, and OPT-1.3B under GPTQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Figure 13: Comparison of PPL across LNWA -equivalent averaging coefficients for OPT-125M, OPT-350M, and OPT-1.3B under AWQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Figure 14: Comparison of validation loss and perplexity under Stable LR, LR decay, and LNWA for OPT-125M.

Figure 15: Comparison of Hessian-trace trajectories and final Hessian trace between LR decay, Stable LR, and weight averaging for OPT-125M.

Explore similar work

Apr 29, 2026cs.LG

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs

Post-training quantization (PTQ) has become an important technique for reducing the inference cost of Large Language Models (LLMs). While recent mixed-precision methods improve ultra-low bit quantization by preserving critical subspaces in high precision, they typically construct these subspaces relying solely on activation statistics. This ignores the fundamental nature of linear operations, where the output perturbation is jointly driven by both activation and weight quantization noise. In this paper, we propose CoQuant, a joint weight-activation subspace projection method. By theoretically modeling the expected output error, CoQuant formulates a closed-form weighted PCA solution that balances activation and weight covariances to select the optimal high-precision subspace. Extensive experiments on Llama-3.2 and Qwen2.5 models show that CoQuant consistently outperforms strong PTQ baselines in both WikiText perplexity and zero-shot common-sense reasoning accuracy. These results demonstrate that joint weight-activation subspace modeling provides a principled and effective direction for low-bit LLM quantization. The source code is available at https://github.com/Zachary5895/CoQuant.

Zhe Ding, Su Pan, Duowei Pan

School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA

Large Language Model Quantization Post-Training Quantization Qantis Subspace

Ask PDF

May 11, 2026cs.LG

ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models

Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (PTQ) is a leading approach for compressing LLMs. Popular weight quantization procedures, including GPTQ and RTN, suffer in model utility, especially at aggressive quantization levels (sub-4-bit). We propose ADMM-Q, a novel weight quantization algorithm that considers the layer-wise quantization problem. Our algorithm is based on a combinatorial variant of the Alternating Direction Method of Multipliers (ADMM). Our operator-splitting procedure updates weights continuously to minimize the layer-wise reconstruction error, while gradually enforcing the quantization constraints with convergence guarantees. We propose additional algorithmic enhancements (e.g., penalty scheduling, preconditioning, and a local search post-processing step) to make ADMM-Q efficient at LLM scale. ADMM-Q is modular and can be used as a drop-in replacement for any weight quantizer within existing quantization pipelines: ADMM-Q is fully composable with existing techniques including range clipping, learned or random rotations, and activation scaling. Using ADMM-Q in place of GPTQ on Qwen3-8B, we decrease WikiText-2 perplexity in: (i) the W3A16 weight-only setting (12.85

\rightarrow

10.06); (ii) the W4A8 SmoothQuant procedure (9.29

\rightarrow

8.68); and (iii) the W2A4KV4 SpinQuant procedure (66.11

\rightarrow

19.42).

Ryan Lucas, Mehdi Makni, Xiang Meng +2

MIT Operations Research Center · MIT Sloan School of Management · MIT Center for Statistics

Large Language Model Quantization Model Weights Alternating Direction Method Of Multiplier

Ask PDF

Sep 10, 2026cs.LG

Why Does Post-Training Quantization Work?

Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.

Yuxiang Chen, Michael Beyer, Jun Zhu +1

Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University · College of AI, Tsinghua University · Bosch AI Research, Renningen, Germany

Post-Training Quantization Quantized Post-Training Growth

Ask PDF