Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Hanzhang Wang; Tianqi Shen; Zonglin Liu; Junze He; Difan Zou; Ziye Ma

cs.LGOct 4, 2026

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Authors: Hanzhang Wang, Tianqi Shen, Zonglin Liu, Junze He, Difan Zou, Ziye Ma

Organizations: City University of Hong Kong · The University of Hong Kong

Abstract

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide a simple way to improve the pretraining-to-quantization transition. But the mechanism behind weight averaging remains insufficiently explained. This leads to inconsistent and fragile performance gains, thereby preventing practitioners from applying such a technique confidently. As a response, we formulate weight averaging as a trade-off between retaining training progress and improving robustness under perturbation. We further derive a continuous family of averaging kernels that unifies conventional strategies and achieves the Pareto frontier between the two competing goals. Critically, a theoretical framework for performing weight averaging under PTQ is developed. It can be shown that coarser quantization is more susceptible to perturbations, whereas finer quantization could be less affected. Thus, our results could provide unified theoretical guidance for performing weight averaging under different PTQ conditions. Experiments validate both the predicted behavior and the proposed averaging strategy. Code is available at https://github.com/MOFA-LAB/weight-averaging-for-ptq.

Figures & tables

Figure 1: Overview of our work. (a) Different averaging strategies’ rankings reverse between 4-bit and 2-bit PTQ. (b) Less lag favors better current performance (FP), while less noise favors greater robustness to quantization (2-bit). (c) We formulate weight averaging as a lag–noise trade-off and derive its Pareto frontier, LNWA . (d) Reduced lag can offset additional noise within a finite tolerance interval; beyond it, further noise leads to degradation. (e) The preferred balance reverses with precision, with lower precision favoring less noise and higher precision favoring less lag.

Figure 2: Illustrative visualizations with a two-layer MLP on MNIST. (a) SGD training trajectories with different data orders and their estimated latent mean. (b) Weight averaging reduces spatial noise but introduces temporal lag compared with the stable training endpoint. (c) LNWA characterizes the Pareto frontier between temporal lag and spatial noise. Colors indicate validation loss.

Methods	Kernel $w_{d}$	Lag $m(w)$	Noise $R(w)$	Exact match
LAWA	$\displaystyle 1/M$	$\displaystyle(M-1)/2$	$\displaystyle 1/M$	$\checkmark$
WMA	$\displaystyle 2(M-d)/[M(M+1)]$	$\displaystyle(M-1)/3$	$\displaystyle 2(2M+1)/[3M(M+1)]$	$\checkmark$
EMA	$\displaystyle(1-\beta)\beta^{d}/(1-\beta^{M})$	$\displaystyle\frac{\beta-M\beta^{M}+(M-1)\beta^{M+1}}{(1-\beta)(1-\beta^{M})}$	$\displaystyle\frac{(1-\beta)(1+\beta^{M})}{(1+\beta)(1-\beta^{M})}$	$M=2$ : $\checkmark$ $M>2$ : $\mathcal{O}(\eta^{2}M)$
Stable	$\displaystyle\bm{e}_{0}=(1,0,\ldots,0)$	$0$	$1$	$\checkmark$

Table 1: Relation between classical averaging kernels and LNWA . LAWA and Stable are the two endpoints, and WMA is an exact interior point. EMA is exact when

M=2

. For

M>2

\lambda_{\textnormal{{EMA}}}

gives a local approximation with error

\mathcal{O}(\eta^{2}M)

, where

\eta=-\log\beta

Figure 3: Comparison of

\Delta

NLL across LNWA coefficients for OPT-125M under GPTQ and AWQ at 4-bit (W4) and 2-bit (W2).

Figure 4: Comparison of PPL across LNWA -equivalent averaging coefficients for OPT-125M under GPTQ and AWQ at 4-bit (W4) and 2-bit (W2).

Figure 5: Comparison of PPL between LR decay, Stable LR, and LNWA for OPT-1.3B under GPTQ and AWQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and their average.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Spatial-noise correlation as the interval between saved weights increases. Left: learning-rate variation at batch size 32. Right: batch-size variation at learning rate

0.03

Figure 7: Spatial-noise variance captured by the first

k

fitted directions. Left: learning-rate variation at batch size 32. Right: batch-size variation at learning rate

0.03

Figure 8: Cross-run estimates of the pairwise inner products between nine consecutive latent-mean progress increments. Left: learning-rate variation. Right: batch-size variation.

Figure 9: Cross-run estimates of the inner products between the window-start latent mean and nine subsequent progress increments. Left: learning-rate variation. Right: batch-size variation.

Figure 10: Comparison of

\Delta

NLL across LNWA coefficients for OPT-125M, OPT-350M, and OPT-1.3B under GPTQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Model size	125M	350M	1.3B
Training tokens	2B	4B	20B
Batch size	256	512	2560
Sequence length	512	512	512
Peak learning rate	$1.0{\times}10^{-3}$	$1.0{\times}10^{-3}$	$1.0{\times}10^{-3}$
LR warmup ratio	$1\%$	$1\%$	$1\%$
LR decay ratio	$10\%$	$10\%$	$10\%$

Appendix

Table 2: Pretraining settings used in the experiments. Global batch size is measured in sequences per optimizer update, and gradient clipping is by global norm.

Figure 11: Comparison of

\Delta

NLL across LNWA coefficients for OPT-125M, OPT-350M, and OPT-1.3B under AWQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Figure 12: Comparison of PPL across LNWA -equivalent averaging coefficients for OPT-125M, OPT-350M, and OPT-1.3B under GPTQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Figure 13: Comparison of PPL across LNWA -equivalent averaging coefficients for OPT-125M, OPT-350M, and OPT-1.3B under AWQ at 4-bit (W4), 3-bit (W3), 2-bit (W2), and full precision (FP).

Figure 14: Comparison of validation loss and perplexity under Stable LR, LR decay, and LNWA for OPT-125M.

Figure 15: Comparison of Hessian-trace trajectories and final Hessian trace between LR decay, Stable LR, and weight averaging for OPT-125M.

Explore similar work

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs
Apr 29, 2026Zhe Ding, Su Pan, Duowei PanLarge Language Model Quantization Post-Training Quantization
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA
ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
May 11, 2026Ryan Lucas, Mehdi Makni, Xiang Meng +2Large Language Model Quantization Model Weights
MIT Operations Research Center · MIT Sloan School of Management · MIT Center for Statistics
Why Does Post-Training Quantization Work?
Sep 10, 2026Yuxiang Chen, Michael Beyer, Jun Zhu +1Post-Training Quantization Quantized
Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University · College of AI, Tsinghua University · Bosch AI Research, Renningen, Germany

Apr 29, 2026, cs.LGJ/K move · Enter open · S save

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs

Zhe Ding, Su Pan, Duowei Pan

School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA

Large Language Model Quantization Post-Training Quantization Qantis Subspace

Open paper AskPDF arXiv

Post-training quantization (PTQ) has become an important technique for reducing the inference cost of Large Language Models (LLMs). While recent mixed-precision methods improve ultra-low bit quantization by preserving critical subspaces in high precision, they typically construct these subspaces relying solely on activation statistics. This ignores the fundamental nature of linear operations, where the output perturbation is jointly driven by both activation and weight quantization noise. In this paper, we propose CoQuant, a joint weight-activation subspace projection method. By theoretically modeling the expected output error, CoQuant formulates a closed-form weighted PCA solution that balances activation and weight covariances to select the optimal high-precision subspace. Extensive experiments on Llama-3.2 and Qwen2.5 models show that CoQuant consistently outperforms strong PTQ baselines in both WikiText perplexity and zero-shot common-sense reasoning accuracy. These results demonstrate that joint weight-activation subspace modeling provides a principled and effective direction for low-bit LLM quantization. The source code is available at https://github.com/Zachary5895/CoQuant.

Abstract

Explore similar work

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs

ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models

Why Does Post-Training Quantization Work?