End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost

Qitao Tan; Xiaoying Song; Ningxi Cheng; Guoming Li; Jun Liu; Lingzi Hong; Caiwen Ding; Jundong Li

cs.LGAug 21, 2025

End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost

Authors: Qitao Tan, Xiaoying Song, Ningxi Cheng, Guoming Li, Jun Liu, Lingzi Hong, Caiwen Ding, Jundong Li,

Organizations: University of Georgia · University of North Texas · Northeastern University · University of Minnesota ·

Abstract

Quantization reduces the deployment cost of large language models (LLMs). However, post-training quantization (PTQ) cannot fine-tune model parameters and often degrades accuracy in low-bit settings, while quantization-aware training (QAT) requires backpropagation and incurs high memory overhead. We propose ZeroQAT, a zeroth-order QAT framework supporting both weight and activation quantization. By using forward-only gradient estimation, ZeroQAT removes backpropagation and substantially reduces memory and computation while retaining end-to-end optimization benefits. We further introduce a lightweight variant of ZeroQAT for quantized fine-tuning, which freezes and pre-quantizes most parameters to further cut memory usage. Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory. For example, ZeroQAT enables fine-tuning of a 13B model at extremely low bit-widths (e.g., 2-4 bits) on a single 8GB GPU, and even allows fine-tuning a 6.7B model on a OnePlus 12 smartphone, demonstrating its practicality for end-to-end QAT on resource-limited edge devices.

Figures & tables

Method	Category	Quant Support	Low-bit performance		Memory Efficiency
Method	Category	Quant Support	Pre-train	Fine-tune	Memory Efficiency
SmoothQuant	Range PTQ	WA	✗	✗	High
GPTQ	Approx PTQ	WO	✗	✗	High
OmniQuant	Approx PTQ	WA	✓	✗	Moderate
LLM-QAT	Full QAT	WA	✓	✓	Low
QLoRA	PEFT QAT	WO	✓	✓	Moderate

Table 1: Comparison of our method with existing methods. PEFT indicates parameter-efficient fine-tuning. WO and WA indicate weight-only and weight-activation quantization, respectively.

Method	Category	Quantized Pre-training (PPL $\downarrow$ )			Quantized Fine-tuning (Acc $\uparrow$ )
Method	Category	W6A6	W2A16	W4A4	W6A6	W2A16g128	W4A4
ZO (FP16)	-	5.47			66.0
Zero-shot	-	-			41.3
SmoothQuant	Range-based PTQ	6.20	100.23 ‡	83.12	57.2	27.7 ‡	32.9 ‡
OmniQuant	Approx-based PTQ	5.87	37.37	14.26	63.9	40.6 ‡	38.8 ‡
EfficientQAT	QAT	5.60	33.40	76.32 ‡	66.4	45.4	28.6 ‡

Table 2: Results of applying different quantization methods on LLama-2-7B. ‡ indicates that the method is intrinsically not suitable for the setting; we report these results to illustrate its limitations.

Figure 1: Comparison of the block-wise reconstruction loss reduction between OmniQuant ( Shao et al., 2023 ) and our method. The X-axis is the index of layer, Y-axis measures the ratio of loss decrease.

Figure 2: Objective inconsistency between the reconstruction loss used by approximation-based PTQ and final evaluation metrics.

LLama / PPL $\downarrow$		LLama1-7B		LLama1-13B		LLama2-7B		LLama2-13B		LLama3-8B
Task		WIKI	C4	WIKI	C4	WIKI	C4	WIKI	C4	WIKI	C4
FP16	-	5.68	7.08	5.09	6.61	5.47	6.97	4.88	6.46	6.14	9.45
W2A16	RTN	1.1e5	1.3e5	6.8e4	5.6e4	3.8e4	4.8e4	4.2e4	1.2e3	4.2e4	5.6e4
	GPTQ	5.6e4	689.13	5.5e3	6.97	7.7e3	NAN	2.1e3	323.12	4.2e3	1.2e3
	OmniQuant	15.47	24.89	13.21	18.31	37.37	90.64	17.21	26.76	28.48	56.64
	ZeroQAT	9.65	13.55	9.09	11.87	17.18	28.36	8.67	13.68	9.14	12.89

Table 3: Weight-only and weight-activation quantization results of LLama-series models on two datasets: WikiText2 (WIKI), and C4.

LLama / Acc $\uparrow$	Method	PIQA	ARC-e	ARC-c	HellaSwag	Winogrande	Avg.
LLama-1-7B	FP16	77.47	72.38	41.46	73.00	67.07	65.26
	Spinquant	73.81	66.58	34.16	67.02	59.95	60.30
	FlatQuant	75.73	68.87	36.88	69.31	65.46	63.17
	ZeroQAT	76.98	69.32	36.19	70.86	64.87	63.64
LLama-2-7B	FP16	79.05	74.54	46.16	75.98	69.06	68.96
	Spinquant	75.30	65.32	39.42	71.45	63.46	62.99

Table 4: Weight-only and weight-activation quantization results of LLama models. This table reports the accuracy of 5 zero-shot tasks. Refer to Table E.1 for more results.

Figure 3: lightweight ZeroQAT for fine-tuning.

OPT / Acc $\uparrow$		OPT-2.7B				OPT-6.7B				OPT-13B
Task		SST-2	CB	SQuAD	DROP	SST-2	CB	SQuAD	DROP	SST-2	CB	SQuAD	DROP
Zero-shot		56.3	50.0	29.8	10.0	64.2	50.0	37.9	13.1	58.8	46.4	46.2	14.6
FP16 (ZO)	-	90.0	69.6	68.7	22.9	90.2	71.4	76.0	26.4	91.4	67.9	84.7	30.9
W2A16g128	RTN	44.4	44.6	0.0	0.0	59.2	50.0	0.0	0.0	53.5	50.0	0.0	0.0
	QuZO	47.2	46.3	0.0	0.0	60.6	55.3	0.0	0.0	59.6	51.8	0.0	0.0
	QLoRA	61.2	51.8	0.0	8.2	64.8	58.9	0.0	0.0	63.8	69.6	0.0	0.0

Table 5: Weight-only and weight-activation quantized fine-tuning results of OPT models. The table reports the accuracy or F1 score of the corresponding tasks. g128 indicates the group size is 128.

Method	#Bits	7B	13B
-	FP	67.0	69.3
QLoRA w/GPTQ	W2A16	31.8	32.4
QA-LoRA	W2A16	34.6	37.3
IR-QLoRA	W2A16	34.4	36.3
PEQA	W2A16	35.2	34.8
EfficientQAT	W2A16	49.1	52.1

Table 6: Averaged accuracy over 5 zero-shot datasets after fine-tuning. Results on MMLU and GSM8K are presented in Table E.2 and Table E.3 .

Method	OPT-1.3B		OPT-2.7B		OPT-6.7B		OPT-13B
Method	Memory	Time	Memory	Time	Memory	Time	Memory	Time
Quantized Pre-training (avg sequence length = 2048)
LLM-QAT (bsz=1)	28.8GB	1.00s	58.6GB	1.64s	$\sim$ 166GB	$\sim$ 5.0s	$\sim$ 337GB	$\sim$ 15.5s
FlatQuant (bsz=1)	7.2GB	1.12s	8.5GB	1.83s	13.9GB	3.04s	20.1GB	5.14s
ZeroQAT (bsz=1)	3.1GB	0.58s	6.1GB	0.98s	14.2GB	1.77s	26.6GB	3.12s
FlatQuant (bsz=4)	17.4GB	2.55s	21.5GB	4.03s	27.2GB	6.35s	33.8GB	11.81s

Table 7: Memory consumption and wallclock time per update during quantized pre-training and fine-tuning under the W2A16g128 setting. Since ZeroQAT only stores the weights in memory, its memory usage remains unaffected by batch size. Four and sixteen is the ideal batch size in FlatQuant and EfficientQAT, respectively.

Stage	Metrics	OPT-1.3B		OPT-2.7B		OPT-6.7B
Stage	Metrics	FP16	ZeroQAT	FP16	ZeroQAT	FP16	ZeroQAT
Fine-tuning	Latency	11.2s	7.8s	19.6s	12.3s	/	29.1s
	Weight memory	2.6GB	0.9GB	5.4GB	1.8GB	13.4GB	4.6GB
	Running memory	3.5GB	1.2GB	8.1GB	2.6GB	OOM	6.4GB
Inference	Token / s	10.9	15.4	7.58	11.0	3.13	4.76
Inference	Speed up	1.0 $\times$	1.41 $\times$	1.0 $\times$	1.45 $\times$	1.0 $\times$	1.52 $\times$

Table 8: Evaluation of memory consumption and speed on a OnePlus 12 smartphone under W4A4 quantization. Prompts of 384 tokens are used in inference, and OOM indicates out of memory.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Hyperparameters	Values
Quantized Pre-training	Batch size	4
	Iteration	10K
	Learning rate	{5e-7, 1e-8}
	Lr for smothing	5e-6
	Lr for clipping	1e-5
	Lr schedule	Linear Decay

Appendix

Table B.1: The hyperparameter for experiments. For ZeroQAT, we only show the setting of extra hyperparameters, and have the same setting in other common hyperparameters as MeZO ( Malladi et al., 2023 ) .

PPL $\downarrow$	LLama-1-7B		LLama-2-13B
Leanable Components	W4A4	W2A16	W4A4	W2A16
Smoothing + Clipping	6.17	9.65	5.27	8.67
W/O Smoothing	1.4e3	29.61	5.2e3	15.97
W/O Clipping	16.64	9.4e3	18.7	2.8e3
W/O Smoothing & Clipping	2.1e3	1.2e4	1.7e4	4.6e3

Appendix

Table C.1: Effect of each component. WikiText2 perplexity is reported in this table. W/O indicates removing the corresponding learnable components. Here, smoothing indicates scaling and transformation.

Method	Fine-tuning Memory	PTQ memory	SST-2	CB	SQuAD	DROP
FP ZO	14.2 GB	-	90.2	71.4	76.0	26.4
OmniQuant (ZO)	14.2 GB	4.4 GB	61.2	48.2	24.7	11.7
OmniQuant (FO)	98.6 GB	4.4 GB	58.7	55.3	31.8	13.5
ZeroQAT	3.7 GB	-	88.6	67.4	61.3	22.4

Appendix

Table C.2: Compare with the PTQ method with different fine-tuned models as starting points. Results of fine-tuning OPT-6.7B under W4A4 setting. ZO and FO indicate the starting fine-tuned checkpoint is from first-order and zeroth-order optimization, respectively.

PPL $\downarrow$	LLama2-7b		LLama2-13b
Method	W2A16	W4A4	W2A16	W4A4
ZeroQAT (LW)	41.05	19.34	21.97	15.45
ZeroQAT	29.61	12.95	15.95	10.41

Appendix

Table C.3: Effect of using lightweight ZeroQAT in quantized pre-training. LW indicates lightweight. Perplexity on Wikitext2 is reported.

Panel of Table 16 — Table C.4: Ablation study for selecting which layers to maintain full-precision and update in Quantized Fine-tuning. The highlighted line with a blue rectangle is the setting used in ZeroQAT. Attn_Q: attention Query layer; Attn_V: attention Value layer; Attn_K: attention Key layer; Attn_O: attention output projection; Dense: dense fully connected layer.

Table 17

Method	Bit	Qwen2.5-7B	Mistral-7B
-	FP16	6.67	5.54
RTN	W2A16	2.4e4	4.7e4
OmniQuant	W2A16	42.84	31.28
ZeroQAT	W2A16	18.37	14.67
OmniQuant	W4A4	15.32	13.74
FlatQuant	W4A4	7.23	5.78

Appendix

Table D.1: Results of perplexity on Wikitext2 on Qwen and Mistral.

OPT / PPL $\downarrow$		OPT-2.7B			OPT-6.7B			OPT-13B
Task		WIKI	PT	C4	WIKI	PT	C4	WIKI	PT	C4
FP16	-	12.47	15.13	13.16	10.86	13.09	11.74	10.13	12.34	11.20
W6A6	SmoothQuant	12.64	15.91	13.34	11.34	13.82	12.14	10.56	12.76	11.40
	RPTQ	13.19	16.37	14.04	11.19	13.98	12.08	11.19	13.98	12.08
	RPTQ*	12.71	15.53	13.33	10.96	13.24	11.86	10.96	13.24	11.86
	OmniQuant	12.62	15.32	13.29	10.96	13.20	11.81	10.21	12.47	11.17

Appendix

Table D.2: Weight-activation quantization results of OPT models on three datasets: WikiText2 (WIKI), Penn Treebank (PT), and C4. RPTQ* represents a variant that quantizes all activations except the softmax output.

LLama / Acc $\uparrow$	Method	PIQA	ARC-e	ARC-c	HellaSwag	Winogrande	Avg.
LLama-1-7B	FP16	77.47	72.38	41.46	73.00	67.07	65.26
	SmoothQuant	49.80	30.40	25.80	27.40	48.00	38.41
	OmniQuant	67.38	53.87	30.63	53.12	55.25	52.15
	Spinquant	73.81	66.58	34.16	67.02	59.95	60.30
	FlatQuant	75.73	68.87	36.88	69.31	65.46	63.17
	ZeroQAT	76.98	69.32	36.19	70.86	64.87	63.64

Appendix

Table E.1: Weight-only and weight-activation quantization results of LLama models. This table reports the accuracy of 5 zero-shot tasks.

LLama-1-7B (FP: 38.41%)	GPTQ	EfficientQAT	SmoothQuant	OmniQuant	ZeroQAT
W2A16	23.71%	24.74%	-	25.65%	32.55%
W4A4	-	-	24.55%	26.93%	34.73%

Appendix

Table E.2: Results of fine-tuning Llama-1-7B on MMLU benchmarks. 5-shot results are reported.

LLama-2-7B (FP: 27.3%)	SmoothQuant	OmniQuant	FlatQuant	ZeroQAT
W2A16	-%	6.3%	18.3%	20.7%
W4A4	0.6%	7.8%	18.8%	22.1%

Appendix

Table E.3: Results of fine-tuning Llama-2-7B on GSM8K benchmarks. 5-shot results are reported.

On-Device Language Model Fine-Tuning

See all 8 papers

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization
Jun 9, 2026Haoyu Wang, Xingyu Yu, Haiyan Zhao +2LLM Quantization Quantization-Aware Training
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.
Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM Quantization Quantization-Aware Training
ISTA · EPFL, ISTA
RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
Mar 18, 2026Arpit Singh Gautam, Saurabh JhaLLM Quantization Neural Network Quantization

Jun 9, 2026J/K move · Enter open · S save

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

Haoyu Wang, Xingyu Yu, Haiyan Zhao, Fengxiang Wang+1

Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.

LLM Quantization Quantization-Aware Training Vector Quantization Weight-Only Quantization Neural Network Quantization Data-Efficient Learning

Open paper AskPDF arXiv

Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%--10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.

Abstract

On-Device Language Model Fine-Tuning

Explore similar work

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference