Beyond 4 bits: Data-Centric Post-training Quantization and Recovery

Rhea Sanjay Sukthanker; Riccardo Grazzi; James Hensman; Pashmina Cameron

cs.AIOct 20, 2026

Beyond 4 bits: Data-Centric Post-training Quantization and Recovery

Authors: Rhea Sanjay Sukthanker, Riccardo Grazzi, James Hensman, Pashmina Cameron

Organizations: Applied Sciences Group (ASG), Microsoft · Microsoft Research

Abstract

Post-training quantization (PTQ) compresses the weights of a trained model with limited compute and data. Down to 4 bits it is near-lossless, but at 2 bits scalar PTQ configurations fall to chance-level accuracy. We study three choices usually treated as implementation details that matter little at 4 bits but decide 2-bit quality: the quantization representation, the calibration data, and the recovery fine-tuning data, across six dense and mixture-of experts models (ranging from $3.8$ to $32$ B parameters). Representation is the largest factor and vector codebooks restore usable models where scalar PTQ pipelines collapse. Most of the remaining loss in accuracy is on generative tasks, and the amount of performance drop is model-dependent. Furthermore, at a short recovery budget, calibration and recovery data are coupled. The best recovery data depends on the calibration data, so the two must be chosen jointly. The resulting recipe involves a vector codebook representation, a small task-aligned calibration mixture, and an inexpensive codebook-only recovery stage. Including codebooks, it uses about $2.3$ effective bits per weight and retains $89$ to $97\%$ of full-precision question-answering (QA) accuracy where scalar PTQ is at chance, and, including recovery, uses $1.4$ to $30\times$ less compute than the strongest quantization-aware-training (QAT) baseline.

Figures & tables

Panel of Figure 1 — Figure 1: Phi-4-mini at 2 bits. (a) Scalar PTQ (SpinQuant, Liu et al., 2025b ) falls to the random floor on question-answering (QA) tasks as bitwidth decreases; a 2-bit vector representation (YAQA, Tseng et al., 2026 ) stays higher. (b) Generation (AvgGen, the GSM8K/HumanEval mean) versus compute: trained scalar baselines (BitDistiller, Du et al., 2024 ; GSQ, Dadgarnia et al., 2026 ) trail no-recovery vector PTQ, and short codebook-only recovery improves generation.

Figure 2: Calibration and recovery data are not separable at 2 bits. 2-bit YAQA bases are calibrated on web or math text (OpenR1-Math), then unrecovered (hollow markers) or given short codebook-only recovery budget on web or math text (

3{,}200

examples). The shared

y

-axis is AvgGen as a percentage of each model’s bf16 AvgGen; labels give absolute AvgGen. Means

\pm

std over three recovery seeds (P-bf16). Circles mark the best cell. Boxes give the interaction contrast and its

95\%

hierarchical-bootstrap interval. Detailed results are in Table 1 .

Figure 3: The interaction stays negative across robustness checks. Interaction contrast with

95\%

intervals; left of zero, the best recovery data flips with the calibration data. Circles: bootstrap over recovery seeds and test items, on the main benchmarks or on held-out tasks (HumanEval+ reuses HumanEval prompts). Squares: intervals over three independently re-quantized bases per arm (Welch) or three disjoint recovery-data samples (

t

Table 4: Cross-paradigm comparison at 2 bits (P-fp16; OLMoE row P-bf16). Best AvgQA ( 4(a) ) and AvgGen ( 4(b) ) per model. VPTQ and YAQA select the best recipe per metric from their recipe grids. GSQ: layer-wise scalar PTQ; BitDistiller: scalar QAT; VPTQ, YAQA: vector PTQ. Shaded bold marks the best per model. N Native (unrotated) VPTQ, P-bf16, where it beats rotated VPTQ (Appendix Table 27 ). S Experts share one codebook per layer and projection type (

2.09

2.14

effective bits

	Phi-4-mini		Llama-3.1		Qwen3-MoE
Recipe	QA	Gen	QA	Gen	QA	Gen
Web	63.3	12.2	65.1	9.6	66.0	54.0
Math	58.5	37.4	62.1	26.6	64.5	61.8
Code	55.9	18.7	60.8	22.6	62.8	48.9
Math+web .5/.5	62.6	24.7	65.7	17.8	66.7	56.5
Web+math .7/.3	63.0	27.7	65.5	15.1	65.8	56.2

Table 5: Calibration mainly changes generation (2-bit YAQA, P-fp16; bold: best;

\Delta

: range over the full sweep).

Figure 4: 2-bit retention of QA and chat-template HumanEval by model. Models are ordered by total parameters (hatched: MoE). Best no-recovery VPTQ/YAQA checkpoint per metric on our recipe grid within the

2.5

-bit budget (YAQA on the MoEs). The dotted Llama-3.1-8B bar is completion-style HumanEval (base model, no chat template); completion-prompt AvgGen is in Table 33 .

Figure 5: Recovery gains saturate early. Gain over the unrecovered base as the recovery budget grows (Phi-4-mini, math-calibrated 2-bit YAQA, codebook-only recovery on web data; Table 22 ). Generation gains

6.3

points by

3{,}200

examples and then plateaus; QA rises slowly.

Precision	Accuracy	Mean CoT	Cap-hit	Markers/1k	Loop ( $4$ -gram)	R-then-A
bf16	0.675	4928	0.25	15.5	0.39	0.69
4-bit	0.650	5265	0.25	12.4	0.41	0.50
2-bit	0.450	5900	0.45	23.7	0.67	0.41

Table 6: Overthinking under 2-bit quantization (Qwen3-30B-A3B, MATH-500, greedy). 2-bit amplifies hesitation and looping and moderately lowers accuracy without collapsing. “Cap-hit” is the length-cap hit rate. “R-then-A” is the reached-then-abandoned rate among wrong answers. We use

40

problems, and the 2-bit model is expert-shared VPTQ (

2.09

bits).

Rep. penalty	Marker penalty	Accuracy	Mean CoT	Cap-hit	Markers/1k	Loop ( $4$ -gram)
1.0	0	0.425	5884	0.475	23.0	0.66
1.0	2	0.475	5156	0.375	10.2	0.62
1.0	6	0.500	4726	0.350	2.2	0.59
1.2	0	0.575	5052	0.300	15.0	0.40
1.3	0	0.625	4799	0.250	13.0	0.30

Table 7: Decoding penalties on the 2-bit model (Qwen3-30B-A3B, MATH-500, 2-bit). A repetition penalty reduces looping and chain-of-thought length and improves accuracy. The same penalty also helps bf16 .

Decision	Recommendation
Representation	vector codebook (YAQA).
Calibration	task-aligned Hessian mixture
Recovery	short codebook-only KD, or none for base models.
Recovery data	choose jointly with calibration.
KD settings	defaults ( $\tau{=}1$ ).
Teacher	4-bit (matches bf16 ) or none.

Table 9: 2-bit decision guide (Section 4 ; KD and teacher rows: Phi-4-mini).

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Calibration	AvgQA	GSM8K	HumanEval	HumanEval c
Qwen3-MoE	web	64.8	69.5	25.6	64.6 (70%)
	instruction	64.8	67.5	19.5	67.7 (73%)
	code	63.8	66.9	39.0	76.2 (82%)
	math	59.8	72.2	17.7	59.8 (64%)
	all-4 mixture	63.7	75.1	18.3	76.2 (82%)
	web/math $0.7/0.3$	66.0	76.3	22.6	67.1 (72%)

Appendix

Table 10: MoE VPTQ with expert-shared codebooks. One codebook per decoder layer and projection type, shared by all experts (

2.09

2.14

effective bits; P-bf16). HumanEval c uses the chat template. Parentheses give retention relative to bf16 under the same protocol. This protocol is bfloat16 for Qwen3-MoE ( bf16 score 92.7) and float16 for OLMoE ( bf16 score 34.8).

		Fixed	Learned, $\eta{=}1.5$		Learned, $\eta{=}0.25$
Model	Bits	AvgQA	AvgQA	AvgGen	AvgQA	AvgGen
Phi-4-mini	4	69.2	68.4–69.3	63.2–68.9	68.2–69.7	62.9–70.4
Phi-4-mini	2	29.3	28.8–29.3	0.0	28.7–29.4	0.0
Llama-3.1-8B	4	71.8	70.4–71.6	36.5–40.5	70.9–71.9	37.1–41.8
Llama-3.1-8B	2	29.8	28.7–29.4	0.0	29.3–29.7	0.0

Appendix

Table 12: Learning SpinQuant’s rotations does not move the 2-bit floor. AvgQA (seven-task mean) and AvgGen (GSM8K/HumanEval mean) for Cayley-optimized

R_{1}

R_{2}

(

100

steps, learning rate

\eta

), shown as ranges over web, math, mixed, and code calibration. Fixed is the best-domain AvgQA of Hadamard-initialized SpinQuant used elsewhere (Table 37 ). Learned rows use bfloat16.

Method (bits/weight)	AvgQA	AvgGen	AvgQA	AvgGen
	web		math
VPTQ ( $2.00$ / $2.33$ )	$51.5$	$1.2$	$41.0$	$6.5$
VPTQ, model-wide ( $2.25$ / $2.26$ )	$58.4$	$4.3$	$47.5$	$11.8$
YAQA ( $2.25$ / $2.27$ )	$\mathbf{63.3}$	$\mathbf{12.2}$	$\mathbf{58.5}$	$\mathbf{37.4}$

Appendix

Table 13: Vector-quantizer rate control (Phi-4-mini). Rows give index / effective bits per weight. Model-wide denotes one

2^{18}

-entry codebook shared by all linear layers, matching YAQA’s effective bits (P-bf16). Its HumanEval is

0

on both recipes. AvgQA is the seven-task mean; AvgGen averages GSM8K and HumanEval (pass@1). Bold marks best per column.

Model	Method	Web	Instruct	Math	Code	Mixture
Phi-4-mini	VPTQ	51.5	53.9	41.0	44.4	47.8
Phi-4-mini	YAQA	63.3	56.8	58.5	55.9	58.8
Llama-3.1-8B	VPTQ	57.6	59.8	45.7	53.4	54.5
Llama-3.1-8B	YAQA	65.1	63.9	62.1	60.8	64.3
Qwen3-30B-A3B	VPTQ S	64.8	64.8	59.8	63.8	63.7
Qwen3-30B-A3B	YAQA	66.0	64.9	64.5	62.8	66.0

Appendix

Table 14: AvgQA by vector quantizer and calibration domain. Bold marks the better method within each model/domain. S Expert-shared codebooks (

2.09

effective bits, P-bf16).

		Hessian mixture weight				Phi-4-mini		Llama-3.1-8B		Qwen3-30B-A3B
Group	Recipe	Code	Web	Math	Instr	QA	Gen	QA	Gen	QA	Gen
Single	Web		1			63.3	12.2	65.1	9.6	66.0	54.0
	Instruct				1	56.8	10.7	63.9	11.8	64.9	49.8
	Math			1		58.5	37.4	62.1	26.6	64.5	61.8
	Code	1				55.9	18.7	60.8	22.6	62.8	48.9
Standard mix	math+web		0.5	0.5		62.6	24.7	65.7	17.8	66.7	56.5

Appendix

Table 16: Task-complete 2-bit YAQA calibration sweep. QA and generation means from complete per-task evaluations. Blank weight entries mean absent sources. Dashes mark cells with only aggregate AvgQA/HEval.

\Delta_{\rm complete}

excludes dashed result cells.

	4-bit scalar (QA / Gen spread)			2-bit YAQA
Model	SpinQuant	QuaRot	KronQ	(QA / Gen)
Phi-4-mini	1.4 / 2.9	0.5 / 6.5	1.3 / 5.0	7.5 / 26.7
Llama-3.1-8B	1.6 / 4.4	1.0 / 2.6	0.6 / 7.3	4.9 / 17.0

Appendix

Table 17: Spread across calibration domains, 4-bit scalar versus 2-bit vector. Max minus min over five calibration domains (4-bit) or the full recipe sweep (2-bit YAQA). AvgQA is the seven-task mean. AvgGen is the GSM8K/HumanEval mean.

Source	HumanEval	MATH-500
web (FineWeb-Edu)	0.0	0.0
math (OpenR1-Math-220k)	0.6	4.8
code (Evol-Instruct-Code-80k)	5.5	1.0
instruction (Alpaca-cleaned)	2.4	0.6

Appendix

Table 18: Benchmark overlap (

13

-gram) with the first

20{,}000

rows of each calibration/recovery source. Entries give the percentage of evaluation items sharing at least one

13

-gram with the source text.

Calibration	Recovery	Core6	GSM8K	HumanEval	AvgGen	IFEval
Web	Web	65.7	23.5	14.0	18.8	28.1
Web	Math	65.0	35.1	14.0	24.6	26.6
Math	Web	61.6	56.3	31.1	43.7	32.0
Math	Math	60.1	60.1	25.0	42.6	27.5

Appendix

Table 20: Calibration and recovery interact at matched budget. AvgGen is the mean of GSM8K and HumanEval. The final column is held out from recipe selection. Core6 averages 6 QA tasks (excluding MMLU)

Phi-4-mini (three recovery seeds)
Calib.	Recovery	AvgQA	GSM8K	HEval	AvgGen	IFEval
bf16		70.8	81.7	62.8	72.3	55.8
web	none	63.1	15.5	6.7	11.1	29.0
web	web	$63.3{\pm}0.1$	$24.2{\pm}0.3$	$13.6{\pm}0.9$	$18.9{\pm}0.5$	$29.8{\pm}1.3$
web	math	$62.8{\pm}0.1$	$33.8{\pm}0.7$	$14.2{\pm}0.9$	$24.0{\pm}0.4$	$23.4{\pm}1.6$
math	none	58.3	53.2	20.1	36.7	31.2

Appendix

Table 21: Single-protocol bfloat16 evaluation of the

2\times 2

design. Mean

\pm

sample std over three recovery seeds per cell. Every row uses bfloat16 weights and activations and the seven-task AvgQA (with MMLU). none rows are the no-recovery 2-bit YAQA bases. The bf16 row averages two independent evaluations. See Table 2 for interaction statistics. “Math” is OpenR1-Math.

Examples	Tokens	PFLOP	AvgQA	AvgGen
$0^{\dagger}$	0	0	58.5	37.4
$200$	0.2M	6	58.8	38.5
$800$	0.8M	25	59.1	41.8
$3{,}200$	3.3M	101	60.3	43.7
$12{,}800$	13.1M	403	61.0	44.2

Appendix

Table 22: Recovery accuracy versus data budget (Phi-4-mini, 2-bit YAQA, math-calibrated base, FineWeb-Edu codebook-only recovery, seed

42

, P-fp16). Tokens are examples

\times 1{,}024

. PFLOP assumes

8N

per token. † No-recovery base (Table 8 ).

Calib.	Recovery	AvgQA	GSM8K	HEval	AvgGen	IFEval
bf16		70.9	87.5	61.6	74.5	25.0
web	none	63.3	53.0	7.9	30.5	15.2
web	web	$65.1{\pm}0.1$	$56.2{\pm}0.4$	$14.6{\pm}0.0$	$35.4{\pm}0.2$	$16.2{\pm}0.6$
web	math	$64.8{\pm}0.0$	$58.2{\pm}0.5$	$19.9{\pm}0.7$	$39.0{\pm}0.2$	$16.8{\pm}0.9$
math	none	60.7	77.9	20.1	49.0	16.5
math	web	$63.9{\pm}0.1$	$78.8{\pm}0.1$	$34.6{\pm}0.4$	$56.7{\pm}0.1$	$16.8{\pm}0.9$

Appendix

Table 23: Pre-registered Qwen3-8B

2\times 2

design (2-bit YAQA, P-bf16; mean

\pm

std over three recovery seeds). none denotes the no-recovery base.

Benchmark	Phi-4-mini	Llama-3.1-8B	Qwen3-8B
MBPP	$\mathbf{-5.6}$ $[-9.7,-1.5]$	$+0.9$ $[-3.0,4.5]$	$\mathbf{-6.8}$ $[-10.8,-2.7]$
HumanEval+	$\mathbf{-12.2}$ $[-19.7,-4.7]$	$\mathbf{-3.7}$ $[-7.1,-0.4]$	$\mathbf{-13.0}$ $[-20.5,-6.1]$
MATH-500	$-2.7$ $[-6.7,1.1]$	$-2.0$ $[-4.8,0.8]$	$+1.7$ $[-2.5,5.9]$
MBPP+MATH-500	$\mathbf{-4.2}$ $[-7.2,-1.3]$	$-0.6$ $[-2.9,1.8]$	$-2.6$ $[-5.6,0.5]$
All three	$\mathbf{-6.8}$ $[-10.1,-3.6]$	$-1.6$ $[-3.6,0.3]$	$\mathbf{-6.0}$ $[-9.2,-3.0]$

Appendix

Table 24: Held-out interaction contrasts by benchmark (

95\%

hierarchical bootstrap). HumanEval+ shares HumanEval’s prompts; MBPP and MATH-500 are independent. Bold marks intervals that exclude zero.

Sample	web $\to$ web	web $\to$ math	math $\to$ web	math $\to$ math	Interaction
original	24.2 / 13.6	33.8 / 14.2	56.9 / 34.6	60.1 / 24.6	$-8.5$
$+20$ k rows	23.2 / 13.4	33.6 / 12.2	56.3 / 35.4	60.1 / 25.0	$-7.9$
$+40$ k rows	25.1 / 14.6	34.4 / 14.6	58.4 / 31.7	61.5 / 27.4	$-5.3$

Appendix

Table 25: Interaction across disjoint recovery-data samples (Phi-4-mini, original bases, P-bf16; GSM8K / HumanEval per cell, interaction on AvgGen). The original row is the

3{,}200

-example sample of Table 1 (mean over three seeds). The

20

k and

40

k rows are samples drawn through the same shuffle buffer after skipping

20

40

k source rows (seed

42

). They share no rows with the original.

Calib.	Base	GSM8K	HEval	AvgGen	GSM8K	HEval	AvgGen	Effect
		web recovery			math recovery
web	original	24.2	13.6	18.9	33.8	14.2	24.0	$+5.1$
web	replicate 1	29.9	8.5	19.2	42.3	6.7	24.5	$+5.3$
web	replicate 2	25.7	10.4	18.0	39.6	17.1	28.3	$+10.3$
math	original	56.9	34.6	45.7	60.1	24.6	42.3	$-3.4$
math	replicate 1	57.3	33.5	45.4	61.0	27.4	44.2	$-1.2$

Appendix

Table 26: Interaction across independently quantized bases (Phi-4-mini, 2-bit YAQA, P-bf16). Each row is one quantized base. The original row is the base of Table 1 (mean over three recovery seeds), and replicates use seed

42

. The recovery-data effect is math minus web recovery (AvgGen).

Model	Method	Web	Instruct	Math	Code	Mixed
Phi-4-mini	VPTQ-rotated	51.4 / 1.5	53.7 / 7.5	40.9 / 6.4	44.6 / 11.4	47.6 / 7.8
	VPTQ-native	57.4 / 7.6	57.6 / 12.9	47.8 / 13.2	51.0 / 29.5	56.2 / 26.0
	YAQA-rotated	63.1 / 11.1	56.8 / 10.9	58.3 / 36.7	55.8 / 19.2	58.8 † / 19.9 †
Llama-3.1-8B	VPTQ-rotated	57.5 / 2.6	59.6 / 10.3	45.8 / 8.5	53.2 / 15.8	54.4 / 13.2
	VPTQ-native	63.3 / 5.5	62.4 / 11.4	57.8 / 16.0	57.6 / 19.7	62.0 / 17.1
	YAQA-rotated	65.1 / 9.5	63.7 / 12.3	62.0 / 27.4	60.7 / 22.9	64.2 / 18.5

Appendix

Table 27: Native versus rotated VPTQ, and YAQA on the same calibration domain (AvgQA / AvgGen, P-bf16; † P-fp16). Effective bits are

2.33

2.27

for VPTQ (Phi-4-mini / Llama-3.1-8B) and

2.27

2.26

for YAQA.

Checkpoint	fp16	8-bit	4-bit	2-bit
YAQA math+web	68.5	68.5	68.5	68.6
YAQA MWI442	68.2	68.1	68.0	68.0

Appendix

Table 29: Qwen3-MoE router-precision ablation. Expert weights remain at their vector-quantized 2-bit setting; only router precision changes. The checkpoint labels are those in the router-ablation runs, scored by their QA average. Compare only within-row precision deltas.

Model	Type	Params (B)	Completion	HumanEval
Phi-4-mini	dense	$3.8$	$51\%$	$19$ – $45\%$
OLMoE-1B-7B	MoE	$6.9$ / $1.3$	$64\%$	$49\%$
Llama-3.1-8B	dense	$8.0$	$62\%$	$19$ – $73\%^{\dagger}$
Qwen3-8B	dense	$8.2$	$66\%^{\lx@sectionsign}$	$42$ – $45\%$
Qwen3-32B	dense	$32.8$	$100\%^{\ast}$	$68\%^{\ast}$
Qwen3-30B-A3B	MoE	$30.5$ / $3.3$	$101\%$	$81\%$

Appendix

Table 33: Generation retention by model size and sparsity. The completion-style column gives the best 2-bit AvgGen relative to bf16 . The HumanEval column gives the range of 2-bit HumanEval retention under the chat template. Parameters are total / active (billions). ∗ Single recipe (VPTQ, web). † Llama-3.1-8B is a base model without a chat template, so HumanEval uses completion-style bfloat16 evaluation. § Math-calibrated base under P-bf16. The separate math+web checkpoint uses P-fp16.

Configuration	AvgQA	GSM8K	HEval	PTB ppl
bf16 ceiling	68.3	33.7	31.1	14.7
VPTQ S web	62.7	11.8	7.9	22.2
VPTQ S web+math	63.0	18.7	11.6	21.7
YAQA web+math	64.0	26.0	15.2	19.3

Appendix

Table 34: OLMoE-1B-7B-Instruct at 2 bits. The model has 64 non-fused experts. S Expert-shared codebooks (

2.14

effective bits, P-bf16).

Model	Configuration	IFEval
Phi-4-mini-instruct	bf16	56.6
	2-bit YAQA (math+web)	37.2
	+ codebook recovery	36.0
Llama-3.1-8B	bf16	8.9
	2-bit YAQA	7.4
Qwen3-30B-A3B	bf16	28.3

Appendix

Table 35: IFEval prompt-strict accuracy (completion prompts).

Phi-4-mini (YAQA)
Calib.	Recovery	GSM8K	HumanEval	IFEval
bf16		81.4	73.8	69.7
web	none	13.0	15.2	43.1
web	web	24.4	18.9	43.8
web	math	36.5	21.3	44.2
math	none	55.0	34.1	40.3

Appendix

Table 36: Recovered checkpoints under the chat template (bfloat16; GSM8K

5

-shot, humaneval_instruct pass@1, IFEval prompt-strict).

Figure 6: Calibration composition primarily moves generation, not QA. Across 2-bit YAQA calibration recipes, QA varies modestly while generation moves more. The full sweep is in Table 5 .

Figure 7: In recovery, data composition dominates; KD hyperparameters move it little across the tested grid. Loss type, temperature, top-

k

, and teacher precision have smaller effects, especially for generation.

Figure 8: Which capabilities survive 2-bit vector PTQ. Per-task retention of one representative balanced 2-bit vector checkpoint per model relative to bf16 . QA and next-word prediction are largely retained, while generation (GSM8K, HumanEval) is fragile.

Figure 9: QA and generation are distinct axes. AvgQA versus AvgGen for every method and model at 2 bits. Points are the per-metric best entries of Table 4 (QA and generation may come from different recipes). Vector PTQ recovers QA; generation stays near bf16 only for Qwen3-MoE. Scalar 2-bit PTQ collapses on both axes.

Figure 10: The tested vector pipelines improve the nominal 2-bit quality–rate trade-off. Effective bits-per-weight versus AvgQA and AvgGen on Phi-4-mini. At

\sim 2

bits/weight, a vector codebook (YAQA and VPTQ) sits far above the scalar 2-bit floor but below scalar 4-bit quality.

Model	Method	bf16	4-bit	2-bit
Phi-4-mini	SpinQuant	71.0	69.2	29.3
Phi-4-mini	QuaRot	71.0	69.3	29.8
Llama-3.1-8B	SpinQuant	73.1	71.8	29.8
Llama-3.1-8B	QuaRot	73.1	71.7	29.3
Qwen3-MoE	SpinQuant	72.6	70.8	29.4
Qwen3-MoE	QuaRot	72.6	70.3	29.3

Appendix

Table 37: The 2-bit scalar cliff is universal across rotation methods. Best-calibration-domain AvgQA at 16, 4, and 2 bits for two scalar PTQ methods (best calibration domain, so entries can exceed the web-only values of Table 3 ). All methods stay near bf16 at 4 bits and collapse at 2 bits. QuaRot is applied through expert-wise rotation on Qwen3-MoE.

Panel of Figure 11 — Figure 11: The 2-bit scalar cliff on the other models (analogs of Figure 1 (a)). Scalar PTQ (SpinQuant) AvgQA falls from near-lossless at 4 bits to the random-accuracy floor at 2 bits, while a 2-bit vector codebook (YAQA, web calibration) recovers much of the gap. AvgQA is the seven-task mean.

Figure 12: Generation cliff on Phi-4-mini. Scalar SpinQuant generation collapses by 3-bit and vanishes at 2-bit. A vector codebook restores part of the gap. AvgGen averages GSM8K and HumanEval.

Panel of Figure 13 — Figure 13: Llama-3.1-8B accuracy and generation vs. training compute (analog of Figure 1 (b)). Scalar $=$ crimson, vector $=$ teal. The $x$ -axis is training compute (log scale). PTQ+RFT is plotted at full-pipeline cost (YAQA plus recovery; the math+web long recovery of Table 8 , P-bf16). VPTQ QA is native VPTQ, and bf16 references are P-fp16.

Panel of Figure 14 — Figure 14: Qwen3-30B-A3B (MoE) accuracy and generation vs. training compute (analog of Figure 1 (b)). The MoE stays near its under-measured completion-prompt reference. Under the chat template it retains more HumanEval than the smaller dense models. VPTQ uses expert-shared codebooks ( $2.09$ effective bits). PTQ+RFT is plotted at full-pipeline cost ( $15+32=47$ PFLOP).

Figure 15: A repetition penalty helps bf16 and 2-bit alike (Qwen3-30B-A3B,

40

MATH-500 level-4/5 problems, greedy, same penalty for both models; the 2-bit model is expert-shared VPTQ). The penalty improves both precisions and lowers the

4

-gram loop score for both. The matched-decoding gap remains (

0.15

1.3

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 386 papers

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization
Oct 8, 2026Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu +6LLM Quantization Post-Training Quantization
University of Science and Technology of China · University College London · Shanghai University of Finance and Economics +5
Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
Jun 8, 2026Hanyang Li, Jianhao Ma, Ying CuiNeural Network Optimization Straight-Through Estimator
Department of IEOR, University of California, Berkeley · Department of Statistics and Data Science, University of Pennsylvania
ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization
Aug 7, 2026Yongge Ma, Guoan Wang, Feiyu Wang +5LLM Quantization Post-Training Quantization
School of Computer Science, Peking University · School of Software and Microelectronics, Peking University · School of Physics, Peking University +2

Oct 8, 2026J/K move · Enter open · S save

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu, Baihua He+5

University of Science and Technology of China · University College London · Shanghai University of Finance and Economics+5

LLM Quantization Post-Training Quantization Weight-Only Quantization Quantization Robustness

Open paper AskPDF arXiv

Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model performance on new tasks. In fact, we find that lower reconstruction loss can even degrade model performance on the same calibration data. Our analysis further shows that weights with lower reconstruction loss on calibration data can have higher loss than other weights when the distribution of input activations changes. Motivated by these observations and our analysis, we propose Distributionally Robust Quantization (DRQ), a post-hoc refinement process that minimizes worst-case reconstruction loss over a constrained set of input activation distributions. DRQ refines the integer codes representing quantized weights within the existing quantization grid, keeping quantization parameters and inference operators unchanged. Extensive experiments show that DRQ improves models quantized by six representative PTQ methods, including AWQ, GPTQ, and ParoQuant, and delivers gains across both dense and mixture-of-experts large language models. These results establish DRQ as a general post-hoc refinement framework for weight-only PTQ, achieving better downstream performance without adding inference overhead.

Abstract

LLM Quantization

Explore similar work

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin

ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization