FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization

Yujie Chen; Weiyan Zheng; Yiwu Yao; Tianchi Hu; Bo Wang; Jiyu Guo; Wenzhan Li; Jingcai Guo

cs.CLOct 10, 2026

FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization

Authors: Yujie Chen, Weiyan Zheng, Yiwu Yao, Tianchi Hu, Bo Wang, Jiyu Guo, Wenzhan Li, Jingcai Guo,

Organizations: The Hong Kong Polytechnic University · Huawei Technologies Ltd. · The University of Hong Kong · Sun Yat-sen University ·

Abstract

Ultra-low-bit scalar quantization offers a regular arithmetic structure for efficient deployment of large language models, but preserving predictive behavior at extreme precision remains challenging: with only a few reconstruction levels, accurate weight representation must translate into consistent predictions from a small calibration set. We propose FidelityQ, a scalar quantization framework that preserves predictive fidelity at weight precisions down to 2 bits. For accurate quantization, we introduce Information-Aligned Analytic Coding (IAC), which shapes each weight row toward a centered symmetric source via randomized Hadamard mixing and fits a mirrored four-level minimum-distortion quantizer to its empirical weights. Every level set in this family decomposes exactly into two sign-valued bases with associated scales, combining source-adaptive reconstruction with lookup-free decoding compatible with scaled integer arithmetic. Our analysis further shows that residual predictive discrepancies are heavy-tailed and concentrate on tokens the teacher predicts confidently, motivating Knowledge-Intensive Calibration (KIC) for consistent quantization. KIC ranks sequences by the tail of token-wise teacher-student divergence, allocates a limited calibration budget to the highest-scoring samples, and restores predictive agreement by updating only quantization levels and scales under teacher supervision, keeping pretrained weights frozen. Experiments on the LLaMA and Qwen families demonstrate the effectiveness of FidelityQ in weight-only and joint quantization settings. On LLaMA-2-7B, FidelityQ achieves a WikiText2 perplexity of 6.62 at W2A16 and improves the prior state-of-the-art W2A4KV4 result from 8.31 to 6.96, while reducing linear-layer weight memory by 8.0x relative to FP16.

Figures & tables

Figure 1: FidelityQ against scalar (SQ) and codebook (VQ) quantization at two bits. (a) FidelityQ improves the accuracy-efficiency tradeoff: no other scalar baseline shown reaches the codebook methods’ WikiText2 perplexity, and FidelityQ reaches it at scalar latency. ∗ is 2.125 bits per weight. (b) The same representation is smaller and faster: on LLaMA-2-7B at batch size 1, its linear-layer weights take 8.0

\times

less memory than FP16 and decode 3.4

\times

faster than QuIP# (Appendix D ).

Figure 2: Overview of FidelityQ. The top pipeline compares a frozen full-precision teacher with the IAC-quantized model, after which KIC selects calibration data and updates only the reconstruction levels. IAC shapes the source with a randomized Hadamard transform, places four symmetric nonzero levels at minimum distortion, and decodes them exactly as two sign planes with row-wise scales. KIC ranks sequences by the tail mean of token-wise divergence, keeps the top-

k

complete sequences, and refines the levels using the teacher distribution and corpus targets.

Panel of Figure 3 — Figure 3: IAC source shaping and adaptive level placement on LLaMA-2-7B at W2. (a) Rotation turns a heavy-tailed row into an approximately symmetric marginal, and the fitted levels improve SQNR over the group-max grid. (b) The reduction in excess kurtosis holds across input-rotated projections. (c) With one block quantized, the fitted levels lie in a low-cross-entropy region of the model-level loss surface. The upper loss range is compressed for visibility.

Figure 4: The damage KIC targets is concentrated, and KIC improves the tokens that carry it. On LLaMA-2-7B at W2A4KV4, (a) the worst decile of tokens carries 45% of the total divergence. On Qwen3-8B-Base at W2A16, against a random draw of the same size on a slice ranked before either arm is trained, (b) the tail perplexity gain grows from 0.08% over the corpus to 17.5% on the worst 1%, and (c) on that slice, top-1 agreement and repaired errors rise by 3.13 and 3.10 points, where tail perplexity is 79.8 against 96.7.

Method	W-A-KV	Perplexity $\downarrow$		HellaS	PIQA	WinoG	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Method	W-A-KV	Wiki2	C4	HellaS	PIQA	WinoG	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
LLaMA-2-7B
FP16	16-16-16	5.47	6.97	76.16	78.78	69.46	44.0	66.67	68.66	68.81	73.65	45.94
AQLM ∗	2-16-16	7.24	8.96	66.74	74.10	62.51	36.6	58.04	60.08	56.26	67.34	33.03
[1.5pt/1.5pt] QuaRot	2-16-16	24.52	39.39	33.30	58.98	52.17	28.2	42.45	41.50	15.72	17.04	25.87
DB-LLM	2-16-16	8.89	19.85	51.99	62.13	55.96	32.6	47.80	48.16	32.19	45.47	24.95

Table 1: FidelityQ achieves the lowest WikiText2 perplexity among the compared scalar methods across model families, under weight-only and joint weight, activation, and KV-cache quantization. The 7-task average covers ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande, BoolQ, and OpenBookQA, and the 5-task average the first five of these; complete task results are in Appendix E .

Variant	W-A-KV	Perplexity $\downarrow$		ARC		HellaS	PIQA	WinoG	BoolQ	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Variant	W-A-KV	Wiki2	C4	Easy	Chg	HellaS	PIQA	WinoG	BoolQ	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
FP16	16-16-16	5.47	6.97	73.86	45.05	76.16	78.78	69.46	79.36	44.0	66.67	68.66	68.81	73.65	45.94
Base	2-16-16	9.56	18.43	49.24	26.71	44.50	66.16	57.93	62.29	29.2	48.00	48.91	24.45	32.37	26.70
Base + IAC	2-16-16	7.78	12.53	51.43	28.84	52.98	67.03	61.33	62.35	34.2	51.17	52.32	39.14	51.64	28.73
Base + KIC	2-16-16	9.56	18.42	49.37	26.79	44.45	66.16	57.46	62.29	29.4	47.99	48.85	24.49	32.52	26.73
Base + IAC + KIC	2-16-16	7.11	9.74	60.19	36.35	65.70	74.05	63.93	68.69	37.4	58.04	60.04	56.90	67.16	29.54

Table 2: Component ablation on LLaMA-2-7B. Base uses rotation, GPTAQ, and a uniform grid without finetuning. Rows add IAC, KIC, or both; Base + IAC + KIC is FidelityQ. All rows use 512 WikiText2 calibration sequences. Bold marks the best quantized result per precision configuration.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

		$\mathrm{CV}(\bm{d})$		$\mathrm{med}_{r}\,\|g_{r}\|$
Proj.	$n$	unrotated	rotated	unrot. ( $\times 10^{-2}$ )	rot. ( $\times 10^{-4}$ )
q	4096	0.205 [0.077, 0.278]	0.015 [0.012, 0.023]	0.95 [0.44, 67.75]	3.22 [2.59, 8.15]
k	4096	0.205 [0.077, 0.278]	0.015 [0.012, 0.023]	2.62 [0.50, 64.68]	3.25 [2.67, 7.12]
v	4096	0.205 [0.077, 0.278]	0.015 [0.012, 0.023]	0.36 [0.11, 27.64]	3.20 [2.65, 4.98]
o	4096	0.782 [0.505, 2.717]	0.017 [0.006, 0.022]	4.89 [1.94, 94.23]	3.57 [1.53, 4.71]
gate	4096	0.207 [0.078, 0.273]	0.014 [0.012, 0.020]	0.57 [0.12, 7.01]	3.10 [2.69, 4.71]

Appendix

Table 3: Sequential sensitivity spread and weighted-to-unweighted residual gaps on LLaMA-2-7B. Brackets give the range across blocks.

Setting	Value
Optimizer	Adam
Learning rate	$5\times 10^{-6}$ , constant
Epochs / batch size	3 / one sequence, no accumulation
Sequence length	2048 tokens
Loss	Hard CE $+$ $0.1\times$ soft-target CE
Teacher support / temperature	Top-512 tokens / $T=1$

Appendix

Table 4: KIC training settings. Only the two reconstruction magnitudes per row are trainable.

Variant	W-A-KV	Perplexity $\downarrow$		ARC		HellaS	PIQA	WinoG	BoolQ	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Variant	W-A-KV	Wiki2	C4	Easy	Chg	HellaS	PIQA	WinoG	BoolQ	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
Base + IAC	2-16-16	7.78	12.53	51.43	28.84	52.98	67.03	61.33	62.35	34.2	51.17	52.32	39.14	51.64	28.73
Clipped uniform (no KIC)	2-16-16	7.89	12.87	49.71	29.01	51.62	67.08	62.12	62.32	33.6	50.78	51.91	39.39	51.39	28.67
Base + IAC + KIC	2-16-16	7.11	9.74	60.19	36.35	65.70	74.05	63.93	68.69	37.4	58.04	60.04	56.90	67.16	29.54
Zero-anchored + KIC	2-16-16	6.87	12.82	57.32	35.07	65.12	73.07	61.33	71.04	36.8	57.11	58.38	48.19	58.84	26.50
Base + IAC + KIC	2-4-4	7.53	10.20	54.46	34.47	63.90	71.49	62.27	63.98	34.4	55.00	57.32	50.28	61.79	26.09

Appendix

Table 5: Level-placement and representation controls on LLaMA-2-7B. Bold marks the best result within each precision setting.

Selection rule	Enrichment	Entropy	Wiki2 $\downarrow$	C4 $\downarrow$
Random	1.00 $\times$	1.01 $\times$	17.05	24.46
Prior-feature ridge	1.20 $\times$	1.01 $\times$	16.92	24.45
Two-stage cascade	1.29 $\times$	0.93 $\times$	16.88	24.47
Tail-damage score	1.45 $\times$	0.89 $\times$	16.69	24.37

Appendix

Table 6: Sequence-selection ablation on Qwen3-8B with a fixed 512-sequence calibration budget. Lower perplexity is better.

Calibration data	Wiki2	C4
C4 only	17.18	24.43
FineWeb only	16.75	24.32
C4 + FineWeb	16.83	24.33
C4 only, 1024 sequences	16.44	23.34
mix, 1024 sequences	16.26	23.44

Appendix

Table 7: Calibration corpus and budget ablation on Qwen3-8B. Rows use 512 sequences unless stated otherwise.

Method	Decoding path	LLaMA-2-7B	Qwen3-8B	Wiki2 $\downarrow$
		(ms)	(ms)
FP16	torch linear	25.26	27.01	5.47
QuaRot	BitBLAS uint2 per-row	4.72	5.01	24.52
ResComp	BitBLAS uint2 per-row	4.72	5.01	10.28
Base (uniform grid)	BitBLAS uint2 per-row	4.72	5.01	9.56
DB-LLM	additive decomposition	5.11	—	8.89

Appendix

Table 8: Linear-stack kernel time per token at batch size 1 on an RTX A6000 at W2A16, with WikiText2 perplexity on LLaMA-2-7B repeated from table 1 . Times exclude the online rotations, which add 1.11 ms for FidelityQ on LLaMA-2-7B. Scalar decoding paths are listed first. A dash marks a configuration we did not time.

Method	Weights (GB)	Prefill (ms/token)	Decode (tokens/s)
FP16	12.06	0.27	35.9
Marlin W4A16	3.02	0.21	39.8
BitBLAS uniform W2	1.51	6.07	35.6
QuIP# E8P	1.51	1.12	13.8
AQLM 1 $\times$ 16	1.73	1.98	23.2
FidelityQ	1.51	7.32	47.0

Appendix

Table 9: LLaMA-2-7B generation in HuggingFace eager mode at batch size 1. Storage covers linear weights and reconstruction metadata.

Method	W-A-KV	Perplexity $\downarrow$		ARC		HellaS	PIQA	WinoG	BoolQ	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Method	W-A-KV	Wiki2	C4	Easy	Chg	HellaS	PIQA	WinoG	BoolQ	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
FP16	16-16-16	5.47	6.97	73.86	45.05	76.16	78.78	69.46	79.36	44.0	66.67	68.66	68.81	73.65	45.94
RTN ∗	2-16-16	diverged		26.22	27.05	25.57	50.54	51.30	37.98	24.8	34.78	36.14	0.00	0.00	24.80
RTN	$2_{g64}$ -16-16	431.97	—	—	—	—	—	—	—	—	—	—	—	—	—
GPTQ ∗	2-16-16	29.83	335.50	41.84	27.30	42.38	62.35	52.09	56.61	29.0	44.51	45.19	21.99	25.58	22.95
QuaRot ∗	2-16-16	24.52	39.39	39.35	23.72	33.30	58.98	52.17	61.41	28.2	42.45	41.50	15.72	17.04	25.87

Appendix

Table 10: Extended results on LLaMA-2-7B. Markers are defined in appendix E .

Method	W-A-KV	Perplexity $\downarrow$		ARC		HellaS	PIQA	WinoG	BoolQ	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Method	W-A-KV	Wiki2	C4	Easy	Chg	HellaS	PIQA	WinoG	BoolQ	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
FP16	16-16-16	4.88	6.47	76.47	48.81	79.62	80.25	72.45	82.11	45.2	69.27	71.52	70.68	76.54	55.43
RTN ∗	2-16-16	5634.55	4815.95	26.01	28.16	26.20	49.35	48.07	37.83	23.4	34.15	35.56	0.00	0.00	22.92
GPTQ ∗	2-16-16	38.55	108.93	28.87	24.83	28.61	51.36	50.36	57.22	25.4	38.09	36.81	0.39	8.00	23.28
QuaRot ∗	2-16-16	12.17	24.11	43.81	25.09	45.14	62.30	56.59	62.54	28.8	46.32	46.59	31.11	40.42	25.44
ResComp ∗	2-16-16	7.90	14.05	55.68	30.20	53.24	68.66	62.83	63.18	33.4	52.46	54.12	38.95	50.01	27.97

Appendix

Table 11: Extended results on LLaMA-2-13B. Markers are defined in appendix E .

Method	W-A-KV	Perplexity $\downarrow$		ARC		HellaS	PIQA	WinoG	BoolQ	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Method	W-A-KV	Wiki2	C4	Easy	Chg	HellaS	PIQA	WinoG	BoolQ	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
FP16	16-16-16	6.14	8.88	77.74	54.10	79.22	80.58	73.16	81.83	44.8	70.20	72.96	68.81	75.59	65.48
DB-LLM m	$2_{g128}$ -16-16	13.62	19.26	52.15	29.52	53.55	69.15	60.46	66.24	34.0	52.15	52.97	38.37	50.36	28.03
DB-LLM ∗	2-16-16	14.70	55.66	25.38	24.15	27.14	50.92	50.51	39.20	27.4	34.96	35.62	0.12	0.47	23.69
QuaRot ∗	2-16-16	22.90	45.88	27.40	24.23	27.59	52.50	49.25	39.60	25.4	35.14	36.19	1.26	3.03	25.37
ResComp ∗	2-16-16	15.17	37.52	36.24	24.57	39.06	58.11	56.43	62.26	29.2	43.70	42.88	19.31	26.99	25.64

Appendix

Table 12: Extended results on LLaMA-3-8B. Markers are defined in appendix E .

Method	W-A-KV	Perplexity $\downarrow$		ARC		HellaS	PIQA	WinoG	BoolQ	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Method	W-A-KV	Wiki2	C4	Easy	Chg	HellaS	PIQA	WinoG	BoolQ	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
FP16	16-16-16	7.00	10.42	79.88	56.23	78.60	79.54	72.06	83.18	41.8	70.18	73.26	66.02	70.83	76.84
AutoRound ∗	2-16-16	87.25	64.30	46.04	29.27	42.28	62.30	55.56	59.36	31.4	46.60	47.09	4.72	6.85	23.19
QuaRot ∗	2-16-16	12.58	23.02	61.15	34.47	52.47	67.79	61.88	63.67	31.6	53.29	55.55	43.78	49.56	43.93
DB-LLM ∗	2-16-16	11.62	23.33	50.42	30.97	50.92	64.64	57.46	58.90	35.6	49.84	50.88	32.18	40.68	27.55
FidelityQ	2-16-16	8.84	13.64	71.93	44.88	70.28	74.97	67.96	74.92	42.0	63.85	66.00	57.91	65.11	59.94

Appendix

Table 13: Extended results on Qwen3-8B-Base. Markers are defined in appendix E .

Method	W-A-KV	Perplexity $\downarrow$		ARC		HellaS	PIQA	WinoG	BoolQ	OBQA	Avg. $\uparrow$		LAMBADA		MMLU 5
Method	W-A-KV	Wiki2	C4	Easy	Chg	HellaS	PIQA	WinoG	BoolQ	OBQA	7-task	5-task	Std	OpenAI	MMLU 5
FP16	16-16-16	4.10	5.98	79.25	53.58	83.01	81.83	75.85	83.79	49.2	72.36	74.70	73.06	77.61	58.30
[1.5pt/1.5pt] ResComp	2-16-16	6.45	10.38	68.01	38.05	63.98	74.32	69.30	68.75	38.0	60.06	62.73	54.57	67.20	32.00
Base (uniform grid)	2-16-16	6.33	10.10	67.05	38.57	64.91	74.70	68.59	66.54	39.0	59.91	62.76	54.63	68.33	31.29
FidelityQ	2-16-16	5.14	7.64	72.10	45.56	79.23	78.67	71.98	79.45	42.4	67.06	69.51	68.21	74.83	49.96
[1.5pt/1.5pt] QuaRot	2-4-4	99.92	961.80	50.46	29.95	46.67	61.04	57.38	61.04	31.6	48.31	49.10	21.42	22.74	25.99

Appendix

Table 14: Extended results on LLaMA-30B. Markers are defined in appendix E .

Low-Bit Quantization

Low-bit quantization reduces neural-network precision to save memory and computation while seeking to preserve task performance.

See all 223 papers

OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
Oct 7, 2026Wenjun Wang, Heng Li, Yanggan Gu +1LLM Quantization Quantization
The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
Jul 3, 2026Hamish OgilvyLLM Inference Acceleration Mixed-Precision Quantization
Algolia Research
WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization
May 26, 2026Phong Nam Huu Nguyen, Khoi M. Le, Cong-Duy T Nguyen +3Mixed-Precision Quantization LLM Quantization
CAIR, VinUniversity, Vietnam

Oct 7, 2026J/K move · Enter open · S save

OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models

Wenjun Wang, Heng Li, Yanggan Gu, Hongxia Yang

The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute

LLM Quantization Quantization Quantization-Aware Training Quantization-Aware Distillation On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Quantization-aware training (QAT) can recover much of the accuracy lost when large language models are compressed below four bits. Existing re- covery stages, however, are commonly optimized on fixed completions or teacher-generated answers, whereas the deployed quantized model condi- tions on prefixes generated by itself. Quantization errors can therefore move the model into states that are absent from offline recovery data. We introduce OnlineQAT, a two-stage framework that first obtains a usable low-bit initialization through block-wise QAT and then performs on-policy distillation (OPD) on student-generated responses. At each visited pre- fix, a frozen full-precision teacher provides a sampled reverse-KL training signal. On Qwen3-1.7B, OnlineQAT obtains the best average among the compared quantized methods: 57.28 at W3A16 and 32.52 at W2A16, im- proving over ReasoningQAT by 2.90 and 0.44 points, respectively. The results suggest that student-visited states provide a useful recovery signal beyond fixed-completion training, particularly at three bits.

Abstract

Low-Bit Quantization

Explore similar work

OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization