Ultra-low-bit quantization of large language models is a challenging task because aggressive compression severely distorts model parameters and disproportionately harms a small number of highly sensitive channels. We present a vector quantization method tailored for this regime. Our approach begins with random preconditioning, which regularizes the weight space and makes it more amenable to quantization. We then represent weights in polar coordinates and decouple the quantization of directions and magnitudes. To quantize directions, we leverage the structure induced by random preconditioning and design a structured codebook based on the E8 lattice, yielding a compact yet expressive representation. To further reduce degradation, we detect extremely important channels using a proxy Hessian and preserve them with an additional cost of only 0.16 bits. Together, these components provide an effective quantization pipeline for LLMs in the extremely low-bit regime. Empirical results demonstrate improved accuracy, highlighting the benefit of combining geometric structure, lattice-based codebooks, and selective channel preservation.
Figures & tables
Figure 1: Pipeline of HADVQ. The method applies random preconditioning, decomposes each weight block into magnitude and direction, quantizes the two components separately, reconstructs the quantized blocks, and preserves highly sensitive channels using proxy Hessian scores.
Model
Method
Bits
BoolQ ↑
WinoGrande ↑
ArcE ↑
ArcC ↑
HellaSwag ↑
PiQA ↑
Llama2-7B
FP16
16
71.10
67.17
69.28
39.93
56.70
78.35
QuIP#
3
[69.70]
[66.50]
[68.40]
[39.20]
-
[77.30]
AQLM
3.04
-
[66.93]
[68.06]
[38.40]
[54.12]
[76.88]
QTIP
3
-
[66.90]
[68.10]
[38.90]
-
[78.10]
ICQuant
3.3
-
66.90
66.60
39.00
-
77.70
HADVQ
3.41
70.95
66.85
69.07
38.48
55.49
77.97
Table 1: Zero-shot accuracy ( ↑ ) comparison on Llama2-7B, Llama2-13B, and Llama3-8B under low-bit quantization. Gray rows denote our method. [⋅] indicates results obtained with any form of fine-tuning. All results are reported using LM Eval Harness v0.3.0.
Llama2-7B
Llama2-13B
Llama3-8B
Method
bits
2048 ↓
4096 ↓
bits
2048 ↓
4096 ↓
bits
2048 ↓
4096 ↓
FP16
16
5.47
5.12
16
4.88
4.57
16
6.14
5.75
QuIP#
3
[5.79]
[5.41]
3
[5.10]
[4.78]
–
–
–
AQLM
3.04
–
[5.46]
3.03
–
[4.82]
–
–
–
QTIP
3
–
[5.28]
3
–
[4.69]
–
–
–
ICQuant
3.3
–
5.35
3.3
–
4.75
–
–
–
Table 2: WikiText-2 perplexity ( ↓ ) for sequence lengths 2048 and 4096. [⋅] indicates results reported with fine-tuning.
Figure 2: Layer-wise quantization loss on Qwen3-0.6B for different projection matrices.
Figure 5
Qwen3-0.6B
Qwen3-4B
Qwen3-8B
Variant
Bits
Avg. Acc ↑
Wiki2 ↓
Avg. Acc ↑
Wiki2 ↓
Avg. Acc ↑
Wiki2 ↓
FP16
16
56.05
20.90
71.40
13.70
73.98
9.71
HADVQ (1% important, final codebook)
2.41
45.73
49.39
64.09
17.73
69.95
12.10
w/o important channels
2.25
45.47
107.71
61.25
21.34
64.60
14.18
1% random channels
2.41
43.90
249.04
59.19
24.56
48.04
69.98
greedy min–max codebook
2.25
40.45
306.02
58.40
26.69
51.82
55.53
Table 3: Ablation of sensitive-channel preservation and directional codebook construction. We report average zero-shot accuracy (%) over BoolQ, WinoGrande, ARC-Easy, ARC-Challenge, and PIQA, together with WikiText-2 perplexity. Results were obtained with LM Eval Harness v0.4.11.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Empirical distributions of the coordinates of the first column vector of the query projection matrix in layer 0 of Qwen3-0.6B, before and after random Hadamard preconditioning. After preconditioning, the distribution becomes noticeably closer to Gaussian, which is consistent with the asymptotic normality stated in the lemma.
Bits
Qwen3-0.6B
Qwen3-4B
Qwen3-8B
tok/s
tok/s
tok/s
16
21.48
17.05
16.73
2.41
9.48
5.90
3.59
Appendix
Table 12: Inference throughput in tokens per second.
Asset
Type
License
Notes
Llama2, Llama3, Llama3.2
Base models
Llama Community License
Use is subject to the corresponding Llama Community License Agreement and Acceptable Use Policy.
Qwen3
Base models
Apache 2.0
We use publicly released Qwen3 checkpoints.
The Pile
Calibration data
MIT (repository)
We use mit-han-lab/pile-val-backup for calibration. The Pile aggregates multiple sources, so users should also respect the terms of the underlying components.
BoolQ
Evaluation dataset
CC BY-SA 3.0
Used only for evaluation.
WinoGrande
Evaluation dataset
Not clearly specified
Used only for evaluation; we cite the original dataset source.
ARC-Easy / ARC-Challenge
Evaluation dataset
Not clearly specified
Used only for evaluation; we cite the original dataset source.
Appendix
Table 13: Main external assets used in this work and their licenses.