Ultra-low-bit quantization of large language models is a challenging task because aggressive compression severely distorts model parameters and disproportionately harms a small number of highly sensitive channels. We present a vector quantization method tailored for this regime. Our approach begins with random preconditioning, which regularizes the weight space and makes it more amenable to quantization. We then represent weights in polar coordinates and decouple the quantization of directions and magnitudes. To quantize directions, we leverage the structure induced by random preconditioning and design a structured codebook based on the E8 lattice, yielding a compact yet expressive representation. To further reduce degradation, we detect extremely important channels using a proxy Hessian and preserve them with an additional cost of only 0.16 bits. Together, these components provide an effective quantization pipeline for LLMs in the extremely low-bit regime. Empirical results demonstrate improved accuracy, highlighting the benefit of combining geometric structure, lattice-based codebooks, and selective channel preservation.
Figures & tables
Figure 1: Pipeline of HADVQ. The method applies random preconditioning, decomposes each weight block into magnitude and direction, quantizes the two components separately, reconstructs the quantized blocks, and preserves highly sensitive channels using proxy Hessian scores.
Model
Method
Bits
BoolQ ↑
WinoGrande ↑
ArcE ↑
ArcC ↑
HellaSwag ↑
PiQA ↑
Llama2-7B
FP16
16
71.10
67.17
69.28
39.93
56.70
78.35
QuIP#
3
[69.70]
[66.50]
[68.40]
[39.20]
-
[77.30]
AQLM
3.04
-
[66.93]
[68.06]
[38.40]
[54.12]
[76.88]
QTIP
3
-
[66.90]
[68.10]
[38.90]
-
[78.10]
ICQuant
3.3
-
66.90
66.60
39.00
-
77.70
HADVQ
3.41
70.95
66.85
69.07
38.48
55.49
77.97
Table 1: Zero-shot accuracy ( ↑ ) comparison on Llama2-7B, Llama2-13B, and Llama3-8B under low-bit quantization. Gray rows denote our method. [⋅] indicates results obtained with any form of fine-tuning. All results are reported using LM Eval Harness v0.3.0.
Llama2-7B
Llama2-13B
Llama3-8B
Method
bits
2048 ↓
4096 ↓
bits
2048 ↓
4096 ↓
bits
2048 ↓
4096 ↓
FP16
16
5.47
5.12
16
4.88
4.57
16
6.14
5.75
QuIP#
3
[5.79]
[5.41]
3
[5.10]
[4.78]
–
–
–
AQLM
3.04
–
[5.46]
3.03
–
[4.82]
–
–
–
QTIP
3
–
[5.28]
3
–
[4.69]
–
–
–
ICQuant
3.3
–
5.35
3.3
–
4.75
–
–
–
Table 2: WikiText-2 perplexity ( ↓ ) for sequence lengths 2048 and 4096. [⋅] indicates results reported with fine-tuning.
Figure 2: Layer-wise quantization loss on Qwen3-0.6B for different projection matrices.
Figure 5
Qwen3-0.6B
Qwen3-4B
Qwen3-8B
Variant
Bits
Avg. Acc ↑
Wiki2 ↓
Avg. Acc ↑
Wiki2 ↓
Avg. Acc ↑
Wiki2 ↓
FP16
16
56.05
20.90
71.40
13.70
73.98
9.71
HADVQ (1% important, final codebook)
2.41
45.73
49.39
64.09
17.73
69.95
12.10
w/o important channels
2.25
45.47
107.71
61.25
21.34
64.60
14.18
1% random channels
2.41
43.90
249.04
59.19
24.56
48.04
69.98
greedy min–max codebook
2.25
40.45
306.02
58.40
26.69
51.82
55.53
Table 3: Ablation of sensitive-channel preservation and directional codebook construction. We report average zero-shot accuracy (%) over BoolQ, WinoGrande, ARC-Easy, ARC-Challenge, and PIQA, together with WikiText-2 perplexity. Results were obtained with LM Eval Harness v0.4.11.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Empirical distributions of the coordinates of the first column vector of the query projection matrix in layer 0 of Qwen3-0.6B, before and after random Hadamard preconditioning. After preconditioning, the distribution becomes noticeably closer to Gaussian, which is consistent with the asymptotic normality stated in the lemma.
Bits
Qwen3-0.6B
Qwen3-4B
Qwen3-8B
tok/s
tok/s
tok/s
16
21.48
17.05
16.73
2.41
9.48
5.90
3.59
Appendix
Table 12: Inference throughput in tokens per second.
Asset
Type
License
Notes
Llama2, Llama3, Llama3.2
Base models
Llama Community License
Use is subject to the corresponding Llama Community License Agreement and Acceptable Use Policy.
Qwen3
Base models
Apache 2.0
We use publicly released Qwen3 checkpoints.
The Pile
Calibration data
MIT (repository)
We use mit-han-lab/pile-val-backup for calibration. The Pile aggregates multiple sources, so users should also respect the terms of the underlying components.
BoolQ
Evaluation dataset
CC BY-SA 3.0
Used only for evaluation.
WinoGrande
Evaluation dataset
Not clearly specified
Used only for evaluation; we cite the original dataset source.
ARC-Easy / ARC-Challenge
Evaluation dataset
Not clearly specified
Used only for evaluation; we cite the original dataset source.
Appendix
Table 13: Main external assets used in this work and their licenses.
Scalar quantization of large language models (LLMs) is fundamentally limited by information-theoretic bounds. While vector quantization (VQ) overcomes these limits by encoding blocks of parameters jointly, practical implementations must avoid the need for expensive lookup mechanisms or other explicit codebook storage. Lattice approaches address this through highly structured and dense packing. This paper explores the Leech lattice, which, with its optimal sphere packing and kissing configurations at 24 dimensions, is the highest dimensional lattice known with such optimal properties. To make the Leech lattice usable for LLM quantization, we extend an existing search algorithm based on the extended Golay code construction, to i) support indexing, enabling conversion to and from bitstrings without materializing the codebook, ii) allow angular search over union of Leech lattice shells, iii) propose fully-parallelisable dequantization kernel. Lastly, we provide a geometric reinterpretation of combining shape--gain quantization with GPTQ-style Hessian corrections: the standard scale-correction step of shape--gain acts as a retraction onto a product of spheres, yielding a Spherical GPTQ primarily acting on directions. We find that low-angular-distortion LLVQ reduces sensitivity to Hadamard/rotation preprocessing, and enables a strong Hadamard-free PTQ in practice. LLVQ delivers state-of-the-art LLM quantization performance, outperforming recent methods such as Quip#, QTIP, and PVQ. The results highlight the effectiveness of high-dimensional lattices for scalable, theoretically grounded model compression.
Tycho F. A. van der Ouderaa, Mart van Baalen, Paul Whatmough +1
Trellis-coded quantization enables high-dimensional compression of large language model (LLM) weights at ultra-low bit widths without the exponentially large codebooks required by conventional vector quantization. Practical deployment, however, presents two challenges: reconstructing compressed weights at sufficient parallel throughput to avoid making dequantization an inference bottleneck, and maintaining quantization accuracy without costly incoherence transformations. We address these challenges with two complementary techniques. First, we introduce an ultra-low-complexity trellis dequantizer that uses a structured, hardware-efficient state-to-value mapping while preserving diverse reconstruction choices for trellis search. Second, we reformulate discrete trellis path optimization with a curvature-aware objective that reflects model sensitivity directly in the original coordinate space. Together, these techniques enable high-quality ultra-low-bit trellis quantization with inexpensive, highly parallel runtime reconstruction and without relying on Hadamard-based incoherence processing.
We present PolarQuant, a post-training weight quantization method for large language models (LLMs) that exploits the distributional structure of neural network weights to achieve near-lossless compression. PolarQuant operates in three stages: (1) block-wise normalization to the unit hypersphere, (2) Walsh-Hadamard rotation to transform coordinates into approximately Gaussian random variables, and (3) quantization with centroids matched to the Gaussian distribution. Our ablation reveals that Hadamard rotation alone accounts for 98% of the quality improvement, reducing Qwen3.5-9B perplexity from 6.90 (absmax Q5) to 6.40 (Delta = +0.03 from FP16), making it practically lossless without any calibration data. Furthermore, PolarQuant functions as an effective preprocessing step for downstream INT4 quantizers: PolarQuant Q5 dequantized and re-quantized by torchao INT4 achieves perplexity 6.56 versus 6.68 for direct absmax INT4, while maintaining 43.1 tok/s throughput at 6.5 GB VRAM. Code and models are publicly available.