Trainable Vector Quantization for Large Language Models
Organizations: Huawei Technologies Co. Ltd.
Abstract
Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ training-free approaches that are limited to layer-wise reconstruction loss minimization. To address this limitation, we propose TVQ, a novel method that leverages straight-through estimation to enable gradient descent-based optimization of vector-quantized LLMs. By reformulating VQ LLM optimization with trainable latent weights, TVQ enables arbitrary objective minimization. We constitute our approach by applying TVQ in a block-wise knowledge distillation paradigm, which balances quality and training efficiency, and propose several practical solutions to accelerate training. In experiments on modern LLMs, we demonstrate that TVQ is compatible with modern VQ schemes. As a result, TVQ achieves state-of-the-art quality in low-bit post-training quantization settings with a modest training budget and straightforward PyTorch implementation. For instance, TVQ achieves near-lossless 2-bit quantization for LLaMA2-70B with only 1% accuracy drop on popular Question-Answering benchmarks.
Figures & tables
| Method | Bits | W2 | C4 | WG | PQ | HS | AE | AC | QA Avg. | |
| Llama3-8B | FP16 | 16 | 5.54 | 7.10 | 73.1 | 79.6 | 60.2 | 80.1 | 50.3 | 68.7 / 70.8 |
| QuIP# | 2.00 | 7.84 | 9.06 | 68.2 | 75.6 | - | 72.9 | 39.2 | - / 64.0 | |
| VPTQ | 2.08 | - | - | 65.9 | 75.1 | 52.2 | 71.0 | 36.9 | 60.2 / 62.2 | |
| QTIP | 2.00 | 7.33 | 8.62 | 70.7 | 77.6 | - | 75.2 | 44.2 | - / 66.9 | |
| PV-Tuning | 2.01 | 6.99 | 8.29 | 69.9 | 77.8 | 55.5 | 75.8 | 42.8 | 64.4 / 66.6 | |
| TVQ+BasicVQ | 2.02 | 7.59 | 8.86 | 71.5 | 76.6 | 54.3 | 73.5 | 41.3 | 63.4 / 65.7 |
| Method | Stage-1 | Stage-2 | Total |
| AQLM | 250 | 50 | 300 |
| VPTQ | 74 | 50 | 124 |
| Quip# | 40 | 50 | 90 |
| QTIP | 50 | 50 | 100 |
| PV-tuning | X | 1500 | >1500 |
| TVQ+Quip# | 54 | 50 | 104 |
| Stage-1 time | Llama2-7B | Llama3-8B | Mistral-7B | ||||
| W2 | QA | W2 | QA | W2 | QA | ||
| 8 | 32h | 6.15 | 59.0 | 8.45 | 63.3 | 5.72 | 65.8 |
| 16 | 21h | 6.17 | 58.9 | 8.60 | 62.2 | 5.71 | 66.0 |
| 32 | 15h | 6.22 | 58.8 | 8.68 | 62.0 | 5.74 | 65.7 |
| 64 | 14h | >100 | <10 | >100 | <10 | >100 | <10 |
| Stage-1 | Llama2-70B | ||
| time | W2 | QA | |
| 1 | 51h | 4.11 | 68.2 |
| 2 | 33h | 4.10 | 68.1 |
| 3 | 24h | 4.11 | 68.1 |
| 4 | 19h | 4.15 | 67.8 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | VQ scheme | |
| Codebook matrix constraints | Index matrix constraints | |
| AQLM, VPTQ | (unconstrained) | (unconstrained) |
| QuIP# | (unconstrained) | |
| QTIP | , where is an unconstrained smaller codebook that fits into L1 GPU cache | , for all , where is the average bitwidth of the packed index matrix |
| Search Accuracy | N | K | Speedup | W2 | C4 |
| 99.9 % | 256 | 16 | 5.0 | 6.70 | 8.70 |
| 99 % | 1024 | 16 | 5.5 | 6.64 | 8.68 |
| 98 % | 256 | 8 | 6.4 | 6.63 | 8.69 |
| 95 % | 512 | 8 | 6.5 | 6.68 | 8.73 |
| 80 % | 1024 | 4 | 6.7 | 6.84 | 8.92 |
| 60 % | 1024 | 2 | 6.7 | 7.13 | 9.15 |
| Method | Index update algorithm | Update time (sec) |
| TVQ+Quip# | E8-lattice NN search | 0.236 ± 0.001 |
| TVQ+BasicVQ | ANN search | 1.31 ± 0.02 |
| TVQ+QTIP | Viterbi search | 41.8 ± 0.2 |
| Model | Method | SL | SN | E | BS | ANNs | |||
| TVQ+BasicVQ | 4096 | 2048 | 2 | 8 | 256/8 | 1 | 5e-5 | 5e-5 | |
| Llama2-7B | TVQ+QuIP# | 4096 | 2048 | 2 | 8 | - | 1 | - | 1e-5 |
| TVQ+QTIP | 4096 | 2048 | 2 | 8 | - | 16 | - | 1e-5 | |
| TVQ+BasicVQ | 4096 | 2048 | 1 | 8 | 128/8 | 1 | 5e-5 | 5e-5 | |
| Llama2-70B | TVQ+QuIP# | 4096 | 2048 | 1 | 8 | - | 2 | - | 1e-5 |
| TVQ+QTIP | 4096 | 2048 | 1 | 8 | - | 16 | - | 1e-5 |
| Model | Method | SL | SN | BS | |||
| TVQ+BasicVQ | 4096 | 4096 | 50 | 1e-5 | - | - | |
| Llama2-7B | TVQ+QuIP# | 4096 | 4096 | 50 | - | 5e-4 | 5e-5 |
| TVQ+QTIP | 4096 | 4096 | 50 | 1e-3 | 5e-4 | 5e-5 | |
| TVQ+BasicVQ | 4096 | 4096 | 50 | 1e-5 | - | - | |
| Llama2-70B | TVQ+QuIP# | 4096 | 8192 | 50 | - | 5e-4 | 5e-5 |
| TVQ+QTIP | 4096 | 8192 | 50 | 1e-3 | 5e-4 | 5e-5 |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers