cs.LGOct 30, 2026

Trainable Vector Quantization for Large Language Models

Authors: Anastasiia Prutianova, Alina Bogdanova, Mikhail Shutov, Dmitrii Poimanov, Anatoly Gorkov, Ilia Zharikov

Organizations: Huawei Technologies Co. Ltd.

Abstract

Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ training-free approaches that are limited to layer-wise reconstruction loss minimization. To address this limitation, we propose TVQ, a novel method that leverages straight-through estimation to enable gradient descent-based optimization of vector-quantized LLMs. By reformulating VQ LLM optimization with trainable latent weights, TVQ enables arbitrary objective minimization. We constitute our approach by applying TVQ in a block-wise knowledge distillation paradigm, which balances quality and training efficiency, and propose several practical solutions to accelerate training. In experiments on modern LLMs, we demonstrate that TVQ is compatible with modern VQ schemes. As a result, TVQ achieves state-of-the-art quality in low-bit post-training quantization settings with a modest training budget and straightforward PyTorch implementation. For instance, TVQ achieves near-lossless 2-bit quantization for LLaMA2-70B with only 1% accuracy drop on popular Question-Answering benchmarks.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

    Jun 9, 2026Haoyu Wang, Xingyu Yu, Haiyan Zhao +2LLM QuantizationQuantization-Aware Training

  2. UniSVQ: 2-bit Unified Scalar-Vector Quantization

    Jun 9, 2026Haoyu Wang, Haiyan Zhao, Xingyu Yu +4LLM QuantizationVector Quantization

  3. SQuAT: Spherical Lattice Vector Quantization-Aware Training

    Oct 22, 2026Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa +4LLM QuantizationVector Quantization