Quantization-aware training (QAT) is an effective method to drastically reduce the memory footprint of LLMs while keeping performance degradation at an acceptable level. However, the optimal choice of quantization format and bit-width presents a challenge in practice. The full design space of quantization is not fully explored in the context of QAT, and the precise trade-off between quantization and downstream performance is poorly understood, as comparisons often rely solely on perplexity-based evaluations. In this work, we address these shortcomings with an empirical study of QAT in the low-bit regime. We show that k-means based weight quantization outperforms integer formats and can be implemented efficiently on standard hardware. Furthermore, we find that, under a fixed inference memory budget, the best performance on generative downstream tasks is achieved with 1-bit quantized weights.
Figures & tables
Figure 2: Precision-to-capacity mapping and the induced memory-normalized efficiency. (Left) The saturating function f(Pw) models diminishing returns as weight precision increases, where higher values are better. (Right) The ratio g(Pw)=f(Pw)/Pw determines the optimal precision under a fixed inference-memory budget M=NPw , where higher values are better.
Benchmark
4 B/ 16 -bit
12 B/ 4 -bit
31 B/ 1 -bit
MMLU
33.21
50.86
51.61
HellaSwag
50.16
55.41
54.70
PIQA
74.97
76.88
77.20
ARC-C
41.21
49.15
50.68
ARC-E
71.34
75.25
77.27
MBPP
19.00
26.40
30.00
Table 1: Memory-matched scaling evaluation results across model sizes and bit-widths.
Figure 3: A theoretical model for speedup of 1 -bit and 4 -bit formats versus bf16 , when decoded to bf16 in software on an L40S GPU.
Pw
Time (Speedup)
Effective BW
16
175.6\upmus ( 1.0× )
764GB/s
4.25
49.5\upmus ( 3.7× )
721GB/s
1.25
24.0\upmus ( 7.6× )
438GB/s
Table 2: Micro-benchmark results for the product of a 1×8192 bf16 vector and 8192×8192 quantized matrix. The 4 -bit format achieves near-optimal speedup. Peak memory bandwidth for the test GPU is 864GB/s . All standard errors are <0.2\upmus .
Tokens/s
Pw
4 B
12 B
31 B
16
88.8
30.0
OOM
4.25
190.1
79.3
35.4
1.25
245.6
120.0
60.7
Table 3: Token generation performance for the models of Section 3.2 ( blue ), for single-user autoregressive generation. These correspond to effective bandwidth of 620 , 524 , and 372GB/s for 4 B 16 -bit, 12 B 4 -bit, and 31 B 1 -bit models respectively. The maximum error bar ( ±2 standard errors) is <1% of the mean.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Nemotron
FineMath-3+
FineMath-4+
Starcoder-V2
Phase
%
Tokens
%
Tokens
%
Tokens
%
Tokens
Phase 1 (72%)
85%
92.4B
3%
3.3B
0%
0.0B
12%
13.0B
Phase 2 (18%)
75%
20.4B
0%
0.0B
16%
4.4B
9%
2.4B
Phase 3 (10%)
63%
9.5B
0%
0.0B
27.4%
4.1B
9.6%
1.4B
Appendix
Table 4: Pretraining curriculum mixture over data sources. Training runs for 36 K steps with 4194304 tokens per step ( ≈151 B tokens total). Phase lengths follow a 72%/18%/10% split of total training. Each entry reports both mixture proportion and the corresponding token count.
Model
Parameters N
Dim d
Layers L
Heads
KV Heads
Max LR
RoPE θ
0.8 B
81871411 2
1536
12
12
6
8⋅10−4
500000
1.4 B
1431373824
2048
16
16
8
6⋅10−4
500000
3.9/4 B
3883551744
3072
24
24
8
4⋅10−4
500000
12 B
11520053248
4096
48
32
8
3⋅10−4
500000
31 B
30643279872
6144
60
48
8
2⋅10−4
500000
Appendix
Table 5: Architectural and optimization hyperparameters for all model scales. All models use SwiGLU with multiplier 1.3 , RMSNorm pre-normalization, grouped query attention, FlexAttention-based SDPA, no dropout or bias terms, and gradient clipping at norm 1.0 .
Figure 4: Normalization ablation. Training loss curves with logarithmic y -axis for different block-wise normalization strategies. Left: 2-bit quantization, where absmean scaling leads to substantially more stable optimization. Right: 4 -bit quantization, where absmax scaling becomes favorable. K-means quantization remains largely insensitive to this choice.
Figure 5: Training progression Evolution of pretraining evaluations during the training of the 4 B bf16 , 12 B 4 -bit, and 31 B 1 -bit models.
Figure 6: Predicted vs. actual loss for k-means (left) and uniform (right) quantization formats. Points are colored by precision Pw (bits). Both methods achieve strong fits with R2>0.96 , with uniform quantization showing slightly better predictive accuracy (RMSE =0.0305 ) compared to k-means (RMSE =0.0391 ). The dashed line indicates perfect prediction ( y=x ).
Figure 7: Effective parameter density Neff(Pw)/M versus backbone bit-width Pw for fixed weight-memory budgets M (GB). Solid lines: uniform quantization; dashed lines: kmeans quantization. The 16-bit special block consists of both token embeddings and the LM head ( E(N)=2Vd(N) ). For large budgets the density is maximized at the smallest feasible Pw , while at very small budgets the optimum can shift upward because the 16-bit special block becomes non-negligible. Interestingly, the optimal bit-width for uniform formats at 8 GB memory-budget is 2 bits, while the kmeans format yields 1 bit as the optimal bit-width.
Pw
Implementation
Time (Speedup)
Effective Bandwidth
16
Torch
181.6\upmus ( 1.0× )
739GB/s
16
Triton
175.6\upmus ( 1.0× )
764GB/s
4.25
torch.compile
772.3\upmus ( 0.2× )
46GB/s
4.25
Marlin
53.5\upmus ( 3.4× )
667GB/s
4.25
Triton
49.5\upmus ( 3.7× )
721GB/s
1.25
torch.compile
770.4\upmus ( 0.2× )
14GB/s
Appendix
Table 6: A micro-benchmark of different kernel implementations, for m=1 and h=8192 . The theoretical maximum speedup for 4 -bit and 1 -bit kernels is 3.8× and 12.8× respectively. Standard errors are <0.2\upmus except for the torch.compile implementation ( <0.6\upmus ).
Figure 8: A micro-benchmark of kernel performance for square matrix multiplication, swept over batch size m , hidden size h and bit-width. The maximum error bar ( ±2 standard errors) is approximately ±1% of the mean, so error bars are not visible at this scale.
Pw
Implementation
4B
12B
31B
16
Torch
88.8
30.0
OOM
4.25
Triton
190.1
79.3
35.4
4.25
Marlin
151.5
64.1
32.4
4.25
Marlin (Linear)
156.2
65.5
32.8
1.25
Triton
245.6
120.0
60.7
Appendix
Table 7: Model tokens/s for autoregressive generation at batch size 1 . Entries in blue correspond to our scaled-up training settings from Section 3.2 The 4 -bit 12 B model is ≈10% slower than the 16 -bit 4 B model, while the 1 -bit 31 B model has somewhat lower tokens/s since the dequantization and higher-precision arithmetic is unable to saturate memory bandwidth (see Table 6 ). The maximum error bar ( ±2 standard errors) is <1% of the mean.
Figure 9: Autoregressive decoding speed for different model size, batch size and bit-width. Larger batches are beneficial to token/s, since they amortize the cost of transferring the model parameters, but this has greater impact on the bf16 baseline than on compact formats. The maximum error bar ( ±2 standard errors) is <0.5% of the mean.
Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%--10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.
Haoyu Wang, Xingyu Yu, Haiyan Zhao +2
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.
The rapid development of LLMs incurs prohibitive memory footprints and intensive computational demands. Quantization-Aware Training (QAT) techniques have emerged as a promising solution to address these challenges by explicitly simulating quantization effects during model training, yielding low-bit models that achieve accuracy comparable to their full-precision counterparts. In this work, we provide a target-centric survey of QAT, aimed at clarifying both its theoretical foundations and its evolving implementation landscape. We systematically review existing QAT methods through a target-centric taxonomy and synthesize cross-target differences in error characteristics, numerical formats, and strategy transferability. We further summarize QAT evaluation paradigms and discuss challenges in optimization and deployment, outlining potential directions for future research.
Jiamin Song, Mengjie Zhao, Zijing Wang +6
1Northeastern University, China · 2Shandong University, China · 3CIS, LMU Munich; MCML, Germany
Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.
Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2