Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by quantization noise from heavy-tailed outlier channels and error accumulation across layers. To address these issues, we present Bit-by-Bit, a progressive QAT framework with outlier channel splitting. Our approach integrates three key components: (1) block-wise progressive training that reduces precision stage by stage, ensuring stable initialization for low-bit optimization; (2) nested structure of integer quantization grids to enable a "train once, deploy any precision" paradigm, allowing a single model to support multiple bit-widths without retraining; (3) rounding-aware outlier channel splitting, which mitigates quantization error while acting as an identity transform that preserves the quantized outputs. Furthermore, we follow microscaling groups with E4M3 scales, capturing dynamic activation ranges in alignment with OCP/NVIDIA standards. To address the lack of efficient 2-bit kernels, we developed custom operators for both W2A2 and W2A16 configurations, achieving up to 11× speedup over BF16. Under W2A2 settings, Bit-by-Bit significantly outperforms baselines like BitDistiller and EfficientQAT on both Llama2/3, achieving a loss of only 2.25 WikiText2 PPL compared to full-precision models.
Figures & tables
Figure 1: Loss landscapes under different precisions. The vertical axis denotes the loss, the horizontal axes ( α,β ) represent random directions in parameter space.
Figure 2: Analysis of QAT challenges. (a) Training loss curve of direct QAT, exhibiting a prominent loss spike. (b) Layer-wise reconstruction loss and relative error across Transformer blocks, illustrating significant error accumulation in deeper layers. (c) Comparison of training budgets; our method (Bit-by-Bit) achieves a 3600 × reduction in token requirements compared to ParetoQ.
Figure 3: Value distributions of various group granularities showing (a) Low-bit values are nested in the high-bit grid, (b) lower bits collapse representations; larger groups improve dynamic-range.
Figure 4: (a) Progressive Bit-by-Bit QAT: Direct 2-bit QAT drives weights into coarse clusters under a non-smooth loss landscape, progressive schedule that lowers precision stage-by-stage, using the higher-precision phase to stabilize and initialize the next stage. (b) Rounding-aware outlier channel splitting: detect outlier channels via metric ∣∣x∣∣2⋅max∣w∣ , then apply identical, rounding-aware halving that keeps the quantized output unchanged.
Figure 5: Bit-shifting from higher to lower precision while sharing the same scale s .
Method
Bits
Group
WikiText2
C4
2-7B
3.2-1B
3.2-3B
3-8B
2-7B
3.2-1B
3.2-3B
3-8B
FP16
-
-
5.47
9.75
7.81
6.13
6.97
12.74
10.44
8.89
Weight Only Quantization (w2a16)
GPTQ
w2a16
32
60.5
2775.63
379.23
43.34
33.7
1875.41
323.24
43.28
AWQ
w2a16
32
2.2e5
1.7e7
7.2e6
5.2e5
1.75e5
1.9e7
7.7e6
5.1e5
OmniQuant
w2a16
32
11.06
6260.71
1.4e51
2.2e6
15.02
2442.55
8315.17
8.3e5
Table 1: Evaluation results on WikiText2 and C4 across different model sizes. Our method Bit-by-Bit is highlighted.
LLaMA-3.2-3B
PIQA
Hella.
Wino.
ARC-c
ARC-e
Avg
bfloat16
77.47
73.62
69.61
45.90
71.71
67.67
ParetoQ
66.70
43.48
52.49
21.93
44.36
45.79
w2a16
EfficientQAT
70.02
57.07
59.35
34.13
58.92
55.89
BitDistiller
70.65
57.42
59.78
34.71
58.34
56.18
Bit-by-Bit (ours)
71.87
58.03
60.38
35.58
58.71
56.91
ParetoQ
51.80
25.76
48.78
23.55
27.53
35.48
Table 2: Zero-shot evaluation of LLaMA-3.2 3B on five downstream tasks accuracy.
Block-wise
Progressive
Ocs
Metric
group size
WikiText2 ppl
Task avg
Memory
-
-
-
-
32
1.7e3
35.09
0.33GB
✓
-
-
-
32
31.88
40.87
0.33GB
✓
✓
-
-
32
24.60
43.26
0.33GB
✓
✓
✓
Kurtosis
32
22.43
43.69
0.36GB
✓
✓
✓
wmax
32
20.37
44.26
0.36GB
✓
✓
✓
xmax
32
19.07
44.30
0.36GB
Table 3: Ablation study on Llama 3.2-1b on w2a16 setting.
Mistral-7B
Bits
Method
C4 ppl
Task avg
bfloat16
8.24
73.99
w8a16
OmniQuant
8.24
73.77
MatQuant
8.43
73.46
Bit-by-Bit (ours)
8.33
73.51
w4a16
OmniQuant
8.47
73.62
Table 4: Evaluation of Mistral-7B under different quantization methods
Figure 6: Kernel latency of Bit-by-Bit relative to native PyTorch and Marlin of Wup , Wdown .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Precision
Gsm8k
MathQA
Mmlu
Ifeval
Llama2-13B
FP16
0.22
0.32
0.52
0.18
Bit-by-Bit w2a16
0.20
0.32
0.50
0.17
Bit-by-Bit w2a2
0.11
0.29
0.40
0.16
Qwen2.5-7B
FP16
0.80
0.43
0.71
0.28
Bit-by-Bit w2a16
0.77
0.42
0.70
0.28
Bit-by-Bit w2a2
0.75
0.38
0.70
0.27
Appendix
Table 5: Performance comparison on complex reasoning and instruction following capabilities (GSM8k, MathQA, MMLU, and IFEval)
Block-wise
Progressive
Ocs
Metric
Calibration
group size
WikiText2 ppl
C4 ppl
-
-
-
-
-
32
2.0e5
1.0e6
✓
-
-
-
-
32
1441.9
4592.8
✓
✓
-
-
-
32
42.2
120.2
✓
✓
✓
Kurtosis
WikiText2
32
41.8
127.4
✓
✓
✓
wmax
WikiText2
32
36.75
97.66
✓
✓
✓
∥x∥2⋅∣w∣
WikiText2
32
32.48
79.95
Appendix
Table 6: Ablation study on LLaMA-3.2-1B under the W2A2 setting. All models are trained for one epoch. “-” indicates the component is disabled.
Model
Shape (N,K)
W2A2 ( μ s)
BF16 ( μ s)
Llama3.2-3B
(1024, 3072)
8.93
6.45
(3072, 3072)
8.64
16.04
(3072, 8192)
8.70
32.80
(8192, 3072)
8.94
20.76
Llama3-8B
(1024, 4096)
8.58
6.87
(4096, 4096)
8.64
29.73
Appendix
Table 7: Latency comparison for weight matrices appearing in the decode layers of LLaMA models.
Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%--10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.
Haoyu Wang, Xingyu Yu, Haiyan Zhao +2
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.
Quantization-Aware Training from scratch has emerged as a promising approach for building efficient large language models (LLMs) with extremely low-bit weights (sub 2-bit), which can offer substantial advantages for edge deployment. However, existing methods still fail to achieve satisfactory accuracy and scalability. In this work, we identify a parameter democratization effect as a key bottleneck: the sensitivity of all parameters becomes homogenized, severely limiting expressivity. To address this, we propose pQuant, a method that decouples parameters by splitting linear layers into two specialized branches: a dominant 1-bit branch for efficient computation and a compact high-precision branch dedicated to preserving the most sensitive parameters. Through tailored feature scaling, we explicitly guide the model to allocate sensitive parameters to the high-precision branch. Furthermore, we extend this branch into multiple, sparsely-activated experts, enabling efficient capacity scaling. Extensive experiments indicate our pQuant achieves state-of-the-art performance in extremely low-bit quantization.
Wenzheng Zhang, Bingzheng Liu, Yang Hu +3
School of Computer Science, Peking University · College of Future Information Technology, Fudan University · Center for Information Research, Academy of Military Sciences +2
Post-training quantization (PTQ) has become an important technique for reducing the inference cost of Large Language Models (LLMs). While recent mixed-precision methods improve ultra-low bit quantization by preserving critical subspaces in high precision, they typically construct these subspaces relying solely on activation statistics. This ignores the fundamental nature of linear operations, where the output perturbation is jointly driven by both activation and weight quantization noise. In this paper, we propose CoQuant, a joint weight-activation subspace projection method. By theoretically modeling the expected output error, CoQuant formulates a closed-form weighted PCA solution that balances activation and weight covariances to select the optimal high-precision subspace. Extensive experiments on Llama-3.2 and Qwen2.5 models show that CoQuant consistently outperforms strong PTQ baselines in both WikiText perplexity and zero-shot common-sense reasoning accuracy. These results demonstrate that joint weight-activation subspace modeling provides a principled and effective direction for low-bit LLM quantization. The source code is available at https://github.com/Zachary5895/CoQuant.
Zhe Ding, Su Pan, Duowei Pan
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA