Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

Binxing Xu; Hao Gu; Lujun Li; Hao Wang; Bei Liu; Jiacheng Liu; Qiyuan Zhu; Xintong Yang

cs.LGApr 9, 2026

Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

Authors: Binxing Xu, Hao Gu, Lujun Li, Hao Wang, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Xintong Yang,

Organizations: Zhejiang University · Hong Kong University of Science and Technology · City University of Hong Kong

Abstract

Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by quantization noise from heavy-tailed outlier channels and error accumulation across layers. To address these issues, we present Bit-by-Bit, a progressive QAT framework with outlier channel splitting. Our approach integrates three key components: (1) block-wise progressive training that reduces precision stage by stage, ensuring stable initialization for low-bit optimization; (2) nested structure of integer quantization grids to enable a "train once, deploy any precision" paradigm, allowing a single model to support multiple bit-widths without retraining; (3) rounding-aware outlier channel splitting, which mitigates quantization error while acting as an identity transform that preserves the quantized outputs. Furthermore, we follow microscaling groups with E4M3 scales, capturing dynamic activation ranges in alignment with OCP/NVIDIA standards. To address the lack of efficient 2-bit kernels, we developed custom operators for both W2A2 and W2A16 configurations, achieving up to 11 $\times$ speedup over BF16. Under W2A2 settings, Bit-by-Bit significantly outperforms baselines like BitDistiller and EfficientQAT on both Llama2/3, achieving a loss of only 2.25 WikiText2 PPL compared to full-precision models.

Figures & tables

Figure 1: Loss landscapes under different precisions. The vertical axis denotes the loss, the horizontal axes ( $\alpha,\beta$ ) represent random directions in parameter space.

Figure 2: Analysis of QAT challenges. (a) Training loss curve of direct QAT, exhibiting a prominent loss spike. (b) Layer-wise reconstruction loss and relative error across Transformer blocks, illustrating significant error accumulation in deeper layers. (c) Comparison of training budgets; our method (Bit-by-Bit) achieves a 3600 $\times$ reduction in token requirements compared to ParetoQ.

Figure 3: Value distributions of various group granularities showing (a) Low-bit values are nested in the high-bit grid, (b) lower bits collapse representations; larger groups improve dynamic-range.

Figure 4: (a) Progressive Bit-by-Bit QAT: Direct 2-bit QAT drives weights into coarse clusters under a non-smooth loss landscape, progressive schedule that lowers precision stage-by-stage, using the higher-precision phase to stabilize and initialize the next stage. (b) Rounding-aware outlier channel splitting: detect outlier channels via metric $||\mathbf{x}||_{2}\cdot max{|w|}$ , then apply identical, rounding-aware halving that keeps the quantized output unchanged.

Figure 5: Bit-shifting from higher to lower precision while sharing the same scale

s

Method	Bits	Group	WikiText2				C4
Method	Bits	Group	2-7B	3.2-1B	3.2-3B	3-8B	2-7B	3.2-1B	3.2-3B	3-8B
FP16	-	-	5.47	9.75	7.81	6.13	6.97	12.74	10.44	8.89
Weight Only Quantization (w2a16)
GPTQ	w2a16	32	60.5	2775.63	379.23	43.34	33.7	1875.41	323.24	43.28
AWQ	w2a16	32	2.2e5	1.7e7	7.2e6	5.2e5	1.75e5	1.9e7	7.7e6	5.1e5
OmniQuant	w2a16	32	11.06	6260.71	1.4e51	2.2e6	15.02	2442.55	8315.17	8.3e5

Table 1: Evaluation results on WikiText2 and C4 across different model sizes. Our method Bit-by-Bit is highlighted.

LLaMA-3.2-3B		PIQA	Hella.	Wino.	ARC-c	ARC-e	Avg
bfloat16		77.47	73.62	69.61	45.90	71.71	67.67
	ParetoQ	66.70	43.48	52.49	21.93	44.36	45.79
w2a16	EfficientQAT	70.02	57.07	59.35	34.13	58.92	55.89
w2a16	BitDistiller	70.65	57.42	59.78	34.71	58.34	56.18
	Bit-by-Bit (ours)	71.87	58.03	60.38	35.58	58.71	56.91
	ParetoQ	51.80	25.76	48.78	23.55	27.53	35.48

Table 2: Zero-shot evaluation of LLaMA-3.2 3B on five downstream tasks accuracy.

Block-wise	Progressive	Ocs	Metric	group size	WikiText2 ppl	Task avg	Memory
-	-	-	-	32	1.7e3	35.09	0.33GB
✓	-	-	-	32	31.88	40.87	0.33GB
✓	✓	-	-	32	24.60	43.26	0.33GB
✓	✓	✓	Kurtosis	32	22.43	43.69	0.36GB
✓	✓	✓	$w_{\max}$	32	20.37	44.26	0.36GB
✓	✓	✓	$x_{\max}$	32	19.07	44.30	0.36GB

Table 3: Ablation study on Llama 3.2-1b on w2a16 setting.

Mistral-7B
Bits	Method	C4 ppl	Task avg
bfloat16		8.24	73.99
w8a16	OmniQuant	8.24	73.77
	MatQuant	8.43	73.46
	Bit-by-Bit (ours)	8.33	73.51
w4a16	OmniQuant	8.47	73.62

Table 4: Evaluation of Mistral-7B under different quantization methods

Figure 6: Kernel latency of Bit-by-Bit relative to native PyTorch and Marlin of $W_{\text{up}}$ , $W_{\text{down}}$ .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Precision	Gsm8k	MathQA	Mmlu	Ifeval
Llama2-13B	FP16	0.22	0.32	0.52	0.18
	Bit-by-Bit w2a16	0.20	0.32	0.50	0.17
	Bit-by-Bit w2a2	0.11	0.29	0.40	0.16
Qwen2.5-7B	FP16	0.80	0.43	0.71	0.28
	Bit-by-Bit w2a16	0.77	0.42	0.70	0.28
	Bit-by-Bit w2a2	0.75	0.38	0.70	0.27

Appendix

Table 5: Performance comparison on complex reasoning and instruction following capabilities (GSM8k, MathQA, MMLU, and IFEval)

Block-wise	Progressive	Ocs	Metric	Calibration	group size	WikiText2 ppl	C4 ppl
-	-	-	-	-	32	2.0e5	1.0e6
✓	-	-	-	-	32	1441.9	4592.8
✓	✓	-	-	-	32	42.2	120.2
✓	✓	✓	Kurtosis	WikiText2	32	41.8	127.4
✓	✓	✓	$w_{\max}$	WikiText2	32	36.75	97.66
✓	✓	✓	$\\|\mathbf{x}\\|_{2}\cdot\|\mathbf{w}\|$	WikiText2	32	32.48	79.95

Appendix

Table 6: Ablation study on LLaMA-3.2-1B under the W2A2 setting. All models are trained for one epoch. “-” indicates the component is disabled.

Model	Shape $(N,K)$	W2A2 ( $\mu$ s)	BF16 ( $\mu$ s)
Llama3.2-3B	(1024, 3072)	8.93	6.45
	(3072, 3072)	8.64	16.04
	(3072, 8192)	8.70	32.80
	(8192, 3072)	8.94	20.76
Llama3-8B	(1024, 4096)	8.58	6.87
	(4096, 4096)	8.64	29.73

Appendix

Table 7: Latency comparison for weight matrices appearing in the decode layers of LLaMA models.

Sequence Length	BF16 (tokens/s)	W2A2 (tokens/s)
64	49.02	76.59
128	48.85	75.13
256	48.59	74.97
512	47.87	74.17

Appendix

Table 8: Throughput comparison on Llama3-8b

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 388 papers

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization
Jun 9, 2026Haoyu Wang, Xingyu Yu, Haiyan Zhao +2LLM Quantization Quantization-Aware Training
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.
pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training
Feb 26, 2026Wenzheng Zhang, Bingzheng Liu, Yang Hu +3LLM Quantization Low-Bit Quantization
School of Computer Science, Peking University · College of Future Information Technology, Fudan University · Center for Information Research, Academy of Military Sciences +2
CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs
Apr 29, 2026Zhe Ding, Su Pan, Duowei PanMixed-Precision Quantization LLM Quantization
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA

Jun 9, 2026J/K move · Enter open · S save

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

Haoyu Wang, Xingyu Yu, Haiyan Zhao, Fengxiang Wang+1

Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.

LLM Quantization Quantization-Aware Training Vector Quantization Weight-Only Quantization Neural Network Quantization Data-Efficient Learning

Open paper AskPDF arXiv

Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%--10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.

Abstract

LLM Quantization

Explore similar work

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs