Recent advances in post-training quantization for large language models (LLMs) have demonstrated that low-bit quantization methods can maintain most of the original model performance. Despite such successes, 1-bit quantization remains particularly challenging. A common strategy in 1-bit quantization is to determine binary weights by matching full-precision parameters, following a weight-driven criterion. However, this objective is not directly aligned with the quantized model's objective, which is to preserve the model's output behavior under the impact of quantization. A natural alternative is to adopt output-driven criteria that minimize discrepancies in model outputs using calibration data. Surprisingly, naive output-driven approaches often perform even worse in the 1-bit regime. In this paper, we show that this failure arises from two fundamental issues: error accumulation across layers and, more critically, anisotropic distortion of the representation space. Based on this observation, we introduce a geometry-preserving output-alignment framework for 1-bit LLM quantization. First, we formulate output alignment as a constrained optimization problem that improves reconstruction while explicitly controlling changes in token similarity, preventing reconstruction gains from being achieved at the expense of representation geometry. Second, we propose a geometry-aware salient-mask selection strategy that identifies quantization configurations according to their ability to preserve token relations, rather than relying solely on reconstruction error. These components enable effective output-driven optimization while retaining the structural information lost by naive alignment. Extensive experiments across multiple LLM families and downstream tasks demonstrate consistent improvements over existing 1-bit PTQ methods.
Figures & tables
Figure 1: Reconstruction error alone does not characterize the functional distortion introduced by extreme quantization. (a) Output-driven optimization allocates its reconstruction improvement unevenly across representation dimensions, disproportionately benefiting high-magnitude channels. (b) Despite reducing reconstruction error, it induces increasing distortion of the pairwise geometry between token representations. (c) We isolate this effect by perturbing the first-block representations with equal-magnitude errors while varying only their direction. Although all perturbations have identical reconstruction error, those that induce larger token-geometry distortion lead to higher final perplexity. Together, these observations suggest that preserving representation direction and token relationships captures information missed by reconstruction MSE.
Dataset
Method
Use
OPT
OPT
LLaMA
LLaMA
OA
Bits
1.3B
2.7B
6.7B
Bits
LLaMA-2 7B
LLaMA-2 13B
LLaMA-3 8B
C4
Full Precision
–
16
16.07
14.34
12.71
16
7.26
6.73
9.45
PB-LLM
{\color[rgb]{1,0,0}\times}
1.7
168.12
222.15
104.78
1.7
80.69
184.67
104.15
BiLLM
{\color[rgb]{1,0,0}\times}
1.11
64.14
44.77
42.13
1.08
39.38
25.87
61.04
ARB-X
{\color[rgb]{0,0.75,0.16}\checkmark}
1.11
47.60
34.97
22.54
1.08
28.02
19.82
41.86
OAC
{\color[rgb]{1,0,0}\times}
1.11
–
–
–
1.08
21.64
15.25
–
Table 1: Perplexity (↓) results of OPT and LLaMA models on C4, WikiText2, and PTB. Separate weight-bit columns are used for each model family. Use OA denotes methods that use an output-alignment objective for quantization.
Model
Method
Bits
Lambada
PIQA
OBQA
WinoGrande
ARC-E
ARC-C
HellaSwag
COPA
Avg. †
LLaMA-2-7B
ARB-RC
1.08
51.66
64.85
19.40
59.51
50.97
24.32
36.79
71.00
47.31
DBELL
1.08
41.63
64.85
17.60
56.91
50.38
23.98
35.65
63.00
44.25
Ours
1.08
58.53 (+6.87)
67.46 (+2.61)
20.80 (+1.40)
60.14 (+0.63)
53.41 (+2.44)
25.60 (+1.28)
38.91 (+2.12)
74.00 (+3.00)
49.86 (+2.55)
LLaMA-2-13B
ARB-RC
1.08
66.02
69.59
23.20
61.48
61.28
29.18
38.73
81.00
53.81
DBELL
1.08
55.75
68.44
20.80
61.64
60.10
28.16
39.08
78.00
51.50
Ours
1.08
69.28 (+3.26)
71.71 (+2.12)
25.00 (+1.80)
66.22 (+4.74)
64.44 (+3.16)
30.89 (+1.71)
43.09 (+4.36)
82.00 (+1.00)
56.58 (+2.77)
Table 2: Zero-shot accuracy (%) on QA reasoning benchmarks. Green values denote the improvement of our method over ARB-RC in percentage points. † The average is computed over all tasks.
Dataset
Metric
Method
Block Size
Qwen2
Qwen2.5
Qwen3
1.5B
7B
1.5B
7B
8B
C4
PPL ( ↓ )
ARB-RC
128
109.83
22.63
128.77
24.32
32.71
DBellQuant
128
607.06
29.03
1428.88
29.52
50.88
Ours
128
74.83 ( ↓ 35.00)
20.54 ( ↓ 2.09)
64.33 ( ↓ 64.44)
20.95 ( ↓ 3.37)
27.68 ( ↓ 5.03)
WikiText2
PPL ( ↓ )
ARB-RC
128
89.59
16.99
107.92
17.08
26.03
DBellQuant
128
542.06
28.52
2416.90
25.10
58.16
Table 3: Perplexity ( ↓ ) on C4, WikiText2, and PTB, and average zero-shot reasoning accuracy ( ↑ ) for Qwen models. All methods use a block size of 128. Green annotations report the absolute improvement over ARB-RC.
Optimization
Qwen-2.5-1.5B
LLaMA-2-7B
LQ
ΔSReg≥0
Mask
C4
Wiki
C4
Wiki
{\color[rgb]{1,0,0}\times}
{\color[rgb]{1,0,0}\times}
{\color[rgb]{1,0,0}\times}
128.77
107.92
20.56
16.36
{\color[rgb]{0,0.75,0.16}\checkmark}
{\color[rgb]{1,0,0}\times}
{\color[rgb]{1,0,0}\times}
115.69
98.18
29.12
26.24
{\color[rgb]{1,0,0}\times}
{\color[rgb]{0,0.75,0.16}\checkmark}
{\color[rgb]{1,0,0}\times}
119.14
99.45
19.97
15.66
{\color[rgb]{1,0,0}\times}
{\color[rgb]{1,0,0}\times}
{\color[rgb]{0,0.75,0.16}\checkmark}
86.00
73.07
19.91
16.17
{\color[rgb]{0,0.75,0.16}\checkmark}
{\color[rgb]{0,0.75,0.16}\checkmark}
{\color[rgb]{1,0,0}\times}
109.53
90.58
18.94
15.20
Table 4: Ablation study about the contribution of each component, using a fixed random seed.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: The difference between the token similarity matrix between full-precision model and quantized model, with and without SReg constraint, demonstrating the effectiveness of the proposed method.
Figure 3: Comparison of attention scores with and without the token-geometry preservation constraint against the full-precision model. Attention scores produced with the SReg constraint more closely match those of the pretrained model than those obtained without SReg constraint, demonstrating the effectiveness of the proposed method.
Model
ARB-X
ARB-RC
Ours
OPT-6.7B
87m
65m
93m
Llama-2-7B
91m
54m
76m
Llama-2-13B
147m
100m
119m
Appendix
Table 5: Quantization time comparison between 1-bit LLM methods and ours across different models.
Model
Prefill
Decode
FP16
2.83
0.87
ARB-RC
11.44
14.15
Ours
12.42
14.52
Appendix
Table 6: Inference throughput (tokens/s) on the Intel Xeon Gold 6548Y+ CPU Wang et al. (2023) . Higher is better.
Size
Method
PPL (↓)
C4
WikiText2
32
ARB-RC
109.45
95.08
Ours
83.44
76.56
128
ARB-RC
109.83
89.59
Ours
74.83
68.45
512
ARB-RC
127.84
89.13
Appendix
Table 7: Ablation study on the calibration set size for Qwen-2-1.5B.
Model
β
PPL (↓)
C4
WikiText2
OPT-6.7B
0
16.24
14.56
0.5
16.21
14.53
1
16.18
14.59
Appendix
Table 8: Ablation study for the hyper-parameter β .
Algorithm 1 Our-RC with Second-Order Relational Projection
Large language models (LLMs) have achieved remarkable progress, yet their massive storage and memory-bandwidth demands still hinder efficient deployment. Weight binarization is a promising solution, but existing binarization-based post-training quantization (PTQ) methods usually far exceed the nominal 1-bit storage target due to hidden overhead. To address this gap, we propose All for 1-Bit (AF1), a genuine 1-bit PTQ framework for LLMs. AF1 comprises two complementary components: (1) Null-space-Aware Binary Factorization (NABF) for improving binary reconstruction through Hessian-aware surrogate reparameterization, null-space-aware binary factorization, and scale-only global reconstruction; and (2) Hierarchical Shapley Allocation (HiSA) for assigning structural capacity using hierarchical Shapley sensitivity. Together, they preserve model accuracy under a strict 1.0-BPW budget in the PTQ setting. Experiments on LLaMA, Qwen, and Gemma families show that AF1 consistently outperforms existing binarization-based PTQ methods in perplexity and zero-shot accuracy. Compared with BF16, AF1 achieves an average 2.5 times inference speedup and over 90% memory reduction across evaluated models, providing a practical path toward deployable genuine 1-bit compression for LLMs. The code for reproducibility is available at https://github.com/Kishon-zzx/AF1.
Zhixiong Zhao, Zukang Xu, Guangyu Sun +2
Houmo AI · School of Integrated Circuits, Peking University
Large language models (LLMs) have shown remarkable performance in various domains, but they are constrained by massive computational and storage costs. Quantization, an effective technique for compressing models to fit resource-limited devices while preserving generative quality, encompasses two primary methods: quantization aware training (QAT) and post-training quantization (PTQ). QAT involves additional retraining or fine-tuning, thus inevitably resulting in high training cost and making it unsuitable for LLMs. Consequently, PTQ has become the research hotspot in recent quantization methods. However, existing PTQ methods usually rely on various complex computation procedures and suffer from considerable performance degradation under low-bit quantization settings. To alleviate the above issues, we propose a simple and effective post-training quantization paradigm for LLMs, named SEPTQ. Specifically, SEPTQ first calculates the importance score for each element in the weight matrix and determines the quantization locations in a static global manner. Then it utilizes the mask matrix which represents the important locations to quantize and update the associated weights column-by-column until the appropriate quantized weight matrix is obtained. Compared with previous methods, SEPTQ simplifies the post-training quantization procedure into only two steps, and considers the effectiveness and efficiency simultaneously. Experimental results on various datasets across a suite of models ranging from millions to billions in different quantization bit-levels demonstrate that SEPTQ significantly outperforms other strong baselines, especially in low-bit quantization scenarios.
Han Liu, Haotian Gao, Xiaotong Zhang +5
Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China +1
Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by quantization noise from heavy-tailed outlier channels and error accumulation across layers. To address these issues, we present Bit-by-Bit, a progressive QAT framework with outlier channel splitting. Our approach integrates three key components: (1) block-wise progressive training that reduces precision stage by stage, ensuring stable initialization for low-bit optimization; (2) nested structure of integer quantization grids to enable a "train once, deploy any precision" paradigm, allowing a single model to support multiple bit-widths without retraining; (3) rounding-aware outlier channel splitting, which mitigates quantization error while acting as an identity transform that preserves the quantized outputs. Furthermore, we follow microscaling groups with E4M3 scales, capturing dynamic activation ranges in alignment with OCP/NVIDIA standards. To address the lack of efficient 2-bit kernels, we developed custom operators for both W2A2 and W2A16 configurations, achieving up to 11× speedup over BF16. Under W2A2 settings, Bit-by-Bit significantly outperforms baselines like BitDistiller and EfficientQAT on both Llama2/3, achieving a loss of only 2.25 WikiText2 PPL compared to full-precision models.
Binxing Xu, Hao Gu, Lujun Li +8
Zhejiang University · Hong Kong University of Science and Technology · City University of Hong Kong