Weight binarization has emerged as a promising strategy to reduce the complexity of large language models (LLMs). Existing approaches fall into post-training binarization, which is simple but causes severe performance loss, and training-aware methods, which depend on full-precision latent weights, adding complexity and limiting efficiency. We propose a novel framework that represents LLMs with multi-kernel Boolean parameters and, for the first time, enables direct finetuning LMMs in the Boolean domain, eliminating the need for latent weights. This enhances representational capacity and dramatically reduces complexity during both finetuning and inference. Extensive experiments across diverse LLMs show our method outperforms recent ultra low-bit quantization and binarization techniques.
Figures & tables
Figure 1: Finetuning opt models ( Zhang et al., 2022 ) using our 3 Boolean kernels ( ), compared to optq ( Frantar et al., 2023 ) ( ), which quantizes the models to 3 bits, and the fp 16 baseline ( ) on the C4 dataset.
Method
Train from Scratch
Post-training Binarization
Finetune from FP Model
Calibration Data
Weight Update
Multiple Binary Bases
Higher-bit Salient Weights
b it n et ( Wang et al., 2023 )
✓
✗
✗
NA
FP latent-weights
✗
✗
b i llm ( Huang et al., 2024 )
✗
✓
✗
✓
NA
✓
✓
pb - llm ( Yuan et al., 2024 )
✗
✓
✗
✓
NA
✗
✓
stbllm ( Dong et al., 2025 )
✗
✓
✗
✓
NA
✓
✓
arb - llm ( Li et al., 2025 )
✗
✓
✗
✓
NA
✓
✓
b it s tack ( Wang et al., 2025 )
✗
✓
✗
✗
NA
✓
✗
Table 1: A summary of sota binarization methods for llms compared to our method.
Figure 2: Illustration of svid .
Figure 3: The computation of a linear layer approximated using multi kernels of Boolean.
Figure 4: Illustration of successive extractions of Boolean kernels from a given fp weight matrix.
Figure 5: Normalized L1 norm difference between the approximated weights at initialization and after finetuning against the fp weights ( ∥Wapprox−WFP∥1/∥WFP∥1 ) , and the final results.
Figure 6: The progression of training losses, number of flips, and perplexity of the resulting models ( opt -125 m ) is examined with respect to the optimization of different kernel configurations.
Model
Method
Wbits
Perplexity ( ↓ )
Zero-shot Accuracy ( ↑ )
Wiki2
C4
BoolQ
PIQA
Hella.
WinoG.
ARC-e
ARC-c
Average
opt -1.3 b
fp 16
16
14.62
14.72
57.82
72.42
53.70
59.51
50.97
29.52
53.99
pb - llm
1.7
272.83
175.42
62.17
54.24
27.25
50.27
27.98
23.72
40.94
b i llm
1.11
69.45
63.92
61.92
59.52
33.81
49.32
34.38
22.35
43.55
o ne b it
1
20.36
20.76
57.85
66.53
39.21
54.61
42.80
23.97
47.50
m o s
1
18.45
18.83
60.34
68.66
41.99
53.99
44.87
26.19
49.34
Table 2: Perplexity and zero-shot accuracy results of Float16, quantized and binarized llms .
OPT Model
WBits
Wiki2
C4
125M
350M
1.3B
2.7B
6.7B
125M
350M
1.3B
2.7B
6.7B
full-precision
16
27.65
22.00
14.63
12.47
10.86
26.56
22.59
16.07
14.34
12.71
rtn ( Yao et al., 2022 ; Dettmers et al., 2022 )
3
1.3e3
64.57
1.3e4
1.6e4
5.8e3
834
55.49
5.2e3
1.1e4
5.3e3
optq ( Frantar et al., 2023 )
3
53.85
33.79
20.97
16.88
14.86
42.41
31.33
21.63
18.17
17.14
mbok [Ours]
3 × 1
29.10
23.12
15.30
13.09
11.03
28.62
22.10
15.68
14.00
12.33
Table 3: opt perplexity results ( lower is better ) on WikiText2 and C4. The results of fp , rount-to-nearest ( rtn ) and optq are taken from ( Frantar et al., 2023 ) .
Figure 7: Comparions between our method and latent-weight approaches.
Figure 11
Figure 9: opt -125 m performance w.r.t. bit budget.
Figure 10: Estimated memory for finetuning for weights ( ) and optimizer states ( ).
weight size
fp 16
quip# ( Tseng et al., 2024a )
qtip ( Tseng et al., 2024b )
mbok (Ours)
5120×5120
0.16540
0.62260 ( 0.27× )
1.96368 ( 0.08× )
0.05074 ( 3.25× )
5120×13824
0.42830
0.62836 ( 0.68× )
5.23681 ( 0.09× )
0.05098 ( 8.40× )
13824×5120
0.43411
0.62840 ( 0.69× )
5.21193 ( 0.08× )
0.04987 ( 8.70× )
Table 4: Measured latency (ms) of linear layers in ll a ma -13 b , with values in parentheses denoting speed-up relative to the fp 16 baseline.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
a
x
¬x
δ(x→¬x)
f(a,x)
f(a,¬x)
δf(x→¬x)
f′(x)
true
true
false
false
false
true
true
false
true
false
true
true
true
false
false
false
false
true
false
false
true
false
false
true
false
false
true
true
false
true
true
true
Appendix
Table 5: Variation truth table of f(x)=xor(a,x) , a,x∈B .
Algorithm 2 Python code of xor linear layer
Algorithm 3 Python code of the backpropagation logic of xor linear layer
Algorithm 4 Backpropagation logic with Boolean received backpropagation
Algorithm 5 Backpropagation logic with real received backpropagation
Algorithm 6 Python code of Boolean optimizer
Algorithm 7 Python code of svid approximation of a fp matrix.
Algorithm 8 Python code of successively extracts kernels from fp matrix using svid .
Figure 11: The training convergence of Lis , and Llogits , measured by Forward kl , and the final results with respect to the choice of Dlogits .
Model
Method
Wbits
Perplexity ( ↓ )
Zero-shot Accuracy ( ↑ )
Wiki2
C4
BoolQ
PIQA
Hella.
WinoG.
ARC-e
ARC-c
Average
opt -1.3 b
mbok (2 kernels)
2 × 1
16.13
16.61
58.53
70.67
48.11
56.75
48.19
27.90
51.69
mbok (3 kernels)
3 × 1
15.30
15.68
60.64
70.78
50.71
56.83
48.82
28.49
52.71
mbok (4 kernels)
4 × 1
14.83
14.92
60.95
70.85
51.02
56.85
49.13
29.24
53.01
ll a ma -7 b
mbok (2 kernels)
2 × 1
6.83
8.53
69.20
74.32
64.80
60.30
49.05
34.90
58.76
mbok (3 kernels)
3 × 1
6.20
7.76
67.89
76.15
68.91
63.30
48.94
37.62
60.47
Appendix
Table 6: Perplexity and zero-shot accuracy results of our mbok method with different number of kernels.
Model
Method
Wbits
Perplexity ( ↓ )
Zero-shot Accuracy ( ↑ )
Wiki2
C4
BoolQ
PIQA
Hella.
WinoG.
ARC-e
ARC-c
Average
ll a ma 2-7 b
fp 16
16
5.47
6.97
71.10
76.88
72.94
67.09
53.58
40.61
63.70
pb - llm
1.7
76.75
85.92
62.17
52.82
26.87
50.11
26.89
24.31
40.53
b i llm
1.11
27.72
36.34
62.14
59.19
35.18
53.11
34.22
26.54
45.06
o ne b it
1
8.60
10.74
63.06
70.40
54.24
56.67
40.82
29.35
52.42
m o s
1
7.88
9.75
65.02
71.55
59.41
56.18
41.84
30.03
54.01
Appendix
Table 7: Perplexity and zero-shot accuracy results of Float16, quantized and binarized ll a ma 2 models.
Prompt:
What are the capitals of China, US and Russia?
FP16:
The capital of China is Beijing. The capital of the United States is Washington D.C. and the capital of Russia is Moscow.
MBOK:
The capital of China is Beijing. The capital of the US is Washington D.C. and the capital of Russia is Moscow.
Prompt:
I have a friend who lives in London and I would like to visit him. How would I go there from Paris?
FP16:
You can take the Eurostar from Gare du Nord in Paris to St. Pancras Station in London. The trip takes about 2 hours and 15 minutes.
MBOK:
The fastest way to get from Paris to London is to take the Eurostar train. The Eurostar departs from the Gare du Nord station in Paris and arrives at the St. Pancras International station in London. The trip takes about 2 hours and 15 minutes.
Prompt:
What is Fete du Travail?
Appendix
Table 8: Comparison of generation quality ( non-curated ) on ll a ma -13 b models with fp 16 and mbok using three kernels.
bit-width
method
opt -125 m
opt -350 m
opt -1.3 b
q u ip ( Chee et al., 2023 )
34.22
25.19
16.21
s hift a dd llm ( You et al., 2024 )
31.29
24.24
21.53
2
mbok [Ours]
29.10
23.12
15.03
q u ip ( Chee et al., 2023 )
347.40
672.30
41.64
s hift a dd llm ( You et al., 2024 )
51.15
40.24
29.03
3
mbok [Our]
28.60
24.54
16.13
Appendix
Table 9: Comparisons with q u ip , s hift a dd llm using opt models.
method
Wiki2 (↓)
ARc-e (↑)
ARC-c (↑)
PIQA (↑)
Hella. (↑)
WinoG. (↑)
awq ( Lin et al., 2024 )
1.8e5
26.3
26.7
50.9
26.5
49.3
b it s tack ( Wang et al., 2025 )
29.93
32.3
25.6
62.4
42.8
53.6
db - llm ( Chen et al., 2024 )
7.23
45.2
33.5
73.1
61.9
61.7
mbok [Ours]
6.87
44.8
34.2
75.0
65.6
61.7
Appendix
Table 10: Comparisons with awq , b it s tack, db - llm using ll a ma 2-7 b with 2-bit setting.
Figure 12: Study on the effect of using knowledge distillation on opt -125 m with 2 Boolean kernels.
Figure 13: Histogram of output-scaling values for the first linear layer of opt -125 m with four kernels, shown at initialization and after finetuning. The Kolmogorov–Smirnov (KS) statistic is also reported to quantify the difference between the scaling-value distributions before and after finetuning.
Figure 14: The training convergences of mbok using 3 kernels with opt models.
Figure 15: Study on the effect of using our successive svid strategy and random initialization on opt -125 m with 2 Boolean kernels.
method
Wiki2 ( ↓ )
C4 ( ↓ )
quip# ( Tseng et al., 2024a )
6.86
8.36
qtip ( Tseng et al., 2024b )
6.52
7.99
mbok [Ours]
6.83
8.53
Appendix
Table 11: Perplexity comparison with SOTA vector quantization methods using ll a ma -7 b .
weight size
fp 16
quip# ( Tseng et al., 2024a )
qtip ( Tseng et al., 2024b )
mbok (Ours)
4096×4096
0.10697
0.46196 ( 0.23× )
1.37137 ( 0.08× )
0.04989 ( 2.14× )
4096×11008
0.27935
0.55526 ( 0.50× )
3.13633 ( 0.09× )
0.05136 ( 5.44× )
11008×4096
0.27664
0.55988 ( 0.49× )
3.16067 ( 0.09× )
0.05117 ( 5.41× )
Appendix
Table 12: Measured latency (ms) of linear layers in ll a ma -7 b , with values in parentheses denoting speed-up relative to the fp 16 baseline.
weight size
fp 16
quip# ( Tseng et al., 2024a )
qtip ( Tseng et al., 2024b )
mbok (Ours)
5120×5120
0.16540
0.62260 ( 0.27× )
1.96368 ( 0.08× )
0.05074 ( 3.25× )
5120×13824
0.42830
0.62836 ( 0.68× )
5.23681 ( 0.09× )
0.05098 ( 8.40× )
13824×5120
0.43411
0.62840 ( 0.69× )
5.21193 ( 0.08× )
0.04987 ( 8.70× )
Appendix
Table 13: Measured latency (ms) of linear layers in ll a ma -13 b , with values in parentheses denoting speed-up relative to the fp 16 baseline.
Deploying large language models (LLMs) in resource-constrained environments is hindered by heavy computational and memory requirements. We present LBLLM, a lightweight binarization framework that achieves effective W(1+1)A4 quantization through a novel three-stage quantization strategy. The framework proceeds as follows: (1) initialize a high-quality quantized model via PTQ; (2) quantize binarized weights, group-wise bitmaps, and quantization parameters through layer-wise distillation while keeping activations in full precision; and (3) training learnable activation quantization factors to dynamically quantize activations to 4 bits. This decoupled design mitigates interference between weight and activation quantization, yielding greater training stability and better inference accuracy. LBLLM, trained only using 0.016B tokens with a single GPU, surpasses existing state-of-the-art binarization methods on W2A4 quantization settings across tasks of language modeling, commonsense QA, and language understanding. These results demonstrate that extreme low-bit quantization of LLMs can be both practical and highly effective without introducing any extra high-precision channels or rotational matrices commonly used in recent PTQ-based works, offering a promising path toward efficient LLM deployment in resource-limited situations.
Siqing Song, Chuang Wang, Yong Lang +2
MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Central Media Technology Institute, Huawei
Large language models (LLMs) have achieved remarkable progress, yet their massive storage and memory-bandwidth demands still hinder efficient deployment. Weight binarization is a promising solution, but existing binarization-based post-training quantization (PTQ) methods usually far exceed the nominal 1-bit storage target due to hidden overhead. To address this gap, we propose All for 1-Bit (AF1), a genuine 1-bit PTQ framework for LLMs. AF1 comprises two complementary components: (1) Null-space-Aware Binary Factorization (NABF) for improving binary reconstruction through Hessian-aware surrogate reparameterization, null-space-aware binary factorization, and scale-only global reconstruction; and (2) Hierarchical Shapley Allocation (HiSA) for assigning structural capacity using hierarchical Shapley sensitivity. Together, they preserve model accuracy under a strict 1.0-BPW budget in the PTQ setting. Experiments on LLaMA, Qwen, and Gemma families show that AF1 consistently outperforms existing binarization-based PTQ methods in perplexity and zero-shot accuracy. Compared with BF16, AF1 achieves an average 2.5 times inference speedup and over 90% memory reduction across evaluated models, providing a practical path toward deployable genuine 1-bit compression for LLMs. The code for reproducibility is available at https://github.com/Kishon-zzx/AF1.
Zhixiong Zhao, Zukang Xu, Guangyu Sun +2
Houmo AI · School of Integrated Circuits, Peking University
Large language models (LLMs) have achieved remarkable performance on Natural Language Processing (NLP) tasks, but they are hindered by high computational costs and memory requirements. Ternarization, an extreme form of quantization, offers a solution by reducing memory usage and enabling energy-efficient floating-point additions. However, applying ternarization to LLMs faces challenges stemming from outliers in both weights and activations. In this work, observing asymmetric outliers and non-zero means in weights, we introduce Dual Learnable Ternarization (DLT), which enables both scales and shifts to be learnable. We also propose Outlier-Friendly Feature Knowledge Distillation (OFF) to recover the information lost in extremely low-bit quantization. The proposed OFF can incorporate semantic information and is insensitive to outliers. At the core of OFF is maximizing the mutual information between features in ternarized and floating-point models using cosine similarity. Extensive experiments demonstrate that our TernaryLLM surpasses previous low-bit quantization methods on the standard text generation and zero-shot benchmarks for different LLM families. Specifically, for one of the most powerful open-source models, LLaMA-3, our approach (W1.58A16) outperforms the previous state-of-the-art method (W2A16) by 5.8 in terms of perplexity on C4 and by 8.2% in terms of average accuracy on zero-shot tasks.
Tianqi Chen, Zhe Li, Weixiang Xu +6
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Advanced Micro Devices, Inc., Beijing, China +1