BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design

Yifu Ding; Xianglong Liu; Shenghao Jin; Jinyang Guo; Jiwen Lu

cs.LGApr 5, 2026

BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design

Authors: Yifu Ding, Xianglong Liu, Shenghao Jin, Jinyang Guo, Jiwen Lu

Organizations: School of Computer Science and Engineering, Beihang University, Beijing, China. · Institute of Artificial Intelligence, Beihang University, Beijing, China. · Department of Automation, Tsinghua University, Beijing, China.

Abstract

Ultra low-bit quantization brings substantial efficiency for Transformer-based models, but the accuracy degradation and limited GPU support hinder its wide usage. In this paper, we analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme, which projects tiny values to zero and preserves the accuracy of extremely low-bit models. For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a Magnitude-Alignment Projection Factor to enable stable and fast convergence. For inference, we develop a BWTA MatMul CUDA kernel with instruction-level parallel bit-packing and comprehensive binary/ternary MatMul implementations for both linear and attention operators, allowing seamless integration across Transformer architectures. Experiments show that BWTA approaches full-precision performance for BERT, with an average 3.5% drop on GLUE and less than 2% drop on five tasks, and achieves comparable perplexity and accuracy for LLMs. In efficiency, it delivers 16 to 24 times kernel-level speedup over FP16 on NVIDIA GPUs, and 216 to 330 tokens/s end-to-end prefill speedup with lower memory footprint on LLMs. As an algorithm-hardware co-design, BWTA demonstrates practical, low-latency ultra-low-bit inference without sacrificing model quality.

Figures & tables

Fig. 1: The overview of the Binary Weight & Ternary Activation (BWTA) framework. The left is the training algorithm named Smooth Multi-stage Quantization , designed for ultra-low bit Transformers with ternary activation, including the (1) binary/ternary definitions to build a BWTA model, (2) smooth strategies for stable convergence, and (3) distillation for fast training. The right is the full stack GPU support for the custom BWTA MatMul Kernel , including the (1) binarized weight for storage reduction, (2) instruction-level parallel bitpack for runtime quantization, and (3) low-bit BWTA MatMul kernels for fast computation.

Fig. 2: Histograms for binary/ternary activation in Self-Attention structure before and after quantized. (a) shows the value matrix $V_{\mathrm{bitwise}}$ and the activation $A_{\mathrm{bitwise}}$ in the linear layer following the multiplication of attention score and value trained by bitwise degradation strategy, while (b) shows the distribution of $V_{\mathrm{levelwise}}$ and $A_{\mathrm{levelwise}}$ by levelwise strategy.

Fig. 3: Illustrations for two practices and the issues of projecting

0_{fp}

to integer space.

Fig. 4: (a) The illustration of the bit/levelwise multi-stage quantization strategies. (b) The grid points shift with projection factors (blue arrows), and convergent again in the next stage with fewer integers (gray arrows).

Fig. 5: Data distribution in threads for weight, activation and mma results using m8n8k128 data layout. Binary weight is packed and stored before inference, and we pick $8\times 128$ elements for each SIMT, with four copies stored in 128 threads. Activation is packed into ternary during runtime with four slices each SIMT without repeated copies.

Fig. 6: Instruction-level parallel bitpack from 32 FP16 (Half) to (a) binary and (b) ternary representations.

Fig. 7: The difference of the algorithm (upper-level) perspective and the hardware (lower-level) perspective in BWTA quantization scheme.

Fig. 8: The arithmetic rules of the three layers. ˆ and & means mma instructions with xor and mma instructions with and , respectively. We dub them as mma.and and mma.xor in the main text.

Fig. 9: Comparison of level/bitwise multi-stage quantization strategies (start with

L_{0}=4

and

k_{0}=3

as an example).

Fig. 10: Comparison of different re-initialization strategies for scaling factor.

Quant	#Bits	Size ${}_{\text{ (MB)}}$	MNLI ${}_{\text{-m/mm}}$	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	Avg.
Full Precision	32/32	418	84.9/85.5	91.4	88.6	93.2	59.7	89.8	86.2	72.2	83.9
Full Precision	32/32	418	84.9/85.5*	91.4*	92.1	93.2	59.7	90.1	86.3	72.2	82.3
TernaryBERT	2/8	28.0	83.3/83.3*	90.1*	90.0	92.9	47.8	84.3	82.6	68.4	77.8
TernaryBERT	2/2	28.0	40.3/40.0*	63.1*	50.0	87.5	20.6	72.5	72.0	47.2	58.3
TernaryBERT	2/1	28.0	32.7/33.0*	74.1*	50.9	80.3	6.5	10.3	71.5	53.4	45.5
BinaryBERT	1/4	16.5	83.9/84.2*	91.2*	91.4	93.7	53.3	88.6	86.0	71.5	80.8

TABLE I: Comparison of BERT quantization methods. All results are trained with data augmentation except BiPFT. “–” means the results are not officially reported. Experiments without data augmentation are provided in Appendix VIII . *Data augmentation is not needed for MNLI and QQP.

Model	Method	W bit /A bit	Wikitext2	C4	CmQA
LLaMA-2-7B	Full Precision	16/16	5.47	–	68.9
	GPTQ	2/16	20.85	–	37.5
	PB-LLM	2/16	20.37	–	40.0
	BiLLM	1.1/16	32.48	–	42.3
OPT-2.7B	RTN	2/16	9505.76	–	–
	GPTQ	2/16	61.59	–	–

TABLE II: Perplexity on WikiText2 and C4, and average accuracy on CommensenseQA for various models and quantization methods. “–” means the result is not officially reported. For all BWN and BWTA results, we replace 30% layers to ultra-low bitwidth based on Bitnet [ 27 ] .

Kernel	W bit /A bit	Time ( $\mu$ s)	FLOPs ( $\times 10^{3}$ )
torch.nn.functional	16/16	10.74	3.05
bnb.nn.Linear4bit	4/16	60.33	0.543
bitlinear_int8xint2 (Bitnet)	2/8	9.08*	3.609
Binary Linear (Supported)	1/1	7.29	4.495
Binary A x V (Supported)	1/1	6.97	4.701
BWTA (Ours)	1/1.5	8.56	3.828

TABLE III: Kernel-level benchmark. All test cases are repeated 50 times with 256

\times

256 multiplying with 256

\times

1 on NVIDIA H800 GPU (96GB VRAM).

Fig. 11: Overall time comparisons for each step in the GEMM kernels with four typical shapes.

Models	W bit /A bit	Storage (GB)	Prefill Phase (SeqLen=2k)				Decode Phase (BS=1)
Models	W bit /A bit	Storage (GB)	BS	Time (s)	Tokens/s	Mem. (GB)	G. Len.	Time (s)	Tokens/s
Llama-3.2-3B	4/16 (bnb)	2.240	1	0.072	28267	15.59	50	1.317	43.4
			4	0.172	11924	54.94	100	2.563	39.0
			8	–	–	OOM	150	3.507	44.2
Gemma-2B	16/16	5.012	1	0.063	32492	20.75	50	0.512	111.6
			4	0.116	17694	64.51	100	1.146	87.3

TABLE IV: End-to-end benchmarks on various models with

\leq

3B parameters. The input token length of prefill phase is uniformly set to

2k

, and the batch size of decode phase is 1. “G. Len.” means the generated token length during decoding.

Fig. 12: The quantized activation of (a) bitwise and (b) levelwise degradation.

Fig. 13: Task loss surface of quantized models and the full-precision counterpart.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	BS	Seqlen	aten::*mm		aten::softmax		aten::*norm		aten::*lu
Model	BS	Seqlen	Time (ms)	Time (%)	Time (ms)	Time (%)	Time (ms)	Time (%)	Time (ms)	Time (%)
Llama-3.2-3B	1	512	11.095	5.20%	1.537	0.72%	6.393	3.00%	0.346	0.16%
		1024	21.057	11.42%	6.308	3.42%	5.961	3.23%	0.568	0.31%
		2048	44.245	9.55%	25.092	5.42%	7.331	1.58%	1.178	0.25%
	4	512	38.365	14.21%	6.141	2.27%	7.387	2.74%	1.181	0.44%
		1024	78.264	12.43%	23.104	3.67%	13.196	2.10%	2.405	0.38%

Appendix

TABLE V: Operator-wise latency breakdown during prefill phase (device-side CUDA time). Results report the total elapsed time for each operator category.

Model	Seqlen	aten::*mm		aten::softmax		aten::*norm		aten::*lu
Model	Seqlen	Time (ms)	Time (%)	Time (ms)	Time (%)	Time (ms)	Time (%)	Time (ms)	Time (%)
Llama-3.2-3B	128	6.336	9.40%	0.129	0.19%	4.154	6.16%	0.120	0.18%
	256	6.333	9.68%	0.129	0.20%	4.071	6.22%	0.119	0.18%
	512	6.358	9.69%	0.130	0.20%	3.959	6.03%	0.121	0.18%
	1024	6.356	9.47%	0.130	0.19%	4.072	6.07%	0.121	0.18%
bert-base	128	1.439	8.64%	0.079	0.48%	0.304	1.83%	0.047	0.28%

Appendix

TABLE VI: Operator-wise latency breakdown during decode phase (device-side CUDA time). Results report the total elapsed time for each operator category.

Fig. 14: The curves of scaling factors and their gradients.

Strategy	QNLI	SST-2	STS-B	CoLA	MRPC
Bitwise (%)	34.2	28.1	30.2	31.5	37.0
Levelwise (%)	11.6	14.4	13.7	10.9	19.8

Appendix

TABLE VII: Scaling factors not converged within the pre-defined training epochs.

Quant	#Bits	Size ${}_{\text{ (MB)}}$	MNLI ${}_{\text{-m/mm}}$	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	Avg.
Full Precision	32/32	418	84.9/85.5	91.4	88.6	93.2	59.7	89.8	86.2	72.2	83.9
Q2BERT	2/8	43.0	47.2/47.3	67.0	61.3	80.6	0	4.4	68.4	52.7	47.7
Q-BERT	2/8	43.0	76.6/77.0	–	–	84.6	–	–	68.3	52.7	–
TernaryBERT	2/8	28.0	83.3/83.3	90.1	–	–	50.7	–	87.5	68.2	–
TernaryBERT	2/2	28.0	40.3/40.0	63.1	50.0	80.7	0	12.4	68.3	54.5	45.5
TernaryBERT	2/1	28.0	32.7/33.0	74.1	59.3	53.1	0	7.1	68.3	53.4	42.3

Appendix

TABLE VIII: Comparison of BERT quantization methods without data augmentation. “–” means the results are not officially reported.

Model	Method	W bit /A bit	BoolQ	PIQA	HS	WG	ARC-E	ARC-C	OBQA	Avg.
LLaMA-2-7B	Full Precision	16/16	77.4	78.8	77.2	69.2	75.2	45.9	58.6	68.9
	GPTQ	2/16	43.9	51.1	26.3	50.8	26.6	28.5	29.0	36.6
	PB-LLM	1.7/16	62.3	53.8	27.7	49.3	28.0	25.0	30.2	39.5
	BiLLM	1.1/16	61.8	60.6	34.8	52.4	36.2	24.4	33.2	43.3
Bitnet-b1.58-3B	Bitnet	1.5/8	58.9	72.3	42.8	60.8	65.0	30.0	26.6	50.9
	BWN (10% layers)	1.4/7	37.8	49.6	25.0	49.6	25.7	22.9	27.0	33.9

Appendix

TABLE IX: Accuracy on commonsense QA benchmarks for various models and quantization methods.

Shape	Method	Latency ( $\mu$ s)	TFLOPs
BWTA with once Ternary Bitpack
$W$ : $[128,128]$ $A$ : $[128,128]$	w/o	8.31	0.505
	Naive	10.47	0.398
	IPB	10.01	0.416
$W$ : $[768,768]$ $A$ : $[128,768]$	w/o	7.59	19.90
	Naive	11.38	13.26

Appendix

TABLE X: Speed ablation of bitpacking implements. w/o means skip the bitpack operation, Naive means one-by-one bitpacking with setp instruction. IPB is our Instruction-level Parallel Bitpack method using set instruction.

Shape	Method	Latency ( $\mu$ s)	TFLOPs
$W$ : $[128,128]$ $A$ : $[128,128]$	w/o	5.44	0.772
	Naive ( mma.and )	7.46	0.562
	BWTA ( mma.xor )	6.96	0.603
$W$ : $[768,768]$ $A$ : $[128,768]$	w/o	10.91	13.84
	Naive ( mma.and )	13.75	10.98
	BWTA ( mma.xor )	11.33	13.33

Appendix

TABLE XI: Speed ablation of MatMul kernels. w/o means we skip the MatMul operation. Naive is a straightforward implementation without optimizing. BWTA is our efficient xor -based BWTA MatMul.

Layer	Shape	Kernel	Latency ( $\mu$ s)
$W\otimes A^{\top}$ (Linear)	$W$ : $[768,768]$ $A$ : $[128,768]$	FP16	215.1
		cuBLAS	41.32
		BWTA	9.24
	$W$ : $[3072,768]$ $A$ : $[128,768]$	FP16	216.3
		cuBLAS	80.17
		BWTA	10.43

Appendix

TABLE XII: The latency comparison to the customized FP16 GEMM which has the same blocks and threads as our BWTA, and cuBLAS which has the optimized parallel management.

Explore similar work

BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers
Mar 10, 2026Chaodong Xiao, Zhengqiang Zhang, Lei ZhangAttention Quantization Attention Mechanisms
The Hong Kong Polytechnic University · OPPO Research Institute
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
May 1, 2026Zhixiong Zhao, Zukang Xu, Dawei YangLLM Quantization LLM Inference Acceleration
1Houmo AI
FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy
Aug 16, 2026Qingyao Yang, Runming Yang, He Xiao +7LLM Quantization LLM Inference Acceleration
The University of Hong Kong · Harbin Institute of Technology

Mar 10, 2026J/K move · Enter open · S save

BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers

Chaodong Xiao, Zhengqiang Zhang, Lei Zhang

The Hong Kong Polytechnic University · OPPO Research Institute

Attention Quantization Attention Mechanisms Vision Transformer Diffusion Transformer

Open paper AskPDF arXiv

Transformers have achieved widespread and remarkable success, while the computational complexity of their attention modules remains a major bottleneck for vision tasks. Existing methods mainly employ 8-bit or 4-bit quantization to balance efficiency and accuracy. In this paper, with theoretical justification, we indicate that binarization of attention preserves the essential similarity relationships, and propose BinaryAttention, an effective method for fast and accurate 1-bit qk-attention. Specifically, we retain only the sign of queries and keys in computing the attention, and replace the floating dot products with bit-wise operations, significantly reducing the computational cost. We mitigate the inherent information loss under 1-bit quantization by incorporating a learnable bias, and enable end-to-end acceleration. To maintain the accuracy of attention, we adopt quantization-aware training and self-distillation techniques, mitigating quantization errors while ensuring sign-aligned similarity. BinaryAttention is more than 2x faster than FlashAttention2 on A100 GPUs. Extensive experiments on vision transformer and diffusion transformer benchmarks demonstrate that BinaryAttention matches or even exceeds full-precision attention, validating its effectiveness. Our work provides a highly efficient and effective alternative to full-precision attention, pushing the frontier of low-bit vision and diffusion transformers. The codes and models can be found at https://github.com/EdwardChasel/BinaryAttention.

Abstract

Explore similar work

BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers

BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy