BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers

Chaodong Xiao; Zhengqiang Zhang; Lei Zhang

cs.CVMar 10, 2026

BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers

Authors: Chaodong Xiao, Zhengqiang Zhang, Lei Zhang

Organizations: The Hong Kong Polytechnic University · OPPO Research Institute

Abstract

Transformers have achieved widespread and remarkable success, while the computational complexity of their attention modules remains a major bottleneck for vision tasks. Existing methods mainly employ 8-bit or 4-bit quantization to balance efficiency and accuracy. In this paper, with theoretical justification, we indicate that binarization of attention preserves the essential similarity relationships, and propose BinaryAttention, an effective method for fast and accurate 1-bit qk-attention. Specifically, we retain only the sign of queries and keys in computing the attention, and replace the floating dot products with bit-wise operations, significantly reducing the computational cost. We mitigate the inherent information loss under 1-bit quantization by incorporating a learnable bias, and enable end-to-end acceleration. To maintain the accuracy of attention, we adopt quantization-aware training and self-distillation techniques, mitigating quantization errors while ensuring sign-aligned similarity. BinaryAttention is more than 2x faster than FlashAttention2 on A100 GPUs. Extensive experiments on vision transformer and diffusion transformer benchmarks demonstrate that BinaryAttention matches or even exceeds full-precision attention, validating its effectiveness. Our work provides a highly efficient and effective alternative to full-precision attention, pushing the frontier of low-bit vision and diffusion transformers. The codes and models can be found at https://github.com/EdwardChasel/BinaryAttention.

Figures & tables

Figure 1: Top : Performance comparison between FlashAttention2 and BinaryAttention on vision tasks. Bottom : Image generation examples by DiT-XL/2 [ 56 ] driven by BinaryAttention.

Panel of Figure 2 — Figure 2: Overview and comparative analysis of BinaryAttention. (a) The computation of BinaryAttention involves three components: converting queries and keys into scaled binary representations, applying a bias enhancement, and quantizing the attention coefficients and values. Sub-figures (b) and (c) show the attention maps (top) and the corresponding activation maps (bottom) for Standard Attention and our BinaryAttention, demonstrating the comparable expressivity of BinaryAttention to Standard Attention despite 1-bit quantization.

Figure 3: Kernel speed comparison on A100 GPUs.

Figure 4: End-to-end throughput and speedup comparisons on A100 GPUs. ViT [ 18 ] models are used.

	Method	Reso.	#Param.	OPs	Top-1
Linear Transformer	Hydra-T [ 5 ]	$\text{224}^{\text{2}}$	6M	1.1G	68.3
	Efficient-T [ 66 ]	$\text{224}^{\text{2}}$	6M	1.1G	70.2
	Angular-T [ 83 ]	$\text{224}^{\text{2}}$	6M	1.1G	70.8
	Enhanced-T [ 7 ]	$\text{224}^{\text{2}}$	6M	1.1G	72.9
	FLatten-T [ 27 ]	$\text{224}^{\text{2}}$	6M	1.1G	74.1
	InLine-T [ 28 ]	$\text{224}^{\text{2}}$	7M	1.1G	74.5

Table 1: Comparison of image classification on ImageNet-1K.

(a) Mask R-CNN
Backbone	AP ${}^{\text{b}}$	AP ${}^{\text{b}}_{\text{50}}$	AP ${}^{\text{b}}_{\text{75}}$	AP ${}^{\text{b}}_{\text{s}}$	AP ${}^{\text{b}}_{\text{m}}$	AP ${}^{\text{b}}_{\text{l}}$	AP ${}^{\text{m}}$	AP ${}^{\text{m}}_{\text{50}}$	AP ${}^{\text{m}}_{\text{75}}$	AP ${}^{\text{m}}_{\text{s}}$	AP ${}^{\text{m}}_{\text{m}}$	AP ${}^{\text{m}}_{\text{l}}$	#Param.	OPs
DeiT-T	41.11	62.24	44.71	23.50	44.00	56.56	37.26	59.20	39.63	17.35	39.55	56.62	28M	338G
SageAttention-T	41.10	62.21	44.65	23.52	44.02	56.43	37.26	59.23	39.62	17.39	39.57	56.61	28M	327G
BinaryAttention-T	41.29	62.04	44.91	23.99	44.62	55.81	37.26	59.10	39.68	17.71	39.89	55.85	28M	313G
DeiT-S	45.49	67.29	49.10	28.91	48.56	61.57	40.87	63.90	43.57	21.95	43.56	60.89	44M	440G
SageAttention-S	45.48	67.23	49.13	28.87	48.65	61.62	40.85	63.90	43.59	21.92	43.52	60.86	44M	418G

Table 2: Comparison of object detection and instance segmentation on COCO. OPs are computed with input resolution of 1024

\times

1024.

Backbone	Crop size	mIoU (SS)	mIoU (MS)	#Param.	OPs
DeiT-T	$\text{512}^{\text{2}}$	39.82	40.68	11M	227G
SageAttention-T	$\text{512}^{\text{2}}$	39.82	40.68	11M	198G
BinaryAttention-T	$\text{512}^{\text{2}}$	39.93	40.89	11M	159G
DeiT-S	$\text{512}^{\text{2}}$	44.67	46.01	43M	744G
SageAttention-S	$\text{512}^{\text{2}}$	44.67	46.01	43M	687G
BinaryAttention-S	$\text{512}^{\text{2}}$	44.75	45.95	43M	610G

Table 3: Comparison of semantic segmentation on ADE20K. ‘SS’ and ‘MS’ represent single-scale and multi-scale testing, respectively. OPs are calculated with input resolution of

512\times 2048

Method	OPs	Steps	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Pre. $\uparrow$	Re. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Pre. $\uparrow$	Re. $\uparrow$
(cfg=1.25)			DiT-S/2					SiT-S/2
FlashAttention2	6.1G	400K	54.73	10.21	26.83	0.42	0.57	46.33	7.89	31.99	0.46	0.59
SageAttention	5.8G	400K	54.74	10.21	26.86	0.42	0.57	46.32	7.89	32.00	0.46	0.59
BinaryAttention	5.5G	200K	49.76	10.40	30.14	0.45	0.57	41.40	7.99	36.49	0.48	0.59
(cfg=1.50)			DiT-S/2					SiT-S/2
FlashAttention2	6.1G	400K	43.87	8.82	35.16	0.48	0.55	36.26	7.09	42.24	0.52	0.56

Table 4: Comparison of class-conditional image generation on ImageNet 256

\times

256.

Figure 5: Qualitative comparison of generated image by DiT-XL/2 (cfg=1.50) using FlashAttention2 and BinaryAttention.

Scale	Bias	Distillation	DeiT-T	DeiT-S	DeiT-B
Scale	Bias	Distillation	Top-1 Accuracy
Baseline (full-precision)			72.2	79.8	81.8
✗	✗	✗	71.95	79.59	81.10
✓	✗	✗	72.42	79.81	81.33
✓	✗	✓	72.44	79.97	81.99
✓	✓	✓	72.88	80.24	82.04

Table 5: Ablation studies on BinaryAttention for the scaled binary representations, bias enhancement and self-distillation strategy on ImageNet-1K benchmark, using DeiT architectures.

	CosSim	Relative L1	RMSE	Precision
Layer (0)	0.9186	0.4840	0.4165	0.7716
Layer (6)	0.8740	0.7353	0.5084	0.7301

Table 6: Attention pattern comparison between FlashAttention2 and BinaryAttention by DeiT-B on ImageNet-1K validation set.

Method	Top-1	Mem. (512)	Mem. (1024)
FlashAttention2	72.2	1705M	5304M
SageAttention	72.11	1705M	5304M
BinaryAttention (den.)	72.88	3246M	29904M
BinaryAttention (dec.)	72.97	1706M	5307M

Table 7: Memory comparison by DeiT-T using FlashAttention2, SageAttention and BinaryAttention at resolutions of 512 and 1024.

FlashAttention2	SageAttention	BinaryAttention	Quant Q&K	Quant V
175.3ms	124.6ms	88.2ms	2.8ms	1.9ms

Table 8: Latency of attention kernels and quantization components measured on A100 GPUs.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Epochs	DeiT-T	DeiT-S	DeiT-B
Baseline	72.2	79.8	81.8
100	71.98	79.44	81.80
300	72.44	79.97	81.99

Appendix

Table 9: Top-1 accuracy of DeiT models using BinaryAttention without bias at 100 and 300 fine-tuning epochs.

Figure 6: More qualitative comparisons of generated image by DiT-XL/2 (cfg=1.50) using FlashAttention2, SageAttention and BinaryAttention.

Explore similar work

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
Sep 14, 2026Xingyang Li, Dongyun Zou, Shining Zhang +8Video Diffusion Models Diffusion Transformer
2MIT · 1Nunchux AI · 3CMU +3
BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design
Apr 5, 2026Yifu Ding, Xianglong Liu, Shenghao Jin +2Low-Bit Quantization Transformer Inference
School of Computer Science and Engineering, Beihang University, Beijing, China. · Institute of Artificial Intelligence, Beihang University, Beijing, China. · Department of Automation, Tsinghua University, Beijing, China.
QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
Apr 28, 2026Sehyeon Oh, Yongin Kwon, Jemin LeeSoftmax Attention GPU Acceleration
University of Science and Technology, Daejeon, Republic of Korea · Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea · Pusan National University, Busan, Republic of Korea +1

Sep 14, 2026J/K move · Enter open · S save

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

Xingyang Li, Dongyun Zou, Shining Zhang, Jiacheng Chen+7

2MIT · 1Nunchux AI · 3CMU+3

Video Diffusion Models Diffusion Transformer Low-Bit Quantization Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical entries confined to a narrow range of representable values. Prior work smooths queries and keys, but value outliers follow no fixed channel or spatiotemporal structure and remain the dominant source of output error. Speed is limited by softmax: low-bit Tensor Cores accelerate only the two matrix multiplications, so the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs. We propose VC-Attention, a training-free low-bit attention framework that addresses both by pairing Value smoothing with a fused probability Cast. V-Smooth reorders value tokens by lightweight online clustering, so the tokens in a hardware block quantize well together. It quantizes only the residual after subtracting the block mean, and restores that mean from the row sum the online softmax already maintains. ExpCast-FP8 maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion. We implement VC-Attention for B200, B300, H200, RTX PRO 6000, and RTX 5090. Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, VC-Attention improves fidelity over low-bit baselines, speeds up the attention kernel over BF16 FlashAttention-4 by 1.46-1.59x on datacenter Blackwell and Hopper and by 2.3-3.6x on workstation cards, and generates a clip 1.13-1.19x and 1.36-1.70x faster end to end.

Abstract

Explore similar work

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention