PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

Haopeng Li; Shitong Shao; Wenliang Zhong; Zikai Zhou; Lichen Bai; Hui Xiong; Zeke Xie

cs.CVFeb 1, 2026

PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

Authors: Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, Zeke Xie

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Abstract

Diffusion Transformers are fundamental for video and image generation, but their efficiency is bottlenecked by the quadratic complexity of attention. While block sparse attention accelerates computation by attending only critical key-value blocks, it suffers from degradation at high sparsity by discarding context. In this work, we discover that attention scores of non-critical blocks exhibit distributional stability, allowing them to be approximated accurately and efficiently rather than discarded, which is essentially important for sparse attention design. Motivated by this key insight, we propose PISA, a training-free Piecewise Sparse Attention that covers the full attention span with sub-quadratic complexity. Unlike the conventional keep-or-drop paradigm that directly drop the non-critical block information, PISA introduces a novel exact-or-approximate strategy: it maintains exact computation for critical blocks while efficiently approximating the remainder through block-wise Taylor expansion. This design allows PISA to serve as a faithful proxy to full attention, effectively bridging the gap between speed and quality. Experimental results demonstrate that PISA achieves 1.91 times and 2.57 times speedups on Wan2.1-14B and Hunyuan-Video, respectively, while consistently maintaining the highest quality among sparse attention methods. Notably, even for image generation on FLUX, PISA achieves a 1.2 times acceleration without compromising visual quality. Code is available at: https://github.com/xie-lab-ml/piecewise-sparse-attention.

Figures & tables

Panel of Figure 1 — Figure 1: PISA accelerates diverse generation tasks. (a, b) Wan2.1-14B image-to-video generation. (c) FLUX.1 image generation. PISA achieves better quality than SpargeAttn at higher sparsity. (d) Wan2.1-1.3B attention patterns: PISA more closely approximates dense attention than block sparse attention at similar cost.

Figure 2: Error of first-order Taylor expansion on pre-softmax attention scores ( $\bm{Q}\bm{K}^{\top}$ ) in Wan2.1-1.3B. (a) Expansion at the block mean $\mu$ (note the $10\times$ $y$ -range on the right). (b) Error vs. within-row score percentile (all layers/heads): the top-10% blocks carry 75% of the error.

Figure 3: The algorithm pipeline of PISA. Prepare Phase: we pre-compute block-wise means of queries and keys ( $\overline{Q},\overline{K}$ ), block-wise sums of values ( $\hat{V}$ ) and the global statistic $\bar{H}$ in a single pass; a block-wise Top-K selection (Eq. ( 7 )) identifies critical blocks. Fused kernel: selected blocks (e.g., 2, 3) are computed exactly ( Phase 1 ), unselected blocks (e.g., 1, 4) use the block-wise zeroth-order expansion with selected columns masked ( Phase 2 ), and Phase 3 applies $\bar{H}$ once to inject the global first-order term, avoiding memory-bound streaming.

Panel of Figure 4 — Figure 4: Kernel efficiency profile. ( a ) Latency comparison across sequence lengths at 12.5% density (87.5% sparsity) under two configurations with notation B-H-D ( batch_size , num_heads , head_dim ). ( b ) Relative speedup against FlashAttention-3 (FA3) across varying densities and sequence lengths under the B2-H16-D128 configuration. The dashed line indicates the baseline performance.

Figure 5: Latency breakdown of PISA at 12.5% density. Exact attention (latency annotated) dominates the runtime at all lengths.

Panel of Figure 6 — Figure 6: Video generation samples of different attention mechanisms. Left : Hunyuan-Video 13B. Right : Wan2.1-14B. All sparse attention are configured under a similar sparsity level ( $\sim$ 87.5%).

Wan2.1-1.3B (Text-to-Video, 480P)
Method	Sparsity	VBench $(\%)\uparrow$			Similarity			Efficiency
Method		S.C.	I.Q.	A.Q.	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Latency $\downarrow$	Speedup $\uparrow$
Dense	0.00%	94.96	66.09	60.79	–	–	–	98 s	1.00 $\times$
Sparge	87.5%	93.76	64.89	58.66	0.761	20.75	0.138	51 s	1.92 $\times$
SVG	87.5%	93.45	64.01	58.83	0.806	22.14	0.121	55 s	1.78 $\times$
SVG2 †	87.1%	93.79	64.51	59.09	0.809	22.85	0.104	62 s	1.58 $\times$

Table 1: Comparison of different sparse attention on video generation with warmup strategy.

\dagger

: using the official-provided configuration for reproducing the 87.13% sparsity reported in SVG2 paper.

Model	Method	Sparsity	Quality				Similarity			Efficiency
Model	Method	Sparsity	FID $\downarrow$	IR $\uparrow$	MPS $\uparrow$	HPSv2 $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Latency $\downarrow$
SD 3.5-M (1024p)	Dense-Attn	0%	15.48	0.912	–	30.13	–	–	–	3.03 s
	MInference	70%	17.30	0.678	39.87%	27.03	0.690	16.44	0.191	2.88 s
	SpargeAttn	70%	19.52	0.465	32.32%	26.30	0.627	14.60	0.269	2.69 s
	Ours	70%	14.78	0.883	49.71%	30.40	0.790	19.55	0.158	2.15 s
FLUX.1 (1024p)	Dense-Attn	0%	16.35	0.965	–	31.31	–	–	–	8.32 s

Table 2: Comparison of different sparse attention on text-to-image generation with warmup strategy.

Panel of Figure 8 — Figure 8: Approximation ablation (85% sparsity): FLUX.1 images (left, middle) and attention maps (right). PISA-0th loses fine details and attention structure, which the hybrid correction recovers.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Model	Method	Sparsity	VBench(%) $\uparrow$		Similarity		Speedup $\uparrow$
Model	Method		I.Q.	A.Q.	PSNR $\uparrow$	LPIPS $\downarrow$
Wan2.1-1.3B	Sparge	87.5%	60.10	55.26	10.92	0.397	2.72 $\times$
	SVG2	87.1%	63.22	57.25	13.42	0.292	2.33 $\times$
	Ours	87.5%	66.08	60.32	14.16	0.267	3.06 $\times$
Wan2.1-14B	Sparge	87.5%	60.03	51.45	9.48	0.520	2.59 $\times$
	SVG2	87.1%	58.85	55.72	10.67	0.489	2.37 $\times$

Appendix

Table 6: Comparison of different sparse attention on video generation without warmup strategy.

Method	Sparsity	Quality				Similarity			Efficiency
Method	Sparsity	FID $\downarrow$	IR $\uparrow$	MPS $\uparrow$	HPSv2 $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Latency $\downarrow$
Dense-Attn	0%	17.57	0.940	–	30.26	–	–	–	48.68 s
MInference	90%	19.66	0.910	44.33%	29.03	0.669	18.38	0.164	35.74 s
SpargeAttn	90%	19.72	0.908	46.19%	28.69	0.655	18.11	0.170	32.90 s
Ours	90%	17.11	0.932	50.44%	30.99	0.714	20.72	0.137	30.04 s

Appendix

Table 7: Text-to-image comparison on FLUX.1 at 2048

\times

2048 resolution with warmup strategy.

Approximation	Rel. $\ell_{2}$ $\downarrow$
Zeroth order	0.1533
Centroid $\beta_{t}$ + global $H$	0.1455
Average $\beta_{t}$ + global $H$	0.1440
Average $\beta_{t}$ + blockwise $H$	0.1420

Appendix

Table 8: Controlled design ablations . Left: fixed routing; right: fixed approximation and budget.

FLUX.1-dev
Block Size	SSIM $\uparrow$	PSNR $\uparrow$	Lat. $\downarrow$
128 $\times$ 128	0.606	12.11	7.12 s
128 $\times$ 0 64	0.661	15.99	6.66 s
64 $\times$ 128	0.648	15.09	7.08 s
64 $\times$ 0 64	0.682	17.07	6.87 s

Appendix

Table 9: Ablations of block size . Block size refers to the Q block and KV block sizes.

Panel of Figure 10 — Figure 10: Text-to-Image generation samples on FLUX.1-dev. The SpargeAttn under the 80% sparsity while PISA under the 85% sparsity.

Panel of Figure 11 — Figure 11: Text-to-Video generation samples on Wan2.1-14B. The PISA under the 87.5% sparsity with 10 steps and 1 layer warmup.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 506 papers

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1
Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
Oct 15, 2026Yun Dai, Yuyun Chen, Jiarui Wen +3Sparse Attention Diffusion Model Inference Acceleration
South China University of Technology

May 22, 2026J/K move · Enter open · S save

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Jie Hu, Zixiang Gao, Yutong He, Kun Yuan

1Peking University.

Video Diffusion Models Diffusion Transformer Transformer Attention Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Open paper AskPDF arXiv

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1

\times

end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Abstract

Diffusion Model Inference Acceleration

Explore similar work

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search