PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

Haopeng Li; Shitong Shao; Wenliang Zhong; Zikai Zhou; Lichen Bai; Hui Xiong; Zeke Xie

cs.CVFeb 1, 2026

PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

Authors: Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, Zeke Xie

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Abstract

Diffusion Transformers are fundamental for video and image generation, but their efficiency is bottlenecked by the quadratic complexity of attention. While block sparse attention accelerates computation by attending only critical key-value blocks, it suffers from degradation at high sparsity by discarding context. In this work, we discover that attention scores of non-critical blocks exhibit distributional stability, allowing them to be approximated accurately and efficiently rather than discarded, which is essentially important for sparse attention design. Motivated by this key insight, we propose PISA, a training-free Piecewise Sparse Attention that covers the full attention span with sub-quadratic complexity. Unlike the conventional keep-or-drop paradigm that directly drop the non-critical block information, PISA introduces a novel exact-or-approximate strategy: it maintains exact computation for critical blocks while efficiently approximating the remainder through block-wise Taylor expansion. This design allows PISA to serve as a faithful proxy to full attention, effectively bridging the gap between speed and quality. Experimental results demonstrate that PISA achieves 1.91 times and 2.57 times speedups on Wan2.1-14B and Hunyuan-Video, respectively, while consistently maintaining the highest quality among sparse attention methods. Notably, even for image generation on FLUX, PISA achieves a 1.2 times acceleration without compromising visual quality. Code is available at: https://github.com/xie-lab-ml/piecewise-sparse-attention.

Figures & tables

Panel of Figure 1 — Figure 1: PISA accelerates diverse generation tasks. (a, b) Wan2.1-14B image-to-video generation. (c) FLUX.1 image generation. PISA achieves better quality than SpargeAttn at higher sparsity. (d) Wan2.1-1.3B attention patterns: PISA more closely approximates dense attention than block sparse attention at similar cost.

Figure 2: Error of first-order Taylor expansion on pre-softmax attention scores ( $\bm{Q}\bm{K}^{\top}$ ) in Wan2.1-1.3B. (a) Expansion at the block mean $\mu$ (note the $10\times$ $y$ -range on the right). (b) Error vs. within-row score percentile (all layers/heads): the top-10% blocks carry 75% of the error.

Figure 3: The algorithm pipeline of PISA. Prepare Phase: we pre-compute block-wise means of queries and keys ( $\overline{Q},\overline{K}$ ), block-wise sums of values ( $\hat{V}$ ) and the global statistic $\bar{H}$ in a single pass; a block-wise Top-K selection (Eq. ( 7 )) identifies critical blocks. Fused kernel: selected blocks (e.g., 2, 3) are computed exactly ( Phase 1 ), unselected blocks (e.g., 1, 4) use the block-wise zeroth-order expansion with selected columns masked ( Phase 2 ), and Phase 3 applies $\bar{H}$ once to inject the global first-order term, avoiding memory-bound streaming.

Panel of Figure 4 — Figure 4: Kernel efficiency profile. ( a ) Latency comparison across sequence lengths at 12.5% density (87.5% sparsity) under two configurations with notation B-H-D ( batch_size , num_heads , head_dim ). ( b ) Relative speedup against FlashAttention-3 (FA3) across varying densities and sequence lengths under the B2-H16-D128 configuration. The dashed line indicates the baseline performance.

Figure 5: Latency breakdown of PISA at 12.5% density. Exact attention (latency annotated) dominates the runtime at all lengths.

Panel of Figure 6 — Figure 6: Video generation samples of different attention mechanisms. Left : Hunyuan-Video 13B. Right : Wan2.1-14B. All sparse attention are configured under a similar sparsity level ( $\sim$ 87.5%).

Wan2.1-1.3B (Text-to-Video, 480P)
Method	Sparsity	VBench $(\%)\uparrow$			Similarity			Efficiency
Method		S.C.	I.Q.	A.Q.	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Latency $\downarrow$	Speedup $\uparrow$
Dense	0.00%	94.96	66.09	60.79	–	–	–	98 s	1.00 $\times$
Sparge	87.5%	93.76	64.89	58.66	0.761	20.75	0.138	51 s	1.92 $\times$
SVG	87.5%	93.45	64.01	58.83	0.806	22.14	0.121	55 s	1.78 $\times$
SVG2 †	87.1%	93.79	64.51	59.09	0.809	22.85	0.104	62 s	1.58 $\times$

Table 1: Comparison of different sparse attention on video generation with warmup strategy.

\dagger

: using the official-provided configuration for reproducing the 87.13% sparsity reported in SVG2 paper.

Model	Method	Sparsity	Quality				Similarity			Efficiency
Model	Method	Sparsity	FID $\downarrow$	IR $\uparrow$	MPS $\uparrow$	HPSv2 $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Latency $\downarrow$
SD 3.5-M (1024p)	Dense-Attn	0%	15.48	0.912	–	30.13	–	–	–	3.03 s
	MInference	70%	17.30	0.678	39.87%	27.03	0.690	16.44	0.191	2.88 s
	SpargeAttn	70%	19.52	0.465	32.32%	26.30	0.627	14.60	0.269	2.69 s
	Ours	70%	14.78	0.883	49.71%	30.40	0.790	19.55	0.158	2.15 s
FLUX.1 (1024p)	Dense-Attn	0%	16.35	0.965	–	31.31	–	–	–	8.32 s

Table 2: Comparison of different sparse attention on text-to-image generation with warmup strategy.

Panel of Figure 8 — Figure 8: Approximation ablation (85% sparsity): FLUX.1 images (left, middle) and attention maps (right). PISA-0th loses fine details and attention structure, which the hybrid correction recovers.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Model	Method	Sparsity	VBench(%) $\uparrow$		Similarity		Speedup $\uparrow$
Model	Method		I.Q.	A.Q.	PSNR $\uparrow$	LPIPS $\downarrow$
Wan2.1-1.3B	Sparge	87.5%	60.10	55.26	10.92	0.397	2.72 $\times$
	SVG2	87.1%	63.22	57.25	13.42	0.292	2.33 $\times$
	Ours	87.5%	66.08	60.32	14.16	0.267	3.06 $\times$
Wan2.1-14B	Sparge	87.5%	60.03	51.45	9.48	0.520	2.59 $\times$
	SVG2	87.1%	58.85	55.72	10.67	0.489	2.37 $\times$

Appendix

Table 6: Comparison of different sparse attention on video generation without warmup strategy.

Method	Sparsity	Quality				Similarity			Efficiency
Method	Sparsity	FID $\downarrow$	IR $\uparrow$	MPS $\uparrow$	HPSv2 $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Latency $\downarrow$
Dense-Attn	0%	17.57	0.940	–	30.26	–	–	–	48.68 s
MInference	90%	19.66	0.910	44.33%	29.03	0.669	18.38	0.164	35.74 s
SpargeAttn	90%	19.72	0.908	46.19%	28.69	0.655	18.11	0.170	32.90 s
Ours	90%	17.11	0.932	50.44%	30.99	0.714	20.72	0.137	30.04 s

Appendix

Table 7: Text-to-image comparison on FLUX.1 at 2048

\times

2048 resolution with warmup strategy.

Approximation	Rel. $\ell_{2}$ $\downarrow$
Zeroth order	0.1533
Centroid $\beta_{t}$ + global $H$	0.1455
Average $\beta_{t}$ + global $H$	0.1440
Average $\beta_{t}$ + blockwise $H$	0.1420

Appendix

Table 8: Controlled design ablations . Left: fixed routing; right: fixed approximation and budget.

FLUX.1-dev
Block Size	SSIM $\uparrow$	PSNR $\uparrow$	Lat. $\downarrow$
128 $\times$ 128	0.606	12.11	7.12 s
128 $\times$ 0 64	0.661	15.99	6.66 s
64 $\times$ 128	0.648	15.09	7.08 s
64 $\times$ 0 64	0.682	17.07	6.87 s

Appendix

Table 9: Ablations of block size . Block size refers to the Q block and KV block sizes.

Panel of Figure 10 — Figure 10: Text-to-Image generation samples on FLUX.1-dev. The SpargeAttn under the 80% sparsity while PISA under the 85% sparsity.

Panel of Figure 11 — Figure 11: Text-to-Video generation samples on Wan2.1-14B. The PISA under the 87.5% sparsity with 10 steps and 1 layer warmup.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 506 papers

May 22, 2026

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1

\times

end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Jie Hu, Zixiang Gao, Yutong He +1

1Peking University.

Video Diffusion Models Diffusion Transformer Transformer Attention Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Ask PDF

Jun 22, 2026

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.

Ruiliang Zhou, Xuecheng Wu, Kang He +6

KlingAI Research · Beijing Institute of Technology · NVIDIA +1

Video Diffusion Models Diffusion Transformer Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Ask PDF

Oct 15, 2026

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55

\times

--2.21

\times

speedup while maintaining generation quality comparable to full attention.

Yun Dai, Yuyun Chen, Jiarui Wen +3

South China University of Technology

Sparse Attention Diffusion Model Inference Acceleration Video Diffusion Models Video Generation Diffusion Transformer

Ask PDF

Abstract

Diffusion Model Inference Acceleration

Explore similar work

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search