Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

Yun Dai; Yuyun Chen; Jiarui Wen; Huiping Zhuang; Cen Chen; Ziqian Zeng

cs.CVOct 15, 2026

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

Authors: Yun Dai, Yuyun Chen, Jiarui Wen, Huiping Zhuang, Cen Chen, Ziqian Zeng

Organizations: South China University of Technology

Abstract

Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55 $\times$ --2.21 $\times$ speedup while maintaining generation quality comparable to full attention.

Figures & tables

Figure 1: Quantitative analysis of attention dynamics in DiT models. The subfigures (a–d) displays the variation of attention entropy and mean attention distance (MAD) across 10 diverse prompts, where curves of different shades represent individual samples. Tokens are reordered via Hilbert curves to preserve spatial-temporal adjacency before computing positional distances (Appendix A.6 ).

Figure 2: Attention recall under a fixed sparse mask. A sparse mask is constructed at an early denoising step and held fixed at subsequent steps. The mask maintains high attention recall throughout subsequent steps on both HunyuanVideo and Wan2.1.

Figure 3: Block-level attention patterns across denoising steps. Each cell represents the attention weight between a query block and a key block. Two representative heads at steps 5, 17, 29, 37, and 49 on HunyuanVideo. Tokens are reordered via Hilbert curves [ 3 ] to preserve spatial-temporal adjacency. As attention becomes more concentrated, blocks with high attention weights early in denoising continue to capture substantial attention weight, while those with low weights tend to retain low weights at later steps.

Figure 4: Overview of GSRS. The top row illustrates growing sparsity . As denoising progresses, progressively tighter Top-p thresholds retain fewer blocks, yielding increasingly sparse masks. The bottom row illustrates reduced search . The first search step evaluates all blocks to establish a baseline sparse mask. Subsequent search steps only evaluate blocks within this baseline mask, while non-search steps directly reuse the most recent mask without re-evaluation.

Model	Method	Quality						Efficiency
Model		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	SubConsist $\uparrow$	ImageQual $\uparrow$	VBench-Q $\uparrow$	Density $\downarrow$	Latency $\downarrow$	Speedup $\uparrow$
Wan 2.1	Full Attention	-	-	–	98.26%	73.29%	83.75%	100%	3689s	1.00 $\times$
	SpargeAttn [ 63 ]	20.55	0.757	0.295	94.88%	72.80%	82.21%	42.00%	2732s	1.35 $\times$
	SVG [ 51 ]	20.87	0.786	0.277	97.74%	71.99%	83.69%	40.14%	2675s	1.38 $\times$
	SVG 2 [ 58 ]	22.83	0.847	0.214	97.80%	72.23%	83.19%	32.08%	2455s	1.50 $\times$
	SVOO [ 33 ]	23.34	0.865	0.195	97.89%	72.42%	83.33%	32.82%	2610s	1.41 $\times$

Table 1: Quantitative evaluation of quality and efficiency. We evaluate on Wan 2.1 and HunyuanVideo. Bold and underline denote the best and second best results.

Table 6

Table 7

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Peak Memory (GB)
Full Attention	65.60
GSRS	62.65

Appendix

Table 6: Peak GPU memory usage during inference on HunyuanVideo (single A800, 125 frames, 720p).

Figure 5: Per-step attention kernel latency at varying density levels (

L=115200,H=24,d=128

Figure 6: Token distribution within a single block under different ordering strategies. (a) Standard row-major ordering groups tokens from a flat spatial-temporal slab. (b) Gilbert curve ordering groups tokens from a compact spatial-temporal region, better preserving spatial-temporal adjacency.

Figure 7: Runtime breakdown on HunyuanVideo. Removing reduced search introduces substantial extra attention and search overhead. GSRS with reduced search achieves 13.75

\times

lower overhead compared to the variant without it.

Model	Method	Subject Consistency	Background Consistency	Temporal Flickering	Motion Smoothness	Aesthetic Quality	Imaging Quality	Dynamic Degree	VBench-Q
Wan 2.1	Full Attention	98.26%	98.25%	99.08%	99.21%	63.07%	73.29%	33.52%	83.75%
	SpargeAttn [ 63 ]	94.88%	95.49%	99.07%	99.15%	61.73%	72.80%	32.95%	82.21%
	SVG [ 51 ]	97.74%	97.90%	99.27%	99.43%	63.18%	71.99%	34.65%	83.69%
	SVG 2 [ 58 ]	97.80%	97.47%	99.26%	99.38%	61.87%	72.23%	31.81%	83.19%
	SVOO [ 33 ]	97.89%	97.76%	99.30%	99.40%	61.63%	72.42%	32.38%	83.33%
	GSRS	98.21%	98.22%	99.07%	99.19%	63.09%	73.22%	33.52%	83.72%

Appendix

Table 7: Detailed VBench Quality dimensions. Full breakdown of VBench Quality Score across all 7 dimensions for Wan 2.1 and HunyuanVideo. Bold and underline denote the best and second best results.

$N_{w}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Latency(s) $\downarrow$
1	24.60	0.835	0.258	1680
3	27.63	0.899	0.203	1728
5	29.39	0.925	0.180	1746
8	31.00	0.944	0.161	1785
10	32.37	0.957	0.138	1825
15	33.79	0.966	0.134	1992

Appendix

Table 8: Ablation on the number of warmup steps

N_{w}

on HunyuanVideo.

Warmup Steps	Subject Consistency	Background Consistency	Temporal Flickering	Motion Smoothness	Aesthetic Quality	Imaging Quality	Dynamic Degree	VBench-Q
1	96.64%	96.44%	98.55%	99.49%	56.02%	70.66%	63.06%	83.80%
3	96.66%	96.50%	98.59%	99.49%	55.89%	70.86%	61.36%	83.71%
5	96.68%	96.50%	98.60%	99.49%	56.03%	70.81%	61.93%	83.78%
8	96.66%	96.52%	98.61%	99.50%	55.94%	70.82%	62.50%	83.81%
10	96.66%	96.91%	98.58%	99.49%	56.51%	70.79%	63.06%	84.01%
15	96.67%	96.49%	98.62%	99.50%	55.89%	70.78%	62.50%	83.80%

Appendix

Table 9: Detailed VBench Quality dimensions under different warmup steps. Full breakdown of VBench Quality Score across all 7 dimensions on HunyuanVideo.

Figure 8: PSNR distribution across VBench prompts. GSRS achieves the highest median PSNR on both Wan2.1 and HunyuanVideo.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench-Q $\uparrow$
HunyuanVideo	$31.35\pm 0.97$	$0.9519\pm 0.0067$	$0.147\pm 0.011$	$84.43\pm 0.65$

Appendix

Table 10: GSRS performance across five random seeds on HunyuanVideo. Values are reported as mean

\pm

standard deviation.

Top-p configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench-Q $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
{0.95, 0.95, 0.95, 0.95}	32.55	0.959	0.136	84.89	1969s	1.55 $\times$
{0.93, 0.93, 0.93, 0.93}	31.20	0.947	0.153	84.90	1852s	1.65 $\times$
{0.90, 0.90, 0.90, 0.90}	30.28	0.937	0.166	84.89	1774s	1.72 $\times$
{0.88, 0.88, 0.88, 0.88}	29.63	0.930	0.170	84.95	1749s	1.75 $\times$
{0.95, 0.80, 0.85, 0.90}	31.64	0.951	0.143	84.98	1850s	1.65 $\times$
{0.90, 0.90, 0.95, 0.95}	30.40	0.939	0.163	84.87	1792s	1.70 $\times$

Appendix

Table 11: GSRS performance under different Top-p configurations on HunyuanVideo.

	Wan2.1		HunyuanVideo
Method	Latency $\downarrow$	Speedup $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
Full Attention	2573s	1.00 $\times$	2265s	1.00 $\times$
SpargeAttn	1934s	1.33 $\times$	1497s	1.51 $\times$
SVG	1796s	1.43 $\times$	1319s	1.72 $\times$
SVG2	1735s	1.48 $\times$	1168s	1.94 $\times$
SVOO	1759s	1.46 $\times$	1143s	1.98 $\times$

Appendix

Table 12: Inference latency and speedup on a single NVIDIA H800 GPU.

Method	PSNR $\uparrow$	SSIM $\uparrow$	Speedup $\uparrow$
MInference	22.53	0.7435	1.27 $\times$
XAttention	29.08	0.9099	1.36 $\times$
LiteAttention	21.85	0.7036	1.33 $\times$
AdaSpa (w/o Head Adaptive)	28.51	0.8825	1.76 $\times$
AdaSpa	29.07	0.8905	1.78 $\times$
GSRS (AdaSpa aligned)	29.66	0.9330	1.86 $\times$

Appendix

Table 13: Comparison with related sparse attention methods on HunyuanVideo. GSRS (AdaSpa aligned) and AdaSpa (w/o Head Adaptive) both use a uniform 20% keep ratio.

Figure 9: Attention entropy and MAD profiles across diverse prompts for HunyuanVideo and Wan2.1. Each line represents a different test prompt. Both metrics exhibit a consistent monotonic decrease across all prompts, confirming that the attention concentration trend is an intrinsic model property independent of input content.

Figure 10: Layer-wise attention entropy and MAD across denoising steps. Both metrics decrease monotonically within each layer. Meanwhile, substantial variation exists across layers, confirming distinct attention concentration patterns at different depths.

Figure 11: Head-level attention entropy and MAD in HunyuanVideo. Each row corresponds to one layer, showing entropy (left) and MAD (right) for all heads. While different heads exhibit diverse behaviors, all follow a consistent decreasing trend across denoising steps.

Figure 12: Head-level attention entropy and MAD in Wan2.1 (conditional branch). Significant variation exists across heads within each layer, while the overall monotonic decrease over denoising is preserved.

Figure 13: Head-level attention entropy and MAD in Wan2.1 (unconditional branch). The patterns are highly consistent with the conditional branch, suggesting that head-wise variation is largely independent of conditioning.

Figure 14: Comparion of Full Attention and GSRS on Wan 2.1 Text-to-Video generation.

Figure 15: Comparion of Full Attention and GSRS on HunyuanVideo Text-to-Video generation.

Sparse Attention

Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.

See all 231 papers

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering
Mar 19, 2026Jiayi Luo, Jiayu Chen, Jiankun Wang +6Efficient Video Generation Video Generation
SKLCCSE, School of Computer Science and Engineering, Beihang University · Zhongguancun Academy · School of Computer Science, Peking University +4
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1

May 22, 2026J/K move · Enter open · S save

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Jie Hu, Zixiang Gao, Yutong He, Kun Yuan

1Peking University.

Video Diffusion Models Diffusion Transformer Transformer Attention Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Open paper AskPDF arXiv

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1

\times

end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Abstract

Sparse Attention

Explore similar work

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers