Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55×--2.21× speedup while maintaining generation quality comparable to full attention.
Figures & tables
Figure 1: Quantitative analysis of attention dynamics in DiT models. The subfigures (a–d) displays the variation of attention entropy and mean attention distance (MAD) across 10 diverse prompts, where curves of different shades represent individual samples. Tokens are reordered via Hilbert curves to preserve spatial-temporal adjacency before computing positional distances (Appendix A.6 ).
Figure 2: Attention recall under a fixed sparse mask. A sparse mask is constructed at an early denoising step and held fixed at subsequent steps. The mask maintains high attention recall throughout subsequent steps on both HunyuanVideo and Wan2.1.
Figure 3: Block-level attention patterns across denoising steps. Each cell represents the attention weight between a query block and a key block. Two representative heads at steps 5, 17, 29, 37, and 49 on HunyuanVideo. Tokens are reordered via Hilbert curves [ 3 ] to preserve spatial-temporal adjacency. As attention becomes more concentrated, blocks with high attention weights early in denoising continue to capture substantial attention weight, while those with low weights tend to retain low weights at later steps.
Figure 4: Overview of GSRS. The top row illustrates growing sparsity . As denoising progresses, progressively tighter Top-p thresholds retain fewer blocks, yielding increasingly sparse masks. The bottom row illustrates reduced search . The first search step evaluates all blocks to establish a baseline sparse mask. Subsequent search steps only evaluate blocks within this baseline mask, while non-search steps directly reuse the most recent mask without re-evaluation.
Model
Method
Quality
Efficiency
PSNR ↑
SSIM ↑
LPIPS ↓
SubConsist ↑
ImageQual ↑
VBench-Q ↑
Density ↓
Latency ↓
Speedup ↑
Wan 2.1
Full Attention
-
-
–
98.26%
73.29%
83.75%
100%
3689s
1.00 ×
SpargeAttn [ 63 ]
20.55
0.757
0.295
94.88%
72.80%
82.21%
42.00%
2732s
1.35 ×
SVG [ 51 ]
20.87
0.786
0.277
97.74%
71.99%
83.69%
40.14%
2675s
1.38 ×
SVG 2 [ 58 ]
22.83
0.847
0.214
97.80%
72.23%
83.19%
32.08%
2455s
1.50 ×
SVOO [ 33 ]
23.34
0.865
0.195
97.89%
72.42%
83.33%
32.82%
2610s
1.41 ×
Table 1: Quantitative evaluation of quality and efficiency. We evaluate on Wan 2.1 and HunyuanVideo. Bold and underline denote the best and second best results.
Table 6Table 7
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Peak Memory (GB)
Full Attention
65.60
GSRS
62.65
Appendix
Table 6: Peak GPU memory usage during inference on HunyuanVideo (single A800, 125 frames, 720p).
Figure 5: Per-step attention kernel latency at varying density levels ( L=115200,H=24,d=128 ).
Figure 6: Token distribution within a single block under different ordering strategies. (a) Standard row-major ordering groups tokens from a flat spatial-temporal slab. (b) Gilbert curve ordering groups tokens from a compact spatial-temporal region, better preserving spatial-temporal adjacency.
Figure 7: Runtime breakdown on HunyuanVideo. Removing reduced search introduces substantial extra attention and search overhead. GSRS with reduced search achieves 13.75 × lower overhead compared to the variant without it.
Model
Method
Subject Consistency
Background Consistency
Temporal Flickering
Motion Smoothness
Aesthetic Quality
Imaging Quality
Dynamic Degree
VBench-Q
Wan 2.1
Full Attention
98.26%
98.25%
99.08%
99.21%
63.07%
73.29%
33.52%
83.75%
SpargeAttn [ 63 ]
94.88%
95.49%
99.07%
99.15%
61.73%
72.80%
32.95%
82.21%
SVG [ 51 ]
97.74%
97.90%
99.27%
99.43%
63.18%
71.99%
34.65%
83.69%
SVG 2 [ 58 ]
97.80%
97.47%
99.26%
99.38%
61.87%
72.23%
31.81%
83.19%
SVOO [ 33 ]
97.89%
97.76%
99.30%
99.40%
61.63%
72.42%
32.38%
83.33%
GSRS
98.21%
98.22%
99.07%
99.19%
63.09%
73.22%
33.52%
83.72%
Appendix
Table 7: Detailed VBench Quality dimensions. Full breakdown of VBench Quality Score across all 7 dimensions for Wan 2.1 and HunyuanVideo. Bold and underline denote the best and second best results.
Nw
PSNR ↑
SSIM ↑
LPIPS ↓
Latency(s) ↓
1
24.60
0.835
0.258
1680
3
27.63
0.899
0.203
1728
5
29.39
0.925
0.180
1746
8
31.00
0.944
0.161
1785
10
32.37
0.957
0.138
1825
15
33.79
0.966
0.134
1992
Appendix
Table 8: Ablation on the number of warmup steps Nw on HunyuanVideo.
Warmup Steps
Subject Consistency
Background Consistency
Temporal Flickering
Motion Smoothness
Aesthetic Quality
Imaging Quality
Dynamic Degree
VBench-Q
1
96.64%
96.44%
98.55%
99.49%
56.02%
70.66%
63.06%
83.80%
3
96.66%
96.50%
98.59%
99.49%
55.89%
70.86%
61.36%
83.71%
5
96.68%
96.50%
98.60%
99.49%
56.03%
70.81%
61.93%
83.78%
8
96.66%
96.52%
98.61%
99.50%
55.94%
70.82%
62.50%
83.81%
10
96.66%
96.91%
98.58%
99.49%
56.51%
70.79%
63.06%
84.01%
15
96.67%
96.49%
98.62%
99.50%
55.89%
70.78%
62.50%
83.80%
Appendix
Table 9: Detailed VBench Quality dimensions under different warmup steps. Full breakdown of VBench Quality Score across all 7 dimensions on HunyuanVideo.
Figure 8: PSNR distribution across VBench prompts. GSRS achieves the highest median PSNR on both Wan2.1 and HunyuanVideo.
Model
PSNR ↑
SSIM ↑
LPIPS ↓
VBench-Q ↑
HunyuanVideo
31.35±0.97
0.9519±0.0067
0.147±0.011
84.43±0.65
Appendix
Table 10: GSRS performance across five random seeds on HunyuanVideo. Values are reported as mean ± standard deviation.
Top-p configuration
PSNR ↑
SSIM ↑
LPIPS ↓
VBench-Q ↑
Latency ↓
Speedup ↑
{0.95, 0.95, 0.95, 0.95}
32.55
0.959
0.136
84.89
1969s
1.55 ×
{0.93, 0.93, 0.93, 0.93}
31.20
0.947
0.153
84.90
1852s
1.65 ×
{0.90, 0.90, 0.90, 0.90}
30.28
0.937
0.166
84.89
1774s
1.72 ×
{0.88, 0.88, 0.88, 0.88}
29.63
0.930
0.170
84.95
1749s
1.75 ×
{0.95, 0.80, 0.85, 0.90}
31.64
0.951
0.143
84.98
1850s
1.65 ×
{0.90, 0.90, 0.95, 0.95}
30.40
0.939
0.163
84.87
1792s
1.70 ×
Appendix
Table 11: GSRS performance under different Top-p configurations on HunyuanVideo.
Wan2.1
HunyuanVideo
Method
Latency ↓
Speedup ↑
Latency ↓
Speedup ↑
Full Attention
2573s
1.00 ×
2265s
1.00 ×
SpargeAttn
1934s
1.33 ×
1497s
1.51 ×
SVG
1796s
1.43 ×
1319s
1.72 ×
SVG2
1735s
1.48 ×
1168s
1.94 ×
SVOO
1759s
1.46 ×
1143s
1.98 ×
Appendix
Table 12: Inference latency and speedup on a single NVIDIA H800 GPU.
Method
PSNR ↑
SSIM ↑
Speedup ↑
MInference
22.53
0.7435
1.27 ×
XAttention
29.08
0.9099
1.36 ×
LiteAttention
21.85
0.7036
1.33 ×
AdaSpa (w/o Head Adaptive)
28.51
0.8825
1.76 ×
AdaSpa
29.07
0.8905
1.78 ×
GSRS (AdaSpa aligned)
29.66
0.9330
1.86 ×
Appendix
Table 13: Comparison with related sparse attention methods on HunyuanVideo. GSRS (AdaSpa aligned) and AdaSpa (w/o Head Adaptive) both use a uniform 20% keep ratio.
Figure 9: Attention entropy and MAD profiles across diverse prompts for HunyuanVideo and Wan2.1. Each line represents a different test prompt. Both metrics exhibit a consistent monotonic decrease across all prompts, confirming that the attention concentration trend is an intrinsic model property independent of input content.
Figure 10: Layer-wise attention entropy and MAD across denoising steps. Both metrics decrease monotonically within each layer. Meanwhile, substantial variation exists across layers, confirming distinct attention concentration patterns at different depths.
Figure 11: Head-level attention entropy and MAD in HunyuanVideo. Each row corresponds to one layer, showing entropy (left) and MAD (right) for all heads. While different heads exhibit diverse behaviors, all follow a consistent decreasing trend across denoising steps.
Figure 12: Head-level attention entropy and MAD in Wan2.1 (conditional branch). Significant variation exists across heads within each layer, while the overall monotonic decrease over denoising is preserved.
Figure 13: Head-level attention entropy and MAD in Wan2.1 (unconditional branch). The patterns are highly consistent with the conditional branch, suggesting that head-wise variation is largely independent of conditioning.
Figure 14: Comparion of Full Attention and GSRS on Wan 2.1 Text-to-Video generation.
Figure 15: Comparion of Full Attention and GSRS on HunyuanVideo Text-to-Video generation.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.
Diffusion Transformers (DiTs) achieve strong video generation quality but suffer from high inference cost due to dense 3D attention, motivating sparse attention techniques for improving efficiency. However, existing training-free sparse attention methods for video generation still face two unresolved limitations: ignoring layer heterogeneity in attention pruning and ignoring query-key coupling in block partitioning, which hinder a better quality-speedup trade-off. In this work, we uncover a critical insight: attention sparsity is an intrinsic layer-wise property, with only minor variation across different inputs. Motivated by this observation, we propose SVOO, a training-free sparse attention framework for fast video generation via offline layer-wise sparsity profiling and online bidirectional co-clustering. Specifically, SVOO adopts a two-stage paradigm: (i) offline layer-wise sensitivity profiling to derive intrinsic per-layer pruning levels, and (ii) online block-wise sparse attention via a bidirectional co-clustering algorithm. Extensive experiments on seven widely used video generation models demonstrate that SVOO achieves a superior quality-speedup trade-off over state-of-the-art methods, delivering up to 1.93x speedup while maintaining a PSNR of up to 29 dB on Wan2.1. Code is available at: https://github.com/Mutual-Luo/SVOO.
Jiayi Luo, Jiayu Chen, Jiankun Wang +6
SKLCCSE, School of Computer Science and Engineering, Beihang University · Zhongguancun Academy · School of Computer Science, Peking University +4
While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.
Ruiliang Zhou, Xuecheng Wu, Kang He +6
KlingAI Research · Beijing Institute of Technology · NVIDIA +1