Video diffusion transformers generate high-quality videos, but attention over long spatiotemporal sequences makes inference computationally expensive. Sparse attention reduces this cost by computing only a subset of query-key interactions exactly, and recent training-free methods further approximate the remaining interactions. These methods, however, degrade quickly at high sparsity, which limits the achievable speedup. We present ESTRA (Extreme Sparsity Through Refined Attention), a training-free acceleration method that preserves generation quality even at extreme sparsity. ESTRA concentrates exact attention computation on a small set of important key-value tokens and approximates the remaining contributions with cluster representatives. To identify these tokens efficiently, it groups similar queries and uses their representatives to score individual keys. A variance-based correction further refines the attention weights of the cluster representatives, improving the accuracy of the approximation. Experiments on Wan2.1-14B and HunyuanVideo-13B show that ESTRA establishes a new quality-efficiency frontier for training-free sparse attention, achieving 3.26× and 4.71× end-to-end speedups at 5% attention density while surpassing Sol-Attn and SVG-EAR in fidelity to dense generation and on most VBench dimensions. Even at 1% density, ESTRA remains more faithful to dense generation than Sol-Attn at 12.5%, while reaching up to 5.39× speedup.
Figures & tables
Figure 1: High-quality video generation at up to 99% attention sparsity. At 5% exact-interaction density, ESTRA more closely preserves the composition and appearance of the dense references than the baselines shown, while using higher sparsity and achieving greater end-to-end speedups. At 1%, the videos remain visually coherent, with particularly small changes in the HunyuanVideo example. Density denotes the fraction of video query–key interactions evaluated exactly; speedups are end to end on an NVIDIA H20. Additional examples are provided in Appendix D .
Figure 2: ESTRA overview. (a) Independent query and key clusters provide query centroids and K/V means. (b) Each centroid scores individual keys for selection shared by its query cluster. (c) VCM uses remaining-key spread and query norms to calibrate representative weights. (d) Original queries attend jointly to selected tokens and fixed cluster means, with log-count and VCM biases on representative logits. Colors identify clusters. Attention scaling is omitted.
Figure 3: Adding VCM and then token-level selection lowers attention-output error on both backbones.
Efficiency
Fidelity to dense
VBench dimensions ↑
Method
Density
Latency (s) ↓
Speedup
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
Subj.
Bkgd.
Imag.
Aesth.
Wan2.1-14B , 1280 × 720, 81 frames; no dense warmup
Dense
100%
6838
1.00 ×
ref.
ref.
ref.
91.78
94.42
66.98
66.02
Sol-Attn
12.8%
2139
3.20 ×
10.71
0.283
0.737
90.00
93.14
65.69
60.23
SVG-EAR
7.6%
2171
3.15 ×
11.57
0.342
0.657
89.80
92.13
66.28
63.79
ESTRA
5.0%
2097
3.26 ×
14.67
0.489
0.467
91.06
93.59
66.08
65.93
Table 1: Video quality, fidelity, and speed at 720p. Fidelity metrics average 251 sparse–dense pairs per backbone; VBench scores use designated prompt subsets. Latency is the median H20 time per video, including VAE decoding and export. Bold marks the best sparse result per backbone. Subj., Bkgd., Imag., and Aesth. denote subject consistency, background consistency, imaging quality, and aesthetic quality.
Figure 4: Attention runtime on NVIDIA H20. Panel (a) compares ESTRA with Sol-Attn and SVG-EAR on both backbones. Panel (b) breaks down ESTRA ’s runtime at 5% and 1% density.
Wan2.1-14B
HunyuanVideo-13B
Method
Density
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
Density
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
Density: 5–6%
Sol-Attn
5.0%
11.31
0.343
0.816
5.0%
12.29
0.453
0.545
SVG-EAR
5.9%
11.39
0.335
0.693
5.8%
18.41
0.693
0.282
ESTRA
5.0%
14.64
0.507
0.455
5.0%
19.29
0.733
0.226
Density: approximately 15%
Table 2: Fidelity at comparable attention budgets. Results average 10 prompts per backbone at the densities shown. Bold marks the best score within each density group and backbone.
Figure 5: Fidelity to dense generation across attention densities. Results average 10 prompts per backbone.
Wan2.1-14B
HunyuanVideo-13B
Variant
Density
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
Cluster selection
4.94%
11.25
0.325
0.698
17.10
0.654
0.336
+ VCM
4.94%
11.52
0.349
0.685
17.54
0.668
0.320
+ Token selection ( ESTRA )
5.00%
14.64
0.507
0.455
19.29
0.733
0.226
Table 3: End-to-end component ablation at a 5% target density. Fidelity to dense generation is averaged over 10 prompts per backbone. Rows successively add VCM and token-level selection. Cluster-level densities are measured; bold marks the best score per backbone.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Sparse-phase density ρ
Latency (s) ↓
Speedup
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
Wan2.1-14B
Sol-Attn
12.8%
3086
2.22 ×
18.88
0.587
0.296
SVG-EAR
7.1%
3077
2.22 ×
21.17
0.727
0.200
ESTRA
5.0%
3020
2.26 ×
21.52
0.746
0.191
ESTRA
1.0%
2884
2.37 ×
20.69
0.708
0.245
HunyuanVideo-13B
Appendix
Table 4: Comparison with a shared dense warmup. All methods use 10 dense steps followed by 40 sparse steps. Fidelity metrics and end-to-end latency are averaged over ten prompts per backbone. Density refers to the sparse phase; including warmup, the trajectory-averaged density is approximately 20%+0.8ρ . Speedups use the dense reference times in Table 1 . Bold marks the best result per backbone.
Cluster selection
Token selection
Budget
Density
w/o VCM
w/ VCM
w/o VCM
w/ VCM
Wan2.1-14B
1%
0.9%
0.3719
0.3484
0.2001
0.1775
3%
2.9%
0.2362
0.2205
0.1172
0.1043
5%
4.9%
0.1854
0.1720
0.0873
0.0769
10%
9.9%
0.1258
0.1149
0.0540
0.0522
Appendix
Table 5: Complete component ablation. Attention-output errors average 10 prompts per backbone. Token selection uses target density labels; whole-cluster selection uses measured densities. Bold marks the lower error within each VCM pair.
Figure 6: Cluster selection and VCM. Mass-ranked selection and SVG-EAR routing are each evaluated with and without VCM on identical inputs. Each VCM pair retains the same selected clusters. Curves use measured densities and cover all requested budgets, including SVG-EAR’s overlapping low-budget points. Errors average 10 prompts per backbone.
Wan2.1-14B
HunyuanVideo-13B
Budget
Density
w/o VCM
w/ VCM
Density
w/o VCM
w/ VCM
1%
5.6%
0.1987
0.1898
5.5%
0.1166
0.0913
3%
5.6%
0.1986
0.1898
5.5%
0.1166
0.0913
5%
5.7%
0.1954
0.1867
5.6%
0.1152
0.0900
10%
10.1%
0.1260
0.1147
10.1%
0.0772
0.0587
15%
15.1%
0.0955
0.0854
15.1%
0.0592
0.0453
Appendix
Table 6: SVG-EAR routing with and without VCM. The two variants use the same selected clusters. Budget is the requested density; density is measured. Errors average 10 prompts per backbone. All requested budgets are retained, including those with the same rounded measured density.
Figure 7: Attention-output error across methods. Production implementations are evaluated on identical QKV inputs from step 10 of dense generation. Errors cover all layers, heads, and video query rows, averaged over 10 prompts per backbone. Baselines use measured densities; ESTRA uses target densities and VCM at every point.
Wan2.1-14B
HunyuanVideo-13B
τ
Density
Error ↓
Density
Error ↓
1.3
20.7%
0.1369
17.8%
0.1251
1.7
15.6%
0.1631
12.4%
0.1540
2.2
10.8%
0.2010
7.6%
0.1934
3.0
5.7%
0.2668
3.3%
0.2626
4.0
2.5%
0.3549
1.1%
0.3512
Appendix
Table 7: Sol-Attn operating points for the offline comparison. The official kernel is evaluated at five thresholds τ ; density is measured. Errors average the same 10 prompts per backbone.
Figure 8: Additional qualitative comparisons. Each example shows the five main operating points, with full frames aligned at the indicated times.
Figure 9: Additional qualitative comparisons. Each example shows the five main operating points, with full frames aligned at the indicated times.
Figure 10: Additional qualitative comparisons. Each example shows the five main operating points, with full frames aligned at the indicated times.
Figure 11: Additional qualitative comparisons. Each example shows the five main operating points, with full frames aligned at the indicated times.
Figure 12: Additional qualitative comparisons. Each example shows the five main operating points, with full frames aligned at the indicated times.
Figure 13: Additional qualitative comparisons. Each example shows the five main operating points, with full frames aligned at the indicated times.
Figure 14: Generation at approximately 1% density on Wan2.1-14B. ESTRA preserves clear subjects and coherent scenes, whereas Sol-Attn exhibits severe block artifacts and the cluster-level variant loses substantial visual detail. Rows show aligned frames at the indicated times.
Figure 15: Generation at approximately 1% density on HunyuanVideo-13B. ESTRA retains detailed subjects and coherent scenes, while Sol-Attn and the cluster-level variant produce block artifacts and substantial blur, respectively. Rows show aligned frames at the indicated times.
Diffusion transformers are essential for high-fidelity video generation, but long token sequences make attention a dominant inference bottleneck. Training-free dynamic sparse attention alleviates this bottleneck by computing only selected key-value blocks, yet existing methods struggle to sparsify attention both efficiently and accurately for two reasons: (1) Rigid, unpredictable, and costly routing: selecting a fixed fraction of top-ranked blocks by proxy score imposes fixed budgets, whereas retaining blocks to reach a target cumulative proxy probability mass yields dynamic but potentially imbalanced budgets; both incur non-negligible overhead from computing and materializing proxy scores. (2) Lossy keep-or-drop sparsification: unselected blocks are discarded entirely, degrading accuracy under aggressive sparsity. These limitations motivate cheaper dynamic-budget routing while limiting accuracy degradation. In this paper, we introduce training-free Sol-Attn (Sparsifying online attention), which unifies dynamic routing, sparse computation, and approximation correction in a single online-softmax pass, achieving a better accuracy-efficiency trade-off in sparse attention. The core of Sol-Attn is on-the-fly block thresholding with proxy-score reuse, which selects critical blocks by comparing block proxy scores against a threshold during online softmax. This design enables dynamic yet controllable block budgets without materializing the proxy map, while directly reusing the proxy scores of unselected blocks to approximate their contribution. Experiments across image and video generation tasks show that Sol-Attn advances the quality-efficiency frontier of training-free sparse attention, delivering 2.1 times and 2.3 times end-to-end speedups for video generation and editing, respectively, while preserving visual quality.
Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55×--2.21× speedup while maintaining generation quality comparable to full attention.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.