The quadratic computational complexity of self-attention mechanisms imposes a critical constraint on transformer-based video generation, particularly for synthesizing ultra-long and high-definition sequences. While most sparse attention techniques employ fixed or heuristic patterns, they often struggle to accommodate the heterogeneous and layer-specific redundancies inherent in video diffusion transformers (VDiTs). We find that different attention heads consistently concentrate most of their attention mass in different coarse query-key regions, a behavior we call functional specialization. Although individual attention values vary with the generated content, these regions remain similar across prompts and random seeds. Leveraging these insights, we introduce Compact Attention, a hardware-aware sparse attention framework for efficient long-video synthesis. We quantify this stability and use it to construct searchable, reusable, head-specific masks. Based on the observation, we propose a deformable dual-window sparse pattern mechanism that flexibly approximates local and axial attention regions within one tile-aligned mask family. To avoid manually choosing a mask for each head, we develop a recall-driven offline auto-search algorithm that identifies high-quality sparse masks while leveraging temporal stability for mask reuse. Extensive evaluations on Wan2.1 (14B) and Hunyuan-Video demonstrate that Compact Attention achieves 2.25x/3.03x attention speedups over FlashAttention-2 and 1.20x/1.74x over FlashAttention-3, respectively, while maintaining visual quality and temporal consistency comparable to full-attention baselines. Project Page: https://yo-ava.github.io/Compact-Attention.github.io/
Figures & tables
Fig. 1: The motivation and overview of Compact Attention. (1) Fixed masks may waste computation or miss important attention regions; blue intensity denotes reference attention mass, while red overlays and outlines denote retained query–key regions. (2) Different heads exhibit local, axial, or dense coarse concentration regions. (3) Compact Attention uses two deformable windows and temporal-distance groups to represent the observed spatial and temporal structures.
Fig. 2: Coarse concentration regions of attention heads in VDiTs. We collect attention scores from Wan2.1-14B across multiple generation runs. Query-centric attention maps, with the query token at the spatial center of the middle frame, are arranged in a film-strip layout. The two examples show a relatively static chameleon and a fast-moving car. Each column presents 11 attention heads over seven frames. Across prompts and random seeds, the same head tends to concentrate attention in similar coarse regions, while the values within those regions remain input dependent.
Fig. 3: Overall pipeline of Compact Attention. (Top) Offline automated search iteratively shrinks dual windows under the recall ( τ ) and cost ( λ ) constraints and stores the resulting sparse mask configurations. (Bottom) Online inference loads the precomputed masks into block-sparse attention kernels and reuses each configuration over a fixed interval of adjacent denoising steps.
Fig. 4: Detailed visualization of the offline mask auto-search process. Starting from full attention coverage across different temporal distance groups, the dual windows are iteratively shrunk while satisfying the recall and shrink-penalty constraints. The resulting binary mask retains the coarse query–key regions that contain most of each head’s attention mass.
Model (seq_len)
Method
Sparsity
Subject Consistency
Background Consistency
Aesthetic Quality
CLIPSIM
CLIP-T
Wan2.1 [ 15 ] (80K)
FlashAttention-2 [ 56 ]
0.00%
0.9507
0.9623
0.6246
0.1933
0.9989
FlashAttention-3 [ 52 ]
0.00%
0.9505
0.9597
0.6286
0.1943
0.9986
Sparse VideoGen1 [ 8 ]
25.02%
0.9487
0.9564
0.6112
0.1926
0.9985
Sparse VideoGen2 [ 47 ]
28.10%
0.9436
0.9592
0.6214
0.1942
0.9986
SpargeAttn [ 10 ]
32.27%
0.9286
0.9434
0.5898
0.1910
0.9982
Compact Attention (Ours)
33.99%
0.9491
0.9586
0.6219
0.1921
0.9985
TABLE I: Quantitative Comparison of Sparse Attention Methods in Wan2.1 and Hunyuan on Prompts from the Open-Sora Benchmark and VBench: Visual Consistency, Aesthetic Quality, Text-Video Alignment, and Temporal Semantic Consistency.
Model (seq_len)
Method
Sparsity
Quality
Attention Efficiency
SSIM ↑
PSNR ↑
MSE ↓
Attention Latency (s)
vs. FA2
vs. FA3
Wan2.1 [ 15 ] (80K)
FlashAttention-2 [ 56 ]
0.00%
-
-
-
1496.7s
1.00x
0.53x
FlashAttention-3 [ 52 ]
0.00%
-
-
-
795.63s
1.88x
1.00x
Sparse VideoGen1 [ 8 ]
25.02%
0.7115
21.1444
0.0109
1107.9s
1.35x
0.72x
Sparse VideoGen2 [ 47 ]
28.10%
0.7428
22.0998
0.0094
1118.8s
1.34x
0.71x
SpargeAttn [ 10 ]
32.27%
0.6767
20.1787
0.0120
1065.8s
1.40x
0.75x
TABLE II: Quality and attention-efficiency comparison on a single H800 GPU. Attention latency is accumulated over the generation process, and speedups are reported relative to FA2 and FA3.
Fig. 5: End-to-end performance and sparsity trends. (a-b) Latency breakdown and PSNR evaluation on Wan2.1 and Hunyuan, showing notable speedups with minimal quality degradation. Here, “baseline” denotes FA2, and “ours (0.0%)” denotes the dense Compact kernel. (c) The impact of different recall thresholds on the achieved sparsity for auto-searched attention patterns in both models.
Fig. 6: Visual qualitative comparison on the Hunyuan-Video model. We compare the generated frames using Full Attention against Compact Attention at a sparsity level of 62.36%. As shown, Compact Attention preserves the semantic details and temporal consistency (e.g., facial features and background structures) while skipping over half of the attention computation.
Mask Components
Spatial-Pattern Sparsity
Temporal-Bias Sparsity
Overall Sparsity
Cubic
Frame-group
Dual Window
Locality
Cross
Global
Time-Variant
Time-Invariant
✓
72.6%
38.5%
7.8%
44.1%
30.6%
36.1%
✓
✓
75.8%
40.6%
8.5%
47.2%
31.7%
37.0% (+0.9 pp)
✓
✓
✓
76.6%
51.6%
9.9%
56.7%
38.5%
45.9% (+8.9 pp)
TABLE III: Ablation study on the sparse mask design. Components are added cumulatively , progressively increasing the achievable sparsity. Category-wise and overall sparsity are reported as percentages. Values in parentheses indicate the incremental increase in percentage points (pp) from the previous row.
Model
Assignment
Sparsity
PSNR
Wan2.1
Correct per-head
33.99%
22.97
Wan2.1
Shared within layer
33.99%
18.65
Wan2.1
Random head assignment
33.99%
18.31
Wan2.1
Random Mask Search
33.99%
19.12
Hunyuan
Correct per-head
62.36%
27.48
Hunyuan
Shared within layer
62.36%
22.86
TABLE IV: Head-specific mask assignment with matched per-layer budgets. Mean PSNR is reported.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.
Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55×--2.21× speedup while maintaining generation quality comparable to full attention.
Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1× end-to-end speedup and a 10.5× self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.
Shihao Han, Hao Yang, Xinting Hu +3
ByteDance Inc. · University of Science and Technology of China · The University of Hong Kong