Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation

Qirui Li; Guangcong Zheng; Qi Zhao; Jie Li; Bin Dong; Yiwu Yao; Xuewei Li; Zequn Qin

cs.CVAug 18, 2025

Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation

Authors: Qirui Li, Guangcong Zheng, Qi Zhao, Jie Li, Bin Dong, Yiwu Yao, Xuewei Li, Zequn Qin,

Organizations: College of Computer Science & Technology, Zhejiang University · Huawei Technologies

Abstract

The quadratic computational complexity of self-attention mechanisms imposes a critical constraint on transformer-based video generation, particularly for synthesizing ultra-long and high-definition sequences. While most sparse attention techniques employ fixed or heuristic patterns, they often struggle to accommodate the heterogeneous and layer-specific redundancies inherent in video diffusion transformers (VDiTs). We find that different attention heads consistently concentrate most of their attention mass in different coarse query-key regions, a behavior we call functional specialization. Although individual attention values vary with the generated content, these regions remain similar across prompts and random seeds. Leveraging these insights, we introduce Compact Attention, a hardware-aware sparse attention framework for efficient long-video synthesis. We quantify this stability and use it to construct searchable, reusable, head-specific masks. Based on the observation, we propose a deformable dual-window sparse pattern mechanism that flexibly approximates local and axial attention regions within one tile-aligned mask family. To avoid manually choosing a mask for each head, we develop a recall-driven offline auto-search algorithm that identifies high-quality sparse masks while leveraging temporal stability for mask reuse. Extensive evaluations on Wan2.1 (14B) and Hunyuan-Video demonstrate that Compact Attention achieves 2.25x/3.03x attention speedups over FlashAttention-2 and 1.20x/1.74x over FlashAttention-3, respectively, while maintaining visual quality and temporal consistency comparable to full-attention baselines. Project Page: https://yo-ava.github.io/Compact-Attention.github.io/

Figures & tables

Fig. 1: The motivation and overview of Compact Attention. (1) Fixed masks may waste computation or miss important attention regions; blue intensity denotes reference attention mass, while red overlays and outlines denote retained query–key regions. (2) Different heads exhibit local, axial, or dense coarse concentration regions. (3) Compact Attention uses two deformable windows and temporal-distance groups to represent the observed spatial and temporal structures.

Fig. 2: Coarse concentration regions of attention heads in VDiTs. We collect attention scores from Wan2.1-14B across multiple generation runs. Query-centric attention maps, with the query token at the spatial center of the middle frame, are arranged in a film-strip layout. The two examples show a relatively static chameleon and a fast-moving car. Each column presents 11 attention heads over seven frames. Across prompts and random seeds, the same head tends to concentrate attention in similar coarse regions, while the values within those regions remain input dependent.

Fig. 3: Overall pipeline of Compact Attention. (Top) Offline automated search iteratively shrinks dual windows under the recall (

\tau

) and cost (

\lambda

) constraints and stores the resulting sparse mask configurations. (Bottom) Online inference loads the precomputed masks into block-sparse attention kernels and reuses each configuration over a fixed interval of adjacent denoising steps.

Fig. 4: Detailed visualization of the offline mask auto-search process. Starting from full attention coverage across different temporal distance groups, the dual windows are iteratively shrunk while satisfying the recall and shrink-penalty constraints. The resulting binary mask retains the coarse query–key regions that contain most of each head’s attention mass.

Model (seq_len)	Method	Sparsity	Subject Consistency	Background Consistency	Aesthetic Quality	CLIPSIM	CLIP-T
Wan2.1 [ 15 ] (80K)	FlashAttention-2 [ 56 ]	0.00%	0.9507	0.9623	0.6246	0.1933	0.9989
	FlashAttention-3 [ 52 ]	0.00%	0.9505	0.9597	0.6286	0.1943	0.9986
	Sparse VideoGen1 [ 8 ]	25.02%	0.9487	0.9564	0.6112	0.1926	0.9985
	Sparse VideoGen2 [ 47 ]	28.10%	0.9436	0.9592	0.6214	0.1942	0.9986
	SpargeAttn [ 10 ]	32.27%	0.9286	0.9434	0.5898	0.1910	0.9982
	Compact Attention (Ours)	33.99%	0.9491	0.9586	0.6219	0.1921	0.9985

TABLE I: Quantitative Comparison of Sparse Attention Methods in Wan2.1 and Hunyuan on Prompts from the Open-Sora Benchmark and VBench: Visual Consistency, Aesthetic Quality, Text-Video Alignment, and Temporal Semantic Consistency.

Model (seq_len)	Method	Sparsity	Quality			Attention Efficiency
Model (seq_len)	Method	Sparsity	SSIM $\uparrow$	PSNR $\uparrow$	MSE $\downarrow$	Attention Latency (s)	vs. FA2	vs. FA3
Wan2.1 [ 15 ] (80K)	FlashAttention-2 [ 56 ]	0.00%	-	-	-	1496.7s	1.00x	0.53x
	FlashAttention-3 [ 52 ]	0.00%	-	-	-	795.63s	1.88x	1.00x
	Sparse VideoGen1 [ 8 ]	25.02%	0.7115	21.1444	0.0109	1107.9s	1.35x	0.72x
	Sparse VideoGen2 [ 47 ]	28.10%	0.7428	22.0998	0.0094	1118.8s	1.34x	0.71x
	SpargeAttn [ 10 ]	32.27%	0.6767	20.1787	0.0120	1065.8s	1.40x	0.75x

TABLE II: Quality and attention-efficiency comparison on a single H800 GPU. Attention latency is accumulated over the generation process, and speedups are reported relative to FA2 and FA3.

Fig. 5: End-to-end performance and sparsity trends. (a-b) Latency breakdown and PSNR evaluation on Wan2.1 and Hunyuan, showing notable speedups with minimal quality degradation. Here, “baseline” denotes FA2, and “ours (0.0%)” denotes the dense Compact kernel. (c) The impact of different recall thresholds on the achieved sparsity for auto-searched attention patterns in both models.

Fig. 6: Visual qualitative comparison on the Hunyuan-Video model. We compare the generated frames using Full Attention against Compact Attention at a sparsity level of 62.36%. As shown, Compact Attention preserves the semantic details and temporal consistency (e.g., facial features and background structures) while skipping over half of the attention computation.

Mask Components			Spatial-Pattern Sparsity			Temporal-Bias Sparsity		Overall Sparsity
Cubic	Frame-group	Dual Window	Locality	Cross	Global	Time-Variant	Time-Invariant	Overall Sparsity
✓			72.6%	38.5%	7.8%	44.1%	30.6%	36.1%
✓	✓		75.8%	40.6%	8.5%	47.2%	31.7%	37.0% (+0.9 pp)
✓	✓	✓	76.6%	51.6%	9.9%	56.7%	38.5%	45.9% (+8.9 pp)

TABLE III: Ablation study on the sparse mask design. Components are added cumulatively , progressively increasing the achievable sparsity. Category-wise and overall sparsity are reported as percentages. Values in parentheses indicate the incremental increase in percentage points (pp) from the previous row.

Model	Assignment	Sparsity	PSNR
Wan2.1	Correct per-head	33.99%	22.97
Wan2.1	Shared within layer	33.99%	18.65
Wan2.1	Random head assignment	33.99%	18.31
Wan2.1	Random Mask Search	33.99%	19.12
Hunyuan	Correct per-head	62.36%	27.48
Hunyuan	Shared within layer	62.36%	22.86

TABLE IV: Head-specific mask assignment with matched per-layer budgets. Mean PSNR is reported.

Efficient Video Generation

See all 31 papers

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
Oct 15, 2026Yun Dai, Yuyun Chen, Jiarui Wen +3Sparse Attention Diffusion Model Inference Acceleration
South China University of Technology
Veda: Scalable Video Diffusion via Distilled Sparse Attention
May 28, 2026Shihao Han, Hao Yang, Xinting Hu +3Video Diffusion Models Diffusion Transformer
ByteDance Inc. · University of Science and Technology of China · The University of Hong Kong

May 22, 2026J/K move · Enter open · S save

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Jie Hu, Zixiang Gao, Yutong He, Kun Yuan

1Peking University.

Video Diffusion Models Diffusion Transformer Transformer Attention Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Open paper AskPDF arXiv

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1

\times

end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Abstract

Efficient Video Generation

Explore similar work

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

Veda: Scalable Video Diffusion via Distilled Sparse Attention