Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering

Jiayi Luo; Jiayu Chen; Jiankun Wang; Cong Wang; Hanxin Zhu; Qingyun Sun; Chen Gao; Zhibo Chen

cs.CVMar 19, 2026

Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering

Authors: Jiayi Luo, Jiayu Chen, Jiankun Wang, Cong Wang, Hanxin Zhu, Qingyun Sun, Chen Gao, Zhibo Chen,

Organizations: SKLCCSE, School of Computer Science and Engineering, Beihang University · Zhongguancun Academy · School of Computer Science, Peking University · Beihang University ·

Abstract

Diffusion Transformers (DiTs) achieve strong video generation quality but suffer from high inference cost due to dense 3D attention, motivating sparse attention techniques for improving efficiency. However, existing training-free sparse attention methods for video generation still face two unresolved limitations: ignoring layer heterogeneity in attention pruning and ignoring query-key coupling in block partitioning, which hinder a better quality-speedup trade-off. In this work, we uncover a critical insight: attention sparsity is an intrinsic layer-wise property, with only minor variation across different inputs. Motivated by this observation, we propose SVOO, a training-free sparse attention framework for fast video generation via offline layer-wise sparsity profiling and online bidirectional co-clustering. Specifically, SVOO adopts a two-stage paradigm: (i) offline layer-wise sensitivity profiling to derive intrinsic per-layer pruning levels, and (ii) online block-wise sparse attention via a bidirectional co-clustering algorithm. Extensive experiments on seven widely used video generation models demonstrate that SVOO achieves a superior quality-speedup trade-off over state-of-the-art methods, delivering up to 1.93x speedup while maintaining a PSNR of up to 29 dB on Wan2.1. Code is available at: https://github.com/Mutual-Luo/SVOO.

Figures & tables

Figure 1: An example acceleration comparison on the Wan2.1-T2V-1.3B ( Wan et al., 2025 ) . All experiments are conducted on a single NVIDIA H200 GPU at a 720p resolution with 81 frames.

Figure 2: Layer-wise attention sparsity across different models. The figure shows that attention density varies substantially across layers ( layer-wise heterogeneity ), while remaining highly stable for each layer across different inputs ( layer-wise stability ) .

Figure 3: Illustration of query-key coupling in block partitioning, which is important for block-wise sparse attention. The example shows that the optimal partitioning of Keys is Query-dependent , as different Queries induce different optimal groupings of Keys.

Figure 4: The framework of SVOO consists of two stages for accelerating video generation. Offline stage (left): we profile the intrinsic attention sparsity of each transformer layer and derive a layer-wise sparsity schedule. Online stage (right): we perform bidirectional co-clustering to partition queries and keys into coupled blocks, and then select salient block pairs according to the offline schedule.

Model	Config	Baseline	Quality							Efficiency
Model	Config	Baseline	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ImageQual $\uparrow$	AesQual $\uparrow$	SubConsist $\uparrow$	BackConsist $\uparrow$	Latency	Speedup
Wan2.1	1.3B-720P-T2V	Origin	-	-	-	66.58%	64.47%	96.74%	97.30%	417s	1.00 $\times$
		SpargeAttn	25.137	0.801	0.234	63.08%	62.75%	95.79%	97.09%	288s	1.45 $\times$
		SVG1	25.712	0.811	0.230	63.26%	61.14%	94.27%	91.16%	266s	1.56 $\times$
		SVG2	29.268	0.886	0.127	61.83%	60.15%	95.88%	96.47%	241s	1.73 $\times$
		Radial	26.305	0.829	0.182	63.67%	62.76%	96.56%	97.07%	257s	1.62 $\times$

Table 1: Quality and efficiency benchmarking results of our proposed SVOO and baselines on Text-to-Video Task.

Model	Config	Baseline	Quality							Efficiency
Model	Config	Baseline	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ImageQual $\uparrow$	AesQual $\uparrow$	SubConsist $\uparrow$	BackConsist $\uparrow$	Latency	Speedup
Wan2.1	14B-720P-I2V	Origin	-	-	-	71.52%	61.86%	94.98%	95.37%	1658s	1.00 $\times$
		SpargeAttn	21.557	0.691	0.297	70.83%	61.77%	95.81%	95.04%	1124s	1.48 $\times$
		SVG1	24.262	0.825	0.174	70.93%	61.02%	94.42%	94.51%	1047s	1.58 $\times$
		SVG2	27.324	0.856	0.125	70.76%	61.45%	95.72%	94.93%	998s	1.66 $\times$
		Radial	23.673	0.759	0.189	70.91%	61.62%	94.52%	95.36%	1046s	1.58 $\times$

Table 2: Quality and efficiency benchmarking results of our proposed SVOO and baselines on Image-to-Video Task.

Figure 5: The ablation study of our proposed SVOO.

Figure 6: Examples of videos generated by our proposed SVOO and dense attention on Wan and HunyuanVideo models.

Figure 7: Quality-efficiency trade-off of SVOO.

Figure 8: Analysis of clustering result across diffusion steps.

Figure 9: Attention recall comparison between SVOO and SVG2.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Additional layer-wise attention sparsity across different models.

Figure 11: Layer attention sparsity across different steps of representative models.

Efficient Video Generation

See all 25 papers

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
Oct 15, 2026Yun Dai, Yuyun Chen, Jiarui Wen +3Sparse Attention Diffusion Model Inference Acceleration
South China University of Technology
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Video Diffusion Models Sparse Attention
NVIDIA

Oct 15, 2026J/K move · Enter open · S save

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

Yun Dai, Yuyun Chen, Jiarui Wen, Huiping Zhuang+2

South China University of Technology

Sparse Attention Diffusion Model Inference Acceleration Video Diffusion Models Video Generation Diffusion Transformer

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55

\times

--2.21

\times

speedup while maintaining generation quality comparable to full attention.

Abstract

Efficient Video Generation

Explore similar work

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification