Accelerating Text-to-Video Generation with Calibrated Sparse Attention

Shai Yehezkel; Shahar Yadin; Noam Elata; Yaron Ostrovsky-Berman; Bahjat Kawar

cs.CVMar 5, 2026

Accelerating Text-to-Video Generation with Calibrated Sparse Attention

Authors: Shai Yehezkel, Shahar Yadin, Noam Elata, Yaron Ostrovsky-Berman, Bahjat Kawar

Organizations: Apple · Tel Aviv University

Abstract

Recent diffusion models enable high-quality video generation, but suffer from slow runtimes. The large transformer-based backbones used in these models are bottlenecked by spatiotemporal attention. In this paper, we identify that a significant fraction of token-to-token connections consistently yield negligible scores across various inputs, and their patterns often repeat across queries. Thus, the attention computation in these cases can be skipped with little to no effect on the result. This observation continues to hold for connections among local token blocks. Motivated by this, we introduce CalibAtt, a training-free method that accelerates video generation via calibrated sparse attention. CalibAtt performs an offline calibration pass that identifies block-level sparsity and repetition patterns that are stable across inputs, and compiles these patterns into optimized attention operations for each layer, head, and diffusion timestep. At inference time, we compute the selected input-dependent connections densely, and skip the unselected ones in a hardware-efficient manner. Extensive experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models at various resolutions show that CalibAtt achieves up to 1.58x end-to-end speedup, outperforming existing training-free methods while maintaining video generation quality and text-video alignment.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of two prompts and resolutions generated with the same seed on Wan2.1 14B text-to-video. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.

Panel of Figure 2 — Figure 2: Attention patterns across timesteps ( $t$ ), layers ( $l$ ), and heads ( $h$ ). We compare post-softmax attention maps (queries $\times$ keys) for different $t,l,h$ for the same prompt with Wan 2.1 14B [ 28 ] . Each row fixes the attention granularity (token-level or block-level). For ease of visualization, we show the first $12544$ tokens out of the full sequence length of ${32760}$ . Notably, the large block structure visible in some of the maps reflects intra-frame token correspondences in the video.

Panel of Figure 3 — Figure 3: Data-independence of block sparsity. (a) Attention maps from layer 20, head 24, at timestep 10 across four different prompts, showing consistent sparsity patterns. (b) Histogram showing how often each block is marked to be kept across calibration prompts. A value of 0 means the block is skipped for all prompts, while 1 means it is always computed. Many blocks cluster near 0 or 1, indicating a largely data-independent sparsity pattern. The curve (purple) shows the cumulative fraction of blocks skipped under different agreement thresholds.

Panel of Figure 4 — Figure 4: Spatial repetition within frames. (a) Token-level attention map at layer 30, head 24, timestep 0, showing two frames ( $3120$ of $32760$ tokens). Frame boundaries are marked in red. (b) Zoomed-in slices from each frame-to-frame block with white grid lines separating spatial rows. The attention pattern repeats across spatial rows within each query frame.

Panel of Figure 5 — Figure 5: Schematic description of CalibAtt. (a) We threshold the top key blocks per query block. Then, we aggregate the resulting masks across prompts and store them in a mask dictionary. In addition, we identify attention heads that exhibit spatial row repetition and store them in a dictionary. (b) At inference time, for non-repetitive heads (top), we load block masks into memory and skip the computation of the unset blocks accordingly. For heads flagged as spatially repetitive (bottom), we compute attention only for selected anchor rows per frame and broadcast the outputs to neighboring rows.

Panel of Figure 6 — Figure 6: Spatial repetition analysis. (a) Complementarity with block sparsity : Spatial similarity vs. calibrated mask sparsity for each attention map. The negative correlation indicates that the two strategies target different maps. (b) Cross-prompt consistency : Standard deviation of spatial similarity across calibration prompts. Low variance for high-similarity maps enables their reliable identification from few samples.

Model	Method	VBench $\uparrow$			Sparsity $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
Model	Method	Semantic	Quality	Total	Sparsity $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
Wan 2.1 14B ${480p}$ $81$ frames	Dense (FA3)	$72.71$	$82.19$	$80.29$	$0\%$	$363$ s	$1.00\times$
	SpargeAttention	$72.87$	$82.03$	$80.20$	$49.5\%$	$317$ s	$1.15\times$
	RadialAttention	$76.69$	$81.24$	$80.33$	$49.0\%$	$302$ s	$1.20\times$
	SVG2	$67.84$	$82.27$	$79.38$	$53.4\%$	$291$ s	$1.25\times$
	CalibAtt (Ours)	$72.80$	$82.30$	$80.40$	$\textbf{68.1}\%$	250 s	$\textbf{1.45}\times$

Table 1: VBench scores, averaged attention sparsity, and end-to-end latency for high-timestep different text-to-video generation models and different resolutions.

Res.	Method	VBench $\uparrow$			Sparsity $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
Res.	Method	Semantic	Quality	Total	Sparsity $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
$\mathbf{480p}$	Dense (FA3)	$76.06$	$82.28$	$81.04$	$0\%$	$14.5$ s	$1.00\times$
	SpargeAttention	$76.40$	$82.08$	$80.94$	$49.6\%$	$13.5$ s	$1.07\times$
	RadialAttention	$77.02$	$82.46$	$81.37$	$48.4\%$	$12.7$ s	$1.14\times$
	SVG2	$76.19$	$82.64$	$81.35$	$62.7\%$	$24.0$ s	$0.60\times$
	CalibAtt (Ours)	$77.17$	$82.29$	$81.26$	$\textbf{70.9}\%$	11.2 s	$\textbf{1.29}\times$

Table 2: VBench scores, averaged attention sparsity, and end-to-end latency on

4

-step distilled LightX2V [ 4 ] at

480p

and

720p

resolutions with 81 frames.

Figure 7: Calibration set size and agreement threshold. VBench Quality Score over sparsity for different calibration set sizes and agreement thresholds

\rho

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S8 : Energy threshold schedule.

\epsilon

as a function of the normalized timestep

t/T

for different models and resolutions.

Figure S9 : Skip-list row length distribution. Histogram of the number of intervals per skip-list row, measured on skip lists calibrated for 720p generation on Wan2.1-14B text-to-video with

N{=}75{,}600

. 95% of query rows require at most

\leq 50

intervals, out of a theoretical maximum of

295

Panel of Figure 12 — Figure S10 : Skip-list memory optimization . Each row stores interval pairs for a (timestep, head, query-block) entry within a layer; columns beyond the data are padding. (a) Full allocation to theoretical maximum width. (b) Per-layer trimming to the widest row used across all entries in the layer (red line). (c) Merging outlier rows (orange) reduces the layer’s maximum width, enabling further trimming.

Merge %	$\tau$	Size	Red.	VBench $\uparrow$			Sparsity $\uparrow$	Latency	Speedup
Merge %	$\tau$	(GB) $\downarrow$	$\uparrow$	Qual.	Sem.	Total	Sparsity $\uparrow$	$\downarrow$	$\uparrow$
Dense (FA3)	–	–	–	$81.27$	$71.65$	$79.35$	$0\%$	$1244$ s	$1.00\times$
2D skip lists
100 (no merge)	–	$21.5$	$0\%$	$81.41$	$72.81$	$79.69$	$62.50\%$	$785$ s	$1.58\times$
99	–	$11.8$	$45\%$	$81.41$	$72.81$	$79.69$	$62.50\%$	$785$ s	$1.58\times$
95	–	$8.1$	$63\%$	$81.38$	$72.76$	$79.65$	$62.41\%$	$786$ s	$1.58\times$

Appendix

Table S3 : Skip-list memory footprint for Wan-T2V 14B at 720p. Reductions are relative to the per-layer trimmed 2D baseline (21.5 GB). Timestep sharing clusters masks with pair-wise IoU

\geq\tau

Figure S11 : Timestep similarity of calibrated masks. Each cell $(t_{1},t_{2})$ reports the intersection-over-union (IoU) between the skipped-block masks at timesteps $t_{1}$ and $t_{2}$ , averaged over all layers $l$ and heads $h$ . Later timesteps show higher cross-timestep similarity (brighter off-diagonal region), suggesting that masks can be shared across consecutive late denoising steps, reducing the memory footprint of storing per-timestep masks. Red contour lines mark constant-similarity levels.

Panel of Figure 15 — Figure S12 : Trade-off between anchor row count $k$ and approximation quality for different similarity thresholds $\gamma$ . Each curve sweeps $k$ from $1$ to $15$ ; circle and square markers highlight the $k{=}1$ and $k{=}15$ endpoints. (a) Sparsity vs. relative error (lower-right is better). (b) Weighted attention speedup vs. relative error, accounting for the fraction of repetitive heads at each $\gamma$ (lower-right is better).

$k$	ms $\downarrow$	Speedup $\uparrow$	Sps. $\uparrow$
30 (baseline)	$31.62$	$1.0\times$	$0\%$
15	$18.60$	$1.7\times$	$50.0\%$
10	$11.52$	$2.8\times$	$66.7\%$
5	$5.99$	$5.3\times$	$83.3\%$
4	$4.79$	$6.6\times$	$86.7\%$
3	$3.72$	$8.5\times$	$90.0\%$

Appendix

Table S4 : Spatial repetition attention benchmark. Speedup of computing attention over

k

anchor query rows versus full attention, measured with FA3 on a single H100 GPU (batch 1, 40 heads, head dim 128, sequence length 32,760). Sparsity is the fraction of query rows skipped per attention map.

Figure S13 : Spatial repetition heads count at $\tau=0.87$ . Percentage of attention heads whose spatial-similarity score exceeds $\tau$ , per timestep and layer for Wan 2.1 14B 480p.

Mask.	Sim.	Quality $\uparrow$	Semantic $\uparrow$	Total $\uparrow$	Sparsity $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$	GPU
Prompts	Prompts	Quality $\uparrow$	Semantic $\uparrow$	Total $\uparrow$	Sparsity $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$	hrs $\downarrow$
Dense (FA3)		$81.27$	$71.65$	$79.35$	$0\%$	$1244$ s	$1.00\times$	–
64	64	$81.41$	$72.81$	$79.69$	$62.5\%$	$785$ s	$1.58\times$	$89.6$
64	1	$81.62$	$71.24$	$79.55$	$63.4\%$	$772$ s	$1.61\times$	$53.0$
32	1	$81.47$	$72.47$	$79.67$	$62.0\%$	$796$ s	$1.56\times$	$27.0$
16	1	$81.36$	$71.81$	$79.45$	$61.3\%$	$801$ s	$1.55\times$	$13.7$

Appendix

Table S5 : Impact of calibration budget on VBench scores for Wan 2.1 14B at 720p. Block energies are computed on all calibration prompts; spatial similarity uses a subset. GPU-hours on a single H100. Dense FA3 as reference.

Panel of Figure 19 — Figure S14 : Effect of block size on achievable sparsity. (a) Fine blocks enable finer-grained selection. (b) Large blocks reduce sparsity.

Method	$B_{q}\times B_{kv}$	Quality $\uparrow$	Semantic $\uparrow$	Total $\uparrow$	Sparsity $\uparrow$	Latency $\downarrow$	Speedup $\uparrow$
Dense (FA3)	$128\times 176$	$82.19$	$72.71$	$80.29$	$0\%$	$363$ s	$1.00\times$
CalibAtt (Ours)	$128\times\ 64$	$82.16$	$72.91$	$80.31$	$64.9\%$	$265$ s	$1.37\times$
	$128\times\ 80$	$82.08$	$73.12$	$80.29$	$64.6\%$	$259$ s	$1.40\times$
	$128\times\ 96$	$81.92$	$74.00$	$80.34$	$64.1\%$	$258$ s	$1.41\times$
	$128\times 128$	$82.23$	$72.91$	$80.37$	$63.4\%$	$257$ s	$1.41\times$
	$128\times 144$	$82.18$	$73.05$	$80.35$	$63.1\%$	$258$ s	$1.41\times$

Appendix

Table S6 : Effect of kernel block size on sparsity and quality for Wan 2.1 14B at 480p. Speedup is relative to dense FlashAttention-3 baseline. Latency is per-prompt wall-clock time on a single H100 GPU.

Panel of Figure 21 — Figure S15 : Comparison of four prompts generated with the same seed on Wan2.1 14B 480p text-to-video. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.

Panel of Figure 22 — Figure S16 : Comparison of four prompts generated with the same seed on Wan2.1 14B 720p text-to-video. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.

Panel of Figure 23 — Figure S17 : Comparison of four prompts generated with the same seed on Mochi. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.

Panel of Figure 24 — Figure S18 : Comparison of four prompts generated with the same seed on LightX2V (distilled Wan2.1 14B) at 480p.

Panel of Figure 25 — Figure S19 : Comparison of four prompts generated with the same seed on LightX2V (distilled Wan2.1 14B) at 720p.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 491 papers

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention
May 14, 2026Xuzhe Zheng, Yuexiao Ma, Jing Xu +3Video Diffusion Models Diffusion Transformer
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
Oct 15, 2026Yun Dai, Yuyun Chen, Jiarui Wen +3Sparse Attention Diffusion Model Inference Acceleration
South China University of Technology

May 22, 2026J/K move · Enter open · S save

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Jie Hu, Zixiang Gao, Yutong He, Kun Yuan

1Peking University.

Video Diffusion Models Diffusion Transformer Transformer Attention Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Open paper AskPDF arXiv

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1

\times

end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Abstract

Diffusion Model Inference Acceleration

Explore similar work

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention

Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search