Recent diffusion models enable high-quality video generation, but suffer from slow runtimes. The large transformer-based backbones used in these models are bottlenecked by spatiotemporal attention. In this paper, we identify that a significant fraction of token-to-token connections consistently yield negligible scores across various inputs, and their patterns often repeat across queries. Thus, the attention computation in these cases can be skipped with little to no effect on the result. This observation continues to hold for connections among local token blocks. Motivated by this, we introduce CalibAtt, a training-free method that accelerates video generation via calibrated sparse attention. CalibAtt performs an offline calibration pass that identifies block-level sparsity and repetition patterns that are stable across inputs, and compiles these patterns into optimized attention operations for each layer, head, and diffusion timestep. At inference time, we compute the selected input-dependent connections densely, and skip the unselected ones in a hardware-efficient manner. Extensive experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models at various resolutions show that CalibAtt achieves up to 1.58x end-to-end speedup, outperforming existing training-free methods while maintaining video generation quality and text-video alignment.
Figures & tables
Figure 1: Comparison of two prompts and resolutions generated with the same seed on Wan2.1 14B text-to-video. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.
Figure 2: Attention patterns across timesteps ( t ), layers ( l ), and heads ( h ). We compare post-softmax attention maps (queries × keys) for different t,l,h for the same prompt with Wan 2.1 14B [ 28 ] . Each row fixes the attention granularity (token-level or block-level). For ease of visualization, we show the first 12544 tokens out of the full sequence length of 32760 . Notably, the large block structure visible in some of the maps reflects intra-frame token correspondences in the video.
Figure 3: Data-independence of block sparsity. (a) Attention maps from layer 20, head 24, at timestep 10 across four different prompts, showing consistent sparsity patterns. (b) Histogram showing how often each block is marked to be kept across calibration prompts. A value of 0 means the block is skipped for all prompts, while 1 means it is always computed. Many blocks cluster near 0 or 1, indicating a largely data-independent sparsity pattern. The curve (purple) shows the cumulative fraction of blocks skipped under different agreement thresholds.
Figure 4: Spatial repetition within frames. (a) Token-level attention map at layer 30, head 24, timestep 0, showing two frames ( 3120 of 32760 tokens). Frame boundaries are marked in red. (b) Zoomed-in slices from each frame-to-frame block with white grid lines separating spatial rows. The attention pattern repeats across spatial rows within each query frame.
Figure 5: Schematic description of CalibAtt. (a) We threshold the top key blocks per query block. Then, we aggregate the resulting masks across prompts and store them in a mask dictionary. In addition, we identify attention heads that exhibit spatial row repetition and store them in a dictionary. (b) At inference time, for non-repetitive heads (top), we load block masks into memory and skip the computation of the unset blocks accordingly. For heads flagged as spatially repetitive (bottom), we compute attention only for selected anchor rows per frame and broadcast the outputs to neighboring rows.
Figure 6: Spatial repetition analysis. (a) Complementarity with block sparsity : Spatial similarity vs. calibrated mask sparsity for each attention map. The negative correlation indicates that the two strategies target different maps. (b) Cross-prompt consistency : Standard deviation of spatial similarity across calibration prompts. Low variance for high-similarity maps enables their reliable identification from few samples.
Model
Method
VBench ↑
Sparsity ↑
Latency ↓
Speedup ↑
Semantic
Quality
Total
Wan 2.1 14B 480p81 frames
Dense (FA3)
72.71
82.19
80.29
0%
363 s
1.00×
SpargeAttention
72.87
82.03
80.20
49.5%
317 s
1.15×
RadialAttention
76.69
81.24
80.33
49.0%
302 s
1.20×
SVG2
67.84
82.27
79.38
53.4%
291 s
1.25×
CalibAtt (Ours)
72.80
82.30
80.40
68.1%
250 s
1.45×
Table 1: VBench scores, averaged attention sparsity, and end-to-end latency for high-timestep different text-to-video generation models and different resolutions.
Res.
Method
VBench ↑
Sparsity ↑
Latency ↓
Speedup ↑
Semantic
Quality
Total
480p
Dense (FA3)
76.06
82.28
81.04
0%
14.5 s
1.00×
SpargeAttention
76.40
82.08
80.94
49.6%
13.5 s
1.07×
RadialAttention
77.02
82.46
81.37
48.4%
12.7 s
1.14×
SVG2
76.19
82.64
81.35
62.7%
24.0 s
0.60×
CalibAtt (Ours)
77.17
82.29
81.26
70.9%
11.2 s
1.29×
Table 2: VBench scores, averaged attention sparsity, and end-to-end latency on 4 -step distilled LightX2V [ 4 ] at 480p and 720p resolutions with 81 frames.
Figure 7: Calibration set size and agreement threshold. VBench Quality Score over sparsity for different calibration set sizes and agreement thresholds ρ .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S8 : Energy threshold schedule. ϵ as a function of the normalized timestep t/T for different models and resolutions.
Figure S9 : Skip-list row length distribution. Histogram of the number of intervals per skip-list row, measured on skip lists calibrated for 720p generation on Wan2.1-14B text-to-video with N=75,600 . 95% of query rows require at most ≤50 intervals, out of a theoretical maximum of 295 .
Figure S10 : Skip-list memory optimization . Each row stores interval pairs for a (timestep, head, query-block) entry within a layer; columns beyond the data are padding. (a) Full allocation to theoretical maximum width. (b) Per-layer trimming to the widest row used across all entries in the layer (red line). (c) Merging outlier rows (orange) reduces the layer’s maximum width, enabling further trimming.
Merge %
τ
Size
Red.
VBench ↑
Sparsity ↑
Latency
Speedup
(GB) ↓
↑
Qual.
Sem.
Total
↓
↑
Dense (FA3)
–
–
–
81.27
71.65
79.35
0%
1244 s
1.00×
2D skip lists
100 (no merge)
–
21.5
0%
81.41
72.81
79.69
62.50%
785 s
1.58×
99
–
11.8
45%
81.41
72.81
79.69
62.50%
785 s
1.58×
95
–
8.1
63%
81.38
72.76
79.65
62.41%
786 s
1.58×
Appendix
Table S3 : Skip-list memory footprint for Wan-T2V 14B at 720p. Reductions are relative to the per-layer trimmed 2D baseline (21.5 GB). Timestep sharing clusters masks with pair-wise IoU ≥τ .
Figure S11 : Timestep similarity of calibrated masks. Each cell (t1,t2) reports the intersection-over-union (IoU) between the skipped-block masks at timesteps t1 and t2 , averaged over all layers l and heads h . Later timesteps show higher cross-timestep similarity (brighter off-diagonal region), suggesting that masks can be shared across consecutive late denoising steps, reducing the memory footprint of storing per-timestep masks. Red contour lines mark constant-similarity levels.
Figure S12 : Trade-off between anchor row count k and approximation quality for different similarity thresholds γ . Each curve sweeps k from 1 to 15 ; circle and square markers highlight the k=1 and k=15 endpoints. (a) Sparsity vs. relative error (lower-right is better). (b) Weighted attention speedup vs. relative error, accounting for the fraction of repetitive heads at each γ (lower-right is better).
k
ms ↓
Speedup ↑
Sps. ↑
30 (baseline)
31.62
1.0×
0%
15
18.60
1.7×
50.0%
10
11.52
2.8×
66.7%
5
5.99
5.3×
83.3%
4
4.79
6.6×
86.7%
3
3.72
8.5×
90.0%
Appendix
Table S4 : Spatial repetition attention benchmark. Speedup of computing attention over k anchor query rows versus full attention, measured with FA3 on a single H100 GPU (batch 1, 40 heads, head dim 128, sequence length 32,760). Sparsity is the fraction of query rows skipped per attention map.
Figure S13 : Spatial repetition heads count at τ=0.87 . Percentage of attention heads whose spatial-similarity score exceeds τ , per timestep and layer for Wan 2.1 14B 480p.
Mask.
Sim.
Quality ↑
Semantic ↑
Total ↑
Sparsity ↑
Latency ↓
Speedup ↑
GPU
Prompts
Prompts
hrs ↓
Dense (FA3)
81.27
71.65
79.35
0%
1244 s
1.00×
–
64
64
81.41
72.81
79.69
62.5%
785 s
1.58×
89.6
64
1
81.62
71.24
79.55
63.4%
772 s
1.61×
53.0
32
1
81.47
72.47
79.67
62.0%
796 s
1.56×
27.0
16
1
81.36
71.81
79.45
61.3%
801 s
1.55×
13.7
Appendix
Table S5 : Impact of calibration budget on VBench scores for Wan 2.1 14B at 720p. Block energies are computed on all calibration prompts; spatial similarity uses a subset. GPU-hours on a single H100. Dense FA3 as reference.
Figure S14 : Effect of block size on achievable sparsity. (a) Fine blocks enable finer-grained selection. (b) Large blocks reduce sparsity.
Method
Bq×Bkv
Quality ↑
Semantic ↑
Total ↑
Sparsity ↑
Latency ↓
Speedup ↑
Dense (FA3)
128×176
82.19
72.71
80.29
0%
363 s
1.00×
CalibAtt (Ours)
128×64
82.16
72.91
80.31
64.9%
265 s
1.37×
128×80
82.08
73.12
80.29
64.6%
259 s
1.40×
128×96
81.92
74.00
80.34
64.1%
258 s
1.41×
128×128
82.23
72.91
80.37
63.4%
257 s
1.41×
128×144
82.18
73.05
80.35
63.1%
258 s
1.41×
Appendix
Table S6 : Effect of kernel block size on sparsity and quality for Wan 2.1 14B at 480p. Speedup is relative to dense FlashAttention-3 baseline. Latency is per-prompt wall-clock time on a single H100 GPU.
Figure S15 : Comparison of four prompts generated with the same seed on Wan2.1 14B 480p text-to-video. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.
Figure S16 : Comparison of four prompts generated with the same seed on Wan2.1 14B 720p text-to-video. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.
Figure S17 : Comparison of four prompts generated with the same seed on Mochi. CalibAtt achieves higher attention sparsity and lower end-to-end latency while maintaining visual quality and prompt alignment.
Figure S18 : Comparison of four prompts generated with the same seed on LightX2V (distilled Wan2.1 14B) at 480p.
Figure S19 : Comparison of four prompts generated with the same seed on LightX2V (distilled Wan2.1 14B) at 720p.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.
Diffusion-based video generation has advanced substantially in visual fidelity and temporal coherence, but practical deployment remains limited by the quadratic complexity of full attention. Training-free sparse attention is attractive because it accelerates pretrained models without retraining, yet existing online top-p sparse attention still spends non-negligible cost on mask prediction and applies shared thresholds despite strong head-level heterogeneity. We show that these two overlooked factors limit the practical speed-quality trade-off of training-free sparse attention in Video DiTs. To address them, we introduce a head-wise adaptive framework with two plug-in components: Temporal Mask Reuse, which skips unnecessary mask prediction based on query-key drift, and Error-guided Budgeted Calibration, which assigns per-head top-p thresholds by minimizing measured model-output error under a global sparsity budget. On Wan2.1-1.3B and Wan2.1-14B, our method consistently improves XAttention and SVG2, achieving up to 1.93 times speedup at 720P while maintaining competitive video quality and similarity metrics.
Xuzhe Zheng, Yuexiao Ma, Jing Xu +3
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention. Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip. To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps. We further observe that blocks selected early continue to capture a large fraction of the total attention weight at later steps. These findings suggest two acceleration opportunities: the sparsity ratio should grow as denoising progresses, and earlier search results can be reused to reduce the search space. Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead. GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead. Experiments on HunyuanVideo and Wan 2.1 validate that GSRS achieves 1.55×--2.21× speedup while maintaining generation quality comparable to full attention.