Video diffusion transformers rely on 3D full attention over tens of thousands of spatiotemporal tokens, and its quadratic cost dominates inference latency. Sparse attention reduces this cost, but existing methods face a dilemma: static methods apply predefined masks and cannot capture input-dependent attention, whereas dynamic methods reconstruct the sparse mask repeatedly, paying online to rediscover positional structure that is largely stable across inputs. To this end, we propose PoCoAttn, a training-free sparse attention framework that separates input-stable from input-dependent attention and resolves the former offline and the latter online. PoCoAttn features: (1) A relative-offset analysis of 3D query-key pairs, which shows that video attention decomposes into an input-stable positional prior that carries most of the attention mass and a small content-dependent residual. (2) A position-content collaborative sparse attention mechanism, which retrieves positional keys from offline-calibrated tables, selects content keys online with a head-adaptive ratio between the two sources, and compensates for the discarded keys to approximate full attention. (3) A step-adaptive budget allocation algorithm, which models how early-step sparse attention errors are amplified along the denoising trajectory and derives the optimal sparsity budget per step. Extensive experiments across four video diffusion models and three tasks show that PoCoAttn consistently achieves a better speed-fidelity Pareto frontier than state-of-the-art (SOTA) methods, improving PSNR by 1.9 to 5.0 dB over baselines and delivering up to 4.1× attention and 2.0× denoising speedups over dense FlashAttention-3 attention.
Figures & tables
Figure 2: Overview of PoCoAttn. (a) PoCoAttn decomposes attention into an input-stable positional part and an input-dependent content part. (b) At inference, each query tile looks up its positional key tiles in the positional table, selects content keys by pooled-query scores, and merges the discarded keys into virtual keys by weighted pooling; the per-head position–content ratio is solved offline. (c) The positional table is compiled offline from the per-head positional prior Ph(Δ) , obtained by averaging calibration attention over each relative offset Δ . (d) The step-wise budget ρs is solved offline by minimizing the local error et(ρt) weighted by its amplification A(t) .
Figure 3: Relative position alone captures substantial attention mass. Attention maps of three representative Wan2.1-14B heads at denoising step 10, for a query in latent frame f=10 and keys in frames f−1 , f , f+1 . Under the same 15% key budget, the positional prior extracted from the same prompt captures head-specific patterns missed by hand-crafted templates and approaches the per-query oracle. Recall measures the retained attention mass; rel. err. measures the relative ℓ2 error of the sparse attention output against the dense one.
Figure 4: Attention decomposes into an input-stable positional prior and an input-dependent content residual. Each column corresponds to a different prompt. The bottom row shows the residual attention that content adds on top of the positional prior.
Figure 5: Positional recall is head-dependent but consistent across inputs. (a) Recall of the 40 heads in layer 30 of Wan2.1-14B at a 25% key budget: positional table alone, gain from moving half the budget to online content keys, and the mass left on discarded keys. (b) Per-head recall of the positional table: offline estimate (8 calibration prompts) versus actual on 8 unseen prompts.
Figure 6: Selection and compensation are complementary. (a) Single-layer attention error without compensation, versus ratio η ; grey bars show the share of heads assigned each η . (b) Error left relative to the positional table alone; compensation removes most of the error that selection leaves on global heads.
Figure 7: Step-dependent error and amplification. One 5-step window runs oracle block-sparse attention; all other steps are dense. (a) Final PSNR. Higher is better. (b) Local attention error induced in the window. (c) Amplification of that error to the final output; Gaussian noise follows the same decay.
Table 1: Quantitative comparison of video generation quality and efficiency. Bold indicates the best sparse-method results. Density is the fraction of query–key pairs actually computed in the sparse attention calls; percentages in parentheses are target density settings. † Lower speedup than originally reported, as our fused dense baseline is faster than their official dense run (App. D ).
Figure 8: Qualitative comparison of video generation. Our method achieves better visual fidelity at lower density.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Density ↓
Speedup ↑
PISA [ 14 ]
33.72
0.949
0.032
25.0%
1.28 ×
Sol-Attn [ 13 ]
33.35
0.947
0.033
26.9%
1.47 ×
PoCoAttn (25%)
37.60
0.968
0.017
25.0%
1.47 ×
PoCoAttn (15%)
35.90
0.961
0.024
15.0%
1.72 ×
Table 2: Video editing on Bernini. Speedup over dense FA3.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Density ↓
Speedup ↑
Dense ( FA4 [ 31 ] )
–
–
–
100%
1.00 ×
PISA [ 14 ]
23.71
0.805
0.154
29.9%
1.03 ×
Sol-Attn [ 13 ]
23.85
0.806
0.152
30.7%
0.99 ×
PoCoAttn
27.48
0.876
0.088
25.0%
1.44 ×
Table 3: Results on the NVIDIA B200 (Wan2.1-T2V-14B, 720p).
Pos.
Cont.
Comp.
PSNR ↑
SSIM ↑
LPIPS ↓
Attn ×↑
Speedup ↑
✓
24.12
0.811
0.149
2.38 ×
1.46 ×
✓
25.30
0.818
0.135
2.24 ×
1.42 ×
✓
✓
26.11
0.850
0.112
2.27 ×
1.43 ×
✓
✓
✓
27.81
0.884
0.081
2.07 ×
1.40 ×
Table 4: Component ablation on Wan2.1. Pos.: positional keys from the table; Cont.: online content keys; Comp.: compensation.
Table 5: Step-schedule ablation on Wan2.1. All rows run PoCoAttn at 25% average density and differ only in how the budget is allocated across denoising steps (bars: early → late). E is the error proxy in Sec. 4.3 ; Solved is our schedule that minimizes it.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Head-adaptive compilation. Positional priors of three Wan2.1 heads for one query (black outline) on a toy 4×6×8 grid. At a 25% key budget, each head selects the box shape ( t×y×x ) among all candidates that retains the most prior mass (red dashed). Percentages give the retained mass of each candidate. The head-adaptive box shapes are decided offline and reused for table construction.
Model
Step Slots
25% density
15% density
Budget Schedule
Mean η
Budget Schedule
Mean η
Wan2.1-T2V-14B
10,15,…,45
.55 .55 .50 .25 .15 .10 .05 .05
0.868
.55 .35 .20 .10 .05 .05 .05 .05
0.864
Wan2.1-I2V-14B
10,15,…,45
.55 .55 .45 .30 .15 .10 .05 .05
0.895
.55 .35 .20 .10 .05 .05 .05 .05
0.894
HunyuanVideo-13B
10,15,…,45
.55 .55 .40 .25 .20 .10 .10 .05
0.886
.45 .30 .20 .15 .10 .05 .05 .05
0.876
LTX-2.3-22B
1,2,3 of stage 2
.40 .20 .15
0.794
.25 .10 .10
0.788
Appendix
Table S1: Deployment details per model . Schedules list our allocated budget of the step slots from early to late and the mean positional ratio η solved by our method.
Model
Resolution
Frames
Steps
Warm-up steps
Wan2.1-T2V-14B
1280×720
81
50
10
Wan2.1-I2V-14B
1280×720
81
50
10
HunyuanVideo-13B
1280×720
129
50
10
LTX-2.3-22B
1920×1088
721
8+3
8 (stage 1)
Bernini
1280×720
81
40
8
Appendix
Table S2 : Generation and warm-up settings. LTX-2.3 runs 8 steps in stage 1 and 3 steps in stage 2 following Sol-Attn [ 13 ] .
Key selection
Offline
Method
Static
Dynamic
calib.
Comp.
RadialAttn [ 15 ]
✓
✗
✗
✗
CalibAtt [ 29 ]
✓
✗
✓
✗
SVG2 [ 28 ]
✗
✓
✗
✗
SVG-EAR [ 40 ]
✗
✓
✗
✓
PISA [ 14 ]
✗
✓
✗
✓
Appendix
Table S3 : Categorization of the sparse attention baselines used in our experiments. Static / Dynamic : keys are selected by fixed masks / by online selection at inference time. Offline calib. : the sparse pattern is calibrated offline before inference. Comp. : the contribution of unselected keys is compensated rather than discarded. PoCoAttn combines all four.
Baseline
Δ PSNR (dB)
Wins
Wan2.1-T2V-14B (Text-to-Video, 720p)
RadialAttn [ 15 ]
+5.31±0.42
50/50
SVG2 [ 28 ]
+4.21±0.82
47/50
SVG-EAR [ 40 ]
+3.52±0.99
40/50
PISA [ 14 ] ‡
+3.38±0.39
50/50
Sol-Attn [ 13 ]
+3.68±0.54
49/50
Appendix
Table S4 : Paired per-prompt comparison of PoCoAttn at 25% density against every baseline in Tab. 1 and Tab. 2 that runs at the same or a higher density than PoCoAttn (25.0–36.6%), over the 50 prompts or image–prompt pairs of each generation setting and the 73 editing cases of Bernini. Δ PSNR is the mean paired difference ± the half-width of its 95% confidence interval, and Wins counts the cases on which PoCoAttn has the higher PSNR. ‡ marks the strongest baseline of each setting.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Density ↓
Attn ×↑
Speedup ↑
Original prompts
Dense ( FA3 [ 21 ] )
–
–
–
100%
1.00 ×
1.00 ×
PISA [ 14 ]
30.37
0.868
0.116
30.0%
1.28 ×
1.18 ×
Sol-Attn [ 13 ] (30%)
31.41
0.882
0.102
31.3%
1.40 ×
1.25 ×
Sol-Attn [ 13 ] (15%)
28.86
0.847
0.145
15.5%
2.95 ×
1.86 ×
PoCoAttn (25%)
33.64
0.917
0.073
25.0%
2.27 ×
1.63 ×
Appendix
Table S5 : Robustness to prompt length on LTX-2.3, on the same 50 prompts in their original and extended forms. Speedups on the extended prompts are those of Tab. 1 ; those on the original prompts are timed on these prompts with the same protocol.
Figure S2 : End-to-end latency and speedup across models as acceleration techniques are progressively applied, 15% density.
Figure S3 : Sparse attention kernel roofline on Wan2.1-T2V-14B, 720p, on the NVIDIA H100.
Figure S4 : Effect of calibration set size on video fidelity, the 20-prompt subset evaluated on LTX-2.3-22B. Uniform 25% density.
Method
Wan2.1
HunyuanVideo
LTX-2.3
T2V
I2V
T2V
T2V
Dense
58.2
64.6
51.0
75.3
SVG2 [ 28 ]
58.7
65.1
58.4
–
SVG-EAR [ 40 ]
58.7
65.1
–
–
PISA [ 14 ]
59.3
65.8
51.0
75.3
Sol-Attn [ 13 ]
58.2
64.6
51.8
77.5
Appendix
Table S6 : Peak GPU memory (GB) allocated during generation.
Tables
Overlap
PSNR ↑
SSIM ↑
LPIPS ↓
Δ PSNR
Cross-resolution transfer (480p → 720p)
Native
–
27.81
0.884
0.081
–
From 480p
75%
27.52
0.879
0.085
− 0.28 ± 0.18
Cross-task transfer (T2V → I2V)
Native
–
28.38
0.883
0.067
–
From T2V
80%
28.56
0.887
0.063
+ 0.18 ± 0.16
Appendix
Table S7 : Calibration table transfer on Wan2.1 at 720p, over the 50 evaluation prompts or image–prompt pairs, at 25% density and nearly the same speedup as the native tables (1.39–1.40 × ). Overlap is the share of keys that the transferred tables select in common with the native ones, against 25% for random selection. Δ PSNR is the paired difference from the native tables ± the half-width of its 95% confidence interval.
Pos.
Cont.
Comp.
PSNR ↑
SSIM ↑
LPIPS ↓
Attn ×↑
Speedup ↑
✓
24.17
0.814
0.147
2.38 ×
1.46 ×
✓
25.58
0.822
0.130
2.24 ×
1.42 ×
✓
✓
26.12
0.850
0.111
2.27 ×
1.43 ×
✓
✓
26.07
0.858
0.109
2.18 ×
1.43 ×
✓
✓
27.12
0.864
0.094
1.87 ×
1.36 ×
✓
✓
✓
27.73
0.885
0.082
2.07 ×
1.40 ×
Appendix
Table S8 : Full component ablation on Wan2.1-T2V-14B, 720p on a 20-prompt subset of the evaluation set. 25% density. Pos.: positional keys from the table; Cont.: online content keys; Comp.: compensation.
Figure S5 : Qualitative comparison on text-to-video generation. Frames generated with dense attention (FA3) and with PoCoAttn at 25% density from the same prompt and seed. Following Sol-Attn [ 13 ] , LTX-2.3 rows use its high-quality generation setting.
Figure S6 : Qualitative comparison on image-to-video generation and video editing. Frames generated with dense attention (FA3) and with PoCoAttn at 25% density from the same input and seed; for video editing, the first row is the source video.
Figure S7 : Detailed comparison with baselines on Wan2.1-T2V-14B and HunyuanVideo-13B at 720p. Each block shows dense attention, PoCoAttn at 25% density, and the three baselines with the highest PSNR on that prompt, with PSNR against the dense video. Red boxes mark the magnified regions.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.
Sparse attention accelerates few-step video diffusion transformers by computing a few key blocks exactly and compensating for the rest. However, existing sparse attention still faces key challenges, including identification overhead from finding critical blocks at every denoising step, suboptimal budget allocation that sets sparsity budgets uniformly or by local metrics, and cross-step error accumulation as reused masks or clusters go stale. Observing that the errors left on skipped blocks cancel each other once every key is compensated, we propose DecoupAttn, a new paradigm of decoupled sparse attention: semantic clusters compensate every key, and exact attention corrects their error on a few blocks fixed once offline by position. To support this paradigm, we introduce Distortion Budget, a rate-distortion objective for budget allocation that places exact computation across timesteps, layers, and heads where it removes the most distortion that compensation leaves in the final latent. Built on this budget, we further propose Reassign Cache, in which each step reassigns the drifting keys to cached centroids and refreshes compensation while the budgeted blocks stay fixed. Moreover, DecoupAttn is training-free: a few calibration clips compile one plan offline, and inference runs standard block-sparse kernels with one fused compensation kernel. Our code will be released after submission.
Yanjin Li, Bingbing Qiu, Qiang Huang +5
University of Illinois Urbana-Champaign · Fudan University · Macao Polytechnic University +1
Diffusion Transformers (DiTs) achieve strong video generation quality but suffer from high inference cost due to dense 3D attention, motivating sparse attention techniques for improving efficiency. However, existing training-free sparse attention methods for video generation still face two unresolved limitations: ignoring layer heterogeneity in attention pruning and ignoring query-key coupling in block partitioning, which hinder a better quality-speedup trade-off. In this work, we uncover a critical insight: attention sparsity is an intrinsic layer-wise property, with only minor variation across different inputs. Motivated by this observation, we propose SVOO, a training-free sparse attention framework for fast video generation via offline layer-wise sparsity profiling and online bidirectional co-clustering. Specifically, SVOO adopts a two-stage paradigm: (i) offline layer-wise sensitivity profiling to derive intrinsic per-layer pruning levels, and (ii) online block-wise sparse attention via a bidirectional co-clustering algorithm. Extensive experiments on seven widely used video generation models demonstrate that SVOO achieves a superior quality-speedup trade-off over state-of-the-art methods, delivering up to 1.93x speedup while maintaining a PSNR of up to 29 dB on Wan2.1. Code is available at: https://github.com/Mutual-Luo/SVOO.
Jiayi Luo, Jiayu Chen, Jiankun Wang +6
SKLCCSE, School of Computer Science and Engineering, Beihang University · Zhongguancun Academy · School of Computer Science, Peking University +4