Self-attention over long token sequences dominates the inference cost of video diffusion transformers. Clustering-based sparse attention clusters both queries and keys and lets all queries in a cluster share one selection of key clusters. We show that this sharing limits accuracy: with key clusters and density fixed, attention grows more accurate as the query clusters are refined, and is most accurate when each query selects on its own, which needs no query clustering at all. We therefore propose Per-Query Routed Sparse Attention (PQRS), a training-free method that clusters only keys, lets each query select its own key clusters, and compensates for the unselected ones with their centroids. Per-query selections leave no query tile with a shared key list, so PQRS traverses key clusters instead of queries, buckets queries with identical selections into dense tasks, and merges their partial results in one pass under a fixed softmax offset carried from the previous denoising step. On four text-to-video and image-to-video models from the Wan and HunyuanVideo families, PQRS is the fastest of all the sparse attention methods we evaluate and the closest to the dense output, reaching 28.9 dB PSNR on Wan2.1-T2V and 35.4 dB on HunyuanVideo at 1.66--2.10× end-to-end speedup.
Figures & tables
Figure 1: PQRS accelerates 720p video generation while preserving visual quality. Each block shows frames from dense attention (top) and PQRS (bottom) generated with the same prompt and seed. On a single H100, PQRS reaches 1.75 × and 2.10 × end-to-end speedup on Wan2.1 and HunyuanVideo, with PSNRs of 27.61 and 33.58 dB on the shown samples. Prompts are truncated.
Figure 2: Shared selection versus per-query routing. Darker cells carry more attention mass, outlined cells are selected, and bars show the recall ri . (a) A shared selection fits some queries in a cluster poorly. (b) Per-query routing keeps a similar recall for every query with one fewer selected pair.
Figure 3: Accuracy versus query granularity. All configurations share the key clusters ( Ck=592 ) and the target density ( ρ=0.22 ) and differ only in the number of query clusters Cq , where Cq=N is per-query routing. (a) Mean recall under centroid and Oracle scoring. (b) Mean relative error of the compensated output. (c) Recall percentiles over the queries of a head under centroid scoring, averaged over heads.
Figure 4: Execution pipeline of PQRS for one attention head. (a) Key clusters with similar centroids are grouped and stored contiguously. (b) Each query records its selection in each group as a 4-bit code and initializes its accumulators with the compensation. (c) The queries are bucketed by (group, code). (d) Each bucket runs as dense tasks that atomically add their partial results under a fixed offset mi . (e) The rows are normalized, and rare rows outside the safe range are recomputed. Bottom: each row’s LSE is its offset at the next step.
Figure 5: The LSE changes little between consecutive steps. Percentiles of ∣δi∣ over the sparse rows at each denoising step on Wan2.1-T2V-14B at 720p, pooled over two prompts. Rows above the dashed repair threshold, li=10±30 , are recomputed.
Model
Method
Quality
Efficiency
PSNR ↑
SSIM ↑
LPIPS ↓
VBench ↑
Density
Latency ↓
Speedup ↑
Wan2.1-T2V-14B (720P, 81 Frames)
Dense
–
–
–
83.64%
100%
1682s
1.00 ×
SVG
22.17
0.769
0.210
83.77%
40.7%
1053s
1.60 ×
SVG2
25.73
0.858
0.138
83.54%
31.4%
1091s
1.54 ×
SVOO
26.02
0.866
0.130
83.58%
35.8%
1137s
1.48 ×
PISA
24.05
0.824
0.164
83.78%
35.5%
1107s
1.52 ×
Table 1: Quality and efficiency on four models. Among sparse methods, the best result is in bold and the second best is underlined .
Figure 6: Latency versus group width G . Each bar is one sparse attention call over all 40 heads of Wan2.1-T2V-14B, averaged over 3 prompts, 3 denoising steps and 39 layers. Other stages covers cluster, route, build and finalize.
Selection
Comp.
Density
PSNR ↑
SSIM ↑
LPIPS ↓
Shared
✗
22.5%
24.91
0.842
0.151
Shared
✓
21.9%
26.61
0.874
0.123
Per-query
✗
22.4%
25.81
0.861
0.134
Per-query
✓
22.0%
28.86
0.909
0.094
Table 2: Selection granularity × centroid compensation. Results on Wan2.1-T2V-14B, where shared selection uses Cq=592 query clusters and the last row is PQRS.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Rows
Calls
Wan2.1-T2V-14B
3.2×10−5
8.7%
Wan2.1-I2V-14B
3.9×10−5
9.1%
Wan2.2-T2V-A14B
3.3×10−5
9.3%
HunyuanVideo-T2V-13B
11 rows in total
3 calls in total
Appendix
Table 3: Row repair on the 50 evaluation videos. Rows is the fraction of (head, row) pairs in sparse calls that are repaired, and Calls is the fraction of sparse calls with at least one repaired row.
Method
Subj.
Back.
Motion
Aesth.
Imag.
Avg.
Wan2.1-T2V-14B
Dense
93.86
95.73
98.62
62.22
67.76
83.64
SVG
93.80
95.77
98.66
62.63
68.01
83.77
SVG2
93.71
95.69
98.60
62.11
67.56
83.54
SVOO
93.86
95.55
98.62
62.12
67.73
83.58
PISA
93.88
95.91
98.64
62.47
67.99
83.78
Appendix
Table 4: VBench scores per dimension (%). Subj., Back., Motion, Aesth. and Imag. are subject consistency, background consistency, motion smoothness, aesthetic quality and imaging quality, and Avg. is their mean as reported in Table 1 .
Model
Frames (fps)
N
Steps
Sampler
Guidance
Dense steps
Dense blocks
Wan2.1-T2V-14B
81 (16)
75,600
50
UniPC, shift 5.0
CFG 5.0
1–10
first of 40
Wan2.1-I2V-14B
81 (16)
75,600
40
UniPC, shift 5.0
CFG 5.0
1–8
first of 40
Wan2.2-T2V-A14B
81 (16)
75,600
40
UniPC, shift 12.0
CFG 4.0 / 3.0
1–8
first per expert
HunyuanVideo-T2V-13B
129 (24)
118,800
50
Euler, shift 7.0
embedded 6.0
1–10
first of each stream type
Appendix
Table 5: Sampling configurations. Dense steps and dense blocks use dense attention for every method. N counts video tokens. Euler denotes the flow-matching Euler sampler.
Table 6: Baseline hyperparameters. Each baseline uses the same density hyperparameter on all four models, except where noted, and its official values for all other hyperparameters.
Method
Setting
Density
Latency
Speedup
PSNR
SSIM
LPIPS
SVG2
0.80
22.3%
994s
1.69 ×
24.66
0.836
0.157
SVG2
0.90
31.4%
1091s
1.54 ×
25.73
0.858
0.138
SVG2
0.95
50.8%
1268s
1.33 ×
26.67
0.875
0.123
SVG-EAR
0.75
19.3%
1017s
1.65 ×
27.00
0.880
0.120
SVG-EAR
0.80
21.7%
1032s
1.63 ×
27.19
0.884
0.116
SVG-EAR
0.85
25.2%
1081s
1.56 ×
27.31
0.886
0.114
Appendix
Table 7: Results of Figure 7 . The setting is p for PQRS and top- p for the baselines, and the middle row of each method is its setting in Table 1 . Speedups are over dense attention (1682 s).
Figure 7: Fidelity versus budget on Wan2.1-T2V-14B. Each method runs at three settings, and circles mark the settings of Table 1 . (a) PSNR versus density. (b) PSNR versus end-to-end latency.
Method
Wall time
Speedup
Density
Attend
Outside attend
Largest stages outside attend
Dense (SDPA)
329.1
1.00 ×
100%
329.1
–
–
SVG
123.3
2.67 × [2.66, 2.68]
40.7%
112.4
10.9 (8.8%)
head profiling 5.7
SVG2
138.6
2.51 × [2.45, 2.57]
32.9%
106.0
32.6 (23.6%)
k-means 17.8, plan 7.2, permutation 3.3
SVOO
152.6
2.25 × [2.20, 2.30]
37.2%
116.3
36.3 (23.8%)
plan 27.6
SVOO, reclustering call
425.3
0.78 ×
37.2%
116.9
308.5
co-clustering 272.2
SVOO, amortized
159.4
2.15 ×
–
–
43.2 (27.1%)
–
Appendix
Table 8: Latency of one sparse attention call on Wan2.1-T2V-14B. Times are in milliseconds per call over all 40 heads, averaged over 351 calls. Speedups are means of per-call ratios with bootstrap 95% intervals. Outside attend is the wall time minus the attend kernel, with its share of the wall time in parentheses. The reclustering row times an SVOO call that recomputes its clusters, and the amortized row spreads one such call over 40 calls.
Figure 8: Wan2.1-T2V-14B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
Figure 9: Wan2.1-I2V-14B. Dense attention (top) and PQRS (bottom) with the same input image, prompt and seed.
Figure 10: Wan2.2-T2V-A14B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
Figure 11: HunyuanVideo-T2V-13B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
Video diffusion transformers generate high-quality videos, but attention over long spatiotemporal sequences makes inference computationally expensive. Sparse attention reduces this cost by computing only a subset of query-key interactions exactly, and recent training-free methods further approximate the remaining interactions. These methods, however, degrade quickly at high sparsity, which limits the achievable speedup. We present ESTRA (Extreme Sparsity Through Refined Attention), a training-free acceleration method that preserves generation quality even at extreme sparsity. ESTRA concentrates exact attention computation on a small set of important key-value tokens and approximates the remaining contributions with cluster representatives. To identify these tokens efficiently, it groups similar queries and uses their representatives to score individual keys. A variance-based correction further refines the attention weights of the cluster representatives, improving the accuracy of the approximation. Experiments on Wan2.1-14B and HunyuanVideo-13B show that ESTRA establishes a new quality-efficiency frontier for training-free sparse attention, achieving 3.26× and 4.71× end-to-end speedups at 5% attention density while surpassing Sol-Attn and SVG-EAR in fidelity to dense generation and on most VBench dimensions. Even at 1% density, ESTRA remains more faithful to dense generation than Sol-Attn at 12.5%, while reaching up to 5.39× speedup.
Video diffusion transformers generate high-quality videos, but the quadratic cost of 3D full attention dominates their inference. Dynamic block-sparse attention reduces this cost by computing exact attention only on key/value blocks selected from current activations. However, its fidelity degrades under aggressive sparsity: blocks are scored from coarse summaries or formed by costly global clustering, pruned blocks vanish from the softmax, and whole tiles are kept or dropped even when only a few of their keys matter. To address these limitations, we propose Accurate Block Attention (AB-Attention), a training-free method that accurately approximates dense attention at high sparsity. Its Intra-Block Clustering Selector (IBCS) estimates block importance faithfully yet cheaply by clustering queries only within each query block, without permuting tokens. Sampling with Inclusion-Probability Weighting (SIP) then samples blocks in proportion to these estimates and reweights them by inverse inclusion probability, so that one pass estimates dense attention rather than truncating it. Finally, its kernel gathers key/value blocks much smaller than a tile into full tensor-core tiles, refining selection at no extra tensor-core cost. On Wan2.1 and HunyuanVideo, AB-Attention skips 89--92% of attention, whereas the baselines compute 1.8--3.6× as much, yet it still improves PSNR by up to 4.4~dB over the strongest baseline and achieves up to 2.23× end-to-end speedup. Our code is available at https://github.com/InfRec-Cardinal/AB-Attention.
Bin Xu, Mo Zhang, Zhengfeiyang Zhang +6
InfRec · The Chinese University of Hong Kong, Shenzhen · Shandong University +5
Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic similarity or fail to adapt to heterogeneous token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity-preserving clustering method to query vectors for higher compression, and designs a euclidean-similarity-preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.
Haoyue Tan, Shengnan Wang, Yulin Qiao +5
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · 3Independent Researcher +2