Self-attention over long token sequences dominates the inference cost of video diffusion transformers. Clustering-based sparse attention clusters both queries and keys and lets all queries in a cluster share one selection of key clusters. We show that this sharing limits accuracy: with key clusters and density fixed, attention grows more accurate as the query clusters are refined, and is most accurate when each query selects on its own, which needs no query clustering at all. We therefore propose Per-Query Routed Sparse Attention (PQRS), a training-free method that clusters only keys, lets each query select its own key clusters, and compensates for the unselected ones with their centroids. Per-query selections leave no query tile with a shared key list, so PQRS traverses key clusters instead of queries, buckets queries with identical selections into dense tasks, and merges their partial results in one pass under a fixed softmax offset carried from the previous denoising step. On four text-to-video and image-to-video models from the Wan and HunyuanVideo families, PQRS is the fastest of all the sparse attention methods we evaluate and the closest to the dense output, reaching 28.9 dB PSNR on Wan2.1-T2V and 35.4 dB on HunyuanVideo at 1.66--2.10× end-to-end speedup.
Figures & tables
Figure 1: PQRS accelerates 720p video generation while preserving visual quality. Each block shows frames from dense attention (top) and PQRS (bottom) generated with the same prompt and seed. On a single H100, PQRS reaches 1.75 × and 2.10 × end-to-end speedup on Wan2.1 and HunyuanVideo, with PSNRs of 27.61 and 33.58 dB on the shown samples. Prompts are truncated.
Figure 2: Shared selection versus per-query routing. Darker cells carry more attention mass, outlined cells are selected, and bars show the recall ri . (a) A shared selection fits some queries in a cluster poorly. (b) Per-query routing keeps a similar recall for every query with one fewer selected pair.
Figure 3: Accuracy versus query granularity. All configurations share the key clusters ( Ck=592 ) and the target density ( ρ=0.22 ) and differ only in the number of query clusters Cq , where Cq=N is per-query routing. (a) Mean recall under centroid and Oracle scoring. (b) Mean relative error of the compensated output. (c) Recall percentiles over the queries of a head under centroid scoring, averaged over heads.
Figure 4: Execution pipeline of PQRS for one attention head. (a) Key clusters with similar centroids are grouped and stored contiguously. (b) Each query records its selection in each group as a 4-bit code and initializes its accumulators with the compensation. (c) The queries are bucketed by (group, code). (d) Each bucket runs as dense tasks that atomically add their partial results under a fixed offset mi . (e) The rows are normalized, and rare rows outside the safe range are recomputed. Bottom: each row’s LSE is its offset at the next step.
Figure 5: The LSE changes little between consecutive steps. Percentiles of ∣δi∣ over the sparse rows at each denoising step on Wan2.1-T2V-14B at 720p, pooled over two prompts. Rows above the dashed repair threshold, li=10±30 , are recomputed.
Model
Method
Quality
Efficiency
PSNR ↑
SSIM ↑
LPIPS ↓
VBench ↑
Density
Latency ↓
Speedup ↑
Wan2.1-T2V-14B (720P, 81 Frames)
Dense
–
–
–
83.64%
100%
1682s
1.00 ×
SVG
22.17
0.769
0.210
83.77%
40.7%
1053s
1.60 ×
SVG2
25.73
0.858
0.138
83.54%
31.4%
1091s
1.54 ×
SVOO
26.02
0.866
0.130
83.58%
35.8%
1137s
1.48 ×
PISA
24.05
0.824
0.164
83.78%
35.5%
1107s
1.52 ×
Table 1: Quality and efficiency on four models. Among sparse methods, the best result is in bold and the second best is underlined .
Figure 6: Latency versus group width G . Each bar is one sparse attention call over all 40 heads of Wan2.1-T2V-14B, averaged over 3 prompts, 3 denoising steps and 39 layers. Other stages covers cluster, route, build and finalize.
Selection
Comp.
Density
PSNR ↑
SSIM ↑
LPIPS ↓
Shared
✗
22.5%
24.91
0.842
0.151
Shared
✓
21.9%
26.61
0.874
0.123
Per-query
✗
22.4%
25.81
0.861
0.134
Per-query
✓
22.0%
28.86
0.909
0.094
Table 2: Selection granularity × centroid compensation. Results on Wan2.1-T2V-14B, where shared selection uses Cq=592 query clusters and the last row is PQRS.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Rows
Calls
Wan2.1-T2V-14B
3.2×10−5
8.7%
Wan2.1-I2V-14B
3.9×10−5
9.1%
Wan2.2-T2V-A14B
3.3×10−5
9.3%
HunyuanVideo-T2V-13B
11 rows in total
3 calls in total
Appendix
Table 3: Row repair on the 50 evaluation videos. Rows is the fraction of (head, row) pairs in sparse calls that are repaired, and Calls is the fraction of sparse calls with at least one repaired row.
Method
Subj.
Back.
Motion
Aesth.
Imag.
Avg.
Wan2.1-T2V-14B
Dense
93.86
95.73
98.62
62.22
67.76
83.64
SVG
93.80
95.77
98.66
62.63
68.01
83.77
SVG2
93.71
95.69
98.60
62.11
67.56
83.54
SVOO
93.86
95.55
98.62
62.12
67.73
83.58
PISA
93.88
95.91
98.64
62.47
67.99
83.78
Appendix
Table 4: VBench scores per dimension (%). Subj., Back., Motion, Aesth. and Imag. are subject consistency, background consistency, motion smoothness, aesthetic quality and imaging quality, and Avg. is their mean as reported in Table 1 .
Model
Frames (fps)
N
Steps
Sampler
Guidance
Dense steps
Dense blocks
Wan2.1-T2V-14B
81 (16)
75,600
50
UniPC, shift 5.0
CFG 5.0
1–10
first of 40
Wan2.1-I2V-14B
81 (16)
75,600
40
UniPC, shift 5.0
CFG 5.0
1–8
first of 40
Wan2.2-T2V-A14B
81 (16)
75,600
40
UniPC, shift 12.0
CFG 4.0 / 3.0
1–8
first per expert
HunyuanVideo-T2V-13B
129 (24)
118,800
50
Euler, shift 7.0
embedded 6.0
1–10
first of each stream type
Appendix
Table 5: Sampling configurations. Dense steps and dense blocks use dense attention for every method. N counts video tokens. Euler denotes the flow-matching Euler sampler.
Table 6: Baseline hyperparameters. Each baseline uses the same density hyperparameter on all four models, except where noted, and its official values for all other hyperparameters.
Method
Setting
Density
Latency
Speedup
PSNR
SSIM
LPIPS
SVG2
0.80
22.3%
994s
1.69 ×
24.66
0.836
0.157
SVG2
0.90
31.4%
1091s
1.54 ×
25.73
0.858
0.138
SVG2
0.95
50.8%
1268s
1.33 ×
26.67
0.875
0.123
SVG-EAR
0.75
19.3%
1017s
1.65 ×
27.00
0.880
0.120
SVG-EAR
0.80
21.7%
1032s
1.63 ×
27.19
0.884
0.116
SVG-EAR
0.85
25.2%
1081s
1.56 ×
27.31
0.886
0.114
Appendix
Table 7: Results of Figure 7 . The setting is p for PQRS and top- p for the baselines, and the middle row of each method is its setting in Table 1 . Speedups are over dense attention (1682 s).
Figure 7: Fidelity versus budget on Wan2.1-T2V-14B. Each method runs at three settings, and circles mark the settings of Table 1 . (a) PSNR versus density. (b) PSNR versus end-to-end latency.
Method
Wall time
Speedup
Density
Attend
Outside attend
Largest stages outside attend
Dense (SDPA)
329.1
1.00 ×
100%
329.1
–
–
SVG
123.3
2.67 × [2.66, 2.68]
40.7%
112.4
10.9 (8.8%)
head profiling 5.7
SVG2
138.6
2.51 × [2.45, 2.57]
32.9%
106.0
32.6 (23.6%)
k-means 17.8, plan 7.2, permutation 3.3
SVOO
152.6
2.25 × [2.20, 2.30]
37.2%
116.3
36.3 (23.8%)
plan 27.6
SVOO, reclustering call
425.3
0.78 ×
37.2%
116.9
308.5
co-clustering 272.2
SVOO, amortized
159.4
2.15 ×
–
–
43.2 (27.1%)
–
Appendix
Table 8: Latency of one sparse attention call on Wan2.1-T2V-14B. Times are in milliseconds per call over all 40 heads, averaged over 351 calls. Speedups are means of per-call ratios with bootstrap 95% intervals. Outside attend is the wall time minus the attend kernel, with its share of the wall time in parentheses. The reclustering row times an SVOO call that recomputes its clusters, and the amortized row spreads one such call over 40 calls.
Figure 8: Wan2.1-T2V-14B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
Figure 9: Wan2.1-I2V-14B. Dense attention (top) and PQRS (bottom) with the same input image, prompt and seed.
Figure 10: Wan2.2-T2V-A14B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
Figure 11: HunyuanVideo-T2V-13B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · 3Independent Researcher +2