Attention is essential for modeling spatio-temporal dependencies in video diffusion transformers, but its quadratic cost makes it a major inference bottleneck. Its high sparsity offers an opportunity to accelerate generation by focusing computation on important interactions. However, accuracy and execution favor different granularities: each query attends to its own key tokens, whereas GPU kernels work on tiles. We present Token2Tile, a training-free sparse attention operator that combines per-query scoring and a per-query residual with tile execution. An online low-rank space, computed directly from queries and keys to preserve attention logits, makes clustering, scoring, and residual approximation efficient. Token2Tile unites query-specific selections to allocate exact work across query clusters, then packs the selected keys into GPU tiles. On MiniMax-H3 and Wan2.1-14B, Token2Tile delivers 1.48-1.54x end-to-end speedup over dense FlashAttention-3 with VBench scores within 0.5 points of dense. At comparable speeds on H3, it improves PSNR over SOL and SubBlock by 5.5-6.8 dB; on Wan, it exceeds SOL by 4.6-4.8 dB while running faster.
Figures & tables
Figure 1: Higher dense-reference fidelity at comparable generation speed. (a) H3 I2V sweep on a common VBench-I2V subset with 95% bootstrap PSNR intervals; green labels are nominal budgets. Near 1.49× on this subset, Token2Tile improves PSNR by 4.9–5.6 dB. (b) Crops of lettering from the dense video and three sparse methods at 1.47 – 1.50× (whole-video PSNR, request time). (c) Dense-left/Token2Tile-right splits of four T2V examples at 1.5%.
Figure 2: Why score per query, select key clusters, and keep a per-query residual. Attention captured from four generators at a 1% per-query key budget; whiskers span the inputs, red labels give the ratio between the green bar and the bar it is compared with. (a) Dense attention mass captured by per-query selection of key clusters and other selection methods. (b) Per-query selection of key clusters versus contiguous 64-key blocks. (c) Reduction in executed pairs when per-query selections are united over query clusters versus contiguous query tiles of the same size. (d) Mass left in the tail, and the output-error increase when the tail is dropped instead of compensated by the per-query residual.
Figure 3: Token-wise selection with tile-wise execution. (a) An SVD of the second-moment square-root product gives matched projections Pq,Pk from 128 to 32 dimensions, in which queries and keys are clustered per token. Each query keeps the shortest prefix of whole key clusters that meets its token budget (colored); a query cluster executes the union of its members’ selections (outlined), and each remaining cluster contributes one centroid term per query (grey, ⋄ ). (b) The original Q multiplies the union keys, packed into tiles across cluster boundaries; QPq multiplies the centroids of the unselected clusters, and an LSE merge yields one softmax per query. Clusters, selections and plans are refreshed every R=4 calls; reuse calls use the current Q , K , V . No N×N tensor is materialized.
Task
Method
Similarity to dense
VBench dimensions (%) ↑
Efficiency (8 GPUs)
PSNR ↑
SSIM ↑
LPIPS ↓
Subject
Backgr.
Motion
Aesthetic
Time (s) ↓
Speedup ↑
MiniMax-H3
T2V
Dense (FA3)
–
–
–
88.24
93.19
99.25
57.16
109.0
1.000×
SOL
21.00
0.722
0.206
88.25
93.37
99.21
57.32
73.5
1.483 ×
SubBlock 0.75
21.55
0.741
0.190
88.26
93.31
99.26
57.20
74.1
1.471 ×
Token2Tile 1.5%
27.83
0.875
0.085
88.10
93.18
99.20
56.96
73.0
1.493 ×
Table 1: Generation fidelity and speed. Individual VBench dimensions (%) use the same videos as the fidelity metrics. Shading marks Token2Tile; bold marks the best fidelity.
T2V
I2V
Model
PSNR
DiT
E2E
PSNR
DiT
E2E
LightX2V [ 15 ]
35.57
1.17 ×
1.11 ×
35.02
1.14 ×
1.09 ×
HyperFlow [ 32 ]
30.53
1.16 ×
1.06 ×
30.71
1.13 ×
1.03 ×
Table 2: Eight-step distilled H3 models. Four GPUs; 5% budget, three dense calls, R=1 . PSNR is relative to each model's dense video; DiT/E2E denote denoiser/full-request speedups on a VBench subset.
b
ρ
Token2Tile
EAR
SVG2
XAttn
SubBl.
PISA
SOL
H3
1.0
11.7
27.2
23.2
19.8
15.7
16.7
17.1
16.5
3.5
25.7
34.6
27.7
27.1
21.3
21.6
21.1
20.6
Wan
1.0
7.6
25.8
25.4 †
15.1
10.7
11.2
13.6
12.8
3.5
20.0
32.9
27.7
20.8
15.4
15.8
18.2
17.9
Hun.
1.0
6.8
24.4
23.4 †
15.4
9.3
9.9
10.4
10.4
3.5
17.2
30.7
26.5
21.2
12.6
13.4
13.8
13.8
Table 3: Accuracy at equal executed density. Attention SNR (dB); b : per-query budget, ρ : executed density (both %). † SVG-EAR exceeds the target density because its minimum is about 10%. On H3, pinned context is exact for every rule except the official SVG-EAR kernel. Full results: Tab. A2 .
Figure 4: Time breakdown of an H3 attention call (3.5% budget, 23.4% executed density). Bars show dense FA3, refresh, reuse, and their R=4 mean; the table lists stage times and mean shares.
Figure 5: H3 kernel speed–accuracy trade-off. Points average captured inputs. Token2Tile sweeps 10–1% budgets; SOL sweeps τ=0.25 –2. Other kernels match Token2Tile's executed density; hollow markers denote SVG-EAR at higher density. Four-model results: Figs. A2 and A4 .
Table 4: Component ablations at matched executed density. Changes from Token2Tile (positive = better). (a) H3 video PSNR at 2.5%. (b,c) Attention SNR at 3.5%: (b) ranges across generators; (c) the same summed-mass decisions and per-query residuals with two cluster budget policies. Shared variants use the query-cluster mean; PCA is principal component analysis. Full controls: Sec. C.4 .
H3
Wan
Hun.
LTX
Measured
1.33
1.25
1.26
1.27
Predicted
1.32
1.26
1.28
1.25
Table 5: Predicting union growth. After/before ratio of union inflation when query-cluster size doubles (3.5% budget). Model: Sec. C.4 .
Figure 7: Denoising-time budget allocation (H3 T2V). (a) Denoiser output SNR when one window is sparsified (higher: less sensitive). (b) Calibrated window budgets. Wan and gains over uniform schedules: Fig. A6 .
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Speedup ( × )
Comparison
Win
Tie
Loss
Score
Ours
Base
All
46
44
10
68 [62, 74]
–
–
Block-sparse
66
29
5
81 [77, 85]
1.38–1.49
1.41–1.49
SVG2 and SVG-EAR
27
58
15
56 [52, 61]
1.38–1.40
0.96–1.07
H3, vs SOL
71
24
5
83 [74, 92]
1.48–1.49
1.48–1.49
H3, vs SubBlock
58
32
11
74 [62, 87]
1.48–1.49
1.47–1.48
Appendix
Table A1: Blind study. Win/tie/loss percentages and score = win +21 tie; 95% intervals bootstrap raters. Block-sparse combines SOL and SubBlock. Speedups span T2V/I2V at the rated settings: main-table budgets on H3, 5% on Wan.
Figure A1: Equal-density accuracy across generators (3.5% budget). Whiskers span inputs; arrows give relative-error ratios of the nearest and farthest alternatives to Token2Tile. Full values: Tab. A2 .
Controls, Δ dB
Other rules at the same ρ
Budget (%)
ρ (%)
Token2Tile
w/o res.
shared res.
128-D res.
shared sel.
SVG-EAR
SVG2
XAttn
SubBlock
PISA
SOL
H3
1.0
11.7
27.2
− 1.8
− 1.0
+0.3
− 2.0
23.2
19.8
15.7
16.7
17.1
16.5
3.5
25.7
34.6
− 1.6
− 1.0
+0.2
− 2.1
27.7
27.1
21.3
21.6
21.1
20.6
10.0
47.2
46.2
− 2.0
− 1.4
+0.3
− 4.2
30.6
36.3
28.4
27.0
26.0
25.2
Wan2.1-14B
Appendix
Table A2: Accuracy and component controls at matched density. Attention SNR (dB), averaged over inputs. Controls give Δ SNR against FP32 Token2Tile with the same state: no residual, shared residual, original-space centroid scores, or shared selection. Shared variants use the query-cluster mean. The Token2Tile column is the real kernel output; controls are FP32 differences. On H3, pinned context is exact for every rule except the official SVG-EAR kernel, unlike the real kernels of Tab. A4 . † SVG-EAR exceeds the target density.
SOL
Top- k
Top- k + corr.
Model (fine block)
b
ρ
Token2Tile
64
fine
64
fine
64
fine
H3 (128 × 36)
1
11.7
27.2
16.5
16.3
16.2
15.8
16.6
16.3
3.5
25.7
34.6
20.6
21.0
20.3
20.6
20.6
20.9
10
47.2
46.2
25.2
26.1
25.2
25.9
25.4
26.1
Wan2.1-14B (128 × 37)
1
7.6
25.8
12.8
13.0
11.0
11.0
13.5
13.5
3.5
20.0
32.9
17.9
18.1
15.7
15.8
17.9
18.1
Appendix
Table A3: Block rules at Token2Tile's granularity. Attention SNR (dB) at Token2Tile's executed density. Each block rule runs with 64×64 blocks and with as many key blocks as Token2Tile has key clusters: 128-query blocks over 2048 contiguous key blocks (block shape after the model name). SOL: row threshold with zeroth-order correction; top- k : per-query-block top- k key blocks, exact only or with the same correction.
Figure A2: Kernel speed–accuracy curves across generators. Points average inputs; bars span them. Competing kernels match Token2Tile's executed density, except hollow SVG-EAR points at higher density. Values: Tab. A4 .
H3
Wan2.1-14B
HunyuanVideo
LTX-2.3
Setting
Sp. ↑
SNR ↑
Sp. ↑
SNR ↑
Sp. ↑
SNR ↑
Sp. ↑
SNR ↑
Token2Tile, per-query budget
1.0%
4.40
27.2
5.80
25.8
7.92
24.4
5.33
29.6
2.0%
3.26
30.9
4.23
29.4
5.31
27.6
4.29
32.1
3.5%
2.48
34.6
3.11
32.9
3.75
30.7
3.47
34.5
5.0%
2.08
36.8
2.53
35.3
2.98
32.9
3.00
36.4
Appendix
Table A4: Kernel speed and accuracy. Sp.: speedup over FA3 from summed times; SNR: mean real-kernel output accuracy (dB). Token2Tile rows average one refresh and three reuse calls ( R=4 ). Densities match except below SVG-EAR's minimum. Timing scope: Sec. B .
Input
QKV
FA3 peak
Workspace
of which G
Peak/call
State/layer (MiB)
Layers
State, all
Resident
H3
0.37
0.12
2.82
1.96
0.13
11.2
50
0.55
3.37
Wan2.1-14B
0.27
0.09
2.07
1.44
0.09
8.1
40
0.31
2.39
HunyuanVideo
0.31
0.10
2.36
1.65
0.12
8.6
60
0.50
2.87
LTX-2.3
0.19
0.06
1.43
1.00
0.08
5.8
48
0.27
1.70
Appendix
Table A5: Allocated attention memory per rank (GiB unless marked). Workspace is shared across layers; G is its cluster-score buffer. Peak/call is transient allocation beyond the workspace. State stores each layer's labels, centroids, unions, and plans; resident memory sums workspace and all-layer state. Budget: 3.5%.
FA3
Fixed-block
Model
b
ρ
(ms)
Token2Tile
routed
kernel
SOL
H3
1
12.1
28.8
4.52
4.90
5.64
3.67
3.5
26.0
28.7
2.55
2.43
2.57
2.14
10
47.5
29.3
1.54
1.39
1.43
1.40
Wan2.1-14B
1
7.9
21.5
5.76
6.55
8.08
3.48
3.5
21.0
21.9
3.13
3.03
3.28
2.37
Appendix
Table A6: Kernels at equal executed density. Speedup over dense FA3 per attention call (GPU kernel time), all kernels interleaved in one process. Token2Tile: full call averaged over R=4 . Fixed-block: the 64×64 block-sparse FlashAttention kernel with SubBlock routing, and the kernel alone on a precomputed plan. SOL: its kernel with τ set to keep the same density. ρ : density of the timed calls. Bold: fastest complete method.
Figure A3: Kernel cost across generators. (a) Stage times relative to FA3 at 3.5%; R=4 averages one refresh and three reuse calls. (b) Speedup at three budgets. Labels and colors identify the stages and budgets.
Backend
Step 16
Step 24
Step 32
Token2Tile exact
0.468 / 16.51
0.358 / 12.68
0.236 / 8.39
FlashInfer var.-block
0.468 / 27.00
0.358 / 20.51
0.236 / 13.52
FA4 128 x 128 cover
0.913 / 28.61
0.828 / 25.68
0.708 / 21.80
FA4 64 x 64 cover
0.776 / 34.97
0.660 / 29.55
0.515 / 23.36
Dense FA3
1.000 / 29.33
1.000 / 28.71
1.000 / 28.01
Dense FA4
1.000 / 28.78
1.000 / 28.88
1.000 / 28.02
Appendix
Table A7: Exact-attention backend controls. H3 captures at a 3.5% budget. Cells: pair density / time (ms). Variable-block attention uses the same mask; fixed-block covers add pairs. These standalone runs exclude planning and the residual; Fig. 4 instead attributes stages within full calls. Library kernels use default settings, including cuDNN SDPA [ 21 ] .
Figure A4: Accuracy of low-rank spaces at 3.5%. Compare the original space, key-only PCA, shared PCA, and our 32-dimensional low-rank space. Each query executes its own prefix without a union, with a per-query residual. Statistics and times: Tabs. A8 and A9 .
Figure A5: Video component ablations. PSNR change from full Token2Tile at matched executed density, interpolated over 15–22%. Changes within 0.1 dB are treated as ties. The 128-D variant runs on the FlashInfer backend and is compared with a rank-32 control on that backend. Captured-attention controls: Tab. A2 .
Space
Logit err. ↓
Spearman ↑
Within-cl. var. ↓
Mass 1% ↑
SNR 1% ↑
Mass 3.5% ↑
H3
Original 128-D
0.000
1.000
26.7
68.0
18.1
82.7
Key-only PCA, 32-D
0.227
0.944
26.4
64.9
15.9
80.9
Shared PCA, 32-D
0.186
0.963
26.1
66.7
17.1
81.8
Our low-rank space, 32-D
0.181
0.965
24.3
68.0
17.4
82.9
Wan2.1-14B
Appendix
Table A8: Comparing low-rank spaces. Means over captured inputs. Logit error and Spearman measure agreement with QK⊤ ; within-cluster variance is relative to total logit variance (%). Mass and SNR use each query's own cluster prefix without a union, with a per-query residual. SNR at 3.5% is in Fig. A4 .
Model
H×N
Assignment, r=32/64/128
Residual, r=32/64/128
128 vs 32
H3
7 × 73,393
0.57
0.64
1.02
0.84
1.02
1.63
1.88 ×
Wan2.1-14B
5 × 75,600
0.44
0.49
0.77
0.63
0.77
0.99
1.65 ×
HunyuanVideo
3 × 144,014
0.61
0.70
1.05
0.72
0.88
1.39
1.84 ×
LTX-2.3
4 × 65,280
0.30
0.33
0.53
0.44
0.54
0.71
1.68 ×
Appendix
Table A9: Stage time versus low-rank dimension (ms). Assignment produces key/query labels; the per-query residual includes centroid scoring and value aggregation. Both use fused kernels with on-chip scores.
Dimension
H3
Wan
Hun.
LTX
1%
3.5%
1%
3.5%
1%
3.5%
1%
3.5%
32 (both)
27.7
36.2
26.4
33.5
24.7
31.0
30.0
35.0
Selection dimension r (residual 32)
8
− 6.05
− 5.38
− 10.81
− 9.70
− 6.70
− 5.91
− 4.40
− 4.27
16
− 0.57
− 0.88
− 3.74
− 2.62
− 1.55
− 1.03
+0.07
+0.35
24
− 0.44
+0.55
− 0.61
− 0.22
− 0.27
+0.02
+0.20
+0.29
Appendix
Table A10: Dimension of the low-rank space. First row: attention SNR (dB) with clusters rebuilt in FP32 for every dimension, so it differs from Tab. A2 ; other rows: Δ SNR. Selection changes recluster and rescore at dimension r , matching the 32-dimensional density; residual changes keep selection fixed. A 16-dimensional residual takes 0.80 ms on H3; Tab. A9 gives 32–128.
H3
Wan2.1-14B
HunyuanVideo
LTX-2.3
Budget (%)
ours
orig.
tiles
rand.
ours
orig.
tiles
rand.
ours
orig.
tiles
rand.
ours
orig.
tiles
rand.
1.0
0.24
0.23
0.14
0.04
0.36
0.35
0.14
0.08
0.35
0.32
0.12
0.05
0.36
0.34
0.13
0.03
3.5
0.30
0.29
0.18
0.05
0.40
0.39
0.17
0.06
0.42
0.38
0.16
0.05
0.45
0.44
0.20
0.06
Appendix
Table A11: Selection overlap within query clusters. Token-weighted Jaccard, excluding context keys. Compare our low-rank space (ours), the original space (orig.), contiguous tiles, and random sets, with equal mean query counts. Values average captured inputs.
H3
Wan2.1-14B
HunyuanVideo
LTX-2.3
Budget (%)
ρ
infl.
tiles
Δ SNR
ρ
infl.
tiles
Δ SNR
ρ
infl.
tiles
Δ SNR
ρ
infl.
tiles
Δ SNR
0.5
7.5
13.9 ×
10.8
+8.0
4.3
8.2 ×
12.6
+6.3
4.0
7.6 ×
12.5
+5.7
4.0
7.7 ×
11.4
+5.2
1.0
11.7
11.2 ×
17.2
+8.7
7.6
7.4 ×
19.9
+6.4
6.8
6.6 ×
20.0
+5.9
6.4
6.2 ×
16.9
+5.2
2.0
18.1
8.9 ×
26.7
+9.9
13.1
6.5 ×
30.9
+6.9
11.4
5.6 ×
31.1
+6.5
10.2
5.0 ×
24.5
+5.4
3.5
25.7
7.3 ×
37.2
+11.0
20.0
5.7 ×
42.7
+7.6
17.2
4.9 ×
43.1
+7.1
14.8
4.2 ×
32.5
+5.6
5.0
31.9
6.3 ×
45.3
+11.3
25.9
5.2 ×
51.5
+8.2
22.3
4.4 ×
52.1
+7.7
18.9
3.8 ×
38.6
+5.9
Appendix
Table A12: Union size and accuracy. Means over captured inputs. ρ : executed density (%); infl.: ratio to each query's own prefix; tiles: density under contiguous query tiles of the same size. Δ SNR: gain from executing the union rather than the own prefix, both with a per-query residual.
H3
Wan2.1-14B
HunyuanVideo
LTX-2.3
CQ
rows
ρ
SNR
Sp.
rows
ρ
SNR
Sp.
rows
ρ
SNR
Sp.
rows
ρ
SNR
Sp.
× 0.25
513
44.5
39.4
1.64
591
33.6
39.9
2.18
511
27.1
37.1
2.60
510
27.7
33.8
2.50
× 0.5
257
35.3
36.8
1.76
295
25.8
37.5
2.52
256
20.9
35.2
3.04
256
21.5
32.4
2.99
deployed
128
27.8
34.3
1.90
148
19.7
35.4
2.79
128
16.3
33.9
3.28
128
17.1
31.2
3.16
× 2
64
21.4
32.2
1.88
74
15.0
33.8
3.04
64
12.7
32.6
3.01
64
13.5
30.1
2.93
× 4
32
16.7
30.2
1.21
37
11.6
32.2
2.01
32
10.1
31.6
1.84
32
10.8
29.2
1.85
Appendix
Table A13: Query-cluster size. Vary CQ at a 3.5% budget on one capture per generator. Rows: queries per cluster; ρ : pair density (%); SNR: attention accuracy (dB); Sp.: exact-branch speedup over FA3 using FlashInfer variable-block attention.
Inflation, meas. / pred.
Rank corr.
1%
10%
Inflation
Overlap
H3
9.8 / 11.7
4.5 / 4.6
0.96–0.98
0.97–0.98
Wan
7.2 / 8.4
4.1 / 4.2
0.97–0.99
0.97–0.99
Hun.
6.4 / 7.6
3.5 / 3.6
0.95–0.97
0.92–0.98
LTX
6.0 / 7.9
2.9 / 3.1
0.96–0.99
0.92–0.98
Appendix
Table A14: Predicting union size from cluster statistics. Measured/predicted inflation, median over query clusters at 1% and 10%, and Spearman correlation across query clusters between predicted and measured inflation and pairwise overlap (range over five budgets). Hun.: HunyuanVideo.
Selection rule
H3
Wan
Hun.
LTX
1%
3.5%
1%
3.5%
1%
3.5%
1%
3.5%
Per-query prefixes, cluster union (deployed)
25.4
33.1
22.2
28.6
23.7
29.9
28.0
32.7
Shared selection, union budget
− 1.38
− 1.47
− 1.18
− 0.88
− 0.57
− 0.43
− 0.79
− 0.72
Per-tile decision (summed mass), union budget
+0.01
+0.15
− 0.52
− 0.36
− 0.08
+0.03
− 0.28
− 0.19
Per-tile decision (squared mass), union budget
+0.03
+0.26
− 0.33
− 0.21
− 0.04
+0.00
− 0.24
− 0.13
Union selection, uniform cluster budgets
− 0.35
− 0.93
− 0.15
− 0.62
− 0.33
− 0.36
− 0.14
− 0.26
Appendix
Table A15: Scores, decisions, and query-cluster budgets. First row: attention SNR (dB); other rows: Δ SNR from the union (positive = better). All variants use per-query residuals and match total pairs within 1%. Rows 2–4 retain individual union budgets; later rows redistribute the total. Errors are RMS over sampled query clusters and every variant keeps the deployed union's token count per cluster, so values differ from Tab. A2 .
Union
Per tile
Δ (s.e.)
Better
Δ SSIM
Δ LPIPS
T2V
28.53
28.69
+0.16 (0.14)
23/40
+0.20
− 0.20
I2V
31.58
31.50
− 0.08 (0.14)
22/40
− 0.01
+0.04
Appendix
Table A16: Per-tile decisions versus the union. Both use the same exact-attention backend. Each query cluster retains its union budget but selects by summed member mass. H3 videos, 2.5% budget; Δ PSNR with standard error, Δ SSIM/ Δ LPIPS ×100 .
H3 T2V
H3 I2V
Kernel speedup
PSNR
Δ
PSNR
Δ
H3
Wan
Hun.
LTX
R=1
29.04
+0.42
31.92
+0.27
2.00
2.42
3.04
2.42
R=2
28.96
+0.34
31.81
+0.16
2.29
2.84
3.48
3.03
R=4
28.62
+0.00
31.65
+0.00
2.48
3.11
3.75
3.47
R=8
28.07
− 0.55
30.62
− 1.03
2.58
3.26
3.90
3.74
R=4 , fresh μc
28.62
− 0.01
31.68
+0.03
2.48
–
–
–
Appendix
Table A17: Structure reuse. Left: H3 video PSNR at 2.5%, Δ against R=4 . Right: kernel speedup at 3.5%, computed from the refresh and reuse times of the R=4 profile. Fresh μc recomputes centroids; reselect also rescores, reselects, and replans. More b uses 3.66/3.50% on T2V/I2V, matching R=2 at 2.5% in the H3 ablation control.
Figure A6: Denoising-time budget allocation. (a) Wan T2V denoiser output SNR when one window is sparsified (higher: less sensitive). (b) Video PSNR gain over uniform allocation at the same nominal budget.
Model/task
Window allocation (%)
Frequency
Δ PSNR [95% CI]
Δ SSIM
Δ LPIPS
H3 T2V
5/2/0.75/1.5 (reference)
5/12
–
–
–
H3 T2V
2.5/2.5/2.5/2.5
1/12
-0.76 [-1.10, -0.44]
-0.0089
+0.0066
H3 T2V
5/2/1/0.75
4/12
+0.17 [-0.01, +0.41]
-0.0008
+0.0006
H3 T2V
5/2.5/0.75/0.5
1/12
+0.15 [-0.01, +0.31]
-0.0015
+0.0012
H3 T2V
5/1.25/1.25/1.5
1/12
-0.03 [-0.08, +0.03]
-0.0003
+0.0001
H3 I2V
2.5/1.5/2/5 (reference)
1/12
–
–
–
Appendix
Table A18: Calibration-input sensitivity. Distinct window budgets from 12 calibrations per task at 2.5%, their frequencies, and video-fidelity changes relative to the reference schedule. Window budgets are normalized to the target at deployment.
Figure A7: Dense and Token2Tile frames at matched timestamps. Cases show lettering (a, b), faces, and hands (c, d). Rows use the main-table settings; video PSNR is relative to dense.
Attention over tens of thousands of visual tokens dominates the inference of video diffusion transformers (DiTs). Existing methods reduce this cost through sparsity, low precision, or both, but use a single precision for all kept tiles. We observe that the output error introduced by approximation varies widely across tiles. Many tiles can be skipped with little error, others must be computed but gain little beyond low precision, and a few retain substantial error even at low precision. We present TriTile, a training-free attention method that assigns each tile one of three approximation tiers: skipping it by replacing it with its block means, skimming it in NVFP4, or refining it with NVFP4 residual correction. A lightweight router predicts each tile's error under each tier from block statistics and spends extra compute on a tile only where it removes enough error; a single knob, a price on compute, controls the whole speed-quality trade-off. A fused kernel executes all three tiers in a single pass on the FP4 tensor cores. Over dense FlashAttention, TriTile speeds up generation by 1.38-1.83x on Wan2.2 at 23.0-27.6 dB PSNR, 2.28-2.32x on HunyuanVideo at 34.5-36.0 dB, and 1.84x on LTX-2.5 at 34.1 dB, and in every setting it Pareto-dominates every baseline in speed and PSNR.
Yeonjae Kim, Omin Kwon, Seong Hoon Seo +1
Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea.
Video diffusion transformers generate high-quality videos, but attention over long spatiotemporal sequences makes inference computationally expensive. Sparse attention reduces this cost by computing only a subset of query-key interactions exactly, and recent training-free methods further approximate the remaining interactions. These methods, however, degrade quickly at high sparsity, which limits the achievable speedup. We present ESTRA (Extreme Sparsity Through Refined Attention), a training-free acceleration method that preserves generation quality even at extreme sparsity. ESTRA concentrates exact attention computation on a small set of important key-value tokens and approximates the remaining contributions with cluster representatives. To identify these tokens efficiently, it groups similar queries and uses their representatives to score individual keys. A variance-based correction further refines the attention weights of the cluster representatives, improving the accuracy of the approximation. Experiments on Wan2.1-14B and HunyuanVideo-13B show that ESTRA establishes a new quality-efficiency frontier for training-free sparse attention, achieving 3.26× and 4.71× end-to-end speedups at 5% attention density while surpassing Sol-Attn and SVG-EAR in fidelity to dense generation and on most VBench dimensions. Even at 1% density, ESTRA remains more faithful to dense generation than Sol-Attn at 12.5%, while reaching up to 5.39× speedup.
Video diffusion transformers generate high-quality videos, but the quadratic cost of 3D full attention dominates their inference. Dynamic block-sparse attention reduces this cost by computing exact attention only on key/value blocks selected from current activations. However, its fidelity degrades under aggressive sparsity: blocks are scored from coarse summaries or formed by costly global clustering, pruned blocks vanish from the softmax, and whole tiles are kept or dropped even when only a few of their keys matter. To address these limitations, we propose Accurate Block Attention (AB-Attention), a training-free method that accurately approximates dense attention at high sparsity. Its Intra-Block Clustering Selector (IBCS) estimates block importance faithfully yet cheaply by clustering queries only within each query block, without permuting tokens. Sampling with Inclusion-Probability Weighting (SIP) then samples blocks in proportion to these estimates and reweights them by inverse inclusion probability, so that one pass estimates dense attention rather than truncating it. Finally, its kernel gathers key/value blocks much smaller than a tile into full tensor-core tiles, refining selection at no extra tensor-core cost. On Wan2.1 and HunyuanVideo, AB-Attention skips 89--92% of attention, whereas the baselines compute 1.8--3.6× as much, yet it still improves PSNR by up to 4.4~dB over the strongest baseline and achieves up to 2.23× end-to-end speedup. Our code is available at https://github.com/InfRec-Cardinal/AB-Attention.
Bin Xu, Mo Zhang, Zhengfeiyang Zhang +6
InfRec · The Chinese University of Hong Kong, Shenzhen · Shandong University +5