Token2Tile: Token-Wise Sparse Attention with Tile Execution through a Low-Rank Space for Video DiTs
Organizations: Beijing Institute of Technology · NVIDIA
Abstract
Attention is essential for modeling spatio-temporal dependencies in video diffusion transformers, but its quadratic cost makes it a major inference bottleneck. Its high sparsity offers an opportunity to accelerate generation by focusing computation on important interactions. However, accuracy and execution favor different granularities: each query attends to its own key tokens, whereas GPU kernels work on tiles. We present Token2Tile, a training-free sparse attention operator that combines per-query scoring and a per-query residual with tile execution. An online low-rank space, computed directly from queries and keys to preserve attention logits, makes clustering, scoring, and residual approximation efficient. Token2Tile unites query-specific selections to allocate exact work across query clusters, then packs the selected keys into GPU tiles. On MiniMax-H3 and Wan2.1-14B, Token2Tile delivers 1.48-1.54x end-to-end speedup over dense FlashAttention-3 with VBench scores within 0.5 points of dense. At comparable speeds on H3, it improves PSNR over SOL and SubBlock by 5.5-6.8 dB; on Wan, it exceeds SOL by 4.6-4.8 dB while running faster.
Figures & tables
| Task | Method | Similarity to dense | VBench dimensions (%) | Efficiency (8 GPUs) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | Subject | Backgr. | Motion | Aesthetic | Time (s) | Speedup | ||
| MiniMax-H3 | ||||||||||
| T2V | Dense (FA3) | – | – | – | 88.24 | 93.19 | 99.25 | 57.16 | 109.0 | |
| SOL | 21.00 | 0.722 | 0.206 | 88.25 | 93.37 | 99.21 | 57.32 | 73.5 | 1.483 | |
| SubBlock 0.75 | 21.55 | 0.741 | 0.190 | 88.26 | 93.31 | 99.26 | 57.20 | 74.1 | 1.471 | |
| Token2Tile 1.5% | 27.83 | 0.875 | 0.085 | 88.10 | 93.18 | 99.20 | 56.96 | 73.0 | 1.493 | |
| T2V | I2V | |||||
|---|---|---|---|---|---|---|
| Model | PSNR | DiT | E2E | PSNR | DiT | E2E |
| LightX2V [ 15 ] | 35.57 | 1.17 | 1.11 | 35.02 | 1.14 | 1.09 |
| HyperFlow [ 32 ] | 30.53 | 1.16 | 1.06 | 30.71 | 1.13 | 1.03 |
| Token2Tile | EAR | SVG2 | XAttn | SubBl. | PISA | SOL | |||
|---|---|---|---|---|---|---|---|---|---|
| H3 | 1.0 | 11.7 | 27.2 | 23.2 | 19.8 | 15.7 | 16.7 | 17.1 | 16.5 |
| 3.5 | 25.7 | 34.6 | 27.7 | 27.1 | 21.3 | 21.6 | 21.1 | 20.6 | |
| Wan | 1.0 | 7.6 | 25.8 | 25.4 † | 15.1 | 10.7 | 11.2 | 13.6 | 12.8 |
| 3.5 | 20.0 | 32.9 | 27.7 | 20.8 | 15.4 | 15.8 | 18.2 | 17.9 | |
| Hun. | 1.0 | 6.8 | 24.4 | 23.4 † | 15.4 | 9.3 | 9.9 | 10.4 | 10.4 |
| 3.5 | 17.2 | 30.7 | 26.5 | 21.2 | 12.6 | 13.4 | 13.8 | 13.8 |
| (a) Video fidelity: PSNR (dB) | ||
|---|---|---|
| Variant | T2V | I2V |
| No per-query residual | 1.80 | 3.25 |
| Shared selection | +0.04 | 0.99 |
| Shared selection, no residual | 1.43 | 2.76 |
| Per-tile decision, union budget | +0.16 | 0.08 |
| Original 128-D space | 0.04 | 0.66 |
| H3 | Wan | Hun. | LTX | |
|---|---|---|---|---|
| Measured | 1.33 | 1.25 | 1.26 | 1.27 |
| Predicted | 1.32 | 1.26 | 1.28 | 1.25 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Speedup ( ) | ||||||
| Comparison | Win | Tie | Loss | Score | Ours | Base |
| All | 46 | 44 | 10 | 68 [62, 74] | – | – |
| Block-sparse | 66 | 29 | 5 | 81 [77, 85] | 1.38–1.49 | 1.41–1.49 |
| SVG2 and SVG-EAR | 27 | 58 | 15 | 56 [52, 61] | 1.38–1.40 | 0.96–1.07 |
| H3, vs SOL | 71 | 24 | 5 | 83 [74, 92] | 1.48–1.49 | 1.48–1.49 |
| H3, vs SubBlock | 58 | 32 | 11 | 74 [62, 87] | 1.48–1.49 | 1.47–1.48 |
| Controls, dB | Other rules at the same | |||||||||||
| Budget (%) | (%) | Token2Tile | w/o res. | shared res. | 128-D res. | shared sel. | SVG-EAR | SVG2 | XAttn | SubBlock | PISA | SOL |
| H3 | ||||||||||||
| 1.0 | 11.7 | 27.2 | 1.8 | 1.0 | +0.3 | 2.0 | 23.2 | 19.8 | 15.7 | 16.7 | 17.1 | 16.5 |
| 3.5 | 25.7 | 34.6 | 1.6 | 1.0 | +0.2 | 2.1 | 27.7 | 27.1 | 21.3 | 21.6 | 21.1 | 20.6 |
| 10.0 | 47.2 | 46.2 | 2.0 | 1.4 | +0.3 | 4.2 | 30.6 | 36.3 | 28.4 | 27.0 | 26.0 | 25.2 |
| Wan2.1-14B | ||||||||||||
| SOL | Top- | Top- + corr. | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model (fine block) | Token2Tile | 64 | fine | 64 | fine | 64 | fine | ||
| H3 (128 36) | 1 | 11.7 | 27.2 | 16.5 | 16.3 | 16.2 | 15.8 | 16.6 | 16.3 |
| 3.5 | 25.7 | 34.6 | 20.6 | 21.0 | 20.3 | 20.6 | 20.6 | 20.9 | |
| 10 | 47.2 | 46.2 | 25.2 | 26.1 | 25.2 | 25.9 | 25.4 | 26.1 | |
| Wan2.1-14B (128 37) | 1 | 7.6 | 25.8 | 12.8 | 13.0 | 11.0 | 11.0 | 13.5 | 13.5 |
| 3.5 | 20.0 | 32.9 | 17.9 | 18.1 | 15.7 | 15.8 | 17.9 | 18.1 | |
| H3 | Wan2.1-14B | HunyuanVideo | LTX-2.3 | |||||
| Setting | Sp. | SNR | Sp. | SNR | Sp. | SNR | Sp. | SNR |
| Token2Tile, per-query budget | ||||||||
| 1.0% | 4.40 | 27.2 | 5.80 | 25.8 | 7.92 | 24.4 | 5.33 | 29.6 |
| 2.0% | 3.26 | 30.9 | 4.23 | 29.4 | 5.31 | 27.6 | 4.29 | 32.1 |
| 3.5% | 2.48 | 34.6 | 3.11 | 32.9 | 3.75 | 30.7 | 3.47 | 34.5 |
| 5.0% | 2.08 | 36.8 | 2.53 | 35.3 | 2.98 | 32.9 | 3.00 | 36.4 |
| Input | FA3 peak | Workspace | of which | Peak/call | State/layer (MiB) | Layers | State, all | Resident | |
|---|---|---|---|---|---|---|---|---|---|
| H3 | 0.37 | 0.12 | 2.82 | 1.96 | 0.13 | 11.2 | 50 | 0.55 | 3.37 |
| Wan2.1-14B | 0.27 | 0.09 | 2.07 | 1.44 | 0.09 | 8.1 | 40 | 0.31 | 2.39 |
| HunyuanVideo | 0.31 | 0.10 | 2.36 | 1.65 | 0.12 | 8.6 | 60 | 0.50 | 2.87 |
| LTX-2.3 | 0.19 | 0.06 | 1.43 | 1.00 | 0.08 | 5.8 | 48 | 0.27 | 1.70 |
| FA3 | Fixed-block | ||||||
|---|---|---|---|---|---|---|---|
| Model | (ms) | Token2Tile | routed | kernel | SOL | ||
| H3 | 1 | 12.1 | 28.8 | 4.52 | 4.90 | 5.64 | 3.67 |
| 3.5 | 26.0 | 28.7 | 2.55 | 2.43 | 2.57 | 2.14 | |
| 10 | 47.5 | 29.3 | 1.54 | 1.39 | 1.43 | 1.40 | |
| Wan2.1-14B | 1 | 7.9 | 21.5 | 5.76 | 6.55 | 8.08 | 3.48 |
| 3.5 | 21.0 | 21.9 | 3.13 | 3.03 | 3.28 | 2.37 | |
| Backend | Step 16 | Step 24 | Step 32 |
|---|---|---|---|
| Token2Tile exact | 0.468 / 16.51 | 0.358 / 12.68 | 0.236 / 8.39 |
| FlashInfer var.-block | 0.468 / 27.00 | 0.358 / 20.51 | 0.236 / 13.52 |
| FA4 128 x 128 cover | 0.913 / 28.61 | 0.828 / 25.68 | 0.708 / 21.80 |
| FA4 64 x 64 cover | 0.776 / 34.97 | 0.660 / 29.55 | 0.515 / 23.36 |
| Dense FA3 | 1.000 / 29.33 | 1.000 / 28.71 | 1.000 / 28.01 |
| Dense FA4 | 1.000 / 28.78 | 1.000 / 28.88 | 1.000 / 28.02 |
| Space | Logit err. | Spearman | Within-cl. var. | Mass 1% | SNR 1% | Mass 3.5% |
|---|---|---|---|---|---|---|
| H3 | ||||||
| Original 128-D | 0.000 | 1.000 | 26.7 | 68.0 | 18.1 | 82.7 |
| Key-only PCA, 32-D | 0.227 | 0.944 | 26.4 | 64.9 | 15.9 | 80.9 |
| Shared PCA, 32-D | 0.186 | 0.963 | 26.1 | 66.7 | 17.1 | 81.8 |
| Our low-rank space, 32-D | 0.181 | 0.965 | 24.3 | 68.0 | 17.4 | 82.9 |
| Wan2.1-14B | ||||||
| Model | Assignment, | Residual, | 128 vs 32 | |||||
|---|---|---|---|---|---|---|---|---|
| H3 | 7 73,393 | 0.57 | 0.64 | 1.02 | 0.84 | 1.02 | 1.63 | 1.88 |
| Wan2.1-14B | 5 75,600 | 0.44 | 0.49 | 0.77 | 0.63 | 0.77 | 0.99 | 1.65 |
| HunyuanVideo | 3 144,014 | 0.61 | 0.70 | 1.05 | 0.72 | 0.88 | 1.39 | 1.84 |
| LTX-2.3 | 4 65,280 | 0.30 | 0.33 | 0.53 | 0.44 | 0.54 | 0.71 | 1.68 |
| Dimension | H3 | Wan | Hun. | LTX | ||||
|---|---|---|---|---|---|---|---|---|
| 1% | 3.5% | 1% | 3.5% | 1% | 3.5% | 1% | 3.5% | |
| 32 (both) | 27.7 | 36.2 | 26.4 | 33.5 | 24.7 | 31.0 | 30.0 | 35.0 |
| Selection dimension (residual 32) | ||||||||
| 8 | 6.05 | 5.38 | 10.81 | 9.70 | 6.70 | 5.91 | 4.40 | 4.27 |
| 16 | 0.57 | 0.88 | 3.74 | 2.62 | 1.55 | 1.03 | +0.07 | +0.35 |
| 24 | 0.44 | +0.55 | 0.61 | 0.22 | 0.27 | +0.02 | +0.20 | +0.29 |
| H3 | Wan2.1-14B | HunyuanVideo | LTX-2.3 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Budget (%) | ours | orig. | tiles | rand. | ours | orig. | tiles | rand. | ours | orig. | tiles | rand. | ours | orig. | tiles | rand. |
| 1.0 | 0.24 | 0.23 | 0.14 | 0.04 | 0.36 | 0.35 | 0.14 | 0.08 | 0.35 | 0.32 | 0.12 | 0.05 | 0.36 | 0.34 | 0.13 | 0.03 |
| 3.5 | 0.30 | 0.29 | 0.18 | 0.05 | 0.40 | 0.39 | 0.17 | 0.06 | 0.42 | 0.38 | 0.16 | 0.05 | 0.45 | 0.44 | 0.20 | 0.06 |
| H3 | Wan2.1-14B | HunyuanVideo | LTX-2.3 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Budget (%) | infl. | tiles | SNR | infl. | tiles | SNR | infl. | tiles | SNR | infl. | tiles | SNR | ||||
| 0.5 | 7.5 | 13.9 | 10.8 | +8.0 | 4.3 | 8.2 | 12.6 | +6.3 | 4.0 | 7.6 | 12.5 | +5.7 | 4.0 | 7.7 | 11.4 | +5.2 |
| 1.0 | 11.7 | 11.2 | 17.2 | +8.7 | 7.6 | 7.4 | 19.9 | +6.4 | 6.8 | 6.6 | 20.0 | +5.9 | 6.4 | 6.2 | 16.9 | +5.2 |
| 2.0 | 18.1 | 8.9 | 26.7 | +9.9 | 13.1 | 6.5 | 30.9 | +6.9 | 11.4 | 5.6 | 31.1 | +6.5 | 10.2 | 5.0 | 24.5 | +5.4 |
| 3.5 | 25.7 | 7.3 | 37.2 | +11.0 | 20.0 | 5.7 | 42.7 | +7.6 | 17.2 | 4.9 | 43.1 | +7.1 | 14.8 | 4.2 | 32.5 | +5.6 |
| 5.0 | 31.9 | 6.3 | 45.3 | +11.3 | 25.9 | 5.2 | 51.5 | +8.2 | 22.3 | 4.4 | 52.1 | +7.7 | 18.9 | 3.8 | 38.6 | +5.9 |
| H3 | Wan2.1-14B | HunyuanVideo | LTX-2.3 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rows | SNR | Sp. | rows | SNR | Sp. | rows | SNR | Sp. | rows | SNR | Sp. | |||||
| 0.25 | 513 | 44.5 | 39.4 | 1.64 | 591 | 33.6 | 39.9 | 2.18 | 511 | 27.1 | 37.1 | 2.60 | 510 | 27.7 | 33.8 | 2.50 |
| 0.5 | 257 | 35.3 | 36.8 | 1.76 | 295 | 25.8 | 37.5 | 2.52 | 256 | 20.9 | 35.2 | 3.04 | 256 | 21.5 | 32.4 | 2.99 |
| deployed | 128 | 27.8 | 34.3 | 1.90 | 148 | 19.7 | 35.4 | 2.79 | 128 | 16.3 | 33.9 | 3.28 | 128 | 17.1 | 31.2 | 3.16 |
| 2 | 64 | 21.4 | 32.2 | 1.88 | 74 | 15.0 | 33.8 | 3.04 | 64 | 12.7 | 32.6 | 3.01 | 64 | 13.5 | 30.1 | 2.93 |
| 4 | 32 | 16.7 | 30.2 | 1.21 | 37 | 11.6 | 32.2 | 2.01 | 32 | 10.1 | 31.6 | 1.84 | 32 | 10.8 | 29.2 | 1.85 |
| Inflation, meas. / pred. | Rank corr. | |||
|---|---|---|---|---|
| 1% | 10% | Inflation | Overlap | |
| H3 | 9.8 / 11.7 | 4.5 / 4.6 | 0.96–0.98 | 0.97–0.98 |
| Wan | 7.2 / 8.4 | 4.1 / 4.2 | 0.97–0.99 | 0.97–0.99 |
| Hun. | 6.4 / 7.6 | 3.5 / 3.6 | 0.95–0.97 | 0.92–0.98 |
| LTX | 6.0 / 7.9 | 2.9 / 3.1 | 0.96–0.99 | 0.92–0.98 |
| Selection rule | H3 | Wan | Hun. | LTX | ||||
| 1% | 3.5% | 1% | 3.5% | 1% | 3.5% | 1% | 3.5% | |
| Per-query prefixes, cluster union (deployed) | 25.4 | 33.1 | 22.2 | 28.6 | 23.7 | 29.9 | 28.0 | 32.7 |
| Shared selection, union budget | 1.38 | 1.47 | 1.18 | 0.88 | 0.57 | 0.43 | 0.79 | 0.72 |
| Per-tile decision (summed mass), union budget | +0.01 | +0.15 | 0.52 | 0.36 | 0.08 | +0.03 | 0.28 | 0.19 |
| Per-tile decision (squared mass), union budget | +0.03 | +0.26 | 0.33 | 0.21 | 0.04 | +0.00 | 0.24 | 0.13 |
| Union selection, uniform cluster budgets | 0.35 | 0.93 | 0.15 | 0.62 | 0.33 | 0.36 | 0.14 | 0.26 |
| Union | Per tile | (s.e.) | Better | SSIM | LPIPS | |
|---|---|---|---|---|---|---|
| T2V | 28.53 | 28.69 | +0.16 (0.14) | 23/40 | +0.20 | 0.20 |
| I2V | 31.58 | 31.50 | 0.08 (0.14) | 22/40 | 0.01 | +0.04 |
| H3 T2V | H3 I2V | Kernel speedup | ||||||
|---|---|---|---|---|---|---|---|---|
| PSNR | PSNR | H3 | Wan | Hun. | LTX | |||
| 29.04 | +0.42 | 31.92 | +0.27 | 2.00 | 2.42 | 3.04 | 2.42 | |
| 28.96 | +0.34 | 31.81 | +0.16 | 2.29 | 2.84 | 3.48 | 3.03 | |
| 28.62 | +0.00 | 31.65 | +0.00 | 2.48 | 3.11 | 3.75 | 3.47 | |
| 28.07 | 0.55 | 30.62 | 1.03 | 2.58 | 3.26 | 3.90 | 3.74 | |
| , fresh | 28.62 | 0.01 | 31.68 | +0.03 | 2.48 | – | – | – |
| Model/task | Window allocation (%) | Frequency | PSNR [95% CI] | SSIM | LPIPS |
|---|---|---|---|---|---|
| H3 T2V | 5/2/0.75/1.5 (reference) | 5/12 | – | – | – |
| H3 T2V | 2.5/2.5/2.5/2.5 | 1/12 | -0.76 [-1.10, -0.44] | -0.0089 | +0.0066 |
| H3 T2V | 5/2/1/0.75 | 4/12 | +0.17 [-0.01, +0.41] | -0.0008 | +0.0006 |
| H3 T2V | 5/2.5/0.75/0.5 | 1/12 | +0.15 [-0.01, +0.31] | -0.0015 | +0.0012 |
| H3 T2V | 5/1.25/1.25/1.5 | 1/12 | -0.03 [-0.08, +0.03] | -0.0003 | +0.0001 |
| H3 I2V | 2.5/1.5/2/5 (reference) | 1/12 | – | – | – |
Sparse Attention
Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.
See all 274 papers