Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers

Tianze Xia; Xiaoling Zhou; Ruya Lan; Yiqiao Liao; Chengru Song; Shiming Xiang

cs.CVOct 21, 2026

Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers

Authors: Tianze Xia, Xiaoling Zhou, Ruya Lan, Yiqiao Liao, Chengru Song, Shiming Xiang

Organizations: UCAS · CASIA · KlingAI Research · PKU

Abstract

Full 3D attention is costly in video Diffusion Transformers. Block-sparse attention reduces this cost by computing only selected attention blocks. However, existing methods either overlook the potential to optimize sparsity allocation or rely on heuristic rules or local metrics to guide it. In this work, we observe that sparsifying attention in different layers and at different denoising steps affects final-video fidelity to substantially different degrees. Yet the identities of the most sensitive layer-step pairs remain nearly unchanged across inputs. Building on these findings, we propose Allocating Layer-step Latency by Output Tests (ALLOT), a training-free sparsity allocation method. Treating each layer-step pair as a cell, ALLOT estimates each cell's contribution to final-video error through (i) cell probing and (ii) trajectory calibration, then jointly optimizes cell densities under a measured-latency budget. It preserves existing selection strategies and kernels, replacing only the allocation component. At equal or shorter wall-clock time, ALLOT outperforms every compared allocation strategy with XAttention, with an 11.8 dB PSNR gain on HunyuanVideo. Extensive experiments across four additional published pipelines on Wan2.1, Wan2.2, and HunyuanVideo show that ALLOT outperforms or ties the default in all nine settings.

Figures & tables

Figure 1: Changing only the allocation at equal wall-clock time. Left: dense, baseline and + ALLOT (top to bottom), with blue-box insets; prompts are near the median gain. Right: paired PSNR gains (ALLOT $-$ baseline) with 95% CIs for Tabs. 1 and 2 ; open markers denote ties.

Figure 2: Layer and step sensitivity. (a) Single-layer one-step error (

b{=}0.30

t{=}0.85

; bars), attention-map metrics [ 14 , 36 ] (lines), and final-video MSE with that layer sparse throughout (diamonds). Titles give the most sensitive layer and its metric ranks. (b) Wan2.1-1.3B window errors versus local metrics [ 33 ] ; annotations give first/last-window ratios. (c) Window errors across four models.

Figure 3: ALLOT overview. \scriptsize1⃝ Offline: measure cell probes $\hat{e}_{\ell}$ , trajectory gains $w(s)$ and kernel latency $r(b)$ ; solve a density table for each latency budget $C$ (§ 4.4 ). \scriptsize2⃝ Inference: look up each cell’s density for any prompt, using dense attention when $b_{\ell s}{=}1$ and the existing sparse kernel otherwise.

Allocation	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench $\uparrow$	Wall-clock (s)	Speedup
Dense	—	—	—	72.35	183.8	1.00 $\times$
Default	18.14	0.659	0.268	70.27	148.7	1.24 $\times$
CalibAtt-style	19.67	0.700	0.222	70.79	149.6	1.23 $\times$
ALLOT	25.37	0.835	0.106	71.24	148.6	1.24 $\times$
HEART (EBC)	19.43	0.699	0.221	72.02	151.7	1.21 $\times$
Efficient-vDiT	19.08	0.702	0.225	72.32	151.1	1.22 $\times$

Table 1: Comparison with other allocation methods (XAttention on Wan2.1-1.3B 480p

\times

81,

n{=}32

). Default: XAttention’s published configuration. Rows grouped by wall-clock time, with ALLOT re-solved for each of the first two groups. The third group applies HEART’s cross-step mask reuse (TMR) to both sides. Bold: best PSNR/SSIM/LPIPS in group. Details in Appendix G .

Figure 4: Qualitative comparison. Dense, baseline and + ALLOT (left to right) at the first and last frames; bottom rows enlarge the boxed regions, with inner boxes marking larger baseline deviations from dense. Headers give PSNR to dense, gain and wall-clock time. Prompts are near the median gain. Settings: Tab. 1 (XAttention), Appendix H (SpargeAttn 720p), and Tab. 2 (SpargeAttn 480p).

Pipeline & setup	Allocation	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench $\uparrow$	Wall-clock (s)	Speedup
Wan2.1-1.3B 480p $\times$ 81	Default	22.67	0.768	0.150	70.03	147.7	1.24 $\times$
SpargeAttn	+ ALLOT	27.70	0.876	0.070	71.58	145.8	1.26 $\times$
Wan2.1-1.3B 720p $\times$ 81	Default	26.44	0.832	0.143	71.71	448.9	1.62 $\times$
Radial	+ ALLOT	26.83	0.837	0.138	71.08	441.5	1.65 $\times$
Wan2.1-1.3B 720p $\times$ 81	Default	24.25	0.808	0.151	71.09	410.9	1.66 $\times$
SVG2	+ ALLOT	25.60	0.819	0.156	70.75	410.5	1.66 $\times$

Table 2: Integration with published pipelines, changing only the allocation (

n{=}32

; speedup = dense wall-clock / row wall-clock; dense times and paired intervals in Appendix H ). Bold: best PSNR/SSIM/LPIPS in each setting.

Figure 5: Fidelity–speedup curves (Wan2.1-1.3B 720p; prompts as in Tab. 2 ; y-ranges differ). Open circles: pipeline settings (SpargeAttn: default, two released and one retuned checkpoint; Radial: default and decay 1.0; SVOO: default). Arrows show gains at the default’s speed.

Model & setup	Speedup	$n$	PSNR $\uparrow$ (uniform $\to$ ALLOT)	$\Delta$ PSNR vs. uniform	$\Delta$ PSNR vs. strengthened
Wan2.1-1.3B, 480p $\times$ 33	1.14 $\times$	32	19.59 $\to$ 25.99	$+$ 6.40	$+$ 2.55
Wan2.1-14B, 720p $\times$ 81	1.29 $\times$	16	24.87 $\to$ 31.25	$+$ 6.38	$+$ 0.76
Wan2.2-A14B, 480p $\times$ 33	1.09 $\times$	32	19.36 $\to$ 25.86	$+$ 6.50	$+$ 1.67

Table 3: Cross-model comparison (mean-pooled top-

k

selection with the VSA kernel; groups differ only in density tables;

\Delta

PSNR is the mean paired difference). Details in Appendix J .

Figure 6: Solved density tables for XAttention ( Tab. 1 ) and HunyuanVideo (equal density; Appendix J ); white contours mark dense cells. Right: mean density per step, including Wan2.1-14B ( Tab. 3 ), versus a rule keeping the first 30% of steps dense and using 0.30 thereafter.

Module	Replaced with	PSNR	$\Delta$ PSNR
—	ALLOT	26.39	—
Cell prob.	Same density for all layers	25.32	$-$ 1.07
Traj. cal.	$w\equiv 1$	19.34	$-$ 7.05
Per-cell solve	Parametric shape rule	24.63	$-$ 1.76
Latency table	Density-proportional cost	26.27	$-$ 0.12

Table 4: Ablation (XAttention, Wan2.1-1.3B 480p

\times

81,

n{=}32

; one module replaced per row at the same cost as ALLOT; full metrics in Appendix I ).

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$t$	Most sensitive layer	Ratio	Rank: coverage density	Rank: $H^{2}$	Spearman (coverage, $\hat{e}$ )	Spearman ( $H^{2}$ , $\hat{e}$ )
Wan2.1-1.3B	0.85	0	53 $\times$	3/30	1/30	$+0.33$	$+0.43$
Wan2.1-1.3B	0.45	1	27 $\times$	26/30	26/30	$+0.29$	$+0.45$
Wan2.1-1.3B	0.15	0	15 $\times$	1/30	1/30	$+0.33$	$+0.39$
Wan2.1-14B	0.85	0	154 $\times$	22/40	13/40	$+0.03$	$+0.08$
Wan2.1-14B	0.45	0	75 $\times$	11/40	6/40	$-0.16$	$-0.02$
Wan2.1-14B	0.15	0	71 $\times$	7/40	4/40	$-0.15$	$+0.11$

Appendix

Table B.1 : Cell probes and attention-map metrics. Ratio: cell probe of the most sensitive layer

\div

that of the median layer; ranks are in decreasing order.

Figure B.1 : SVOO’s released per-layer coverage density (bars) and each layer’s cell probe (line,

b{=}0.30

). (a) Wan2.1-1.3B; (b) Wan2.1-14B; (c) step-wise trend: both decrease along denoising.

Window	Steps	Wan2.1-1.3B	Wan2.1-14B	Wan2.2-A14B	HunyuanVideo
1	0–4	10.71 (463)	10.70 (186)	12.95 (109)	20.89 (69)
2	5–9	18.46 (78)	16.99 (44)	18.48 (31)	23.55 (38)
3	10–14	23.19 (26)	21.89 (14)	24.28 (8)	28.43 (12)
4	15–19	26.46 (12)	26.77 (5)	25.36 (6)	33.12 (4)
5	20–24	29.58 (6)	27.73 (4)	26.70 (5)	36.74 (2)
6	25–29	32.61 (3)	30.13 (2)	29.74 (2)	38.34 (1)

Appendix

Table C.1 : Window experiments. Each entry: PSNR (in parentheses, final-video MSE relative to window 10, from 8-prompt mean).

Figure D.1 : Sensitivity is stable across inputs. (a–d) Final-video MSE when only one window is sparsified (8 prompts each; boxes: median leave-one-out Spearman correlation). (e–h) Per-video cell probes and layer-ranking Spearman on 1.3B and OpenVid videos.

Test	Reading	Conclusion
Separability: single layer sparse over all steps (§ E.3 )	ranking of the 6 layers identical on 1.3B and HunyuanVideo, Spearman 0.94 on 14B; ratio between the most and the least sensitive layer predicted 9.3 $\times$ , measured 4.8 $\times$ (1.3B), predicted 146 $\times$ , measured 7.7 $\times$ (14B), predicted 286 $\times$ , measured 30 $\times$ (HunyuanVideo)	ranking holds, magnitude compressed
Layer groups within one window: final MSE per unit of measured injected energy	at density 0.05, largest/smallest group ratio 2.17 [1.89, 2.50] in window 4 and 1.25 [0.91, 1.71] in window 6; energy-matched layer 0/comparison-group ratios in windows 1–2 are 0.56–0.82 ( $n{=}8$ )	point estimates within $2\times$ in window 6 and the energy-matched early windows; 2.17 $\times$ in window 4
Across steps: two windows sparse together $\div$ sum of each alone	over 7 window pairs, 1.3B 0.86–1.26 and 14B 0.83–1.36; adjacent windows within steps 15–34 super-additive by 1.16–1.36 $\times$ (4 of 6 pairs at $p\leq 0.04$ , $n{=}8$ )	approximately additive
Across layers: all layers sparse together $\div$ sum of each layer alone	1.3B 6.5 at high noise to 1.1 at low noise, 14B 1.9 to 0.8, HunyuanVideo 0.74–0.89 to 1.2–1.5; for two layers sparse together within one step, median ratio $1\pm 0.08$	not additive in magnitude; absorbed by $w$
Additive prediction of isolated probes vs. marginals measured in context	Spearman: 1.3B 0.63 in the deployed table’s context and 0.36 in a uniform 0.15 context; HunyuanVideo 0.77 in the deployed table’s context	less accurate in sparser contexts
Additive prediction vs. final video for arbitrary tables (80 tables, § E.5 )	Kendall $\tau$ : pooled 0.83 ( $R^{2}$ 0.90), within one budget 0.78 (per budget 0.70 / 0.77 / 0.86); by family of tables: convex combinations 0.91, jitters 0.82, random tables 0.72, log–log slopes 0.98–1.14 / 0.48–0.75 / 0.26–0.44	rankings within one budget usable; weaker for tables further from the solved ones

Appendix

Table E.1 : Tests of the two approximations.

Figure E.1 : Additive-model prediction vs. final-video MSE for 80 tables at three budgets (color: budget; marker: family of the table; 4 held-out prompts each). Kendall

\tau

per budget and pooled in the last row of Table E.1 .

Assignment	PSNR	$\Delta$ PSNR	Additive-model prediction
ALLOT	25.99	–	0.37
Shuffle layers within each step	22.60	$-3.40$ [ $-3.94$ , $-2.86$ ] 0/32	1.07
Shuffle steps within each layer	13.77	$-12.23$ [ $-13.60$ , $-10.85$ ] 0/32	10.6
Fully random (seed 1)	10.21	$-15.78$ [ $-17.48$ , $-14.09$ ] 0/32	45.6
Fully random (seed 0)	9.82	$-16.17$ [ $-17.85$ , $-14.49$ ] 0/32	56.1
Reversed estimate	7.95	$-18.05$ [ $-19.65$ , $-16.44$ ] 0/32	146

Appendix

Table E.2 : Reassigning the same densities (

\Delta

= row

-

ALLOT).

Model and mask	Window	Sparse	Random signs	Gaussian
1.3B, top- $k$	1	10.71	29.57	25.68
1.3B, top- $k$	5	29.58	38.55	38.64
1.3B, sliding window	1	9.16	28.23	26.66
1.3B, sliding window	5	26.21	38.23	38.48
14B, top- $k$	1	10.70	25.93	26.54
14B, top- $k$	5	27.73	37.17	37.45

Appendix

Table E.3 : Oracle diagnostic: mean final-video PSNR (dB) to each model’s dense reference. All rows use 8 prompts, 480p

\times

33, 50 steps and seed 0; windows 1 and 5 cover steps 0–4 and 20–24. Only the indicated window is perturbed.

Figure F.1 : Latency tables. (a) Kernels of each pipeline on Wan2.1-1.3B 720p

\times

81;

\blacktriangledown

marks the density at which cost equals dense. (b) The VSA kernel is nearly identical across shapes and head counts.

Model	Cell probes	Trajectory gain	Latency table	Total
Wan2.1-1.3B	2.8	3.1	minutes	$\approx$ 6
Wan2.1-14B	17.4	shared with 1.3B	minutes	$\approx$ 17
HunyuanVideo	18.9	3.8	minutes	$\approx$ 23

Appendix

Table F.1 : Offline cost per model (GPU-h, including model loading).

Setting	Baseline	$n$	$\Delta$ PSNR	$\Delta$ SSIM	$\Delta$ LPIPS	$\Delta$ DISTS	Verdict
XAttention 1.3B 480p $\times$ 81 (ALLOT at the default’s wall-clock time)	default	32	$+7.23$ $[+6.60,\,+7.85]$ 32/0	$+0.176$ $[+0.148,\,+0.204]$ 32/0	$-0.163$ $[-0.187,\,-0.138]$ 32/0	$-0.059$ $[-0.073,\,-0.044]$ 31/1	win
XAttention 1.3B 480p $\times$ 81	HEART (EBC)	32	$+6.96$ $[+6.35,\,+7.56]$ 32/0	$+0.155$ $[+0.127,\,+0.183]$ 32/0	$-0.132$ $[-0.154,\,-0.111]$ 32/0	$-0.045$ $[-0.055,\,-0.035]$ 31/1	win
XAttention 1.3B 480p $\times$ 81 (TMR on both sides)	HEART (EBC + TMR)	32	$+6.59$ $[+5.99,\,+7.19]$ 32/0	$+0.142$ $[+0.115,\,+0.168]$ 32/0	$-0.120$ $[-0.141,\,-0.099]$ 32/0	$-0.040$ $[-0.050,\,-0.030]$ 31/1	win
XAttention 1.3B 720p $\times$ 81 (ALLOT at the default’s wall-clock time)	default	32	$+6.35$ $[+5.61,\,+7.09]$ 32/0	$+0.110$ $[+0.092,\,+0.127]$ 32/0	$-0.141$ $[-0.165,\,-0.117]$ 32/0	$-0.062$ $[-0.078,\,-0.046]$ 29/3	win
XAttention 1.3B 720p $\times$ 81 (ALLOT at the default’s wall-clock time)	HEART (EBC)	32	$+5.16$ $[+4.40,\,+5.92]$ 32/0	$+0.079$ $[+0.065,\,+0.093]$ 32/0	$-0.095$ $[-0.114,\,-0.076]$ 31/1	$-0.032$ $[-0.045,\,-0.019]$ 27/5	win
XAttention 1.3B 720p $\times$ 81 (TMR on both sides)	HEART (EBC + TMR)	32	$+5.21$ $[+4.44,\,+5.98]$ 32/0	$+0.083$ $[+0.070,\,+0.097]$ 32/0	$-0.100$ $[-0.120,\,-0.079]$ 31/1	$-0.035$ $[-0.048,\,-0.022]$ 28/4	win

Appendix

Table G.1 : Paired differences (ALLOT

-

baseline) for the XAttention rows default, EBC and HEART (EBC + TMR) of Table 1 and XAttention on Wan2.1-1.3B 720p

\times

81 (§ G.4 ); verdicts follow § A.1 .

Setting	Construction	Dense cells	Time (s)	PSNR	SSIM	LPIPS	DISTS
Dense	–	1500	183.8	–	–	–	–
Default	5 dense steps; threshold 0.90	150	148.7	18.14	0.659	0.268	0.154
Default, density cutoff	Default per-cell densities; density cutoff	150	151.5	20.41	0.745	0.181	0.111
HEART (EBC)	Per-head thresholds; integer programming	150	151.7	19.43	0.699	0.221	0.130
HEART (EBC + TMR)	EBC + cross-step mask reuse	150	145.0	19.54	0.706	0.214	0.128
Efficient-vDiT	Per-layer density; layers 0–2 dense	285	151.1	19.08	0.702	0.225	0.133

Appendix

Table G.2 : XAttention allocation settings (

n{=}32

; shared latency table and budget). Top: construction and absolute metrics. Bottom: paired differences,

\Delta=\text{ALLOT}-\text{row}

. Default and CalibAtt-style use ALLOT at the default’s time; HEART (EBC + TMR) uses ALLOT + TMR.

Allocation	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench $\uparrow$	Wall-clock (s)	Speedup
Dense	—	—	—	70.99	595.4	1.00 $\times$
Default (threshold 0.90)	15.65	0.568	0.476	65.62	481.3	1.24 $\times$
HEART (EBC)	16.61	0.608	0.428	67.20	483.0	1.23 $\times$
ScalingAttention (FAST)	20.33	0.705	0.328	67.72	494.4	1.20 $\times$
ALLOT	27.49	0.875	0.122	69.96	481.4	1.24 $\times$

Appendix

Table G.3 : XAttention on HunyuanVideo 480p

\times

129 (720

\times

480;

n{=}32

; speedup = dense wall-clock / row wall-clock). Paired differences in § G.3 .

Allocation	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench $\uparrow$	Wall-clock (s)
Dense	—	—	—	71.12	715.9
Default (threshold 0.90)	21.16	0.731	0.287	71.08	499.3
HEART (EBC)	22.36	0.762	0.241	71.05	498.5
ALLOT	27.51	0.841	0.146	72.07	493.2
HEART (EBC + TMR)	22.17	0.751	0.252	70.73	441.5
ALLOT + TMR	27.38	0.835	0.152	72.02	442.3

Appendix

Table G.4 : XAttention on Wan2.1-1.3B 720p

\times

81 (

n{=}32

Pipeline	Wan2.1-1.3B	Wan2.1-14B	Wan2.2-A14B	HunyuanVideo
SpargeAttn	183.8	–	–	–
Radial	728.1	3576.1	–	3192.3
SVOO	700.1	3406.0	3355.9	3225.5
SVG2	682.9	–	–	–

Appendix

Table H.1 : Dense wall-clock times (s) used for the speedups of Table 2 . Wan models at 720p

\times

81 except SpargeAttn (480p

\times

81); HunyuanVideo at 720p

\times

129.

Setting	Baseline	$n$	$\Delta$ PSNR	$\Delta$ SSIM	$\Delta$ LPIPS	$\Delta$ DISTS	Verdict
SVOO 14B 720p	default	32	$+3.55$ $[+2.95,\,+4.15]$ 32/0	$+0.027$ $[+0.019,\,+0.035]$ 31/1	$-0.024$ $[-0.031,\,-0.017]$ 31/1	$-0.011$ $[-0.013,\,-0.008]$ 30/2	win
SVOO Wan2.2 720p	default	32	$+4.87$ $[+4.37,\,+5.38]$ 32/0	$+0.057$ $[+0.042,\,+0.072]$ 32/0	$-0.045$ $[-0.055,\,-0.035]$ 32/0	$-0.014$ $[-0.016,\,-0.012]$ 32/0	win
SVOO 1.3B 720p	default	32	$+0.66$ $[+0.19,\,+1.13]$ 21/11	$-0.003$ $[-0.009,\,+0.004]$ 14/18	$+0.012$ $[+0.004,\,+0.021]$ 9/23	$+0.017$ $[+0.011,\,+0.023]$ 4/28	tie
SVOO HunyuanVideo 720p $\times$ 129	default	32	$+3.38$ $[+2.51,\,+4.25]$ 31/1	$+0.039$ $[+0.029,\,+0.050]$ 30/2	$-0.047$ $[-0.060,\,-0.035]$ 29/3	$-0.019$ $[-0.023,\,-0.014]$ 29/3	win

Appendix

Table H.2 : Paired differences (ALLOT

-

default) for the SVOO rows of Table 2 ; verdicts follow § A.1 .

Setting	Baseline	$n$	$\Delta$ PSNR	$\Delta$ SSIM	$\Delta$ LPIPS	$\Delta$ DISTS	Verdict
SpargeAttn 1.3B 480p $\times$ 81, official configuration	default	32	$+5.03$ $[+4.49,\,+5.57]$ 32/0	$+0.108$ $[+0.083,\,+0.133]$ 32/0	$-0.080$ $[-0.100,\,-0.061]$ 32/0	$-0.041$ $[-0.051,\,-0.030]$ 32/0	win
SpargeAttn 1.3B 720p, official configuration	default	32	$+2.24$ $[+1.82,\,+2.65]$ 31/1	$+0.035$ $[+0.022,\,+0.047]$ 29/3	$-0.032$ $[-0.044,\,-0.019]$ 28/4	$-0.015$ $[-0.025,\,-0.005]$ 23/9	win
SpargeAttn 1.3B 720p, retuned	default	32	$+1.99$ $[+1.63,\,+2.36]$ 31/1	$+0.028$ $[+0.016,\,+0.041]$ 30/2	$-0.029$ $[-0.041,\,-0.018]$ 28/4	$-0.014$ $[-0.022,\,-0.006]$ 27/5	win

Appendix

Table H.3 : Paired differences (ALLOT

-

default) for the SpargeAttn rows; verdicts follow § A.1 . The first row is from Table 2 , the other two are the 720p settings above.

Setting	Baseline	$n$	$\Delta$ PSNR	$\Delta$ SSIM	$\Delta$ LPIPS	$\Delta$ DISTS	Verdict
Radial 1.3B 720p, decay 0.2	default	32	$+0.40$ $[+0.11,\,+0.69]$ 26/6	$+0.005$ $[+0.000,\,+0.010]$ 24/8	$-0.005$ $[-0.013,\,+0.002]$ 23/9	$-0.004$ $[-0.010,\,+0.003]$ 21/11	win
Radial 1.3B 720p, decay 1.0	default	32	$+2.00$ $[+1.46,\,+2.54]$ 30/2	$+0.016$ $[+0.009,\,+0.024]$ 26/6	$-0.012$ $[-0.021,\,-0.004]$ 25/7	$-0.007$ $[-0.013,\,+0.000]$ 23/9	win
Radial 14B 720p	default	32	$+0.53$ $[+0.37,\,+0.70]$ 28/4	$+0.011$ $[+0.006,\,+0.015]$ 28/4	$-0.013$ $[-0.017,\,-0.009]$ 28/4	$-0.007$ $[-0.011,\,-0.003]$ 29/3	win
Radial HunyuanVideo 720p $\times$ 129	default	32	$-0.55$ $[-1.17,\,+0.07]$ 16/16	$-0.000$ $[-0.008,\,+0.007]$ 18/14	$+0.010$ $[-0.003,\,+0.023]$ 12/20	$+0.003$ $[-0.004,\,+0.010]$ 16/16	tie

Appendix

Table H.4 : Paired differences (ALLOT

-

default) for the Radial rows of Table 2 and the decay 1.0 setting; verdicts follow § A.1 .

Figure H.1 : Fidelity–speedup curves at 720p

\times

81 (left Wan2.1-1.3B, right Wan2.1-14B). Red: ALLOT; gray: uniform baseline at equal wall-clock time; diamonds: strengthened baseline.

Setting	Baseline	$n$	$\Delta$ PSNR	$\Delta$ SSIM	$\Delta$ LPIPS	$\Delta$ DISTS
XAttention 1.3B 480p $\times$ 81	same density for all layers (trajectory gain only)	32	$-1.07$ $[-1.34,\,-0.80]$ 2/30	$-0.023$ $[-0.029,\,-0.018]$ 1/31	$+0.022$ $[+0.017,\,+0.027]$ 2/30	$+0.021$ $[+0.016,\,+0.026]$ 3/29
XAttention 1.3B 480p $\times$ 81	trajectory gain all 1	32	$-7.05$ $[-7.52,\,-6.58]$ 0/32	$-0.148$ $[-0.173,\,-0.123]$ 0/32	$+0.131$ $[+0.110,\,+0.152]$ 0/32	$+0.042$ $[+0.033,\,+0.051]$ 1/31
XAttention 1.3B 480p $\times$ 81	parametric shape rule	32	$-1.76$ $[-2.02,\,-1.50]$ 1/31	$-0.028$ $[-0.034,\,-0.023]$ 1/31	$+0.023$ $[+0.018,\,+0.028]$ 1/31	$+0.014$ $[+0.010,\,+0.018]$ 3/29
XAttention 1.3B 480p $\times$ 81	density-proportional cost	32	$-0.12$ $[-0.26,\,+0.03]$ 14/18	$+0.002$ $[-0.000,\,+0.005]$ 20/12	$-0.001$ $[-0.003,\,+0.001]$ 16/16	$-0.002$ $[-0.004,\,-0.000]$ 20/12

Appendix

Table I.1 : Paired differences (row

-

ALLOT) for the rows of Table 4 (ablation, no verdict).

Setting	PSNR / LPIPS / DISTS	$\Delta$ PSNR	LPIPS W/L	DISTS W/L
ALLOT	25.99 / 0.098 / 0.107	–	–	–
Same density for all layers (trajectory gain only)	22.77 / 0.189 / 0.177	$-3.22$ [ $-3.78$ , $-2.67$ ] 0/32	0/32	0/32
Trajectory gain all 1	14.33 / 0.389 / 0.187	$-11.66$ [ $-13.17$ , $-10.16$ ] 0/32	0/32	2/30
Parametric shape rule	24.86 / 0.121 / 0.129	$-1.13$ [ $-1.27$ , $-0.99$ ] 0/32	0/32	0/32

Appendix

Table I.2 : Ablation on mean-pooled top-

k

(

\Delta

= row

-

ALLOT).

Model and setting	Speedup	$n$	Uniform baseline	Strengthened baseline	ALLOT	ALLOT $-$ strengthened baseline	ALLOT $-$ uniform baseline
Wan2.1-14B, 720p $\times$ 81	1.29 $\times$	16	24.87	30.49	31.25	$+0.76$ [ $+0.46$ , $+1.06$ ] 15/1	$+6.38$ [ $+5.55$ , $+7.21$ ] 16/0
Wan2.1-1.3B, 480p $\times$ 33	1.14 $\times$	32	19.59	23.44	25.99	$+2.55$ [ $+2.16$ , $+2.94$ ] 32/0	$+6.40$ [ $+5.63$ , $+7.17$ ] 32/0
Wan2.1-14B, 480p $\times$ 33	1.09–1.11 $\times$	32	19.20	27.26	27.85	$+0.60$ [ $+0.41$ , $+0.79$ ] 28/4	$+8.65$ [ $+7.84$ , $+9.46$ ] 32/0
Wan2.2-A14B, 480p $\times$ 33	1.09 $\times$	32	19.36	24.20 ∗	25.86	$+1.67$ [ $+1.37$ , $+1.96$ ] 32/0 ∗	$+6.50$ [ $+5.92$ , $+7.07$ ] 32/0
HunyuanVideo, 480p $\times$ 33	equal density	32	27.82	27.46	28.42	$+0.96$ [ $+0.02$ , $+1.90$ ] 24/8	$+0.60$ [ $-0.34$ , $+1.53$ ] 22/10

Appendix

Table J.1 : PSNR (dB) of four models under the same budget. ∗ : the uniform baseline with layer 0 additionally dense over all steps (the number of warm-up steps was not searched).

Budget	$k{=}5$	10	15	20	25	30	35	ALLOT	ALLOT $-$ best $k$	ALLOT $-$ $k{=}20$
0.20	16.92	19.73	20.23	19.13	15.84	–	–	22.79	$+2.56$ [ $+2.18$ , $+2.94$ ] 32/0	$+3.66$ [ $+3.26$ , $+4.06$ ] 32/0
0.30	17.85	21.00	22.71	23.44	23.14	–	–	25.99	$+2.55$ [ $+2.16$ , $+2.94$ ] 32/0	$+2.55$ [ $+2.16$ , $+2.94$ ] 32/0
0.45	–	–	25.15	26.74	27.77	28.60	29.05	31.15	$+2.09$ [ $+1.73$ , $+2.45$ ] 32/0	$+4.41$ [ $+3.94$ , $+4.88$ ] 32/0

Appendix

Table J.2 : PSNR (dB,

n{=}32

) of the strengthened baseline as a function of

k

on Wan2.1-1.3B 480p

\times

33, and ALLOT.

Variable	Change	Agreement (%)	Correlation of $w$ ’s shape
Probing videos	4 $\to$ 1	XAttention 96.1–98.5; mean-pooled: 1.3B 95.2–98.0, 14B 97.9–98.4, HunyuanVideo 93.4–96.3	–
Probing videos	4 $\to$ 2	XAttention 95.7–99.3; mean-pooled: 1.3B 96.7–97.5, 14B 98.5–99.4, HunyuanVideo 96.7–97.7	–
Calibration prompts	8 $\to$ 1	XAttention 92.3–98.5; mean-pooled: 1.3B 89.1–98.5, HunyuanVideo 86.1–97.9	–
Calibration prompts	8 $\to$ 4	96.7–99.4	–
Window density	0.30 $\to$ 0.15 / 0.50	XAttention 98.9 / 98.5; mean-pooled 99.6 / 97.1	0.995 / 0.995
Window selection strategy	mean-pooled $\to$ SpargeAttn	96.5 / 96.0 (two budgets)	0.993

Appendix

Table K.1 : Robustness of the measurement setup. Agreement: share of cells whose dense/sparse status matches the original table ( Wan2.1-1.3B unless noted).

Setting	Allocation	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Wall-clock (s)
30 steps	Default	19.87	0.696	0.232	92.5
	ALLOT	25.80	0.830	0.110	91.9
CFG 7.5	Default	17.68	0.668	0.255	150.1
	ALLOT	24.48	0.827	0.101	147.7
Euler sampler	Default	18.79	0.664	0.278	147.8
	ALLOT	25.97	0.832	0.113	147.1

Appendix

Table K.2 : Changing one inference setting after calibration (XAttention, Wan2.1-1.3B 480p

\times

81,

n{=}32

; other settings as in Table 1 ). Fidelity is measured against the dense output under the same setting; wall-clock time follows Appendix A .

Sparse Attention

Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.

See all 244 papers

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
Aug 12, 2026Enhuai Liu, Yunke Wang, Yutong Wang +2Video Diffusion Models Diffusion Transformer
1The University of Sydney · 2CSIRO
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1

May 22, 2026J/K move · Enter open · S save

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Jie Hu, Zixiang Gao, Yutong He, Kun Yuan

1Peking University.

Video Diffusion Models Diffusion Transformer Transformer Attention Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Open paper AskPDF arXiv

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1

\times

end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Abstract

Sparse Attention

Explore similar work

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers