Competition Domains Matter: Selective Head Globalization for Training-Free Sparse Attention in Video Diffusion Transformers

Tiantian Liu; Meng Wan; Zhiqiang Liang; Jue Wang; Ningming Nie; Yangang Wang

cs.CVOct 10, 2026

Competition Domains Matter: Selective Head Globalization for Training-Free Sparse Attention in Video Diffusion Transformers

Authors: Tiantian Liu, Meng Wan, Zhiqiang Liang, Jue Wang, Ningming Nie, Yangang Wang

Organizations: University of Chinese Academy of Sciences, Beijing, China

Abstract

Training-free sparse attention is a practical route to efficient video diffusion transformer inference, yet existing methods typically fix the competition domain at each query row for every attention head. Prior work primarily addresses how to estimate token-level importance within this fixed domain; whether the domain itself should adapt to head heterogeneity remains under-specified. We introduce EG-Sparse, a training-free framework that routes low-entropy heads to a grouped cross-head pool while retaining per-row local selection for high-entropy heads, using only scores already materialized by the sparse pipeline. A controlled diagnostic fixes the ranking score and varies only the competition rule: grouped cross-head selection reduces low-entropy-head reconstruction error by 9.8%, whereas forcing high-entropy heads into an all-head global pool increases it by 7.9%. Across four configurations, EG-Sparse achieves higher video quality while reducing attention density and GPU denoising latency compared with strong training-free sparse-attention baselines. Mechanism ablations further show that selective domain adaptation outperforms uniform, random, and fixed domain-assignment policies. These results support competition-domain adaptation as a distinct design axis within centroid-guided training-free sparse attention.

Figures & tables

Figure 1: Attention heads exhibit substantial structural heterogeneity. (a) Normalized attention entropy $\widetilde{H}_{h}=H_{h}/\log C_{k}$ varies across layers and heads. (b–c) Two representative heads from the same layer show distinct dense centroid-attention patterns: a low-entropy head concentrates its mass on a small set of query–key cluster pairs, whereas a high-entropy head distributes attention broadly across keys. This heterogeneity motivates head-adaptive competition domains.

Figure 2: Overview of EG-Sparse. A fixed row-wise domain constrains low-entropy heads because the budget cannot flow across rows to where it is most needed, whereas indiscriminate all-head global competition penalizes high-entropy heads. EG-Sparse resolves both: entropy routes low-entropy heads to a grouped cross-head pool, retains per-row local selection for the remaining heads, and merges both masks for execution by the sparse attention kernel.

Figure 3: Competition-Domain Mismatch under a Fixed Ranking Score. Bars report the relative reconstruction-error change from the per-row local baseline for low- and high-entropy heads; error bars show mean $\pm$ 95% CI over 200 HunyuanVideo-T2V-13B prompts at 480P. The overlaid curves report the retained-budget ratio relative to the per-row local baseline. Low-entropy heads benefit increasingly from domain expansion, whereas high-entropy heads incur a $+7.9\%$ error increase under all-head global selection, where low-entropy heads capture a disproportionate share of the pooled budget.

Model	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VBench $\uparrow$	Density $\downarrow$	Latency (s) $\downarrow$	Speedup vs. Dense $\uparrow$
Wan2.1 I2V-14B 720P	Dense	–	–	–	79.31	100%	1823	1.00 $\times$
	SVG	17.80	0.635	0.318	77.40	28%	974	1.87 $\times$
	SpargeAttn	17.60	0.628	0.325	77.20	41%	1050	1.74 $\times$
	SVG2	18.26	0.649	0.307	77.73	27%	902	2.02 $\times$
	EG-Sparse (Ours)	18.75 †	0.672	0.252	78.52	25%	889	2.05 $\times$
Wan2.1 T2V-14B 720P	Dense	–	–	–	78.16	100%	1815	1.00 $\times$

Table 1: Main results in the evaluated model configurations. All sparse methods use no dense warmup. Density is the fraction of retained centroid pairs; lower is better. GPU denoising latency is the total GPU time for the 50-step DiT denoising process, and speedup is relative to Dense.

\dagger

p<0.05

vs. SVG2 (Wilcoxon).

Selection Rule	PSNR $\uparrow$	VBench $\uparrow$	Speedup $\uparrow$
SVG2: per-row local	21.56	76.79	1.15 $\times$
All-head global	21.47	76.69	1.18 $\times$
Random 50/50 split	21.18	76.41	1.16 $\times$
Fixed head-index split	21.28	76.52	1.16 $\times$
Entropy split	21.95 †	77.35	1.13 $\times$
w/o Value weighting	21.79 †	77.15	1.14 $\times$

Table 2: Selective-globalization ablation on Wan2.1-T2V-1.3B at 480P. All variants share the same upstream candidate representation and use

p{=}0.9

\dagger

p<0.05

vs. SVG2 (Wilcoxon).

Grouped Cross-Head (of 24)	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
6 ( $25\%$ )	19.86	0.701	0.280
10 ( $40\%$ )	20.04	0.700	0.272
12 ( $50\%$ )	20.35	0.740	0.252
14 ( $60\%$ )	20.11	0.710	0.267
18 ( $75\%$ )	19.78	0.708	0.277

Table 3: Sensitivity to the fraction of low-entropy heads routed to grouped cross-head selection on HunyuanVideoT2V13B at 480P with

p{=}0.9

. Heads are added in ascending entropy order. Fidelity improves as heads predicted to benefit from grouped cross-head selection enter the pool, peaks at the median split, and declines as the pool expands into the higher-entropy regime where inter-group budget interference outweighs the gain.

Component	Latency (ms)	Share
Candidate preprocessing	13.64	41.8%
Head-entropy computation	0.13	0.4%
Selective domain routing and mask selection	2.80	8.6%
FlashInfer sparse attention	16.06	49.2%
Total per attention block	32.63	100%

Table 4: Per-attention-block latency breakdown on HunyuanVideo-T2V-13B at 480P (GH200, ms). Computing the entropy signal costs only

0.13

ms; the complete selective domain routing and sparse execution costs are reported separately.

Sparse Attention

Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.

See all 244 papers

HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention
May 14, 2026Xuzhe Zheng, Yuexiao Ma, Jing Xu +3Video Diffusion Models Diffusion Transformer
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention
Aug 13, 2026Qi Zhao, Qirui Li, Hanlin Tang +10Video Diffusion Models Diffusion Transformer
Zhejiang University · Alibaba Group
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1

May 14, 2026J/K move · Enter open · S save

HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention

Xuzhe Zheng, Yuexiao Ma, Jing Xu, Xiawu Zheng+2

1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China

Video Diffusion Models Diffusion Transformer Sparse Attention Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Diffusion-based video generation has advanced substantially in visual fidelity and temporal coherence, but practical deployment remains limited by the quadratic complexity of full attention. Training-free sparse attention is attractive because it accelerates pretrained models without retraining, yet existing online top-

p

sparse attention still spends non-negligible cost on mask prediction and applies shared thresholds despite strong head-level heterogeneity. We show that these two overlooked factors limit the practical speed-quality trade-off of training-free sparse attention in Video DiTs. To address them, we introduce a head-wise adaptive framework with two plug-in components: Temporal Mask Reuse, which skips unnecessary mask prediction based on query-key drift, and Error-guided Budgeted Calibration, which assigns per-head top-

p

thresholds by minimizing measured model-output error under a global sparsity budget. On Wan2.1-1.3B and Wan2.1-14B, our method consistently improves XAttention and SVG2, achieving up to 1.93 times speedup at 720P while maintaining competitive video quality and similarity metrics.

Abstract

Sparse Attention

Explore similar work

HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention

SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers