SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning

Jintao Zhang; Kai Jiang; Chendong Xiang; Weiqi Feng; Yuezhou Hu; Haocheng Xi; Jianfei Chen; Jun Zhu

cs.CVFeb 13, 2026

SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning

Authors: Jintao Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng, Yuezhou Hu, Haocheng Xi, Jianfei Chen, Jun Zhu

Organizations: Tsinghua University · UC Berkeley

Abstract

Many training-free sparse attention methods are effective for accelerating diffusion models. Recently, several works suggest that making sparse attention trainable can further increase sparsity while preserving generation quality. We study three key questions: (1) when do the two common masking rules, i.e., Top-k and Top-p, fail, and how can we avoid these failures? (2) why can trainable sparse attention reach higher sparsity than training-free methods? (3) what are the limitations of fine-tuning sparse attention using the diffusion loss, and how can we address them? Based on this analysis, we propose SpargeAttention2, a trainable sparse attention method that achieves high sparsity without degrading generation quality. SpargeAttention2 includes (i) a hybrid masking rule that combines Top-k and Top-p for more robust masking at high sparsity, (ii) an efficient trainable sparse attention implementation, and (iii) a distillation-inspired fine-tuning objective to better preserve generation quality during fine-tuning using sparse attention. Experiments on video diffusion models show that SpargeAttention2 reaches 95% attention sparsity and a 16.2x attention speedup while maintaining generation quality, consistently outperforming prior sparse attention methods.

Figures & tables

Figure 1: Qualitative examples of text-to-video generation. We compare the original full-attention model with SpargeAttention2 under high attention sparsity. SpargeAttention2 preserves visual quality, temporal coherence, and text–video alignment comparable to full attention, while substantially reducing attention computation. The prompts used for generation are in Appendix C

Model	Scale-Res.	AQ $\uparrow$	VR $\uparrow$	VA $\uparrow$	VT $\uparrow$
Original	1.3B-480p	64.41	0.1084	81.28	85.80
Fine-tuning	1.3B-480p	61.83	0.0936	75.45	80.87
Original	14B-720p	64.66	0.1238	86.15	87.00
Fine-tuning	14B-720p	63.48	0.1181	79.86	81.95

Table 3: Full-attention diffusion fine-tuning degrades alignment under distribution mismatch. Without access to the original pre-training data, optimizing the diffusion loss alone leads to consistent degradation in aesthetic quality, vision reward, and VQA accuracy, even when full attention is used.

Method	IQ $\uparrow$	OC $\uparrow$	AQ $\uparrow$	VR $\uparrow$	VQA-a $\uparrow$	VQA-t $\uparrow$	Sparsity $\uparrow$	Attn Time $\downarrow$	E2E Time $\downarrow$
Full Attention	63.67	20.27	64.41	0.1084	81.28	85.80	0%	97s	159s
SpargeAttn	35.28	13.41	40.53	-0.1398	3.258	0.608	89%	12.6s	74.6s
VSA	59.57	19.27	50.60	-0.0881	33.35	48.36	90%	25s	87s
VMoBA	65.31	20.82	64.14	0.0936	78.99	86.69	90%	36s	98s
SLA	63.14	21.09	62.91	0.0881	72.66	80.51	95%	11s	73s
SpargeAttn2	67.68	21.57	65.05	0.1010	83.86	87.73	95%	6s	68s

Table 4: Effectiveness comparison on Wan2.1-1.3B at 480p resolution. Bold indicates the best result among sparse-attention methods.

Method	IQ $\uparrow$	OC $\uparrow$	AQ $\uparrow$	VR $\uparrow$	VQA-a $\uparrow$	VQA-t $\uparrow$	Sparsity $\uparrow$	Attn Time $\downarrow$	E2E Time $\downarrow$
Full Attention	68.01	22.44	64.66	0.1238	86.15	87.00	0%	2550s	3043s
SpargeAttn	38.46	16.26	42.16	-0.1306	9.926	4.451	86%	415s	908s
VSA	64.03	21.27	63.37	0.1074	77.63	85.60	90%	651s	1144s
VMoBA	67.18	20.85	63.64	0.1117	81.66	83.21	90%	832s	1325s
SLA	64.43	20.89	61.89	0.1078	76.90	82.60	95%	285s	778s
SpargeAttn2	69.08	21.62	64.92	0.1149	85.21	87.48	95%	157s	650s

Table 5: Effectiveness comparison on Wan2.1-14B at 720p resolution. Bold indicates the best result among sparse-attention methods.

	Wan2.1-1.3B-480p					Wan2.1-14B-720p
Variant	IQ	OC	AQ	VR	VQA	IQ	OC	AQ	VR	VQA
SpargeAttn2	67.68	21.57	65.05	0.1010	86.73	68.41	21.06	65.02	0.1119	88.22
Top-k only	65.84	21.51	64.57	0.0916	86.90	65.24	20.64	63.99	0.0935	84.25
Top-p only	60.56	21.12	60.12	0.0312	62.57	63.37	21.33	63.62	0.1090	86.43
Training-free	53.18	19.87	48.93	-0.0650	20.40	62.17	19.41	57.01	-0.0272	45.85
–VD	67.23	21.26	63.34	0.0939	85.05	66.40	20.56	64.59	0.1082	85.05

Table 6: Ablation studies on SpargeAttention2. “–VD” replaces velocity distillation with standard diffusion fine-tuning. The 1.3B and 14B ablations use 500 and 100 training steps, respectively (Appendix B ). VQA denotes the overall score combining VQA-a and VQA-t.

Figure 4: A representative example of text-to-video generation under high attention sparsity, evaluated on Wan2.1-14B at 720p. SpargeAttention2 produces a semantically correct video. In contrast, SLA and VSA produce videos in which the male character walks backward, while VMoBA fails to generate the female character specified in the prompt. The prompt used for generation is in Appendix C

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Wan2.1-1.3B
Method	Quality	Semantic	Total	Sparsity
Full Attention	83.95	82.38	83.63	0%
SpargeAttn	63.10	35.84	57.65	89%
VSA	77.21	64.32	74.63	90%
VMoBA	80.03	78.92	79.81	90%
SLA	81.24	78.79	80.75	95%

Appendix

Table 7: Complete VBench evaluation on Wan2.1. Higher scores are better. Bold indicates the best score among the sparse-attention methods.

Method	Quality	Semantic	Total	Sparsity
Full Attention	81.55	83.01	81.84	0%
SLA	80.02	73.90	78.80	95%
SpargeAttention2	81.43	81.56	81.46	95%

Appendix

Table 8: VBench evaluation on a private 720p MM-DiT model. Higher scores are better. Bold indicates the better score between the two sparse-attention methods.

Sparse Attention

Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.

See all 243 papers

MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
Oct 5, 2026Jiarui Chen, Zeqiang Lai, Jiangshan Wang +5Video Diffusion Models Diffusion Transformer
Fudan University · Tencent HY · Shanghai Innovation Institute +4
Training-free sparse attention based on cumulative energy filtering
Jun 15, 2026Chunlu Li, Yixuan Pan, Bai Du +5Video Diffusion Models Diffusion Transformer
Huawei Technologies
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.

Oct 5, 2026J/K move · Enter open · S save

MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

Jiarui Chen, Zeqiang Lai, Jiangshan Wang, Ziheng Ouyang+4

Fudan University · Tencent HY · Shanghai Innovation Institute+4

Video Diffusion Models Diffusion Transformer Sparse Attention Diffusion Model Caching Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Sparse attention is a primary approach to reducing the latency of diffusion transformers in long-sequence generation tasks, such as video and high-resolution 3D asset generation. However, existing methods can degrade generation quality and fidelity at high sparsity levels. Through controlled oracle comparisons, we trace this degradation to three sources: constraints imposed by token grouping, inaccurate interaction selection, and the attention contributions lost when tokens are discarded. Guided by this analysis, we propose Meta-Cached Sparse Attention (MC-Sparse), a training-free framework that selects individual key-value (KV) tokens while organizing similar queries into tile-aligned groups for efficient GPU execution. MC-Sparse caches metadata comprising query groups, KV indices selected using exact attention probabilities, and residuals between dense and sparse attention outputs, and reuses them across subsequent denoising steps. Across video and 3D generation models, MC-Sparse achieves higher fidelity to dense-attention outputs and larger denoising speedups than existing sparse-attention baselines, without visible quality degradation. Relative to dense attention, it delivers a

1.80\times

denoising speedup on Minimax-H3-Base and a

2.32\times

speedup on 3D asset generation, both with negligible quality loss.

Abstract

Sparse Attention

Explore similar work

MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

Training-free sparse attention based on cumulative energy filtering

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation