SLA2: Sparse-Linear Attention with Learnable Routing and Quantization-aware Training

Jintao Zhang; Haoxu Wang; Kai Jiang; Kaiwen Zheng; Youhe Jiang; Ion Stoica; Jianfei Chen; Jun Zhu

cs.LGFeb 13, 2026

SLA2: Sparse-Linear Attention with Learnable Routing and Quantization-aware Training

Authors: Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, Youhe Jiang, Ion Stoica, Jianfei Chen, Jun Zhu,

Organizations: Tsinghua University · UC Berkeley

Abstract

Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality.

Figures & tables

Figure 1: Computation pipeline of SLA2 .

Model	Method	Quality						Efficiency
Model	Method	IQ $\uparrow$	OC $\uparrow$	AQ $\uparrow$	MS $\uparrow$	SC $\uparrow$	VR $\uparrow$	FLOPs $\downarrow$	Sparsity $\uparrow$
Wan2.1 -T2V -1.3B -480P	Full Attention	63.67	20.27	64.41	98.95	95.40	0.1084	52.75T	0%
	VMoBA	65.31	20.82	64.14	97.80	86.69	0.0936	5.28T	90%
	VSA	59.57	19.27	50.60	97.44	87.98	-0.0881	5.40T
	SLA	63.10	20.88	64.34	97.90	92.54	0.0872	5.40T
	SLA2	67.70	21.62	64.86	98.69	95.54	0.1093	5.51T

Table 1: Quality and efficiency metrics of SLA2 and the baseline methods. Bold quality scores indicate the best result within each sparse-method group.

Figure 2: Visible examples of SLA2 and baselines on Wan2.1-T2V-1.3B-480P model. The prompt used for generation is in Appendix C .

Figure 3: Visible examples of SLA2 and baselines on Wan2.1-T2V-14B-720P model. The prompt used for generation is in Appendix C .

Figure 5: End-to-end generation latency of SLA2 and baselines with different sparsities.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	IQ	OC	AQ	MS	SC	VR
Without $\alpha$	64.05	20.66	61.79	94.71	91.37	0.1008
With $\alpha$	66.64	21.42	64.62	98.04	94.83	0.1039

Appendix

Table 3: Ablation of the

\alpha

-based correction. Higher scores are better.

Method	Fine-tuning steps	IQ	OC	VR
Full Attention	0	63.67	20.27	0.1084
Full Attention	500	65.79	21.10	0.1014
SLA2 (90% sparsity)	500	67.70	21.62	0.1093

Appendix

Table 4: Comparison under matched fine-tuning data and step budgets. Higher scores are better.

Sparse Attention

Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.

See all 243 papers

SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
Feb 13, 2026Jintao Zhang, Kai Jiang, Chendong Xiang +5Sparse Attention Diffusion Model Acceleration
Tsinghua University · UC Berkeley
LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
Aug 12, 2026Enhuai Liu, Yunke Wang, Yutong Wang +2Video Diffusion Models Diffusion Transformer
1The University of Sydney · 2CSIRO
LVSA: Training-Free Sparse Attention for Long Video Diffusion
May 29, 2026Gael Glorian, Ioannis Lamprou, Zhen Zhang +2Video Diffusion Models Long-Horizon Video Generation
Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.

Feb 13, 2026J/K move · Enter open · S save

SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning

Jintao Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng+4

Tsinghua University · UC Berkeley

Sparse Attention Diffusion Model Acceleration Diffusion Model Fine-Tuning Video Diffusion Models

Open paper AskPDF arXiv

Many training-free sparse attention methods are effective for accelerating diffusion models. Recently, several works suggest that making sparse attention trainable can further increase sparsity while preserving generation quality. We study three key questions: (1) when do the two common masking rules, i.e., Top-k and Top-p, fail, and how can we avoid these failures? (2) why can trainable sparse attention reach higher sparsity than training-free methods? (3) what are the limitations of fine-tuning sparse attention using the diffusion loss, and how can we address them? Based on this analysis, we propose SpargeAttention2, a trainable sparse attention method that achieves high sparsity without degrading generation quality. SpargeAttention2 includes (i) a hybrid masking rule that combines Top-k and Top-p for more robust masking at high sparsity, (ii) an efficient trainable sparse attention implementation, and (iii) a distillation-inspired fine-tuning objective to better preserve generation quality during fine-tuning using sparse attention. Experiments on video diffusion models show that SpargeAttention2 reaches 95% attention sparsity and a 16.2x attention speedup while maintaining generation quality, consistently outperforming prior sparse attention methods.

Abstract

Sparse Attention

Explore similar work

SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning

LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration

LVSA: Training-Free Sparse Attention for Long Video Diffusion