Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

Pardis Taghavi; Reza Langari; Gaurav Pandey

cs.CVAug 19, 2026

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

Authors: Pardis Taghavi, Reza Langari, Gaurav Pandey

Organizations: Texas A&M University

Abstract

Block-sparse attention accelerates video transformers by grouping queries and keys and computing attention only between selected groups. However, it faces two challenges: (1) queries sharing a block route may attend to different keys; and (2) retaining most attention mass does not necessarily preserve the attention output. We find that partition choice affects both the overlap among grouped queries' preferred supports and how well an affine function of the sparse output can represent the difference between dense and sparse outputs. We introduce SparsePR, a training-free method combining Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction. We first group paired key/value tokens by how their keys respond to sampled queries. We then group queries by their responses to the resulting key-group centroids for shared sparse routing. We compute exact attention on a small set of query rows and fit an affine correction to the differences between their dense and sparse outputs using weighted ridge regression. The fitted map predicts a residual for each unprobed query, which we add to its sparse output. We measure attention-output error as the relative L2 error with respect to each query's dense output. Across four video generation and world models, SparsePR reduces this error, averaged over unprobed queries, by 61.6--90.5% compared with semantic block-sparse attention without residual correction, at 22% executed-pair density including exact probes. SparsePR achieves 1.48x--2.61x end-to-end speedups over dense attention, with dense-reference PSNR of 24.42--31.84 dB at 21.9--26.0% realized executed-pair density. Project page: https://pardistaghavi.github.io/SparsePR-website/

Figures & tables

Figure 1: Qualitative results for the Cosmos-Predict2.5-14B model.

Figure 2: Overview of SparsePR. Response-Coupled Partitioning builds executable K/V and query groups, and Probe-Fitted Residual Reconstruction uses exact probe rows to correct the sparse output.

Figure 3: Structural observations under the semantic reference partition. (a) Pooling eight queries expands the support required to retain $90\%$ attention mass. Markers denote model medians. (b) Nearly identical retained mass can yield substantially different output errors.

		Dense-Reference Fidelity			VBench / VBench++		PBench	Efficiency
Model	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ImgQual $\uparrow$	SubCons $\uparrow$	Quality $\uparrow$	Density $\downarrow$	PFLOPs $\downarrow$	E2E Speedup $\uparrow$
Content-generation video models
HunyuanVideo 13B, 720p Text-to-Video	Dense	–	–	–	0.850	0.976	–	100.0%	612.38	1.00 $\times$
	SpargeAttn Zhang et al. (2025a)	25.31	0.832	0.217	0.763	0.928	–	40.19%	389.76	1.38 $\times$
	SVG2 Yang et al. (2026)	29.87	0.907	0.121	0.850	0.927	–	25.45%	299.02	2.30 $\times$
	SVOO Luo et al. (2026)	24.87	0.843	0.224	0.679	0.976	–	33.26%	345.81	2.17 $\times$

Table 1: Quality–efficiency comparison across content-generation and physical video world models.

Configuration	HunyuanVideo	Wan2.2	Cosmos- Predict2.5	Cosmos3-Nano
Partition construction, repair disabled
Semantic partition	0.089 / 0.714	0.164 / 1.734	0.791 / 7.532	0.359 / 3.356
Key-response K/V partition	0.085 / 0.812	0.156 / 1.659	0.769 / 7.270	0.341 / 3.174
Response-coupled partition	0.073 / 0.696	0.108 / 0.647	0.761 / 7.227	0.331 / 3.150
Partition–repair interaction
Semantic partition + probe repair	0.053 / 0.356	0.105 / 0.819	0.262 / 0.826	0.124 / 0.668

Table 2: Partition and reconstruction ablations at 22% executed-pair density, including exact probes. Entries report mean/p99 normalized attention output error on unprobed rows. Lower is better. Repair comparisons fix probe indices and estimator settings. The key-response K/V variant changes only K/V grouping, retaining semantic query groups.

Configuration	HunyuanVideo	Wan2.2	Cosmos- Predict2.5	Cosmos3-Nano
Probe selection: full weighted ridge fixed
Random probes	0.039 / 0.235	0.080 / 0.356	0.075 / 0.503	0.083 / 0.467
Uniform spatiotemporal probes	0.037 / 0.246	0.089 / 0.394	0.090 / 0.543	0.096 / 0.549
Query group stratified probes	0.030 / 0.207	0.063 / 0.323	0.075 / 0.489	0.076 / 0.447
Correction family: query group stratified probes fixed
Mean-only	0.043 / 0.362	0.089 / 0.455	0.350 / 3.538	0.191 / 1.376

Table 4: Probe-selection and correction-family ablations with response-coupled partitioning,

M=64

\lambda=1

. Within each panel, only the indicated component varies. Entries report mean/p99 normalized attention-output error on a shared set of unprobed query rows. Lower is better. Bold row labels mark the reference configuration, repeated in both panels.

Figure 4: Partitioning and runtime analysis. (a) Error reduction from response-coupled vs. semantic partitioning at 22% density. (b) Wan2.2 full generation latency for Dense, SVG2, and SparsePR.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Requested query clusters $C_{q}$	$300$
Requested K/V clusters $C_{k}$	$1000$
Metric query sampling budget per query head	$2048$
Initial / subsequent $k$ means iteration limits	$25/2$
$k$ means convergence tolerance	$10^{-4}$
Key response rank $r_{K}$	$48$

Appendix

Table C.1: Method configuration used in the evaluated models.

Model	$\bm{N_{q}}$	$\bm{M/N_{q}}$
HunyuanVideo	$118{,}800$	$0.054\%$
Wan2.2	$75{,}600$	$0.085\%$
Cosmos-Predict2.5	$84{,}480$	$0.076\%$
Cosmos3-Nano	$44{,}160$	$0.145\%$

Appendix

Table C.2: Probe fractions for the evaluated sequence lengths with

M=64

Figure D.1: Matched retained mass across models. Dark points are heads from the same input, layer, and generation step as Cases A and B. Stars mark two heads with nearly identical retained mass. The logarithmic y axis shows multiplicative differences in p99 normalized output error.

Model	Per query RMS	Global RMS	Per query RMS + log magnitude
HunyuanVideo	0.073	0.074	0.083
Wan2.2	0.108	0.109	0.166
Cosmos-Predict2.5	0.761	0.762	0.777
Cosmos3-Nano	0.331	0.332	0.337

Appendix

Table D.1: Query RMS normalization ablation. Mean normalized attention output error

\lVert O_{i}^{\mathrm{sp}}-O_{i}^{\mathrm{dense}}\rVert_{2}/(\lVert O_{i}^{\mathrm{dense}}\rVert_{2}+10^{-8})

. Lower is better. Bold entries mark the lowest error in each row.

(a) Probe count with fixed $\lambda=1$ and no output truncation
Model	$M=16$	$M=32$	$\bm{M=64}$	$M=128$
HunyuanVideo	0.037	0.034	0.030	0.027
Wan2.2	0.080	0.071	0.063	0.057
Cosmos-Predict2.5	0.118	0.092	0.075	0.065
Cosmos3-Nano	0.097	0.082	0.076	0.074

Appendix

Table D.2: Sensitivity to probe count and estimator settings. Entries report mean normalized attention output error on the same unprobed query rows across settings. Lower is better. Each panel varies one parameter while holding the others fixed.

Figure D.2: Oracle affine residual representability under response coupled versus semantic partitioning. The left panel shows the increase in the fraction of residual squared norm explained by the best affine fit. The right panel shows the reduction in squared error remaining after fitting. We fit all 2048 evaluated queries per head and sum energies across cases before computing ratios. Cosmos-Predict2.5 uses the subset matched by density.

Figure E.1: Dense and SparsePR outputs for Cosmos3-Nano-16B on VBench++.

Figure E.2: Dense and SparsePR outputs for Cosmos3-Nano-16B on PBench.

Figure E.3: Qualitative comparison on HunyuanVideo-13B.

Figure E.4: Qualitative comparison on Wan2.2-I2V-A14B.

Sparse Attention

Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.

See all 244 papers

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Video Diffusion Models Sparse Attention
NVIDIA
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering
Mar 19, 2026Jiayi Luo, Jiayu Chen, Jiankun Wang +6Efficient Video Generation Video Generation
SKLCCSE, School of Computer Science and Engineering, Beihang University · Zhongguancun Academy · School of Computer Science, Peking University +4

Jul 27, 2026J/K move · Enter open · S save

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

Haopeng Li, Yitong Li, Junsong Chen, Tian Ye+7

NVIDIA

Video Diffusion Models Sparse Attention Diffusion Model Inference Acceleration Block-Sparse Attention

Open paper AskPDF arXiv

Diffusion transformers are essential for high-fidelity video generation, but long token sequences make attention a dominant inference bottleneck. Training-free dynamic sparse attention alleviates this bottleneck by computing only selected key-value blocks, yet existing methods struggle to sparsify attention both efficiently and accurately for two reasons: (1) Rigid, unpredictable, and costly routing: selecting a fixed fraction of top-ranked blocks by proxy score imposes fixed budgets, whereas retaining blocks to reach a target cumulative proxy probability mass yields dynamic but potentially imbalanced budgets; both incur non-negligible overhead from computing and materializing proxy scores. (2) Lossy keep-or-drop sparsification: unselected blocks are discarded entirely, degrading accuracy under aggressive sparsity. These limitations motivate cheaper dynamic-budget routing while limiting accuracy degradation. In this paper, we introduce training-free Sol-Attn (Sparsifying online attention), which unifies dynamic routing, sparse computation, and approximation correction in a single online-softmax pass, achieving a better accuracy-efficiency trade-off in sparse attention. The core of Sol-Attn is on-the-fly block thresholding with proxy-score reuse, which selects critical blocks by comparing block proxy scores against a threshold during online softmax. This design enables dynamic yet controllable block budgets without materializing the proxy map, while directly reusing the proxy scores of unselected blocks to approximate their contribution. Experiments across image and video generation tasks show that Sol-Attn advances the quality-efficiency frontier of training-free sparse attention, delivering 2.1 times and 2.3 times end-to-end speedups for video generation and editing, respectively, while preserving visual quality.

Abstract

Sparse Attention

Explore similar work

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering