cs.CVNov 15, 2026

PoCoAttn: Position-Content Collaborative Sparse Attention for Fast and High-Fidelity Video Generation

Authors: Jianwen Wu, Meiling Wang, Junkang Zhou, Yipeng Shen, Haoran You

Organizations: Purdue University

Abstract

Video diffusion transformers rely on 3D full attention over tens of thousands of spatiotemporal tokens, and its quadratic cost dominates inference latency. Sparse attention reduces this cost, but existing methods face a dilemma: static methods apply predefined masks and cannot capture input-dependent attention, whereas dynamic methods reconstruct the sparse mask repeatedly, paying online to rediscover positional structure that is largely stable across inputs. To this end, we propose PoCoAttn, a training-free sparse attention framework that separates input-stable from input-dependent attention and resolves the former offline and the latter online. PoCoAttn features: (1) A relative-offset analysis of 3D query-key pairs, which shows that video attention decomposes into an input-stable positional prior that carries most of the attention mass and a small content-dependent residual. (2) A position-content collaborative sparse attention mechanism, which retrieves positional keys from offline-calibrated tables, selects content keys online with a head-adaptive ratio between the two sources, and compensates for the discarded keys to approximate full attention. (3) A step-adaptive budget allocation algorithm, which models how early-step sparse attention errors are amplified along the denoising trajectory and derives the optimal sparsity budget per step. Extensive experiments across four video diffusion models and three tasks show that PoCoAttn consistently achieves a better speed-fidelity Pareto frontier than state-of-the-art (SOTA) methods, improving PSNR by 1.9 to 5.0 dB over baselines and delivering up to 4.1×\times attention and 2.0×\times denoising speedups over dense FlashAttention-3 attention.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DecoupAttn: Decoupled Sparse Attention for Training-Free Acceleration of Video Diffusion

    Nov 4, 2026Yanjin Li, Bingbing Qiu, Qiang Huang +5Diffusion Model Inference AccelerationSparse Attention

  2. Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering

    Mar 19, 2026Jiayi Luo, Jiayu Chen, Jiankun Wang +6Efficient Video GenerationVideo Generation