Long-context video understanding and generation pose a significant computational challenge for Transformer-based video models due to the quadratic complexity of self-attention. While existing sparse attention methods employ coarse-grained patterns to improve efficiency, they typically incur redundant computation and suboptimal performance. To address this issue, in this paper, we propose VecAttention, a novel framework of vector-wise sparse attention that achieves superior accuracy-efficiency trade-offs for video models. We observe that video attention maps exhibit a strong vertical-vector sparse pattern, and further demonstrate that this vertical-vector pattern offers consistently better accuracy-sparsity trade-offs compared with existing coarse-grained sparse patterns. Based on this observation, VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection that minimizes memory access overhead and an optimized kernel of vector sparse attention. Comprehensive evaluations on video understanding (VideoMME, LongVideoBench, and VCRBench) and generation (VBench) tasks show that VecAttention delivers a 2.65× speedup over full attention and a 1.83× speedup over state-of-the-art sparse attention methods, with comparable accuracy to full attention. Our code is available at https://github.com/anminliu/VecAttention.
Figures & tables
Figure 1: (a–b) Under the same sparsity budget, vertical-vector (V-Vec) achieves higher recall and better approximates the oracle pattern than horizontal-vector (H-Vec) and other coarse line/block patterns, for both VLMs (video understanding) and DiTs (video generation). Markers such as (68.9%, 0.95) denote (sparsity, recall), with sparsity in percentage. (c) On VideoMME at matched full-accuracy settings, VecAttention attains higher effective sparsity and faster attention computation, with low important-region selection overhead, compared to existing coarse-grained methods.
Figure 2: Attention maps visualized on different layers across tasks of video understanding and video generation.
Figure 3: Sparsity–recall tradeoff of different sparse patterns across long-context video understanding (left three columns) and video generation (right column) tasks. The top row shows results on the causal InternVL-3.5-8B VLM and non-causal HunyuanVideo DiT, while the bottom row presents the causal Qwen2.5-VL-7B VLM and non-causal Wan2.1-T2V-14B DiT. The vertical-vector pattern consistently yields higher recall under the same sparsity than horizontal-vector pattern and coarse line/block pattern, closely tracking the oracle pattern across different datasets and models.
Figure 4: The overview of VecAttention, which can be divided into two stages: (1) Important-Vector Selection via TilingSelect and minS Filter; (2) Vector Sparse Attention on selected vectors.
Figure 5: Recall heatmaps of different filter strategies under the same sparsity. The minS filter strategy achieves comparable recall under the same sparsity level. Markers such as (75.0%, 0.96) denote (sparsity, recall), with sparsity in percentage.
Figure 6: Time and memory-traffic breakdown at context length N=64K and sparsity 0.9 . Compared with a naive materialize-then-filter baseline (e.g., topP), TilingSelect with minS avoids writing estimated attention map to HBM, cutting memory traffic from 18.3 GB to 1.8 GB and reducing selection latency by 2.42 × , yielding an overall 9.12 × speedup.
Model
Avg. Sp.
Video MME
Long Video
VCR- Bench
Avg. Acc.
InternVL-3.5-8B #Frames: 64 #Tokens: 17K
Full Atten.
0.0
65.7
59.4
32.9
52.7
FlexPref.
76.5
52.3
59.0
30.0
47.1
XAtten.
78.1
56.0
59.9
32.5
49.5
AnchorAtten.
78.6
57.4
59.4
31.3
49.4
VecAtten.
78.6
60.6
59.0
33.8
51.1
Table 1: Performance of different models and different methods on video understanding tasks. All values are percentages (i.e., multiplied by 100) without the % symbol.
Model
Sparsity(%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
Wan 2.1-T2V-14B 720P, #Tokens 76K
XAttention
54.6
19.7
0.658
0.348
SVG
52.2
18.7
0.639
0.381
VecAttention
52.3
19.7
0.668
0.339
HunyuanVideo-T2V-13B 720P, #Tokens 119K
XAttention
60.8
21.2
0.734
0.348
Table 2: Performance of different models and methods on video generation tasks, with 50 inference steps and a 6% warm-up ratio.
Figure 7: Microbenchmarks of efficiency-accuracy trade-offs across different sparse attention methods on VideoMME. VecAttention preserves accuracy at up to 93% sparsity and translates it into practical gains, which are 2.65 × acceleration for attention and 1.17 × end-to-end TTFT.
Figure 8: Ablation study on VideoMME, which contains diverse video inputs ranging from 11 seconds to 1 hour in duration, for three key parameters.
Figure 9: Visualization of attention maps across various layers and attention heads in tasks of video understanding and video generation.
Figure 10: Sparsity-recall trade-off of various sparse patterns across different layers and attention heads of Qwen2.5-VL-7B-Instruct, evaluated on the VideoMME benchmark.
Figure 11: Sparsity-recall trade-off of various sparse patterns across different layers and attention heads of Wan2.1, evaluated on VBench.
Figure 12: The sparse pattern in Meta-Llama3-8B-Instruct evaluated on LongBench.
Figure 13: The sparse pattern in Qwen2.5-Omni-7B evaluated on OmniBench [ 23 ] .
Figure 14: Attention latency breakdown normalized to full attention across different context lengths on V-NIAH [ 56 ] . Markers (e.g., 64%) indicate the sparsity ratio.
Figure 15: Qualitative comparison of videos generated with different sparse attention methods. The input prompt is sampled from VBench: “In the vast desert, an aerial shot shows an oasis nestled among the dunes, featuring tall palm trees and exuding an air of serenity. Realistic, Natural lighting, Peaceful.”
Dense self-attention is the compute and quality bottleneck of long-video diffusion inference: cost grows quadratically with the sequence length, and beyond the training horizon the model converges to near-static output, that is, "frozen" repetitive video. State of the art approaches are either too costly, e.g., they require retraining, or fail to satisfy both performance and quality objectives in a scalable manner. To this end, we introduce Long Video Sparse Attention (LVSA), a training-free model-agnostic block-sparse attention for video diffusion transformers that combines a structured window pattern with rotating global anchors, thus removing the fixed-grid bias which causes long-range temporal artifacts. LVSA, combined with a FlashInfer kernel, reduces compute up to 3.17x on Wan 2.1 1.3B at a 6x horizon, 2.98x on Wan 2.1 14B at a 6x horizon, and 3.33x on HunyuanVideo 1.5 at a 1.5x horizon, compared to dense attention. Beyond reducing compute, LVSA enables HunyuanVideo 1.5 generation at a 2x horizon, which is otherwise out-of-memory on a single GPU. Moreover, LVSA provides speedups up to 2.41x compared to RIFLEx and 3.27x compared to UltraViCo on Wan 2.1 1.3B. To demonstrate applicability across diverse platforms, we apply LVSA on NPUs and achieve speedups up to 2.71x on Wan 2.2 A14B and 3.24x on Wan 2.1 1.3B compared to dense attention. To evaluate quality in a fair way, we introduce VQeval, a tool properly scoring loopy video failures, which instead are rewarded in state of the art evaluators like VBench-Long. LVSA is quality-neutral for generation at training horizon length and quality-positive at extended lengths.
Gael Glorian, Ioannis Lamprou, Zhen Zhang +2
Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.
Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1× end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.
Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1× end-to-end speedup and a 10.5× self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.
Shihao Han, Hao Yang, Xinting Hu +3
ByteDance Inc. · University of Science and Technology of China · The University of Hong Kong