VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference

Anmin Liu; Ruixuan Yang; Huiqiang Jiang; Bin Lin; Minmin Sun; Yong Li; Chen Zhang; Tao Xie

cs.CVMar 31, 2026

VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference

Authors: Anmin Liu, Ruixuan Yang, Huiqiang Jiang, Bin Lin, Minmin Sun, Yong Li, Chen Zhang, Tao Xie

Organizations: SCS, Peking University, Beijing, China · Key Lab of HCST (PKU), MOE, Beijing, China · Fudan University, Shanghai, China · Alibaba Group, China · Shanghai Jiao Tong University, Shanghai, China

Abstract

Long-context video understanding and generation pose a significant computational challenge for Transformer-based video models due to the quadratic complexity of self-attention. While existing sparse attention methods employ coarse-grained patterns to improve efficiency, they typically incur redundant computation and suboptimal performance. To address this issue, in this paper, we propose VecAttention, a novel framework of vector-wise sparse attention that achieves superior accuracy-efficiency trade-offs for video models. We observe that video attention maps exhibit a strong vertical-vector sparse pattern, and further demonstrate that this vertical-vector pattern offers consistently better accuracy-sparsity trade-offs compared with existing coarse-grained sparse patterns. Based on this observation, VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection that minimizes memory access overhead and an optimized kernel of vector sparse attention. Comprehensive evaluations on video understanding (VideoMME, LongVideoBench, and VCRBench) and generation (VBench) tasks show that VecAttention delivers a 2.65 $\times$ speedup over full attention and a 1.83 $\times$ speedup over state-of-the-art sparse attention methods, with comparable accuracy to full attention. Our code is available at https://github.com/anminliu/VecAttention.

Figures & tables

Panel of Figure 1 — Figure 1: (a–b) Under the same sparsity budget, vertical-vector (V-Vec) achieves higher recall and better approximates the oracle pattern than horizontal-vector (H-Vec) and other coarse line/block patterns, for both VLMs (video understanding) and DiTs (video generation). Markers such as (68.9%, 0.95) denote (sparsity, recall), with sparsity in percentage. (c) On VideoMME at matched full-accuracy settings, VecAttention attains higher effective sparsity and faster attention computation, with low important-region selection overhead, compared to existing coarse-grained methods.

Panel of Figure 2 — Figure 2: Attention maps visualized on different layers across tasks of video understanding and video generation.

Panel of Figure 3 — Figure 3: Sparsity–recall tradeoff of different sparse patterns across long-context video understanding (left three columns) and video generation (right column) tasks. The top row shows results on the causal InternVL-3.5-8B VLM and non-causal HunyuanVideo DiT, while the bottom row presents the causal Qwen2.5-VL-7B VLM and non-causal Wan2.1-T2V-14B DiT. The vertical-vector pattern consistently yields higher recall under the same sparsity than horizontal-vector pattern and coarse line/block pattern, closely tracking the oracle pattern across different datasets and models.

Figure 4: The overview of VecAttention, which can be divided into two stages: (1) Important-Vector Selection via TilingSelect and minS Filter; (2) Vector Sparse Attention on selected vectors.

Figure 5: Recall heatmaps of different filter strategies under the same sparsity. The minS filter strategy achieves comparable recall under the same sparsity level. Markers such as (75.0%, 0.96) denote (sparsity, recall), with sparsity in percentage.

Figure 6: Time and memory-traffic breakdown at context length

N{=}64\mathrm{K}

and sparsity

0.9

. Compared with a naive materialize-then-filter baseline (e.g., topP), TilingSelect with minS avoids writing estimated attention map to HBM, cutting memory traffic from 18.3 GB to 1.8 GB and reducing selection latency by 2.42

\times

, yielding an overall 9.12

\times

speedup.

InternVL-3.5-8B #Frames: 64 #Tokens: 17K
Model	Avg. Sp.	Video MME	Long Video	VCR- Bench	Avg. Acc.
Full Atten.	0.0	65.7	59.4	32.9	52.7
FlexPref.	76.5	52.3	59.0	30.0	47.1
XAtten.	78.1	56.0	59.9	32.5	49.5
AnchorAtten.	78.6	57.4	59.4	31.3	49.4
VecAtten.	78.6	60.6	59.0	33.8	51.1

Table 1: Performance of different models and different methods on video understanding tasks. All values are percentages (i.e., multiplied by 100) without the % symbol.

Wan 2.1-T2V-14B 720P, #Tokens 76K
Model	Sparsity(%) $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
XAttention	54.6	19.7	0.658	0.348
SVG	52.2	18.7	0.639	0.381
VecAttention	52.3	19.7	0.668	0.339
HunyuanVideo-T2V-13B 720P, #Tokens 119K
XAttention	60.8	21.2	0.734	0.348

Table 2: Performance of different models and methods on video generation tasks, with 50 inference steps and a 6% warm-up ratio.

Panel of Figure 7 — Figure 7: Microbenchmarks of efficiency-accuracy trade-offs across different sparse attention methods on VideoMME. VecAttention preserves accuracy at up to 93% sparsity and translates it into practical gains, which are 2.65 $\times$ acceleration for attention and 1.17 $\times$ end-to-end TTFT.

Panel of Figure 8 — Figure 8: Ablation study on VideoMME, which contains diverse video inputs ranging from 11 seconds to 1 hour in duration, for three key parameters.

Panel of Figure 9 — Figure 9: Visualization of attention maps across various layers and attention heads in tasks of video understanding and video generation.

Figure 10: Sparsity-recall trade-off of various sparse patterns across different layers and attention heads of Qwen2.5-VL-7B-Instruct, evaluated on the VideoMME benchmark.

Figure 11: Sparsity-recall trade-off of various sparse patterns across different layers and attention heads of Wan2.1, evaluated on VBench.

Panel of Figure 12 — Figure 12: The sparse pattern in Meta-Llama3-8B-Instruct evaluated on LongBench.

Panel of Figure 13 — Figure 13: The sparse pattern in Qwen2.5-Omni-7B evaluated on OmniBench [ 23 ] .

Figure 14: Attention latency breakdown normalized to full attention across different context lengths on V-NIAH [ 56 ] . Markers (e.g., 64%) indicate the sparsity ratio.

Panel of Figure 15 — Figure 15: Qualitative comparison of videos generated with different sparse attention methods. The input prompt is sampled from VBench: “In the vast desert, an aerial shot shows an oasis nestled among the dunes, featuring tall palm trees and exuding an air of serenity. Realistic, Natural lighting, Peaceful.”

Explore similar work

LVSA: Training-Free Sparse Attention for Long Video Diffusion
May 29, 2026Gael Glorian, Ioannis Lamprou, Zhen Zhang +2Video Diffusion Models Long-Horizon Video Generation
Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Veda: Scalable Video Diffusion via Distilled Sparse Attention
May 28, 2026Shihao Han, Hao Yang, Xinting Hu +3Video Diffusion Models Diffusion Transformer
ByteDance Inc. · University of Science and Technology of China · The University of Hong Kong

May 29, 2026J/K move · Enter open · S save

LVSA: Training-Free Sparse Attention for Long Video Diffusion

Gael Glorian, Ioannis Lamprou, Zhen Zhang, Yujie Yuan+1

Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.

Video Diffusion Models Long-Horizon Video Generation Sparse Attention Diffusion Model Inference Acceleration Temporal Consistency in Video Generation Block-Sparse Attention

Open paper AskPDF arXiv

Dense self-attention is the compute and quality bottleneck of long-video diffusion inference: cost grows quadratically with the sequence length, and beyond the training horizon the model converges to near-static output, that is, "frozen" repetitive video. State of the art approaches are either too costly, e.g., they require retraining, or fail to satisfy both performance and quality objectives in a scalable manner. To this end, we introduce Long Video Sparse Attention (LVSA), a training-free model-agnostic block-sparse attention for video diffusion transformers that combines a structured window pattern with rotating global anchors, thus removing the fixed-grid bias which causes long-range temporal artifacts. LVSA, combined with a FlashInfer kernel, reduces compute up to 3.17x on Wan 2.1 1.3B at a 6x horizon, 2.98x on Wan 2.1 14B at a 6x horizon, and 3.33x on HunyuanVideo 1.5 at a 1.5x horizon, compared to dense attention. Beyond reducing compute, LVSA enables HunyuanVideo 1.5 generation at a 2x horizon, which is otherwise out-of-memory on a single GPU. Moreover, LVSA provides speedups up to 2.41x compared to RIFLEx and 3.27x compared to UltraViCo on Wan 2.1 1.3B. To demonstrate applicability across diverse platforms, we apply LVSA on NPUs and achieve speedups up to 2.71x on Wan 2.2 A14B and 3.24x on Wan 2.1 1.3B compared to dense attention. To evaluate quality in a fair way, we introduce VQeval, a tool properly scoring loopy video failures, which instead are rewarded in state of the art evaluators like VBench-Long. LVSA is quality-neutral for generation at training horizon length and quality-positive at extended lengths.

Abstract

Explore similar work

LVSA: Training-Free Sparse Attention for Long Video Diffusion

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Veda: Scalable Video Diffusion via Distilled Sparse Attention