VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling

Chen Guanzhong

cs.LGMar 3, 2026

VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling

Authors: Chen Guanzhong

Organizations: School of Computer Science and Technology, Tongji University

Abstract

The quadratic complexity of self-attention during the prefill phase impedes long-context inference in large language models. Existing sparse attention methods face a trade-off among context adaptivity, sampling overhead, and fine-tuning costs. We propose VSPrefill, a mechanism requiring lightweight training that uses the vertical-slash structural pattern in attention distributions. Our compact VSIndexer module predicts context-aware importance scores for vertical columns and slash diagonals from key-value representations augmented with RoPE. This approach constructs sparse masks with linear complexity without modifying the backbone parameters. During inference, an adaptive cumulative-threshold strategy allocates sparsity budgets per layer, while a fused kernel executes attention with on-the-fly index merging. Evaluated on Qwen3-4B-Instruct and LLaMA-3.1-8B-Instruct across the LongBench and RULER benchmarks, VSPrefill preserves 98.35% of the full attention accuracy while delivering a 4.95x average speedup at a context length of 128k. These results establish a new Pareto frontier in the trade-off between accuracy and efficiency.

Figures & tables

Figure 1: Overview of VSPrefill. The VSIndexer employs a shared-weight bilayer linear network that accepts concatenated key-value pairs as input and outputs vertical and slash attention scores, denoted as $\hat{A}_{v}$ and $\hat{A}_{s}$ . These scores are trained to approximate the ground-truth full attention weights aggregated along the corresponding directional patterns. During inference, $\hat{A}_{v}$ and $\hat{A}_{s}$ undergo top-k selection with a dynamic sparsity budget to construct the vertical-slash sparse attention mask. Implementation details, including the architecture of VSIndexer, RoPE, and sparse attention computation, are omitted for clarity.

Figure 2: Accuracy and Perplexity Trends Across Different Attention Recall Levels on HotPotQA dataset.

Panel of Figure 3 — Figure 3: Visualization of dynamic attention sparsity patterns. (a-c) Comparisons reveal high structural similarity within the same KV group (Intra-Group) contrasted with distinct topologies across different groups (Inter-Group). (d) Sparsity patterns evolve significantly as network depth increases. (e-f) The distribution of salient weights shifts in response to varying input prompts and model architectures.

Figure 4: Diagonal-aggregated attention heatmap for Layer 0 of Qwen3-4B-Instruct on LongBench. The x-axis denotes the relative diagonal offset ( $i-j$ ) where 0 corresponds to the main diagonal. The emergence of distinct vertical bands at distal offsets validates the existence of consistent slash patterns, characterized by strong correlations at fixed relative distances across multiple heads.

Model	Method	Length						Avg. Score	Avg. Speedup
		4k	8k	16k	32k	64k	128k	Avg. Score	Avg. Speedup
Qwen3-4B- Instruct	FlashAttn	77.99	79.40	80.23	78.96	84.00	77.41	79.67	–
	StrLLM	70.85	64.84	58.47	52.31	47.56	36.16	55.03	4.79
	FlexPre	74.32	76.62	76.87	76.87	79.44	74.44	76.43	1.83
	SeerAttn	77.10	78.24	79.92	77.16	81.43	64.45	76.38	1.67
	VSPrefill	77.46	77.27	80.00	78.29	83.47	75.15	78.61	1.91

Table 1: Performance evaluation on the RULER benchmark across context lengths ranging from 4k to 128k. Bold values indicate the highest score within each model group (excluding FlashAttn); underlined values indicate the second-highest score. Speedup is relative to full attention.

Model	Method	Qasper	MNews	Trec	2WikiQA	LCC	MF-en	GovRep	PsgRetr	PsgCnt	SamSum	RBench	HotPot	TriviQA	Avg.
Qwen3-4B- Instruct	FlashAttn	40.66	22.12	72.67	40.28	6.41	50.53	30.75	100.00	1.45	35.98	8.25	57.61	85.49	42.48
	StrLLM	31.62	22.22	73.00	26.57	6.07	33.76	30.60	38.17	2.89	36.64	7.13	35.83	84.29	32.98
	FlexPre	37.90	22.40	74.00	31.50	4.68	50.85	30.22	51.78	2.06	36.72	7.24	47.91	83.24	36.96
	SeerAttn	41.90	22.11	70.33	38.68	5.00	47.31	30.64	100.00	1.48	34.08	7.46	57.39	84.04	41.57
	VSPrefill	42.39	22.33	73.67	37.50	5.09	48.11	30.71	100.00	1.21	36.15	9.24	52.59	84.21	41.78
LLaMA-3.1- 8B-Instruct	FlashAttn	42.98	26.18	8.00	43.46	26.28	55.25	34.93	99.67	11.72	8.13	22.81	60.94	88.76	40.70

Table 2: Performance evaluation on the LongBench benchmark across 13 diverse tasks. Bold values indicate the highest score within each model group (excluding FlashAttn); underlined values indicate the second-highest score.

Panel of Figure 5 — Figure 5: Accuracy vs. Speedup across different Attention mechanism on context length ranging from 32k to 128k.

Method	Sparsity Rate
Method	50%	90%	95%	99%
Random	47.70	9.73	4.94	0.98
Sampling	66.33	57.36	46.57	27.78
VSPrefill	97.50	88.48	83.68	72.15

Table 3: Attention Recall (%) comparison across different sparsity rates.

Loss Function	Recall (%)
KL Divergence	92.65
MSE	85.00
MSLE	85.65
Cosine Similarity	89.88

Table 4: Ablation study on loss functions (70% sparsity).

Input Type	Recall (%)	Loss
Query (Q)	91.08	5.78
Key (K)	92.34	3.59
Value (V)	91.97	4.23
Query-Key (QK)	92.26	3.61
Key-Value (KV)	92.66	3.46

Table 5: Ablation study on VSIndexer input feature combinations.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Attention weights aggregated along the vertical direction across different heads on Qwen3-4B-Instruct using the LongBench dataset.

Panel of Figure 7 — Figure 7: Slash-aggregated attention weights under four configurations of query-key averaging before RoPE application: (a) no averaging, (b) averaging along the sequence dimension, (c) averaging along the feature dimension, and (d) averaging along both dimensions. In each case, $Q$ and $K$ are first averaged along the specified dimension(s), then element-wise replicated to restore their original shapes, followed by RoPE encoding.

Panel of Figure 8 — Figure 8: Dimension-wise histogram statistics of Query and Key vectors with Gaussian fits. The distributions display dimension-dependent means and variances, revealing heterogeneous statistical properties of attention projections.

Explore similar work

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving
Aug 20, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +2LLM Inference Acceleration Long-Context Language Model Inference
MAIS&NLPR, CASIA · UCAS · WeChat, Tencent
RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models
Sep 17, 2026Chuxu Song, Jiuqi Wei, Zhencan PengLong-Context Language Model Inference LLM Inference Acceleration
Department of Computer Science, Rutgers University · OceanBase, Ant Group
FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling
Mar 6, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +3Long-Context Language Model Inference LLM Inference Acceleration
MAIS&NLPR, CASIA · UCAS · WeChat, Tencent

Aug 20, 2026J/K move · Enter open · S save

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

Qihang Fan, Huaibo Huang, Zhiying Wu, Bingning Wang+1

MAIS&NLPR, CASIA · UCAS · WeChat, Tencent

LLM Inference Acceleration Long-Context Language Model Inference Block-Sparse Attention

Open paper AskPDF arXiv

Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. Our previous work, FlashPrefill, mitigates this cost through instantaneous pattern discovery and max-based dynamic thresholding; however, it remains an algorithmic prototype that is still distant from production deployment. In this paper, we present FlashPrefill V2, which evolves FlashPrefill from a prototype toward practical long-context serving along three dimensions. First, we introduce a mean correction term that effectively suppresses the approximation error, keeping performance degradation manageable even at extreme sparsity levels. Second, we redesign the sparse attention operator with PackGQA memory access, warp specialization, and pingpong pipelining, fully aligning with the latest FlashAttention-3/4 implementations and supporting FP8 inference to meet practical quantization requirements. Third, FlashPrefill V2 natively supports paged KV cache and continuous batching, allowing integration as an attention backend in modern inference frameworks such as SGLang. Extensive evaluations on NVIDIA H20 GPUs---among the most widely deployed inference accelerators---demonstrate that FlashPrefill V2 delivers up to 47.26x and 27.19x speedups over FlashAttention-2 at 128K context length under FP8 and BF16 precision, respectively, and, in FP8, still achieves a 30.49x speedup against an FA3/4-aligned dense baseline.

Abstract

Explore similar work

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models

FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling