AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

Yuxuan Hu; Jianchao Tan; Jiaqi Zhang; Wen Zan; Pingwei Sun; Yifan Lu; Yerui Sun; Yuchen Xie

cs.CLApr 9, 2026

AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

Authors: Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie,

Organizations: School of Information, Renmin University of China, Beijing, China · Meituan, Beijing, China

Abstract

Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse attention offers superior accuracy, its indexing overhead is costly; block-level methods improve efficiency but sacrifice precision. We propose AsyncTLS, a hierarchical sparse attention system that combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency, coupled with an asynchronous offloading engine that overlaps KV cache transfers with computation via temporal locality exploitation. Evaluated on Qwen3 and GLM-4.7-Flash across GQA, and MLA architectures, AsyncTLS achieves accuracy comparable to full attention while delivering 1.2x - 10.0x operator speedups and 1.3x - 4.7x end-to-end throughput improvements on 48k - 96k contexts.

Figures & tables

Figure 1: Illustration of Two-Level Sparse Attention. We construct hierarchical indices for KV blocks at two granularities (Block-level and Token-level). The decoding process first selects Top- $K_{b}$ blocks via block-level indexing, then selects Top- $K_{t}$ tokens within these blocks via token-level indexing for sparse attention computation.

Figure 2: Workflow of AsyncTLS. During prefilling, we first construct hierarchical indices for the input sequence. Based on the query representation of the final token, we retain critical KV pairs in the GPU resident cache and offload the remaining KV cache to the host (CPU) memory. During decoding, we perform sparse attention by retrieving KV pairs from the resident cache using token-level indexing. Concurrently, we asynchronously prefetch additional KV blocks from host memory via block-level indices to update the resident cache.

		Single-Doc QA			Multi-Doc QA			Summarization			Few-shot	Synthetic	Code		Avg.
		NQA	QQA	MFQ	HQA	2WM	Mus	GvR	QMS	MNs	TQA	PRetr	LCC	RBP
Qwen3-8B (32k)	Full	25.61	44.17	53.40	53.48	38.29	32.14	33.17	23.53	24.93	90.71	100.0	67.56	64.92	50.14
	Quest	20.64	40.13	51.01	45.74	38.46	27.25	32.13	22.21	24.95	87.55	98.5	66.86	61.95	47.49
	DS	22.50	44.76	54.05	54.22	36.26	33.13	33.51	23.47	24.65	90.06	100.0	68.85	65.94	50.10
	AsyncTLS	24.77	44.35	52.67	53.59	38.02	31.1	33.58	23.67	24.72	89.88	99.5	67.67	65.17	49.90
Qwen3-14B (32k)	Full	27.77	44.58	49.53	60.93	48.9	36.02	33.29	23.59	24.89	92.25	100.0	69.75	66.63	52.16

Table 1: Experiment results of AsyncTLS and baseline methods on LongBench.

		S1	S2	MK1	MK2	MQ	MV	QA-1	QA-2	VT	FWE	Avg.
Qwen3-8B	Full	100.0	100.0	99.60	98.20	99.75	99.60	49.00	60.90	99.80	97.67	90.45
	Quest	100.0	30.20	28.00	4.00	11.25	5.40	32.60	39.35	50.48	87.87	38.92
	DS	100.0	100.0	99.60	98.20	99.65	99.75	49.20	62.92	99.76	95.27	90.44
	AsyncTLS	100.0	100.0	99.80	98.60	99.70	99.45	52.20	59.62	99.80	96.27	90.54
Qwen3-14B	Full	100.0	100.0	99.80	99.60	99.95	100.0	57.80	66.60	100.0	38.67	86.24
	Quest	100.0	34.60	41.00	5.00	17.85	10.20	35.60	44.37	40.24	50.00	37.89

Table 2: Experiment results of AsyncTLS and baseline methods on RULER for Qwen3 models.

	MQ	MV	QA-1	QA-2	VT	FWE	Avg.
Full	99.55	99.75	53.60	66.25	100.0	95.20	85.73
Quest	57.45	42.25	30.40	45.80	60.84	83.20	53.32
DS	92.90	99.75	49.00	59.40	100.0	93.20	82.38
AsyncTLS	95.10	98.80	48.20	59.50	99.88	87.53	81.50

Table 3: Experiment results of AsyncTLS and baseline methods on RULER for GLM-4.7-Flash.

Panel of Figure 3 — Figure 3: Performance of AsyncTLS and baseline methods under various token budgets on Qwen3-14B.

Panel of Figure 4 — Figure 4: Latency comparison of full attention (FA), token-level sparse attention (DS), block-level sparse attention (Quest), and two-level sparse attention (AsyncTLS) kernels across varying batch sizes and sequence lengths.

Panel of Figure 5 — Figure 5: End-to-end latency comparison of Qwen3-8B and GLM4.7-Flash with different attention methods across varying sequence lengths.

Panel of Figure 6 — Figure 6: Throughput comparison of Qwen3-8B and GLM4.7-Flash with full attention (FA) and AsyncTLS across varying sequence lengths.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
Jun 4, 2026Yutao Sun, Yanqi Zhang, Li Dong +2Long-Context Language Model Inference KV-Cache Management
Microsoft Research · Tsinghua University
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3KV Caching Long-Context Language Model Inference
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel

Jun 4, 2026J/K move · Enter open · S save

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang+1

Microsoft Research · Tsinghua University

Long-Context Language Model Inference KV-Cache Management LLM Inference Acceleration Sparse Attention Cross-Layer Attention

Open paper AskPDF arXiv

Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive. In this work, we propose cross-layer sparse attention (CLSA), which is built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-level top-k selection once and reuses the resulting index across layers, thereby preserving the fine-grained selectivity of token sparse attention while amortizing the routing overhead. The resulting architecture improves all major inference bottlenecks jointly, including pre-filling, KV-cache storage, and long-context decoding. Experiments across short-context and long-context benchmarks show that CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context. These results suggest a more complete architectural solution for long-context LLMs that jointly advances model quality and inference efficiency.

Abstract

LLM Inference Acceleration

Explore similar work

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

Block Sparse Flash Attention