ART: Attention Run-time Termination for Efficient Large Language Model Decoding

Chen Qiu; Guozhong Li; Cristian McGee; Aritra Dutta; Panos Kalnis

cs.CLApr 15, 2026

ART: Attention Run-time Termination for Efficient Large Language Model Decoding

Authors: Chen Qiu, Guozhong Li, Cristian McGee, Aritra Dutta, Panos Kalnis

Organizations: King Abdullah University of Science and Technology · University of Central Florida

Abstract

Long-context decoding in Large Language Models (LLMs) is constrained by the cost of accessing and processing the Key-Value (KV) cache. Despite evidence that attention outputs depend jointly on keys and values, most existing KV management methods rely on key-only pruning, since incorporating values incurs prohibitive overhead. In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. Rather than replacing KV selection, ART dynamically terminates redundant KV traversal on top of existing dense or sparse attention policies. We introduce a stability-based criterion that monitors both magnitude and directional changes of intermediate attention outputs and provideds a theoretical characterization of the resulting truncation error. Experiments on the LongBench and RULER Needle-in-a-Haystack tasks show that ART increases the generation throughput of existing KV-cache methods by up to 20%, without compromising the result quality.

Figures & tables

Figure 1: Attention score versus output contribution. Orange points denote high-contribution tokens missed by attention-score ranking.

Figure 2: Overview of the ART. As KV blocks are processed sequentially, ART monitors the intermediate attention output and triggers an early termination once the output stabilizes, skipping the remaining KV blocks during decoding.

Figure 3: ART integrated into the FlashAttention execution pipeline. FlashAttention overlaps DMA-based KV block prefetch from HBM with Tensor Core computation in a block-wise manner. ART performs a lightweight run-time check on the evolving attention output during computation. Once convergence is detected ( e.g. , at Step

i

), ART terminates the pipeline early by preventing further KV block prefetch and computation, reducing memory traffic and computation, with negligible impact on downstream generation quality.

Figure 4: Convergence of attention output with recent KV retention. The plot illustrates the relative

L_{2}

error between the attention output from the full KV cache and that from a truncated recent-KV window, as a function of the loaded fraction.

	Mistral-7B-Instruct-v0.3				Llama-3.1-70B-Instruct
	LB Score		TPOT $\times$		LB Score		TPOT $\times$
Method	Base	+ART	Base	+ART	Base	+ART	Base	+ART
Baseline	46.29	45.74	1.00	1.16	48.25	47.48	1.00	1.15
StrLLM(0.8)	40.29	40.20	1.08	1.24	42.31	41.80	1.13	1.27
StrLLM(0.2)	31.11	31.09	1.18	1.30	33.88	33.51	1.52	1.66
SnapKV(0.8)	43.92	43.12	1.07	1.17	46.49	45.54	1.08	1.18

Table 1: Average LongBench score and decoding TPOT speedup (TPOT

\times

) on Mistral-7B-Instruct-v0.3 and Llama-3.1-70B-Instruct. For each base method, we report the original result and the result after integrating ART.

Figure 5: Generation throughput (tokens/s) versus batch size for Full-KV (Baseline) and SnapKV, combined with ART. The left and right panels correspond to KV cache retention ratios of

0.8

and

0.2

, respectively. SnapKV+ART performs consistently better, especially for large batch size.

Method	4K		8K		16K		32K
Method	Acc.	TPOT $\times$	Acc.	TPOT $\times$	Acc.	TPOT $\times$	Acc.	TPOT $\times$
Baseline	99.95	1.00	99.88	1.00	99.73	1.00	99.34	1.00
+ ART	89.11	1.10	95.24	1.15	85.77	1.17	78.65	1.27
SnapKV	92.65	1.11	98.63	1.13	99.54	1.16	99.37	1.18
+ ART	92.73	1.11	98.73	1.15	98.93	1.21	95.68	1.36
StreamingLLM	80.13	1.12	82.72	1.15	81.42	1.20	77.78	1.21

Table 2: Accuracy (%) and decoding TPOT speedup (TPOT

\times

) of Qwen3-8B on RULER NIAH, For each base method, we report the original result and the result after integrating ART.

Method	QA	Sum	Fewshot	LB Score	$\Delta$ Score	Time (ms)	$\Delta$ Time (%)
ART (full KV)	35.58	26.40	63.04	45.74	–	0.633	–
w/o $d_{\mathrm{scale}}$	14.78	21.18	27.74	19.34	$-26.57$	0.135	$-78.7$
w/o $d_{\mathrm{direction}}$	35.32	26.25	62.64	43.87	$-2.04$	0.628	$-0.8$
w/o patience	33.58	25.96	58.11	41.54	$-4.37$	0.506	$-20.1$

Table 3: Ablation study of ART on LongBench. We report the average score of Single-Document QA and Multi-Document QA (QA), summarization (Sum), few-shot learning (Fewshot) and overall LongBench score; the average FlashAttention kernel time per call; and their changes relative to the full ART configuration.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Sequence length (layer 35, needle @ 50%)
Condition	$n$	Agree $\uparrow$	FP $\downarrow$	FN $\downarrow$	$\Delta$ blk
xx 1 024 tokens	800	100.00%	0.00%	0.00%	—
xx 2 048 tokens	800	100.00%	0.00%	0.00%	—
xx 4 096 tokens	800	100.00%	0.00%	0.00%	—
xx 8 192 tokens	800	99.88%	0.13%	0.00%	—
(b) Layer depth (4 096 tokens, needle @ 50%)

Appendix

Table 4: Consistency between the probe and the oracle on Qwen3-8B (36 layers, head_dim = 128). FP : probe stops before oracle, or probe stops while oracle never does (premature termination risk). FN : oracle stops before probe, or oracle stops while probe never does (conservative; no accuracy risk).

\boldsymbol{\Delta}

blk : mean

\pm

std block offset among mismatched stopping cases (i.e., both probe and oracle stop, but at different blocks)..

Method	Paper Notation / Term	Symbol	Value
StreamingLLM	Initial Tokens (Attention Sinks)	-	4
SnapKV	Observation Window	$L_{obs}$	32
	Pooling Kernel Size	-	5
PyramidKV	Instruction Tokens (Local Window)	$\alpha$	128
	Pooling Kernel Size	$\beta$	5

Appendix

Table 5: Hyperparameter configurations mapped to original paper notations.

Setting	Avg. FA kernel time (ms)
Baseline	0.78
patience= $\infty$	0.79

Appendix

Table 6: ART incurs negligible overhead when early termination is disabled. With

p=\infty

, the ART detector executes at every FlashAttention call without triggering termination, increasing kernel time by only 1.3%.

Figure 6: Sensitivity analysis of the scale parameter

\tau

. The blue solid line (left axis) tracks the LongBench score, while the red dashed line (right axis) shows the kernel execution time. The results highlight a trade-off:

\tau=10^{-5}

and

\phi=10^{-3}

offers an optimal balance between stability and speed, while larger values provide maximum acceleration with a slight performance cost.

Figure 7: Sensitivity analysis of the parameter

p

. The blue solid line (left axis) tracks the LongBench score, while the red dashed line (right axis) shows the kernel execution time.

Figure 8: Impact of context length on inference efficiency and performance. The dataset is categorized into three length intervals: 0-4k, 4-8k, and 8k+. The integration of ART significantly reduces kernel running time on long context length with minimal effect on the quality of the results.

Mistral-7B-Instruct-v0.3
Method	Single QA	Multi QA	Summ.	Few-shot	Synth.	Code	Avg.	$\boldsymbol{\Delta}$	TPOT speedup
Baseline	37.88	34.65	26.06	63.43	65.63	63.56	46.29	–	1.00 $\times$
+ ART	37.20	33.96	26.40	63.04	64.85	61.80	45.74	-0.55	1.16 $\times$
StrLLM(0.8)	28.34	27.61	25.81	62.27	47.21	64.16	40.29	–	1.08 $\times$
+ ART	28.05	27.34	25.83	62.42	47.51	63.52	40.20	-0.09	1.24 $\times$
StrLLM(0.2)	20.04	19.69	22.71	56.57	18.20	61.27	31.11	–	1.18 $\times$

Appendix

Table 7: Category-level LongBench score and decoding TPOT speedup comparison on Mistral-7B-Instruct-v0.3 and Llama-3.1-70B-Instruct. Each category score is averaged over its corresponding tasks. The Avg. column reports the original LongBench mean score over all 21 tasks,

\Delta

denotes the marginal score change compared with the corresponding base method.

Long-Context Language Model Inference

Long-context language model inference makes processing and generating over lengthy prompts faster and more memory-efficient.

See all 224 papers

CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
Aug 31, 2026Haoyun Jiang, Haolin Li, Jianwei Zhang +7Long-Context Language Model Inference LLM Inference Acceleration
1CMIC, Shanghai Jiao Tong University · 2Alibaba Group · 3Fudan University
Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
May 28, 2026Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir SinghKV Caching Long-Context Language Modeling
Indian Institute of Technology Guwahati, India
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3KV Caching Long-Context Language Model Inference
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China

Aug 31, 2026J/K move · Enter open · S save

CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

Haoyun Jiang, Haolin Li, Jianwei Zhang, Fei Huang+6

1CMIC, Shanghai Jiao Tong University · 2Alibaba Group · 3Fudan University

Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this work, we discover that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm. Inspired by this observation, we propose CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead, while preserving the majority of KV pairs in adaptive heads to ensure high accuracy. We show the unique characteristics of our algorithm and its extension with existing acceleration methods. Comprehensive evaluations on long-context benchmarks show that, while maintaining accuracy comparable to full attention, CateKV reduces memory usage by up to

2.72\times

and accelerates decoding by

2.18\times

in single-sample inputs, and boosts throughput by

3.96\times

in batch scenarios.

Abstract

Long-Context Language Model Inference

Explore similar work

CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference