ATTUNER: Recomputation-Free Agentic Context KV Reuse

Xinghao Chen; Junnan Dong; Cai Ke; Chak Tou Leong; Haocheng Sun; Keyu Chen; Siyu An; Ruizhi Qiao

cs.CLSep 29, 2026

ATTUNER: Recomputation-Free Agentic Context KV Reuse

Authors: Xinghao Chen, Junnan Dong, Cai Ke, Chak Tou Leong, Haocheng Sun, Keyu Chen, Siyu An, Ruizhi Qiao,

Organizations: EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Tencent Youtu Lab

Abstract

Large language model (LLM) agents repeatedly load reusable content, such as skills, documents, and memory entries, into the current context. Re-encoding this content for every request wastes computation. Position-independent caching (PIC) alleviates this by encoding each artifact independently and reusing its key-value (KV) states at arbitrary positions, but it incurs a quality loss relative to full-context prefill. Existing methods repair this loss by restoring global position IDs or recomputing selected tokens. In this work, we isolate the source of the loss, finding that the positional mismatch has minor effect, and independently cached artifacts retain faithful representations: reading a provided artifact stays largely accurate, and performance degrades only when the model must select among multiple artifacts. Moreover, replacing PIC's attention scores with full-prefill scores recovers performance with the cached KV unchanged, localizing the failure to the attention rather than KV recomputation. Motivated by this, we propose Attuner, a query-side adaptation method that learns to read a frozen artifact cache. Attuner inserts low-rank adapters into the query projections and is trained by distilling full-prefill distribution into the student. It trains fewer than 0.05% of the model parameters and, at inference, requires neither cache recomputation nor a full-context reference. On Qwen3-4B and Qwen3-8B across seven benchmarks covering skills, documents, memory, and code, Attuner substantially outperforms prior PIC baselines in both in-domain and out-of-domain settings, matches full-context prefill quality while providing up to $3.73\times$ speedup.

Figures & tables

Figure 1: Three approaches to reusable artifact caching. (a) Inference-time adaptation repairs cached states or adjusts attention during inference. (b) Training-based adaptation modifies cache construction through learned encoders or linking tokens. (c) Attuner adapts online queries while preserving independently constructed artifact KV, learning to read the cache without rewriting it.

Figure 2: Task quality and TTFT speedup on Qwen3-4B. (a) Increasing distractors while retaining all relevant skills or supporting documents. (b) Supplying only task-annotated relevant artifacts, grouped into single- and multi-artifact settings.

Figure 3: Routing recovery with unchanged artifact KV. Cosine similarities for one MuSiQue example: Full vs. PIC for (a) Full–PIC similarity of cached values across artifact tokens, (b) Full–PIC similarity of attention-weight vectors overartifact tokens, and (c–d) Similarity of per-head attention outputs to Full, under PIC and the routing oracle. Additional case studies appear in Appendix .

Figure 4: Inference workflow of Attuner . Artifact KV is prepared once and reused unchanged. Warm-cache TTFT includes KV loading and assembly, online prefill, and first-token selection, excluding offline cache construction.

Figure 5: Training Attuner . A frozen Full-context teacher supervises the PIC student’s output distributions under identical inputs and response prefixes. Only query adapters are updated; the backbone and cached artifact KV remain frozen.

\rectanglecolor black!66-26-20 \rectanglecolor black!611-211-20 \rectanglecolor black!617-217-20 \rectanglecolor black!622-222-20 \rectanglecolor green!1014-214-20 \rectanglecolor green!1025-225-20 \Block 3-1		In-domain										Out-of-domain
	Method	BCB		MSQ		PMem		Quality		Efficiency		TQA		CHP		Quality		Efficiency
		Score $\uparrow$	TTFT $\downarrow$	Score $\uparrow$	TTFT $\downarrow$	Score $\uparrow$	TTFT $\downarrow$	Avg $\uparrow$	$\Delta$	Geo. $\downarrow$	Speedup $\uparrow$	Score $\uparrow$	TTFT $\downarrow$	Score $\uparrow$	TTFT $\downarrow$	Avg $\uparrow$	$\Delta$	Geo. $\downarrow$	Speedup $\uparrow$
\Block 11-1 Qwen3-4B	Full	44.53	420.77	61.01	39.06	53.28	62.38	52.94	–	100.83	$1.00\times$	71.22	89.08	69.06	139.01	70.14	–	111.28	$1.00\times$
	PIC	39.58	99.60	48.69	35.51	50.34	47.56	46.20	$\downarrow$ 6.74	55.20	$1.83\times$	66.80	38.14	64.13	46.70	65.47	$\downarrow$ 4.68	42.20	$2.64\times$
	Training-free
	APE	39.58	114.59	41.02	39.17	43.72	51.92	41.44	$\downarrow$ 11.50	61.54	$1.64\times$	59.57	42.88	60.54	49.83	60.06	$\downarrow$ 10.09	46.22	$2.41\times$

Table 2: Quality and efficiency on training-domain and held-out tasks. BCB, MSQ, PMem, TQA, and CHP denote BigCodeBench, MuSiQue, PersonaMem, TheoremQA, and CHAMP, respectively. Avg is the arithmetic mean of scores within each domain. Geo. denotes the geometric mean of TTFT within each domain. Full and PIC are the references and excluded from ranking. The best and second-best results among the remaining methods are bold and underlined .

Figure 7: Effect of training data size on Qwen3-4B for 1 epoch. Zero examples denotes PIC without adaptation;

Figure 8: Attention-output alignment before and after training. Cosine similarity of

O

to Full on Qwen3-4B. The first two panels show online prompt positions; the last two show teacher-forced answer positions. Each pair presents MuSiQue followed by BigCodeBench, with 128 examples.

(a) Layer-range comparison on MuSiQue
Method	Layers	EM	F1	$\Delta$ F1
Full	–	46.09	60.06	–
PIC	–	34.38	49.00	–
PIC + Attuner	16–33	50.78	65.14	$+16.13$
PIC + Attuner	18–35	50.00	63.34	$+14.34$
(b) Full versus PIC adaptation

Table 3: Adapter controls on Qwen3-4B. (a) Layer-range comparison on 128 MuSiQue test examples after training on 1,024 examples (b) The same L0–35 adapter under Full and PIC; The two panels use different training and evaluation sets.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

1 Dataset	Artifact unit	Train	Eval.	Artifacts	Metric
TheoremQA	Skill	0	747	1	Accuracy
LogicBench	Skill	0	760	1	Accuracy
MedCalc-Bench	Skill	0	1,100	1	Accuracy
CHAMP	Skill	0	223	1–5	Accuracy
BigCodeBench	Skill	4,000	667	2–5	pass@1
MuSiQue	GT document	4,000	2,417	2–4	F1 / EM

Appendix

Table 4: Main evaluation sets. Artifact counts are per request. Training counts refer to the shared 12K mixture; BigCodeBench training examples are synthetic. MuSiQue uses its official development set for held-out evaluation.

1 Operation	Timed?
Model loading, tokenization, artifact-boundary identification	No
Offline artifact KV construction	No
CPU RAM cache lookup and GPU transfer	Yes
KV assembly and online computation, including repair	Yes
Greedy first-token selection	Yes
Subsequent generation, text decoding, and scoring	No

Appendix

Table 5: Warm-cache TTFT boundary. Reusable KV is already resident in CPU memory before timing begins.

KV-Cache Reuse

Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.

See all 73 papers

KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Apr 14, 2026Chuangtao Chen, Grace Li Zhang, Xunzhao Yin +3KV Caching KV-Cache Reuse
Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China +1
RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse
Sep 27, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +5Retrieval-Augmented Generation KV-Cache Management
Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom (TeleAI) · State University of New York at Buffalo
CacheReforge: Bounded Recovery for Stale KV Caches under Evolving Adapters
Sep 25, 2026Yuhang Cao, Yanzhou Mu, Chunrong Fang +1KV-Cache Management KV-Cache Reuse
Nanjing University

Apr 14, 2026J/K move · Enter open · S save

KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs

Chuangtao Chen, Grace Li Zhang, Xunzhao Yin, Cheng Zhuo+2

Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China+1

KV Caching KV-Cache Reuse LLM Inference Acceleration

Open paper AskPDF arXiv

Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable ``packets'' wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen2.5 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline.

Abstract

KV-Cache Reuse

Explore similar work

KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs

RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse

CacheReforge: Bounded Recovery for Stale KV Caches under Evolving Adapters