Large language model (LLM) agents repeatedly load reusable content, such as skills, documents, and memory entries, into the current context. Re-encoding this content for every request wastes computation. Position-independent caching (PIC) alleviates this by encoding each artifact independently and reusing its key-value (KV) states at arbitrary positions, but it incurs a quality loss relative to full-context prefill. Existing methods repair this loss by restoring global position IDs or recomputing selected tokens. In this work, we isolate the source of the loss, finding that the positional mismatch has minor effect, and independently cached artifacts retain faithful representations: reading a provided artifact stays largely accurate, and performance degrades only when the model must select among multiple artifacts. Moreover, replacing PIC's attention scores with full-prefill scores recovers performance with the cached KV unchanged, localizing the failure to the attention rather than KV recomputation. Motivated by this, we propose Attuner, a query-side adaptation method that learns to read a frozen artifact cache. Attuner inserts low-rank adapters into the query projections and is trained by distilling full-prefill distribution into the student. It trains fewer than 0.05% of the model parameters and, at inference, requires neither cache recomputation nor a full-context reference. On Qwen3-4B and Qwen3-8B across seven benchmarks covering skills, documents, memory, and code, Attuner substantially outperforms prior PIC baselines in both in-domain and out-of-domain settings, matches full-context prefill quality while providing up to 3.73× speedup.
Figures & tables
Figure 1: Three approaches to reusable artifact caching. (a) Inference-time adaptation repairs cached states or adjusts attention during inference. (b) Training-based adaptation modifies cache construction through learned encoders or linking tokens. (c) Attuner adapts online queries while preserving independently constructed artifact KV, learning to read the cache without rewriting it.
Figure 2: Task quality and TTFT speedup on Qwen3-4B. (a) Increasing distractors while retaining all relevant skills or supporting documents. (b) Supplying only task-annotated relevant artifacts, grouped into single- and multi-artifact settings.
Figure 3: Routing recovery with unchanged artifact KV. Cosine similarities for one MuSiQue example: Full vs. PIC for (a) Full–PIC similarity of cached values across artifact tokens, (b) Full–PIC similarity of attention-weight vectors overartifact tokens, and (c–d) Similarity of per-head attention outputs to Full, under PIC and the routing oracle. Additional case studies appear in Appendix .
Figure 4: Inference workflow of Attuner . Artifact KV is prepared once and reused unchanged. Warm-cache TTFT includes KV loading and assembly, online prefill, and first-token selection, excluding offline cache construction.
Figure 5: Training Attuner . A frozen Full-context teacher supervises the PIC student’s output distributions under identical inputs and response prefixes. Only query adapters are updated; the backbone and cached artifact KV remain frozen.
Table 2: Quality and efficiency on training-domain and held-out tasks. BCB, MSQ, PMem, TQA, and CHP denote BigCodeBench, MuSiQue, PersonaMem, TheoremQA, and CHAMP, respectively. Avg is the arithmetic mean of scores within each domain. Geo. denotes the geometric mean of TTFT within each domain. Full and PIC are the references and excluded from ranking. The best and second-best results among the remaining methods are bold and underlined .
Figure 7: Effect of training data size on Qwen3-4B for 1 epoch. Zero examples denotes PIC without adaptation;
Figure 8: Attention-output alignment before and after training. Cosine similarity of O to Full on Qwen3-4B. The first two panels show online prompt positions; the last two show teacher-forced answer positions. Each pair presents MuSiQue followed by BigCodeBench, with 128 examples.
(a) Layer-range comparison on MuSiQue
Method
Layers
EM
F1
Δ F1
Full
–
46.09
60.06
–
PIC
–
34.38
49.00
–
PIC + Attuner
16–33
50.78
65.14
+16.13
PIC + Attuner
18–35
50.00
63.34
+14.34
(b) Full versus PIC adaptation
Table 3: Adapter controls on Qwen3-4B. (a) Layer-range comparison on 128 MuSiQue test examples after training on 1,024 examples (b) The same L0–35 adapter under Full and PIC; The two panels use different training and evaluation sets.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
1 Dataset
Artifact unit
Train
Eval.
Artifacts
Metric
TheoremQA
Skill
0
747
1
Accuracy
LogicBench
Skill
0
760
1
Accuracy
MedCalc-Bench
Skill
0
1,100
1
Accuracy
CHAMP
Skill
0
223
1–5
Accuracy
BigCodeBench
Skill
4,000
667
2–5
pass@1
MuSiQue
GT document
4,000
2,417
2–4
F1 / EM
Appendix
Table 4: Main evaluation sets. Artifact counts are per request. Training counts refer to the shared 12K mixture; BigCodeBench training examples are synthetic. MuSiQue uses its official development set for held-out evaluation.
1 Operation
Timed?
Model loading, tokenization, artifact-boundary identification
No
Offline artifact KV construction
No
CPU RAM cache lookup and GPU transfer
Yes
KV assembly and online computation, including repair
Yes
Greedy first-token selection
Yes
Subsequent generation, text decoding, and scoring
No
Appendix
Table 5: Warm-cache TTFT boundary. Reusable KV is already resident in CPU memory before timing begins.
Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable ``packets'' wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen2.5 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline.
Chuangtao Chen, Grace Li Zhang, Xunzhao Yin +3
Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China +1
Cross-request KV caching reduces the prefill cost of Retrieval-Augmented Generation (RAG), but conventional prefix caching severely limits cache reuse across requests. Position-Independent Caching (PIC) removes this constraint by reusing independent chunks, but their KV states miss cross-chunk interactions. Existing methods selectively recompute token states to recover these missing interactions, but primarily allocate the recomputation budget to selecting which states to recompute, while fixing the recomputation context to the full causal prefix. We introduce RelaxKV, which formulates selective cache repair as a joint allocation problem over repair targets and recomputation context. Guided by the user query, RelaxKV identifies layer-specific repair targets and restricts their recomputation to a query-relevant context, reducing attention computation. Across four decoder models, RelaxKV at a 15% anchor ratio improves aggregate LongBench performance over ProphetKV on all models. On Qwen3-14B, RelaxKV provides a stronger quality-TTFT trade-off than ProphetKV across a 5%-30% anchor-ratio sweep, and achieves the best selective results on RULER-MV and LV-Eval at 16K and 32K context lengths. Controlled ablations further demonstrate the importance of recomputation context selection.
Ruoling Qi, Yirui Liu, Xuaner Wu +5
Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom (TeleAI) · State University of New York at Buffalo
Large language models rely on KV caching to reduce repeated prefill computation in long context and interactive applications. As lightweight adapters evolve, cached states reflect earlier versions, so stale reuse distorts current model outputs, while complete affected suffix recomputation restores fidelity at substantial cost. We seek minimal recomputation that recovers current adapter behavior. Existing systems track token, context, or stable adapter identity, but neither represent caches from earlier adapter versions nor distinguish update propagation from the recomputation required for behavioral recovery. To address these gaps, we introduce CacheReforge, which represents stale KV caches as layerwise mixed-version objects. It combines per-layer adapter anchors, calibrated sensitivity, accumulated drift, and executable restart boundaries to select direct reuse, bounded recomputation, or complete affected-suffix recovery. We distinguish dependency depth from the functional recomputation horizon and use cumulative tail influence to characterize when bounded recovery preserves current-model behavior. We evaluate CacheReforge on Qwen2.5-1.5B and Qwen2.5-7B with continual LoRA updates, including 16K HotpotQA and 2WikiMQA workloads. CacheReforge reduces mean KL divergence by 92.4% relative to stale reuse, while recomputing only 5.44% of layers and reducing cache-maintenance time by 93.2% relative to fresh full prefill. These results show that version-aware recovery preserves model fidelity and most KV caching gains.