Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.
Figures & tables
Figure 1: Full-context inference, context engineering, and cache engineering. Full-context inference reuses all historical KV. Context engineering removes irrelevant text and recomputes the retained history. Cache engineering keeps the text history but drops selected KV entries, reusing retained states without recomputation. Nested color blocks illustrate how retained KV can still carry information from dropped history.
Figure 2
Figure 5: Cache operation trajectories and history-dependent KV states. (a) ① appends C to A and B, then drops A’s KV; ② drops A’s KV before appending C; ③ starts from B alone and appends C. All three retain B and C. (b) B’s KV is identical in ① and ②, both conditioned on A, but differs in ③, where B has no prefix. C’s KV differs in all three: ① is computed with A and B; ② and ③ cannot attend to A but use different B caches. (c) The extended radix tree shares the A–B prefix of ① and ②, then branches according to drop timing, distinguishing the cached states used to generate D.
Figure 4
Figure 8: repos in the KV pool and radix tree. (a) Two requests share A and B; Request 2 drops A before appending C. (b) B and C initially occupy positions 1 and 2. (c) repos creates copies at positions 0 and 1 by re-encoding K and copying V. The original B remains shared with Request 1, while the original C becomes evictable. The tree branches at B to distinguish positional versions and records a reposition node after C. Labels p denote position p .
Figure 9: Task performance and serving throughput with GPT-OSS-120B on SWE-bench Verified. (a) Online evaluation on the first 100 tasks: mean cumulative pass rate versus GPU hours across three runs per method, with shaded regions indicating ± 1 standard deviation. (b) Cumulative pass rate versus cumulative logical input tokens over all 500 tasks. (c) Serving throughput versus concurrency.
Method
BCP
GAIA-Text
SWE-Verified
xbench-DS
Average
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Score
Cost
MiniMax-M2.7 (230B total, 10B active)
Standard
58.9
5.2
35.0
2.4
75.2
7.0
40.5
5.5
52.4
5.0
Text Engineering
62.3
5.3
36.9
2.7
72.0
9.4
40.0
5.2
52.8
5.7
AKV
58.9
4.8
40.8
2.6
74.4
5.5
40.5
4.7
53.7
4.4
GPT-OSS-120B (5.1B active)
Table 1: Benchmark scores (pass rate, %) and costs (cents) by model and management method. Bold indicates the highest score and lowest cost within each model, including ties.
C
Method
Request latency
TTFT
TPOT
Throughput
mean
P95
mean
P95
mean
P95
Input
Output
8
Standard
12.16
43.57
3.18
8.42
26.57
31.38
25200
217
Text Engineering
12.84
43.57
4.94
22.99
23.23
25.68
17313
228
AKV
7.79
25.93
1.87
5.99
23.50
29.42
39227
245
10
Standard
12.82
45.03
3.93
9.76
26.47
30.83
30108
254
Text Engineering
15.17
50.27
7.35
34.07
22.57
25.09
18533
246
Table 2: Serving performance (latency, s; TTFT, s; TPOT, ms/token; throughput, tokens/s) of GPT-OSS-120B on Top-80 SWE-Verified across concurrency ( C ) levels.
Table 9
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Restore with segment-batch attention. The request in (a) produces the four attention segments in (b). Numbers denote query/key positions; colored cells indicate allowed attention.
Figure 11: Encoding drop and reposition histories in the extended radix tree. (a) Three message trajectories interleave token appends, KV drops, and reposition operations. Although their retained tokens overlap, applying the operations at different points produces different KV states. (b) AKV records each operation in the radix tree, distinguishing cache versions by position and operation history. Numbers beside token nodes denote stored position encoding indices; RX denotes repos after X .
Setting
Value
Model and weight precision
GPT-OSS-120B, BF16
Hardware
Eight NVIDIA A800 GPUs, 80 GB each
Parallelism
Prefill: TP4; decode: TP4
KV transfer
Mooncake over intra-node NVLink
Attention backend
Triton
MoE backend
triton_kernel
Appendix
Table 5: Serving configuration.
Model
Cache-miss
Cache-hit
Output
MiniMax-M2.7
0.300
0.060
1.200
GPT-OSS-120B
0.150
0.075
0.600
MiroThinker-1.7-Mini
0.200
0.040
2.400
GPT-OSS-20B
0.075
0.0375
0.300
Appendix
Table 6: Token prices for API-equivalent cost rate (USD per million tokens).
Figure 12: Latency breakdown. Decode-step time for GPT-OSS-120B at the same batch size, broken down by operation.
Figure 15
Figure 15: Attention density across message history. Attention density per token by layer and message index averaged over 10 selected BrowseComp-Plus trajectories, divided into old reasoning, old tool responses, and recent history.
Method
BCP
GAIA-Text
SWE-Verified
xbench-DS
Average
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Standard
49.0
6.5
60.2
3.5
41.8
12.5
25.0
2.4
44.0
6.2
Text Engineering
43.0
4.4
65.1
3.0
25.6
8.8
19.0
2.0
38.2
4.6
AKV
50.5
6.0
62.1
2.5
44.2
8.9
24.0
1.8
45.2
4.8
Appendix
Table 7: Benchmark scores (pass rate, %) and costs (cents) of GPT-OSS-20B across different management methods. Bold indicates the highest score and lowest cost, including ties.
Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.
Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck. Existing eviction policies generally apply the same attention- or recency-based rule to every token, ignoring semantic structure already available to the agent orchestrator. We introduce MemDecay, a training-free, region-aware KV-cache eviction policy. MemDecay assigns tokens region-specific base priorities and decay rates, refreshes retention scores when tokens receive attention, and evicts the lowest-scoring pages under a fixed cache budget while allowing critical regions to be pinned. We also provide a procedure for calibrating decay rates from measured attention lifetimes. We evaluate MemDecay at approximately 450 and 1,700 token contexts using Qwen2.5-1.5B and 3B. Across all settings, attention lifetimes differ by an order of magnitude across regions: system-token half-lives range from 148 to 189 decoding steps, compared with 14 to 16 for scratchpad tokens. Pinning preserves system-region facts at full-cache accuracy in every setting, while no baseline preserves more than 13 of 24. Region-aware retention remains effective as context grows, whereas recency-based retention collapses. Accumulated-attention retention performs better on unpinned content, however, and ablations identify attention-score normalization as the main limitation of the current formulation. These results establish semantic prompt structure as a robust signal for KV-cache management while clarifying how it should be combined with attention-based importance.
Modern KV cache management assumes the chatbot workload: prompts arrive once and the cache grows append-only, so prefix caching and forward-only eviction are correct by construction. Agentic LLMs break this assumption. Their conversations evolve through policy-driven editing: failed tool calls are retried, stale outputs dropped, trajectories pivoted. Two distinct cache problems result. First, identical content moves to new positions between turns, invalidating exact-prefix caches even though the underlying KV would still be valid; recent work on position-independent caching for MLA addresses this reuse problem. Second, and this paper's focus, a policy may need to direct the serving system to actively remove or replace a span of cached content and continue without re-prefilling everything that came after. No existing primitive offers this. Production agentic harnesses fall back to re-prefill on every edit, paying full prefix-recomputation cost; kernel-level eviction methods make their own decisions and cannot accept policy directives from outside the kernel. We introduce Leyline, a serving-side primitive that closes this gap. A declarative directive 4-tuple separates what to edit from how to preserve position correctness. The policy declares the edit and its mode (in-place splice or prefix-trimmed re-prefill for semantic forgetting); an architecture-agnostic interface routes to a per-architecture kernel that restores attention math via a closed-form RoPE-rotation correction. The splice kernel lifts replay cache-hit by +11.2 pp and cuts latency by up to 241 ms. A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 pp on debug-gym. The mechanism is open; the policy space it enables is the agenda.
Bole Ma, Jan Eitzinger, Harald Koestler
Erlangen National High Performance Computing Center, Erlangen, Germany