Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.
Figures & tables
Figure 1: Full-context inference, context engineering, and cache engineering. Full-context inference reuses all historical KV. Context engineering removes irrelevant text and recomputes the retained history. Cache engineering keeps the text history but drops selected KV entries, reusing retained states without recomputation. Nested color blocks illustrate how retained KV can still carry information from dropped history.
Figure 2
Figure 5: Cache operation trajectories and history-dependent KV states. (a) ① appends C to A and B, then drops A’s KV; ② drops A’s KV before appending C; ③ starts from B alone and appends C. All three retain B and C. (b) B’s KV is identical in ① and ②, both conditioned on A, but differs in ③, where B has no prefix. C’s KV differs in all three: ① is computed with A and B; ② and ③ cannot attend to A but use different B caches. (c) The extended radix tree shares the A–B prefix of ① and ②, then branches according to drop timing, distinguishing the cached states used to generate D.
Figure 4
Figure 8: repos in the KV pool and radix tree. (a) Two requests share A and B; Request 2 drops A before appending C. (b) B and C initially occupy positions 1 and 2. (c) repos creates copies at positions 0 and 1 by re-encoding K and copying V. The original B remains shared with Request 1, while the original C becomes evictable. The tree branches at B to distinguish positional versions and records a reposition node after C. Labels p denote position p .
Figure 9: Task performance and serving throughput with GPT-OSS-120B on SWE-bench Verified. (a) Online evaluation on the first 100 tasks: mean cumulative pass rate versus GPU hours across three runs per method, with shaded regions indicating ± 1 standard deviation. (b) Cumulative pass rate versus cumulative logical input tokens over all 500 tasks. (c) Serving throughput versus concurrency.
Method
BCP
GAIA-Text
SWE-Verified
xbench-DS
Average
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Score
Cost
MiniMax-M2.7 (230B total, 10B active)
Standard
58.9
5.2
35.0
2.4
75.2
7.0
40.5
5.5
52.4
5.0
Text Engineering
62.3
5.3
36.9
2.7
72.0
9.4
40.0
5.2
52.8
5.7
AKV
58.9
4.8
40.8
2.6
74.4
5.5
40.5
4.7
53.7
4.4
GPT-OSS-120B (5.1B active)
Table 1: Benchmark scores (pass rate, %) and costs (cents) by model and management method. Bold indicates the highest score and lowest cost within each model, including ties.
C
Method
Request latency
TTFT
TPOT
Throughput
mean
P95
mean
P95
mean
P95
Input
Output
8
Standard
12.16
43.57
3.18
8.42
26.57
31.38
25200
217
Text Engineering
12.84
43.57
4.94
22.99
23.23
25.68
17313
228
AKV
7.79
25.93
1.87
5.99
23.50
29.42
39227
245
10
Standard
12.82
45.03
3.93
9.76
26.47
30.83
30108
254
Text Engineering
15.17
50.27
7.35
34.07
22.57
25.09
18533
246
Table 2: Serving performance (latency, s; TTFT, s; TPOT, ms/token; throughput, tokens/s) of GPT-OSS-120B on Top-80 SWE-Verified across concurrency ( C ) levels.
Table 9
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Restore with segment-batch attention. The request in (a) produces the four attention segments in (b). Numbers denote query/key positions; colored cells indicate allowed attention.
Figure 11: Encoding drop and reposition histories in the extended radix tree. (a) Three message trajectories interleave token appends, KV drops, and reposition operations. Although their retained tokens overlap, applying the operations at different points produces different KV states. (b) AKV records each operation in the radix tree, distinguishing cache versions by position and operation history. Numbers beside token nodes denote stored position encoding indices; RX denotes repos after X .
Setting
Value
Model and weight precision
GPT-OSS-120B, BF16
Hardware
Eight NVIDIA A800 GPUs, 80 GB each
Parallelism
Prefill: TP4; decode: TP4
KV transfer
Mooncake over intra-node NVLink
Attention backend
Triton
MoE backend
triton_kernel
Appendix
Table 5: Serving configuration.
Model
Cache-miss
Cache-hit
Output
MiniMax-M2.7
0.300
0.060
1.200
GPT-OSS-120B
0.150
0.075
0.600
MiroThinker-1.7-Mini
0.200
0.040
2.400
GPT-OSS-20B
0.075
0.0375
0.300
Appendix
Table 6: Token prices for API-equivalent cost rate (USD per million tokens).
Figure 12: Latency breakdown. Decode-step time for GPT-OSS-120B at the same batch size, broken down by operation.
Figure 15
Figure 15: Attention density across message history. Attention density per token by layer and message index averaged over 10 selected BrowseComp-Plus trajectories, divided into old reasoning, old tool responses, and recent history.
Method
BCP
GAIA-Text
SWE-Verified
xbench-DS
Average
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Standard
49.0
6.5
60.2
3.5
41.8
12.5
25.0
2.4
44.0
6.2
Text Engineering
43.0
4.4
65.1
3.0
25.6
8.8
19.0
2.0
38.2
4.6
AKV
50.5
6.0
62.1
2.5
44.2
8.9
24.0
1.8
45.2
4.8
Appendix
Table 7: Benchmark scores (pass rate, %) and costs (cents) of GPT-OSS-20B across different management methods. Bold indicates the highest score and lowest cost, including ties.