EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

Kunming Shao; Jierun Chen; Jiangnan Yu; Xiao-Hui Li; Chaofan Tao; Yanli Wang; Huanxin Lin; Kwang-Ting Cheng

cs.DCSep 27, 2026

EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

Authors: Kunming Shao, Jierun Chen, Jiangnan Yu, Xiao-Hui Li, Chaofan Tao, Yanli Wang, Huanxin Lin, Kwang-Ting Cheng, +2 more

Organizations: The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · The University of Hong Kong · Sun Yat-sen University

Abstract

LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key-value (KV) state and, when GPU memory runs out, by offloading that state to host memory. For agents, offloading gives inconsistent results: on the same coding-agent workload it speeds up one deployment, slows down another, and changes nothing on a third, even where loading a token back is several times cheaper than recomputing it. The reason is that cached state must survive until it is used again. While one agent waits for its tool, the server processes the contexts of all other agents, so an agent's prefix is reused only if the host tier holds the reusable context of the whole agent pool, which we call the reuse working set. A smaller tier keeps writing state that is evicted before anyone reads it. We present EfficientAgent, which sizes and manages the host tier by this working set. A stack-distance model estimates the working set from agent histories to size the host tier; its predictions, made before the experiments, located the capacity at which offloading starts to pay. When the tier is too small, a runtime policy stops writing large refills of evicted context and keeps extending prefixes that are still cached; when the tier is large enough, it writes everything. On SWE-bench Verified coding agents, a host tier sized to the estimated working set cuts recomputed prompt tokens by 93% and end-to-end time by 39%. With a small fixed tier, the policy cuts recomputation by 35%; with a large tier, it avoids the 4.3-fold increase caused by always filtering writes. Across three GPU types and two models, offloading pays off when the GPU has little compute per byte of host bandwidth and the host tier holds the working set. Code is available at https://github.com/KunmingSHAO/efficientagent_release.

Figures & tables

GPU	BF16 TFLOPS	Host link (GB/s)	Memory (GB)	FLOP/byte	Offload/recompute
RTX 3090	71	PCIe Gen4, 32	24	2.2K	0.91 ; dense 0.92
H20	148	PCIe Gen5, 64	96	2.3K	0.60 (tier $\geq$ working set)
H800	989.5	PCIe Gen5, 64	80	15.5K	1.08–1.87; dense 1.23

Table 1: Offload pays on low-ratio GPUs once host hits survive and slows every H800 run. Peak dense BF16 TFLOPS, host link per direction, and memory per GPU; offload/recompute for the MoE and the dense model. Bold: offload faster than recomputation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Model / precision	Qwen3-Coder-30B-A3B-Instruct / BF16
Maximum model context	262,144 tokens
Running-request cap	16 ( max_num_seqs )
Scheduler token budget	8,192; chunked prefill enabled
GPU prefix caching	Enabled
Active pool $A$	16; 8 in the concurrency comparison

Appendix

Table 2: Serving configuration. Upper part: the H20 stack; the active pool limits the tasks in progress, and the running-request cap limits the requests the engine executes at once. Lower part: memory configuration per deployment of Qwen3-Coder-30B-A3B-Instruct and of the dense Qwen2.5-Coder-32B-Instruct; GPU KV capacity is in tokens, each GPU holding its shard of every token.

Configuration	$A$	CPU GiB	Time (min)	Prefill M	Retrieved M	Stored M	Preempt.
Recompute	16	0	211.69	89.66	0	0	2,190
Recompute, repeat	16	0	205.98	89.82	0	0	2,203
Offload	16	5	208.91	76.74	12.57	81.69	1,808
Offload	16	10	161.06	23.53	65.20	22.42	437
Offload	16	20	128.05	5.28	82.77	4.06	46
Offload	16	40	124.01	5.27	82.43	4.06	52

Appendix

Table 3: Complete H20 replay measurements. Every completed row processes the same 4,427 calls. Each row is one run; repeats are retained individually ( ∗ : see Baselines and policy variants).

A

: active pool; host (CPU) budgets are per rank. Retrieved and stored token volumes are summed over TP workers and divided by eight. Preempt.: engine preemptions during the replay. Bold: capacity-conditioned admission.

Host GiB	Predicted prefill	Measured prefill	Predicted restore	Measured restore
10	27.64–31.20	23.53	55.28–58.88	65.20
20	5.19–5.78	5.28	80.84–81.36	82.77
80	5.19–5.31	5.29	81.31–81.36	82.89

Appendix

Table 4: Prefix-survival predictions and subsequent measurements. Prefill and restored volumes are millions of tokens; restored volume is per rank. Prediction ranges use the two reference interleavings.

Host GiB	Predicted prefill	Measured prefill	Predicted restore	Measured restore
0	41.87–60.80	41.48	0	0
3	29.59–52.01	28.99	7.51–12.28	13.01
4.5	18.84–31.73	17.11	22.25–29.36	13.22

Appendix

Table 5: Predictions and measurements for an active pool of

A=8

. Units match Table 4 .

	Qwen3-Coder-30B-A3B-Instruct			Qwen2.5-Coder-32B-Instruct
TP ranks	Heads/rank	KiB/rank	Aggregate KiB	Heads/rank	KiB/rank	Aggregate KiB
1	4	96	96	8	256	256
2	2	48	96	4	128	256
4	1	24	96	2	64	256
8	1	24	192	1	32	256

Appendix

Table 6: BF16 KV footprint of the two evaluated models.

Parameter	Value
Cache occupancy threshold $\theta$	0.95
Task activity and eviction window	60 seconds
Prompt-length window	Last 256 first prefix lookups
Telemetry report interval	1 second
Scheduler report-read interval	0.5 seconds
Maximum age of a fresh report	5 seconds

Appendix

Table 7: Write-admission parameters, fixed before the admission runs.

Workflow wall-clock
Hardware	Model	Recompute	Offload	Ratio
$8\!\times$ RTX 3090	MoE	407 min	369 min	0.91
$8\!\times$ RTX 3090	Dense	443 min	406 min	0.92
$8\!\times$ H20	MoE	20.58 h	20.80 h	1.01
Engine window
$2\!\times$ H800	MoE	62 min	73 min	1.18

Appendix

Table 8: Offload shortens the RTX 3090 runs and lengthens every H800 run for both models. Both configurations enable GPU prefix caching. Rows are grouped by timing scope; each ratio (offload/recompute) compares the two within one deployment and scope. MoE: Qwen3-Coder-30B-A3B-Instruct; dense: Qwen2.5-Coder-32B-Instruct. † Complete SWE-bench Verified benchmark. Bold: offload faster than recomputation.

Hardware	Without prefix caching	With prefix caching	Speedup
$8\!\times$ RTX 3090	13 h 35 min	1 h 13 min	11.1 $\times$
$2\!\times$ H800	1 h 58 min	54 min	2.19 $\times$

Appendix

Table 9: Prefix caching shortens coding-agent runs by up to 11.1

\times

. Qwen3-Coder-30B-A3B-Instruct; workflow wall-clock.

		Recompute	Offload
Hardware	GPU KV usage	Total	Total	GPU	External
$8\!\times$ RTX 3090	$<$ 80%	62.9	79.4	62.7	16.6
	80–90%	34.7	43.7	16.9	26.7
	90–95%	34.4	44.0	17.1	27.2
	95–100%	33.8	44.3	17.3	25.9
$2\!\times$ H800	$<$ 80%	65.7	98.4	97.5	0.2

Appendix

Table 10: Prefix-hit medians by GPU KV usage. The Qwen3-Coder-30B-A3B-Instruct RTX 3090 and two-H800 subset pairs of Table 8 , both at GPU memory fraction 0.65. Entries are medians (%) within each usage band. Without offload, all prefix hits are GPU hits. With offload, the GPU, external, and total rates are separately computed medians; the GPU and external medians need not sum to the total. Bold: net hit gain with offload at high GPU KV usage.

GPU memory fraction	Wall-clock (min)	Prefix hit rate
.65	407	44.6%
.70	333	53.5%
.75	245	58.6%
.80	114	79.7%
.85	78	96.2%
.95	73	98.1%

Appendix

Table 11: Complete RTX 3090 GPU memory-fraction sweep (Qwen3-Coder-30B-A3B-Instruct, SWE-bench Verified subset).

Regime	Evidence	Action
High FLOP per host-link byte	Offload/recompute 1.08–1.87 (H800)	Favor GPU prefix caching
Low FLOP per byte, $\gamma_{G}>1$	0.60 (H20, 20 GiB); 0.91 (RTX 3090)	Offload; admit every write if $\gamma_{H}\leq 1$
$\gamma_{H}>1$	Makespan $-10.4\%$ at 5 GiB	Conditioned admission or larger tier
Context editing	Early edits rekey all later chunks	Judge edits by cache-stable length

Appendix

Table 12: Deployment guide.

Condenser	GPU KV utilization	Prefix-hit rate
None (NoOp)	52.85%	96.50%
Recent-event truncation	25.57%	60.85%
Observation masking	29.15%	50.80%
LLM summarization	41.12%	92.15%

Appendix

Table 13: Condensers that rewrite early history lower the prefix-hit rate. Qwen3-Coder-30B-A3B-Instruct on

8\!\times

RTX 3090 with GPU prefix caching; run means.

Explore similar work

AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3Depthweave-Kv Key-Value Cache Eviction
Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA
Latent Cache Flow: Model-to-Model Communication Without Text
May 19, 2026Maximillian Rossi, Prajwal Raghunath, Eugene WuLarge Language Model Agents Latent Flow
Columbia University
TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
Jul 31, 2026Zhenyu Zhang, Zhichao CaoTime-To-First-Token Large Language Model Serving
Arizona State University · Tempe, Arizona, USA

Sep 14, 2026, cs.LGJ/K move · Enter open · S save

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao, Bowen Yang+2

Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA

Depthweave-Kv Key-Value Cache Eviction Large Language Model Serving Agentic

Open paper AskPDF arXiv

Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

Abstract

Explore similar work

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Latent Cache Flow: Model-to-Model Communication Without Text

TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving