Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

Yakov Pyotr Shkolnikov

cs.LGFeb 17, 2026

Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

Authors: Yakov Pyotr Shkolnikov

Abstract

Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory

Figures & tables

Device	Mem	BW	SSD	Type
	(GB)	(GB/s)	(GB/s)
M4 Pro (MacBook Pro)	24	273	${\sim}$ 7	Unified
M4 Max (MacBook)	128	546	7	Unified
DGX Spark	128	273	11	Unified
RTX 5090 (VRAM)	32	1792	64 ∗	Discrete
RTX 4090 (VRAM)	24	1008	32 ∗	Discrete

Table 1: Edge device memory and bandwidth. Unified memory devices share RAM between CPU and GPU. Discrete GPUs (RTX) have separate VRAM; KV cache offload to host RAM drops to PCIe bandwidth.

Panel of Figure 1 — Figure 1: System architecture. Multiple agents maintain isolated KV caches in a persistent block pool. The Q4 pipeline quantizes cache data on save and operates directly on quantized tensors during attention. Disk persistence enables sub-100ms reload (warm) vs seconds of re-prefill (cold).

Context	FP16/agent	Q4/agent	FP16 fits	Q4 fits
4K	1.5 GB	0.42 GB	6	24
8K	3.0 GB	0.84 GB	3	12
16K	6.0 GB	1.7 GB	1	6
32K	12.0 GB	3.4 GB	0	3

Table 2: Agent capacity on M4 Pro (10.2 GB cache budget). Gemma 3 12B, 48 layers, 8 KV heads, head dim 256.

Panel of Figure 2 — Figure 2: TTFT scaling across cache states for all three models (Gemma solid, DeepSeek dashed, Llama dotted). Cold prefill scales linearly with context length. Hot and warm caches reduce TTFT by up to 136 $\times$ at 32K (Gemma) and 111 $\times$ at 16K (Llama), with sub-second reload up to 16K context. Llama 3.1 8B falls between Gemma and DeepSeek in cold prefill, but achieves the highest warm speedups (111 $\times$ at 16K) due to its smaller 8B parameter count enabling fast cache reload relative to cold.

Model	Cache	1K	2K	4K	8K	16K	32K
Gemma 3	Cold	3964	7119	15736	35009	74219	172096
	Warm	475	495	577	626	795	1819
	Hot	683	709	719	807	934	1264
DeepSeek	Cold	1043	1737	3970	8296	19396	47315
	Warm	234	246	271	338	434	633
	Hot	345	368	366	386	490	624

Table 3: TTFT (ms) by cache state. Streaming, batch=1, median of 6 passes.

		Gemma 3		DeepSeek		Llama 3.1
Context	Cache	SysTPS	Per	SysTPS	Per	SysTPS	Per
1K	Cold	10.4	5.2	35.5	17.8	16.6	8.3
1K	Warm	22.6	11.2	63.0	31.5	37.8	18.9
1K	Hot	22.1	11.1	73.1	36.6	38.3	19.1
4K	Cold	3.5	1.8	13.5	6.8	5.9	2.9
4K	Warm	19.6	9.9	52.6	26.3	34.5	17.2

Table 4: Single vs concurrent throughput (non-streaming, median of 6 passes). Single: batch=1. Concurrent: batch=2, SysTPS = total tokens/second across both agents.

	agent-memory (Q4 KV)			vllm-mlx (FP16 KV)
Context	Cold	Warm	Restart	Cold	Prefix	Restart
1K	2,500	260	260	2,289	273	2,262
2K	4,530	274	274	2,204	204	4,134
4K	10,235	290	290	4,394	290	8,079
8K	21,536	331	331	9,472 †	244 †	—
16K	47,629	431	431	37,935 †	371 ‡	—

Table 5: Head-to-head TTFT comparison: agent-memory (Q4 KV cache, disk-persistent) vs vllm-mlx [ 3 ] (FP16 KV cache, volatile). Llama 3.1 8B Q4 weights, streaming, batch=1, median of 3 passes. “Warm” = agent-memory Q4 cache loaded from SSD. “Prefix” = vllm-mlx FP16 prefix cache hit (in-memory only). “Restart” = server killed and restarted, then same prompt re-sent. vllm-mlx run with --continuous-batching --cache-memory-percent 0.40 (9.6 GB of 24 GB UMA for FP16 prefix cache). Benchmark structured as cold

\to

prefix per context to prevent FP16 cache eviction (Section 4.4 ).

Component	Metric	With	Without	Effect
Persistence	TTFT (ms), Gemma 4K	577	15736	27 $\times$
Q4 vs FP16	Agents at 8K	12	3	4.0 $\times$
Batching	SysTPS, Gemma 1K warm	22.6	11.2 ∗	2.0 $\times$
Cross-phase	TTFT (ms), Phase 5	1705	3292	1.9 $\times$
∗ Per-agent TPS = SysTPS/2. The 2.0 $\times$ is definitional (system vs per-agent),
not a measured throughput gain over batch=1.

Table 6: Component contributions. Each row compares the system with vs without one component, holding others constant.

	Gemma 3			DeepSeek
Phase	Cold	Pers	$\times$	Cold	Pers	$\times$
1: Interrogation A	1136	1079	1.1	477	460	1.0
2: Interrogation B	1119	976	1.1	465	430	1.1
3: The Yard	1648	1019	1.6	532	474	1.1
4: Final Reckoning	2195	1250	1.8	664	542	1.2
5: Verdict	3292	1705	1.9	874	649	1.3

Table 7: Measured per-phase average TTFT (ms). Cold: caches cleared each phase. Persistent: caches accumulate. 25 turns total per run.

	Gemma 3		DeepSeek
Phase	TTFT	Quality	TTFT	Quality
1: Priming (cold)	20514	8/10	5140	3/10
2: Queries (warm)	847	8/10	396	4/10
3: Repeated (hot)	860	3/3	424	2/3
Warm/cold speedup	24.2 $\times$		13.0 $\times$
Hot/cold speedup	23.9 $\times$		12.1 $\times$

Table 8: Wikipedia routing TTFT (ms) by phase. 10 experts, 5 queries, 3 repeated. Articles are 3K words (

{\sim}

4K tokens) each.

Model	FP16 PPL	Q4 PPL	$\Delta$ PPL	$\Delta$ %
Gemma 3 12B	14.40	14.30	$-$ 0.10	$-$ 0.7
DeepSeek-V2-Lite 16B	6.26	6.45	$+$ 0.19	$+$ 3.0
Llama 3.1 8B	4.28	4.40	$+$ 0.12	$+$ 2.8

Table 9: Perplexity with actual Q4 KV caches. FP16 baseline uses standard KV cache; Q4 uses the production QuantizedKVCache (group size 64). 7,935 tokens, 512-token windows, 256-token stride.

	RAG	KV Persist	Msg Pass
Restore cost	O( $n$ ) prefill	O( $n$ ) I/O	O( $n$ ) rebuild
Stores	Text chunks	Attn state	Structured msgs
Scope	External KB	Conv. history	Inter-agent
Model-specific	No	Yes	No
Hardware	Vector DB	SSD/RAM	Network

Table 10: Context restoration approaches for multi-turn agents.

System	Pool	BQ4	WM	Edge	Disk
vLLM [ 12 ]	Paged	FP8 †	No	No	No
SGLang [ 36 ]	Radix	No	No	No	No
vllm-mlx [ 3 ]	Prefix	No	No	Yes	No
llama.cpp [ 7 ]	Slot	Q4 ‡	No	Yes	Slot §
mlx-engine [ 18 ]	No	Q8 ∥	No	Yes	No
KVSwap [ 35 ]	No	No	No	Yes	No

Table 11: Feature comparison with related systems. Pool: per-agent cache isolation. BQ4: batched Q4 inference. WM: cross-phase KV persistence. Edge: UMA device support. Multi: dense + MoE architectures. Disk: persistent cache survives restart.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symptom	Root Cause	Fix
Cache appears empty after prefill	Missing mx.eval() after cache update	Evaluate after update
OOM during batch	Graph accumulates without clearing	Evaluate per iteration
Zeros after disk reload	loaded tensors not evaluated	Evaluate after load
Quantization corruption	Scales/biases lazy	Evaluate quantize output
Attention NaNs	Q4 tensors invalid post-load	Validate dtype/shape
Batch hangs	Merge built graph but not executed	Evaluate before attention

Appendix

Table 12: MLX lazy evaluation failure modes relevant to KV cache persistence.

Model	Cache	1K	4K	16K	32K
Gemma	Cold	3815–3964–4055	15478–15736–16076	74120–74219–74260	171718–172096–172436
	Warm	455–475–539	504–577–608	782–795–817	1748–1819–1921
	Hot	668–683–701	697–719–735	920–934–964	1240–1264–1345
DeepSeek	Cold	1019–1043–1077	3861–3970–4050	19257–19396–19466	47066–47315–47359 †
	Warm	227–234–271	263–271–287	407–434–505	631–633–650 †
	Hot	323–345–457	352–366–406	483–490–497	614–624–656 †

Appendix

Table 13: TTFT measurement ranges (ms). Min–median–max of 6 passes.

	4K		8K		16K		32K
Model	FP16	Q4	FP16	Q4	FP16	Q4	FP16	Q4
Gemma 3	6	24	3	12	1	6	0	3
DeepSeek	9	33	4	16	2	8	1	4
Llama 3.1	19	70	9	35	4	17	2	8

Appendix

Table 14: Agent capacity comparison, all models. M4 Pro, 10.2 GB cache budget.

Panel of Figure 3 — Figure 3: Staggered request arrivals (4K cold context, Agent B arrives 2 s after Agent A). Total wall time is identical between sequential and batched modes. However, Agent B’s own perceived TTFT (submit to first token) is worse in batched mode: 34.1 s vs 16.7 s for Gemma (2.0 $\times$ ), 20.2 s vs 10.3 s for Llama (2.0 $\times$ ), 6.8 s vs 3.9 s for DeepSeek (1.7 $\times$ ). In sequential mode, B runs alone with full GPU bandwidth after A completes. In batched mode, B starts 2 s after A but shares prefill bandwidth via interleaved chunking, approximately doubling its individual TTFT.

Panel of Figure 4 — Figure 4: Architecture comparison. The block pool abstracts away architectural differences through ModelCacheSpec. Gemma 3 uses grouped-query attention with hybrid sliding-window layers, requiring 5D mask expansion and window-aware chunked prefill. DeepSeek uses multi-latent attention with asymmetric K/V dimensions (192 vs 128) and MoE routing, requiring larger memory budgets for intermediate tensors.

Panel of Figure 5 — Figure 5: Agent cache state across prisoner’s dilemma phases. Permanent agents (Warden, Marco, Danny) start cold and transition to warm/hot as context accumulates via cross-phase injection. Each phase extends the cached prefix rather than re-computing. The Analyst appears only in Phase 5 (cold start). TTFT annotations show projected latency from Table 3 at equivalent context lengths.

Panel of Figure 6 — Figure 6: Wikipedia multi-agent routing. Ten expert agents are primed with article content (cold prefill). Cross-topic queries route to 2–3 relevant experts whose caches are warm/hot from priming. A reporter agent synthesizes responses. Repeated queries to the same experts benefit from hot cache (projected 10–30 $\times$ TTFT reduction vs cold).

Category	Count	Representative Example
1: External dependency	2	Transformers v5.0.0rc1 SentencePiece regression: space markers stripped during decode, all words run together. Fix: pin transformers $<$ 5.0 .
2: Integration	5	DeepSeek MLA caches K at dim=192 (128+64 rope) but V at dim=128; spec extractor fell back to 128 for both, causing 20% memory undercount. DeepSeek chat template closes every assistant turn with EOS, killing identity injection.
3: Concurrency	6	Batch=2 warm cache produces EOS on first token. Root cause: scheduler thread runs submit() $\to$ mx.eval() while event loop runs _save_to_disk() $\to$ safetensors save with different locks $\to$ SIGSEGV. Initial hypothesis (lazy tensor chain) was wrong; verification script masked the race with sequential timing.
4: Silent correctness	4	Gemma 3 sliding window mask ignored during chunked prefill: QuantizedKVCache.make_mask() returns causal mask regardless of window_size . 40/48 layers get full attention instead of 1024-token window. Output looks plausible; perplexity scaling with context length was the tell.

Appendix

Table 15: Bug taxonomy from development. 17 distinct bugs across 135 fix-related commits. Categories ordered by diagnosis difficulty.

Model	Input	Output	Cache W	Cache R	Total
Opus 4.5	$7	$120	$908	$1,225	$2,260
Sonnet 4.5	$2	$50	$306	$544	$902
Opus 4.6	$2	$23	$238	$522	$783
Haiku 4.5	$0	$0	$1	$1	$2
Total	$11	$193	$1,453	$2,291	$3,947

Appendix

Table 16: AI development cost. 44 sessions, 19 days (Jan 22 – Feb 9, 2026). 1,587 human messages, 301 commits.

Memory Management for LLM Agents

See all 58 papers

May 5, 2026

QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs

Multi-agent LLM systems on edge devices need to hand off latent context efficiently, but the practical choices today are expensive re-prefill or full-precision KV transfer. We study QKVShare, a framework for quantized KV-cache handoff between agents that combines token-level mixed-precision allocation, a self-contained CacheCard representation, and a HuggingFace-compatible cache injection path. Our current results support a narrower but clearer story than the original draft: on 150 GSM8K problems with Llama-3.1-8B-Instruct, adaptive quantization remains competitive under repeated handoff and shows its clearest gains against uniform quantization in deeper-hop, higher budget settings; for handoff latency, the QKVShare path reduces TTFT relative to full re prefill at every tested context, from 130.7 ms vs. 150.2 ms at nominal 1K context to 397.1 ms vs. 1029.7 ms at nominal 8K context;. Stage timing shows that post-injection generation, not card creation, dominates the current QKVShare latency path. These results position quantized KV handoff as a promising on-device systems direction while also highlighting the need for stronger controller ablations and apples-to-apples runtime comparisons.

Pratik Honavar, Tejpratap GVSL

Mixed-Precision Quantization Multi-Agent LLM Systems KV Caching KV-Cache Management KV-Cache Quantization On-Device Inference

Ask PDF

Apr 27, 2026

PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to preserve softmax stability, while Values are compressed using TurboQuant MSE -- a Fast Walsh-Hadamard Transform (FWHT) rotation followed by 3-bit Lloyd-Max quantization with centroids tuned to N(0,1). We evaluate across two model scales (SmolLM2-1.7B-Instruct and Llama-3-8B-Instruct), three context lengths (600-7,194 tokens), and up to 15 concurrent agents. PolyKV achieves a stable 2.91x compression ratio across all configurations. On Llama-3-8B with 15 agents sharing a 4K-token context, PolyKV reduces KV cache memory from 19.8 GB to 0.45 GB -- a 97.7% reduction -- while maintaining only +0.57% perplexity degradation and a mean BERTScore F1 of 0.928. PPL delta does not grow with agent count and improves as context length increases, inverting to -0.26% at 1,851 coherent tokens. To our knowledge, no prior work combines a single shared, lossy-compressed KV pool with multi-reader concurrent agent access.

Ishan Patel, Ishan Joshi

Independent Researcher

LLM Inference Efficiency Multi-Agent LLM Systems KV Caching KV-Cache Reuse KV-Cache Compression KV-Cache Quantization

Ask PDF

Jul 12, 2026

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck. Existing eviction policies generally apply the same attention- or recency-based rule to every token, ignoring semantic structure already available to the agent orchestrator. We introduce MemDecay, a training-free, region-aware KV-cache eviction policy. MemDecay assigns tokens region-specific base priorities and decay rates, refreshes retention scores when tokens receive attention, and evicts the lowest-scoring pages under a fixed cache budget while allowing critical regions to be pinned. We also provide a procedure for calibrating decay rates from measured attention lifetimes. We evaluate MemDecay at approximately 450 and 1,700 token contexts using Qwen2.5-1.5B and 3B. Across all settings, attention lifetimes differ by an order of magnitude across regions: system-token half-lives range from 148 to 189 decoding steps, compared with 14 to 16 for scratchpad tokens. Pinning preserves system-region facts at full-cache accuracy in every setting, while no baseline preserves more than 13 of 24. Region-aware retention remains effective as context grows, whereas recency-based retention collapses. Accumulated-attention retention performs better on unpinned content, however, and ablations identify attention-score normalization as the main limitation of the current formulation. These results establish semantic prompt structure as a robust signal for KV-cache management while clarifying how it should be combined with attention-based importance.

Venkatesha Matam, Keon Kim

Independent Researcher New York City, USA

KV Caching KV-Cache Eviction KV-Cache Management LLM Inference Acceleration

Ask PDF

Abstract

Memory Management for LLM Agents

Explore similar work

QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs

PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference