Modern LLM agents operate in persistent workspaces whose accumulated history can exceed both GPU KV capacity and the model's native context window. Existing systems typically compact older context into summaries or retrieve it later as text, either losing fine-grained execution evidence or repeatedly prefilling content that the model has already processed. We present KVMem, a KV-context virtualization system that preserves overflowed workspace history as paged KV state across GPU memory, host memory, and NVMe. KVMem uses lightweight, model-native attention-space indexes to select relevant historical blocks and materializes a query-dependent execution view bounded by the model's native context window. Extensive evaluations on long-context agent benchmarks spanning histories up to one million tokens, including LongMemEval, MemoryAgentBench, and AgentLongBench, show that KVMem generally achieves higher task utility and greater inference efficiency than compaction-based approaches, the de facto standard for handling context overflow. In the DeepSWE long-context test with Qwen3.8-27B, KVMem improves task success from 43.8% with compaction-only context management to 48.4%. In our local-deployment evaluation, KVMem runs Qwen3.6/3.8-27B NVFP4 with MTP on an off-the-shelf laptop equipped with a 24,GB RTX 5090 Laptop GPU, virtualizing agent workspaces of up to 1M tokens-four times the model's native 256K-token context window. In a single-session setting, KVMem generates ∼50 tokens/s, providing interactive responsiveness for local agent execution. More broadly, by decoupling addressable workspace size from the LLM's native context window, KVMem provides a practical path toward long-running agents whose workspaces can grow beyond that window.
Figures & tables
Figure 1: Overview of KVMem ’s three core designs. First, step-level memory scheduling reselects a bounded working set once per agent step, after prefill and before decoding (Sec. 4.2 ). Second, query-conditioned retrieval represents the full history with compact block-level indexes, enabling efficient block selection with little GPU memory overhead (Sec. 4.3 ). Third, tiered KV management proactively stages out completed blocks, reuses overlapping GPU pages, retains frequently retrieved blocks in host memory, and packs the remaining GPU misses into a pipelined rematerialization path (Sec. 4.4 ).
Figure 2: Attention behavior across eight OpenHands SWE-bench Lite rollouts. The top panel shows adjacent-window KL for one example rollout, where KL remains low within an agent step and spikes across step boundaries. Red markers and dashed lines denote step switches. The bottom panel reports mean top- k coverage for 128-token KV blocks across all eight rollouts, with the shaded band showing the P10–P90 range.
Figure 3: Packed and pipelined KV rematerialization. Fragmented historical KV turns stage-in into many small H2D transfers and reconstruction launches, underutilizing PCIe bandwidth and amplifying launch and synchronization overhead. KVMem gathers the scattered blocks into contiguous host buffers, transfers them in bulk, and overlaps CPU gather of batch n+1 , H2D transfer of batch n , and GPU scatter/re-RoPE of batch n−1 .
Benchmark
Active Context
Metric
Full Context (ref.)
Sliding Window
Compact Only
Compact +RAG
KVMem
LongMemEval-S
32K
Answer Acc. (%) ↑
86.60
26.80
45.60
86.20
85.60
Latency (s) ↓
0.30
0.19
18.92
26.63 (10.63)
0.48
MemoryAgentBench ( >256 K)
64K
Overall Score (%) ↑
–
17.95
27.54
34.80
40.99
Latency (s) ↓
–
0.26
85.87
106.02 (20.72)
1.81
AgentLongBench ( ≤ 256K)
32K
Task Success (%) ↑
59.54
25.36
15.84
47.49
60.87
Latency (s) ↓
0.17
0.11
97.04
111.39 (14.80)
0.38
Table 1: Utility and efficiency results across controlled long-history benchmarks. The active context column reports the context budget used by Sliding Window, Compact-only, Compact+RAG, and KVMem ; Full Context is not subject to this limit. Pre-answer latency includes memory construction and retrieval when applicable, together with final-query input processing. For Compact+RAG, values in parentheses exclude the time required to generate the compact summary. All methods are executed with the same QW3 backend and model configuration.
Utility
Efficiency
Method
Answer Acc. (%) ↑
Total input (K) ↓
Fresh prefill (K) ↓
Pre-answer latency (s) ↓
Full Context
86.60
109.74
0.08
0.30
Sliding Window
26.80
27.99
0.08
0.19
Compact-only
45.60
114.00
2.83
18.92
Compact+RAG
86.20
142.16
31.00
26.63
KVMem
85.60
109.74
0.08
0.48
Table 2: Utility and efficiency results on LongMemEval-S. Higher answer accuracy is better; lower values are better for all efficiency metrics. Full Context assumes sufficient context and KV-cache capacity to include the complete history; all other methods use a 32K-token active context window. Total input includes the input used to construct the compact summary for Compact-only and Compact+RAG. Fresh prefill denotes the total cache-miss input. Pre-answer latency includes memory construction and retrieval when applicable, together with final-query input processing.
System
Execution View / Context
Virtual Workspace
Execution Model
vLLM
10K
–
Direct context
llama.cpp
80K
–
Direct context
KVMem ( QW3 )
80K
1M
Virtual workspace
Table 3: Maximum context and workspace capacity on the same consumer laptop with a 24 GB RTX 5090 Laptop GPU.
Workspace
GPU Memory
Host Memory
NVMe Storage
Retrieval
TTFT
Prefill
Decode
(Model+KV, GiB)
(KV+Index, GiB)
(KV, GiB)
Latency (s)
(s)
(tok/s)
(tok/s)
256K
33.9
18.0
8.5
0.174
0.427
2582.1
78.69
512K
34.7
26.5
17.0
0.209
0.466
2558.2
85.56
1M
34.9
43.6
34.0
0.213
0.441
2639.3
83.25
2M
34.8
64.0
68.0
0.393
0.625
2641.1
81.02
4M
34.9
64.0
136.0
0.828
1.070
2597.5
75.67
Table 4: Workspace scalability on the server platform with 32-token blocks. The execution view is fixed at 64K tokens while the addressable workspace grows from 256K to 10M tokens. GPU Memory reports the total GPU-process footprint, Host Memory includes host-resident KV state and the retrieval index and is capped at 64 GiB, and NVMe Storage reports the persistent KV-store footprint. Retrieval latency and TTFT are medians over 16 frozen-query trials after one warm-up, using the same 128-token queries at all workspace sizes. Retrieval latency covers scoring, selection, and KV materialization, while TTFT additionally includes query prefill. Prefill throughput is measured with 2,048-token chunks.
Model and configuration
Agent harness
Runs/task
Pass@1 (%) ↑
Pass@4 (%) ↑
Public DeepSWE trajectories
Gemini 3.7 Flash [high]
mini-swe-agent
4
64.1
81.3
DeepSeek V4 Flash [max]
mini-swe-agent
4
53.1
75.0
Claude Sonnet 4.6 [high]
mini-swe-agent
4
32.8
56.3
Kimi K2.7 Code [default]
mini-swe-agent
4
29.7
75.0
Our paired Qwen3.8-27B experiment
Table 5: Success on the first sixteen tasks in the canonical DeepSWE v1.1 task ordering. We select the first sixteen tasks directly, without task-level filtering or cherry-picking, and sample each task four times, yielding 64 trajectories per local configuration. Public-model Pass@1 and Pass@4 values are reaggregated from released scored trajectories over exactly the same task subset. Pass@4 denotes the percentage of tasks with at least one passing scored rollout. Public configurations use mini-swe-agent, whereas our paired Qwen3.8-27B experiment uses Claude Code.
Sample group
Policy
Agent time (min)
Request wall time (min)
Prefill time (s)
Decoded tokens (K)
All selected tasks ( n=16 )
KVMem
45.8 (1.15 × )
40.7 (1.18 × )
95.0 (2.23 × )
125.6 (1.23 × )
Compaction-only
52.5
48.1
211.5
154.4
Table 6: Efficiency on the first 16 DeepSWE tasks. Compared with the compaction-only Qwen3.8-27B baseline, KVMem achieves a 1.15 × agent speedup, a 1.18 × request-time speedup, a 2.23 × prefill speedup, and a 1.23 × reduction in decoded tokens. Values are per-task averages, and parenthetical factors report the Compaction-only value divided by the KVMem value.