The Missing Memory Hierarchy: Demand Paging for LLM Context Windows

Tony Mason

cs.OSMar 9, 2026

The Missing Memory Hierarchy: Demand Paging for LLM Context Windows

Authors: Tony Mason

Organizations: University of British Columbia Canada · Georgia Institute of Technology USA

Abstract

The context window of a large language model is not memory. It is L1 cache: a small, fast, expensive resource that the field treats as the entire memory system. There is no L2, no virtual memory, no paging. Every tool definition, every system prompt, and every stale tool result occupies context for the lifetime of the session. The result is measurable: across 857 production sessions and 4.45 million effective input tokens, 21.8% is structural waste. We present Pichay, a demand paging system for LLM context windows. Implemented as a transparent proxy between client and inference API, Pichay interposes on the message stream to evict stale content, detect page faults when the model re-requests evicted material, and pin working-set pages identified by fault history. In offline replay across 1.4 million simulated evictions, the fault rate is 0.0254%. In live production deployment over 681turns, the system reduces context consumption by up to 93% (5,038KB to 339KB); under extreme sustained pressure, the system remains operational but exhibits the expected thrashing pathology, with repeated fault-in of evicted content. The key observation is that the problems the field faces, such as context limits, attention degradation, cost scaling, lost state across sessions, are virtual memory problems wearing different clothes. The solutions exist: working set theory (Denning, 1968), demand paging, fault-driven replacement policies, and memory hierarchies with multiple eviction-managed levels. We describe the architecture of a full memory hierarchy for LLM systems (L1 through persistent storage), report on the first three levels deployed in production use (L1 eviction, L2 fault-driven pinning, L3 model-initiated conversation compaction), and identify cross-session memory as the remaining frontier.

Figures & tables

OS Concept	Context Window Equivalent
Physical memory	Context window (200K tokens)
Virtual memory	Persistent state (disk, databases)
Page table	Retrieval handles (UUIDs, anchors)
Page fault	Model re-requests evicted content
MMU	Proxy layer between client and API
Working set	Currently relevant context subset

Table 1. Virtual memory analogy for context window management.

Level	Contents	Eviction	Fault
L1	Generation window	—	—
L2	Working set (pinned)	Pressure-based	Re-read
L3	Session history	Age-based	Summary expansion
L4	Cross-session memory	LRU / relevance	Retrieval query
Storage	Full corpus	None	Search + ingest

Table 2. Memory hierarchy for LLM systems.

Panel of Figure 1 — Figure 1. Tool adoption rate across 801 active sessions. The median session uses 3 of 18 available tools. Seven tools see zero or near-zero adoption, yet their full schemas (collectively $\sim$ 24,500 bytes) are sent on every API call.

Category	Bytes	%	Mechanism
Dead tool output	6,468,360	26.5	Stale results never re-referenced
Tool definition stubs	4,924,950	20.2	Schemas for unused tools
Static re-send	2,680,794	11.0	Unchanged system prompt content
Skill triplication	700,582	2.9	Same skill listed $3\times$
Total addressable	14,774,686	60.5

Table 3. Waste taxonomy. Percentages are of total request bytes across 99 API calls.

Total simulated evictions	1,393,000
Page faults detected	354
Fault rate	0.0254%
Content evicted	8.49 GB

Table 4. Eviction safety results from offline replay.

Metric	Baseline	Trimmed	Compact+Trim
API calls	3	4	3
Effective input tokens	114,222	88,421	71,816
Cache read tokens	79,712	38,639	32,228
Task completed correctly	Yes	Yes	Yes
Token reduction	—	22.6%	37.1%

Table 5. Treatment comparison on a standardized task.

Intervention	Tokens Saved	% of Input
Tool stub trimming	487.5M	11.0%
Skill deduplication	95.8M	2.2%
Static re-send	387.0M	8.7%
Total addressable	970.4M	21.8%

Table 6. Corpus-scale token savings projection.

Context free before compaction	7%
Context free after compaction	43%
Total evictions	15
Garbage collected	11
Read evictions (pageable)	4
Page faults	1
Fault rate (Read only)	25%

Table 7. Production Session A: steady-state coding.

Turns	681
Evictions (total)	680
Garbage collected	74
Read evictions (pageable)	606
Page faults	659
Fault rate (total evictions)	97% (659/680)
Peak compression	5,038KB $\to$ 339KB

Table 8. Production Session B: 681-turn sustained session.

Metric	Baseline	Treatment
Judges preferring	15 (28%)	20 (37%)
Ties	19 (35%)
Mean correctness	3.89	3.74
Mean completeness	3.59	3.59
Mean coherence	3.74	3.69
Max score $\Delta$	0.15

Table 9. Non-inferiority evaluation: 18 sessions, 54 verdicts. Treatment evicts consumed tool results outside a 20-message recency window (mean compression 48%).

Panel of Figure 2 — Figure 2. Cumulative input tokens processed over an 88-turn session. Without context management, every turn re-processes all prior content, producing superlinear cumulative cost. Pichay’s eviction reduces cumulative token spend by 45% ($11.67 at Opus pricing). The shaded area represents wasted computation.

LLM Context Management

Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.

See all 248 papers

Feb 25, 2026

Contextual Memory Virtualisation: DAG-Based State Management and Structurally Lossless Trimming for LLM Agents

As large language models engage in extended reasoning tasks, they accumulate significant state -- architectural mappings, trade-off decisions, codebase conventions -- within the context window. This understanding is lost when sessions reach context limits and undergo lossy compaction. We propose Contextual Memory Virtualisation (CMV), a system that treats accumulated LLM understanding as version-controlled state. Borrowing from operating system virtual memory, CMV models session history as a Directed Acyclic Graph (DAG) with formally defined snapshot, branch, and trim primitives that enable context reuse across independent parallel sessions. We introduce a three-pass structurally lossless trimming algorithm that preserves every user message and assistant response verbatim while reducing token counts by a mean of 20% and up to 86% for sessions with significant overhead by stripping mechanical bloat such as raw tool outputs, base64 images, and metadata. A single-user case-study evaluation across 76 real-world coding sessions demonstrates that trimming remains economically viable under prompt caching, with the strongest gains in mixed tool-use sessions, which average 39% reduction and reach break-even within 10 turns. A reference implementation is available at https://github.com/IngressionLabs/contextual-memory-virtualisation.

Cosmo Santoni

Imperial College London

LLM Context Management Agent Memory Management Persistent Memory for Language Models Graph-Based Agent Memory Context Compression

Ask PDF

Jun 15, 2026

TokenPilot: Cache-Efficient Context Management for LLM Agents

As LLM agents are deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimize token footprints; however, their unconstrained sequence mutations alter layouts, introducing prefix mismatches and cache invalidation. This reveals a critical trade-off between text sparsity and prompt cache continuity. To address this, we present TokenPilot, a dual-granularity context management framework. Globally, Ingestion-Aware Compaction acts as a framework harness to stabilize prompt prefixes and eliminate open-world environmental noise at the ingestion gate. Locally, Lifecycle-Aware Eviction monitors the ongoing residual utility of context segments, enforcing a conservative batch-turn schedule to offload content segments only when task relevance expires. Experiments on PinchBench and Claw-Eval under both isolated and continuous modes demonstrate that TokenPilot reduces costs by 61% and 56% in isolated mode, and 61% and 87% in continuous mode, while maintaining competitive performance compared to prior systems. TokenPilot has been integrated into LightMem2 at https://github.com/zjunlp/LightMem2.

Buqiang Xu, Zirui Xue, Dianmou Chen +12

1Zhejiang University · University of Electronic Science and Technology of China · 3Xi’an University of Electronic Science and Technology +1

LLM Agent Memory KV-Cache Management Long-Horizon LLM Agents LLM Context Management Context Compression

Ask PDF

Jul 7, 2026

Akashic: A Low-Overhead LLM Inference Service with MemAttention

Recent LLM-based agent systems continuously accumulate context across multi-turn interactions, tool invocations, and cross-session workflows. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality. We propose Akashic, a low-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross-chunk evidence without repeatedly rewriting the full history. Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead. Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10.2 points, throughput by up to 1.21x, and sustainable request rate by up to 1.88x over strong prior memory baselines.

Yang Liu, Zhaokai Luo, Huayi Jin +7

Xiaohongshu Inc., China · ShangHai JiaoTong University · Peking University

LLM Serving Memory-Augmented Language Models Memory-Efficient Inference LLM Context Management Memory-Augmented Language Model Agents

Ask PDF

Abstract

LLM Context Management

Explore similar work

Contextual Memory Virtualisation: DAG-Based State Management and Structurally Lossless Trimming for LLM Agents

TokenPilot: Cache-Efficient Context Management for LLM Agents

Akashic: A Low-Overhead LLM Inference Service with MemAttention