CACHEFORGE: LLM-Guided End-to-End Generative Cache Replacement Policy for Performance and Hardware Efficiency

Kaushal Mhapsekar; Bita Aslrousta; Brijesh Kumar Bhayana; Paula Contreras; Azam Ghanbari; Ethan Goodman; Anna Andriiko; Samira Mirbagher Ajorpaz

cs.AROct 6, 2026

CACHEFORGE: LLM-Guided End-to-End Generative Cache Replacement Policy for Performance and Hardware Efficiency

Authors: Kaushal Mhapsekar, Bita Aslrousta, Brijesh Kumar Bhayana, Paula Contreras, Azam Ghanbari, Ethan Goodman, Anna Andriiko, Samira Mirbagher Ajorpaz

Organizations: Department of Electrical and Computer Engineering North Carolina State University, Raleigh, NC, USA

Abstract

Modern cache replacement designs saturate because they operate within fixed representational structures, hand-crafted and heuristic based feature-engineered predictors, or offline imitation models that cannot generate new decision logic on their own. At the same time, replacement is shaped by the causal interaction of prefetching, thrashing, spatial locality, and access-type behavior, producing an enormous design space that is difficult to traverse manually. Prior approaches typically rely on heuristics, parameter tuning, or imitation of an offline optimal policy, capturing correlations rather than synthesizing new mechanisms. As a result, their performance gains often plateau and they overfit under dynamic workload conditions. CACHEFORGE is the first framework to evolve cache-replacement policies end-to-end by embedding a large language model inside a governed hardware-aware loop. In each iteration, the LLM proposes new C++ replacement logic, the policy is evaluated under a trace-based CRC-2 ChampSim simulator, and the framework enforces feasibility through reward shaping, structural checks, dynamic mutation, temperature scheduling, and cross-policy crossover. This closed-loop generation-evolution loop specifically designed for cache replacement policy enables the discovery of compact policies that satisfy hardware constraints while exploring algorithmic transformations beyond fixed predictor structures. Across SPEC CPU2006, CACHEFORGE outperforms all CRC-2 baselines. It improves the total hit rate by 27.36%, 19.69%, 13.72%, 13.15%, 11.83%, and 5.73% over MPPPB, ReD, Hawk-eye, SHiP++, LIME, and LRU, respectively. On memory-intensive workloads, it increases IPC by 10.15%, 7.89%, 6.34%, 3.64%, 3.12%, and 2.71% over LRU, MPPPB, LIME, ReD, SHiP++, and Hawkeye.

Figures & tables

Fig. 1: CacheForge system diagram.

Component	Configuration
Processor	1 core; 4 GHz; 6-wide fetch/decode/execute; 4-wide retire; 256-entry ROB; gshare branch predictor
L1 I/D-Cache	32 KB, 64 sets, 8 ways, 4-cycle latency, next-line prefetcher;
L2 Cache	256 KB, 512 sets, 8 ways; 12-cycle latency, PC-based stride prefetcher;
LLC	2 MB, 2048 sets, 16 ways; 32-cycle latency;
L1 TLB (I/D)	4KB; 16 sets, 8 ways
L2 TLB	96KB; 128 sets, 12 ways

TABLE I: Processor and Memory Configuration.

Panel of Figure 3 — Fig. 2: Anti-correlation of IPC and Cache Hit Rate observed.

Panel of Figure 4 — Fig. 3: Optimizing for IPC over iterations.

Panel of Figure 5 — Fig. 4: Speedup of CacheForge relative to LRU across SPEC workloads, compared against state-of-the-art baselines.

Panel of Figure 6 — Fig. 5: Cache Hit Rate improvement using 4o-mini+Gemini 2.5 pro. The first vertical line shows the first instance where CacheForge surpasses a state-of-the-art policy and the second vertical line signifies the model change from 4o-mini to Gemini 2.5 pro.

Panel of Figure 7 — Fig. 6: Comparison of Load and RFO Hit Rate evolution over iterations. Two blue tones denote the access types, with dashed lines indicating RFO progression.

Panel of Figure 8 — Fig. 7: Sorted cache hit rates for policies evolved under Hit Rate, Average, and Weighted Average rewards and sorted IPC for policies evolved under IPC, and Average rewards.

Fig. 8: Successful replacement policy hardware overhead budget alignment

Panel of Figure 10 — Fig. 9: Storage Overhead (KB) of CacheForge Proposed Policies vs. Baseline Policies

Panel of Figure 11 — Fig. 10: IPC per KB improvement over Multiperspective.

Novel Design Component	Objective	Impact	CacheForge Capability
Reinforcement-Learning	Learn eviction value online	Qualitative only	Explores unconventional algorithmic spaces
Q-Value Candidate Ranking	Score all candidates	High granularity eviction decisions	Applies ranking-based eviction strategies
Online Reward Adaptation	Adjust RL policy using hits/misses	Adaptive learning	Experiments with reinforcement learning
Dead-Block Prediction	Identify low-reuse lines	Improves reuse filtering	Rediscovers classical dead block mechanisms
Evicted-Address Filter	Detect recently-dead lines	Reduces pollution	Uses compact deadness filters effectively
Prefetch Usefulness	Reduce unused prefetches	Reduces cache pollution	Models prefetch awareness

TABLE II: Novel Design Components explored by CacheForge.

Fig. 11: Heatmap showing the impact of novel features explored in CacheForge over Hit Rate.

Component	Description	Multiperspective					CacheForge					Tuning
		A	B	E	W	X	A	B	E	W	X
Perceptron Features
F_PC	PC: Bits B-E of the PC of the $W$ th memory access instruction. A is the recency-stack position beyond which the block is considered dead.	7	14	43	11	0	6	0	16	8	0	Changed
F_PC		16	3	11	16	1	8	6	22	3	0	Changed
F_PC		10	1	53	10	0	10	4	20	1	0	Changed
F_PC		16	8	16	5	0	12	4	20	0	2	Changed

TABLE III: Design comparison of CacheForge vs. Multiperspective.

Fig. 12: Seed analysis across various replacement policies.

Panel of Figure 16 — Fig. 13: Impact of number of seeds to cross over

Fig. 14: Performance across various LLM models

Fig. 15: Impact of memory-augmented design.

Fig. 17: Impact of dynamic mutation rate and temperature.

Fig. 18: Cache hit rate distributions for the undergraduate and graduate cohorts. and the LRU baseline.

Explore similar work

Nov 12, 2024cs.PF

Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies

LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources. While LLM inference systems are capable of serving millions of requests from multiple users, they often lack theoretical models to determine whether they achieve the performance upper bounds of underlying hardware resources. Beyond online workload serving, merely analyzing existing systems-or developing yet another one-is both GPU-intensive and labor-intensive. This paper provides a comprehensive survey of LLM inference systems, focusing on their cache management policies and availability. We then show that simulations can be an effective tool to save GPU hours in the development and analysis phase of inference systems, revealing useful insights for developing better inference techniques, unlike how existing studies used simulations to find the best parameters inside a given system. Finally, we provide theoretical tools to estimate the optimal performance and formulate new ideas. Based on the theoretical analysis, especially on the cache management in LLM inference, we propose a simple yet effective cache replacement policy that can be easily plugged into existing preemptive schedulers and systems. We show that such a simple policy inspired from database systems can substantially save GPU hours in actual inference systems on online workloads. We share our experience submitting a journal paper to a database venue in November 2025 for anyone considering a similar path.

Kyoungmin Kim, Jiacheng Li, Kijae Hong +3

EPFL, Switzerland · NUS, Singapore · POSTECH, South Korea

LLM Inference Optimization Graphics Processing Unit Resources Cpu-Gpu Hybrid Designs Workload Large Language Model Memory

Ask PDF

Jul 1, 2026cs.DB

When Classic Cache Policies Fail: Learning-Augmented Replacement for Semantic Retrieval Buffers

LLM agents increasingly rely on retrieval buffers to store and reuse past experience, yet the cache management policies governing these buffers remain largely ad-hoc. We formalize this as an online semantic cache replacement problem with switching costs, where items are matched by embedding similarity and hit quality is continuous rather than binary. Through experiments on two datasets from MemoryBench-Full (LoCoMo, DialSim) with 8 replacement policies, we reveal a surprising finding: classic heuristics (LRU, LFU) \emph{consistently underperform} the naive FIFO baseline on semantic workloads, due to the absence of temporal locality and frequency concentration. We propose SOLAR, a learning-augmented framework that derives modification timing from regret accumulation (achieving

\sim

17% modification rate) and content selection from Bayesian online learning over implicit retrieval feedback. We prove SOLAR achieves a constant competitive ratio

\leq 3

, independent of cache size and horizon (vs.\

Ω(K)

for FIFO), and eviction regret

O(\sqrt{KT\log T})

, matching the

Ω(\sqrt{KT})

lower bound up to logarithmic factors. Experiments demonstrate 5--75% relative improvement over FIFO at tight cache sizes, with a clearly characterized phase transition at the working set boundary. Synthetic experiments with 5000-item pools further reveal an inverted-U relationship between pool size and retrieval quality, justifying capacity constraints as a retrieval noise phenomenon rather than a storage limitation.

Yushi Sun, Bowen Cao, Wai Lam

LIGHTSPEED, Tencent · Shenzhen, China · The Chinese University of Hong Kong +1

Cache Feedback Regret

Ask PDF

May 25, 2026cs.OS

LearnedCache: An eBPF-Integrated Perceptron-Based Eviction Policy for the Linux Page Cache

Linux is the foundation of the digital age, accounting for the majority of the cloud and mobile OS markets. Any device that runs Linux uses the Linux page cache, a central pillar in OS and application performance, serving to reduce extraneous disk access. Many page cache eviction policies have been developed but remain bound by the rigidity of heuristics. The rise of AI-driven tools in recent years, melded with the ever-increasing variety of workloads for Linux devices, sets the stage for machine-learning-driven cache eviction policies. Promising research has been done in this field, but only in the field of user-space applications such as CDNs. We develop LearnedCache, an eBPF-integrated single-layer perceptron-based cache eviction policy for the Linux page cache, trained on real kernel data from diverse workloads. We demonstrate median AUCs of nearly 80% over multiple linear models modeling page reuse time, then take a step further by embedding these models inside the Linux kernel for real-time performance evaluation. Through statistical testing over 50 paired trials against a baseline of FIFO for each workload, LearnedCache reveals that machine-learning-derived cache eviction policies are practical in the Linux kernel under representative empirical workloads and are able to surpass conventional FIFO by statistically significant margins of up to 10% in insertion rate, a frequency-adjusted derivation of cache hit rate, in specific workloads while incurring minimal overhead.

Zejia Qi

Eviction Cache Preemption Workload Robotwin

Ask PDF