In this work, we identify and address the core challenges of agentic memory management in LLM serving, where large-scale storage, frequent updates, and multiple coexisting agents jointly introduce complex and high-cost approximate nearest neighbor (ANN) searching problems. We present Pancake, a multi-tier agentic memory system that unifies three key techniques: (i) multi-level index caching for single agents, (ii) coordinated index management across multiple agents, and (iii) collaborative GPU-CPU acceleration. Pancake exposes easy-to-use interface that can be integrated into memory-based agents like Mem-GPT, and is compatible with agentic frameworks such as LangChain and LlamaIndex. Experiments on realistic agent workloads show that Pancake substantially outperforms existing frameworks, achieving more than 4.29x end-to-end throughput improvement.
Figures & tables
Figure 1: Memory-based workflow of agentic LLMs.
Figure 2: An example of multi-agent memory in Pancake.
Figure 3: Memory-based agents and their workflows.
Figure 4: Direct in-place updates scatter the new vectors into a large number of existing clusters, leading to degradation in efficiency and recall. A naive solution is to leverage intra-agent locality and maintain dedicated clusters for a agent.
Figure 5: For more complex workloads, locality across multiple reasoning steps of different requests can be observed. This makes naive dedicated clusters for the agent inefficient, as it fails to capture step-wise clustering.
Figure 6: Coarse search costs with different index methods.
Figure 7: Coarse and Fine Search Challenges in Multi-Agent Memory. (a) Coarse search overhead grows rapidly as the number of agents increases. (b) When two agents access the same cluster in the static memory, their access patterns exhibit non-uniform distributions; circles denote accessed vectors, and stars denote the centroids formed by those vectors.
Figure 8: Comparison of operation costs on GPU and CPU, including (a) search time and (b) data transfer and allocation. The results are sampled on the MS MARCO [ 51 ] dataset.
Figure 9: Three-level memory index cache to optimize search efficiency, with FSM-based modeling for access patterns.
Figure 10: Multi-agent index management with hybrid graph and agent-specific pattern profiling on shared clusters.
Figure 11: GPU-CPU coordinated index management to enable hotspot cluster computation acceleration.
Figure 12: End-to-end throughput comparison between Pancake and other agentic frameworks, in a single agent scenario across four different access patterns. The experiments are conducted with vLLM [ 35 ] for Llama models [ 62 ] and API calls for GPT-5.
Figure 13: End-to-end throughput comparison in two-agent mixed workload, conducted with Llama3.1-8B.
Figure 14: Scaling behavior of end-to-end throughput with an increasing agent number. The experiments are conducted with Llama3.1-8B.
Figure 15: Query throughput of Pancake and existing vector database implementations, with the batch size set as 8.
Figure 16: Tradeoff between recall and query latency over different indexing strategies.
Figure 17: Number of scanned vectors to achieve fully recall of top-5 memory items. The insertion-to-search ratio is 1:1.
Figure 18: Efficiency improvements from multi-level index management on (a) coarse index search cost and (b) total computation cost.
Figure 19: (a) GPU speedups under varying pre-allocated GPU cache sizes. (b) Computation latency of each query over the input workload with AgentGym dataset and 10GB GPU memory cache size. The insertion-to-search ratio is 1:1.