MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference

Joris Köster; Zixuan Liu; Siavash Khajavi; Zizhan Zheng

cs.CLMar 27, 2026

MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference

Authors: Joris Köster, Zixuan Liu, Siavash Khajavi, Zizhan Zheng

Organizations: Department of Computer Science, Aalto University, Finland · Department of Computer Science, Tulane University, LA, USA · Department of Industrial Engineering and Management, Aalto University, Finland

Abstract

Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline.

Figures & tables

Figure 1: Overview of MemBoost. For each incoming query, the AME retrieves a set of relevant memory entries. The MC then either composes an answer from the retrieved results or escalates to the Oracle. When the Oracle is used, the MC decides whether to write the new answer back into the AME for future reuse.

Component	Hyperparameter	Value
Traffic simulation	Workload distribution	Zipfian
	Zipf exponent ( $\alpha$ )	$\{0.8,1.1,1.4\}$
	Number of requests ( $N$ )	5,000
	Random seed	1
Associative Memory Engine	Embedding model	all-MiniLM-L6-v2
	Similarity metric	Cosine (FAISS inner product)

Table 1: System and hyperparameter configuration for the continuous serving simulation.

Model	Accuracy (%)
	Zipf 0.8	Zipf 1.1	Zipf 1.4
Baselines
Qwen3.5-2B	50.0	43.5	37.1
Ministral-3-3B-Instruct-2512	53.8	46.4	38.2
Qwen3-4B-Instruct-2507-FP8	74.5	75.6	80.5
Qwen3-14B-FP8-dynamic (Oracle)	76.4	79.9	85.0

Table 2: Accuracy (%) of different methods on MMLU-Pro under Zipf-sampled query streams with varying repetition rates (Zipf

\alpha

Figure 2: Average memory-use rate

\overline{I}_{t}

(200-step window) over a 5,000-step Zipf-sampled query stream. Higher

\overline{I}_{t}

indicates more queries served from AME and fewer oracle calls, implying lower total inference cost.

Figure 3: Response latency over time under Zipf-sampled workloads (average over the previous 100 steps). MemBoost reduces latency relative to the oracle-only baseline as an increasing fraction of queries are served from AME.

LLM Inference Serving

Large language model inference serving studies how to run model requests efficiently while meeting latency, throughput, cost, and energy goals.

See all 72 papers

InferScale: GPU-Native KV Injection for Personalized LLM Serving
Jul 29, 2026Peter Li, Prashant PandeyLLM Serving Memory-Augmented Language Models
Northeastern University Boston, Massachusetts, USA
Akashic: A Low-Overhead LLM Inference Service with MemAttention
Jul 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +7LLM Serving Memory-Augmented Language Models
Xiaohongshu Inc., China · ShangHai JiaoTong University · Peking University
MeMo: Memory as a Model
May 14, 2026Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong +6Retrieval-Augmented Generation Memory-Augmented Language Models
Institute of Data Science, National University of Singapore, Singapore · Integrative Sciences and Engineering Programme, NUSGS, Singapore · Agency for Science, Technology, Research (A*STAR), Singapore +6

Jul 29, 2026J/K move · Enter open · S save

InferScale: GPU-Native KV Injection for Personalized LLM Serving

Peter Li, Prashant Pandey

Northeastern University Boston, Massachusetts, USA

LLM Serving Memory-Augmented Language Models KV Caching Persistent Memory for Language Models LLM Inference Acceleration KV-Cache Reuse

Open paper AskPDF arXiv

Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.

Abstract

LLM Inference Serving

Explore similar work

InferScale: GPU-Native KV Injection for Personalized LLM Serving

Akashic: A Low-Overhead LLM Inference Service with MemAttention

MeMo: Memory as a Model