Organizations: Sun Yat-sen University Zhuhai, China · Independent Researcher Shanghai, China · Independent Researcher Hangzhou, China · EPFL Lausanne, Switzerland · Peng Cheng Laboratory Shenzhen, China
Modern LLMs strain device memory through large model weights and attention states, particularly in resource-constrained scenarios. Existing systems separately provision memory for model weights and attention states, imposing fixed fixed boundaries that cannot adapt to the changing memory demands of workloads. Under heavy workloads, insufficient state capacity limits request concurrency and reduces throughput; under light workloads, idle memory in one region cannot be reallocated to another, wasting device memory. Meanwhile, modern LLM architectures introduce heterogeneous tensors with diverse layouts and lifetimes, further complicating memory management. These problems motivate unified memory management for heterogeneous tensors. However, a shared pool alone does not make memory directly reusable across tensors with different allocation granularities and execution layouts. This paper presents Rimuru, a modern LLM serving system that unifies memory management through elastic expert offloading. Rimuru establishes a shared page pool to enables elastic memory sharing across heterogeneous tensors. To accommodate different allocation granularities, Rimuru introduces memory views that allow each tensor type to interpret the shared memory space using its own page size and page table. Furthermore, Rimuru combines huge-page alignment with a multi-level page allocator that splits and merges pages, enabling cross-type capacity reuse. A page-table-aware fused MoE kernel directly accesses non-contiguous expert weights, avoiding copy overhead. Building on these mechanisms, Rimuru employs a workload-aware adjuster that dynamically reallocates pages across tensor types according to runtime demands. Implemented on top of vLLM, Rimuru remains compatible with modern LLMs. Evaluations show that Rimuru achieves up to 19.03x baseline throughput and reduces TTFT by 98.1% on average.
Figures & tables
Figure 1. Different memory management. (a) Conventional serving systems leave little memory for attention states. (b) Expert offloading creates headroom but still exist a fixed boundary between the expert cache and attention states. (c) Rimuru with unified memory management.
Figure 2. A comparison of the architecture of a classic transformer-based LLM and a representative modern LLM.
Figure 3. Request arrival rate and sequence length vary over time in a production workload.
Figure 4. GPU-memory usage and request concurrency while serving Qwen1.5-MoE-14B on one NVIDIA RTX 5090. All configurations serve the same ShareGPT requests ( Dataset, 2023 ) , submitted at 10 requests/s, with GPU memory utilization capped at 0.9. Stacked areas show memory attributed to model weights and the KV cache; the line shows running-request concurrency. The horizontal axis reports forward steps.
Tensor Type
Tensor Shape (single)
Qwen3-30B-A3B
Qwen3.5-35B-A3B
Expert
w1, w3
[2048, 768]
[2048, 512]
w2
[768, 2048]
[512, 2048]
KV cache
[2, 4, 128]
[2, 2, 256]
Mamba state
conv
/
[3, 8192]
SSM
/
[32, 128, 128]
Table 1. Tensor heterogeneity in representative modern LLMs. Shapes are reported for each expert‘s three projection matrix, the key and value tensors for one token, and an individual SSM or convolutional state.
Figure 5. System overview of Rimuru.
Figure 6. Multi-view memory layout of Rimuru. Each tensor type has its own tensor-specific view over the shared storage.
Figure 7. Multi-level page allocation and subdivision in Rimuru.
Figure 8. Single buffer vs. Dual buffers. Design for overlapping expert prefetch and execution, avoiding local DtoD copy.
Figure 9. Comprehensive serving evaluation of expert offloading. Columns show the four model and hardware configurations, and rows report TTFT, TPOT, and output-token throughput as request rate increases.
Figure 10. Serving evaluation of SGLang, vLLM, and Rimuru.
Figure 11. Ablation study of Rimuru. UMM: Unified Memory Management, CA: Cache Adjuster, DF: Discrete FusedMoE.
On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the operating system. When a new inference request arrives, the inference system must restore the evicted memory through slow storage reads or recompute the entire KV cache, severely degrading responsiveness. To address this, we present mzCache, an on-device LLM inference system with specialized memory management for multitasking environments. Under unpredictable memory pressure, mzCache elastically evicts LLM memory and leverages the unified memory of mobile SoCs to enable zero-wait inference on the GPU with concurrent CPU-side restoration. mzCache realizes this through restoration-oriented memory management: LLM memory is partitioned into fine-grained shared buffers to enable partial eviction and restoration with concurrent cross-processor access, while hybrid swap and backward-out eviction policies ensure low-latency restoration from any eviction state. Implemented on llama.cpp and deployed as an Android application, mzCache achieves 2.1-5.5× reduction in Time-to-First-Token compared to storage-backed partial offload and demonstrates its effectiveness in real multitasking scenarios.
Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning. We show that these optimizations can be unified into a four-step memory processing pipeline: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference. Through systematic profiling, we identify a 22%-97% memory processing overhead in LLM inference and strong heterogeneity in its computational characteristics. Motivated by this insight, we argue that heterogeneous systems are well-suited to accelerate memory processing and thus end-to-end inference. We demonstrate this approach on a GPU-FPGA system by offloading sparse, irregular, and memory-bounded operations to FPGAs while retaining compute-intensive operations on GPUs. Evaluated on an AMD MI210 GPU and an Alveo U55C FPGA, our system is up to 2.2× faster and achieves up to 4.7× less energy across multiple LLM inference optimizations than the GPU baseline (similar results hold on NVIDIA A100). These results establish heterogeneous systems as a practical direction for efficient LLM memory processing and inform future heterogeneous hardware design.
Zifan He, Rui Ma, Yizhou Sun +1
Department of Computer Science, University of California, Los Angeles · Microsoft Research Asia
Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects. At the same time, workloads such as LLM inference exhibit highly dynamic memory demands, which can cause one GPU to exhaust its local memory while others remain underutilized. This mismatch motivates a model of elastic resource sharing across GPUs. We present EMA, a memory sharing system that allows GPUs within a server to borrow and reclaim memory from each other, forming an elastic pool of capacity. EMA ensures performance transparency for both borrowers and lenders. For borrowers, prefetching hides remote access costs so that applications experience remote and local memory as indistinguishable in performance. For lenders, borrowed resources remain reclaimable on demand, guaranteeing that performance never falls below that of static partitioning. While our design focuses on memory, the same principle naturally extends to other GPU resources. Our evaluation shows that EMA improves individual user throughput by up to 52%, achieves 96% of the throughput of a system provisioned with 2X capacity, and maintains latency similar to the static local baseline.