cs.LGSep 29, 2026

Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging

Authors: Sanjali Yadav, Bahar Asgari

Organizations: University of Maryland, College Park

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-constrained, single-GPU systems. However, this benefit is difficult to realize because expert parameters dominate memory, and token-level routing is dynamic, unpredictable, and skewed. Prior work using offloading and caching remains fundamentally reactive, as systems wait for router outputs before moving experts, leading to inefficient cache utilization and an inability to overlap transfers with compute under tight VRAM budgets. To address these challenges, we propose Mira, an algorithm-system co-design that enables high-capacity MoE inference on a single GPU. Mira shifts from a reactive to a proactive stance by coupling predictive expert management with a tailored quantization format. It introduces lightweight per-layer predictors that anticipate expert usage two layers ahead, enabling proactive prefetching. These predictions feed a two-tier HOT+STAGE GPU cache managed by token-level routing telemetry to retain frequently used experts while staging predicted ones. To minimize transfer overhead, Mira implements a custom compression for expert parameters, which reduces metadata and improves packing efficiency, while minimally degrading accuracy. Mira is implemented as a fully integrated runtime that coordinates predictors, caching policies, and quantized transfers to maximize overlap between communication and compute. Our experiments show that Mira reduces expert-induced stalls. Compared against state-of-the-art baselines, Mira achieves a 5.71x speedup in average throughput on a memory-constrained GPU. It accelerates Time-to-First-Token by 11.71x and achieves a 3.84$x average speedup in beam search inference, demonstrating its effectiveness across diverse inference scenarios.

Figures & tables

Fig. 1: The architecture of dense transformer block compared to sparse MoE block.
Fig. 2: Aggregated expert activation patterns for Mixtral-8x7B [ 15 ] on four distinct MMLU tasks . Each heatmap shows the activation frequency (color) for each of the 8 experts (y-axis) across the 32 MoE layers (x-axis).
 FP16INT8INT8 (Mira)INT4INT3INT2
Perplexity ↓\downarrow3.93243.96953.95544.10485.26531147.5236
TABLE I: Perplexity on WikiText-2 for varying quantization bit widths.
Fig. 3: Top-15 expert activations across decode tokens on the movie recommendation dataset using Mixtral-8x7B. Each column tracks the activation frequency of one expert over a decode sequence.
Fig. 4: Mira’s Runtime Components: At runtime, Mira maintains a two-level GPU expert cache with HOT and STAGE regions. During inference, experts are fetched from this cache and processed using a custom kernel that fuses dequantization with FFN computation. A lightweight predictor forecasts which experts will be needed shortly and triggers asynchronous prefetching into the STAGE region. Meanwhile, token-level telemetry informs a rebalancer that periodically adjusts the HOT expert set for optimal reuse. Offline, on CPU side, all expert FFN weights are converted into a unified INT8 format.
Fig. 5: Mira’s memory layout. Compressed expert weights W1W_{1} , W2W_{2} , and W3W_{3} are combined (denoted by ++ ) on the CPU. The weights and a shared scale are packed (indicated by the dashed line) and transferred to the GPU.
Fig. 6: Timeline illustration of expert computation under varying hardware setups.
Fig. 7: Inference performance comparison of Mira+P (with prefetcher) and Mira (without prefetcher against baselines in Environment 1 (48GB VRAM), measured by Time-to-First-Token (TTFT) and throughput (Tokens/s) for varying input and output token lengths. Higher is better for Tokens/s; lower is better for TTFT.
Fig. 8: Inference performance comparison of Mira+P (with prefetcher) and Mira (without prefetcher against baselines in Environment 2 (24GB VRAM), measured by Time-to-First-Token (TTFT) and throughput (Tokens/s) for varying input and output token lengths. Higher is better for Tokens/s; lower is better for TTFT.
 Arc-ESCIQWinograndeTriviaQA
FP160.8700.9700.7600.57
Mira0.8710.9750.7500.54
TABLE II: Accuracy comparison on downstream tasks
Fig. 9: Mass-weighted Coverage@k by source layer for Mixtral-8x7B. The predictor selects the top-k experts to prefetch for a future MoE layer; Coverage@k reports the fraction of routed token mass that lands on those prefetched experts. Higher coverage indicates more effective prefetching.
Fig. 10: Mira’s performance gain over Fiddler with beam width scaling.
VRAM (GB)Mira (Tokens/s)Mira+P (Tokens/s)Speedup
161.702.531.49
242.773.871.40
323.654.481.23
405.267.291.39
488.979.661.08
TABLE III: Mira and Mira+P throughput across varying VRAM budgets.

Explore similar work

CardsList
May 18, 2026

CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution

The Mixture-of-Experts (MoE) architecture improves computational efficiency via sparse expert activation, but throughput-oriented inference faces substantial GPU memory pressure due to a significant parameter size and intermediate data. Prior works attempt to mitigate this using expert offloading with micro-batching or by offloading computation to the CPU. However, the fragmented workload resulting from micro-batching degrades operational intensity, causing expert execution to become memory-bound. Meanwhile, CPU offloading is constrained by slow PCIe transfers and its limited applicability to attention computation in the decode stage. Consequently, these inefficiencies prevent effective system utilization, severely restricting the end-to-end throughput of MoE inference. To address these challenges, this paper proposes CoX-MoE, an Advanced Matrix Extensions (AMX)-enabled CPU-GPU collaborative system that comprehensively optimizes MoE inference by combining coalesced expert execution with strategic workload orchestration for higher throughput. CoX-MoE introduces (i) a coalescing-aware orchestration policy to jointly optimize resource allocation by adopting ordinary batch, instead of micro-batch, for expert computation and selective attention offloading, and (ii) a static expert-aware stratification scheme that pre-assigns frequently activated experts to the GPU, mitigating PCIe transfer overhead and balancing workload for the CPU and GPU during inference. Compared to state-of-the-art frameworks, CoX-MoE delivers significant gains, achieving up to 7.1x and 2.4x higher throughput than FlexGen and MoE-Lightning, respectively.

Muyoung Son, Yi Chen, Seungjae Yoo +2

KAIST · Daejeon, Republic of Korea

Expert OffloadingGPU AccelerationMixture-of-Experts InferenceLLM Inference AccelerationHeterogeneous Computing
AskPDF
May 12, 2026

Fast MoE Inference via Predictive Prefetching and Expert Replication

The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-specific expertise in LLMs and scaling model capacity without proportionally increasing their computational overhead. However, MoE inference often suffers from suboptimal GPU utilization, load imbalance, and elevated latency arising from multiple tokens waiting on the same experts for their computation which arises from sparsity of expert activation. To address these challenges, we propose a dynamic expert replication strategy that predicts which experts are likely to be overloaded and replicates them for upcoming batches of tokens. The replicated experts process batch tokens concurrently across layers, which leads to improved parallelism, shorter GPU idle time, and significantly faster inference. Experimental evaluations conducted on large-scale MoE models, including Switch-base-128 and Switch-base-256, demonstrate that our method achieves near-complete GPU utilization (approx 100%), leading to upto 3x improvement in inference speed while preserving approximately 90-95% of the performance of baseline architectures

Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1

Iowa State University Ames, USA

Expert Load BalancingGPU AccelerationMixture of ExpertsMixture-of-Experts InferenceLLM Inference Acceleration
AskPDF
Apr 3, 2026

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the key-value cache in large-batch, long-output offline workloads. We present FluxMoE, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new expert paging abstraction. FluxMoE combines PagedTensor for transparent remapping, a bandwidth-balanced hierarchy spanning losslessly compressed GPU memory and host DRAM, and a budget-aware residency planner. Unlike CPU-GPU co-inference and whole-layer offloading, FluxMoE streams weights on demand while keeping expert computation on GPUs. We implement FluxMoE atop vLLM and evaluate it on three MoE models. For GLM-4.5 on 8×\timesH20 GPUs, FluxMoE delivers up to 7.2×\times vLLM's throughput and 79.0% lower average Time-Per-Output-Token (TPOT), without measurable model-quality loss using lossless compression. For Mixtral-8×\times7B-Instruct on 2×\timesL40S GPUs, where weight-resident vLLM cannot fit, FluxMoE delivers 4.3×\times KTransformers's throughput and 29.1% lower average TPOT.

Qingxiu Liu, Yongchao He, Runhan Jiang +4

CUHK · Unaffiliated · SCITIX +1

Expert OffloadingGPU AccelerationMixture of ExpertsMixture-of-Experts InferenceMemory-Efficient InferenceLLM Inference Acceleration
AskPDF