Expert offloading enables large mixture-of-experts (MoE) language models to run locally, but conventional batching stalls the GPU while missing experts are fetched. A popular remedy, known as prefetching, is to predict what experts will be needed soon and fetch them using idle link bandwidth. Imprecise prediction, however, wastes bandwidth and GPU memory on fetching unnecessary experts. Our key finding is that prefetching can be harmful when there is an expert cache in GPU memory. This is because as the cache includes more necessary experts, expert prediction becomes less precise. Our key insight is that proper request scheduling beats prefetching. Building on this insight, we present Bevy, an MoE inference system based on runahead inference for concurrent requests. A request whose experts are available can run ahead to later layers while other requests wait for fetches, revealing the experts it needs next. Bevy can then fetch the missing experts so identified while computing ready requests, overlapping computation and fetching without predicting expert selections. Running requests separately, however, repeats reads of shared weights in GPU memory and increases compute time. To limit this cost, Bevy's self-balancing scheduler batches ready requests at each layer and prioritizes lagging requests, adapting their grouping as expert availability changes. Across two local platforms, Bevy delivers up to 3.46× the decode throughput of llama.cpp and up to 1.55× that of the fastest prefetching baseline.
Figures & tables
Figure 1: Timeline of two concurrent requests, R1 and R2. (a) With batching, the GPU idles during fetches and the link idles during computation. (b) Same as (a), but predicted experts are prefetched during computation. Because prediction is never perfect, prefetching unnecessary experts wastes both bandwidth and GPU memory. (c) Each request runs without waiting for the other’s fetches, so fetches overlap with computation.
Figure 2: Prefetch precision under caching at 80% coverage of nonresident target-layer demand. To test whether the smaller fraction of needed candidates alone explains the decline, we compare the actual cache (solid) with a randomized cache (dashed). For each cache snapshot, the randomized cache holds the same numbers of needed and unneeded experts as the actual cache, selected randomly within each group. The dotted line is the no-cache reference. Region (1) shows the precision drop from this reference to the randomized cache; region (2) shows the additional drop to the actual cache.
Figure 3: Example timeline of runahead inference with three requests that reach layer l together. (a) Each request runs without waiting for the other requests’ fetches. (b) Same as (a), but R2 and R3 are grouped. (c) Same as (a), but the experts of R3 are fetched before those of R2.
Figure 4: Compute and fetch time per step for six concurrent requests (Gemma4-26B-A4B, Agentic Coding, 30% expert cache, Apple M6). (a) Compute time when the requests run in 1, 2, 3, or 6 groups. (b) Fetch time of each step over 256 steps.
Figure 5: Accumulated extra misses over the mean for four concurrent requests ranked by their misses in layers 0–3 (Qwen3.6-35B-A3B, ShareGPT, 30% expert cache).
Figure 6: System overview of Bevy.
JB
Blocked jobs with remaining missing experts
JR
Ready jobs
l(j)
Layer of job j
m(j)
Number of remaining missing experts of job j
Algorithm 1 Bevy’s Scheduling Policy
Figure 7: Operation of opportunistic layerwise grouping. (a) With many misses, only R2 is ready when R1 finishes, so R2 runs alone. (b) With few misses, R2, R3, and R4 are all ready and run together as one batch.
Figure 8: Throughput as the number of concurrent requests grows. The numbers above the bars give Bevy’s speedup over the fastest baseline. Even without prefetching, Bevy achieves the highest throughput in every configuration.
Figure 9: Throughput across expert cache sizes. As the cache grows, prefetch precision drops and the prefetching baselines approach llama.cpp, while Bevy achieves the highest throughput at every cache size.
Figure 10: (a) Step time split by whether the link and the GPU are busy or idle. (b) Expert fetches per step: missing experts fetched on demand, and prefetched experts that the step needed or wasted. Bevy keeps the link and the GPU busy together for most of the step while fetching only the experts that the requests need.
Figure 11: Mean step time against the number of missing experts per step, for static groupings and Bevy’s opportunistic layerwise grouping. Which static grouping is fastest depends on the number of missing experts, while Bevy achieves the lowest step time at every point by adapting groups to the missing experts at each layer.
Figure 12: (a) Change in GPU busy time, GPU idle time, and step time relative to Bevy when the fetch order, the compute order, or both are changed. (b) Share of groups by group size, the number of jobs that run together as one batch. Lowest-layer-first compute ordering creates batching opportunities: it forms larger groups, and the GPU busy time it saves outweighs the increase in GPU idle time.
Figure 13: (a) Throughput of Bevy and Bevy + Prefetch as the number of concurrent requests grows. (b) Expert prefetches per step, split into needed and wasted. Prefetching does not improve the throughput of Bevy: with few requests, many prefetches are wasted, and with more requests, the link has no idle time left for prefetching.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.
The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-specific expertise in LLMs and scaling model capacity without proportionally increasing their computational overhead. However, MoE inference often suffers from suboptimal GPU utilization, load imbalance, and elevated latency arising from multiple tokens waiting on the same experts for their computation which arises from sparsity of expert activation. To address these challenges, we propose a dynamic expert replication strategy that predicts which experts are likely to be overloaded and replicates them for upcoming batches of tokens. The replicated experts process batch tokens concurrently across layers, which leads to improved parallelism, shorter GPU idle time, and significantly faster inference. Experimental evaluations conducted on large-scale MoE models, including Switch-base-128 and Switch-base-256, demonstrate that our method achieves near-complete GPU utilization (approx 100%), leading to upto 3x improvement in inference speed while preserving approximately 90-95% of the performance of baseline architectures
Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1
Mixture-of-Experts (MoE) models reduce inference computation through sparse expert activation, but their large expert memory footprint often exceeds GPU capacity, making expert offloading necessary for memory-constrained inference. However, existing systems typically handle GPU-missed experts by either transferring them to the GPU, incurring substantial PCIe transfer latency, or executing them on the CPU, making the slower CPU computation the bottleneck. To address these limitations, we present \method, a CPU-GPU collaborative MoE inference system that adaptively assigns these missed experts to the CPU or GPU for efficient execution. By exploiting differences in expert token-loads, \method selectively transfers high-load experts to the GPU while executing low-load experts on the CPU. To minimize inference latency, \method adaptively adjusts expert assignments to balance the CPU computation path against the GPU transfer-and-computation path, allowing both execution paths to complete as early as possible. Extensive experiments on three representative MoE LLMs show that \method improves prefill speed by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline across the evaluated batch sizes, with consistent gains under different GPU-memory budgets.