EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs
Authors: Junming Zhang, Zhenzhe Zheng, Fan Wu, Xiaoyao Huang, Jie Wu
Organizations: Shanghai Jiao Tong University Shanghai, China · Cloud Computing Research Institute, China Telecom China · Temple University Philadelphia, Pennsylvania, USA
Mobile vendors and application developers increasingly deploy LLMs on smartphones for diverse prefill-only services. Yet current systems rely mainly on dense models whose regular computation maps efficiently to mobile NPUs, leaving more capable MoEs underused. MoE prefill does not fit mobile NPUs: NPU graphs are fixed at compile time, yet MoE picks experts at runtime; and one request touches most experts, more than a phone can hold in memory. We present EStream, which resolves both by separating what the NPU must fix from what MoE decides at runtime. A single compiled expert graph serves every expert, with each expert's routed tokens and weight address bound at call time, so dynamic MoE execution runs entirely on the NPU without padding or CPU/GPU fallback. Expert virtualization keeps the expert pool in UFS flash storage and pages it through a fixed-size NPU-addressable arena, group by group, with loading hidden behind computation, so memory is bounded by the arena rather than by the model. It further introduces a hardware-aware configuration algorithm that automatically configures the UFS--NPU pipeline and maximizes loading--computation overlap. Across 18 comparative settings covering three 7B--16B MoEs and 256--4,096-token prompts, we evaluate EStream on a commercial Snapdragon smartphone. Compared to the fastest baseline at each setting, EStream achieves a 2.25--27.57X pure-prefill TTFT speedup and reduces peak physical memory by 1.19--12.29X. EStream further scales to MoE models with up to 46.7B parameters.
Figures & tables
Figure 1. On-device prefill-only applications process private data, but MoE prefill remains inefficient on mobile devices. A two-panel diagram. The left panel shows private user profiles, documents, and reviews entering a phone and producing sentiment, correctness, and intent decisions. The right panel shows a long tensor routed to nearly all experts, irregular matrix tiles that do not map cleanly to a mobile NPU, and slower, power-hungry CPU and GPU alternatives.
Figure 2. Longer prompts expose more expert computation to NPU acceleration. Each bar stacks the execution latencies of all experts during OLMoE CPU prefill at one input length. Stacked per-expert projection latency for OLMoE CPU prefill from 1 to 2048 tokens. Each bar stacks experts in increasing routed-token count, with color indicating route count. Curves mark the boundaries above which experts receive at least 32 or 64 routes. The region above these curves grows with prompt length and is labeled as an NPU acceleration opportunity.
Figure 3. MoE prefill activates most experts, which dominate the model weights. (a) Activated layer–expert pairs. (b) Q4 expert and non-expert weight sizes. Two plots characterize the expanded MoE parameter working set. Panel (a) shows the percentage of layer–expert pairs activated by C4 prefixes from 1 to 4096 tokens; OLMoE and LFM2.5 approach 100 percent, while Qwen3 reaches 92.4 percent. Panel (b) compares the Q4 payload of non-expert and expert weights; expert weights account for more than 91 percent in all three models.
Figure 4. The system overview of EStream. A two-panel diagram. The left panel shows private user profiles, documents, and reviews entering a phone and producing sentiment, correctness, and intent decisions. The right panel shows a long tensor routed to nearly all experts, irregular matrix tiles that do not map cleanly to a mobile NPU, and slower, power-hungry CPU and GPU alternatives.
Figure 5. Topology-invariant graph sharing improves efficiency. (a) A shared graph binds data at runtime. (b) Latency and memory versus per-expert QNN contexts. Two subfigures. The left compares parameter-bound expert graphs with one shared graph whose parameter buffer is selected at runtime. The right uses grouped bars to compare EStream and per-expert QNN contexts in expert-path latency and peak physical memory from 64 to 2048 input tokens.
Figure 6. Expert-loading schedules for 1,024-token OLMoE prefill. Earlier prefetch increases I/O–compute overlap and lowers bubble rate, 1−max(Tio,Texec)/Ttotal . Three timelines compare expert loading and NPU execution. On-demand loading alternates orange loading blocks and blue expert-compute blocks and has a 34.41 percent bubble rate. Prefetch after routing overlaps the two after non-FFN execution and has a 21.36 percent bubble rate. Prefetch before routing starts orange loading blocks during gray non-FFN execution and reduces the bubble rate to 9.05 percent.
Figure 7. Mean request latency–memory tradeoffs across seven application workloads. Rows denote models and columns denote datasets; both axes are logarithmic and lower left is better. A three-by-seven grid of log-scale scatter plots comparing mean request latency and memory across seven application datasets. EStream has a stable bounded memory footprint and becomes the fastest runtime on the longer reasoning and retrieval workloads.
Figure 8. Pure-prefill latency–memory tradeoffs across input lengths on v81. Rows denote models and columns denote input lengths; both axes are logarithmic and lower left is better. A three-by-six grid of log-scale scatter plots comparing EStream with six baselines. EStream is marked by red stars and generally occupies the lower-left region, indicating lower TTFT and memory.
Model
Layers
E
Non-exp. (GB)
Expert (GB)
OLMoE-1B-7B
16 (16)
64
0.30
3.62
LFM2.5-8B-A1B
24 (22)
32
0.41
4.36
DeepSeek-V2-Lite
27 (26)
64
0.99
8.10
Qwen3-30B-A3B
48 (48)
128
0.91
16.31
Mixtral-8 × 7B
32 (32)
8
1.04
25.37
Table 1. Model structures and deployed payloads. Layers gives total (MoE) blocks and E experts per block.
Dataset
Task
Metric
Mean tokens
MASSIVE
Intent
Accuracy
16
WinoGrande
Commonsense
Accuracy
23
ToxicChat
Safety
F1
66
HellaSwag
Completion
Norm. accuracy
80
ARC-C
Science QA
Accuracy
368
NFCorpus
Retrieval
nDCG@10
376
Table 2. Datasets, metrics, and mean prompt lengths.
Model
Tokens
Pure TTFT (s) ↓
Peak mem. (GiB) ↓
Tokens/J ↑
Qwen3-30B-A3B
1K
4.838
1.461
43.59
2K
5.043
1.594
62.36
3K
5.368
1.729
70.64
4K
6.836
1.866
72.18
Mixtral-8 × 7B
1K
7.086
2.767
23.41
2K
8.742
3.352
29.65
Table 3. EStream performance on large MoE models on v81. TTFT is the three-run median; memory and QPT energy are measured separately.
Figure 9. Steady-state prefill energy efficiency across input lengths; higher is better. Bars show medians and whiskers show ranges over completed runs. The unavailable DeepSeek-V2-Lite EdgeMoE result at 1.5K is omitted. Three side-by-side grouped bar charts compare the tokens per joule of EStream with llama.cpp-CPU, MNN-CPU, MNN-OpenCL, GenAI-CPU, EdgeMoE, and PowerInfer on OLMoE, LFM2.5, and DeepSeek-V2-Lite from 128 to 2,048 input tokens. EStream's efficiency rises substantially with input length, while the CPU baselines remain comparatively flat.
Figure 10. Measured 1K latency–memory tradeoffs across Q/B/G configurations. Curves show Pareto frontiers, crosses mark unconstrained minima, and stars mark auto-configured points under three memory budgets. Four scatter plots show pure-prefill latency against peak memory for OLMoE, LFM2.5, DeepSeek-V2-Lite, and Qwen3-30B-A3B. Each plot overlays a measured Pareto frontier, the global sweep minimum, three memory budgets, and the configuration selected automatically under each budget.
Figure 11. System ablation across compute and expert-loading strategies. The top row shows pure-prefill latency, and the bottom row shows peak physical memory. A two-by-three plot compares OLMoE, LFM2.5, and DeepSeek-V2-Lite. The top row uses lines to show pure-prefill latency, while the bottom row uses grouped bars to show peak physical memory. Each column compares fully resident CPU and NPU execution, per-expert QNN context streaming, naive NPU streaming with Q equals 1, G equals 1, and B equals 3, two cross-layer loading strategies, and the complete EStream system from 256 to 4,096 input tokens.
Checkpoint
Exec.
C4 ↓
Wino. ↑
ARC-C ↑
LAMB. ↑
Hella. ↑
BoolQ ↑
SciFact ↑
OLMo-1B
Host
33.52
65.00
28.00
64.50
65.50
63.50
18.59
OLMoE-1B-7B
Host
29.15
68.00
63.50
75.00
78.50
76.50
20.97
EStream
29.15
68.50
63.50
74.00
77.50
76.50
22.12
LFM2.5-1.2B
Host
51.44
59.00
74.00
49.50
63.50
75.00
20.17
LFM2.5-8B-A1B
Host
36.51
67.00
85.00
59.50
73.50
72.50
17.82
EStream
36.48
67.50
85.00
59.50
74.50
74.00
17.27
Table 4. Unified Q4 quality on the same 200 source examples per dataset. C4 is word PPL ( ↓ ); other values are percentages ( ↑ ). For paired families, bold compares Host Dense against EStream MoE.
Apple Neural Engine (ANE) is a dedicated neural processing unit (NPU) present in every Apple Silicon chip. Mixture-of-Experts (MoE) LLMs improve inference efficiency via sparse activation but are challenging for NPUs in three ways: expert routing is unpredictable and introduces dynamic tensor shapes that conflict with the shape-specific constraints of NPUs; several irregular operators, e.g., top-k, scatter/gather, etc., are not NPU-friendly; and launching many small expert kernels incurs substantial dispatch and synchronization overhead. NPUs are designed to offload AI compute from CPU and GPU; our goal is to enable such offloading for MoE inference, particularly during prefill, where long-context workloads consume substantial system resources. This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon by offloading dense, static computation to NPU, while preserving a CPU/GPU fallback path for dynamic operations. NPUMoE uses offline calibration to estimate expert capacity and popularity that drives three key techniques: (1) Static tiers for expert capacity to address dynamic expert routing (2) Grouped expert execution to mitigate NPU concurrency limits (3) Load-aware expert compute graph residency to reduce CPU-NPU synchronization overhead. Experiments on Apple M-series devices using three representative MoE LLMs and four long-context workloads show that NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.
Afsara Benazir, Felix Xiaozhu Lin
University of Virginia Charlottesville, Virginia, USA
Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose ZeRO-Prefill, a prefill-only serving system built on this idea. Its backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than by activation; its co-designed frontend enforces a physically derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.34-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.
Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5
University of Virginia · Snowflake AI Research · Harvard University
Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.