EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs

Junming Zhang; Zhenzhe Zheng; Fan Wu; Xiaoyao Huang; Jie Wu

cs.DCSep 6, 2026

EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs

Authors: Junming Zhang, Zhenzhe Zheng, Fan Wu, Xiaoyao Huang, Jie Wu

Organizations: Shanghai Jiao Tong University Shanghai, China · Cloud Computing Research Institute, China Telecom China · Temple University Philadelphia, Pennsylvania, USA

Abstract

Mobile vendors and application developers increasingly deploy LLMs on smartphones for diverse prefill-only services. Yet current systems rely mainly on dense models whose regular computation maps efficiently to mobile NPUs, leaving more capable MoEs underused. MoE prefill does not fit mobile NPUs: NPU graphs are fixed at compile time, yet MoE picks experts at runtime; and one request touches most experts, more than a phone can hold in memory. We present EStream, which resolves both by separating what the NPU must fix from what MoE decides at runtime. A single compiled expert graph serves every expert, with each expert's routed tokens and weight address bound at call time, so dynamic MoE execution runs entirely on the NPU without padding or CPU/GPU fallback. Expert virtualization keeps the expert pool in UFS flash storage and pages it through a fixed-size NPU-addressable arena, group by group, with loading hidden behind computation, so memory is bounded by the arena rather than by the model. It further introduces a hardware-aware configuration algorithm that automatically configures the UFS--NPU pipeline and maximizes loading--computation overlap. Across 18 comparative settings covering three 7B--16B MoEs and 256--4,096-token prompts, we evaluate EStream on a commercial Snapdragon smartphone. Compared to the fastest baseline at each setting, EStream achieves a 2.25--27.57X pure-prefill TTFT speedup and reduces peak physical memory by 1.19--12.29X. EStream further scales to MoE models with up to 46.7B parameters.

Figures & tables

Figure 1. On-device prefill-only applications process private data, but MoE prefill remains inefficient on mobile devices. A two-panel diagram. The left panel shows private user profiles, documents, and reviews entering a phone and producing sentiment, correctness, and intent decisions. The right panel shows a long tensor routed to nearly all experts, irregular matrix tiles that do not map cleanly to a mobile NPU, and slower, power-hungry CPU and GPU alternatives.

Figure 2. Longer prompts expose more expert computation to NPU acceleration. Each bar stacks the execution latencies of all experts during OLMoE CPU prefill at one input length. Stacked per-expert projection latency for OLMoE CPU prefill from 1 to 2048 tokens. Each bar stacks experts in increasing routed-token count, with color indicating route count. Curves mark the boundaries above which experts receive at least 32 or 64 routes. The region above these curves grows with prompt length and is labeled as an NPU acceleration opportunity.

Panel of Figure 3 — Figure 3. MoE prefill activates most experts, which dominate the model weights. (a) Activated layer–expert pairs. (b) Q4 expert and non-expert weight sizes. Two plots characterize the expanded MoE parameter working set. Panel (a) shows the percentage of layer–expert pairs activated by C4 prefixes from 1 to 4096 tokens; OLMoE and LFM2.5 approach 100 percent, while Qwen3 reaches 92.4 percent. Panel (b) compares the Q4 payload of non-expert and expert weights; expert weights account for more than 91 percent in all three models.

Figure 4. The system overview of EStream. A two-panel diagram. The left panel shows private user profiles, documents, and reviews entering a phone and producing sentiment, correctness, and intent decisions. The right panel shows a long tensor routed to nearly all experts, irregular matrix tiles that do not map cleanly to a mobile NPU, and slower, power-hungry CPU and GPU alternatives.

Panel of Figure 5 — Figure 5. Topology-invariant graph sharing improves efficiency. (a) A shared graph binds data at runtime. (b) Latency and memory versus per-expert QNN contexts. Two subfigures. The left compares parameter-bound expert graphs with one shared graph whose parameter buffer is selected at runtime. The right uses grouped bars to compare EStream and per-expert QNN contexts in expert-path latency and peak physical memory from 64 to 2048 input tokens.

Figure 6. Expert-loading schedules for 1,024-token OLMoE prefill. Earlier prefetch increases I/O–compute overlap and lowers bubble rate, $1-\max(T_{\mathrm{io}},T_{\mathrm{exec}})/T_{\mathrm{total}}$ . Three timelines compare expert loading and NPU execution. On-demand loading alternates orange loading blocks and blue expert-compute blocks and has a 34.41 percent bubble rate. Prefetch after routing overlaps the two after non-FFN execution and has a 21.36 percent bubble rate. Prefetch before routing starts orange loading blocks during gray non-FFN execution and reduces the bubble rate to 9.05 percent.

Figure 7. Mean request latency–memory tradeoffs across seven application workloads. Rows denote models and columns denote datasets; both axes are logarithmic and lower left is better. A three-by-seven grid of log-scale scatter plots comparing mean request latency and memory across seven application datasets. EStream has a stable bounded memory footprint and becomes the fastest runtime on the longer reasoning and retrieval workloads.

Figure 8. Pure-prefill latency–memory tradeoffs across input lengths on v81. Rows denote models and columns denote input lengths; both axes are logarithmic and lower left is better. A three-by-six grid of log-scale scatter plots comparing EStream with six baselines. EStream is marked by red stars and generally occupies the lower-left region, indicating lower TTFT and memory.

Model	Layers	$E$	Non-exp. (GB)	Expert (GB)
OLMoE-1B-7B	16 (16)	64	0.30	3.62
LFM2.5-8B-A1B	24 (22)	32	0.41	4.36
DeepSeek-V2-Lite	27 (26)	64	0.99	8.10
Qwen3-30B-A3B	48 (48)	128	0.91	16.31
Mixtral-8 $\times$ 7B	32 (32)	8	1.04	25.37

Table 1. Model structures and deployed payloads. Layers gives total (MoE) blocks and

E

experts per block.

Dataset	Task	Metric	Mean tokens
MASSIVE	Intent	Accuracy	16
WinoGrande	Commonsense	Accuracy	23
ToxicChat	Safety	F1	66
HellaSwag	Completion	Norm. accuracy	80
ARC-C	Science QA	Accuracy	368
NFCorpus	Retrieval	nDCG@10	376

Table 2. Datasets, metrics, and mean prompt lengths.

Model	Tokens	Pure TTFT (s) $\downarrow$	Peak mem. (GiB) $\downarrow$	Tokens/J $\uparrow$
Qwen3-30B-A3B	1K	4.838	1.461	43.59
	2K	5.043	1.594	62.36
	3K	5.368	1.729	70.64
	4K	6.836	1.866	72.18
Mixtral-8 $\times$ 7B	1K	7.086	2.767	23.41
	2K	8.742	3.352	29.65

Table 3. EStream performance on large MoE models on v81. TTFT is the three-run median; memory and QPT energy are measured separately.

Figure 9. Steady-state prefill energy efficiency across input lengths; higher is better. Bars show medians and whiskers show ranges over completed runs. The unavailable DeepSeek-V2-Lite EdgeMoE result at 1.5K is omitted. Three side-by-side grouped bar charts compare the tokens per joule of EStream with llama.cpp-CPU, MNN-CPU, MNN-OpenCL, GenAI-CPU, EdgeMoE, and PowerInfer on OLMoE, LFM2.5, and DeepSeek-V2-Lite from 128 to 2,048 input tokens. EStream's efficiency rises substantially with input length, while the CPU baselines remain comparatively flat.

Figure 10. Measured 1K latency–memory tradeoffs across

Q/B/G

configurations. Curves show Pareto frontiers, crosses mark unconstrained minima, and stars mark auto-configured points under three memory budgets. Four scatter plots show pure-prefill latency against peak memory for OLMoE, LFM2.5, DeepSeek-V2-Lite, and Qwen3-30B-A3B. Each plot overlays a measured Pareto frontier, the global sweep minimum, three memory budgets, and the configuration selected automatically under each budget.

Figure 11. System ablation across compute and expert-loading strategies. The top row shows pure-prefill latency, and the bottom row shows peak physical memory. A two-by-three plot compares OLMoE, LFM2.5, and DeepSeek-V2-Lite. The top row uses lines to show pure-prefill latency, while the bottom row uses grouped bars to show peak physical memory. Each column compares fully resident CPU and NPU execution, per-expert QNN context streaming, naive NPU streaming with Q equals 1, G equals 1, and B equals 3, two cross-layer loading strategies, and the complete EStream system from 256 to 4,096 input tokens.

Checkpoint	Exec.	C4 $\downarrow$	Wino. $\uparrow$	ARC-C $\uparrow$	LAMB. $\uparrow$	Hella. $\uparrow$	BoolQ $\uparrow$	SciFact $\uparrow$
OLMo-1B	Host	33.52	65.00	28.00	64.50	65.50	63.50	18.59
OLMoE-1B-7B	Host	29.15	68.00	63.50	75.00	78.50	76.50	20.97
OLMoE-1B-7B	EStream	29.15	68.50	63.50	74.00	77.50	76.50	22.12
LFM2.5-1.2B	Host	51.44	59.00	74.00	49.50	63.50	75.00	20.17
LFM2.5-8B-A1B	Host	36.51	67.00	85.00	59.50	73.50	72.50	17.82
LFM2.5-8B-A1B	EStream	36.48	67.50	85.00	59.50	74.50	74.00	17.27

Table 4. Unified Q4 quality on the same 200 source examples per dataset. C4 is word PPL (

\downarrow

); other values are percentages (

\uparrow

). For paired families, bold compares Host Dense against EStream MoE.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs
Apr 20, 2026Afsara Benazir, Felix Xiaozhu LinAI Accelerator Inference Energy-Efficient ML
University of Virginia Charlottesville, Virginia, USA
ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Expert Parallelism LLM Serving
University of Virginia · Snowflake AI Research · Harvard University
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
Jul 6, 2026Guanyu Cai, Ruiming Tian, Lang Yang +4LLM Inference Efficiency AI Accelerator Inference
Tsinghua University · Beijing Jiaotong University

Apr 20, 2026J/K move · Enter open · S save

Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs

Afsara Benazir, Felix Xiaozhu Lin

University of Virginia Charlottesville, Virginia, USA

AI Accelerator Inference Energy-Efficient ML Mixture-of-Experts Inference LLM Inference Acceleration On-Device Inference

Open paper AskPDF arXiv

Apple Neural Engine (ANE) is a dedicated neural processing unit (NPU) present in every Apple Silicon chip. Mixture-of-Experts (MoE) LLMs improve inference efficiency via sparse activation but are challenging for NPUs in three ways: expert routing is unpredictable and introduces dynamic tensor shapes that conflict with the shape-specific constraints of NPUs; several irregular operators, e.g., top-k, scatter/gather, etc., are not NPU-friendly; and launching many small expert kernels incurs substantial dispatch and synchronization overhead. NPUs are designed to offload AI compute from CPU and GPU; our goal is to enable such offloading for MoE inference, particularly during prefill, where long-context workloads consume substantial system resources. This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon by offloading dense, static computation to NPU, while preserving a CPU/GPU fallback path for dynamic operations. NPUMoE uses offline calibration to estimate expert capacity and popularity that drives three key techniques: (1) Static tiers for expert capacity to address dynamic expert routing (2) Grouped expert execution to mitigate NPU concurrency limits (3) Load-aware expert compute graph residency to reduce CPU-NPU synchronization overhead. Experiments on Apple M-series devices using three representative MoE LLMs and four long-context workloads show that NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.

Abstract

Mixture-of-Experts Inference

Explore similar work

Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs

ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference