Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. To address this, we identify three key observations for on-device MoE inference: (a) cache hit rates are fundamentally bounded even under Belady's optimal policy; (b) these bounds are insufficient for low-latency on-device inference; and (c) prior cache-aware routing based on binary cache presence can induce routing instability. Based on these insights, we propose CECAR (Cache & Expert Co-Aware Routing), a MoE inference system that jointly considers expert routing and cache management. Unlike prior cache-aware routing methods, CECAR uses an ML-based cache policy to predict the expected reuse distance of each expert and prioritizes routing and execution toward experts with smaller predicted reuse distances. On the Qwen3-30B-A3B model, CECAR achieves 12.5 tokens/s on a consumer-grade GPU with only 16 GB of VRAM, where non-resident experts are fetched from SSD, delivering a 4.5x speedup over conventional MoE inference and exceeding the decode speed of baselines that keep the entire model fully resident in DRAM. Code is available at https://github.com/hoonki2002/CECAR.
Figures & tables
Figure 1: Memory bottlenecks for MoE inference on consumer devices.
Figure 2: Decoding Speed (Tokens/s) Across MoE Models under Different Routing Strategies.
Figure 3: Latency and memory implications of cache hit rate and expert cache size.
Figure 4: Illustration of the CECAR Routing Algorithm. Darker colors denote experts that CECAR expects to be routed in the near future.
Figure 5: Merging FFNHot and FFNCold for Eviction Score.
ML
Bonus
Pruning
HumanEval
Hit-rate (%)
Decoding Speed
×
×
×
47.56
54.83
3.40
✓
×
×
47.56
57.95
3.54
×
✓
×
44.51
75.91
5.45
×
×
✓
47.56
63.47
7.35
✓
✓
×
51.83
76.57
5.60
✓
×
✓
49.39
66.87
7.40
Table 1: Ablation study on ML-based cache policy, CECAR bonus routing, and expert pruning on DeepSeek-V2-Lite-Chat. Accuracy, hit-rate, and decoding speed are all measured on HumanEval only, whereas Table 3 averages hit-rate and speed over four benchmarks.
Policy
Cache Size (TriviaQA)
Dataset Generalization (Cache Size = 24)
8
16
24
32
TriviaQA
HumanEval
MBPP
GPQA-Diamond
MATH-500
Belady’s Optimal
52.30
72.47
82.56
88.68
82.56
79.51
79.62
82.37
79.49
ML (CECAR) ∗
44.18
62.42
73.63
82.04
73.63
68.37
69.31
73.73
69.63
ML (FlashMoE)
34.41
58.63
73.22
82.23
73.22
44.84
46.17
53.07
46.17
LRU
33.39
58.45
69.49
78.28
69.49
65.50
65.26
70.86
66.04
LFU
11.33
30.65
47.53
60.96
47.53
39.50
47.25
45.66
43.49
Table 2: Comparison of cache hit rates (%) across replacement policies.
Model
Method
HumanEval
MBPP
GPQA-D
MATH-500
Avg. top- k ∗
Hit-Rate (%)
Tokens/s
Qwen3-A3B
Baseline
67.7 ± 1.2
64.4 ± 0.9
36.7 ± 1.5
56.7 ± 2.2
8.00
68.3
2.8 ± 0.31
DES
65.4 ± 2.1
52.1 ± 0.7
34.0 ± 0.8
38.0 ± 1.2
4.67
71.3
4.7 ± 0.97
ODP
66.3 ± 0.9
55.4 ± 1.2
31.0 ± 0.8
44.1 ± 0.8
4.71
73.1
5.0 ± 1.43
MoCCE
67.5 ± 1.9
61.8 ± 0.5
38.0 ± 2.8
55.3 ± 2.8
8.00
92.3
8.6 ± 1.80
CECAR
67.9 ± 3.0
61.1 ± 1.4
36.4 ± 1.8
56.4 ± 1.7
7.13
96.6
12.5 ± 1.19
DeepSeek-A2.4B
Baseline
47.4 ± 0.9
45.7 ± 0.4
16.8 ± 1.8
24.1 ± 0.8
6.00
58.2
4.4 ± 4.87
Table 3: Benchmark accuracy comparison & performance of cache-aware MoE inference methods. Accuracy is mean ± SD over 3 decoding seeds; bold and underline mark the best and second-best cache-aware or pruning method (baseline excluded).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
ML Cache (FlashMoE)
ML Cache (CECAR)
Notes
# FFNs per MoE Layer
1
2
Hot and Cold networks for CECAR
# Layers of FFN
3
3
Base depth per FFN; sweep over 2–4 in Appendix K
Input Dimension
2NE
2
2 features per expert (recency, frequency)
Hidden Size
128
32
Base size; sweep over 8–128 in Appendix K
Output Dimension
NE
1
Eviction score per expert
Activation
ReLU
SiLU
Non-linear activation
Appendix
Table 4: Training Hyperparameters for ML-Based Cache Policies. Full training details for CECAR are given in Appendix K .
De
FlashMoE
CECAR
Binary Target
Hot Target ( Dhot=6 )
Cold Target ( Dcold=15 )
1
0
0.0
0.0
2
0
0.2
0.0
3
0
0.4
0.0
4
0
0.6
0.0
5
0
0.8
0.0
Appendix
Table 5: Example Training Targets for FlashMoE and CECAR.
λ
Qwen3-30B-A3B
DeepSeek-V2-Lite-Chat
Baseline
67.68
47.56
0.2
69.51
47.56
0.4
64.02
46.95
0.6
62.19
46.34
0.8
60.36
46.34
Appendix
Table 6: MoCCE parameter sweep on HumanEval.
Top- k
Qwen3-30B-A3B
Top- k
DeepSeek-V2-Lite-Chat
8
67.68
6
48.78
7
68.29
5
48.78
6
65.85
4
45.12
5
64.02
3
39.63
4
57.93
2
29.88
3
36.59
1
6.71
Appendix
Table 7: Top- k ablation on HumanEval.
Top- k
Qwen3-30B-A3B
Top- k
DeepSeek-V2-Lite-Chat
Baseline
67.68
Baseline
47.56
6
68.90
4
49.39
5
67.68
3
50.61
4
70.73
2
42.07
3
55.49
1
32.32
Appendix
Table 8: CECAR Top- k threshold sweep on HumanEval.
Top- k
Policy
HumanEval
MBPP
GPQA-D
MATH-500
4
LRU
43.90
43.40
20.71
22.80
ML
50.61
44.00
17.68
26.60
3
LRU
48.78
42.40
20.71
26.00
ML
49.39
46.20
19.70
28.40
Appendix
Table 9: Effect of cache policy on DeepSeek-V2-Lite-Chat.
Table 11: Specifications of Desktop System used for CECAR. The system includes ample DRAM to facilitate experimentation, but during inference only a small portion is used to emulate memory-constrained edge environments.
Strategy
Qwen3-A3B
DeepSeek-A2.4B
OLMoE-A1B
Baseline
2.8 ± 0.31
4.4 ± 4.87
8.5 ± 1.97
DES
4.7 ± 0.97
5.0 ± 3.48
16.2 ± 2.93
ODP
5.0 ± 1.43
4.6 ± 2.19
18.4 ± 5.69
MoCCE
8.6 ± 1.80
7.6 ± 0.55
27.7 ± 6.96
CECAR
12.5 ± 1.19
10.3 ± 3.59
29.9 ± 4.99
Appendix
Table 12: Decode speed (tokens/s), mean ± SD across repeated on-device runs (cache size 24 experts/layer, 100 prompts per benchmark).
Model
Size
PCIe 5.0 Latency (ms)
PCIe 4.0 Latency (ms)
PCIe 5.0 TP (GB/s)
PCIe 4.0 TP (GB/s)
OLMoE-1B-7B-0125-Instruct
12.6M
5.137
6.256
2.45
2.01
DeepSeek-V2-Lite-Chat
17.3M
7.111
8.187
2.43
2.11
Qwen3-30B-A3B
9.4M
4.464
4.424
2.12
2.13
Appendix
Table 13: Expert loading latency and throughput across PCIe generations.
Strategy
Example 1
Example 2
Avg. top- k ∗
Hit-Rate (%)
Tokens/s
Baseline
False
True
6.00
58.78
3.24
DES
False
False
4.98
57.08
3.74
ODP
False
False
5.08
56.95
3.79
MoCCE
True
False
6.00
87.54
8.78
CECAR
True
True
5.48
88.47
10.12
Appendix
Table 14: Comparison of strategy outputs on Examples.
Dataset
MoE-Infinity
LRU
Belady (original routing)
CECAR
n (trace / CECAR)
MBPP
0.516 ± 0.028
0.642 ± 0.021
0.797 ± 0.013
0.950 ± 0.019
50 / 100
HumanEval
0.670 ± 0.026
0.627 ± 0.027
0.731 ± 0.016
0.949 ± 0.021
6 † / 100
MATH-500
0.598 ± 0.042
0.629 ± 0.017
0.718 ± 0.013
0.968 ± 0.014
6 † / 100
GPQA-Diamond
0.604 ± 0.030
0.646 ± 0.033
0.737 ± 0.016
0.982 ± 0.011
6 † / 100
Appendix
Table 15: Hit rate on the four evaluation benchmarks (Qwen3-30B-A3B, 24 experts/layer), mean ± std over sequences.
Operating point
Misses
Expert load (ms)
Fixed compute (ms)
CECAR scorer (ms)
Total (ms)
Tokens/s
On-demand (0% hit)
384
1006.1
42.73
–
1048.8
0.95
LRU (64.4% hit, MBPP)
137
358.2
42.73
–
400.9
2.49
CECAR (96.6% hit)
13
34.2
42.73
5.27
82.2
12.16
Appendix
Table 16: Per-token latency at three operating points (Qwen3-30B-A3B, 24 experts/layer).
Schedule
pass@1
Hit rate
Fixed-budget reference
S1 ( 24→8→24 )
72/164
85.4%
C=23 (realized average 22.8): 73/164, 86.5%
S2 ( 32→16 )
71/164
90.4%
C=32 (realized average 31.7 † ): 71/164, 90.7%
S3 ( 24↔12 )
72/164
82.1%
C=18 (realized average 20.6): 70/164, 81.4%
Appendix
Table 17: Accuracy and hit rate under changing budgets, with a fixed budget of the same time-averaged size for reference (DeepSeek-V2-Lite-Chat, HumanEval, 164 problems).
Transition
After transition
Steady state at new C
S1: 24→8
59.2%
56.9%
S2: 32→16
81.4%
78.3%
S1: 8→24
80.2%
87.2% (reached as the cache refills)
Appendix
Table 18: Hit rate immediately after each budget change, compared with the steady-state hit rate of the new budget.
Policy
# Params
TriviaQA (ID)
Korean
MBPP
GSM8K
Retrained ( h=32 , d=3 ), 3 seeds
2,370
1.000
1.000
1.000
1.000
Deployed scorer ( h=32 , d=3 )
2,370
0.995
0.995
0.994
0.996
Sweep range (15 configs × 3 seeds)
66 – 67,074
0.998 – 1.004
0.999 – 1.001
0.999 – 1.003
0.999 – 1.002
Belady (upper bound)
–
1.135
1.074
1.202
1.186
Appendix
Table 19: Hit rate relative to the retrained h=32 , d=3 scorer.
Fine-grained Mixture-of-Experts (MoE) models sparsely activate only a subset of experts per token, reducing activated computation while maintaining high model capacity. However, in memory-constrained inference scenarios, only a small set of experts can be cached. Experts not in the cache must be fetched from slow external storage (e.g., UFS), leading to frequent evictions and substantial I/O overhead. We propose ReMoE, a router fine-tuning framework designed to boost token-wise expert reuse. ReMoE biases the router toward recently selected experts, producing temporally stable routing that better matches cache locality constraints. By increasing short-horizon expert reuse, ReMoE reduces expert fetches from storage without adding inference-time computation. Experiments on DeepSeek and Qwen models show that ReMoE improves expert reuse by 26% while maintaining downstream task performance. Real-system evaluations further confirm these benefits, improving output throughput by 8.4% under vLLM GPU-CPU expert offloading and reducing TPOT by 43.6-49.8% under llama.cpp on Jetson Orin NX, corresponding to a 1.77-1.99× decode speedup across diverse workloads. Checkpoints and usage instructions are available at https://github.com/BUAA-OSCAR/ReMoE.
Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang +3
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Huawei Technologies Ltd.
Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter language models, yet its advantages at sub-billion scales for on-device deployment remain largely unexplored. To close this gap, we present MobileMoE, a family of on-device MoE language models with sub-billion active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier for on-device LLMs. We first formulate an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints, identifying an on-device sweet spot - moderate sparsity with fine-grained and shared experts - that is simultaneously memory and compute-optimal. Building on the derived architectures, we train MobileMoE with a four-stage recipe covering pre-training, mid-training, instruction fine-tuning, and quantization-aware training, all on open-source datasets. Across 14 benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs with 2-4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters. To bridge the last mile to mobile deployment, we provide the first efficient MoE inference on commodity smartphones with comprehensive on-device profiling. At comparable INT4 weight memory, MobileMoE-S delivers 1.8-3.8× faster prefill and 2.2-3.4× faster decode than the dense baseline MobileLLM-Pro.
Recently, Mixture-of-Experts (MoE) models have gained attention for efficiently scaling large language models. Although these models are extremely large, their sparse activation enables inference to be performed by accessing only a fraction of the model at a time. This property opens the possibility of on-device inference of MoE, which was previously considered infeasible for such large models. Consequently, various systems have been proposed to leverage this sparsity and enable efficient MoE inference for edge devices. However, previous MoE inference systems like Fiddler[8] or DAOP[13] rely on DRAM-based offloading and are not suitable for memory constrained on-device environments. As recent MoE models grow to hundreds of gigabytes, RAM-offloading solutions become impractical. To address this, we propose FlashMoE, a system that offloads inactive experts to SSD, enabling efficient MoE inference under limited RAM. FlashMoE incorporates a lightweight ML-based caching strategy that adaptively combines recency and frequency signals to maximize expert reuse, significantly reducing storage I/O. In addition, we built a user-grade desktop platform to demonstrate the practicality of FlashMoE. On this real hardware setup, FlashMoE improves cache hit rate by up to 51% over well-known offloading policies such as LRU and LFU, and achieves up to 2.6x speedup compared to existing MoE inference systems.