Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. To address this, we identify three key observations for on-device MoE inference: (a) cache hit rates are fundamentally bounded even under Belady's optimal policy; (b) these bounds are insufficient for low-latency on-device inference; and (c) prior cache-aware routing based on binary cache presence can induce routing instability. Based on these insights, we propose CECAR (Cache & Expert Co-Aware Routing), a MoE inference system that jointly considers expert routing and cache management. Unlike prior cache-aware routing methods, CECAR uses an ML-based cache policy to predict the expected reuse distance of each expert and prioritizes routing and execution toward experts with smaller predicted reuse distances. On the Qwen3-30B-A3B model, CECAR achieves 12.5 tokens/s on a consumer-grade GPU with only 16 GB of VRAM, where non-resident experts are fetched from SSD, delivering a 4.5x speedup over conventional MoE inference and exceeding the decode speed of baselines that keep the entire model fully resident in DRAM. Code is available at https://github.com/hoonki2002/CECAR.
Figures & tables
Figure 1: Memory bottlenecks for MoE inference on consumer devices.
Figure 2: Decoding Speed (Tokens/s) Across MoE Models under Different Routing Strategies.
Figure 3: Latency and memory implications of cache hit rate and expert cache size.
Figure 4: Illustration of the CECAR Routing Algorithm. Darker colors denote experts that CECAR expects to be routed in the near future.
Figure 5: Merging FFNHot and FFNCold for Eviction Score.
ML
Bonus
Pruning
HumanEval
Hit-rate (%)
Decoding Speed
×
×
×
47.56
54.83
3.40
✓
×
×
47.56
57.95
3.54
×
✓
×
44.51
75.91
5.45
×
×
✓
47.56
63.47
7.35
✓
✓
×
51.83
76.57
5.60
✓
×
✓
49.39
66.87
7.40
Table 1: Ablation study on ML-based cache policy, CECAR bonus routing, and expert pruning on DeepSeek-V2-Lite-Chat. Accuracy, hit-rate, and decoding speed are all measured on HumanEval only, whereas Table 3 averages hit-rate and speed over four benchmarks.
Policy
Cache Size (TriviaQA)
Dataset Generalization (Cache Size = 24)
8
16
24
32
TriviaQA
HumanEval
MBPP
GPQA-Diamond
MATH-500
Belady’s Optimal
52.30
72.47
82.56
88.68
82.56
79.51
79.62
82.37
79.49
ML (CECAR) ∗
44.18
62.42
73.63
82.04
73.63
68.37
69.31
73.73
69.63
ML (FlashMoE)
34.41
58.63
73.22
82.23
73.22
44.84
46.17
53.07
46.17
LRU
33.39
58.45
69.49
78.28
69.49
65.50
65.26
70.86
66.04
LFU
11.33
30.65
47.53
60.96
47.53
39.50
47.25
45.66
43.49
Table 2: Comparison of cache hit rates (%) across replacement policies.
Model
Method
HumanEval
MBPP
GPQA-D
MATH-500
Avg. top- k ∗
Hit-Rate (%)
Tokens/s
Qwen3-A3B
Baseline
67.7 ± 1.2
64.4 ± 0.9
36.7 ± 1.5
56.7 ± 2.2
8.00
68.3
2.8 ± 0.31
DES
65.4 ± 2.1
52.1 ± 0.7
34.0 ± 0.8
38.0 ± 1.2
4.67
71.3
4.7 ± 0.97
ODP
66.3 ± 0.9
55.4 ± 1.2
31.0 ± 0.8
44.1 ± 0.8
4.71
73.1
5.0 ± 1.43
MoCCE
67.5 ± 1.9
61.8 ± 0.5
38.0 ± 2.8
55.3 ± 2.8
8.00
92.3
8.6 ± 1.80
CECAR
67.9 ± 3.0
61.1 ± 1.4
36.4 ± 1.8
56.4 ± 1.7
7.13
96.6
12.5 ± 1.19
DeepSeek-A2.4B
Baseline
47.4 ± 0.9
45.7 ± 0.4
16.8 ± 1.8
24.1 ± 0.8
6.00
58.2
4.4 ± 4.87
Table 3: Benchmark accuracy comparison & performance of cache-aware MoE inference methods. Accuracy is mean ± SD over 3 decoding seeds; bold and underline mark the best and second-best cache-aware or pruning method (baseline excluded).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
ML Cache (FlashMoE)
ML Cache (CECAR)
Notes
# FFNs per MoE Layer
1
2
Hot and Cold networks for CECAR
# Layers of FFN
3
3
Base depth per FFN; sweep over 2–4 in Appendix K
Input Dimension
2NE
2
2 features per expert (recency, frequency)
Hidden Size
128
32
Base size; sweep over 8–128 in Appendix K
Output Dimension
NE
1
Eviction score per expert
Activation
ReLU
SiLU
Non-linear activation
Appendix
Table 4: Training Hyperparameters for ML-Based Cache Policies. Full training details for CECAR are given in Appendix K .
De
FlashMoE
CECAR
Binary Target
Hot Target ( Dhot=6 )
Cold Target ( Dcold=15 )
1
0
0.0
0.0
2
0
0.2
0.0
3
0
0.4
0.0
4
0
0.6
0.0
5
0
0.8
0.0
Appendix
Table 5: Example Training Targets for FlashMoE and CECAR.
λ
Qwen3-30B-A3B
DeepSeek-V2-Lite-Chat
Baseline
67.68
47.56
0.2
69.51
47.56
0.4
64.02
46.95
0.6
62.19
46.34
0.8
60.36
46.34
Appendix
Table 6: MoCCE parameter sweep on HumanEval.
Top- k
Qwen3-30B-A3B
Top- k
DeepSeek-V2-Lite-Chat
8
67.68
6
48.78
7
68.29
5
48.78
6
65.85
4
45.12
5
64.02
3
39.63
4
57.93
2
29.88
3
36.59
1
6.71
Appendix
Table 7: Top- k ablation on HumanEval.
Top- k
Qwen3-30B-A3B
Top- k
DeepSeek-V2-Lite-Chat
Baseline
67.68
Baseline
47.56
6
68.90
4
49.39
5
67.68
3
50.61
4
70.73
2
42.07
3
55.49
1
32.32
Appendix
Table 8: CECAR Top- k threshold sweep on HumanEval.
Top- k
Policy
HumanEval
MBPP
GPQA-D
MATH-500
4
LRU
43.90
43.40
20.71
22.80
ML
50.61
44.00
17.68
26.60
3
LRU
48.78
42.40
20.71
26.00
ML
49.39
46.20
19.70
28.40
Appendix
Table 9: Effect of cache policy on DeepSeek-V2-Lite-Chat.
Table 11: Specifications of Desktop System used for CECAR. The system includes ample DRAM to facilitate experimentation, but during inference only a small portion is used to emulate memory-constrained edge environments.
Strategy
Qwen3-A3B
DeepSeek-A2.4B
OLMoE-A1B
Baseline
2.8 ± 0.31
4.4 ± 4.87
8.5 ± 1.97
DES
4.7 ± 0.97
5.0 ± 3.48
16.2 ± 2.93
ODP
5.0 ± 1.43
4.6 ± 2.19
18.4 ± 5.69
MoCCE
8.6 ± 1.80
7.6 ± 0.55
27.7 ± 6.96
CECAR
12.5 ± 1.19
10.3 ± 3.59
29.9 ± 4.99
Appendix
Table 12: Decode speed (tokens/s), mean ± SD across repeated on-device runs (cache size 24 experts/layer, 100 prompts per benchmark).
Model
Size
PCIe 5.0 Latency (ms)
PCIe 4.0 Latency (ms)
PCIe 5.0 TP (GB/s)
PCIe 4.0 TP (GB/s)
OLMoE-1B-7B-0125-Instruct
12.6M
5.137
6.256
2.45
2.01
DeepSeek-V2-Lite-Chat
17.3M
7.111
8.187
2.43
2.11
Qwen3-30B-A3B
9.4M
4.464
4.424
2.12
2.13
Appendix
Table 13: Expert loading latency and throughput across PCIe generations.
Strategy
Example 1
Example 2
Avg. top- k ∗
Hit-Rate (%)
Tokens/s
Baseline
False
True
6.00
58.78
3.24
DES
False
False
4.98
57.08
3.74
ODP
False
False
5.08
56.95
3.79
MoCCE
True
False
6.00
87.54
8.78
CECAR
True
True
5.48
88.47
10.12
Appendix
Table 14: Comparison of strategy outputs on Examples.
Dataset
MoE-Infinity
LRU
Belady (original routing)
CECAR
n (trace / CECAR)
MBPP
0.516 ± 0.028
0.642 ± 0.021
0.797 ± 0.013
0.950 ± 0.019
50 / 100
HumanEval
0.670 ± 0.026
0.627 ± 0.027
0.731 ± 0.016
0.949 ± 0.021
6 † / 100
MATH-500
0.598 ± 0.042
0.629 ± 0.017
0.718 ± 0.013
0.968 ± 0.014
6 † / 100
GPQA-Diamond
0.604 ± 0.030
0.646 ± 0.033
0.737 ± 0.016
0.982 ± 0.011
6 † / 100
Appendix
Table 15: Hit rate on the four evaluation benchmarks (Qwen3-30B-A3B, 24 experts/layer), mean ± std over sequences.
Operating point
Misses
Expert load (ms)
Fixed compute (ms)
CECAR scorer (ms)
Total (ms)
Tokens/s
On-demand (0% hit)
384
1006.1
42.73
–
1048.8
0.95
LRU (64.4% hit, MBPP)
137
358.2
42.73
–
400.9
2.49
CECAR (96.6% hit)
13
34.2
42.73
5.27
82.2
12.16
Appendix
Table 16: Per-token latency at three operating points (Qwen3-30B-A3B, 24 experts/layer).
Schedule
pass@1
Hit rate
Fixed-budget reference
S1 ( 24→8→24 )
72/164
85.4%
C=23 (realized average 22.8): 73/164, 86.5%
S2 ( 32→16 )
71/164
90.4%
C=32 (realized average 31.7 † ): 71/164, 90.7%
S3 ( 24↔12 )
72/164
82.1%
C=18 (realized average 20.6): 70/164, 81.4%
Appendix
Table 17: Accuracy and hit rate under changing budgets, with a fixed budget of the same time-averaged size for reference (DeepSeek-V2-Lite-Chat, HumanEval, 164 problems).
Transition
After transition
Steady state at new C
S1: 24→8
59.2%
56.9%
S2: 32→16
81.4%
78.3%
S1: 8→24
80.2%
87.2% (reached as the cache refills)
Appendix
Table 18: Hit rate immediately after each budget change, compared with the steady-state hit rate of the new budget.
Policy
# Params
TriviaQA (ID)
Korean
MBPP
GSM8K
Retrained ( h=32 , d=3 ), 3 seeds
2,370
1.000
1.000
1.000
1.000
Deployed scorer ( h=32 , d=3 )
2,370
0.995
0.995
0.994
0.996
Sweep range (15 configs × 3 seeds)
66 – 67,074
0.998 – 1.004
0.999 – 1.001
0.999 – 1.003
0.999 – 1.002
Belady (upper bound)
–
1.135
1.074
1.202
1.186
Appendix
Table 19: Hit rate relative to the retrained h=32 , d=3 scorer.