Organizations: Xiaohongshu Inc. · Shenzhen Graduate School, Peking University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China
Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference phases with fundamentally different bottlenecks: prefill is dominated by token-wise expert computation, whereas decode is constrained by memory traffic from the batch-wise activated expert set. However, existing training-free acceleration methods optimize only a single resource proxy, either the experts each token executes or the experts a batch activates, and either discard the excluded experts' contribution or leave it only implicitly approximated. In this paper, we propose ExFold, a unified training-free expert-folding framework for jointly accelerating MoE prefill and decode. ExFold casts both prefill and decode as one budgeted output-approximation problem: execute only a phase-specific constrained expert set while projecting the contribution of budget-excluded experts onto retained experts using calibrated scalar projectors. Motivated by the observation that many expert outputs are directionally aligned but differ in magnitude, ExFold calibrates a pairwise scalar-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts. Under this view, prefill acceleration becomes token-level Top-K folding, and decode acceleration becomes batch-level expert-pool folding. The two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared folding mechanism. We implement ExFold as a plug-and-play plugin in vLLM, with a lightweight expert-folding CUDA kernel, delivering up to 1.41x TTFT and 2.45x TPOT speedups while retaining about 99% of the original average quality.
Figures & tables
Figure 1: MoE prefill and decode have different bottlenecks.
Figure 2: Quality–speed trade-offs for ExFold prefill and decode acceleration.
Figure 3: (a-c) MoE expert outputs can be aligned across experts and layers, and (d) their L2 norms vary substantially .
Figure 4: Overview of ExFold : training-free projector calibration and unified expert folding.
Method
Prefill
Decode
MATH 500
AIME24
IFEval
IFBench
GPQA
LCB (P/A.@8)
Eval+
MMLU pro
Avg.
Baseline
Original TopK=8
100%
100%
97.40
64.48
83.73
29.31
63.13
68.26 / 56.89
77.44
68.57
69.04
Prefill-Only Acceleration
Prefill TopK=6
75%
100%
95.80
64.79
83.18
26.27
64.14
69.46 / 57.34
75.61
66.33
68.20
Prefill TopK=4
50%
100%
96.00
66.56
76.00
29.02
60.61
66.47 / 57.63
73.17
65.28
66.64
MC-MoE-P4
50%
100%
96.00
64.58
79.85
26.48
45.45
68.26 / 57.11
77.44
65.44
65.44
Table 1: Qwen3-30B-A3B quality. Bold marks the best result per setting. P4 denotes 4 experts per token in prefill, D m means a decode pool with size m , and S∗ means SERE’s dynamic set. P/A.@8 means the score of Pass@8 and Avg@8.
Method
AIME24
IFEval
GPQA
Eval+
MMLU pro
Avg.
Baseline
Original Top-8
48.75
83.55
76.26
71.34
58.98
67.78
Prefill-Only Acceleration
Prefill TopK=6
46.67
83.55
75.25
73.78
56.53
67.16
Prefill TopK=4
45.00
80.59
69.19
75.61
40.41
62.16
ExFold-P4
48.75
80.78
73.74
67.68
58.57
65.90
Table 2: GLM-4.5-Air quality. Bold for the best per setting.
Figure 5: Online TTFT/TPOT speedups and offline serving throughput.
Method
MATH 500
AIME24
IFEval
IFBench
GPQA
Eval+
MMLU pro
PPL ↓
Δ Storage
Δ Token Cost
Kernel
Original TopK=8
97.40
64.48
83.73
29.31
63.13
77.44
68.57
3.020
0
O(4LCFFN)
FusedMoE
Prefill TopK=4
96.00
66.56
76.00
29.02
60.61
73.17
65.28
3.470
0
0
FusedMoE
Global scalar
96.00
65.21
80.96
25.73
59.60
75.00
65.60
3.373
4 B
O(4L)
FusedMoE
Layer scalar
96.60
67.60
82.44
26.30
61.11
74.39
64.70
3.350
192 B
O(4L)
FusedMoE
Expert scalar
97.00
66.46
84.00
29.53
62.63
76.83
66.94
3.352
6.00 MiB
O(4L)
FusedMoE
Diagonal
96.60
67.50
83.92
27.63
60.61
72.56
65.87
3.308
316.7 MiB
O(4LH)
Unfused
Table 3: Projector ablation under Top-4 prefill. Overheads use FP32 state and are relative to Prefill TopK=4. Here ΔK=4 ; R8/R16 incur 2ΔKR=64/128 operations per hidden dimension. L,H,CFFN : layers, hidden size, and one expert-call time.
Table 4: Calibration-corpus ablation for ExFold-P4.
Selection criterion
MATH 500
IFEval
GPQA
Eval+
Avg.
Prefill (ExFold-P4)
rank by Si
97.00
82.62
64.65
73.17
79.36
rank by Hi
97.00
67.47
57.58
75.61
74.42
rank by Si×Hi (Ours)
97.00
84.00
62.63
76.83
80.11
Decode (ExFold-D64)
rank by Si
97.00
83.92
60.10
73.17
78.55
Table 5: Retained-expert selection on Qwen3-30B-A3B.
Method
Prefill
Decode
MATH 500
AIME25
AIME26
IFEval
IFBench
GPQA
LCB
Eval+
MMLU pro
Avg.
Baseline
Original TopK=6
100%
100%
93.40
70.42
72.50
82.44
36.23
73.23
54.90
89.25
81.76
72.68
Prefill-Only Acceleration
Prefill TopK=3
50%
100%
92.00
69.17
69.17
80.22
35.67
69.70
54.12
85.59
79.20
70.54
ExFold-P3
50%
100%
93.20
65.83
70.42
82.99
35.28
72.22
55.67
87.65
80.48
71.53
Decode-Only Acceleration
Table 6: DeepSeek-V4-Flash quality. Bold marks the best result per setting. P3 denotes 3 experts per token in prefill, and D m means a decode pool with size m .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Layers
MoE Layers
Routed Experts
Shared Experts
Top- K
Hidden Size
Expert FFN
Qwen3-30B-A3B ( 26 )
48
48
128
0
8
2048
768
GLM-4.5-Air ( 32 )
46
45
128
1
8
4096
1408
DeepSeek-V2-Lite-Chat ( 5 )
27
26
64
2
6
2048
1408
Qwen3.5-35B-A3B ( 27 )
40
40
256
1
8
2048
512
Appendix
Table 8: Architectures of the evaluated MoE checkpoints.
Benchmark
Domain
Cases
Samples
Metric
MATH500 ( 10 ; 13 )
Mathematics
500
1
Symbolic-equivalence accuracy
AIME24 ( 23 )
Competition mathematics
30
32 (Qwen3); 8 (GLM)
Mean symbolic-equivalence accuracy
IFEval ( 33 )
Instruction following
541
1
Prompt-level strict accuracy
IFBench ( 25 )
Instruction following
294
1
OpenCompass four-way mean
GPQA-Diamond ( 28 )
Graduate-level reasoning
198
1
Accuracy
LiveCodeBench v5 ( 14 )
Code generation
167
8
Pass@8 / mean@8
Appendix
Table 9: Benchmarks, evaluation sizes, and reported metrics.
Projector
Storage
Online
Kernel
Global scalar
1
Scalar
Fused
Layer scalar
L
Scalar
Fused
Pair scalar
P
Scalar
Fused
Pair diagonal
PH
Element-wise
Unfused
Scalar + low-rank
P+LHR+PRH
Rank- R
Unfused
Appendix
Table 10: Implementation differences among the evaluated projector families.
Method
IFEval
GPQA
Eval+
MMLU pro
Avg.
Original Top-6
42.14
23.23
46.34
26.52
34.56
Direct Top-4
40.85
24.24
46.95
24.71
34.19
ExFold P4+D32
41.96
26.77
48.17
24.75
35.41
Appendix
Table 11: DeepSeek-V2-Lite-Chat quality; bold marks the column best.
Method
MATH 500
IFEval
IFBench
GPQA
Eval+
Avg.
Original Top-8
89.00
87.06
29.20
82.83
88.41
75.30
Direct Top-4
32.80
81.52
33.53
82.32
87.20
63.47
ExFold P4
88.40
82.62
32.59
82.32
92.68
75.72
ExFold D64
89.20
85.21
38.40
81.31
89.63
76.75
ExFold P4+D64
88.40
83.36
33.77
86.36
92.07
76.79
Appendix
Table 12: Qwen3.5-35B-A3B quality; bold marks the column best.
Figure 7: Expert-output magnitudes across all Qwen3 MoE layers.
Figure 8: GLM-4.5-Air expert-output magnitudes (continued); hatching marks one isolated outlier.
Figure 9: Raw expert-output similarity across all Qwen3 MoE layers.
Figure 10: Scalar-aligned similarity for observed Qwen3 expert pairs.
Figure 11: Raw co-routed expert similarity across all GLM-4.5-Air MoE layers.
Figure 12: Scalar-aligned co-routed expert similarity across all GLM-4.5-Air MoE layers.
Figure 13: Common-input scalar-projector matrices across all Qwen3 MoE layers.
Figure 14: Common-input scalar-aligned Frobenius losses across all Qwen3 MoE layers.
Figure 15: Token-level proxy reconstructions. Expert contributions are accumulated strictly head-to-tail; each chain terminates at y or y .
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence--expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average, raises draft acceptance to 8487%, and achieves 8688% prefetch hit rates.
Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video interaction systems. To meet the extremely low response latency requirements of these scenarios, practitioners commonly employ small-batch decoding, under which MoE inference becomes memory-bound and is severely bottlenecked by expert weight loading. However, this bottleneck has received limited attention, and existing solutions such as post-training weight compression or fine-grained expert design during pre-training either degrade model accuracy or introduce additional computation and communication overhead. To tackle this issue, we propose DeaMoE, a decoding-efficient MoE architecture, in which the experts are grouped into several departments, and the experts belonging to the same department share most parameters since they come from the same professional field, and additionally each expert contains a few private parameters to reflect its uniqueness. Moreover, we design customized two-stage routing strategy for DeaMoE to avoid redundant loading, under which DeaMoE greatly improves the efficiency during LLM decoding. Compared with vanilla MoE, DeaMoE reduces per-step loaded weights by up to 50.9% and achieves up to 1.33 end-to-end TPOT speedup for the pre-trained 7B model on A40, and up to 2.00x and 1.97x peak speedup for DeepSeek-V3 on A40 and H100 in microbenchmarks.
Zewen Jin, Shen Fu, Zeping Duan +8
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · Shanxi University