ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

Juntong Wu; Yifei Liu; Junyi Chen; Siqi Fan; Chaoran Feng; Minghao Li; Liujie Zhang; Weihang Chen

cs.LGAug 24, 2026

ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

Authors: Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen,

Organizations: Xiaohongshu Inc. · Shenzhen Graduate School, Peking University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China

Abstract

Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference phases with fundamentally different bottlenecks: prefill is dominated by token-wise expert computation, whereas decode is constrained by memory traffic from the batch-wise activated expert set. However, existing training-free acceleration methods optimize only a single resource proxy, either the experts each token executes or the experts a batch activates, and either discard the excluded experts' contribution or leave it only implicitly approximated. In this paper, we propose ExFold, a unified training-free expert-folding framework for jointly accelerating MoE prefill and decode. ExFold casts both prefill and decode as one budgeted output-approximation problem: execute only a phase-specific constrained expert set while projecting the contribution of budget-excluded experts onto retained experts using calibrated scalar projectors. Motivated by the observation that many expert outputs are directionally aligned but differ in magnitude, ExFold calibrates a pairwise scalar-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts. Under this view, prefill acceleration becomes token-level Top-K folding, and decode acceleration becomes batch-level expert-pool folding. The two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared folding mechanism. We implement ExFold as a plug-and-play plugin in vLLM, with a lightweight expert-folding CUDA kernel, delivering up to 1.41x TTFT and 2.45x TPOT speedups while retaining about 99% of the original average quality.

Figures & tables

Figure 1: MoE prefill and decode have different bottlenecks.

Figure 2: Quality–speed trade-offs for ExFold prefill and decode acceleration.

Figure 3: (a-c) MoE expert outputs can be aligned across experts and layers, and (d) their L2 norms vary substantially .

Figure 4: Overview of ExFold : training-free projector calibration and unified expert folding.

Method	Prefill	Decode	MATH 500	AIME24	IFEval	IFBench	GPQA	LCB (P/A.@8)	Eval+	MMLU pro	Avg.
Baseline
Original TopK=8	100%	100%	97.40	64.48	83.73	29.31	63.13	68.26 / 56.89	77.44	68.57	69.04
Prefill-Only Acceleration
Prefill TopK=6	75%	100%	95.80	64.79	83.18	26.27	64.14	69.46 / 57.34	75.61	66.33	68.20
Prefill TopK=4	50%	100%	96.00	66.56	76.00	29.02	60.61	66.47 / 57.63	73.17	65.28	66.64
MC-MoE-P4	50%	100%	96.00	64.58	79.85	26.48	45.45	68.26 / 57.11	77.44	65.44	65.44

Table 1: Qwen3-30B-A3B quality. Bold marks the best result per setting. P4 denotes 4 experts per token in prefill, D

m

means a decode pool with size

m

, and

S^{\ast}

means SERE’s dynamic set. P/A.@8 means the score of Pass@8 and Avg@8.

Baseline
Method	AIME24	IFEval	GPQA	Eval+	MMLU pro	Avg.
Original Top-8	48.75	83.55	76.26	71.34	58.98	67.78
Prefill-Only Acceleration
Prefill TopK=6	46.67	83.55	75.25	73.78	56.53	67.16
Prefill TopK=4	45.00	80.59	69.19	75.61	40.41	62.16
ExFold-P4	48.75	80.78	73.74	67.68	58.57	65.90

Table 2: GLM-4.5-Air quality. Bold for the best per setting.

Panel of Figure 5 — Figure 5: Online TTFT/TPOT speedups and offline serving throughput.

Method	MATH 500	AIME24	IFEval	IFBench	GPQA	Eval+	MMLU pro	PPL $\downarrow$	$\Delta$ Storage	$\Delta$ Token Cost	Kernel
Original TopK=8	97.40	64.48	83.73	29.31	63.13	77.44	68.57	3.020	0	$O(4LC_{\mathrm{FFN}})$	FusedMoE
Prefill TopK=4	96.00	66.56	76.00	29.02	60.61	73.17	65.28	3.470	0	0	FusedMoE
Global scalar	96.00	65.21	80.96	25.73	59.60	75.00	65.60	3.373	4 B	$O(4L)$	FusedMoE
Layer scalar	96.60	67.60	82.44	26.30	61.11	74.39	64.70	3.350	192 B	$O(4L)$	FusedMoE
Expert scalar	97.00	66.46	84.00	29.53	62.63	76.83	66.94	3.352	6.00 MiB	$O(4L)$	FusedMoE
Diagonal	96.60	67.50	83.92	27.63	60.61	72.56	65.87	3.308	316.7 MiB	$O(4LH)$	Unfused

Table 3: Projector ablation under Top-4 prefill. Overheads use FP32 state and are relative to Prefill TopK=4. Here

\Delta K=4

; R8/R16 incur

2\Delta KR=64/128

operations per hidden dimension.

L,H,C_{\mathrm{FFN}}

: layers, hidden size, and one expert-call time.

Figure 6: Per-token MoE output reconstruction under expert reduction. Error denotes

\|\widehat{y}-y\|_{2}

Calibration	MATH 500	GPQA	LCB (P/A.@8)	Eval+	Avg.
GPQA	96.40	65.15	70.06 / 56.96	71.95	72.10
MATH	96.40	60.10	69.46 / 56.36	73.17	71.10
CODE	96.60	59.09	70.06 / 56.74	75.61	71.62
PRETRAIN	97.00	63.13	69.46 / 57.34	76.83	72.75
MIXED	97.00	62.63	70.66 / 57.86	76.83	73.00

Table 4: Calibration-corpus ablation for ExFold-P4.

Prefill (ExFold-P4)
Selection criterion	MATH 500	IFEval	GPQA	Eval+	Avg.
rank by $S_{i}$	97.00	82.62	64.65	73.17	79.36
rank by $H_{i}$	97.00	67.47	57.58	75.61	74.42
rank by $S_{i}\times H_{i}$ (Ours)	97.00	84.00	62.63	76.83	80.11
Decode (ExFold-D64)
rank by $S_{i}$	97.00	83.92	60.10	73.17	78.55

Table 5: Retained-expert selection on Qwen3-30B-A3B.

Baseline
Method	Prefill	Decode	MATH 500	AIME25	AIME26	IFEval	IFBench	GPQA	LCB	Eval+	MMLU pro	Avg.
Original TopK=6	100%	100%	93.40	70.42	72.50	82.44	36.23	73.23	54.90	89.25	81.76	72.68
Prefill-Only Acceleration
Prefill TopK=3	50%	100%	92.00	69.17	69.17	80.22	35.67	69.70	54.12	85.59	79.20	70.54
ExFold-P3	50%	100%	93.20	65.83	70.42	82.99	35.28	72.22	55.67	87.65	80.48	71.53
Decode-Only Acceleration

Table 6: DeepSeek-V4-Flash quality. Bold marks the best result per setting. P3 denotes 3 experts per token in prefill, and D

m

means a decode pool with size

m

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Layers	MoE Layers	Routed Experts	Shared Experts	Top- $K$	Hidden Size	Expert FFN
Qwen3-30B-A3B ( 26 )	48	48	128	0	8	2048	768
GLM-4.5-Air ( 32 )	46	45	128	1	8	4096	1408
DeepSeek-V2-Lite-Chat ( 5 )	27	26	64	2	6	2048	1408
Qwen3.5-35B-A3B ( 27 )	40	40	256	1	8	2048	512

Appendix

Table 8: Architectures of the evaluated MoE checkpoints.

Benchmark	Domain	Cases	Samples	Metric
MATH500 ( 10 ; 13 )	Mathematics	500	1	Symbolic-equivalence accuracy
AIME24 ( 23 )	Competition mathematics	30	32 (Qwen3); 8 (GLM)	Mean symbolic-equivalence accuracy
IFEval ( 33 )	Instruction following	541	1	Prompt-level strict accuracy
IFBench ( 25 )	Instruction following	294	1	OpenCompass four-way mean
GPQA-Diamond ( 28 )	Graduate-level reasoning	198	1	Accuracy
LiveCodeBench v5 ( 14 )	Code generation	167	8	Pass@8 / mean@8

Appendix

Table 9: Benchmarks, evaluation sizes, and reported metrics.

Projector	Storage	Online	Kernel
Global scalar	$1$	Scalar	Fused
Layer scalar	$L$	Scalar	Fused
Pair scalar	$P$	Scalar	Fused
Pair diagonal	$PH$	Element-wise	Unfused
Scalar + low-rank	$P+LHR+PRH$	Rank- $R$	Unfused

Appendix

Table 10: Implementation differences among the evaluated projector families.

Method	IFEval	GPQA	Eval+	MMLU pro	Avg.
Original Top-6	42.14	23.23	46.34	26.52	34.56
Direct Top-4	40.85	24.24	46.95	24.71	34.19
ExFold P4+D32	41.96	26.77	48.17	24.75	35.41

Appendix

Table 11: DeepSeek-V2-Lite-Chat quality; bold marks the column best.

Method	MATH 500	IFEval	IFBench	GPQA	Eval+	Avg.
Original Top-8	89.00	87.06	29.20	82.83	88.41	75.30
Direct Top-4	32.80	81.52	33.53	82.32	87.20	63.47
ExFold P4	88.40	82.62	32.59	82.32	92.68	75.72
ExFold D64	89.20	85.21	38.40	81.31	89.63	76.75
ExFold P4+D64	88.40	83.36	33.77	86.36	92.07	76.79

Appendix

Table 12: Qwen3.5-35B-A3B quality; bold marks the column best.

Figure 7: Expert-output magnitudes across all Qwen3 MoE layers.

Figure 8: GLM-4.5-Air expert-output magnitudes (continued); hatching marks one isolated outlier.

Figure 9: Raw expert-output similarity across all Qwen3 MoE layers.

Figure 10: Scalar-aligned similarity for observed Qwen3 expert pairs.

Figure 11: Raw co-routed expert similarity across all GLM-4.5-Air MoE layers.

Figure 12: Scalar-aligned co-routed expert similarity across all GLM-4.5-Air MoE layers.

Figure 13: Common-input scalar-projector matrices across all Qwen3 MoE layers.

Figure 14: Common-input scalar-aligned Frobenius losses across all Qwen3 MoE layers.

Figure 15: Token-level proxy reconstructions. Expert contributions are accumulated strictly head-to-tail; each chain terminates at

\mathbf{y}

\widehat{\mathbf{y}}

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 179 papers

Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Jul 27, 2026Dengke HanExpert Offloading On-Device Language Model Inference
DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding
Aug 14, 2026Zewen Jin, Shen Fu, Zeping Duan +8Mixture-of-Experts Inference Mixture-of-Experts Models
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · Shanxi University

Mar 9, 2026J/K move · Enter open · S save

Speculating Experts Accelerates Inference for Mixture-of-Experts

Vivan Madan, Prajwal Singhania, Abhinav Bhatele, Tom Goldstein+1

University of Maryland · TogetherAI

Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Language Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.

Abstract

Mixture-of-Experts Inference

Explore similar work

Speculating Experts Accelerates Inference for Mixture-of-Experts

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding