Among parallel decoding paradigms, diffusion large language models (dLLMs) have emerged as a promising candidate that balances generation quality and throughput. However, their integration with Mixture-of-Experts (MoE) architectures is constrained by an expert explosion: as the number of tokens generated in parallel increases, the number of distinct experts activated grows nearly linearly. This results in substantial memory traffic that pushes inference into a memory-bound regime, negating the efficiency gains of both MoE and parallel decoding. To address this challenge, we propose Dynamic Expert Sharing (DES), a novel technique that shifts MoE optimization from token-centric pruning and conventional expert skipping methods to sequence-level coreset selection. To maximize expert reuse, DES identifies a compact, high-utility set of experts to satisfy the requirements of an entire parallel decoding block. We introduce two innovative selection strategies: (1) Intra-Sequence Sharing (DES-Seq), which adapts optimal allocation to the sequence level, and (2) Saliency-Aware Voting (DES-Vote), a novel mechanism that allows tokens to collectively elect a coreset based on aggregated router weights. Extensive experiments on MoE dLLMs demonstrate that DES reduces unique expert activations by over 55% and latency by up to 38%, while retaining 99% of vanilla accuracy, effectively decoupling memory overhead from the degree of parallelism.
Figures & tables
Figure 1: From left to right: (a) Growth of unique activated experts vs. block size. (b) Roofline model showing MoE dLLMs are more memory-bound than dense counterparts and how DES mitigates this bottleneck. (c) Activated experts vs. relative accuracy, where DES-Vote extends the Pareto frontier. (d) MoE kernel latency vs. relative accuracy. Accuracy in (c, d) is relative to the vanilla baseline. Setup : single B200, batch size 1, generation length 2048; block size 32 except (a) sweeps 4–128. (a–c) average over HumanEval, MBPP, MATH500, GSM8K; (d) uses MBPP.
Figure 2: Overview of the Dynamic Expert Sharing method. (a) Vanilla MoE dLLM independently routes multiple tokens, leading to many unique activated experts. (b) Dynamic Expert Skipping reduces local per-token computation (indicated by dotted lines) but often fails to optimize the global unique expert load. (c) Dynamic Expert Sharing employs sequence-level coreset selection (via DES-Seq or DES-Vote) to identify high-utility experts globally, minimizing unique expert weights transferred from HBM. (d) DES-Vote enforces sequence-level consensus by aggregating router weights across the parallel block, contrasting with the independent per-token selection in DES-Seq. Greyed out boxes and bars represent experts not selected and not fetched to on-chip memory.
Figure 3: Left: Latency breakdown across MoE FFN, attention, and other components. Right: MoE kernel latency versus activated experts for different block lengths.
Figure 4: Analysis of Dynamic Expert Sharing (DES) in LLaDA-MoE-7B (8 tokens, Layer 10). (Left) Expert Importance: Heatmap of reconstruction loss sensitivity to expert ablation. Darker regions indicate an increase in reconstruction loss when an expert is removed, revealing strong dependencies between specific token positions and experts. (Middle) Selection Overlays: DES-Seq (orange), DES-Vote (pink), and shared (yellow) selections overlaid on log-routing weights. DES-Vote captures high-importance experts (see index 42) that local ranking misses. (Right) Performance vs. Coreset Size ( Mcore ): DES-Vote achieves higher Top- k recall (top) and lower residual reconstruction loss (bottom) than DES-Seq across Mcore . Red dashed lines denote the specific Mcore in the heatmaps.
Figure 5
Method
MBPP
GSM8K
HumanEval
MATH500
Avg. T
Mem.
Avg. R. Acc
T
R. Acc.
T
R. Acc.
T
R. Acc.
T
R. Acc.
GB
%
LLaDA2.0-Mini 16B
Vanilla
78
100.0
87
100.0
84
100.0
85
100.0
84
0.98
100.0
top-k ( k=4 )
52
70.5
58
53.8
56
91.4
58
83.6
56
0.66
74.8
NAEE ( β=0.6 )
60
44.0
67
61.6
67
20.5
66
60.5
65
0.76
46.6
MC-MOE ( β=0.6 )
61
45.9
67
60.2
68
18.0
67
62.7
66
0.77
46.7
Table 1: Main results on generative benchmarks for LLaDA2.0-Mini and LLaDA-MoE-7B. T denotes the average number of unique activated experts per layer. R.Acc denotes accuracy relative to the Vanilla model. Mem. is the average memory footprint of unique activated MoE parameters per layer. All experiments use a block length of 32 with 16 prefix and 16 suffix cache tokens. Best results are bolded ; the superior performer in adjacent DES-Seq vs. DES-Vote pairs is underlined .
Figure 5: (a) Latency measurements for the MoE kernel (left) and total end-to-end GPU kernel execution time (right) across models. (b) Profiling of coreset selection latency: our custom fused kernel achieves a large speedup over the PyTorch baseline.
BS
Vanilla
DES-Vote
T
MoE
E2E
T
MoE
E2E
Lat. (ms)
Lat. (ms)
Lat. (ms)
Lat. (ms)
Homogeneous (MBPP)
1
63.6
8.6
19.4
25.0
5.8
17.5
2
74.1
9.4
20.7
27.9
5.7
17.6
4
77.3
10.2
23.8
28.4
6.0
19.2
Table 2: Batched serving of LLaDA2.0-Mini on a single B200 ( β=0.10 ).
Method
T
R. Acc. (%)
MoE Lat. (ms)
Vanilla
67.9
100.0
5.7
DES-Vote ( β=0.10 )
25.0
100.0
5.1
DES-Vote ( β=0.05 )
12.0
96.4
4.9
Table 3: DES on the 103B LLaDA2.0-flash on 4 × B200 with tensor parallelism (MBPP).
Figure 6: Analysis of efficiency-accuracy trade-offs. All experiments are done with LLaDA2.0-Mini.
Figure 7: (a) Both configurations activate the same number of unique experts, differing only in per-token expert computation ( k=8 vs. k=4 ). (b) (Left) Normalized hit-rate heatmaps across a parallel block, where s denotes the cosine similarity between the coreset and vanilla expert hit rate vectors. (Right) Expert activation frequencies obtained by ranking experts by their average hit counts within each method. Results are on the 10th layer of LLaDA-MoE 7B.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
2 × B6000 (EP=2)
4 × B200 (EP=4)
MoE (ms)
Speedup
MoE (ms)
Speedup
Vanilla
5.1
—
1.4
—
DES-Vote
2.9
1.76 ×
1.3
1.10 ×
Appendix
Table 4: Fused MoE kernel latency under EP for LLaDA2.0-Mini on MBPP. DES-Vote uses β=0.10 .
Setting
MATH500
MBPP
T
R. Acc. (%)
T
R. Acc. (%)
NAEE, flat β=0.2
69.5
39.5
63.0
15.4
NAEE, flat β=0.35 – 0.70
≤ 59.5
≤ 1.3
≤ 52.2
0.0
NAEE, calibrated 25pct
65.5
87.5
≈ 60
76.7
NAEE, calibrated 50pct
52.9
4.0
≈ 44
1.3
DES-Vote ( Mcore=38 )
38
97.4
38
104.7
Appendix
Table 5: NAEE sweep and calibration on LLaDA2.0-Mini. Accuracy is relative to Vanilla .
Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide block expert routing in a coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 16.6 while retaining 99.55% of the original performance. Meanwhile, it reduces memory usage by 74.36% to 76.78% and achieves 1.36× to 2.04× end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE.
Semi-autoregressive diffusion large language models (dLLMs) improve decoding parallelism through iterative block-wise denoising, but scaling them with mixture-of-experts (MoE) layers introduces a large expert parameter footprint that exceeds memory-constrained GPU capacity. Expert offloading is a natural remedy, yet existing MoE serving systems target autoregressive decoding and rely on intra-iteration layer-wise prefetching: while computing one layer, they predict and load experts for subsequent layers. Under dLLM inference, block-wise routing expands the active expert working set within each iteration, making such prefetches difficult to complete in time and costly when mispredicted. Consequently, existing prefetch-based solutions often degenerate into on-demand expert loading with high decoding latency. We propose OLED-MoE, an expert offloading system that shifts the optimization target from intra-iteration prefetching to inter-iteration expert retention. Its key insight is that adjacent denoising iterations exhibit strong expert routing overlap, and token confidence indicates which experts are likely to be reused. OLED-MoE uses confidence-guided inter-iteration prediction to retain high-value experts in GPU memory without introducing extra prefetch traffic. It further compensates unavoidable cache misses through CPU-GPU cooperative execution, jointly considering dynamic expert computation load and predicted future reuse. Across diverse dLLM workloads, OLED-MoE reduces time per output token (TPOT) by 1.23x-7.93x and improves expert cache utilization by 1.44x-4.23x over state-of-the-art offloading systems. Notably, OLED-MoE approaches full-residency performance while using only 40% of the expert GPU memory, incurring merely 23% higher TPOT despite a 60% reduction in expert memory footprint. OLED-MoE's source code is publicly available at https://github.com/flashserve/OLED-MoE.
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.