One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink
Authors: Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou
Expert parallelism (EP) shards the experts of a Mixture-of-Experts (MoE) layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with training, inference processes relatively few tokens per GPU at a time, and at small batch sizes fixed costs that do not grow with the number of tokens, rather than data movement, dominate the latency. We present NVLinkOneSided, a one-sided EP all-to-all for GPUs within a single NVLink domain. Each source rank pushes its tokens, without first exchanging metadata, into rank-major buffers on the target GPUs, and the grouped GEMMs read them in place instead of from a per-expert copy. We provide two protocols for the synchronization that the communication requires, namely Fence and Sentinel, for high-throughput and low-latency scenarios respectively. On Blackwell GPUs, NVLinkOneSided completes dispatch and combine 1.2-3.0× faster than DeepEP, with the largest speedups at small batch sizes. The resulting MoE layer is up to 2.4× faster than DeepEP with DeepGEMM. It even outperforms the fused DeepGEMM MegaMoE at up to 128 tokens per GPU, making it still competitive for decoding. We expose NVLinkOneSided through a modular interface, so that it is agnostic to the activation quantization format, decoupled from the MoE compute backend and easy to integrate into inference frameworks.
Figures & tables
Figure 1: Example used throughout the paper: two ranks, experts E0 , E1 on R0 and E2 , E3 on R1 , and top-2 routing. R0 holds tokens T0 – T3 and R1 holds T4 – T6 . Beside each token are its two expert ids and, through the routing (arrow), the target rank of each: both experts of T1 are on R0 , both experts of T3 and T5 are on R1 , and every other token reaches both ranks. The braces on R1 mark its number of tokens, T=3 , and the largest number of tokens of any rank, T^=4 (Table 2 ).
NVLink domain
RDMA NIC
Initiator
any thread, load/store
work request to the NIC
Transfer size
store ( ≤ 32 B) or TMA copy
message, KB scale
Ordering
PTX fences, system scope
per queue pair
Arrival in a message
whole 32-bit words
unspecified
Table 1: Remote access within an NVLink domain and over RDMA (Blackwell, per GPU).
Symbol
Meaning
Name in the implementation
P
EP ranks
ep_size
k
experts per token
top_k
E
experts of the layer
num_experts
Eℓ
experts per rank
num_local_experts
T
tokens of a rank in the step
local_num_tokens
T^
largest T across ranks
runtime_max_tokens_per_rank
Table 2: Notation and the corresponding parameter names in the implementation.
Figure 2: Data path for the routing of Figure 1 , with the Fence synchronization protocol. Dispatch: each source rank stores its tokens into its own region of the dispatch workspaces of the target ranks and records the target rank and slot of each selection; T1 selects E0 and E1 , both on R0 , so it takes one slot there and its second selection records no slot ( −1 ). MoE compute turns the tokens from each source rank into partial results at the same slots of the combine workspace. Combine: each source rank loads the partial results from the target ranks and slots recorded in dispatch and adds them. White slots are empty.
Figure 3: The MoE layer on R0 , detailing Figure 2 ; MoE compute comprises indexing, FC1, FC2 and finalize, and slots are numbered 0–7 across the two regions. The indexing kernel turns the expert ids of the received slots into per-expert lists of slot indices (empty slots 3, 6 and 7 carry invalid ids); FC1 gathers tokens through these lists and multiplies them by the expert weights, FC2 multiplies the FC1 output by its own weights, and finalize adds the outputs of each token into its slot of the combine workspace.
Figure 4: The dispatch workspace of R1 at the end of dispatch. (a) Fence gathers the tokens of each source rank into a prefix of its region, so each source rank sends its slot count C ; the slots past the count get invalid expert ids ( −1 ). (b) Sentinel pushes token i of each source rank into slot i of its region, so every slot signals its own arrival; a slot without a token keeps the sentinel payload (hatched) and gets invalid expert ids. Bottom: the critical path of dispatch until MoE compute starts; Sentinel shortens it, which pays off at small batches.
DeepSeek-V4.1-Flash
Kimi-K3
Hidden size
5120
3584 a
Experts, top- k
384, 6
896, 16
Expert intermediate size
2304
3072
Dispatch bytes (MXFP8)
5328
3824
Combine bytes (BF16)
10240
7168
a Latent MoE; the original hidden size is 7168.
Table 3: Configuration of the MoE layers of the tested models, with the bytes per token and target rank.
Figure 5: NVLinkOneSided completes dispatch and combine faster than DeepEP at every batch. Round trip on eight B300 GPUs; bottom: speedup over DeepEP.
Figure 6: Sentinel shortens the work around the wait for peers, not the wait. Critical path inside the dispatch and combine kernels, from IKET timestamps; DeepSeek-V4.1-Flash on eight B300 GPUs. Launch, PDL waits include Fence ’s counter-reset kernel; election, fence, flags: Fence ’s serial chain; sanitize: Fence ’s kernel that invalidates the expert ids past the slot counts, mostly overlapped with dispatch; copy-out, reset: Sentinel ’s handling of received slots; other: kernel exit and Sentinel ’s local reduction.
Figure 7: Low-precision formats accelerate dispatch and combine at large batches; dashed: the byte ratio, their theoretical limit. NVLinkOneSided with Fence , DeepSeek-V4.1-Flash on eight B300 GPUs; top: latency of each phase; bottom: speedup over BF16.
NVLinkOneSided Fence
NVLinkOneSided Sentinel
DE+DG
MegaMoE
Tokens/rank
MoE layer
stages
MoE layer
stages
MoE layer
stages
MoE layer
DeepSeek-V4.1-Flash
1
46.0
12.8 / 19.6 / 14.4
36.7
8.3 / 23.9 / 9.8
74.7
21.4 / 26.0 / 27.2
49.8
16
79.6
13.3 / 49.9 / 17.2
70.5
8.5 / 54.7 / 12.9
122.0
22.0 / 68.7 / 31.4
90.3
128
188.5
16.3 / 150.0 / 23.1
183.3
13.5 / 150.6 / 21.5
235.9
25.8 / 168.7 / 41.4
188.7
256
208.1
22.2 / 153.4 / 33.6
206.5
20.5 / 154.9 / 33.8
251.2
29.8 / 170.6 / 50.8
194.1
Table 4: MoE layer latency (µs) on eight B300 GPUs: layer time and dispatch / MoE compute / combine stages (MegaMoE: one kernel); fastest layer in bold.
DeepSeek-V4.1-Flash
Kimi-K3
Communication
MoE layer
Communication
MoE layer
Tokens/rank
α
Imba.
Ours
DeepEP
Ours
DE+DG
MegaMoE
Imba.
Ours
DeepEP
Ours
DE+DG
MegaMoE
1
0
1.50
16.1
47.6
52.2
97.5
68.3
1.38
22.4
58.0
84.6
148.0
114.2
0.5
1.50
15.8
49.4
57.8
105.2
73.0
1.25
28.2
68.5
81.0
146.1
106.8
1
1.50
15.8
52.6
52.4
99.3
67.6
1.38
30.0
65.4
71.1
133.4
96.6
1.25
1.67
15.9
51.6
52.3
97.9
69.2
1.62
22.1
62.2
66.4
129.4
95.4
Table 5: MoE layer under Zipf routing with exponent α ( α=0 : uniform) on eight B300 GPUs, latency in µs. Imba. (load imbalance) is the ratio of the token–expert routing pairs that the busiest rank computes to those that each rank computes under perfect routing; 1 means balanced. Communication is the dispatch and combine of the busiest rank. Ours: NVLinkOneSided with the faster protocol + Prims-TS; DE+DG: DeepEP + DeepGEMM.
Figure 8: At one token per rank, Sentinel ’s round trip grows least from 8 to 64 GPUs. All-to-all round trip on NVL72 relative to the same implementation on 8 GPUs, at a fixed number of tokens per rank.
EP size (GPUs)
T
8
16
32
64
1
Ours
35.4
35.4
35.7
36.9
DE+DG
78.1
78.0
79.1
82.3
MegaMoE
45.7
46.3
47.2
49.7
16
Ours
66.4
51.9
47.7
49.0
DE+DG
124.5
100.6
88.9
90.9
Table 6: MoE layer latency (µs) of DeepSeek-V4.1-Flash on one GB200 NVL72 rack, by EP size; Ours: the faster protocol. Fastest layer in bold.
Table A.3: MoE layer latency (µs), DeepSeek-V4.1-Flash, 8 × B300, MXFP4 weights and MXFP8 activations: layer time and dispatch / MoE compute / combine, each stage from its first kernel’s start to its last kernel’s end.
NVLinkOneSided Fence
NVLinkOneSided Sentinel
MegaMoE
DE+DG
Tokens/rank
MoE layer
stages
MoE layer
stages
MoE layer
MoE layer
stages
1
47.7
13.1 / 23.5 / 15.1
42.1
8.8 / 28.1 / 13.5
60.4
99.7
26.5 / 37.8 / 35.3
2
52.7
13.0 / 27.8 / 16.0
47.2
8.9 / 32.6 / 14.9
67.5
109.2
26.0 / 45.5 / 37.4
4
61.8
13.1 / 37.7 / 15.5
57.4
8.9 / 42.6 / 15.2
78.6
120.6
24.2 / 58.3 / 38.2
8
84.7
14.1 / 59.4 / 15.5
80.7
9.2 / 65.2 / 16.0
106.7
146.2
23.4 / 85.0 / 37.8
16
128.4
13.4 / 100.9 / 15.0
122.0
9.4 / 106.5 / 12.5
148.6
191.2
23.5 / 129.4 / 38.2
Appendix
Table A.4: MoE layer latency (µs), Kimi-K3, 8 × B300, MXFP4 weights and MXFP8 activations: layer time and dispatch / MoE compute / combine, each stage from its first kernel’s start to its last kernel’s end.
Dispatch
Combine
Bytes
Tokens per rank
disp. / comb.
1
128
1024
4096
DeepSeek-V4.1-Flash
BF16
BF16
10288 / 10240
29.0
46.3
194.3
684.5
MXFP8
BF16
5328 / 10240
28.7
40.7
154.9
532.7
FP8 (1 × 128)
BF16
5328 / 10240
28.6
41.1
155.3
532.7
NVFP4
BF16
2928 / 10240
28.1
37.6
137.4
456.0
Appendix
Table A.5: Fence with other payload formats on eight B300 GPUs: bytes per token and target rank, and round trip (µs). The last two rows of each model include quantization to MXFP8, on the source rank or, after a BF16 dispatch, on the target rank.
DeepSeek-V4.1-Flash
Kimi-K3
Routing
T
Ours
DE+DG
MegaMoE
Ratio
Imba.
Ours
DE+DG
MegaMoE
Ratio
Imba.
Perfect
1
36.7
74.7
49.8
0.74
1.00
42.1
99.7
60.4
0.70
1.00
16
70.5
122.0
90.3
0.78
1.00
122.0
191.2
148.6
0.82
1.00
128
183.3
235.9
188.7
0.97
1.00
346.4
430.8
358.5
0.97
1.00
1024
353.8 †
409.6
255.3
1.39
1.00
719.5 †
976.3
535.8
1.34
1.00
Zipf, α=0
1
52.2
97.5
68.3
0.76
1.50
84.6
148.0
114.2
0.74
1.38
Appendix
Table A.6: MoE layer under Zipf routing with exponent α on eight B300 GPUs: layer time (µs) of Ours, DE+DG and MegaMoE. Ratio is ours relative to MegaMoE, and Imba. is the load imbalance of Table 5 . Ours: NVLinkOneSided with the faster protocol + Prims-TS ( † : Fence ); DE+DG: DeepEP + DeepGEMM.
DeepSeek-V4.1-Flash
Kimi-K3
Ranks
T
Fence
Sentinel
DeepEP
Fence
Sentinel
DeepEP
8
1
43.0
17.5
44.7
34.6
17.5
53.6
16
43.1
17.8
44.2
36.0
18.2
57.6
128
48.9
31.7
57.8
44.1
31.5
70.9
1024
160.1
151.6
173.9
150.6
144.0
179.0
16
1
41.8
16.6
45.9
48.4
21.5
80.0
Appendix
Table A.7: All-to-all round trip (µs, dispatch + combine) on one GB200 NVL72 rack with perfect routing, by expert-parallel ranks (on ranks / 4 compute trays) and tokens per rank.
DeepSeek-V4.1-Flash
Kimi-K3
Ranks
T
Ours
DE+DG
MegaMoE
Ours
DE+DG
MegaMoE
8
1
35.4
78.1
45.7
40.7
116.5
57.3
16
66.4
124.5
83.4
117.5
200.6
143.8
128
181.7
237.7
179.3
341.5
429.9
344.7
256
208.4
256.9
188.1
491.5
517.4
373.3
512
348.2
317.2
222.0
624.3
728.2
473.1
Appendix
Table A.8: MoE layer latency (µs) on one GB200 NVL72 rack with perfect routing, by expert-parallel ranks and tokens per rank: NVLinkOneSided (the faster protocol; † : Fence ), DeepEP + DeepGEMM and MegaMoE. –: not tested, as the Prims-TS grouped GEMMs need at least k experts per rank (Kimi-K3 has 14 on 64 GPUs).
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University