Mixture-of-experts (MoE) models expand model capacity by activating only a subset of experts for each token. On accelerators with static execution plans, expert capacities and placement are fixed before routing reveals each expert's load. Uniform worst-case capacities waste work on padding; smaller capacities tied to expert identities can overflow. We present RankLane, which ranks each device's experts by current load and assigns them to fixed-capacity lanes. Runtime indices select experts and token rows while tier populations, shapes and device placement remain fixed. An aggregation graph preserves device ownership and reduces each token's local contributions on one core. Routing calibration and measured execution costs guide tier selection; overflowing chunks rerun at full capacity. We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights on four Cloud AI 100 Ultra SoCs, using 128-2048-token chunks and including overflow detection and in-program recovery support. Against a dense baseline, RankLane achieves end-to-end prefill speedups of 1.27-4.18x on Qwen3 and 1.93-3.46x on Gemma.
Figures & tables
Figure 1: Static execution and changing expert demand. (a) The evaluated Cloud AI 100 graph path fixes lane capacities, shapes and placement. (b) For eight top-2 tokens on one device, uniform capacity reserves 32 rows for 16 assignments; smaller capacities tied to expert identities overflow when the busy experts change. (c) Ranking fits both illustrated chunks into the same 16-row lane layout, followed by local aggregation.
Figure 2: Expert-load envelopes from device routing captures. Each panel uses 200 calibration and 200 held-out chunks, with matched counts per workload across eight workloads. Calibration fixes the mean-load expert order; curves summarize held-out loads. Red: fixed order. Blue: runtime rank. Black: deployed RankLane capacities. Dotted: capacity T .
Figure 3: RankLane overview. Offline calibration and measured costs select fixed lane counts and capacities. At run time, each SoC (1) ranks its experts, (2) gathers weights and tokens into fixed-capacity lanes, and (3) combines outputs by token on local cores; (4) an elementwise sum combines device partials. After all layers, a host check triggers a capacity- T rerun if any lane overflows.
Figure 4: Aggregation placement on the evaluated compiler. (a) Global indices cause expert-output redistribution. (b) A SoC-batched gather preserves device locality, but lane-indexed reduction exchanges operands across cores. (c) RankLane also completes each local token sum on one core. All variants retain the final cross-SoC sum; § 5.4 measures these placement effects.
T
C1/C2/C3
Tiered rows
Uniform rows
128
128/48/16
1,664
4,096
256
256/64/32
3,072
8,192
512
512/128/64
6,144
16,384
Table 1: Qwen3 tiers: 8/8/16 lanes per device. Provisioned rows are 8C1+8C2+16C3 , compared with 32T for uniform capacity- T lanes.
Figure 5: We measure batch-one prefill with MXFP6 weights on a four-SoC Cloud AI 100 Ultra card across 128–2048-token chunks. RankLane speeds up full-model prefill by 1.27–4.18 × on Qwen3 and 1.93–3.46 × on Gemma over the dense baseline.
Figure 6: Under the setup of Figure 5 , we successively add token-owned reduction, ranked tiers and recovery support to the dense baseline. Across T=128 –2048, reduction gives 1.21–2.32 × on Qwen3 and 1.22–1.95 × on Gemma over the dense baseline. Ranked tiers add a further 1.10–2.11 × and 1.60–2.00 × , respectively, over reduction alone, before recovery overhead.
T
Assignment
Rows/SoC
Attempt (ms)
Overflow chunks
128
Calibration-fixed
1,664
111.1
782/782
Runtime-ranked
1,664
105.4
0/782
256
Calibration-fixed
3,072
145.3
609/609
Runtime-ranked
3,072
141.7
0/609
512
Calibration-fixed
6,144
235.3
443/443
Runtime-ranked
6,144
225.5
0/443
Table 2: Qwen3 assignment at matched tier capacities. In held-out trace replay, fixed assignment overflows every chunk; runtime ranking overflows none. Attempt timings exclude recovery, required by all fixed-assignment timing runs.
Figure 7: Qwen3 full-model prefill under uniform and ranked capacities. Token-owned aggregation enables 9–27% latency reductions from tiering; dense aggregation yields a 37% slowdown at T=128 and only a 4% reduction at T=512 .
Selection
Widths
Rows/SoC
ms
Manual
8/8/8/8
6,144
309.9
Alternative
12/11/9
7,488
294.2
Latency-guided
11/12/9
7,264
283.5
Table 3: Gemma prefill at T=512 with recovery support. Latency-guided tiers cut latency by 8.5% versus manual tiers despite 18.2% more rows, showing that reserved capacity alone does not predict performance.
Figure 8: Predicted versus measured Qwen3 MoE spans at T=512 for 48 programs, using the full fitted cost table. Predictions track the measured cost differences with 4.3% mean absolute percentage error; close candidates still require measurement to resolve their ordering.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
T
r=1
2
4
8
9
16
17
32
128
127
122
105
33
32
14
12
3
256
256
223
134
58
50
25
25
4
512
508
424
240
101
89
51
50
10
Appendix
Table 4: Largest load of the rank- r expert of a SoC over all calibration chunks, layers and SoCs.
T=128
T=256
T=512
Qwen3-30B-A3B
B gathered, fixed order, T
—
160.2
302.4
C gathered, ranked, T
115.6
163.3
308.3
D ranked, two tiers
106.4
138.1
269.8
E RankLane, three tiers
105.1
—
224.8
C/E speedup
1.10 ×
—
1.37 ×
Appendix
Table 5: Tier-assignment ablation: full-model latency (ms per chunk), with token-owned reduction in every program. B → C adds ranking; C → D/E adds tiers. Dashes omit historical variants outside the common configuration. All tiered programs here are single-profile; recovery is included in Figure 6 . Row budgets are per SoC.
Figure 9: Equivalent arithmetic, different placement. Three expressions of the same per-SoC token aggregation, with bit-identical logits at uniform capacity T with static weights. Left: full-model latency relative to the layout-preserving SoC axis. Right: cross-SoC traffic inside one MoE layer at T=512 , from instrumented two-layer profiles.
Lane-indexed
Token-owned
Full-model latency (ms)
401.9
349.3
Cross-SoC traffic (MiB)
12.1
14.0
Local-sum inter-core traffic (MiB)
241.9
1.9
Local-sum busy time (core-ms)
47.1
2.4
Appendix
Table 6: Local-reduction ablation at T=512 , uniform capacity T and static weights. Full-model latency is uninstrumented; traffic and local-sum busy time are from layer 1 of instrumented two-layer cuts, summed across SoCs.
Figure 10: Qwen3 cost table at T=512 : (a) stage GEMM span and (b) transition gap, measured on layer 1 of instrumented two-layer cuts; boxes mark deployed tiers. Figure 8 compares predicted and measured MoE spans.
T
Maxima
Deployed
h=1.28
h=1
128
127/30/12
128/48/16
128/48/16
128/32/16
256
254/56/26
256/64/32
256/80/48
256/64/32
512
509/98/44
512/128/64
512/128/64
512/112/48
Appendix
Table 7: Qwen3 search inputs: maxima at ranks 1/9/17 from expanded captures (401/400/252 chunks). Deployed capacities use the original captures in Appendix C . Headroom rules define separate comparisons.
T
h
Widths
Predicted
Measured
128
1.28
8/8/16 ∗
2.04
1.94
16/16
2.09
1.98
256
1.00
8/16/8 ∗
2.38
2.35
9/7/16
2.43
2.32
8/8/16
2.48
2.39
512
1.28
8/8/16 ∗
3.38
3.32
Appendix
Table 8: Qwen3 candidate selection: predicted and measured layer-1 MoE spans (ms) in instrumented two-layer cuts. Each group fixes calibration and h ; ∗ marks the selected candidate ( T=512 : the deployed choice, within 0.1% of the minimum score). Capacities follow Eq. ( 4 ); The comparison excludes arbitrary capacity sweeps and the 12-layer cut.
Program
T=128
T=256
T=512
Dense baseline
0.199 ✓
0.072 ✓
0.087 ✓
RankLane
0.188 ✓
0.091 ×
0.096 ✓
Appendix
Table 9: Relative L2 error of last-position logits against FP32; ✓ indicates the same next token.
Model / T
128
256
512
1024
2048
Qwen3
127.4
202.2
344.4
1065.9
2285.2
Gemma
140.1
214.7
390.6
1197.6
2476.6
Appendix
Table 10: Additional total latency when recovery is invoked (ms), including the capacity- T rerun and host checks. Adding it to the normal-path latency in Figure 5 gives the recovered-chunk cost in § 5.6 , up to rounding.
Stressed chunk
Direct reference
With actual overflow
0
710.5
863.7
1
709.3
859.5
3
706.8
861.7
Appendix
Table 11: Gemma actual-overflow validation at T=256 : median four-chunk prompt latency (ms), pooled over three inputs and two mirrored passes. The direct reference uses capacity T at the specified chunk, with no failed attempt. The three-profile normal schedule takes 659.4 ms; these stress timings are separate from the deployed two-profile results.
T=128
T=256
T=512
Same capacities as deployed RankLane
Rows
1,664
3,072
6,144
Runtime rank, overflow
0%
0%
0%
Each of four fixed policies
100%
100%
100%
Capacities selected for zero calibration overflow
Calibration-best, rows
3,840
7,040
13,056
Appendix
Table 12: Fixed assignment versus runtime ranking. Overflow is the percentage of evaluation chunks with any lane over capacity. Rows are per SoC. The lower panel reports minimum row budgets with zero calibration overflow and their evaluation overflow in parentheses. The evaluation-set qualification in § 5.6 applies.
T
C1/C2/C3
Tiered rows
Uniform rows
128
128/32/16
1,936
4,096
256
256/64/16
3,728
8,192
512
512/112/32
7,264
16,384
1024
1024/192/64
14,144
32,768
2048
2048/384/128
28,288
65,536
Appendix
Table 13: Delivered Gemma capacities for 11/12/9 lanes. Rows are provisioned per SoC; uniform capacity reserves 32T .
Workload
T=128
T=256
T=512
GSM8K
1.91 ×
2.23 ×
3.45 ×
HumanEval
1.94 ×
2.25 ×
3.46 ×
JavaScript
1.93 ×
2.25 ×
3.48 ×
MMLU
1.91 ×
2.23 ×
3.45 ×
SWE-bench
1.93 ×
2.24 ×
3.45 ×
Alpaca
1.87 ×
2.18 ×
3.37 ×
Appendix
Table 14: Final Gemma program: per-workload dense-to-complete speedup at 11/12/9 widths. Each entry uses all held-out inputs of that workload at the given length; zero actual overflows occur.
Table 16: Gemma simulator errors relative to measured layer-1 MoE spans. All values are percentages; signed error is (predicted − measured) / measured.
Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose ZeRO-Prefill, a prefill-only serving system built on this idea. Its backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than by activation; its co-designed frontend enforces a physically derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.34-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.
Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5
University of Virginia · Snowflake AI Research · Harvard University
Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference phases with fundamentally different bottlenecks: prefill is dominated by token-wise expert computation, whereas decode is constrained by memory traffic from the batch-wise activated expert set. However, existing training-free acceleration methods optimize only a single resource proxy, either the experts each token executes or the experts a batch activates, and either discard the excluded experts' contribution or leave it only implicitly approximated. In this paper, we propose ExFold, a unified training-free expert-folding framework for jointly accelerating MoE prefill and decode. ExFold casts both prefill and decode as one budgeted output-approximation problem: execute only a phase-specific constrained expert set while projecting the contribution of budget-excluded experts onto retained experts using calibrated scalar projectors. Motivated by the observation that many expert outputs are directionally aligned but differ in magnitude, ExFold calibrates a pairwise scalar-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts. Under this view, prefill acceleration becomes token-level Top-K folding, and decode acceleration becomes batch-level expert-pool folding. The two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared folding mechanism. We implement ExFold as a plug-and-play plugin in vLLM, with a lightweight expert-folding CUDA kernel, delivering up to 1.41x TTFT and 2.45x TPOT speedups while retaining about 99% of the original average quality.
Juntong Wu, Yifei Liu, Junyi Chen +6
Xiaohongshu Inc. · Shenzhen Graduate School, Peking University · Shanghai Jiao Tong University +1
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.