RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities
Organizations: University of California, San Diego · Qualcomm AI Research
Abstract
Mixture-of-experts (MoE) models expand model capacity by activating only a subset of experts for each token. On accelerators with static execution plans, expert capacities and placement are fixed before routing reveals each expert's load. Uniform worst-case capacities waste work on padding; smaller capacities tied to expert identities can overflow. We present RankLane, which ranks each device's experts by current load and assigns them to fixed-capacity lanes. Runtime indices select experts and token rows while tier populations, shapes and device placement remain fixed. An aggregation graph preserves device ownership and reduces each token's local contributions on one core. Routing calibration and measured execution costs guide tier selection; overflowing chunks rerun at full capacity. We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights on four Cloud AI 100 Ultra SoCs, using 128-2048-token chunks and including overflow detection and in-program recovery support. Against a dense baseline, RankLane achieves end-to-end prefill speedups of 1.27-4.18x on Qwen3 and 1.93-3.46x on Gemma.
Figures & tables
| Tiered rows | Uniform rows | ||
|---|---|---|---|
| 128 | 128/48/16 | 1,664 | 4,096 |
| 256 | 256/64/32 | 3,072 | 8,192 |
| 512 | 512/128/64 | 6,144 | 16,384 |
| Assignment | Rows/SoC | Attempt (ms) | Overflow chunks | |
|---|---|---|---|---|
| 128 | Calibration-fixed | 1,664 | 111.1 | 782/782 |
| Runtime-ranked | 1,664 | 105.4 | 0/782 | |
| 256 | Calibration-fixed | 3,072 | 145.3 | 609/609 |
| Runtime-ranked | 3,072 | 141.7 | 0/609 | |
| 512 | Calibration-fixed | 6,144 | 235.3 | 443/443 |
| Runtime-ranked | 6,144 | 225.5 | 0/443 |
| Selection | Widths | Rows/SoC | ms |
|---|---|---|---|
| Manual | 8/8/8/8 | 6,144 | 309.9 |
| Alternative | 12/11/9 | 7,488 | 294.2 |
| Latency-guided | 11/12/9 | 7,264 | 283.5 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| 2 | 4 | 8 | 9 | 16 | 17 | 32 | ||
|---|---|---|---|---|---|---|---|---|
| 128 | 127 | 122 | 105 | 33 | 32 | 14 | 12 | 3 |
| 256 | 256 | 223 | 134 | 58 | 50 | 25 | 25 | 4 |
| 512 | 508 | 424 | 240 | 101 | 89 | 51 | 50 | 10 |
| Qwen3-30B-A3B | |||
|---|---|---|---|
| B gathered, fixed order, | — | 160.2 | 302.4 |
| C gathered, ranked, | 115.6 | 163.3 | 308.3 |
| D ranked, two tiers | 106.4 | 138.1 | 269.8 |
| E RankLane, three tiers | 105.1 | — | 224.8 |
| C/E speedup | 1.10 | — | 1.37 |
| Lane-indexed | Token-owned | |
|---|---|---|
| Full-model latency (ms) | 401.9 | 349.3 |
| Cross-SoC traffic (MiB) | 12.1 | 14.0 |
| Local-sum inter-core traffic (MiB) | 241.9 | 1.9 |
| Local-sum busy time (core-ms) | 47.1 | 2.4 |
| Maxima | Deployed | |||
|---|---|---|---|---|
| 128 | 127/30/12 | 128/48/16 | 128/48/16 | 128/32/16 |
| 256 | 254/56/26 | 256/64/32 | 256/80/48 | 256/64/32 |
| 512 | 509/98/44 | 512/128/64 | 512/128/64 | 512/112/48 |
| Widths | Predicted | Measured | ||
| 128 | 1.28 | 8/8/16 ∗ | 2.04 | 1.94 |
| 16/16 | 2.09 | 1.98 | ||
| 256 | 1.00 | 8/16/8 ∗ | 2.38 | 2.35 |
| 9/7/16 | 2.43 | 2.32 | ||
| 8/8/16 | 2.48 | 2.39 | ||
| 512 | 1.28 | 8/8/16 ∗ | 3.38 | 3.32 |
| Program | |||
|---|---|---|---|
| Dense baseline | 0.199 ✓ | 0.072 ✓ | 0.087 ✓ |
| RankLane | 0.188 ✓ | 0.091 | 0.096 ✓ |
| Model / | 128 | 256 | 512 | 1024 | 2048 |
|---|---|---|---|---|---|
| Qwen3 | 127.4 | 202.2 | 344.4 | 1065.9 | 2285.2 |
| Gemma | 140.1 | 214.7 | 390.6 | 1197.6 | 2476.6 |
| Stressed chunk | Direct reference | With actual overflow |
|---|---|---|
| 0 | 710.5 | 863.7 |
| 1 | 709.3 | 859.5 |
| 3 | 706.8 | 861.7 |
| Same capacities as deployed RankLane | |||
|---|---|---|---|
| Rows | 1,664 | 3,072 | 6,144 |
| Runtime rank, overflow | 0% | 0% | 0% |
| Each of four fixed policies | 100% | 100% | 100% |
| Capacities selected for zero calibration overflow | |||
| Calibration-best, rows | 3,840 | 7,040 | 13,056 |
| Tiered rows | Uniform rows | ||
|---|---|---|---|
| 128 | 128/32/16 | 1,936 | 4,096 |
| 256 | 256/64/16 | 3,728 | 8,192 |
| 512 | 512/112/32 | 7,264 | 16,384 |
| 1024 | 1024/192/64 | 14,144 | 32,768 |
| 2048 | 2048/384/128 | 28,288 | 65,536 |
| Workload | |||
|---|---|---|---|
| GSM8K | 1.91 | 2.23 | 3.45 |
| HumanEval | 1.94 | 2.25 | 3.46 |
| JavaScript | 1.93 | 2.25 | 3.48 |
| MMLU | 1.91 | 2.23 | 3.45 |
| SWE-bench | 1.93 | 2.24 | 3.45 |
| Alpaca | 1.87 | 2.18 | 3.37 |
| Held-out source | CV headroom | Fixed |
|---|---|---|
| GSM8K | 0/149 | 0/149 |
| HumanEval + JavaScript | 1/81 | 0/81 |
| MMLU | 0/400 | 0/400 |
| SWE-bench | 0/153 | 0/153 |
| Alpaca | 0/14 | 0/14 |
| CNN/DailyMail | 0/84 | 0/84 |
| Validation programs | Mean abs. | Max abs. | Mean signed |
|---|---|---|---|
| 19 independent structures | 3.9 | 9.5 | |
| 20 search candidates | 10.9 | 16.8 |
Mixture-of-Experts Inference
Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.
See all 200 papers