GC-MoE on SM120: Overlapping Expert-Parallel Token Dispatch with Warp-Level FP8xFP4 Block-Scaled MMA on Consumer Blackwell
Organizations: Alibaba Inc.
Abstract
Expert-parallel Mixture-of-Experts decoding is latency bound: each layer serializes a token exchange, a grouped GEMM over routed experts, and a combine, while only a few tokens per rank are in flight. Prior GC-MoE fused these phases into one persistent, Green-Context-partitioned CUDA Graph on Hopper (SM90) and datacenter Blackwell (SM100). This paper rebuilds the operator for SM120 (compute capability 12.0, e.g. RTX PRO 5000) over PCIe. SM120 exposes block-scaled tensor cores only through warp-level mma.sync.aligned.kind::mxf8f6f4: no tensor memory, no two-CTA MMA, no cluster multicast. We rebuild the kernel around a swap-AB warp-level block-scaled MMA with FP4 weights in the register A operand and FP8 activations in B, plus SM120-specific mechanisms: scale-byte selection through MMA immediates, a compact-weights path, an adaptive token-tile geometry, an expert-granular streaming dispatch with a batched bulk-pull engine and a 64-bit deduplicating pull cache, full-width BF16 returns drained by resident relay copiers over a position-major arena, and shared experts folded into the compute role. On 8xRTX PRO 5000 over PCIe the operator is 1.522x and 1.387x faster than the FlashInfer CuTe DSL W4A8 MegaMoE and the Cake MegaMoE across 18 decode points (18/18 wins each), and 1.388x faster than Cake with shared experts fused; all 94 numerical cases and 752 per-rank checks pass, at cross-operator cosine similarity of at least 0.9999997.
Figures & tables
| Role | Threads | Context | Work |
|---|---|---|---|
| 1 | 256 | dispatch (6 SMs) | scan routes, pull peer rows, publish experts |
| 2 | 192 | compute | grouped GEMM1+SwiGLU, GEMM2, return staging, relay, |
| shared-expert FC1/FC2 | |||
| 3 | 384 / 256 | compute | combine partial sums, BF16 store of |
| Geometry | GLM5.2 ( ) | DSV4.1-Flash ( ) |
|---|---|---|
| : rows/expert, tile, stages | , , | , , |
| : rows/expert | ||
| tile_n / stages | / | / |
| compact weights | yes | yes |
| handoff: global + shared chunks | yes | yes |
| batched BF16 returns | yes | yes |
| Item | Value |
|---|---|
| Hardware | 8 NVIDIA RTX PRO 5000 72 GB (SM120), PCIe |
| Software | GC MoE 0.6.0, FlashInfer 0.7.3, ptxas 13.2.78 |
| Precision | FP8 (E4M3) activations, MXFP4 routed weights |
| Baselines | FlashInfer CuTe DSL W4A8 MegaMoE; Cake MegaMoE ( all_lsa ) |
| Routing | same pre-quantized inputs, weights and routes for all operators |
| Capacity | 256 tokens per rank (largest ); SwiGLU clamp 10 |
| Model | routed experts | top- | ||
|---|---|---|---|---|
| GLM5.2 | 6144 | 2048 | 256 | 8 |
| DSV4.1-Flash | 5120 | 2304 | 384 | 6 |
| tokens | CuTe DSL | Cake | GC | DSL/GC | Cake/GC | |
|---|---|---|---|---|---|---|
| 1 | 8 | 268.47 | 320.95 | 190.39 | 1.410 | 1.686 |
| 2 | 16 | 429.01 | 464.45 | 306.73 | 1.399 | 1.514 |
| 4 | 32 | 571.58 | 592.61 | 418.89 | 1.365 | 1.415 |
| 8 | 64 | 743.89 | 726.91 | 522.10 | 1.425 | 1.392 |
| 16 | 128 | 798.96 | 768.70 | 567.56 | 1.408 | 1.354 |
| 32 | 256 | 809.59 | 793.29 | 560.41 | 1.445 | 1.416 |
| tokens | CuTe DSL | Cake | GC | DSL/GC | Cake/GC | |
|---|---|---|---|---|---|---|
| 1 | 8 | 273.78 | 337.25 | 190.89 | 1.434 | 1.767 |
| 2 | 16 | 398.74 | 435.42 | 281.11 | 1.418 | 1.549 |
| 4 | 32 | 575.03 | 577.39 | 401.79 | 1.431 | 1.437 |
| 8 | 64 | 852.16 | 810.27 | 589.13 | 1.446 | 1.375 |
| 16 | 128 | 1031.32 | 952.48 | 706.83 | 1.459 | 1.348 |
| 32 | 256 | 1115.22 | 1006.87 | 746.20 | 1.495 | 1.349 |
| GLM5.2 | DSV4.1-Flash | |||||
|---|---|---|---|---|---|---|
| Cake | GC | ratio | Cake | GC | ratio | |
| 1 | 339.10 | 199.51 | 1.700 | 352.95 | 201.33 | 1.753 |
| 2 | 481.78 | 319.19 | 1.509 | 450.63 | 286.77 | 1.571 |
| 4 | 608.91 | 432.63 | 1.407 | 598.97 | 413.08 | 1.450 |
| 8 | 741.81 | 536.62 | 1.382 | 828.64 | 601.50 | 1.378 |
| 16 | 783.59 | 599.50 | 1.307 | 969.48 | 721.93 | 1.343 |
| HBM GB/s / GPU | payload GB/s / GPU | TFLOPS / GPU | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | DSL | Cake | GC | DSL | Cake | GC | DSL | Cake | GC |
| GLM5.2 | 550.9 | 460.8 | 776.8 | 0.49 | 0.41 | 0.70 | 2.25 | 1.88 | 3.17 |
| GLM5.2 | 792.7 | 809.0 | 1145.1 | 5.16 | 5.27 | 7.46 | 23.9 | 24.4 | 34.5 |
| GLM5.2 | 458.7 | 720.5 | 924.9 | 12.0 | 18.8 | 24.1 | 55.3 | 86.8 | 111.4 |
| GLM5.2 | 392.4 | 582.9 | 592.9 | 20.5 | 30.4 | 30.9 | 94.5 | 140.5 | 142.9 |
| DSV4.1 | 412.0 | 334.5 | 590.9 | 0.28 | 0.23 | 0.41 | 1.55 | 1.26 | 2.22 |
| Variant | quantity | GLM5.2 | DSV4.1-Flash | threshold |
|---|---|---|---|---|
| routed | min cosine vs reference | 0.97610 | 0.97699 | 0.97 |
| min cosine vs Cake | 0.99999976 | 0.99999976 | 0.97 | |
| max GC Cake | 0.000000 | 0.000000 | – | |
| cases / rank checks | 47/47, 376/376 | 47/47, 376/376 | – | |
| fused SE | min cosine vs reference | 0.99999952 | 0.99999958 | 0.99 |
| min GC shared residual | 0.99999785 | 0.99999791 | 0.99 |
GPU Kernel Optimization
GPU kernel optimization improves the speed and memory efficiency of computations by tailoring code to GPU hardware and workloads.
See all 238 papers