Fuse-All MoE Inference Kernel with Load Balancing on TPUs
Organizations: Meta SuperIntelligence Lab
Abstract
Mixture-of-Experts (MoE) architectures activate only a subset of experts for each token, allowing large language models (LLMs) to scale parameter capacity without a proportional increase in computation per token. However, intermediate data movement and uneven expert loads limit the efficiency of MoE inference on TPUs. To address these bottlenecks, we introduce Fuse-All MoE, a TPU kernel design that jointly optimizes expert computation, output communication, and load balancing. The kernel fuses token collection, expert projections, gated activation, and output writes, retaining expert computation intermediates on chip. Fixed token-slot destinations allow scheduling and expert placement to change while preserving routing semantics. In prefill-decode (PD) disaggregated serving, Fuse-All MoE balances EP16 prefill workloads through scheduling and expert redistribution, while symmetric EP8TP2 shares active expert computation between the two TensorCore devices within each chip during decode. Our experiments show that Fuse-All MoE reduces kernel runtime for a measured skewed EP16 prefill workload from 2.3x to 1.09x the same fixed reference time for a balanced workload. Across 32-512 decode tokens, symmetric EP8TP2 reduces kernel latency by up to 21.3% relative to fused EP16. With accompanying serving stack optimizations, the integrated system improves serving throughput by up to 34.5% over the Megablox EP16 baseline.
Figures & tables
| Resource | Specification | Consequence |
|---|---|---|
| TensorCore devices / SparseCores | 2 / 4 per chip | EP device count and physical chip count differ. |
| VMEM | 64 MiB per TensorCore | Weights, intermediates, and DMA staging compete for capacity. |
| HBM | 192 GiB capacity; approximately 7.38 TB/s | Use chip-level bytes and chip-level peak throughput consistently. |
| Dense compute | 2,307 TFLOPs/s BF16; 4,614 TFLOPs/s FP8 | A roofline must match the arithmetic mode actually used. |
| ICI | 1,200 GB/s bidirectional per chip | This is aggregate bandwidth, not a per-destination guarantee. |
| Intra-chip link | Documented as six times a 1-D ICI link | Chipmate weight transfers avoid inter-chip torus traffic. |
| Symbol | Meaning |
|---|---|
| , | Token count of the compiled tensor shape, including padding, and the valid-token count used to plan work. |
| , | Experts selected per token and total (token, expert-choice) assignments, M = Tk for complete routing. |
| , | Total experts and EP devices/ranks. Initial expert assignment has experts per rank. |
| , | Hidden width and expert intermediate width. |
| , | Owned tokens per rank, = , and routing slot in [0, ). |
| , | Rows assigned to expert e and the maximum rows in one compute tile for that expert. |
| Boundary or storage | Fused path |
|---|---|
| Grouped activation materialization | Removed: token collection goes directly into VMEM. |
| gate/up matmul / activation / down-projection matmul intermediates in HBM | Removed within the fused expert-compute path. |
| Auxiliary receive-and-scatter buffer | Removed: writes target final token-slot addresses. |
| Final expert-slot output | Retained; the main HBM-output path materializes it before postprocessing. |
| Routing preparation and upstream activation exchange | Outside this fused launch; include their cost in full-layer measurements. |
| Weighted sum and per-expert output normalization | Outside this launch, implemented by a Pallas postprocessing kernel or JAX path. |
| Buffer | Multiplicity | Lifetime or purpose |
|---|---|---|
| Collected input rows | 2 tiles | Current compute and next-tile token collection use different slots. |
| W1 and its block scales | Circular weight buffers | Current-expert consumption, cross-tile prefetch, and conditional residency. |
| W2 and its block scales | Tuned ring | Issued early enough to overlap token collection/gate/up matmul; repeated-expert hits skip loads. |
| Activated intermediate | 1 tile | Written by gate/up matmul's activation and consumed by down-projection matmul. |
| Shared accumulator / down-projection matmul tile | Shared scratch / 1 output tile | Computation and conversion before temporary send buffers. |
| Temporary send buffers | 3 tiles | Rows remain stable until local/remote source DMA completion. |
| Parameter | Value |
|---|---|
| Hidden / intermediate width | 4096 / 4096 |
| Experts / top-k | 256 / 8 |
| Hardware slice | TPUv7 (Ironwood), 2 2 2 physical chips |
| EP16 devices / experts per device | 16 / 16 |
| W1 / W2 shape per expert | 4096 8192 / 4096 4096 |
| Packed W1 / W2 bytes per expert | 16 MiB / 8 MiB, excluding scales |
| Global tokens | Useful projection GFLOPs | Weight-only FLOPs/byte |
| 32 | 25.8 | 4 |
| 128 | 103.1 | 16 |
| 512 | 412.3 | 64 |
| 2048 | 1649.3 | 256 |
| 8192 | 6597.1 | 1024 |
Mixture-of-Experts Inference
Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.
See all 200 papers