FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?
Organizations: University of California, San Diego
Abstract
Attention-FFN disaggregation (AFD) separates attention from the experts of a mixture-of-experts (MoE) model and has been reported to raise decode throughput over colocated serving. Whether this advantage holds on rack-scale NVLink systems such as GB200 NVL72 remains unclear: prior results were obtained on other hardware, against narrower colocated baselines, or with analytical models, whereas production engines serve large MoE models on these systems with wide expert parallelism (wide EP). We build and tune FastAFD, an AFD engine for GB200 NVL72 that overlaps the MoE of one microbatch with the attention of the other, and compare it against the best vLLM wide EP setting. Experiments show that AFD has a throughput advantage of up to 1.17x over wide EP on Qwen3 and 1.16x on Qwen3.5, and matches it on MiniMax-M2.5. We derive an analytical model that predicts AFD throughput within 3% using only profiles of colocated vLLM, telling whether switching to AFD would pay off before deploying it. The model yields a concise insight: AFD's speedup factors into a batch gain (larger batches, since attention GPUs hold no expert weights), an overlap gain (shorter steps, since the MoE runs on the expert GPUs behind attention), and a disaggregation cost (the GPUs given to experts). AFD has the advantage when the two gains outweigh the cost. This view explains which models benefit from AFD, and shows that speculative decoding helps light-expert models most, where the enlarged overlap gain outweighs the larger disaggregation cost and adds a further 8% to AFD's advantage over wide EP. Looking ahead, serving the experts on accelerators that keep their weights in on-chip SRAM, such as the Groq LPX accelerators paired with Vera Rubin NVL72, would remove most of the disaggregation cost, raising the ideal speedup to 2.48x with speculative decoding. FastAFD is open source at https://github.com/hao-ai-lab/FastAFD.
Figures & tables
| Best configuration | Speedup ( ) | |||||
|---|---|---|---|---|---|---|
| Model | Ctx | Wide EP | A:F | |||
| MiniMax-M2.5 | 4K | EP32 | 9:1 | 1.40 | 1.00 | 1.12 |
| 8K | EP32 | 15:1 | 1.52 | 1.02 | 1.09 | |
| 16K | EP32 | 17:1 † | 1.56 | 1.02 | 1.08 ▼ | |
| Qwen3 | 4K | EP16 | 5:1 | 1.30 ▼ | 1.06 | 1.28 |
| 8K | EP16 | 7:1 | 1.41 | 1.10 | 1.26 | |
| Best configuration | Speedup ( ) | ||||
|---|---|---|---|---|---|
| Model | Ctx | Wide EP | A:F | ||
| MiniMax-M2.5 | 4K | EP16 | 9:1 4:1 | 1.08 | 1.34 |
| 8K | EP16 | 15:1 5:1 | 1.02 ▼ | 1.22 | |
| 16K | EP16 | 17:1 8:1 | 1.08 | 1.22 ▼ | |
| Qwen3 | 4K | EP8 | 5:1 2:1 | 1.13 ▲ | 1.70 ▲ |
| 8K | EP16 | 7:1 3:1 | 1.10 | 1.46 | |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Attention | Total / Active | Experts | Top- | |
|---|---|---|---|---|---|
| MiniMax-M2.5 | Full (GQA) | 229B / 10B | 62 | 256 | 8 |
| Qwen3-235B-A22B | Full (GQA) | 235B / 22B | 94 | 128 | 8 |
| Qwen3.5-397B-A17B | 15 Full + 45 Linear | 397B / 17B | 60 | 512 | 10 |
| Model | Context | AFD | Throughput | Baseline | Throughput | / EP4 | / best EP |
|---|---|---|---|---|---|---|---|
| MiniMax-M2.5 | 4K | AFD 9:1 | 4118.0 | vLLM EP32 | 4098.0 | 1.403 | 1.005 |
| MiniMax-M2.5 | 8K | AFD 15:1 | 2372.0 | vLLM EP32 | 2317.0 | 1.518 | 1.024 |
| MiniMax-M2.5 | 16K | AFD 17:1 | 1234.0 | vLLM EP32 | 1208.0 | 1.556 | 1.022 |
| Qwen3 | 4K | AFD 5:1 | 4256.0 | vLLM EP16 | 4004.0 | 1.302 | 1.063 |
| Qwen3 | 8K | AFD 7:1 | 2624.0 | vLLM EP16 | 2389.0 | 1.412 | 1.098 |
| Qwen3 | 16K | AFD 11:1 | 1452.0 | vLLM EP32 | 1240.0 | 1.425 | 1.171 |
| Configuration | 4K | 8K | 16K |
|---|---|---|---|
| AFD 1:1 | 2341.0 | 1281.0 | 657.0 |
| AFD 3:1 | 3500.0 | 1920.0 | 987.0 |
| AFD 5:1 | 3883.0 | 2128.0 | 1038.0 |
| AFD 7:1 | 4042.0 | 2230.0 | 1148.0 |
| AFD 9:1 | 4118.0 | 2294.0 | 1180.0 |
| AFD 11:1 | 3553.0 | 2331.0 | 1201.0 |
| Configuration | 4K | 8K | 16K |
|---|---|---|---|
| AFD 1:1 | 2721.0 | 1542.0 | 810.0 |
| AFD 3:1 | 4055.0 | 2289.0 | 1195.0 |
| AFD 5:1 | 4256.0 | 2542.0 | 1329.0 |
| AFD 7:1 | 3907.0 | 2624.0 | 1396.0 |
| AFD 9:1 | — | 2530.0 | 1434.0 |
| AFD 11:1 | — | — | 1452.0 |
| Configuration | 4K | 8K | 16K |
|---|---|---|---|
| AFD 1:1 | 4366.0 | 3532.0 | 2543.0 |
| AFD 3:1 | 6494.0 | 5220.0 | 3711.0 |
| AFD 5:1 | 6968.0 | 5627.0 | 3955.0 |
| AFD 7:1 | 4716.0 | 5054.0 | 3854.0 |
| vLLM EP4 | 3092.0 | 2382.0 | 1871.0 |
| vLLM EP8 | 5186.0 | 4188.0 | 3226.0 |
| Standard | Speculative | |
| (4K–16K) | ( , 4K–16K) | |
| Workloads | 9 | 6 |
| Mean error at the peak | 3.41% | 4.07% |
| Mean error | 3.04% | 2.64% |
| Max error at the peak | 7.36% | 5.66% |
| Max error | 7.36% | 5.66% |
| Decoding | Runs | Mean | Max | Sel. loss | |
|---|---|---|---|---|---|
| Standard ( ) | all | 53 | 4.25 | 43.38 | 4.72 |
| 44 | 3.04 | 7.36 | |||
| peaks | 9 | 3.41 | 7.36 | ||
| all | 42 | 4.32 | 11.05 | 3.49 | |
| 26 | 3.19 | 8.77 | |||
| peaks | 8 | 4.32 | 8.30 |
| Model | Context | A:F | (ms) | (ms) | (%) | Scope | ||
|---|---|---|---|---|---|---|---|---|
| MiniMax-M2.5 | 4K | 0 | 1:1 | 156 | 33.320 | 34.047 | -2.13 | Y |
| MiniMax-M2.5 | 4K | 0 | 3:1 | 156 | 33.420 | 34.047 | -1.84 | Y |
| MiniMax-M2.5 | 4K | 0 | 5:1 | 156 | 33.480 | 34.047 | -1.66 | Y |
| MiniMax-M2.5 | 4K | 0 | 7:1 | 154 | 33.340 | 34.047 | -2.08 | Y |
| MiniMax-M2.5 | 4K | 0 | 9:1 | 154 | 33.660 | 34.047 | -1.14 | Y |
| MiniMax-M2.5 | 4K | 0 | 11:1 | 152 | 39.210 | 35.257 | +11.21 | N |
| Model | Context | A:F | (ms) | (ms) | (%) | Scope | ||
|---|---|---|---|---|---|---|---|---|
| Qwen3.5 | 4K | 0 | 7:1 | 491 | 91.092 | 63.532 | +43.38 | N |
| Qwen3.5 | 8K | 0 | 1:1 | 383 | 54.220 | 52.110 | +4.05 | Y |
| Qwen3.5 | 8K | 0 | 3:1 | 373 | 53.590 | 51.127 | +4.82 | Y |
| Qwen3.5 | 8K | 0 | 5:1 | 365 | 54.050 | 53.966 | +0.16 | Y |
| Qwen3.5 | 8K | 0 | 7:1 | 357 | 61.804 | 58.526 | +5.60 | N |
| Qwen3.5 | 16K | 0 | 1:1 | 247 | 48.570 | 47.466 | +2.33 | Y |
| Model | Context | A:F | (ms) | (ms) | (%) | Scope | ||
|---|---|---|---|---|---|---|---|---|
| MiniMax-M2.5 | 4K | 3 | 1:1 | 148 | 36.596 | 36.438 | +0.43 | Y |
| MiniMax-M2.5 | 4K | 3 | 3:1 | 150 | 38.874 | 38.711 | +0.42 | Y |
| MiniMax-M2.5 | 4K | 3 | 4:1 | 150 | 41.367 | 43.745 | -5.44 | Y |
| MiniMax-M2.5 | 4K | 3 | 5:1 | 150 | 44.305 | 48.779 | -9.17 | N |
| MiniMax-M2.5 | 4K | 3 | 7:1 | 148 | 54.269 | 58.362 | -7.01 | N |
| MiniMax-M2.5 | 4K | 7 | 1:1 | 150 | 46.196 | 42.815 | +7.90 | Y |
| Model | Context | A:F | (ms) | (ms) | (%) | Scope | ||
|---|---|---|---|---|---|---|---|---|
| MiniMax-M2.5 | 32K | 7 | 8:1 | 20 | 37.605 | 35.835 | +4.94 | Y |
| MiniMax-M2.5 | 32K | 7 | 9:1 | 20 | 38.077 | 37.177 | +2.42 | N |
| Qwen3 | 4K | 3 | 1:1 | 194 | 46.352 | 45.227 | +2.49 | Y |
| Qwen3 | 4K | 3 | 2:1 | 194 | 52.397 | 53.096 | -1.32 | Y |
| Qwen3 | 4K | 3 | 3:1 | 192 | 62.060 | 65.843 | -5.75 | N |
| Qwen3 | 4K | 3 | 5:1 | 192 | 98.167 | 91.895 | +6.83 | N |
| Symbol | Meaning | Source |
| Configuration and measured quantities | ||
| attention / FFN nodes (four GPUs each) | config | |
| A:F node ratio | config | |
| per-GPU decode batch: FastAFD attention GPU / colocated vLLM | measured | |
| requests per microbatch, | config | |
| decode step period: FastAFD / vLLM | measured | |
| Model | Context | (ms) | |||
|---|---|---|---|---|---|
| MiniMax-M2.5 | 4K | 0 | 94 | 3.8310 | — |
| MiniMax-M2.5 | 8K | 0 | 49 | 3.6400 | — |
| MiniMax-M2.5 | 16K | 0 | 25 | 3.5070 | — |
| Qwen3.5 | 4K | 0 | 83 | 7.5140 | — |
| Qwen3.5 | 8K | 0 | 62 | 7.2230 | — |
| Qwen3.5 | 16K | 0 | 45 | 7.0110 | — |
| Model | Context | AFD | tok/s/GPU | EP | tok/s/GPU | |||
|---|---|---|---|---|---|---|---|---|
| MiniMax-M2.5 | 4K | 0 | 9:1 | 4118.1 | EP32 | 4098.0 | 1.005 | 1.117 |
| MiniMax-M2.5 | 4K | 3 | 4:1 | 5213.2 | EP16 | 4849.4 | 1.075 | 1.344 |
| MiniMax-M2.5 | 4K | 7 | 3:1 | 3770.2 | EP8 | 3551.7 | 1.062 | 1.415 |
| MiniMax-M2.5 | 8K | 0 | 15:1 | 2372.1 | EP32 | 2317.0 | 1.024 | 1.092 |
| MiniMax-M2.5 | 8K | 3 | 5:1 | 3302.6 | EP16 | 3238.4 | 1.020 | 1.224 |
| MiniMax-M2.5 | 8K | 7 | 4:1 | 2544.2 | EP16 | 2497.4 | 1.019 | 1.273 |
| Config | Variant | (ms) | tok/s/GPU | |
|---|---|---|---|---|
| MiniMax-M2.5 8K 7:1 | MegaMoE (full FastAFD) | 76 | 29.813 | 2230.6 |
| DeepEP+DeepGEMM | 76 | 53.209 | 1249.8 | |
| full FastAFD (second session) | 76 | 29.893 | 2224.6 | |
| one microbatch | 76 | 39.880 | 1667.5 | |
| no cross-step overlap | 76 | 43.820 | 1517.6 | |
| Qwen3 8K 7:1 | MegaMoE, reported batch (MR 0.91) | 106 | 35.322 | 2625.8 |
| Microbatches | 3:1 | 3:1 repeat | 7:1 | 7:1, |
|---|---|---|---|---|
| 1 | 37.770 | 37.720 | 39.902 | 38.214 |
| 2 | 29.607 | 29.605 | 29.873 | 28.825 |
| 3 | 35.061 | 35.033 | 35.139 | 34.774 |
| 4 | 36.699 | 36.634 | 38.797 | 38.649 |
| Reported cell | 29.688 | 29.688 | 29.827 | — |
| Ctx | Scope | A:F | (ms) | tok/s/GPU | |
| 4K | kvfull | 1:1 | 154 | 31.315 | 2458.9 |
| 4K | kvfull | 2:1 | 154 | 36.918 | 2781.0 |
| 4K | kvfull | 3:1 | 154 | 42.285 | 2731.4 |
| 4K | kvfull | 4:1 | 154 | 45.889 | 2684.7 |
| 8K | kvfull | 2:1 | 82 | 32.796 | 1666.9 |
| 8K | kvfull | 3:1 | 82 | 35.839 | 1716.0 |
| Preset | Cells | OSL |
| FastAFD, MiniMax-M2.5 and Qwen3 | ||
| afd-cell-v3.4 | 10 | 1024 |
| afd-cell-v3.4.1 | 25 | 1024 |
| afd-cell-v3.4.2 | 3 | 1024 |
| afd-cell-v3.4.3 | 9 | 1024 |
| FastAFD, Qwen3.5 | ||
LLM Inference Acceleration
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
See all 1,228 papers