Attention-FFN disaggregation (AFD) separates attention from the experts of a mixture-of-experts (MoE) model and has been reported to raise decode throughput over colocated serving. Whether this advantage holds on rack-scale NVLink systems such as GB200 NVL72 remains unclear: prior results were obtained on other hardware, against narrower colocated baselines, or with analytical models, whereas production engines serve large MoE models on these systems with wide expert parallelism (wide EP). We build and tune FastAFD, an AFD engine for GB200 NVL72 that overlaps the MoE of one microbatch with the attention of the other, and compare it against the best vLLM wide EP setting. Experiments show that AFD has a throughput advantage of up to 1.17x over wide EP on Qwen3 and 1.16x on Qwen3.5, and matches it on MiniMax-M2.5. We derive an analytical model that predicts AFD throughput within 3% using only profiles of colocated vLLM, telling whether switching to AFD would pay off before deploying it. The model yields a concise insight: AFD's speedup factors into a batch gain (larger batches, since attention GPUs hold no expert weights), an overlap gain (shorter steps, since the MoE runs on the expert GPUs behind attention), and a disaggregation cost (the GPUs given to experts). AFD has the advantage when the two gains outweigh the cost. This view explains which models benefit from AFD, and shows that speculative decoding helps light-expert models most, where the enlarged overlap gain outweighs the larger disaggregation cost and adds a further 8% to AFD's advantage over wide EP. Looking ahead, serving the experts on accelerators that keep their weights in on-chip SRAM, such as the Groq LPX accelerators paired with Vera Rubin NVL72, would remove most of the disaggregation cost, raising the ideal speedup to 2.48x with speculative decoding. FastAFD is open source at https://github.com/hao-ai-lab/FastAFD.
Figures & tables
Figure 1: AFD’s speedup over wide EP balances two gains against one cost (Eq. 1 ). (1) Batch gain : moving expert weights off attention GPUs frees HBM for larger KV caches and decode batches. (2) Overlap gain : two-microbatch ping-pong hides each microbatch’s MoE behind the other’s attention stage. (3) Disaggregation cost : attention nodes hold KV caches and serve requests; the FFN node does not, so one node in every attention:FFN group produces no tokens.
Figure 2: FastAFD architecture and intra-step ping-pong overlap on GB200 NVL72. (a) Attention nodes (blue) hold paged KV caches and run the attention stage (norms, QKV/O projections, TensorRT-LLM FMHA attention, and top- K routing), while one NVLink round trip per layer ( N2M / M2N ) dispatches routed tokens to a dedicated FFN node (coral) running a single persistent fused MegaMoE kernel. (b) Splitting each attention GPU’s batch into two ping-pong microbatches ( mb0 , mb1 ) overlaps the attention stage on the attention pool with the FFN stage ( MegaMoE ) on the FFN pool across consecutive layers.
Figure 3: Context-tuned wide EP narrows homogeneous AFD gains from 1.30–2.36× to 0.99–1.17× across 4K–16K contexts. (a) FastAFD speedups over EP4 ( SEP4 , slate) and over best vLLM wide EP counting all GPUs ( Sall , coral) or attention GPUs only ( Satt , teal; stems show the disaggregation cost). (b) Per-GPU batch ratios ( bA/bEP4 vs. bA/bV ). (c) Colocated vLLM scaling across EP4–EP64 normalized to EP4 (outlined markers: per-curve optima).
Figure 4: Speculative decoding ( d=3,q=4,aˉ≈1.8 ) raises Satt on every workload and keeps Sall at 1.02–1.13× while contracting optimal A:F ratios (4K–16K). (a) Same-engine speculative self-speedup over d=0 (FastAFD up to 1.54× ). (b) FastAFD speedup over tuned wide EP ( d=0 circles vs. d=3 diamonds): hiding 4× MoE work on the FFN pool raises Satt on all six workloads (up to 1.70× ) and lifts MiniMax-M2.5 Sall to 1.08× at 4K and 16K. (c) Optimal A:F ratios contract ( d: 0→3 ) as the FFN pool takes rnq routed tokens.
Best configuration
Speedup ( × )
Model
Ctx
Wide EP
A:F
SEP4
Sall
Satt
MiniMax-M2.5
4K
EP32
9:1
1.40
1.00
1.12
8K
EP32
15:1
1.52
1.02
1.09
16K
EP32
17:1 †
1.56
1.02
1.08 ▼
Qwen3
4K
EP16
5:1
1.30 ▼
1.06
1.28
8K
EP16
7:1
1.41
1.10
1.26
Table 1: Standard decoding ( d=0 ): context-tuned wide EP erases most of FastAFD’s gain over EP4. Each engine runs at its measured best configuration: the vLLM EP width (Wide EP) and the FastAFD attention:FFN node ratio (A:F). SEP4 and Sall compare per-GPU decode throughput over EP4 and the best wide EP, counting every allocated GPU; Satt counts attention GPUs only. ▲ / ▼ : largest/smallest in each column; † : still rising at the 18-node rack limit (a lower bound).
Best configuration
Speedup ( × )
Model
Ctx
Wide EP
A:F
Sall
Satt
MiniMax-M2.5
4K
EP16
9:1 → 4:1
1.08
1.34
8K
EP16
15:1 → 5:1
1.02 ▼
1.22
16K
EP16
17:1 → 8:1
1.08
1.22 ▼
Qwen3
4K
EP8
5:1 → 2:1
1.13 ▲
1.70 ▲
8K
EP16
7:1 → 3:1
1.10
1.46
Table 2: Speculative decoding ( d=3 ) at least halves FastAFD’s best A:F ratio and raises Satt on every workload. Acceptance is held near aˉ≈1.8 tokens per step. A:F shows the best ratio at d=0→d=3 ; Wide EP is the best vLLM width at d=3 . Speedups as in Table 1 .
Figure 5: Three-factor throughput decomposition, step breakdown, and coverage boundary. (a) Equation 1 factors across 4K–16K standard decoding: batch gain bA/bV , overlap gain TV/TA , disaggregation cost r/(r+1) , attention-GPU speedup Satt , and all-GPU speedup Sall . (b) Normalized 8K step share: FastAFD hides colocated MoE while running attention-stage projections twice per step. (c) Ratio of remote FFN latency to the attention stage at 8K, crossing 100% at each model’s measured peak (15:1, 7:1, and 5:1).
Figure 6: Zero-refit model validation across candidate A:F ratios. (a) At 16K standard decoding ( d=0 ), predicted throughput (dashed) against measured throughput (solid) across candidate ratios. (b) At 16K speculative decoding ( d=3 ), the model captures the leftward shift of the optimal ratio. (c) Relative error ∣X/X−1∣ across all 65 deployments up to each workload’s optimal ratio ( 2.9% mean error, dashed line).
Figure 7: Runtime ablations at 8K and MoE-boundary vs. tuned CAD across 4K–16K. (a) Fused MegaMoE vs. DeepEP + DeepGEMM at 7:1. (b) Microbatch count k on MiniMax-M2.5 at 3:1 and 7:1. (c) One-step-ahead coordinator on vs. off at 7:1. (d) FastAFD vs. peak-tuned CAD on MiniMax-M2.5 across 4K–16K.
Figure 8: Rack-scale A:F provisioning and heterogeneous FFN placement across 4K–16K. (a) Optimal homogeneous node ratios r∗=NA/NF in standard decoding ( d=0 ): full-attention MoEs fan out from 9:1→17:1 (MiniMax-M2.5) and 5:1→11:1 (Qwen3), whereas hybrid Qwen3.5 holds 5:1 across 4K–16K. (b, c) Measured speedup over best wide EP in standard ( d=0 ) and speculative ( d=3 ) decoding when counting all GPUs ( Sall , coral) or attention GPUs only ( Satt , solid teal; up to 1.39× and 1.70× ), plus the modeled zero-FFN-cost ceiling (hatched; up to 1.76× and 2.48× ).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Attention
Total / Active
L
Experts
Top- K
MiniMax-M2.5
Full (GQA)
229B / 10B
62
256
8
Qwen3-235B-A22B
Full (GQA)
235B / 22B
94
128
8
Qwen3.5-397B-A17B
15 Full + 45 Linear
397B / 17B
60
512
10
Appendix
Table A1: Evaluated FP8 MoE models and architectural parameters.
Model
Context
AFD
Throughput
Baseline
Throughput
/ EP4
/ best EP
MiniMax-M2.5
4K
AFD 9:1
4118.0
vLLM EP32
4098.0
1.403
1.005
MiniMax-M2.5
8K
AFD 15:1
2372.0
vLLM EP32
2317.0
1.518
1.024
MiniMax-M2.5
16K
AFD 17:1
1234.0
vLLM EP32
1208.0
1.556
1.022
Qwen3
4K
AFD 5:1
4256.0
vLLM EP16
4004.0
1.302
1.063
Qwen3
8K
AFD 7:1
2624.0
vLLM EP16
2389.0
1.412
1.098
Qwen3
16K
AFD 11:1
1452.0
vLLM EP32
1240.0
1.425
1.171
Appendix
Table A2: Best measured standard-decode configuration of each engine per workload (tokens/s/GPU, all GPUs counted). Optima are over the measured A:F ratios and EP widths.
Configuration
4K
8K
16K
AFD 1:1
2341.0
1281.0
657.0
AFD 3:1
3500.0
1920.0
987.0
AFD 5:1
3883.0
2128.0
1038.0
AFD 7:1
4042.0
2230.0
1148.0
AFD 9:1
4118.0
2294.0
1180.0
AFD 11:1
3553.0
2331.0
1201.0
Appendix
Table A3: MiniMax-M2.5 standard-decode throughput (tokens/s/GPU, counting FFN GPUs). Dashes mark configurations without a valid run.
Configuration
4K
8K
16K
AFD 1:1
2721.0
1542.0
810.0
AFD 3:1
4055.0
2289.0
1195.0
AFD 5:1
4256.0
2542.0
1329.0
AFD 7:1
3907.0
2624.0
1396.0
AFD 9:1
—
2530.0
1434.0
AFD 11:1
—
—
1452.0
Appendix
Table A4: Qwen3 standard-decode throughput (tokens/s/GPU, counting FFN GPUs). Dashes mark configurations without a valid run.
Configuration
4K
8K
16K
AFD 1:1
4366.0
3532.0
2543.0
AFD 3:1
6494.0
5220.0
3711.0
AFD 5:1
6968.0
5627.0
3955.0
AFD 7:1
4716.0
5054.0
3854.0
vLLM EP4
3092.0
2382.0
1871.0
vLLM EP8
5186.0
4188.0
3226.0
Appendix
Table A5: Qwen3.5 standard-decode throughput (tokens/s/GPU, counting FFN GPUs). Dashes mark configurations without a valid run.
Standard
Speculative
(4K–16K)
( d=3 , 4K–16K)
Workloads
9
6
Mean error at the peak
3.41%
4.07%
Mean error
3.04%
2.64%
Max error at the peak
7.36%
5.66%
Max error
7.36%
5.66%
Appendix
Table A6: Model accuracy and ratio-choice summary against the 88 measured FastAFD runs at 4K–16K (Section 4.4). Error is ∣X/X−1∣ .
Decoding
Runs
n
Mean
Max
Sel. loss
Standard ( d=0 )
all
53
4.25
43.38
4.72
r≤r∗
44
3.04
7.36
peaks
9
3.41
7.36
d=3
all
42
4.32
11.05
3.49
r≤r∗
26
3.19
8.77
peaks
8
4.32
8.30
Appendix
Table A7: Extended model error summary including 32K and d=7 : mean and maximum ∣e∣ (%) over the listed runs, and the worst selection loss.
Model
Context
d
A:F
bA
T (ms)
T (ms)
e (%)
Scope
MiniMax-M2.5
4K
0
1:1
156
33.320
34.047
-2.13
Y
MiniMax-M2.5
4K
0
3:1
156
33.420
34.047
-1.84
Y
MiniMax-M2.5
4K
0
5:1
156
33.480
34.047
-1.66
Y
MiniMax-M2.5
4K
0
7:1
154
33.340
34.047
-2.08
Y
MiniMax-M2.5
4K
0
9:1
154
33.660
34.047
-1.14
Y
MiniMax-M2.5
4K
0
11:1
152
39.210
35.257
+11.21
N
Appendix
Table A8: Standard-decode model candidates 1–27: measured period T , predicted period T , and signed error e=100(T/T−1) . Scope Y marks r≤r∗ .
Model
Context
d
A:F
bA
T (ms)
T (ms)
e (%)
Scope
Qwen3.5
4K
0
7:1
491
91.092
63.532
+43.38
N
Qwen3.5
8K
0
1:1
383
54.220
52.110
+4.05
Y
Qwen3.5
8K
0
3:1
373
53.590
51.127
+4.82
Y
Qwen3.5
8K
0
5:1
365
54.050
53.966
+0.16
Y
Qwen3.5
8K
0
7:1
357
61.804
58.526
+5.60
N
Qwen3.5
16K
0
1:1
247
48.570
47.466
+2.33
Y
Appendix
Table A9: Standard-decode model candidates 28–53: measured period T , predicted period T , and signed error e=100(T/T−1) . Scope Y marks r≤r∗ .
Model
Context
d
A:F
bA
T (ms)
T (ms)
e (%)
Scope
MiniMax-M2.5
4K
3
1:1
148
36.596
36.438
+0.43
Y
MiniMax-M2.5
4K
3
3:1
150
38.874
38.711
+0.42
Y
MiniMax-M2.5
4K
3
4:1
150
41.367
43.745
-5.44
Y
MiniMax-M2.5
4K
3
5:1
150
44.305
48.779
-9.17
N
MiniMax-M2.5
4K
3
7:1
148
54.269
58.362
-7.01
N
MiniMax-M2.5
4K
7
1:1
150
46.196
42.815
+7.90
Y
Appendix
Table A10: Speculative model candidates 1–38: measured period T , predicted period T , and signed error e=100(T/T−1) . Scope Y marks r≤r∗ .
Model
Context
d
A:F
bA
T (ms)
T (ms)
e (%)
Scope
MiniMax-M2.5
32K
7
8:1
20
37.605
35.835
+4.94
Y
MiniMax-M2.5
32K
7
9:1
20
38.077
37.177
+2.42
N
Qwen3
4K
3
1:1
194
46.352
45.227
+2.49
Y
Qwen3
4K
3
2:1
194
52.397
53.096
-1.32
Y
Qwen3
4K
3
3:1
192
62.060
65.843
-5.75
N
Qwen3
4K
3
5:1
192
98.167
91.895
+6.83
N
Appendix
Table A11: Speculative model candidates 39–76: measured period T , predicted period T , and signed error e=100(T/T−1) . Scope Y marks r≤r∗ .
Table A12: Notation used in the main paper. Source: config = chosen setting; measured = run records; profile = colocated vLLM EP4 profiles; modeled = this appendix’s model.
Model
Context
d
bp
DV (ms)
κ
MiniMax-M2.5
4K
0
94
3.8310
—
MiniMax-M2.5
8K
0
49
3.6400
—
MiniMax-M2.5
16K
0
25
3.5070
—
Qwen3.5
4K
0
83
7.5140
—
Qwen3.5
8K
0
62
7.2230
—
Qwen3.5
16K
0
45
7.0110
—
Appendix
Table A13: Colocated profile inputs of the model. Standard rows use the vLLM EP4 profile; speculative rows use the profile with the same draft length. A dash marks no speculative attention factor.
Model
Context
d
AFD
tok/s/GPU
EP
tok/s/GPU
Sall
Satt
MiniMax-M2.5
4K
0
9:1
4118.1
EP32
4098.0
1.005
1.117
MiniMax-M2.5
4K
3
4:1
5213.2
EP16
4849.4
1.075
1.344
MiniMax-M2.5
4K
7
3:1
3770.2
EP8
3551.7
1.062
1.415
MiniMax-M2.5
8K
0
15:1
2372.1
EP32
2317.0
1.024
1.092
MiniMax-M2.5
8K
3
5:1
3302.6
EP16
3238.4
1.020
1.224
MiniMax-M2.5
8K
7
4:1
2544.2
EP16
2497.4
1.019
1.273
Appendix
Table A14: Best measured configurations for standard ( d=0 ) and speculative ( d=3,7 ) decoding (tokens/s/GPU, all GPUs counted). Acceptance is held near 1.8 useful tokens per step. ∗ MiniMax-M2.5 32K, d=3 : EP64 has no valid run, so the wide-EP peak is not bracketed.
Figure A1: Operator split boundary comparison. (a) FastAFD keeps projections and routing on the attention pool around an off-path N2M / M2N round trip to fused MegaMoE . (b) Core Attention Disaggregation (CAD) moves projections to the FFN pool, placing its round trip on the attention critical path and shrinking the attention stage.
Config
Variant
bA
T (ms)
tok/s/GPU
MiniMax-M2.5 8K 7:1
MegaMoE (full FastAFD)
76
29.813
2230.6
DeepEP+DeepGEMM
76
53.209
1249.8
full FastAFD (second session)
76
29.893
2224.6
one microbatch
76
39.880
1667.5
no cross-step overlap
76
43.820
1517.6
Qwen3 8K 7:1
MegaMoE, reported batch (MR 0.91)
106
35.322
2625.8
Appendix
Table A15: Runtime ablation legs behind Figure 7a and c: measured wall-clock periods, one run per row. MR is the run’s KV memory ratio.
Microbatches
3:1
3:1 repeat
7:1
7:1, bA=72
1
37.770
37.720
39.902
38.214
2
29.607
29.605
29.873
28.825
3
35.061
35.033
35.139
34.774
4
36.699
36.634
38.797
38.649
Reported cell
29.688
29.688
29.827
—
Appendix
Table A16: Microbatch-count sweep behind Figure 7b on MiniMax-M2.5 8K: measured wall-clock periods (ms) at bA=76 ( bA=75 with three microbatches, so every microbatch keeps a captured decode graph). The repeat is an independent 3:1 session on other nodes; the last column uses bA=72 . The bottom row is the reported configuration with two microbatches.
Ctx
Scope
A:F
b
T (ms)
tok/s/GPU
4K
kvfull
1:1
154
31.315
2458.9
4K
kvfull
2:1
154
36.918
2781.0
4K
kvfull
3:1
154
42.285
2731.4
4K
kvfull
4:1
154
45.889
2684.7
8K
kvfull
2:1
82
32.796
1666.9
8K
kvfull
3:1
82
35.839
1716.0
Appendix
Table A17: Complete 15-cell CAD measurements on MiniMax-M2.5 across 4K–16K. Each context brackets its throughput peak on both sides ( kvfull maximizes per-GPU KV batch; aligned matches the selected AFD batch bA ). Throughput counts all GPUs.
Preset
Cells
OSL
FastAFD, MiniMax-M2.5 and Qwen3
afd-cell-v3.4
10
1024
afd-cell-v3.4.1
25
1024
afd-cell-v3.4.2
3
1024
afd-cell-v3.4.3
9
1024
FastAFD, Qwen3.5
Appendix
Table A18: Run presets behind the measured standard-decode cells (62 FastAFD, 60 vLLM). OSL: output tokens per request.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Modern large language model (LLM) inference has progressively disaggregated to keep pace with growing model sizes and tight TTFT and TPOT service-level objectives: from chunked-prefill aggregation, to prefill-decode (P/D) disaggregation, and most recently to operator-level Attention-FFN Disaggregation (AFD). This trend is especially important for mixture-of-experts (MoE) models, where memory-bound attention, compute-intensive expert FFNs, and MoE dispatch/combine communication create distinct resource demands. AFD further exposes this heterogeneity by placing attention and MoE-FFN execution on separate GPU groups. Each level of disaggregation deepens the scheduling design space across workload characteristics, resource allocation, and interconnect topology, raising the central question: when does each level actually pay off? We systematically characterize this trade-off for MoE inference across realistic workloads spanning input/output sequence lengths, prefix-KV reuse, and per-user latency constraints. Using chunked-prefill and P/D disaggregation as baselines, we study the benefits and limits of AFD at scale through a framework that fuses on-device kernel measurements with high-fidelity network simulation. Under strict TTFT/TPOT SLOs, AFD sustains around 4k tokens/s of system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads, where non-AFD deployments are infeasible. We distill concrete takeaways for jointly optimizing throughput and interactivity, including how to partition attention and FFN across GPUs as a function of workload and model architecture, providing design principles for current rack- and cluster-scale deployments as well as future disaggregated AI infrastructure.
Attention--Feed-Forward Network (FFN) Disaggregation (AFD) is emerging as a promising architecture for serving Mixture-of-Experts (MoE) language models. While existing AFD systems improve the efficiency of disaggregated execution, they leave a deployment question unanswered: under the same model, workload, time-per-output-token (TPOT) service-level objective (SLO), hardware budget, hardware catalog, and runtime capabilities, does AFD provide higher throughput than the best collocated deployment? Answering this question requires jointly optimizing hardware assignment and deployment organization for both architectures, making exhaustive provisioning prohibitively expensive. We present AFD-Ledger, an offline analytical provisioning system that independently provisions AFD and collocated deployments using an analytical execution model and an evaluation-bounded hardware search. Across deployment spaces where exhaustive provisioning is feasible, AFD-Ledger reduces complete deployment evaluations by 68.8%--83.5% while still recovering the globally optimal deployment. On three physical LongCat 2.0 deployments, it preserves the correct architecture decision while predicting AFD-to-collocated throughput within 6.6%--9.6% of measurement. Using this validated framework, we show that homogeneous AFD improves fixed-budget throughput in only a minority of the studied settings, heterogeneous AFD requires deployment-level hardware complementarity rather than heuristic device selection, and role-specific hardware improvements matter primarily when they enable better deployment organizations by crossing deployment capability--price boundaries.
Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suffers from severe all-to-all communication bottlenecks, which is exaggerated by the limited inter-node network bandwidth as the growing model size requires distributing experts across GPU nodes. Prior work focused on overlapping these all-to-all communications with feed-forward network (FFN) and self-attention computations, which often leaves residual network-bound stalls due to inherent imbalance in attention and FFN layers' computation-communication ratios. We present DisagMoE, a disaggregated MoE training system that jointly optimizes model placement and scheduling for maximal efficiency. DisagMoE separates attention and FFN layers into disjoint GPU groups, introduces a multi-stage pipeline with uni-directional, many-to-many communications, and employs a computation-communication roofline model to balance GPU and network bandwidth allocation among the attention and FFN groups. DisagMoE is implemented on Megatron-LM, and evaluation shows that DisagMoE improves training efficiency across multiple MoE models with up to 1.8x speedup on 16-node 8xH800 clusters.
Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10
ByteDance Seed · University of Washington · Cornell University