cs.DCOct 28, 2026
SaveThe Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch
Organizations: KlingAI Research, Shanghai, China · KlingAI Research, Beijing, China
Abstract
Fully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alone accounts for 6.4% of step time in Llama-3.1 70B training at 8K tokens per GPU. Communication kernels impose a second cost: a default NCCL reduce-scatter can slow a concurrent GEMM that would otherwise complete in a single wave by 69%. We present TIDE, a set of zero-copy collectives built on public NCCL 2.31+ APIs. TIDE preserves FSDP2's per-parameter storage and compute kernels while folding layout placement into collective addressing. TIDE-AG delivers shards directly to parameter storage using NVSwitch multicast and paced recursive-doubling RDMA, with no SMs used for transport or placement. TIDE-RS consumes per-parameter bf16 gradients directly and reduces them into fp32 shards (a mixed-precision mode that NCCL does not provide) using copy engines and short-lived thread blocks (CTAs) for node-local accumulation and NVSwitch for cross-node reduction. A two-phase pipeline separates gradient reads from network progress; Slot-Reuse optionally stages gradients in vacated all-gather slots. On 64 GPUs, TIDE reduces step time by up to 27.1% relative to Stock FSDP2 and 6.0% relative to FSDP2+SymmMem across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B. It reduces reduce-scatter SM time by factors of 12.5-14.5 and cuts interference with concurrent compute by up to an order of magnitude. Its reduce-scatter buffer remains nearly constant from 16 to 64 GPUs at a fixed per-GPU output size, and its peak memory is up to 4.7 GiB below FSDP2+SymmMem's and, with Slot-Reuse on Wan2.2, below Stock FSDP2's.
Figures & tables
Figure 1. FSDP2’s contract boundary for one parameter group (3 parameters, ranks, shard size ; tile color marks the owning rank, and colored lines trace rank 1’s pieces). Stock FSDP2 converts layouts with SM kernels and temporary buffers; TIDE writes each piece once, at its final address. Two rows of tensor-layout diagrams comparing stock FSDP2 and TIDE for all-gather and reduce-scatter.
| Dimension | Stock FSDP2 | FSDP2+SymmMem ( PyTorch Contributors, 2026 ) | Flat-Buffer FSDP ( NVIDIA Corporation, 2025a ; Wang et al., 2026 ) | PACE ( NVIDIA Corporation, 2026c ) | TIDE (Ours) |
|---|---|---|---|---|---|
| Preserves DTensor storage | Yes | Yes | No (flat/uneven) | Yes | Yes |
| AG direct landing | No | No | Yes | Yes | Yes |
| RS input preparation | Full-group pack | Full-group pack | Fused wgrad / copy | SM staging / cast | In-Place Read |
| AG SM budget | 16–32 | 0 | / | 0–1 | 0 |
| RS SM budget | 16–32 | / | 32 | 8 § | |
| Extra buffer footprint | private |
Table 1. Design space of fully sharded communication subsystems.
Figure 2. TIDE ’s collective paths (two nodes; four of eight GPUs shown). (a) All-gather: local multicast 1 , rail RDMA 2 , destination multicast 3 . (b) Reduce-scatter: copy-engine staging 4 , TMA fp32 sums 5 , rail RDMA 6 , NVSwitch reduction 7 . Two panels, each with two nodes of four GPUs, an NVSwitch per node, and rail NICs, showing numbered all-gather and reduce-scatter steps.
| Symbol | Meaning |
|---|---|
| , , | nodes, GPUs per NVSwitch domain, ranks |
| node-major rank of local GPU on node | |
| , | parameters in an FSDP2 group, parameter index |
| , | padded per-rank rows and row length of |
| , | per-rank shard bytes and prefix offset of |
| , | per-rank packed shard bytes; unsharded group bytes |
Table 2. Notation.
Figure 3. SM wave quantization for the GEMM (schematic; slot counts and CTA displacement measured with an SM-id probe, Table 5 ). Each block is a slot for one four-CTA cluster; the bottom row holds the spare slot and the 16 SMs outside any slot. Three panels, each with 33 two-by-two cluster slots and 16 single SMs, showing GEMM CTAs, resident communication clusters, and transient TIDE CTAs, with GEMM latency bars.
Figure 4. Slowdown of the GEMM versus communication CTAs at a fixed SM clock. Lines: spin kernels resident for the whole GEMM. Markers: 32 MiB reduce-scatters on 8 nodes (64-GPU median), with TIDE ’s short-lived CTAs at their peak count (labels: sum + landing CTAs; 16+4 is the training configuration); runs separate from Table 4 . Log-scale slowdown versus CTA count. Two gray curves rise from 2 to 390 percent; NCCL markers at 8 to 16 CTAs sit near 60 to 71 percent, while NCCL with 4 CTAs or cluster size 1 and both TIDE configurations stay below 7 percent.
| Served traffic (GB/s) | Victim slowdown | |||
|---|---|---|---|---|
| Initiator | Pull | Push | Pull | Push |
| SM | 788 | 721 | ||
| Copy engine | 838 | 798 | ||
Table 3. HBM interference by traffic direction.
Figure 5. TIDE -AG on one rail. (a) Recursive doubling gathers shards in rounds. (b) One strided copy per parameter multicasts a run of stage slots into every local GPU; hatched rows belong to the other local GPU. A matrix of stage buffers over recursive-doubling rounds for five nodes, and a strided copy from stage slots to parameter storage.
Figure 6. Two-level TIDE -RS. (a) Copy engines stage local bf16 slices and bounded TMA CTAs form fp32 node partials. (b) Latin-square placement distributes contributors across destination GPUs so NVSwitch reduces each aligned sub-slice. Level-1 node-local reduction into an fp32 partial, and a four-by-four Latin square of contributor nodes on the destination node's GPUs.
Figure 7. Backward scheduling in the two modes. In-Place lets the next all-gather run during reduce-scatter but briefly keeps two gradient groups live; Slot-Reuse copies gradients into a vacated all-gather slot and gates the slot’s reuse on read completion. Stream timelines of the two modes over three backward layers, showing compute, all-gather, local reads, network phases, and gradient lifetimes.
Figure 8. TIDE is fastest at every point and uses less memory than FSDP2+SymmMem (64 GPUs; Llama and Qwen3: In-Place; Wan: Slot-Reuse). (a–c) Time saved versus Stock; labels are the relative saving, Stock’s step time is under each group, and error bars summarize run-to-run variation. (d) Peak memory versus Stock (repeats within 0.1 GiB). Grouped bars of the per-step time that FSDP2+SymmMem and TIDE save relative to Stock FSDP2 on Llama-3.1 70B at 8K to 32K tokens per GPU, on Wan2.2 at 7.8K to 31.2K tokens per GPU (17 frames), and on Qwen3-30B-A3B at 16K to 32K tokens per GPU, with Stock's step time under each group, and bars of peak-memory change relative to Stock annotated with Stock's absolute peak.
Figure 9. TIDE -AG barely slows concurrent compute; TIDE -RS cuts that slowdown several-fold (training shapes, fixed SM clock; lower is better; labels are TIDE ’s values). (a) Isolated calls. (b) MLP GEMM (Llama) or self-attention (Wan). Grouped bars comparing Stock FSDP2, FSDP2+SymmMem, and TIDE on Llama-3.1 70B and Wan2.2 all-gather and reduce-scatter: median latency, slowdown of concurrent compute, and SM time per call.
Figure 10. TIDE -AG stays faster from 8 to 64 GPUs; its reduce-scatter buffer stays flat ( Tables 7 and 8 ). (a) Median speedup over FSDP2’s fastest all-gather. (b) Buffer per GPU. Two line plots over 8 to 64 GPUs. Left: TIDE all-gather speedup in median latency over the best FSDP2 path ranges from 1.06 to 1.42 for 32 and 128 MiB shards; for 128 MiB it rises from 1.13 at 16 GPUs to 1.42 at 64. Right, log scale: FSDP2's reduce-scatter buffer doubles with the GPU count, reaching 2.03 and 8.12 GiB at 64 GPUs, while TIDE's stays near 0.6 and 2.4 GiB from 16 GPUs on, a 3.2 times gap at 64 GPUs.
Figure 11. Reduce-scatter time versus one-wave GEMM slowdown ( Table 4 ). Hollow markers: pre-packed fp32 input. Scatter plot of reduce-scatter time alone against the slowdown of a concurrent one-wave GEMM. Stock FSDP2 and NCCL configurations with default or eight CTAs sit near 68 to 71 percent slowdown, and NCCL's symmetric kernel at 11 percent; NCCL capped at four CTAs reaches 2.7 percent but takes 26.4 ms; TIDE with 16 or 4 sum CTAs sits at 1.3 to 2.9 percent and 7.0 to 12.0 ms.
| RS alone | Victim slowdown (%) | ||||
|---|---|---|---|---|---|
| Configuration | CTAs | (ms) | 1-wave | 28-wave | Elementwise |
| TIDE , training | 6.96 | 2.9 | 4.2 | 1.6 | |
| TIDE , 4+4 | 12.02 | 1.3 | 4.0 | 0.6 | |
| Stock FSDP2 | 9.01 | 71.0 | 13.6 | 13.3 | |
| FSDP2+SymmMem | 7.45 | 9.7 | 10.6 | 13.6 | |
| NCCL default | 6.75 | 69.0 | 13.5 | 7.4 | |
Table 4. Concurrent 32 MiB reduce-scatters on 8 nodes at a fixed SM clock (medians of three launches). Bold: lowest slowdown among rows no slower than Stock FSDP2.
| Configuration | Displaced | Windows | Resident |
|---|---|---|---|
| CTAs (mode) | hit (%) | CTAs (mean) | |
| TIDE , training | 0 | 0 | 2.0 |
| TIDE , 4+4 | 0 | 0 | 1.5 |
| Stock FSDP2 | 12 | 65 | 13.4 |
| FSDP2+SymmMem | 0 | 0 | 12.0 |
| NCCL default | 12 | 64 | 14.7 |
Table 5. SM-id probe during a 32 MiB reduce-scatter on 8 nodes at a fixed SM clock (two launches). The probe launches the one-wave GEMM’s 32 four-CTA clusters, each CTA filling an SM and recording its SM id and entry and exit timestamps; displaced counts probe CTAs that start after the first wave. Resident CTAs come from a second probe with one CTA on each of the 148 SMs.
| Latency | Victim | SM-ms | ||
| Op | System | p50 (ms) | slowdown | per call |
| Llama-3.1 70B | ||||
| AG | Stock FSDP2 | 8.07 | 6.60% | 218.2 |
| AG | FSDP2+SymmMem | 7.23 | 3.05% | 86.3 |
| AG | TIDE | 4.90 | 0.33% | 0.0 |
| RS | Stock FSDP2 | 14.12 | 11.61% | 651.4 |
Table 6. Training-shape replay at a fixed SM clock (medians of three launches of 30 isolated calls).
| Nodes (GPUs) | Shard | TIDE -AG (0 SM) | NCCL Ring | NCCL CE | Ring + Copy-out | CE + Copy-out | vs. Best FSDP2 |
|---|---|---|---|---|---|---|---|
| 1 (8 GPUs) | 32 MiB | 0.38 (0.38) | 0.40 (0.41) | 0.35 (0.36) | 0.50 (0.51) | 0.45 (0.46) | 1.22 |
| 2 (16 GPUs) | 32 MiB | 1.11 (1.12) | 1.35 (1.59) | 1.08 (1.10) | 1.53 (1.58) | 1.26 (1.28) | 1.14 |
| 4 (32 GPUs) | 32 MiB | 2.57 (2.60) | 2.71 (2.79) | 3.14 (3.22) | 3.11 (3.14) | 3.56 (3.62) | 1.21 |
| 8 (64 GPUs) | 32 MiB | 5.52 (5.86) | 5.46 (5.78) | 7.25 (7.42) | 6.15 (6.21) | 7.96 (8.19) | 1.06 |
| 1 (8 GPUs) | 128 MiB | 1.40 (1.41) | 1.50 (1.52) | 1.27 (1.28) | 1.85 (1.87) | 1.61 (1.63) | 1.16 |
| 2 (16 GPUs) | 128 MiB | 4.19 (4.20) | 5.29 (6.09) | 4.04 (4.13) | 5.99 (6.84) | 4.73 (4.76) | 1.13 |
Table 7. All-gather latency, min (median) ms, for tensors. “+ Copy-out” adds FSDP2’s fused split_with_sizes_copy ; TIDE lands directly. Bold: the fastest path that yields FSDP2’s parameter layout; the last column compares medians.
| Nodes (GPUs) | Output | TIDE -RS | Stock FSDP2 | FSDP2+SymmMem | Bare NCCL | Buffer ( TIDE / FSDP2) |
|---|---|---|---|---|---|---|
| 1 (8 GPUs) | 32 MiB | 0.41 (0.42) | 0.59 (0.60) | 0.60 (0.61) | 0.39 (0.40) | 0.29 / 0.28 |
| 2 (16 GPUs) | 32 MiB | 2.08 (2.12) | 1.67 (1.75) | 1.16 (1.27) | 1.30 (1.45) | 0.58 / 0.53 |
| 4 (32 GPUs) | 32 MiB | 3.70 (3.75) | 3.39 (4.06) | 3.06 (3.07) | 2.69 (2.78) | 0.61 / 1.03 |
| 8 (64 GPUs) | 32 MiB | 6.61 (8.12) | 6.82 (7.96) | 6.67 ( 6.84 ) | 5.40 (5.52) | 0.64 / 2.03 |
| 1 (8 GPUs) | 128 MiB | 1.29 (1.33) | 2.23 (2.25) | 2.36 (2.37) | 1.44 (1.46) | 0.38 / 1.12 |
| 2 (16 GPUs) | 128 MiB | 7.63 (7.68) | 6.70 (6.78) | 4.62 (4.96) | 5.17 (5.27) | 2.26 / 2.12 |
Table 8. Reduce-scatter latency, min (median) ms, and buffer (GiB/GPU) for bf16 gradients. FSDP2 columns include _chunk_cat ; Bare NCCL starts from packed fp32 . Bold: lowest among per-parameter paths, and the smaller buffer.
| Metric | Static or CLC-normal | CLC-high |
|---|---|---|
| Communication admission | Final GEMM wave | tile boundaries |
| Added RS latency | ms | ms |
| GEMM penalty | (RS waits) | ms |
Table 9. Scheduler probes during a persistent GEMM. CLC-normal/high: normal/high-priority communication.
Figure 12. Qwen3-30B-A3B trained from scratch on public C4 (64 GPUs, 1,000 steps). (a) 50-step moving-average loss. (b) Difference from the first Stock run; Section 7.5 reports maxima over the shaded steps 500–1,000. Two panels over 1,000 training steps. Top: moving-average loss of two Stock FSDP2 runs and TIDE In-Place, which overlap as the loss falls from about 6 to 3.5. Bottom: percent difference from the first Stock run; after step 500, TIDE stays within 0.73 percent and the second Stock run within 1.12 percent.
Figure 13. Host-time allocation: the host allocates later layers’ gradients before the GPU finishes reading layer , so a GPU-safe release is too late for them to reuse its storage. A host timeline that enqueues three backward layers and allocates gradients early, above a GPU timeline whose read of layer ell finishes later; the GPU-safe release point lies after the host allocations.
More in this topic
Communication-Efficient Distributed Training
Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.
See all 230 papers