Fully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alone accounts for 6.4% of step time in Llama-3.1 70B training at 8K tokens per GPU. Communication kernels impose a second cost: a default NCCL reduce-scatter can slow a concurrent GEMM that would otherwise complete in a single wave by 69%. We present TIDE, a set of zero-copy collectives built on public NCCL 2.31+ APIs. TIDE preserves FSDP2's per-parameter storage and compute kernels while folding layout placement into collective addressing. TIDE-AG delivers shards directly to parameter storage using NVSwitch multicast and paced recursive-doubling RDMA, with no SMs used for transport or placement. TIDE-RS consumes per-parameter bf16 gradients directly and reduces them into fp32 shards (a mixed-precision mode that NCCL does not provide) using copy engines and short-lived thread blocks (CTAs) for node-local accumulation and NVSwitch for cross-node reduction. A two-phase pipeline separates gradient reads from network progress; Slot-Reuse optionally stages gradients in vacated all-gather slots. On 64 GPUs, TIDE reduces step time by up to 27.1% relative to Stock FSDP2 and 6.0% relative to FSDP2+SymmMem across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B. It reduces reduce-scatter SM time by factors of 12.5-14.5 and cuts interference with concurrent compute by up to an order of magnitude. Its reduce-scatter buffer remains nearly constant from 16 to 64 GPUs at a fixed per-GPU output size, and its peak memory is up to 4.7 GiB below FSDP2+SymmMem's and, with Slot-Reuse on Wan2.2, below Stock FSDP2's.
Figures & tables
Figure 1. FSDP2’s contract boundary for one parameter group (3 parameters, N=4 ranks, shard size S ; tile color marks the owning rank, and colored lines trace rank 1’s pieces). Stock FSDP2 converts layouts with SM kernels and temporary buffers; TIDE writes each piece once, at its final address. Two rows of tensor-layout diagrams comparing stock FSDP2 and TIDE for all-gather and reduce-scatter.
Dimension
Stock FSDP2
FSDP2+SymmMem ( PyTorch Contributors, 2026 )
Flat-Buffer FSDP ( NVIDIA Corporation, 2025a ; Wang et al., 2026 )
PACE ( NVIDIA Corporation, 2026c )
TIDE (Ours)
Preserves DTensor storage
Yes
Yes
No (flat/uneven)
Yes
Yes
AG direct landing
No
No
Yes
Yes
Yes ‡
RS input preparation
Full-group pack
Full-group pack
Fused wgrad / copy
SM staging / cast
In-Place Read †
AG SM budget
16–32
0
/
0–1
0
RS SM budget
16–32
≈13
/
32
8 §
Extra buffer footprint
3NS
3NS private
Θ(NS)
2NS+rings
≈1.1NS
Table 1. Design space of fully sharded communication subsystems.
Figure 2. TIDE ’s collective paths (two nodes; four of eight GPUs shown). (a) All-gather: local multicast 1 , rail RDMA 2 , destination multicast 3 . (b) Reduce-scatter: copy-engine staging 4 , TMA fp32 sums 5 , rail RDMA 6 , NVSwitch reduction 7 . Two panels, each with two nodes of four GPUs, an NVSwitch per node, and rail NICs, showing numbered all-gather and reduce-scatter steps.
Symbol
Meaning
M , L , N=ML
nodes, GPUs per NVSwitch domain, ranks
r=nL+l
node-major rank of local GPU l on node n
P , p
parameters in an FSDP2 group, parameter index
Rp , Kp
padded per-rank rows and row length of p
sp , op
per-rank shard bytes and prefix offset of p
S , NS
per-rank packed shard bytes; unsharded group bytes
Table 2. Notation.
Figure 3. SM wave quantization for the 256×8192×8192 GEMM (schematic; slot counts and CTA displacement measured with an SM-id probe, Table 5 ). Each 2×2 block is a slot for one four-CTA cluster; the bottom row holds the spare slot and the 16 SMs outside any slot. Three panels, each with 33 two-by-two cluster slots and 16 single SMs, showing GEMM CTAs, resident communication clusters, and transient TIDE CTAs, with GEMM latency bars.
Figure 4. Slowdown of the 256×8192×8192 GEMM versus communication CTAs at a fixed SM clock. Lines: spin kernels resident for the whole GEMM. Markers: 32 MiB reduce-scatters on 8 nodes (64-GPU median), with TIDE ’s short-lived CTAs at their peak count (labels: sum + landing CTAs; 16+4 is the training configuration); runs separate from Table 4 . Log-scale slowdown versus CTA count. Two gray curves rise from 2 to 390 percent; NCCL markers at 8 to 16 CTAs sit near 60 to 71 percent, while NCCL with 4 CTAs or cluster size 1 and both TIDE configurations stay below 7 percent.
Served traffic (GB/s)
Victim slowdown
Initiator
Pull
Push
Pull
Push
SM
788
721
46.7%
12.1%
Copy engine
838
798
43.9%
11.0%
Table 3. HBM interference by traffic direction.
Figure 5. TIDE -AG on one rail. (a) Recursive doubling gathers M=5 shards in ⌈log2M⌉=3 rounds. (b) One strided copy per parameter multicasts a run of stage slots into every local GPU; hatched rows belong to the other local GPU. A matrix of stage buffers over recursive-doubling rounds for five nodes, and a strided copy from stage slots to parameter storage.
Figure 6. Two-level TIDE -RS. (a) Copy engines stage local bf16 slices and bounded TMA CTAs form fp32 node partials. (b) Latin-square placement distributes contributors across destination GPUs so NVSwitch reduces each aligned sub-slice. Level-1 node-local reduction into an fp32 partial, and a four-by-four Latin square of contributor nodes on the destination node's GPUs.
Figure 7. Backward scheduling in the two modes. In-Place lets the next all-gather run during reduce-scatter but briefly keeps two gradient groups live; Slot-Reuse copies gradients into a vacated all-gather slot and gates the slot’s reuse on read completion. Stream timelines of the two modes over three backward layers, showing compute, all-gather, local reads, network phases, and gradient lifetimes.
Figure 8. TIDE is fastest at every point and uses less memory than FSDP2+SymmMem (64 GPUs; Llama and Qwen3: In-Place; Wan: Slot-Reuse). (a–c) Time saved versus Stock; labels are the relative saving, Stock’s step time is under each group, and error bars summarize run-to-run variation. (d) Peak memory versus Stock (repeats within 0.1 GiB). Grouped bars of the per-step time that FSDP2+SymmMem and TIDE save relative to Stock FSDP2 on Llama-3.1 70B at 8K to 32K tokens per GPU, on Wan2.2 at 7.8K to 31.2K tokens per GPU (17 frames), and on Qwen3-30B-A3B at 16K to 32K tokens per GPU, with Stock's step time under each group, and bars of peak-memory change relative to Stock annotated with Stock's absolute peak.
Figure 9. TIDE -AG barely slows concurrent compute; TIDE -RS cuts that slowdown several-fold (training shapes, fixed SM clock; lower is better; labels are TIDE ’s values). (a) Isolated calls. (b) MLP GEMM (Llama) or self-attention (Wan). Grouped bars comparing Stock FSDP2, FSDP2+SymmMem, and TIDE on Llama-3.1 70B and Wan2.2 all-gather and reduce-scatter: median latency, slowdown of concurrent compute, and SM time per call.
Figure 10. TIDE -AG stays faster from 8 to 64 GPUs; its reduce-scatter buffer stays flat ( Tables 7 and 8 ). (a) Median speedup over FSDP2’s fastest all-gather. (b) Buffer per GPU. Two line plots over 8 to 64 GPUs. Left: TIDE all-gather speedup in median latency over the best FSDP2 path ranges from 1.06 to 1.42 for 32 and 128 MiB shards; for 128 MiB it rises from 1.13 at 16 GPUs to 1.42 at 64. Right, log scale: FSDP2's reduce-scatter buffer doubles with the GPU count, reaching 2.03 and 8.12 GiB at 64 GPUs, while TIDE's stays near 0.6 and 2.4 GiB from 16 GPUs on, a 3.2 times gap at 64 GPUs.
Figure 11. Reduce-scatter time versus one-wave GEMM slowdown ( Table 4 ). Hollow markers: pre-packed fp32 input. Scatter plot of reduce-scatter time alone against the slowdown of a concurrent one-wave GEMM. Stock FSDP2 and NCCL configurations with default or eight CTAs sit near 68 to 71 percent slowdown, and NCCL's symmetric kernel at 11 percent; NCCL capped at four CTAs reaches 2.7 percent but takes 26.4 ms; TIDE with 16 or 4 sum CTAs sits at 1.3 to 2.9 percent and 7.0 to 12.0 ms.
RS alone
Victim slowdown (%)
Configuration
CTAs
(ms)
1-wave
28-wave
Elementwise
TIDE , training
16+4
6.96
2.9
4.2
1.6
TIDE , 4+4
4+4
12.02
1.3
4.0
0.6
Stock FSDP2
16
9.01
71.0
13.6
13.3
FSDP2+SymmMem
13
7.45
9.7
10.6
13.6
NCCL default
16
6.75
69.0
13.5
7.4
Table 4. Concurrent 32 MiB reduce-scatters on 8 nodes at a fixed SM clock (medians of three launches). Bold: lowest slowdown among rows no slower than Stock FSDP2.
Configuration
Displaced
Windows
Resident
CTAs (mode)
hit (%)
CTAs (mean)
TIDE , training
0
0
2.0
TIDE , 4+4
0
0
1.5
Stock FSDP2
12
65
13.4
FSDP2+SymmMem
0
0
12.0
NCCL default
12
64
14.7
Table 5. SM-id probe during a 32 MiB reduce-scatter on 8 nodes at a fixed SM clock (two launches). The probe launches the one-wave GEMM’s 32 four-CTA clusters, each CTA filling an SM and recording its SM id and entry and exit timestamps; displaced counts probe CTAs that start after the first wave. Resident CTAs come from a second probe with one CTA on each of the 148 SMs.
Latency
Victim
SM-ms
Op
System
p50 (ms)
slowdown
per call
Llama-3.1 70B
AG
Stock FSDP2
8.07
6.60%
218.2
AG
FSDP2+SymmMem
7.23
3.05%
86.3
AG
TIDE
4.90
0.33%
0.0
RS
Stock FSDP2
14.12
11.61%
651.4
Table 6. Training-shape replay at a fixed SM clock (medians of three launches of 30 isolated calls).
Nodes (GPUs)
Shard
TIDE -AG (0 SM)
NCCL Ring
NCCL CE
Ring + Copy-out
CE + Copy-out
vs. Best FSDP2
1 (8 GPUs)
32 MiB
0.38 (0.38)
0.40 (0.41)
0.35 (0.36)
0.50 (0.51)
0.45 (0.46)
1.22 ×
2 (16 GPUs)
32 MiB
1.11 (1.12)
1.35 (1.59)
1.08 (1.10)
1.53 (1.58)
1.26 (1.28)
1.14 ×
4 (32 GPUs)
32 MiB
2.57 (2.60)
2.71 (2.79)
3.14 (3.22)
3.11 (3.14)
3.56 (3.62)
1.21 ×
8 (64 GPUs)
32 MiB
5.52 (5.86)
5.46 (5.78)
7.25 (7.42)
6.15 (6.21)
7.96 (8.19)
1.06 ×
1 (8 GPUs)
128 MiB
1.40 (1.41)
1.50 (1.52)
1.27 (1.28)
1.85 (1.87)
1.61 (1.63)
1.16 ×
2 (16 GPUs)
128 MiB
4.19 (4.20)
5.29 (6.09)
4.04 (4.13)
5.99 (6.84)
4.73 (4.76)
1.13 ×
Table 7. All-gather latency, min (median) ms, for P=30 tensors. “+ Copy-out” adds FSDP2’s fused split_with_sizes_copy ; TIDE lands directly. Bold: the fastest path that yields FSDP2’s parameter layout; the last column compares medians.
Nodes (GPUs)
Output
TIDE -RS
Stock FSDP2
FSDP2+SymmMem
Bare NCCL
Buffer ( TIDE / FSDP2)
1 (8 GPUs)
32 MiB
0.41 (0.42)
0.59 (0.60)
0.60 (0.61)
0.39 (0.40)
0.29 / 0.28
2 (16 GPUs)
32 MiB
2.08 (2.12)
1.67 (1.75)
1.16 (1.27)
1.30 (1.45)
0.58 / 0.53
4 (32 GPUs)
32 MiB
3.70 (3.75)
3.39 (4.06)
3.06 (3.07)
2.69 (2.78)
0.61 / 1.03
8 (64 GPUs)
32 MiB
6.61 (8.12)
6.82 (7.96)
6.67 ( 6.84 )
5.40 (5.52)
0.64 / 2.03
1 (8 GPUs)
128 MiB
1.29 (1.33)
2.23 (2.25)
2.36 (2.37)
1.44 (1.46)
0.38 / 1.12
2 (16 GPUs)
128 MiB
7.63 (7.68)
6.70 (6.78)
4.62 (4.96)
5.17 (5.27)
2.26 / 2.12
Table 8. Reduce-scatter latency, min (median) ms, and buffer (GiB/GPU) for P=16 bf16 gradients. FSDP2 columns include _chunk_cat ; Bare NCCL starts from packed fp32 . Bold: lowest among per-parameter paths, and the smaller buffer.
Metric
Static or CLC-normal
CLC-high
Communication admission
Final GEMM wave
≤2 tile boundaries
Added RS latency
0.38–0.90 ms
0.09–0.54 ms
GEMM penalty
≈0 (RS waits)
0.015–0.14 ms
Table 9. Scheduler probes during a persistent GEMM. CLC-normal/high: normal/high-priority communication.
Figure 12. Qwen3-30B-A3B trained from scratch on public C4 (64 GPUs, 1,000 steps). (a) 50-step moving-average loss. (b) Difference from the first Stock run; Section 7.5 reports maxima over the shaded steps 500–1,000. Two panels over 1,000 training steps. Top: moving-average loss of two Stock FSDP2 runs and TIDE In-Place, which overlap as the loss falls from about 6 to 3.5. Bottom: percent difference from the first Stock run; after step 500, TIDE stays within 0.73 percent and the second Stock run within 1.12 percent.
Figure 13. Host-time allocation: the host allocates later layers’ gradients before the GPU finishes reading layer ℓ , so a GPU-safe release is too late for them to reuse its storage. A host timeline that enqueues three backward layers and allocates gradients early, above a GPU timeline whose read of layer ell finishes later; the GPU-safe release point lies after the host allocations.
Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.
Fully Sharded Data Parallel (FSDP), also known as Zero Redundancy Optimizer (ZeRO), is widely used for large-scale model training, because of its memory efficiency and minimal intrusion on model code. However, existing FSDP systems rely on fixed element-wise or row-wise sharding formats that conflict with block-structured computations. As a result, they struggle to support modern structure-aware training methods, including block-wise quantization and non-element-wise optimizers such as Shampoo and Muon. In addition, today's implementations incur communication and memory overheads that degrade efficiency at the scale of tens of thousands of GPUs. We introduce veScale-FSDP, a novel FSDP system that combines RaggedShard, a flexible sharding format, with a structure-aware planning algorithm to deliver both flexibility and performance. veScale-FSDP enables zero-copy FSDP communications and natively supports block-wise quantization and non-element-wise optimizers, achieving 5% to 66% higher throughput and 16% to 30% lower memory usage than existing FSDP systems, while scaling efficiently to tens of thousands of GPUs.
The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across multiple GPUs. However, it incurs prohibitive communication overhead when deployed at scale, particularly on multi-tier interconnects with heterogeneous performance. Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations. Such approaches decouple large DP deployments into smaller, loosely-coupled federation groups, requiring minimal inter-group traffic while keeping the global batch size bounded by the groups' size. Formal analysis of communication costs and experimental validation prove their scalability and flexibility. A Llama3.1 8B pre-training on 512 A100 GPUs shows that, under identical hyperparameters, FL+FSDP and FL+HSDP achieve up to 8.04 faster data processing and 4.48 lower evaluation perplexity than their counterparts, demonstrating superior computational efficiency and improved model quality. These properties stem from reduced communication overhead and the bounded growth of the global batch size relative to the federation group size.
We present tensor and sequence parallelism (TSP), a parallel execution strategy that folds tensor parallelism and sequence parallelism onto a single device axis. In conventional multi-dimensional parallelism layouts, tensor parallelism (TP) shards model weights while sequence parallelism (SP) shards tokens, reducing per-device parameter or activation memory, respectively. Traditionally, each scheme is assigned its own mesh dimension. TSP instead assigns each rank both a weight shard and a sequence shard, reducing both parameter and activation memory along the same device axis. We implement this design with two runtime schedules. For attention, ranks iterate over broadcast parameter shards and reconstruct context through a sequence-wise key/value exchange. For gated MLPs, weight shards circulate in a ring while partial outputs accumulate locally. By sharding both weights and activations across the same devices, TSP trades additional communication volume for reduced memory overhead. We provide a theoretical communication and memory analysis, describe our implementation of TSP attention and gated MLP blocks, and benchmark TSP against TP, SP, and TP+SP. These results position TSP as a hardware-aware alternative for long-context and memory-constrained model training, and as a viable axis of parallelism in concert with existing parallelism schemes such as pipeline and expert parallelism for dense and mixture-of-expert models.