cs.NIFeb 16, 2026

When Scaling Fails: Network and Fabric Effects on Distributed GPU Training Performance

Authors: Dinesh Gopalan, Ratul Ali

Organizations: Principal Member of Technical Staff, AI Enablement AMD Dallas, Texas, USA · Institute of Information Technology (Postgraduate Researcher) Jahangirnagar University Dhaka, Bangladesh

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scaling distributed GPU training is commonly assumed to yield predictable performance gains as additional nodes are added. In practice, many large-scale deployments encounter diminishing returns and unstable behavior well before theoretical limits are reached. This paper examines why scaling fails in real systems, with a focus on the role of network and fabric effects that are often overlooked by higher-level training frameworks. We present an empirical study of distributed GPU training performance across multiple production-scale clusters. Our results show that network topology, congestion dynamics, collective synchronization behavior, and GPU locality frequently dominate end-to-end training performance once workloads move beyond a small number of nodes. Identical models and software stacks can exhibit sharply different scaling characteristics depending on fabric design and runtime communication patterns. We identify recurring failure modes that emerge as training transitions from single-node to multi-node execution, including synchronization amplification, topology-induced contention, and locality-driven performance variance. These effects are often invisible to standard profiling tools and are therefore misdiagnosed as framework or model-level inefficiencies. Based on these findings, we outline practical diagnostic principles that system builders can apply to understand scaling limits, improve predictability, and reduce the cost of large-scale distributed training.

Figures & tables

Figure 1: Scaling behavior of distributed GPU training throughput.
Figure 2: Coordination Bottleneck Timeline in Distributed GPU Training.
Figure 3: System architecture overview.
Figure 4: Experimental system architecture showing multi-GPU nodes connected via a shared network fabric. A lightweight coordination layer operates alongside existing training frameworks to observe synchronization behavior and apply adaptive pacing.
NodesMetricBaselineCoordination
4Throughput10241018 (–0.6%)
 CV0.020.02
8Throughput19801995 (+0.8%)
 CV0.040.03
16Throughput36003720 (+3.3%)
 CV0.090.05
Table 1: Performance comparison: Baseline vs. Coordination enabled. Throughput (samples/sec) and coefficient of variation (CV) of iteration time.
Figure 5: Performance comparison of distributed GPU training throughput as node count increases. Baseline execution exhibits early saturation at moderate scale. With coordination enabled, throughput improves at higher node counts and remains more stable due to reduced synchronization skew.
More in this topic

Distributed Training

Distributed training coordinates computation across multiple devices or machines to train models that exceed single-device limits.

See all 129 papers

Explore similar work

CardsList
May 23, 2026

ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training

The rapid scaling of large language model training requires distributing GPU resources across multiple data center buildings and regions. We refer to such paradigm as "scale-across" training. As infrastructure expands, the system design space becomes increasingly intricate, encompassing new model architectures, hardware heterogeneity, and evolving communication patterns. Drawing from Meta's production experience, we highlight the complexities of deploying training jobs across a few data centers housing hundreds of thousands of GPUs. To accelerate exploration of the large design space and to enable efficient training for frontier model development, we conduct in-depth characterization of three key design dimensions: parallelism placement, parallelism scheduling, and network layer technologies. We then propose ScaleAcross Explorer, an optimizer that considers the interplay of design dimensions and holistically optimizes scale-across training. Testbed experiments and simulations demonstrate up to 64.62% training speedups over production configuration and up to 37.59% training speedups over the state-of-the-art baseline across a wide range of design points.

Minghao Li, Alicia Golden, Samuel Hsia +14

1Harvard University · 2Meta Platforms, Inc.

Communication-Efficient Distributed TrainingDistributed TrainingLLM Training
AskPDF
May 18, 2026

Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training

Training frontier-scale foundation models involves coordinating tens of thousands of GPUs over multi-month runs, where even minor performance degradations can accumulate into substantial efficiency losses. Existing health-check mechanisms, such as NCCL tests or GPU burn-in, primarily focus on functional correctness and often fail to detect fail-slow behaviors that silently degrade system performance. In this paper, we present Guard, a scalable system for detecting stragglers and ensuring node health in large-scale training clusters. Guard combines lightweight online performance monitoring during training with an offline node-sweep mechanism that systematically evaluates and qualifies nodes before they participate in production workloads. This design enables Guard to detect both acute failures and long-running fail-slow behaviors that traditional diagnostics cannot capture. Deployed on large-scale foundation model pretraining workloads, Guard improves mean FLOPs utilization by up to 1.7x, reduces run-to-run training step variance from 20% to 1%, increases mean time to failure (MTTF), and significantly reduces operational and debugging overhead. These results demonstrate that proactive straggler detection and systematic node qualification are critical for maintaining stable and efficient large-scale training.

Guanliang Liu, Abhinandan Patni, Congzhu Lin +14

Store Foundational AI, Amazon, Seattle, WA, USA.

Distributed TrainingAnomaly Detection
AskPDF
Aug 24, 2026

ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork

Training large-scale AI models often outgrows a single data center, demanding sharded, multi-cluster, and decentralized training. However, the huge space of resource allocations makes exhaustive benchmarking and manual tuning impractical, while performance depends on tightly coupled factors like model size, GPU memory, batch size, bandwidth, and sharding strategy. We introduce ShardMeter, a lightweight analytical performance model that predicts the end-to-end runtime of transformer-based workloads across arbitrary sharded, distributed, and even decentralized training. Given a model's characteristics and a target hardware topology, ShardMeter estimates per-GPU and per-island throughput, training cost, total wall-clock time, and identifies performance bottlenecks. Our analysis reveals diminishing-return regimes as island size increases, quantifies transitions between compute- and communication-bound scaling, evaluates hyperparameter trade-offs, and models cost-throughput for large-scale decentralized training. ShardMeter exposes these insights to quickly explore the configuration space, choose near-optimal deployment plans, and avoid costly trial and error.

Tim Beringer, Patrick Diem, Felix Wolf +1

Technical University of Darmstadt · PanocularAI

Distributed TrainingDecentralized Learning
AskPDF