The rapid scale-up of AI infrastructure has made datacenter reliability a cornerstone of the cloud business economy. Since hardware failures in hyper-scale AI clusters can stall multi-million dollar training workloads, the downtime of individual machines can cause outsized economic losses. Minimizing machine downtime, therefore, is more important than ever. To address this, we present Hardware Diagnoser (HardwareDx), an AI-driven hardware diagnosis system that applies a graph foundation model approach to automated datacenter maintenance. It allows a single model to diagnose many platforms and accelerate onboarding of new hardware products. Deployed at production scale at Google, HardwareDx yields substantial efficiency gains across diverse compute architectures. Real-world evaluations demonstrate that it reduces production downtime by 18.3% on GPU platforms and 16.1% on TPUs. Furthermore, it significantly accelerates hardware deployment, cutting quality assurance (QA) time by 28.3% for GPUs and 25.1% for TPUs. These results establish HardwareDx as a field-tested solution for maximizing the reliability and cost efficiency of next generation AI fleets.
Figures & tables
Figure 1: Simplified flow of datacenter operations.
Figure 2: HardwareDx Model Architecture. The machine topology and diagnostic signals are represented as a heterogeneous graph. Each node type is encoded according to the features it has and the topology is homogenized to create the universal graph format . This graph then goes through the GNN backbone before predictions are made for each task and component type
Figure 3: Repair time improvements from deploying HardwareDx on a high-volume TPU platform. HardwareDx improves repair times across the distribution.
Hardware Platform
Mean TTR Reduction
QA Delivery Reduction
Automated Diagnostic % (improvement)
H100
18.3% ± 7.6%
28.3% ± 10.7%
93.9% (+10.7%)
TPU
16.1% ± 2.6%
25.1% ± 2.5%
82.8% (+9.9%)
Table 1: A/B Production Results: Repair Time Improvements and Automation Coverage. ± denotes one standard error (Appendix H ).
Figure 5Table 6
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Illustration of topology-aware diagnosis in Case 1 (Multiple GPUs Missing). Identical Missing telemetry signals on GPUs lead HardwareDx to distinct repair predictions based on the underlying PCIe hierarchy: (a) two missing GPUs sharing a PCIe chain implicate the shared PCIe chain (76.3%), (b) two missing GPUs across separate PCIe chains localize the fault to the individual GPUs (8.5%), and (c) more than two missing GPUs across chains escalate to the GPU backplane (6.8%) or software (8.5%).
Category
Percentage
GPU Backplane
6.8%
GPU repairs
8.5%
PCIe repairs
76.3%
Software
8.5%
Appendix
Table 4: Breakdown of recommendations by HardwareDx when encountering multiple missing GPUs. Cases are categorized by the most involved repair recommended (Backplane > GPU > PCIe > Software). Cases with comorbid symptoms outside these categories (9.7%) were excluded.
Feature Node
Feature Keys
DIMM Error
Error (message / error type); Max Rate (max errors per minute); Threshold (configured error-rate threshold)
CPU Error
Error (message / error type); Max Rate (max errors per minute); Threshold (configured error-rate threshold)
Sensor
Name (sensor identifier); Unit (degrees, volts, RPM, watts, etc.); State (health state: GOOD, DETECTED, MISSING, HOT, COLD, etc.); Violations (number of out-of-range readings)
Component Bandwidth
Flow (connection type: GPU-to-GPU or GPU-to-CPU); Bandwidth (measured value); Threshold
Name (symptom identifier); Test Name (test that produced the symptom); Pass/Fail Status; Message (detailed error or diagnostic output)
Appendix
Table 5: Feature node types and their feature keys.
Ablated Feature Node Type
Component F1 Score
Job Accuracy
Job Recall
Full Model (Reference)
0.9715
0.9625
0.9764
symptom
0.3532
0.2426
0.2443
pcie error
0.9708
0.9599
0.9695
sensor
0.9720
0.9634
0.9686
dimm error
0.9723
0.9642
0.9808
bus error
0.9747
0.9625
0.9764
Appendix
Table 6: Leave-one-out feature ablation on TPU v7x at 10k training steps, ordered from lowest to highest F1 without the feature.
Kept Feature Node Type(s)
Component F1 Score
Job Accuracy
Job Recall
Full model (all 13 features)
0.9691±0.0006
0.9688±0.0002
0.9783±0.0006
All except symptom (12 features)
0.9270±0.0060
0.9075±0.0034
0.9151±0.0037
only symptom
0.9697±0.0013
0.9707±0.0007
0.9791±0.0006
only bus error
0.5725±0.0003
0.5031±0.0002
0.5053±0.0002
only field diag
0.5531±0.0030
0.6686±0.0017
0.6740±0.0016
only sensor
0.4654±0.0012
0.5892±0.0003
0.5909±0.0009
Appendix
Table 7: Single-feature inclusion (“keep-only”) ablation on H100 (mean ± one standard error across 3 seeds at 30k steps). The top two reference rows show the full 13-feature model and the 12-feature model with symptom removed. Each remaining row keeps only the listed feature node type and removes the other 12, ordered by descending F1.
Platform
Unsupervised Dataset
Labeled
Median Graph
Unique
Size (Trajectories)
Cases
Size (Nodes)
Labels
TPU v5p
111,764
5,236
874
59
TPU v7x
43,987
7,529
992
66
TPU v7
1,381
245
1,142
29
H100
81,015
6,780
1,613
85
H200
7,558
1,838
1,276
80
Appendix
Table 8: HardwareDx Component Training Dataset Statistics
Metric
Existing System
HardwareDx Predict Repair Actions
Final Action Replication Rate
56.23%
63.42%
Action-Weighted Final Action Replication Rate
62.50%
73.04%
Average Estimated Iterations ( Δ over actual iterations)
-0.29
-0.50
Appendix
Table 9: Performance Comparison: Existing Automated System vs. HardwareDx Predict Repair Actions with Success Rate
B200 (Finetune Platform)
Model
Job-level Acc
Job-level Rec
Component F1
FM-2Platform-FT
0.9848
0.9939
0.9817
Single
0.9787
0.9939
0.9728
Appendix
Table 10: Results of finetuning a foundation model pretrained on H100 and H200 data on the B200 platform.
GNN Layer
Expected Calibration Error ( ↓ )
Brier Score ( ↓ )
GATv2
0.0015
0.0062
GIN
0.0020
0.0067
Appendix
Table 11: GNN Model Calibration on TPU v7x using Expected Calibration Error and Brier Scores (lower is better). The evaluated models both include transformer layers but include different local GNN layers.
Figure 7: Comprehensive ablation study on the number of layers.
Fault diagnosis identifies, localizes, and explains failures in engineered and computational systems using sensor data, logs, models, and domain knowledge.
Large-scale AI training is fundamentally a distributed systems problem, where hardware failures are routine operating conditions rather than rare exceptions, yet public operational evidence from production training clusters remains limited. This report presents an empirical analysis of a 63-node NVIDIA B200 production cluster (504 GPUs), using 55 days of Prometheus time-series data and 73 days of operational logs covering 224 multi-node training sessions. The environment is cross-organizational: five parties (SKT, Upstage, Lablup, NVIDIA Korea, VAST Data) share a unified monitoring pipeline. This enabled joint diagnosis of a 60-node-scale storage I/O bottleneck absent in 2-4-node tests, a production-scale phenomenon no single team could isolate alone. We perform three quantitative analyses yielding four findings. First, over 751 Prometheus metrics and 10 XID-identified GPU failures, no single metric is consistently dominant across failure types, motivating multi-signal detection. Second, 523 checkpoint events trace the save/load path from GPU VRAM to the NFS server: restart loading reaches 21.5% of maximum read bandwidth (700 GB/s) and save bursts 16.0% of maximum write bandwidth (250 GB/s), with NFS/RPC queueing and transport-layer backlog rising together. Third, across 224 sessions over 73 days, node exclusions concentrate so the top 3 of 63 nodes account for over 50%. Fourth, auto-retry chain analysis shows a 33.3% success rate over 12 chains (73 attempts), 2.7x the 12.5% manual rate, with a median retry interval of 11 minutes (IQR 10-11). All analyses are grounded in production infrastructure providing session-level workload management, GPU-centric scheduling, and unified observability.
Large-scale machine learning workloads on the cloud suffer from silent inefficiencies, which do not trigger failures or obvious anomalies, making them hard to detect and diagnose. Existing diagnosis systems rely on guest-side instrumentation or dedicated healthy baselines. These assumptions do not hold in modern production cloud environments, where operators cannot observe inside the guest, and a healthy baseline for each setup is impractical because of system diversity. We present Unveil, a hardware-centric observability framework for detecting, identifying, and remediating silent inefficiencies using only host-visible telemetry. Unveil requires neither guest instrumentation nor application modification. It uses unsupervised learning models to detect inefficiencies and identifies known ones through a novel fingerprinting algorithm. An extensible knowledge base containing curated remediation scripts is used to correct problems without interrupting workloads when possible. We evaluate Unveil on distributed machine learning workloads, such as large language model serving, introducing less than 2% monitoring overhead. It successfully recovered from a 7% throughput loss and reduced inference latency by up to 43.48%. These results demonstrate that host-visible telemetry is sufficient for practical and continuous diagnosis of production machine learning systems.
Training frontier-scale foundation models involves coordinating tens of thousands of GPUs over multi-month runs, where even minor performance degradations can accumulate into substantial efficiency losses. Existing health-check mechanisms, such as NCCL tests or GPU burn-in, primarily focus on functional correctness and often fail to detect fail-slow behaviors that silently degrade system performance. In this paper, we present Guard, a scalable system for detecting stragglers and ensuring node health in large-scale training clusters. Guard combines lightweight online performance monitoring during training with an offline node-sweep mechanism that systematically evaluates and qualifies nodes before they participate in production workloads. This design enables Guard to detect both acute failures and long-running fail-slow behaviors that traditional diagnostics cannot capture. Deployed on large-scale foundation model pretraining workloads, Guard improves mean FLOPs utilization by up to 1.7x, reduces run-to-run training step variance from 20% to 1%, increases mean time to failure (MTTF), and significantly reduces operational and debugging overhead. These results demonstrate that proactive straggler detection and systematic node qualification are critical for maintaining stable and efficient large-scale training.