The rapid scale-up of AI infrastructure has made datacenter reliability a cornerstone of the cloud business economy. Since hardware failures in hyper-scale AI clusters can stall multi-million dollar training workloads, the downtime of individual machines can cause outsized economic losses. Minimizing machine downtime, therefore, is more important than ever. To address this, we present Hardware Diagnoser (HardwareDx), an AI-driven hardware diagnosis system that applies a graph foundation model approach to automated datacenter maintenance. It allows a single model to diagnose many platforms and accelerate onboarding of new hardware products. Deployed at production scale at Google, HardwareDx yields substantial efficiency gains across diverse compute architectures. Real-world evaluations demonstrate that it reduces production downtime by 18.3% on GPU platforms and 16.1% on TPUs. Furthermore, it significantly accelerates hardware deployment, cutting quality assurance (QA) time by 28.3% for GPUs and 25.1% for TPUs. These results establish HardwareDx as a field-tested solution for maximizing the reliability and cost efficiency of next generation AI fleets.
Figures & tables
Figure 1: Simplified flow of datacenter operations.
Figure 2: HardwareDx Model Architecture. The machine topology and diagnostic signals are represented as a heterogeneous graph. Each node type is encoded according to the features it has and the topology is homogenized to create the universal graph format . This graph then goes through the GNN backbone before predictions are made for each task and component type
Figure 3: Repair time improvements from deploying HardwareDx on a high-volume TPU platform. HardwareDx improves repair times across the distribution.
Hardware Platform
Mean TTR Reduction
QA Delivery Reduction
Automated Diagnostic % (improvement)
H100
18.3% ± 7.6%
28.3% ± 10.7%
93.9% (+10.7%)
TPU
16.1% ± 2.6%
25.1% ± 2.5%
82.8% (+9.9%)
Table 1: A/B Production Results: Repair Time Improvements and Automation Coverage. ± denotes one standard error (Appendix H ).
Figure 5Table 6
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Illustration of topology-aware diagnosis in Case 1 (Multiple GPUs Missing). Identical Missing telemetry signals on GPUs lead HardwareDx to distinct repair predictions based on the underlying PCIe hierarchy: (a) two missing GPUs sharing a PCIe chain implicate the shared PCIe chain (76.3%), (b) two missing GPUs across separate PCIe chains localize the fault to the individual GPUs (8.5%), and (c) more than two missing GPUs across chains escalate to the GPU backplane (6.8%) or software (8.5%).
Category
Percentage
GPU Backplane
6.8%
GPU repairs
8.5%
PCIe repairs
76.3%
Software
8.5%
Appendix
Table 4: Breakdown of recommendations by HardwareDx when encountering multiple missing GPUs. Cases are categorized by the most involved repair recommended (Backplane > GPU > PCIe > Software). Cases with comorbid symptoms outside these categories (9.7%) were excluded.
Feature Node
Feature Keys
DIMM Error
Error (message / error type); Max Rate (max errors per minute); Threshold (configured error-rate threshold)
CPU Error
Error (message / error type); Max Rate (max errors per minute); Threshold (configured error-rate threshold)
Sensor
Name (sensor identifier); Unit (degrees, volts, RPM, watts, etc.); State (health state: GOOD, DETECTED, MISSING, HOT, COLD, etc.); Violations (number of out-of-range readings)
Component Bandwidth
Flow (connection type: GPU-to-GPU or GPU-to-CPU); Bandwidth (measured value); Threshold
Name (symptom identifier); Test Name (test that produced the symptom); Pass/Fail Status; Message (detailed error or diagnostic output)
Appendix
Table 5: Feature node types and their feature keys.
Ablated Feature Node Type
Component F1 Score
Job Accuracy
Job Recall
Full Model (Reference)
0.9715
0.9625
0.9764
symptom
0.3532
0.2426
0.2443
pcie error
0.9708
0.9599
0.9695
sensor
0.9720
0.9634
0.9686
dimm error
0.9723
0.9642
0.9808
bus error
0.9747
0.9625
0.9764
Appendix
Table 6: Leave-one-out feature ablation on TPU v7x at 10k training steps, ordered from lowest to highest F1 without the feature.
Kept Feature Node Type(s)
Component F1 Score
Job Accuracy
Job Recall
Full model (all 13 features)
0.9691±0.0006
0.9688±0.0002
0.9783±0.0006
All except symptom (12 features)
0.9270±0.0060
0.9075±0.0034
0.9151±0.0037
only symptom
0.9697±0.0013
0.9707±0.0007
0.9791±0.0006
only bus error
0.5725±0.0003
0.5031±0.0002
0.5053±0.0002
only field diag
0.5531±0.0030
0.6686±0.0017
0.6740±0.0016
only sensor
0.4654±0.0012
0.5892±0.0003
0.5909±0.0009
Appendix
Table 7: Single-feature inclusion (“keep-only”) ablation on H100 (mean ± one standard error across 3 seeds at 30k steps). The top two reference rows show the full 13-feature model and the 12-feature model with symptom removed. Each remaining row keeps only the listed feature node type and removes the other 12, ordered by descending F1.
Platform
Unsupervised Dataset
Labeled
Median Graph
Unique
Size (Trajectories)
Cases
Size (Nodes)
Labels
TPU v5p
111,764
5,236
874
59
TPU v7x
43,987
7,529
992
66
TPU v7
1,381
245
1,142
29
H100
81,015
6,780
1,613
85
H200
7,558
1,838
1,276
80
Appendix
Table 8: HardwareDx Component Training Dataset Statistics
Metric
Existing System
HardwareDx Predict Repair Actions
Final Action Replication Rate
56.23%
63.42%
Action-Weighted Final Action Replication Rate
62.50%
73.04%
Average Estimated Iterations ( Δ over actual iterations)
-0.29
-0.50
Appendix
Table 9: Performance Comparison: Existing Automated System vs. HardwareDx Predict Repair Actions with Success Rate
B200 (Finetune Platform)
Model
Job-level Acc
Job-level Rec
Component F1
FM-2Platform-FT
0.9848
0.9939
0.9817
Single
0.9787
0.9939
0.9728
Appendix
Table 10: Results of finetuning a foundation model pretrained on H100 and H200 data on the B200 platform.
GNN Layer
Expected Calibration Error ( ↓ )
Brier Score ( ↓ )
GATv2
0.0015
0.0062
GIN
0.0020
0.0067
Appendix
Table 11: GNN Model Calibration on TPU v7x using Expected Calibration Error and Brier Scores (lower is better). The evaluated models both include transformer layers but include different local GNN layers.
Figure 7: Comprehensive ablation study on the number of layers.
Fault diagnosis identifies, localizes, and explains failures in engineered and computational systems using sensor data, logs, models, and domain knowledge.