When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry

Michael Bidollahkhani; Freja Nordsiek; Julian M. Kunkel

cs.DCMar 17, 2026

When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry

Authors: Michael Bidollahkhani, Freja Nordsiek, Julian M. Kunkel

Organizations: Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG) Georg-August-Universität Göttingen Göttingen, Germany

Abstract

GPU nodes are central to modern HPC and AI workloads, yet many failures do not manifest as immediate hard faults. While some instabilities emerge gradually as weak thermal or efficiency drift, a significant class occurs abruptly with little or no numeric precursor. In these detachment-class failures, GPUs become unavailable at the driver or interconnect level and the dominant observable signal is structural, including disappearance of device metrics and degradation of monitoring payload integrity. This paper proposes an observability-aware early-warning framework that jointly models (i) utilization-aware thermal drift signatures in GPU telemetry and (ii) monitoring-pipeline degradation indicators such as scrape latency increase, sample loss, time-series gaps, and device-metric disappearance. The framework is evaluated on production telemetry from GPU nodes at GWDG, where GPU, node, monitoring, and scheduler signals can be correlated. Results show that detachment failures exhibit minimal numeric precursor and are primarily observable through structural telemetry collapse, while joint modeling increases early-warning lead time compared to GPU-only detection. The dataset used in this study is publicly available at https://doi.org/10.5281/zenodo.19052367.

Figures & tables

Failure pattern	Numeric precursor	Dominant observable signal
Thermal / efficiency drift	Gradual (weak)	Temperature drift and trend anomalies
Load-triggered instability	Moderate (workload-correlated)	Thermal and power excursions under load
GPU detachment (“fallen off bus”)	None or negligible	Loss of device metrics, scrape sample drop, gaps
Chronic detachment recurrence	None per event	Repeated structural anomalies on same node

TABLE I: Qualitative characterization of GPU-node instability patterns observed in GWDG production telemetry.

Incident category	Count
GPU error / problem	31
GPU fell off bus	24
GPU unknown	5
GPU lost	3
GPU ECC	2
GPU failed	2

TABLE II: GPU-related incident categories and counts in the analyzed GWDG incident catalog.

Node	$t_{0}$ (UTC)	Category	Signals ( numSignalsLong )	Top signals by $\|\Delta\|$
ggpu121	2025-02-09 15:00	gpu error/problem	70	node_memory_MemAvailable_bytes ( $\Delta=13615104$ ), node_load1 ( $\Delta=-0.09$ ), node_load5 ( $\Delta=-0.04$ ), node_load15 ( $\Delta=-0.03$ )
ggpu139	2025-03-21 09:45	gpu fell off bus	510	nodes_total_gpus_when_good ( $\Delta=-4$ )
ggpu142	2025-02-16 12:50	gpu fell off bus	340	node_memory_MemAvailable_bytes ( $\Delta=-3244032$ )
ggpu149	2025-06-12 07:30	gpu fell off bus	350	node_memory_MemAvailable_bytes ( $\Delta=111666880512$ ), node_load15 ( $\Delta=-0.05$ ), node_load5 ( $\Delta=-0.01$ )

TABLE III: Weak-event summary extracted from weak_events_summary.csv . For each incident-anchored weak event ( label=pre_failure ), we report the number of available signals ( numSignalsLong ) and the dominant delta-ranked feature shifts within the forensic comparison window. Variance-shift statistics ( diffStd ) are frequently zero in this slice and therefore do not provide a stable ranking axis.

Node	Alignment time $t_{0}$	Category	Dominant pre-fault observability signal
ggpu142	2025-02-16 12:50	GPU detachment	Partial GPU metric-family loss and scrape payload collapse
ggpu142	2025-03-21 09:10	GPU detachment	GPU telemetry disappearance with preceding temperature/utilization activity
ggpu149	2025-03-21 10:40	GPU detachment	Abrupt GPU metric disappearance with scrape degradation
ggpu149	2025-06-12 07:30	GPU detachment	GPU telemetry loss preceding delayed operator detection
ggpu149	2026-01-18 12:40	GPU detachment	GPU metric disappearance without stable numeric precursor

TABLE IV: Incident-anchored pre-fault observability behavior for GPU detachment incidents. Dominant signals are structural (GPU telemetry disappearance) rather than stable numeric precursors.

Node	$t_{0}^{incident}$	$t_{0}^{used}$	Tidy archive
ggpu142	2025-02-17 00:00	2025-02-16 12:50	ggpu142_2025-02-17_gpus-fallen-off-bus_tidy.csv.bz2
ggpu142	2025-03-21 00:00	2025-03-21 09:10	ggpu142_2025-03-19_gpu-problem_tidy.csv.bz2
ggpu149	2025-03-21 00:00	2025-03-21 10:40	ggpu149_2025-03-21_gpus-dropped-off-bus_tidy.csv.bz2
ggpu149	2025-06-12 00:00	2025-06-12 07:30	ggpu149_2025-06-12_gpus-dropped-off-bus_tidy.csv.bz2
ggpu149	2026-01-19 00:00	2026-01-18 12:40	ggpu149_2026-01-19_gpus-dropped-off-bus_tidy.csv.bz2

TABLE V: Detachment-class incidents used for forensic alignment ( failureClass = gpu error / fallen off bus ). The alignment time

t_{0}^{used}

is derived from scrapeCountDrop .

Panel of Figure 6 — Fig. 1: Average lead time under a fixed 1% alert budget. Joint (GPU + observability) increases early warning, especially for Isolation Forest (IF), compared to GPU-only features.

Plane	Method	Avg Lead	Median Lead	Max Lead	Avg RunLen	Runs
GPU	z-score	1.667	0.0	8.0	4.706	17
GPU	Isolation Forest	2.000	0.0	10.0	8.000	10
GPU	One-Class SVM	2.333	0.0	11.0	7.273	11
Joint	z-score	1.667	1.0	5.0	5.000	16
Joint	Isolation Forest	7.000	1.5	29.0	4.706	17
Joint	One-Class SVM	4.833	0.0	26.0	8.000	10

TABLE VI: Baseline plane comparison at 1% alert budget. Lead time is reported in windows within a lookback of 48 windows.

Fault Detection

Fault detection identifies abnormal behavior in machines, sensors, infrastructure, software, and autonomous systems before it causes harm or disruption.

See all 104 papers

GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures
Jul 1, 2026Parv Agarwal, Asif EkbalFault Detection Fault Diagnosis
AI-NLP-ML Research Laboratory, Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, Bihar, India.
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
May 10, 2026Daemyung Kang, Eunjin Hwang, Hanjeong Lee +11Fault-Tolerant Distributed Training Distributed Training
Lablup Inc.∗
Detecting Hidden ML Training With Zero-Overhead Telemetry
Jun 17, 2026Robi Rahman, Sabiha TajdariAdversarial Attacks Adversarial Robustness
Machine Intelligence Research Institute · University of Virginia

Jul 1, 2026J/K move · Enter open · S save

GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures

Parv Agarwal, Asif Ekbal

AI-NLP-ML Research Laboratory, Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, Bihar, India.

Fault Detection Fault Diagnosis

Open paper AskPDF arXiv

GPU training jobs fail often, roughly two in five on large production clusters, yet the operator typically learns of a failure only by reconnecting hours later. Experiment trackers require editing the training script and maintaining a cloud connection; the scheduler's mail hook delivers a single status line with no cause and no logs. GPUAlert is a command-line wrapper that monitors any training command at the process boundary, and with no change to that command, emails a structured notification on completion carrying a classified failure cause, durable logs, and output artifacts. The tool is organized around three reliability primitives: a pre-launch log guarantee that establishes the durable destination before the child process can crash, notifier isolation that makes the wrapper's exit code a pure function of the child's status regardless of whether the email succeeds, and a non-silent artifact budget that bounds attachment size without ever dropping output silently. We release a labelled corpus of 474 GPU training logs across 15 failure classes and a reproducible evaluation harness. On the twelve hardware-reproduced classes, the ordered-rule classifier reaches 0.997 macro-F1, against 0.830 for unordered keyword matching and 0.133 for exit-code inspection. Wrapper overhead is a constant approximately 3ms per job; the pre-launch guarantee preserves a log where a shell redirect yields nothing; and across all 15 failure modes the wrapper returns the child's exit code unchanged even when the SMTP relay is unreachable.

Abstract

Fault Detection

Explore similar work

GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures

From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs

Detecting Hidden ML Training With Zero-Overhead Telemetry