Large-scale machine learning workloads on the cloud suffer from silent inefficiencies, which do not trigger failures or obvious anomalies, making them hard to detect and diagnose. Existing diagnosis systems rely on guest-side instrumentation or dedicated healthy baselines. These assumptions do not hold in modern production cloud environments, where operators cannot observe inside the guest, and a healthy baseline for each setup is impractical because of system diversity. We present Unveil, a hardware-centric observability framework for detecting, identifying, and remediating silent inefficiencies using only host-visible telemetry. Unveil requires neither guest instrumentation nor application modification. It uses unsupervised learning models to detect inefficiencies and identifies known ones through a novel fingerprinting algorithm. An extensible knowledge base containing curated remediation scripts is used to correct problems without interrupting workloads when possible. We evaluate Unveil on distributed machine learning workloads, such as large language model serving, introducing less than 2% monitoring overhead. It successfully recovered from a 7% throughput loss and reduced inference latency by up to 43.48%. These results demonstrate that host-visible telemetry is sufficient for practical and continuous diagnosis of production machine learning systems.
Figures & tables
Tool
VM-compat.
Cont. Deploy
No Code △
HW Indep.
No-Base.
Scope
Plumber ( Kuchnik et al., 2022 )
✗
✗
✓
✓ (A)
data loading
SuperBench ( Xiong et al., 2024 )
✓
✗
✓
✓
✗ (H)
component
Aegis ( Dong et al., 2025 )
✗
✓
✓
✓
(P)
compute or communication
Greyhound ( Wu et al., 2025 )
✗
✓
✓
✓
(S+P)
compute or communication
Pulse ( Xiao et al., 2026 )
✓
✓
✓
✗
(S+P)
compute or communication
Mycroft ( Deng et al., 2025 )
✗
✓
✓
✓
(P)
Process rank + RDMA or GPU (communication only)
Table 1. Comparison with existing monitoring and diagnosis tools.
Figure 1. GPU–NIC NUMA placement ( subsection 2.6 ). Diagram of one host with two NUMA nodes. NUMA 0 keeps the NCCL host buffer beside the Ethernet NIC. NUMA 1 places that buffer on the other node, so traffic crosses NUMA before reaching the NIC and the other hosts.
Figure 2. Comparison of Prodigy, eACGM, and Unveil (Our work) under GPU–NIC NUMA misalignment ( subsection 2.6 ). Three panels for GPU–NIC NUMA misalignment. Prodigy reconstruction-error distributions for healthy and inefficient runs overlap, and many healthy samples sit above the alert threshold. eACGM log-likelihood distributions for the two runs are nearly the same. Unveil records fingerprint hits on inefficient runs and none on the healthy run.
Figure 3. Distribution of fingerprints at the subsystem-level for four silent inefficiency types (panels (a)–(d), red). Red bars denote fingerprints observed under the injected inefficiencies, while green bars denote fingerprints from anomalous windows detected during healthy executions ("False Positives") ( subsubsection 3.4.2 ). Four bar charts of fingerprint share across CPU, GPU, memory, interconnect, network, and storage. Panels are NUMA misplacement, IRQ locality misses, reduced huge-page coverage, and GPU power throttling. Red bars are injected inefficiencies and green bars are false positives from healthy runs. The red bars concentrate on different subsystems from the green bars.
Figure 4. Impact of GPU power limits on prefill throughput and decode bandwidth ( subsubsection 3.5.1 ). Line charts of LLaMA and Qwen feed-forward and attention layers as the GPU power limit falls from 350 W to 100 W. Prefill throughput drops earlier and more steeply than decode bandwidth. LLaMA feed-forward throughput declines almost linearly, while the other series stay flat until a sharp drop at the lowest power limits.
Figure 5. Hardware-centric observability and remediation workflow in Unveil ( section 4 ). Flowchart of Unveil on one host. Monitoring agents collect telemetry from CPUs, GPUs, memory, disks, and NICs while containerized or bare-metal workloads run. Each window is turned into features, tested for an anomaly, and attributed to the top contributing factors. The result is reported and can be remediated automatically or manually.
Figure 6. Runtime overhead (left axis) and telemetry data rate (right axis) across relevant tools ( subsubsection 5.2.1 ). Grouped bars for Prodigy at 1 second, LDMS at 100 ms, Unveil at 100 ms, PyProf, eACGM, and Nsight Systems. Blue bars are normalized runtime and stay near one for Prodigy, LDMS, and Unveil, while the profilers are higher. Hatched bars are the log-scale telemetry rate, lowest for Prodigy and Unveil and highest for eACGM.
Figure 7. CPU overhead of Unveil at different sampling intervals, compared with perf stat at 100 ms ( subsubsection 5.2.2 ). Box plots of CPU utilization for Unveil at sampling intervals from 100 ms to 600 ms, next to perf stat at 100 ms. The median falls from about 1.3 percent at 100 ms toward about 0.5 percent at longer intervals, close to perf stat. A few samples exceed 2 percent.
Figure 8. NVTX-annotated execution timeline. Detected inefficiencies (yellow) rarely coincide with lifecycle transition boundaries ( subsubsection 5.3.1 ). Timeline of GPU utilization and NVTX phases, including tokenizer load, dataset load, model load, supervised fine-tuning start, training steps, and forward or backward. Yellow bands mark detected inefficiency windows. Those bands rarely line up with the vertical lines at phase changes.
Configuration
Signature Hit Rate
Signature Coverage
Detection– Signature Agreement
Healthy Contamination
Z∨IF∨Maha
6.1%
81.3%
11.2%
3.3%
Z∨IF
6.1%
81.3%
11.2%
3.3%
Z∨Maha
6.2%
81.3%
11.2%
3.2%
IF∨Maha
37.2%
51.2%
40.6%
1.2%
Z
6.2%
81.3%
11.2%
3.2%
IF
41.6%
39.5%
37.8%
2.3%
Table 2. Detection and fingerprint matching performance between different combinations. Z = Z-score, IF = Isolation Forest, and Maha = Mahalanobis distance ( subsubsection 5.3.2 ).
Figure 9. Detected anomalies (red markers) and knowledge-base fingerprint matches (black-ringed markers) over time ( subsubsection 5.3.2 ). Four anomaly-score traces over 285 seconds for NUMA misconfiguration, IRQ locality misses, GPU power throttling, and reduced huge-page coverage. Colored circles are windows matched to a knowledge-base fingerprint. Red circles are detected but unmatched. One huge-page window is also marked as a GPU-power fingerprint.
The training and deployment of machine learning (ML) models have become extremely energy-intensive. While existing optimization efforts focus primarily on hardware energy efficiency, a significant but overlooked source of inefficiency is software energy waste caused by poor software design. This often includes redundant or poorly designed operations that consume more energy without improving performance. These inefficiencies arise in widely used ML frameworks and applications, yet developers often lack the visibility and tools to detect and diagnose them. We propose differential energy debugging, a novel approach that leverages the observation that competing ML systems often implement similar functionality with vastly different energy consumption. Building on this insight, we design and implement Magneton, an energy profiler that compares energy consumption between similar ML systems at the operator level and automatically pinpoints code regions and configuration choices responsible for excessive energy use. Applied to 9 popular ML systems spanning LLM inference, general ML frameworks, and image generation, Magneton detects and diagnoses 16 known cases of software energy inefficiency and further discovers 8 previously unknown cases, 7 of which have been confirmed by developers.
Yi Pan, Wenbo Qian, Dedong Xie +3
University of Washington · Shanghai Jiao Tong University · Boston University
We present Overall FLOP Utilization (OFU), a hardware-level, precision-agnostic GPU efficiency metric for AI workloads on HPC systems, derived from two on-chip performance counters: Tensor Pipe Activity and SM clock frequency. OFU requires no application instrumentation and works across GPU generations and numeric precisions. We characterize five properties of the OFU approximation -- tile quantization, floating-point precision scaling, clock sampling noise, Tensor Core clock domains, and non-tensor undercounting -- through controlled GEMM experiments on H100 and GB200 across FP16, TF32, FP8, and NVFP4. After tile-quantization correction, OFU predicts application-level MFU to within <=2 percentage points. Against 608 production training jobs, OFU achieves r = 0.78 correlation with application-level MFU and surfaces two framework-level FLOPs miscalculations. Deployed across large-scale GPU fleets, OFU has detected a 2.5x efficiency regression and tracked precision-dependent utilization changes in mixed-precision pretraining. Our evaluation and operational experience suggest OFU is a practical, deployment-ready complement to application-level MFU for continuous fleet-wide efficiency monitoring.
The increasing demand for GPU-accelerated machine learning inference has exposed limitations in centralized GPU infrastructures, including high cost, limited elasticity, and long wait times under resource constraints. At the same time, modern Apple Silicon-based macOS devices are equipped with powerful GPUs that remain largely idle during normal usage. This paper presents Minions, a dynamic and distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to execute large-scale machine learning inference workloads. Minions enables teams to leverage existing hardware without requiring additional GPU procurement. The system provides flexible GPU inference task execution, fault tolerance, and efficient resource utilization. Our experience shows that idle macOS devices can be transformed into a practical distributed GPU inference platform in environments where centralized GPU resources are constrained.