Seeing is believing: The case for hardware-centric inefficiency detection and remediation

Ziji Chen; Steven W. D. Chien; Peng Qian; Noa Zilberman

cs.PFOct 29, 2025

Seeing is believing: The case for hardware-centric inefficiency detection and remediation

Authors: Ziji Chen, Steven W. D. Chien, Peng Qian, Noa Zilberman

Organizations: University of Oxford

Abstract

Large-scale machine learning workloads on the cloud suffer from silent inefficiencies, which do not trigger failures or obvious anomalies, making them hard to detect and diagnose. Existing diagnosis systems rely on guest-side instrumentation or dedicated healthy baselines. These assumptions do not hold in modern production cloud environments, where operators cannot observe inside the guest, and a healthy baseline for each setup is impractical because of system diversity. We present Unveil, a hardware-centric observability framework for detecting, identifying, and remediating silent inefficiencies using only host-visible telemetry. Unveil requires neither guest instrumentation nor application modification. It uses unsupervised learning models to detect inefficiencies and identifies known ones through a novel fingerprinting algorithm. An extensible knowledge base containing curated remediation scripts is used to correct problems without interrupting workloads when possible. We evaluate Unveil on distributed machine learning workloads, such as large language model serving, introducing less than 2% monitoring overhead. It successfully recovered from a 7% throughput loss and reduced inference latency by up to 43.48%. These results demonstrate that host-visible telemetry is sufficient for practical and continuous diagnosis of production machine learning systems.

Figures & tables

Tool	VM-compat.	Cont. Deploy	No Code $\triangle$	HW Indep.	No-Base.	Scope
Plumber ( Kuchnik et al., 2022 )	✗		✗	✓	✓ (A)	data loading
SuperBench ( Xiong et al., 2024 )	✓	✗	✓	✓	✗ (H)	component
Aegis ( Dong et al., 2025 )	✗	✓	✓	✓	(P)	compute or communication
Greyhound ( Wu et al., 2025 )	✗	✓	✓	✓	(S+P)	compute or communication
Pulse ( Xiao et al., 2026 )	✓	✓	✓	✗	(S+P)	compute or communication
Mycroft ( Deng et al., 2025 )	✗	✓	✓	✓	(P)	Process rank + RDMA or GPU (communication only)

Table 1. Comparison with existing monitoring and diagnosis tools.

Figure 1. GPU–NIC NUMA placement ( subsection 2.6 ). Diagram of one host with two NUMA nodes. NUMA 0 keeps the NCCL host buffer beside the Ethernet NIC. NUMA 1 places that buffer on the other node, so traffic crosses NUMA before reaching the NIC and the other hosts.

Figure 2. Comparison of Prodigy, eACGM, and Unveil (Our work) under GPU–NIC NUMA misalignment ( subsection 2.6 ). Three panels for GPU–NIC NUMA misalignment. Prodigy reconstruction-error distributions for healthy and inefficient runs overlap, and many healthy samples sit above the alert threshold. eACGM log-likelihood distributions for the two runs are nearly the same. Unveil records fingerprint hits on inefficient runs and none on the healthy run.

Figure 3. Distribution of fingerprints at the subsystem-level for four silent inefficiency types (panels (a)–(d), red). Red bars denote fingerprints observed under the injected inefficiencies, while green bars denote fingerprints from anomalous windows detected during healthy executions ("False Positives") ( subsubsection 3.4.2 ). Four bar charts of fingerprint share across CPU, GPU, memory, interconnect, network, and storage. Panels are NUMA misplacement, IRQ locality misses, reduced huge-page coverage, and GPU power throttling. Red bars are injected inefficiencies and green bars are false positives from healthy runs. The red bars concentrate on different subsystems from the green bars.

Figure 4. Impact of GPU power limits on prefill throughput and decode bandwidth ( subsubsection 3.5.1 ). Line charts of LLaMA and Qwen feed-forward and attention layers as the GPU power limit falls from 350 W to 100 W. Prefill throughput drops earlier and more steeply than decode bandwidth. LLaMA feed-forward throughput declines almost linearly, while the other series stay flat until a sharp drop at the lowest power limits.

Figure 5. Hardware-centric observability and remediation workflow in Unveil ( section 4 ). Flowchart of Unveil on one host. Monitoring agents collect telemetry from CPUs, GPUs, memory, disks, and NICs while containerized or bare-metal workloads run. Each window is turned into features, tested for an anomaly, and attributed to the top contributing factors. The result is reported and can be remediated automatically or manually.

Figure 6. Runtime overhead (left axis) and telemetry data rate (right axis) across relevant tools ( subsubsection 5.2.1 ). Grouped bars for Prodigy at 1 second, LDMS at 100 ms, Unveil at 100 ms, PyProf, eACGM, and Nsight Systems. Blue bars are normalized runtime and stay near one for Prodigy, LDMS, and Unveil, while the profilers are higher. Hatched bars are the log-scale telemetry rate, lowest for Prodigy and Unveil and highest for eACGM.

Figure 7. CPU overhead of Unveil at different sampling intervals, compared with perf stat at 100 ms ( subsubsection 5.2.2 ). Box plots of CPU utilization for Unveil at sampling intervals from 100 ms to 600 ms, next to perf stat at 100 ms. The median falls from about 1.3 percent at 100 ms toward about 0.5 percent at longer intervals, close to perf stat. A few samples exceed 2 percent.

Figure 8. NVTX-annotated execution timeline. Detected inefficiencies (yellow) rarely coincide with lifecycle transition boundaries ( subsubsection 5.3.1 ). Timeline of GPU utilization and NVTX phases, including tokenizer load, dataset load, model load, supervised fine-tuning start, training steps, and forward or backward. Yellow bands mark detected inefficiency windows. Those bands rarely line up with the vertical lines at phase changes.

Configuration	Signature Hit Rate	Signature Coverage	Detection– Signature Agreement	Healthy Contamination
$Z\lor IF\lor Maha$	6.1%	81.3%	11.2%	3.3%
$Z\lor IF$	6.1%	81.3%	11.2%	3.3%
$Z\lor Maha$	6.2%	81.3%	11.2%	3.2%
$IF\lor Maha$	37.2%	51.2%	40.6%	1.2%
$Z$	6.2%	81.3%	11.2%	3.2%
$IF$	41.6%	39.5%	37.8%	2.3%

Table 2. Detection and fingerprint matching performance between different combinations. Z = Z-score, IF = Isolation Forest, and Maha = Mahalanobis distance ( subsubsection 5.3.2 ).

Figure 9. Detected anomalies (red markers) and knowledge-base fingerprint matches (black-ringed markers) over time ( subsubsection 5.3.2 ). Four anomaly-score traces over 285 seconds for NUMA misconfiguration, IRQ locality misses, GPU power throttling, and reduced huge-page coverage. Colored circles are windows matched to a knowledge-base fingerprint. Red circles are detected but unmatched. One huge-page window is also marked as a GPU-power fingerprint.

Cloud Computing

Cloud computing studies on-demand delivery and management of compute, storage, and services across distributed infrastructure.

See all 48 papers

Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging
Dec 9, 2025Yi Pan, Wenbo Qian, Dedong Xie +3Energy-Efficient ML
University of Washington · Shanghai Jiao Tong University · Boston University
Instant GPU Efficiency Visibility at Fleet Scale
May 20, 2026Connor Pedersen, Dong H. Ahn, Michel Migdal +2GPU Acceleration High-Performance Computing
NVIDIA
Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices
Oct 9, 2026Jianhua Li, Yun Xing, Girisha De Silva +3Distributed Inference GPU Acceleration
Apple USA

Dec 9, 2025J/K move · Enter open · S save

Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging

Yi Pan, Wenbo Qian, Dedong Xie, Ruiyan Hu+2

University of Washington · Shanghai Jiao Tong University · Boston University

Energy-Efficient ML

Open paper AskPDF arXiv

The training and deployment of machine learning (ML) models have become extremely energy-intensive. While existing optimization efforts focus primarily on hardware energy efficiency, a significant but overlooked source of inefficiency is software energy waste caused by poor software design. This often includes redundant or poorly designed operations that consume more energy without improving performance. These inefficiencies arise in widely used ML frameworks and applications, yet developers often lack the visibility and tools to detect and diagnose them. We propose differential energy debugging, a novel approach that leverages the observation that competing ML systems often implement similar functionality with vastly different energy consumption. Building on this insight, we design and implement Magneton, an energy profiler that compares energy consumption between similar ML systems at the operator level and automatically pinpoints code regions and configuration choices responsible for excessive energy use. Applied to 9 popular ML systems spanning LLM inference, general ML frameworks, and image generation, Magneton detects and diagnoses 16 known cases of software energy inefficiency and further discovers 8 previously unknown cases, 7 of which have been confirmed by developers.

Abstract

Cloud Computing

Explore similar work

Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging

Instant GPU Efficiency Visibility at Fleet Scale

Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices