IOSage: Detecting HPC I/O Bottleneck Patterns to Guide LLM Recommendations

Mahdi Banisharif; Tabassum Mahmud; Ahmad Tarraf; Suren Byna; Ali Jannesari

cs.DCOct 26, 2026

IOSage: Detecting HPC I/O Bottleneck Patterns to Guide LLM Recommendations

Authors: Mahdi Banisharif, Tabassum Mahmud, Ahmad Tarraf, Suren Byna, Ali Jannesari

Abstract

HPC applications often leave much of their parallel file system's bandwidth unused, and turning their Darshan logs into concrete I/O fixes still requires a scarce I/O expert. Rule-based tools flag counters against thresholds set by experts, and in tools built on large language models (LLMs), such as ION and IOAgent, the LLM decides which bottlenecks a log contains. We present IOSage, in which a classifier trained on many labeled logs, not the LLM, makes that decision, since a single log gives an LLM no baseline for what is abnormal. The LLM receives the I/O bottleneck patterns the classifier detects, the job's configuration files, per-file Darshan evidence, and for each pattern a card built from benchmark training runs, and writes recommended changes. Applied by hand and measured in repeated paired runs, the changes IOSage names speed up median total wall time 4.38x for an openPMD kernel (4.02x on a second storage tier) and 1.42x for hourly WRF restarts. For 5 of the 6 application scenarios with a helpful change, IOSage with Claude Sonnet 4 names it, including the change for 10-minute WRF restarts (up to 1.93x faster); with GPT-4o and Llama 3.1 as well, the classifier and cards name more helpful changes than the LLM alone. On 78 benchmark test runs without a pattern, the LLM deciding alone proposes changes for 64; with the classifier, for 9. The classifier reaches 0.929 Micro-F1 on 488 labeled benchmark test runs (0.916 as the gate; 0.832 on the 158 from configurations and jobs outside training), above Drishti's rules, WisIO, and IOAgent's pipeline without its literature retrieval. On TraceBench's nine expert-labeled traces from production systems, it has the highest precision of the tools compared (0.82), with 2 false positives against 12 for the keyword-mapped LLM diagnoses shipped with TraceBench and 5 for Drishti, on the five issue labels that match registered patterns.

Figures & tables

Fig. 1: IOSage architecture. (a) Preparation: configuration-labeled benchmark runs and rule-labeled Polaris logs train one classifier per pattern; the training runs yield one pattern card per pattern (typical counter values and a change template). (b) Per-log analysis: the LLM’s prompt holds the detected patterns with their cards, the log’s summary and per-file evidence, and the job’s configuration files. (c) Validation: a recommended change is applied by hand and its wall-time effect measured. Grey: inputs and outputs; orange: computation; blue: stored products; dashed arrows: products of (a) used in (b).

Fig. 2: Share of rank time in recorded I/O calls for the 131,151 cleaned Polaris logs: POSIX and STDIO read, write, and metadata time summed over processes, divided by processes times run time. Values above 1 occur when calls overlap (397 logs).

Pattern	Behavior	Inducing configurations	Card change template	When the change does not apply
access granularity	many small reads or writes	IOR with 64 B to 16 KiB transfers; DLIO with small records; h5bench with small blocks; HACC-IO with few particles per rank	increase the transfer size to at least 1 MB; aggregate small writes; use collective I/O	the small requests carry a negligible share of the I/O time
metadata intensity	metadata time dominates	mdtest with 1,000 to 50,000 creates, cross-node stat, deep trees	reduce the file count; batch metadata operations; use shared files for small data	the files are required outputs, such as per-step files read by other tools
parallelism efficiency	unequal I/O across ranks	mpi4py programs in which rank 0 writes 2 to 20 times more	balance I/O across ranks; use collective I/O	one rank owns the data by design, such as configuration or log output
access pattern	nonsequential offsets	IOR random offsets on a shared file; DLIO with shuffled files	sort offsets; access sequentially; enable data sieving	the data layout fixes the order, or sieving would read far more than needed
interface choice	shared-file I/O without collective MPI-IO	IOR POSIX on a shared file; MPI-IO independent with collective buffering off; h5bench independent data	use collective calls ( MPI_File_write_all ); enable collective buffering	ranks write large disjoint blocks, where independent I/O performs as well
file strategy	too many files	IOR file per process with 64 to 256 ranks; mdtest with 100 to 5,000 files per rank; HACC-IO file per process	use shared files with collective MPI-IO, or HDF5 or ADIOS2 [ 19 ]	downstream tools need one file per process

TABLE I: Registered bottleneck patterns, the benchmark configurations we ran to induce each one, the change template of its pattern card, and when the change does not apply

Fig. 3: Labels of the 689 configuration-labeled benchmark runs, by benchmark; a run can carry several labels.

Application	Change	Wall time (s)		Speedup	Write
and scenario		without	with	median [3rd, 9th]	step
Faster: interval above 1
openPMD kernel	HDF5 collective	76.0	16.6	4.38 [2.41, 5.35]	4.94
on flash tier		66.1	16.1	4.02 [3.68, 4.80]	5.32
WaComM++	no input copy	59.1	30.3	1.95 [1.47, 2.62]	n/a
WRF restart, hourly	PnetCDF restart	48.4	33.9	1.42 [1.37, 1.50]	2.09

TABLE II: Measured changes on real applications. Wall time: median of 11 runs. Speedup: median of the 11 pair ratios, with its 3rd to 9th ratio as a 93.5% interval (Section III-F )

Setting	Claude	GPT-4o	Llama
IOSage	5-6 ( 4 )	4-5 ( 4 )	3-4 ( 3 )
no cards	3 (2)	3 (2)	2 (1)
no classifier	4 (3)	3 (2)	2 (2)
LLM alone	2 (1)	2 (1)	1 (0)

TABLE III: Helpful changes named on the 6 application scenarios (ranked first in parentheses), by setting and LLM; one run per cell, and the IOSage row gives the range over four runs

Other learners, same data
Method	Micro-F1	Macro-F1	Hamming	Subset
IOSage classifier	0.929 $\pm$ 0.003	0.929 $\pm$ 0.004	0.019 $\pm$ 0.001	0.878 $\pm$ 0.006
no production logs	0.909 $\pm$ 0.001	0.904 $\pm$ 0.001	0.026 $\pm$ <0.001	0.852 $\pm$ 0.003
Random forest	0.916 $\pm$ 0.003	0.917 $\pm$ 0.005	0.023 $\pm$ 0.001	0.850 $\pm$ 0.008
MLP	0.881 $\pm$ 0.004	0.884 $\pm$ 0.006	0.033 $\pm$ 0.001	0.814 $\pm$ 0.005
Drishti rules	0.364	0.285	0.227	0.283

TABLE IV: Detection on the benchmark test runs: all eight label models at a threshold of 0.5 (the gate, seven pattern models at

\tau=0.3

, scores 0.916)

Fig. 5: Mean absolute SHAP value of the top counter features per pattern on the 488 test runs (TreeSHAP on the reported classifier); each bar segment is one pattern.

High-Performance Computing

High-performance computing (HPC) uses parallel systems and optimized software to solve computationally demanding scientific and engineering problems.

See all 147 papers

Apr 6, 2026

Instruction-Tuned LLMs for Parsing and Mining Unstructured Logs on Leadership HPC Systems

Leadership-class HPC systems generate massive volumes of heterogeneous, largely unstructured system logs. Because these logs originate from diverse software, hardware, and runtime layers, they exhibit inconsistent formats, making structure extraction and pattern discovery extremely challenging. Therefore, robust log parsing and mining is critical to transform this raw telemetry into actionable insights that reveal operational patterns, diagnose anomalies, and enable reliable, efficient, and scalable system analysis. Recent advances in large language models (LLMs) offer a promising new direction for automated log understanding in leadership-class HPC environments. To capitalize on this opportunity, we present a domain-adapted, instruction-following, LLM-driven framework that leverages chain-of-thought (CoT) reasoning to parse and structure HPC logs with high fidelity. Our approach combines domain-specific log-template data with instruction-tuned examples to fine-tune an 8B-parameter LLaMA model tailored for HPC log analysis. We develop a hybrid fine-tuning methodology that adapts a general-purpose LLM to domain-specific log data, enabling privacy-preserving, locally deployable, fast, and energy-efficient log-mining approach. We conduct experiments on a diverse set of log datasets from the LogHub repository. The evaluation confirms that our approach achieves parsing accuracy on par with significantly larger models, such as LLaMA 70B and Anthropic's Claude. We further validate the practical utility of our fine-tuned LLM model by parsing over 600 million production logs from the Frontier supercomputer over a four-week window, uncovering critical patterns in temporal dynamics, node-level anomalies, and workload-error log correlations.

Ahmad Maroof Karimi, Jong Youl Choi, Charles Qing Cao +1

Oak Ridge National Laboratory, Oak Ridge, USA · University of Tennessee, Knoxville, USA

Document Parsing LLM Information Extraction LLM Fine-Tuning High-Performance Computing

Ask PDF

Feb 23, 2026

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.

Ayush Nangia, Shikhar Mishra, Aman Gokrani +1

Coding Agent Benchmarks AI Coding Agents AI Agent Benchmarks Code Generation Evaluation Code Optimization LLM Inference Efficiency+1

Ask PDF

Jun 26, 2026

KernelSight-LM: A Kernel-Level LLM Inference Simulator

As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets. However, the end-to-end behavior of LLMs couples serving-layer policies with low-level GPU kernel execution and rapidly evolving architectures, forcing slow, deployment-specific benchmarking that is hard to generalize. We present KernelSight-LM, a fine-grained inference simulator that models token-level execution and produces kernel-level latency breakdowns. It decomposes each serving step into a roofline kernel model with a learned efficiency term, a communication model, and a host-overhead model, composed through a discrete-event scheduler that also captures mechanisms like prefix caching and continuous batching. KernelSight-LM offers two prediction tiers that trade target-GPU data for accuracy. The cross-generation tier uses no target-GPU measurements, only hardware specifications and kernel microbenchmarks from previously profiled GPUs, and predicts per-kernel latency on an unseen GPU generation to 12.1% error, a 1.8x improvement over the roofline baseline (22.0%). A second target-measured tier adds one model-agnostic kernel-microbenchmark sweep on the target GPU, sharpening per-kernel error to 3.8%, a 7.3x improvement over a comparable baseline (27.7%). Both tiers require far less target-GPU data than the prior systems they extend. In our simulator, these predictions yield end-to-end median (p50) errors across six model families of 15.4%, 12.8%, and 3.0% (TTFT, TPOT, throughput) in the cross-generation tier and 14.3%, 6.2%, and 2.7% in the target-measured tier, matching dedicated profiling tools while collecting far less on-device data. Beyond prediction, its kernel-level bottleneck breakdowns support hardware/software co-design and capacity planning.

Xiteng Yao, Taeho Kim, Hengzhi Pei +7

Amazon Web Services Santa Clara, CA, USA · Boston University Boston, MA, USA

LLM Inference LLM Inference Serving

Ask PDF

Abstract

High-Performance Computing

Explore similar work

Instruction-Tuned LLMs for Parsing and Mining Unstructured Logs on Leadership HPC Systems

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

KernelSight-LM: A Kernel-Level LLM Inference Simulator