cs.DCOct 19, 2026

SafeWall: A Risk-Aware Job Walltime Recommendation for HPC Scheduling

Authors: Kanglin Xu, Michael Papka, Zhiling Lan

Organizations: University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In high-performance computing (HPC), users must submit resource requirements for their jobs, with user-supplied walltime being a critical parameter. However, these estimates are notoriously inaccurate, as users tend to overestimate walltime to prevent the scheduler from terminating their jobs prematurely. While existing work on learning-based walltime prediction primarily focuses on maximizing prediction accuracy, such predictions are inherently risky for walltime recommendations. Specifically, underestimation leads to job termination before completion, resulting in lost progress and wasted resources. In this study, we present SafeWall, a novel recommendation method that adjusts walltime predictions by incorporating an explicit, nonnegative safety buffer to mitigate underestimation risk. The core of our approach is a risk-aware buffer that dynamically calibrates safety margins based on individual user's historical behavior. When user data is limited, the system employs statistical shrinkage to align these margins with a global baseline, ensuring robustness. This yields a lightweight mechanism that optimizes the trade-off between prediction accuracy and termination risk. SafeWall is modular and applicable to various learning-based prediction models. We evaluate SafeWall using five public workload traces, including the full-lifetime job log from the 4,392-node Theta machine at the Argonne Leadership Computing Facility. Across all traces, SafeWall substantially reduces underestimation risk compared to the unbuffered base predictors, while preserving competitive accuracy. The source code is publicly available at https://github.com/SPEAR-UIC/JSSPP26-SafeWall.

Figures & tables

Figure 1: SafeWall overview. The pipeline consists of a base predictor that estimates job runtime and a risk-aware buffer adapter that adjusts runtime estimation based on user-specific historic information.
SWF FieldTheta Log Field
Submit timequeued timestamp normalized to seconds since trace start
Wait timeELIGIBLE_WAIT_SECONDS
Run timeRUNTIME_SECONDS
Requested time (walltime)WALLTIME_SECONDS
Allocated nodesNODES_USED
Exit statusEXIT_STATUS
Table 1: Mapping from Theta log fields to Standard Workload Format (SWF) fields. SWF fields not available in the log are set to −1-1 .
Figure 2: Illustrative comparison of predicted versus actual job runtimes on a log scale.
Figure 3: Joint distribution of job runtime and resource scale for the 6.5-year Theta workload. Each cell represents the frequency of completed jobs within specific runtime (rows) and node-count (columns) bins; color intensity is capped at 1,000 jobs for visual clarity.
MethodUR (%)Accuracy (%)RMSE(s)MAE(s)R2R^{2}
XGBoost47.848997.20961.4457.340.9835
XGBoost + SafeWall4.305190.27963.8991.810.9834
Quantile GBR5.556885.44685.90159.200.9916
Quantile GBR + SafeWall3.461487.70770.25147.550.9895
Percentage buffer1.061382.831808.66547.430.9418
Fixed-value buffer7.052189.91961.8577.540.9835
Table 2: Comprehensive comparison of buffering strategies on the Theta log.
TracePeriodJobsUsersNodesCens. (%)
ThetaJun 2017–Dec 2023540,5031,8274,3920.1
CoriJan–Dec 20222,462,2791,31810,0000.0
KIT-FH2Jun 2016–Jan 2018114,0241641,1736.6
MetaCentrumJan–Dec 202310,139,1981,61847 clust.4.9
CERIT-SC-K8sJul 2021–Jun 202264,093103200.0
Table 3: Workload traces used for cross-system evaluation. “Cens.” denotes the fraction of censored jobs whose true runtimes are not observed in the trace (e.g., terminated at the walltime limit or otherwise missing a completion time).
  XGBoostSafeWall
Trace#JobsUR (%)Acc (%)UR (%)Acc (%)
Cori2,465,28647.9293.475.0683.21
KIT114,35549.2390.136.7974.76
CERIT64,26849.1192.145.0083.02
MetaCentrum10,145,23148.1684.034.9268.64
Table 4: Runtime prediction across traces: base model and SafeWall.
Figure 4: Joint distribution of job runtime and job size across cross-system traces. Each cell shows the number of completed jobs in a runtime bin (rows) and size bin (columns); color intensity is capped for visual clarity.
More in this topic

High-Performance Computing

High-performance computing (HPC) uses parallel systems and optimized software to solve computationally demanding scientific and engineering problems.

See all 143 papers

Explore similar work

CardsList
Mar 7, 2026

Duration-Informed Workload Scheduler

High-performance computing systems are complex machines whose behaviour is governed by the correct functioning of its many subsystems. Among these, the workload scheduler has a crucial impact on the timely execution of the jobs continuously submitted to the computing resources. Making high-quality scheduling decisions is contingent on knowing the duration of submitted jobs before their execution--a non-trivial task for users that can be tackled with Machine Learning. In this work, we devise a workload scheduler enhanced with a duration prediction module built via Machine Learning. We evaluate its effectiveness and show its performance using workload traces from a Tier-0 supercomputer, demonstrating a decrease in mean waiting time across all jobs of around 11%. Lower waiting times are directly connected to better quality of service from the users' point of view and higher turnaround from the system's perspective.

Daniela Loreti, Davide Leone, Andrea Borghesi

DISI, University of Bologna, Bologna, Viale Risorgimento 2, Italy

High-Performance Computing
AskPDF
Sep 12, 2026

Leakage-Safe and Scheduler-Aware Machine Learning for Grid Job Runtime Prediction

Accurate job runtime prediction can improve scheduling-aware resource management in grid and distributed computing environments, but prediction models must be evaluated under realistic deployment constraints. This paper revisits CPU burst time prediction on the GWA-T-4 AuverGrid workload trace and reformulates it as leakage-safe pre-execution job runtime prediction. We define the target as job-level runtime, use only submission-time attributes, exclude post-execution variables, and evaluate models under temporal and cold-start settings rather than relying only on random cross-validation. We compare standard regressors, chronological historical baselines, categorical encoding strategies, and CatBoost with native categorical handling. We further add temporal hyperparameter tuning, runtime predictability analysis, feature ablation, error analysis by job length, and a minimal scheduling simulation. After temporal-validation tuning, CatBoost achieves the strongest deployment-oriented result with R^2=0.239, MAE=27,019, RMSE=46,587, and LogMAE=2.646 on the held-out temporal test set. A single-server simulation over all 69,523 held-out temporal test jobs shows that prediction-informed SJF reduces average waiting time by 50.92% relative to FCFS. The results show that random-split evaluation overestimates performance, categorical-native boosting improves temporal generalization, and long-job underestimation remains a scheduler-relevant challenge.

Ashfaq Ali Shafin, Khandaker Mamun Ahmed

Augustana College, USA · Florida International University, USA · Dakota State University, USA

High-Performance Computing
AskPDF
Oct 14, 2026

Deep Reinforcement Learning for HPC Job Scheduling: A Statistical Evaluation

Deep Reinforcement Learning (DRL) offers learned job selection for High-Performance Computing (HPC), but performance against heuristics alone does not establish value beyond action masking. Six DRL treatments are compared: Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), and Deep Q-Network (DQN), each with and without masking. Three heuristics and a seed-paired masked uniform random control provide baselines across ten seeds on two real Slurm (Simple Linux Utility for Resource Management) workload traces, one homogeneous and one heterogeneous, under a common non-parametric framework. All masked treatments complete evaluation, whereas unmasked treatments are truncated. On the homogeneous trace, no masked DRL treatment significantly improves average waiting time or average and maximum bounded slowdown over the control. On the heterogeneous trace, MaskablePPO improves all three metrics relative to the control. Its average waiting time is equivalent to that of the three heuristics within a 10% margin, and its maximum bounded slowdown is 22% lower than that of Last-Come, First-Served (LCFS). MaskablePPO has the leading average waiting-time rank among complete treatments on both traces, but Nemenyi comparisons detect no significant difference from MaskableA2C. Among masked variants, MaskablePPO has the lowest recorded mean training time, while MaskableDQN has the lowest mean decision latency. These findings distinguish completion from baseline-specific scheduling gains and support including a masked random control when assessing the value of learned selection.

Justin M. Cheney, Clement N. Nyirenda

Computer Science Department, University of the Western Cape, Robert Sobukwe Rd, Bellville, Cape Town, 7535, South Africa · eResearch Office, University of the Western Cape, Bellville, Cape Town, South Africa

Reinforcement LearningHigh-Performance ComputingDeep Q-Learning
AskPDF