In high-performance computing (HPC), users must submit resource requirements for their jobs, with user-supplied walltime being a critical parameter. However, these estimates are notoriously inaccurate, as users tend to overestimate walltime to prevent the scheduler from terminating their jobs prematurely. While existing work on learning-based walltime prediction primarily focuses on maximizing prediction accuracy, such predictions are inherently risky for walltime recommendations. Specifically, underestimation leads to job termination before completion, resulting in lost progress and wasted resources. In this study, we present SafeWall, a novel recommendation method that adjusts walltime predictions by incorporating an explicit, nonnegative safety buffer to mitigate underestimation risk. The core of our approach is a risk-aware buffer that dynamically calibrates safety margins based on individual user's historical behavior. When user data is limited, the system employs statistical shrinkage to align these margins with a global baseline, ensuring robustness. This yields a lightweight mechanism that optimizes the trade-off between prediction accuracy and termination risk. SafeWall is modular and applicable to various learning-based prediction models. We evaluate SafeWall using five public workload traces, including the full-lifetime job log from the 4,392-node Theta machine at the Argonne Leadership Computing Facility. Across all traces, SafeWall substantially reduces underestimation risk compared to the unbuffered base predictors, while preserving competitive accuracy. The source code is publicly available at https://github.com/SPEAR-UIC/JSSPP26-SafeWall.
Figures & tables
Figure 1: SafeWall overview. The pipeline consists of a base predictor that estimates job runtime and a risk-aware buffer adapter that adjusts runtime estimation based on user-specific historic information.
SWF Field
Theta Log Field
Submit time
queued timestamp normalized to seconds since trace start
Wait time
ELIGIBLE_WAIT_SECONDS
Run time
RUNTIME_SECONDS
Requested time (walltime)
WALLTIME_SECONDS
Allocated nodes
NODES_USED
Exit status
EXIT_STATUS
Table 1: Mapping from Theta log fields to Standard Workload Format (SWF) fields. SWF fields not available in the log are set to −1 .
Figure 2: Illustrative comparison of predicted versus actual job runtimes on a log scale.
Figure 3: Joint distribution of job runtime and resource scale for the 6.5-year Theta workload. Each cell represents the frequency of completed jobs within specific runtime (rows) and node-count (columns) bins; color intensity is capped at 1,000 jobs for visual clarity.
Method
UR (%)
Accuracy (%)
RMSE(s)
MAE(s)
R2
XGBoost
47.8489
97.20
961.44
57.34
0.9835
XGBoost + SafeWall
4.3051
90.27
963.89
91.81
0.9834
Quantile GBR
5.5568
85.44
685.90
159.20
0.9916
Quantile GBR + SafeWall
3.4614
87.70
770.25
147.55
0.9895
Percentage buffer
1.0613
82.83
1808.66
547.43
0.9418
Fixed-value buffer
7.0521
89.91
961.85
77.54
0.9835
Table 2: Comprehensive comparison of buffering strategies on the Theta log.
Trace
Period
Jobs
Users
Nodes
Cens. (%)
Theta
Jun 2017–Dec 2023
540,503
1,827
4,392
0.1
Cori
Jan–Dec 2022
2,462,279
1,318
10,000
0.0
KIT-FH2
Jun 2016–Jan 2018
114,024
164
1,173
6.6
MetaCentrum
Jan–Dec 2023
10,139,198
1,618
47 clust.
4.9
CERIT-SC-K8s
Jul 2021–Jun 2022
64,093
103
20
0.0
Table 3: Workload traces used for cross-system evaluation. “Cens.” denotes the fraction of censored jobs whose true runtimes are not observed in the trace (e.g., terminated at the walltime limit or otherwise missing a completion time).
XGBoost
SafeWall
Trace
#Jobs
UR (%)
Acc (%)
UR (%)
Acc (%)
Cori
2,465,286
47.92
93.47
5.06
83.21
KIT
114,355
49.23
90.13
6.79
74.76
CERIT
64,268
49.11
92.14
5.00
83.02
MetaCentrum
10,145,231
48.16
84.03
4.92
68.64
Table 4: Runtime prediction across traces: base model and SafeWall.
Figure 4: Joint distribution of job runtime and job size across cross-system traces. Each cell shows the number of completed jobs in a runtime bin (rows) and size bin (columns); color intensity is capped for visual clarity.
Computer Science Department, University of the Western Cape, Robert Sobukwe Rd, Bellville, Cape Town, 7535, South Africa · eResearch Office, University of the Western Cape, Bellville, Cape Town, South Africa