Duration-Informed Workload Scheduler

Daniela Loreti; Davide Leone; Andrea Borghesi

cs.DCMar 7, 2026

Duration-Informed Workload Scheduler

Authors: Daniela Loreti, Davide Leone, Andrea Borghesi

Organizations: DISI, University of Bologna, Bologna, Viale Risorgimento 2, Italy

Abstract

High-performance computing systems are complex machines whose behaviour is governed by the correct functioning of its many subsystems. Among these, the workload scheduler has a crucial impact on the timely execution of the jobs continuously submitted to the computing resources. Making high-quality scheduling decisions is contingent on knowing the duration of submitted jobs before their execution--a non-trivial task for users that can be tackled with Machine Learning. In this work, we devise a workload scheduler enhanced with a duration prediction module built via Machine Learning. We evaluate its effectiveness and show its performance using workload traces from a Tier-0 supercomputer, demonstrating a decrease in mean waiting time across all jobs of around 11%. Lower waiting times are directly connected to better quality of service from the users' point of view and higher turnaround from the system's perspective.

Figures & tables

Figure 1: Histogram of the target variable run_time

Feature Name	Description
cpu	Number of CPU cores requested by the job.
mem (GB)	Amount of memory requested by the hob.
node	Number of nodes requested for the job.
gres/gpu	GPU resources requested by the job.
user_id	Identifier of the user submitting the job.
qos	Quality of Service level associated with the job.

Table 1: Brief description of the features.

	CPU	mem(GB)	nodes	GRES/GPU	user_id	QoS	time_limit	run_time
mean	121.379	236.068	1.693	5.630	110.895	0.051	1038.069	43.433
std	246.657	1008.594	6.961	27.927	118.594	0.368	506.318	168.719
min	1.000	0.098	1.000	1.000	0.000	0.000	1.000	0.017
25%	4.000	7.813	1.000	1.000	2.000	0.000	720.000	0.017
50%	80.000	230.000	1.000	4.000	93.000	0.000	1440.000	0.83
75%	128.000	237.5000	1.000	4.000	191.000	0.000	1440.000	22.700

Table 2: Brief statistical description of the dataset.

OVERESTIMATION
	Decision Tree	Random Forest	Gradient Boosting	Neural Network
MAE	23.51	23.53	40.11	21.95
MSE	8001.99	7968.58	13060.00	9202.41
RMSE	89.45	89.27	114.28	95.93
$R^{2}$	0.72	0.72	0.54	0.68
Confidence interval (95%)	[0.00, 326.70]	[0.00, 325.60]	[0.00, 227.48]	[0.00, 344.92]

Table 3: Performance of the four considered ML models

OVERESTIMATION
	Decision Tree	Random Forest	Gradient Boosting	Neural Network
MAE	22.24	22.26	26.01	20.53
MSE	7312.82	7275.61	8406.57	8623.19
RMSE	85.52	85.30	91.69	92.86
$R^{2}$	0.71	0.72	0.67	0.66
Confidence interval (95%)	[0.00, 307.77]	[0.00, 306.92]	[0.00, 291.46]	[0.00, 319.62]

Table 4: Performance of the four considered ML models with data augmentation.

OVERESTIMATION
	Decision Tree	Random Forest	Gradient Boosting	Neural Network
MAE	8.33	8.35	22.90	8.22
MSE	3438.32	3432.47	5086.70	3674.63
RMSE	58.64	58.59	71.32	60.62
$R^{2}$	0.62	0.62	0.44	0.60
Confidence interval (95%)	[0.00, 155.35]	[152.11]	[0.00, 104.60]	[0.00, 165.33]

Table 5: Performance of the four considered ML models with a consecutive split of train and test sets.

	DIWS	EasyBF	Improvement
makespan	86272.0068	86272.0024	+0.00%
scheduling time	37.8449	240.7396	-84.28%
mean waiting time	846.5391	953.3813	-11.21%
mean turnaround time	2351.1828	2458.0250	-4.35%
mean slowdown	2.3089	45.8519	-94.96%
max waiting time	17003.0928	12608.0384	+34.88%

Table 6: Comparison of DIWS and EasyBF performance when scheduling jobs on a large HPC system ( Setup A ). Negative values in the “Improvement” column highlight desirable situations where DIWS brings a decrease in the corresponding metric.

	DIWS	EasyBF	Improvement
makespan	1029198.2116	1090869.2938	-5.99%
scheduling time	210.3460	194.5042	+7.53%
mean waiting time	127474.5570	163846.3107	-28.54%
mean turnaround time	128979.2008	165350.9545	-28.21%
mean slowdown	22785.2399	20097.1711	+11.80%
max waiting time	994211.2116	1026349.2598	-3.21%

Table 7: Comparison of DIWS and EasyBF performance when scheduling jobs on a constrained HPC system ( Setup B ).

Panel of Figure 2 — Figure 2: Scheduling performance comparison

High-Performance Computing

High-performance computing (HPC) uses parallel systems and optimized software to solve computationally demanding scientific and engineering problems.

See all 146 papers

Leakage-Safe and Scheduler-Aware Machine Learning for Grid Job Runtime Prediction
Sep 12, 2026Ashfaq Ali Shafin, Khandaker Mamun AhmedHigh-Performance Computing
Augustana College, USA · Florida International University, USA · Dakota State University, USA
Duration Aware Scheduling for ASR Serving Under Workload Drift
Mar 11, 2026Darshan Makwana, Yash Jogi, Harsh Kotta +1Real-Time Scheduling Automatic Speech Recognition
Sprinklr Gurugram, India
Hyperion: An AI-powered HPC cluster for sciences and humanities research that utilizes ML for predicting job turnaround time
Sep 14, 2026Jun Zhou, Nathan Elgar, Tawnee Benedetto +5High-Performance Computing
University of South Carolina Columbia, South Carolina, USA · University of South Carolina Columbia, USA · Boston University Boston, USA

Sep 12, 2026J/K move · Enter open · S save

Leakage-Safe and Scheduler-Aware Machine Learning for Grid Job Runtime Prediction

Ashfaq Ali Shafin, Khandaker Mamun Ahmed

Augustana College, USA · Florida International University, USA · Dakota State University, USA

High-Performance Computing

Open paper AskPDF arXiv

Accurate job runtime prediction can improve scheduling-aware resource management in grid and distributed computing environments, but prediction models must be evaluated under realistic deployment constraints. This paper revisits CPU burst time prediction on the GWA-T-4 AuverGrid workload trace and reformulates it as leakage-safe pre-execution job runtime prediction. We define the target as job-level runtime, use only submission-time attributes, exclude post-execution variables, and evaluate models under temporal and cold-start settings rather than relying only on random cross-validation. We compare standard regressors, chronological historical baselines, categorical encoding strategies, and CatBoost with native categorical handling. We further add temporal hyperparameter tuning, runtime predictability analysis, feature ablation, error analysis by job length, and a minimal scheduling simulation. After temporal-validation tuning, CatBoost achieves the strongest deployment-oriented result with R^2=0.239, MAE=27,019, RMSE=46,587, and LogMAE=2.646 on the held-out temporal test set. A single-server simulation over all 69,523 held-out temporal test jobs shows that prediction-informed SJF reduces average waiting time by 50.92% relative to FCFS. The results show that random-split evaluation overestimates performance, categorical-native boosting improves temporal generalization, and long-job underestimation remains a scheduler-relevant challenge.

Abstract

High-Performance Computing

Explore similar work

Leakage-Safe and Scheduler-Aware Machine Learning for Grid Job Runtime Prediction

Duration Aware Scheduling for ASR Serving Under Workload Drift

Hyperion: An AI-powered HPC cluster for sciences and humanities research that utilizes ML for predicting job turnaround time