Learning to Score: Tuning Cluster Schedulers through Reinforcement Learning

Martin Asenov; Qiwen Deng; Gingfung Yeung; Adam Barker

cs.LGMar 11, 2026

Learning to Score: Tuning Cluster Schedulers through Reinforcement Learning

Authors: Martin Asenov, Qiwen Deng, Gingfung Yeung, Adam Barker

Organizations: Edinburgh Research Centre, Central Software Institute, Huawei · School of Computer Science, University of St Andrews

Abstract

Efficiently allocating incoming jobs to nodes in large-scale clusters can lead to substantial improvements in both cluster utilization and job performance. In order to allocate incoming jobs, cluster schedulers usually rely on a set of scoring functions to rank feasible nodes. Results from individual scoring functions are usually weighted equally, which could lead to sub-optimal deployments as the one-size-fits-all solution does not take into account the characteristics of each workload. Tuning the weights of scoring functions, however, requires expert knowledge and is computationally expensive. This paper proposes a reinforcement learning approach for learning the weights in scheduler scoring algorithms with the overall objective of improving the end-to-end performance of jobs for a given cluster. Our approach is based on percentage improvement reward, frame-stacking, and limiting domain information. We propose a percentage improvement reward to address the objective of multi-step parameter tuning. The inclusion of frame-stacking allows for carrying information across an optimization experiment. Limiting domain information prevents overfitting and improves performance in unseen clusters and workloads. The policy is trained on different combinations of workloads and cluster setups. We demonstrate the proposed approach improves performance on average by 33% compared to fixed weights and 12% compared to the best-performing baseline in a lab-based serverless scenario.

Figures & tables

Fig. 1 : Filtering and scoring steps in a job scheduler. Assigning pods to nodes in a cluster job scheduler is typically a two-step process of filtering feasible nodes, followed by scoring functions. In this work, we focus on optimizing the relative weighting ( w 1 , w 2 , w 3 , … , w k ) of the different scoring functions in different cluster and workload scenarios, with the goal of optimizing a given metric.

Fig. 2 : Reinforcement learning for tuning weights of scoring functions. We pose the optimization of weights of scoring functions as a parameter tuning problem and propose a reinforcement learning based solution. In this work, we propose using percentage improvement reward 4 , encoding past samples information through the use of frame stacking or recurrent policies 5 , and limiting domain information to prevent overfitting 6 . We develop an extensive gym wrapper 2 , including the option for parallel environments 7 , and demonstrate the capability of our approach in an example FaaS benchmark scenario 1 .

Scoring Func.	Description
LeastAllocated	Favors nodes with the lowest utilization
MostAllocated	Favors nodes with the highest utilization
RTCRatio	Piecewise linear function of utilization
LocalityType	Tag for the type of machine, e.g. edge vs cloud
DataLocality	Estimated time to download necessary data
Capability	Tag for capability of the machine, e.g. GPU

TABLE I : Skippy Scheduler Scoring Functions. A total of eight scoring functions are used as part of scheduling. For fixed weights baseline, and initial weights selection for optimization algorithms, all weights are set to 1, except LeastAllocated and RTCRatio which are set to 0.

Fig. 3 : Different heterogeneous cluster configurations used for training and evaluation. Distributions of the types of machines used for benchmark experiments. Only cloud_cpu , cloud_gpu and edge_cloudlet cluster configurations are used during training. We use additional cluster configurations to evaluate how well the proposed approach is able to adapt to unseen machines‘ distributions.

Panel of Figure 5 — Fig. 4 : Example network configurations within the cluster setup. We use two types of cluster connectivity across benchmark experiments.

Configuration	Train environment	Test environment
Cluster setups	3	8
Workload functions	5	8
Requests per second	10	5-30
% of nodes to score	100	10-100
Min # nodes per func	1	1-10
Max # nodes per func	100	50-100

TABLE II : Cluster and workload configuration across experiments. To evaluate how well the proposed approach generalizes to novel scenarios, we test the trained agent on novel scenarios with different cluster setups, novel workload functions, and different scheduling options.

Fig. 5 : Tuning weights of scoring functions on similar cluster and workload configurations. Example results for six experiments visualized across two columns. For each experiment the following three characteristics are described (from left to right): best score (as defined in eq. 2 ) from the set of explored weights’ configurations; mean and standard deviation, best weights selection from the reinforcement learning algorithm; short description of the experiment. We compare the proposed approach against four baselines, including fixed weights (Fix), random search (RS), Bayesian Optimization (BO), and Tree-structured Parzen Estimator (TPE). In each experiment, the fixed weight configuration was used as an initial sample (same as Fix), followed by four optimization steps. A total of eight scoring functions were used.

Fig. 6 : Example results on novel cluster and workload configurations. Follows the same notation as Figure 5 . Results of unseen cluster and workload configurations as described in Table II

Panel of Figure 9 — Fig. 7 : Summary results for the proposed method. Mean of the best-achieved score (as defined in eq. 2 ) across ten experiments. A total of five samples of weights were used per method, with the exception of Fixed, which uses the default weights. The initial weights sample in every experiment is the same as Fixed. Configurations of the experiments in (a) and (b) are uniformly sampled from Table II .

Learning to Optimize

Learning to Optimize uses data to adapt algorithms, solver decisions, and solution maps for faster or better optimization.

See all 138 papers

When Does Learning Beat Heuristics? A Case Study in Kubernetes Scheduler Score Plugins
Aug 25, 2026Wang Xuying, Zhibek SarypbekovaLearning to Rank
I. Razzakov Kyrgyz State Technical University
MCRL2: Multi-resource Cross-attention-based Representation Learning-augmented Reinforcement Learning for Cloud Microservice Scheduling
Sep 11, 2026Tiangang Li, Shi Ying, Xiangbo Tian +2Cross-Attention Cloud Computing
School of Computer Science, Wuhan University, Wuhan 430072, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
SCALE: Scalable Cross-Attention Learning with Extrapolation for Agentic Workflow Scheduling
Jun 5, 2026Zhifei Xu, Jierui Lan, Zixuan Liang +2RL for LLM Agents
Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China

Aug 25, 2026J/K move · Enter open · S save

When Does Learning Beat Heuristics? A Case Study in Kubernetes Scheduler Score Plugins

Wang Xuying, Zhibek Sarypbekova

I. Razzakov Kyrgyz State Technical University

Learning to Rank

Open paper AskPDF arXiv

Kubernetes scheduler plugins that score candidate nodes are, in production, hand-tuned heuristics. We ask whether a learned scoring function - trained on real placement decisions from a production cluster trace - can match or exceed these heuristics, and if not, why. We implement an external, HTTP-backed scoring plugin for a widely used scheduler simulator and evaluate two learned models (a Random Forest over engineered features, and a graph neural network encoder over per-job task-dependency graphs) trained on a large-scale production cluster trace. Under standard regression fit (R^2), both models improve modestly but monotonically across four feature-engineering iterations, reaching R^2 of about 0.042. However, on the metric that actually matters for scheduling - Top-1 ranking accuracy, whether the model scores the machine the production scheduler actually chose highest - both learned models are outperformed by a trivial single-feature heuristic (rank by free CPU: 74-84% vs. 65-66% for either model). We show this gap is best explained by objective mismatch: both models were trained with pointwise regression (MSE) rather than a ranking-specific objective, echoing a long-standing distinction in the learning-to-rank literature. This parallels prior evidence that RL-trained schedulers can outperform heuristics when the training objective is aligned with the deployment task, suggesting objective misalignment, not architecture, is the primary obstacle here. We further report an ablation of the occupancy reconstruction required to make offline trace data usable (naive features yield R^2 near 0), a controlled comparison isolating feature richness and data volume between the two model families, and a sensitivity analysis of inference latency and serving-container memory constraints. Code, data pipelines, and experiment scripts are released for reproducibility.

Abstract

Learning to Optimize

Explore similar work

When Does Learning Beat Heuristics? A Case Study in Kubernetes Scheduler Score Plugins

MCRL2: Multi-resource Cross-attention-based Representation Learning-augmented Reinforcement Learning for Cloud Microservice Scheduling

SCALE: Scalable Cross-Attention Learning with Extrapolation for Agentic Workflow Scheduling