PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problem

Mateo Toro Diz; Jonathan Hoss; Noah Klarmann

cs.LGSep 25, 2026

PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problem

Authors: Mateo Toro Diz, Jonathan Hoss, Noah Klarmann

Organizations: Department of Industrial Engineering Rosenheim University of Applied Sciences Rosenheim, Germany

Abstract

The Job Shop Scheduling Problem (JSSP) is a fundamental combinatorial optimization problem in industrial optimization. This work introduces Pretrained Offline Reinforcement Learning (PORL), a hybrid approach that combines simulation-based online pretraining with offline fine-tuning on production-specific data. Reinforcement learning through online interaction enables exploration of general scheduling strategies, but typically relies on simulation environments and may suffer from a simulation-to-reality gap. In contrast, offline RL avoids direct interaction with the environment by learning from historical data, but its performance is strongly influenced by dataset quality and coverage. PORL combines the strengths of both paradigms by first learning a general scheduling policy through online interaction and subsequently adapting it offline to a target distribution. A KL-divergence-based policy constraint is introduced to limit deviations from the pretrained policy during fine-tuning. The approach is evaluated on JSSP instances with distribution shift and datasets generated from heuristic, noisy-expert, and random behavioral policies. The results show that PORL consistently achieves lower optimality gaps than standalone offline RL and the considered general scheduling baselines. Furthermore, its advantage over standalone offline RL increases as dataset quality decreases, indicating reduced sensitivity to the quality and coverage of the available offline data. The results suggest that offline adaptation of pretrained policies is a promising approach for industrial scheduling environments where direct online exploration is impractical.

Figures & tables

Fig. 1: Training pipeline of (a) the offline Reinforcement Learning baseline and (b) the proposed Pretrained Offline Reinforcement Learning approach.

Parameter	Value
Input feature dimension	5
Hidden dimension	64
Number of Graph Isomorphism Network layers ( $K$ )	3

TABLE I: GNN architecture hyperparameters.

Hyperparameter	Value
Learning rate	$1.5\times 10^{-4}$
Discount factor ( $\gamma$ )	0.99
TD loss function	smooth_l1
CQL regularization ( $\alpha$ )	0.0
Training timesteps	250000
Replay buffer size	10000

TABLE II: Online DQN learning hyperparameters.

Hyperparameter	Value
Learning rate	$1.0\times 10^{-5}$
Discount factor ( $\gamma$ )	0.99
Training timesteps	100000
TD loss function	smooth_l1
CQL regularization ( $\alpha$ )	1.0
Batch size	64

TABLE III: Offline CQL-DQN learning hyperparameters.

Hyperparameter	Value
$\lambda_{\mathrm{KL}}$	0.5
Discount factor ( $\gamma$ )	0.99
Training timesteps	100000
TD loss function	smooth_l1
Batch size	64
Target network update frequency	500

TABLE IV: Pretrained Offline agent learning hyperparameters.

Parameter	$\alpha_{\mathrm{CQL}}$	Learning rate
Initial value	0.0	$5.0\times 10^{-6}$
Final value	1.0	$2.0\times 10^{-6}$
Ramp start	0	40000
Ramp end	40000	80000

TABLE V: PORL hyperparameters ramping.

Approach	Optimality gap
Generally trained PPO agent	$16.19$ %
Pretrained DQN agent (PORL baseline)	$14.50$ %
Flow Due Date / Most Work Remaining	$13.63$ %
Offline Reinforcement Learning agent	$12.48\pm 0.63$ %
Pretrained Offline Reinforcement Learning agent	$11.54\pm 0.35$ %

TABLE VI: Makespan optimality gap (%) for different scheduling approaches, evaluated on high distribution shift instances.

Fig. 2: PORL and Offline RL optimality gap comparison

Approach	Combined Heuristics	Noisy-Expert	Random
Offline RL agent	$12.48\pm 0.63$ %	$16.89\pm 2.15$ %	$26.60\pm 1.24$ %
Pretrained Offline RL agent	$11.54\pm 0.35$ %	$13.19\pm 0.88$ %	$21.08\pm 1.74$ %

TABLE VII: Makespan optimality gap (%) for offline and PORL agents with different dataset compositions.

RL for Combinatorial Optimization

Reinforcement learning for combinatorial optimization learns sequential decisions and search strategies for difficult discrete problems.

See all 106 papers

Scalable Production Scheduling: Linear Complexity via Unified Homogeneous Graphs
Apr 26, 2026Jonathan Hoss, Moritz Link, Noah KlarmannGraph Neural Networks Reinforcement Learning
Faculty of Management and Engineering, Rosenheim Technical University of Applied Sciences, Germany
Variational Approach for Job Shop Scheduling
Jan 30, 2026Seung Heon Oh, Jiwon Baek, Hyunjin Oh +3Job Shop Scheduling
Republic of Korea Navy · Department of Naval Architecture and Ocean Engineering Seoul National University · Research Institute of Marine Systems Engineering Seoul National University
Learning Memory-Enhanced Improvement Heuristics for Flexible Job Shop Scheduling
Mar 3, 2026Jiaqi Wang, Zhiguang Cao, Peng Zhao +4Flexible Job Shop Scheduling Job Shop Scheduling
Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University · College of Software, Jilin University · Singapore Management University +2

Apr 26, 2026J/K move · Enter open · S save

Scalable Production Scheduling: Linear Complexity via Unified Homogeneous Graphs

Jonathan Hoss, Moritz Link, Noah Klarmann

Faculty of Management and Engineering, Rosenheim Technical University of Applied Sciences, Germany

Graph Neural Networks Reinforcement Learning Graph Representation Learning Job Shop Scheduling

Open paper AskPDF arXiv

Efficiently solving the Job Shop Scheduling Problem in real-world industrial applications requires policies that are both computationally lean and topologically robust. While Reinforcement Learning has shown potential in automating dispatching rules, existing models often struggle with a scalability bottleneck caused by quadratic graph complexity or the architectural overhead of heterogeneous layers. We introduce a unified graph framework that employs feature-based homogenization to project distinct node roles into a shared latent space. This allows a standard homogeneous Graph Isomorphism Network to capture complex resource contention with linear complexity, ensuring low-latency inference for large-scale industrial applications. Our empirical results demonstrate that our framework achieves state-of-the-art performance while exhibiting consistent zero-shot generalization. We identify the job-to-machine ratio as the primary driver of policy effectiveness, rather than absolute problem size. Based on this, we propose a hypothesis of structural saturation, demonstrating that policies trained on critically congested instances (

\mathcal{J} \approx \mathcal{M}

) learn scale-invariant resolution strategies. Agents trained at this saturation point internalize invariant conflict-resolution logic, allowing them to treat massive rectangular instances as a sequential concatenation of saturated sub-problems. This approach eliminates the need for expensive scale-specific retraining and prevents overfitting to statistical shortcuts, providing a robust and efficient pathway for deploying RL solutions in dynamic production environments.

Abstract

RL for Combinatorial Optimization

Explore similar work

Scalable Production Scheduling: Linear Complexity via Unified Homogeneous Graphs

Variational Approach for Job Shop Scheduling

Learning Memory-Enhanced Improvement Heuristics for Flexible Job Shop Scheduling