A Novel Hybrid Heuristic-Reinforcement Learning Optimization Approach for a Class of Railcar Shunting Problems

Ruonan Zhao; Joseph Geunes

cs.LGMar 5, 2026

A Novel Hybrid Heuristic-Reinforcement Learning Optimization Approach for a Class of Railcar Shunting Problems

Authors: Ruonan Zhao, Joseph Geunes

Organizations: Wm Michael Barnes ’64 Department of Industrial and Systems Engineering, Texas A&M University

Abstract

Railcar shunting is a core planning task in freight railyards, where yard planners need to disassemble and reassemble groups of railcars to form outbound trains. Classification tracks with access from one side only can be considered as stack structures, where railcars are added and removed from only one end, leading to a last-in-first-out (LIFO) retrieval order. In contrast, two-sided tracks function like queue structures, allowing railcars to be added from one end and removed from the opposite end, following a first-in-first-out (FIFO) order. We consider a problem requiring assembly of multiple outbound trains using two locomotives in a railyard with two-sided classification track access. To address this combinatorially challenging problem class, we decompose the problem into two subproblems, each with one-sided classification track access and a locomotive on each side. We present a novel Hybrid Heuristic-Reinforcement Learning (HHRL) framework that integrates railway-specific heuristic solution approaches with a reinforcement learning method, specifically Q-learning. The proposed framework leverages methods to decrease the state-action space and guide exploration during reinforcement learning. The results of a series of numerical experiments demonstrate the efficiency and quality of the HHRL algorithm in both one-sided access, single-locomotive problems and two-sided access, two-locomotive problems.

Figures & tables

Figure 1: One-sided railyard layout example.

Figure 2: Two-sided railyard layout example

Number of Railcar groups \ Tracks	0–10	10–20	20–40
0–10	Small	Small	Medium
10–20	Small	Medium	Large
20–40	Medium	Large	Large

Table 1: Problem scale by number of car groups and tracks

Parameter	Symbol	Small	Medium	Large
Number of total tracks	$\|S\|$	$\mathcal{U}(4,10)$	$\mathcal{U}(10,40)$	$\mathcal{U}(10,40)$
Number of departure tracks	$\|S_{D}\|$	$\mathcal{U}\!\bigl(2,\min(\|S\|-2,4)\bigr)$	$\mathcal{U}\!\bigl(5,\min(\|S\|-2,7)\bigr)$	$\mathcal{U}\!\bigl(8,\min(\|S\|-2,10)\bigr)$
Number of total car groups	$\|R\|$	$\mathcal{U}(2,20)$	$\mathcal{U}(2,40)$	$\mathcal{U}(10,40)$

Table 2: Scale-dependent parameter settings for problem instance generation

Parameter	Symbol	Value(s)
Number of classification tracks	$\|S_{C}\|$	$\|S\|-\|S_{D}\|$
Number of total car groups in $R_{C}$	$\|R_{C}\|$	$\mathcal{U}(0,\min(\|R\|-1,10))$
Number of total car groups in $R_{D}$	$\|R_{D}\|$	$\|R\|-\|R_{D}\|$
Length of car group $r$ , $\forall r\in R$	$l_{r}$	1
Cost of moving car groups from track $i$ to $j$ , $\forall i,j\in S,j\neq i$	$c_{ij}$	$\|i-j\|$
Head-pair index threshold	$\delta$	3

Table 3: Scale-independent parameter settings for problem instance generation

Figure 6: Exploration rate decay over training episodes

Note: “–” indicates no results were obtained within the 12-hour time limit.
	MIP		ARG-DP			HHRL
Scale	Obj.	Time (s)	Obj.	Opt. Gap (%)	Time (s)	Obj.	Opt. Gap (%)	Time (s)
Small (20 cases)	6.80	21.12	7.15	4.78	1.65	7.05	3.05	117.15
Medium (6 cases)	19	14.14	19.5	2.30	688.10	19	0	13.01
Medium (14 cases)	–	–	–	–	–	33.50	–	178.68
Large (20 cases)	–	–	–	–	–	61.20	–	332.68

Table 4: Performance comparison of MIP, ARG-DP, and HHRL for OS-RSP

	HHRL-APS			HHRL-ROBS
Scale	Obj.	Makespan	Time (s)	Obj.	Makespan	Time (s)
Small (20)	8.0	3.75	37.70	8.1	3.50	36.72
Medium (20)	32.1	10.60	107.72	36.1	8.75	118.20
Large (20)	72.1	21.35	293.16	73.7	17.15	302.90

Table 5: Performance comparison of APS and ROBS under HHRL for TS–RSP

Explore similar work

Towards Autonomous Railway Operations: A Semi-Hierarchical Deep Reinforcement Learning Approach to the Vehicle Rescheduling Problem
May 11, 2026Alberto Castagna, Stefan Zahlner, Adrian Egli +4Multi-Agent Coordination Intelligent Transportation Systems
∗enliteAI, AT · †SBB CFF FFS, CH · ‡Flatland Association, CH
Reducing Complexity for Quantum Approaches in Train Load Optimization
Mar 31, 2026Zhijie Tang, Albert Nieto-Morales, Arit Kumar BishwasCombinatorial Optimization Supply Chain Management
Commercial Technology and Innovation, Shanghai, China · Tech and Innovation, San Francisco, US
A Sliding-Window-Based Reinforcement Learning for Dynamic Assembly Flow Shop Scheduling with Multi-Product Delivery
Jul 3, 2026Junhao Qiu, Jianjun Liu, Ting Liu +3Combinatorial Optimization RL for Combinatorial Optimization
School of Electromechanical Engineering, Guangdong University of Technology, Guangzhou, China · Department of Computer Science, City University of Hong Kong, Hong Kong

May 11, 2026J/K move · Enter open · S save

Towards Autonomous Railway Operations: A Semi-Hierarchical Deep Reinforcement Learning Approach to the Vehicle Rescheduling Problem

Alberto Castagna, Stefan Zahlner, Adrian Egli, Christian Eichenberger+3

∗enliteAI, AT · †SBB CFF FFS, CH · ‡Flatland Association, CH

Multi-Agent Coordination Intelligent Transportation Systems Hierarchical RL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Managing disruptions in railway traffic management is a major challenge. Rising traffic density and infrastructure limits increase complexity, making the Vehicle Routing and Scheduling Problem (VRSP) difficult to solve reliably and in real time. While Operational Research (OR) methods are widely used, most dispatching still relies on human expertise due to the problem's exponential combinatorial complexity. Reinforcement Learning (RL) has gained attention for its potential in multi-agent coordination, but existing RL approaches often underperform OR methods and struggle to scale in dense rail networks. This paper addresses this gap from a machine learning perspective by introducing a semi-hierarchical RL formulation tailored to operational railway constraints. The method separates dispatching from routing through dedicated action and observation spaces, enabling policies to specialise in distinct decision scopes and addressing the imbalance between rare dispatch decisions and frequent routing updates. The approach is evaluated on the Flatland-RL simulator across five difficulty levels and 50 random seeds, with 7 to 80 trains. Results show substantially improved coordination, resource utilisation, and robustness compared with heuristic baselines and monolithic RL, nearly doubling the number of trains reaching their destinations, while keeping deadlock rates below 5% and adaptively sequencing, delaying, or cancelling trains under heavy congestion.

Abstract

Explore similar work

Towards Autonomous Railway Operations: A Semi-Hierarchical Deep Reinforcement Learning Approach to the Vehicle Rescheduling Problem

Reducing Complexity for Quantum Approaches in Train Load Optimization

A Sliding-Window-Based Reinforcement Learning for Dynamic Assembly Flow Shop Scheduling with Multi-Product Delivery