Efficient Detailed Scheduling of Multiproduct Pipelines using Action-Decoupled Deep Reinforcement Learning
Authors: Kai Shao, Ling Jian, Liming Liu, Serguey V. Bogomolov, Xinmin Wang
Organizations: School of Economics and Management, China University of Petroleum (East China), Qingdao, Shandong, China · Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Leninskie Gory, Moscow, 119899, Russia
Multiproduct pipelines transport refined products to delivery stations in sequential batches, and their detailed scheduling must meet market demand while respecting nonlinear hydraulic constraints. Retaining the full hydraulic nonlinearity yields a mixed-integer nonlinear program that is computationally intractable, while the linearization commonly adopted for solver-based optimization degrades hydraulic accuracy without resolving the heavy computational burden. This paper develops an action-decoupled proximal policy optimization (PPO) framework that efficiently produces detailed schedules while preserving hydraulic fidelity. The scheduling process is cast as a semi-Markov decision process (SMDP) with flexible action durations. To handle the discrete-continuous hybrid action space, a cascaded dual-actor architecture decouples the discrete operating duration from the continuous flow decisions, eliminating the gradient interference that destabilizes conventional coupled-action policies and thereby yielding high-quality schedules far more efficiently. Validated on a real-world multiproduct pipeline in China and benchmarked against a single-actor baseline and a linearized MILP solved by Gurobi, the proposed framework achieves delivery accuracy close to the exact optimum while substantially improving flow rate stability and substantially reducing solution time.
Figures & tables
Figure 1: Schematic of sequential batch transportation in a multiproduct pipeline
Figure 2: Schematic diagram of batch lengths and elevation differences within pipeline segment i .
Figure 3: Interaction between the cascaded dual-actor agent and the multiproduct pipeline simulation environment under the SMDP-PPO framework.
Station ID
Altitude (m)
Min delivery flow rate (m 3 /h)
Max delivery flow rate (m 3 /h)
Max head of pump station (m)
IS
0
–
–
1092
DPS1
7
100
900
1096
DS2
111
100
600
–
DS3
53
100
600
–
TS
51
100
600
–
Table 1: Physical and operational parameters of each station.
Pipeline segment
Length (km)
Inner diameter (mm)
Min allowable flow rate (m 3 /h)
Max allowable flow rate (m 3 /h)
IS–DPS1
130.4
492.2
600
1100
DPS1–DS2
139.5
492.2
500
1100
DS2–DS3
66.9
441.2
100
1100
DS3–TS
38.7
441.2
100
1100
Table 2: Physical and operational parameters of pipeline segments.
Product Type
Density ( kg/m3 )
Viscosity ( m2/s )
0# diesel
845
6.11×10−6
92# gasoline
740
2.05×10−6
95# gasoline
750
1.07×10−6
Table 3: Physical properties of refined products.
Station
0# Diesel
92# Gasoline
95# Gasoline
Total
(m 3 )
(m 3 )
(m 3 )
(m 3 )
DPS1
22,485
35,810
14,666
72,961
DS2
8,876
20,270
8,000
37,146
DS3
10,651
18,919
4,667
34,236
TS
31,361
23,648
6,000
61,009
Total demand
73,373
98,647
33,333
205,353
Table 4: Planned supply and demand for the case study.
Category
Parameters
Value
Network Architecture
Actor 1&2 network (Layers × Neurons)
2×256
Critic network (Layers × Neurons)
2×256
PPO Algorithm
Learning rate
3×10−4
Discount factor ( γ )
0.99
PPO clip coefficient ( ϵ )
0.2
GAE parameter ( λ )
0.95
Table 5: Hyper-parameters for the Action-Decoupled SMDP-PPO Framework
Figure 4: Comparison of training performance between the proposed Action-Decoupled and Single-Actor SMDP-PPO. (a) Convergence of the delivery deviation f1 ; (b) convergence of the flow rate stability f2 . The solid line and shaded band denote the mean and standard deviation over three independent runs.
Figure 5: Batch migration chart of a representative detailed scheduling scheme found by the Action-Decoupled SMDP-PPO during training.
Figure 6: Flow rate and hydraulic loss of each pipeline segment under the obtained scheduling scheme. The red step lines and blue lines denote the segment flow rate and hydraulic loss, respectively, and the dashed lines indicate the minimum flow rate limits. (a) Segment 1 (IS–DPS1); (b) Segment 2 (DPS1–DS2); (c) Segment 3 (DS2–DS3); (d) Segment 4 (DS3–TS).
Station
0# Diesel
92# Gasoline
95# Gasoline
(m 3 )
(m 3 )
(m 3 )
Demand
Prop.
SA
Gurobi
Demand
Prop.
SA
Gurobi
Demand
Prop.
SA
Gurobi
DPS1
22,485
22,875
22,625
22,485
35,810
35,829
35,045
35,810
14,666
14,719
14,651
14,666
DS2
8,876
10,674
11,861
15,005
20,270
20,219
20,333
20,391
8,000
8,280
7,694
8,000
DS3
10,651
11,082
11,434
10,651
18,919
19,088
19,678
18,919
4,667
4,717
4,803
4,667
TS
31,361
35,031
33,407
31,361
23,648
23,681
23,998
23,648
6,000
5,792
6,093
6,000
Table 6: Planned demand and actual delivered volumes at each station for the representative schedules found by the three methods during training/solving.
Method
f1 ( m3 )
f2 ( m3/h )
Solution time ( s )
Gap ( % )
Action-Decoupled SMDP-PPO
7,153
561
1,503
–
Single-Actor SMDP-PPO
8,442
1,449
2,059
–
Gurobi
6,250
1,697
38,911
13.8
Table 7: Performance comparison of the proposed method, the single-actor baseline, and the Gurobi-based MILP model.
Figure 7: Comparison between the Action-Decoupled and Single-Actor SMDP-PPO. (a) Time to the first feasible solution; (b) number of feasible solutions over 30,000 episodes. Bars and error bars denote mean and standard deviation over three runs; circles mark individual runs.
Deep reinforcement learning (DRL) approaches for flexible job shop scheduling (FJSP) heavily rely on attention-centric architectures to achieve state-of-the-art performance. However, these models suffer from excessive parameter counts and prohibitive inference latency as problem scales expand. While liquid neural networks (LNNs) offer a parameter-efficient alternative for modeling adaptive state evolution, their inherently sequential dynamics bottleneck computational efficiency. To resolve this trade-off, we propose PLAN (Parallel Liquid-inspired Approximation Network), a lightweight representation learning framework that reformulates continuous liquid-state dynamics into a discretized and parallelizable formulation. PLAN structurally decouples state evolution from context aggregation, where liquid-inspired updates handle the primary evolving state representation, and a lightweight context aggregation module provides complementary global context. Furthermore, PLAN acts as a versatile, plug-and-play backbone that generalizes to complex FJSP variants, pairing with a compact stochastic module for stochastic FJSP and replacing heavy heterogeneous graph transformers in multi-faceted dynamic FJSP. Extensive evaluations across deterministic, stochastic, and multi-faceted dynamic FJSP benchmarks show that PLAN reduces the average makespan by 1.2%, 1.4%, and 2.3%, respectively, compared with the corresponding state-of-the-art baselines, with the improvement reaching 10.2% in one benchmark setting. PLAN also reduces average inference latency by 13.2%, 31.7%, and 26.9%, respectively, with a maximum reduction of 69.2% on the largest instances, while using only 22−47% of the baseline parameters.
Dhivya Dharshini Kannan, Wei Zhang, Jieyi Bi +5
Singapore Institute of Technology (SIT) · Nanyang Technological University (NTU) · Shanghai Jiao Tong University +1
This paper extends Mixed-Integer Differentiable Predictive Control (MI-DPC) to multi-modal discrete decisions and nonconvex polynomial dynamics arising in Underground Pumped Hydro Energy Storage Systems (UPHES). A neural policy mapping problem parameters to continuous setpoints and integer mode selections via a Gumbel-Softmax layer is trained in a self-supervised manner by differentiating the expectation of the finite horizon control objective through the nonlinear dynamics model. Three methodological contributions enable this extension: a parallel differentiable simulator that preserves gradient magnitude, a Transformer encoder that captures long-range temporal dependencies, and a Gumbel-Softmax temperature annealing schedule that regularizes the combinatorial search. We demonstrate the framework on day-ahead scheduling of a UPHES, a large-scale mixed-integer optimal control problem with nonlinear unit performance curves and volume-head coupling. MI-DPC achieves only 1.6% suboptimality relative to a piecewise mixed-integer quadratic programming baseline, while providing five orders of magnitude speedup in online scheduling time.
Honghui Zheng, Ján Boldocký, Yury Dvorkin +1
Johns Hopkins University, Baltimore MD, US · Slovak University of Technology in Bratislava, Slovakia
Many operational problems are constrained sequential decision processes with combinatorial action spaces and interdependent constraints. Mixed-integer linear programs (MILPs) handle such constraints flexibly but scale poorly in stochastic environments. Deep reinforcement learning (DRL) promises scalable decision rules, but existing methods either penalize constraints rather than enforce them, or rely on feasibility mechanisms that break down once constraints interact. We bridge this gap by embedding a differentiable optimization module inside the policy: a neural network proposes continuous action targets, a quadratic program projects them onto the relaxed feasible set, and a dual-informed integer mapping restores integrality while preserving feasibility. Given a differentiable simulator, the policy trains end-to-end from sampled trajectories using pathwise gradients, while handling hard constraints with similar flexibility to MILPs. We show that our approach has bounded error relative to an exact integer projection and ensures the entire feasible action space is reachable. We apply the method to multi-echelon production-inventory planning under shared resource and material constraints. Our policy attains an average optimality gap of 1% on 3-item instances and consistently outperforms a rolling-horizon multi-stage stochastic program by at least 5.6% in 10-item networks. Compared to state-of-the-art echelon base-stock policies, savings reach 15.3% on 40-item instances. On an industry-scale case study with nonstationary demand from ASML, we reduce average cost by up to 1.8% relative to the best-known benchmark policy. The savings are largest where planning is hardest: in tightly capacitated systems with high demand variability. More broadly, we show that DRL can improve upon state-of-the-art baselines in sequential decision problems with hard constraints, which are widespread in practice.
Patrick Helm, Jan-Niklas Doerr, Joren Gijsbrechts +1
TUM School of Management, Technical University of Munich · Esade, Ramon Llull University · Munich Data Science Institute (MDSI), Technical University of Munich