Efficient Detailed Scheduling of Multiproduct Pipelines using Action-Decoupled Deep Reinforcement Learning
Authors: Kai Shao, Ling Jian, Liming Liu, Serguey V. Bogomolov, Xinmin Wang
Organizations: School of Economics and Management, China University of Petroleum (East China), Qingdao, Shandong, China · Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Leninskie Gory, Moscow, 119899, Russia
Multiproduct pipelines transport refined products to delivery stations in sequential batches, and their detailed scheduling must meet market demand while respecting nonlinear hydraulic constraints. Retaining the full hydraulic nonlinearity yields a mixed-integer nonlinear program that is computationally intractable, while the linearization commonly adopted for solver-based optimization degrades hydraulic accuracy without resolving the heavy computational burden. This paper develops an action-decoupled proximal policy optimization (PPO) framework that efficiently produces detailed schedules while preserving hydraulic fidelity. The scheduling process is cast as a semi-Markov decision process (SMDP) with flexible action durations. To handle the discrete-continuous hybrid action space, a cascaded dual-actor architecture decouples the discrete operating duration from the continuous flow decisions, eliminating the gradient interference that destabilizes conventional coupled-action policies and thereby yielding high-quality schedules far more efficiently. Validated on a real-world multiproduct pipeline in China and benchmarked against a single-actor baseline and a linearized MILP solved by Gurobi, the proposed framework achieves delivery accuracy close to the exact optimum while substantially improving flow rate stability and substantially reducing solution time.
Figures & tables
Figure 1: Schematic of sequential batch transportation in a multiproduct pipeline
Figure 2: Schematic diagram of batch lengths and elevation differences within pipeline segment i .
Figure 3: Interaction between the cascaded dual-actor agent and the multiproduct pipeline simulation environment under the SMDP-PPO framework.
Station ID
Altitude (m)
Min delivery flow rate (m 3 /h)
Max delivery flow rate (m 3 /h)
Max head of pump station (m)
IS
0
–
–
1092
DPS1
7
100
900
1096
DS2
111
100
600
–
DS3
53
100
600
–
TS
51
100
600
–
Table 1: Physical and operational parameters of each station.
Pipeline segment
Length (km)
Inner diameter (mm)
Min allowable flow rate (m 3 /h)
Max allowable flow rate (m 3 /h)
IS–DPS1
130.4
492.2
600
1100
DPS1–DS2
139.5
492.2
500
1100
DS2–DS3
66.9
441.2
100
1100
DS3–TS
38.7
441.2
100
1100
Table 2: Physical and operational parameters of pipeline segments.
Product Type
Density ( kg/m3 )
Viscosity ( m2/s )
0# diesel
845
6.11×10−6
92# gasoline
740
2.05×10−6
95# gasoline
750
1.07×10−6
Table 3: Physical properties of refined products.
Station
0# Diesel
92# Gasoline
95# Gasoline
Total
(m 3 )
(m 3 )
(m 3 )
(m 3 )
DPS1
22,485
35,810
14,666
72,961
DS2
8,876
20,270
8,000
37,146
DS3
10,651
18,919
4,667
34,236
TS
31,361
23,648
6,000
61,009
Total demand
73,373
98,647
33,333
205,353
Table 4: Planned supply and demand for the case study.
Category
Parameters
Value
Network Architecture
Actor 1&2 network (Layers × Neurons)
2×256
Critic network (Layers × Neurons)
2×256
PPO Algorithm
Learning rate
3×10−4
Discount factor ( γ )
0.99
PPO clip coefficient ( ϵ )
0.2
GAE parameter ( λ )
0.95
Table 5: Hyper-parameters for the Action-Decoupled SMDP-PPO Framework
Figure 4: Comparison of training performance between the proposed Action-Decoupled and Single-Actor SMDP-PPO. (a) Convergence of the delivery deviation f1 ; (b) convergence of the flow rate stability f2 . The solid line and shaded band denote the mean and standard deviation over three independent runs.
Figure 5: Batch migration chart of a representative detailed scheduling scheme found by the Action-Decoupled SMDP-PPO during training.
Figure 6: Flow rate and hydraulic loss of each pipeline segment under the obtained scheduling scheme. The red step lines and blue lines denote the segment flow rate and hydraulic loss, respectively, and the dashed lines indicate the minimum flow rate limits. (a) Segment 1 (IS–DPS1); (b) Segment 2 (DPS1–DS2); (c) Segment 3 (DS2–DS3); (d) Segment 4 (DS3–TS).
Station
0# Diesel
92# Gasoline
95# Gasoline
(m 3 )
(m 3 )
(m 3 )
Demand
Prop.
SA
Gurobi
Demand
Prop.
SA
Gurobi
Demand
Prop.
SA
Gurobi
DPS1
22,485
22,875
22,625
22,485
35,810
35,829
35,045
35,810
14,666
14,719
14,651
14,666
DS2
8,876
10,674
11,861
15,005
20,270
20,219
20,333
20,391
8,000
8,280
7,694
8,000
DS3
10,651
11,082
11,434
10,651
18,919
19,088
19,678
18,919
4,667
4,717
4,803
4,667
TS
31,361
35,031
33,407
31,361
23,648
23,681
23,998
23,648
6,000
5,792
6,093
6,000
Table 6: Planned demand and actual delivered volumes at each station for the representative schedules found by the three methods during training/solving.
Method
f1 ( m3 )
f2 ( m3/h )
Solution time ( s )
Gap ( % )
Action-Decoupled SMDP-PPO
7,153
561
1,503
–
Single-Actor SMDP-PPO
8,442
1,449
2,059
–
Gurobi
6,250
1,697
38,911
13.8
Table 7: Performance comparison of the proposed method, the single-actor baseline, and the Gurobi-based MILP model.
Figure 7: Comparison between the Action-Decoupled and Single-Actor SMDP-PPO. (a) Time to the first feasible solution; (b) number of feasible solutions over 30,000 episodes. Bars and error bars denote mean and standard deviation over three runs; circles mark individual runs.
TUM School of Management, Technical University of Munich · Esade, Ramon Llull University · Munich Data Science Institute (MDSI), Technical University of Munich