Epigraph Form Reinforcement Learning-based Safe Trajectory Tracking Control for Connected and Autonomous Vehicles
Authors: Xiaoke Yang, Jin Chen, Yafei Wang, Qi Zhang, Ming Li, Haiwen Wu, Chengliang Yin, Yong Zhang
Organizations: School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · KTH Royal Institute of Technology, Stockholm SE-10044, Sweden · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong 999077, China
Safe trajectory tracking (STT) aims to enable autonomous vehicles to follow a reference trajectory accurately while avoiding collisions. This problem is particularly challenging for Connected and Autonomous Vehicles (CAVs) in highly dynamic environments. Existing non-learning-based STT methods often use simplified vehicle and interaction models to meet real-time requirements, which may result in suboptimal tracking accuracy. Although Deep Reinforcement Learning (DRL) can help address this limitation, widely used Lagrangian DRL methods often suffer from unstable training because increasing Lagrange multipliers amplify policy-gradient magnitudes. To address these issues, we propose a novel reinforcement learning framework for STT, called STT Control Epigraph Form RL (STTC-EFRL). In this framework, the STT problem is formulated as an epigraph-form min-max optimization problem. Based on this formulation, we develop a learning mechanism that uses a decreasing scalar related to the epigraph to adjust the policy gradient descent direction without increasing the gradient magnitude, thereby improving training stability. We also develop a computationally efficient ray-casting method to compute a polyhedron scaling factor for safety quantification. Experimental results, including Vehicle-in-the-Loop tests, show that STTC-EFRL achieves higher tracking accuracy than the considered non-learning-based baseline and more stable training than widely used Lagrangian DRL methods, while satisfying safety requirements in connected environments.
Figures & tables
Fig. 1: Illustration of a typical STT task for a CAV in a V2X-enabled environment. The ego vehicle tracks a reference trajectory while interacting with roadside infrastructure and other connected vehicles.
Fig. 2: Comparison of STT performance in an obstructed-lane scenario. Rectangles denote vehicles at historical time steps. In (a), the hierarchical controller deviates from the reference trajectory to remain safe, which reduces tracking accuracy. In (b), the integrated controller better follows the reference trajectory while maintaining safety.
Fig. 3: Diagram of the 3-DOF bicycle model of a vehicle.
Fig. 4: STTC-EFRL framework architecture. The solid lines denote the information flow during evaluation and implementation, and the dashed lines denote the information flow during training.
Fig. 5: Examples of calculating the optimal scaling factor γ∗ between two rectangles. In (a), γ∗>1 , which means no collision occurs between the two rectangles. In (b), 0<γ∗≤1 , which means P1 collides with P2 , where a smaller γ∗ indicates a more severe collision. In (c), γ∗=0 , which means the scaling center of P1 lies inside P2 , and γ∗ is consistently 0.
Fig. 6: The structure of the ViL platform.
Name
Range
x coordinate of the start of the lane change segment
[0,40] m
x coordinate of the end of the lane change segment
[60,100] m
x coordinate of the start of the ego vehicle
[0,30] m
x coordinate of the static vehicle
[50,80] m
initial speed of the ego vehicle
[60,90] km/h
TABLE I: Random Parameters in Type-1 and Type-2 training scenarios generation.
Name
Range
x coordinate of the start of the ego vehicle
[−100,−80] m
initial speed of the ego vehicle
[60,90] km/h
speed of the moving vehicle
[20,100] km/h
speed in the reference trajectory at the entrance
[60,90] km/h
speed in the reference trajectory during turning
[30,40] km/h
TABLE II: Random Parameters in Type-3 training scenarios generation.
Name
Value
Name
Value
lane width
3 m
turning area
35 m ×35 m
ego size
2.63 m ×1.65 m
obstacle size
4.10 m ×2.05 m
Cf
95700 N/rad
Cr
93020 N/rad
lf
0.8475 m
lr
0.9025 m
M
940 kg
Iz
752.25 kg ⋅m2
TABLE III: Other parameters.
elateralmean
σlateral
Cost
Hkmax
Safety
I
STTC-EFRL
-1.33
1.39
5.67
-0.61
✓
PPO-L
-1.73
1.82
5.79
0.38
×
DDPG-L
-1.07
2.31
8.54
-0.09
✓
UFTSTC
-1.17
2.73
6.72
-0.24
✓
II
STTC-EFRL
-0.59
1.02
5.23
-0.23
✓
TABLE IV: Overall performance of ViL experiments.
Fig. 7: Schematic of three types of training scenarios. (a) is Type-1: the lane-changing scenario. (b) is Type-2: the overtaking scenario. (c) is Type-3: the intersection scenario.
Fig. 8: Total tracking cost in (a) and unsafe fraction in (b) in Type-1 and Type-2 scenarios, and total tracking cost in (c) and unsafe fraction in (d) in Type-3 scenarios, w.r.t. training iterations.
Fig. 9: Policy gradient norm in (a), tracking critic loss in (c), safety critic loss in (e) in Type-1 and Type-2 scenarios, and policy gradient norm in (b), tracking critic loss in (d), safety critic loss in (f) in Type-3 scenarios, w.r.t. training iterations.
Fig. 10: Total tracking cost in (a) and unsafe fraction in (b) w.r.t. training iterations with different hyper parameters. Numbers in brackets beside STTC-EFRL denote zmax and beside PPO-L denote λinit .
Fig. 11: The tracking trajectories in the x–y coordinate system of Scenario I in (a), Scenario II in (b), Scenario III in (c), Scenario IV in (d), Scenario V in (e) and Scenario VI in (f). The red squares denote the static and moving obstacle vehicles.
Fig. 12: The lateral errors w.r.t. the traveled reference trajectory distance of Scenario I in (a), Scenario II in (b), Scenario III in (c), Scenario IV in (d), Scenario V in (e) and Scenario VI in (f).
Fig. 13: Key intermediate frames of ViL experiments of Scenario I in (a), Scenario III in (b) and Scenario V in (c). Transparent primitives represent vehicles at historical time steps.
Fig. 14: Three repeated ViL trajectories of STTC-EFRL in Scenario I under fixed experimental conditions.
Comparison
Mean (m)
Std. (m)
Trial 1 vs. Trial 2
0.0606
0.0557
Trial 1 vs. Trial 3
0.0466
0.0618
Trial 2 vs. Trial 3
0.0935
0.0831
TABLE V: Mean and standard deviation of absolute lateral trajectory differences for STTC-EFRL in Scenario I.
Robot trajectory tracking develops methods that make robots follow desired paths or motions accurately despite disturbances, uncertainty, and physical constraints.
College of Automotive and Energy Engineering, Tongji University, Shanghai, 201804, China · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore, 639798, Singapore
School of Physics, Northeast Normal University, Changchun, 130024, China. · 2Clean Energy Automotive Engineering Center, School of Automotive Studies, Tongji University, Shanghai, 201804, China. · 3Mengshi Automobile Technology Company, Dongfeng Motor Corporation, Wuhan, 430010, China.