This paper studies safe policy improvement for real-time operation control of flex-route transit (FRT) systems, where robustness is critical and any performance decrease is undesirable. While reinforcement learning (RL) has shown strong empirical success, its adoption remains limited in safety and cost critical applications, such as intelligent transportation, since learned policies lack statistically certified performance guarantees before deployment. In this paper, we propose the High Confidence Lower Bound Approximation (Hi-CoLA) RL framework, which combines importance sampling and concentration inequalities with a differentiable surrogate network to efficiently estimate the high-confidence performance lower bound and optimize it via policy gradient for robust performance improvement. On the FRT operation control problem, Hi-CoLA outperforms state-of-the-art offline and safe RL methods in both average and tail-risk performance. A case study on an actual route in Chicago, implemented in the SUMO simulator, serves as a proof of concept. Hi-CoLA enhances real-time FRT control with guaranteed performance and is broadly applicable to other safety critical decision-making scenarios.
Figures & tables
Fig. 1: Illustration of Flex-Route Transit System
Notations
Descriptions
W×L
dimensions of service area
G(V,E)
the routing network (represented as a graph)
V
set of total stops, ∣V∣=N
C
set of fixed stops, ∣C∣=n
F
set of flexible stops, ∣F∣=m
E
set of directed edges connecting stops
TABLE I: List of Major Notations Used in the Paper
Fig. 2: Flex-Route Transit Operation System Description
Fig. 3: Hi-CoLA framework and the abstract flow process. The framework has two phases: Hi-CoLA network pre-training and policy improvement. In pre-training, the behavioral policy π(θb) perturbs neighborhood candidate policies and generates trajectories for confidence lower bound calculation. The perturbed policies pass through a KL filter, and the qualified ones serve as training features; their confidence lower bounds, computed via HCOPE, serve as training labels for the Hi-CoLA Net. The trained network is then deployed to improve π(θb) : starting from θt=θb , it predicts μ^δ(θt) , which gradient-based policy optimization maximizes. Updates iterate until leaving the KL trust region, after which the behavioral policy is updated and the cycle repeats for continual improvement.
Fig. 4: Training process
Fig. 5: Performance Distribution Comparison
Method
CVaR 5%
CVaR 10%
CVaR 5% Improve
CVaR 10% Improve
Behavioral
57.07
57.41
0.00%
0.00%
IQL
61.39
61.75
7.57%
7.56%
CQL
61.95
62.28
8.55%
8.48%
TD3+BC
59.97
60.35
5.08%
5.12%
Hi-CoLA
63.52
63.88
11.30%
11.27%
Daedalus
57.51
57.86
0.77%
0.78%
TABLE II: Hourly Ridership CVaR Performance
Method
CVaR 5%
CVaR 10%
Δ CVaR 5%
Δ CVaR 10%
Behavioral
53.56
53.92
-6.15%
-6.08%
IQL
61.13
61.48
-0.42%
-0.43%
CQL
55.72
56.08
-10.06%
-9.96%
TD3+BC
53.78
54.17
-10.32%
-10.24%
Hi-CoLA
63.41
63.79
-0.17%
-0.14%
Daedalus
56.29
56.46
-2.12%
-2.42%
TABLE III: Hourly Ridership CVaR Performance with Random Deviation Policy and Comparison with Table II
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
Taxiway routing and on-surface conflict avoidance are coupled safety-critical decision problems in airport surface operations. Existing planning and optimization methods are often limited by online computational cost, while reinforcement learning methods may struggle to represent downstream traffic conflicts and balance multiple objectives. This paper presents Conflict-aware Taxiway Routing (CaTR), a reinforcement learning framework for real-time multi-aircraft taxiway routing. CaTR constructs a grid-based airport surface environment with action masking, introduces a hierarchical foresight traffic representation to encode current and downstream conflict-related traffic conditions, and adopts a value-decomposed reinforcement learning strategy to prioritize sparse but safety-critical objectives. Experiments are conducted on a realistic environment based on Changsha Huanghua International Airport under multiple traffic density levels. Results show that CaTR achieves better safety--efficiency trade-offs than representative planning, optimization, and reinforcement learning baselines while maintaining practical runtime.
Shizhong Zhou, Haifeng Liu, Zheng Zhang +3
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu 610000, China · College of Computer Science, Sichuan University, Chengdu 610000, China
Supervised open-loop training has been widely adopted for training traffic simulation models; however, it fails to capture the inherently dynamic, multi-agent interactions common in complex driving scenarios. We introduce RLFTSim, a reinforcement-learning-based fine-tuning framework that enhances scenario realism by aligning simulator rollouts with real-world data distributions and provides a method for distilling goal-conditioned controllability in scenario generation. We instantiate RLFTSim on top of a pre-trained simulation model, design a reward that balances fidelity and controllability, and perform comprehensive experiments on the Waymo Open Motion Dataset. Our results show improvements in realism, achieving state-of-the-art performance. Compared with other heuristic search-based fine-tuning methods, RLFTSim requires significantly fewer samples due to a proposed low-variance and dense reward signal, and it directly addresses the realism alignment issue by design. We also demonstrate the effectiveness of our approach for distilling traffic simulation controllability through goal conditioning. The project page is available at https://ehsan-ami.github.io/rlftsim.
When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL methods, although offering high sample efficiency, suffer from constraint violations due to cost-agnostic exploration and estimation bias in cumulative cost. To address this issue, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy safe RL algorithm that integrates cost-bounded online exploration and conservative offline distributional value learning. First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space and adaptively adjusts the trust region to control the training cost. Second, we adopt truncated quantile critics to stabilize the cost value learning. Quantile critics also quantify epistemic uncertainty to guide exploration. Experiments on safe velocity, safe navigation, and autonomous driving tasks demonstrate that COX-Q achieves high sample efficiency, competitive test safety performance, and controlled data collection cost. The results highlight COX-Q as a promising RL method for safety-critical applications.
Guopeng Li, Matthijs T. J. Spaan, Julian F. P. Kooij
Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, the Netherlands