Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance

Zeqiao Li; Yijing Wang; Haoyu Wang; Zheng Li; Zhiqiang Zuo

cs.ROOct 13, 2026

Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance

Authors: Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Zhiqiang Zuo

Abstract

Online reinforcement learning (RL) enables driving policies to adapt to changing deployment conditions, but requires safe exploration throughout training. Safety filtering alone does not ensure stable policy improvement or reduced dependence on corrections. We propose Safe Online Adaptive Refinement (SOAR-Flow), a framework for safe and stable online fine-tuning of offline-initialized Flow-based driving policies. A control barrier function (CBF)--model predictive control (MPC) filter corrects potentially unsafe actions before execution. To support stable policy updates, we introduce Gaussian-return-aware value calibration based on calibrated Q-learning (Cal-QL), using recorded behavior returns and predicted return variability to smooth conservative critic updates and mitigate policy degradation. Paired corrective learning teaches feasible action adjustments, while an intervention-to-go predictor encourages decisions requiring fewer future corrections. Its predictions also regulate guidance strength, allowing recovery supervision to weaken as predicted dependence decreases. Together with gradually reduced calibration regularization, this supports continued task improvement while preserving hard execution constraints. We analyze conditions for safe execution at control steps and diminishing auxiliary learning influences. We evaluate SOAR-Flow in simulation and on an uncrewed ground vehicle (UGV). The results demonstrate improved driving performance and reduced reliance on safety corrections during protected online fine-tuning. The demonstration video and project overview are available at https://lzqw.github.io/SOAR-Flow/.

Figures & tables

Panel of Figure 2 — Fig. 2: MetaDrive scenario and experimental workstation.

Panel of Figure 3 — Fig. 3: Simulation learning curves.

Method	Return $\uparrow$	SS (%) $\uparrow$	Eval. Cost $\downarrow$	Train Cost $\downarrow$	Corr. Rate (%) $\downarrow$
CBF–MPC	$350.0\!\pm\!0.6$	$\boldsymbol{100.0\!\pm\!0.0}$	$\boldsymbol{0.00\!\pm\!0.00}$	–	–
Flow-BC	$309.1\!\pm\!4.8$	$70.0\!\pm\!7.1$	$5.42\!\pm\!0.76$	–	$3.33\!\pm\!0.35$
Cal-QL [ 2 ]	$335.9\!\pm\!3.5$	$88.0\!\pm\!3.2$	$1.94\!\pm\!0.54$	$6.94\!\pm\!1.46$	$5.25\!\pm\!0.40$
PROTO-SAC [ 16 ]	$340.3\!\pm\!3.4$	$91.6\!\pm\!2.6$	$1.14\!\pm\!0.48$	$4.66\!\pm\!1.02$	$3.86\!\pm\!0.40$
Gaussian-DSAC	$343.5\!\pm\!3.0$	$96.0\!\pm\!3.2$	$0.38\!\pm\!0.28$	$0.85\!\pm\!0.32$	$2.68\!\pm\!0.28$
Flow-Q-only	$278.2\!\pm\!12.1$	$42.0\!\pm\!9.5$	$6.24\!\pm\!0.87$	$18.22\!\pm\!3.20$	$37.11\!\pm\!4.40$

TABLE I: Comparison in the simulation experiment, including safe success (SS).

	Blocked-Side Bypass			8-shaped Maneuver			Held-out Cornering
Method	SS (%) $\uparrow$	Return $\uparrow$	Corr. Rate (%) $\downarrow$	SS (%) $\uparrow$	Time (s) $\downarrow$	Min. Clr. (m) $\uparrow$	SS (%) $\uparrow$
CBF–MPC	$\mathbf{100}$	$345.7\!\pm\!1.5$	–	$95$	$16.72\!\pm\!0.31$	$0.779\!\pm\!0.018$	$85$
Flow-BC	$85$	$311.2\!\pm\!3.8$	$4.31\!\pm\!0.41$	$65$	$18.21\!\pm\!0.56$	$0.681\!\pm\!0.039$	$55$
Gaussian-DSAC	$95$	$343.9\!\pm\!2.6$	$2.46\!\pm\!0.32$	$90$	$15.96\!\pm\!0.29$	$0.776\!\pm\!0.017$	$90$
Ours	$\mathbf{100}$	$\boldsymbol{350.8\!\pm\!1.8}$	$\boldsymbol{0.68\!\pm\!0.11}$	$\mathbf{100}$	$\boldsymbol{15.36\!\pm\!0.24}$	$\boldsymbol{0.801\!\pm\!0.019}$	$\mathbf{95}$

TABLE II: UGV comparison. SS is reported as a percentage; other metrics are mean

\pm

SD.

Fig. 5: UGV refinement with SOAR-Flow. (a) Multimodal initialization. (b–c) Recorded blocked-side and 8-shaped trajectories at different refinement stages. Red segments indicate corrections; the late 8-shaped run uses PO execution with MPC tracking retained. (d) Held-out SOAR-Flow success and fixed CBF–MPC collision. (e–f) Blocked-side curves; vertical dashed lines mark training stages.

Fig. 6: 8-shaped refinement with SOAR-Flow: early protected execution, intermediate refinement, and late execution without CBF correction. Red segments mark correction intervals in the early and intermediate runs.

RL for Autonomous Driving

Reinforcement learning for autonomous driving trains vehicle policies through interaction to plan, control, and coordinate safely in traffic.

See all 120 papers

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
Mar 16, 2026Mumuksh Tayal, Manan Tayal, Ravi PrakashFlow Matching Offline RL
Indian Institute of Science, India
Safe Online Learning via Smooth Safety-Structured Policy Composition
Jun 30, 2026Hongpeng Cao, Liqun Zhao, Yuliang Gu +3Safety-Critical Control Safe RL
School of Engineering and Design Technical University of Munich Garching, Munich 85748, Germany · Department of Engineering Science University of Oxford Oxford OX1 3PJ, United Kingdom · Department of Mechanical Science and Engineering University of Illinois Urbana-Champaign Urbana, IL 61801, USA +1
Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning
Aug 11, 2026Xincong Hu, Lei Ou, Maosen Li +3Adversarial Scenario Generation Safe RL
Nanjing University · Yinwang Intelligent Technology Co., Ltd

Mar 16, 2026J/K move · Enter open · S save

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Mumuksh Tayal, Manan Tayal, Ravi Prakash

Indian Institute of Science, India

Flow Matching Offline RL Safe RL Flow-Based Policy Learning

Open paper AskPDF arXiv

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

Abstract

RL for Autonomous Driving

Explore similar work

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Safe Online Learning via Smooth Safety-Structured Policy Composition

Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning