RL for Autonomous Driving

RL: Reinforcement Learning

Ask about this topic

Overview

The field studies how vehicles can learn driving decisions and controls through rewards and interaction, including how to navigate traffic, avoid hazards, and coordinate with other road users. Methods range from PPO, SAC, and DQN to model-based and world-model learning, multi-agent reinforcement learning, and combinations of imitation learning with diffusion or vision-language-action planners. Safety is addressed through reward design, expert guidance, control barrier functions, and reachability constraints, with evaluation commonly using CARLA, nuPlan, NAVSIM, and Bench2Drive. A central challenge is achieving sample-efficient learning that remains safe and reliable in closed-loop, unfamiliar, and real-world conditions, where simulator fidelity and trade-offs among safety, progress, and comfort matter.

Latest papers

CardsList

120 papers21 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Apr 3, 2026

ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving

End-to-end autonomous driving models based on Vision-Language-Action (VLA) architectures have shown promising results by learning driving policies through behavior cloning on expert demonstrations. However, imitation learning inherently limits the model to replicating observed behaviors without exploring diverse driving strategies, leaving it brittle in novel or out-of-distribution scenarios. Reinforcement learning (RL) offers a natural remedy by enabling policy exploration beyond the expert distribution. Yet VLA models, typically trained on offline datasets, lack directly observable state transitions, necessitating a learned world model to anticipate action consequences. In this work, we propose a unified understanding-and-generation framework that leverages world modeling to simultaneously enable meaningful exploration and provide dense supervision. Specifically, we augment trajectory prediction with future RGB and depth image generation as dense world modeling objectives, requiring the model to learn fine-grained visual and geometric representations that substantially enrich the planning backbone. Beyond serving as a supervisory signal, the world model further acts as a source of intrinsic reward for policy exploration: its image prediction uncertainty naturally measures a trajectory's novelty relative to the training distribution, where high uncertainty indicates out-of-distribution scenarios that, if safe, represent valuable learning opportunities. We incorporate this exploration signal into a safety-gated reward and optimize the policy via Group Relative Policy Optimization (GRPO). Experiments on the NAVSIM and nuScenes benchmarks demonstrate the effectiveness of our approach, achieving a state-of-the-art PDMS score of 93.7 and an EPDMS of 88.8 on NAVSIM. The code is available at https://zihaosheng.github.io/ExploreVLA/.

Zihao Sheng, Xin Ye, Jingru Luo +2

Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI) · University of Wisconsin–Madison

World Model LearningVLMs for Autonomous DrivingEnd-to-End Autonomous DrivingRL ExplorationVision-Language-Action ModelsWorld Models for Autonomous Driving+1
AskPDF
Mar 28, 2026

Robust Global-Local Behavior Arbitration via Continuous Command Fusion Under LiDAR Errors

Modular autonomous driving systems must coordinate global progress objectives with local safety-driven reactions under imperfect sensing and strict real-time constraints. This paper presents a ROS2-native arbitration module that continuously fuses the outputs of two unchanged and interpretable controllers: a global reference-tracking controller based on Pure Pursuit and a reactive LiDAR-based Gap Follow controller. At each control step, both controllers propose Ackermann commands, and a PPO-trained policy predicts a continuous gate from a compact feature observation to produce a single fused drive command, augmented with practical safety checks. For comparison under identical ROS topic inputs and control rate, we implement a lightweight sampling-based predictive baseline. Robustness is evaluated using a ROS2 impairment protocol that injects LiDAR noise, delay, and dropout, and additionally sweeps forward-cone false short-range outliers. In a repeatable close-proximity passing scenario, we report safe success and failure rates together with per-step end-to-end controller runtime as sensing stress increases. The study is intended as a command-level robustness evaluation in a modular ROS2 setting, not as a replacement for planning-level interaction reasoning.

Mohamed Elgouhary, Amr S. El-Wakeel

The Authors are with the Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA.

Autonomous DrivingAutonomous Driving Safety EvaluationRL for Autonomous Driving
AskPDF
Mar 28, 2026

Autonomous overtaking trajectory optimization using reinforcement learning and opponent pose estimation

Vehicle overtaking is one of the most complex driving maneuvers for autonomous vehicles. To achieve optimal autonomous overtaking, driving systems rely on multiple sensors that enable safe trajectory optimization and overtaking efficiency. This paper presents a reinforcement learning mechanism for multi-agent autonomous racing environments, enabling overtaking trajectory optimization, based on LiDAR and depth image data. The developed reinforcement learning agent uses pre-generated raceline data and sensor inputs to compute the steering angle and linear velocity for optimal overtaking. The system uses LiDAR with a 2D detection algorithm and a depth camera with YOLO-based object detection to identify the vehicle to be overtaken and its pose. The LiDAR and the depth camera detection data are fused using a UKF for improved opponent pose estimation and trajectory optimization for overtaking in racing scenarios. The results show that the proposed algorithm successfully performs overtaking maneuvers in both simulation and real-world experiments, with pose estimation RMSE of (0.0816, 0.0531) m in (x, y).

Matej Rene Cihlar, Luka Šiktar, Branimir Ćaran +1

University of Zagreb, Zagreb 10000, Croatia, Faculty of mechanical engineering and naval architecture

Autonomous DrivingAutonomous RacingRL for Autonomous DrivingAutonomous Driving Perception
AskPDF
Mar 26, 2026

Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Control

Recent advancements in computer vision have accelerated the development of autonomous driving. Despite these advancements, training machines to drive in a way that aligns with human expectations remains a significant challenge. Human factors are still essential, as humans possess a sophisticated cognitive system capable of rapidly interpreting scene information and making accurate decisions. Aligning machine with human intent has been explored with Reinforcement Learning with Human Feedback (RLHF). Conventional RLHF methods rely on collecting human preference data by manually ranking generated outputs, which is time-consuming and indirect. In this work, we propose an electroencephalography (EEG)-guided decision-making framework to incorporate human cognitive insights without behaviour response interruption into reinforcement learning (RL) for autonomous driving. We collected EEG signals from 20 participants in a realistic driving simulator and analyzed event-related potentials (ERP) in response to sudden environmental changes. Our proposed framework employs a neural network to predict the strength of ERP based on the cognitive information from visual scene information. Moreover, we explore the integration of such cognitive information into the reward signal of the RL algorithm. Experimental results show that our framework can improve the collision avoidance ability of the RL algorithm, highlighting the potential of neuro-cognitive feedback in enhancing autonomous driving systems. Our project page is: https://alex95gogo.github.io/Cognitive-Reward/.

Zhuoli Zhuang, Yu-Cheng Chang, Yu-Kai Wang +2

University of Technology Sydney

RL for Autonomous DrivingReinforcement LearningRL from Human FeedbackMultimodal Reward Modeling
AskPDF
Mar 26, 2026

Macroscopic Characteristics of Mixed Traffic Flow with Deep Reinforcement Learning Based Automated and Human-Driven Vehicles

Automated Vehicle (AV) control in mixed traffic, where AVs coexist with human-driven vehicles, poses significant challenges in balancing safety, efficiency, comfort, fuel efficiency, and compliance with traffic rules while capturing heterogeneous driver behavior. Traditional car-following models, such as the Intelligent Driver Model (IDM), often struggle to generalize across diverse traffic scenarios and typically do not account for fuel efficiency, motivating the use of learning-based approaches. Although Deep Reinforcement Learning (DRL) has shown strong microscopic performance in car-following conditions, its macroscopic traffic flow characteristics remain underexplored. This study focuses on analyzing the macroscopic traffic flow characteristics and fuel efficiency of DRL-based models in mixed traffic. A Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is implemented for AVs' control and trained using the NGSIM highway dataset, enabling realistic interaction with human-driven vehicles. Traffic performance is evaluated using the Fundamental Diagram (FD) under varying driver heterogeneity, heterogeneous time-gap penetration levels, and different shares of RL-controlled vehicles. A macroscopic level comparison of fuel efficiency between the RL-based AV model and the IDM is also conducted. Results show that traffic performance is sensitive to the distribution of safe time gaps and the proportion of RL vehicles. Transitioning from fully human-driven to fully RL-controlled traffic can increase road capacity by approximately 7.52%. Further, RL-based AVs also improve average fuel efficiency by about 28.98% at higher speeds (above 50 km/h), and by 1.86% at lower speeds (below 50 km/h) compared to the IDM. Overall, the DRL framework enhances traffic capacity and fuel efficiency without compromising safety.

Pankaj Kumar, Pranamesh Chakraborty, Subrahmanya Swamy Peruru

Department of Civil Engineering, Indian Institute of Technology Kanpur, Kanpur-208016, U.P., India · Department of Electrical Engineering Indian Institute of Technology Kanpur, Kanpur-208016, U.P., India

Intelligent Transportation SystemsRL for Autonomous DrivingAutonomous Driving Simulation
AskPDF
Mar 26, 2026

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems

Multi-Agent Self-Driving (MASD) systems provide an effective solution for coordinating autonomous vehicles to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems. Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel collaborative(CO-) interaction-aware(-IN) MARL framework, named COIN. Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic policy gradient (CIG-TD3) algorithm, crafted in a "centralized training, decentralized execution" (CTDE) manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations. Supplementary videos are available at https://marmotlab.github.io/COIN/

Yifeng Zhang, Jieming Chen, Tingguang Zhou +4

Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

Centralized Training with Decentralized ExecutionMulti-Agent CoordinationRL for Autonomous DrivingCooperative MARLMulti-Agent Reinforcement Learning
AskPDF
Mar 25, 2026

DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving

We introduce DreamerAD, the first latent world model framework that enables efficient reinforcement learning for autonomous driving by compressing diffusion sampling from 100 steps to 1 - achieving 80x speedup while maintaining visual interpretability. Training RL policies on real-world driving data incurs prohibitive costs and safety risks. While existing pixel-level diffusion world models enable safe imagination-based training, they suffer from multi-step diffusion inference latency (2s/frame) that prevents high-frequency RL interaction. Our approach leverages denoised latent features from video generation models through three key mechanisms: (1) shortcut forcing that reduces sampling complexity via recursive multi-resolution step compression, (2) an autoregressive dense reward model operating directly on latent representations for fine-grained credit assignment, and (3) Gaussian vocabulary sampling for GRPO that constrains exploration to physically plausible trajectories. DreamerAD achieves 87.7 EPDMS on NavSim v2, establishing state-of-the-art performance and demonstrating that latent-space RL is effective for autonomous driving.

Pengxuan Yang, Yupeng Zheng, Deheng Qian +11

Institute of Automation, CAS · Chongqing Chang’an Technology Co., Ltd · School of Advanced Interdisciplinary Sciences, UCAS +1

RL for Autonomous DrivingLatent World ModelsModel-Based RLDiffusion Model Acceleration
AskPDF
Mar 23, 2026

Partial Attention in Deep Reinforcement Learning for Safe Multi-Agent Control

Attention mechanisms excel at learning sequential patterns by discriminating data based on relevance and importance. This provides state-of-the-art performance in advanced generative artificial intelligence models. This paper applies this concept of an attention mechanism for multi-agent safe control. We specifically consider the design of a neural network to control autonomous vehicles in a highway merging scenario. The environment is modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). Within a QMIX framework, we include partial attention for each autonomous vehicle, thus allowing each ego vehicle to focus on the most relevant neighboring vehicles. Moreover, we propose a comprehensive reward signal that considers the global objectives of the environment (e.g., safety and vehicle flow) and the individual interests of each agent. Simulations are conducted in the Simulation of Urban Mobility (SUMO). The results show better performance compared to other driving algorithms in terms of safety, driving speed, and reward.

Turki Bin Mohaya, Peter Seiler

Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109, USA.

RL for Autonomous DrivingMulti-Agent Reinforcement LearningSafe RLDecentralized MARL
AskPDF
Mar 18, 2026

DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving

Traditional reinforcement learning (RL) methods rely on manually engineered rewards or sparse collision signals, which fail to capture the rich contextual understanding required for safe driving and make unsafe exploration unavoidable in real-world settings. Recent vision-language models (VLMs) offer promising semantic understanding capabilities; however, their high inference latency and susceptibility to hallucination hinder direct application to real-time vehicle control. To address these limitations, this paper proposes DriveVLM-RL, a neuroscience-inspired framework that integrates VLMs into RL through a dual-pathway architecture for safe and deployable autonomous driving. Inspired by the human brain's habitual and deliberative visual processing, DriveVLM-RL decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment via CLIP-based contrasting language goals, and a Dynamic Pathway for attention-gated multi-frame semantic risk reasoning via a lightweight detection model and large VLM (LVLM). A hierarchical reward synthesis mechanism fuses these signals with vehicle state information, while an asynchronous training pipeline decouples expensive LVLM inference from environment interaction. Critically, all VLM components operate exclusively during offline training and are completely removed at deployment, eliminating inference latency at test time. Extensive experiments in the CARLA simulator demonstrate that DriveVLM-RL significantly outperforms state-of-the-art baselines in collision avoidance and task success, attaining the highest success rate while reducing collision severity from 10.09 to 1.75 km/h relative to the strongest VLM-based baseline. The demo video, code, and model checkpoints are available at: https://zilin-huang.github.io/DriveVLM-RL-website/

Zilin Huang, Zihao Sheng, Zhengyang Wan +4

Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA · Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Civil Engineering, McGill University, Montreal, QC, H3A 0C3, Canada

VLMs for Autonomous DrivingLLM-Guided RLMultimodal Reward ModelingRL for Autonomous DrivingSafe RL
AskPDF
Mar 16, 2026

CorrectionPlanner: Self-Correction Planner with Reinforcement Learning in Autonomous Driving

Autonomous driving requires safe planning, but most learning-based planners lack explicit self-correction ability: once an unsafe action is proposed, there is no mechanism to correct it. Thus, we propose CorrectionPlanner, an autoregressive planner with self-correction that models planning as motion-token generation within a propose, evaluate, and correct loop. At each planning step, the policy proposes an action, namely a motion token, and a learned collision critic predicts whether it will induce a collision within a short horizon. If the critic predicts a collision, we retain the sequence of historical unsafe motion tokens as a self-correction trace, generate the next motion token conditioned on it, and repeat this process until a safe motion token is proposed or the safety criterion is met. This self-correction trace, consisting of all unsafe motion tokens, represents the planner's correction process in motion-token space, analogous to a reasoning trace in language models. We train the planner with imitation learning followed by model-based reinforcement learning using rollouts from a pretrained world model that realistically models agents' reactive behaviors. Closed-loop evaluations show that CorrectionPlanner reduces collision rate by over 20% on Waymax and achieves state-of-the-art planning scores on nuPlan.

Yihong Guo, Dongqiangzi Ye, Sijia Chen +2

Department of Computer Science, Johns Hopkins University, Baltimore, USA. · XPENG Motors.

Autonomous Driving PlanningSafe Motion PlanningRL for Autonomous Driving
AskPDF
Mar 16, 2026

PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning

End-to-end autonomous driving policies based on Imitation Learning (IL) often struggle in closed-loop execution due to the misalignment between inadequate open-loop training objectives and real driving requirements. While Reinforcement Learning (RL) offers a solution by directly optimizing driving goals via reward signals, the rendering-based training environments introduce the rendering gap and are inefficient due to high computational costs. To overcome these challenges, we present a novel Pseudo-simulation-based RL method for closed-loop end-to-end autonomous driving, PerlAD. Based on offline datasets, PerlAD constructs a pseudo-simulation that operates in vector space, enabling efficient, rendering-free trial-and-error training. To bridge the gap between static datasets and dynamic closed-loop environments, PerlAD introduces a prediction world model that generates reactive agent trajectories conditioned on the ego vehicle's plan. Furthermore, to facilitate efficient planning, PerlAD utilizes a hierarchical decoupled planner that combines IL for lateral path generation and RL for longitudinal speed optimization. Comprehensive experimental results demonstrate that PerlAD achieves state-of-the-art performance on the Bench2Drive benchmark, surpassing the previous E2E RL method by 10.29% in Driving Score without requiring expensive online interactions. Additional evaluations on the DOS benchmark further confirm its reliability in handling safety-critical occlusion scenarios.

Yinfeng Gao, Qichao Zhang, Deqing Liu +8

School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China. · Xiaomi EV. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.

End-to-End Autonomous DrivingRL for Autonomous DrivingAutonomous Driving PlanningAutonomous Driving SimulationWorld Models for Autonomous Driving
AskPDF
Mar 14, 2026

Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving

End-to-end autonomous driving is typically built upon imitation learning (IL), yet its performance is constrained by the quality of human demonstrations. To overcome this limitation, recent methods incorporate reinforcement learning (RL) through sequential fine-tuning. However, such a paradigm remains suboptimal: sequential RL fine-tuning can introduce policy drift and often leads to a performance ceiling due to its dependence on the pretrained IL policy. To address these issues, we propose PaIR-Drive, a general Parallel framework for collaborative Imitation and Reinforcement learning in end-to-end autonomous driving. During training, PaIR-Drive separates IL and RL into two parallel branches with conflict-free training objectives, enabling fully collaborative optimization. This design eliminates the need to retrain RL when applying a new IL policy. During inference, RL leverages the IL policy to further optimize the final plan, allowing performance beyond prior knowledge of IL. Furthermore, we introduce a tree-structured trajectory neural sampler to group relative policy optimization (GRPO) in the RL branch, which enhances exploration capability. Extensive analysis on NAVSIMv1 and v2 benchmark demonstrates that PaIR-Drive achieves Competitive performance of 91.2 PDMS and 87.9 EPDMS, building upon Transfuser and DiffusionDrive IL baselines. PaIR-Drive consistently outperforms existing RL fine-tuning methods, and could even correct human experts' suboptimal behaviors. Qualitative results further confirm that PaIR-Drive can effectively explore and generate high-quality trajectories.

Zhexi Lian, Haoran Wang, Xuerun Yan +4

Tongji University · Nanyang Technological University · Chery Automobile

End-to-End Autonomous DrivingRL for Autonomous DrivingImitation Learning
AskPDF
Mar 11, 2026

Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planning

Vision-language models (VLMs) enhance the planning capability of end-to-end (E2E) driving policy by leveraging high-level semantic reasoning. However, existing approaches often overlook the dual-system consistency between VLM's high-level decision and E2E's low-level planning. As a result, the generated trajectories may misalign with the intended driving decisions, leading to weakened top-down guidance and decision-following ability of the system. To address this issue, we propose Senna-2, an advanced VLM-E2E driving policy that explicitly aligns the two systems for consistent decision-making and planning. Our method follows a consistency-oriented three-stage training paradigm. In the first stage, we conduct driving pre-training to achieve preliminary decision-making and planning, with a decision adapter transmitting VLM decisions to E2E policy in the form of implicit embeddings. In the second stage, we align the VLM and the E2E policy in an open-loop setting. In the third stage, we perform closed-loop alignment via bottom-up Hierarchical Reinforcement Learning in 3DGS environments to reinforce the safety and efficiency. Extensive experiments demonstrate that Senna-2 achieves superior dual-system consistency (19.3% F1 score improvement) and significantly enhances driving safety in both open-loop (5.7% FDE reduction) and closed-loop settings (30.6% AF-CR reduction).

Yuehao Song, Shaoyu Chen, Hao Gao +8

Huazhong University of Science & Technology · Horizon Robotics

End-to-End Autonomous DrivingAutonomous Driving PlanningVLMs for Autonomous DrivingRL for Autonomous Driving
AskPDF
Mar 8, 2026

Emergency Lane-Change Simulation: A Behavioral Guidance Approach for Risky Scenario Generation

In contemporary autonomous driving testing, virtual simulation has become an important approach due to its efficiency and cost effectiveness. However, existing methods usually rely on reinforcement learning to generate risky scenarios, making it difficult to efficiently learn realistic emergency behaviors. To address this issue, we propose a behavior guided method for generating high risk lane change scenarios. First, a behavior learning module based on an optimized sequence generative adversarial network is developed to learn emergency lane change behaviors from an extracted dataset. This design alleviates the limitations of existing datasets and improves learning from relatively few samples. Then, the opposing vehicle is modeled as an agent, and the road environment together with surrounding vehicles is incorporated into the operating environment. Based on the Recursive Proximal Policy Optimization strategy, the generated trajectories are used to guide the vehicle toward dangerous behaviors for more effective risk scenario exploration. Finally, the reference trajectory is combined with model predictive control as physical constraints to continuously optimize the strategy and ensure physical authenticity. Experimental results show that the proposed method can effectively learn high risk trajectory behaviors from limited data and generate high risk collision scenarios with better efficiency than traditional methods such as grid search and manual design.

Chen Xiong, Cheng Wang, Yuhang Liu +2

School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen 518107, China · Guangdong Provincial Key Laboratory of Intelligent Transportation System, Sun Yat-sen University, Guangzhou 510275, China · Department of Civil and Environmental Engineering, The Pennsylvania State University, State College, PA 16802, USA +1

Autonomous Driving Scenario GenerationAutonomous Driving SimulationAdversarial Scenario GenerationAutonomous DrivingRL for Autonomous Driving
AskPDF
Mar 6, 2026

Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models

We identify a fundamental Narrow Policy limitation undermining the performance of autonomous VLA models, where driving Imitation Learning (IL) tends to collapse exploration and limit the potential of subsequent Reinforcement Learning (RL) stages, which often saturate prematurely due to insufficient feedback diversity. Thereby, we propose Curious-VLA, a framework that alleviates the exploit-explore dilemma through a two-stage design. During IL, we introduce a Feasible Trajectory Expansion (FTE) strategy to generate multiple physically valid trajectories and a step-wise normalized trajectory representation to adapt this diverse data. In the RL stage, we present Adaptive Diversity-Aware Sampling (ADAS) that prioritizes high-diversity samples and introduce Spanning Driving Reward (SDR) with a focal style weighting to amplify reward's value span for improving sensitivity to driving quality. On the Navsim benchmark, Curious-VLA achieves SoTA results (PDMS 90.3, EPDMS 85.4) and a Best-of-N PDMS of 94.8, demonstrating its effectiveness in unlocking the exploratory potential of VLA models. Code: https://github.com/Mashiroln/curious_vla.git.

Canyu Chen, Yuguang Yang, Zhewen Tan +10

National Superior College for Engineers, Beihang University · Institute for AI Industry Research, Tsinghua University · School of Electronic Information Engineering, Beihang University +5

Autonomous DrivingVLMs for Autonomous DrivingRL ExplorationRL for Autonomous Driving
AskPDF
Mar 6, 2026

TADPO: Reinforcement Learning Goes Off-road

Off-road autonomous driving poses significant challenges such as navigating unmapped, variable terrain with uncertain and diverse dynamics. Addressing these challenges requires effective long-horizon planning and adaptable control. Reinforcement Learning (RL) offers a promising solution by learning control policies directly from interaction. However, because off-road driving is a long-horizon task with low-signal rewards, standard RL methods are challenging to apply in this setting. We introduce TADPO, a novel policy gradient formulation that extends Proximal Policy Optimization (PPO), leveraging off-policy trajectories for teacher guidance and on-policy trajectories for student exploration. Building on this, we develop a vision-based, end-to-end RL system for high-speed off-road driving, capable of navigating extreme slopes and obstacle-rich terrain. We demonstrate our performance in simulation and, importantly, zero-shot sim-to-real transfer on a full-scale off-road vehicle. To our knowledge, this work represents the first deployment of RL-based policies on a full-scale off-road platform.

Zhouchonghao Wu, Raymond Song, Vedant Mundheda +3

Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 · National Robotics Engineering Center, Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15201

End-to-End Autonomous DrivingAutonomous DrivingProximal Policy OptimizationRL for Autonomous DrivingOff-Policy RL
AskPDF
Mar 6, 2026

Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving

Autonomous driving involves multiple, often conflicting objectives such as safety, efficiency, and comfort. In reinforcement learning (RL), these objectives are typically combined through weighted summation, which collapses their relative priorities and often yields policies that violate safety-critical constraints. To overcome this limitation, we introduce the Preordered Multi-Objective MDP (Pr-MOMDP), which augments standard MOMDPs with a preorder over reward components. This structure enables reasoning about actions with respect to a hierarchy of objectives rather than a scalar signal. To make this structure actionable, we extend distributional RL with a novel pairwise comparison metric, Quantile Dominance (QD), that evaluates action return distributions without reducing them into a single statistic. Building on QD, we propose an algorithm for extracting optimal subsets, the subset of actions that remain non-dominated under each objective, which allows precedence information to shape both decision-making and training targets. Our framework is instantiated with Implicit Quantile Networks (IQN), establishing a concrete implementation while preserving compatibility with a broad class of distributional RL methods. Experiments in Carla show improved success rates, fewer collisions and off-road events, and deliver statistically more robust policies than IQN and ensemble-IQN baselines. By ensuring policies respect rewards preorder, our work advances safer, more reliable autonomous driving systems.

Ahmed Abouelazm, Jonas Michel, Daniel Bogdoll +2

FZI Research Center for Information Technology, Germany · Karlsruhe Institute of Technology, Germany

Multi-Objective Reinforcement LearningRL for Autonomous DrivingSafe RLDistributional RL
AskPDF
Mar 3, 2026

Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving

Reinforcement learning (RL) is a fundamental methodology in autonomous driving systems, where generative policies exhibit considerable potential by leveraging their ability to model complex distributions to enhance exploration. However, their inherent high inference latency severely impedes their deployment in real-time decision-making and control. To address this issue, we propose diffusion actor-critic with entropy regulator via flow matching (DACER-F) by introducing flow matching into online RL, enabling the generation of competitive actions in a single inference step. By leveraging Langevin dynamics and gradients of the Q-function, DACER-F dynamically optimizes actions from experience replay toward a target distribution that balances high Q-value information with exploratory behavior. The flow policy is then trained to efficiently learn a mapping from a simple prior distribution to this dynamic target. In complex multi-lane and intersection simulations, DACER-F outperforms baselines diffusion actor-critic with entropy regulator (DACER) and distributional soft actor-critic (DSAC), while maintaining an ultra-low inference latency. DACER-F further demonstrates its scalability on standard RL benchmark DeepMind Control Suite (DMC), achieving a score of 775.8 in the humanoid-stand task and surpassing prior methods. Collectively, these results establish DACER-F as a high-performance and computationally efficient RL algorithm.

Tianze Zhu, Yinuo Wang, Wenjun Zou +6

School of Vehicle and Mobility, Tsinghua University, Beijing, China · College of Artificial Intelligence, Tsinghua University, Beijing, China

Reinforcement LearningRL for Autonomous DrivingRL for Generative Models
AskPDF
Mar 1, 2026

Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures

Vision-Language-Action (VLA) models for autonomous driving often hit a performance plateau during Reinforcement Learning (RL) optimization. This stagnation arises from exploration capabilities constrained by previous Supervised Fine-Tuning (SFT), leading to persistent failures in long-tail scenarios. In these critical situations, all explored actions yield a zero-value driving score. This information-sparse reward signals a failure, yet fails to identify its root cause -- whether it is due to incorrect planning, flawed reasoning, or poor trajectory execution. To address this limitation, we propose VLA with Explicit Learning from Failures (ELF-VLA), a framework that augments RL with structured diagnostic feedback. Instead of relying on a vague scalar reward, our method produces detailed, interpretable reports that identify the specific failure mode. The VLA policy then leverages this explicit feedback to generate a Feedback-Guided Refinement. By injecting these corrected, high-reward samples back into the RL training batch, our approach provides a targeted gradient, which enables the policy to solve critical scenarios that unguided exploration cannot. Extensive experiments demonstrate that our method unlocks the latent capabilities of VLA models, achieving state-of-the-art (SOTA) performance on the public NAVSIM benchmark for overall PDMS, EPDMS score and high-level planning accuracy.

Yuechen Luo, Qimao Chen, Fang Li +5

Tsinghua University · University of Macau · Beijing Jiaotong University

VLMs for Autonomous DrivingRL for Autonomous DrivingAutonomous Driving PlanningRL for VLMs
AskPDF
Feb 27, 2026

TSC: Topology-Conditioned Stackelberg Coordination for Multi-Agent Reinforcement Learning in Interactive Driving

Safe and efficient autonomous driving in dense traffic is fundamentally a decentralized multi-agent coordination problem, where interactions at conflict points such as merging and weaving must be resolved reliably under partial observability. With only local and incomplete cues, interaction patterns can change rapidly, often causing unstable behaviors such as oscillatory yielding or unsafe commitments. Existing multi-agent reinforcement learning (MARL) approaches either adopt synchronous decision-making, which exacerbate non-stationarity, or depend on centralized sequencing mechanisms that scale poorly as traffic density increases. To address these limitations, we propose Topology-conditioned Stackelberg Coordination (TSC), a learning framework for decentralized interactive driving under communication-free execution, which extracts a time-varying directed priority graph from braid-inspired weaving relations between trajectories, thereby defining local leader-follower dependencies without constructing a global order of play. Conditioned on this graph, TSC endogenously factorizes dense interactions into graph-local Stackelberg subgames and, under centralized training and decentralized execution (CTDE), learns a sequential coordination policy that anticipates leaders via action prediction and trains followers through action-conditioned value learning to approximate local best responses, improving training stability and safety in dense traffic. Experiments across four dense traffic scenarios show that TSC achieves superior performance over representative MARL baselines across key metrics, most notably reducing collisions while maintaining competitive traffic efficiency and control smoothness.

Xiaotong Zhang, Gang Xiong, Yuanjing Wang +3

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Department of Natural Sciences, University of Durham, Durham DH1 3LE, United Kingdom +4

Multi-Agent CoordinationDecentralized MARLRL for Autonomous DrivingMulti-Agent Reinforcement LearningSequential MARL
AskPDF
Feb 20, 2026

Interacting safely with cyclists using Hamilton-Jacobi reachability and reinforcement learning

In this paper, we present a framework for enabling autonomous vehicles to interact with cyclists in a manner that balances safety and optimality. The approach integrates Hamilton-Jacobi reachability analysis with deep Q-learning to jointly address safety guarantees and time-efficient navigation. A value function is computed as the solution to a time-dependent Hamilton-Jacobi-Bellman inequality, providing a quantitative measure of safety for each system state. This safety metric is incorporated as a structured reward signal within a reinforcement learning framework. The method further models the cyclist's latent response to the vehicle, allowing disturbance inputs to reflect human comfort and behavioral adaptation. The proposed framework is evaluated through simulation and comparison with human driving behavior and an existing state-of-the-art method.

Aarati Andrea Noronha, Jean Oh

Autonomous Driving PlanningSafe RLRL for Autonomous Driving
AskPDF
Feb 16, 2026

DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving

Vision-Language-Action (VLA) models for autonomous driving increasingly adopt generative planners trained with imitation learning followed by reinforcement learning. Diffusion-based planners suffer from modality alignment difficulties, low training efficiency, and limited generalization. Token-based planners are plagued by cumulative causal errors and irreversible decoding. In summary, the two dominant paradigms exhibit complementary strengths and weaknesses. In this paper, we propose DriveFine, a masked diffusion VLA model that combines flexible decoding with self-correction capabilities. In particular, we design a novel plug-and-play block-MoE, which seamlessly injects a refinement expert on top of the generation expert. By enabling explicit expert selection during inference and gradient blocking during training, the two experts are fully decoupled, preserving the foundational capabilities and generic patterns of the pretrained weights, which highlights the flexibility and extensibility of the block-MoE design. Furthermore, we design a hybrid reinforcement learning strategy that encourages effective exploration of refinement expert while maintaining training stability. Extensive experiments on NAVSIM v1, v2, and Navhard benchmarks demonstrate that DriveFine exhibits strong efficacy and robustness. The code will be released at https://github.com/MSunDYY/DriveFine.

Chenxu Dang, Sining Ang, Yongkang Li +7

Huazhong University of Science and Technology · Xiaomi EV · Institute for AI Industry Research (AIR), Tsinghua University

VLMs for Autonomous DrivingDiffusion-Based PlanningMasked Diffusion ModelsRobustness of VLA ModelsMixture-of-Experts ModelsRL for Autonomous Driving
AskPDF
Jan 26, 2026

Preference-Conditioned Multi-Objective Reinforcement Learning for Tactical Decision Making in Heavy-Duty Vehicles

A central difficulty in applying reinforcement learning to real-world applications such as autonomous driving is that conventional scalar reward formulations obtained by aggregating multiple often competing objectives may obscure the structure of their trade-offs. These challenges are particularly relevant to decision-making for heavy-duty vehicles, which requires balancing safety, efficiency, and operational costs under demanding commercial requirements. We present a preference-conditioned multi-objective reinforcement learning framework that explicitly represents these objectives and utilizes Generalized Policy Improvement and Proximal Policy Optimization to learn policies capturing different trade-offs between them. The proposed method approximates the Pareto front across three conflicting objectives: safety, quantified in terms of collisions and successful completion; energy efficiency, quantified using energy cost; and time efficiency, quantified using driver cost. We evaluate the framework in a scalable highway traffic simulation environment tailored for tactical decision making in heavy-duty trucks. It yields smooth and interpretable Pareto fronts, and evaluations across traffic densities and against an analytical baseline show that the learned policies capture meaningful trade-offs. The preference-conditioned policy enables transitions between different driving behaviors without retraining, demonstrating the potential of multi-objective reinforcement learning for adaptive decision making in autonomous heavy-duty vehicles.

Deepthi Pathare, Leo Laine, Morteza Haghir Chehreghani

Department of Computer Science and Engineering Chalmers University of Technology and and University of Gothenburg · Department of Mechanics and Maritime Sciences Chalmers University of Technology

Multi-Objective Reinforcement LearningRL for Autonomous DrivingAutonomous Driving
AskPDF
Oct 28, 2025

Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorer

Human demonstrations are widely considered the cornerstone of end-to-end (E2E) autonomous driving despite human demonstration's scarcity for long-tail and safety-critical scenarios. Nonetheless, current E2E autonomous driving (AD) training paradigms continue to rely on human demonstrations. Imitation learning (IL) requires human demonstrations for training, whereas reinforcement learning (RL) has emerged as a promising alternative to reduce this dependency. However, most existing RL methods for E2E AD still rely implicitly on human demonstrations. A pure rewards-based RL method can overcome the need for human demonstrations, but general RL policy gradient methods suffer from the cold-start problem. In this paper, we propose ZTRS (Zero-human demonstration end-to-end autonomous driving with TRajectory Scorer) - a complete RL-based E2E planning paradigm trained solely on real-world images and rule-based rewards, entirely without human demonstration. Through our proposed Exhaustive Policy Optimization (EPO), a policy gradient variant tailored for enumerable trajectory actions and dense supervision, ZTRS enables the model to generalize better to long-tail driving scenarios. We demonstrate this generalization through our SOTA performance against IL approaches on both long-tail Navhard and closed-loop HUGSIM datasets. Project page: https://zhenxinli.net/ZTRS/.

Zhenxin Li, Nadine Chang, Wenhao Yao +9

Institute of Trustworthy Embodied AI, Fudan University · *Work done during an internship at NVIDIA. · NVIDIA +3

Long-Tail Autonomous DrivingReinforcement LearningEnd-to-End Autonomous DrivingAutonomous DrivingAutonomous Driving PlanningRL for Autonomous Driving
AskPDF
Aug 23, 2025

Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving

Personalized autonomous driving requires understanding the driving needs implicit in user requests. This is challenging when driving demonstrations lack style annotations and behavioral adaptation risks degrading planning quality. We propose M-Diffusion Planner with a dual-decoupling design: separating intent understanding from trajectory planning, and general planning capability learning from strategy alignment. A large language model infers driving needs and selects a strategy, which is executed by the corresponding head of a shared diffusion planner. Following imitation pretraining, strategy-specific rewards specialize individual heads without style annotations. We innovatively adapt Group Relative Policy Optimization using surrogate policy-ratio estimation and constrained head-specific updates to mitigate degradation of pretrained planning capabilities. Open-loop and closed-loop evaluations on nuPlan demonstrate distinct strategy-aligned behaviors and competitive planning performance. Further experiments assess intent-to-strategy mapping and collision risk, and analyze the effects of reward design and optimization on strategy alignment and planning quality.

Ding Fan, Xuewen Luo, Hwa Hui Tew +3

Monash University, Malaysia · University of Utah, USA · Monash University, Australia

Autonomous Driving PlanningAutonomous DrivingDiffusion-Based PlanningRL for Autonomous Driving
AskPDF
Jul 5, 2025

DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving

Most end-to-end autonomous driving methods rely on imitation learning from single expert demonstrations, often leading to conservative and homogeneous behaviors that limit generalization in complex real-world scenarios. In this work, we propose DIVER, an end-to-end driving framework that integrates reinforcement learning with diffusion-based generation to produce diverse and feasible trajectories. At the core of DIVER lies a reinforced diffusion-based generation mechanism. First, the model conditions on map elements and surrounding agents to generate multiple reference trajectories from a single ground-truth trajectory, alleviating the limitations of imitation learning that arise from relying solely on single expert demonstrations. Second, reinforcement learning is employed to guide the diffusion process, where reward-based supervision enforces safety and diversity constraints on the generated trajectories, thereby enhancing their practicality and generalization capability. Furthermore, to address the limitations of L2-based open-loop metrics in capturing trajectory diversity, we propose a novel Diversity metric to evaluate the diversity of multi-mode predictions.Extensive experiments on the closed-loop NAVSIM and Bench2Drive benchmarks, as well as the open-loop nuScenes dataset, demonstrate that DIVER significantly improves trajectory diversity, effectively addressing the mode collapse problem inherent in imitation learning.

Ziying Song, Lin Liu, Hongyu Pan +7

Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · School of Artificial Intelligence (School of Software), Yanshan University, Qinhuangdao, China · Horizon Robotics +3

End-to-End Autonomous DrivingDiffusion-Based PlanningImitation LearningRL for Diffusion ModelsRL for Autonomous DrivingTrajectory Prediction
AskPDF
Mar 31, 2025

A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective

Reinforcement learning (RL), with its ability to explore and optimize policies in complex, dynamic decision-making tasks, has emerged as a promising approach to addressing motion planning (MoP) challenges in autonomous driving (AD). Despite rapid advancements in RL and AD, a systematic description and interpretation of the RL design process tailored to diverse driving tasks remains underdeveloped. This survey provides a comprehensive review of RL-based MoP for AD, focusing on lessons from task-specific perspectives. We first outline the fundamentals of RL methodologies, and then survey their applications in MoP, analyzing scenario-specific features and task requirements to shed light on their influence on RL design choices. Building on this analysis, we summarize key design experiences, extract insights from various driving task applications, and provide guidance for future implementations. Additionally, we examine the frontier challenges in RL-based MoP, review recent efforts to addresse these challenges, and propose strategies for overcoming unresolved issues.

Zhuoren Li, Guizhe Jin, Ran Yu +8

College of Automotive and Energy Engineering, Tongji University, Shanghai 201804, China

Autonomous Driving PlanningRL for Autonomous Driving
AskPDF
Feb 17, 2025

Does Knowledge About Perceptual Uncertainty Help an Agent in Automated Driving?

Agents in real-world scenarios like automated driving deal with uncertainty in their environment, in particular due to perceptual uncertainty. Although, reinforcement learning is dedicated to autonomous decision-making under uncertainty these algorithms are typically not informed about the uncertainty currently contained in their environment. On the other hand, uncertainty estimation for perception itself is typically directly evaluated in the perception domain, e.g., in terms of false positive detection rates or calibration errors based on camera images. Its use for deciding on goal-oriented actions remains largely unstudied. In this paper, we investigate how an agent's behavior is influenced by an uncertain perception and how this behavior changes if information about this uncertainty is available. Therefore, we consider a proxy task, where the agent is rewarded for driving a route as fast as possible without colliding with other road users. For controlled experiments, we introduce uncertainty in the observation space by perturbing the perception of the given agent while informing the latter. Our experiments show that an unreliable observation space modeled by a perturbed perception leads to a defensive driving behavior of the agent. Furthermore, when adding the information about the current uncertainty directly to the observation space, the agent adapts to the specific situation and in general accomplishes its task faster while, at the same time, accounting for risks.

Natalie Grabowsky, Annika Mütze, Joshua Wendland +2

Institute of Mathematics Technical University of Berlin Berlin, Germany · Department of Mathematics University of Wuppertal Wuppertal, Germany · Faculty of Computer Science Ruhr University Bochum Bochum, Germany

Autonomous Driving PlanningRL for Autonomous DrivingReinforcement Learning
AskPDF
Aug 18, 2023

Fading Expert Guidance: Bridging Model-Based and Learning-Based Control for Abortable Autonomous Overtaking

Overtaking on two-lane roads is a safety-critical decision-making problem for autonomous vehicles, since oncoming traffic may force the ego vehicle to abort and merge back to its original lane. Deep reinforcement learning (DRL) is promising for such continuous-control tasks, but it requires substantial interaction data and may produce unsafe exploratory actions during training. Model-based controllers provide structured behavior, but depend on model fidelity and hand-designed logic. This paper proposes a fading expert-guidance framework that uses a model-based controller to guide DRL training for autonomous overtaking. The expert combines a constrained iterative LQR (CiLQR) planner with PID-based auxiliary controllers, activated when the optimized plan becomes infeasible. The expert action enters the actor objective through a fading term, interpreted as a time-varying soft trust region around the expert policy. This term first biases the policy toward the expert and then vanishes, allowing optimization according to the reinforcement-learning objective. The method is evaluated with PPO, TD3, and SAC. Simulations with bidirectional traffic show improved sample efficiency and final task performance. The safety effects are algorithm-dependent: guidance reduces vehicle collisions for SAC, eliminates boundary collisions for TD3, and leaves the PPO collision profile unchanged. Overall, the results support fading expert guidance as a practical mechanism for transferring model-based driving knowledge to learning-based controllers without restricting the final policy to imitation in an abortable overtaking task with bidirectional traffic.

Jinxiong Lu, Gokhan Alcan, Ville Kyrki

Intelligent Robotics Group, Department of Electrical Engineering and Automation, Aalto University, 02150 Espoo, Finland

Autonomous Driving PlanningRL for Autonomous Driving
AskPDF
Dec 12, 2022

Security of Deep Reinforcement Learning for Autonomous Driving: A Survey

Reinforcement learning (RL) enables agents to learn optimal behaviors through interaction with their environment and has been increasingly deployed in safety-critical applications, including autonomous driving. Despite its promise, RL is susceptible to attacks designed either to compromise policy learning or to induce erroneous decisions by trained agents. Although the literature on RL security has grown rapidly and several surveys exist, existing categorizations often fall short in guiding the selection of appropriate defenses for specific systems. In this work, we present a comprehensive survey of 86 recent studies on RL security, addressing these limitations by systematically categorizing attacks and defenses according to defined threat models and single- versus multi-agent settings. Furthermore, we examine the relevance and applicability of state-of-the-art attacks and defense mechanisms within the context of autonomous driving, providing insights to inform the design of robust RL systems.

Ambra Demontis, Srishti Gupta, Maura Pintor +6

University of Cagliari · Sapienza, University of Rome · University of Genoa +3

Robust RLRL for Autonomous DrivingAdversarial MARL
AskPDF