Autonomous Driving

Follow this topic Ask about this topic

Autonomous driving studies how vehicles can sense their surroundings, anticipate other road users, and choose safe, effective actions in traffic. Systems range from modular stacks that connect camera, LiDAR, and radar perception to prediction and model-predictive control, to end-to-end policies trained with imitation learning, reinforcement learning, or vision-language and world models. Researchers evaluate these methods on datasets such as nuScenes, Waymo, and Argoverse, in simulators such as CARLA and nuPlan, and through physical vehicle tests. A central challenge is turning strong benchmark results into reliable closed-loop behavior across rare events, changing conditions, and real-world deployment constraints.

Latest papers

538 papers53 papers in the last four weeks, down 9% on the four weeks before. 0.3% of all new papers.

Feb 9, 2026

karl. -- A Research Vehicle for Automated and Connected Driving

As highly automated driving is transitioning from single-vehicle closed-access testing to commercial deployments of public ride-hailing in selected areas (e.g., Waymo), automated driving and connected cooperative intelligent transport systems (C-ITS) remain active fields of research. Even though simulation is omnipresent in the development and validation life cycle of automated and connected driving technology, the complex nature of public road traffic and software that masters it still requires real-world integration and testing with actual vehicles. Dedicated vehicles for research and development allow testing and validation of software and hardware components under real-world conditions early on. They also enable collecting and publishing real-world datasets that let others conduct research without vehicle access, and support early demonstration of futuristic use cases. In this paper, we present karl., our new research vehicle for automated and connected driving. Apart from major corporations, few institutions worldwide have access to their own L4-capable research vehicles, restricting their ability to carry out independent research. This paper aims to help bridge that gap by sharing the reasoning, design choices, and technical details that went into making karl. a flexible and powerful platform for research, engineering, and validation in the context of automated and connected driving. More impressions of karl. are available at https://karl.ac.

Jean-Pierre Busch, Lukas Ostendorf, Guido Linden +5

Institute for Automotive Engineering (ika), RWTH Aachen University, Germany

Intelligent Transportation Systems Autonomous Driving

Feb 9, 2026

Learning to Drive in New Cities Without Human Demonstrations

While autonomous vehicles have achieved reliable performance within specific operating regions, their deployment to new cities remains costly and slow. A key bottleneck is the need to collect many human demonstration trajectories when adapting driving policies to new cities that differ from those seen in training in terms of road geometry, traffic rules, and interaction patterns. In this paper, we show that self-play multi-agent reinforcement learning can adapt a driving policy to a substantially different target city using only the map and meta-information, without requiring any human demonstrations from that city. We introduce NO data Map-based self-play for Autonomous Driving (NOMAD), which enables policy adaptation in a simulator constructed based on the target-city map. Using a simple reward function, NOMAD substantially improves both task success rate and trajectory realism in target cities, demonstrating an effective and scalable alternative to data-intensive city-transfer methods. Project page: https://zerlinwang.github.io/NOMAD/

Zilin Wang, Saeed Rahmani, Daphne Cornelisse +4

WhiRL, University of Oxford · FLAIR, University of Oxford · Delft University of Technology +1

Autonomous Driving Self-Play RL Autonomous Driving Simulation

Feb 1, 2026

HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving

End-to-end autonomous driving models increasingly benefit from large vision-language models for semantic understanding, yet safe and reliable planning under long-tail conditions remains challenging, particularly in mixed-traffic environments involving heterogeneous road users and rare safety-critical interactions. This paper proposes HERMES, a holistic risk-aware end-to-end multimodal driving framework that explicitly incorporates long-tail semantic knowledge into trajectory planning. HERMES employs a foundation-model-assisted annotation pipeline to construct structured Long-Tail Scene Context and Long-Tail Planning Context, capturing hazard-centric scene information, maneuver intent, and risk-aware planning guidance. A Tri-Modal Driving Module then integrates multi-view visual observations, historical ego-motion, and long-tail semantic instructions through intent- and risk-aware conditioning for trajectory generation. Extensive experiments on a large-scale real-world long-tail driving benchmark demonstrate consistent improvements over representative recent baselines in overall planning performance and across diverse safety-critical scenarios. Ablation studies further validate the effectiveness and complementary roles of the major components within HERMES.

Weizhe Tang, Junwei You, Jiaxi Liu +7

Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53711 USA

Long-Tail Autonomous Driving VLMs for Autonomous Driving End-to-End Autonomous Driving Autonomous Driving Risk-Aware Path Planning Safe Motion Planning+1

Jan 29, 2026

FlexMap: Robust HD Map Construction under Flexible Camera Configurations

High-definition (HD) maps provide essential semantic information about road structures for autonomous driving, but existing HD map construction methods typically require calibrated multi-camera rigs and explicit 2D-to-BEV transformations. Such pipelines degrade when camera views are missing or pose estimates are inaccurate, limiting their use across heterogeneous fleet configurations. We introduce FlexMap, a vectorized HD mapping framework that adapts to varying camera configurations without architectural changes or per-configuration retraining and does not require camera parameters as model input. FlexMap replaces explicit geometric projection with a geometry foundation model that encodes cross-view 3D structure. A spatial-temporal enhancement module then separates cross-view spatial reasoning from temporal aggregation, while a camera-aware decoder uses the token produced for each input view to adapt its attention without camera poses. Experiments on nuScenes and Argoverse 2 show that FlexMap outperforms pose-dependent baselines using estimated poses and maintains comparable accuracy across all evaluated camera configurations, including those with missing views.

Run Wang, Chaoyi Zhou, Amir Salarpour +5

School of Computing, Clemson University · 2Tacoma School of Engineering and Technology, University of Washington

HD Map Construction Autonomous Driving

Jan 26, 2026

Preference-Conditioned Multi-Objective Reinforcement Learning for Tactical Decision Making in Heavy-Duty Vehicles

A central difficulty in applying reinforcement learning to real-world applications such as autonomous driving is that conventional scalar reward formulations obtained by aggregating multiple often competing objectives may obscure the structure of their trade-offs. These challenges are particularly relevant to decision-making for heavy-duty vehicles, which requires balancing safety, efficiency, and operational costs under demanding commercial requirements. We present a preference-conditioned multi-objective reinforcement learning framework that explicitly represents these objectives and utilizes Generalized Policy Improvement and Proximal Policy Optimization to learn policies capturing different trade-offs between them. The proposed method approximates the Pareto front across three conflicting objectives: safety, quantified in terms of collisions and successful completion; energy efficiency, quantified using energy cost; and time efficiency, quantified using driver cost. We evaluate the framework in a scalable highway traffic simulation environment tailored for tactical decision making in heavy-duty trucks. It yields smooth and interpretable Pareto fronts, and evaluations across traffic densities and against an analytical baseline show that the learned policies capture meaningful trade-offs. The preference-conditioned policy enables transitions between different driving behaviors without retraining, demonstrating the potential of multi-objective reinforcement learning for adaptive decision making in autonomous heavy-duty vehicles.

Deepthi Pathare, Leo Laine, Morteza Haghir Chehreghani

Department of Computer Science and Engineering Chalmers University of Technology and and University of Gothenburg · Department of Mechanics and Maritime Sciences Chalmers University of Technology

Multi-Objective Reinforcement Learning RL for Autonomous Driving Autonomous Driving

Dec 30, 2025

Evolutionary Physics-Informed Temporal Fusion for Lane-Change Intention Prediction

Early lane-change intention prediction is essential for autonomous driving and ADAS, but it remains challenging because lane-changing behavior depends on evolving traffic risk, surrounding-vehicle interactions, and target-lane feasibility rather than only instantaneous vehicle states. This study proposes an evolutionary physics-informed temporal fusion framework for three-class lane-change intention prediction, including left lane change, right lane change, and no lane change. Instead of using static physics-informed variables alone, the proposed method derives temporal descriptors from conventional traffic signals, including risk evolution, gap persistence, counterfactual lane utility, interaction pressure gradient, maneuver feasibility, and intent consistency. These descriptors are fused with temporal embeddings learned from raw trajectory sequences through a sequence encoder, and the fused representation is used for final classification. Experiments are conducted on the highD and exiD datasets under 1,s, 2,s, and 3,s prediction horizons. The proposed model achieves Macro F1-scores of 0.9514, 0.9256, and 0.8872 on highD, and 0.9386, 0.9070, and 0.8531 on exiD, respectively. The improvement is especially pronounced in exiD ramp-adjacent scenarios, indicating that temporal physical evolution is particularly useful in interaction-rich environments. These results demonstrate that combining evolutionary physics-informed descriptors with learned temporal representations provides a more dynamic and interpretable solution for early lane-change intention prediction.

Jiazhao Shi, Qiyang Xie, Ziyu Wang +7

Tandon School of Engineering New York University New York, USA · Khoury College of Computer Science Northeastern University Boston, USA · School of Business Wake Forest University Winston Salem, USA +6

Autonomous Driving Physics-Informed ML

Dec 15, 2025

A Convex Obstacle Avoidance Formulation

Autonomous driving requires reliable collision avoidance in dynamic environments. Nonlinear Model Predictive Controllers (NMPCs) are suitable for this task, but struggle in time-critical scenarios requiring high frequency. To meet this demand, optimization problems are often simplified via linearization, narrowing the horizon window, or reduced temporal nodes, each compromising accuracy or reliability. This work presents the first general convex obstacle avoidance formulation, enabled by a novel approach to integrating logic. This facilitates the incorporation of an obstacle avoidance formulation into convex MPC schemes, enabling a convex optimization framework with substantially improved computational efficiency relative to conventional nonconvex methods. A key property of the formulation is that obstacle avoidance remains effective even when obstacles lie outside the prediction horizon, allowing shorter horizons for real-time deployment. In scenarios where nonconvex formulations are unavoidable, the proposed method meets or exceeds the performance of representative nonconvex alternatives. The method is evaluated in autonomous vehicle applications, where system dynamics are highly nonlinear.

Ricardo Tapia, Iman Soltani

Laboratory for AI, Robotics and Automation, University of California at Davis, Davis, CA 95616 USA

Collision Avoidance Nonlinear MPC Autonomous Driving Model Predictive Control Dynamic Obstacle Avoidance

Dec 11, 2025

Closing the Navigation Compliance Gap in End-to-end Autonomous Driving

Trajectory-scoring planners achieve high navigation compliance when following the expert's original command, yet they struggle at intersections when presented with alternative commands; over 30 percent of such commands are ignored. We attribute this navigation compliance gap to two root causes: (1) existing metrics like Ego Progress do not explicitly measure navigation adherence, diluting the gap between on-route and off-route trajectories; and (2) current datasets pair each scenario with a single command, preventing models from learning command-dependent behavior. We address the metric gap by introducing the binary Navigation Compliance metric (NAVI) and the derived Controllability Measure (CM), and the data gap with the NavControl dataset, 14,918 intersection scenarios augmented with all feasible alternative commands and routing annotations, yielding over 34,000 direction samples. Building on these, we propose NaviHydra, a trajectory-scoring planner incorporating NAVI distillation and Bird's Eye View (BEV)-based trajectory gathering for context-position-aware trajectory feature extraction. NaviHydra achieves 92.7 PDM score on NAVSIM navtest split and 77.5 CM on NavControl test split. Training with NavControl improves controllability across diverse architectures, confirming it as a broadly effective augmentation for navigation compliance. Our project page is https://hanfeng-wu.github.io/closing/.

Hanfeng Wu, Marlon Steiner, Michael Schmidt +2

Karlsruhe Institute of Technology Karlsruhe, Germany · BMW Group Munich, Germany

Autonomous Driving Autonomous Driving Planning End-to-End Autonomous Driving Autonomous Driving Datasets

Nov 25, 2025

WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving

Recent advancements in multimodal large language models (MLLMs) have shown strong understanding of driving scenes, drawing interest in their application to autonomous driving. However, high-level reasoning in safety-critical scenarios, where avoiding one traffic risk can create another, remains a major challenge. Such reasoning is often infeasible with only a single front view and requires a comprehensive view of the environment, which we achieve through multi-view inputs. We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge. Then, we distill Safety-Critical Reasoning into two stages: first resolve the immediate risk, then mitigate the decision-induced downstream risks. To support this, we introduce WaymoQA, a dataset of 35,000 human-annotated question-answer pairs covering complex, high-risk driving scenarios. The dataset includes multiple-choice and open-ended formats across both image and video modalities. Experiments reveal that existing MLLMs underperform in safety-critical scenarios compared to normal scenes, but fine-tuning with WaymoQA significantly improves their reasoning ability, highlighting the effectiveness of our dataset in developing safer and more reasoning-capable driving agents. Our code and data are provided in https://github.com/sjyu001/WaymoQA

Seungjun Yu, Seonho Lee, Namho Kim +5

KAIST Seoul, Republic of Korea · Hanyang University Seoul, Republic of Korea

Autonomous Driving Datasets Visual Question Answering Autonomous Driving Multimodal QA

Nov 14, 2025

GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving

Vision-language models have recently emerged as promising planners for autonomous driving, where success hinges on reasoning over spatial structure and dynamic interactions from multimodal input. However, existing models are typically trained without supervision that explicitly encodes relational dependencies, requiring them to infer interactions between traffic entities implicitly from raw sensor data. In this work, we introduce GraphPilot, a model-agnostic approach that conditions language-based driving models on structured relational context in the form of traffic scene graphs. We serialize scene graphs at different abstraction levels and formats and inject them through structured prompts without modifying the underlying architecture or training objective. Extensive closed-loop evaluations on LangAuto and Bench2Drive show persistent improvements across LMDrive, BEVDriver, and SimLingo. Controlled ablations and hidden-state probing further indicate that relational supervision contributes substantially to these improvements while increasing the decodability of scene relations. Open-loop SpaceDrive results on nuScenes further demonstrate transfer to real-world data. These results indicate that diverse architectures can effectively internalize and ground relational priors through scene graph-conditioned training, even without requiring scene graph input at test-time. Code, fine-tuned models, and our scene graph dataset are publicly available at https://github.com/iis-esslingen/GraphPilot.

Fabian Schmidt, Markus Enzweiler, Abhinav Valada

Esslingen University of Applied Sciences · University of Freiburg

Vision-Language Models VLMs for Autonomous Driving Language Model-Based Control Autonomous Driving Autonomous Driving Planning Graph Reasoning

Oct 28, 2025

Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorer

Human demonstrations are widely considered the cornerstone of end-to-end (E2E) autonomous driving despite human demonstration's scarcity for long-tail and safety-critical scenarios. Nonetheless, current E2E autonomous driving (AD) training paradigms continue to rely on human demonstrations. Imitation learning (IL) requires human demonstrations for training, whereas reinforcement learning (RL) has emerged as a promising alternative to reduce this dependency. However, most existing RL methods for E2E AD still rely implicitly on human demonstrations. A pure rewards-based RL method can overcome the need for human demonstrations, but general RL policy gradient methods suffer from the cold-start problem. In this paper, we propose ZTRS (Zero-human demonstration end-to-end autonomous driving with TRajectory Scorer) - a complete RL-based E2E planning paradigm trained solely on real-world images and rule-based rewards, entirely without human demonstration. Through our proposed Exhaustive Policy Optimization (EPO), a policy gradient variant tailored for enumerable trajectory actions and dense supervision, ZTRS enables the model to generalize better to long-tail driving scenarios. We demonstrate this generalization through our SOTA performance against IL approaches on both long-tail Navhard and closed-loop HUGSIM datasets. Project page: https://zhenxinli.net/ZTRS/.

Zhenxin Li, Nadine Chang, Wenhao Yao +9

Institute of Trustworthy Embodied AI, Fudan University · *Work done during an internship at NVIDIA. · NVIDIA +3

Long-Tail Autonomous Driving Reinforcement Learning End-to-End Autonomous Driving Autonomous Driving Autonomous Driving Planning RL for Autonomous Driving

Oct 21, 2025

OmniNWM: Omniscient Driving Navigation World Models

Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. However, existing methods are typically restricted to fragmented modality modeling, short-horizon drift, and imprecise action control, while lacking intrinsic mechanisms for policy evaluation. In this paper, we introduce OmniNWM, an Omniscient panoramic Navigation World Model that addresses all three dimensions within a consistent probabilistic framework. For State, OmniNWM generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy, ensuring pixel-level alignment across modalities with joint distribution modeling. To mitigate autoregressive exposure bias, we propose a structured panoramic forcing strategy to stabilize long-horizon generation via stochastic manifold thickening. For Action, we introduce canonical geometric action encoding with normalized panoramic Plücker ray-maps. This representation decouples motion dynamics from sensor intrinsics, enabling precise, zero-shot trajectory control across heterogeneous datasets and camera configurations. For Reward, we derive intrinsic occupancy-grounded dense rewards directly from generated 3D volumes, establishing a reliable closed-loop simulation cycle for evaluating diverse planning agents. Extensive experiments demonstrate that OmniNWM achieves SOTA performance in generation fidelity and control precision, with remarkable zero-shot robustness to novel scenes on NuPlan and in-house datasets with distinct camera rigs. Project page is available at https://arlo0o.github.io/OmniNWM/.

Bohan Li, Zhuang Ma, Dalong Du +10

Shanghai Jiao Tong University · Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China · PhiGent +3

Action-Conditioned Video Generation World Model Learning World Model-Based Planning Autonomous Driving World Models for Autonomous Driving World Model Evaluation

Oct 15, 2025

Safe Driving in Occluded Environments

Ensuring safe autonomous driving in the presence of occlusions poses a significant challenge in its policy design. While existing model-driven control techniques based on set invariance can handle visible risks, occlusions create latent risks in which safety-critical states are not observable. Data-driven techniques also struggle to handle latent risks because direct mappings from risk-critical objects in sensor inputs to safe actions cannot be learned without visible risk-critical objects. Motivated by these challenges, in this paper, we propose a probabilistic safety certificate for latent risk. Our key technical enabler is the application of probabilistic invariance: It relaxes the strict observability requirements imposed by set-invariance methods that demand the knowledge of risk-critical states. The proposed techniques provide linear action constraints that confine the latent risk probability within tolerance. Such constraints can be integrated into model predictive controllers or embedded in data-driven policies to mitigate latent risks. The proposed method is tested using the CARLA simulator and compared with a few existing techniques. The theoretical and empirical analysis jointly demonstrate that the proposed methods assure long-term safety in real-time control in occluded environments without being overly conservative and with transparency to exposed risks.

Zhuoyuan Wang, Tongyao Jia, Pharuj Rajborirug +5

Department of Electrical and Computer Engineering, Carnegie Mellon Universty · Faculty of Medicine, King Mongkut’s Institute of Technology Ladkrabang, Thailand · Department of Mechanical Systems Engineering, Nagoya University, Japan

Autonomous Driving

Oct 9, 2025

Injecting Hallucinations in Autonomous Vehicles: A Component-Agnostic Safety Evaluation Framework

Perception failures in autonomous vehicles (AV) remain a major safety concern because they can directly compromise decision-making and contribute to unsafe behavior. To study how these failures affect safety, researchers typically inject artificial faults into real-world or simulated components and observe the resulting outcomes. However, existing fault injection (FI) studies often target a single sensor within the machine perception (MP) module, resulting in siloed frameworks that are difficult to generalize or integrate into unified simulation environments. This work addresses that limitation by reframing perception failures as hallucinations, that is, false perceptions that distort an AV's situational awareness and may trigger unsafe control actions. Building on this concept, we propose a configurable, component-agnostic hallucination injection framework that induces six plausible hallucination types in an open-source simulation environment. A total of 18,356 simulations (8,695 baseline and 9,661 with hallucinations injected) were executed while AVs crossed an unsignalized intersection with transversal traffic. The results statistically validate the framework and quantify the impact of each hallucination type on collisions and on the minimum distance between the AV and the nearest vehicle, used as a continuous proxy for the safety margin. Certain hallucinations, especially missed detection, blind region, angular drift, phantom perception, and latency, significantly increase the odds of collision, while linear drift significantly reduces minimum distances without producing a statistically significant increase in collision odds in the evaluated scenario. The proposed framework provides a scalable and statistically grounded method for evaluating AV safety under perception distortions and supports future research on fault tolerance and resilient AV design.

Alexandre Moreira Nascimento, Gabriel Kenji Godoy Shimanuki, Lúcio Flavio Vismari +5

Escola Politécnica da Universidade de São Paulo, São Paulo 05508-010, Brazil · Center for Design Research, Department of Mechanical Engineering, Stanford, CA 94305 USA · Department of Communication, University of Miami, Coral Gables, FL 33124 USA +1

Autonomous Driving Safety Evaluation Autonomous Driving Autonomous Driving Perception Autonomous Driving Simulation

Sep 30, 2025

Learn2Drive: A neural network-based framework for socially compliant automated vehicle control

This study introduces a novel control framework for adaptive cruise control (ACC) in automated driving, leveraging neural networks and physics-informed constraints. As automated vehicles (AVs) adopt advanced features like ACC, transportation systems are becoming increasingly intelligent and efficient. However, existing AV control strategies primarily focus on optimizing the performance of individual vehicles or platoons, often neglecting their interactions with human-driven vehicles (HVs) and the broader impact on traffic flow. This oversight can exacerbate congestion and reduce overall system efficiency. To address this critical research gap, we propose a neural network-based, socially compliant AV control framework that incorporates social value orientation (SVO). This framework enables AVs to account for their influence on HVs and traffic dynamics. By leveraging AVs as mobile traffic regulators, the proposed approach promotes adaptive driving behaviors that reduce congestion, improve traffic efficiency, and lower energy consumption. Numerical results demonstrate the effectiveness of the proposed method in adapting to varying traffic conditions, thereby enhancing system-wide efficiency. Specifically, when the AV's control mode shifts from prioritizing its own energy conservation to optimizing collective traffic flow efficiency, the controlled AV proactively adapts its acceleration profile. This prosocial behavior yields at least a 38.39% improvement in the average speed of downstream vehicles and effectively dampens traffic oscillations, demonstrating significant enhancements in system-wide dynamics. The implementation code is available https://github.com/lilab2024/Learn2Drive-SVO_v1.git.

Yuhui Liu, Samannita Halder, Shian Wang +1

Department of Civil Engineering Saint Louis University · Department of Electrical and Computer Engineering Saint Louis University · Department of Civil, Environmental, and Architectural Engineering The University of Kansas

Intelligent Transportation Systems Autonomous Driving Neural Network Control

Sep 17, 2025

AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving

While reasoning technology like Chain of Thought (CoT) has been widely adopted in Vision Language Action (VLA) models, it demonstrates promising capabilities in end to end autonomous driving. However, recent efforts to integrate CoT reasoning often fall short in simple scenarios, introducing unnecessary computational overhead without improving decision quality. To address this, we propose AdaThinkDrive, a novel VLA framework with a dual mode reasoning mechanism inspired by fast and slow thinking. First, our framework is pretrained on large scale autonomous driving (AD) scenarios using both question answering (QA) and trajectory datasets to acquire world knowledge and driving commonsense. During supervised fine tuning (SFT), we introduce a two mode dataset, fast answering (w/o CoT) and slow thinking (with CoT), enabling the model to distinguish between scenarios that require reasoning. Furthermore, an Adaptive Think Reward strategy is proposed in conjunction with the Group Relative Policy Optimization (GRPO), which rewards the model for selectively applying CoT by comparing trajectory quality across different reasoning modes. Extensive experiments on the Navsim benchmark show that AdaThinkDrive achieves a PDMS of 90.3, surpassing the best vision only baseline by 1.7 points. Moreover, ablations show that AdaThinkDrive surpasses both the never Think and always Think baselines, improving PDMS by 2.0 and 1.4, respectively. It also reduces inference time by 14% compared to the always Think baseline, demonstrating its ability to balance accuracy and efficiency through adaptive reasoning.

Yuechen Luo, Fang Li, Shaoqing Xu +11

Tsinghua University · Xiaomi EV · Nanyang Technological University +2

Autonomous Driving Reasoning for VLA Models Efficient VLA Model Inference RL for VLMs

Aug 23, 2025

Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving

Personalized autonomous driving requires understanding the driving needs implicit in user requests. This is challenging when driving demonstrations lack style annotations and behavioral adaptation risks degrading planning quality. We propose M-Diffusion Planner with a dual-decoupling design: separating intent understanding from trajectory planning, and general planning capability learning from strategy alignment. A large language model infers driving needs and selects a strategy, which is executed by the corresponding head of a shared diffusion planner. Following imitation pretraining, strategy-specific rewards specialize individual heads without style annotations. We innovatively adapt Group Relative Policy Optimization using surrogate policy-ratio estimation and constrained head-specific updates to mitigate degradation of pretrained planning capabilities. Open-loop and closed-loop evaluations on nuPlan demonstrate distinct strategy-aligned behaviors and competitive planning performance. Further experiments assess intent-to-strategy mapping and collision risk, and analyze the effects of reward design and optimization on strategy alignment and planning quality.

Ding Fan, Xuewen Luo, Hwa Hui Tew +3

Monash University, Malaysia · University of Utah, USA · Monash University, Australia

Autonomous Driving Planning Autonomous Driving Diffusion-Based Planning RL for Autonomous Driving

Aug 19, 2025

ROVR-Open-Dataset: A Large-Scale Depth Dataset for Autonomous Driving

Depth estimation is a fundamental component of spatial perception for autonomous driving and other unmanned systems operating in open urban environments. Existing depth datasets such as KITTI, nuScenes, and DDAD have advanced the field but are limited in diversity and scalability, and benchmark performance on them is approaching saturation. A less discussed constraint is sensor economics: the bespoke multi-LiDAR rigs behind these datasets are expensive, power-hungry, and difficult to replicate at fleet scale, which caps the geographic and temporal diversity that any single benchmark can cover. We present ROVR, a large-scale, diverse, and cost-efficient depth dataset designed to capture the complexity of real-world driving. ROVR comprises 200K high-resolution frames across highway, rural, and urban scenarios, spanning day/night cycles and adverse weather conditions, collected across North America, Europe, and Asia. We additionally release the calibration, synchronization, preprocessing, and privacy pipeline so that the platform can be reproduced by third parties. The lightweight acquisition pipeline enables scalable collection, while sparse but statistically sufficient ground truth -- validated by a density ablation -- supports robust model training. Extensive ablation studies further characterize performance across scene types, illumination, weather conditions, and ground-truth sparsity levels, and identify three qualitatively distinct failure modes -- photometric collapse, geometric confusion, and range saturation -- that current architectures share. The dataset, data loaders, calibration and privacy pipelines, and evaluation code are publicly available at https://xiandaguo.net/ROVR-Open-Dataset.

Xianda Guo, Ruijun Zhang, Yiqun Duan +9

School of Computer Science, Wuhan University, China · Institute of Automation, Chinese Academy of Sciences (CASIA), China · University of Technology Sydney, Australia +6

Autonomous Driving Datasets Depth Estimation Autonomous Driving

Aug 15, 2025

OEL-AD: An Online Evolutive Learning Framework for Cross-Region Adaptive Autonomous Driving

Autonomous driving models have achieved great progress in recent years, yet most existing approaches remain static, following a train-then-deploy paradigm that lacks the ability to adjust internal parameters when encountering new environments. This limitation is particularly critical in cross-region scenarios, which are common in real-world deployments in autonomous driving where differences in road layouts, traffic patterns, and environmental conditions introduce distribution shifts. In this paper, we argue for a paradigm shift towards Online Evolutive Learning (OEL), a learning paradigm designed to enable adaptive, online model optimization through the interaction and coordination of intelligent components in dynamic environments. We introduce OEL-AD, a novel framework that instantiates this paradigm for autonomous driving. Specifically, OEL-AD leverages trajectory uncertainty from the planning module to determine whether adaptive updates are necessary. When triggered, the framework uses discrepancies between predicted agent behaviors and subsequent observations as a self-supervised learning signal to refine perception and prediction modules, thereby indirectly improving planning accuracy and safety. Experiments on the nuScenes dataset with a cross-region split between Singapore and Boston demonstrate that OEL-AD consistently improves perception and prediction performance. This enhancement in perception and prediction consequently results in improved planning capabilities, thereby confirming the promise of online evolutive learning as a fundamental approach for building adaptive and robust autonomous driving systems.

Jiayue Jin, Lang Qian, Jingyu Zhang +2

College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China

Autonomous Driving Adaptive Learning Autonomous Driving Perception Domain Adaptation Autonomous Driving Planning

Jul 7, 2025

Agentic Vehicles for Human-Centered Mobility: Definition, Prospects, and Synergistic Co-Development with Vehicle Autonomy

Autonomy, from the Greek autos (self) and nomos (law), refers to the capacity to operate according to internal rules without external control. Autonomous vehicles (AuVs) are therefore understood as vehicular systems that perceive their environment and execute tasks with minimal human intervention, consistent with the direction indicated by the SAE levels of automated driving. However, recent research and deployments increasingly showcase vehicular capabilities that, while not contradicting autonomy, are not entailed by it, including ambiguous goal handling, purposeful social engagement, external tool use, proactive problem solving, continuous learning, and context-sensitive reasoning in unseen and ethically salient situations, enabled in part by multimodal language models. These developments reveal a gap between technical autonomy and the broader social cognitive functions required for human-centered mobility, which are more precisely captured by the notion of agency. Therefore, rather than adding increasingly elaborate modifiers to "autonomous," we introduce agentic vehicles (AgVs) and suggest that autonomy and agency are intertwined but conceptually distinct: if autonomy concerns what to do and how to do it (task executions under internal rules), agency pertains to why to do it and what else can be done (goal-directed, adaptive actions). We present autonomy and agency as orthogonal yet synergistic dimensions with co-development implications. Vehicle agency marks a novel dimension of mobility service intelligence, heralding vehicles as purposeful actors in society.

Jiangbo Yu, Raphael Frank, Luis Miranda-Moreno +6

Department of Civil Engineering, McGill University, Montreal, QC H3A 0C3, Canada · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-4365 Esch-sur-Alzette, Luxembourg

Autonomous Driving Human-Centered AI Agentic AI

May 21, 2025

Generative AI for Autonomous Driving: A Review

Generative AI (GenAI) is rapidly advancing the field of Autonomous Driving (AD), extending beyond traditional applications in text, image, and video generation. We explore how generative models can enhance automotive tasks, such as static map creation, dynamic scenario generation, trajectory forecasting, and vehicle motion planning. By examining multiple generative approaches ranging from Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to Generative Transformers (GTs) and Diffusion Models (DMs), we highlight and compare their capabilities and limitations for AD-specific applications. Additionally, we discuss hybrid methods integrating conventional techniques with generative approaches, and emphasize their improved adaptability and robustness. We also identify relevant datasets and outline open research questions to guide future developments in GenAI. Finally, we discuss five core challenges: safety, interpretability, realtime capabilities, training and benchmarking. We also present recommendations for image generation, dynamic scenario generation, and planning.

Katharina Winter, Abhishek Vivekanandan, Rupert Polley +17

Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany · FZI Research Center for Information Technology, 76131 Karlsruhe, Germany · Continental Automotive Technologies GmbH, Guericke Str.7, 60488 Frankfurt a. M., Germany +6

Autonomous Driving Autonomous Driving Planning Autonomous Driving Scenario Generation Trajectory Prediction

May 13, 2025

Generative AI for Autonomous Driving: Frontiers and Opportunities

Generative Artificial Intelligence (GenAI) constitutes a transformative technological wave that reconfigures industries through its unparalleled capabilities for content creation, reasoning, planning, and multimodal understanding. This revolutionary force offers the most promising path yet toward solving one of engineering's grandest challenges: achieving reliable, fully autonomous driving, particularly the pursuit of Level 5 autonomy. This survey delivers a comprehensive and critical synthesis of the emerging role of GenAI across the autonomous driving stack. We begin by distilling the principles and trade-offs of modern generative modeling, encompassing VAEs, GANs, Diffusion Models, and Large Language Models (LLMs). We then map their frontier applications in image, LiDAR, trajectory, occupancy, video generation as well as LLM-guided reasoning and decision making. We categorize practical applications, such as synthetic data workflows, end-to-end driving strategies, high-fidelity digital twin systems, smart transportation networks, and cross-domain transfer to embodied AI. We identify key obstacles and possibilities such as comprehensive generalization across rare cases, evaluation and safety checks, budget-limited implementation, regulatory compliance, ethical concerns, and environmental effects, while proposing research plans across theoretical assurances, trust metrics, transport integration, and socio-technical influence. By unifying these threads, the survey provides a forward-looking reference for researchers, engineers, and policymakers navigating the convergence of generative AI and advanced autonomous mobility. An actively maintained repository of cited works is available at https://github.com/taco-group/GenAI4AD.

Yuping Wang, Shuo Xing, Cui Can +46

Autonomous Driving Autonomous Driving Scenario Generation

May 1, 2025

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. To systematically examine this assumption, we introduce LightEMMA, a longitudinal framework for evaluating the autonomous driving performance of VLMs. LightEMMA uses a lightweight, unified evaluation protocol that assesses each model's intrinsic driving capability without model-specific fine-tuning, architectural changes, or prompt engineering. Using this protocol, we evaluate 15 models from five major families on the challenging nuScenes prediction benchmark. Empirical findings show that, despite increased model scale and enhanced general reasoning capabilities, successive VLM generations do not consistently achieve better driving performance. Further analysis of driving scenarios reveals recurring failure modes, including overreliance on historical actions and difficulty reconciling conflicting visual cues. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems. The source code is available at https://github.com/michigan-traffic-lab/LightEMMA.

Zhijie Qiao, Haowei Li, Zhong Cao +1

Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA

VLM Evaluation Autonomous Driving Benchmarks VLMs for Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving

Mar 24, 2025

Tube-Based Robust Control Strategy for Vision-Guided Autonomous Vehicles

A robust control strategy for autonomous vehicles can improve system stability, enhance riding comfort, and prevent driving accidents. This paper presents a novel interpolation-tubebased constrained iterative linear quadratic regulator (itubeCILQR) algorithm for autonomous computer-vision-based vehicle lane-keeping. The goal of the algorithm is to enhance robustness during high-speed cornering on tight turns. Compared with standard tube-based approaches, the proposed itube-CILQR algorithm reduces system conservatism and exhibits higher computational speed. Numerical simulations and vision-based experiments were conducted to examine the feasibility of using the proposed algorithm for controlling autonomous vehicles. The results indicated that the proposed algorithm achieved superior vehicle lane-keeping performance to variational CILQRbased methods and model predictive control (MPC) approaches involving the use of a classical interior-point optimizer. Specifically, itube-CILQR required an average runtime of 3.45 ms to generate a control signal for guiding a self-driving vehicle. By comparison, itube-MPC typically required a 4.32 times longer computation time to complete the same task. Moreover, the influence of conservatism on system behavior was investigated by exploring the variations in the interpolation variables derived using the proposed itube-CILQR algorithm during lane-keeping maneuvers.

Der-Hau Lee

Department of Electrophysics, National Yang Ming Chiao Tung University, Hsinchu 300, Taiwan

Autonomous Driving Robust Control Linear Quadratic Regulator

Mar 4, 2025

L2G-Map: Local-to-Global Mapping via Hierarchical Diffusion Refinement and Elliptical Bayesian Fusion

Offline high-definition maps provide essential geometric and topological priors for autonomous driving systems. Pure-vision solutions have become the predominant paradigm for offline mapping due to their cost-effectiveness and scalability. However, local-to-global mapping under visual conditions confronts two fundamental challenges: single-shot local observations are susceptible to viewpoint variation and environmental interference, leading to geometric deviations, while multi-source local information exhibits heterogeneous confidence, rendering globally consistent aggregation difficult. To address these, this paper proposes L2G-Map, a framework comprising hierarchical prior diffusion refinement and elliptical space Bayesian fusion. The former jointly embeds temporal context and centerline priors to guide structure completion and topology recovery during denoising, alleviating the information incompleteness inherent in pure-vision settings. The latter incorporates an adaptive weighting strategy driven by elliptical distance propagation, enabling probabilistically optimal aggregation of multi-source information under the Bayesian posterior update paradigm. Extensive experiments on nuScenes and Argoverse benchmark datasets verify the effectiveness of L2G-Map. The proposed refinement component yields consistent local map accuracy improvements across different datasets. Under sensor-degraded conditions, a 3.27% mIoU gain is achieved. Furthermore, the adaptive fusion component significantly enhances the accuracy of global maps. The fused global map can be flexibly embedded into different online map models, yielding an 18.26% mIoU improvement in semantic map construction and a 20.00% enhancement in vectorized map construction, demonstrating the overall advantages of the proposed closed-loop pipeline. Source code will be available at https://github.com/lynn-yu/L2G-Map.

Siyu Li, Xinying Hong, Fei Teng +5

School of Automation and Electrical Engineering, Zhejiang University of Science and Technology, China · School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · College of Computer Science and Electronic Engineering, Hunan University, China +1

HD Map Construction Autonomous Driving Diffusion Models

Dec 3, 2024

Highway Congestion Reduction through Reinforcement Learning Based Eulerian Headway Control

Connected automated vehicles (CAVs) equipped with adaptive cruise control (ACC) create new opportunities for highway congestion mitigation. Traditional practice relies on Eulerian variable speed limits (VSL) which regulate traffic through roadside signs, but suffer from infrequent updates and limited driver compliance. Recent research explored Lagrangian strategies that directly control individual vehicles, offering high reactivity and compliance, yet in realistic multi-lane settings they depend on drivers' latent lane-change intentions, making robust vehicle-level decisions difficult. Hence, we propose an Eulerian control system optimized through reinforcement learning, that (i) leverages ACC for reactivity and compliance, and (ii) obviates dependence on latent driver intentions by regulating aggregate density near bottlenecks, crucially via headway commands rather than speed commands. We evaluate two variants of our system, time-headway and distance-headway control, in large-scale simulations across a range of traffic conditions. Both variants outperform baselines, improving traffic flow by up to 10.6% over human traffic and 6.7% over traditional VSL. To strengthen evaluation, we propose a novel boundary-aware speed metric addressing a recognized flaw in simulation studies with dynamic vehicle entry and exit. The empirical results, together with our emphasis on deployable system design, suggest a path towards practical, safe, and scalable highway congestion mitigation.

Yaron Veksler, Sharon Hornstein, Han Wang +2

Research and Development Labs, General Motors, Herzliya, Israel. · Department of Civil and Environmental Engineering, UC Berkeley.

RL Control Intelligent Transportation Systems Autonomous Driving

Nov 3, 2024

ROAD-Waymo: A Large-Scale Action Awareness Dataset for Autonomous Driving

Autonomous Vehicle (AV) perception systems require more than simply seeing, via e.g., object detection or scene segmentation. They need a holistic understanding of what is happening within the scene for safe interaction with other road users. Few datasets exist for the purpose of developing and training algorithms to comprehend the actions of other road users. This paper presents ROAD-Waymo, an extensive dataset for the development and benchmarking of techniques for agent, action, location and event detection in road scenes, provided as a layer upon the (US) Waymo Open dataset. Considerably larger and more challenging than any existing dataset (and encompassing multiple cities), it comes with 198k annotated video frames, 54k agent tubes, 3.9M bounding boxes and a total of 12.4M labels. The integrity of the dataset has been confirmed and enhanced via a novel annotation pipeline designed for automatically identifying violations of requirements specifically designed for this dataset. As ROAD-Waymo is compatible with the original (UK) ROAD dataset, it provides the opportunity to tackle domain adaptation between real-world road scenarios in different countries within a novel benchmark: ROAD++.

Salman Khan, Izzeddin Teeti, Reza Javanmard Alitappeh +5

Oxford Brookes University · MAZUST · University of Oxford +2

Autonomous Driving Datasets Autonomous Driving Benchmarks Autonomous Driving Autonomous Driving Perception

Aug 22, 2024

Human-In-The-Loop Machine Learning for Safe and Ethical Autonomous Vehicles: Principles, Challenges, and Opportunities

Machine Learning (ML) has become central to Autonomous Vehicles (AVs), supporting perception, prediction, planning, control, and decision-making in dynamic environments. However, achieving full autonomy in cluttered and complex scenarios, such as intricate intersections, diverse scenes, varied trajectories, and complex missions, remains challenging; moreover, data labeling is still a major bottleneck. These limitations motivate Human-in-the-Loop Machine Learning (HITL-ML), in which human input is incorporated through validation, annotation, task organization, reward design, action correction, preference feedback, and supervisory intervention. To advance safe and ethical autonomy, this paper presents a tutorial survey of HITL-ML for AVs, focusing on Curriculum Learning (CL), Human-in-the-Loop Reinforcement Learning (HITL-RL), Human-in-the-Loop Large Language Models (HITL-LLMs), Active Learning (AL), and ethical principles. We first review CL methods that structure training from simple to complex tasks, covering navigation, path planning, obstacle avoidance, data collection, landing, intersection handling, motion planning, and UAV swarm coordination. We then examine HITL-RL through reward shaping, action injection, demonstrations, preference-based feedback, and interactive learning, emphasizing improved learning efficiency, safer policy exploration, and real-time intervention. Next, we review HITL-LLM through collaboration and oversight and specify key challenges. After that, we discuss AL for perception, anomaly detection, semantic mapping, object detection, vehicle recognition, and security-related classification. Ethical principles are reviewed as technical requirements for transparency, accountability, human oversight, safety, security, regulatory compliance, and reliability of human input.

Yousef Emami, Mohammadhossein Homaei, Miguel Gutiérrez Gaitán +4

Active Learning Autonomous Driving Human-in-the-Loop AI