VLMs for Autonomous Driving
VLM: Vision-Language Model
Overview
This field studies how models that combine visual and language understanding can interpret traffic scenes, reason about motion and hazards, and help vehicles choose safe actions. Approaches include adapting pretrained models with driving data, grounding them in 3D or bird’s-eye-view geometry, fusing camera, LiDAR, and radar inputs, and connecting reasoning to trajectory planners or predictive world models. Researchers evaluate scene understanding and prediction on datasets such as nuScenes, and test interactive driving in closed-loop simulators such as Bench2Drive and CARLA; reinforcement learning and structured feedback are increasingly used to improve planning. A central challenge is that fluent explanations and strong static-scene recognition do not guarantee grounded, temporally reliable, safe actions under real-time and changing conditions.
Latest papers
192 papers23 papers in the last four weeks, up 10% on the four weeks before. 0.1% of all new papers.