cs.CVNov 16, 2026

Competent Autonomous Driving Through Fine Tuning Vision Language Action Models with Symbolic Traffic Rules

Authors: Barkin Dagda, Vinal Asodia, Iman Sharifi, Peng Wei, Saber Fallah

Organizations: Connected and Autonomous Vehicles Lab (CAV-Lab), University of Surrey, Guildford, GU2 7XH, UK. · Department of Mechanical and Aerospace Engineering, George Washington University, Washington, DC 20052, USA.

Abstract

Vision-language-action (VLA) models offer a unified approach to perception, reasoning, and control for autonomous driving, but accurate action prediction alone does not establish compliance with traffic regulations. This paper presents a rule-guided VLA framework that incorporates symbolic traffic-rule feedback into policy optimisation. The framework combines supervised fine-tuning on expert demonstrations with reinforcement learning (RL) refinement using a reward that encourages encoded-rule compliance while penalising deviations from expert actions. Selected provisions of the United Kingdom Highway Code are formalised as executable first-order logic rules using PyReason, covering speed adaptation, following distance, and road-user priority. A frozen vision-language model extracts scene predicates, which are combined with vehicle-state information to assess the projected consequences of candidate control actions. Offline evaluation on held-out data collected in the CARLA simulator compares three group-based RL methods. The results show improvements in both action prediction accuracy and encoded-rule compliance over the supervised baseline. The best-performing variant increases the overall rule pass rate from 92.9% to 97.6%, with zero observed violations of the encoded road-user-priority check in the evaluated samples. These findings support symbolic rule feedback as a means of improving VLA policy alignment with explicit traffic-rule criteria.

Figures & tables

Explore similar work

CardsList
  1. Teaching Vision-Language-Action Models What to See and Where to Look

    Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10End-to-End Autonomous DrivingVLM Distillation

  2. DriveMA: Driving Vision-Language-Action Models with verifiable Meta-Actions

    May 29, 2026Weicheng Zheng, Yixin Huang, Qiao Sun +2VLMs for Autonomous DrivingAutonomous Driving