RedLight-VLA: Models for traffic-rule grounding and behavioral emphasis in driving policies

Bala Murali Manoghar Sai Sudhakar; Sourab Bapu Sridhar; Sandipan Das; Rahul Ahuja; Meda Lazar; Ashish Garg; Pratik Likhar; Senthil Yogamani

cs.ROAug 21, 2026

RedLight-VLA: Models for traffic-rule grounding and behavioral emphasis in driving policies

Authors: Bala Murali Manoghar Sai Sudhakar, Sourab Bapu Sridhar, Sandipan Das, Rahul Ahuja, Meda Lazar, Ashish Garg, Pratik Likhar, Senthil Yogamani

Organizations: Qualcomm Technologies, Inc., USA · Qualcomm Auto Ltd Sweden Filial, Sweden · Qualcomm India Private Limited, India · Arriver System Software S.r.l., Romania

Abstract

Behavior-cloned Vision-Language-Action (VLA) driving policies struggle with rare rule-governed maneuvers at signalized intersections. Braking and launching examples contribute little to averaged trajectory loss, while fused representations lack explicit supervision for the governing traffic-light and stop-line state. We present RedLight-VLA, a training objective that uses expert futures and automatically generated perception targets without additional manual rule annotation. First, trajectory-derived behavioral reweighting (BR) emphasizes rare deceleration and acceleration using rotation-invariant longitudinal dynamics and a scale-preserving reduction that exactly recovers the baseline when disabled. Second, parallel auxiliary (AUX) heads ground traffic-light and stop-line state in continuous post-fusion rule tokens, without autoregressive language generation or changes to the trajectory decoder. We evaluate on a curated set of 20 s sequences with a 5 s prediction horizon. Controlled variants share the same backbone, training data, decoder, and evaluation population. Against an otherwise identical VLA baseline, RedLight-VLA reduces red-light stop-line overshoot from 7.3% to6.8%, reduces stop-line velocity error by 12.7%, and improves 3 s trafficlight-sliced ADE/FDE from 0.274/0.964 m to 0.247/0.897 m. Green-light false stops increase from 3.2% to 3.9%; however, combining BR with AUX supervision mitigates the larger increase observed for AUX alone (4.0%). The combined model also improves non-traffic-light ADE/FDE from 0.268/0.956 m to 0.241/0.876 m and outperforms either mechanism alone on all four sliced displacement measures.

Figures & tables

Split	Sequences	Frames	Any TL (raw)	Ego-rel. TL (raw)	Resolved TL (GT)	Stop-line (GT)
Train	$235{,}985$	$1{,}651{,}895$	$40.66\%$ ( $671$ K)	$38.34\%$ ( $633$ K)	$25.70\%$ ( $425$ K)	$10.10\%$ ( $167$ K)
Val	$16{,}318$	$114{,}226$	$40.62\%$ ( $46{,}395$ )	$38.26\%$ ( $43{,}703$ )	$25.59\%$ ( $29{,}236$ )	$10.01\%$ ( $11{,}429$ )
Test	$16{,}404$	$114{,}828$	$40.69\%$ ( $46{,}727$ )	$38.43\%$ ( $44{,}128$ )	$25.81\%$ ( $29{,}640$ )	$10.19\%$ ( $11{,}696$ )

Table 1: Dataset scale and split composition. Percentages are relative to all frames in each split; parentheses report frame counts, with “K” denoting thousands for the training split. “Resolved TL” is the red/green target used by the auxiliary task, and “Stop-line” is the capped target used by the stop-line head.

Table 2

Table 3

	TL slice			non-TL slice
Scenario	Base	Full	$\Delta$ Full	Base	Full	$\Delta$ Full
Stationary	0.3361	0.3377	+0.0016	0.4300	0.3645	-0.0655
Low speed	0.5942	0.5550	-0.0392	0.8185	0.7234	-0.0951
Medium speed	0.8302	0.7318	-0.0984	0.7861	0.6821	-0.1040
High speed	0.7154	0.6667	-0.0487	0.5664	0.5356	-0.0308
Low curvature	0.6397	0.5825	-0.0572	0.6131	0.5598	-0.0533

Table 4: Scenario-bucketed ADE at

5

s on the curated validation extract (m,

\downarrow

). The TL slice contains frames with an ego-relevant traffic light or stop line; the non-TL slice contains the remaining eligible frames. Full has lower point-estimate ADE than Base in 19 of 20 slice–scenario cells.

Stop for red
Red $\rightarrow$ green

Figure 4: Qualitative red-to-green sequence. Predictions (green) stop at red (top) and accelerate after green (bottom), compared with reference futures (blue). Insets show speed and TLR GT/PRED; this example is illustrative, not causal evidence.

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 534 papers

VLALight: A Vision-Language-Action Model for Traffic Signal Control
Sep 29, 2026Pan Zhang, Siqi Lai, Kemu Dong +1Intelligent Transportation Systems Vision-Language-Action Models
The Hong Kong University of Science and Technology (Guangzhou) · Dalian University of Technology
VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control
Sep 25, 2026Kemou Jiang, Maonan Wang, Xingchen Zou +6Traffic Signal Control Efficient VLA Models
School of Transportation Science and Engineering, Beihang University, Beijing, China · The Chinese University of Hong Kong (Shenzhen), Shenzhen, China · The Chinese University of Hong Kong, Hong Kong SAR, China +2
Inference-Time Attention Steering for Vision-Language-Action Driving Models
Aug 17, 2026Darshan Nagendra Prasad, Lars Ullrich, Knut GraichenVision-Language-Action Models VLMs for Autonomous Driving
FAU Erlangen-Nürnberg, Germany

Sep 29, 2026J/K move · Enter open · S save

VLALight: A Vision-Language-Action Model for Traffic Signal Control

Pan Zhang, Siqi Lai, Kemu Dong, Hao Liu

The Hong Kong University of Science and Technology (Guangzhou) · Dalian University of Technology

Intelligent Transportation Systems Vision-Language-Action Models Traffic Signal Control

Open paper AskPDF arXiv

Traffic signal control (TSC) is essential for improving urban mobility and reducing congestion. Although roadside cameras are widely deployed at signalized intersections and provide rich visual observations of evolving traffic, existing TSC methods typically rely on manually engineered traffic states or separate perception modules, creating a gap between physical observations and control decisions. We present VLALight, the first vision-language-action (VLA) model for end-to-end traffic signal control from multi-view roadside videos. VLALight directly maps visual observations to coordinated signal actions through multi-target spatiotemporal traffic reasoning and topology-aware cooperative perception across intersections. To establish this capability, we develop a two-stage supervised cold-start training strategy for visual traffic understanding and signal decision-making, followed by cooperative agentic reinforcement learning that jointly optimizes local control and network-wide traffic efficiency. Furthermore, VLALight introduces adaptive fast and slow reasoning modes, enabling the policy to allocate deeper reasoning only when additional deliberation provides sufficient control benefits. Through balanced mode-aware rollouts and relative advantage optimization, VLALight learns to trade off decision quality and inference cost. Extensive experiments on seven real-world traffic-flow datasets across three urban networks demonstrate that VLALight consistently outperforms transportation-based, RL-based, and LLM/VLM-based baselines. Ablation studies validate the effectiveness of cooperative perception, network-level optimization, and adaptive reasoning. These results demonstrate the potential of VLA models for real-world physical traffic control. Our project is available at https://github.com/usail-hkust/VLALight.git.

Abstract

Autonomous Driving

Explore similar work

VLALight: A Vision-Language-Action Model for Traffic Signal Control

VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control

Inference-Time Attention Steering for Vision-Language-Action Driving Models