One Perception, All Maneuvers: Directional Traffic Signal Understanding for Maneuver-Level Signal Intent Prediction

Ang Zou; Runzhe Zheng; Zhigang li; Zhen Yang; Han Xia; Xuewei Li; Zequn Qin; Xi Li

cs.CVSep 26, 2026

One Perception, All Maneuvers: Directional Traffic Signal Understanding for Maneuver-Level Signal Intent Prediction

Authors: Ang Zou, Runzhe Zheng, Zhigang li, Zhen Yang, Han Xia, Xuewei Li, Zequn Qin, Xi Li

Organizations: College of Computer Science and Technology, Zhejiang University · BYD Company Limited · School of Electronic and Information Engineering, Shanghai DianJi University

Abstract

Traffic lights are a key regulatory signal for autonomous driving at urban intersections, yet existing traffic signal perception is still predominantly formulated as instance-level detection or color recognition. Such formulations identify where traffic lights are and what colors they display, but leave a critical semantic gap before downstream planning: which ego maneuver is controlled by each visible signal and what dynamic permission the signal expresses for that maneuver. In this paper, we formulate Directional Traffic Signal Understanding, a decision-oriented task that predicts structured signal states for straight, left-turn, right-turn, and U-turn maneuvers from a front-view image. Each state contains the associated signal color and signal-implied passability. Based on OpenLane-V2, we provide a direction-level benchmark with maneuver-level supervision and metrics for color recognition, passability, full-frame consistency, and safety-critical errors. A direction-aware baseline combines global context, localized traffic-light evidence, and maneuver-specific representations. Experiments show that direction-level modeling improves passability prediction over image-level classifiers and detection-oriented pipelines, particularly at complex multi-signal intersections. The resulting representation provides a direct and interpretable traffic-signal interface for downstream planning together with topology, route, and surrounding-agent information.

Figures & tables

Figure 1: Overview of the proposed Directional Traffic Signal Understanding task. Conventional traffic light recognition focuses on detecting signal instances and recognizing their colors, which leaves an ambiguity between visible signals and ego-vehicle maneuvers. We reformulate traffic signal perception as maneuver-level passability understanding, where a model predicts structured signal states for going straight, turning left, turning right, and making a U-turn. The none label indicates that no dedicated signal controls the corresponding direction, whose signal-implied passability is treated as go .

Figure 2: Representative scenarios and direction-level annotations in the proposed passability benchmark. The examples illustrate a standard signalized intersection, label transitions when the ego vehicle passes through an intersection, and a dedicated right-turn signal. For each frame, the benchmark provides color and passability states for left-turn, U-turn, straight, and right-turn maneuvers, while excluding traffic lights that control adjacent lanes or distant intersections.

Figure 3: Color-state data distributions for straight, left-turn, right-turn, and U-turn maneuvers.

Figure 4: Overview of the proposed framework. Global scene features and localized traffic-light ROI features are combined to predict color and passability for four ego maneuvers: left, U-turn, straight, and right.

Method	Acc color $\uparrow$	Acc pass $\uparrow$	EM $\uparrow$	TSS $\uparrow$	False-Go $\downarrow$	False-Stop $\downarrow$
ResNet-18	0.8514	0.8977	0.7253	0.8248	0.3224	0.0335
ResNet-50	0.8775	0.9142	0.7745	0.8554	0.2651	0.0297
ResNet-101	0.8795	0.9165	0.7702	0.8554	0.2209	0.0405
ViT-B/16	0.8698	0.9054	0.7702	0.8485	0.1612	0.0738
Swin-Tiny	0.8916	0.9266	0.8019	0.8734	0.1831	0.0391
DeiT-Small	0.8634	0.9062	0.7528	0.8408	0.2387	0.0485

Table 1: Comparison with representative generic visual models on Subset-A. All models use the same four-maneuver output space and a unified input resolution. Higher is better for the first four metrics, while lower is better for False-Go and False-Stop. The best result in each column is highlighted in bold .

Method	Params	Setting	Acc color $\uparrow$	Acc pass $\uparrow$	EM $\uparrow$	TSS $\uparrow$	False-Go $\downarrow$	False-Stop $\downarrow$
Qwen2.5-VL-7B-Instruct	7B	SFT-LoRA	0.8779	0.9126	0.7892	0.8599	0.2814	0.0268
MiniCPM-V 4.5	8B	SFT-LoRA	0.8712	0.9043	0.7888	0.8548	0.2607	0.0442
Llama-3.2-11B-Vision	11B	SFT-LoRA	0.8731	0.9031	0.7830	0.8531	0.2615	0.0454
Qwen3.6-35B-A3B	35B-A3B	Zero-shot	0.5910	0.7780	0.1014	0.4901	0.2745	0.2055
Gemma 4 31B	31B	Zero-shot	0.4645	0.6700	0.0104	0.3817	0.3601	0.3206
Nemotron 3 Nano Omni	30B-A3B	Zero-shot	0.5437	0.6253	0.0000	0.3897	0.0130	0.4877

Table 2: Comparison with recent multimodal large language models on Subset-A. SFT models are fine-tuned on the training split, while zero-shot models use no task-specific supervision. The best and second-best results are shown in bold and underlined , respectively.

Method	Inference mechanism	Acc color $\uparrow$	Acc pass $\uparrow$	EM $\uparrow$	TSS $\uparrow$	False-Go $\downarrow$	False-Stop $\downarrow$
TLD-READY	Pictogram mapping	0.8563	0.9187	0.7516	0.8422	0.3017	0.0124
TopoPoint	Predicted topology	0.7180	0.8681	0.5439	0.7100	0.5229	0.0096
Reusing Attention / One-Stage	Reused-attention topology	0.6647	0.8532	0.4762	0.6647	0.5721	0.0138
Ours	Direct maneuver prediction	0.9019	0.9356	0.8031	0.8802	0.1372	0.0416

Table 3: Comparison with relevance- and topology-based traffic-light understanding methods on Subset-A. All methods are evaluated under the same directional output protocol using the same six task metrics. TLD-READY derives directional states from traffic-light and arrow-direction relevance cues, while TopoPoint and One-Stage use predicted lane–traffic topology. All structured baselines use approach-signal fallback when explicit directional evidence is unavailable. The best results are highlighted in bold .

Autonomous Driving Perception

Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.

See all 379 papers

VLALight: A Vision-Language-Action Model for Traffic Signal Control
Sep 29, 2026Pan Zhang, Siqi Lai, Kemu Dong +1Intelligent Transportation Systems Vision-Language-Action Models
The Hong Kong University of Science and Technology (Guangzhou) · Dalian University of Technology
Hierarchical Flow Decomposition for Turning Movement Prediction at Signalized Intersections
Apr 10, 2026Md Atiqur Rahman Mallick, Kamrul Hasan, Pulock Das +2Intelligent Transportation Systems Traffic Flow Estimation
Department of Electrical and Computer Engineering, Tennessee State University, Nashville, TN, USA
Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Driving
Aug 18, 2026Zongzheng Zhang, Jijun Wang, Saining Zhang +8End-to-End Autonomous Driving Autonomous Driving Perception
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Bosch Corporate Research, Shanghai, China

Sep 29, 2026J/K move · Enter open · S save

VLALight: A Vision-Language-Action Model for Traffic Signal Control

Pan Zhang, Siqi Lai, Kemu Dong, Hao Liu

The Hong Kong University of Science and Technology (Guangzhou) · Dalian University of Technology

Intelligent Transportation Systems Vision-Language-Action Models Traffic Signal Control

Open paper AskPDF arXiv

Traffic signal control (TSC) is essential for improving urban mobility and reducing congestion. Although roadside cameras are widely deployed at signalized intersections and provide rich visual observations of evolving traffic, existing TSC methods typically rely on manually engineered traffic states or separate perception modules, creating a gap between physical observations and control decisions. We present VLALight, the first vision-language-action (VLA) model for end-to-end traffic signal control from multi-view roadside videos. VLALight directly maps visual observations to coordinated signal actions through multi-target spatiotemporal traffic reasoning and topology-aware cooperative perception across intersections. To establish this capability, we develop a two-stage supervised cold-start training strategy for visual traffic understanding and signal decision-making, followed by cooperative agentic reinforcement learning that jointly optimizes local control and network-wide traffic efficiency. Furthermore, VLALight introduces adaptive fast and slow reasoning modes, enabling the policy to allocate deeper reasoning only when additional deliberation provides sufficient control benefits. Through balanced mode-aware rollouts and relative advantage optimization, VLALight learns to trade off decision quality and inference cost. Extensive experiments on seven real-world traffic-flow datasets across three urban networks demonstrate that VLALight consistently outperforms transportation-based, RL-based, and LLM/VLM-based baselines. Ablation studies validate the effectiveness of cooperative perception, network-level optimization, and adaptive reasoning. These results demonstrate the potential of VLA models for real-world physical traffic control. Our project is available at https://github.com/usail-hkust/VLALight.git.

Abstract

Autonomous Driving Perception

Explore similar work

VLALight: A Vision-Language-Action Model for Traffic Signal Control

Hierarchical Flow Decomposition for Turning Movement Prediction at Signalized Intersections

Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Driving