Traffic-VL: Reinforced Vision-Language Models for Generalizable End-to-End Traffic Signal Control
Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · the Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China · Beihang University, Beijing, China · PCITECH, Guangzhou, China ·
Abstract
Existing traffic signal control methods typically rely on structured traffic states produced by dedicated perception pipelines. Emerging visual approaches often use idealized simulator images and convert them into intermediate representations before control, limiting their ability to directly exploit roadside visual observations. To bridge both the visual-domain and perception-to-decision gaps, we introduce SynTra-V and Traffic-VL. SynTra-V converts physics-based traffic simulations into photorealistic multi-view surveillance imagery across diverse road topologies and sensing conditions, and automatic semantic auditing checks that control-critical geometry, vehicle movements, and lane semantics match the simulator state. Traffic-VL is an eight-billion-parameter vision-language model that directly maps synchronized multi-view images and natural-language phase descriptions to signal decisions without intermediate structured states. Building on Traffic-R1's two-stage reinforced reasoning framework, Traffic-VL adapts its vision encoder via offline reinforcement learning on SynTra-V and freezes it during online visual stepwise trajectory policy optimization in closed-loop simulation. Experiments show that Traffic-VL matches structured-state Traffic-R1 while outperforming image-based controllers and general-purpose vision-language models, including Gemini 3 Pro. Evaluations on unseen topologies, degraded sensing conditions, real surveillance footage, and a three-week deployment across ten intersections further demonstrate robust visual generalization and effective constrained online control.
Figures & tables
| Aspect | Conference Version (Traffic-R1) | Journal Version (Traffic-VL) |
| Perception | Structured text states from upstream perception | Raw images with natural-language phase descriptions |
| Reasoning | LLM reasoning over pre-extracted states | Joint visual and language reasoning for phase selection |
| Phase Schema | Fixed four-phase lane template | Language descriptions for variable and complex phase sets |
| Training Data | Fixed textual symbolic traffic states | Photorealistic imagery across topologies and environments |
| Deployment | Dedicated V2X detectors and CV pipelines | Roadside-camera views with natural-language phase descriptions |
| Generalization | Standard intersections with fixed topology | Generalization across topological and environmental variation |
| Method | Input | Setting | Depl. | Jinan1 | Jinan2 | Jinan3 | Hangzhou1 | Hangzhou2 | |||||
| ATT | AWT | ATT | AWT | ATT | AWT | ATT | AWT | ATT | AWT | ||||
| FixedTime | Structured-Oracle | Rule-based | ✗ | 453.41 | 51.32 | 370.34 | 35.15 | 384.53 | 36.95 | 497.54 | 36.41 | 408.53 | 53.94 |
| MaxPressure | Structured-Oracle | Rule-based | ✗ | 274.34 | 32.04 | 246.35 | 22.56 | 245.66 | 24.31 | 289.55 | 21.52 | 349.53 | 67.52 |
| MPLight | Structured-Oracle | Zero-shot | ✗ | 455.34 | 72.45 | 471.14 | 78.03 | 427.37 | 64.91 | 491.32 | 64.05 | 425.42 | 69.85 |
| CoLight | Structured-Oracle | Zero-shot | ✗ | 472.44 | 91.09 | 450.41 | 78.59 | 498.84 | 89.94 | 494.61 | 72.18 | 435.32 | 81.11 |
| Advanced-CoLight | Structured-Oracle | Zero-shot | ✗ | 347.31 | 56.54 | 345.78 | 35.96 | 342.56 | 37.55 | 485.32 | 54.11 | 523.19 | 72.56 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Baseline | Category | Observation | Description |
| FixedTime [ 44 ] | Rule-based | Structured-Oracle | Fixed-cycle plan with preset phase durations; no learning. |
| MaxPressure [ 45 ] | Rule-based | Structured-Oracle | Greedily activates the phase of maximum pressure each step. |
| MPLight [ 14 ] | RL | Structured-Oracle | Pressure-based deep RL with FRAP backbone and parameter sharing. |
| CoLight [ 5 ] | RL | Structured-Oracle | Graph-attention RL that coordinates with neighboring intersections. |
| Advanced-CoLight [ 20 ] | RL | Structured-Oracle | Strongest RL baseline; advanced traffic-state features over CoLight. |
| AttendLight [ 19 ] | RL | Structured-Oracle | Attention policy across intersection structures; evaluated zero-shot without target-intersection fine-tuning in the main comparison. |
| Method | Setting | Jinan1 | Jinan2 | Jinan3 | Hangzhou1 | Hangzhou2 | |||||
| ATT | AWT | ATT | AWT | ATT | AWT | ATT | AWT | ATT | AWT | ||
| FixedTime | Rule-based | 453.41 | 51.32 | 370.34 | 35.15 | 384.53 | 36.95 | 497.54 | 36.41 | 408.53 | 53.94 |
| MaxPressure | Rule-based | 274.34 | 32.04 | 246.35 | 22.56 | 245.66 | 24.31 | 289.55 | 21.52 | 349.53 | 67.52 |
| MPLight | Full-shot | 310.54 | 50.45 | 270.14 | 48.03 | 272.37 | 42.91 | 319.32 | 44.05 | 365.42 | 69.85 |
| CoLight | Full-shot | 272.44 | 41.09 | 250.41 | 38.59 | 248.84 | 39.94 | 294.61 | 42.18 | 335.32 | 61.11 |
| Advanced-CoLight | Full-shot | 247.31 | 32.54 | 235.78 | 25.96 | 242.56 | 27.55 | 285.32 | 24.11 | 323.19 | 52.56 |
Traffic Signal Control
Traffic signal control studies how to set intersection phases and green times to reduce congestion while balancing safety, emissions, and priority needs.
See all 50 papers