ProcessLight: Process Supervision for Large Language Model Based Traffic Signal Control

Huaitao Zhao; Tianlong Zhou; Weijie Wang; Jiasheng Shi; Weixiong Rao

cs.AISep 19, 2026

ProcessLight: Process Supervision for Large Language Model Based Traffic Signal Control

Authors: Huaitao Zhao, Tianlong Zhou, Weijie Wang, Jiasheng Shi, Weixiong Rao

Organizations: School of Computer Science and Technology, Tongji University, Shanghai, China

Abstract

Large Language Models (LLMs) have recently been introduced into traffic signal control (TSC) as decision agents due to their strengths in human-readable reasoning generation. Yet, existing LLM TSC methods optimize only from final outcomes and fail to distinguish valid from flawed reasoning steps, causing useful or misleading steps to be jointly updated and thus impairing the model's learning of effective reasoning. To bridge this gap, we propose an LLM-based framework ProcessLight to decompose signal decisions into verifiable semantic steps. Building on ProcessLight, we further develop Step-wise Traffic Process Policy Optimization (STeP-PO), a novel reinforcement learning framework that optimizes structured reasoning processes through step-level credit assignment. Specifically, STeP-PO uses step quality scores to evaluate local reasoning quality and step importance to measure each step's influence on the final action, and then assigns step-level advantages over a semantic step tree structure. The resulting step-level advantages are propagated to reasoning tokens, enabling fine-grained policy optimization beyond outcome-only rewards. Extensive experiments over multiple real-world datasets demonstrate the superiority of our methods. Our code is available at https://github.com/wenzhaoabc/processlight.

Figures & tables

Figure 1: Outcome-level supervision cannot effectively distinguish correct and incorrect reasoning steps in the traffic signal reasoning process, weakening the model’s ability to learn correct reasoning.

Figure 2: Overview of ProcessLight with two stages, (1) task prompt construction and (2) step-wise reasoning. The proposed STeP-PO algorithm is used for policy optimization.

Figure 3: Semantic step tree construction. Reasoning steps with soft-matched claims are merged into the same node. The global state value is recursively propagated from child nodes, while the local step-level state value is computed from the step quality rewards.

Flow dataset	Structure	Vehicles	Arrival rate (vehicles/5min)
Flow dataset	Structure	Vehicles	Mean	Std	Max	Min
Jinan 1	3 $\times$ 4	6295	523.67	98.52	671	255
Jinan 2		4365	362.83	74.81	493	236
Jinan 3		5494	456.92	46.38	544	362
Hangzhou 1	4 $\times$ 4	2983	247.67	40.44	332	211
Hangzhou 2	4 $\times$ 4	6984	581.08	318.43	1145	202

Table 1: Statistics of datasets.

Traditional Methods
Method	Jinan 1			Jinan 2			Jinan 3			Hangzhou 1			Hangzhou 2
Method	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$
FixedTime	410.07	157.00	190.36	521.91	232.12	405.89	463.07	184.99	282.31	564.91	204.65	169.57	553.78	137.69	267.12
MaxPressure	305.15	59.93	72.67	318.12	77.98	136.36	298.02	62.23	94.96	343.97	47.08	39.01	438.54	102.06	197.99
RL Methods
EfficientPressLight	316.85	77.93	136.27	310.08	66.08	80.12	302.16	67.20	102.56	472.23	178.99	148.31	502.59	172.35	334.37

Table 2: Zero-shot traffic control performance on real-world traffic flows. Lower values indicate better performance. The best results are in bold and the second-best results are underlined .

Variant	Average Travel Time (s) $\downarrow$
Variant	Jinan 1	Jinan 2	Jinan 3	Hangzhou 1	Hangzhou 2
Qwen2.5-3B	327.90	356.37	327.34	353.87	438.37
+ Warm-up	320.91	344.55	319.17	345.61	432.92
+ SQ	300.64	315.18	298.33	330.99	418.24
+ SI	293.23	312.87	292.18	318.72	409.82
+ SSTA (ProcessLight)	289.34	293.18	284.33	314.99	407.24

Table 3: Ablation study on five real-world datasets.

Training Method	Average Travel Time (s) $\downarrow$
Training Method	Jinan 1	Hangzhou 1
Warm-up only	321.23	352.12
Warm-up + Outcome-level Reward	305.74	334.25
Warm-up + Step-wise Reward	293.64	321.47
Warm-up + STeP-PO	289.34	314.99

Table 4: Effectiveness of process-level supervision.

Panel of Figure 4 — Figure 4: Training dynamics of reward-side indicators.

Supervision	Advantage-Sign Accuracy
Outcome-only	78%
Step-Quality	100%

Table 5: Accuracy of step-level advantage signs under controlled noise injection.

Figure 5: Step importance across traffic regimes. The x-axis denotes reasoning steps, and the y-axis groups traffic regimes by the dominant factor behind the simulator-optimal phase. Each cell shows the normalized importance of a step; higher values indicate a larger change in phase-selection probability after masking that step.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustration of (a) intersections, (b) traffic movements, and (c) signal phases.

#	$\eta$	$\lambda$	$G$	Jinan1 ATT $\downarrow$	Hangzhou1 ATT $\downarrow$
1	0.3	0.2	4	307.87	337.24
2	0.3	0.3	8	291.62	319.33
3	0.6	0.3	8	289.34	314.99
4	0.6	0.4	8	296.76	320.52
5	0.9	0.3	12	294.48	321.05
6	0.9	0.5	12	302.15	329.41

Appendix

Table 6: Hyperparameter sensitivity of STeP-PO on Jinan1 and Hangzhou1.

Traditional Methods
Method	Jinan 1			Hangzhou 1
Method	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$
FixedTime	409.73 $\pm$ 0.00	156.76 $\pm$ 0.00	190.07 $\pm$ 0.00	564.86 $\pm$ 0.00	204.63 $\pm$ 0.00	169.56 $\pm$ 0.00
MaxPressure	305.26 $\pm$ 0.00	60.19 $\pm$ 0.00	72.97 $\pm$ 0.00	343.94 $\pm$ 0.00	47.09 $\pm$ 0.00	39.02 $\pm$ 0.00
RL Methods
EfficientPressLight	316.85 $\pm$ 0.00	77.93 $\pm$ 0.00	136.27 $\pm$ 0.00	472.22 $\pm$ 0.01	178.99 $\pm$ 0.00	148.31 $\pm$ 0.00

Appendix

Table 7: Mean and standard deviation over five random seeds on Jinan1 and Hangzhou1. Table 2 reports results under the default seed. Best results are in bold.

Component	Execution Accuracy
Claim extraction	99.81 $\pm$ 0.02%
Semantic matching	96.87 $\pm$ 0.04%
Observation validity	97.25 $\pm$ 0.46%

Appendix

Table 8: Execution accuracy of the three verification stages.

Stage	Model / Environment	Calls	Token Usage	Runtime	Cost
Reference claims	DeepSeek-V4-Pro, Cloud API	3k	7.07M in + 1.70M out	–	$9.1
Step importance	Qwen2.5-3B, 1 $\times$ RTX PRO 6000	21k	57.81M in; no output	1.5 GPU-h	$1.2

Appendix

Table 9: Cost of the two data-construction stages.

Method	Jinan 1			Jinan 2			Jinan 3			Hangzhou 1			Hangzhou 2
Method	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$
LLMLight	297.28	52.28	63.39	310.48	70.66	123.56	291.86	55.96	85.40	337.21	41.60	34.47	419.60	84.91	164.73
ProcessLight w/o Neighbor Info.	300.91	48.87	65.63	304.32	68.35	119.47	299.48	56.23	81.59	332.06	39.14	34.38	414.50	83.65	168.67

Appendix

Table 10: Zero-shot performance under restricted observation without neighboring-intersection information. Best results are in bold.

Method	Teacher	Jinan 1			Jinan 2			Jinan 3			Hangzhou 1			Hangzhou 2
Method	Teacher	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$	ATT $\downarrow$	AWT $\downarrow$	AQL $\downarrow$
LLMLight	–	297.28	52.28	63.39	310.48	70.66	123.56	291.86	55.96	85.40	337.21	41.60	34.47	419.60	84.91	164.73
CoLLMLight	–	298.99	54.35	65.90	311.23	71.06	124.25	292.25	56.58	86.35	336.25	41.30	34.22	419.96	74.02	143.59
Traffic-R1	–	299.14	54.15	65.65	310.79	70.54	123.34	291.96	56.06	85.56	337.46	40.94	33.92	421.28	81.03	157.20
ProcessLight (Qwen2.5-3B)	DeepSeek-V4-Pro	289.34	45.03	55.94	293.18	54.16	102.67	284.33	50.97	78.20	314.99	32.82	30.48	407.24	73.13	154.21
ProcessLight (Qwen3-8B)	DeepSeek-V4-Pro	288.81	43.18	54.49	295.60	54.57	100.39	285.15	48.43	77.18	306.95	30.98	30.13	401.47	71.06	152.25

Appendix

Table 11: Zero-shot performance with different backbones and teacher models. Best results are in bold.

Language Model-Based Control

Language model-based control uses language models to interpret goals, choose actions, and steer AI systems or physical processes.

See all 178 papers

DGLight: DQN-Guided GRPO Fine-Tuning of Large Language Models for Traffic Signal Control
Apr 28, 2026Chenbo YuLanguage Model-Based Control Intelligent Transportation Systems
National University of Singapore Singapore
CuraLight: Debate-Guided Data Curation for LLM-Centered Traffic Signal Control
Apr 7, 2026Qing Guo, Xinhang Li, Junyu Chen +4Traffic Signal Control Language Model-Based Control
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China · Beijing Big Data Center, Beijing, China
VLALight: A Vision-Language-Action Model for Traffic Signal Control
Sep 29, 2026Pan Zhang, Siqi Lai, Kemu Dong +1Intelligent Transportation Systems Vision-Language-Action Models
The Hong Kong University of Science and Technology (Guangzhou) · Dalian University of Technology

Apr 28, 2026J/K move · Enter open · S save

DGLight: DQN-Guided GRPO Fine-Tuning of Large Language Models for Traffic Signal Control

Chenbo Yu

National University of Singapore Singapore

Language Model-Based Control Intelligent Transportation Systems Traffic Signal Control RL Fine-Tuning

Open paper AskPDF arXiv

Traffic signal control (TSC) plays a central role in reducing congestion and maintaining urban mobility. This dissertation introduces DGLight, a critic-guided reinforcement-learning framework for adapting a pretrained large language model to TSC. DGLight first trains a CoLight-based Deep Q-Network critic to estimate traffic-aware action values from structured intersection states, then uses the frozen critic to score candidate language-model actions and optimize the policy with Group Relative Policy Optimization (GRPO). The resulting controller maps traffic states to interpretable reasoning traces and signal decisions while learning from dense per-state supervision rather than raw cumulative environment rewards. Experiments on TSC benchmarks covering Jinan and Hangzhou show that DGLight is the strongest overall method among the compared LLM-based controllers, remains competitive with strong RL baselines, and transfers well to city datasets not used to fit the critic. Qualitative examples further show that the model's generated reasoning is interpretable and aligned with the chosen signal phase. The project code is available here.

Abstract

Language Model-Based Control

Explore similar work

DGLight: DQN-Guided GRPO Fine-Tuning of Large Language Models for Traffic Signal Control

CuraLight: Debate-Guided Data Curation for LLM-Centered Traffic Signal Control

VLALight: A Vision-Language-Action Model for Traffic Signal Control