Reinforcement Learning based Traffic Signal Control for Queue Minimization: From Lane-based to Laneless Traffic
Organizations: Department of Civil Engineering, IIT Madras · Wadhwani School of Data Science & AI, IIT Madras · School of Electrical Engineering & Robotics, QUT, Brisbane, Australia · Robert Bosch Centre for Data Science & AI, IIT Madras
Abstract
Efficient traffic signal control (TSC) is essential for reducing congestion and delays. Traditional TSC methods struggle to capture the dynamics of laneless, heterogeneous traffic. We propose a reinforcement learning (RL) based adaptive TSC framework whose reward depends only on approach-level queue length, motivated by the difficulty of obtaining per-vehicle measurements in laneless traffic, and evaluate it using SUMO on a conventional lane-based, homogeneous (LBHO) intersection and on a non-orthogonal laneless, heterogeneous (LLHE) intersection. On the LBHO site, the lowest mean configuration has about 23% lower mean queue length than a Dynamic Webster baseline, and the unselected Full-State configuration about 15% lower. On the LLHE site, all RL configurations have 66-73% lower mean queue length than a Dynamic Webster baseline under nominal demand, and 63--74% lower under two higher-demand scenarios evaluated without retraining. Our algorithm also outperforms actuated control and a RESCO-DQN baseline on mean queue length. We additionally release a laneless, heterogeneous-vehicle SUMO framework integrated with RL based signal control, to our knowledge the first such open implementation on a current SUMO version.
Figures & tables
| Representation | Dimensions | Key Features | Motivation |
|---|---|---|---|
| PPO (Queue) | 8-D | Basic traffic state | Minimal complexity |
| PPO (Full State) | 19-D | Detailed per-lane info | Capture spatial patterns |
| Autoencoder | Variable | Learned compression | Lower input dimensionality |
| Parameter | Value |
|---|---|
| Encoder layer widths | 19–64–128– |
| Decoder layer widths | –128–64–19 |
| Activation function | ReLU (linear output) |
| Optimizer | Adam |
| Learning rate | |
| Batch size | 256 |
| Vehicle Class | Share (%) |
|---|---|
| Two-wheeler | 64.0 |
| Car | 25.7 |
| Auto-rickshaw | 6.3 |
| Heavy Motor Vehicle | 3.9 |
| Parameter | SUMO Default | This Work |
| Lane-change model | LC2013 (discrete) | SL2015 (sublane) |
| Lateral alignment | center | arbitrary |
| Min. lateral gap ( minGapLat ) | 0.6 m | 0.5 m |
| Lateral resolution | none (no sublanes) | 0.8 m |
| Lateral tracking | Implicit (lane ID) | Continuous offset |
| Handedness | Right-hand | Left-hand |
| Hyperparameter | LBHO | LLHE |
|---|---|---|
| Learning Rate | ||
| Learning-Rate Schedule | constant | linear decay |
| Optimizer | Adam | Adam |
| Batch Size | 64 | 256 |
| Number of Steps | 200 | 2048 |
| Number of Epochs | 10 | 5 |
| Algorithm | Avg. Queue/Cycle | vs. Ref. | Hedges’ |
|---|---|---|---|
| ( ) | (95% CI) | ||
| PPO (Queue AE-16D) † | 53.01 9.63 | – | – |
| PPO (Delay) | 54.26 7.59 | [-8.99, +10.16] | +0.13 |
| PPO (Speed) | 55.36 5.76 | [-7.07, +10.19] | +0.27 |
| PPO (Queue AE-8D) | 57.41 17.37 | [-6.42, +15.23] | +0.28 |
| PPO (Pressure) | 58.57 5.75 | [-3.98, +13.60] | +0.63 |
| Latent | MSE | MSE | Queue Length | vs. 16-D | Hedges’ |
|---|---|---|---|---|---|
| Dim | Mean | Std | (Mean Std) | (95% CI) | |
| 4 | 0.000072 | 0.000011 | 68.22 13.03 | [-2.47, +28.37] | +1.20 |
| 8 | 0.000062 | 0.000002 | 57.41 17.37 | [-6.42, +15.23] | +0.28 |
| 16 † | 0.000063 | 0.000009 | 53.01 9.63 | – | – |
| 19 | 0.000061 | 0.000004 | 58.94 9.21 | [-4.68, +16.55] | +0.57 |
| 32 | 0.000063 | 0.000009 | 62.34 7.28 | [-4.13, +17.81] | +0.99 |
| Latent | MSE | MSE | Queue Length | vs. Ref. | Hedges’ |
|---|---|---|---|---|---|
| Dim | Mean | Std | (Mean Std) | (95% CI) | |
| 4 | 56.27 10.85 | [-1.38, +9.47] | +0.34 | ||
| 8 | 53.85 15.64 | [-12.96, +14.59] | +0.06 | ||
| 16 | 62.80 8.61 | [+4.61, +16.52] | +1.22 | ||
| 19 † | 53.03 5.52 | – | – | ||
| 32 | 59.65 9.53 | [-2.90, +15.60] | +0.77 |
| Controller | Avg Queue | Wait | Speed | vs. Ref. | Hedges’ |
|---|---|---|---|---|---|
| (Mean Std) | (s) | (m/s) | (95% CI) | ||
| PPO (Queue AE-16D) | 30.67 5.46 | 0.80 | 9.53 | [-2.03, +6.02] | +0.45 |
| PPO (Delay) | 34.85 2.89 | 0.83 | 9.52 | [+3.21, +9.15] | +2.47 |
| PPO (Queue AE-8D) † | 28.68 1.36 | 0.73 | 9.62 | – | – |
| PPO (Pressure) | 35.96 5.02 | 0.88 | 9.52 | [+3.83, +10.49] | +1.79 |
| PPO (Full State) | 35.24 7.22 | 0.87 | 9.39 | [+1.67, +13.35] | +1.14 |
| Controller | Avg Queue | Wait | Speed | vs. Ref. | Hedges’ |
|---|---|---|---|---|---|
| (Mean Std) | (s) | (m/s) | (95% CI) | ||
| PPO (Queue AE-16D) | 29.77 5.00 | 0.73 | 9.60 | [-2.05, +5.29] | +0.40 |
| PPO (Delay) | 35.68 4.64 | 0.82 | 9.53 | [+3.31, +11.76] | +1.99 |
| PPO (Queue AE-8D) † | 28.15 1.31 | 0.67 | 9.68 | – | – |
| PPO (Pressure) | 36.18 4.43 | 0.83 | 9.58 | [+5.10, +10.98] | +2.22 |
| PPO (Full State) | 40.21 14.45 | 0.70 | 9.43 | [+2.52, +24.79] | +1.06 |
| Controller | Avg Queue | Wait | Speed | vs. Ref. | Hedges’ |
|---|---|---|---|---|---|
| (Mean Std) | (s) | (m/s) | (95% CI) | ||
| PPO (Queue AE-16D) | 29.69 4.64 | 0.74 | 9.60 | [-2.33, +4.80] | +0.33 |
| PPO (Delay) | 35.49 4.20 | 0.81 | 9.55 | [+3.20, +10.86] | +2.05 |
| PPO (Queue AE-8D) † | 28.46 1.24 | 0.67 | 9.69 | – | – |
| PPO (Pressure) | 36.76 4.18 | 0.80 | 9.60 | [+5.10, +11.19] | +2.44 |
| PPO (Full State) | 31.24 4.09 | 0.69 | 9.57 | [-0.12, +5.69] | +0.83 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Avg. Queue/Cycle | Wait (s) | Speed (m/s) | |
|---|---|---|---|
| ( ) | |||
| (1.0, 0.0) | 32.45 3.71 | 0.80 | 9.53 |
| (0.8, 0.2) | 34.10 3.46 | 0.87 | 9.51 |
| (0.6, 0.4) | 32.41 2.51 | 0.78 | 9.58 |
| (0.4, 0.6) | 30.09 1.59 | 0.71 | 9.61 |
| (0.2, 0.8) | 35.12 2.76 | 0.90 | 9.50 |
| Dim | Features added | Avg. queue (veh) | Wait (s) | Speed (m/s) | 95% CI of | |
|---|---|---|---|---|---|---|
| 3 † | temporal ( , , ) | 54.58 6.49 | 25.87 | 2.810 | – | – |
| 7 | phase one-hot | 60.60 10.72 | 20.61 | 2.114 | [-6.62, +18.38] | +0.61 |
| 11 | per-approach queue | 62.73 8.30 | 26.86 | 2.277 | [-11.46, +15.55] | +0.20 |
| 15 | queue change | 53.70 16.98 | 20.57 | 2.760 | [-22.48, +0.44] | -0.61 |
| 19 | green times (= Full State) | 58.86 5.68 | 26.46 | 2.378 | [-7.28, +17.60] | +0.37 |
| Dim | Features added | Avg. queue (veh) | Wait (s) | Speed (m/s) | 95% CI of | |
|---|---|---|---|---|---|---|
| 3 † | temporal ( , , ) | 59.09 1.48 | 21.80 | 2.386 | – | – |
| 7 | phase one-hot | 63.90 6.22 | 23.56 | 1.985 | [-0.44, +9.43] | +0.96 |
| 11 | per-approach queue | 57.17 8.06 | 26.01 | 2.749 | [-13.62, +0.29] | -0.84 |
| 15 | queue change | 58.97 9.97 | 24.46 | 2.365 | [-4.31, +9.34] | +0.18 |
| 19 | green times (= Full State) | 70.31 1.85 | 29.58 | 1.672 | [+3.36, +19.15] | +1.43 |
| Dim | Features added | Avg. queue (veh) | Wait (s) | Speed (m/s) | 95% CI of | |
|---|---|---|---|---|---|---|
| 3 † | temporal ( , , ) | 35.67 2.16 | 0.79 | 9.549 | – | – |
| 7 | phase one-hot | 30.73 3.39 | 0.83 | 9.513 | [-8.16, -1.63] | -1.57 |
| 11 | per-approach queue | 29.91 3.20 | 0.77 | 9.538 | [-2.80, +1.16] | -0.22 |
| 15 | queue change | 31.29 3.68 | 0.79 | 9.526 | [-2.24, +5.57] | +0.36 |
| 19 | green times (= Full State) | 30.56 2.37 | 0.69 | 9.617 | [-4.29, +2.83] | -0.21 |
Traffic Signal Control
Traffic signal control studies how to set intersection phases and green times to reduce congestion while balancing safety, emissions, and priority needs.
See all 49 papers