Discovering Programmatic Policies from Reinforcement Learning-Based Traffic Signal Controllers
Organizations: Department of Electrical and Electronic Engineering The Hong Kong Polytechnic University, Hong Kong, China · School of Computer Science and Engineering University of New South Wales, Sydney, Australia
Abstract
Traffic signal control (TSC) is critical for mitigating urban congestion. Deep reinforcement learning (DRL) has achieved strong performance by learning adaptive policies from traffic observations, but learned neural controllers are often opaque, structurally complex, and difficult to generalize across scenarios. We propose PPL-TSC, a programmatic policy learning framework that distills compact and human-readable TSC policies from DRL teachers. PPL-TSC formulates policy learning as an evolutionary search over phase utility functions (PUFs), which score candidate signal phases and select the highest-utility phase for execution. Each generation consists of four steps: policy generation, policy review, policy calibration, and policy evaluation. In the generation step, a large language model (LLM) acts as a semantics-aware evolutionary operator to propose and refine candidate PUFs. Generated PUFs are screened through explicit checks for syntactic validity, mathematical soundness, and consistency with TSC principles, with invalid ones repaired through iterative feedback. Valid PUFs are then calibrated to improve behavioral fidelity to the DRL teacher, and evaluated using a joint score combining teacher fidelity and traffic-control performance to select candidates for the next generation. Extensive experiments across diverse traffic scenarios and representative DRL-based controllers show that PPL-TSC produces compact and human-readable policies, achieves competitive performance for most teacher families, improves transfer generalization, and substantially reduces computational cost. Code is available at https://github.com/ForrestXie9/PPL-TSC.
Figures & tables
| Teacher | PPL-TSC (Tailored) | PPL-TSC (Unified) | ||||
|---|---|---|---|---|---|---|
| ATT (%) | ATT wins | ABF | ATT (%) | ATT wins | ABF | |
| CoLight | 17/35 | 18/35 | ||||
| FRAP | 24/35 | 22/35 | ||||
| MPLight | 21/35 | 17/35 | ||||
| EcoLight (native) | 15/35 | 8/35 | ||||
| EcoLight (phase-selection) | 30/35 | 30/35 | ||||
| (a) Same-scenario, different-traffic-flow | ||
|---|---|---|
| Teacher | ATT (%) | ATT wins |
| CoLight | 61/140 | |
| EcoLight (native) | 122/140 | |
| EcoLight (phase-selection) | 120/140 | |
| FRAP | 89/140 | |
| MPLight | 55/140 | |
| (a) Policy calibration | |||
|---|---|---|---|
| Variant | ATT (%) | ATT wins | ABF |
| Full PPL-TSC | 7.62 28.80 | 113/175 | 0.64 0.14 |
| w/o calibration | 9.78 34.96 | 116/175 | 0.55 0.18 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Check | Description |
| Syntactic | Parseability | The generated function must have valid Python syntax and compile successfully. |
| Executability | The generated function must be loadable and callable at runtime. | |
| Structural validity | The generated function must have a valid structure and return a value on every execution path. | |
| Mathematical | Finite output | Output must be a finite real number (no NaN or infinite values). |
| Domain safety | No invalid operations (e.g., division by zero, invalid logarithms, or square roots of negative inputs). | |
| Runtime robustness | No exceptions under real teacher states and extreme input scenarios (e.g., all-zero, all-max, mixed, single-feature extremes). |
| Component | Complexity |
|---|---|
| Policy generation | |
| Policy review | per reviewed candidate |
| Policy calibration | per candidate |
| Policy evaluation | for cached agreement; for simulation |
| Deployment | per decision step |
| Scenario | ATT | ABF | |||
|---|---|---|---|---|---|
| Teacher | Tailored | Unified | Tailored | Unified | |
| Hangzhou1 | 96.00 5.20 | 94.31 3.45 | 98.63 4.76 | 0.66 0.06 | 0.60 0.06 |
| Hangzhou2 | 122.47 4.06 | 123.52 2.71 | 127.24 3.66 | 0.68 0.08 | 0.62 0.09 |
| Hangzhou3 | 80.08 1.87 | 82.39 1.15 | 84.65 2.37 | 0.73 0.04 | 0.71 0.04 |
| Atlanta | 311.00 24.27 | 311.36 58.31 | 325.75 65.71 | 0.73 0.17 | 0.65 0.14 |
| Jinan | 1109.23 311.54 | 937.99 433.65 | 358.95 27.54 | 0.60 0.31 | 0.15 0.04 |
| Scenario | ATT | ABF | |||
|---|---|---|---|---|---|
| Teacher | Tailored | Unified | Tailored | Unified | |
| Hangzhou1 | 128.94 31.88 | 109.79 19.76 | 99.04 5.85 | 0.69 0.16 | 0.61 0.23 |
| Hangzhou2 | 150.24 56.75 | 128.03 14.62 | 122.65 1.15 | 0.62 0.27 | 0.59 0.31 |
| Hangzhou3 | 132.47 110.34 | 134.01 115.03 | 85.05 2.30 | 0.72 0.05 | 0.58 0.30 |
| Atlanta | 258.77 25.82 | 333.29 31.27 | 364.11 13.07 | 0.75 0.10 | 0.66 0.11 |
| Jinan | 307.61 15.17 | 312.17 2.58 | 316.78 1.92 | 0.62 0.02 | 0.59 0.03 |
| Scenario | ATT | ABF | |||
|---|---|---|---|---|---|
| Teacher | Tailored | Unified | Tailored | Unified | |
| Hangzhou1 | 100.83 3.36 | 97.33 6.23 | 100.04 1.62 | 0.76 0.07 | 0.72 0.07 |
| Hangzhou2 | 121.81 1.32 | 121.97 1.33 | 122.89 2.44 | 0.74 0.10 | 0.71 0.10 |
| Hangzhou3 | 82.72 2.73 | 82.48 2.10 | 83.51 2.42 | 0.89 0.04 | 0.88 0.04 |
| Atlanta | 249.45 15.03 | 236.32 5.22 | 241.55 4.61 | 0.79 0.11 | 0.71 0.15 |
| Jinan | 297.07 7.04 | 307.66 4.84 | 314.77 6.00 | 0.72 0.10 | 0.68 0.08 |
| Scenario | ATT | ABF | |||
|---|---|---|---|---|---|
| Teacher | Tailored | Unified | Tailored | Unified | |
| Hangzhou1 | 205.88 37.67 | 183.19 25.05 | 194.82 15.38 | 0.81 0.05 | 0.77 0.05 |
| Hangzhou2 | 135.03 0.54 | 134.63 2.11 | 135.92 2.79 | 0.91 0.04 | 0.90 0.04 |
| Hangzhou3 | 98.05 1.77 | 96.31 1.24 | 97.23 1.52 | 0.93 0.02 | 0.92 0.03 |
| Atlanta | 311.47 32.78 | 329.75 20.98 | 380.94 38.96 | 0.79 0.11 | 0.77 0.05 |
| Jinan | 378.74 3.97 | 765.00 314.15 | 736.17 166.60 | 0.84 0.01 | 0.81 0.02 |
| Scenario | ATT | ABF | |||
|---|---|---|---|---|---|
| Teacher | Tailored | Unified | Tailored | Unified | |
| Hangzhou1 | 205.88 42.11 | 126.10 31.32 | 115.29 26.62 | 0.43 0.09 | 0.29 0.06 |
| Hangzhou2 | 135.03 0.60 | 127.90 3.37 | 124.61 4.31 | 0.49 0.03 | 0.38 0.08 |
| Hangzhou3 | 98.05 1.98 | 83.73 2.59 | 86.99 1.66 | 0.65 0.03 | 0.62 0.02 |
| Atlanta | 311.47 36.65 | 358.34 55.18 | 381.22 11.42 | 0.56 0.05 | 0.48 0.07 |
| Jinan | 378.74 4.44 | 324.74 9.79 | 320.34 3.63 | 0.39 0.02 | 0.28 0.03 |
| Scenario | ATT | ABF | |||
|---|---|---|---|---|---|
| Teacher | Tailored | Unified | Tailored | Unified | |
| Hangzhou1 | 98.76 5.81 | 97.11 6.78 | 105.11 15.31 | 0.67 0.07 | 0.57 0.06 |
| Hangzhou2 | 121.84 1.20 | 119.83 0.78 | 123.16 5.12 | 0.66 0.08 | 0.57 0.09 |
| Hangzhou3 | 81.17 1.22 | 81.53 1.11 | 83.86 3.85 | 0.82 0.08 | 0.70 0.09 |
| Atlanta | 278.47 12.85 | 359.23 54.13 | 375.47 30.47 | 0.69 0.05 | 0.65 0.07 |
| Jinan | 299.54 2.83 | 324.91 89.72 | 397.69 128.98 | 0.31 0.11 | 0.42 0.10 |
| Teacher | Configuration | ABF | ATT (%) |
|---|---|---|---|
| EcoLight (native) | Original inputs | 0.84 | |
| Aligned inputs | 0.94 | 2.71 | |
| TinyLight | Search budget | 0.64 | |
| Search budget | 0.71 | 1.51 |
| Source Scenario | Teacher ATT | PPL-CoLight ATT | ATT (%) | ATT wins |
|---|---|---|---|---|
| Hangzhou1 | 96.74 4.70 | 99.36 2.41 | -2.96 5.72 | 4/20 |
| Hangzhou2 | 122.85 3.26 | 122.36 1.07 | 0.33 2.61 | 6/20 |
| Hangzhou3 | 81.04 1.28 | 86.72 1.18 | -7.04 2.17 | 0/20 |
| Atlanta | 313.68 20.93 | 402.94 28.94 | -28.56 6.21 | 0/20 |
| Jinan | 1114.97 254.41 | 751.00 474.67 | 35.48 32.05 | 15/20 |
| Los Angeles | 846.71 16.73 | 636.83 188.32 | 24.57 23.06 | 16/20 |
| Same-Scenario, Different-Flow | ||||
|---|---|---|---|---|
| Source Scenario | Teacher ATT | PPL-FRAP ATT | ATT (%) | ATT wins |
| Hangzhou1 | 126.50 27.48 | 100.03 2.89 | 18.10 13.19 | 20/20 |
| Hangzhou2 | 142.02 40.37 | 122.95 1.49 | 8.99 15.76 | 18/20 |
| Hangzhou3 | 134.77 101.22 | 86.61 1.07 | 12.54 30.97 | 7/20 |
| Atlanta | 254.60 16.28 | 389.20 25.79 | -53.10 8.86 | 0/20 |
| Jinan | 307.89 13.30 | 319.69 1.61 | -4.00 4.01 | 4/20 |
| Same-Scenario, Different-Flow | ||||
|---|---|---|---|---|
| Source Scenario | Teacher ATT | PPL-MPLight ATT | ATT (%) | ATT wins |
| Hangzhou1 | 100.72 5.51 | 98.96 7.21 | 1.80 3.13 | 15/20 |
| Hangzhou2 | 122.42 1.72 | 122.19 1.44 | 0.18 0.91 | 11/20 |
| Hangzhou3 | 82.86 1.23 | 84.43 1.50 | -1.89 1.48 | 2/20 |
| Atlanta | 251.50 17.14 | 243.17 11.01 | 2.87 7.81 | 10/20 |
| Jinan | 297.09 6.17 | 310.32 1.13 | -4.50 2.12 | 1/20 |
| Same-Scenario, Different-Flow | ||||
|---|---|---|---|---|
| Source Scenario | Teacher ATT | PPL-EcoLight ATT | ATT (%) | ATT wins |
| Hangzhou1 | 209.20 35.75 | 98.73 3.15 | 51.52 7.63 | 20/20 |
| Hangzhou2 | 134.42 1.02 | 122.49 1.27 | 8.87 1.15 | 20/20 |
| Hangzhou3 | 97.86 3.08 | 86.62 1.21 | 11.40 2.95 | 20/20 |
| Atlanta | 312.77 31.81 | 373.94 5.14 | -20.60 10.34 | 2/20 |
| Jinan | 379.52 6.98 | 320.75 1.82 | 15.46 1.39 | 20/20 |
| Same-Scenario, Different-Flow | ||||
|---|---|---|---|---|
| Source Scenario | Teacher ATT | PPL-EcoLight ATT | ATT (%) | ATT wins |
| Hangzhou1 | 209.20 35.75 | 107.25 12.43 | 47.05 11.14 | 20/20 |
| Hangzhou2 | 134.42 1.02 | 123.58 1.66 | 8.06 1.43 | 20/20 |
| Hangzhou3 | 97.86 3.08 | 87.07 1.80 | 10.92 3.59 | 20/20 |
| Atlanta | 312.77 31.81 | 376.35 1.39 | -21.42 10.78 | 0/20 |
| Jinan | 379.52 6.98 | 324.88 7.63 | 14.34 3.33 | 20/20 |
| Source Scenario | Teacher ATT | PPL-TinyLight ATT | ATT (%) | ATT wins |
|---|---|---|---|---|
| Hangzhou1 | 99.85 7.36 | 95.99 3.09 | 3.56 4.48 | 16/20 |
| Hangzhou2 | 121.68 1.44 | 119.59 1.27 | 1.71 0.96 | 20/20 |
| Hangzhou3 | 81.76 1.10 | 81.19 1.30 | 0.70 1.30 | 15/20 |
| Atlanta | 276.85 16.05 | 320.72 46.04 | -16.94 22.24 | 5/20 |
| Jinan | 299.86 3.14 | 284.36 1.94 | 5.16 1.18 | 20/20 |
| Los Angeles | 466.54 39.77 | 490.47 98.35 | -5.02 17.37 | 10/20 |
| Teacher interface | ATT (%) | ATT wins |
|---|---|---|
| CoLight | 0.66 | 35/70 |
| EcoLight (phase-selection) | 2.66 | 48/70 |
| EcoLight (native) | 1.07 | 30/70 |
| FRAP | 1.26 | 38/70 |
| MPLight | 1.19 | 40/70 |
| TinyLight | -0.15 | 32/70 |
| Method | Cross-scenario ATT (s) | Best pairs | Mean rank |
|---|---|---|---|
| PPL-TSC | 232.51 154.20 (217.01) | 24/49 | 1.94 |
| -Light-Distill | 305.99 274.68 (230.98) | 3/49 | 3.21 |
| GPLight+-Distill | 301.32 286.78 (234.88) | 11/49 | 2.54 |
| SymLight-Distill | 275.23 249.27 (220.53) | 11/49 | 2.31 |
| Source / Target | Atlanta | Hangzhou1 | Hangzhou2 | Hangzhou3 | Jinan | Los Angeles | Manhattan |
|---|---|---|---|---|---|---|---|
| Atlanta | 239.05 / 236.23 / 236.24 / 235.12 | 98.57 / 100.18 / 100.94 / 98.20 | 121.98 / 122.96 / 122.89 / 122.32 | 85.06 / 85.96 / 86.08 / 85.35 | 310.48 / 972.14 / 451.72 / 327.69 | 446.30 / 494.71 / 473.96 / 523.34 | 216.61 / 551.18 / 284.36 / 217.70 |
| Hangzhou1 | 238.91 / 307.04 / 246.65 / 262.26 | 98.97 / 97.50 / 100.13 / 101.19 | 121.98 / 122.89 / 122.34 / 122.11 | 85.09 / 84.01 / 85.24 / 84.90 | 311.48 / 342.63 / 330.78 / 325.05 | 458.47 / 763.85 / 604.76 / 700.44 | 217.37 / 245.91 / 256.26 / 229.73 |
| Hangzhou2 | 239.32 / 245.45 / 243.67 / 248.47 | 99.13 / 96.18 / 95.20 / 96.18 | 122.21 / 122.07 / 122.25 / 122.19 | 85.33 / 84.66 / 84.35 / 84.66 | 316.33 / 331.47 / 308.03 / 313.59 | 454.33 / 487.48 / 453.35 / 490.67 | 220.93 / 230.42 / 216.49 / 222.03 |
| Hangzhou3 | 246.89 / 245.41 / 244.26 / 308.25 | 95.70 / 100.12 / 93.81 / 95.97 | 122.14 / 122.49 / 121.99 / 122.03 | 84.43 / 85.07 / 83.86 / 83.56 | 316.08 / 321.09 / 309.96 / 303.85 | 470.13 / 484.35 / 542.14 / 763.90 | 210.03 / 221.39 / 224.13 / 218.08 |
| Jinan | 238.90 / 238.67 / 238.57 / 239.04 | 99.03 / 98.66 / 98.94 / 98.31 | 121.98 / 122.32 / 122.62 / 122.10 | 85.09 / 85.51 / 85.51 / 84.82 | 310.53 / 310.86 / 309.31 / 309.12 | 445.59 / 460.05 / 483.16 / 472.98 | 216.53 / 218.87 / 214.51 / 213.95 |
| Los Angeles | 240.33 / 243.00 / 341.65 / 241.37 | 96.87 / 102.88 / 106.42 / 100.10 | 122.00 / 123.42 / 125.53 / 123.30 | 85.32 / 86.08 / 91.05 / 86.08 | 313.00 / 941.63 / 339.10 / 355.16 | 466.45 / 490.53 / 461.39 / 466.50 | 217.82 / 226.46 / 215.96 / 221.52 |
| Teacher / task | Phase-scoring component | Traffic-control interpretation | ABF | ATT (s) | |
|---|---|---|---|---|---|
| Teacher | Student | ||||
| CoLight Atlanta / flow 3 | Prioritizes demand; a quadratic bonus retains a busy active phase. | 0.94 | 339.20 | 317.90 | |
| EcoLight (phase-selection) Hangzhou2 / flow 2 | Combines total and peak demand with an active-phase retention bonus. | 0.94 | 135.93 | 134.13 | |
| EcoLight (native, aligned) Hangzhou2 / flow 2 | Keeps green for served demand; switches as red-side demand grows relative to green-side demand. | 0.99 | 135.93 | 135.05 | |
| FRAP Atlanta / flow 1 | Scores aggregate vehicle demand linearly and adds a fixed bonus for the active phase. | 0.89 | 255.03 | 245.01 | |
| MPLight Hangzhou3 / flow 4 | Combines aggregate and bottleneck pressure with a retention bonus, following MaxPressure [ 7 ] . | 0.93 | 86.01 | 84.04 | |
| Metric | Mean |
|---|---|
| Final joint-score CV | 2.08% |
| Token usage per search | 67.1K |
| Generation cost per search | USD 0.0353 |
| LLM API latency per search | 104.2 s |
Traffic Signal Control
Traffic signal control studies how to set intersection phases and green times to reduce congestion while balancing safety, emissions, and priority needs.
See all 50 papers