Scenario Orchestration for Closed-loop Policy Evaluation of Autonomous Driving
Organizations: Department of Computer Science, University of Toronto, Toronto, ON M5S 2E4, Canada. · Department of Electrical and Computer Engineering, McGill University, Montreal, QC H3A 0E9, Canada.
Abstract
Scenario-based simulation is central to evaluating autonomous-driving policies, but a scenario scripted before execution (i.e., with predetermined, fixed actor routes and speeds) may not preserve its intended interaction once the tested ego policy responds in closed loop. Scripted scenarios cannot adapt to unanticipated ego behavior, while learned traffic actors may remain reactive without reliably realizing a particular scenario intent. We introduce a scenario orchestrator that continuously monitors the joint traffic state and adapts non-ego actors through re-timing, re-routing, and role reassignment for creating a specific scenario intent. At each step, the orchestrator solves a constrained optimization problem that preserves the intended interaction while limiting interventions and maintaining traffic feasibility. We further distinguish scenario realization from policy failure, allowing a test to first be judged based on whether the intended interaction occurred and then by how the ego policy responded. Experiments in the CARLA simulator across multiple scenario families and driving policies show that orchestration realizes targeted interactions more reliably than scripted and learned actor-level baselines, particularly under variations in initial conditions and ego behavior. These results support closed-loop scenario orchestration as a tool for controlled and reproducible policy evaluation.
Figures & tables
| Scenarios | Initial Scene | Intention Scene | Terminal Scene |
| Red-light | Ego approaches a signalised junction on green, heading straight; cross traffic approaches on red | A cross-traffic actor runs its red light toward the ego’s path, within the ego’s view ( s in view) and within 1 m of the ego (box to box) | Violator occupies the conflict zone while the ego is still approaching it (within 40 m), with TTC s |
| Left-turn | Ego approaches a junction to make an unprotected left turn; oncoming traffic approaches on the opposite side of the junction | An oncoming actor drives through the junction across the ego’s turning path, within the ego’s view ( s in view) and within 1 m of the ego (box to box) | Oncoming actor occupies the conflict zone while the ego is still approaching it (within 40 m), with TTC s |
| Right-turn | Ego approaches a junction to turn right; crossing traffic approaches from the ego’s left along the lane the ego turns into | That actor keeps driving along the lane toward the ego’s merge point, within the ego’s view ( s in view) and within 1 m of the ego (box to box) | Crossing actor reaches the ego’s merge point while the ego is within 40 m of it, before or after it, with TTC s |
| Cut-in | Actor starts on adjacent lane of the ego | Actor changes lanes into the ego’s lane, in front of the ego, within the ego’s view ( s in view) and within 1 m of the ego (box to box) | Actor completes the merge 0.5–10 m ahead of the ego and keeps driving forward at a nominal speed |
| Hard-brake | Lead actor cruises ahead in the ego’s lane ( m/s); a second actor drives in the adjacent lane, 2–35 m from the lead | Once the ego is within 4.5 s headway (or 6 m), the lead brakes hard ( m/s 2 ) while the adjacent actor keeps its speed | Ego closes to within 4 m of the braked lead, or pulls out of its lane to go round it |
| Over-take | Ego follows a blocker in its lane on a two-way road; an oncoming actor approaches in the otherwise clear opposite lane, m beyond the blocker | Blocker brakes to a stop, so the only way past is the opposite lane, against the oncoming actor | Oncoming actor passes within 4 m of the ego while the ego is held behind the blocker or pulls out to pass |