cs.RONov 6, 2026

Scenario Orchestration for Closed-loop Policy Evaluation of Autonomous Driving

Authors: Frieda Rong, Zhaoyue Wang, Veer Kapadia, Vishaka Rajani, Aren A. Babikian, Xujie Si, Mahsa Golchoubian, Raquel Urtasun, 

Organizations: Department of Computer Science, University of Toronto, Toronto, ON M5S 2E4, Canada. · Department of Electrical and Computer Engineering, McGill University, Montreal, QC H3A 0E9, Canada.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scenario-based simulation is central to evaluating autonomous-driving policies, but a scenario scripted before execution (i.e., with predetermined, fixed actor routes and speeds) may not preserve its intended interaction once the tested ego policy responds in closed loop. Scripted scenarios cannot adapt to unanticipated ego behavior, while learned traffic actors may remain reactive without reliably realizing a particular scenario intent. We introduce a scenario orchestrator that continuously monitors the joint traffic state and adapts non-ego actors through re-timing, re-routing, and role reassignment for creating a specific scenario intent. At each step, the orchestrator solves a constrained optimization problem that preserves the intended interaction while limiting interventions and maintaining traffic feasibility. We further distinguish scenario realization from policy failure, allowing a test to first be judged based on whether the intended interaction occurred and then by how the ego policy responded. Experiments in the CARLA simulator across multiple scenario families and driving policies show that orchestration realizes targeted interactions more reliably than scripted and learned actor-level baselines, particularly under variations in initial conditions and ego behavior. These results support closed-loop scenario orchestration as a tool for controlled and reproducible policy evaluation.

Figures & tables

Fig. 1: Taxonomy of scenario execution methods categorized by how they condition on test intent and what they model. The icon indicates our ideal scenario execution method.
Fig. 2: Our orchestrator monitors the arrival of the ego to the conflict region and dynamically adjusts the non-ego (“ado”) actors to ensure a safety-critical interaction. In this example, the orchestrator responds to a delayed ego by changing the currently assigned hero actor to an NPC that is further away from the intersection so that a collision ( ) is ensured.
Table 3
ScenariosInitial SceneIntention SceneTerminal Scene
Red-lightEgo approaches a signalised junction on green, heading straight; cross traffic approaches on redA cross-traffic actor runs its red light toward the ego’s path, within the ego’s view ( ≥0.5\geq 0.5 s in view) and within 1 m of the ego (box to box)Violator occupies the conflict zone while the ego is still approaching it (within 40 m), with TTC ≤3\leq 3 s
Left-turnEgo approaches a junction to make an unprotected left turn; oncoming traffic approaches on the opposite side of the junctionAn oncoming actor drives through the junction across the ego’s turning path, within the ego’s view ( ≥0.5\geq 0.5 s in view) and within 1 m of the ego (box to box)Oncoming actor occupies the conflict zone while the ego is still approaching it (within 40 m), with TTC ≤3\leq 3 s
Right-turnEgo approaches a junction to turn right; crossing traffic approaches from the ego’s left along the lane the ego turns intoThat actor keeps driving along the lane toward the ego’s merge point, within the ego’s view ( ≥0.5\geq 0.5 s in view) and within 1 m of the ego (box to box)Crossing actor reaches the ego’s merge point while the ego is within 40 m of it, before or after it, with TTC ≤3\leq 3 s
Cut-inActor starts on adjacent lane of the egoActor changes lanes into the ego’s lane, in front of the ego, within the ego’s view ( ≥0.5\geq 0.5 s in view) and within 1 m of the ego (box to box)Actor completes the merge 0.5–10 m ahead of the ego and keeps driving forward at a nominal speed
Hard-brakeLead actor cruises ahead in the ego’s lane ( ≥8\geq 8 m/s); a second actor drives in the adjacent lane, 2–35 m from the leadOnce the ego is within 4.5 s headway (or 6 m), the lead brakes hard ( ≤−2.5\leq-2.5 m/s 2 ) while the adjacent actor keeps its speedEgo closes to within 4 m of the braked lead, or pulls out of its lane to go round it
Over-takeEgo follows a blocker in its lane on a two-way road; an oncoming actor approaches in the otherwise clear opposite lane, ≥12\geq 12 m beyond the blockerBlocker brakes to a stop, so the only way past is the opposite lane, against the oncoming actorOncoming actor passes within 4 m of the ego while the ego is held behind the blocker or pulls out to pass
TABLE III: Metric for measuring scenario realization for all scenarios. Each scenario is defined by a sequence of scenes, each being the precondition of the subsequent. If a prior scene is not satisfied, the following scene becomes unreachable and is labeled as failed. A set of trajectories generated from a given method for a given scenario is said to have successfully realized the scenario if all three scenes are satisfied.
Fig. 3: Visualization of our method in CARLA, on the cut-in scenario, satisfying all 3 scenes. Frame 1 shows the orchestrator assigning cut-in to actor 3. Frame 2 shows the orchestrator assigning cut-in to actor 2 as the ego’s speed made actor 3 infeasible to perform the intention. Frame 3 and 4 shows actor 2 performing the intention and terminating as required.
Fig. 4: Solvability across time-to-collision (TTC) thresholds ranging from 0.0s to 2.0s at regular 0.5s intervals. Scenario: left turn; policy: PlanT 2.0.
Fig. 5: Scenario realization rate of different methods given different actor spawning offset, same IDM ego policy. Each method is tuned to be successful on the 0m column, then the same set of parameters is applied to the other offsets.
More in this topic

Scenario-Based Testing

See all 13 papers

Explore similar work

CardsList
May 7, 2026

Traffic Scenario Orchestration from Language via Constraint Satisfaction

Autonomous vehicles (AVs) require extensive testing in simulation, but test case generation for driving scenarios is laborious. The desired scenarios are often out-of-distribution and have precise requirements on interactions with the AV policy under test. Manually programming scenarios allows for precise controllability but is difficult to scale. On the other hand, statistical models can leverage compute and data, but struggle with precise controllability when out-of-distribution. We cast scenario orchestration as a constraint-solving problem and present a language-in, simulation-out scenario orchestrator for closed-loop testing AVs. Our approach leverages foundation model reasoning to translate general, natural language descriptions into a set of constraints as a scenario representation. This then allows us to leverage off the shelf solvers to solve for actor behaviors which meet precise testing intentions in closed-loop. Under a benchmark of carefully crafted and diverse scenario descriptions, our approach greatly outperforms our baselines in orchestration success rate. We further show that our closed-loop approach is especially important for scenarios which require ego-reactive specifications.

Frieda Rong, Chris Zhang, Kelvin Wong +1

University of Toronto · 2Waabi

Autonomous Driving SimulationAutomated Test GenerationAutonomous Driving Scenario Generation
AskPDF
Jul 5, 2026

Agent-driven Long-tail Simulation for Autonomous Driving

Evaluating autonomous driving systems in closed-loop settings requires realistic and interactive simulation, yet existing simulators largely rely on log replay or rule-based agents, limiting behavioral diversity and long-tail coverage. We propose an agent-driven simulation framework in which surrounding road participants are controlled by instruction-following large language models through a structured action interface, enabling intentional and reactive behaviors while preserving physical plausibility. Furthermore, we introduce SemanticPlan, a benchmark of closed-loop planning in long-tail and semantically rich scenarios that augment real nuPlan scenes with multiple interactive agents following diverse language instructions. Evaluation results show that state-of-the-art planners still struggle to consistently achieve safe and effective task completion, suggesting that these long-tail scenarios remain challenging.

Junru Gu, Lijin Yang, Jianing Huang +3

1IIIS, Tsinghua University · 2Bosch Research

Autonomous Driving BenchmarksLanguage Model-Based ControlAutonomous Driving PlanningAutonomous Driving SimulationAutonomous Driving Scenario GenerationLong-Tail Autonomous Driving
AskPDF
May 15, 2026

PCASim: Promptable Closed-loop Adversarial Simulation for Urban Traffic Environment

Real-world autonomous driving, particularly in urban environments with numerous corner cases, requires rigorous testing to ensure product safety and robustness. However, few studies have explored integrating adversarial scenario generation with the training of safety agents in closed-loop testing, enabling efficient co-evolution and mutual enhancement of both. To address this challenge, an adversarial behavior knowledge repository is constructed by applying rule-based filtering to an open-source dataset, combined with knowledge retrieval modules tailored for simulation environments. A large language model (LLM) is employed to integrate knowledge-, data-, and adversarial-driven approaches, generating safety-critical traffic scenarios customized to user needs. Additionally, while evaluating the generated scenarios, we employ reinforcement learning models to train the behaviors of different types of vehicles, thereby enriching scenario diversity beyond existing datasets while preserving realism. Experimental results demonstrate that the proposed framework improves the accuracy of domain-specific language generation by 12%. Moreover, the success rate of newly generated scenario transformations increases by 8%, while obstacle-avoidance capability is enhanced by 30%. For the complete manuscript, please refer to: https://zhenhaooo.github.io/PCASim.github.io/

Chuancheng Zhang, Zhenhao Wang, Kaizheng Li +3

Shenzhen Research Institute of Shandong University, Shenzhen, 518057, Guangdong, China · School of Airspace Science and Engineering, Shandong University, 264209 Weihai, China. · School of Computer Science and Technology, Shandong University of Finance and Economics, 250014, Jinan, China.

Autonomous Driving Safety EvaluationSynthetic Data GenerationAdversarial Scenario GenerationAutonomous DrivingAutonomous Driving SimulationAutonomous Driving Scenario Generation
AskPDF