SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions

Yicheng Zhu; Tianmu Zhao; Haoxin Leng; Fan Zuo; Tao Li; Zilin Bian

cs.ROAug 25, 2026

SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions

Authors: Yicheng Zhu, Tianmu Zhao, Haoxin Leng, Fan Zuo, Tao Li, Zilin Bian

Organizations: Rochester Institute of Technology · CVS Health · City University of Hong Kong · New York University

Abstract

Emergency vehicles (EMVs) can reorganize surrounding traffic as civilian vehicles brake, change lanes, or form rescue corridors in response to their passage. Evaluating these safety-critical interactions requires behavior-level control over both EMV privileges and civilian responses, together with consistent sensing and ground truth. Existing datasets and simulation benchmarks do not directly provide this combination. We present SIREN, a behavior-driven SUMO--CARLA co-simulation platform for generating EMV--civilian interactions. SIREN couples SUMO's network-level traffic evolution and behavior logic with CARLA's continuous vehicle control and synchronized onboard sensing; depending on the active behavior, the interaction is controlled by SUMO, CARLA, or jointly. We instantiate the platform as SIREN-Bench-v1, comprising seven parameterized interaction templates across emergency levels L1--L3 and three behavior families, with synchronized sensor observations and simulator-native annotations. We demonstrate the benchmark through three representative tasks: 3D object detection, trajectory prediction, and vision-language risk understanding. Evaluations of nine trajectory predictors, four LiDAR-based detectors, and five vision-language models reveal behavior-dependent failure modes. Traffic-clearance interactions are hardest for detection, privileged intersection traversal is hardest for prediction, and no learned predictor outperforms the constant-velocity reference on average. Vision-language models perform substantially better on normal traffic than on near-miss and collision events. These results demonstrate the value of behavior-centered benchmarking and establish SIREN as an extensible data-generation and evaluation platform for autonomous-driving and transportation safety research.

Figures & tables

Figure 1: Overview of SIREN and SIREN-Bench. (1) Existing scenario-based benchmarks typically evaluate a fixed set of predefined scenarios with prescribed actor motion and timing. SIREN instead specifies EMV and civilian behavior policies and generates scenario realizations through closed-loop interaction. (2) Representative EMV–civilian interactions generated by the SIREN SUMO–CARLA co-simulation. Reading in row order, the panels illustrate the rescue-corridor pull-over and lane-clearing responses of the clearance family (S1–S2), privileged traversal of controlled intersections (S3–S6), and use of nonstandard road space (S7); template identifiers are defined in Table 2 . (3) SIREN-Bench evaluates three complementary AI capabilities: 3D object detection, trajectory prediction under EMV-induced interaction, and scene-level risk understanding from onboard visual observations.

Figure 2: Overview of the SIREN behavior-driven generation pipeline. Rather than prescribing complete scenarios, SIREN configures EMV and civilian behaviors whose closed-loop interaction is resolved through SUMO–CARLA co-simulation. Simulator-native states and sensor observations are then used to construct SIREN-Bench for downstream evaluation. The emergency-level axis (L0–L3) is defined in Table 1 ; SIREN-Bench-v1 releases the L1–L3 settings as seven templates, with a risk-understanding set of 105 videos (15 per template and 190 frames per video; 19,950 frames in total).

Level	Siren	EMV route mode	Civilian response	Tem- plates
L0	off	Nominal; no privilege	Nominal traffic control	—
L1	off	Next-link; intent override	Nominal traffic control	S3–S6
L2	on	As L1	Feasible yield lane change	S2
L3	on	Next-link/intent; copied SUMO route; gap escape	Pull over, then braking hold	S1, S7

Table 1: Level-Dependent Control in SIREN

Figure 3: Illustrations of SIREN. (1) SUMO view of the synchronized traffic state. (2) CARLA rendering of the corresponding scene. (3) Supported EMV assets. (4) EMV passes between queued vehicles (template S1). (5) Civilian lane clearing for EMV passage (S2). (6) Privileged EMV traversal at a controlled intersection (S3–S6). (7) EMV use of nonstandard road space while turning (S7). Trajectory color indicates EMV speed.

ID	Behavior family	Instantiated interaction	$N_{\mathrm{veh}}$
S1	Clearance	EMV passes between two queues of vehicles.	30
S2	Clearance	Vehicles merge out of the EMV lane to create a clear passage.	20
S3	Traffic control	EMV traverses a red signal with same-direction traffic in the intersection.	20
S4	Traffic control	EMV traverses a red signal with crossing traffic in the intersection.	20
S5	Traffic control	EMV traverses a stop sign with same-direction traffic in the intersection.	20
S6	Traffic control	EMV traverses a stop sign with crossing traffic in the intersection.	30

Table 2: Seven Parameterized Interaction Templates in SIREN-Bench-v1

	Clearance		Traffic-control traversal				Road-space
Model	S1	S2	S3	S4	S5	S6	S7	Mean
CV	1.63/4.30	1.17/3.11	0.96/2.40	0.68/1.65	1.95/5.03	2.86/7.56	0.85/2.08	1.44/3.73
IDM	2.23/5.14	1.58/3.75	1.18/2.69	1.04/2.18	2.68/6.49	3.29/8.34	1.06/2.55	1.86/4.45
CSP	2.63/6.12	3.57/7.79	2.92/6.65	2.54/5.83	3.25/7.10	4.63/9.69	1.82/4.02	3.05/6.74
STDAN	3.99/8.12	6.02/12.13	4.20/9.14	3.11/6.81	4.58/9.94	7.87/15.41	2.38/5.93	4.59/9.64
BAT	9.14/17.35	9.55/16.14	8.17/15.75	6.06/10.52	4.80/7.85	8.40/13.16	3.42/5.06	7.08/12.26

Table 3: Trajectory Prediction on SIREN-Bench

	Clearance		Traffic-control traversal				Road-space
Model	S1	S2	S3	S4	S5	S6	S7	Mean
PointPillars	.398/.407	.434/.436	.583/.525	.589/.549	.451/.456	.530/.540	.569/.528	.508/.492
SECOND	.420/.420	.388/.438	.631/.563	.633/.574	.461/.480	.523/.540	.590/.556	.521/.510
VoxelNeXt	.438/.393	.413/.434	.641/.571	.616/.558	.468/.454	.568/.540	.578/.543	.532/.499
TransFusion-L	.424/.411	.423/.456	.648/.566	.618/.565	.491/.481	.562/.555	.610/.571	.539/.515

Table 4: 3D Object Detection on SIREN-Bench

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Sensor	$x$	$y$	$z$	yaw	Key attributes
LiDAR (64-beam)	0.0	0.0	1.93	0	120 m, 1.3 Mpps, 20 Hz
Cam. front	2.5	0.0	1.0	0	$800{\times}600$ , $100^{\circ}$
Cam. right-rear	0.0	0.3	1.8	100	$800{\times}600$ , $100^{\circ}$
Cam. left-rear	0.0	$-0.3$	1.8	$-100$	$800{\times}600$ , $100^{\circ}$
Cam. rear	$-2.0$	0.0	1.5	180	$800{\times}600$ , $100^{\circ}$
GNSS	0.0	0.0	1.93	0	co-located w/ LiDAR

Appendix

Table 7: EMV-mounted sensor suite and placement (CARLA vehicle frame:

x

forward,

y

right,

z

up, in meters; yaw in degrees). All cameras are

800\times 600

100^{\circ}

horizontal FOV.

Constant	Meaning	Value
Co-simulation and control
Interaction region (in / out)	EMV-centered hysteresis radius	40 / 50 m
Leader query range	SUMO leader look-ahead (gap_escape)	80 m
Lane-change cooldown	min. interval between forced-LC attempts	2.0 s
EMV-passed clearance	buffer for EmvFullyPassed	1.0 m
Heading-aligned tolerance	max. error for HeadingAligned	$8^{\circ}$

Appendix

Table 8: Control constants used by the co-simulation (from the controller implementation).

Agent	Control mode	Target (km/h)
EMV	Next-link route-following	50
EMV	Intent-override lane change	50
EMV	copy_sumo corridor	30
EMV	gap_escape recovery	20
Civilian	Nominal (Traffic Manager)	TM-governed
Civilian	L2 feasible-yield lane change	10

Appendix

Table 9: Per-mode target speeds enforced by the shared waypoint navigator.

Model	Task	Training data or checkpoint	Reference
CV	Traj.	— (analytical)	—
IDM	Traj.	— (car-following)	—
CSP	Traj.	NGSIM	[ 23 ]
STDAN	Traj.	NGSIM	[ 24 ]
BAT	Traj.	NGSIM	[ 25 ]
EMP-M / EMP-D	Traj.	Argoverse 2	[ 26 ]

Appendix

Table 10: Provenance of evaluated models. All learned models run zero-shot from released pretrained checkpoints (no fine-tuning on SIREN-Bench).

Autonomous Driving Benchmarks

Autonomous driving benchmarks evaluate how well vehicles perceive, predict, plan, and act safely across varied roads and conditions.

See all 140 papers

May 17, 2026

Generating Realistic Safety-Critical Scenarios for Vehicle-Pedestrian Interactions

Automated driving system deployment requires rigorous validation across safety-critical vehicle-pedestrian interactions, yet real-world datasets rarely capture high-risk scenarios while simulation platforms lack realistic behavior. In response, this study proposes a three-stage framework that combines real-world grounding with adaptive simulation to generate behaviorally realistic safety-critical scenarios at scale. Stage 1 pre-trains multi-agent state-space Transformer-enhanced DDPG (MA-SST-DDPG) agents on real-world safety-critical data to learn human-like interactive evasive behaviors through data-driven learning. Stage 2 deploys pre-trained multi-agents in CARLA for online reinforcement learning to generalize across diverse scenarios, integrating real-world knowledge with simulation experience to produce a refined MA-SST-DDPG model. Stage 3 uses CARLA with the refined model to generate over 198,000 high-resolution interaction episodes from eight intersection scenarios, culminating in the Vehicle-Pedestrian Safety-Critical Interaction (VPSCI) dataset. The Refined MA-SST-DDPG model outperformed baseline methods in reproducing realistic evasive behaviors, achieving the lowest trajectory errors (ADE = 0.072 m, FDE = 0.142 m). Statistical comparison confirmed distributional equivalence between the generated and real-world data in both conflict severity and behavioral response. A Turing test confirmed that the three-stage framework generated evasive behaviors were indistinguishable from real-world interactions. These results demonstrate the framework's effectiveness in producing high-fidelity safety-critical data, offering valuable sources for the development of ADS and simulation-based safety evaluations.

Qingwen Pu, Kun Xie, Yuan Zhu +1

Transportation Informatics Lab, Department of Civil and Environmental Engineering, Old Dominion University, Norfolk, VA 23529, United States · Inner Mongolia Center for Transportation Research, Inner Mongolia University, Rm A357A, Transportation Building, Inner Mongolia University South Campus, 49 S Xilin Rd, Hohhot, Inner Mongolia, 010020, China · School of Transportation and Logistics, National Engineering Laboratory of Integrated Transportation Big Data Application Technology, National and Local Joint Engineering Research Center of Integrated Transportation Intelligence, Southwest Jiaotong University, Chengdu 611756, China

Human Behavior Simulation Autonomous Driving Safety Evaluation Synthetic Data Generation for Robotics RL for Autonomous Driving Autonomous Driving Simulation Autonomous Driving Scenario Generation

Ask PDF

Jun 12, 2026

ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving

Reactive capability is a key property of data-driven behavior world model simulators for autonomous driving simulation systems. With this capability, simulated world agents can respond feasibly to autonomous vehicle (AV) behaviors that differ from the log. However, existing behavior simulation benchmarks do not directly measure reactive capability. They often let the simulator jointly control the AV and surrounding agents and evaluate realism through log similarity or open-loop prediction metrics. In this work, we introduce ReactSim-Bench for evaluating the reactive capability of behavior world model simulation in autonomous driving. We decouple the control of agents and the AV, using AV behaviors that differ from the log and require agents to respond as independent AV inputs. To obtain these AV behaviors, we construct a pipeline that uses an AV planner model to generate candidate behaviors and filters the data using rules and manual verification. Collision metrics, map-based metrics, and kinematic feasibility metrics are used to evaluate the safety and rule compliance of reactive responses. We construct 2,636 test scenarios with three categories and conduct a systematic evaluation of state-of-the-art models across multiple architectures, including Transformer-based, diffusion-based, and next-token-prediction-based models. We further analyze how replan frequency affects performance and provide insights for future studies.

Zhiyuan Zhang, Yanlun Peng, Jianing Zhang +7

Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University · Great Wall Motor · Institute of Trustworthy Embodied AI (TEAI), Fudan University +2

Autonomous Driving Benchmarks World Models for Autonomous Driving World Model Evaluation Autonomous Driving Simulation

Ask PDF

Sep 28, 2026

VehicleArena: A Realistic Urban Environment for Multi-Agent Driving

Real-world embodied agents often pursue independent objectives within a shared physical environment, where their actions can alter the conditions faced by others. Existing benchmarks, however, typically assume shared goals or explicitly prescribed interaction protocols, leaving such emergent physical coupling underexplored. We introduce VehicleArena, a 3D urban-driving benchmark for studying independently operating agents in a dynamic shared world. In VehicleArena, LLM-controlled agents must fulfill evolving passenger requests while navigating complex traffic, and each agent's driving decisions can reshape traffic flow, delays, risks, and subsequent observations for surrounding agents. The benchmark provides 112 evaluation tasks spanning single-agent and multi-agent driving. Across nine evaluated models, the highest arrival rates reach only 65.0% on single-agent tasks and 65.6% on multi-agent tasks, while strong passenger-request or cabin scores do not reliably translate into successful trip completion. Moreover, in matched multi-agent runs, every tested focal policy reduces the arrival rate of surrounding vehicles relative to the simulator's native traffic controller, revealing measurable externalities beyond the focal vehicle itself.

Jie Yang, Jiajun Chen, Jiazheng Zhou +4

Fudan University · Shanghai Innovation Institute

Autonomous Driving Benchmarks Autonomous Driving AI Agent Benchmarks LLM Agents Autonomous Driving Simulation Emergent Behavior in Multi-Agent Systems

Ask PDF

Abstract

Autonomous Driving Benchmarks

Explore similar work

Generating Realistic Safety-Critical Scenarios for Vehicle-Pedestrian Interactions

ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving

VehicleArena: A Realistic Urban Environment for Multi-Agent Driving