ScenarioCharacterization: A Modular Toolkit for Characterizing Safety across Trajectory Datasets

Ingrid Navarro; Yutong Duan; Jonathan Francis; Jean Oh

cs.ROAug 17, 2026

ScenarioCharacterization: A Modular Toolkit for Characterizing Safety across Trajectory Datasets

Authors: Ingrid Navarro, Yutong Duan, Jonathan Francis, Jean Oh

Organizations: Robotics Institute, School of Computer Science, Carnegie Mellon University · Work done as part of an internship at Stack AV · Stack AV · Bosch Center for Artificial Intelligence

Abstract

We introduce ScenarioCharacterization, an open-source framework for automated, dataset-agnostic profiling of driving scenarios in trajectory datasets. Our framework is packaged as a modular, configuration-driven pipeline of three layers: a dataset adapter that maps custom datasets onto an open Scenario representation, a characterizer that performs feature extraction, behavior probing, and criticality scoring at scenario and agent levels, and an analysis layer for scenario visualization and feature, score, and probe analyses. Because the layers communicate only through Pydantic-validated schemas composed via configurations, a new dataset can easily plug in without rewriting the characterization and analysis stack. This technical report describes the design and APIs, shows example outputs on Waymo Open Motion, Argoverse2, and nuPlan, and discusses downstream uses of the approach. The framework is available at https://github.com/navarrs/ScenarioCharacterization.

Figures & tables

Component Base Class	Entry point	Output
BaseDataset ( IV )	transform(Raw Scenario)	Scenario
BaseFeature ( V-A )	compute( Scenario )	ScenarioFeatures
BaseScorer ( V-B )	compute( Scenario , ScenarioFeatures )	ScenarioScores
BehaviorProber ( V-C )	probe( Scenario )	ScenarioProbe
BaseVisualizer ( VI-B )	visualize( Scenario )	Rendered Scenario

TABLE I: Framework API Overview. Each component is extended by subclassing its base class and implementing the corresponding entry point.

Individual Features	Interaction Features
Speed, Acceleration, Jerk [ 4 ]	Time-to-Collision (TTC) [ 4 , 13 ]
Speed Limit Diff [ 4 ]	Time Headway (THW) [ 4 , 13 ]
Waiting Period at Conflict Point [ 4 , 13 ]	Deceleration Rate to Avoid Crash (DRAC) [ 4 , 13 ]
Trajectory Type [ 12 ]	Minimum Time to Conflict Point (mTTCP) [ 4 , 13 ]
Kalman Difficulty [ 12 , 22 ]	Collisions [ 4 , 13 ]

TABLE II: List of low-level per-axis features.

Fig. 1: Supported visualizations, shown on two WOMD [ 1 ] scenarios. Top : the ego waits at a traffic light while relevant agents approach and stop behind it. Bottom : the ego approaches an intersection as a cyclist initiates a left turn across its path.

	Trajectories			Ego interactions
Dataset	Vehicles	Pedestrians	Cyclists	Ego-Vehicle	Ego-Pedestrian	Ego-Cyclist
WOMD [ 1 ]	49.2 (290k)	3.4 (37k)	0.0 (2.6k)	31.5 (215k)	0.8 (18k)	0.0 (2.1k)
Argoverse2 [ 2 ]	37.8 (202k)	2.8 (25k)	0.5 (6.0k)	25.8 (144k)	1.1 (17k)	0.2 (4.5k)
nuPlan [ 3 ]	32.6 (174k)	21.0 (194k)	0.3 (3.2k)	20.6 (127k)	3.5 (112k)	0.2 (2.5k)

TABLE III: Individual trajectories and interactions per dataset sample, reported as the per-scenario interquartile mean with the sample total in parentheses. Interaction columns only count ego pairs ; global interaction counts are available in the code.

Fig. 2: Feature distribution analysis examples. Top : within-dataset comparison on WOMD , showing the speed-limit difference distributions for vehicles (left), pedestrians (center) and cyclists (right). The red dashed lines show the 25th, 75th and 90th percentile values. Bottom : cross-dataset comparison across WOMD , nuPlan , and Argoverse2 , showing DRAC distributions for ego-vehicle (left), ego-pedestrian (center) and ego-cyclist (right) pairs. Both rows use a logarithmic $y$ -axis.

Fig. 3: Score distribution analysis examples. Top : shows the score distribution across supported aggregators on WOMD . Bottom : shows the SafeShift distribution across WOMD , nuPlan and Argoverse2 .

Fig. 4: Probe outcome distribution across datasets. For each of the $5{,}000$ scenarios per dataset, the constant-velocity sweep either finds no critical probe or retains one that perturbs the ego-agent or a non-ego agent; bars give the share of each outcome, annotated with raw counts.

Fig. 5: Per-scenario interaction score before vs. after probing, over the scenarios in which a probe was retained. Left three : per-dataset scatters for WOMD , nuPlan , and Argoverse2 . Rightmost : all three overlaid. The dashed line marks $y=x$ ; vertical displacement from it is the probe-induced score delta.

Fig. 6: Mining scenarios by probe-induced criticality delta. Left : per-scenario scores before vs. after probing, color-coded by whether the probed agent is the ego or not. Points above the identity line are amplified by the probe. Right : the interaction marked with a red $\times$ in the scatter, shown before (left) and after (right) probing, where the constant-velocity probe raises the pair score by $10.5$ .

Autonomous Driving Safety Evaluation

Autonomous driving safety evaluation studies how to measure, test, and improve the safety of self-driving systems in routine and hazardous conditions.

See all 158 papers

Scene Graph-based Driving Scenario Extraction for Automotive Egocentric Datasets
Aug 31, 2026Stefan Ramdhan, Kyanna Dagenais, Vera Pantelic +2Autonomous Driving Datasets Autonomous Driving
McMaster Centre for Software Certification (McSCert) · McMaster University · Hamilton, Ontario, Canada
PersonaDrive: Controllable Trajectory Prediction with Multi-Dimensional Driving Personas
Aug 15, 2026Chan Lee, Kimin Yun, Yuseok Bae +2Multimodal Trajectory Prediction Autonomous Driving Datasets
Kyung Hee University, Yong-in, South Korea · ETRI, Daejeon, South Korea
Multimodal Scenario Similarity Search for Autonomous Driving
Jul 10, 2026Tamás Matuszka, András Tamásy, Balázs SzolárAutonomous Driving Datasets Multimodal IR
aiMotive

Aug 31, 2026J/K move · Enter open · S save

Scene Graph-based Driving Scenario Extraction for Automotive Egocentric Datasets

Stefan Ramdhan, Kyanna Dagenais, Vera Pantelic, Victor Bandur+1

McMaster Centre for Software Certification (McSCert) · McMaster University · Hamilton, Ontario, Canada

Autonomous Driving Datasets Autonomous Driving

Open paper AskPDF arXiv

Extracting scenarios from unlabelled real-world sensor data streams is a critical but challenging task in the development process of automated driving systems (ADS). Automatically sifting through large datasets to spatially and temporally locate critical scenarios can enable scenario-based coverage analysis of ADS datasets. In this paper, we present a method for extracting scenarios from egocentric datasets using scene graphs and Linear Temporal Logic (LTL). We first process egocentric sensor data and HD maps to generate a sequence of scene graphs representing a driving scenario. Next, we use LTL to formally specify driving scenarios of interest, then extract all instances of the scenarios from the dataset using an off-the-shelf model checker, which evaluates the LTL formula against the sequence of scene graphs. Our approach can be used on both simulated and real world datasets. We evaluate the method on the training and validation datasets from Argoverse 2 consisting of 850 15-second real-world driving logs, and several videos of dashcam footage. We demonstrate the effectiveness of our approach for extracting and querying scenarios by evaluating against a rule-based benchmark based on track annotations and HD maps.

Abstract

Autonomous Driving Safety Evaluation

Explore similar work

Scene Graph-based Driving Scenario Extraction for Automotive Egocentric Datasets

PersonaDrive: Controllable Trajectory Prediction with Multi-Dimensional Driving Personas

Multimodal Scenario Similarity Search for Autonomous Driving