Approval of ADS depends on evaluating its behavior within representative real-world traffic scenarios. A common way to obtain such scenarios is to extract them from real-world data recordings. These can then be grouped and serve as basis on which the ADS is subsequently tested. This poses two central challenges: how scenarios are extracted and how they are grouped. Existing extraction methods rely on heterogeneous definitions, hindering scenario comparability. For the grouping of scenarios, rule-based or ML-based methods can be utilized. However, while modern ML-based approaches can handle the complexity of traffic scenarios, unlike rule-based approaches, they lack interpretability and may not align with domain-knowledge. This work contributes to a standardized scenario extraction based on the Scenario-as-Specification concept, as well as a domain-knowledge-guided scenario clustering process. Experiments on the highD dataset demonstrate that scenarios can be extracted reliably and that domain-knowledge can be effectively integrated into the clustering process. As a result, the proposed methodology supports a more standardized process for deriving scenario categories from highway data recordings and thus enables a more efficient validation process of automated vehicles.
Figures & tables
Fig. 1 : Overview of the proposed method for knowledge guided scenario extraction and clustering.
Fig. 2 : In the Scenario Preprocessing stage, ego–behavior changes are detected and used to extract scenarios from the traffic dataset. In addition, the interaction matrix T(m) and the pseudo-class label vector s(m) are computed for each scenario. For clustering, a Clustering Vector Quantized - Variational Autoencoder with a predefined number of codebook entries Q is employed. The model receives only the scenario trajectories ξ(m) as input, produces a discretized representation zq(m) , and predicts both the interaction matrix and the behavior class from this latent representation.
Method
Precision ↑
Recall ↑
TP
FP
FN
Rule-based
0.741
0.916
109
38
10
EMA [ 29 ]
0.196
0.244
29
119
90
CVQ-VAE
0.302
0.723
86
199
33
TABLE I: Evaluation of behavior–change detection on the manually annotated subset. Best values are in bold .
Fig. 3 : Exemplary augmented scenario where the additional vehicle (green) has no influence on the ego vehicle (red).
Backend
no DK
DK
purity ↓
accuracy ↑
purity ↓
accuracy ↑
CVQ-VAE
3.014
0.068
1.243
0.568
k-means
3.014
0.091
1.127
0.182
Hierarchical
3.010
0.068
1.153
0.250
TABLE II: Clustering on the train split: cluster-purity ( ↓ ) and augmentation cluster accuracy ( ↑ ). DK = Domain-Knowledge. Best values per row are in bold .
Extracting scenarios from unlabelled real-world sensor data streams is a critical but challenging task in the development process of automated driving systems (ADS). Automatically sifting through large datasets to spatially and temporally locate critical scenarios can enable scenario-based coverage analysis of ADS datasets. In this paper, we present a method for extracting scenarios from egocentric datasets using scene graphs and Linear Temporal Logic (LTL). We first process egocentric sensor data and HD maps to generate a sequence of scene graphs representing a driving scenario. Next, we use LTL to formally specify driving scenarios of interest, then extract all instances of the scenarios from the dataset using an off-the-shelf model checker, which evaluates the LTL formula against the sequence of scene graphs. Our approach can be used on both simulated and real world datasets. We evaluate the method on the training and validation datasets from Argoverse 2 consisting of 850 15-second real-world driving logs, and several videos of dashcam footage. We demonstrate the effectiveness of our approach for extracting and querying scenarios by evaluating against a rule-based benchmark based on track annotations and HD maps.
Stefan Ramdhan, Kyanna Dagenais, Vera Pantelic +2
McMaster Centre for Software Certification (McSCert) · McMaster University · Hamilton, Ontario, Canada
Large-scale autonomous-driving datasets contain vast numbers of recorded scenarios, creating a need for efficient retrieval methods that can identify situations similar to a given query. Existing approaches typically rely on either visual representations or motion-based descriptions, making it difficult to understand their relative strengths and limitations for scenario retrieval. In this work, we present a multimodal framework for autonomous-driving scenario retrieval that combines visual and trajectory-based representations within a unified retrieval pipeline. We investigate two trajectory-based approaches: Exo-Trajectory, an explicit matching method based on surrounding-agent motion, and ScenarioFormer, a transformer-based representation learned from object trajectories using contrastive learning. We compare these approaches against strong vision-based baselines and analyze their behavior across a diverse set of driving scenarios. Experimental results show that trajectory representations provide strong retrieval performance for motion-centric events such as cut-ins, turning maneuvers, and traffic queueing, while visual embeddings excel when appearance cues are informative. Most importantly, combining visual and trajectory information consistently improves retrieval quality, yielding the best overall performance. These findings demonstrate that appearance and motion capture are complementary notions of scenario similarity and motivate multimodal retrieval systems for autonomous-driving data mining, dataset curation, and scenario-based validation.
Autonomous vehicles (AVs) face driving scenarios ranging from routine traffic to rare events. To assess safety it is crucial to reproduce these scenarios in a controllable, repeatable, and scalable manner, with simulation playing a key role. This paper introduces D-V2S, a novel framework that automatically generates simulatable driving scenarios from driving videos. D-V2S operates in two stages: a Driving Record Analyzer (DRA) uses a vision language model (VLM) with our designed prompt to produce natural-language descriptions from input videos, capturing road layouts and dynamic traffic interactions; subsequently, a Scenario Generator (SG) uses a large language model (LLM) and our conditioning context to translate these descriptions into executable scenarios. Using simulations, we show that D-V2S generates scenarios where 90% of the relevant semantic elements of the videos are present. We also provide qualitative results demonstrating D-V2S's capability to transform real-world driving videos into simulatable scenarios. Moreover, we provide both semantic and human driven ablative analyses of D-V2S's modules. In particular, we show how the VLM choice matters for DRA, and how our SG achieves a 75% preference rate over other state-of-the-art methods.
Alexandre Levy, Ernest Valveny Llobet, Antonio Manuel López
Dept. Computer Science, Universitat Aut`onoma de Barcelona (UAB) · Computer Vision Center (CVC), UAB