ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

Kevin Kai-Chun Chang; Ekin Beyazit; Alberto Sangiovanni-Vincentelli; Tichakorn Wongpiromsarn; Sanjit A. Seshia

cs.ROFeb 17, 2026

ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

Authors: Kevin Kai-Chun Chang, Ekin Beyazit, Alberto Sangiovanni-Vincentelli, Tichakorn Wongpiromsarn, Sanjit A. Seshia

Organizations: University of California, Berkeley · Iowa State University

Abstract

Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, obeying traffic rules, and making efficient progress. In many situations, these objectives cannot be satisfied simultaneously, and explicit priority relations naturally arise. Also, driving rules require context, so it is important to formally model the environment scenarios within which such rules apply. Existing benchmarks for evaluating autonomous vehicles lack such combinations of multi-objective prioritized rules and formal environment models. In this work, we introduce ScenicRules, a benchmark for evaluating autonomous driving systems in stochastic environments under prioritized multi-objective specifications. We first formalize a diverse set of objectives to serve as quantitative evaluation metrics. Next, we design a Hierarchical Rulebook framework that encodes multiple objectives and their priority relations in an interpretable and adaptable manner. We then construct a compact yet representative collection of scenarios spanning diverse driving contexts and near-accident situations, formally modeled in the Scenic language. Experimental results show that our formalized objectives and Hierarchical Rulebooks align well with human driving judgments and that our benchmark effectively exposes agent failures with respect to the prioritized objectives. Our benchmark can be accessed at https://github.com/BerkeleyLearnVerify/ScenicRules/.

Figures & tables

Panel of Figure 1 — Fig. 1 : Real-world autonomous driving examples with multiple conflicting objectives.

Fig. 2 : An example of a Rulebook.

Idx	Name	Description	Formula
1	No collisions with VRUs	The ego vehicle aims to avoid collisions with vulnerable road users (VRUs), such as pedestrians, cyclists, etc.	STL: $\operatorname{G}_{[T_{1},T_{2}]}(\forall j\in\mathcal{V}\cdot(p_{0}(t)\cap p_{j}(t)=\emptyset))$ VS: $\sum_{j\in\mathcal{J}}\mathcal{E}_{\text{ego\_loss}}+\mathcal{E}_{\text{vru}_{j}\text{\_gain}}$ , where $\mathcal{E}$ is kinetic energy.
2	No collisions with vehicles	The ego vehicle aims to avoid collisions with other vehicles.	STL: $\operatorname{G}_{[T_{1},T_{2}]}(\forall i\in\mathcal{A}\cdot(p_{0}(t)\cap p_{i}(t)=\emptyset))$ VS: $\sum_{i\in\mathcal{A}}\mathcal{E}_{\text{ego\_loss}}+\mathcal{E}_{\text{vehicle}_{i}\text{\_loss}}$ , where $\mathcal{E}$ is kinetic energy.
3	Staying within the drivable area	The ego vehicle aims to remain within the drivable area.	STL: $\operatorname{G}_{[T_{1},T_{2}]}(p_{0}(t)\subseteq\mathcal{R}_{driv}(t))$ VS: $\max_{t}(\\|p_{0}(t)\setminus\mathcal{R}_{driv}(t)\\|+\operatorname{d}(p_{0}(t),\mathcal{R}_{driv}(t))^{2})$
4	Staying on the correct side of road	The ego vehicle aims to remain on the correct side of road.	STL: $\operatorname{G}_{[T_{1},T_{2}]}(p_{0}(t)\cap(\mathcal{R}_{incorrect}(t)\setminus\mathcal{R}_{correct}(t))=\emptyset)$ VS: $\sum_{t}(\\|p_{0}(t)\cap(\mathcal{R}_{incorrect}(t)\setminus\mathcal{R}_{correct}(t))\\|)$
5	VRU time to collision	The ego aims to maintain a time to collision (TTC) above a threshold for all VRUs, assuming the ego maintains its current velocity while VRUs stop suddenly.	STL: $\operatorname{G}_{[T_{1},T_{2}]}(\forall j\in\mathcal{V}\cdot(TTC_{j}(t)\geq T_{VRU,ttc}))$ VS: $\max_{t,j}(T_{VRU,ttc}-TTC_{j}(t))$
6	VRU acknowledgment	If the ego is expected to come within a minimum safe distance of a VRU and its current velocity towards the VRU is higher than a threshold, it aims to decelerate sufficiently.	STL: $\operatorname{G}_{[T_{1},T_{2}]}(\forall\tau\in[t,t+T_{ack}]\cdot((\exists j\in\mathcal{V}\cdot((v_{0,j}(t)>V_{ack})$ $\land(\operatorname{d}(p_{0}(\tau),p_{j}(\tau))<D_{ack}))\implies a_{0,j}(t)<A_{ack}))$ VS: $\max_{t_{k},j}(a_{0,j}(t)-A_{ack})$

TABLE I : Collected objectives and their formalizations. The notations used in the formulas are defined below the table.

Fig. 4 : Categorization of rules.

Fig. 5 : Our automated Scenic program generator.

Attribute	Supported Options
agent type	car, pedestrian
vehicle maneuver	going straight, left turn, right turn, lane change, lane following
spatial relation to ego	ahead of, behind, faster lane, slower lane, opposing lanes, conflicting lanes
pedestrian maneuver	crossing street, walking along sidewalk

TABLE II : Supported attributes in the Scenic generator.

Model	Accuracy (%)
Ours	$80.9\pm 1.8$
RB + DT	$78.4\pm 2.8$
RF	$82.3\pm 2.4$
LR	$79.1\pm 2.5$
w/ parameter	$81.8\pm 2.0$

TABLE III : Evaluation results against human preferences.

Restricted	# Iterations	Accuracy (%)
Yes	$2$	$85.08$
No	$2$	$85.97$
Brute Force Search		$85.97$

TABLE IV : Scenario-specific priority optimization results.

Formalization	Accuracy (%)	# Different Preferences	# Reason Changes
Clearance – Sum	$80.92$	$6$	$1$
Clearance – Heading	$81.63$	$23$	$251$
Correct Side – Centroid	$78.95$	$69$	$171$
Original	$80.97$	–	–

TABLE V : Evaluation results under alternative rule formalizations.

Config.	Space Size	Dim.	Max. Dist.	Ego Man.	Adv. Man.	Adv. Spat.	Spat.- Man.
2 adversaries no pedestrian	$1616$	$5$	$2$	$100\%$	$100\%$	$100\%$	$100\%$
2 adversaries 1 pedestrian	$3232$	$7$	$2$	$100\%$	$100\%$	$100\%$	$100\%$

TABLE VI : Scenario coverage of common scenarios.

Fig. 6 : The falsification flow.

Scenario	Policy	Avg. EV	Max EV	CE Ratio	Violated Rules (%)	#Unique Viols.
Common no VRU	rule-based	$0.127$	$0.979$	$76.6\%$	$92.9\%$	$168$
Common no VRU	ppo	$0.134$	$0.995$	$96.9\%$	$100.0\%$	$205$
Common with VRU	rule-based	$0.124$	$0.979$	$86.4\%$	$94.7\%$	$291$
Common with VRU	ppo	$0.161$	$0.933$	$97.7\%$	$100.0\%$	$350$
Near-accident no VRU	rule-based	$0.470$	$0.999$	$99.3\%$	$92.9\%$	$144$
Near-accident no VRU	ppo	$0.252$	$0.979$	$96.5\%$	$100.0\%$	$100$

TABLE VII : Falsification results on our benchmark.

Panel of Figure 14 — Fig. 7 : Different failure modes identified under the same driving context (Scenic program).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 8 : Example from the Reasonable Crowd dataset [ 3 ] . Each scenario contains multiple feasible trajectories, and human annotators provide pairwise preference labels.

Fig. 9 : Evaluation flow for comparing hierarchical Rulebook preferences with human preferences.

Explore similar work

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving
Sep 29, 2026Victoria Smirnova, Viktoriia Zinkovich, Gregorii Bukhtuev +4Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation
Lomonosov Moscow State University · FusionBrain Lab · Moscow Institute of Physics and Technology +3
HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving
May 11, 2026Zhongyu Xia, Guanyu Zhu, Guo Tang +2Autonomous Driving Benchmarks End-to-End Autonomous Driving
Wangxuan Institute of Computer Technology, Peking University
DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving
Mar 2, 2026Enhui Ma, Jiahuan Zhang, Guantian Zheng +10Autonomous Driving Benchmarks VLMs for Autonomous Driving
Autolab, Westlake University · Li Auto Inc · Tsinghua University +1

Sep 29, 2026J/K move · Enter open · S save

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving

Victoria Smirnova, Viktoriia Zinkovich, Gregorii Bukhtuev, Artem Belyaev+3

Lomonosov Moscow State University · FusionBrain Lab · Moscow Institute of Physics and Technology+3

Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation Autonomous Driving Autonomous Driving Simulation Autonomous Driving Scenario Generation

Open paper AskPDF arXiv

Autonomous driving planners are typically evaluated using aggregate metrics such as driving score, destination rate, and collision rate, which do not explicitly measure compliance with traffic rules. As a result, planners can achieve high benchmark scores while still exhibiting unsafe or illegal behaviors, limiting their applicability to real-world deployment. To address this gap, we introduce TrafficSignBench, a large-scale, traffic sign-centric benchmark for systematic and interpretable evaluation of traffic-rule compliance in autonomous driving. Our framework combines real-map-based simulation for realistic road layouts with rule-targeted procedural scenario generation for scalable and balanced coverage of underrepresented rules. We implement traffic rules corresponding to 34 traffic signs, each equipped with an automatic rule checker for detecting violations during closed-loop execution. This design yields 29,000 diverse road scenes and 29 distinct testing scenario types, enabling controlled evaluation of rule-specific planner behavior. We construct 5,800 testing scenes and demonstrate that current autonomous driving planners can exhibit poor traffic-rule compliance despite strong performance on standard evaluation metrics. To address this limitation, we transform existing planners into rule-compliant trajectory experts via explicit traffic-sign constraints, enabling scalable generation of high-quality oracle trajectories for fine-tuning.

Abstract

Explore similar work

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving