Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152

Vahid Zolfaghari; Nenad Petrovic; AndrÉ Schamschurko; Alois Knoll

cs.AIAug 17, 2026

Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152

Authors: Vahid Zolfaghari, Nenad Petrovic, AndrÉ Schamschurko, Alois Knoll

Organizations: Chair of Robotics, Artificial Intelligence (AI) and Embedded Systems, Technical University of Munich, Munich, Germany

Abstract

Generating regulation-compliant test scenarios is essential for validating safety-critical automotive systems, yet Large Language Models (LLMs) struggle to ground outputs in long, hierarchical standards. We present RegulaRAG, a Retrieval-Augmented Generation (RAG) pipeline that couples SmartChunking, reference-aware enrichment of paragraphs and tables via graph traversal, with Smart Retrieve & Rerank over these enriched units. To test our system, we evaluate on a manually curated dataset covering all scenarios in UN Regulation No. 152 (AEBS). Our study comprises: (i) a three-step progressive search that identifies near-optimal retrieval parameters without exhaustive grid search; (ii) head-to-head comparisons against five baseline RAG systems; and (iii) a robustness stress test that scales the source corpus with distractor content. Outputs are evaluated using a customized penalized scoring metric. Across all experiments, RegulaRAG achieves the highest average Meta-Score (82.99), outperforming the next-best system by 43% (NoRAG: 57.94), while operating at 14k-25k tokens per query versus up to 500k for graphcentric baselines. It maintains strong performance, remaining stable even as the number of regulatory sources grows, whereas competing RAG systems degrade sharply in both quality and robustness.

Figures & tables

Fig. 1: Example of a reference and table chain in UN Regulation No. 152 .

Category	Condition	Num Scenarios
CtoStC	unladen	12
CtoStC	laden	12
CtoMoC	unladen	8
CtoMoC	laden	7
CtoP	MassRun	10
CtoP	MaxMass	10

TABLE I: Summary of Test Scenarios by Category and Condition

RAG System	Model	CtoStC ( $\uparrow$ )	CtoMoC ( $\uparrow$ )	CtoP ( $\uparrow$ )	Mean ( $\uparrow$ )	stdev ( $\downarrow$ )	Meta-Score ( $\uparrow$ )	Avg. 3 LLMs ( $\uparrow$ )
RegulaRAG	GPT-4o	76.92	96.55	100.00	91.16	10.16	80.99	82.99
	DeepSeek-chat	85.71	96.55	97.44	93.23	5.33	87.90
	Llama-3.3	76.92	96.55	91.89	88.45	8.37	80.08
R&R+RCS	GPT-4o	8.00	88.89	91.89	62.93	38.86	24.07	8.88
	DeepSeek-chat	0.00	0.00	97.44	32.48	45.93	-13.45
	Llama-3.3	76.92	0.00	91.89	56.27	40.26	16.01

TABLE II: Performance, Stability, and Meta-Score Across RAG Systems and Models

Fig. 6: Converted table (HTML) after PDF split

Two vehicles of Category M1 AA saloon shall be positioned. The subject vehicle that performs the braking and the lead vehicle.

Both vehicles should face in the same direction of travel.

The subject vehicle shall approach the lead vehicle in a straight line for at least 2 s before the functional part of the test commences.

The subject vehicle should travel at the speed of 10 km/h (with a tolerance of

+

-

2 km/h) when the vehicle brakes.

The functional part of the test shall start at a distance corresponding to a Time To Collision (TTC) of at least 4 seconds from the target.

The subject vehicle is unladen.

The lead vehicle is stationary.

TABLE III: Ground-truth scenario Test_CtoStC_unladen_10 — expected LLM output for a CtoStC test at 10 km/h (unladen).

Fig. 7: F1-score scaling of RegulaRAG and HippoRAG as corpus size increases.

Fig. 8: Runtime scaling for RegulaRAG and HippoRAG .

Fig. 9: Token usage comparison across increasing corpus sizes.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving
Jul 15, 2026Yuan Gao, Wenting Miao, Mattia Piccinini +3Autonomous Driving Benchmarks Retrieval-Augmented Generation
Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, 85748 Garching, Germany; Munich Institute of Robotics and Machine Intelligence (MIRMI) · University College London, London, United Kingdom
AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems
Mar 10, 2026Athanasios Davvetas, Michael Papademas, Xenia Ziouvelou +1RAG Evaluation
National Centre for Scientific Research “Demokritos”, Institute of Informatics and Telecommunications, Aghia Paraskevi, Greece · Department of Communication, Media and Culture, Panteion, University of Social and Political Sciences, Athens, Greece
RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models
Jul 26, 2026Ange Maiztegi, Jon Ayerdi, Miren Illarramendi +1RAG Evaluation Automated Test Generation
Mondragon University, Arrasate-Mondragon, Gipuzkoa, Spain

Jul 15, 2026J/K move · Enter open · S save

Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving

Yuan Gao, Wenting Miao, Mattia Piccinini, Haoyu Wang+2

Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, 85748 Garching, Germany; Munich Institute of Robotics and Machine Intelligence (MIRMI) · University College London, London, United Kingdom

Autonomous Driving Benchmarks Retrieval-Augmented Generation Autonomous Driving Safety Evaluation Autonomous Driving Simulation Autonomous Driving Scenario Generation

Open paper AskPDF arXiv

Validating autonomous driving systems requires diverse, regulation-compliant test scenarios. In simulation-based testing, scenarios are defined as executable scripts. Yet automatically generating such scripts from regulatory descriptions remains an open challenge, and existing approaches face fundamental trade-offs. Retrieval-assemble methods achieve reasonable compilation rates but lack scalability, whereas retrieval-based full-script generation suffers from low compilation success rates. We present Chat2Scenic, the first iterative retrieval-augmented framework to generate scenario scripts in Domain Specific Language (DSL). Specifically, Chat2Scenic provides a chatbot interface that supports interactive scenario refinement and integrates Retrieval-augmented Generation (RAG) to ground scenario generation in regulatory knowledge and DSL syntax. Furthermore, we propose an open benchmark for scenario generation comprising 123 scenarios from various regulations, including NHTSA and United Nations Vehicle Regulations, as well as other sources. Extensive evaluation with State-of-the-Art (SOTA) Large Language Models (LLMs) demonstrates that Chat2Scenic achieves 76.42% Compilation Success Rate (CSR) and 58.17% Framework Accuracy (FA), outperforming existing methods (Retrieval Assemble with 30.08% CSR, 11.03% FA and Retrieval full script generation with 16.26% CSR, 10.86% FA). To facilitate future research, we release our code as open source at https://github.com/TUM-AVS/chat2scenic.

Abstract

Retrieval-Augmented Generation

Explore similar work

Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving

AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models