DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant

Lev Sorokin; Ivan Vasilev; Samuele Pasini

cs.AIApr 14, 2026

DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant

Authors: Lev Sorokin, Ivan Vasilev, Samuele Pasini

Organizations: BMW Group · Technical University of Munich, Munich, Germany · Università della Svizzera italiana, Lugano, Switzerland

Abstract

This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at ICSE 2026. Four tools competed in benchmarking an LLM-based car manual information retrieval application, with the objective of identifying user inputs for which the system fails to appropriately mention warnings contained in the manual. The testing solutions were evaluated based on their effectiveness in exposing failures and the diversity of the discovered failure-revealing tests. We report on the experimental methodology, the competitors, and the results.

Figures & tables

Figure 1. Example user request for guidance on activating ACC to an LLM-based manual assistant. The left response is unsafe because it ignores weather-related obstructions, while the right response addresses potential ACC malfunctions. Left system output could lead to potentially safety critical situations.

Model	Tool	SUT-I – Manual-1				SUT-I – Manual-2				SUT-II – Manual-1				SUT-II – Manual-2
Model	Tool	W	Rate	Cov	S	W	Rate	Cov	S	W	Rate	Cov	S	W	Rate	Cov	S
GPT-4o-Mini	ATLAS	0.61	0.52	0.74	0.62	0.64	0.51	0.92	0.69	0.57	0.32	0.89	0.59	0.60	0.36	0.89	0.62
	CRISP	0.42	0.32	0.50	0.42	0.46	0.32	0.68	0.49	0.34	0.15	0.61	0.37	0.35	0.17	0.69	0.41
	Exida	0.39	0.65	0.60	0.55	0.41	0.67	0.91	0.66	0.34	0.45	0.84	0.54	0.36	0.45	0.85	0.55
	Random	0.49	0.35	0.79	0.54	0.51	0.34	0.94	0.60	0.38	0.23	0.88	0.50	0.40	0.24	0.90	0.51
	Warnless	0.51	0.58	0.78	0.62	0.55	0.59	0.86	0.67	0.36	0.30	0.84	0.50	0.40	0.32	0.86	0.53

Table 1. Results for submitted tools and random as baseline for different systems and manuals, averaged over six runs.

Panel of Figure 2 — Figure 2. Metric results for SUT-I (left) and SUT-II (right) with manual Manual-2 and LLM GPT-4o-Mini over 6 runs.

Tool	Utterance
ATLAS	URGENT! How do I, um, secure the child seat in the cargo area? I am worried it is not fitting properly!
Exida	Can I pull over to check the child seat as I am rushing to the airport with my young child in freezing temperatures?
Warnless	How can I adjust the backrest to ensure the child restraint system fits securely?
CRISP	What should I do for child restraint system if on icy roads?
Random	How do I properly install the child seat when the backrest is tilted?

Table 2. Example utterances generated when targeting a warning related to the installation of a child seat.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Specification-Based Testing of Large Language Models
Oct 10, 2026Kausar Moshood, Jace Bolante, Roshan Aditya Murali +1LLM Safety Evaluation LLM Evaluation
School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA
Prioritizing Repeated LLM Evaluation for Hidden Failure Discovery
Sep 26, 2026Keita Broadwater, Akin BroadwaterLLM Evaluation LLM Reliability
Independent Researcher San Jose, CA USA · Bellarmine College Prepatory San Jose, CA USA
SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
May 21, 2026Yuxuan Sun, Yuze Zhao, Yufeng Wang +6LLM Evaluation Automated Software Testing
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Independent Researcher. · 3Beihang University. +3

Oct 10, 2026J/K move · Enter open · S save

Specification-Based Testing of Large Language Models

Kausar Moshood, Jace Bolante, Roshan Aditya Murali, Manish Motwani

School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA

LLM Safety Evaluation LLM Evaluation LLM Safety Benchmarks

Open paper AskPDF arXiv

Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined benchmarks rather than asking which documented claims those benchmarks actually test. We introduce LEST (LLM Evaluation through Specification-driven Testing), a framework that treats documented safety claims as specifications for black-box testing. In LEST, each claim determines which benchmark prompt tests it and whether a Refuse-to-Comply transition, where a model refuses a prompt but complies with its mutated version, counts as evidence against it. LEST mutates these prompts to search for counterexamples, manually validates each candidate, and traces validated counterexamples back to the claims they contradict. We evaluate LEST using the system cards of Claude Opus 4.8 and GPT-5.5 with 97 seed prompts from SORRY-Bench across Mental Health and Self-Harm, Harmful and Violent Content, and Fairness and Bias. Of 56 identified safety claims, 29 (51.8%) are prompt-testable, but only 17 (58.6%) are exercised by an applicable benchmark prompt. Both models refuse >95% of benchmark prompts. Evaluating 1,836 mutated prompts on both models reveals 312 candidate Refuse-to-Comply transitions flagged by automated response classifiers. However, human validation confirms only 19 (6.1%) as both intent-preserving and genuinely compliant, showing automated evaluation substantially overestimates behavioral failures. Tracing these 19 transitions back to the claims reveals counterexamples for 6 of 9 covered claims for which refusal is an applicable oracle. Our results show that benchmark prompts may not be sufficient to validate all documented LLM safety behavior, and automated failure detection requires manual validation. LEST provides end-to-end traceability from natural-language safety documentation to executable tests and human-validated counterexamples.

Abstract

LLM Evaluation

Explore similar work

Specification-Based Testing of Large Language Models

Prioritizing Repeated LLM Evaluation for Hidden Failure Discovery

SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?