Organizations: Volvo Cars | University of Gothenburg and Chalmers University of Technology Gothenburg, Sweden · University of Groningen Groningen, Netherlands · Volvo Cars Gothenburg, Sweden · University of Gothenburg and Chalmers University of Technology Gothenburg, Sweden
Data leakage is the inadvertent transfer of information between training and evaluation datasets that poses a subtle, yet critical, risk to the reliability of machine learning (ML) models in safety-critical systems such as automotive perception. While leakage is widely recognized in research, little is known about how industrial practitioners actually perceive and manage it in practice. This study investigates practitioners' knowledge, experiences, and mitigation strategies around data leakage through ten semi-structured interviews with system design, development, and verification engineers working on automotive perception functions development. Using reflexive thematic analysis, we identify that knowledge of data leakage is widespread and fragmented along role boundaries: ML engineers conceptualize it as a data-splitting or validation issue, whereas design and verification roles interpret it in terms of representativeness and scenario coverage. Detection commonly arises through generic considerations and observed performance anomalies rather than implying specific tools. However, data leakage prevention is more commonly practiced, which depends mostly on experience and knowledge sharing. These findings suggest that leakage control is a socio-technical coordination problem distributed across roles and workflows. We discuss implications for ML reliability engineering, highlighting the need for shared definitions, traceable data practices, and continuous cross-role communication to institutionalize data leakage awareness within automotive ML development.
Figures & tables
Figure 1. A graphical overview of the study.
ID
Role
SDLC Category
Experience (Yrs.)
P1
System Design Engineer
Design
1–1.5
P2
System Design Engineer
Design
1
P3
Data Architect
Implementation
2
P4
Data Scientist
Implementation
10+
P5
ML Engineer
Implementation
5
P6
ML Engineer
Implementation
4
Table 1. Participant demographics (N=10) with their role in the development process and the number of years of experience within the current roles.
Question No.
Questions
Related RQs
Q1
How would you define data leakage generally? How in the context of image data?
RQ1
Q2
Do you think data leakage is a common problem in ML projects? Why or why not?
RQ1
Q3
Do you consider data leakage when planning training and validation?
RQ1
Q4
Have you come across any notable examples of data leakage (from research papers, case studies, or real-world projects)?
RQ1
Q5
Have you ever encountered data leakage in your own work? If so, can you describe the situation?
RQ2
Q6
What was the impact of the data leakage (e.g., overestimated performance, poor generalization)?
RQ2
Table 2. The list of interview questions mapped into the corresponding research questions.
Figure 2. A mindmap of the main themes and sub-themes produced from the participants’ responses and also motivated by the RQs. The leaf nodes contain the most frequent codes mentioned by the practitioners. Some codes were repeatedly mentioned while answering questions related to different themes, such as Duplicate Images and Image Similarity Analysis . Duplicate Images were used to give examples, so it also came out as one of the mentioned causes of data leakage for obvious reasons. Practitioners also repeatedly mentioned Image Similarity Analysis , which can be used to detect data leakage as well as to prevent it.
Data leakage has been identified in 648 published papers across 30 scientific fields. The knowledge to prevent it has existed for over a decade; the problem persists because the tools do not enforce what the textbooks teach. This paper presents a grammar (eight typed primitives connected by a directed acyclic graph with four hard constraints) that makes the most damaging leakage types structurally unrepresentable within the grammar's scope. The core mechanism is a terminal assessment gate: the first call-time-enforced evaluate/assess boundary documented in the peer-reviewed ML methodology literature (to my knowledge, as of May 2026), backed by a specification precise enough for independent reimplementation. A companion landscape study across 2,047 datasets grounds the constraints in measured effect sizes. Two reference implementations (Python, R) are available.
Twenty-eight within-subject counterfactual experiments across 2,047 iid tabular datasets, plus a boundary experiment on 129 temporal datasets, measure the severity of four data leakage classes in machine learning. Class I (estimation: fitting scalers on full data) is negligible: all nine conditions produce ∣ΔAUC∣≤0.005. Class II (selection: peeking, seed cherry-picking) is substantial: the measured effect is consistent with about 90% noise exploitation inflating reported scores. Class III (memorization) scales with model capacity: dz = 0.37 (Naive Bayes) to 1.11 (Decision Tree) at 10% duplication. Class IV (boundary) is invisible under random cross-validation. Within this iid tabular regime, the textbook emphasis is inverted: normalization leakage matters least; selection leakage at practical dataset sizes matters most.
We present MLPerf Automotive, the first standardized public performance benchmark for evaluating Machine Learning systems that are deployed for AI acceleration in automotive systems. Developed through a collaborative partnership within MLCommons, this benchmark addresses the need for standardized performance evaluation methodologies in automotive machine learning systems. Existing benchmark suites cannot be utilized for these systems since automotive workloads have unique constraints including sensor suites, safety, and real-time processing that distinguish them from the domains that previously introduced benchmarks target. Our implemented and adopted MLPerf Automotive benchmark is a framework for evaluation and methodology for benchmarking automotive systems with reproducible performance metrics. The benchmark consists of automotive perception tasks in 2D object detection, 2D semantic segmentation, 3D object detection, end-to-end driving, and an infotainment system application. We carefully curated and customized models for automotive use cases. We describe the methodology behind the benchmark design including the task selection, reference models, and submission rules. We also discuss the challenges involved in acquiring the datasets and the engineering efforts to develop the reference implementations. Our benchmark code is available at https://github.com/mlcommons/mlperf_automotive.