Coverage-Driven Verification for Safety-by-Design in AI-Based Collision Avoidance Systems

Thomas Stefani; Johann Maximilian Christensen; Elena Hoemann; Frank Köster; Sven Hallerbach

cs.AIAug 21, 2026

Coverage-Driven Verification for Safety-by-Design in AI-Based Collision Avoidance Systems

Authors: Thomas Stefani, Johann Maximilian Christensen, Elena Hoemann, Frank Köster, Sven Hallerbach

Organizations: Institute for AI Safety and Security, German Aerospace Center (DLR), Ulm, Germany · Institute for AI Safety and Security, German Aerospace Center (DLR), Sankt Augustin, Germany

Abstract

Artificial Intelligence (AI) offers significant potential for future aviation systems; however, its integration into safety-critical applications requires compliance with the aviation sector's stringent safety standards. For AI and Machine Learning (ML)-based systems, the European Union Aviation Safety Agency (EASA) emphasizes the need to demonstrate the representativeness and completeness of the Operational Design Domain (ODD) and the associated data distributions used during development and verification. Despite this requirement, a structured engineering process for defining target distributions and evaluating representativeness within ODDs remains largely unexplored. This work presents a method for representativeness assessment of AI/ML constituent ODDs in the context of aviation safety assurance. Starting from the methodical identification of suitable target distributions, a process flow is proposed that guides developers from ODD definition and parameter distribution modeling to the quantitative assessment and interpretation of coverage results with respect to EASA's learning assurance objectives. As quantitative measures, the chi-squared goodness-of-fit test is examined and found unsuitable for the large data sets arising in this setting, leading to the adoption of the Kullback--Leibler divergence and Cramér's $V$ for the representativeness assessment. The method is demonstrated using the example of AI-based airborne collision avoidance, employing experimental data from previous Horizontal Collision Avoidance System (HCAS) and Vertical Collision Avoidance System (VCAS) simulations. The results illustrate how statistical distribution comparison methods can support the assessment of representativeness for safety-critical AI applications and contribute toward a systematic Safety-by-Design AI engineering process aligned with emerging EASA guidance.

Figures & tables

Figure 1: Activity Diagram for the Coverage Assessment loop aligning with EASA’s MOC DM-08 through a two-stage process consisting of demonstrating data representativeness and completeness.

Panel of Figure 2 — Figure 2: Generic process flow for identifying the target distributions for the ODD parameters that are subsequently used for the coverage assessment. Starting from real-world data, over simulation data, physically motivated assumptions, to expert knowledge.

$D_{\mathrm{KL}}$ Ranges	Interpretation	Coverage Assessment Implication
$0\leq D_{\mathrm{KL}}<0.05$	Excellent	Strong evidence of representativeness
$0.05\leq D_{\mathrm{KL}}<0.15$	Good	Generally acceptable with rationale
$0.15\leq D_{\mathrm{KL}}<0.30$	Moderate	Mitigation or targeted sampling recommended
$0.30\leq D_{\mathrm{KL}}<0.50$	Poor	Coverage argument weakened
$D_{\mathrm{KL}}\geq 0.50$	Fail	Representativeness claim at risk

Table 1: Interpretation of KL divergence values demonstrating parameter representativeness (EASA MOC DM-07-2) [ Belov2011 , Bonnici2024 , Kvalseth2017 ] . Lower

D_{\mathrm{KL}}

indicates a better fit between the two distributions.

$V$ Ranges	Interpretation	Coverage Assessment Implication
$0\leq V<0.10$	Excellent	Strong evidence of representativeness
$0.10\leq V<0.20$	Good	Generally acceptable with rationale
$0.20\leq V<0.30$	Moderate	Mitigation or targeted sampling recommended
$0.30\leq V<0.40$	Poor	Coverage argument weakened
$V\geq 0.40$	Fail	Representativeness claim at risk

Table 2: Interpretation of Cramér’s

V

values for the

2\times k

goodness-of-fit framing, demonstrating parameter representativeness (EASA MOC DM-07-2). The thresholds are adopted from Cohen [ Cohen1988 ] and translated into finer granularity.

Variable	Description	Range
$\rho$	Range to intruder	[ $0\text{\,}\mathrm{ft}$ , $56\,000\text{\,}\mathrm{ft}$ ]
$\theta$	Bearing angle	[ − 180 ° , 180 ° ]
$\psi$	Relative heading	[ − 180 ° , 180 ° ]
$v_{\mathrm{own}}$	Ownship speed	$865\text{\,}\mathrm{ft}\text{\,}{\mathrm{s}}^{-1}$
$v_{\mathrm{int}}$	Intruder speed	[ $845\text{\,}\mathrm{ft}\text{\,}{\mathrm{s}}^{-1}$ , $868\text{\,}\mathrm{ft}\text{\,}{\mathrm{s}}^{-1}$ ]
$\tau$	Time to CPA	[ $0\text{\,}\mathrm{s}$ , $55\text{\,}\mathrm{s}$ ]

Table 3: Derived AI/ML constituent

ODD_{HCAS}

from the HCAS State Variables [ Julian2019 ] . The values are oriented on the GitHub release at https://github.com/sisl/HorizontalCAS but differ based on previous work [ Christensen2024a , Christensen2024 , Stefani2024a , Stefani2024b ] where

v_{\mathrm{own}}

was kept the same throughout the executed simulations and therefore remained at a constant value.

Variable	Description	Range
$h$	Relative altitude	[ $-2800\text{\,}\mathrm{ft}$ , $2800\text{\,}\mathrm{ft}$ ]
$\dot{h}_{\mathrm{own}}$	Ownship vert. rate	[ $-3200\text{\,}\mathrm{ft}\text{\,}{\mathrm{min}}^{-1}$ , $3200\text{\,}\mathrm{ft}\text{\,}{\mathrm{min}}^{-1}$ ]
$\dot{h}_{\mathrm{int}}$	Intruder vert. rate	[ $0\text{\,}\mathrm{ft}\text{\,}{\mathrm{min}}^{-1}$ ]
$\tau{}$	Time to CPA	[ $0\text{\,}\mathrm{s}$ , $40\text{\,}\mathrm{s}$ ]

Table 4: Derived AI/ML constituent

ODD_{VCAS}

from the VCAS State Variables [ Julian2019 ] . The values are oriented on the GitHub release at https://github.com/sisl/VerticalCAS but differ based on previous work [ Christensen2024a , Christensen2024 , Stefani2024a , Stefani2024b ] where only the ownship avoided the collision, therefore

\dot{h}_{\mathrm{int}}

is set to

0

Figure 3: Excerpt from the executed collision avoidance scenarios. The blue dots indicate the available data for the ownship’s vertical rate over the time to CPA $\tau$ . At approximately $30\text{\,}\mathrm{s}$ to CPA the ownship performs an avoidance maneuver through either climbing or descending.

Parameter	Target	Rationale / Source
	Distribution
$ODD_{VCAS}$ : $\dot{h}_{own}$	Normal distribution	The ownship vertical rate is assumed to follow a normal distribution based on expert knowledge of typical aircraft flight behavior. During nominal operations, aircraft predominantly operate around stable climb or descent rates, while extreme vertical rates occur less frequently. The normal distribution, therefore, reflects the expected concentration around standard operational conditions while still accounting for rare deviations.
$ODD_{HCAS}$ : $\tau$	Uniform distribution	In approximately unaccelerated encounters, time to CPA ( $\tau$ ) decays at a constant ratio with elapsed time. Recording potential encounters at a fixed sampling rate over a set window (e.g., from 55 to 0 seconds) therefore yields an approximately uniform distribution.

Table 5: Exemplary documentation of target distribution assumptions for ODD parameters in accordance with EASA MOC DM-07-2 traceability objectives.

Panel of Figure 4 — Figure 4: Sensitivity of the $\chi^{2}$ test and Cramér’s $V$ to the number of bins for the $ODD_{VCAS}$ parameter $\tau$ which has an approximately uniform distribution.

$ODD_{VCAS}$	Nr.	Target	Range	Bin Cov.	$D_{\mathrm{KL}}$	$D_{\mathrm{KL}}$	Cramér’s $V$	Cramér’s $V$
	Bins	Dist.	Util. ( $\mathrm{\%}$ )	( $\mathrm{\%}$ )	Value	Result	Value	Result
$h$	$100$	Normal	92.96	89.65	1.0382	Fail	$0.1682$	Good
$\dot{h}_{\mathrm{own}}$	$100$	Normal	87.80	64.35	1.5507	Fail	$0.2357$	Moderate
$\tau$	$100$	Uniform	374.85	100.00	0.0002	Excellent	$0.0013$	Excellent

Table 6: Coverage report for the

ODD_{VCAS}

with bin coverage, KL divergence and Cramér’s

V

$ODD_{HCAS}$	Nr.	Target	Range	Bin Cov.	$D_{\mathrm{KL}}$	$D_{\mathrm{KL}}$	Cramér’s $V$	Cramér’s $V$
	Bins	Dist.	Util. ( $\mathrm{\%}$ )	( $\mathrm{\%}$ )	Value	Result	Value	Result
$\rho$	$100$	Uniform	173.95	100.00	0.0236	Excellent	0.0212	Excellent
$\theta$	$100$	Uniform	100.00	99.41	1.1164	Fail	0.2782	Moderate
$\psi$	$100$	Uniform	100.00	95.29	1.1599	Fail	0.5563	Fail
$v_{\mathrm{int}}$	$20$	Uniform	469.36	100.00	0.0444	Excellent	0.0202	Excellent
$\tau$	$100$	Uniform	272.62	100.00	0.0019	Excellent	0.0060	Excellent

Table 7: Coverage report for the

ODD_{HCAS}

with bin coverage, KL divergence and Cramér’s

V

Figure 5: Kullback–Leibler divergence of relative altitude

h

from

ODD_{VCAS}

against normal distribution.

Figure 6: Cramér’s

V

of time to CPA

\tau

from

ODD_{VCAS}

against uniform distribution.

EASA item	Data-quality requirement	Addressed by
DA-04	Capture of data quality requirements, incl. completeness and representativeness, per ODD parameter, with documented origin and traceability	Target-distribution definition and documentation ( Figure 2 , Table 5 )
DM-07-2	Verification that data are sampled per the prescribed distribution; $\chi^{2}$ /KS goodness-of-fit anticipated for low-dimensional operating parameters	Representativeness assessment via $D_{\mathrm{KL}}$ and Cramér’s $V$ ( Section 4.3 )
DM-08	Data-verification step confirming ODD coverage with the necessary completeness and representativeness	Two-stage coverage loop ( Figure 1 )

Table 8: Mapping of the proposed coverage method to EASA’s anticipated objectives and means of compliance (Issue 02) for the data representativeness properties [ EUASA2024 ] .

AI Assurance

Artificial intelligence assurance develops evidence and controls to evaluate, govern, and certify AI systems across their lifecycle and deployment contexts.

See all 77 papers

Defining Operational Conditions for Safety-Critical AI-Based Systems from Data
Jan 29, 2026Johann Maximilian Christensen, Elena Hoemann, Frank Köster +1AI Assurance AI Safety
Institute for AI Safety and Security · German Aerospace Center (DLR) · Sankt Augustin, Germany
On the Applicability of Safety Nets: A Safety-By-Design Solution for Certifying Neural Networks
Aug 20, 2026Johann Maximilian Christensen, Thomas Stefani, Elena Hoemann +2Aviation Safety Neural Network Verification
Institute for AI Safety and Security, German Aerospace Center (DLR), Sankt Augustin, Germany · Institute for AI Safety and Security, German Aerospace Center (DLR), Ulm, Germany
Mechanistic Interpretability for Learning Assurance of a Vision-Based Landing System
May 20, 2026Romeo Valentin, Olivia Beyer Bruvik, Marc R. Schlichting +1Aviation Safety Representation Disentanglement
Stanford Intelligent Systems Laboratory, Stanford University, Stanford, CA, USA

Jan 29, 2026J/K move · Enter open · S save

Defining Operational Conditions for Safety-Critical AI-Based Systems from Data

Johann Maximilian Christensen, Elena Hoemann, Frank Köster, Sven Hallerbach

Institute for AI Safety and Security · German Aerospace Center (DLR) · Sankt Augustin, Germany

AI Assurance AI Safety

Open paper AskPDF arXiv

Artificial Intelligence (AI) has been on the rise in many domains, including numerous safety-critical applications. However, for complex systems in the real world, defining the underlying environmental conditions in which the AI-based system must operate---the Operational Design Domain (ODD)---is extremely challenging. This often results in an incomplete description of the ODD, which contrasts with the requirements of many domains for certifying AI-based systems. Traditionally, the ODD is created in the early stages of the development process, drawing on sophisticated expert knowledge and related standards. This paper presents a novel method for defining the ODD a posteriori from previously collected data using a multidimensional kernel-based representation. This approach is validated through both synthetic benchmarks and a real-world aviation use case. Moreover, the paper defines similarity of two ODDs if they generate the same outputs up to Lebesgue-null input sets and proves convergence in volume of the calibrated representation under the stated assumptions. The novel, safety-by-design, deterministic kernel-based ODD representation is derived fully automatically, given documented assurance inputs, permutation-stable, bounded by construction, and, under affine-equivariant per-dimension normalization, invariant to the choice of units. Utilizing the proposed ODD representation supports future certification of data-driven, safety-critical AI-based systems.

Abstract

AI Assurance

Explore similar work

Defining Operational Conditions for Safety-Critical AI-Based Systems from Data

On the Applicability of Safety Nets: A Safety-By-Design Solution for Certifying Neural Networks

Mechanistic Interpretability for Learning Assurance of a Vision-Based Landing System