Sufficient and Necessary Causal Explanations for the Analysis of Autonomous Systems

Tom Yaacov; Khen Elimelech; Hana Chockler; Moshe Y. Vardi

cs.ROOct 21, 2026

Sufficient and Necessary Causal Explanations for the Analysis of Autonomous Systems

Authors: Tom Yaacov, Khen Elimelech, Hana Chockler, Moshe Y. Vardi

Organizations: King’s College London · Rice University

Abstract

Understanding failures of autonomous systems is challenging, particularly when their behavior depends on black-box components such as neural networks. In this work, we make a distinction between two objectives of failure analysis: reproduction, which seeks to identify conditions sufficient to reproduce an observed failure, and avoidance, which seeks to identify the changes required to prevent it. Previous work on causal explanations of system failures is based on the concept of sufficient explanations and naturally supports reproduction. Nevertheless, we observe that it does not address the problem of avoiding the failure. In this paper, we introduce and formalize the concept of necessary explanations for system analysis as the correct formalism for failure avoidance. We place necessary explanations in the framework of actual causality and show that, together with sufficient explanations, they provide a unified causal framework for addressing both objectives of failure analysis. We present an efficient approximation algorithm for generating necessary explanations based on the notion of causal responsibility. We evaluate the efficiency of our approach across multiple environments. We demonstrate the practical utility of necessary explanations for failure avoidance through counterfactual generation and show that it outperforms existing methods.

Figures & tables

Fig. 1 : The autonomous car in the LiteRacer simulator [ 11 ] .

Panel of Figure 2 — Fig. 2 : A graphical representation of the causal model used to explain failures of our autonomous-car system. The red circles represent the variables in the causal model. The variable $O$ is determined from the simulation outcome, with the system treated as a black box.

Panel of Figure 3 — Fig. 3 : An illustration of the difference between sufficient and necessary explanations.

Fig. 4 : The Metadrive simulation [ 23 ] . The arrows point to obstacles that form a necessary explanation.

		15		16		17		18		19
	Env.	Exh.	Resp.	Exh.	Resp.	Exh.	Resp.	Exh.	Resp.	Exh.	Resp.
LiteRacer	Sin-SN	442	161	877	559	1270	901	2584	1909	2816	2076
	Sin-SW	224	93	240	114	298	140	394	142	386	171
	Sin-LN	290	148	485	129	726	475	745	580	808	481
	Sin-LW	208	94	232	56	339	132	328	158	383	176
	Seg-SN	737	501	710	520	1050	690	1764	1126	3169	2305

TABLE I : Efficiency of explanation generation algorithms (number of simulation runs; lower is better)

		15			16			17			18			19
	Env.	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra
LiteRacer	Sin-SN	1.0	.85	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	1.0	1.0
	Sin-SW	1.0	1.0	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	.90	1.0	1.0	1.0	1.0
	Sin-LN	1.0	1.0	1.0	1.0	.95	1.0	1.0	.85	1.0	1.0	1.0	1.0	1.0	1.0	1.0
	Sin-LW	1.0	1.0	1.0	1.0	.90	1.0	1.0	.90	1.0	1.0	.90	1.0	1.0	1.0	1.0
	Seg-SN	1.0	.95	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	.90	1.0	1.0	.95	1.0

TABLE II : Counterfactual validity (correctness) rate for the necessary explanations (our approach), DeCaF, and random methods with a sample budget of

400

		15			16			17			18			19
	Env.	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra
LiteRacer	Sin-SN	.25	.38	.58	.14	.27	.49	.29	.40	.88	.16	.51	.61	.42	.67	1.03
	Sin-SW	.15	.18	.31	.15	.24	.43	.16	.25	.39	.22	.40	.56	.25	.33	.49
	Sin-LN	.18	.38	.62	.15	.42	.73	.30	.53	.79	.28	.52	.74	.34	.62	1.43
	Sin-LW	.13	.28	.40	.18	.31	.57	.26	.32	.62	.13	.22	.65	.23	.38	.80
	Seg-SN	.28	.54	.76	.25	.35	.72	.25	.44	.82	.35	.59	.94	.48	.74	1.07

TABLE III : Quality of counterfactuals generated using necessary explanations, DeCaF, and random methods (average counterfactual distance; lower is better).

Panel of Figure 8 — Fig. 5 : Quality of counterfactuals as a function of the number of simulation runs ( $100$ – $400$ ) for necessary explanations (our approach), DeCaF, and random methods, measured by the average distance from the original example (lower is better).

		15			16			17			18			19
	Env.	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra
LiteRacer	Sin-SN	11	18	11	10	16	9	9	17	9	9	15	8	8	15	8
	Sin-SW	11	20	11	10	17	10	10	16	9	8	14	8	9	15	9
	Sin-LN	19	27	18	16	23	15	14	20	14	15	23	14	15	22	14
	Sin-LW	19	27	19	16	22	16	13	19	14	15	21	15	15	23	15
	Seg-SN	16	25	15	12	20	11	10	15	9	9	16	9	11	19	10

TABLE IV : Elapsed time (minutes) for the necessary explanations, DeCaF, and random methods.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 25 — Fig. 21 : Execution time as a function of the number of simulation runs.

		15			16			17			18			19
	Env.	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra
LiteRacer	Sin-SN	1.0	.95	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	1.0	1.0
	Sin-SW	1.0	.80	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	1.0	1.0	1.0	.95	1.0
	Sin-LN	1.0	.80	1.0	1.0	.90	1.0	1.0	.95	1.0	1.0	.90	1.0	1.0	.95	1.0
	Sin-LW	1.0	.85	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	.80	1.0	1.0	.80	1.0
	Seg-SN	1.0	.90	1.0	1.0	.95	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	1.0	1.0

Appendix

TABLE V : Counterfactual validity (correctness) rate for the necessary explanations (our approach), DeCaF, and random methods with a sample budget of

100

		15			16			17			18			19
	Env.	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra
LiteRacer	Sin-SN	1.0	.95	1.0	1.0	.95	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	1.0	1.0
	Sin-SW	1.0	.90	1.0	1.0	.85	1.0	1.0	.90	1.0	1.0	.95	1.0	1.0	.95	1.0
	Sin-LN	1.0	1.0	1.0	1.0	.80	1.0	1.0	.85	1.0	1.0	.90	1.0	1.0	.95	1.0
	Sin-LW	1.0	.90	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	.90	1.0	1.0	.95	1.0
	Seg-SN	1.0	.90	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	1.0	1.0

Appendix

TABLE VI : Counterfactual validity (correctness) rate for the necessary explanations (our approach), DeCaF, and random methods with a sample budget of

200

		15			16			17			18			19
	Env.	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra	NE	De	Ra
LiteRacer	Sin-SN	1.0	.90	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	.90	1.0	1.0	1.0	1.0
	Sin-SW	1.0	1.0	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	.90	1.0	1.0	1.0	1.0
	Sin-LN	1.0	.95	1.0	1.0	.85	1.0	1.0	1.0	1.0	1.0	.95	1.0	1.0	1.0	1.0
	Sin-LW	1.0	.95	1.0	1.0	.90	1.0	1.0	.95	1.0	1.0	.75	1.0	1.0	.85	1.0
	Seg-SN	1.0	.90	1.0	1.0	.95	1.0	1.0	.95	1.0	1.0	1.0	1.0	1.0	.95	1.0

Appendix

TABLE VII : Counterfactual validity (correctness) rate for the necessary explanations (our approach), DeCaF, and random methods with a sample budget of

300

Panel of Figure 29 — Fig. 22 : Quality of counterfactuals as a function of the number of simulation run samples for settings with $15$ obstacles, for necessary explanations (our approach), DeCaF, and random methods, measured by the average distance from the original example (lower is better).

Panel of Figure 30 — Fig. 23 : Quality of counterfactuals as a function of the number of simulation run samples for settings with $16$ obstacles, for necessary explanations (our approach), DeCaF, and random methods, measured by the average distance from the original example (lower is better).

Panel of Figure 31 — Fig. 24 : Quality of counterfactuals as a function of the number of simulation run samples for settings with $17$ obstacles, for necessary explanations (our approach), DeCaF, and random methods, measured by the average distance from the original example (lower is better).

Panel of Figure 32 — Fig. 25 : Quality of counterfactuals as a function of the number of simulation run samples for settings with $18$ obstacles, for necessary explanations (our approach), DeCaF, and random methods, measured by the average distance from the original example (lower is better).

Causal Explanation

Causal explanation uses causal models and interventions to identify why outcomes or model predictions occur.

See all 26 papers

Explaining Failures of Cyber-Physical Systems with Actual Causality
Jun 23, 2026Khen Elimelech, Tom Yaacov, David A. Kelly +2Agent Failure Analysis Cyber-Physical Systems
King’s College London · Rice University
Causal Analysis of Failures of Cyber-Physical Systems with Continuous Inputs
Oct 22, 2026Tom Yaacov, David A. Kelly, Khen Elimelech +1Causal Explanation Actual Causality
King’s College London
Coverage Aware Active Evaluation for Failure Discovery with Paired Systems
Aug 13, 2026Anjali Parashar, Rachel Luo, Apoorva Sharma +6Robot Failure Detection Adaptive Sampling
Laboratory of Information & Decision Systems (LIDS), MIT, USA · NVIDIA Research, USA

Jun 23, 2026J/K move · Enter open · S save

Explaining Failures of Cyber-Physical Systems with Actual Causality

Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler+1

King’s College London · Rice University

Agent Failure Analysis Cyber-Physical Systems Actual Causality Causal Reasoning Causal Explanation

Open paper AskPDF arXiv

Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, face increasingly complex demands, and yet are expected to act reliably. The black-box nature often characterizing such systems, especially those relying on neural components, makes it impossible to fully verify the system behavior prior to deployment. Unfortunately, unexpected failures-when the system does not comply with its specification-are inevitable and may have catastrophic implications. To improve trust in the system and facilitate future mitigation after a failure occurs, it is important to try to derive an explanation for the unexpected system behavior. This paper introduces the novel concept of leveraging the framework of actual causality for CPS failure explanation. Up until now, this framework was only used to derive explanations in the context of simple systems, such as image classifiers. This paper addresses the theoretical gaps and provides the guidance needed to allow for correct explanation derivation in the CPS domain. Beyond the theoretical contribution, the paper presents two novel, practical, system-agnostic explanation derivation algorithms, allowing to prioritize either explanation optimality or derivation efficiency. The approach is demonstrated and evaluated in the context of a neural-network-controlled autonomous car, designed to avoid collisions.

Abstract

Causal Explanation

Explore similar work

Explaining Failures of Cyber-Physical Systems with Actual Causality

Causal Analysis of Failures of Cyber-Physical Systems with Continuous Inputs

Coverage Aware Active Evaluation for Failure Discovery with Paired Systems