Software Frameworks for Explainable AI in Time Series Classification: A Systematic Review

Louis Peter; Nils Gumpfer; Jana Fischer; Christin Seifert; Jennifer Hannig

cs.AIAug 19, 2026

Software Frameworks for Explainable AI in Time Series Classification: A Systematic Review

Authors: Louis Peter, Nils Gumpfer, Jana Fischer, Christin Seifert, Jennifer Hannig

Organizations: Technische Hochschule Mittelhessen - University of Applied Science, Friedberg, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · Marburg University, Marburg, Germany

Abstract

Time series arise in a wide range of application domains and are analyzed using machine learning in decision-critical settings. Time series classification (TSC) is one of the most widely studied and relevant tasks. In this context, ensuring the transparency and trustworthiness of TSC models has become an important requirement, motivating the use of explainable artificial intelligence (XAI) methods. Despite growing interest, research on XAI for TSC remains fragmented, and a systematic understanding of the available software frameworks for explanation generation, their evaluation practices, and practical limitations is still lacking. Prior work largely focused on individual explanation methods, while cross-framework consistency, time-series-specific evaluation, and reproducibility have received little attention. In this survey, we analyze existing software frameworks for explanation generation and evaluation in TSC. We compare them along multiple dimensions, including supported XAI methods, evaluation metrics, usability, benchmarking support, and reproducibility, providing the first time-series-specific survey of frameworks with implementation comparisons and an analysis of frequency-domain support. We identify six frameworks that explicitly support time series and reveal common limitations: only one method supports frequency-domain explanations despite their relevance; only two evaluation metrics have been developed specifically for time series; and identical XAI methods can yield substantially different explanations across frameworks. Based on these findings, we discuss open challenges and outline directions for future research, highlighting the need for unified, time-series-specific XAI frameworks that enable faithful, reproducible, and time-series-aware explanations.

Figures & tables

Figure 1: The electrocardiogram (ECG) signal is a multivariate time series in which each channel corresponds to a distinct measurement location (lead). All channels reflect the same underlying cardiac process with strong temporal and synchronous cross-channel dependencies. An example with right bundle branch block (RBBB) shows characteristic patterns in leads V1 and V6, highlighting the need for multichannel analysis.

Figure 2: Flowchart of selection and review process.

Figure 3: Dependency ecosystem of XAI frameworks (arrows indicate dependencies). White nodes support time series; black nodes are core dependencies without time series support. For XAI method frameworks, o n line, colback=black!80, colframe=black, boxrule=0pt, arc=2pt,left=0pt, right=0pt, top=0pt, bottom=0pt] X indicates the number of time-series-specific XAI methods. For XAI evaluation frameworks, o n line, colback=black!80, colframe=black, boxrule=0pt, arc=2pt,left=0pt, right=0pt, top=0pt, bottom=0pt] X/Y indicates the number of time-series-specific XAI methods (X) and evaluation metrics (Y).

							XAI Methods				Metrics
XAI Method Frameworks
TSInterpret [ 18 ]	11/2025	144	18	P	T, P, S	5-4-5	6 + (10)	2	4	0	0	0
tsCaptum [ 30 ]	11/2024	10	1	P	P	3-2-4	(5)	0	0	0	0	0
SIGN-XAI-2 [ 16 ]	12/2025	6	0	P	P	5-5-5	2 + (9)	1	0	1	0	0
XAI Evaluation Frameworks
Quantus [ 17 ]	07/2025	631	83	P	T, P	5-4-5	(26)	0	0	0	36	0

Table 1: Overview of the frameworks with a reference to the papers proposing frameworks or to the software release (URLs are given in the reference entries). : last update on GitHub; : number of stars on GitHub; : number of forks on GitHub; : availability local (L) or as installable package (P); : supported machine-learning backend (TensorFlow (T), PyTorch (P), and Scikit-learn (S)); : Usability scores in order of active maintenance, interaction with community, and documentation (3 scores denoted as X–X–X, each in the range 0–5 from lowest to best); : number of supported XAI methods (implemented methods and, in parenthesis, wrapped or reused implementations of other frameworks); : number of methods specific to univariate time series; : number of methods specific to multivariate time series; : number of methods generating explanations in the frequency domain; : number of supported XAI metrics (implemented metrics and, in parenthesis, wrapped or reused implementations of other frameworks); : number of metrics specific to time series.

Figure 4: Spoken digit nine from the AudioMNIST data by a female (upper) and a male (lower) speaker. DFT-LRP explanations (SIGN-XAI-2 framework) are shown in the time (left) and frequency domain (right). Red dots indicate features (time steps or frequencies) with positive relevance scores; higher intensity corresponds to higher relevance. The expected ground truth in the frequency domain is highlighted in orange.

Dataset	Frameworks	Channels		L	S	GT

Arma [ 7 ]	time_interpret	✓	✓	v	✓	✓
BioBank [ 31 ]	time_interpret
Hawkes [ 3 ]	time_interpret	✓	✓	v	✓	✓
Hidden Markow Model [ 7 ]	time_interpret	✓	✓	v	✓	✓
Mimic III [ 21 ]	time_interpret	✗	31	48	✗	✗

Table 2: Datasets of frameworks. Frameworks (F) : Frameworks supporting the dataset; Channels and Length (L) : Univariate ( ) or multivariate ( ), synthetic datasets have variable (v) lengths and channel number; Synthetic (S) : ✓ indicates dataset is synthetic, mixed (m), and ✗ otherwise; Ground Truth (GT) : ✓ indicates dataset contains GT information and ✗ otherwise; *no open access; † Archives consist of multiple datasets; here we only indicate their variability at the collection level.

Panel of Figure 5 — Figure 5: Electrocardiogram (ECG) example with right bundle branch block (RBBB). Left: Illustration of characteristic “M” -shaped pattern (rSR’). Middle and right: Explanations generated using Integrated Gradients as implemented in TSInterpret (middle, based on Captum) and SIGN-XAI-2 (right, based on Zennit) for the same ECG instance, illustrating differences in relevance distributions across implementations. Relevance scores are indicated by red dots; higher intensity indicates higher relevance.

Interpretable Time-Series Classification

See all 16 papers

What-If Explanations Over Time: Counterfactuals for Time Series Classification
Mar 29, 2026Udo Schlegel, Thomas SeidlGraph Counterfactual Explanations Counterfactual Explanations
LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany
L2GTX: From Local to Global Time Series Explanations
Mar 13, 2026Ephrem Tibebe Mekonnen, Luca Longo, Lucas Rizzo +1Explainable Artificial Intelligence Interpretable Time-Series Classification
School of Computer Science, Technological University Dublin Dublin, Ireland · Artificial Intelligence and Cognitive Load Research Lab, University College Cork Cork, Ireland · School of Computer Science and Information Technology, University College Cork Cork, Ireland +1
Towards A Unified Information Bottleneck Framework for Time Series Explanations
Aug 26, 2026Xu Zheng, Zichuan Liu, Zhuomin Chen +7Explainable Time Series Forecasting Explainable Artificial Intelligence
Florida International University · Carnegie Mellon University · University of Houston +2

Mar 29, 2026J/K move · Enter open · S save

What-If Explanations Over Time: Counterfactuals for Time Series Classification

Udo Schlegel, Thomas Seidl

LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany

Graph Counterfactual Explanations Counterfactual Explanations Counterfactual Evaluation Interpretable Time-Series Classification Time Series Classification

Open paper AskPDF arXiv

Counterfactual explanations emerge as a powerful approach in explainable AI, providing what-if scenarios that reveal how minimal changes to an input time series can alter the model's prediction. This work presents a survey of recent algorithms for counterfactual explanations for time series classification. We review state-of-the-art methods, spanning instance-based nearest-neighbor techniques, pattern-driven algorithms, gradient-based optimization, and generative models. For each, we discuss the underlying methodology, the models and classifiers they target, and the datasets on which they are evaluated. We highlight unique challenges in generating counterfactuals for temporal data, such as maintaining temporal coherence, plausibility, and actionable interpretability, which distinguish the temporal from tabular or image domains. We analyze the strengths and limitations of existing approaches and compare their effectiveness along key dimensions (validity, proximity, sparsity, plausibility, etc.). In addition, we implemented an open-source implementation library, Counterfactual Explanations for Time Series (CFTS), as a reference framework that includes many algorithms and evaluation metrics. We discuss this library's contributions in standardizing evaluation and enabling practical adoption of explainable time series techniques. Finally, based on the literature and identified gaps, we propose future research directions, including improved user-centered design, integration of domain knowledge, and counterfactuals for time series forecasting.

Abstract

Interpretable Time-Series Classification

Explore similar work

What-If Explanations Over Time: Counterfactuals for Time Series Classification

L2GTX: From Local to Global Time Series Explanations

Towards A Unified Information Bottleneck Framework for Time Series Explanations