TopoMap: A Feature-based Discriminator of the Topographical Regions in the Test Input Space

Gianmarco De Vita; Nargiz Humbatova; Paolo Tonella

cs.LGSep 3, 2025

TopoMap: A Feature-based Discriminator of the Topographical Regions in the Test Input Space

Authors: Gianmarco De Vita, Nargiz Humbatova, Paolo Tonella

Organizations: Università della Svizzera italiana, Lugano, Switzerland

Abstract

Testing Deep Learning (DL)-based systems is an open challenge. Although it is relatively easy to find inputs that cause a DL model to misbehave, the grouping of such inputs by features that make them problematic for the DL model under test is largely unexplored. In this work, we propose TopoMap, a model-agnostic approach that relies solely on features that characterise inputs to construct a partitioning of the input space where each region represents a specific set of distinguishable input features. Such a partitioning gives users a possibility to identify and analyse areas that are under-represented, show higher failure rates, or exhibit prediction changes after a model update. The identified regions of interest can then be used for additional data generation or can help prioritise already generated inputs that require labelling efforts. To discriminate inputs according to the specific features they share, we first apply dimensionality reduction to obtain input embeddings that are then subjected to clustering. Each DL model might require specific embedding computations and clustering algorithms, and specific configurations of the two, to achieve a meaningful separation of inputs into discriminative groups. We propose a novel way to evaluate alternative configurations of embedding and clustering techniques. We use a deep neural network (DNN) as an approximation of a human evaluator who could tell whether a pair of clusters can be discriminated based on the features of the included elements. The evaluation results show that the maps generated by TopoMap consist of distinguishable and meaningful regions. In addition, we evaluate the effectiveness of TopoMap using mutation analysis. In particular, we assess whether the clusters in our topographical map allow for an effective selection of mutation-killing inputs.

Figures & tables

Figure 1: Topographical map of aggregations of test input clusters killing various mutants ( Udacity steering-angle prediction dataset)

Figure 2: Cluster generation and evaluation pipeline

Figure 3: Computation of the weighted pairwise accuracy on a three-cluster prediction example

Figure 4: Empirical design of the human evaluation study

Figure 5: Structure of the mutation-based evaluation pipeline

Data	#Inputs	#Classes	Data dim.	Embedding dim.	Retained dim.
MN	10,000	10	784	86	10.97%
CF	10,000	10	3,072	98	3.19%
SR	1,350	5	8,000	565	7.06%
RE	2,246	46	10,000	1,910	19.10%
UD	2,432	( 3 )	51,200	274	0.53%
UE	25,857	( 9 )	2,162	35	1.62%

Table 1: Size of the test set and number of components in the feature space in their original raw form compared to the reduced dimensionality in the embedded form for each data scenario. Although both UD and UE datasets concern regression problems, we can group instances into buckets for the sake of having a starting point with respect to the number of clusters for the generation of the topographical map.

	MN	CF	SR	RE	UD	UE
Embedding	SVD	PCA	t-SNE	PCA	t-SNE	LDA
Clustering	K-means	K-means	K-means	BIRCH	BIRCH	K-means
$\bm{k^{*}}$	90	40	40	138	9	27
Sil. score (↑)	0.065	0.023	0.376	-0.088	0.380	0.120
DB index (↓)	2.453	2.806	0.806	3.607	0.820	1.677
CH index (↑)	120.077	246.947	2,555.596	6.164	2,422.404	2,057.630

Table 2: Selected topographical map for each data scenario by embedding/clustering algorithm configuration with respect to the accuracy of the cluster pair most difficult to discriminate by a DNN. Widely used metrics for cluster quality assessment (Silhouette score, DB/CH index) are included for comparison.

		$\bm{w_{acc}^{A,B}}$		$\bm{t_{acc}^{A,B}}$		$\bm{{t^{\prime}}_{acc}^{A,B}}$
	Metric	PCC	Mean $\bm{\pm}$ 3SE	PCC	Mean $\bm{\pm}$ 3SE	PCC	Mean $\bm{\pm}$ 3SE
MN	$h_{\textit{acc}}^{A,B}$	0.735	0.915 $\pm$ 0.043	0.736	0.919 $\pm$ 0.041	0.622	0.913 $\pm$ 0.053
MN	$h_{\textit{diff}}^{A,B}$	-0.691	0.915 $\pm$ 0.043	-0.631	0.919 $\pm$ 0.041	-0.585	0.913 $\pm$ 0.053
CF	$h_{\textit{acc}}^{A,B}$	0.351	0.856 $\pm$ 0.079	0.401	0.882 $\pm$ 0.058	0.347	0.866 $\pm$ 0.080
CF	$h_{\textit{diff}}^{A,B}$	0.086	0.856 $\pm$ 0.079	0.050	0.882 $\pm$ 0.058	-0.050	0.866 $\pm$ 0.080

Table 3: PCC, mean and 3 times the standard error (3SE) computed between human-based metrics and DNN-based metrics with respect to classification according to the cluster labels. Highlighted correlation coefficient indicates statistical significance (

p<

0.05).

		MN							CF							SR
	MO	#	$\bm{t_{c}}$	$\bm{\rho_{\bm{k}}}$	$\bm{\rho_{\bm{c}}}$	$\bm{\rho^{(R)}_{\bm{c}}}$	$\bm{\rho_{d}}$	$\bm{\rho^{(R)}_{d}}$	#	$\bm{t_{c}}$	$\bm{\rho_{\bm{k}}}$	$\bm{\rho_{\bm{c}}}$	$\bm{\rho^{(R)}_{\bm{c}}}$	$\bm{\rho_{d}}$	$\bm{\rho^{(R)}_{d}}$	#	$\bm{t_{c}}$	$\bm{\rho_{\bm{k}}}$	$\bm{\rho_{\bm{c}}}$	$\bm{\rho^{(R)}_{\bm{c}}}$	$\bm{\rho_{d}}$	$\bm{\rho^{(R)}_{d}}$
Killable	TAN	2	0.014	0.025	0.070	0.016	1.000	0.300	4	0.462	0.027	0.541	0.465	1.000	0.575	—	—	—	—	—	—	—
	RAW	3	0.013	0.030	0.060	0.013	1.000	0.233	12	0.433	0.053	0.511	0.427	1.000	0.750	—	—	—	—	—	—	—
	ACH	9	0.019	0.032	0.096	0.018	1.000	0.322	5	0.573	0.019	0.653	0.576	1.000	1.000	1	0.035	0.034	0.311	0.029	1.000	0.000
	HBS	3	0.015	0.022	0.075	0.016	1.000	0.033	1	0.391	0.033	0.466	0.375	1.000	0.100	—	—	—	—	—	—	—
	RCP	—	—	—	—	—	—	—	—	—	—	—	—	—	—	4	0.109	0.039	0.548	0.113	1.000	0.450

Table 4: Proportion

\rho_{k}

of input instances included into the killing cluster aggregation for three classification data scenarios (MN, CF and SR) averaged over ten empirical runs. The density of contributing inputs

\rho_{c}

quantifies the concentration of killing inputs in the considered cluster aggregations. Results are compared with the concentration of contributors

\rho_{c}

and the probability

\rho_{d}

of the mutant being killed by the aggregation obtained from the feature regions identified by a random selection.

		RE							UD							UE
	MO	#	$\bm{t_{c}}$	$\bm{\rho_{\bm{k}}}$	$\bm{\rho_{\bm{c}}}$	$\bm{\rho^{(R)}_{\bm{c}}}$	$\bm{\rho_{d}}$	$\bm{\rho^{(R)}_{d}}$	#	$\bm{t_{c}}$	$\bm{\rho_{\bm{k}}}$	$\bm{\rho_{\bm{c}}}$	$\bm{\rho^{(R)}_{\bm{c}}}$	$\bm{\rho_{d}}$	$\bm{\rho^{(R)}_{d}}$	#	$\bm{t_{c}}$	$\bm{\rho_{\bm{k}}}$	$\bm{\rho_{\bm{c}}}$	$\bm{\rho^{(R)}_{\bm{c}}}$	$\bm{\rho_{d}}$	$\bm{\rho^{(R)}_{d}}$
Killable	AAL	—	—	—	—	—	—	—	—	—	—	—	—	—	—	7	0.671	0.035	0.911	0.674	1.000	0.943
	TAN	1	0.089	0.995	0.089	0.089	1.000	1.000	—	—	—	—	—	—	—	3	0.713	0.035	0.801	0.716	1.000	1.000
	RAW	6	0.230	0.001	0.742	0.204	1.000	0.317	—	—	—	—	—	—	—	11	0.669	0.034	0.795	0.670	1.000	1.000
	ACH	14	0.431	0.005	0.757	0.438	1.000	0.493	—	—	—	—	—	—	—	5	0.793	0.035	0.917	0.791	1.000	1.000
	HBS	3	0.098	0.013	0.545	0.085	1.000	0.167	—	—	—	—	—	—	—	2	0.687	0.036	0.879	0.684	1.000	1.000

Table 5: Proportion

\rho_{k}

of input instances included into the killing cluster aggregation for one classification and two regression data scenarios (RE, UD and UE) averaged over ten empirical runs. The density of contributing inputs

\rho_{c}

quantifies the concentration of killing inputs in the considered cluster aggregations. Results are compared with the concentration of contributors

\rho_{c}

and the probability

\rho_{d}

of the mutant being killed by the aggregation obtained from the feature regions identified by a random selection.

	MO	#	$\bm{t_{c}}$	$\bm{\rho_{\bm{k}}}$	$\bm{\rho^{(DH)}_{\bm{k}}}$	$\bm{\rho_{\bm{c}}}$	$\bm{\rho^{(DH)}_{\bm{c}}}$	$\bm{\rho_{d}}$	$\bm{\rho^{(DH)}_{d}}$
Killable	TAN	2	0.014	1.5e-04	3.5e-04	0.983	0.800	1.000	1.000
	RAW	3	0.013	2.8e-04	1.0e-03	0.954	0.634	1.000	1.000
	ACH	9	0.019	2.8e-04	3.5e-04	0.942	0.966	1.000	1.000
	HBS	3	0.015	2.1e-04	5.4e-04	0.957	0.872	1.000	1.000
	HNE	3	0.023	3.2e-04	9.1e-04	0.884	0.847	1.000	1.000
	TCL	6	0.043	5.0e-04	3.5e-04	0.904	0.947	1.000	1.000

Table 6: Proportion

\rho_{k}

of inputs included into the killing cluster aggregation for the MN dataset averaged over ten empirical runs. The density of contributing inputs

\rho_{c}

quantifies the concentration of killing inputs in the considered cluster aggregations. Results are compared with the concentration of contributors

\rho_{c}

and the probability

\rho_{d}

of the mutant being killed by the aggregation obtained from the feature regions identified by DeepHyperion.

		DeepGini		Vanilla SM		Misclassified
Model		TM	DH	TM	DH	TM	DH
7-layer CNN	[ 16 ]	0.046	0.049	0.042	0.058	0.029	0.039
8-layer CNN	[ 14 ]	0.109	0.087	0.109	0.087	0.040	0.021
19-layer CNN	[ 31 ]	0.101	0.090	0.101	0.090	0.025	0.021
LeNet1	[ 2 ]	0.027	0.028	0.039	0.045	0.135	0.184
LeNet5	[ 2 ]	0.052	0.059	0.062	0.067	0.131	0.192

Table 7: Gini impurity for each mapping technique indicating their ability of separating critical (high-uncertainty and misclassified) inputs from non-critical ones across different architectures. Bold values indicate statistical significance (

p

-value

<0.05

MSC	MN	CF	SR	RE	UD	UE
First	67.74%	27.17%	45.23%	39.13%	65.38%	57.21%
Second	35.16%	19.78%	31.82%	33.33%	34.62%	26.72%
Third	24.52%	16.30%	17.95%	21.74%	11.54%	5.08%

Table 8: Proportion

\rho_{a}

of aggregations that contain the 1st, 2nd, 3rd most selected cluster (MSC) across the killable mutants per subject.

	#Configurations
MO	MN	CF	SR	RE	UD	UE	Avg#Cfgs	KillHalf	KillFull
AAL	—	—	—	—	—	7	7.00	100%	0%
TAN	2	4	—	1	—	3	2.50	78%	50%
RAW	3	12	—	6	—	11	8.00	55%	12%
ACH	9	5	1	14	—	5	6.80	72%	36%
HBS	3	1	—	3	—	2	2.25	95%	62%

Table 9: Probability that the input cluster aggregation selected by TopoMap kills at least half ( KillHalf ) or all ( KillFull ) of the MO configurations across different MO categories and subjects.

Figure 6: Topographical map computed on the MNIST dataset (SVD, K-means, 90)

Figure 7: Average contribution probability within the most selected cluster on the MNIST dataset (SVD, K-means, 90)

Panel of Figure 8 — Figure 8: Cohesion and separability of clusters in MNIST and CIFAR-10

Panel of Figure 9 — Figure 9: Graph-based visualisation of the topographical maps for MNIST and Speaker Recognition subjects

DNN Testing

See all 11 papers

Aug 15, 2026

Look Before You Lift: Visual and Quantitative Diagnostics for Topological Deep Learning

Topological deep learning (TDL) methods rely on lifting raw data into higher-order discrete domains such as simplicial complexes, cell complexes, and hypergraphs. In practice, this lifting step is often treated as a black box: practitioners select a lifting and then tune architectures, with limited visibility into whether the induced higher-order connectivity is meaningful for the downstream task. To address this missing diagnostic layer, we propose a visualization technique called TopoExplorer that leverages the strictly augmented Hasse graph form of topological datasets for exploratory data analysis. For the first time, practitioners can easily visualize the incidence- and adjacency-based neighborhoods that define the lifted dataset, as well as read off key graph metrics that describe its structural and feature landscape. Via an extensive set of experiments across many datasets and liftings, we show that several of these metrics correlate with downstream model performance, suggesting they can help inform TDL preprocessing design. Our perspective reframes the TDL workflow from lift-train to lift-look-design-train, enabling more principled, interpretable, and efficient model development. TopoExplorer is hosted at https://topoexplorer.pagekite.me, and its source code is available at github.com/geometric-intelligence/topoexplorer.

Mathilde Papillon, Guillermo Bernárdez, Álvaro Ballón Barreiro +4

University of California Santa Barbara · Sapienza Universitá di Roma · Arlequin AI

Topological Representation Learning Topological Data Analysis Scientific Visualization

Ask PDF

May 7, 2026

No Triangulation Without Representation: Generalization in Topological Deep Learning

Despite an ever-increasing interest in topological deep learning models that target higher-order datasets, there is no consensus on how to evaluate such models. This is exacerbated by the fact that topological objects permit operations, such as structural refinements, that are not appropriate for graph data. In this work, we extend MANTRA, a benchmark dataset containing manifold triangulations, to a larger class of manifolds with more diverse homeomorphism types. We show that, unlike prior claims, both graph neural networks (GNNs) and higher-order message passing (HOMP) methods can saturate the benchmark. However, we find that this is contingent on the right representation and feature assignment, emphasizing their importance in baseline models. We thus provide a novel evaluation protocol based on representational diversity and triangulation refinement. Surprisingly, we find no indication that existing models are capable of generalizing beyond the combinatorial structure of the data. This points towards a research gap in developing models that understand topological structure independent of scale. Our work thus provides the necessary scaffolding to evaluate future models and enable the development of topology-aware inductive biases.

Johannes S. Schmidt, Martin Carrasco, Ernst Röell +3

Dept. of Math. & Statistics University of Montréal Montréal, Canada · Dept. of Informatics University of Fribourg Fribourg, Switzerland

Neural Network Generalization Graph Neural Networks Message Passing Neural Networks Topological Representation Learning Topological Data Analysis

Ask PDF

Oct 15, 2026

Deep Learning of Topological Equivalence Classes

We demonstrate an application of deep learning to the problem of identifying whether two generating vectors correspond to topologically equivalent finite group actions on Riemann surfaces, showing how architectures developed for semantic understanding have use for pure mathematical problems. In doing so we discuss how desired algebraic structure influences model design, and explore which structural properties of topological equivalence classes the model has successfully learnt.

Linden Disney-Hogg, Max Zuev

Topic pending

Ask PDF

Abstract

DNN Testing

Explore similar work

Look Before You Lift: Visual and Quantitative Diagnostics for Topological Deep Learning

No Triangulation Without Representation: Generalization in Topological Deep Learning

Deep Learning of Topological Equivalence Classes