Stable Dimensionality Reduction via Isometric Centroid-Encoder

Tomojit Ghosh; Siva Rama Krishna Prasad Changala; Prathyusha Kanakamalla; Harigovind Harikumar

cs.LGOct 24, 2026

Stable Dimensionality Reduction via Isometric Centroid-Encoder

Authors: Tomojit Ghosh, Siva Rama Krishna Prasad Changala, Prathyusha Kanakamalla, Harigovind Harikumar

Organizations: Colonel Glenn Hwy, Dayton, OH 45435-0001, USA Department of Computer Science and Engineering Wright State University

Abstract

Many supervised dimensionality reduction methods promote intra-class compactness and inter-class separation but do not explicitly constrain class geometry, leading to variability in the relative arrangement of classes across independent runs. Such variability limits the reproducibility and interpretability of low-dimensional embeddings. We propose the Isometric Centroid-Encoder (ICE), a supervised dimensionality reduction framework in which class centroids are learned directly in the latent space while preserving a prescribed pairwise class-centroid geometry. Unlike existing centroid- or prototype-based methods that primarily encourage intra-class compactness, ICE explicitly regularizes the relative arrangement of latent class centroids. Unlike multidimensional scaling and many metric-learning approaches that preserve sample-level relationships, ICE operates on class centroids, preserving inter-class geometry without requiring large numbers of sample-pair or triplet constraints. We further show that the intrinsic dimensionality required to represent the centroid-distance configuration of $M$ classes is at most $M-1$ , explaining the observed saturation of distance-preservation measures as the latent dimensionality increases. To quantify embedding reproducibility, we introduce the Centroid Neighborhood Stability (CNS) metric, which measures the consistency of class-level neighborhood topology across independent runs and complements existing embedding quality measures. Experiments on image and biological datasets demonstrate that ICE produces stable embeddings with improved preservation of class-centroid geometry while remaining computationally efficient. Additional analyses of optimization dynamics, sensitivity, and preprocessing further provide practical insights into latent dimensionality and explicit geometry modeling for supervised dimensionality reduction.

Figures & tables

Figure 1: Illustration of Isometric Centroid-Encoder (ICE). Samples are dispersed around class centroids in 3D, with pairwise distances encoding class geometry. ICE maps data to a 2D space by collapsing samples to centroids while preserving these distances, producing compact classes with retained geometric structure.

Property	Centroid-Encoder (CE)	ICE
Centroid location	Fixed in ambient space	Learned in latent space
Target mapping	$x_{i}\rightarrow c_{j}\in\mathbb{R}^{d}$	$x_{i}\rightarrow\mu_{j}\in\mathbb{R}^{p}$
Centroid update	Pre-computed	Updated during training
Embedding constraint	Not enforced	Intra-class compactness + centroid isometry
Global class geometry	Not enforced	Explicitly preserved
Objective perspective	Centroid reconstruction	Centroid encoding + geometry preservation

Table 2: Comparison between Centroid-Encoder (CE) and the proposed ICE framework.

Figure 2: Optimization dynamics of ICE on the MNIST dataset. (a) Latent centroid trajectories in the two-dimensional embedding space. Circles and squares denote the initial and final centroid locations, respectively. Intermediate points illustrate the centroid evolution during training. (b) Mean epoch-to-epoch centroid drift averaged across all classes. The shaded region indicates one standard deviation across classes.

Figure 3: Optimization dynamics of ICE on the FMNIST dataset. (a) Evolution of the latent class centroids in the two-dimensional embedding space throughout training. (b) Mean epoch-to-epoch centroid drift averaged across all classes. The shaded region denotes one standard deviation across classes.

Figure 4: Evolution of the centroid-consistency loss, centroid-isometry loss, and total objective during ICE optimization on FMNIST and MNIST using direct two-dimensional embeddings and class-dimensional embeddings ( $p=M$ ). The total objective is computed as $\mathcal{L}_{\mathrm{cons}}+\lambda\mathcal{L}_{\mathrm{iso}}$ with $\lambda=0.01$ , while the isometry loss is shown in its unweighted form. The plots show that the isometry loss quickly plateaus in the two-dimensional embedding but continues to decrease in the higher-dimensional latent space, consistent with the additional representational capacity available for preserving centroid geometry.

Figure 5: Representative comparison from the controlled synthetic experiment designed to distinguish Procrustes Disparity and CNS. Two configurations with identical embedding coordinates but different class-label assignments are compared. (a) Reference configuration. (b) Configuration obtained by permuting the labels assigned to entire clusters while keeping all sample coordinates fixed. (c) Corresponding Procrustes Disparity and CNS values. (d) $k=5$ nearest-centroid neighborhood graphs illustrating how class-label permutations alter class-neighborhood relationships despite identical embedding geometry. Procrustes Disparity remains zero because the geometric configuration is unchanged, whereas CNS captures the resulting change in class-level neighborhood topology.

Dataset	#Features	#Classes	#Samples	Domain
USPS	256	10	9298	Image
MNIST	784	10	60000	Image
FMNIST	784	10	60000	Image
10X PBMC	16653	8	4271	RNA-Seq
Mouse Bladder Cells	20670	16	2746	RNA-Seq
Worm Neuron Cells	13488	10	4186	RNA-Seq

Table 3: Descriptions of datasets used for benchmarking.

Panel of Figure 6 — Figure 6: Kruskal Stress and Pearson Correlation as functions of latent dimensionality for USPS, Mouse Bladder, Worm Neuron, and 10xPBMC using the deep ICE architecture (1000, 500, and 125 hidden units). Results are averaged over five independent runs. The red vertical line denotes the number of classes $M$ .

Panel of Figure 7 — Figure 7: Kruskal Stress and Pearson Correlation of class-centroid distance preservation for USPS, Mouse Bladder, Worm Neuron, and 10xPBMC using linear ICE (no hidden layers). The latent dimension is varied from 2 to 26, and results are averaged over five independent runs. The red vertical line indicates the number of classes $M$ .

Panel of Figure 8 — Figure 8: Kruskal Stress and Pearson Correlation of class-centroid distance preservation for USPS, Mouse Bladder, Worm Neuron, and 10xPBMC using shallow ICE (one hidden layer with 500 neurons). The latent dimension ranges from 2 to 26, and results are averaged over five independent runs. The red vertical line denotes the number of classes $M$ .

Figure 9: Normalized eigenspectra of the centered class-centroid matrix for (a) Mouse Bladder (16 cell types) and (b) Worm Neuron (10 cell types). The eigenvalues are shown in descending order after normalization by their total sum. Both datasets exhibit a rapidly decaying eigenspectrum, indicating that the dominant centroid geometry is captured by only a small number of principal directions. This suggests that the effective latent dimensionality required to preserve the class-centroid geometry can be substantially smaller than the theoretical upper bound of $M-1$ . The eigenspectra were computed using Dual PCA to efficiently perform the eigendecomposition in the sample space, avoiding the construction of the high-dimensional feature covariance matrix.

$\lambda$	CNS	Kruskal Stress	5NN Accuracy (%)
0.01	0.8080	$0.2243\pm 0.0011$	$97.27\pm 0.05$
0.1	0.8933	$0.2258\pm 0.0005$	$96.94\pm 0.09$
1.0	0.8400	$0.2253\pm 0.0006$	$96.39\pm 0.20$
10.0	0.7787	$0.2268\pm 0.0035$	$96.48\pm 0.08$
100.0	0.7573	$0.2378\pm 0.0100$	$95.93\pm 0.12$

Table 6: Effect of the regularization parameter

\lambda

on centroid neighborhood stability (CNS), Kruskal Stress, and 5NN accuracy for MNIST in a two-dimensional embedding. Results are averaged over five independent runs.

$\lambda$	CNS	Kruskal Stress	5NN Accuracy (%)
0.01	0.8747	$0.2086\pm 0.0035$	$96.24\pm 0.22$
0.1	0.8160	$0.2164\pm 0.0118$	$96.15\pm 0.31$
1.0	0.8933	$\mathbf{0.2085\pm 0.0036}$	$96.09\pm 0.25$
10.0	0.7893	$0.2147\pm 0.0075$	$96.80\pm 0.14$
100.0	0.8027	$0.2137\pm 0.0088$	$95.28\pm 0.40$

Table 7: Effect of the regularization parameter

\lambda

on USPS. Results are averaged over five independent runs using an

80{:}20

train-test split.

$\lambda$	CNS	Kruskal Stress	5NN Accuracy (%)
0.01	0.8700	$0.1693\pm 0.0128$	$82.61\pm 1.42$
0.1	0.8967	$0.1621\pm 0.0065$	$81.87\pm 2.71$
1.0	0.9100	$0.1655\pm 0.0033$	$80.98\pm 1.27$
10.0	0.9133	$0.1580\pm 0.0065$	$83.24\pm 1.42$
100.0	0.9000	$\mathbf{0.1515\pm 0.0044}$	$78.79\pm 1.87$

Table 8: Effect of the regularization parameter

\lambda

on the 10X PBMC dataset. Results are averaged over five independent runs using an

80{:}20

train-test split.

Dataset	Latent Dim.	Stress	Correlation	CNS	5NN Accuracy (%)
MNIST	$2$	$1.00\pm 0.00$	$-0.00\pm 0.19$	$0.46\pm 0.01$	$14.12\pm 3.68$
MNIST	$10$	$1.00\pm 0.00$	$0.00\pm 0.12$	$0.48\pm 0.01$	$9.91\pm 0.39$
FMNIST	$2$	$1.00\pm 0.00$	$0.13\pm 0.32$	$0.50\pm 0.02$	$20.65\pm 6.13$
FMNIST	$10$	$1.00\pm 0.00$	NA	$0.77\pm 0.04$	$10.00\pm 0.01$

Table 9: Ablation results obtained by removing the centroid-isometry term (

\lambda=0

). Lower Kruskal Stress is better, whereas higher Correlation, CNS, and 5NN accuracy are better.

Dataset	Latent Dim.	Kruskal Stress	Correlation	CNS	5NN Accuracy (%)
MNIST	$2$	$0.23\pm 0.01$	$0.83\pm 0.03$	$0.72\pm 0.02$	$94.26\pm 0.32$
FMNIST	$2$	$0.15\pm 0.02$	$0.94\pm 0.03$	$0.95\pm 0.02$	$84.69\pm 0.41$
10X PBMC	$2$	$0.16\pm 0.01$	$0.94\pm 0.01$	$0.91\pm 0.03$	$85.55\pm 2.10$
Mouse Bladder	$2$	$0.20\pm 0.01$	$0.94\pm 0.00$	$0.77\pm 0.02$	$65.71\pm 1.69$
MNIST	$M$	$0.01\pm 0.00$	$1.00\pm 0.00$	$0.95\pm 0.02$	$97.39\pm 0.10$
FMNIST	$M$	$0.02\pm 0.01$	$1.00\pm 0.00$	$0.99\pm 0.01$	$87.83\pm 0.17$

Table 10: Performance of isometry-only ICE after removing the centroid-consistency term. Results are averaged over 15 independent runs.

$k$	$R$	CNS ranking (higher is better)
1	3	IVIS (0.8667) $>$ ICE (0.7867) $>$ SE (0.7778) $>$ CE (0.6444) $>$ SUMAP (0.6044)
1	5	IVIS (0.8507) $>$ ICE (0.7600) $>$ SE (0.7387) $>$ CE (0.5680) $>$ SUMAP (0.5627)
1	15	IVIS (0.8267) $>$ ICE (0.6853) $>$ SE (0.6836) $>$ SUMAP (0.4996) $>$ CE (0.4844)
3	3	IVIS (0.9111) $>$ SE (0.8281) $>$ ICE (0.7793) $>$ CE (0.7689) $>$ SUMAP (0.7437)
3	5	IVIS (0.8969) $>$ SE (0.7796) $>$ ICE (0.7422) $>$ SUMAP (0.7271) $>$ CE (0.7067)
3	15	IVIS (0.8498) $>$ SE (0.7390) $>$ SUMAP (0.6942) $>$ ICE (0.6833) $>$ CE (0.6361)

Table 11: Sensitivity of CNS to neighborhood size

k

and number of runs

R

for the MNIST two-dimensional embedding. Methods are ranked according to their average CNS for each

(k,R)

configuration.

$k$	$R$	CNS ranking (higher is better)
1	3	ICE (0.9911) $>$ SE (0.9422) $>$ IVIS (0.9289) $>$ CE (0.7733) $>$ SUMAP (0.6222)
1	5	ICE (0.9893) $>$ SE (0.9360) $>$ IVIS (0.9067) $>$ CE (0.7173) $>$ SUMAP (0.5547)
1	15	ICE (0.9852) $>$ SE (0.9262) $>$ IVIS (0.8927) $>$ CE (0.6539) $>$ SUMAP (0.4793)
3	3	ICE (0.9674) $>$ IVIS (0.9111) $>$ SE (0.8993) $>$ CE (0.8267) $>$ SUMAP (0.7748)
3	5	ICE (0.9609) $>$ IVIS (0.8969) $>$ SE (0.8791) $>$ CE (0.7813) $>$ SUMAP (0.7271)
3	15	ICE (0.9495) $>$ IVIS (0.8498) $>$ SE (0.8407) $>$ CE (0.7146) $>$ SUMAP (0.6942)

Table 12: Sensitivity of CNS to neighborhood size

k

and number of runs

R

for the MNIST class-dimensional embedding (

p=M

). Methods are ranked according to their average CNS for each

(k,R)

configuration.

Figure 10: Two-dimensional scatter plots of MNIST test digits across five independent runs for each method.

Figure 11: Two-dimensional scatter plots of MNIST test digits across five independent runs for each method. The 2D embeddings are obtained by applying PCA to the corresponding $M$ -dimensional representations.

Figure 12: Two-dimensional scatter plots of FMNIST fashion item from test set across five independent runs for each method.

Figure 13: Two-dimensional scatter plots of FMNIST fashion item from test set across five independent runs for each method. The 2D embeddings are obtained by applying PCA to the corresponding $M$ -dimensional representations.

Figure 14: Two-dimensional scatter plots of 10XPBMC test samples across five independent runs for each method.

Figure 15: Two-dimensional scatter plots of 10XPBMC test set across five independent runs for each method. The 2D embeddings are obtained by applying PCA to the corresponding $M$ -dimensional representations.

Dimensionality Reduction

Dimensionality reduction maps high-dimensional data to compact representations for visualization, prediction, compression, and scientific modeling.

See all 145 papers

May 22, 2026

MEDAL: Manifold Embedding Distillation via Autoencoder Learning

Low-dimensional embeddings are widely used as visual summaries of high-dimensional data and to enable downstream scientific discoveries. Yet, popular nonlinear dimension reduction methods, such as t-SNE and UMAP, are often selected based on visual appeal alone and without rigorous quantitative validation. A major reason is that manifold embeddings typically do not provide an out-of-sample map nor an inverse back to the original feature space; this makes held-out validation, the gold standard in supervised learning, all but impossible. To address these challenges, we develop a novel framework, MEDAL (Manifold Embedding Distillation via Autoencoder Learning), which distills a fitted manifold embedding into a reusable encoder--decoder model. MEDAL trains a constrained autoencoder whose bottleneck exactly matches any teacher embedding while the decoder reconstructs the original input; this yields an explicit map for new samples, an approximate inverse, and a pointwise reconstruction-based measure of distortion in the manifold space. This converts static manifold embeddings into models that can be evaluated on held-out data, enabling quantitative validation including comparing different dimension reduction methods as well as hyperparameter tuning. Across multiple benchmark and scientific case studies, we show that MEDAL enables held-out validation to determine optimal manifold embeddings and hyperparameters, reveals biologically coherent regions that are difficult to preserve in two dimensional embeddings, and detects distribution shift when new samples are mapped into a fixed reference manifold. MEDAL provides a general validation wrapper to any existing dimension reduction technique that will improve the rigor and reliability of dimension reduction in scientific workflows.

Irene Chang, Tarek M. Zikry, Genevera I. Allen

Department of Statistics, Columbia University, New York, NY 10027 · School of Data and Information Sciences, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599 · Irving Institute for Cancer Dynamics, Columbia University, New York, NY 10027 +1

Autoencoders Manifold Learning Dimensionality Reduction

Ask PDF

Jul 22, 2025

Beyond Correlation: Learning Supervised, Sample-Distinct, and Eigenimage-Interpretable Representations

Conventional dimensionality reduction methods mainly optimize variance or correlation, leaving statistical dependence, data diversity, contrast, and interpretability under addressed. We propose three new independence criteria for designing supervised and unsupervised dimensionality reduction (DR) methods, aiming to improve feature extraction and representation quality. Our framework combines linear and nonlinear formulations and is evaluated using contrast, classification accuracy, and interpretability measures. The interpretability of eigenfaces helps to effectively summarize dominant class-specific structures and trends within representative images. Evaluated on MNIST and a Gender face dataset for classification and reconstruction, our methods achieve significant improvements in contrast (up to

+

20.1%), accuracy (up to

+

17.4%), and interpretability (up to

+

120.0%) over Principal Component Analysis (PCA), t-distributed Stochastic Neighbor Embedding (t-SNE), Linear Discriminant Analysis (LDA), and Variational Autoencoder (VAE) baselines, while also improving VAE reconstruction performance by 9.5%. These results suggest a promising direction for interpretable representation learning based on statistical dependence and independence criteria.

Mojtaba Moattari

Independent Researcher

Representation Learning Dimensionality Reduction

Ask PDF

Aug 15, 2026

Shape Operator PCA: Curvature-Aware Projections for Geometric Machine Learning

In this paper, we propose SHOPCA (Shape Operator-based Principal Component Analysis), a novel method for unsupervised metric learning and dimensionality reduction that incorporates differential geometric information into the covariance structure of classical PCA. SHOPCA regularizes the global covariance matrix using the mean shape operator, defined as the average of the absolute local shape operators estimated from the data manifold, steering principal components toward directions of both maximum variance and informative curvature. A single trace-normalized mixing coefficient

α

controls the regularization, recovering standard PCA at

α= 0

and a curvature-driven embedding as

α\to \infty

. We further introduce a fully unsupervised criterion for selecting

α

based on the spectral eigengap of the regularized covariance matrix, maximizing the relative separation between the top-

d

and remaining eigenvalues without using class labels. We evaluate SHOPCA on more than 50 real-world benchmark datasets, comparing it with PCA, ISOMAP, and UMAP using Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), Fowlkes-Mallows index (FM), and V-measure. Results show that SHOPCA consistently improves clustering quality over PCA across a broad range of datasets and surpasses UMAP on small-sample settings, where iterative neighborhood-based manifold estimation can degrade. SHOPCA is computationally tractable, parameter-efficient, and applicable to domains requiring fully unsupervised, geometry-aware dimensionality reduction.

Alexandre L. M. Levada

Federal University of São Carlos 13565-905, São Carlos-SP, Brazil

Dimensionality Reduction Principal Component Analysis Metric Learning

Ask PDF

Abstract

Dimensionality Reduction

Explore similar work

MEDAL: Manifold Embedding Distillation via Autoencoder Learning

Beyond Correlation: Learning Supervised, Sample-Distinct, and Eigenimage-Interpretable Representations

Shape Operator PCA: Curvature-Aware Projections for Geometric Machine Learning