Dimensionality Reduction

Follow this topic Ask about this topic

Dimensionality reduction turns data with many variables into a smaller representation that is easier to visualize, analyze, store, or use in a predictive model. Linear methods such as PCA and supervised discriminant projections select directions of variation, while t-SNE, UMAP, graph and spectral methods, optimal transport, and autoencoders capture nonlinear structure. Researchers apply these approaches to images, text embeddings, biological data, and dynamical systems, evaluating them through reconstruction, classification, neighborhood, density, and topology preservation. A central tension is that preserving variance or local neighborhoods may discard task-relevant signals or distort global geometry, so methods and evaluations increasingly account for the intended use, uncertainty, and out-of-sample behavior.

Latest papers

145 papers9 papers in the last four weeks, down 36% on the four weeks before. 0.1% of all new papers.

May 9, 2026

Privacy-Aware Video Anomaly Detection through Orthogonal Subspace Projection

Video anomaly detection (VAD) systems often prioritize accuracy while overlooking privacy concerns, limiting their suitability for real-world deployment. We propose the Orthogonal Projection Layer (OPL), a lightweight module that removes task-irrelevant variations to produce representations focused on anomaly-relevant cues. To address privacy risks in human-centered scenarios, we introduce Guided OPL (G-OPL), which suppresses facial attributes using weak supervision from face-presence signals while preserving non-identifying features such as pose and motion. A cosine alignment objective enforces consistent capture and removal of facial information without identity labels or adversarial training. We further present a privacy-aware evaluation framework that jointly assesses detection performance and privacy preservation, and enables analysis of how sensitive information is filtered. Experiments show that embedding privacy constraints into model design reduces sensitive information while maintaining or improving detection accuracy, supporting projection-based architectures as a principled approach for privacy-aware VAD.

Lei Wang, Wenxiang Diao, Andrew Busch +2

School of Engineering and Built Environment, Griffith University · Data61/CSIRO · School of Computer Science and Engineering, University of New South Wales +1

Privacy-Preserving ML Video Anomaly Detection Dimensionality Reduction

May 5, 2026

Capabilities of Auto-encoders and Principal Component Analysis of the Reduction of Microstructural Images; Application on the Acceleration of Phase-Field Simulations

In this work, a data-driven framework based on Phase-Field simulations data is proposed to highlight the capabilities of neural networks to ensure accurate low dimensionality reduction of simulated microstructural images and to provide time-series analysis. The dataset was indeed constructed from high-fidelity Phase-Field simulations. Analyses demonstrated that the association of auto-encoder neural networks and principal component analyses leads to ensure efficient and significant dimensionality reduction: 1/196 of reduction ratio with more than 80% of accuracy. These findings give insight to apply analyses on data from the latent dimension. Application of Long Short Term Memory (LSTM) neural networks showed the possibility of making next frame predictions; that makes possible the acceleration of Phase-Field simulation without the need of high computing resources. We discussed the application of such a framework on various areas of research. Different methods are proposed from the conducted analyses, in order to ensure dimensionality reduction, including auto-encoders, principal component analysis and Artificial Neural Networks, and time-series analysis, including LSTM and Gated Recurrent Unit (GRU).

Seifallah Fetni, Thinh Quy Duc Pham, Truong Vinh Hoang +4

University of Liège, UEE Research Unit, MSM division, allée de la Découverte, 9 B52/3, B 4000 Liège, Belgium · Chair of Mathematics for Uncertainty Quantification, RWTH-Aachen University, 52056 Aachen, Germany · Institute of Strategy Development, Thu Dau Mot University, 75100 Binh Duong Province, Viet Nam

Neural Surrogate Modeling Autoencoders Dimensionality Reduction

May 5, 2026

Provable Accuracy Collapse in Embedding-Based Representations under Dimensionality Mismatch

Embedding-based representations in Euclidean space

\mathbb{R}^d

are a cornerstone of modern machine learning, where a major goal is to use the smallest dimension that faithfully captures data relations. In this work, we prove sharp dimension--accuracy tradeoffs and identify a fundamental information-theoretic limitation: unless the embedding dimension

d

is chosen close to the ground-truth dimension

D

, accuracy undergoes a sudden collapse. Our main result shows that this phenomenon arises even in standard contrastive learning settings, where supervision is limited to a set of

m

anchor--positive--negative triplets

(i,j,k)

encoding distance comparisons

\mathrm{dist}(i,j) < \mathrm{dist}(i,k)

. Specifically, given triplets realizable by an unknown ground-truth embedding in

D

dimensions, we prove that there exists constant

c < 1

, such that \emph{every embedding of dimension at most

cD

violates half of the triplets}, yielding accuracy as low as a trivial one-dimensional solution that ignores the input. We complement our information-theoretic bounds with strong computational hardness results: under the Unique Games Conjecture, even if the given triplets are nearly realizable in

D=1

dimension, no polynomial-time algorithm -- regardless of its dimension -- can achieve accuracy above the trivial

50\%

baseline.

Dionysis Arvanitakis, Vaggos Chatziafratis, Yiyuan Luo

Northwestern University, IL, USA · University of California

Intrinsic Dimensionality Contrastive Learning Dimensionality Reduction Information-Theoretic Lower Bounds

May 3, 2026

DR-SNE: Density-Regularized Stochastic Neighbor Embedding

Dimensionality reduction methods such as t-SNE are designed to preserve local neighborhood structure but do not explicitly account for how probability mass is distributed, often leading to distortions of data density. We reformulate dimensionality reduction as the joint alignment of two components: (i) conditional structure, capturing local relationships, and (ii) relative density structure, captured via local density statistics. Based on this perspective, we introduce Density-Regularized SNE (DR-SNE), which augments the stochastic neighbor embedding objective with a density regularization term derived from normalized log-density estimates. Unlike prior approaches such as DensMAP and DenSNE, which rely on local scale consistency, DR-SNE directly aligns normalized density estimates, providing a simple and scale-invariant mechanism for preserving relative density variations. Empirically, DR-SNE improves density preservation while maintaining competitive neighborhood fidelity, and yields gains on density-sensitive tasks such as anomaly detection across multiple datasets. These results suggest that incorporating density information complements geometry-focused objectives in dimensionality reduction.

Maksim Kazanskii

Independent Researcher

Dimensionality Reduction

May 1, 2026

Bridging Graph Drawing and Dimensionality Reduction with Stochastic Stress Optimization

Both Dimensionality Reduction (DR) and Graph Drawing (GD) aim to visualize abstract, non-linear structures, yet rely on different optimization paradigms. This contrast is evident in Multidimensional Scaling (MDS), which typically depends on the SMACOF algorithm despite graph drawing results showing that simpler stochastic optimization schemes can be more effective for the same objective. We bridge these domains by adapting Stochastic Gradient Descent (SGD) techniques from graph drawing to vector data embedding. We present a scikit-learn compatible estimator that minimizes global stress through local pairwise updates, improving upon the existing implementation. Experiments on standard high-dimensional benchmarks show that our stochastic solver converges substantially faster than SMACOF while achieving comparable or lower stress.

Daniel Hangan, Stephen Kobourov, Jacob Miller

Technical University of Munich

Scientific Visualization Data Visualization Multidimensional Scaling Dimensionality Reduction Stochastic Gradient Descent

May 1, 2026

Class Angular Distortion Index for Dimensionality Reduction

Dimensionality reduction (DR) techniques are often characterized by whether they preserve global, high-level structures in the data or local, neighborhood structures. This distinction matters in visualization: global methods can obscure clusters while local methods can over-emphasize them. Yet, even when clusters appear distinct, their relative arrangement in the projection may be arbitrary or misleading, a common issue in techniques such as t-SNE and UMAP. Existing cluster quality metrics either only measure cluster separability or assume spherical, globular clusters in the original space. We introduce the Class Angular Distortion Index (CADI), a metric that uses internal angles among point triples to determine the faithfulness of cluster organization in a projection. We show cases on both real and synthetic data where existing cluster metrics fail, but CADI provides an interpretable result. Since it relies on computing angles, CADI is also differentiable, enabling optimization. We demonstrate this with a CADI-based DR technique.

Kaviru Gunaratne, Stephen Kobourov, Jacob Miller

Technical University of Munich

Data Visualization Dimensionality Reduction

Apr 28, 2026

Deflation-Free Optimal Scoring

Sparse Optimal Scoring (SOS) reformulates linear discriminant analysis to enable feature selection through elastic net regularization, making it well-suited for high-dimensional settings where the number of features exceeds observations. Most existing SOS methods use deflation-based strategies that compute discriminant vectors sequentially, which can propagate errors and produce suboptimal solutions. We propose a novel approach that estimates all discriminant vectors simultaneously under an explicit global orthogonality constraint, which we call Deflation-Free Sparse Optimal Scoring (DFSOS). DFSOS combines Bregman iteration with orthogonality-constrained optimization, decomposing the problem into tractable subproblems for scoring vectors, discriminant vectors, and orthogonality enforcement. We establish convergence to stationary points of the augmented Lagrangian under mild conditions. Extensive experiments using synthetic data and real-world time series data demonstrate that DFSOS achieves classification accuracy comparable to or better than existing deflation-based methods. These results indicate that deflation-free approaches offer a robust and effective framework for sparse discriminant analysis in high-dimensional problems.

Sharmin Afroz, Brendan Ames

Department of Computational & Informational Sciences, Stillman College, Tuscaloosa, AL, USA · School of Mathematical Sciences, University of Southampton, Southampton, UK

Orthogonal Optimization Constrained Optimization Dimensionality Reduction

Apr 28, 2026

DiRe-RAPIDS: Topology-faithful dimensionality reduction at scale

Dimensionality reduction methods such as UMAP and t-SNE are central tools for visualising high-dimensional data, but their local-neighborhood objectives can preserve sampling noise while distorting global topology. We show that standard local metrics reward this noise memorisation: top-performing embeddings invent cycles and disconnected islands absent from the data. We introduce a topology-faithfulness benchmark based on noisy manifolds with known homology, tune DiRe against it, and find Pareto-optimal configurations that match or beat GPU-accelerated UMAP on classification while recovering exact first Betti numbers on stress tests. On 723K arXiv paper embeddings, DiRe preserves 3-4 times more topological structure than UMAP at comparable wall-clock.

Alexander Kolpakov, Igor Rivin

CSTEM, University of Austin, Austin, TX, USA · Department of Mathematics, Temple University, Philadelphia, PA, USA

Data Visualization Manifold Learning Dimensionality Reduction Topological Representation Learning Topological Mapping Topological Data Analysis

Apr 27, 2026

Diffusion-Guided Feature Selection via Nishimori Temperature: Noise-Based Spectral Embedding

We propose Noise-Based Spectral Embedding (NBSE), a physics-informed framework for selecting informative features from high-dimensional data without greedy search. NBSE constructs a sparse similarity graph on the samples and identifies the Nishimori temperature

β_N

the critical inverse temperature at which the Bethe Hessian becomes singular. The corresponding smallest eigenvector captures the dominant mode of an intrinsically degree-corrected diffusion process, naturally reweighting nodes to prevent hub dominance. By transposing the data matrix and applying NBSE in feature space, we obtain a one-dimensional spectral embedding that reveals groups of redundant or semantically related dimensions; balanced binning then selects one representative per group. We prove that coloured Gaussian perturbations shift

β_N

by at most

O(\barσ^2)

, guaranteeing robustness to measurement noise. Experiments on ImageNet embeddings from MobileNetV2 and EfficientNet-B4 show that NBSE preserves classification accuracy even under aggressive compression: on EfficientNet-B4 the accuracy drop is below

1\%

when retaining only

30\%

of features, outperforming ANOVA

F

-test and random selection by up to

6.8\%

.

Vasiliy S. Usatyuk, Denis A. Sapozhnikov, Sergey I. Egorov

R&D Department T8 LLC Moscow, Russia · Department of Computer Science South-West State University Kursk, Russia

Feature Selection Dimensionality Reduction Spectral Methods Spectral Graph Embedding

Apr 26, 2026

Gromov-Wasserstein Methods for Multi-View Relational Embedding and Clustering

Learning low-dimensional representations from multi-view relational data is challenging when underlying geometries differ across views. We propose Bary-GWMDS, a Gromov-Wasserstein-based method that operates directly on distance matrices to learn a consensus embedding preserving shared relational structure. By leveraging intrinsic distances, the approach naturally handles nonlinear distortions across views. We also introduce Mean-GWMDS-C, a clustering-oriented formulation that averages distance matrices and learns reduced-support representations via a consensus Gromov-Wasserstein transport. Experiments on synthetic and real-world datasets show that the proposed framework yields stable and geometrically meaningful embeddings.

Rafael Pereira Eufrazio, Eduardo Fernandes Montesuma, Charles Casimiro Cavalcante

Instituto Federal de Educação, Ciência e Tecnologia do Ceará, Canindé, Ceará, Brazil · Federal University of Ceara, Fortaleza, Ceará, Brazil · Sigma Nova, Paris, France

Representation Learning Multidimensional Scaling Multi-View Learning Multi-View Clustering Dimensionality Reduction Optimal Transport

Apr 23, 2026

Assessing the impact of dimensionality reduction on clustering performance - a systematic study

Dimensionality reduction is a critical preprocessing step for clustering high-dimensional data, yet comprehensive evaluation of its impact across diverse methods and data types remains limited. In this study, we systematically assess the influence of five dimensionality reduction techniques - Principal Component Analysis (PCA), Kernel Principal Component Analysis (Kernel PCA), Variational Autoencoder (VAE), Isometric Mapping (Isomap), and Multidimensional Scaling (MDS) - on the performance of four popular clustering algorithms - k-means, Agglomerative Hierarchical Clustering (AHC), Gaussian Mixture Models (GMM), and Ordering Points to Identify the Clustering Structure (OPTICS). We evaluate clustering quality using the Adjusted Rand Index (ARI), comparing results without and with dimensionality reduction at different reduction levels recommended in the literature (i.e., k-1, where k is the number of clusters, and 25% and 50% of the original number of dimensions). Our findings underscore the importance of a careful selection of the dimensionality reduction technique and the dimensionality reduction level that should be tailored to intrinsic data geometry and clustering algorithm(s) under consideration.

Ousmane Assani Amate, Elyes Lounissi, Mohammadreza Bakhtyari +3

Université du Québec à Montréal, Montreal, Quebec, Canada · Mila - Quebec AI Institute, Montreal, Quebec, Canada

Principal Component Analysis Clustering Dimensionality Reduction

Apr 23, 2026

Local Neighborhood Instability in Parametric Projections: Quantitative and Visual Analysis

Parametric projections let analysts embed new points in real time, but input variations from measurement noise or data drift can produce unpredictable shifts in the 2D layout. Whether and where a projection is locally stable remains largely unexamined. In this paper, we present a stability evaluation framework that probes parametric projections with Gaussian perturbations around selected anchor points and assesses how neighborhoods deform in the 2D embedding. Our approach combines quantitative measures of mean displacement, bias, and nearest-anchor assignment error with per-anchor visualizations of displacement vectors, local PCA ellipsoids, and Voronoi misassignment for detailed inspection. We demonstrate the framework's effectiveness on UMAP- and t-SNE-based neural projectors of varying network sizes and study the effect of Jacobian regularization as a gradient-based robustness strategy. We apply our framework to the MNIST and Fashion-MNIST datasets. The results show that our framework identifies unstable projection regions invisible to reconstruction error or neighborhood-preservation metrics.

Frederik L. Dennig, Daniel A. Keim

University of Konstanz, Germany

Distribution Shift Robustness Jacobian Regularization Dimensionality Reduction

Apr 20, 2026

Curvature-Aware PCA with Geodesic Tangent Space Aggregation for Semi-Supervised Learning

Principal Component Analysis (PCA) is a fundamental tool for representation learning, but its global linear formulation fails to capture the structure of data supported on curved manifolds. In contrast, manifold learning methods model nonlinearity but often sacrifice the spectral structure and stability of PCA. We propose Geodesic Tangent Space Aggregation PCA (GTSA-PCA), a geometric extension of PCA that integrates curvature awareness and geodesic consistency within a unified spectral framework. Our approach replaces the global covariance operator with curvature-weighted local covariance operators defined over a

k

-nearest neighbor graph, yielding local tangent subspaces that adapt to the manifold while suppressing high-curvature distortions. We then introduce a geodesic alignment operator that combines intrinsic graph distances with subspace affinities to globally synchronize these local representations. The resulting operator admits a spectral decomposition whose leading components define a geometry-aware embedding. We further incorporate semi-supervised information to guide the alignment, improving discriminative structure with minimal supervision. Experiments on real datasets show consistent improvements over PCA, Kernel PCA, Supervised PCA and strong graph-based baselines such as UMAP, particularly in small sample size and high-curvature regimes. Our results position GTSA-PCA as a principled bridge between statistical and geometric approaches to dimensionality reduction.

Alexandre L. M. Levada

Computing Department Federal University of São Carlos 13565-905, São Carlos-SP, Brazil

Principal Component Analysis Manifold Learning Dimensionality Reduction

Apr 18, 2026

Neighbor Embedding for High-Dimensional Sparse Poisson Data

Across many scientific fields, measurements often represent the number of times an event occurs. For example, a document can be represented by word occurrence counts, neural activity by spike counts per time window, or online communication by daily email counts. These measurements yield high-dimensional count data that often approximate a Poisson distribution, frequently with low rates that produce substantial sparsity and complicate downstream analysis. A useful approach is to embed the data into a low-dimensional space that preserves meaningful structure, commonly termed dimensionality reduction. Yet existing dimensionality reduction methods, including both linear (e.g., PCA) and nonlinear approaches (e.g., t-SNE), often assume continuous Euclidean geometry, thereby misaligning with the discrete, sparse nature of low-rate count data. Here, we propose p-SNE (Poisson Stochastic Neighbor Embedding), a nonlinear neighbor embedding method designed around the Poisson structure of count data, using KL divergence between Poisson distributions to measure pairwise dissimilarity and Hellinger distance to optimize the embedding. We test p-SNE on synthetic Poisson data and demonstrate its ability to recover meaningful structure in real-world count datasets, including weekday patterns in email communication, research area clusters in OpenReview papers, and temporal drift and stimulus gradients in neural spike recordings.

Noga Mudrik, Adam S. Charles

Department of Biomedical Engineering Center for Imaging Science Kavli Neuroscience Discovery Institute Johns Hopkins University Baltimore, MD 21218, USA

Manifold Learning Dimensionality Reduction

Apr 15, 2026

Joint Representation Learning and Clustering via Gradient-Based Manifold Optimization

Clustering and dimensionality reduction have been crucial topics in machine learning and computer vision. Clustering high-dimensional data has been challenging for a long time due to the curse of dimensionality. For that reason, a more promising direction is the joint learning of dimension reduction and clustering. In this work, we propose a Manifold Learning Framework that learns dimensionality reduction and clustering simultaneously. The proposed framework is able to jointly learn the parameters of a dimension reduction technique (e.g. linear projection or a neural network) and cluster the data based on the resulting features (e.g. under a Gaussian Mixture Model framework). The framework searches for the dimension reduction parameters and the optimal clusters by traversing a manifold,using Gradient Manifold Optimization. The obtained The proposed framework is exemplified with a Gaussian Mixture Model as one simple but efficient example, in a process that is somehow similar to unsupervised Linear Discriminant Analysis (LDA). We apply the proposed method to the unsupervised training of simulated data as well as a benchmark image dataset (i.e. MNIST). The experimental results indicate that our algorithm has better performance than popular clustering algorithms from the literature.

Sida Liu, Yangzi Guo, Mingyuan Wang

Department of Statistics, Florida State University, Tallahassee, Florida, USA · Department of Mathematics, Florida State University, Tallahassee, Florida, USA

Unsupervised Clustering Dimensionality Reduction

Apr 14, 2026

Some Theoretical Limitations of t-SNE

t-SNE has gained popularity as a dimension reduction technique, especially for visualizing data. It is well-known that all dimension reduction techniques may lose important features of the data. We provide a mathematical framework for understanding this loss for t-SNE by establishing a number of results in different scenarios showing how important features of data are lost by using t-SNE.

Rupert Li, Elchanan Mossel

Dimensionality Reduction Data Visualization

Apr 14, 2026

Does Dimensionality Reduction via Random Projections Preserve Landscape Features?

Exploratory Landscape Analysis (ELA) provides numerical features for characterizing black-box optimization problems. In high-dimensional settings, however, ELA suffers from sparsity effects, high estimator variance, and the prohibitive cost of computing several feature classes. Dimensionality reduction has therefore been proposed as a way to make ELA applicable in such settings, but it remains unclear whether features computed in reduced spaces still reflect intrinsic properties of the original landscape. In this work, we investigate the robustness of ELA features under dimensionality reduction via Random Gaussian Embeddings (RGEs). Starting from the same sampled points and objective values, we compute ELA features in projected spaces and compare them to those obtained in the original search space across multiple sample budgets and embedding dimensions. Our results show that linear random projections often alter the geometric and topological structure relevant to ELA, yielding feature values that are no longer representative of the original problem. While a small subset of features remains comparatively stable, most are highly sensitive to the embedding. Moreover, robustness under projection does not necessarily imply informativeness, as apparently robust features may still reflect projection-induced artifacts rather than intrinsic landscape characteristics.

Iván Olarte Rodríguez, Anja Jankovic, Thomas Bäck +1

Leiden Institute of Advanced Computer Science, Leiden University · Leiden, The Netherlands · RTWH Aachen University +1

Black-Box Optimization Randomized Sketching Dimensionality Reduction Fitness Landscape Analysis

Apr 13, 2026

On the continuum limit of t-SNE for data visualization

This work is concerned with the continuum limit of a graph-based data visualization technique called the t-Distributed Stochastic Neighbor Embedding (t-SNE), which is widely used for visualizing data in a variety of applications, but is still poorly understood from a theoretical standpoint. The t-SNE algorithm produces visualizations by minimizing the Kullback-Leibler divergence between similarity matrices representing the high dimensional data and its low dimensional representation. We prove that as the number of data points

n \to \infty

, after a natural rescaling and in applicable parameter regimes, the Kullback-Leibler divergence is consistent as the number of data points

n \to \infty

and the similarity graph remains sparse with a continuum variational problem that involves a non-convex gradient regularization term and a penalty on the magnitude of the probability density function in the visualization space. These two terms represent the continuum limits of the attraction and repulsion forces in the t-SNE algorithm. Due to the lack of convexity in the continuum variational problem, the question of well-posedeness is only partially resolved. We show that when both dimensions are

1

, the problem admits a unique smooth minimizer, along with an infinite number of discontinuous minimizers (interpreted in a relaxed sense). This aligns well with the empirically observed ability of t-SNE to separate data in seemingly arbitrary ways in the visualization. The energy is also very closely related to the famously ill-posed Perona-Malik equation, which is used for denoising and simplifying images. We present numerical results validating the continuum limit, provide some preliminary results about the delicate nature of the limiting energetic problem in higher dimensions, and highlight several problems for future work.

Jeff Calder, Zhonggan Huang, Ryan Murray +1

School of Mathematics, University of Minnesota · Department of Mathematics, University of Utah · Department of Mathematics, North Carolina State University

Data Visualization Dimensionality Reduction

Apr 13, 2026

Classification of Epileptic iEEG using Topological Machine Learning

Epileptic seizure detection from EEG signals remains challenging due to the high dimensionality and nonlinear, potentially stochastic, dynamics of neural activity. In this work, we investigate whether features derived from topological data analysis (TDA) can improve the classification of brain states in preictal, ictal and interictal iEEG recordings from epilepsy patients using multichannel data. We analyze data from 55 patients, significantly larger than many previous studies that rely on patient-specific models. Persistence diagrams derived from iEEG signals are vectorized using several TDA representations, including Carlsson coordinates, persistence images, and template functions. To understand how topological representations interact with modern machine learning pipelines, we conduct a large-scale ablation study across multiple iEEG frequency bands, dimensionality reduction techniques, feature representations, and classifier architectures. Our experiments show that dimension-reduced topological representations achieve up to 80% balanced accuracy for three-class classification. Interestingly, classical machine learning models perform comparably to deep learning models, achieving up to 79.17% balanced accuracy, suggesting that carefully designed topological features can substantially reduce model complexity requirements. In contrast, pipelines preserving the full multichannel feature structure exhibit severe overfitting due to the high-dimensional feature space. These findings highlight the importance of structure-preserving dimensionality reduction when applying topology-based representations to multichannel neural data.

Sunia Tanweer, Narayan Puthanmadam Subramaniyam, Firas A. Khasawneh

Department of Mechanical Engineering, Michigan State University, East Lansing, 48823, MI, USA. · Department of Computational Mathematics, Science and Engineering, Michigan State University, East Lansing, 48823, MI, USA. · Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland. +1

EEG Seizure Detection Topological Data Analysis Multivariate Time Series Classification Topological Representation Learning Dimensionality Reduction

Apr 11, 2026

Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models

With the emergence of 3D foundation models, there is growing interest in fine-tuning them for downstream tasks, where LoRA is the dominant fine-tuning paradigm. As 3D datasets exhibit distinct variations in texture, geometry, camera motion, and lighting, there are interesting fundamental questions: 1) Are there LoRA subspaces associated with each type of variation? 2) Are these subspaces disentangled (i.e., orthogonal to each other)? 3) How do we compute them effectively? This paper provides answers to all these questions. We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation. We show that these subspaces are approximately disentangled. Integrating them leads to a reduced LoRA subspace that enables efficient LoRA fine-tuning with improved prediction accuracy for downstream tasks. In particular, we show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets. An ablation study validates the effectiveness of the choices in our approach.

Yu Jiang, Hanwen Jiang, Ahmed Abdelkader +4

The University of Texas at Austin · Shanghai Jiao Tong University · Adobe Research +1

Parameter-Efficient Fine-Tuning Low-Rank Adaptation 3D Foundation Models Dimensionality Reduction

Apr 8, 2026

Bi-Lipschitz Autoencoder With Injectivity Guarantee

Autoencoders are widely used for dimensionality reduction, based on the assumption that high-dimensional data lies on low-dimensional manifolds. Regularized autoencoders aim to preserve manifold geometry during dimensionality reduction, but existing approaches often suffer from non-injective mappings and overly rigid constraints that limit their effectiveness and robustness. In this work, we identify encoder non-injectivity as a core bottleneck that leads to poor convergence and distorted latent representations. To ensure robustness across data distributions, we formalize the concept of admissible regularization and provide sufficient conditions for its satisfaction. In this work, we propose the Bi-Lipschitz Autoencoder (BLAE), which introduces two key innovations: (1) an injective regularization scheme based on a separation criterion to eliminate pathological local minima, and (2) a bi-Lipschitz relaxation that preserves geometry and exhibits robustness to data distribution drift. Empirical results on diverse datasets show that BLAE consistently outperforms existing methods in preserving manifold structure while remaining resilient to sampling sparsity and distribution shifts. Code is available at https://github.com/qipengz/BLAE.

Qipeng Zhan, Zhuoping Zhou, Zexuan Wang +2

University of Pennsylvania

Autoencoders Representation Learning Dimensionality Reduction Manifold Learning

Apr 5, 2026

Supervised Dimensionality Reduction Revisited: Why LDA on Frozen CNN Features Deserves a Second Look

Frozen pretrained image representations are widely used for transfer learning: a backbone is kept fixed, feature vectors are extracted, and a lightweight classifier is trained on top. This pipeline usually feeds the full feature vector to the classifier, even when the target task has far fewer classes than the pretraining task. We revisit a classical alternative: supervised dimensionality reduction with Linear Discriminant Analysis (LDA) before linear probing. We evaluate ten dimensionality-reduction strategies on frozen features from six backbones -- ResNet-18, ResNet-50, MobileNetV3-Small, EfficientNet-B0, ViT-B/16, and DINOv2-ViT-S/14 -- across CIFAR-100, Tiny ImageNet, and CUB-200-2011. Under a fixed logistic-regression protocol, LDA improves accuracy over full features in 11 of 12 coarse-grained configurations, with gains up to 4.5 percentage points while reducing feature dimensionality by 48-87%. The same projection consistently hurts on fine-grained CUB-200, where full features win across all six backbones. This establishes a practical boundary condition: LDA is useful when class-level structure is coarse enough to be captured by mean-separating directions, but it can discard subtle cues needed for fine-grained recognition. We also compare LDA with PCA, PCA+LDA, regularized LDA, Local Fisher Discriminant Analysis, Neighbourhood Components Analysis, and three lightweight LDA extensions. The results show that plain LDA offers the best accuracy-cost tradeoff for most coarse-grained settings, while more complex supervised reduction methods rarely justify their additional cost. Overall, the study provides concrete guidance for when post-hoc supervised projection should, and should not, be inserted into frozen-feature image classification pipelines.

Indar Kumar, Girish Karhana, Sai Krishna Jasti +1

independent researchers

Dimensionality Reduction Transfer Learning

Apr 3, 2026

Structure-Preserving Multi-View Embedding Using Gromov-Wasserstein Optimal Transport

Multi-view data analysis seeks to integrate multiple representations of the same samples in order to recover a coherent low-dimensional structure. Classical approaches often rely on feature concatenation or explicit alignment assumptions, which become restrictive under heterogeneous geometries or nonlinear distortions. In this work, we propose two geometry-aware multi-view embedding strategies grounded in Gromov-Wasserstein (GW) optimal transport. The first, termed Mean-GWMDS, aggregates view-specific relational information by averaging distance matrices and applying GW-based multidimensional scaling to obtain a representative embedding. The second strategy, referred to as Multi-GWMDS, adopts a selection-based paradigm in which multiple geometry-consistent candidate embeddings are generated via GW-based alignment and a representative embedding is selected. Experiments on synthetic manifolds and real-world datasets show that the proposed methods effectively preserve intrinsic relational structure across views. These results highlight GW-based approaches as a flexible and principled framework for multi-view representation learning.

Rafael Pereira Eufrazio, Eduardo Fernandes Montesuma, Charles Casimiro Cavalcante

Instituto Federal de Educação, Ciência e Tecnologia do Ceará, Canindé, Ceará, Brazil · Department of Teleinformatics Engineering, Federal University of Ceará, Fortaleza, Ceará, Brazil · Sigma Nova, Paris, France

Multidimensional Scaling Multi-View Learning Dimensionality Reduction Multi-View Geometry Optimal Transport

Apr 2, 2026

A Spectral Decomposition Framework for Multiscale Nonlinear Dimensionality Reduction

Dimensionality reduction (DR) involves two longstanding trade-offs. First, preserving local neighborhoods can come at the cost of global structure. Neighbor embedding methods such as t-SNE and UMAP prioritize local similarity preservation but do not explicitly constrain global organization, whereas standard spectral methods such as Laplacian Eigenmaps capture smooth, coarse-scale graph structure but offer limited flexibility to depict finer local structure. Second, the flexibility of nonlinear DR methods often comes at the cost of analytical transparency. Many methods do not explicitly reveal how high-dimensional structure produces patterns in the embedding. We introduce SDMP (Spectral Decomposition for Multiscale Projection), a nonlinear DR framework built on an explicit spectral decomposition. In this formulation, each embedding dimension is expressed as a weighted combination of Laplacian eigenvectors derived from a neighborhood graph, with the weights learned via a UMAP-style cross-entropy objective. By progressively expanding the spectral subspace to capture increasingly fine graph structure, SDMP produces a sequence of embeddings, making the evolving balance between global organization and local detail explicit, controllable, and inspectable. The explicit decomposition also reveals which spectral scales shape the overall embedding and how individual eigenvectors influence point positions. Quantitative evaluations on synthetic, image, and single-cell data show competitive local and global structure preservation, while case studies illustrate how the decomposition supports interpretation of clusters and developmental trajectories across spectral scales.

Zeyang Huang, Angelos Chatzimparmpas, Thomas Höllt +1

Linköping University, Sweden · Utrecht University, The Netherlands · University of Bergen, Norway +1

Dimensionality Reduction Spectral Methods

Apr 1, 2026

The Rashomon Effect for Visualizing High-Dimensional Data

Dimension reduction (DR) is inherently non-unique: multiple embeddings can preserve the structure of high-dimensional data equally well while differing in layout or geometry. In this paper, we formally define the Rashomon set for DR -- the collection of `good' embedding -- and show how embracing this multiplicity leads to more powerful and trustworthy representations. Specifically, we pursue three goals. First, we introduce PCA-informed alignment to steer embeddings toward principal components, making axes interpretable without distorting local neighborhoods. Second, we design concept-alignment regularization that aligns an embedding dimension with external knowledge, such as class labels or user-defined concepts. Third, we propose a method to extract common knowledge across the Rashomon set by identifying trustworthy and persistent nearest-neighbor relationships, which we use to construct refined embeddings with improved local structure while preserving global relationships. By moving beyond a single embedding and leveraging the Rashomon set, we provide a flexible framework for building interpretable, robust, and goal-aligned visualizations.

Yiyang Sun, Haiyang Huang, Gaurav Rajesh Parikh +1

Duke University

Dimensionality Reduction

Mar 30, 2026

Nonlinear Factor Decomposition via Kolmogorov-Arnold Networks: A Spectral Approach to Asset Return Analysis

KAN-PCA is an autoencoder that uses a KAN as encoder and a linear map as decoder. It generalizes classical PCA by replacing linear projections with learned B-spline functions on each edge. The motivation is to capture more variance than classical PCA, which becomes inefficient during market crises when the linear assumption breaks down and correlations between assets change dramatically. We prove that if the spline activations are forced to be linear, KAN-PCA yields exactly the same results as classical PCA, establishing PCA as a special case. Experiments on 20 S&P 500 stocks (2015-2024) show that KAN-PCA achieves a reconstruction R^2 of 66.57%, compared to 62.99% for classical PCA with the same 3 factors, while matching PCA out-of-sample after correcting for data leakage in the training procedure.

David Breazu

Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania

Dimensionality Reduction Principal Component Analysis Financial Time Series Kolmogorov-Arnold Networks

Mar 30, 2026

Diffusion Maps is not Dimensionality Reduction

Diffusion maps (DMAP) are often used as a dimensionality-reduction tool, but more precisely they provide a spectral representation of the intrinsic geometry rather than a complete charting method. To illustrate this distinction, we study a Swiss roll with known isometric coordinates and compare DMAP, Isomap, and UMAP across latent dimensions. For each representation, we fit an oracle affine readout to the ground-truth chart and measure reconstruction error. Isomap most efficiently recovers the low-dimensional chart, UMAP provides an intermediate tradeoff, and DMAP becomes accurate only after combining multiple diffusion modes. Thus the correct chart lies in the span of diffusion coordinates, but standard DMAP do not by themselves identify the appropriate combination.

Julio Candanedo, Alejandro Patiño

SparseTrace.ai. Appleton, WI, USA. · Universidad Nacional de Colombia sede Manizales. Manizales, Caldas, Colombia

Manifold Learning Dimensionality Reduction

Mar 24, 2026

Learning Actuator-Aware Spectral Submanifolds for Precise Control of Continuum Robots

Continuum robots exhibit high-dimensional, nonlinear dynamics which are often coupled with their actuation mechanism. Spectral submanifold (SSM) reduction has emerged as a leading method for reducing high-dimensional nonlinear dynamical systems to low-dimensional invariant manifolds. Our proposed control-augmented SSMs (caSSMs) extend this methodology by explicitly incorporating control inputs into the state representation, enabling these models to capture nonlinear state-input couplings. Training these models relies solely on controlled decay trajectories of the actuator-augmented state, thereby removing the additional actuation-calibration step commonly needed by prior SSM-for-control methods. We learn a compact caSSM model for a tendon-driven trunk robot, enabling real-time control and reducing open-loop prediction error by 40% compared to existing methods. In closed-loop experiments with model predictive control (MPC), caSSM reduces tracking error by 52%, demonstrating improved performance against Koopman and SSM based MPC and practical deployability on hardware continuum robots.

Paul Leonard Wolff, Hugo Buurmeijer, Luis Pabon +6

Autonomous Systems Lab, Stanford University, USA. · Soft Robotics Lab, ETH Zurich, Switzerland. · Institute of Mechanical Systems, ETH Zurich, Switzerland. +1

Continuum Robot Control Robot Dynamics Learning Dimensionality Reduction

Mar 19, 2026

Spectral Tempering for Embedding Compression in Dense Passage Retrieval

Dimensionality reduction is critical for deploying dense retrieval systems at scale, yet mainstream post-hoc methods face a fundamental trade-off: principal component analysis (PCA) preserves dominant variance but underutilizes representational capacity, while whitening enforces isotropy at the cost of amplifying noise in the heavy-tailed eigenspectrum of retrieval embeddings. Intermediate spectral scaling methods unify these extremes by reweighting dimensions with a power coefficient

γ

, but treat

γ

as a fixed hyperparameter that requires task-specific tuning. We show that the optimal scaling strength

γ

is not a global constant: it varies systematically with target dimensionality

k

and is governed by the signal-to-noise ratio (SNR) of the retained subspace. Based on this insight, we propose Spectral Tempering (SpecTemp), a learning-free method that derives an adaptive

γ(k)

directly from the corpus eigenspectrum using local SNR analysis and knee-point normalization, requiring no labeled data or validation-based search. Extensive experiments demonstrate that Spectral Tempering consistently achieves near-oracle performance relative to grid-searched

γ^*(k)

while remaining fully learning-free and model-agnostic. Our code is publicly available at https://github.com/liyongkang123/SpecTemp.

Yongkang Li, Panagiotis Eustratiadis, Evangelos Kanoulas

University of Amsterdam, Amsterdam, The Netherlands

Embedding Compression Dimensionality Reduction Dense Retrieval

Mar 17, 2026

Manifold-Matching Autoencoders

We study a simple unsupervised regularization scheme for autoencoders called Manifold-Matching (MMAE): we align the pairwise distances in the latent space to those of the input data space by minimizing mean squared error. Because alignment occurs on pairwise distances rather than coordinates, it can also be extended to a lower-dimensional representation of the data, adding flexibility to the method. We find that this regularization outperforms similar methods on metrics based on preservation of nearest-neighbor distances and persistent homology-based measures. We also observe that MMAE provides a scalable approximation of Multi-Dimensional Scaling (MDS).

Laurent Cheret, Vincent Létourneau, Isar Nejadgholi +3

Department of Computer Science, University of Ottawa, Ottawa, Canada · MILA, Montréal, Canada · National Research Council of Canada, Ottawa, Canada

Multidimensional Scaling Autoencoders Latent-Space Regularization Dimensionality Reduction Latent Space Geometry Latent Space Alignment+2