Overview

Dimensionality reduction turns data with many variables into a smaller representation that is easier to visualize, analyze, store, or use in a predictive model. Linear methods such as PCA and supervised discriminant projections select directions of variation, while t-SNE, UMAP, graph and spectral methods, optimal transport, and autoencoders capture nonlinear structure. Researchers apply these approaches to images, text embeddings, biological data, and dynamical systems, evaluating them through reconstruction, classification, neighborhood, density, and topology preservation. A central tension is that preserving variance or local neighborhoods may discard task-relevant signals or distort global geometry, so methods and evaluations increasingly account for the intended use, uncertainty, and out-of-sample behavior.

Latest papers

159 papers16 papers in the last four weeks, up 23% on the four weeks before. 0.1% of all new papers.

Dec 15, 2025

Verification of Unknown Dynamical Systems via Autoencoder Latent Space

Formal verification provides a powerful framework for proving that dynamical systems satisfy their specifications. However, these techniques face scalability challenges in high-dimensional settings, as they often rely on state-space discretization which grows exponentially with dimension. Learning-based approaches to dimensionality reduction have shown great potential to alleviate this problem. However, ensuring correctness of latent space verification results remains an open question. In this work, we provide a formal approach to reduce the dimensionality of systems via convex autoencoders and learn the dynamics in the latent space through a kernel-based method. We then construct a finite abstraction from the learned model in the latent space and guarantee that the abstraction contains the true behaviors of the original system. We show that the verification results in the latent space can be mapped back to the original system. Finally, we demonstrate the approach on multiple systems, including a 26D system controlled by a neural network, showing significant scalability improvements.
Oct 17, 2025

Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

Bayesian optimisation (BO) enables sample-efficient global optimisation of expensive black-box functions but remains challenging in high dimensions. We investigate nonlinear dimensionality reduction to a sequence of low-dimensional latent-space BO (LSBO) problems. Early LSBO used linear random and supervised embeddings; building on Grosnit et al., we employ variational autoencoders (VAEs), deep metric loss for structured latent manifolds, and retraining to adapt the encoder-decoder pair to newly sampled regions. We couple LSBO with sequential domain reduction (SDR) directly in latent space (SDR-LSBO), narrowing search domains as evidence accumulates. Implemented in GPU-accelerated BoTorch with Mat'ern-5/2 Gaussian-process surrogates, our methods improve benchmark optimisation quality, and retraining can enhance BO performance. Comparisons with adaptive supervised linear random embeddings demonstrate the effectiveness of VAE-based BO for nonlinear low-dimensional structures. We analyse BO-VAE with a fixed pretrained representation, decomposing ambient-space simple regret into latent BO error and a fixed VAE-induced representation gap. Under a PAC-Bayes-certified reconstruction condition and standard fixed-prior assumptions for expected improvement with a Mat'ern-5/2 kernel, latent BO error vanishes as the evaluation budget increases, whereas the representation gap remains fixed and may impose a non-vanishing error floor. Visualisations empirically assess accessibility of the ambient optimum through the learned decoder. To our knowledge, this is the first study combining SDR with VAE-based LSBO. Our analysis clarifies metric shaping and retraining choices critical for scalable latent-space BO. For reproducibility, source code is available at https://github.com/L-Lok/Nonlinear-Dimensionality-Reduction-Techniques-for-Bayesian-Optimization.git.
Aug 16, 2025

DE-VAE: Revealing Uncertainty in Parametric and Inverse Projections with Variational Autoencoders using Differential Entropy

Recently, autoencoders (AEs) have gained interest for creating parametric and invertible projections of multidimensional data. Parametric projections make it possible to embed new, unseen samples without recalculating the entire projection, while invertible projections allow the synthesis of new data instances. However, existing methods perform poorly when dealing with out-of-distribution samples in either the data or embedding space. Thus, we propose DE-VAE, an uncertainty-aware variational AE using differential entropy (DE) to improve the learned parametric and invertible projections. Given a fixed projection, we train DE-VAE to learn a mapping into 2D space and an inverse mapping back to the original space. We conduct quantitative and qualitative evaluations on four well-known datasets, using UMAP and t-SNE as baseline projection methods. Our findings show that DE-VAE can create parametric and inverse projections with comparable accuracy to other current AE-based approaches while enabling the analysis of embedding uncertainty.
Jul 22, 2025

Beyond Correlation: Learning Supervised, Sample-Distinct, and Eigenimage-Interpretable Representations

Conventional dimensionality reduction methods mainly optimize variance or correlation, leaving statistical dependence, data diversity, contrast, and interpretability under addressed. We propose three new independence criteria for designing supervised and unsupervised dimensionality reduction (DR) methods, aiming to improve feature extraction and representation quality. Our framework combines linear and nonlinear formulations and is evaluated using contrast, classification accuracy, and interpretability measures. The interpretability of eigenfaces helps to effectively summarize dominant class-specific structures and trends within representative images. Evaluated on MNIST and a Gender face dataset for classification and reconstruction, our methods achieve significant improvements in contrast (up to ++20.1%), accuracy (up to ++17.4%), and interpretability (up to ++120.0%) over Principal Component Analysis (PCA), t-distributed Stochastic Neighbor Embedding (t-SNE), Linear Discriminant Analysis (LDA), and Variational Autoencoder (VAE) baselines, while also improving VAE reconstruction performance by 9.5%. These results suggest a promising direction for interpretable representation learning based on statistical dependence and independence criteria.
May 30, 2025

Equilibrium Distribution for t-Distributed Stochastic Neighbor Embedding with Generalized Kernels

We extend the results by Auffinger and Fletcher in 2023 on the large-sample limit of t-SNE to a broader class of input and output kernels. We prove that under natural assumptions on the input distribution and the decay of the output kernel, if perplexity scales linearly with data size, the empirical measures and values of minimizers converge to an equilibrium distribution. We conclude with numerical examples comparing different kernels.
May 10, 2025

Out-of-Sample Embedding with Proximity Data: Projection versus Restricted Reconstruction

The problem of using proximity (similarity or dissimilarity) data for the purpose of "adding a point to a vector diagram" was first studied by J.C. Gower in 1968. Since then, a number of methods -- mostly kernel methods -- have been proposed for solving what has come to be called the problem of out-of-sample embedding. We survey the various kernel methods that we have encountered and show that each can be derived from one or the other of two competing strategies: projection or restricted reconstruction. Projection can be analogized to a well-known formula for adding a point to a principal component analysis. Restricted reconstruction poses a different challenge: how to best approximate redoing the entire multivariate analysis while holding fixed the vector diagram that was previously obtained. This strategy results in a nonlinear optimization problem that can be simplified to a unidimensional search. Various circumstances may warrant either projection or restricted reconstruction.
Jan 31, 2025

Supervised Quadratic Feature Analysis: Information Geometry Approach for Dimensionality Reduction

Supervised dimensionality reduction maps labeled data into a low-dimensional feature space while preserving class separation. A common strategy is to learn features that maximize a measure of statistical dissimilarity between the class-conditional probability distributions. Information geometry, which is rooted in Riemannian geometry, provides an alternative framework for measuring class dissimilarity. It treats probability distributions as points in a statistical manifold and uses the Fisher information metric to define a geodesic distance--the Fisher-Rao distance--between distributions The Fisher-Rao distance is an appealing candidate for measuring class separation because the Fisher information metric is a local measure of discriminability, and because it allows a geometric interpretation. Here, we present Supervised Quadratic Feature Analysis (SQFA), a supervised dimensionality reduction method which learns linear features that maximize Fisher-Rao distances between class-conditional distributions, under Gaussian assumptions. In multiple real world datasets, we find that SQFA features support classification accuracy that is competitive with features that maximize more popular measures of dissimilarity, or that are learned by other state-of-the-art dimensionality reduction methods. Notably, the best classification accuracy is achieved by SQFA-H features, a variant of SQFA that maximizes the Hellinger distance, a rarely used objective for dimensionality reduction. These results demonstrate the potential of information geometry as a tool for supervised dimensionality reduction. We provide a Python implementation of SQFA at https://github.com/dherrera1911/sqfa.
Feb 14, 2023

Linearized Wasserstein dimensionality reduction with approximation guarantees

We introduce LOT Wassmap, a computationally feasible algorithm to uncover low-dimensional structures in the Wasserstein space. The algorithm is motivated by the observation that many datasets are naturally interpreted as probability measures rather than points in Rn\mathbb{R}^n, and that finding low-dimensional descriptions of such datasets requires manifold learning algorithms in the Wasserstein space. Most available algorithms are based on computing the pairwise Wasserstein distance matrix, which can be computationally challenging for large datasets in high dimensions. Our algorithm leverages approximation schemes such as Sinkhorn distances and linearized optimal transport to speed-up computations, and in particular, avoids computing a pairwise distance matrix. We provide guarantees on the embedding quality under such approximations, including when explicit descriptions of the probability measures are not available and one must deal with finite samples instead. Experiments demonstrate that LOT Wassmap attains correct embeddings and that the quality improves with increased sample size. We also show how LOT Wassmap significantly reduces the computational cost when compared to algorithms that depend on pairwise distance computations.
Nov 12, 2018

Analytical Standard Errors for Exploratory Factor Solutions

Inference for factor models is often hampered by the lack of tractable and accurate variance estimates, which can materially distort downstream analyses. In practice, uncertainty in the residual covariance matrix is frequently either ignored or addressed through computationally intensive resampling methods that tend to be unstable. This paper develops a unified analytical framework for inference in exploratory factor analysis under several widely used extraction rules, including least-squares, principal-factor, iterative principal-component, alpha, and image factoring. By treating these estimators as implicitly defined functions of the sample covariance matrix, we derive closed-form Jacobians that translate perturbations in the covariance matrix into changes in the resulting factor solutions. Combined with the delta method and consistent estimators of the sample covariance matrix, the proposed approach yields standard errors that are straightforward to compute and remain valid under non-Gaussianity, heteroskedasticity, and serial or cross-sectional dependence. Simulation evidence confirms that the analytical standard errors accurately capture finite-sample variability while avoiding both the instability of bootstrap procedures and the restrictive assumptions underlying Fisher information-based inference. An application to a factor-augmented structural vector autoregressive (SVAR) model further demonstrates how accounting for this source of uncertainty can substantially affect impulse-response inference. Taken together, the results provide a practical and general tool for propagating estimation uncertainty in settings where factor extraction serves as an intermediate step.