Dimensionality Reduction

Ask about this topic

Overview

Dimensionality reduction turns data with many variables into a smaller representation that is easier to visualize, analyze, store, or use in a predictive model. Linear methods such as PCA and supervised discriminant projections select directions of variation, while t-SNE, UMAP, graph and spectral methods, optimal transport, and autoencoders capture nonlinear structure. Researchers apply these approaches to images, text embeddings, biological data, and dynamical systems, evaluating them through reconstruction, classification, neighborhood, density, and topology preservation. A central tension is that preserving variance or local neighborhoods may discard task-relevant signals or distort global geometry, so methods and evaluations increasingly account for the intended use, uncertainty, and out-of-sample behavior.

Latest papers

CardsList

145 papers9 papers in the last four weeks, down 36% on the four weeks before. 0.1% of all new papers.

Aug 15, 2026

Sufficient Dimesion Reduction via Generalized Stein's Lemma

Sufficient dimension reduction (SDR) seeks the minimal subspace of the predictors that captures the full conditional distribution of the response, which is known as the central subspace (CS). When the response is multivariate, the problem becomes considerably more challenging, particularly when the sample size is limited. Existing methods face different limitations:inverse regression approaches rely on strong distributional assumptions and matrix inversion, and their multi-response extensions suffer from severe slice sparsity; forward regression methods depend on computationally intensive iterative smoothing whose cost grows with the response dimension; and deep learning-based approaches demand large amounts of labeled data. To circumvent these shortcomings, we propose an SDR framework based on the generalized Stein's lemma. Our method constructs a cross-moment matrix between the multivariate response and the marginal score function of the predictors, and recovers the CS via its singular value decomposition. The proposed method does not rely on the linearity condition, avoids matrix inversion as well as iterative smoothing, and can leverage unlabeled data when available. We establish convergence guarantees for the proposed estimator under standard regularity conditions. Moreover, we propose a practical rank-selection algorithm to estimate the dimension of the CS. Extensive simulation studies and a real data application demonstrate that the proposed methods consistently outperform existing approaches across a variety of settings, particularly in moderate-dimensional, label-scarce scenarios with high noise levels.

Ye Tian

Key Laboratory of Applied Statistics of MOE, Key Laboratory of Big Data Analysis of Jilin Province, School of Mathematics and Statistics, Northeast Normal University, Renmin Street, Changchun, 130024, Jilin, China

Dimensionality Reduction
AskPDF
Aug 12, 2026

Function Lives Where Variance Doesn't: Task-Weighted Charts of a Language Model's Computation

How many dimensions does a language model's computation actually use? The question is ill-posed until one names a functional. Task-weighted charts make it well-posed: low-dimensional coordinate systems fit against a chosen functional of the representation, under the functional's own metric, turning distillation into plain least squares. Across six models from three families, spanning 70m to 7B parameters, next-token prediction needs 70--90% of the residual stream's width to stay within 5% of intact perplexity, a width consumed by the rare tail of language, and the variance profile predicts none of it: two directions carry 90% of GPT-2's activation variance and almost none of its function. Dimension is per-functional: the model's own uncertainty reads from six coordinates where the full predictive distribution needs hundreds; and it grows with depth. The dissociation is exploitable: when only a few dimensions can be kept, charts trained under the functional's metric preserve the model's predictions better than variance-based or optimal linear compression.

Alexandre Quemy

Hother, Krak´ow, Poland

Representation Geometry in Language ModelsIntrinsic DimensionalityDimensionality ReductionLLM Compression
AskPDF
Aug 11, 2026

When Is a General Factor Distinguishable? Non-Proportionality, Stable Structure, and the Bifactor Decision

Whether an added general dimension is necessary beyond correlated first-order factors is a property of population covariance, not an estimator. A bifactor structure is covariance-equivalent to correlated factors when general and group loadings are proportional within every cluster. When every cluster violates proportionality, at least three indicators per cluster and mild conditions rule out exact reproduction by a K-factor model with diagonal uniquenesses. Mixed configurations remain only partly characterized, motivating graded distinguishability. We develop a two-step procedure that delivers a stable first-order structure only when it persists across direct wider-count comparisons, then compares oblique and bifactor representations conditionally on delivery. A preliminary simulation study provides tentative design evidence. A major study develops three non-nested persistence profiles, designates .80/r2 as the practical default, and evaluates the frozen family on held-out replications from the same generators. Within the studied scenarios, anchor-only persistence outperforms anchor-zero, minimum congruence modestly outperforms maximum RMSD, and depth four adds no precision at the fixed .70 cutoff. Held-out results also expose stable-yet-wrong delivery under a weak core and show that the conditional comparison is highly accurate after correct recovery. Four empirical applications illustrate agreement, adjacent-count uncertainty, and non-delivery. Persistence, not count alone, carries the structural decision.

Jinsong Chen

Faculty of Education, The University of Hong Kong

Latent Variable ModelsParameter IdentifiabilityDimensionality Reduction
AskPDF
Aug 11, 2026

CosMAP: Contrastive Manifold Approximation and Projection for Dimensionality Reduction of Omics and Genealogical Data

Omics datasets, particularly single-cell RNA sequencing data, are high-dimensional, sparse, noisy, and dominated by zero values, making faithful low-dimensional representation challenging. Existing dimensionality-reduction methods may distort local neighbourhoods, global organization, or the cohesion of meaningful populations, with similar limitations arising in genealogical data. We introduce Contrastive Manifold Approximation and Projection (CosMAP), a graph-based unsupervised dimensionality-reduction method for producing faithful and interpretable embeddings. CosMAP extends the graph-based framework of UMAP by combining cosine-similarity neighbourhoods with temperature-normalized contrastive affinities, which are optimized in the embedding space using an attractive--repulsive objective. It further employs a two-phase refinement strategy: an intermediate higher-dimensional representation is first learned and then used to reconstruct the neighbourhood graph and initialize the final low-dimensional embedding. We evaluate CosMAP on MNIST and USPS handwritten-digit datasets, mouse retina and cortex single-cell RNA-sequencing datasets, and a large genealogical kinship dataset derived from BALSAC-CARTaGENE. Compared with state-of-the-art dimensionality-reduction methods, CosMAP produces more coherent visual representations, improves neighbourhood preservation, and provides clearer global organization of digit classes, biological cell populations, and regional genealogical patterns. These results indicate that CosMAP offers a robust framework for exploratory analysis of complex, sparse, high-dimensional data. The implementation is publicly available at https://github.com/FenosoaRandrianjatovo/CosMAP-dr.

Fenosoa Randrianjatovo, Maya Saleh, Simon Girard +1

Institut national de la recherche scientifique (INRS) · Centre Armand-Frappier Santé Biotechnologie (INRS-AFSB) · Unité mixte de recherche en santé durable (UMR INRS–UQAC) +1

Manifold LearningDimensionality Reduction
AskPDF
Aug 10, 2026

A Convolutional Layer Activation Dimensionality Reduction for Out-of-Distribution and Adversarial Attack Detection Methods

Despite the success of convolutional neural networks in image classification tasks and their general application in multi-modal models, their susceptibility to out-of-distribution and adversarial attack samples raises concerns regarding trustworthiness and safety. Among the approaches to tackle such issues, detection methods that analyze the model's intermediate activations to estimate a confidence score are a promising family that evaluates the decision process, relying on a dimensionality reduction step to enable efficient downstream processing of the high-dimensional activations. However, when considering convolutional layers, the dimensionality reduction methods in the literature either lack a mechanism to control the compression/information-loss trade-off or yield large representations. In this paper, we carefully analyze two state-of-the-art detection methods and their dimensionality reductions for convolutional layers and develop a novel reduction method with a controllable high-compression level. We extend these two state-of-the-art detection methods, enabling the usage of any dimensionality reduction, and evaluate their performance on out-of-distribution and adversarial attack detection. Results show that the detection methods with the proposed dimensionality reduction consistently perform better than, or comparable to, the strongest alternative. Furthermore, the proposed method is shown to reduce computation and memory footprints, given that it has the highest compression among the compared methods.

Leandro de Souza Rosa, Lorenzo Capelli, Clara Nunes Barrancos +2

Dipartimento di Ingegneria dell’Energia Elettrica e dell’Informazione “Guglielmo Marconi” (DEI) - Alma Mater Studiorum Università di Bologna, Bologna, Italy · Advanced Research Center on Electronic Systems “Ercole De Castro” (ARCES) - Alma Mater Studiorum Università di Bologna, Bologna, Italy

Convolutional Neural NetworksDimensionality ReductionOOD DetectionAdversarial Example Detection
AskPDF
Aug 7, 2026

Understanding Differentiable Embeddings Through Differential and Integral Geometry

How can an analyst decide whether a nonlinear dimensionality reduction embedding can be trusted? Existing diagnostics provide only partial answers: projection glyphs characterize local sensitivity, map-continuity scores measure local conditioning, and transport-based analyses reveal path-dependent inconsistencies. However, these methods appear unrelated and provide no common framework for understanding when they agree or not. We show that they are all derived from a single geometric object induced by every differentiable embedding, whether defined implicitly through optimization or explicitly by a learned mapping. This framework provides two complementary geometric views of an embedding. The differential view explains local behavior: its first-order term recovers projection glyphs, while its second-order curvature quantifies how far their linear approximation remains reliable. The integral view follows the same geometry along high dimensional paths and determines whether an embedding depends only on the current state or also on the path taken to reach it. We further show that map-continuity is a prerequisite for the other analyses. The framework is theoretically complete for diagnostics derived from the embedding geometry, and we prove the integral view irreducible: no amount of local measurement at any number of points, to any order of derivative, reproduces what it detects. Classical rank-based metrics form a complementary class based on finite-scale neighborhood relationships. Experiments on synthetic and real datasets validate theoretical predictions, demonstrate accurate curvature-based trust estimates on single-cell embeddings, and show that the integral analysis distinguishes single-valued embeddings from path-dependent optimization-based embeddings in ways that existing pointwise diagnostics cannot.

Xinyu Zhang, Klaus Mueller

Stony Brook University · Department of Computer Science, Stony Brook University, Stony Brook, NY, USA

Dimensionality Reduction
AskPDF
Aug 4, 2026

Physics-informed reduced-order modelling with equivariant spectral submanifolds

Spectral submanifold (SSM) reduction has emerged as a mathematically principled route to reliable nonlinear reduced-order models, capturing dynamics beyond the reach of linear techniques such as Dynamic Mode Decomposition (DMD). The computation of SSMs, however, remains computationally expensive, particularly for high-dimensional systems. In this work, we introduce equivariant spectral submanifold (eSSM) reduction, a novel extension of the SSM framework that explicitly incorporates symmetries of the full-order model into the reduction process. We establish the mathematical foundations of this approach by showing that SSMs are naturally equivariant submanifolds and that the associated charts and reduced dynamics inherit the appropriate induced group actions. Building on this framework, we develop a novel equivariant SSM reduction algorithm that exploits these symmetries to achieve substantially faster computations while also improving model robustness. We demonstrate the advantages of this approach on several benchmark problems including a test from the Common Task Framework for Science.

Georg Maierhofer

Department of Applied Mathematics and Theoretical Physics · University of Cambridge, UK

Reduced-Order ModelingScientific MLDimensionality Reduction
AskPDF
Jul 30, 2026

Measuring Distortion in the Empty Regions of Dimensionality Reduction Scatterplots with the Gap Index

Quality metrics play a crucial role in the proper use of dimensionality reduction projections for visual analysis of high-dimensional data. They quantify the degree of distortion of a projection compared to the high-dimensional data and provide a reliable indication of how confident users can be in the structures they see in the resulting layouts. However, most popular metrics focus on capturing direct relationships between points (e.g., distances or neighborhoods) while neglecting distortions in empty areas of the layout, even though these often compose visually relevant features of a 2D layout. In this paper, we introduce the Gap Index (GI), a quality metric for 2D projections that captures visual distortion by measuring spatial distortion in empty areas of a projection. It does so by decomposing the space into empty triangles, which are then compared to their high-dimensional counterparts to compute the deformation. This per-triangle deformation can be aggregated into a single scalar value or overlaid on a projection to visualize regional distortion patterns. Results show that, contrary to popular quality metrics, the GI is sensitive to small structural deformations that have high visual impact. It is also fast to compute and interpretable.

Jaume Ros, Alessio Arleo, Fernando Paulovich

Eindhoven University of Technology

Data VisualizationVisual AnalyticsDimensionality Reduction
AskPDF
Jul 30, 2026

On a joint simultaneous learning of relevant feature subsets and subspaces in regression-like problems

We extend a recently introduced Entropy-Optimal Manifold Clustering (EOMC) to allow for a joint simultaneous identification of subsets and subspaces of relevant features in nonstationary and nonlinear regression problems. It is shown that the proposed extension - that we coin as Entropy-Optimal Manifold Regression (EOMR) - allows a robust learning with linearly-scaling iteration and memory complexities. EOMR is compared to the most complete set of state-of-the-art tools from the Artificial Intelligence (AI) and Machine Learning (ML) that is available to the author, on the very challenging problems from chaotic and fluid dynamics: (i) on predicting the Lorenz-96 systems dynamics in strongly- and very-strongly chaotic regimes (with forcing parameter being F=8F=8 and F=12F=12, respectively); and, (ii) on a data from the Hasegawa-Wakatani model on the edge of the tokamak plasma. It is demonstrated that the proposed benchmarks (i) and (ii), indeed, are the very challenging problems for the state of the art ML and AI tools - since both the general-purpose gradient boosted random forests and deep neuronal networks, as well as transformer-based AI tools like TabPFN v.03 (more spezialised for large-dimensional small data learning problems) - result in orders of magnitude inferior root mean squared prediction errors, and orders of magnitude larger model complexities, when compared to the EOMR. For a Hasegawa-Wakatani example, EOMR distills a very simple entropy-optimal and skilful description of the leading Essential Orthogonal Function (EOF) dynamics, given by linear, causal and weakly-stationary autoregressive process described by just 8 parameters.

Illia Horenko

Chair for Mathematics of AI, Faculty of Mathematics, RPTU Kaiserslautern-Landau, Gottlieb-Daimler-Str. 48, Kaiserslautern, 67663, Germany.

Scientific MLFeature SelectionDimensionality ReductionChaotic Time-Series Prediction
AskPDF
Jul 29, 2026

FADEx: Feature Attribution and Distortion-based Explanation of Dimensionality Reduction

Dimensionality Reduction (DR) is a fundamental tool for high-dimensional data exploration, reducing the complexity of latent spaces of machine learning models, and assisting in the explanation of complex opaque models. However, non-linear DR techniques often function as opaque transformations themselves, making it challenging to understand how individual features influence instance positioning in the reduced space. This lack of transparency complicates the analysis and interpretation of structural patterns, hindering the ability to reason about the organization of high-dimensional data based on the projected layout. In order to address this challenge, dimensionality reduction explanation methods have shown promise in improving the understanding of the observed groups and cluster structures. Unfortunately, existing DR explanation approaches tend to suffer from limitations such as multiple attributions per feature and restricted applicability to specific dimensionality reduction methods, which hinder their use. In this work, we propose FADEx, a novel local per-instance feature attribution method that leverages local linear approximation via first-order Taylor expansion and Singular Value Decomposition to provide explanations. FADEx computes the local linear models via weighted least squares, eliminating the need for out-of-sample data mapping, making it agnostic to the DR method, while simultaneously providing local feature attributions and distortion analysis. Through qualitative and quantitative evaluations, comparisons with existing methods, and case studies, we demonstrate FADEx's effectiveness and versatility in providing explanations and analytical resources for analyzing the behavior of DR methods. The results indicate FADEx yields robust and reliable explanations, outperforming existing approaches in several aspects.

Lucas Greff Meneses, Evandro S. Ortigossa, Claudio Silva +1

University of São Paulo · Weizmann Institute of Science · New York University

Feature AttributionInverse Dimensionality ReductionDimensionality Reduction
AskPDF
Jul 28, 2026

FloDR: An invertible dimensionality reduction method based on a normalising flow

It is common for two-dimensional embeddings of high-dimensional data to be read far beyond what they can support. Distances in and between clusters, the meaning behind empty spaces, and the amount of structure hidden at each point are generally invisible in the output of methods such as t-SNE and UMAP. This is because the information that could support the meaning of these properties is discarded during the optimisation process. Here, we present FloDR, a dimensionality reduction method that embeds data through an invertible normalising flow. While FloDR only uses the first two output coordinates to create a two-dimensional embedding, it retains the remaining coordinates rather than discarding them. In addition to the embedding, an exact inverse and an exact density are properties of a trained mapping, which enable diagnostic visualisations that are computed from the exact inverse of the model that drew the layout rather than from an approximate one. Specifically, we draw two fields, the conditional spread, which measures how much of the original data remains undetermined at each embedding position in input units, and the hidden contrast, which measures how much information about a labelled contrast the two plotted coordinates discard. Both fields are rendered with a prespecified test against a held out portion of the input data and a bootstrap confidence. A field that fails the test is reported as refused.

Abdallah Baraka, Daniel Probst

Bioinformatics Group Wageningen University & Research Wageningen 6708PB, The Netherlands

Data VisualizationVisual AnalyticsDimensionality Reduction
AskPDF
Jul 28, 2026

A Riemannian View on Active Subspaces

Active subspaces provide an explainable, eigenvalue-ordered principle for studying how scalar-valued quantities of interest change the most, on average, over a reduced basis of Euclidean domains. Composition with parallel transport generalizes this principle from Euclidean space to quantities of interest defined over Riemannian manifolds, and the resulting intrinsic formulation is contrasted with the extrinsic, embedding-based gradient average of manifold learning. Either strategy is studied in an intrinsically local sense, restricted to mean-centered geodesic-balls, and within that scope the two are not identical: on the central tangent space, eigenvalues agree to second order in the geodesic radius of the sampled domain, while dominant eigenspaces agree at the same order relative to the spectral gap. Extending activity beyond that central space then calls for either recomputed decompositions over changing tangent spaces or, intrinsically, parallel transport of a single central frame. Hyperspheres are emphasized throughout as a particular manifold of interest, motivated by applications over preshape spaces for statistical shape analysis. Numerical examples over the 2-sphere illustrate the formalism, including the derived ridge recovery at a curvature-limited quadratic rate.

Zachary Grey

Dimensionality ReductionRiemannian Geometry
AskPDF
Jul 27, 2026

The balance between compactness and forecast accuracy of data-driven latent-space reduced-order models in controlled wake flows

Model-based active flow control requires predictive models that are accurate, stable, and fast enough for real-time optimisation. In controlled wake flows, this is often achieved through Reduced-Order Models (ROMs) that first compress high-dimensional velocity snapshots into a latent space and then learn a time- stepping predictor for the dynamics in the latent space. Here, we study how the choice of the spatial encoder affects the predictability of the resulting latent coordinates for wake flows under control inputs. Using two actuated 2D wake configurations, a simplified truck wake and the fluidic pinball, we compare Proper Orthogonal Decomposition (POD) against nonlinear Convolutional Autoencoders (CAEs) and two types of variational autoencoders for compression, and evaluate several temporal predictors based on Long Short-Term Memory networks. CAEs achieve higher compression efficiency and sharper short-term reconstructions, but they produce latent dynamics that are more irregular and with broadband spectral content. As a consequence, long-horizon forecasts degrade faster and show a higher probability of catastrophic divergence than POD-based models. POD yields smoother latent trajectories that are easier to learn and extrapolate, leading to more reliable predictions beyond the short- term regime. These results reveal a clear trade-off between compactness and forecast accuracy, and suggest that the stability of the latent dynamics prediction can outweigh maximal compression. This is particularly relevant for control strategies rooted in forecasts of the dynamics, such as model predictive control and reinforcement learning. The findings provide practical guidance for designing actuation-aware, hardware-feasible predictive ROMs for real-time flow control.

Alberto Solera-Rico, Patricia García-Caspueñas, Carlos Sanmiguel Vila +1

Sub-directorate general of aeronautical systems, Spanish National Institute for Aerospace Technology (INTA), ctra. M-301, km. 10.5, 28330 San Mart´ın de la Vega, Madrid, Spain. · Departamento de Ingenier´ıa Aeroespacial, Universidad Carlos III de Madrid, Avenida de la Universidad 30, 28911 Legan´es, Madrid, Spain. · Department of Mechanical Engineering, University of Washington, 1410 NE Campus Parkway, Seattle, WA 98195, USA.

Variational AutoencodersLatent Dynamics ModelingAutoencodersProper Orthogonal DecompositionConvolutional AutoencoderFeedback Control+2
AskPDF
Jul 22, 2026

Non--negative matrix factorization using the R package nnmf

Non--negative matrix factorization (NMF) has become an established dimensionality reduction technique for extracting latent structures from non--negative data and has found widespread applications in fields such as bioinformatics, text mining, image analysis, and recommender systems. As the popularity of NMF has increased, numerous R packages implementing different optimization strategies and computational frameworks have been developed. Despite their widespread availability, comprehensive evaluations of these implementations under real--world data conditions remain limited. Consequently, researchers often lack objective guidance when selecting an appropriate package for practical applications. This study introduces a new R package for NMF and offers asystematic performance comparison with two widely available R packages for NMF analysis. Rather than relying on simulated datasets, the evaluation is conducted using real--world data to better reflect the complexity, heterogeneity, and noise characteristics encountered in practical analytical settings. The packages are assessed using a consistent experimental framework, with emphasis on computational efficiency, convergence behavior, reconstruction accuracy, memory utilization, and the stability of the resulting matrix factorization.

Volkan Sevinç, Nikolas Kontemeniotis, Theodoros Perdikis +1

Muğla Sıtkı Koçman University, Faculty of Science, Department of Statistics, Kötekli Kampüsü, Muğla, Türkiye · Department of Economics, University of Crete, Gallos Campus, Rethimno, Greece · Department of Tourism Studies, University of Piraeus, Piraeus, Greece +1

Non-Negative Matrix FactorizationDimensionality ReductionMatrix Factorization
AskPDF
Jul 21, 2026

A Bayesian Framework for Built-in Input Dimension Reduction for Gaussian Process Modeling

Gaussian process (GP) modeling is widely used in computational science and engineering. However, fitting a GP to high-dimensional inputs remains challenging due to the curse of dimensionality. While various methods have been proposed to reduce input dimensionality, they typically follow a two-stage approach, performing dimension reduction and GP fitting separately. We introduce a Bayesian framework that seamlessly integrates dimensionality reduction with GP modeling and inference. Our approach, built on a hierarchical Bayesian model with priors on the Stiefel manifold, enforces orthonormality on the projection matrix and enables posterior inference via Hamiltonian Monte Carlo with geodesic flow. Additionally, we extend this framework by incorporating Deep Gaussian Processes (DGP) with built-in dimension reduction, providing a more flexible and powerful tool for complex datasets. Through extensive numerical studies, we demonstrate that while the proposed Bayesian method incurs higher computational costs, it improves predictive performance and uncertainty quantification, providing a principled and robust alternative to existing methods.

Eric Herrison Gyamfi, Emily L. Kang, Bledar A. Konomi +1

Division of Statistics and Data Science, Department of Mathematical Sciences, University of Cincinnati, Cincinnati, OH · Department of Mathematics, Purdue University, West Lafayette, IN

Bayesian InferenceDeep Gaussian ProcessesGaussian ProcessesDimensionality ReductionGaussian Process Regression
AskPDF
Jul 17, 2026

Scaling Time Series Classification via XAI-Driven Data Reduction

Explainable AI (XAI) for time series has seen significant algorithmic growth, but its utility in providing measurable performance gains for downstream tasks remains under-explored. This paper bridges this gap by introducing drXAI, a novel methodology that repurposes XAI attribution methods for effective data reduction in Time Series Classification (TSC). The core challenge in modern TSC is scalability; state-of-the-art models, such as Transformers, exhibit quadratic complexity relative to sequence length and linear complexity relative to the number of channels. This renders them computationally prohibitive for massive datasets. drXAI addresses this by using a fast, GPU-accelerated classifier (Hydra) to generate local attributions. We aggregate these into global feature importance scores and employ an automated elbow-cut heuristic to select the most salient features without requiring manual thresholds. We evaluate our approach on both synthetic and real-world univariate and multivariate datasets. On synthetic benchmarks, drXAI successfully recovers ground-truth features where traditional baselines fail. On real-world data, drXAI achieves between 80% and 90% data reduction while maintaining classification accuracy comparable to models trained on the full dataset. Most importantly, we show that drXAI allows resource-intensive models like ConvTran to scale to datasets that were previously inaccessible due to memory constraints. Our results show the benefits of using XAI not just for interpretability, but as a robust tool for feature selection and scalability in time series analysis. All our code and data are openly available.

Davide Italo Serramazza, Thach Le Nguyen, Georgiana Ifrim

School of Computer Science, University College Dublin, Ireland

Explainable Artificial IntelligenceTime Series ClassificationFeature SelectionDimensionality Reduction
AskPDF
Jul 11, 2026

Mathematics of Data Science

This book is about the mathematical foundations of data science. 1. Introduction 2. Curses, Blessings, and Surprises in High Dimensions 3. Singular Value Decomposition and Principal Component Analysis 4. Linear Regression and Regularization 5. Graphs, Networks, and Clustering 6. Nonlinear Dimension Reduction and Diffusion Maps 7. Linear Dimension Reduction via Random Projections 8. Optimization for Data Science 9. Classification 10. A Mathematical Introduction to Deep Learning 11. Large Sample Limit of Graph Laplacians 12. Community 13. Concentration of Measure and Gaussian Analysis 14. Matrix Concentration Inequalities 15. Compressive Sensing and Sparsity 16. Low-Rank Matrix Recovery

Afonso S. Bandeira, Amit Singer, Thomas Strohmer

ETH Zürich · Princeton University · UC Davis

Dimensionality Reduction
AskPDF
Jul 10, 2026

Terminal Dimension Reduction for Time Series with Applications

Terminal embeddings have emerged as a powerful tool for dimension reduction. Given a set of points P⊂RdP\subset \mathbb{R}^d, a terminal embedding is a mapping f:Rd→Rtf:\mathbb{R}^d\rightarrow \mathbb{R}^t that preserves the pairwise distance between any pair of points p∈Pp\in P and q∈Rdq\in \mathbb{R}^d up to small distortion under this mapping. Terminal embeddings have been particularly fruitful for constructing kk-means and kk-median coresets, where the objective is to find a typically weighted subset ΩΩ of PP such that for any candidate solution, the cost of the clustering objective on ΩΩ approximates the cost of the clustering objective on PP up to small distortion. Unfortunately, these techniques have not been extended to more complicated structures such as clustering time-series data under common straight-line interpolation between measurements. The main issue is that terminal embeddings, arguably the central technique in this line of research, cannot be linear and are thus not immediately suitable to preserve linear structures. In this work, we develop a generalization of terminal embeddings to affine line-segments that overcomes this issue. We showcase their applicability by using our lines-preserving terminal embeddings to obtain the first dimension-free coresets for clustering time-series under the Fréchet distance. The underlying dimension reduction uses Johnson-Lindenstrauss (JL) embeddings, and our experiments indicate that terminal embeddings perform similarly to JL and favorably against PCA for synthetic and real-world time-series, while only terminal embeddings extend pairwise distance preservation to the full ambient space.

Alexander Munteanu, Matteo Russo, David Saulpic +1

TU Dortmund, Germany · EPFL, Switzerland · CNRS & Université Paris Cité, IRIF, Paris, France +1

Coreset SelectionDimensionality ReductionTime Series Clustering
AskPDF
Jul 9, 2026

Group Invariant Spectral Embedding

Spectral embedding methods are widely used for dimensionality reduction and clustering of high-dimensional datasets with intrinsic low-dimensional structures. Although many datasets of practical interest exhibit invariance under symmetries such as rotations, standard spectral embedding methods do not account for this, treating symmetry-related data points as unrelated. Our approach to this problem is to incorporate the symmetries directly into the affinity kernels used for spectral embedding. We analyze the case of a Riemannian data manifold MM with symmetries given by a compact Lie group~GG and prove that, under suitable conditions, graph Laplacians constructed from three types of invariant kernels converge pointwise to explicit second-order differential operators on the quotient space M/GM/G. Our analysis implies improved convergence rates, as the effective dimension drops according to the dimension of the group. We validate our approach on datasets with SO(2)\mathrm{SO}(2) or SO(3)\mathrm{SO}(3) symmetry, and show that GG-invariant spectral embedding recovers the intrinsic geometry of the data, in contrast to standard spectral embedding, which fails to do so even in the limit of infinite data.

Yeari Vigder, Paulina Hoyos, David Thong +3

Department of Statistics and Operations Research, Tel Aviv University, Tel Aviv, Israel · Department of Mathematics, University of Texas at Austin, Austin, TX, USA · Department of Mathematics, KTH Royal Institute of Technology, Stockholm, Sweden

Manifold LearningDimensionality ReductionSpectral MethodsInvariant Representation LearningSpectral Graph Embedding
AskPDF
Jul 9, 2026

Dimensionality Reduction Meets Network Science: Sensemaking on UMAP's kNN Graph

While UMAP is widely used for exploring high-dimensional data, typical workflows focus on its lower-dimensional embedding, largely overlooking the rich k-nearest-neighbor (kNN) graph that UMAP constructs internally. This graph encodes the data manifold in its original high-dimensional space, before the distortion that UMAP's 2D projection introduces. We demonstrate the untapped potential of this internal representation, showing how standard graph algorithms applied to this graph enhance data sensemaking: (1) PageRank identifies representative data points, (2) k-core decomposition reveals dense core regions versus sparse periphery, and (3) clustering coefficient detects tight-knit neighborhoods with highly-similar data points. Through quantitative and qualitative evaluation on MNIST and Fashion MNIST, we show that these graph-based analyses are not only practical but also competitive with or complementary to purpose-built methods (e.g., k-medoids for exemplar selection, HDBSCAN for density-based clustering).

Duen Horng Chau, Donghao Ren, Fred Hohman +1

Apple

Dimensionality Reduction
AskPDF
Jul 6, 2026

Orthogonal Dendritic Intrinsic Networks: An Architecture for Significance-Ordered, Orthogonal Latent Spaces

Principal Component Analysis or PCA-like properties (orthogonality, variance ranking) are seldom realized in deep autoencoder architectures. In this work, we present ODIN (Orthogonal Dendritic Intrinsic Network), a novel autoencoder architecture that recovers PCA-like latent structure in a fully non-linear regime. By incorporating a set of geometric constraints directly into the training objective, ODIN encourages latent dimensions to be mutually orthogonal and ordered by explained variance, mirroring the interpretable decomposition of PCA while retaining the expressive power of deep networks. We provide theoretical grounding for these constraints and demonstrate their compatibility with standard encoder-decoder frameworks. We also establish empirical results for both synthetic and real world datasets, establishing a principled path toward interpretable, structured feature learning and dimensionality reduction.

Jeanie Schreiber, Tyrus Berry, Zeeshan Ahmed

Department of Mathematical Sciences George Mason University Fairfax, VA 22030, USA · Sensor Science Division, Physical Measurement Laboratory NIST Gaithersburg, MD 20899, USA

Representation LearningPrincipal Component AnalysisAutoencodersUnsupervised Representation LearningOrthogonal Representation LearningDimensionality Reduction+1
AskPDF
Jul 6, 2026

msPCA: An R Package for Sparse PCA with Multiple Components

We present msPCA: an open-source R package for sparse principal component analysis with multiple components. It implements an alternating maximization algorithm to generate a set of sparse loading vectors that collectively explain a large fraction of the variance in a dataset, while remaining non-redundant. The algorithm supports two definitions of non-redundancy: either orthogonality of the loading vectors or zero pairwise correlation between principal components (PCs). In the reported benchmarks, msPCA solves sparse PCA problems with thousands of features, achieving competitive runtimes while producing sparse components with controlled feasibility violations and a high fraction of variance explained.

Ryan Cory-Wright, Jean Pauphilet

Analytics, Marketing and Operations Imperial Business School · Management Science and Operations London Business School

Principal Component AnalysisDimensionality Reduction
AskPDF
Jul 3, 2026

Reduced-Order Models: The Mother of World Models

World models -- compressed latent representations of an environment that support action-conditioned prediction and planning -- are typically presented as a product of modern self-supervised learning. This paper argues that the functional anatomy of a world model was independently developed, deployed, and formally analyzed decades earlier in the model-order-reduction (MOR) and control literature, under different names and for a different purpose: the real-time operation of physical systems. We trace the anatomy across three communities. Low-dimensional models of turbulence built on proper orthogonal decomposition (POD) supplied latent dynamics learned from data of a chaotic environment; eigenface methods in early computer vision supplied the encoder-decoder half, including a primitive runtime validity check; and measurement-based POD frameworks for facility thermal control assembled the complete loop -- POD coefficients as latent state, parametric dependence on actuator setpoints as action conditioning, modal reconstruction as decoding, and, critically, a priori analytical error bounds as a verification layer that certified when the model's predictions could be trusted in closed loop. We then examine what each tradition possesses that the other lacks: MOR contributes verification, physical grounding, and extreme data efficiency; learned world models contribute nonlinear representation, transferability, and horizon. We argue that the outstanding obstacle to deploying world models in systems that cannot fail -- power, thermal, process control -- is not predictive fidelity but verifiability, and we outline a research agenda for physics-grounded, verifiable world models that unifies the two lineages.

Rajat Ghosh

Independent Researcher

World ModelsAction-Conditioned World ModelsProper Orthogonal DecompositionReduced-Order ModelingDimensionality Reduction
AskPDF
Jul 3, 2026

Dimension Reduction for Curves: Simplified and Generalized

We revisit random projections for reducing the dimension of high-dimensional polygonal curves. Drawing from the toolbox of randomized linear algebra, we give a considerably simplified proof of the known O(ε−2log⁡(nm))O(\varepsilon^{-2}\log(nm)) bound on the target dimension of a random projection that preserves the continuous Fréchet distance of polygonal curves up to a factor (1±ε)(1\pm\varepsilon). Our proof is based on the concept of sparse oblivious subspace embeddings. While previous techniques were limited to the case of the Fréchet distance, our techniques are fairly general and extend to all possible distance measures that involve the maximum, a sum or an integral over Euclidean distances between pairs of points on both input curves. We define a generalized dissimilarity measure for curves that includes several popular measures such as Fréchet, qq-DTW, Hausdorff, etc. as special cases and show that the same dimension reduction technique works for this generalized dissimilarity measure. Finally, we apply the same framework for dimension reduction to piecewise linear surfaces, after extending the distance measure suitably to such surfaces.

Matthijs Ebbens, Jie Lu, Alexander Munteanu

TU Dortmund, Germany

Dimensionality Reduction
AskPDF
Jun 29, 2026

Adjusted Wasserstein distances for bridging empirical and true distributions with applications to MDS

This paper examines how metric adjustments to Multidimensional Scaling (MDS) can enhance its effectiveness as a visual tool for pattern recognition. The distance under consideration, referred to as Max-D-SW, is an adjustment of the Max-Sliced Wasserstein distance. In contrast to the original formulation, which optimizes over single unit directions, Max-D-SW aggregates contributions over orthonormal bases. This modification provides a clear numerical advantage in MDS outcomes, particularly when applied to heavy-tailed distributions. We also establish sample-complexity bounds showing that Max-D-SW remains statistically tractable, with rates comparable to those of its max-sliced counterpart. Moreover, we show that a better sample complexity for a metric does not necessarily translate into better performance when the metric is used as an input for MDS.

Flor Martinez-Sermeno, Arturo Jaramillo, Johan Van Horebeek

Multidimensional ScalingDimensionality ReductionHeavy-Tailed DistributionsSliced Wasserstein DistanceWasserstein Distance
AskPDF
Jun 26, 2026

Autoencoder Architectures for Athlete Performance Scoring from Wearable Telemetry

Wearable devices produce large, high dimensional training logs for everyday runners, and interpretation rather than data collection is now the limiting step. This paper evaluates five dimensionality reduction models, three autoencoder variants, PCA, and a Variational Autoencoder, on their ability to compress nine sensor runner profiles into a single scalar performance indicator, the latent score. Because the setting is fully unsupervised, model quality is assessed along two complementary axes: reconstruction error (Mean Squared Error) and latent score interpretability, measured via Spearman and Kendall rank correlations, Mutual Information, and Permutation Importance. These are combined into a composite selection criterion that prevents selecting models on reconstruction accuracy alone. Feature rankings from the four metrics are aggregated via a modified Borda count, and their stability is confirmed by bootstrap validation. A two feature linear baseline is included to anchor the comparison. Deep autoencoder achieved the lowest reconstruction error and the highest composite score. Once the PCA hidden layers were widened, the deeper variants became closely competitive with Deep AE on the composite criterion, indicating that the limiting factor was hidden layer capacity rather than the one dimensional bottleneck. Running pace, aerobic decoupling, and average heart rate emerged as the dominant latent score drivers across all models and resampling runs, consistent with established physiology.

Mateusz Kubita, Jan Zubalewicz, Krzysztof Siwek

Warsaw University of Technology Warsaw, Poland

Variational AutoencodersAutoencodersDimensionality Reduction
AskPDF
Jun 26, 2026

Graph Dimensionality Reduction for Contextual Bandits: Structure-Specific Regret Bounds under Approximate Smoothness and Noisy Eigenspaces

Contextual bandits with graph-structured arms arise in recommendation, citation retrieval, and social advertising, where arms connected on a graph tend to share reward signal. Standard dimensionality reduction ignores this structure, inflating exploration cost by a factor of d/kd/k. We propose GraphDR-LinUCB, which projects arm features onto the graph's low-frequency spectral subspace and runs linear UCB in the resulting kk-dimensional space. We prove the first \wtO(kT)\wtO(k\sqrt{T}) regret bound for spectral-projection-based contextual bandits, reducing dimension dependence from dd to kk; a perturbation argument extends this to noisy graphs, with an explicit penalty for reward-smoothness mismatch and graph-estimation error. Our central theoretical finding is that the high-frequency reward component need not incur a worst-case linear-in-TT penalty: its actual cost depends on its realized impact along the played path, not on its total energy. A simple spectral comparison between subspaces (ΓkΓ_k) predicts which reducer wins on a given dataset, correctly calling five of six real-dataset outcomes without any fitted threshold. Across a synthetic benchmark and six real datasets (MovieLens, Amazon, LastFM, ogbn-arxiv, MIND), GraphDR-LinUCB reduces cumulative regret by 15×15\times over full-dimensional LinUCB and outperforms competing graph-aware methods on five of six; the single failure is precisely where the graph's spectral subspace is misaligned with the reward.

Joyanta Jyoti Mondal, Ibne Farabi Shihab, Anuj Sharma

Department of Computer and Information Sciences, University of Delaware, USA · Department of Computer Science, Iowa State University, USA · Department of Civil, Construction & Environmental Engineering, Iowa State University, USA

Multi-Armed BanditsGraph BanditsContextual BanditsDimensionality ReductionLinear Contextual Bandits
AskPDF
Jun 17, 2026

Dimensionality Reduction of QAOA Parameter Space with Kernel PCA for Max-Cut

The Quantum Approximate Optimization Algorithm (QAOA) is a leading variational algorithm for combinatorial optimization on near term quantum devices. As circuit depth increases, the number of optimization parameters grows, making the search landscape increasingly nonlinear and difficult to optimize. Previous studies have shown that optimal QAOA parameters often lie on a low dimensional manifold that can be approximated using Principal Component Analysis (PCA) at shallow circuit depths. However, the effectiveness of PCA decreases at higher depths because the underlying parameter manifold becomes increasingly nonlinear. In this work, we investigate Kernel Principal Component Analysis (KPCA) with a radial basis function kernel as a nonlinear dimensionality reduction technique for QAOA parameter optimization. The model is trained using 200 graphs from each of 3 graph families, namely Erdos-Renyi, Barabasi-Albert, and Watts-Strogatz, with graph sizes ranging from 7 to 10 nodes. Performance is evaluated on 30 test graphs containing 12 nodes at circuit depths 1, 2, 4, and 8. Experimental results demonstrate that KPCA consistently outperforms PCA at deeper circuit depths across all graph families. At depth 8, KPCA achieves approximation ratios above 0.86, while PCA declines to approximately 0.81 to 0.83. Both methods reduce the number of quantum circuit evaluations by more than 93 percent relative to unrestricted QAOA optimization. These findings suggest that nonlinear kernel methods more effectively capture the structure of the QAOA parameter manifold and provide a practical approach for scaling variational quantum optimization to deeper circuits.

Sidharth Brahmandam, Vayd Ramkumar

Naperville North High School · Independent Researcher

Principal Component AnalysisCombinatorial OptimizationDimensionality ReductionQuantum Approximate Optimization AlgorithmMax-Cut
AskPDF
Jun 15, 2026

Analytic Torsion and Spectral Gap Capture Persistent-Laplacian Performance

While persistent Laplacians (PL) offer a richer geometric representation of data than persistent homology, utilizing their full eigenspectrum for learning tasks is often hampered by high dimensionality and the ``varying length'' problem across different filtration scales. We propose a compact spectral representation that distills the persistent Laplacian into three mathematically grounded invariants: Betti numbers, the spectral gap, and analytic torsion. Across benchmark datasets including MNIST, QM-3D, and SKEMPI WT, we demonstrate that this reduced feature space captures the essential predictive signal of the full spectrum, and in some cases outperforms it, while significantly reducing computational overhead and preventing the noise introduced by higher-frequency eigenvalues. Our results suggest that these invariants provide a principled, fixed-length interface between spectral geometry and topological learning.

Jernej Grlj, Aaron D. Lauda

Department of Mathematics University of Southern California Los Angeles, CA 90089 · Department of Mathematics Department of Physics University of Southern California Los Angeles, CA 90089

Dimensionality ReductionSpectral MethodsTopological Representation Learning
AskPDF
Jun 12, 2026

The Risk Shadow of Principal Component Analysis: When 99.9999% Variance Preservation Causes Catastrophic Decision Errors

Principal Component Analysis (PCA) preserves variance, not the information needed to detect rare catastrophic events. This paper proves the existence of a {\it Risk Shadow}: PCA can retain over 99.9999 percent of total variance while completely erasing all signal about rare, high-impact failures. When this happens, even the best possible classifier operating on the PCA representation reduces to a constant predictor. The root cause is a fundamental mismatch between variance maximization and tail risk awareness. To break the shadow, we introduce Expectile PCA (ExPCA) and Tail-Preserving PCA (TP-PCA), two methods that reweight the data covariance toward high-impact events. We prove theoretically that ExPCA strictly outperforms PCA in retaining rare-event information, and we validate our claims on synthetic data and a real-world credit card fraud detection benchmark. Our results call for a fundamental rethinking of variance-based dimensionality reduction in high-stakes decisions.

Hamidou Tembine

Department of EECS, School of Engineering, UQTR, Canada. Learning and Game Theory Laboratory (LnG Lab), TIMADIE.

Principal Component AnalysisDimensionality ReductionAnomaly Detection
AskPDF