Dimension reduction (DR) is inherently non-unique: multiple embeddings can preserve the structure of high-dimensional data equally well while differing in layout or geometry. In this paper, we formally define the Rashomon set for DR -- the collection of `good' embedding -- and show how embracing this multiplicity leads to more powerful and trustworthy representations. Specifically, we pursue three goals. First, we introduce PCA-informed alignment to steer embeddings toward principal components, making axes interpretable without distorting local neighborhoods. Second, we design concept-alignment regularization that aligns an embedding dimension with external knowledge, such as class labels or user-defined concepts. Third, we propose a method to extract common knowledge across the Rashomon set by identifying trustworthy and persistent nearest-neighbor relationships, which we use to construct refined embeddings with improved local structure while preserving global relationships. By moving beyond a single embedding and leveraging the Rashomon set, we provide a flexible framework for building interpretable, robust, and goal-aligned visualizations.
Figures & tables
Figure 1: Three goals for generating and exploring the Rashomon set for dimension reduction
Figure 2: PaCMAPparam embedding with and without PCA-Informed Alignment. The colored curves overlaid on the embeddings are generated by applying the learned parametric DR mapping to points sampled along the first two principal component directions in the original high-dimensional space, thereby visualizing how the DR mapping transforms the PCA axes.
Figure 3: (a) MNIST PaCMAP param embedding, (b) PCA embedding, (c) PCA-informed embedding with λPCA set to be 0.1 . It is nicely aligned with the first two principal components while capturing the detailed cluster structure. (d) Triplet PCA score has improved after PCA-informed alignment.
UMAPparam
NCVis
InfoNCE
NegtSNE
PaCMAPparam
Dataset
Diff
Before
After
Diff
Before
After
Diff
Before
After
Diff
Before
After
Diff
Before
After
AirPlane
0.07
0.67
0.73
0.23
0.71
0.93
0.28
0.65
0.93
0.19
0.73
0.92
0.12
0.77
0.89
COIL20
0.12
0.68
0.8
0.11
0.68
0.79
0.10
0.73
0.83
0.12
0.68
0.8
0.05
0.74
0.79
FMNIST
0.00
0.87
0.87
0.00
0.87
0.88
0.01
0.87
0.88
0.00
0.87
0.87
0.02
0.83
0.85
MNIST
0.04
0.75
0.79
0.03
0.76
0.79
0.02
0.78
0.8
0.02
0.76
0.79
0.02
0.74
0.76
Mammoth
0.02
0.91
0.93
0.02
0.92
0.93
0.01
0.95
0.96
-0.01
0.91
0.9
-0.03
0.92
0.89
Table 1: Triplet PCA Score for embeddings before and after PCA-alignment within different datasets, orange are the scores that slightly decrease and the teal are the scores that improve. The almost uniform improvement indicates better alignment with the principal components; the fact that the improvements are small is consistent with preservation of local structure, i.e., staying within the Rashomon set.
Figure 4: (a) Concept-informed aligned PaCMAPparam embedding. Alignment is along the horizontal axis from feet (left) to head (right). Footwear is labeled in shades of red to orange, trousers in yellow, dresses in light yellow, pullovers and coats in green, shirts and t-shirts in blue, handbags in purple. (b) Evaluation metrics and losses for FMNIST before and after concept alignment, which remain generally unchanged.
Figure 5: (a) Original PaCMAPparam embedding of USPS dataset. (b) Common knowledge embedding using only stable neighbor pairs within the Rashomon set. (c) Quantitative comparison of original vs. combined DR embeddings across three evaluation metrics for five methods.
Figure 6: (a) MNIST embedding before (left) and after (right) common NN pairs are selected, (b) Examples of rejected NN pairs when finding common knowledge through the Rashomon set. These pairs are close in pixel space (but not close along the manifold of any digit).
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Samples
# features
Tasks Types
Labels
Airplane ( Wu et al., 2015 )
24,141
3
PCA
Airplane Structure
COIL20 ( Nene et al., 1996 )
1,440
16,384
Concept, PCA
20 Objects
FICO ( FICO, 2018 )
9,861
23
Concept
External Risk Score
FMNIST ( Xiao et al., 2017 )
70,000
784
Concept, PCA
10 Clothes Type
Human Cortex ( Zhu et al., 2023 )
43,349
100
Concept, PCA
9 Cell Types
Kang et al. ( Kang et al., 2018 )
13,999
100
Concept, PCA
13 Cell Types
Appendix
Table 2: Data Descriptions and Task Types each dataset corresponds to
Figure 7: Comparison of original COIL20 embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance, triplet PCA score, random triplet PCA score, and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure.
Figure 8: Comparison of original FMNIST embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure. Random Triplet PCA score and Triplet PCA score have shown an improvement after the alignment.
Figure 9: Comparison of original Human Cortex embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure. Random Triplet PCA score and Triplet PCA score have shown an improvement after the alignment.
Figure 10: Comparison of original Kang et al. embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure. Random Triplet PCA score and Triplet PCA score have shown an improvement after the alignment.
Figure 11: Comparison of original Mammoth embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure. Random Triplet PCA score and Triplet PCA score have shown an improvement after the alignment.
Figure 12: Comparison of original Airplane embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure. Random Triplet PCA score and Triplet PCA score have shown an improvement after the alignment.
Figure 13: Comparison of original MNIST embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance, and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure.
Figure 14: Comparison of original Stuart et al . embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure. Random Triplet PCA score and Triplet PCA score have shown an improvement after the alignment.
Figure 15: Comparison of original CBMC embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure. We show that soft Jaccard distance, triplet PCA score, random triplet PCA score, and LDR (bottom) remain mostly unchanged and that aligned embeddings consistently maintain structure.
Figure 16: Comparison of original USPS embedding (left) PCA embedding (middle) and PCA informed embeddings (right) across different methods. We see alignment to principal components across all methods while preserving structure and LDRLDR (bottom) remaining mostly unchanged and that aligned embeddings consistently maintain structure. Random Triplet PCA score and Triplet PCA score have shown an improvement after the alignment.
Figure 17: Comparison of original and aligned embeddings for MNIST using a concept-aware regularizer. The embeddings’ horizontal axis is aligned with the digit index and embedding evaluation metrics haven’t changed significantly after alignment.
Figure 18: Comparison of original and aligned embeddings for FMNIST using a concept-aware regularizer. The embeddings’ horizontal axis is aligned with the label index (from head to toe as mentioned in Section 6.1 ) and embedding evaluation metrics haven’t changed significantly after alignment.
Figure 19: Comparison of original and aligned embeddings for COIL20 using a concept-aware regularizer. The embeddings’ horizontal axis is aligned with the label index and embedding evaluation metrics haven’t changed significantly after alignment.
Figure 20: Comparison of original and aligned embeddings for FICO using a concept-aware regularize. The embeddings’ horizontal axis is aligned with the external risk score. Since the label is a continuous feature, we are not displaying barplot here.
Figure 21: Comparison of original and aligned embeddings for Human Cortex Single Cell dataset using a concept-aware regularizer. The embeddings’ horizontal axis is aligned with the cell type index, and the embedding performance hasn’t been changed significantly.
Figure 22: Comparison of original and aligned embeddings for Kang et al . dataset using a concept-aware regularizer. The embeddings’ horizontal axis is aligned with the cell type index, and the embedding performance hasn’t been changed significantly.
Figure 23: Comparison of original and aligned embeddings for the Stuart dataset using a concept-aware regularizer. The embeddings’ horizontal axis is aligned with the cell type index, and the embedding performance hasn’t been changed significantly.
Figure 24: Comparison of original and aligned embeddings for the CMBC dataset using a concept-aware regularizer. The embeddings’ horizontal axis is aligned with the cell type index, and the embedding performance hasn’t been changed significantly.
Figure 25: Comparison of original and aligned embeddings for USPS dataset using a concept-aware regularizer. Clusters are shown to align with the digits index and structure preservation across multiple DR methods.
Figure 26: MNIST PaCMAPparam Embedding under different label missingness ratios (rows) and label weights (columns). High label weight with high missingness ratio breaks the original structure.
Figure 27: LDR and Jaccard Distance under different label missingness ratios.
Figure 28: MNIST embeddings and improved embedding using common knowledge. Combined DR improves structure and boosts downstream metrics.
Figure 29: FMNIST embeddings and improved embedding using common knowledge. Combined DR improves structure and boosts downstream metrics. NCVis and Neg-t-SNE and UMAP are showing a bit lower silhouette score due to the mixed area at the bottom right corner
Figure 30: USPS embeddings and improved embedding using common knowledge. Combined DR improves structure and boosts downstream metrics.
Figure 31: Kang et al. embeddings and improved embedding using common knowledge. Combined DR improves structure and boosts downstream metrics.
Figure 32: Human Cortex embeddings and improved embedding using common knowledge. Combined DR improves structure and boosts downstream metrics. NCVis and Neg-t-SNE and UMAP are showing a bit lower silhouette score due to the mixed area at the bottom right corner
Figure 33: Stuart et. al. embeddings and improved embedding using common knowledge. Combined DR improves structure and boosts downstream metrics.
Figure 34: COIL20 embeddings and improved embedding using common knowledge. Combined DR improves structure and boosts downstream metrics.
Dimensionality Reduction (DR) methods are widely used to visualize high-dimensional data. One key task in DR-based analysis is discovering neighborhoods, which relies on analyzing the fine-grained local structure of a projection. However, DR is an inherently lossy process; no technique can perfectly preserve the high-dimensional relationships, and projections therefore contain visual artifacts. In this paper, we highlight a typically overlooked source of visual artifacts: ambiguous instances. These are instances that are highly similar to multiple mutually dissimilar neighborhoods in the high-dimensional space. Standard DR methods cannot faithfully project such instances, since each data instance is mapped to a single point in the visual space. As a result, such an instance is placed in only one of its neighborhoods (or in none at all), so only part of its neighborhood structure is represented. We call this distortion partial neighborhood embedding. In this paper, we introduce a graph-based approach that identifies ambiguous instances and replicates them as multiple points in the projection, placing each copy within its respective neighborhood. We use UMAP for our results, but our approach also generalizes to other local graph-based DR techniques, and we show that our approach reveals previously hidden neighborhood memberships in projections and reduces partial neighborhood embedding across multiple examples, and is further supported by quantitative analyses.
Diede P. M. van der Hoorn, Alessio Arleo, Fernando V. Paulovich
Graphs are commonly visualized in 2D, where humans readily interpret spatial relationships, yet such layouts often distort higher-dimensional structure. We propose to embed graphs in high-dimensional space and search for informative 2D viewpoints that optimize aesthetic and readability metrics (e.g., edge crossings and angular resolution), enabled by a novel differentiable surrogate for edge crossings. Numerical experiments show that these viewpoints consistently outperform standard 2D layouts, and can even surpass methods explicitly designed to optimize these metrics. We further introduce DataFly, an interactive system for exploring multiple candidate viewpoints through seamless navigation. A usability study demonstrates that our approach reveals structural patterns that remain hidden in conventional 2D visualizations.
Ya Ji, Xuefeng Li, Timo Brand +4
Khoury College of Computer Sciences, Northeastern University, Seattle · School of Computation, Information and Technology, Technical University of Munich, Heilbronn, Germany
While UMAP is widely used for exploring high-dimensional data, typical workflows focus on its lower-dimensional embedding, largely overlooking the rich k-nearest-neighbor (kNN) graph that UMAP constructs internally. This graph encodes the data manifold in its original high-dimensional space, before the distortion that UMAP's 2D projection introduces. We demonstrate the untapped potential of this internal representation, showing how standard graph algorithms applied to this graph enhance data sensemaking: (1) PageRank identifies representative data points, (2) k-core decomposition reveals dense core regions versus sparse periphery, and (3) clustering coefficient detects tight-knit neighborhoods with highly-similar data points. Through quantitative and qualitative evaluation on MNIST and Fashion MNIST, we show that these graph-based analyses are not only practical but also competitive with or complementary to purpose-built methods (e.g., k-medoids for exemplar selection, HDBSCAN for density-based clustering).