Manifold-Preserving Superpixel Hierarchies and Embeddings for the Exploration of High-Dimensional Images
Authors: Alexander Vieth, Boudewijn Lelieveldt, Elmar Eisemann, Anna Vilanova, Thomas Höllt
Organizations: Leiden University Medical Center, Leiden, The Netherlands · Delft University of Technology, Delft, The Netherlands · Eindhoven University of Technology, Eindhoven, The Netherlands
High-dimensional images, or images with a high-dimensional attribute vector per pixel, are commonly explored with coordinated views of a low-dimensional embedding of the attribute space and a conventional image representation. Nowadays, such images can easily contain several million pixels. For such large datasets, hierarchical embedding techniques are better suited to represent the high-dimensional attribute space than flat dimensionality reduction methods. However, available hierarchical dimensionality reduction methods construct the hierarchy purely based on the attribute information and ignore the spatial layout of pixels in the images. This impedes the exploration of regions of interest in the image space, since there is no congruence between a region of interest in image space and the associated attribute abstractions in the hierarchy. In this paper, we present a superpixel hierarchy for high-dimensional images that takes the high-dimensional attribute manifold into account during construction. Through this, our method enables consistent exploration of high-dimensional images in both image and attribute space. We show the effectiveness of this new image-guided hierarchy in the context of embedding exploration by comparing it with classical hierarchical embedding-based image exploration in two use cases.
Figures & tables
Figure 1: Image Hierarchies: Classical image-space-based hierarchies, like image pyramids, progressively blur and subsample the image (a). Attribute-space-based hierarchies, as used in hierarchical DR methods (b), ignore the image space entirely and only aim to preserve manifold structure of the attribute space. A pixel, as highlighted in the hierarchy, might actually be represented by multiple landmarks in abstraction levels. And, in turn, a landmark can represent scattered pixels. Combined, this complicates an image-based exploration of the high-dimensional space. Our superpixel hierarchy (c) combines both image layout and attribute space manifold structure. and color two superpixels on the middle abstraction level, the respective coarse-grained vertices in the attribute graph and corresponding embedding points.
Figure 2: Graph structures : 4-connected image graph I (left) and attribute-based graph G (right). Different neighborhoods of the same node 1 are highlighted in grey.
Figure 3: Method overview: Each image pixel is associated with a high-dimensional attribute vector (top left). (1) We compute a neighborhood graph G , whose vertices correspond to pixels and edges are based on attribute similarities. The data level embedding is computed from this graph, like in, e.g., t-SNE. (2) We compute a feature vector per vertex, describing the local graph structure, using random walks. (3) For the next abstraction level, vertices in the attribute graph G are merged with the most similar neighbor in I . (4) The new vertex retains all outgoing connections of the merged vertices. All vertex features are added and re-normalized. (5) This is repeated for each vertex. (6) Similarities between merged vertices are used for creating embeddings where each point corresponds to one superpixel (right).
Figure 4: Indian Pines Exploration: (a) a false color image based on data channels 20 (587 nm, red), 76 (1090 nm, green) and 130 (1591 nm, blue) with a ROI marked in red. (b) and (c) show the 4th abstraction level embedding of our superpixel embedding and HSNE, respectively; (d) and (e) highlight the superpixels and landmarks which correspond to the ROI. (f) and (g) show refined embeddings of the highlighted subsets on a lower abstraction level. (h) and (i) recolor a cutout of the image based on the refined embeddings using an overlaid 2D colormap Finally, (j) indicates the pixels in the full images which are represented by the HSNE refinement, whereas our superpixel refinement extends only slightly around the ROI.
Figure 5: Non-exact refinement as discussed in Section 4.3 . Here, using γ=0.01 can improve the embedding structure by including additional superpixels not in the ROI (c.f. Figure 4 ).
Figure 6: CyCIF Data. (a) Grayscale image based on two structural markers and DNA with two selections highlighted, corresponding to two clusters in the level 4 superpixel embedding (b, c).
Figure 7: Hierarchical Embeddings on CyCIF data: (a-d) Embeddings on several levels of abstraction and corresponding recolored images, using the colormapping discussed in Section 5.1 . (e, f) Zoom in on a ROI as highlighted in (b), recolored according to the embedding and with random gray values to better distinguish between superpixels, respectively.
Pines [ 4 ]
Salinas [ 29 ]
Method
AUE
AEV
AUE
AEV
FH [ 11 ]
11.17
10.45
5.66
1.75
ERS [ 27 ]
15.46
12.78
8.46
3.93
SLIC [ 1 ]
13.72
13.11
5.16
4.15
BB [ 2 ]
16.25
9.15
10.27
4.53
SPH geodesic (ours)
13.06
8.92
7.35
1.56
Table 1: Explained Variation and Undersegmentation Error: normalized area under the UE and (1 - EV) value curves in % , lower is better for both AUE and AEV.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Explained Variation and Undersegmentation Error: For Pines data.
Figure 9: Explained Variation and Undersegmentation Error: For Salinas data.
Pines [ 4 ]
Salinas [ 29 ]
Method
log AUE
log AEV
log AUE
log AEV
FH [ 11 ]
53.04
50.48
26.85
19.28
ERS [ 27 ]
52.64
47.80
41.66
37.30
SLIC [ 1 ]
50.09
43.16
32.95
27.75
BB [ 2 ]
50.71
39.32
39.14
24.62
SPH geodesic (ours)
46.56
38.94
34.05
25.86
Appendix
Table 2: Explained Variation and Undersegmentation Error: normalized area under the log 10 -plotted UE and (1 - EV) value curves in % , i.e. the area under the lines in s 8 and 9 . Lower values are better for both. Note, that the FH curves do not start at 1, which skews these aggregate measure in its favor.
Figure 10: Indian Pines Ground Truth: Ground Truth segmentation and labels with map overlay.
Figure 11: Salinas Ground Truth: Ground Truth segmentation. White is background.
Figure 12: Indian Pines comparison: Segmentations on a selection of representative levels for all evaluated methods. Each superpixel is randomly assigned a gray-value for distinction.
Figure 13: Salinas comparison: Segmentations on a selection of representative levels for all evaluated methods. Each superpixel is randomly assigned a gray-value for distinction.
Figure 14: CyCIF Focus Region: As used in section 6.2. DNA corresponds to the Hoechst channel.
Figure 15: Indian Pines superpixel hierarchy: Data level embedding alongside seven superpixel abstraction level embedding. Each level shows a false color image based on the average spectrum per superpixel of channel 20 (587 nm, red), 76 (1090 nm, green) and 130 (1591 nm, blue). Next to them are recolorings of the superpixel based on the embedding layout using a 2D colormap which is superimposed on the respective embeddings (as shown in Figure 4 . Superpixels are hardly visible in these down-scaled version of the originally 614×2,678 (w x h) images, but more abstract levels clearly show more and more high-level structure. The percentages indicates the reduction of components, i.e., the number of components (superpixels) in level 1 reduces to 27.28% of the previous level. In contrast to Figure 17 , embeddings starting from level 1 are initialized randomly.
Figure 16: Geodesic distance approach: RGB image of a bus (top left) and 6 levels of abstraction. Superpixels are recolored with the average color of all the image pixels they contain. Below, embeddings of each level using the same coloring. The embeddings starting from level 1 are initialized based on the previous embedding. Numbers of components: 19090 (166x115 pixels), 5518, 1433, 356, 86, 21, 6. Photograph (downscaled by factor 3) of the bus from Fig. 6 in [ 52 ] , available at https://arxiv.org/abs/1605.06325 .
Figure 17: Indian Pines: Embeddings and image space recolorings (using a 2D colormap as shown in 4 ) for (a) t-SNE and (b) UMAP probabilities. The embeddings starting from level 1 are initialized based on the previous embedding.
Superpixels offer a compact image representation by grouping pixels into coherent regions. Recent methods have reached a plateau in terms of segmentation accuracy by generating noisy superpixel shapes. Moreover, most existing approaches produce a single fixed-scale partition that limits their use in vision pipelines that would benefit multi-scale representations. In this work, we introduce H-SPAM (Hierarchical Superpixel Anything Model), a unified framework for generating accurate, regular, and perfectly nested hierarchical superpixels. Starting from a fine partition, guided by deep features and external object priors, H-SPAM constructs the hierarchy through a two-phase region merging process that first preserves object consistency and then allows controlled inter-object grouping. The hierarchy can also be modulated using visual attention maps or user input to preserve important regions longer in the hierarchy. Experiments on standard benchmarks show that H-SPAM strongly outperforms existing hierarchical methods in both accuracy and regularity, while performing on par with most recent state-of-the-art non-hierarchical methods. Code and pretrained models are available: https://github.com/waldo-j/hspam.
Julien Walther, Rémi Giraud, Michaël Clément
Univ. Bordeaux, CNRS, Bordeaux INP, IMS, UMR 5218, France · Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, France
We propose a multi-dimensional persistent sheaf Laplacian (MPSL) framework on simplicial complexes for image analysis. The proposed method is motivated by the strong sensitivity of commonly used dimensionality reduction techniques, such as principal component analysis (PCA), to the choice of reduced dimension. Rather than selecting a single reduced dimension or averaging results across dimensions, we exploit complementary advantages of multiple reduced dimensions. At a given dimension, image samples are regarded as simplicial complexes, and persistent sheaf Laplacians are utilized to extract a multiscale localized topological spectral representation for individual image samples. Statistical summaries of the resulting spectra are then aggregated across scales and dimensions to form multiscale multi-dimensional image representations. We evaluate the proposed framework on the COIL20 and ETH80 image datasets using standard classification protocols. Experimental results show that the proposed method provides more stable performance across a wide range of reduced dimensions and achieves consistent improvements to PCA-based baselines in moderate dimensional regimes.
Xiang Xiang Wang, Guo-Wei Wei
Department of Mathematics, Michigan State University, East Lansing, MI 48824, USA. · Department of Biochemistry and Molecular Biology, Michigan State University, East Lansing, MI 48824, USA. · Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI 48824, USA.
Dimension reduction (DR) is inherently non-unique: multiple embeddings can preserve the structure of high-dimensional data equally well while differing in layout or geometry. In this paper, we formally define the Rashomon set for DR -- the collection of `good' embedding -- and show how embracing this multiplicity leads to more powerful and trustworthy representations. Specifically, we pursue three goals. First, we introduce PCA-informed alignment to steer embeddings toward principal components, making axes interpretable without distorting local neighborhoods. Second, we design concept-alignment regularization that aligns an embedding dimension with external knowledge, such as class labels or user-defined concepts. Third, we propose a method to extract common knowledge across the Rashomon set by identifying trustworthy and persistent nearest-neighbor relationships, which we use to construct refined embeddings with improved local structure while preserving global relationships. By moving beyond a single embedding and leveraging the Rashomon set, we provide a flexible framework for building interpretable, robust, and goal-aligned visualizations.