Bi-Lipschitz Autoencoder With Injectivity Guarantee

Qipeng Zhan; Zhuoping Zhou; Zexuan Wang; Qi Long; Li Shen

cs.LGApr 8, 2026

Bi-Lipschitz Autoencoder With Injectivity Guarantee

Authors: Qipeng Zhan, Zhuoping Zhou, Zexuan Wang, Qi Long, Li Shen

Organizations: University of Pennsylvania

Abstract

Autoencoders are widely used for dimensionality reduction, based on the assumption that high-dimensional data lies on low-dimensional manifolds. Regularized autoencoders aim to preserve manifold geometry during dimensionality reduction, but existing approaches often suffer from non-injective mappings and overly rigid constraints that limit their effectiveness and robustness. In this work, we identify encoder non-injectivity as a core bottleneck that leads to poor convergence and distorted latent representations. To ensure robustness across data distributions, we formalize the concept of admissible regularization and provide sufficient conditions for its satisfaction. In this work, we propose the Bi-Lipschitz Autoencoder (BLAE), which introduces two key innovations: (1) an injective regularization scheme based on a separation criterion to eliminate pathological local minima, and (2) a bi-Lipschitz relaxation that preserves geometry and exhibits robustness to data distribution drift. Empirical results on diverse datasets show that BLAE consistently outperforms existing methods in preserving manifold structure while remaining resilient to sampling sparsity and distribution shifts. Code is available at https://github.com/qipengz/BLAE.

Figures & tables

Figure 1: Toy example demonstrating the non-injective encoder bottleneck. (a) 20 training points sampled from a V-shaped manifold with two classes (red and blue). (b) Ground truth: an ideal encoder separates the two classes in a 1D latent space. (c) (e) Reconstructed manifolds by BLAE (ours) with hidden dimensions 16 and 256, respectively. (d) (f) Corresponding latent representations learned by BLAE showing proper class separation. (g) (i) (k) Reconstructed manifolds by vanilla autoencoders with hidden dimensions 2, 16, and 256. (h) (j) (l) Corresponding latent representations showing non-injective collapse, where distant manifold regions (red vs. blue) map to overlapping latent codes, resulting in pathological local minima.

Figure 2: Loss landscapes of autoencoders on Swiss roll data. Warmer colors indicate lower loss. (a) Log-scale contour of reconstruction loss, showing the presence of local minima. (b) Log-scale contour of reconstruction loss combined with regularization, where local minima are eliminated.

Measure	BLAE	SPAE	TAE	DN	GRAE	CAE	GGAE	IRAE	GAE	Vanilla AE
$k$ -NN	1.8 $\pm$ 1.3	3.2 $\pm$ 1.3	3.8 $\pm$ 0.8	4.5 $\pm$ 2.7	4.0 $\pm$ 2.7	5.8 $\pm$ 1.8	7.5 $\pm$ 2.1	7.2 $\pm$ 0.4	9.0 $\pm$ 1.7	7.8 $\pm$ 0.8
$\text{KL}_{0.01}$	1.0 $\pm$ 0.0	3.0 $\pm$ 0.0	2.5 $\pm$ 0.9	6.0 $\pm$ 1.2	4.5 $\pm$ 1.5	7.0 $\pm$ 0.7	8.2 $\pm$ 2.5	6.8 $\pm$ 2.4	6.5 $\pm$ 1.1	9.2 $\pm$ 0.4
$\text{KL}_{0.1}$	1.0 $\pm$ 0.0	3.2 $\pm$ 1.1	2.8 $\pm$ 0.8	5.5 $\pm$ 2.2	3.5 $\pm$ 0.9	7.5 $\pm$ 0.9	9.0 $\pm$ 1.7	7.2 $\pm$ 1.6	6.5 $\pm$ 0.9	8.8 $\pm$ 0.4
$\text{KL}_{1}$	1.0 $\pm$ 0.0	4.2 $\pm$ 2.3	3.2 $\pm$ 1.3	5.2 $\pm$ 2.8	4.2 $\pm$ 1.9	7.0 $\pm$ 1.6	8.0 $\pm$ 1.6	7.2 $\pm$ 1.3	6.0 $\pm$ 2.2	8.2 $\pm$ 0.8
MSE	1.2 $\pm$ 0.4	4.8 $\pm$ 3.3	4.0 $\pm$ 0.7	5.2 $\pm$ 3.1	4.5 $\pm$ 3.0	5.5 $\pm$ 1.5	7.5 $\pm$ 2.1	7.0 $\pm$ 2.1	8.0 $\pm$ 1.6	7.2 $\pm$ 1.3
Accuracy	1	5	7	4	6	2	10	8	3	9

Table 1: Average ranks of evaluation metrics across all datasets (lower is better). Detailed metric values for each dataset are provided in Appendix B.4 and Appendix C.6 . The best is shown in bold.

Figure 3: (a) 3-D Swiss roll data. (b) Ground truth: 2-D latent representations to generate a Swiss Roll. Others: 3-D reconstruction and 2-D latent representations learned by AE methods.

Figure 4: (a) Two parallel planes: 3-D latent representation of square (blue) and heart (red) clusters. Others: 2-D latent representations learned by BLAE and other baseline methods.

Figure 5: (a) Digit ‘3’ at various scales and rotations. (b) (c) Ground truth: 2D concentric circle latent representation for uniform and non-uniform training sets. Others: 2D latent representations learned by BLAE and baseline methods on uniform (left) and non-uniform (right) training sets.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Parameter	SPAE	TAE	DN	GRAE	CAE	GGAE	IRAE	GAE
Swiss Roll	$\lambda$	2	5	100	100	0.1	1	0.01	0.01
	$\eta$	$/\penalty$	$/\penalty$	0.001	$/\penalty$	$/\penalty$	$/\penalty$	0	$/\penalty$
	n_neighbor	$/\penalty$	$/\penalty$	10	5	$/\penalty$	10	$/\penalty$	$/\penalty$
	bandwidth	$/\penalty$	$/\penalty$	$/\penalty$	$/\penalty$	$/\penalty$	2	$/\penalty$	$/\penalty$
dSprites	$\lambda$	10	0.01	0.001	0.1	0.1	0.1	0.1	0.01
	$\eta$	$/\penalty$	$/\penalty$	0.1	$/\penalty$	$/\penalty$	$/\penalty$	0	$/\penalty$

Appendix

Table 2: Hyperparameter settings for baselines across all evaluated datasets.

Datasets	Swiss Roll	dSprites	MNIST	ssREAD
$\lambda_{\text{reg}}$	1	2	30	2
$\lambda_{\text{bi-Lip}}$	0.3	0.1	0.1	0.1
$\kappa$	1	1.1	2	1.2
$\epsilon$	0.3	0.3	0.6	0.6

Appendix

Table 3: Hyperparameter settings for BLAE across all evaluated datasets.

Model	$k$ -NN( $\uparrow$ )	KL 0.01 ( $\downarrow$ )	KL 0.1 ( $\downarrow$ )	KL 1 ( $\downarrow$ )	MSE( $\downarrow$ )
BLAE	9.61e-01 $\pm$ 3.59e-03	1.28e-04 $\pm$ 2.95e-05	1.78e-05 $\pm$ 5.98e-06	1.39e-06 $\pm$ 1.02e-06	9.36e-05 $\pm$ 1.92e-05
SPAE	8.95e-01 $\pm$ 2.90e-02	7.95e-03 $\pm$ 1.13e-02	5.15e-03 $\pm$ 8.95e-03	4.73e-04 $\pm$ 8.81e-04	2.10e-03 $\pm$ 3.25e-03
TAE	8.14e-01 $\pm$ 1.54e-02	4.28e-03 $\pm$ 7.20e-04	1.57e-03 $\pm$ 6.38e-04	5.64e-05 $\pm$ 1.72e-05	6.94e-03 $\pm$ 5.07e-03
DN	7.22e-01 $\pm$ 1.58e-02	5.26e-02 $\pm$ 7.95e-03	1.32e-02 $\pm$ 7.50e-03	7.43e-04 $\pm$ 7.03e-04	1.02e-01 $\pm$ 1.73e-02
GRAE	5.70e-01 $\pm$ 3.03e-02	4.09e-02 $\pm$ 1.26e-02	9.94e-03 $\pm$ 4.36e-03	8.34e-04 $\pm$ 4.11e-04	7.72e-02 $\pm$ 2.16e-02
CAE	6.14e-01 $\pm$ 1.71e-02	6.11e-02 $\pm$ 1.66e-02	4.10e-02 $\pm$ 7.76e-03	2.31e-03 $\pm$ 4.27e-04	5.37e-02 $\pm$ 7.41e-03

Appendix

Table 4: Evaluation metrics (mean

\pm

standard deviation over 5 runs) on the Swiss Roll dataset. For MSE and KL metrics, lower values are better; for

k

-NN, higher values are better.

Model	$k$ -NN( $\uparrow$ )	KL 0.01 ( $\downarrow$ )	KL 0.1 ( $\downarrow$ )	KL 1 ( $\downarrow$ )	MSE( $\downarrow$ )
BLAE	7.39e-01 $\pm$ 5.17e-03	6.42e-03 $\pm$ 9.52e-04	3.98e-03 $\pm$ 1.84e-04	3.79e-05 $\pm$ 1.74e-06	1.69e-02 $\pm$ 1.46e-03
SPAE	7.24e-01 $\pm$ 4.60e-03	2.95e-02 $\pm$ 4.67e-04	9.11e-03 $\pm$ 5.36e-05	7.83e-05 $\pm$ 4.92e-07	1.72e-02 $\pm$ 1.02e-03
TAE	5.58e-01 $\pm$ 5.09e-02	3.15e-02 $\pm$ 2.44e-02	1.46e-02 $\pm$ 5.37e-03	1.25e-03 $\pm$ 2.23e-03	2.84e-02 $\pm$ 4.65e-03
DN	4.81e-01 $\pm$ 4.59e-02	8.41e-02 $\pm$ 4.45e-02	7.32e-02 $\pm$ 5.16e-02	6.38e-03 $\pm$ 5.67e-03	3.47e-02 $\pm$ 3.55e-03
GRAE	5.23e-01 $\pm$ 1.99e-02	2.94e-02 $\pm$ 4.97e-04	9.10e-03 $\pm$ 5.59e-05	7.84e-05 $\pm$ 4.72e-07	3.06e-02 $\pm$ 8.94e-03
CAE	6.79e-01 $\pm$ 6.31e-02	3.99e-02 $\pm$ 2.38e-02	2.05e-02 $\pm$ 2.54e-02	1.79e-03 $\pm$ 3.83e-03	2.59e-02 $\pm$ 8.76e-03

Appendix

Table 5: Evaluation metrics (mean

\pm

standard deviation over 5 runs) on the dSprites dataset. For MSE and KL metrics, lower values are better; for

k

-NN, higher values are better.

Model	$k$ -NN( $\uparrow$ )	KL 0.01 ( $\downarrow$ )	KL 0.1 ( $\downarrow$ )	KL 1 ( $\downarrow$ )	MSE( $\downarrow$ )
BLAE	9.03e-01 $\pm$ 9.38e-03	4.79e-02 $\pm$ 5.32e-03	4.01e-02 $\pm$ 1.39e-02	1.22e-02 $\pm$ 6.28e-03	2.92e-03 $\pm$ 1.53e-03
SPAE	8.59e-01 $\pm$ 3.10e-02	7.50e-02 $\pm$ 1.71e-02	6.35e-02 $\pm$ 1.42e-02	1.58e-02 $\pm$ 7.63e-03	6.91e-03 $\pm$ 2.29e-03
TAE	8.29e-01 $\pm$ 2.68e-02	5.17e-02 $\pm$ 9.51e-03	6.69e-02 $\pm$ 1.99e-02	1.64e-02 $\pm$ 8.08e-03	4.93e-03 $\pm$ 6.78e-04
DN	8.64e-01 $\pm$ 3.17e-02	1.33e-01 $\pm$ 4.48e-02	9.15e-02 $\pm$ 3.60e-02	1.49e-02 $\pm$ 6.72e-03	3.43e-03 $\pm$ 9.97e-04
GRAE	8.70e-01 $\pm$ 5.67e-02	1.11e-01 $\pm$ 5.63e-02	6.87e-02 $\pm$ 2.43e-02	1.41e-02 $\pm$ 7.06e-03	3.51e-03 $\pm$ 1.32e-03
CAE	7.69e-01 $\pm$ 3.23e-02	2.58e-01 $\pm$ 3.32e-02	1.62e-01 $\pm$ 4.66e-02	1.93e-02 $\pm$ 8.74e-03	5.54e-03 $\pm$ 8.10e-04

Appendix

Table 6: Evaluation metrics (mean

\pm

standard deviation over 5 runs) on the (Uniform) MNIST dataset. For MSE and KL metrics, lower values are better; for

k

-NN, higher values are better.

Model	$k$ -NN( $\uparrow$ )	KL 0.01 ( $\downarrow$ )	KL 0.1 ( $\downarrow$ )	KL 1 ( $\downarrow$ )	MSE( $\downarrow$ )
BLAE	8.65e-01 $\pm$ 4.50e-02	2.89e-02 $\pm$ 1.80e-02	1.66e-02 $\pm$ 2.60e-03	1.19e-03 $\pm$ 7.91e-05	3.88e-03 $\pm$ 7.87e-04
SPAE	8.22e-01 $\pm$ 1.05e-01	5.59e-02 $\pm$ 3.13e-02	6.11e-02 $\pm$ 4.39e-02	5.44e-03 $\pm$ 5.09e-03	7.78e-03 $\pm$ 2.90e-03
TAE	8.77e-01 $\pm$ 1.22e-02	3.35e-02 $\pm$ 8.59e-03	2.23e-02 $\pm$ 1.01e-03	1.32e-03 $\pm$ 1.66e-05	5.61e-03 $\pm$ 1.88e-03
DN	8.81e-01 $\pm$ 6.48e-02	7.19e-02 $\pm$ 4.58e-02	4.10e-02 $\pm$ 1.74e-02	4.23e-03 $\pm$ 2.03e-03	5.48e-03 $\pm$ 1.65e-03
GRAE	9.14e-01 $\pm$ 2.22e-02	1.06e-01 $\pm$ 1.57e-02	4.84e-02 $\pm$ 1.72e-02	5.23e-03 $\pm$ 1.30e-03	3.08e-03 $\pm$ 8.84e-04
CAE	7.58e-01 $\pm$ 4.05e-02	1.48e-01 $\pm$ 1.95e-02	1.12e-01 $\pm$ 5.71e-03	4.55e-03 $\pm$ 2.44e-04	8.50e-03 $\pm$ 1.33e-03

Appendix

Table 7: Evaluation metrics (mean

\pm

standard deviation over 5 runs) on the (non-uniform) MNIST dataset. For MSE and KL metrics, lower values are better; for

k

-NN, higher values are better.

Model	BLAE	SPAE	TAE	DN	GRAE	CAE	GGAE	IRAE	GAE	Vanilla AE
Runtime (s)	170.9	162.1	601.3	204.4	147.3	199.6	415.8	189.3	207.6	136.2

Appendix

Table 8: Runtime for training BLAE and other baseline models. All experiments were conducted on a Mac Mini equipped with an Apple M4 chip (16GB RAM).

Figure 6: 2-D latent representations of Swiss Roll data learned by BLAE and gradient-based baselines trained with different values of $b$ (0.1, 0.12, 0.15) on the Swiss Roll data. All models were trained using 1500 sample points, and all figures were plotted using the entire 10,000 sample points.

Figure 7: 2-D latent representations learned of Swiss Roll data by BLAE and graph-based baselines trained with different lengths ([-2, 10], [1, 10]) on the Swiss Roll data. All models were trained using 1500 sample points, and all figures were plotted using the entire 10,000 sample points.

Figure 8: 2-D latent representations learned by BLAE and graph-based baselines trained with different sample sizes (400, 1000, 3000) on the Swiss Roll data. All models were trained on the indicated sample sizes, while visualizations use the full set of 10,000 data points.

Sample size = 400
Measure	BLAE	GGAE	SPAE	TAE	Diffusion Net	GRAE
MSE( $\downarrow$ )	1.52e-03 $\pm$ 1.07e-04	9.69e-02 $\pm$ 7.98e-03	1.86e-02 $\pm$ 6.07e-03	5.39e-02 $\pm$ 2.96e-03	1.34e-01 $\pm$ 2.84e-02	1.80e-01 $\pm$ 3.93e-03
$k$ -NN( $\uparrow$ )	9.19e-01 $\pm$ 3.10e-03	4.55e-01 $\pm$ 2.89e-02	7.30e-01 $\pm$ 2.38e-02	6.81e-01 $\pm$ 4.18e-03	5.73e-01 $\pm$ 3.05e-02	4.76e-01 $\pm$ 6.31e-03
KL 0.01 ( $\downarrow$ )	2.49e-03 $\pm$ 2.25e-04	1.14e-01 $\pm$ 1.08e-02	2.39e-02 $\pm$ 4.56e-03	3.01e-02 $\pm$ 1.52e-03	4.41e-02 $\pm$ 1.23e-02	1.01e-01 $\pm$ 3.86e-03
KL 0.1 ( $\downarrow$ )	3.55e-04 $\pm$ 3.89e-05	1.81e-02 $\pm$ 1.84e-03	5.85e-03 $\pm$ 9.64e-04	7.37e-03 $\pm$ 3.53e-04	3.83e-02 $\pm$ 4.01e-03	2.29e-02 $\pm$ 1.56e-04
KL 1 ( $\downarrow$ )	1.78e-05 $\pm$ 2.22e-06	1.52e-03 $\pm$ 6.69e-05	2.57e-04 $\pm$ 1.94e-04	2.56e-04 $\pm$ 6.34e-06	2.21e-03 $\pm$ 7.78e-05	2.30e-03 $\pm$ 7.54e-05

Appendix

Table 9: Evaluation metrics (over 5 runs) of BLAE and graph-based baselines trained with different sample sizes (400, 1000, 3000) on the Swiss Roll data. For MSE and KL metrics, lower values are better; for

k

-NN, higher values are better. The best performance for each metric is shown in bold.

Panel of Figure 9 — Figure 9: Performance evaluation across different hyperparameter settings. (a) shows the impact of varying $\epsilon$ on $k$ -NN accuracy and error metrics. (b) demonstrates the effect of $\kappa$ on the same metrics. Error metrics are displayed on a logarithmic scale.

Panel of Figure 10 — Figure 10: Sensitivity analysis of $\kappa$ : 2-D latent representation of Swiss Roll data learned by BLAE with different $\kappa$ values.

Panel of Figure 11 — Figure 11: Sensitivity analysis of $\epsilon$ : 2-D latent representation of Swiss Roll data learned by BLAE with different $\epsilon$ values.

Figure 12: 2-D latent representation of Swiss Roll data learned by autoencoders with (a) only CAE regularization, (b) only GAE regularization, (c) CAE + injective regularizations, (d) CAE + injective regularizations.

Figure 13: 2-D latent representation of Swiss Roll data learned by autoencoders with (a) only injective regularization, (b) only SPAE regularization, (c) injective + 1-bi-Lipschitz regularizations, (d) SPAE + 1-bi-Lipschitz regularizations, (e) only 1-bi-Lipschitz regularization.

Figure 14: Left: Single cell type in the AD00109 dataset from the ssREAD database. Others: 2-D latent representations learned by BLAE and other baseline methods.

Model	Latent dimension = 2	Latent dimension = 32
BLAE	0.9250 $\pm$ 0.0033	0.9626 $\pm$ 0.0013
SPAE	$0.9146\pm 0.0206$	$0.9576\pm 0.0019$
TAE	$0.9107\pm 0.0121$	$0.9524\pm 0.0026$
DN	$0.9167\pm 0.0064$	$0.4200\pm 0.0821$
GRAE	$0.9131\pm 0.0119$	$0.9476\pm 0.0022$
CAE	$0.9222\pm 0.0030$	$0.9517\pm 0.0025$

Appendix

Table 10: Classification accuracy (over 5 runs) of SVM trained on latent representations learned by different models. BLAE achieves the highest accuracy, demonstrating better preservation of cell-type-specific information in the latent space. The best performance is shown in bold.

Model	Recon MSE $\downarrow$	FID $\downarrow$	KL	MIG $\uparrow$
VAE	1.42e-1 $\pm$ 3.65e-3	1.23e-2 $\pm$ 2.32e-3	3.06e+0 $\pm$ 1.33e-2	5.34e-1 $\pm$ 1.96e-1
BL-VAE	7.09e-2 $\pm$ 1.41e-3	1.15e-2 $\pm$ 1.15e-3	5.58e+0 $\pm$ 4.06e-2	8.28e-1 $\pm$ 1.93e-2

Appendix

Table 11: Swiss Roll Results (Mean

\pm

Std over 5 seeds)

Figure 15: Extended toy example demonstrating the non-injective encoder bottleneck. (a) 200 training points sampled from a V-shaped manifold with two classes (red and blue). (b) Ground truth: an ideal encoder separates the two classes in a 1D latent space. (c) Reconstructed manifolds by Vanilla AE with hidden dimension 16. (d) Corresponding latent representations learned by Vanilla AE. (e) Reconstructed manifolds by BLAE with hidden dimension 16. (f) Corresponding latent representations learned by BLAE show proper class separation.

Autoencoders

Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.

See all 145 papers

Manifold-Matching Autoencoders
Mar 17, 2026Laurent Cheret, Vincent Létourneau, Isar Nejadgholi +3Multidimensional Scaling Autoencoders
Department of Computer Science, University of Ottawa, Ottawa, Canada · MILA, Montréal, Canada · National Research Council of Canada, Ottawa, Canada
Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction
Aug 30, 2026Louen Pottier, Louis Lesueur, Anders ThorinAutoencoders Manifold Learning
CEA List, Université Paris-Saclay · Palaiseau, F-91120, France · LMS, École Polytechnique, Institut Polytechnique de Paris +1
Geometric regularization of autoencoders via observed stochastic dynamics
Apr 17, 2026Sean Hill, Felix X. -F. YeLatent Dynamics Modeling Autoencoders
Department of Mathematics and Statistics, University at Albany, SUNY

Mar 17, 2026J/K move · Enter open · S save

Manifold-Matching Autoencoders

Laurent Cheret, Vincent Létourneau, Isar Nejadgholi, Chris Drummond+2

Department of Computer Science, University of Ottawa, Ottawa, Canada · MILA, Montréal, Canada · National Research Council of Canada, Ottawa, Canada

Multidimensional Scaling Autoencoders Latent-Space Regularization Geometric Representation Learning Dimensionality Reduction Latent Space Alignment+1

Open paper AskPDF arXiv

We study a simple unsupervised regularization scheme for autoencoders called Manifold-Matching (MMAE): we align the pairwise distances in the latent space to those of the input data space by minimizing mean squared error. Because alignment occurs on pairwise distances rather than coordinates, it can also be extended to a lower-dimensional representation of the data, adding flexibility to the method. We find that this regularization outperforms similar methods on metrics based on preservation of nearest-neighbor distances and persistent homology-based measures. We also observe that MMAE provides a scalable approximation of Multi-Dimensional Scaling (MDS).

Abstract

Autoencoders

Explore similar work

Manifold-Matching Autoencoders

Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction

Geometric regularization of autoencoders via observed stochastic dynamics