math.NAJun 13, 2025

Deep Symmetric Autoencoders from the Eckart-Young-Schmidt Perspective

Authors: Simone Brivio, Nicola Rares Franco

Organizations: †MOX, Department of Mathematics, Politecnico di Milano, Milan, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deep autoencoders have become a fundamental tool in various machine learning applications, ranging from dimensionality reduction and reduced order modeling of partial differential equations to anomaly detection and neural machine translation. Despite their empirical success, a solid theoretical foundation for their expressiveness remains elusive, particularly when compared to classical projection-based techniques. In this work, we aim to take a step forward in this direction by presenting a comprehensive analysis of what we refer to as symmetric autoencoders, a broad class of deep learning architectures that have surfaced frequently in the recent literature. Specifically, we introduce a formal distinction between different classes of symmetric architectures, analyzing their strengths and limitations from a mathematical perspective. For instance, we show that the reconstruction error of symmetric autoencoders with orthonormality constraints can be understood by leveraging the well-renowned Eckart-Young-Schmidt (EYS) theorem. As a byproduct of our analysis, we end up developing the EYS initialization strategy for symmetric autoencoders, which is based on an iterated application of the Singular Value Decomposition (SVD). To validate our findings, we conduct a series of numerical experiments where we benchmark our proposal against conventional deep autoencoders, discussing the importance of model design and initialization.

Figures & tables

Figure 1: Schematic representation of a deep symmetric autoencoder.
Figure 2: Visualization of the (α,β)(\alpha,\beta) -LeakyReLU and θ\theta -Hyperbolic activations for different values of their hyperparameters.
Figure 3: Pictorial representation of the EYS initialization and its relationship with the three manifolds, of decreasing dimension, representing the three (nested) classes of symmetric autoencoder architectures: SAE , SBAE and SOAE .
Figure 4: Random samples of u\mathbf{u} for the three test cases considered in this work: PGA (top), ELS (middle), and ROD (bottom). We denote with μ\bm{\mu} the collection of problem parameters associated to each simulation.
Case studyParameterRangeDescription
PGAμ\mu[0.3, 0.7]wave position
ELSρ\rho[0.5, 2]body force
λ\lambda[0.9, 1.1]first Lamé coefficient
μ\mu[0.5, 1]second Lamé coefficient
mm[1.0, 1.5]mass of the falling object
δ\delta[0.05, 0.1]size of the falling object
Table 1: Problem parameters for the three case studies considered in the numerical experiments. The variability of such parameters is responsible of the stochasticity of the high-dimensional datum u\mathbf{u} .
Figure 5: Comparison between the EYS ( blue ) and the He initialization ( red ) for the PGA case study. Models are compared in terms of the initial MSE over the validation set. Results refer to SAE s with HypActθHypAct_{\theta} activation of varying sharpness ∠=∠(θ)\angle=\angle(\theta) . Left: two-level architecture {n0,n1,n2}\{n_{0},n_{1},n_{2}\} with n1=20n_{1}=20 and varying latent dimension n2n_{2} . Right: SAE s of variable depth but fixed latent dimension, obtained following the pattern {n0,65,3},{n0,65,5,3},…,{n0,65,33,17,9,5,3}\{n_{0},65,3\},\{n_{0},65,5,3\},\ldots,\{n_{0},65,33,17,9,5,3\} . NB: to account for the stochasticity of the He initialization, we repeatedly initialized the architectures 100 times and only kept the trial reporting the best performance.
Figure 6: Visualization of the training dynamics and computational performance of AE , SAE , SBAE , and SOAE , for all the test cases. We compare our proposed initialization (solid line) with the literature standard (dashed line) . The reported result refer to the LeakyReLUα,5/4\textnormal{LeakyReLU}_{\alpha,5/4} activation, with ∠(α)=0.5\angle(\alpha)=0.5 . The validation MSE is computed on normalized data.We refer the reader to Table 2 for the specifics on the employed architectures.
Test casePGAELSROD
Skeleton {nj}j=0l\{n_{j}\}_{j=0}^{l}{514, 64, 15, 3}{882, 15, 10, 8}{4347, 15, 7, 5}
Table 2: Architecture skeletons for the comparative study. We use the criteria proposed in [ 12 ] to set the latent dimension, while we use the error bounds developed in this paper to heuristically set the other levels’ complexity.
   MSEMRE
HypActθ\text{HypAct}_{\theta}∠(θ)=0.5\angle(\theta)=0.5AE(1.09±0.16)×102(1.09\pm 0.16)\times 10^{2}(1.36±0.07)×10−3(1.36\pm 0.07)\times 10^{-3}
SAE(1.25±0.15)×102(1.25\pm 0.15)\times 10^{2}(1.52±0.07)×10−3(1.52\pm 0.07)\times 10^{-3}
SBAE(5.28±0.76)×101(5.28\pm 0.76)\times 10^{1}(9.54±0.50)×10−4(9.54\pm 0.50)\times 10^{-4}
SOAE(1.54±0.22)×102(1.54\pm 0.22)\times 10^{2}(1.62±0.09)×10−3(1.62\pm 0.09)\times 10^{-3}
∠(θ)=3.0\angle(\theta)=3.0AE(2.65±0.52)×102(2.65\pm 0.52)\times 10^{2}(2.09±0.13)×10−3(2.09\pm 0.13)\times 10^{-3}
SAE(1.28±0.14)×102(1.28\pm 0.14)\times 10^{2}(1.56±0.06)×10−3(1.56\pm 0.06)\times 10^{-3}
Table 3: Average test reconstruction accuracy (with 95% confidence interval) of AE , SAE , SBAE , and SOAE for the ROD test case, when considering the EYS initialization, but varying the activation function and the sharpness parameter.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Comparison between the EYS ( blue ) and the standard random initialization ( red ) for the PGA case study. Top: SOAE s with HypActθHypAct_{\theta} . Middle: SAE s with LeakyReLUα,5/4LeakyReLU_{\alpha,5/4} . Bottom: SOAE s with LeakyReLUα,5/4LeakyReLU_{\alpha,5/4} .
   MSEMRE
HypActθ\text{HypAct}_{\theta}∠(θ)=0.5\angle(\theta)=0.5AE(7.55±1.90)×10−3(7.55\pm 1.90)\times 10^{-3}(1.30±0.07)×10−2(1.30\pm 0.07)\times 10^{-2}
SAE(7.13±1.23)×10−3(7.13\pm 1.23)\times 10^{-3}(1.33±0.06)×10−2(1.33\pm 0.06)\times 10^{-2}
SBAE(1.21±0.20)×10−2(1.21\pm 0.20)\times 10^{-2}(1.75±0.07)×10−2(1.75\pm 0.07)\times 10^{-2}
SOAE(3.27±0.23)×100(3.27\pm 0.23)\times 10^{0}(3.02±0.09)×10−1(3.02\pm 0.09)\times 10^{-1}
∠(θ)=3.0\angle(\theta)=3.0AE(1.81±0.41)×10−3(1.81\pm 0.41)\times 10^{-3}(6.53±0.32)×10−3(6.53\pm 0.32)\times 10^{-3}
SAE(1.79±0.35)×10−3(1.79\pm 0.35)\times 10^{-3}(6.60±0.31)×10−3(6.60\pm 0.31)\times 10^{-3}
Appendix
Table 4: PGA test case, with EYS initialization (see Section D.2 ).
   MSEMRE
HypActθ\text{HypAct}_{\theta}∠(θ)=0.5\angle(\theta)=0.5AE(7.86±0.48)×10−3(7.86\pm 0.48)\times 10^{-3}(1.42±0.06)×10−2(1.42\pm 0.06)\times 10^{-2}
SAE(2.67±0.21)×10−3(2.67\pm 0.21)\times 10^{-3}(7.93±0.25)×10−3(7.93\pm 0.25)\times 10^{-3}
SBAE(3.10±0.28)×10−3(3.10\pm 0.28)\times 10^{-3}(8.26±0.29)×10−3(8.26\pm 0.29)\times 10^{-3}
SOAE(2.62±0.22)×10−3(2.62\pm 0.22)\times 10^{-3}(7.70±0.26)×10−3(7.70\pm 0.26)\times 10^{-3}
∠(θ)=3.0\angle(\theta)=3.0AE(1.24±0.47)×10−2(1.24\pm 0.47)\times 10^{-2}(1.46±0.06)×10−2(1.46\pm 0.06)\times 10^{-2}
SAE(2.95±0.23)×10−3(2.95\pm 0.23)\times 10^{-3}(8.15±0.27)×10−3(8.15\pm 0.27)\times 10^{-3}
Appendix
Table 5: ELS test case, with EYS initialization (see Section D.2 ).
Figure 8: Visualization of the training dynamics, with LeakyReLUα,5/4LeakyReLU_{\alpha,5/4} , with α\alpha such that ∠(α)=3.0\angle(\alpha)=3.0 .
Figure 9: Visualization of the training dynamics, with HypAct(θ)HypAct(\theta) , with θ\theta such that ∠(θ)=0.5\angle(\theta)=0.5 .
Figure 10: Visualization of the training dynamics, with HypAct(θ)HypAct(\theta) , with θ\theta such that ∠(θ)=3.0\angle(\theta)=3.0 .
PGARODELS
[514,64,3][514,64,3][4347,15,5][4347,15,5][882,8,5][882,8,5]
[514,64,15,3][514,64,15,3][4347,15,7,5][4347,15,7,5][882,15,10,8][882,15,10,8]
[514,64,15,7,3][514,64,15,7,3][4347,20,7][4347,20,7][882,15,10][882,15,10]
[514,32,15,3][514,32,15,3][4347,30,20,10,5][4347,30,20,10,5][882,10,8][882,10,8]
[514,16,10,3][514,16,10,3][4347,30,10,5][4347,30,10,5][882,30,15,10,8][882,30,15,10,8]
[514,64,15,5][514,64,15,5][4347,20,10,7][4347,20,10,7][882,30,10][882,30,10]
Appendix
Table 6: Architecture skeletons for the hyperparameter sweep of Section D.4 .
Figure 11: For any choice of the design and training hyperparameters, we compare the validation MSE of the best (among 3 runs) EYS- and standard- initialized architectures after training (first column). The POD baseline having the same latent dimension as the trained SAE is reported for reference (second and third columns).
Figure 12: We report the results of the analysis of the sample size sensitivity study, using continuous and dashed lines to indicate architectures initialized with EYS and the standard strategy, respectively. The POD baseline is added for reference. We report the median value (among the 5 different runs), along with min-max bands (whenever they do not degrade the visualization).

Explore similar work

CardsList

  1. Orthogonal Dendritic Intrinsic Networks: An Architecture for Significance-Ordered, Orthogonal Latent Spaces

    Jul 6, 2026Jeanie Schreiber, Tyrus Berry, Zeeshan AhmedPrincipal Component AnalysisAutoencoder Architectures

    Department of Mathematical Sciences George Mason University Fairfax, VA 22030, USA · Sensor Science Division, Physical Measurement Laboratory NIST Gaithersburg, MD 20899, USA

  2. Sharp Reconstruction Bounds for Autoencoders Using the Same Forward Map

    Sep 17, 2026Patricia Medina, Hy P. G. LamReconstruction ErrorAutoencoder Architectures

    New York City College of Technology, CUNY, Brooklyn, NY, USA; The Graduate Center, CUNY, New York, NY, USA. · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA, USA.

  3. Aligned Training: A Parameter-Free Method to Improve Feature Quality and Stability of Sparse Autoencoders (SAE)

    May 18, 2026Michał Brzozowski, Neo Christopher ChungSparse Autoencoder FeaturesFeature Learning

    Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland