Organizations: Department of Statistics, Rutgers University · Department of Biostatistics, Peking University Health Science Center; Beijing International Center for Mathematical Research · Department of Statistics, Carnegie Mellon University · Inria, École Normale Supérieure, PSL Research University · Center for Data Science, Zhejiang University.
When synthesizing multi-source high-dimensional data, a key objective is to extract low-dimensional representations that effectively approximate the original features across different sources. Such representations facilitate the discovery of transferable structures and help mitigate systematic biases such as batch effects. We introduce Stable Principal Component Analysis (StablePCA), a distributionally robust framework for constructing stable latent representations by maximizing the worst-case explained variance over multiple sources. A primary challenge in extending classical PCA to the multi-source setting lies in the nonconvex rank constraint, which renders the StablePCA formulation a nonconvex optimization problem. To overcome this challenge, we conduct a convex relaxation of StablePCA and develop an efficient Mirror-Prox algorithm to solve the relaxed problem, with global convergence guarantees. Since the relaxed problem generally differs from the original formulation, we further introduce a data-dependent certificate to assess how well the algorithm solves the original nonconvex problem and establish the condition under which the relaxation is tight. Finally, we explore alternative distributionally robust formulations of multi-source PCA based on different loss functions.
Figures & tables
Figure 1: Comparison of subspaces yielded by StablePCA and PooledPCA in four realizations of latent factor models ( 10 ). Ellipses show two-standard-deviation covariance contours, and dots are sampled observations for each source.
d\n
100
300
600
1200
2500
5000
10000
20000
40000
10
-0.0001
0.0018
0.0019
0.0024
0.0010
-0.0002
0.0027
0.0004
0.0002
20
-0.0015
-0.0008
-0.0006
-0.0005
-0.0004
-0.0003
-0.0002
-0.0003
-0.0003
30
0.0017
0.0002
0.0009
-0.0014
-0.0024
-0.0014
-0.0017
-0.0028
-0.0021
Table 1: Summary of the values of the certificate τ across samples sizes n∈{100,300,...,40000} and ambient dimensions d∈{10,20,30} . We set the number of iterations T=500 in Algorithm 1 . The table reports mean values averaged over 100 replications.
Figure 2: Comparison of subspaces yielded by StablePCA, SquaredPCA, and FairPCA subspaces for the same setups as Figure 1 . Ellipses show two-standard-deviation covariance contours, and dots are sampled observations, for each source.
Method
d=30
d=50
d=70
d=100
d=200
d=300
MP
0.263±0.023
0.324±0.035
0.511± 0.035
0.706±0.067
1.86±0.04
4.49±0.07
SDP
0.057±0.007
0.281±0.135
0.519±0.057
9.99±5.68
42.96±22.66
173.6±70.2
SDP/MP
0.22
0.89
1.02
14.25
23.07
38.79
Table 2: Comparison of the proposed Mirror-Prox (MP) algorithm and the SDP-based method ( Samadi et al., 2018 ) for solving FairPCA. We report the average runtime (in seconds) and standard deviations for each method in 100 independent replications, across varying ambient dimensions d∈{30,50,70,100,200,300} . The ratio SDP/MP is defined as the averaged runtime of the SDP method divided by that of Mirror-Prox.
Figure 3: Comparison of StablePCA, PooledPCA, SquaredPCA, and FairPCA across different numbers of source domains L∈{2,4,6,8,10} . (a) Recovery error ∥P−VV⊺∥F of the shared subspace. (b) Worst-case explained variance among the training (in-distribution) sources. (c) Worst-case explained variance among new (out-of-distribution) sources. Results are averaged over 100 replications, with error bars representing one standard error.
Figure 4: Worst-case explained variance of methods on the single-cell RNA dataset. (a) Worst-case explained variance among the training batches (in-distribution). (b) Worst-case explained variance among the held-out batches (out-of-distribution). Each boxplot summarizes the results across 200 random train-test splits, and the red stars mark the lowest worst-case explained variance observed among the 200 splits.
Figure 5: t-SNE and UMAP visualizations of the StablePCA embedding with subspace dimension k=50 on the single-cell RNA dataset. Top row: cells colored by 12 experimental batches. Bottom row: cells colored by cell types (B cell, Monocyte, NK cell, and T cell).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: StablePCA and PooledPCA under model ( 48 ). The ellipses show the three sources. StablePCA recovers the shared subspace span(e1) , whereas PooledPCA selects an orthogonal direction.
Figure 8: Finite-sample convergence behavior of Fantope-relaxed StablePCA across varying sample sizes n over [100,40000] and ambient dimensions d∈{10,30} . (a) Objective gap minl⟨Σ(l),M∗⟩−minl⟨Σ(l),MT⟩ . (b) Estimation error ∥M−M∗∥F . Results are averaged over 20 replications, with error bars representing one standard deviation.
Figure 9: Comparison of StablePCA, PooledPCA, SquaredPCA, and FairPCA across different numbers of source domains L∈{2,4,6,8,10} , when the methods are fitted with k=5 . (a) Subspace capture error 1−⟨VV⊺,P⟩/3 of the shared biological subspace, which replaces the recovery error of Figure 3 since rank(P)=5>3 ; (b) and (c) as in Figure 3 .
Figure 10: Comparison of StablePCA, PooledPCA, SquaredPCA, and FairPCA across different numbers of source domains L∈{2,4,6,8,10} , when the methods are fitted with k=10 . (a) Subspace capture error 1−⟨VV⊺,P⟩/3 of the shared biological subspace, which replaces the recovery error of Figure 3 since rank(P)=10>3 ; (b) and (c) as in Figure 3 .
Figure 11: Worst-case explained variance ratio achieved by StablePCA, PooledPCA, SquaredPCA, and FairPCA on the single-cell RNA dataset with k=100 .
Figure 13: t-SNE and UMAP visualizations of the StablePCA embedding with subspace dimension k=25 on the single-cell RNA dataset. Top row: cells colored by 12 experimental batches. Bottom row: cells colored by cell types (B cell, Monocyte, NK cell, and T cell).
Figure 14: t-SNE and UMAP visualizations of the StablePCA embedding with subspace dimension k=75 on the single-cell RNA dataset. Top row: cells colored by 12 experimental batches. Bottom row: cells colored by cell types (B cell, Monocyte, NK cell, and T cell).
Principal component analysis (PCA) finds orthogonal directions of greatest variance for data compression, visualization, denoising, and feature discovery.
While principal component analysis (PCA) is a fundamental tool for dimensionality reduction, its dense representations make it ill-suited for high-dimensional data. Existing methods address this by promoting sparsity through explicit ℓ1-penalties, but these are not obvious to tune due to the unsupervised nature of the task. In contrast, we propose Adversarial PCA (AdvPCA), which leverages robust optimization to achieve sparsity by optimizing the reconstruction objective against bounded, worst-case latent space perturbations. We show that this formulation admits a closed-form reduction, leading to a practical iterative algorithm that alternates between adversarial linear regression-style updates for the sparse encoder and orthogonal updates for the decoder. By theoretically characterizing the solution, we derive a data-adaptive parameterization that allows the algorithm to perform effectively out of the box. We validate these claims through numerical experiments on synthetic and real-world genomics data.
David Vävinggren, Francis Bach, André M. H. Teixeira +2
Uppsala University, Sweden · PSL Research University / INRIA, France · Science for Life Laboratory, Sweden
Removing noise is difficult, but adding noise is easy. In this work, we show how to eliminate mean-shift noisy components from PCA by deliberately introducing knockoff mean-shift perturbation. Standard PCA is highly sensitive to shifts in the sample mean: a small fraction of samples from a shifted distribution can cause large deviations in the leading principal components. In high-dimensional regimes, existing Robust PCA approaches cannot handle the mean-shift contamination structure inherent in the mixture model. Using tools from Random Matrix Theory, we prove that the mean-shift spikes are spectrally separable from the stable eigenvalues of the original covariance. Furthermore, the original eigenspace remains asymptotically invariant to the contamination, independent of the mixture weight. Exploiting this spectral stability, we propose a simple, two-stage PCA algorithm by adding knockoff mean that identifies and removes the mean-shift component using only standard PCA operations.
Mengda Li, Zeng Li, Jianfeng Yao
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, China
Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques. We study PCA for data from multiple related domains. Since principal components generally differ across domains, one way to obtain a shared low-rank embedding is to perform PCA on the pooled data. However, this approach can focus on spurious directions that exhibit high variation in only a few domains. To find a robust embedding that still explains most variance in unseen but similar domains, we propose instead to focus on shared directions of variation. To this end, we introduce Anchor PCA which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. Anchor PCA amounts to PCA on a modified target matrix and thus can be solved efficiently. Moreover, we show that Anchor PCA recovers a maximal invariant subspace and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations. On simulated and real-world gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA recovers the maximally invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. Taken together, these findings establish Anchor PCA as a promising approach to robust unsupervised dimension reduction from multi-domain data.
Benedikt Seiter, Anya Fries, Julius von Kügelgen +1