StablePCA: Distributionally Robust Learning of Shared Representations from Multi-Source Data

Zhenyu Wang; Molei Liu; Jing Lei; Francis Bach; Zijian Guo

cs.LGMay 2, 2025

StablePCA: Distributionally Robust Learning of Shared Representations from Multi-Source Data

Authors: Zhenyu Wang, Molei Liu, Jing Lei, Francis Bach, Zijian Guo

Organizations: Department of Statistics, Rutgers University · Department of Biostatistics, Peking University Health Science Center; Beijing International Center for Mathematical Research · Department of Statistics, Carnegie Mellon University · Inria, École Normale Supérieure, PSL Research University ·

Abstract

When synthesizing multi-source high-dimensional data, a key objective is to extract low-dimensional representations that effectively approximate the original features across different sources. Such representations facilitate the discovery of transferable structures and help mitigate systematic biases such as batch effects. We introduce Stable Principal Component Analysis (StablePCA), a distributionally robust framework for constructing stable latent representations by maximizing the worst-case explained variance over multiple sources. A primary challenge in extending classical PCA to the multi-source setting lies in the nonconvex rank constraint, which renders the StablePCA formulation a nonconvex optimization problem. To overcome this challenge, we conduct a convex relaxation of StablePCA and develop an efficient Mirror-Prox algorithm to solve the relaxed problem, with global convergence guarantees. Since the relaxed problem generally differs from the original formulation, we further introduce a data-dependent certificate to assess how well the algorithm solves the original nonconvex problem and establish the condition under which the relaxation is tight. Finally, we explore alternative distributionally robust formulations of multi-source PCA based on different loss functions.

Figures & tables

Figure 1: Comparison of subspaces yielded by StablePCA and PooledPCA in four realizations of latent factor models ( 10 ). Ellipses show two-standard-deviation covariance contours, and dots are sampled observations for each source.

$d\backslash n$	100	300	600	1200	2500	5000	10000	20000	40000
$10$	-0.0001	0.0018	0.0019	0.0024	0.0010	-0.0002	0.0027	0.0004	0.0002
$20$	-0.0015	-0.0008	-0.0006	-0.0005	-0.0004	-0.0003	-0.0002	-0.0003	-0.0003
$30$	0.0017	0.0002	0.0009	-0.0014	-0.0024	-0.0014	-0.0017	-0.0028	-0.0021

Table 1: Summary of the values of the certificate

\tau

across samples sizes

n\in\{100,300,...,40000\}

and ambient dimensions

d\in\{10,20,30\}

. We set the number of iterations

T=500

in Algorithm 1 . The table reports mean values averaged over 100 replications.

Figure 2: Comparison of subspaces yielded by StablePCA, SquaredPCA, and FairPCA subspaces for the same setups as Figure 1 . Ellipses show two-standard-deviation covariance contours, and dots are sampled observations, for each source.

Method	$d=30$	$d=50$	$d=70$	$d=100$	$d=200$	$d=300$
MP	0.263±0.023	0.324±0.035	0.511± 0.035	0.706±0.067	1.86±0.04	4.49±0.07
SDP	0.057±0.007	0.281±0.135	0.519±0.057	9.99±5.68	42.96±22.66	173.6±70.2
SDP/MP	0.22	0.89	1.02	14.25	23.07	38.79

Table 2: Comparison of the proposed Mirror-Prox (MP) algorithm and the SDP-based method ( Samadi et al., 2018 ) for solving FairPCA. We report the average runtime (in seconds) and standard deviations for each method in 100 independent replications, across varying ambient dimensions

d\in\{30,50,70,100,200,300\}

. The ratio SDP/MP is defined as the averaged runtime of the SDP method divided by that of Mirror-Prox.

Figure 3: Comparison of StablePCA, PooledPCA, SquaredPCA, and FairPCA across different numbers of source domains $L\in\{2,4,6,8,10\}$ . (a) Recovery error $\|\widehat{P}-VV^{\intercal}\|_{F}$ of the shared subspace. (b) Worst-case explained variance among the training (in-distribution) sources. (c) Worst-case explained variance among new (out-of-distribution) sources. Results are averaged over 100 replications, with error bars representing one standard error.

Figure 4: Worst-case explained variance of methods on the single-cell RNA dataset. (a) Worst-case explained variance among the training batches (in-distribution). (b) Worst-case explained variance among the held-out batches (out-of-distribution). Each boxplot summarizes the results across 200 random train-test splits, and the red stars mark the lowest worst-case explained variance observed among the 200 splits.

Figure 5: t-SNE and UMAP visualizations of the StablePCA embedding with subspace dimension $k=50$ on the single-cell RNA dataset. Top row: cells colored by 12 experimental batches. Bottom row: cells colored by cell types (B cell, Monocyte, NK cell, and T cell).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: StablePCA and PooledPCA under model ( 48 ). The ellipses show the three sources. StablePCA recovers the shared subspace ${\rm span}(e_{1})$ , whereas PooledPCA selects an orthogonal direction.

Figure 8: Finite-sample convergence behavior of Fantope-relaxed StablePCA across varying sample sizes $n$ over $[100,40000]$ and ambient dimensions $d\in\{10,30\}$ . (a) Objective gap $\min_{l}\langle\Sigma^{(l)},M^{*}\rangle-\min_{l}\langle\Sigma^{(l)},\widehat{M}_{T}\rangle$ . (b) Estimation error $\|\widehat{M}-M^{*}\|_{F}$ . Results are averaged over 20 replications, with error bars representing one standard deviation.

Figure 9: Comparison of StablePCA, PooledPCA, SquaredPCA, and FairPCA across different numbers of source domains $L\in\{2,4,6,8,10\}$ , when the methods are fitted with $k=5$ . (a) Subspace capture error $1-\langle VV^{\intercal},\widehat{P}\rangle/3$ of the shared biological subspace, which replaces the recovery error of Figure 3 since ${\rm rank}(\widehat{P})=5>3$ ; (b) and (c) as in Figure 3 .

Figure 10: Comparison of StablePCA, PooledPCA, SquaredPCA, and FairPCA across different numbers of source domains $L\in\{2,4,6,8,10\}$ , when the methods are fitted with $k=10$ . (a) Subspace capture error $1-\langle VV^{\intercal},\widehat{P}\rangle/3$ of the shared biological subspace, which replaces the recovery error of Figure 3 since ${\rm rank}(\widehat{P})=10>3$ ; (b) and (c) as in Figure 3 .

Panel of Figure 11 — Figure 11: Worst-case explained variance ratio achieved by StablePCA, PooledPCA, SquaredPCA, and FairPCA on the single-cell RNA dataset with $k=100$ .

Figure 13: t-SNE and UMAP visualizations of the StablePCA embedding with subspace dimension $k=25$ on the single-cell RNA dataset. Top row: cells colored by 12 experimental batches. Bottom row: cells colored by cell types (B cell, Monocyte, NK cell, and T cell).

Figure 14: t-SNE and UMAP visualizations of the StablePCA embedding with subspace dimension $k=75$ on the single-cell RNA dataset. Top row: cells colored by 12 experimental batches. Bottom row: cells colored by cell types (B cell, Monocyte, NK cell, and T cell).

Principal Component Analysis

Principal component analysis (PCA) finds orthogonal directions of greatest variance for data compression, visualization, denoising, and feature discovery.

See all 67 papers

A Robust Optimization Approach to Sparse Principal Component Analysis
Jun 2, 2026David Vävinggren, Francis Bach, André M. H. Teixeira +2Principal Component Analysis Robust Optimization
Uppsala University, Sweden · PSL Research University / INRIA, France · Science for Life Laboratory, Sweden
Mean-Shift PCA by Knockoff Mean
May 25, 2026Mengda Li, Zeng Li, Jianfeng YaoDistribution Shift Robustness Principal Component Analysis
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, China
Anchor PCA
Jun 4, 2026Benedikt Seiter, Anya Fries, Julius von Kügelgen +1Unsupervised Learning Principal Component Analysis
ETH Zürich

Jun 2, 2026J/K move · Enter open · S save

A Robust Optimization Approach to Sparse Principal Component Analysis

David Vävinggren, Francis Bach, André M. H. Teixeira, Dave Zachariah+1

Uppsala University, Sweden · PSL Research University / INRIA, France · Science for Life Laboratory, Sweden

Principal Component Analysis Robust Optimization Dimensionality Reduction

Open paper AskPDF arXiv

While principal component analysis (PCA) is a fundamental tool for dimensionality reduction, its dense representations make it ill-suited for high-dimensional data. Existing methods address this by promoting sparsity through explicit

\ell_1

-penalties, but these are not obvious to tune due to the unsupervised nature of the task. In contrast, we propose Adversarial PCA (AdvPCA), which leverages robust optimization to achieve sparsity by optimizing the reconstruction objective against bounded, worst-case latent space perturbations. We show that this formulation admits a closed-form reduction, leading to a practical iterative algorithm that alternates between adversarial linear regression-style updates for the sparse encoder and orthogonal updates for the decoder. By theoretically characterizing the solution, we derive a data-adaptive parameterization that allows the algorithm to perform effectively out of the box. We validate these claims through numerical experiments on synthetic and real-world genomics data.

Abstract

Principal Component Analysis

Explore similar work

A Robust Optimization Approach to Sparse Principal Component Analysis

Mean-Shift PCA by Knockoff Mean

Anchor PCA