Recovering Latent Structures after Variational Bayesian Variable Selection: Fit Assessment and Factor-Number Selection in Partially Exploratory Factor Analysis

Whether an additional general dimension is necessary beyond correlated first-order factors is a property of the population covariance matrix, not of any estimator or design. This research establishes when that property can be decided. Where the general and group loadings are proportional within every cluster the bifactor structure is covariance-equivalent to correlated factors, so no sample size separates them (Proposition 1); where that proportionality fails in every cluster, three items per cluster and some mild regularities leave no $K$-factor model with diagonal uniquenesses able to reproduce the covariance matrix (Theorem 1); and between them lies a mixed boundary, located numerically here and turning on cluster resistance. Distinguishability is therefore graded, measured by the population distance to the $K$-factor class. Because that question is conditional on a first-order structure which is itself uncertain, a two-step procedure is developed within partially exploratory factor analysis, delivering a structure only when it reproduces across adjacent counts and treating non-delivery as legitimate. Simulation shows that a unanimous count can accompany a structure that fails to reproduce, and that absorbed local dependence can imitate a general factor, the error growing with sample size while stability indicators stay clean. Four empirical datasets illustrate the possible outcomes.

The marginal likelihood, also known as the evidence, is regarded as a mathematical embodiment of Occam's razor, enabling model selection that avoids overfitting. The evidence lower bound (ELBO) objective from variational inference has also been used for similar purposes. Prior work has shown that restricting the approximate posterior family via a mean-field approximation can lead the ELBO to underfit. In this paper, we show how ELBO-based hyperparameter learning in a simple over-parameterized regression model can also produce overfitting, depending on the assumed rank of the covariance matrix in a Gaussian approximate posterior. Surprisingly, among only the underfit and overfit options, Bayesian model selection via the evidence itself sometimes prefers the overfit version, while the ELBO does not. Bayesian practitioners hoping to scale to large models should be cautious about how reduced-rank assumptions needed for tractability may impact the potential for model selection.

We present Variance-Preserving Orthogonal Selection (VPOS), an unsupervised feature-selection method that performs sequential orthogonal deflation in the variance-weighted principal component analysis (PCA) loading space $\mathbf{V}_d\mathbfΛ_d^{1/2}$. After each feature is selected, its loading direction is projected out of all remaining candidates, so subsequent selections cover complementary directions of the rank-$d$ covariance approximation while returning original variables. We establish rank-reduction guarantees and a determinant-growth interpretation, and distinguish VPOS from greedy selection on raw data, unweighted eigenvector pivoting, Principal Feature Analysis (PFA), and Principal Variable Selection (PVS). Experiments enforce $k\leq d$, tune method-specific parameters on validation observations, and evaluate on unseen outer folds. Across seven labelled benchmarks, VPOS improves held-out normalised reconstruction error over matched PCA without deflation on every dataset, with reductions of 1--78%. It obtains the lowest mean reconstruction error on Wine, Breast Cancer, and MNIST and is within 1.7% of the lowest error on CIFAR-10 and HighDim. On CIFAR-10, VPOS is approximately 24$\times$ faster than the closely related PVS baseline while incurring a 1.7% reconstruction gap. These results establish VPOS as an efficient covariance-coverage method, particularly when correlated high-dimensional data must be represented by a small set of identifiable original variables.