Latent Variable Models

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 105

Apr 19, 2026stat.ML

StrEBM: A Structured Latent Energy-Based Model for Blind Source Separation

This paper proposes StrEBM, a structured latent energy-based model for source-wise structured representation learning. The framework is motivated by a broader goal of promoting identifiable and decoupled latent organization by assigning different latent dimensions their own learnable structural biases, rather than constraining the entire latent representation with a single shared energy. In this sense, blind source separation is adopted here as a concrete and verifiable testbed, through which the evolution of latent dimensions toward distinct underlying components can be directly examined. In the proposed framework, latent trajectories are optimized directly together with an observation-generation map and source-wise structural parameters. Each latent dimension is associated with its own energy-based formulation, allowing different latent components to gradually evolve toward distinct source-like roles during training. In the present study, this source-wise energy design is instantiated using Gaussian-process-inspired energies with learnable length-scales, but the framework itself is not restricted to Gaussian processes and is intended as a more general structured latent EBM formulation. Experiments on synthetic multichannel signals under linear and nonlinear mixing settings show that the proposed model can recover source components effectively, providing an initial empirical validation of the framework. At the same time, the study reveals important optimization characteristics, including slow late-stage convergence and reduced stability under nonlinear observation mappings. These findings not only clarify the practical behavior of the current GP-based instantiation, but also establish a basis for future investigation of richer source-wise energy families and more robust nonlinear optimization strategies.

Yuan-Hao Wei

Disentangled Representation Learning Latent Variable Models Energy-Based Models

Apr 18, 2026cs.LG

Covariance-Based Structural Equation Modeling in Small-Sample Settings with $p>n$

Factor-based Structural Equation Modeling (SEM) relies on likelihood-based estimation assuming a nonsingular sample covariance matrix, which breaks down in small-sample settings with

p>n

. To address this, we propose a novel estimation principle that reformulates the covariance structure into self-covariance and cross-covariance components. The resulting framework defines a likelihood-based feasible set combined with a relative error constraint, enabling stable estimation in small-sample settings where

p>n

for sign and direction. Experiments on synthetic and real-world data show improved stability, particularly in recovering the sign and direction of structural parameters. These results extend covariance-based SEM to small-sample settings and provide practically useful directional information for decision-making.

Hiroki Hasegawa, Aoba Tamura, Yukihiko Okada

Graduate School of Science and Technology, University of Tsukuba, Tsukuba, Ibaraki, Japan · Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Ibaraki, Japan · Tsukuba Institute for Advanced Research, University of Tsukuba, Tsukuba, Ibaraki, Japan +1

Latent Variable Models

Feb 4, 2026cs.CL

Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induction heads, function vectors, and the Hydra effect. Some of these individual phenomena have been independently tied to different data distributional properties, while some have been loosely associated with model architecture and how Transformers process information. However, a unified understanding of the relationship between data, model architecture, and optimization remains lacking, failing to answer the fundamental question: why do these three phenomena appear universally across different model families and scales, despite their seeming disconnect? In this work, we answer this question by unifying these three phenomena as consequences of hierarchical latent structures in the data generation process, coupled with decorrelated gradients across additive model components and directional concavity in the representation geometry. We validate our theoretical results in a toy model regime and in a large-scale synthetic data regime, comparing them with language models trained on natural language data.

Jonas Rohweder, Subhabrata Dutta, Iryna Gurevych

UKP Lab, TU Darmstadt

Transformer Interpretability Latent Variable Models Induction Heads

Feb 4, 2026cs.CL

LLM surprisal is necessary but not sufficient to capture English garden-path effects: Evidence from joint latent modeling of reading paradigms

Temporarily ambiguous garden-path sentences ("While the team trained the striker wondered... ") are known to cause processing difficulty, which can manifest itself in a variety of reading behaviors (in-situ slowdowns, rereading), as well as in miscomprehension or outright rejection of the sentence as ungrammatical. Which types of reading behavior are observed critically depends on the experimental method used to collect the data, which makes comparing results between reading paradigms difficult. To address this problem, we present a latent-process multinomial processing tree (MPT) model of human reading and comprehension/judgment behavior in garden-path sentences that we fit to combined data from four different reading paradigms (eye tracking, uni- and bidirectional self-paced reading, Maze). The model distinguishes between the probability of adopting an incorrect initial analysis, the cost of encountering an incompatible continuation, and the cost of syntactic reanalysis. By taking into account trials with inattentive reading, more realistic estimates of the cost parameters are obtained. Cross-validation reveals that the MPT model has a better predictive fit to human reading patterns and end-of-trial task data than a model based solely on LLM-derived surprisal values. We also test several models that assume an influence of surprisal within the MPT architecture, and find that adding surprisal as an additional predictor or reading time and/or garden-path cost further improves predictive fit.

Dario Paape, Tal Linzen, Shravan Vasishth

Department of Linguistics, University of Potsdam · Department Linguistik, Universität Potsdam, Karl-Liebknecht-Straße 24–25, D-14476, Germany. · Center for Data Science/Department of Linguistics, New York University

Latent Variable Models Cognitive Modeling Computational Psycholinguistics

Dec 25, 2025stat.ML

Nonnegative matrix factorizations and related compositional models: Equivalence, identifiability, and an application on the grain-size analysis of sediments

Across fields such as machine learning, social science, and geology, considerable attention has been given to models that factorize a nonnegative matrix into the product of two or three matrices, subject to nonnegative or row-sum-to-1 constraints. Although these models are to a large extent similar or even equivalent, they are presented under different names, and their similarity is not well known. This paper highlights similarities among five models, latent budget analysis (LBA) and latent class analysis (LCA) from social science, end-member analysis (EMA) from geology, probabilistic latent semantic analysis (PLSA) and nonnegative matrix factorization (NMF) from machine learning. We focus on the identifiability of these models. We prove that the solution of LBA, EMA, LCA, PLSA is unique if and only if the solution of NMF is unique. Consequently, existing uniqueness theorems for NMF directly apply to LBA, EMA, LCA, PLSA, and vice versa. We also provide a brief review of algorithms for the estimation of these models. We illustrate NMF on a sedimentary grain-size distribution dataset from sedimentary geology, and end the paper with a discussion of closely related model: archetypal analysis.

Qianqian Qi, Peter G. M. van der Heijden, Maarten A. Prins

Hangzhou Dianzi University, China · Utrecht University, the Netherlands · University of Southampton, UK +1

Latent Variable Models Parameter Identifiability Non-Negative Matrix Factorization

Dec 12, 2025cond-mat.stat-mech

Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling

We show that nonequilibrium dynamics can play a constructive role in unsupervised machine learning by inducing the spontaneous emergence of latent-state cycles. We introduce a model in which visible and hidden variables interact through two independently parametrized transition matrices, defining a Markov chain whose steady state is intrinsically out of equilibrium. Likelihood maximization drives this system toward nonequilibrium steady states with finite entropy production, reduced self-transition probabilities, and persistent probability currents in the latent space. These cycles are not imposed by the architecture but arise from training, and models that develop them reproduce the empirical distribution of data classes more faithfully, with a clear correlation between agreement with the data and entropy production. Compared with equilibrium approaches such as restricted Boltzmann machines, our model breaks the detailed balance between the forward and backward conditional transitions and relies on a log-likelihood gradient that depends explicitly on the last two steps of the Markov chain. Hence, this exploration of the interface between nonequilibrium statistical physics and modern machine learning suggests that introducing irreversibility into latent-variable models can improve the fidelity of the generated data distribution.

Marco Baiesi, Alberto Rosso

Department of Physics and Astronomy, University of Padova, Via Marzolo 8, I-35131 Padova, Italy · INFN, Sezione di Padova, Via Marzolo 8, I-35131 Padova, Italy · LPTMS, CNRS, Universit´e Paris-Saclay, 91405, Orsay, France

Latent Variable Models Latent Dynamics Modeling Markov Models Generative Modeling

Dec 7, 2025stat.ME

Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length

The proliferation of Large Language Models (LLMs) necessitates valid evaluation methods to provide guidance for both downstream applications and actionable future improvements. The Item Response Theory (IRT) model with Computerized Adaptive Testing has recently emerged as a promising framework for evaluating LLMs via their response accuracy. Beyond simple response accuracy, LLMs' chain of thought (CoT) lengths serve as a vital indicator of their reasoning ability. To leverage the CoT length information to assist LLM evaluation, we propose the \textbf{La}tency-\textbf{R}esponse \textbf{T}heory (LaRT) model, which jointly models both the response accuracy and CoT length by introducing a key correlation parameter between the latent ability and the latent speed. We derive an efficient stochastic approximation Expectation-Maximization algorithm for parameter estimation. We establish rigorous identifiability results for the latent ability and latent speed parameters to ensure the statistical validity of their estimation. Through both theoretical asymptotic analyses and simulation studies, we demonstrate LaRT's advantages over IRT in terms of superior estimation accuracy and shorter confidence intervals for latent trait estimation. To evaluate LaRT in real data, we collect responses from diverse LLMs on popular benchmark datasets. We find that LaRT yields different LLM rankings than IRT and outperforms IRT across multiple key evaluation metrics including predictive power, item efficiency, ranking validity, and LLM evaluation efficiency. Code and data are available at https://github.com/Toby-X/Latency-Response-Theory-Model

Zhiyu Xu, Jia Liu, Yixin Wang +1

Department of Statistics, Columbia University. · Department of Statistics, University of Michigan

LLM Evaluation Latent Variable Models CoT Evaluation Item Response Theory

Nov 22, 2025stat.ML

Improving Forecasts of Suicide Attempts for Patients with Little Data

Ecological Momentary Assessment (EMA) studies provide real-time data on suicidal thoughts and behaviors, but forecasting suicide attempts remains challenging: attempts are rare, and the pathways patients take to them are heterogeneous. Here, we investigate a cohort of patients from an EMA study with recorded suicide-related events. We show that a single model fit to all patients forecasts poorly, while idiographic (per-patient) models show improvement but overfit for those with little data. Based on this result, one may hypothesize that patients should be partitioned into subgroups---this way, similar patients' data can be pooled together to improve forecasts. However, we show that grouping patients at random already improves forecasts, with performance increasing monotonically with the number of groups. Moreover, we show that grouping patients by demographics yields worse forecasts than random groupings. From these results, we hypothesize that patient similarity is continuous, rather than discrete, and must be inferred from the data. This motivated us to use Latent Variable Multiple Output Gaussian Processes (LVMOGPs), adapted to our data. Preliminary results show that, even without careful kernel design, LVMOGPs already match the strongest baseline models on most metrics, and their latent spaces yield a similarity between patients that we can inspect directly. Because the cohort is conditioned on the outcome and the splits are not temporal, we read these results as evidence that idiographic structure exists and can be recovered, not as deployable forecasting performance---an area for future work.

Genesis Hang, Annie Chen, Hope Neveux +2

Wellesley College · Harvard University

Latent Variable Models Gaussian Processes Clinical Prediction Suicide Risk Modeling Clinical Risk Prediction

Oct 14, 2025stat.ML

Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamics

We develop interacting particle algorithms for learning latent variable models with energy-based priors. To do so, we leverage recent developments in particle-based methods for solving maximum marginal likelihood estimation (MMLE) problems. Specifically, we provide a continuous-time framework for learning latent energy-based models, by defining stochastic differential equations (SDEs) that provably solve the MMLE problem. We obtain a practical algorithm as a discretisation of these SDEs and provide theoretical guarantees for the convergence of the proposed algorithm. Finally, we empirically validate the effectiveness of our method on synthetic and image datasets and demonstrate that using a particle based approach offers significant improvement in computational efficiency.

Joanna Marks, Tim Y. J. Wang, O. Deniz Akyildiz

Department of Mathematics, Imperial College London

Parameter Estimation Latent Variable Models Energy-Based Models

Jun 20, 2025cs.LG

Variational Learning of Disentangled Representations

Disentangled representations separate factors that are shared across conditions from those that are condition-specific. Such separation is needed for generalization to new domains, treatments, patients, or species. A dominant line of work pursues this goal through variational formulations. While these approaches achieve partial disentanglement, they often exhibit three common limitations: they either do not remove all condition-specific information from the condition-specific representation, allow the condition-specific representation to become uninformative, or impose independence assumptions that do not reflect the underlying generative process. In this work, we introduce DisCoVR, a variational framework that addresses these limitations. Its objective is aligned with the probabilistic structure of the data-generating process, and includes an adversarial term that prevents condition-specific information from being encoded in the condition-specific representation.DisCoVR reconstructs the data from both shared and condition-specific representations, ensuring that each remains informative, and uses a structured prior that further reinforces the informativeness of both representations. We show that across synthetic, image, and single-cell RNA-sequencing datasets, DisCoVR achieves stronger disentanglement compared to previous approaches.

Yuli Slavutsky, Ozgur Beker, David Blei +1

Department of Statistics, Columbia University, New York City, USA · Irving Institute for Cancer Dynamics, Columbia University, New York City, USA · Department of Computer Science, Columbia University, New York City, USA +1

Variational Autoencoders Disentangled Representation Learning Latent Variable Models

Feb 11, 2025stat.ML

The observational partial order of causal structures with latent variables

For two causal structures with the same set of visible variables, one is said to observationally dominate the other if the set of distributions over the visible variables realizable by the first contains the set of distributions over the visible variables realizable by the second. Knowing such dominance relations is useful for adjudicating between these structures given observational data. Here, we consider the problem of determining the partial order of equivalence classes of causal structures with latent variables relative to observational dominance. We provide a complete characterization of the dominance order in the case of three visible variables, and a partial characterization in the case of four visible variables. Our techniques also help to identify which observational equivalence classes have a set of realizable distributions that is characterized by nontrivial inequality constraints, analogous to Bell inequalities and instrumental inequalities. We find evidence that as one increases the number of visible variables, the equivalence classes satisfying nontrivial inequality constraints become ubiquitous. (Because such classes are the ones for which there can be a difference in the distributions that are quantumly and classically realizable, this implies that the potential for quantum-classical gaps is also ubiquitous.) Furthermore, we find evidence that constraint-based causal discovery algorithms that rely solely on conditional independence constraints have a significantly weaker distinguishing power among observational equivalence classes than algorithms that go beyond these (i.e., algorithms that also leverage nested Markov constraints and inequality constraints).

Marina Maciel Ansanelli, Elie Wolfe, Robert W. Spekkens

Perimeter Institute for Theoretical Physics, 31 Caroline Street North, Waterloo, Ontario Canada N2L 2Y5 · Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario, Canada, N2L 3G1

Latent Variable Models Causal Discovery Causal Structure Learning Causal Discovery with Latent Confounding

May 29, 2024cs.LG

Hierarchical Bayesian Crowdsourcing with Item Difficulty

In applied statistics and machine learning, the gold standards used for training are often biased and almost always noisy. Dawid and Skene's justifiably popular crowdsourcing model adjusts for rater sensitivity and specificity, but fails to capture distributional properties of rating data gathered for training, which in turn biases training. In this study, we introduce a general purpose measurement-error model with which we can infer consensus categories by adding item-level effects for difficulty, discriminativeness, and guessability. We further show how to constrain the bimodal posterior of these models to avoid adversarial raters. We validate our model's goodness of fit with posterior predictive checks, the Bayesian analogue of

χ^2

tests, and assess its predictive accuracy using leave-one-out cross-validation. We illustrate our new model with two well-studied data sets, binary rating data for caries in dental X-rays and implication in natural language.

Seong Woo Han, Ozan Adıgüzel, Bob Carpenter

University of Pennsylvania · Columbia University · Center for Computational Mathematics, Flatiron Institute

Inter-Rater Reliability Latent Variable Models Bayesian Inference Hierarchical Bayesian Modeling Learning with Noisy Labels

Dec 12, 2023cs.LG

Interpretable factorization of clinical questionnaires to identify latent factors of psychopathology

Psychiatry research seeks to understand the manifestations of psychopathology in behavior, as measured in questionnaire data, by identifying a small number of latent factors that explain them. While factor analysis is the canonical tool for this purpose, the resulting factors may not be interpretable, and may also be subject to confounding variables. Moreover, missing data are common, and explicit imputation is often required. To overcome these limitations, we introduce Interpretability Constrained Questionnaire Factorization (ICQF), a non-negative matrix factorization method with regularization tailored for questionnaire data. Our method aims to promote factor interpretability and solution stability. We provide an optimization procedure with theoretical convergence guarantees, and an automated procedure to determine latent dimensionality accurately. We validate these procedures using realistic synthetic data. We demonstrate the effectiveness of our method in a widely used general-purpose questionnaire, in two independent datasets (the Healthy Brain Network and Adolescent Brain Cognitive Development studies). Specifically, we show that ICQF preserves diagnostic information across a range of disorders, outperforming competing methods for smaller dataset sizes, and improves interpretability, as assessed by our clinical research collaborators and co-authors. This suggests that the regularization in our method matches domain characteristics, in addition to satisfying qualitative desiderata.

Ka Chun Lam, Francisco Pereira, Bridget W Mahony +1

Machine Learning Core National Institute of Mental Health, National Institutes of Health · Section on Developmental Neurogenomics, Human Genetics Branch National Institute of Mental Health, National Institutes of Health

Latent Variable Models Non-Negative Matrix Factorization Mental Health Psychometric Validation Matrix Factorization

Nov 13, 2023cs.LG

Causal Discovery in Mixtures of Populations

Causal discovery aims to learn causal structures up to certain symmetries. Diverse populations or changing environments give rise to heterogeneous data in the following sense: each population/environment is a ``source'' which idiosyncratically determines the forms of causal effects. From this perspective, the source is a latent common cause for every observed variable. While some methods for causal discovery can work around latent confounding in special cases, a global confounder poses a significant challenge. The only known ways to deal with latent global confounding involve making assumptions that limit structural equations and/or noise functions. We demonstrate that globally confounded causal structures can still be identified with arbitrary structural equations and noise functions, so long as the number of latent classes remains small relative to the size and sparsity of the underlying DAG. The approach relies on agglomerating variables into large-enough matrices of moments, whose ranks directly reveal graphical properties of the causal structure. We also provide a statistical test to test the rank of these matrices.

Bijan Mazaheri, Spencer Gordon, Yuval Rabani +1

Thayer School of Engineering, Dartmouth College, Hanover, New Hampshire, USA · Fragment Data Technologies, San Francisco, California, USA · Department of Computer Science, Hebrew Univeristy of Jerusalem, Jerusalem, Israel +1

Latent Variable Models Causal Discovery Causal Identifiability Causal Discovery with Latent Confounding

Date pendingstat.ME

Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effects

Mixed-effects models are widely used to model data with complex grouping structures and high-cardinality categorical predictor variables. However, for high-dimensional crossed random effects, current standard computations relying on Cholesky decompositions can become prohibitively slow. In this work, we present Krylov subspace-based methods that address existing computational bottlenecks, and we analyze them both theoretically and empirically. In particular, we derive new results on the convergence and accuracy of the preconditioned stochastic Lanczos quadrature and conjugate gradient methods for mixed-effects models, and we develop scalable methods for calculating predictive variances. In experiments with simulated and real-world data, the proposed methods yield speedups of several orders of magnitude and are more computationally robust than Cholesky-based computations, while maintaining essentially the same accuracy.

Pascal Kündig, Fabio Sigrist

Lucerne University of Applied Sciences and Arts · University of Basel · Seminar for Statistics, ETH Zurich

Latent Variable Models Krylov Subspace Methods Mixed-Effects Models