Understanding When Poisson Log-Normal Models Outperform Penalized Poisson Regression for Microbiome Count Data

Daniel Agyapong; Julien Chiquet; Jane Marks; Toby Dylan Hocking

cs.LGApr 4, 2026

Understanding When Poisson Log-Normal Models Outperform Penalized Poisson Regression for Microbiome Count Data

Authors: Daniel Agyapong, Julien Chiquet, Jane Marks, Toby Dylan Hocking

Organizations: School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, Arizona, U.S.A. · UMR MIA Paris-Saclay, Université Paris-Saclay, AgroParisTech, INRAE, Palaiseau, France · Department of Biological Sciences, Northern Arizona University, Flagstaff, Arizona, U.S.A. · Département d’informatique, Université de Sherbrooke, Sherbrooke, Québec, Canada

Abstract

Multivariate count models are often justified by their ability to capture latent dependence, but researchers receive little guidance on when this added structure improves on simpler penalized marginal Poisson regression. We study this question using real microbiome data under a unified held-out evaluation framework. For count prediction, we compare PLN and GLMNet(Poisson) on 20 datasets spanning 32 to 18,270 samples and 24 to 257 taxa, using held-out Poisson deviance under leave-one-taxon-out prediction with 3-fold sample cross-validation rather than synthetic or in-sample criteria. For network inference, we compare PLNNetwork and GLMNet(Poisson) neighborhood selection on five publicly available datasets with experimentally validated microbial interaction truth. PLN outperforms GLMNet(Poisson) on most count-prediction datasets, with gains up to 38 percent. The primary predictor of the winner is the sample-to-taxon ratio, with mean absolute correlation as the strongest secondary signal and overdispersion as an additional predictor. PLNNetwork performs best on broad undirected interaction benchmarks, whereas GLMNet(Poisson) is better aligned with local or directional effects. Taken together, these results provide guidance for choosing between latent multivariate count models and penalized Poisson regression in biological count prediction and interaction recovery.

Figures & tables

Study (year)	Data / setting	Algorithms tested	Evaluation metrics
Kenney et al. (2021)	Simulations; microbiome example	Poisson PCA, PLN	Subspace recovery (principal angles); robustness to noise and exposure; runtime
Chiquet et al. (2021)	Multiple ecology/microbiome case studies	PLN, PLNPCA, PLNNetwork, PLNMixture	BIC/ICL for model order selection; illustrative fits across tasks
Xiao et al. (2022)	Synthetic count networks; real scRNA-seq data	PLNet, VPLN, latentcor, glasso	AUPR/AUC for edge recovery; runtime
Virta and Artemiou (2023)	Simulations; real count matrices	Poisson PCA, vectorization PCA, matrix-normal methods	Subspace estimation error (principal angles); latent-rank selection accuracy; runtime
Qian et al. (2024)	Synthetic microbial communities (5 sizes, 4 topologies)	PLN, MIC, Pearson, MA	3-class accuracy for interaction type recovery; topology and size effects
Batardière et al. (2025)	High-zero simulations; cow microbiome	ZIPLN, PLN	In-sample log-likelihood (ELBO/BIC/AIC); latent dispersion; group discrimination

Table 1: Summary of prior benchmarks of PLN and Poisson algorithms.

Figure 1: Conceptual contrast between earlier evaluation practice and the count-prediction protocol used in this study. Previous work often assessed count models with in-sample fit criteria computed on the observed abundance matrix. Our benchmark instead withholds a target taxon, predicts its counts out of sample from the remaining taxa, and scores those predictions with held-out Poisson deviance under leave-one-taxon-out cross-validation.

Panel of Figure 2 — Figure 2: Dataset-level winner as a function of $N/D$ and MAC across the 20-dataset real-data benchmark. Blue circles: PLN achieves lower held-out Poisson deviance. Red squares: GLMNet(Poisson) achieves lower held-out Poisson deviance. The orange triangle marks American Gut 1, a GLMNet win at low $N/D$ and high MAC that runs counter to the general pattern.

Figure 3: Representative datasets illustrating both directions of the PLN vs. GLMNet(Poisson) comparison. Top: datasets where PLN achieves lower held-out Poisson deviance. Bottom: datasets where GLMNet(Poisson) achieves lower held-out Poisson deviance. Each panel reports held-out Poisson deviance (mean $\pm$ SE across three outer cross-validation folds) for PLN, GLMNet(Poisson), and the featureless baseline. The annotated difference and $p$ -value are from the paired comparison of PLN and GLMNet(Poisson) across folds.

Figure 4: Edge-recovery F1 score on five experimental ground-truth datasets for PLNNetwork and GLMNet(Poisson) . Datasets are grouped by ground-truth type: broad edge recovery (OMM12, OMM12 keystone 2023, PairInteraX) and local/directional (butyrate assembly 2021, host fitness 2018). Bars show mean F1 score over $B=20$ bootstrap resamples of the abundance matrix; error bars are $\pm 1$ SE.

Figure 5: Ground-truth and inferred networks for the OMM12 defined community (12 taxa). Each panel shows the experimental ground truth (left), the PLNNetwork prediction (centre), and the GLMNet(Poisson) prediction (right). Node labels are strain codes: KB1 ( E. faecalis ), YL2 ( B. animalis ), KB18 ( A. muris ), YL27 ( M. intestinale ), YL31 ( F. plautii ), YL32 ( E. clostridioformis ), YL44 ( A. muciniphila ), YL45 ( T. muris ), I46 ( C. innocuum ), I48 ( B. caecimuris ), I49 ( L. reuteri ), YL58 ( B. coccoides ). Edge colour indicates interaction sign (blue: positive, orange: negative).

Figure 6: Median wall time (left) and peak memory (right) as a function of the number of taxa $D$ , profiled on the Vogtmann et al. stool species dataset ( $N=110$ ) using the atime R package. Each point is the median of three runs; $D$ ranges from 10 to 1000. Both axes are on a log scale.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	$N/D$	MAC	GLMNet	PLN	$\Delta$	$p$ -value	$q$ -value	Winner
Castro-Nallar 2015	0.30	0.167	0.4857 $\pm$ 0.0695	0.3538 $\pm$ 0.0248	+27.2%	0.020	0.024	PLN
ChngKR 2016	0.37	0.092	0.2139 $\pm$ 0.0152	0.1793 $\pm$ 0.0104	+16.2%	5.3e-4	8.8e-4	PLN
HMP 2012	0.41	0.118	0.3365 $\pm$ 0.0433	0.2075 $\pm$ 0.0214	+38.3%	2.9e-4	5.3e-4	PLN
Zeller CRC	0.50	0.090	1.0870 $\pm$ 0.0559	0.9439 $\pm$ 0.0448	+13.2%	1.4e-4	3.4e-4	PLN
Ross obesity	0.51	0.130	1.2938 $\pm$ 0.0885	0.9046 $\pm$ 0.0422	+30.1%	1.8e-6	5.0e-6	PLN
Lozupone HIV	0.89	0.163	1.3591 $\pm$ 0.1128	0.9587 $\pm$ 0.0824	+29.5%	2.0e-4	4.1e-4	PLN

Appendix

Table 8

Generalized Linear Models

Generalized linear models connect predictors to non-Gaussian outcomes through a link function, supporting interpretable prediction and statistical inference.

See all 25 papers

A New Family of Poisson Non-negative Matrix Factorization Methods Using the Shifted Log Link
Jan 9, 2026Eric Weine, Peter Carbonetto, Rafael A. Irizarry +1Non-Negative Matrix Factorization Generalized Linear Models
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Human Genetics, University of Chicago, Chicago, IL, USA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA, USA +1
Knowledge-Inclusive Adaptive Physics-Informed Neural Network for Microbial Interaction Modelling
Jun 5, 2026Ravisha Rupasinghe, Rajith Vidanaarachchi, Asela Hevapathige +3Physics-Informed ML
AI, Optimisation and Pattern Recognition Lab, Department of Mechanical Engineering, University of Melbourne, Melbourne, Victoria, Australia · Biodiversity Research Center, Academia Sinica, Taipei, Taiwan
LinearPFN: Amortized Variable Selection for Linear Models with Interactions
Oct 5, 2026Louis Schiekiera, Max Zimmer, Christophe Roux +3Model Selection Prior-Data Fitted Networks
Institute of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany · Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany · Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany +1

Jan 9, 2026J/K move · Enter open · S save

A New Family of Poisson Non-negative Matrix Factorization Methods Using the Shifted Log Link

Eric Weine, Peter Carbonetto, Rafael A. Irizarry, Matthew Stephens

Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Human Genetics, University of Chicago, Chicago, IL, USA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA, USA+1

Non-Negative Matrix Factorization Generalized Linear Models

Open paper AskPDF arXiv

Poisson non-negative matrix factorization (NMF) is a widely used method to find interpretable "parts-based" decompositions of count data. While many variants of Poisson NMF exist, existing methods assume that the "parts" in the decomposition combine additively. This assumption may be natural in some settings, but not in others. Here we introduce Poisson NMF with the shifted-log link function to relax this assumption. The shifted-log link function has a single tuning parameter, and as this parameter varies the model changes from assuming that parts combine additively (i.e., standard Poisson NMF) to assuming that parts combine more multiplicatively. We provide an algorithm to fit this model by maximum likelihood, and also an approximation that substantially reduces computation time for large, sparse datasets (computations scale with the number of non-zero entries in the data matrix). We illustrate these new methods on a variety of real datasets. Our examples show how the choice of link function in Poisson NMF can substantively impact the results, and how in some settings the use of a shifted-log link function may improve interpretability compared with the standard, additive link.

Abstract

Generalized Linear Models

Explore similar work

A New Family of Poisson Non-negative Matrix Factorization Methods Using the Shifted Log Link

Knowledge-Inclusive Adaptive Physics-Informed Neural Network for Microbial Interaction Modelling

LinearPFN: Amortized Variable Selection for Linear Models with Interactions