Deep Neural Mixed Models for Nonlinear Longitudinal Prediction with Subject-Specific Heterogeneity
Authors: Mahmud Hasan, Niloofar Ramezani, Hrishikesh Chakraborty
Organizations: Department of Mathematics and Statistics, Penn State Altoona, PA, USA · Department of Biostatistics, Virginia Commonwealth University, VA, USA · Department of Biostatistics & Bioinformatics, Duke University, NC, USA
Longitudinal health data often exhibit within-subject correlation, subject-specific heterogeneity, and complex nonlinear predictor--response relationships that may not be adequately captured by conventional mixed models. We propose a Deep Generalized Neural Network Mixed Model (DGNMM) that extends the GLMM framework by using a deep neural network to learn nonlinear population-level relationships while incorporating directly trainable, regularized subject-specific effects to capture individual heterogeneity. The model is estimated using prediction-oriented stochastic optimization with separate regularization of the neural-network parameters and subject-specific effects. This formulation allows us to investigate when increased nonlinear representation capacity provides additional predictive benefit in the presence of subject-specific heterogeneity. A simulation study varies population-level nonlinearity and subject-specific heterogeneity and compares DGNMM with GLMM, artificial neural network (ANN), Maity--Pal (MP), and GNMM alternatives for continuous and binary longitudinal outcomes. In the real-data application, DGNMM achieved the lowest MSPE for continuous depression and anxiety prediction, while alternative models remained competitive for binary outcomes. Exploratory subject-level analyses showed heterogeneous predictive gains across individuals, but the magnitude of the estimated subject-specific effects was not significantly associated with predictive gain. Overall, increased nonlinear representation capacity was most beneficial for continuous prediction under nonlinear response structures, while greater subject-specific heterogeneity did not systematically yield larger predictive gains.
Figures & tables
Figure 1: Architecture of the proposed Deep Generalized Neural Network Mixed Model (DGNMM). The population-level nonlinear component fθDeep(xij) is represented by a multilayer feed-forward neural network, while a regularized subject-specific latent effect ei captures persistent individual deviation from the population-level response surface. The resulting predictor is ηij=fθDeep(xij)+ei , with the conditional mean obtained through the inverse link function. Both components are estimated jointly for longitudinal prediction.
ηij=fθDeep(xij)+ei.
Algorithm 1 Parameter Estimation Procedure for DGNMM
Figure 2: MSPE comparison of DGNMM, GNMM-1, GNMM-2, ANN, GLMM, and MP across different levels of population-level nonlinearity γ and subject-specific heterogeneity τ for continuous outcomes. Results are presented for competing neural-network specifications with 5 nodes in the first hidden layer. The proposed DGNMM employs a deep neural network architecture with hidden layers of sizes 64, 32, and 16. For each combination of γ and τ , the final observation from each subject was reserved as a holdout test sample, and prediction performance was evaluated using MSPE over 500 simulation replications. Lower MSPE indicates better predictive performance.
Figure 3: MSPE comparison of DGNMM, GNMM-1, GNMM-2, ANN, GLMM, and MP across different levels of population-level nonlinearity γ and subject-specific heterogeneity τ for continuous outcomes. Results are presented for competing neural-network specifications with 10 nodes in the first hidden layer. The proposed DGNMM employs a deep neural network architecture with hidden layers of sizes 64, 32, and 16. For each combination of γ and τ , the final observation from each subject was reserved as a holdout test sample, and prediction performance was evaluated using MSPE over 500 simulation replications. Lower MSPE indicates better predictive performance.
Figure 4: AUC comparison of DGNMM, GNMM-1, GNMM-2, ANN, GLMM, and MP across different levels of population-level nonlinearity γ and subject-specific heterogeneity τ for binary outcomes. Results are presented for competing neural-network specifications with 5 nodes in the first hidden layer. The proposed DGNMM employs a deep neural network architecture with hidden layers of sizes 64, 32, and 16. For each combination of γ and τ , the final observation from each subject was reserved as a holdout test sample, and prediction performance was evaluated using AUC over 500 simulation replications. Larger AUC indicates better discrimination.
Figure 5: AUC comparison of DGNMM, GNMM-1, GNMM-2, ANN, GLMM, and MP across different levels of population-level nonlinearity γ and subject-specific heterogeneity τ for binary outcomes. Results are presented for competing neural-network specifications with 10 nodes in the first hidden layer. The proposed DGNMM employs a deep neural network architecture with hidden layers of sizes 64, 32, and 16. For each combination of γ and τ , the final observation from each subject was reserved as a holdout test sample, and prediction performance was evaluated using AUC over 500 simulation replications. Larger AUC indicates better discrimination.
Outcome
Method
MSPE
Depression
DGNMM
0.324665
GNMM-1
0.731966
GNMM-2
0.748917
ANN
0.986315
GLMM Main Effects
0.546328
GLMM + 3 Interactions
0.608985
Table 1: MSPE comparison of DGNMM, GNMM-1, GNMM-2, GLMM, ANN, and MP for depression and anxiety prediction. Lower MSPE indicates better predictive performance.
Outcome
Method
AUC
Depression
DGNMM
0.989
GNMM-1
0.982
GNMM-2
0.961
ANN
0.870
GLMM Main Effects
0.875
GLMM + 3 Interactions
1.000
Table 2: AUC comparison of DGNMM, GNMM-1, GNMM-2, GLMM, ANN, and MP for classifying high versus low depression and anxiety symptoms. Larger AUC values indicate better classification performance.
Figure 6: Relationship between the magnitude of the estimated DGNMM subject-specific effect ∣e^i∣ and subject-specific predictive gain relative to the main-effects GLMM for (a) depression and (b) anxiety. Positive predictive gain indicates a smaller squared prediction error for DGNMM, whereas negative gain indicates a smaller squared prediction error for GLMM. The Spearman correlations between ∣e^i∣ and predictive gain were ρ=−0.346 ( p=0.206 ) for depression and ρ=−0.407 ( p=0.132 ) for anxiety. Neither association was statistically significant.
Generalized linear mixed-effects models (GLMMs) are widely used to analyze grouped and hierarchical data. In a GLMM, each response is assumed to follow an exponential-family distribution where the natural parameter is given by a linear function of observed covariates and a latent group-specific random effect. Since exact marginalization over the random effects is typically intractable, model parameters are estimated by maximizing an approximate marginal likelihood. In this paper, we replace the linear function with neural networks. The result is a more flexible model, the neural generalized mixed-effects model (NGMM), which captures complex relationships between covariates and responses. To fit NGMM to data, we introduce an efficient optimization procedure that maximizes the approximate marginal likelihood and is differentiable with respect to network parameters. We show that the approximation error of our objective decays at a Gaussian-tail rate in a user-chosen parameter. On synthetic data, NGMM improves over GLMMs when covariate-response relationships are nonlinear, and on real-world datasets it outperforms prior methods. Finally, we analyze a large dataset of student proficiency to demonstrate how NGMM can be extended to more complex latent-variable models.
Yuli Slavutsky, Sebastian Salazar, David M. Blei
Department of Statistics, Columbia University, New York, USA · Department of Computer Science, Columbia University, New York, USA
The experience sampling method (ESM) is a longitudinal research design where participants report their thoughts, emotional states and behaviours multiple times a day. Our work is motivated by such data collected by the GrowIt! app, which was released to investigate daily emotions among adolescents during the COVID-19 pandemic. Current procedures to analyse ESM data face various challenges. While standard statistical techniques may not scale well to a high-dimensional setting, machine learning procedures can give biased results due to selection bias introduced by missingness. In our motivating dataset, adolescents dropped out due to previous strong feelings of negative emotions. Hence, the implied missing data are of the missing-at-random type that standard machine learning procedures cannot accommodate. We develop a novel neural network architecture that generalises mixed effects models to deep learning to overcome these challenges. It allows semi-parametric and flexible modelling of data's mean and correlation structure through fixed and random effects. For estimation, we use an adaptation of variational auto-encoders and a Bayesian data augmentation algorithm. Through this approach, the model can accommodate longitudinal outcomes following generic distributions, scale well to high-dimensional settings and provide valid inference when data are missing-at-random. We applied the Deep Generalised Mixed Model to the GrowIt! study and various simulations. The results show potential for the Deep Generalised Mixed Model, yet suboptimal performance due to model instability.
Nina van Gerwen, Dimitris Rizopoulos, Manon Hillegers +2
Department of Biostatistics, Erasmus University Medical Center, the Netherlands · Department of Epidemiology, Erasmus University Medical Center, the Netherlands · Department of Child Psychiatry, Erasmus University Medical Center, the Netherlands +1
Longitudinal cohort studies produce repeated data that enable the assessment of time-varying association patterns between exposures and health outcomes. Classical linear mixed-effects models (LMMs) can accommodate a large variety of association patterns while accounting for the irregularly spaced, partially observed measurement. But they require the analyst to pre-specify the functional form linking the exposure history to the outcome. We propose the Neural ODE-LMM, which embeds a Neural Ordinary Differential Equation (Neural ODE) within the linear mixed-effects framework: a learned vector field encodes covariate trajectories into a continuous-time latent state that drives both the fixed- and random-effect design, while preserving the standard LMM observation model. This retains classical likelihood-based inference while learning complex, potentially cumulative, covariate effects flexibly. All parameters are estimated by maximising a penalised marginal likelihood. To quantify covariate effects, we introduce contrasts of counterfactual predictions that compare the expected outcome under alternative covariate trajectories with variance estimated via the delta method. In simulations, the model recovers both instantaneous and cumulative-burden effects without prior specification of the functional form. Applied to the Trois-Cités (3C) cohort, a population-based study of 7{,}324 participants, the method reveals trajectory-dependent associations of BMI and fasting glucose with cognitive decline.
Univ. Bordeaux, Inserm, BPH, UMR 1219, 33000 Bordeaux, France · INRIA SISTM, 33000 Bordeaux, France · Vaccine Research Institute, 94010 Creteil, France