A statistical perspective on transformers for small longitudinal cohort data

Kiana Farhadyar; Maren Hackenberg; Kira Ahrens; Charlotte Schenk; Bianca Kollmann; Oliver Tüscher; Klaus Lieb; Michael M. Plichta

stat.MEFeb 18, 2026

A statistical perspective on transformers for small longitudinal cohort data

Authors: Kiana Farhadyar, Maren Hackenberg, Kira Ahrens, Charlotte Schenk, Bianca Kollmann, Oliver Tüscher, Klaus Lieb, Michael M. Plichta,

Organizations: Institute of Medical Biometry and Statistics (IMBI), University of Freiburg, Freiburg, Germany · Freiburg Center for Data Analysis, Modeling and AI, University of Freiburg, Freiburg, Germany · Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, University Hospital Frankfurt, Frankfurt, Germany · Department of Neuropsychology and Psychological Resilience Research, Central Institute of Mental Health (ZI), Mannheim, Germany · Leibniz Institute for Resilience Research (LIR), Mainz, Germany · Department of Psychiatry and Psychotherapy, Johannes Gutenberg University Medical Center Mainz, Mainz, Germany · Department of Psychiatry, Psychotherapy and Psychosomatic Medicine, University Medicine Halle, Martin-Luther University Halle-Wittenberg, Halle, Germany · German Center for Mental Health (DZPG), partner site Halle-Jena-Magdeburg, Halle, Germany · Neuroimaging Center (NIC), Focus Program Translational Neuroscience (FTN), Johannes Gutenberg University Medical Center, Mainz, Germany

Abstract

Modeling of longitudinal cohort data typically involves complex temporal dependencies between multiple variables. There, the transformer architecture, which has been highly successful in language and vision applications, allows us to account for the fact that the most recently observed time points in an individual's history may not always be the most important for the immediate future. This is achieved by assigning attention weights to observations of an individual based on a transformation of their values. One reason why these ideas have not yet been fully leveraged for longitudinal cohort data is that typically, large datasets are required. Therefore, we present a simplified transformer architecture that retains the core attention mechanism while reducing the number of parameters to be estimated, to be more suitable for small datasets with few time points. Guided by a statistical perspective on transformers, we use an autoregressive model as a starting point and incorporate attention as a kernel-based operation with temporal decay, where aggregation of multiple transformer heads, i.e. different candidate weighting schemes, is expressed as accumulating evidence on different types of underlying characteristics of individuals. This also enables a permutation-based statistical testing procedure for identifying contextual patterns. In a simulation study, the approach is shown to recover contextual dependencies even with a small number of individuals and time points. In an application to data from a resilience study, we identify temporal patterns in the dynamics of stress and mental health. This indicates that properly adapted transformers can not only achieve competitive predictive performance, but also uncover complex context dependencies in small data settings.

Figures & tables

Approach	Metric	$n_{\text{train}}=100$	$n_{\text{train}}=200$	$n_{\text{train}}=500$	$n_{\text{train}}=1000$
Average	MSE	$0.209\pm 0.002$	$0.21\pm 0.002$	$0.209\pm 0.002$	$0.209\pm 0.002$
Average	MSE ${}_{j_{3}}$	$0.218\pm 0.007$	$0.217\pm 0.009$	$0.214\pm 0.01$	$0.211\pm 0.007$
Regression	MSE	$0.209\pm 0.003$	$0.206\pm 0.002$	$0.204\pm 0.002$	$0.204\pm 0.002$
Regression	MSE ${}_{j_{3}}$	$0.155\pm 0.005$	$0.153\pm 0.006$	$0.152\pm 0.005$	$0.151\pm 0.004$
Informed	MSE	$\mathbf{0.204\pm 0.002}$	$0.204\pm 0.002$	$0.204\pm 0.002$	$0.204\pm 0.002$
Informed	MSE ${}_{j_{3}}$	$0.163\pm 0.008$	$0.163\pm 0.008$	$0.161\pm 0.008$	$0.159\pm 0.005$

Table 1: Prediction performance of the MiniTransformer for simulated data with

p=10

variables in training datasets with different numbers of sequences

n_{\text{train}}

, compared to a simple averaging approach, regression models, and an approach informed by the true structure. Mean squared error is considered across all variables (MSE) and specifically for the variable that is affected by the true pattern (MSE

{}_{j_{3}}

), together with standard deviations. Smallest values are indicated by boldface.

Figure 1: Individual simulated and fitted trajectories for a random selection of individuals, with true underlying structure

z_{t}

, observed target

x_{t,j_{3}}

, predicted target

\hat{x}_{t,j_{3}}

, and learned cumulants ( 3 ).

Variable	$V=8$	$V=12$	$V=16$
1	$0.084\pm 0.13$	$0.01\pm 0.02$	$<0.001$
2	$0.2\pm 0.29$	$0.049\pm 0.06$	$0.002$
3	$0.084\pm 0.16$	$0.01\pm 0.02$	$0.007$
4	$0.607\pm 0.24$	$0.633\pm 0.18$	$0.635$

Table 2: Average p-values (and standard deviations from 10 repetitions) resulting from the permutation testing approach when using sampling, with either

V=8

V=12

patterns of

\mathbf{x}_{v}^{\mathrm{visit}}

sampled, contrasted with the full permutation approach with

V=16

, in a setting with

p=4

variables (where variables 1-3 are part of the true pattern) and 50 sequences.

Approach	Metric	Dataset 1	Dataset 2
Average	MSE	$0.170\pm 0.008$	$0.149\pm 0.006$
	MSE tar	$0.229\pm 0.027$	$0.208\pm 0.018$
Carry forward	MSE	$0.220\pm 0.013$	$0.212\pm 0.020$
	MSE tar	$0.304\pm 0.013$	$0.298\pm 0.020$
Regression	MSE	$0.147\pm 0.009$	$0.135\pm 0.008$
	MSE tar	$0.199\pm 0.024$	$0.190\pm 0.023$

Table 3: Prediction performance of the MiniTransformer for two resilience datasets, each with

p=10

, compared to a simple averaging approach, regression models, and a carry forward approach. Mean squared error is considered across all variables (MSE) and for a specific target variable (MSE tar ), together with standard deviations. Smallest values are indicated by boldface.

Rank	Dataset 1		Dataset 2
Rank	Variable	pval	Variable	pval
$1$	dh_38	$0.0131\pm 0.02$	dh_38	$\mathbf{0.0001\pm 0.00}$
$2$	dh_53	$0.0720\pm 0.08$	dh_37	$0.0346\pm 0.05$
$3$	le_8	$0.1321\pm 0.16$	dh_46	$0.0850\pm 0.05$
$4$	dh_37	$0.1559\pm 0.08$	dh_11	$0.0996\pm 0.03$
$5$	dh_45	$0.2240\pm 0.11$	le_17	$0.3613\pm 0.10$

Table 4: Results of the permutation testing approach for the two resilience application datasets. Average p-values are reported together with standard deviations. Variables are ranked in ascending order of p-values, with rank 1 corresponding to the most important variable.

Panel of Figure 2 — Figure 2: Context-target effects for resilience dataset 1 (left) and dataset 2 (right). Dark blue indicates smaller values of the visualized test statistic $s^{(r)}_{j}$ (i.e., the context has a smaller effect), while dark red indicates larger values.

Permutation Testing

See all 12 papers

On the "Induction Bias" in Sequence Models
Feb 20, 2026M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal +1Transformer State Tracking
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Personalized and Context-Aware Transformer Models for Predicting Post-Intervention Physiological Responses from Wearable Sensor Data
Apr 16, 2026Esther Brown, Victoria Dean, Finale Doshi-VelezMultivariate Time Series Forecasting Wearable Health Monitoring
Harvard University, Cambridge, MA · Princeton University, Princeton, NJ
Universal Redundancies in Time Series Foundation Models
Feb 2, 2026Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai +1Transformer Interpretability Kernel Regression
ECE Department, UT Austin · Department of Physics, UT Austin · Oden Institute, UT Austin

Feb 20, 2026J/K move · Enter open · S save

On the "Induction Bias" in Sequence Models

M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal, Roland Memisevic

Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

Transformer State Tracking Inductive Bias Data-Efficient Learning Recurrent Neural Networks Length Generalization

Open paper AskPDF arXiv

Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growing body of literature has shown this limitation primarily through failures in out-of-distribution (OOD) generalization, such as length extrapolation. In this work, we shift attention to the in-distribution implications of these limitations. We conduct a large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes. We find that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs. Furthermore, we analyze the extent to which learned state-tracking mechanisms are shared across different sequence lengths. We show that transformers exhibit negligible or even detrimental weight sharing across lengths, indicating that they learn length-specific solutions in isolation. In contrast, recurrent models exhibit effective amortized learning by sharing weights across lengths, allowing data from one sequence length to improve performance on others. Together, these results demonstrate that state tracking remains a fundamental challenge for transformers, even when training and evaluation distributions match.

Abstract

Permutation Testing

Explore similar work

On the "Induction Bias" in Sequence Models

Personalized and Context-Aware Transformer Models for Predicting Post-Intervention Physiological Responses from Wearable Sensor Data

Universal Redundancies in Time Series Foundation Models