A statistical perspective on transformers for small longitudinal cohort data

Kiana Farhadyar; Maren Hackenberg; Kira Ahrens; Charlotte Schenk; Bianca Kollmann; Oliver Tüscher; Klaus Lieb; Michael M. Plichta

stat.MEFeb 18, 2026

A statistical perspective on transformers for small longitudinal cohort data

Authors: Kiana Farhadyar, Maren Hackenberg, Kira Ahrens, Charlotte Schenk, Bianca Kollmann, Oliver Tüscher, Klaus Lieb, Michael M. Plichta,

Organizations: Institute of Medical Biometry and Statistics (IMBI), University of Freiburg, Freiburg, Germany · Freiburg Center for Data Analysis, Modeling and AI, University of Freiburg, Freiburg, Germany · Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, University Hospital Frankfurt, Frankfurt, Germany · Department of Neuropsychology and Psychological Resilience Research, Central Institute of Mental Health (ZI), Mannheim, Germany · Leibniz Institute for Resilience Research (LIR), Mainz, Germany · Department of Psychiatry and Psychotherapy, Johannes Gutenberg University Medical Center Mainz, Mainz, Germany · Department of Psychiatry, Psychotherapy and Psychosomatic Medicine, University Medicine Halle, Martin-Luther University Halle-Wittenberg, Halle, Germany · German Center for Mental Health (DZPG), partner site Halle-Jena-Magdeburg, Halle, Germany · Neuroimaging Center (NIC), Focus Program Translational Neuroscience (FTN), Johannes Gutenberg University Medical Center, Mainz, Germany

Abstract

Modeling of longitudinal cohort data typically involves complex temporal dependencies between multiple variables. There, the transformer architecture, which has been highly successful in language and vision applications, allows us to account for the fact that the most recently observed time points in an individual's history may not always be the most important for the immediate future. This is achieved by assigning attention weights to observations of an individual based on a transformation of their values. One reason why these ideas have not yet been fully leveraged for longitudinal cohort data is that typically, large datasets are required. Therefore, we present a simplified transformer architecture that retains the core attention mechanism while reducing the number of parameters to be estimated, to be more suitable for small datasets with few time points. Guided by a statistical perspective on transformers, we use an autoregressive model as a starting point and incorporate attention as a kernel-based operation with temporal decay, where aggregation of multiple transformer heads, i.e. different candidate weighting schemes, is expressed as accumulating evidence on different types of underlying characteristics of individuals. This also enables a permutation-based statistical testing procedure for identifying contextual patterns. In a simulation study, the approach is shown to recover contextual dependencies even with a small number of individuals and time points. In an application to data from a resilience study, we identify temporal patterns in the dynamics of stress and mental health. This indicates that properly adapted transformers can not only achieve competitive predictive performance, but also uncover complex context dependencies in small data settings.

Figures & tables

Approach	Metric	$n_{\text{train}}=100$	$n_{\text{train}}=200$	$n_{\text{train}}=500$	$n_{\text{train}}=1000$
Average	MSE	$0.209\pm 0.002$	$0.21\pm 0.002$	$0.209\pm 0.002$	$0.209\pm 0.002$
Average	MSE ${}_{j_{3}}$	$0.218\pm 0.007$	$0.217\pm 0.009$	$0.214\pm 0.01$	$0.211\pm 0.007$
Regression	MSE	$0.209\pm 0.003$	$0.206\pm 0.002$	$0.204\pm 0.002$	$0.204\pm 0.002$
Regression	MSE ${}_{j_{3}}$	$0.155\pm 0.005$	$0.153\pm 0.006$	$0.152\pm 0.005$	$0.151\pm 0.004$
Informed	MSE	$\mathbf{0.204\pm 0.002}$	$0.204\pm 0.002$	$0.204\pm 0.002$	$0.204\pm 0.002$
Informed	MSE ${}_{j_{3}}$	$0.163\pm 0.008$	$0.163\pm 0.008$	$0.161\pm 0.008$	$0.159\pm 0.005$

Table 1: Prediction performance of the MiniTransformer for simulated data with

p=10

variables in training datasets with different numbers of sequences

n_{\text{train}}

, compared to a simple averaging approach, regression models, and an approach informed by the true structure. Mean squared error is considered across all variables (MSE) and specifically for the variable that is affected by the true pattern (MSE

{}_{j_{3}}

), together with standard deviations. Smallest values are indicated by boldface.

Figure 1: Individual simulated and fitted trajectories for a random selection of individuals, with true underlying structure

z_{t}

, observed target

x_{t,j_{3}}

, predicted target

\hat{x}_{t,j_{3}}

, and learned cumulants ( 3 ).

Variable	$V=8$	$V=12$	$V=16$
1	$0.084\pm 0.13$	$0.01\pm 0.02$	$<0.001$
2	$0.2\pm 0.29$	$0.049\pm 0.06$	$0.002$
3	$0.084\pm 0.16$	$0.01\pm 0.02$	$0.007$
4	$0.607\pm 0.24$	$0.633\pm 0.18$	$0.635$

Table 2: Average p-values (and standard deviations from 10 repetitions) resulting from the permutation testing approach when using sampling, with either

V=8

V=12

patterns of

\mathbf{x}_{v}^{\mathrm{visit}}

sampled, contrasted with the full permutation approach with

V=16

, in a setting with

p=4

variables (where variables 1-3 are part of the true pattern) and 50 sequences.

Approach	Metric	Dataset 1	Dataset 2
Average	MSE	$0.170\pm 0.008$	$0.149\pm 0.006$
	MSE tar	$0.229\pm 0.027$	$0.208\pm 0.018$
Carry forward	MSE	$0.220\pm 0.013$	$0.212\pm 0.020$
	MSE tar	$0.304\pm 0.013$	$0.298\pm 0.020$
Regression	MSE	$0.147\pm 0.009$	$0.135\pm 0.008$
	MSE tar	$0.199\pm 0.024$	$0.190\pm 0.023$

Table 3: Prediction performance of the MiniTransformer for two resilience datasets, each with

p=10

, compared to a simple averaging approach, regression models, and a carry forward approach. Mean squared error is considered across all variables (MSE) and for a specific target variable (MSE tar ), together with standard deviations. Smallest values are indicated by boldface.

Rank	Dataset 1		Dataset 2
Rank	Variable	pval	Variable	pval
$1$	dh_38	$0.0131\pm 0.02$	dh_38	$\mathbf{0.0001\pm 0.00}$
$2$	dh_53	$0.0720\pm 0.08$	dh_37	$0.0346\pm 0.05$
$3$	le_8	$0.1321\pm 0.16$	dh_46	$0.0850\pm 0.05$
$4$	dh_37	$0.1559\pm 0.08$	dh_11	$0.0996\pm 0.03$
$5$	dh_45	$0.2240\pm 0.11$	le_17	$0.3613\pm 0.10$

Table 4: Results of the permutation testing approach for the two resilience application datasets. Average p-values are reported together with standard deviations. Variables are ranked in ascending order of p-values, with rank 1 corresponding to the most important variable.

Panel of Figure 2 — Figure 2: Context-target effects for resilience dataset 1 (left) and dataset 2 (right). Dark blue indicates smaller values of the visualized test statistic $s^{(r)}_{j}$ (i.e., the context has a smaller effect), while dark red indicates larger values.

Permutation Testing

See all 12 papers

Feb 20, 2026

On the "Induction Bias" in Sequence Models

Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growing body of literature has shown this limitation primarily through failures in out-of-distribution (OOD) generalization, such as length extrapolation. In this work, we shift attention to the in-distribution implications of these limitations. We conduct a large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes. We find that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs. Furthermore, we analyze the extent to which learned state-tracking mechanisms are shared across different sequence lengths. We show that transformers exhibit negligible or even detrimental weight sharing across lengths, indicating that they learn length-specific solutions in isolation. In contrast, recurrent models exhibit effective amortized learning by sharing weights across lengths, allowing data from one sequence length to improve performance on others. Together, these results demonstrate that state tracking remains a fundamental challenge for transformers, even when training and evaluation distributions match.

M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal +1

Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

Transformer State Tracking Inductive Bias Data-Efficient Learning Recurrent Neural Networks Length Generalization

Ask PDF

Apr 16, 2026

Personalized and Context-Aware Transformer Models for Predicting Post-Intervention Physiological Responses from Wearable Sensor Data

Consumer wearables enable continuous measurement of physiological data related to stress and recovery, but turning these streams into actionable, personalized stress-management recommendations remains a challenge. In practice, users often do not know how a given intervention, defined as an activity intended to reduce stress, will affect heart rate (HR), heart rate variability (HRV), or inter-beat intervals (BBI) over the next 15 to 120 minutes. We present a framework that predicts post-intervention trajectories and the direction of change for these physiological indicators across time windows. Our methodology combines a Transformer model for multi-horizon trajectories of percent change relative to a pre-intervention baseline, direction-of-change calls (positive, negative, or neutral) at each horizon, and an empirical study using wearable sensor data overlaid with user-tagged events and interventions. This proof of concept shows that personalized post-intervention prediction is feasible. We encourage future integration into stress-management tools for personalized intervention recommendations tailored to each person's day following further validation in larger studies and, where applicable, appropriate regulatory review.

Esther Brown, Victoria Dean, Finale Doshi-Velez

Harvard University, Cambridge, MA · Princeton University, Princeton, NJ

Multivariate Time Series Forecasting Wearable Health Monitoring

Ask PDF

Feb 2, 2026

Universal Redundancies in Time Series Foundation Models

Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations on standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.

Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai +1

ECE Department, UT Austin · Department of Physics, UT Austin · Oden Institute, UT Austin

Transformer Interpretability Kernel Regression Mechanistic Interpretability Transformer Pruning Time-Series Foundation Models Neural Network Pruning

Ask PDF

Abstract

Permutation Testing

Explore similar work

On the "Induction Bias" in Sequence Models

Personalized and Context-Aware Transformer Models for Predicting Post-Intervention Physiological Responses from Wearable Sensor Data

Universal Redundancies in Time Series Foundation Models