Non-Stationarity in the Embedding Space of Time Series Foundation Models

Jinmyeong Choi; Brad Shook; Artur Dubrawski

cs.LGApr 6, 2026

Non-Stationarity in the Embedding Space of Time Series Foundation Models

Authors: Jinmyeong Choi, Brad Shook, Artur Dubrawski

Organizations: Auton Lab Robotics Institute Carnegie Mellon University Pittsburgh, PA 15213, USA

Abstract

Time series foundation models (TSFMs) are widely used as generic feature extractors, yet the notion of non-stationarity in their embedding spaces remains poorly understood. Recent work often conflates non-stationarity with distribution shift, blurring distinctions fundamental to classical time-series analysis and long-standing methodologies such as statistical process control (SPC). In SPC, non-stationarity signals a process leaving a stable regime - via shifts in mean, variance, or emerging trends - and detecting such departures is central to quality monitoring and change-point analysis. Motivated by this diagnostic tradition, we study how different forms of distributional non-stationarity - mean shifts, variance changes, and linear trends - become linearly accessible in TSFM embedding spaces under controlled conditions. We further examine temporal non-stationarity arising from persistence, which reflects violations of weak stationarity due to long-memory or near-unit-root behavior rather than explicit distributional shifts. By sweeping shift strength and probing multiple TSFMs, we find that embedding-space detectability of non-stationarity degrades smoothly and that different models exhibit distinct, model-specific failure modes.

Figures & tables

Panel of Figure 1 — Figure 1: Distributional non-stationarity in raw and embedding space (Chronos2).

Panel of Figure 2 — Figure 2: Macro-F1 as a function of shift strength. Strong shifts are easily detected by both TSFM embeddings and statistics-based baselines. As shift magnitude decreases, separability degrades smoothly, with statistics-based baselines degrading more rapidly than Chronos2 and MOMENT. The qualitative behavior remains unchanged under random persistence, indicating that shift-type decodability is driven primarily by distributional structure rather than autoregressive persistence.

Panel of Figure 3 — Figure 3: Temporal non-stationarity in TSFM embeddings under an AR(1) persistence sweep. (a) Chronos2 embeddings shift smoothly with increasing $\phi$ , without a sharp transition at $\phi=1$ . (b) Cosine discrepancy (raw) increases monotonically with persistence. (c) The same trend persists after z-score normalization, indicating that the TSFM embeddings capture temporal dependence beyond scale effects.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Representative AR(1) windows with increasing persistence. As $\phi$ increases from $0.6$ to $1.1$ , the process transitions from a mean-reverting regime to a unit-root boundary and eventually to an explosive regime. Unlike distributional shifts, these changes modify temporal dependence rather than introducing explicit within-window shifts.

Category	Parameter	Value / Range
Baseline AR(1)
	Window length $L$	128
	Mean $\mu$	0.5
	Innovation std. $\sigma$	0.06
	Persistence $\phi$ (fixed)	0.6
	Persistence $\phi$ (nuisance)	$\text{Uniform}(0.3,0.9)$

Appendix

Table 1: Parameter ranges used for synthetic data generation. Shift magnitudes are scaled by strength

s\in(0,1]

to control task difficulty.

Figure 5: Representative AR(1) windows used for distributional non-stationarity experiments (

L=128

): stationary, mean shift, variance shift, and trend.

Model	MAE $\downarrow$	Pearson $r$ $\uparrow$	$R^{2}$ $\uparrow$
Chronos2	0.041	0.982	0.964
MOMENT	0.058	0.965	0.931
TOTEM	0.121	0.812	0.659
Stats-LR	0.089	0.901	0.811
Stats+Dynamics-LR	0.061	0.954	0.910

Appendix

Table 2: Regression performance for predicting AR(1) persistence

\phi

from embeddings. Lower MAE and higher

r

and

R^{2}

indicate better preservation of temporal dependence.

Strong Shifts ( $s=1.0$ )
Strength	Model	$\mathbf{L=64}$	$\mathbf{L=128}$	$\mathbf{L=256}$	$\mathbf{L=512}$
TSFMs	Chronos2	0.891 $\pm$ 0.007	0.924 $\pm$ 0.005	0.948 $\pm$ 0.005	0.969 $\pm$ 0.002
	MOMENT	0.859 $\pm$ 0.007	0.891 $\pm$ 0.004	0.914 $\pm$ 0.004	0.933 $\pm$ 0.003
	TOTEM	0.711 $\pm$ 0.005	0.741 $\pm$ 0.005	0.760 $\pm$ 0.007	0.780 $\pm$ 0.008
Baselines	Stats-LR	0.927 $\pm$ 0.002	0.959 $\pm$ 0.001	0.978 $\pm$ 0.001	0.990 $\pm$ 0.001
Baselines	Stats+Dynamics-LR	0.962 $\pm$ 0.002	0.975 $\pm$ 0.002	0.983 $\pm$ 0.001	0.990 $\pm$ 0.000

Appendix

Table 3: Sequence-length ablation for shift-type probing under fixed persistence (

\phi=0.6

). Values are Macro-F1 (mean

\pm

std over 5 seeds).

Strong Shifts ( $s=1.0$ )
Strength	Model	$\mathbf{L=64}$	$\mathbf{L=128}$	$\mathbf{L=256}$	$\mathbf{L=512}$
TSFMs	Chronos2	0.866 $\pm$ 0.005	0.898 $\pm$ 0.003	0.927 $\pm$ 0.003	0.953 $\pm$ 0.003
	MOMENT	0.836 $\pm$ 0.005	0.873 $\pm$ 0.005	0.903 $\pm$ 0.004	0.927 $\pm$ 0.004
	TOTEM	0.702 $\pm$ 0.006	0.727 $\pm$ 0.013	0.752 $\pm$ 0.006	0.767 $\pm$ 0.008
Baselines	Stats-LR	0.899 $\pm$ 0.003	0.922 $\pm$ 0.002	0.940 $\pm$ 0.002	0.952 $\pm$ 0.002
Baselines	Stats+Dynamics-LR	0.929 $\pm$ 0.002	0.948 $\pm$ 0.002	0.961 $\pm$ 0.001	0.969 $\pm$ 0.001

Appendix

Table 4: Sequence-length ablation for shift-type probing under random persistence (

\phi\sim U(0.3,0.9)

). Values are Macro-F1 (mean

\pm

std over 5 seeds).

Fixed Shift ( $\phi=0.6$ )
Type	Model	$\mathbf{s=1.0}$	$\mathbf{s=0.25}$	$\mathbf{s=0.12}$
TSFMs	Chronos2	0.924 $\pm$ 0.005	0.680 $\pm$ 0.005	0.396 $\pm$ 0.007
	MOMENT	0.891 $\pm$ 0.004	0.620 $\pm$ 0.006	0.373 $\pm$ 0.011
	TOTEM	0.741 $\pm$ 0.005	0.436 $\pm$ 0.007	0.268 $\pm$ 0.005
Baselines	Stats-LR	0.959 $\pm$ 0.001	0.501 $\pm$ 0.006	0.323 $\pm$ 0.012
Baselines	Stats+Dynamics-LR	0.975 $\pm$ 0.002	0.617 $\pm$ 0.003	0.384 $\pm$ 0.005

Appendix

Table 5: Shift-type linear probing under varying shift strengths (

\phi=0.6

and

\phi\sim U(0.3,0.9)

Figure 6: Confusion matrices for shift-type probing at sequence length $L=128$ with fixed persistence ( $\phi=0.6$ ). As shift strength decreases (top to bottom), errors increase and reveal model-specific failure modes. Chronos2 and MOMENT retain strong diagonal structure for variance shifts, whereas TOTEM exhibits substantial collapse of mean shifts into the stationary class.

Figure 7: Confusion matrices under random persistence ( $\phi\sim U(0.3,0.9)$ ). Qualitative failure patterns remain consistent with the fixed- $\phi$ setting, indicating that shift-type confusion is driven by distributional structure rather than persistence.

Time-Series Foundation Models

Time-series foundation models are pretrained sequence models designed to transfer across forecasting, classification, anomaly detection, and other temporal tasks.

See all 298 papers

Post-Training in Time Series Foundation Models: A Unifying Framework
Jul 22, 2026Shifeng Xie, Ambroise Odonnat, Zehao Xiao +7Model Compression Time-Series Foundation Models
Noah’s Ark Lab · Université Paris Cité · Inria +2
Universal Redundancies in Time Series Foundation Models
Feb 2, 2026Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai +1Transformer Interpretability Kernel Regression
ECE Department, UT Austin · Department of Physics, UT Austin · Oden Institute, UT Austin
Normality Constraint Learning: Adapting Foundation Models for Time Series Anomaly Detection
Oct 5, 2026Xiaohui Zhou, Yijie Wang, Hongzuo Xu +2Reconstruction-Based Anomaly Detection Time-Series Anomaly Detection
National Key Laboratory of Parallel and Distributed Computing · College of Computer Science and Technology, National University of Defense Technology · Intelligent Game and Decision Lab (IGDL) +1

Jul 22, 2026J/K move · Enter open · S save

Post-Training in Time Series Foundation Models: A Unifying Framework

Shifeng Xie, Ambroise Odonnat, Zehao Xiao, Lei Zan+6

Noah’s Ark Lab · Université Paris Cité · Inria+2

Model Compression Time-Series Foundation Models

Open paper AskPDF arXiv

Time series foundation models (TSFMs) have emerged as general-purpose models for time series analysis, but pretraining alone is often insufficient for reliable downstream deployment. Bridging this gap requires further intervention to handle domain shift, task heterogeneity, limited supervision, and computational constraints, which motivates post-training as a broad class of methods to adapt, augment, compose, calibrate, or specialize pretrained TSFMs for downstream tasks. In this work, we analyze TSFM post-training methods based on their locus of intervention in the prediction pipeline, yielding five categories: parameter adaptation, context augmentation, model composition, output processing and uncertainty control, and compression and specialization. Within each category, we study main representative methods and discuss their current limitations. We further identify future directions toward controlled adaptation, reliable context construction, uncertainty-aware model composition, calibrated output processing, and deployment-aware specialization. Overall, by providing a unifying framework for the emerging TSFM post-training landscape, this work aims to support future research to navigate the design space between a pretrained TSFM and its reliable downstream deployment.

Abstract

Time-Series Foundation Models

Explore similar work

Post-Training in Time Series Foundation Models: A Unifying Framework

Universal Redundancies in Time Series Foundation Models

Normality Constraint Learning: Adapting Foundation Models for Time Series Anomaly Detection