Scale-Invariant Training for Time Series Foundation Models

Ignacy Stepka; Willa Potosnak; Kin G. Olivares; Artur Dubrawski

cs.LGOct 5, 2026

Scale-Invariant Training for Time Series Foundation Models

Authors: Ignacy Stepka, Willa Potosnak, Kin G. Olivares, Artur Dubrawski

Organizations: Carnegie Mellon University · Amazon · Nixtla

Abstract

Time series foundation models (TSFMs) are trained on large collections of time series datasets that span various morphologies and domains. This setting exposes models to series whose scales -- typical magnitudes of their values -- can differ substantially. Affine scaling methods such as Reversible Instance Normalization (ReVIN) scale model inputs and reverse the transform before computing the loss. We show that this inversion multiplies each series' gradient by $b^p$ relative to loss on scaled targets, where $b$ is the scaling denominator (e.g., standard deviation) and $p$ is the loss degree. We call this scale-contaminated training (ScaleCon), because the scale of each series consequently becomes an importance weight, causing high-scale series to dominate training. For any scale-equivariant scaler and residual loss that is homogeneous of degree $p$ , including MSE, MAE, and Quantile Loss, we prove that computing loss on scaled targets makes every mini-batch gradient and, consequently, the full optimization trajectory invariant to arbitrary independent rescaling of the training series, yielding scale-invariant training (ScaleIn). Notably, existing TSFMs use both objectives, with neither consistent reporting nor a common convention on how to compute training loss. We isolate the convergence disparity induced by ScaleCon and its correction under ScaleIn in controlled studies on synthetic and real data. In pretraining across four TSFM architectures, ScaleIn lowers MASE in all 24 architecture-benchmark comparisons, with average reductions across TSFMs of 18.8% on GIFT-Eval and 21.9% on the M-competitions. The gains extend to supervised neural forecasting, where it lowers MASE in 16 of 20 matched settings. Most existing time series forecasting pipelines can adopt ScaleIn with a one-line code change.

Figures & tables

Figure 1: ScaleIn removes scale-driven gradient bias (Theorem 3.2 ). (a) Synthetic series scales span

b\in\{1,4,13,52\}

. (b) Under ScaleCon , MSE gradients with respect to

\hat{z}

scale as

b^{2}

, giving

b{=}52

approximately

2{,}700\times

the gradient magnitude of

b{=}1

for equal scaled residuals. (c) ScaleIn leaves gradients unchanged by rescaling, on the same

y

-axis. (d) Gradient distributions under ScaleCon (solid) and ScaleIn (hatched). (e) Growing scale induces gradient drift under ScaleCon but not ScaleIn . Lower-row curves use a 15-step moving average of

|\nabla\mathcal{L}|

Model		Loss Function	Explicit	Inferred	Code
TimeGPT-1	garza2023timegpt1	–	–	–	–
LagLLama	rasul2023lag	Student-t NLL	–	ScaleCon ¶	ScaleCon ¶
Time-LLM	jin2023time	MSE	–	ScaleCon	ScaleCon
Moirai	salesforce2023moirai	Mixture NLL	–	ScaleCon ¶	ScaleCon ¶
MOMENT	goswami2024moment	MSE	–	ScaleCon	ScaleCon ‡
TimesFM	das2024timesfm	MSE	–	ScaleCon	ScaleCon ∗‡

Table 1: A survey of loss spaces used in time series foundation models. Explicit : the paper states the loss space explicitly in text or in equations that define the normalized or denormalized quantities entering the loss. Inferred : it is possible to trace the target from the input through the normalization to the loss and infer with high likelihood what is the loss space. Code : the loss space is determined by the publicly released code. A dash means that we were not able to determine the loss space (or loss function) from that source.

Codebase	Reference	ScaleCon	Invariant ( ScaleIn )	Fixed/User-specified
GluonTS	alexandrov2020gluonts	NLL, CRPS, EnergyScore	—	Fixed
PyTorch Forecasting	beitner2020pytorch_forecasting	MAE, RMSE, MAPE, SMAPE, QuantileLoss, NormalDist, LogNormal, NegBinomial, Beta, MQF2	—	Fixed
TSLib	wu2023tslib	—	MSE, MAPE, MASE, SMAPE	Fixed
NeuralForecast	olivares2022neuralforecast	NLL (Normal, StudentT, NegBinomial, Poisson, Tweedie)	MAE, MSE, RMSE, MAPE, SMAPE, MASE, QuantileLoss, MQLoss	Fixed

Table 2: Training properties of time series forecasting codebases.

Figure 2: Loss placement determines whether inverse scaling introduces

b^{p}

gradient weighting. Both pipelines scale inputs using context statistics

a,b

. Top: ScaleCon reverses scaling before computing the loss, introducing the

b^{p}

weight. Bottom: ScaleIn computes loss on scaled targets and reverses scaling only for inference, yielding scale-invariant training (Theorem 3.2 ).

Panel of Figure 3 — Figure 3: Per-source validation nMSE over training steps. We report the mean $\pm 1$ standard deviation over five seeds. (a) The scale-invariant loss yields parallel convergence. (b) The scale-contaminated loss produces scale-ordered convergence. (c) The disparity disappears when all sources share the same scale. (d) The disparity persists after matching the global gradient norm. The dashed grey curve is global nMSE, aggregated rather than disaggregated by source.

Panel of Figure 4 — Figure 4: Inflating scale leaves trajectory-level error nearly unchanged under ScaleIn (a) , but lowers it on average under ScaleCon (b) . Bars show $A^{(k)}_{b,c}$ averaged over five seeds and 15 complementary assignment pairs. Error bars show 95% Student- $t$ intervals across assignment pairs.

Panel of Figure 5 — Figure 5: ScaleIn removes scale-driven convergence disparities and improves TSFM convergence. (a) Median per-dataset nMSE over 8 scale-swap datasets, aggregated across 15 assignment pairs and five seeds. The $b{=}1$ and $b{=}10$ curves overlap under ScaleIn and separate under ScaleCon . (b) Median per-dataset MASE for four architectures over $80{,}000$ updates. Plotted are averages across 4 seeds, with 95% Student- $t$ intervals.

	Chronos2			Moirai-2.0			TimesFM2.5			PatchTST			Reference
Benchmark	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	S.Naive
MASE
M1	1.879	2.388	-21.3	2.013	2.123	-5.2	2.731	2.863	-4.6	2.091	3.859	-45.8	2.117
M3	1.608	1.963	-18.1	1.621	1.800	-9.9	2.233	2.573	-13.2	2.041	4.840	-57.8	1.764
M4	1.845	2.294	-19.6	1.879	1.986	-5.4	2.661	3.208	-17.0	2.235	4.555	-50.9	2.057
Tourism	2.350	2.816	-16.5	2.316	2.564	-9.7	3.820	4.108	-7.0	2.509	3.123	-19.7	2.412

Table 3: Forecasting performance across benchmark datasets. The results were averaged across 4 seeds. ScaleIn uses scale-invariant loss and ScaleCon uses scale-contaminated loss. These are the only differences between the two runs of each architecture.

\Delta\%

reports ScaleIn relative to ScaleCon . Green denotes an improvement and red a regression. The better loss is bold within each architecture. References are fitted on the series they forecast and appear behind the double rule.

		Neural Forecast Baselines										Reference
Benchmark	Metric	N-HiTS		N-BEATS		PatchTST		TCN		LSTM		S.Naive	ETS	ARIMA
Benchmark	Metric	Sc.In	Sc.Con	Sc.In	Sc.Con	Sc.In	Sc.Con	Sc.In	Sc.Con	Sc.In	Sc.Con	S.Naive	ETS	ARIMA
M1	MASE	1.826	$1.855$	1.822	$1.835$	$1.917$	1.911	1.941	$1.987$	1.998	$2.057$	$2.478$	$1.904$	$1.977$
M1	WQL	$0.099$	0.098	$0.099$	0.098	0.106	0.106	0.100	$0.101$	0.105	$0.106$	$0.167$	$0.108$	$0.132$
M3	MASE	$1.652$	1.640	1.677	$1.767$	2.802	$3.083$	1.896	$2.060$	2.133	$2.134$	$2.011$	$1.477$	$1.687$
M3	WQL	0.090	$0.091$	0.089	$0.092$	0.116	$0.123$	0.099	$0.103$	$0.109$	0.107	$0.103$	$0.087$	$0.090$

Table 4: Forecasting performance by benchmark and architecture. For each seed, we select the checkpoint with the lowest validation MAE. Within each frequency, we average MASE across forecasts and compute WQL as a ratio of sums. We then macro-average both metrics across frequencies and report their means across 4 seeds. ScaleIn uses scale-invariant loss and ScaleCon uses scale-contaminated loss. Lower is better, and bold marks the better loss within each architecture.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example time series from the M1 competition dataset. The train and validation sets consist of all observations preceding the first dotted line. The cross validation test set is comprised of a set of forecast creation dates with lengths defined in Table 5 .

	Frequency	Seasonality	Horizon	Series	Min Length	Max Length	% Erratic
M1	Monthly	12	18	617	48	150	0
	Quarterly	4	8	203	18	114	0
	Yearly	1	6	181	15	58	0
M3	Other	4	8	174	71	104	0
	Monthly	12	18	1428	66	144	2
	Quarterly	4	8	756	24	72	1

Appendix

Table 5: Summary of forecasting datasets used in our empirical study.

Dataset	Source	Domain	Frequency	# Series	Avg	Min	Max	# Obs
Jena Weather	Autoformer [ wu2021autoformer ]	Nature	10T	1	52,704	52,704	52,704	52,704
Jena Weather	Autoformer [ wu2021autoformer ]	Nature	H	1	8,784	8,784	8,784	8,784
Jena Weather	Autoformer [ wu2021autoformer ]	Nature	D	1	366	366	366	366
BizITObs - Application	AutoMixer [ palaskar2024automixer ]	Web/CloudOps	10S	1	8,834	8,834	8,834	8,834
BizITObs - Service	AutoMixer [ palaskar2024automixer ]	Web/CloudOps	10S	21	8,835	8,835	8,835	185,535
BizITObs - L2C	AutoMixer [ palaskar2024automixer ]	Web/CloudOps	5T	1	31,968	31,968	31,968	31,968

Appendix

Table 6: Summary of GiftEval forecasting datasets used in our empirical study.

Figure 7: Adam, per-source nMSE over the first

200

steps (linear axis, mean over 5 seeds with

\pm 1

std bands; dashed grey = global nMSE). Left: under the scale-invariant loss all three sources converge together. Right: under the scale-contaminated loss the curves spread out in scale order (high-scale first, low-scale last), the same disparity SGD shows but compressed into the first

{\sim}150

steps.

Figure 8: Adam control experiments (first

200

steps, linear axis, mean

\pm 1

std over 5 seeds). Left: on equal-scale data the scale-contaminated spread collapses to the scale-invariant baseline. Right: under gradient-norm-matched scale-contaminated training the scale-ordered spread persists.

Panel of Figure 9 — Figure 9: Adam, gradient magnitude and global convergence (mean $\pm 1$ std over 5 seeds). The initialization gradient ratio is unchanged from SGD and scales as $b^{p}$ under the scale-contaminated loss. Adam rescales coordinate magnitude but not the scale-driven ordering of convergence.

Panel of Figure 10 — Figure 10: Adam, qualitative per-source forecasts at training steps $0$ , $10$ , $50$ , $100$ , $500$ , and $30{,}000$ .

Figure 11: Dataset-level trajectories under scale-invariant loss and learning-rate-adjusted scale-contaminated loss. Under scale-invariant loss, the

b=1

and

b=10

trajectories overlap because the loss is computed on scaled targets. Lines and bands show the mean and 95% Student-

t

interval across

15

assignment contexts after averaging

5

seeds within each context.

Figure 12: Cross-dataset variance of per-dataset MASE over

80{,}000

updates for all four architectures. Lines show the mean across

4

seeds and bands show 95% Student-

t

intervals. Lower variance indicates more uniform accuracy across datasets, but does not by itself imply lower forecast error.

Figure 13: Pooled validation MASE over

80{,}000

updates for all four architectures. The pooled mean weights usable forecast cases in their natural dataset proportions, unlike the dataset-balanced median in Figure 5(b) . Lines show the mean across

4

seeds and bands show 95% Student-

t

intervals. Lower is better.

		Chronos2			Moirai-2.0			TimesFM2.5			PatchTST			Reference
Benchmark	Freq.	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	S.Naive
M1	Y	4.124 $\pm 0.445$	6.492 $\pm 0.450$	-36.5	4.894 $\pm 0.150$	5.175 $\pm 0.406$	-5.4	8.067 $\pm 0.509$	7.765 $\pm 0.291$	+3.9	4.707 $\pm 0.104$	7.600 $\pm 0.823$	-38.1	4.894
	Q	1.827 $\pm 0.056$	1.997 $\pm 0.035$	-8.5	1.814 $\pm 0.056$	1.885 $\pm 0.053$	-3.8	1.943 $\pm 0.042$	2.394 $\pm 0.088$	-18.8	1.963 $\pm 0.078$	3.625 $\pm 0.917$	-45.9	2.078
	M	1.238 $\pm 0.019$	1.312 $\pm 0.013$	-5.6	1.234 $\pm 0.011$	1.306 $\pm 0.024$	-5.5	1.425 $\pm 0.018$	1.579 $\pm 0.015$	-9.8	1.366 $\pm 0.018$	2.838 $\pm 1.050$	-51.9	1.314
M3	Y	3.070 $\pm 0.045$	4.085 $\pm 0.241$	-24.8	3.119 $\pm 0.127$	3.643 $\pm 0.171$	-14.4	4.880 $\pm 0.198$	5.309 $\pm 0.188$	-8.1	3.889 $\pm 0.465$	9.529 $\pm 2.353$	-59.2	3.154
	Q	1.258 $\pm 0.060$	1.461 $\pm 0.072$	-13.9	1.255 $\pm 0.039$	1.295 $\pm 0.077$	-3.1	1.476 $\pm 0.016$	1.846 $\pm 0.110$	-20.1	1.590 $\pm 0.142$	3.867 $\pm 1.043$	-58.9	1.425

Appendix

Table 7: In-domain MASE by benchmark and frequency after 80,000 updates, means with sample standard deviations (

\pm

, below) across 4 seeds. A star marks a base frequency that pools anchored or multiplied GIFT-Eval aliases. ScaleIn uses scale-invariant loss and ScaleCon uses scale-contaminated loss. These are the only differences between the two runs of each architecture.

\Delta\%

		Chronos2			Moirai-2.0			TimesFM2.5			PatchTST			Reference
Benchmark	Freq.	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	Sc.In	Sc.Con	$\Delta\%$	S.Naive
M1	Y	0.110 $\pm 0.006$	0.126 $\pm 0.018$	-12.8	0.154 $\pm 0.007$	0.157 $\pm 0.006$	-2.1	0.248 $\pm 0.009$	0.287 $\pm 0.018$	-13.6	0.196 $\pm 0.009$	0.307 $\pm 0.036$	-36.1	0.184
	Q	0.065 $\pm 0.008$	0.076 $\pm 0.007$	-13.7	0.094 $\pm 0.004$	0.095 $\pm 0.008$	-1.2	0.115 $\pm 0.013$	0.142 $\pm 0.008$	-18.9	0.118 $\pm 0.009$	0.148 $\pm 0.024$	-20.2	0.117
	M	0.162 $\pm 0.016$	0.131 $\pm 0.007$	+23.5	0.207 $\pm 0.024$	0.188 $\pm 0.021$	+10.0	0.227 $\pm 0.006$	0.261 $\pm 0.021$	-12.9	0.272 $\pm 0.015$	0.322 $\pm 0.066$	-15.7	0.150
M3	Y	0.101 $\pm 0.001$	0.112 $\pm 0.006$	-10.1	0.132 $\pm 0.004$	0.143 $\pm 0.004$	-7.3	0.218 $\pm 0.009$	0.221 $\pm 0.006$	-1.3	0.170 $\pm 0.010$	0.283 $\pm 0.039$	-40.0	0.118
	Q	0.063 $\pm 0.003$	0.066 $\pm 0.002$	-4.8	0.079 $\pm 0.003$	0.080 $\pm 0.003$	-0.5	0.104 $\pm 0.001$	0.126 $\pm 0.007$	-17.7	0.110 $\pm 0.006$	0.202 $\pm 0.034$	-45.4	0.082

Appendix

Table 8: In-domain WQL by benchmark and frequency after 80,000 updates, means with sample standard deviations (

\pm

\Delta\%

Frequency	Horizon	Context	Eligible series
Yearly	8	16	7,841
Quarterly	8	16	23,239
Monthly	24	48	34,183
Weekly	13	26	359
Daily	14	28	4,227
Hourly	48	96	414

Appendix

Table 9: Frequency-specific dimensions and eligible training populations in the supervised experiment. The population is fixed before comparing loss spaces and is shared by the neural and statistical methods.

Setting	Value
Optimizer	Adam
Training updates	20,000
Series batch size	32
Sampled windows per batch	512
Learning rate	$10^{-3}$
Learning-rate decays	3, evenly spaced

Appendix

Table 10: Shared supervised training and evaluation settings.

Architecture	Parameters
NHITS	Three identity stacks, one block per stack, two hidden layers of 512 units per block, pooling kernels $(2,2,1)$ , frequency downsampling $(4,2,1)$ , ReLU activation, and linear interpolation.
NBEATS	Identity, trend, and seasonality stacks, one block per stack, two hidden layers of 512 units per block, ReLU activation, and no shared block weights.
PatchTST	Three encoder layers, 16 attention heads, hidden width 128, feed-forward width 256, patch length 16, stride 8, dropout 0.2, GELU activation, and internal RevIN disabled.
TCN	Kernel size 2, dilations $(1,2,4,8,16)$ , encoder width 128, context width 10, and a two-layer decoder of width 128.
LSTM	Two encoder layers of width 128, no encoder dropout, and a two-layer decoder of width 128. Forecasts use the direct decoder rather than recursive generation.

Appendix

Table 11: Architecture-specific parameters for the supervised experiment. Parameters not shown use the NeuralForecast 3.2.1 defaults.

Architecture	Benchmark	Frequency	ScaleIn	ScaleCon	S.Naive	ETS	ARIMA
N-HiTS	M1	Y	2.368 $\pm 0.051$	2.461 $\pm 0.171$	3.046	2.476	2.632
		Q	1.859 $\pm 0.004$	1.857 $\pm 0.012$	2.527	1.899	1.966
		M	1.251 $\pm 0.012$	1.248 $\pm 0.009$	1.859	1.338	1.333
	M3	Y	2.972 $\pm 0.346$	2.934 $\pm 0.092$	3.333	2.384	3.030
		Q	1.121 $\pm 0.006$	1.127 $\pm 0.014$	1.457	1.115	1.130
		M	0.863 $\pm 0.012$	0.860 $\pm 0.009$	1.243	0.932	0.901

Appendix

Table 12: Best-checkpoint MASE by benchmark and frequency in the supervised experiment. Neural values are mean

\pm

sample standard deviation across four seeds. ScaleIn uses scale-invariant MAE and ScaleCon uses scale-contaminated MAE, both with the same input-window standard scaling. References are fitted at every rolling origin on the same eligible series. Lower is better, and bold marks the better loss within each matched pair.

Architecture	Benchmark	Frequency	ScaleIn	ScaleCon	S.Naive	ETS	ARIMA
N-HiTS	M1	Y	0.119 $\pm 0.003$	0.117 $\pm 0.007$	0.219	0.144	0.204
		Q	0.096 $\pm 0.002$	0.096 $\pm 0.001$	0.139	0.094	0.106
		M	0.083 $\pm 0.001$	0.081 $\pm 0.003$	0.144	0.088	0.086
	M3	Y	0.105 $\pm 0.005$	0.107 $\pm 0.006$	0.105	0.093	0.100
		Q	0.081 $\pm 0.001$	0.082 $\pm 0.001$	0.092	0.079	0.081
		M	0.083 $\pm 0.001$	0.083 $\pm 0.001$	0.111	0.089	0.088

Appendix

Table 13: Best-checkpoint WQL by benchmark and frequency in the supervised experiment. Neural values are mean

\pm

Figure 14: MOMENT reconstruction on held-out GiftEvalPretrain series. Plotted are average per-seed median dataset MASE, with

95\%

Student-

t

intervals across 4 seeds.

Explore similar work

Loss-Guided Pretraining Data Selection for Time-Series Foundation Models
Sep 29, 2026Yike Li, Shaoxu Song, Jianmin WangTime Series Foundation Models Pretraining
Tsinghua University
Self-Adaptive Scale Handling for Forecasting Time Series with Scale Heterogeneity
Jun 18, 2026Xu Zhang, Zhengang Huang, Yunzhi Wu +6Time Series Forecasting Time Series
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Ant Group, Shanghai, China
Into the ORBIT for Time Series: Training Regimes for Foundation Models
Aug 13, 2026Hongjie Xia, Yiding Liu, Yifan Hu +2Time Series Foundation Models Zero-Shot Forecasting

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

Yike Li, Shaoxu Song, Jianmin Wang

Tsinghua University

Time Series Foundation Models Pretraining Foundation Model Loss Function Cross-Architecture Generalization

Open paper AskPDF arXiv

Time series foundation models (TSFMs) are pretrained on heterogeneous collections containing billions of observations, yet their training windows are typically sampled without estimating whether they provide useful learning signal. We introduce a static data-selection framework that scores each window with a reference forecaster and retains an intermediate interval within every source dataset. Specifically, we connect forecasting loss to optimization difficulty by showing that normalized squared loss controls the per-sample gradient norm under a local Jacobian condition. We then define a reference loss score and apply dataset-stratified selection to preserve the diversity of samples. Across various TSFM architectures, our method outperforms random selection by an absolute margin and even improves both relative MASE and CRPS over full-data pretraining by retaining fewer candidate pretraining windows. Further analyses show strong cross-scale and cross-architecture score correlations, indicating that a small reference model can often select data for larger targets, provided that the reference and target share compatible difficulty orderings.

Abstract

Explore similar work

Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

Self-Adaptive Scale Handling for Forecasting Time Series with Scale Heterogeneity

Into the ORBIT for Time Series: Training Regimes for Foundation Models