IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models

Vebjørn Haugland; Pavel Gulin; David Foss; Peet Cremer; Erlend Aune; Simone Casolo; Andreas Solbrå; Milad Mobarhan

cs.LGOct 12, 2026

IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models

Authors: Vebjørn Haugland, Pavel Gulin, David Foss, Peet Cremer, Erlend Aune, Simone Casolo, Andreas Solbrå, Milad Mobarhan

Organizations: Cognite

Abstract

We introduce IND-Eval, a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks. Relative to established benchmarks such as GIFT-Eval, it emphasizes finer sampling resolutions and a greater prevalence of multivariate forecasting. Evaluating Chronos-2, TiRex-2, and TimesFM-3, we find that TimesFM-3 achieves the lowest source-balanced aggregate nMASE and nCRPS and leads seven of nine sources under each metric; source-specific exceptions reinforce the need for domain-representative evaluation for model selection. These findings also motivate improving domain alignment during pretraining, where real industrial data are often too scarce to close domain-specific gaps directly; we therefore investigate whether a domain's statistical structure can guide simulator design. In a case study on two offshore well-monitoring regimes, we improve a general simulator's coverage using an oscillator-augmented simulation pipeline. This lowers forecasting error in the target domain but reduces performance on the subset of GIFT-Eval with natively univariate targets. A 50/50 mixture of the general and augmented simulation recipes achieves the lowest error among the three training recipes for all four event-metric pairs in the case study while recovering most of the lost GIFT-Eval performance. Together, these results illustrate how domain-representative evaluation can guide model selection and the composition of simulated data while revealing a trade-off between specialization and generality.

Figures & tables

Panel of Figure 1 — Figure 1: Structural coverage of IND-Eval and GIFT-Eval. Left: Source-balanced empirical cumulative distribution functions of sampling interval, context span, and forecast span. Right: Source-balanced distribution of the number of target variates per task.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Recorded system	Task construction	Tasks
NEATS	Offshore drilling streams with up to 12 canonical drilling variables at 1, 5, 10, or 12-second cadence	21 well–event groups, each evaluated at 512- and 1,024-sample horizons	42
3W	Multivariate oil-well instances organized into ten operating-event groups	One task per event and 512- or 1,024-sample horizon	20
PRONTO	Six annotated multiphase-flow transients with 12 targets and three known-future control variables	One task per annotated event and 512- or 1,024-sample horizon	12
Mesa del Sol	Fifteen months of microgrid power, voltage, frequency, and temperature measurements	Gap-bounded segments at 10-second, 2-minute, and 5-minute resolutions; 512-sample horizon	3
Tablet compression	Commercial batches with five direct-compression process targets	25 product-code groups at 30-, 60-, and 180-sample horizons; one shared origin per batch	75
MetroPT-3	Five channels from a metro train air-production unit	190 leak-free, gap-bounded instances at 10-second cadence, evaluated at 30-, 60-, and 180-sample horizons	3

Appendix

Table 3: Current IND-Eval source and task construction. Each task combines one source-specific series group and forecast horizon.

Source	Cleaning and resampling	Forecast-origin and exclusion rule
NEATS	Canonicalize channel aliases, units, and timestamps; retain up to 12 drilling targets; split at every cadence discontinuity; remove a channel if it is more than 80% missing, has at most three distinct values, or is flat on more than 75% of valid transitions.	Take up to five forward-anchored windows per source instance, 2,048 samples apart. An instance contributes only to horizons for which a full context and target fit.
3W	Apply event-specific extraction, including a stitched transient and steady interval for event 1; mean-downsample to 2-second cadence except event 4, which remains at 1 second; interpolate sensor gaps; remove sparse, frozen, spike-dominated, and low-range channels.	Take up to two forward-anchored windows per instance. Except for event 9, drop a window only if every retained channel’s final 1,000 context points plus target span less than 5% of its earlier context range. This rule inspects the target interval.
PRONTO	Rename and align 15 process channels, then average the native 1-Hz data into 2-second bins. Twelve response channels are targets; Air In, Water In, and Air Valve are known-future covariates.	Fix origins at the six annotated events, use up to 4,096 pre-event samples, and skip an event–horizon pair when its pre-event history or post-event target is shorter than the horizon.
Mesa del Sol	Mask telemetry sentinels, known invalid codes, and short negative spikes; aggregate most channels by mean and one setpoint by its last value at 10-second, 2-minute, and 5-minute cadence. Mean bins require 80% native-sample coverage. Remove two near-constant setpoint channels and split at fully missing bins.	Take every non-overlapping 512-step window that fits within each gap-bounded segment.
Tablet compression	Preserve acquisition order; split when the batch or active-production state changes, time moves backwards, or a gap occurs; mask physically invalid SREL >15. Five process channels are targets.	Choose the earliest origin with at least 512 active context rows and a finite 180-step future for every target. Reuse that origin for the 30-, 60-, and 180-step horizons. This rule inspects the longest target interval.
MetroPT-3	Sort by timestamp, remove four reported leak intervals, and split at the resulting row discontinuities or timestamp gaps over 15 seconds. No value resampling, interpolation, imputation, or scaling is applied to the output.	Use the latest origin with a shared 180-step future and 512–4,096 context samples. Remove dead-tail instances and require mean normalized context variability of at least 0.60 across four activity channels, with at least two channels at 0.10 or above. The dead-tail rule inspects the target interval.

Appendix

Table 4: Source-specific preprocessing, forecast-origin selection, and exclusions. Rules that inspect the target interval are stated explicitly; remaining implementation details and exact source versions are included with the released preprocessing scripts.

Source	Cadence	Horizon (samples)	Horizon (physical)	Seasonal period $m$
NEATS	1, 5, 10, 12 s	512, 1,024	8.5 min–3.4 h	1
3W	2 s (event 4: 1 s)	512, 1,024	8.5–34 min	1
PRONTO	2 s	512, 1,024	17–34 min	1
Mesa del Sol	10 s, 2 min, 5 min	512	85 min, 17 h, 43 h	1, 720, 288
Tablet compression	10 s (nominal)	30, 60, 180	5–30 min	1
MetroPT-3	10 s	30, 60, 180	5–30 min	1

Appendix

Table 5: Cadence, forecast horizon, and seasonal-naive period per source. Physical horizons are horizon samples times cadence; seasonal periods are measured in samples and follow the cadence order shown in each row.

Figure 3: Source-balanced densities of the eight benchmark-comparison statistics: ACF-1, ACF-5, spectral entropy, ARCH proxy

R^{2}

, KPSS, skewness, linear-trend correlation, and normalized flat-spot length. Within each benchmark, every source dataset contributes equal total weight. KPSS and skewness use

\log(1+x)

and

\mathrm{asinh}(x)

scales, respectively, with tick labels shown in the original units.

Statistic	GIFT-Eval	IND-Eval
ACF-1	0.895	0.960
ACF-5	0.555	0.823
Spectral entropy	0.559	0.428
ARCH proxy $R^{2}$	0.667	0.845
KPSS	1.061	2.164
Skewness	0.581	0.091

Appendix

Table 6: Source-balanced medians for the eight statistics in Figure 3 . Each source dataset contributes equal total weight within its benchmark; values are in the original units.

	nMASE $\downarrow$		nCRPS $\downarrow$
Source	Naive	S-naive	Naive	S-naive
Aggregate	1.217	1.000	1.294	1.000
3W	1.000	1.000	1.000	1.000
NEATS	1.000	1.000	1.000	1.000
PRONTO	1.000	1.000	1.000	1.000
Mesa del Sol	1.280	1.000	1.410	1.000

Appendix

Table 7: Naive and seasonal-naive baselines on IND-Eval. Sources are ordered as in Table 1 ; Aggregate weights all nine sources equally.

Setting	Chronos-2	TiRex-2	TimesFM-3
Checkpoint	amazon/chronos-2	NX-AI/TiRex-2	google/timesfm-3.0-pytorch ; timesfm 3.0.1
Targets	Joint within multivariate windows; separate across windows	Joint within multivariate windows; separate across windows	Joint within multivariate windows; separate across windows
PRONTO controls	Split into past and known-future covariate dictionaries	Supplied as one context-plus-future covariate tensor	Supplied as past_future_covariates
Quantiles	Requested at 0.1–0.9	Checkpoint-native 0.1–0.9	Checkpoint-native 0.1–0.9
Horizon handling	Native for all tasks (adapter cap 2,048)	Chunks above checkpoint future_len , feeding back the median forecast	Native for all tasks (adapter cap 2,048)
Other settings	cross_learning=False ; float32 inference	Checkpoint-native forecast routine	use_symmetric_averaging , make_positive , and sort_quantiles enabled

Appendix

Table 8: Model-specific inference settings for Table 1 .

Industrial core sources and documented benchmark overlap
			Pretraining-overlap evidence
Source	Rel.	Prior forecasting benchmark(s)	Chronos-2	TiRex-2	TimesFM-3
3W	2019	none	unresolved	unresolved	unresolved
NEATS	unknown	none	unresolved	unresolved	unresolved
PRONTO	2019	none	unresolved	unresolved	unresolved
Mesa del Sol	2023	none	unresolved	unresolved	unresolved

Appendix

Table 9: Model–source contamination audit for Table 1 . Rel. is the first verified public release year. For NEATS, the records were accessed in 2026 but their first public release could not be verified. Prior forecasting benchmarks are GIFT-Eval ( Aksu et al., 2024 ) , TIME ( Qiao et al., 2026 ) , the Monash archive ( Godahewa et al., 2021 ) , and the LTSF benchmark ( Zeng et al., 2023 ) . Evidence is documented when official model documentation names the source or a linked derivative, possible for plausible indirect overlap, and unresolved when public documentation does not establish either case. These labels do not establish exact sample duplication. No source is evaluated only on a segment that postdates a model cut-off.

Figure 4: Example harmonic-oscillator augmentation. A directed force drives a random planar spring network (left), where node size indicates displacement magnitude. Projected displacements at sensor nodes produce rich temporal responses (right).

Statistic	3W	SarSim0	SarSim0-oscillator
Robust skewness	$-$ 0.035	0.000	$-$ 0.000
Roughness	0.0013	1.110	0.0046
Trend strength	0.132	0.082	0.735
Spectral entropy	0.204	0.782	0.297
ACF level	0.999	0.386	1.000
ACF decay	0.002	0.682	0.001

Appendix

Table 10: Median values of the ten comparison statistics. Roughness is shown before the logarithmic transform used for PCA.

Statistic	Leading component
ACF level	0.505
Spectral entropy	$-$ 0.498
ACF decay	$-$ 0.442
Log roughness	$-$ 0.384
Trend strength	0.369
Isolated spike freq.	$-$ 0.093

Appendix

Table 11: Loadings of the standardized statistics on the leading principal component (34% explained variance).

Hyperparameter	Value
Optimizer	NorMuon + AdamW
Learning rate	$3{\times}10^{-4}$
Weight decay	0.01
Training steps reported	50k
Effective batch size	4,096 (2,048 $\times$ 2 accumulation)
Context length	4,096

Appendix

Table 12: Model and training hyperparameters for the simulation experiment. All recipes use the same settings; only the simulator recipe differs.

Figure 5: Joint t-SNE projections of windows from 3W ev3/ev4 and both simulators in (a) the 256-dimensional resampled-shape space and (b) the 29-dimensional log-band power spectrum space. The panels are independent embeddings of two different representations of the same windows, not transforms of one another. Axes and global geometry have no intrinsic interpretation.

Representation (dim.)	Simulator	NN rate (%)	Sibling-free (%)
Shape (256)	SarSim0	6.7	7.6
	SarSim0-oscillator	19.9	27.6
Log-band power (29)	SarSim0	2.5	3.3
	SarSim0-oscillator	17.0	24.9
Autocorrelation (29)	SarSim0	4.4	6.3
	SarSim0-oscillator	17.1	27.8

Appendix

Table 13: Window-level alignment of each simulator with 3W ev3/ev4. NN rate is the fraction of 3W windows whose nearest neighbor in the pool comes from the case-study simulator; the sibling-free variant excludes candidates from the query window’s own 3W instance.

Explore similar work

TempusBench: An Evaluation Framework for Time-Series Forecasting
Apr 13, 2026Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade +14Time Series Forecasting Forecasting Benchmarks
Simulacrum New York City, NY, USA
fev-bench: A Realistic Benchmark for Time Series Forecasting
Sep 30, 2025Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5Benchmark Design Time Series Forecasting
AWS · Keystone AI · Prior Labs +2
QuitoBench: A High-Quality Open Time Series Forecasting Benchmark
Mar 27, 2026Siqiao Xue, Zhaoyang Zhu, Wei Zhang +7Time Series Forecasting Forecasting Benchmarks
Ant Group

Apr 13, 2026J/K move · Enter open · S save

TempusBench: An Evaluation Framework for Time-Series Forecasting

Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade, Alif Abdullah+13

Simulacrum New York City, NY, USA

Time Series Forecasting Forecasting Benchmarks Time-Series Foundation Models

Open paper AskPDF arXiv

Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://smlcrm.com/tempusbench.

Abstract

Explore similar work

TempusBench: An Evaluation Framework for Time-Series Forecasting

fev-bench: A Realistic Benchmark for Time Series Forecasting

QuitoBench: A High-Quality Open Time Series Forecasting Benchmark