LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models

Haomin Wen; Ziyu Zhou; Qingxiang Liu; Siru Zhong; Yuxuan Liang

cs.AIAug 18, 2026

LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models

Authors: Haomin Wen, Ziyu Zhou, Qingxiang Liu, Siru Zhong, Yuxuan Liang

Organizations: Shanghai Innovation Institute · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Time Series Foundation Models (TSFMs) have recently emerged as a highly promising paradigm for cross-domain zero-shot forecasting. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how models behave in continuously evolving real-world environments characterized by seasonal variations, distribution shifts, and unexpected events. To bridge this gap, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. By evaluating models prequentially on real future data in open-world environments, LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. Rather than acting as a one-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol.

Figures & tables

Figure 1. Comparison between proposed LiveHouse-TS and static benchmark. LiveHouse-TS introduce a streaming evaluation protocol to capture model robustness under real-world operational conditions. It eliminates data leakage by requiring models to forecast at the current world time before ground truth is observed, with evaluation and the leaderboard updated continuously as new data arrives.

Benchmark	Year	Dataset			Evaluation				Leaderboard
		Live/Static	#Domain	#Data	LR	Test zero-shot?	FT	Multivariate	RTL	OND	Metrics
Monash ( Godahewa et al., 2021 )	2021	Static	7	20	✘	✘	Point	✘	✘	✘	MASE, sMAPE, msMAPE, MAE, RMSE
BasicTS ( Shao et al., 2023 )	2023	Static	5	20	✘	✘	point	✔	✘	✘	MAE, RMSE, MAPE, WAPE
TFB ( Qiu et al., 2024 )	2024	Static	10	41	✘	✘	Point	✔	✘	✔	MAE, MSE, MASE, MSMAPE
ProbTS ( Zhang et al., 2024 )	2024	Static	6	12	✘	✔	Point/Prob.	✔	✘	✘	NMAE, CRPS
CiK ( Williams et al., 2025 )	2025	Static	7	9	✘	✔	Prob.	✘	✘	✘	CRPS

Table 1. Comparison between representative TSFM benchmarks and LiveHouse-TS. (Abbr; LR: Leakage-resistant by nature; RTL: Real-time Leaderboard; OND: Open-to-new-data; FT: Forecasting Task, Prob.: Probability Forecasting). LiveHouse-TS provides the first systematic solution for benchmarking the live performance of time series foundation models.

Figure 2. Overall architecture of LiveHouse-TS, as an analogy in which models are performers and the benchmark is a live house . It contains three decoupled yet coordinated components: $i)$ a Model Entrance that “checks tickets” and standardizes heterogeneous forecasters before they take the stage; $ii)$ Live Data that turns public streams into a rolling set of forecasting tasks according to the dataset description (e.g., frequency, context length, and forecast horizon); and $iii)$ an Evaluation House (the live house) that enforces the future-only rule, and updates the leaderboard.

Figure 3. Dataset examples with representative temporal patterns. Each panel shows one selected dataset window at its native cadence; axes are normalized independently to highlight temporal shape rather than absolute magnitude. The upper-right label gives the displayed window span, and the lower-right label is the native sampling frequency.

Group	#	Domains	Native freq.	Eval freq.	Sources	Targets
Environment	5	weather, air quality, weather-energy	1h (4), 1d	1h (4), 1d	Open-Meteo, NASA POWER, NWS, NOAA NCEI	temperature, PM 2.5
Water	3	hydrology, ocean	6min, 10min, 15min	1h	USGS Water, NOAA CO-OPS, NOAA NDBC	discharge, water level, wave height
Mobility	1	traffic	15min	15min	GBFS Citi Bike	available bikes
Finance	4	finance	1s, 1h (2), 1mo	1s, 1h (2), 1mo	Binance, CoinGecko	close, market price
Society/economy	2	web attention, macro-economy	1d, 1y	1d, 1y	Wikimedia, World Bank	pageviews, GDP
Events	2	news events, disaster events	15min, 1h	1d	GDELT, USGS Earthquake	event volume, count

Table 2. Overview of streaming data in LiveHouse-TS (

n{=}17

datasets, 15 public sources). Native freq. is the sampling rate; Eval freq. is the recommended evaluation frequency. Both exclude crawler polling frequency.

Figure 4. Four measurements for dataset diversity. LiveHouse-TS is diverse enough both in terms of domain breadth, temporal scale, statistical variability, and task structure.

Figure 5. Coverage by domain and native sampling frequency. LiveHouse-TS evaluation covers from short-horizon high-rate forecast to long-horizon low-rate forecast.

Figure 6. Future-only evaluation under asynchronous model entry: models are scored only on targets released after they join the live leaderboard, to avoid leakage.

Measure	Metric	Foundation models								Classical baselines
Measure	Metric	Moi2	TFM	Chr2	TiRex	TabPFN	Sundial	ChrB	Toto1	ARIMA	MovAvg	ETS	SNaive
Average Rank ( $\downarrow$ )	RMSE	4.70	3.30	3.80	4.00	5.80	6.40	7.10	10.70	7.45	7.75	8.00	9.00
	MAPE	3.70	3.85	4.90	4.30	5.10	6.45	6.15	5.05	8.70	9.65	9.80	10.35
	CRPS	2.30	9.20	7.25	7.05	4.70	3.15	3.90	2.95	8.70	9.35	9.45	10.00
Win Rate ( $\uparrow$ )	RMSE	0.66	0.79	0.75	0.73	0.56	0.51	0.45	0.12	0.41	0.39	0.36	0.27
	MAPE	0.76	0.74	0.65	0.70	0.63	0.51	0.53	0.63	0.30	0.21	0.20	0.15

Table 3. Overall performance on LiveHouse-TS evaluated by RMSE, MAPE, and CRPS using Average Rank (

\downarrow

), Win Rate (

\uparrow

), and Elo Rating (

\uparrow

). All baselines are ordered by their average rank across the three metrics. The best and second-best results in each row are highlighted in red bold and blue underline , respectively. TSFMs consistently outperform classical statistical methods. Moirai-2.0 dominates probabilistic forecasting, while TimesFM-2.5 and Chronos–2 excel primarily in point accuracy.

Figure 7. CRPS-based average rank across seven domains. The results show that different TSFMs excel in distinct domains. Notably, the top-ranked Moirai-2.0 performs poorly on weather-related domains (Weather, Air Quality, and Ocean).

Figure 8. CRPS ranking of the shared TSFMs across three static benchmarks and the live benchmark LiveHouse-TS. The dotted segment for TabPFN-TS indicates that it is not evaluated in TIME. The three static benchmarks reach a similar consensus, whereas the live benchmark produces a pronounced ranking inversion, showing that offline (static) rankings can diverge from online performance and that a live benchmark is necessary to assess models under real deployment conditions.

Figure 9. Average Rank of Temporal Stability (blue) and Improvement (yellow). Moirai-2.0 ranks highest on both drift-robustness metrics, while several top static performers exhibit substantially lower temporal stability.

Panel of Figure 10 — Figure 10. Forecasting visualization on Wiki, Wave, PM 2.5, and Potomac datasets. The left of the dashed line denotes the historical context and the right the forecasting horizon. Per-model visualizations are provided in Figure 13 and Figure 14 . Although different TSFMs perform well on different datasets, they share common failure modes under evolving data distributions, including oversmoothing, delayed adaptation, and underestimated distribution shifts.

Figure 11. CRPS rankings over four consecutive weekly snapshots. The dynamic rankings demonstrate that model performance evolves with the incoming data stream, highlighting the need for continuous live evaluation rather than a single leaderboard snapshot.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Entity & region	Primary target	Public endpoint	Background notes
Open-Meteo Shanghai weather	weather	Shanghai grid (31.23 ∘ N, 121.47 ∘ E)	2 m temperature	https://api.open-meteo.com/v1/forecast	Smooth diurnal weather; humidity, wind, precipitation covariates
Open-Meteo Shanghai air quality	air quality	Shanghai grid	PM 2.5	https://air-quality-api.open-meteo.com/v1/air-quality	Pollutant spikes; PM 10 , NO 2 , CO covariates
NASA POWER Shanghai meteorology	weather-energy	Shanghai point	Air temperature (T2M)	https://power.larc.nasa.gov/api/temporal/hourly/point	Energy-oriented meteorology; evaluated daily despite hourly native series
USGS Potomac discharge	hydrology	USGS site 01646500, Potomac River	River discharge	https://waterservices.usgs.gov/nwis/iv/	High-frequency hydrology; gauge height covariate
NOAA CO-OPS San Francisco water level	ocean	Tide station 9414290, San Francisco	Water level	https://api.tidesandcurrents.noaa.gov/api/prod/datagetter	Six-minute coastal measurements with tide-like oscillation
NOAA NDBC buoy 46013	ocean	NDBC buoy 46013	Significant wave height	https://www.ndbc.noaa.gov/data/realtime2/46013.txt	Marine buoy observations; wind, pressure, temperature covariates

Appendix

Table 4. Background and public access information for each benchmark dataset. Endpoints are the base URLs used for collection; query parameters depend on entity, time window, and variables. Covariates may be omitted in univariate runs. We use the following abbreviations in subsequent tables and figures: BTC for Binance BTCUSDT, PM2.5 for Open-Meteo air quality, Quake for USGS earthquake aggregates, Potomac for USGS Potomac discharge, Water for NOAA CO-OPS water level, Wave for NOAA NDBC buoy observations, T2M for NASA POWER meteorology, KSFO for NWS KSFO observations, Temp2m for Open-Meteo weather, and Wiki for Wikimedia pageviews.

Dataset	Domain	Data freq.	Eval freq.	Target	Hist.	Hor.	Type	$N$	Mean	Std	Range
Open-Meteo weather	weather	1h	1h	temperature_2m	336	24	direct	32	26.97	2.746	24.0–33.7
Open-Meteo air quality	air_quality	1h	1h	pm2_5	336	24	direct	32	50.68	22.38	27.1–99.4
NASA POWER meteorology	weather_energy	1h	1d	T2M	336	24	direct	112	23.62	3.679	17.4–30.9
USGS Potomac water	hydrology	15min	1h	usgs_00060	96	24	direct	275	9,824	652.9	8,770–11,100
NOAA CO-OPS water level	ocean	6min	1h	water_level	240	60	direct	240	1.012	0.558	$-$ 0.042–1.84
NOAA NDBC buoy	ocean	10min	1h	wave_height	144	72	direct	76	2.034	0.187	1.8–2.4

Appendix

Table 5. Per-dataset registry and verification statistics. Data freq. and Eval freq. are native series frequency and recommended scoring frequency. Hist. and Hor. are history and forecast horizon in native-frequency steps. Type distinguishes directly reported time series from event-derived time series.

N

is row_count in the diversity verification slice (not a fixed train/test size). Mean, Std, and Range summarize the selected target variable in that slice.

Domain group	Observed pattern in verification samples
Weather & air quality	Diurnal structure (Open-Meteo, NWS) and pollutant spikes (PM 2.5 )
Weather-energy	Smooth exogenous meteorology suitable as covariate-rich context
Hydrology & ocean	Slower physical dynamics; sub-hourly coastal tides and buoy waves
Urban mobility	System-level bike availability rises/falls with commuting demand
Finance	Heavy-tailed movement from 1 s to monthly BTC series; CoinGecko adds market-level context
Web & macro	Sparse daily pageviews and slow annual GDP growth with long history

Appendix

Table 6. Representative dynamics observed in diversity verification samples (June 2026).

Dataset	Domain	$N$	CV	Observed dynamic pattern
Open-Meteo weather	weather	32	0.102	Smooth hourly diurnal temperature cycle
Open-Meteo air quality	air_quality	32	0.442	Pollutant variation with short spikes
NASA POWER meteorology	weather_energy	112	0.156	Smooth exogenous meteorology over several days
USGS Potomac water	hydrology	275	0.066	High-frequency river discharge with slow physical variation
NOAA CO-OPS water level	ocean	240	0.551	Sub-hourly tidal oscillation with values around zero
NOAA NDBC buoy	ocean	76	0.092	Marine wave-height fluctuations with meteorological covariates

Appendix

Table 7. Per-dataset verification variability used to support the dynamic-richness criterion in Figure 4 . CV is computed on the selected target variable in the verification slice as

\mathrm{std}/|\mathrm{mean}|

. These short slices illustrate temporal behavior but are not fixed train/test windows.

Figure 12. Verification-slice target trajectories for all 17 registry datasets. Each panel plots the locally parsed public sample for one dataset and target variable; panel titles give the dataset identifier and selected target.

Model	BTC	PM2.5	Quake	Potomac	Water	Wave	T2M	KSFO	Temp2m	Wiki
Chronos-2	483.614	11.420	2.830	19.675	0.318	0.892	1.293	1.078	1.074	8.420
TiRex	497.749	10.843	2.882	16.958	0.339	0.899	1.329	1.003	1.025	9.889
TimesFM-2.5	485.407	11.156	2.812	17.502	0.234	0.844	1.332	1.060	0.988	8.711
Toto-1.0	1.162e+03	40.693	5.314	58.600	0.539	1.093	1.578	1.208	1.835	96.250
Moirai-2.0	569.016	19.226	3.619	24.663	0.361	0.308	0.909	0.568	1.511	13.782
Chronos-Bolt	802.832	20.359	3.917	28.495	0.634	0.299	0.943	0.578	1.630	33.431

Appendix

Table 8. Per-dataset RMSE (

\downarrow

) on the online benchmark. In each column the best available model is in red bold and the second best is in blue underline .

Model	BTC	PM2.5	Quake	Potomac	Water	Wave	T2M	KSFO	Temp2m	Wiki
Chronos-2	0.010	0.329	0.565	0.009	0.469	0.158	0.043	0.043	0.043	0.061
TiRex	0.010	0.315	0.578	0.007	0.591	0.164	0.043	0.038	0.042	0.057
TimesFM-2.5	0.010	0.314	0.493	0.008	0.283	0.145	0.046	0.040	0.049	0.060
Toto-1.0	0.011	0.333	0.509	0.010	0.426	0.171	0.034	0.031	0.040	0.210
Moirai-2.0	0.008	0.341	0.483	0.009	0.437	0.178	0.031	0.031	0.045	0.046
Chronos-Bolt	0.011	0.364	0.546	0.010	0.722	0.176	0.030	0.031	0.045	0.134

Appendix

Table 9. Per-dataset MAPE (

\downarrow

) on the online benchmark. In each column the best available model is in red bold and the second best is in blue underline .

Model	BTC	PM2.5	Quake	Potomac	Water	Wave	T2M	KSFO	Temp2m	Wiki
Chronos-2	0.112	0.445	0.345	0.139	0.166	0.445	0.059	0.438	0.092	0.122
TiRex	0.118	0.436	0.347	0.139	0.174	0.428	0.056	0.408	0.100	0.118
TimesFM-2.5	0.136	0.518	0.481	0.172	0.147	0.505	0.071	0.499	0.111	0.161
Toto-1.0	0.010	0.245	0.273	0.010	0.186	0.128	0.027	0.027	0.036	0.132
Moirai-2.0	0.008	0.260	0.262	0.008	0.207	0.124	0.027	0.028	0.040	0.050
Chronos-Bolt	0.010	0.272	0.278	0.010	0.365	0.129	0.027	0.028	0.041	0.111

Appendix

Table 10. Per-dataset CRPS (

\downarrow

) on the online benchmark. In each column the best available model is in red bold and the second best is in blue underline .

Model	BTC	PM2.5	Quake	Potomac	Water	Wave	T2M	KSFO	Temp2m	Wiki
Chronos-2	7.329e+05	494.984	8.908	2.860e+03	0.271	3.756	1.756	27.049	1.953	129.300
TiRex	6.805e+05	463.452	8.575	2.414e+03	0.310	3.479	1.653	27.449	1.788	167.781
TimesFM-2.5	6.407e+05	499.814	8.574	2.685e+03	0.137	3.345	1.964	27.191	1.673	133.699
Toto-1.0	3.767e+06	8.411e+03	26.727	9.212e+03	0.653	7.889	7.376	3.933	6.619	8.228e+03
Moirai-2.0	4.540e+05	185.713	5.250	2.734e+03	0.192	0.088	0.375	0.737	1.538	64.521
Chronos-Bolt	6.818e+05	169.701	5.282	3.085e+03	0.358	0.052	1.235	0.981	2.238	238.712

Appendix

Table 11. Per-dataset Temporal Stability (

\downarrow

) on the online benchmark, computed from release-level MSE histories. Cells with fewer than two releases are shown as –. In each column the best available model is in red bold and the second best is in blue underline .

Model	BTC	PM2.5	Quake	Potomac	Water	Wave	T2M	KSFO	Temp2m	Wiki
Chronos-2	0.560	0.588	0.398	0.224	-0.033	-0.500	-0.138	-0.025	0.527	0.704
TiRex	0.565	0.545	0.429	0.242	-0.010	-0.504	0.144	-0.016	0.547	0.873
TimesFM-2.5	0.531	0.548	0.431	0.247	-0.051	-0.428	-0.002	-0.034	0.585	0.479
Toto-1.0	-0.309	0.059	-0.273	0.072	0.022	-0.085	0.344	0.123	0.022	1.000
Moirai-2.0	-0.355	-0.013	-0.351	0.132	-0.028	-0.639	-0.048	-0.195	-0.385	-1.000
Chronos-Bolt	0.090	0.307	-0.371	0.151	0.014	-0.098	-0.048	-0.143	-0.429	1.000

Appendix

Table 12. Per-dataset Improvement (

\downarrow

; more negative is better) on the online benchmark, computed as Kendall

\tau

on release-level MSE histories. Cells with fewer than two releases are shown as –. In each column the best available model is in red bold and the second best is in blue underline .

Metric	Chr2	TiRex	TFM	Toto1	Moi2	ChrB	TabPFN	Sundial	ARIMA	ETS	MovAvg	SNaive
Rank (RMSE) $\downarrow$	3.80	4.00	3.30	10.70	4.70	7.10	5.80	6.40	7.45	8.00	7.75	9.00
Win Rate (RMSE) $\uparrow$	0.745	0.727	0.791	0.118	0.664	0.445	0.564	0.509	0.414	0.364	0.386	0.273
Elo (RMSE) $\uparrow$	1225	1194	1254	637	1218	947	1161	1070	834	789	867	804
Rank (MAPE) $\downarrow$	4.90	4.30	3.85	5.05	3.70	6.15	5.10	6.45	8.70	9.80	9.65	10.35
Win Rate (MAPE) $\uparrow$	0.645	0.700	0.741	0.632	0.755	0.532	0.627	0.505	0.300	0.200	0.214	0.150
Elo (MAPE) $\uparrow$	1163	1251	1139	1055	1253	1064	1177	998	846	650	739	666

Appendix

Table 13. Overall zero-shot standing on the online benchmark aggregated over all datasets. For each metric we report Average Rank (

\downarrow

), Win Rate (

\uparrow

), and Elo (

\uparrow

) computed from the per-dataset values. In each row the best available model is in red bold and the second best is in blue underline .

Group	Metric	Chr2	TiRex	TFM	Toto1	Moi2	ChrB	TabPFN	Sundial	ARIMA	ETS	MovAvg	SNaive
Air Quality	Rank (RMSE) $\downarrow$	3.00	1.00	2.00	12.00	9.00	11.00	10.00	8.00	4.00	7.00	5.00	6.00
	Win Rate (RMSE) $\uparrow$	0.818	1.000	0.909	0.000	0.273	0.091	0.182	0.364	0.727	0.455	0.636	0.545
	Elo (RMSE) $\uparrow$	1451	1786	1601	218	684	397	548	807	1326	931	1191	1059
	Rank (MAPE) $\downarrow$	4.00	3.00	2.00	5.00	6.00	8.00	7.00	1.00	9.00	12.00	10.00	11.00
	Win Rate (MAPE) $\uparrow$	0.727	0.818	0.909	0.636	0.545	0.364	0.455	1.000	0.273	0.000	0.182	0.091
	Elo (MAPE) $\uparrow$	1312	1438	1588	1192	1068	817	950	1783	695	206	553	397

Appendix

Table 14. Zero-shot rankings on the online benchmark aggregated by domain. For each metric we report Average Rank (

\downarrow

), Win Rate (

\uparrow

), and Elo (

\uparrow

) computed from the per-dataset values within each domain. In each row the best available model is in red bold and the second best is in blue underline .

Group	Metric	Chr2	TiRex	TFM	Toto1	Moi2	ChrB	TabPFN	Sundial	ARIMA	ETS	MovAvg	SNaive
6min	Rank (RMSE) $\downarrow$	3.00	4.00	1.00	7.00	5.00	8.00	6.00	2.00	9.50	12.00	9.50	11.00
	Win Rate (RMSE) $\uparrow$	0.818	0.727	1.000	0.455	0.636	0.364	0.545	0.909	0.227	0.000	0.227	0.091
	Elo (RMSE) $\uparrow$	1436	1310	1780	944	1186	820	1062	1602	626	208	625	400
	Rank (MAPE) $\downarrow$	4.00	6.00	1.00	2.00	3.00	8.00	5.00	7.00	10.00	9.00	11.00	12.00
	Win Rate (MAPE) $\uparrow$	0.727	0.545	1.000	0.909	0.818	0.364	0.636	0.455	0.182	0.273	0.091	0.000
	Elo (MAPE) $\uparrow$	1310	1068	1783	1599	1446	817	1182	947	549	692	399	206

Appendix

Table 15. Zero-shot rankings on the online benchmark aggregated by sampling frequency. For each metric we report Average Rank (

\downarrow

), Win Rate (

\uparrow

), and Elo (

\uparrow

) computed from the per-dataset values within each frequency group. In each row the best available model is in red bold and the second best is in blue underline .

Pred. length	Metric	Chr2	TiRex	TFM	Toto1	Moi	Bolt	TabPFN	Sundial	ARIMA	ETS	MovAvg	SNaive
7	Rank (RMSE)	1	3	2	12	4	7	5	6	10	9	8	11
	Rank (MAPE)	3	2	5	8	1	7	4	6	9	11	10	12
	Rank (CRPS)	6	5	8	7	2	4	3	1	9	11	10	12
24	Rank (RMSE)	2	3	1	12	4	6	5	9	7	8	10	11
	Rank (MAPE)	5	4	6	2	1	8	3	7	10	9	11	12
	Rank (CRPS)	7	6	9	2	1	4	5	3	8	10	12	11

Appendix

Table 16. Zero-shot rankings on the online benchmark aggregated by forecast horizon. For each prediction length and metric, models are first ranked within each dataset and then ranked by their average rank across datasets sharing that prediction length (best

{}={}1

). In each row the best available model is in red bold and the second best is in blue underline .

Figure 13. Forecasting visualizations on Wiki (left column) and Wave (right column) for all baselines. Each row corresponds to one baseline. The left side of the dash line shows the context windows, and the right side shows the forecasting horizons.

Figure 14. Forecasting visualizations on PM 2.5 (left column) and Potomac (right column) for all baselines. Each row corresponds to one baseline. The left side of the dash line shows the context windows, and the right side shows the forecasting horizons.

Figure 16. Overall model ranking in the latest leaderboard snapshot. Bars show the dataset-balanced pairwise win rate aggregated over eligible metric-specific comparisons, historical releases, and opponents. Chronos-2 achieves the highest overall score, followed closely by TiRex. Higher values are better.

Figure 17. MSE-based ranking within the point-forecast track. Bars report dataset-balanced pairwise win rates over eligible historical releases and opponents. TimesFM-2.5 achieves the highest score. Higher values are better.

Figure 18. CRPS-based ranking restricted to the eight TSFMs whose model families natively support probabilistic forecasting through quantiles or forecast samples. Chronos-2 achieves the highest score. Higher values are better.

Forecasting Benchmarks

Forecasting Benchmarks are datasets and evaluation protocols for comparing predictions of future time-series values, events, and outcomes.

See all 224 papers

TempusBench: An Evaluation Framework for Time-Series Forecasting
Apr 13, 2026Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade +14Time Series Forecasting Forecasting Benchmarks
Simulacrum New York City, NY, USA
Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting
Mar 9, 2026Azul Garza, Renée Rosillo, Rodrigo Mendoza-Smith +5Forecasting Benchmarks Time Series Forecasting
TimeCopilot · Independent Researcher · ELLIS Institute Tübingen +3
Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation Models
Oct 19, 2026Hongyin Zhang, Mingyi Xiang, Rongzhao Chen +7Forecasting Benchmarks Time-Series Foundation Models
School of Software, BNRist, Tsinghua University · Timecho Ltd, Beijing, China · Harbin Institute of Technology

Apr 13, 2026J/K move · Enter open · S save

TempusBench: An Evaluation Framework for Time-Series Forecasting

Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade, Alif Abdullah+13

Simulacrum New York City, NY, USA

Time Series Forecasting Forecasting Benchmarks Time-Series Foundation Models Time Series Modeling

Open paper AskPDF arXiv

Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://smlcrm.com/tempusbench.

Abstract

Forecasting Benchmarks

Explore similar work

TempusBench: An Evaluation Framework for Time-Series Forecasting

Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting

Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation Models