WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting

Arjun Ashok; Mihir Parmar; Palash Goyal; Bhavana Dalvi Mishra; Chun-Liang Li; Irina Rish; Alexandre Drouin; Jinsung Yoon

cs.AIOct 19, 2026

WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting

Authors: Arjun Ashok, Mihir Parmar, Palash Goyal, Bhavana Dalvi Mishra, Chun-Liang Li, Irina Rish, Alexandre Drouin, Jinsung Yoon,

Organizations: Google · Mila - Quebec AI Institute · Université de Montréal · Université Laval

Abstract

Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors. This enables scenario analysis: forecasting how a target responds to events that set covariates to chosen future values. Whether they can do this reliably is an open question. We introduce WhatIf-Bench, a benchmark measuring how well forecasters use covariates to forecast a target's response to future events. WhatIf-Bench is semi-synthetic: time series come from mechanistic simulators of data centers, power grids, road traffic, and water networks, with controlled covariate-target relationships grounded in each system. Each task presents a history of events intervening on covariates, then asks to forecast a future event that either replays seen values (Replay) or sets them to unseen ones (Novel). Varying the number of covariates, historical events, and event length yields 37,665 tasks. Evaluating TSFMs, frontier LLMs, and simple regression baselines, we find the LLMs reach the highest skill (improvement over a naive baseline), ahead of the best TSFMs. Every model forecasts well on Replay futures (48 to 85% skill) but drops to -7 to 50% on Novel futures, with the regression baselines falling the most. Skill on Novel futures also falls as events set more covariates jointly. More historical events raise skill on Novel futures across the board, but large differences remain. WhatIf-Bench reveals generalization to Novel futures as the core challenge, and offers a testbed toward arbitrary scenario analysis.

Figures & tables

Figure 1: Performance of , , and on WhatIf-Bench tasks. Every model’s skill score drops sharply from Replay to Novel futures.

Figure 2: Illustration of WhatIf-Bench tasks. Left: a WhatIf-Bench task contains a history of events, in which covariates are set and the target responds, and asks a model to forecast the target under one further event. Right: the same events in covariate space. A Replay future repeats a past event (here, the third), so it lies inside the range of values the history has shown; a Novel future sets the covariates beyond that range, and is one of many possible what-if scenarios. Novel futures thus test whether a model extends what it learned to unseen values. Figure 1 shows that current models do this poorly: their skill on Novel futures is far below their skill on Replay futures.

Benchmark	Data source	Includes covariates	Covariate effect known exactly	Evaluates arbitrary what-if scenarios	Systematic difficulty (# covariates, historical evidence)	Tasks with covariates
Monash ( Godahewa et al., 2021 )	Real-world	✗	✗	✗	✗	0
GIFT-Eval ( Aksu et al., 2024 )	Real-world	✗	✗	✗	✗	0
TSFM-Bench ( Li et al., 2025 )	Real-world	✗	✗	✗	✗	0
TIME ( Qiao et al., 2026 )	Real-world	✗	✗	✗	✗	0
fev-bench ( Shchur et al., 2025 )	Real-world	✓	✗	✗	✗	46
Berthelier et al. (2026)	Semi-synthetic (toy relations)	✓	✓	✗	✗	– †

Table 1: Comparison of WhatIf-Bench with existing forecasting benchmarks. WhatIf-Bench supports evaluation on arbitrary what-if scenarios and lets difficulty be varied systematically, from few to many covariates and from few to many past observations of their effects, exposing differences between models that existing benchmarks cannot show.

Figure 3: The WhatIf-Bench construction recipe. : We (1) pick a simulator and a base setting; (2) choose a target and the covariates that drive it; (3) select the parameters of the task; finally (4) construct the events in the history, to produce two futures over the same history. This yields

2

tasks. We generate the benchmark by repeating this process for every target across all simulators, and over all axes: number of covariates, number of events, event length, and context length.

Benchmark composition
Simulator	Domain	# Targets
OpenDC ( Iosup et al., 2017 )	Data center	$22$
GridLAB-D ( Chassin et al., 2008 )	Power grid	$11$
SUMO ( Lopez et al., 2018 )	Road traffic	$30$
EPANET ( Rossman, 2000 )	Water network	$37$
Total		$\mathbf{100}$

Table 2: The WhatIf-Bench instantiation. The number of series contributed by the four simulators, and the axes we vary over them to build the tasks. The exact context and prediction lengths per simulator are given in Table 4 .

Model	wQL Skill	WAPE Skill	Inf. time / series (s)
Frontier LLMs
Claude-Opus-4.8 (High)	72.5%	72.5%	270.13
Claude-Opus-4.8 (Medium)	70.3%	70.3%	206.29
Gemini-3.1-Pro (High)	70.0%	70.0%	355.28
Gemini-3.1-Pro (Medium)	58.2%	58.2%	97.35
Numerical Foundation Models

Table 3: Performance of models on WhatIf-Bench , measured as skill score under both wQL (probabilistic) and WAPE (point) metrics, alongside inference latency. The best score per column is bolded and the second-best is underlined . Naive* represents a naive baseline which does not consume covariates. † The simplest, untuned versions of the regression models; tuned variants are in Appendix J . Win rates are provided in Appendix K .

Figure 4: Skill scores on tasks with different number of covariates on WhatIf-Bench on Replay (left) and Novel (right) future types respectively.

Figure 5: Skill scores with different number of historical events on WhatIf-Bench , on tasks with Replay (left) and Novel (right) futures respectively. Lines are least-squares trends of skill on

\log_{2}

of the number of events; they summarise the direction of change.

Appendix figures & tables56 assets

Supplementary material from the paper’s appendix.

Appendix

	Context / prediction length
Pair	EPANET, GridLAB-D	OpenDC	SUMO
$C_{1}$	$1024$ / $128$	$1024$ / $128$	$1024$ / $128$
$C_{2}$	$2048$ / $256$	$2048$ / $256$	$2048$ / $256$
$C_{3}$	$4096$ / $384$	$4096$ / $384$	$4096$ / $384$
$C_{4}$	$8192$ / $512$	$7328$ / $864$	$7776$ / $864$

Appendix

Table 4: Nominal context and prediction lengths (in timesteps) of the four pairs

C_{1}

–

C_{4}

, per simulator.

Target	Primary covariate	Combination
cpu_demand_host-0	coreSpeed_host-0	multiplicative
cpu_usage_host-0	coreSpeed_host-0	multiplicative
power_draw_host-0	coreSpeed_host-0	additive
cpu_utilization_host-0	coreCount_host-0	additive
cpu_time_active_host-0	coreCount_host-0	multiplicative
cpu_demand_1019	coreSpeed_host-0	multiplicative

Appendix

Table 5: OpenDC target–covariate pairing ( host-0 side). Each target is driven by one primary covariate; whether secondary covariates combine multiplicatively or additively is set by the target’s class. The host-1 targets and VM 1026 pair with the host-1 covariates in the same way.

Target class	Secondary covariate	Weight $w_{c}$	Effect ( $r$ or $o$ )
CPU demand / usage (mult.)	core count, same host	$0.60$	$r=f^{2}$
	memory size, same host ( $f<1$ )	$0.40$	$r=10/f$
	core speed, other host ( $f<1$ )	$0.35$	$r=5\,(2-f)$
	core count, other host	$0.30$	$r=(2-f)^{3}$
	memory size, other host	$0.20$	$r=1+3/f$
Active CPU time (mult.)	core speed, same host	$0.80$	$r=1/f^{3}$

Appendix

Table 6: OpenDC secondary covariate effects ( host-0 targets). For a target, each secondary covariate contributes an effect scaled by its weight

w_{c}

and the joint dampening

s

. Multiplicative classes use the raw effect

r

; additive classes use the offset

o

. Here

f

is the secondary covariate’s own factor, and

f_{m}

the memory-size factor. Effects marked (

f<1

) apply only when that covariate is reduced. host-1 targets use the same template with mirrored hosts.

Target	Primary covariate	Combination
actual_power	base_power	multiplicative
total_load	nominal_power	multiplicative
fan_power	power_pf	multiplicative
hvac_load	voltage_factor	multiplicative
hvac_power	duty_cycle	multiplicative
heating_demand	rate_of_heating	multiplicative

Appendix

Table 7: GridLAB-D target–covariate pairing. Each target is driven by one primary covariate; whether secondary covariates combine multiplicatively or additively is set by the target’s class.

Target class	Secondary covariate group	Weight $w_{c}$	Effect ( $r$ or $\delta$ )
Power (mult.)	voltage_factor	$0.50$	$r=f^{2}$
	breaker_amps	$0.40$	$r=1+0.1\,(f-1)$
	phi	$0.30$	$r=\cos^{2}\!\big(\operatorname{clip}(1.57\,(f-1),\pm 1.57)\big)$
	other non-thermal	$0.20$	$r=f+0.05\,(f-1)$
	thermal	$0.30$	$r=1+0.15\,g(f)$
Demand (mult.)	thermal	$0.40$	$r=1+0.15\,g(f)$

Appendix

Table 8: GridLAB-D secondary covariate effects. For a target, each secondary covariate contributes an effect scaled by its weight

w_{c}

and the joint dampening

s

. Multiplicative classes use the raw effect

r

; additive classes use the offset

\delta

. Here

f

is the secondary covariate’s own factor and

g(f)=\operatorname{sign}(f-1)\,\lvert f-1\rvert^{0.4}

. Covariates split into a thermal group ( heat_fraction , temperature , thermostatic_control_range , rate_of_heating , rate_of_cooling , duty_cycle , recovery_duty_cycle , period ), which act on the building’s thermal state, and a non-thermal group of all remaining covariates, which act on the electrical load.

Target	Primary covariate	$a$	$c$	Combination
pressure_j (junctions)	junction_j_base_demand	$0.09$	$1+0.05\,(j\bmod 10)$	multiplicative
pressure_2 (tank)	tank_2_init_level	$0.09$	$1$	multiplicative
demand_9 (reservoir)	reservoir_9_base_head	$0.09$	$1$	multiplicative
demand_2 (tank)	tank_2_init_level	$0.09$	$1$	multiplicative
flowrate_p (pipes)	pipe_p_diameter	$0.09$	$1+0.1\,(p\bmod 10)$	multiplicative
flowrate_9 (pump)	pump_9_setting	$0.09$	$1$	multiplicative

Appendix

Table 9: EPANET target–covariate pairing. Each target is driven by one primary covariate through the shared form in the text, with per-element constants

a

and

c

. All targets combine multiplicatively. Here

j

ranges over the nine junction ids and

p

over the twelve pipe ids.

Target class	Weight $w_{c}$	Ratio $r$
Pressure	$0.10+0.05\,(h\bmod 5)$	$r=1+(s\,w_{c})\,(f^{0.4}-1)+0.01\,(s\,w_{c})\,(f-1)$
Demand	$0.10+0.02\,(h\bmod 5)$
Flowrate	$0.10+0.04\,(h\bmod 6)$
Velocity	$0.10+0.03\,(h\bmod 6)$

Appendix

Table 10: EPANET secondary covariate effects. Each secondary covariate contributes a multiplicative ratio

r

scaled by its weight

w_{c}

and the joint dampening

s

. All classes share the same ratio shape and differ only in the weight. Here

f

is the secondary covariate’s own factor, and

h

is a fixed per-covariate constant (the sum of the character codes of its name) that spreads the weights across covariates.

Target	Primary covariate	Combination
meanMaxJamLengthInMeters_e2_e_{0,1}	minGap	additive
meanTimeLoss_e2_e_{0,1}	accel	additive
meanHaltingDuration_e2_e_{0,1}	offset	additive
meanVehicleNumber_e2_e_{0,1}	impatience	additive
speed_loop_e_{0,1}_near	decel	multiplicative
meanSpeed_e2_e_{0,1}	maxSpeed	multiplicative

Appendix

Table 11: SUMO target–covariate pairing. Each target is driven by one primary covariate. The Combination column is the rule by which secondary covariates combine, set by the target’s class (additive for delay and jam length, multiplicative for emissions, flow, occupancy, and speed).

	minGap	accel	decel	maxSpeed	sigma	tau	length	speedDev	offset	width	height	lcStrategic	lcCooperative	lcSpeedGain	impatience	actionStepLength
$w^{\text{add}}_{c}$	$.60$	$.20$	$.20$	$.20$	$.20$	$.20$	$.50$	$.20$	$.70$	$.20$	$.20$	$.20$	$.20$	$.20$	$.45$	$.20$
$w^{\text{mul}}_{c}$	$.45$	$.15$	$.30$	$.15$	$.35$	$.50$	$.35$	$.25$	$.30$	$.15$	$.15$	$.20$	$.20$	$.20$	$.15$	$.15$

Appendix

Table 12: SUMO secondary covariate weights.

w^{\text{add}}_{c}

is used for the additive classes (delay, jam length) and

w^{\text{mul}}_{c}

for the multiplicative classes (emissions, flow, occupancy, speed). The effect shapes

e

and

\rho

are given in the text.

Model	Imputed windows	% of windows
Claude-Opus-4.8 (High)	16	0.04%
Claude-Opus-4.8 (Medium)	2	0.01%
Gemini-3.1-Pro (High)	10	0.03%
Gemini-3.1-Pro (Medium)	0	0.00%

Appendix

Table 13: Number of task windows on which each LLM returned no valid forecast after all retries, out of

37{,}665

windows. These windows are imputed with the naive baseline forecast, so they receive zero skill. All other models return a forecast on every window and have no imputed windows.

Figure 6: Skill scores with different event lengths on WhatIf-Bench , on tasks with Replay (left) and Novel (right) futures respectively. Lines are least-squares trends of skill on

\log_{2}

of the event length; they summarise the direction of change.

Figure 7: Skill scores with different context lengths on WhatIf-Bench , on tasks with Replay (left) and Novel (right) futures respectively. All models receive the same input and produce the same output: each consumes the full history at every context length, without truncation, and forecasts the same horizon. Tasks are grouped by their nominal context length; for the longest events, the actual context is longer ( Section B.3 ). Lines are least-squares trends of skill on

\log_{2}

of the context length; they summarise the direction of change.

Simulator	Fewest	Most	Change	Worse	Fewest	Most	Change	Worse
	Replay				Novel
OpenDC	$80.2\%$	$54.4\%$	$-25.8$	22/22	$28.7\%$	$-35.2\%$	$-63.9$	11/22
GridLAB-D	$34.8\%$	$39.8\%$	$+5.0$	4/10	$6.2\%$	$18.2\%$	$+12.0$	4/10
EPANET	$24.4\%$	$39.4\%$	$+15.1$	6/33 †	$10.9\%$	$32.6\%$	$+21.7$	10/33 †
SUMO	$79.8\%$	$5.5\%$	$-74.3$	25/29	$-4.2\%$	$-29.7\%$	$-25.5$	22/29

Appendix

Table 14: Skill score (wQL) of Linear Regression with the fewest and the most historical events, on Replay and Novel futures. Each comparison uses tasks that share the target, event length, context length and number of covariates, where the most events are at least

8\times

the fewest. Change is in percentage points. Worse is the number of targets whose skill is lower with the most events, out of the benchmark targets ( Table 2 ) that have at least one such comparison. † Out of the

37

EPANET targets: the other

4

( demand_9 , flowrate_10 , flowrate_9 , velocity_10 ) have at most

2

events in every task, so no such comparison exists for them. Ridge Regression is within

1

point of Linear Regression in every cell.

Simulator	Fewest	Most	Change	Worse	Fewest	Most	Change	Worse
	Replay				Novel
OpenDC	$63.3\%$	$58.8\%$	$-4.4$	17/22	$23.1\%$	$-24.9\%$	$-48.1$	14/22
GridLAB-D	$53.8\%$	$55.7\%$	$+1.9$	4/10	$35.1\%$	$37.3\%$	$+2.2$	3/10
EPANET	$40.9\%$	$43.6\%$	$+2.7$	15/33 †	$43.6\%$	$47.7\%$	$+4.0$	13/33 †
SUMO	$78.5\%$	$55.3\%$	$-23.2$	29/29	$26.4\%$	$18.5\%$	$-7.9$	27/29

Appendix

Table 15: Skill score (wQL) of TiRex-2 with the fewest and the most historical events, on Replay and Novel futures. Each comparison uses tasks that share the target, event length, context length and number of covariates, where the most events are at least

8\times

37

EPANET targets: the other

4

( demand_9 , flowrate_10 , flowrate_9 , velocity_10 ) have at most

2

events in every task, so no such comparison exists for them.

	Replay		Novel
Model	Context $C_{3}\rightarrow C_{4}$	Events fewest $\rightarrow$ most	Context $C_{3}\rightarrow C_{4}$	Events fewest $\rightarrow$ most
Gemini-3.1-Pro (High)	$87.2\rightarrow 73.7$ ( $\mathbf{-13.5}$ )	$82.9\rightarrow 80.2$ ( $-2.7$ )	$52.3\rightarrow 35.1$ ( $\mathbf{-17.2}$ )	$42.8\rightarrow 49.6$ ( $+6.9$ )
Gemini-3.1-Pro (Medium)	$75.9\rightarrow 63.1$ ( $\mathbf{-12.9}$ )	$73.0\rightarrow 64.6$ ( $-8.4$ )	$35.9\rightarrow 24.2$ ( $\mathbf{-11.6}$ )	$31.6\rightarrow 31.1$ ( $-0.5$ )
Claude-Opus-4.8 (High)	$86.6\rightarrow 83.8$ ( $-2.8$ )	$83.2\rightarrow 85.5$ ( $+2.3$ )	$57.8\rightarrow 51.3$ ( $-6.5$ )	$48.8\rightarrow 60.7$ ( $+11.9$ )
Claude-Opus-4.8 (Medium)	$85.5\rightarrow 82.2$ ( $-3.3$ )	$81.5\rightarrow 84.0$ ( $+2.5$ )	$53.7\rightarrow 44.3$ ( $-9.4$ )	$42.6\rightarrow 55.1$ ( $+12.5$ )
TimesFM-3	$78.2\rightarrow 76.4$ ( $-1.8$ )	$73.5\rightarrow 75.5$ ( $+2.0$ )	$50.9\rightarrow 44.6$ ( $-6.3$ )	$45.2\rightarrow 49.7$ ( $+4.6$ )

Appendix

Table 16: Skill score (wQL, %) of the LLMs and TimesFM-3 when moving from the second-longest to the longest context (

C_{3}\rightarrow C_{4}

), and from the fewest to the most historical events at a fixed target, event length, context length and number of covariates (most events at least

8\times

the fewest). Change in percentage points in brackets. Gemini-3.1-Pro drops at the longest context, but changes little with the number of events.

Figure 8: Aggregate skill leaderboard on WhatIf-Bench as per the WAPE metric.

Figure 9: Replay vs Novel skill of , , and on WhatIf-Bench as per the WAPE metric.

Figure 10: Skill scores as per the WAPE metric on tasks with different number of covariates on WhatIf-Bench , on Replay (left) and Novel (right) future types respectively.

Figure 11: Skill scores as per the WAPE metric with different number of historical events on WhatIf-Bench , on tasks with Replay (left) and Novel (right) futures respectively. Lines as in Figure 5 .

Figure 12: Skill scores as per the WAPE metric with different event lengths on WhatIf-Bench , on tasks with Replay (left) and Novel (right) futures respectively. Lines as in Figure 5 .

Figure 13: Skill scores as per the WAPE metric with different context lengths on WhatIf-Bench , on tasks with Replay (left) and Novel (right) futures respectively. All models receive the same input and produce the same output: each consumes the full history at every context length, without truncation, and forecasts the same horizon. Tasks are grouped by their nominal context length; for the longest events, the actual context is longer ( Section B.3 ). Lines as in Figure 5 .

Model (# Tasks)	OpenDC (8,800)	GridLAB-D (4,000)	EPANET (13,265)	SUMO (11,600)	All (37,665)
Frontier LLMs
Claude-Opus-4.8 (High)	75.1%	58.7%	45.9%	88.1%	72.5%
Claude-Opus-4.8 (Medium)	73.4%	56.2%	41.4%	87.0%	70.3%
Gemini-3.1-Pro (High)	73.0%	52.4%	52.0%	83.7%	70.0%
Gemini-3.1-Pro (Medium)	62.9%	35.0%	25.7%	79.6%	58.2%
Numerical Foundation Models

Appendix

Table 17: Skill score (wQL) of each model on WhatIf-Bench , separately for each of the four simulators, and pooled over all of them (All). The number under each simulator name is its number of tasks (# Tasks). The best score per column is bolded and highlighted and the second-best is underlined . Note that Naive* represents a naive baseline which does not consume covariates. All other models are evaluated with covariates. † The simplest, untuned versions of the regression models; tuned variants are in Appendix J .

Panel of Figure 14 — Figure 14: Skill score (wQL) of , , and on Replay and Novel futures, separately for each simulator. In each panel, the orange segment is a model’s skill on Novel futures and the full bar is its skill on Replay futures; models are sorted by Replay skill within each panel, and all panels share the same horizontal axis. (H) and (M) mark High and Medium reasoning effort.

Figure 15: Replay–Novel gap (Replay skill minus Novel skill, wQL, in percentage points) for each model and simulator. Larger values mean a larger drop from Replay to Novel futures. The last column is the mean over simulators and the last row is the mean over models. Rows are sorted by the mean gap within each group of models.

Model	vs. Naive	vs. Linear	vs. Naive	vs. Linear
	wQL		WAPE
Frontier LLMs
Claude-Opus-4.8 (High)	72.5%	63.1%	72.5%	63.1%
Claude-Opus-4.8 (Medium)	70.3%	60.1%	70.3%	60.1%
Gemini-3.1-Pro (High)	70.0%	59.6%	70.0%	59.6%
Gemini-3.1-Pro (Medium)	58.2%	43.4%	58.2%	43.4%

Appendix

Table 18: Skill score of each model on WhatIf-Bench under wQL and WAPE, measured against two reference models: the naive baseline that copies the last value (as in the main text) and a covariate-aware linear regression (Linear Regression). Each reference has a skill of 0 by construction. The best score per column is bolded and highlighted and the second-best is underlined . Note that Naive* represents a naive baseline which does not consume covariates. All other models are evaluated with covariates. † The simplest, untuned versions of the regression models; tuned variants are in Appendix J .

Figure 16: Skill score against linear regression of , , and on Replay and Novel futures, under wQL (left) and WAPE (right). The orange segment is a model’s skill on Novel futures, and the full bar is its skill on Replay futures. Models are sorted by Replay skill.

Figure 17: Skill score (wQL) against linear regression for different numbers of covariates

K

, on Replay (left) and Novel (right) futures.

Figure 18: Skill score (WAPE) against linear regression for different numbers of covariates

K

, on Replay (left) and Novel (right) futures.

Figure 19: Skill score (wQL) against linear regression for different numbers of historical events, on Replay (left) and Novel (right) futures. Lines as in Figure 5 .

Figure 20: Skill score (WAPE) against linear regression for different numbers of historical events, on Replay (left) and Novel (right) futures. Lines as in Figure 5 .

Figure 21: Skill score (wQL) against linear regression for different event lengths, on Replay (left) and Novel (right) futures. Lines as in Figure 5 .

Figure 22: Skill score (WAPE) against linear regression for different event lengths, on Replay (left) and Novel (right) futures. Lines as in Figure 5 .

Linear Regression
Training	Features	Lags	Skill	Replay	Novel	Gap
Plain (main text)	$\mathbf{z}_{t+h}$	—	26.2%	49.0%	-6.7%	55.7
Paired	$\mathbf{z}_{t+h}$	—	25.8%	48.7%	-7.1%	55.8
Paired	$\mathbf{z}_{t+h},s_{t}$	—	30.7%	52.3%	-0.6%	52.9
Paired	$\bm{\mathbf{z}_{t+h},\mathbf{z}_{t},s_{t},h}$	—	32.9%	54.8%	0.2%	54.6
Paired	all	—	28.9%	52.5%	-6.5%	59.0

Appendix

Table 19: Skill score (wQL) of the tuned regression variants, measured against the naive baseline. Rows add features and target lags step by step; the best variant of each model is in bold. all is

\mathbf{z}_{t+h},\mathbf{z}_{t+h-1},\mathbf{z}_{t},\Delta\mathbf{z},s_{t},h

. Ridge Regression uses a regularization strength of

10

in every row shown. Gap is Replay skill minus Novel skill, in percentage points. The variants are selected on the test set and are not valid baselines.

Figure 23: Skill score (wQL) on Replay and Novel futures, with the tuned variants of Linear Regression, Ridge Regression, and XGBoost (best on the test set) in place of the plain ones. The orange segment is a model’s skill on Novel futures, and the full bar is its skill on Replay futures. Models are sorted by Replay skill.

Figure 24: Skill score (wQL) for different numbers of covariates, on Replay (left) and Novel (right) futures, with the tuned regression variants.

Figure 25: Skill score (wQL) for different numbers of historical events, on Replay (left) and Novel (right) futures, with the tuned regression variants. Lines as in Figure 5 .

Figure 26: Skill score (wQL) for different event lengths, on Replay (left) and Novel (right) futures, with the tuned regression variants. Lines as in Figure 5 .

Figure 27: Skill score (wQL) for different context lengths, on Replay (left) and Novel (right) futures, with the tuned regression variants. Lines as in Figure 5 .

Model	wQL Win Rate	WAPE Win Rate	Inf. time / series (s)
Frontier LLMs
Claude-Opus-4.8 (High)	63.2%	68.6%	270.13
Claude-Opus-4.8 (Medium)	60.1%	65.5%	206.29
Gemini-3.1-Pro (High)	58.4%	63.4%	355.28
Gemini-3.1-Pro (Medium)	46.0%	50.9%	97.35
Numerical Foundation Models

Appendix

Table 20: Average win rate of each model on WhatIf-Bench under wQL (probabilistic) and WAPE (point) metrics, alongside inference latency. A model’s win rate is the fraction of task windows on which it has a lower error than another model, averaged over all other models in the table (ties count half). The best score per column is bolded and the second-best is underlined . Note that Naive* represents a naive baseline which does not consume covariates. All other models are evaluated with covariates. † The simplest, untuned versions of the regression models; tuned variants are in Appendix J .

Figure 28: Pairwise win rates on WhatIf-Bench under the wQL metric. Each cell is the win rate (%) of the row model against the column model, with a bootstrap 95% confidence interval below it; the two cells across the diagonal add up to $100\%$ . Models are ordered by their average win rate. Naive* does not consume covariates.

Figure 29: Pairwise win rates on WhatIf-Bench under the WAPE metric. Each cell is the win rate (%) of the row model against the column model, with a bootstrap 95% confidence interval below it; the two cells across the diagonal add up to $100\%$ . Models are ordered by their average win rate. Naive* does not consume covariates.

Figure 30: A Replay task and a Novel task over a shared history. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 31: A Replay task and a Novel task over a shared history. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 32: A Replay task and a Novel task over a shared history. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 33: Tasks that differ only in event length

\ell=32,64,128,256,512

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 34: Tasks that differ only in event length

\ell=32,64,128,256,512

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 35: Tasks that differ only in event length

\ell=32,64,128,256,512

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 36: Tasks that differ only in event length

\ell=32,64,128,256,512

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 37: Tasks that differ only in the number of events (few versus many). The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 38: Tasks that differ only in the number of events (few versus many). The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 39: Tasks that differ only in the number of events (few versus many). The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 40: Tasks that differ only in context/prediction pair,

C_{1}

through

C_{4}

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 41: Tasks that differ only in context/prediction pair,

C_{1}

through

C_{4}

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 42: Tasks that differ only in context/prediction pair,

C_{1}

through

C_{4}

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 43: Tasks that differ only in the number of covariates,

K=1

through

5

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Figure 44: Tasks that differ only in the number of covariates,

K=1

through

5

. The dotted pre-intervention target is shown for illustration only and is not part of the task.

Time Series Forecasting

Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.

See all 839 papers

Sep 21, 2026

$t_0$ : A Time-Series Foundation Model for Forecasting with Context

We present

t_0

, a family of open-weights foundation models for forecasting with multivariate context. We release its first two members:

\texttt{t0-alpha}

and

\texttt{t0-beta}

, respectively 102M and 256M parameters. Both condition their forecasts on target history, past covariates, and known-future covariates, without task-specific retraining. Their transformer layers alternate attention along time and across variates. They produce probabilistic forecasts through quantile predictions. Pretraining combines curated public data with synthetic generator families constructed to contain covariate-to-target dependencies. On GIFT-Eval,

\texttt{t0-alpha}

reaches an aggregate CRPS of 0.4941, and

\texttt{t0-beta}

a CRPS of 0.4738 and a MASE of 0.6865, third on both and within 4.0% of the best zero-shot TSFM. On fev-bench they score 42.2 and 46.7 in skill, the latter third again and 2.0 points behind the leader. We analyze

\texttt{t0-alpha}

in depth. Known-future covariates raise its skill by 6.3 percentage points across 30 tasks. The report also examines its calibration, its rollout strategy on long horizons, and its robustness to missing data. On the Victoria electricity-demand benchmark,

\texttt{t0-beta}

is among the most accurate models with a context of nearly a year. In an independent Macrocosm evaluation of hourly ERCOT prices over 29 months, both cut the MAE of the lagged-price baseline by 38%.

Lucas Meyer, Claudio Sole, Huikan Xiang +6

The Forecasting Company · Macrocosm

Time Series Forecasting Zero-Shot Time Series Forecasting Probabilistic Time Series Forecasting Multivariate Time Series Forecasting Time Series Forecasting with Covariates Time-Series Foundation Models+1

Ask PDF

Apr 7, 2026

TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems

We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg.

\sim40.2\%\to56.6\%)

, validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io

Md Atik Ahamed, Mihir Parmar, Palash Goyal +7

Google · University of Kentucky

Forecasting Benchmarks Time Series Reasoning Explainable Time Series Forecasting LLM-Based Time Series Forecasting

Ask PDF

Sep 30, 2025

fev-bench: A Realistic Benchmark for Time Series Forecasting

Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of pretrained models. Existing benchmarks often have limited domain coverage or overlook real-world settings such as tasks with covariates. Their aggregation procedures frequently lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks lack consistent evaluation infrastructure or are too rigid for integration into existing pipelines. To address these gaps, we propose fev-bench, a benchmark of 100 forecasting tasks across seven domains, including 46 with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for forecasting evaluation emphasizing reproducibility and integration with existing workflows. Using fev, fev-bench employs principled aggregation with bootstrapped confidence intervals to report performance along two dimensions: win rates and skill scores. We report results on fev-bench for pretrained, statistical, and baseline models and identify promising future research directions.

Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5

AWS · Keystone AI · Prior Labs +2

Benchmark Design Time Series Forecasting Forecasting Benchmarks Time Series Forecasting with Covariates Benchmark Validity

Ask PDF

Abstract

Time Series Forecasting

Explore similar work

t0t_0t0​: A Time-Series Foundation Model for Forecasting with Context

TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems

fev-bench: A Realistic Benchmark for Time Series Forecasting

$t_0$ : A Time-Series Foundation Model for Forecasting with Context