TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems

Md Atik Ahamed; Mihir Parmar; Palash Goyal; Yiwen Song; Long T. Le; Qiang Cheng; Chun-Liang Li; Hamid Palangi

cs.AIApr 7, 2026

TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems

Authors: Md Atik Ahamed, Mihir Parmar, Palash Goyal, Yiwen Song, Long T. Le, Qiang Cheng, Chun-Liang Li, Hamid Palangi, +2 more

Organizations: Google · University of Kentucky

Abstract

We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. $\sim40.2\%\to56.6\%)$ , validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io

Figures & tables

Figure 1: An example task from the proposed TFRBench . We augment the forecasting process by providing past time series and text data to generate a comprehensive reasoning output. This output explicitly captures cross-channel analysis, trend/seasonality component plan and residuals $\&$ event plan. The comprehensive reasoning may guide the forecasting system further for accurate forecasting.

Benchmark	Domains	Context Source	Primary Task	Reasoning	Goal
M4 Competition ( Makridakis et al., 2020 )	Diverse	✗	Statistical Forecast	✗	Pattern Matching
Monash Archive ( Godahewa et al., 2021 )	Diverse	✗	Statistical Forecast	✗	Pattern Matching
GIFT-Eval ( Aksu et al., 2024 )	Diverse	✗	Zero-shot Forecast	✗	Scaling Laws
MTBench ( Chen et al., 2025 )	Fin/Weather	Provided (Static)	Trend QA & Forecast	✗	Multimodal Fusion
InvestorBench ( Li et al., 2025 )	Finance	Vector DB (RAG)	Trading Decision	✗	Profit Maximization
TFRBench (Ours)	Diverse	Agentic Web Search	Step-by-Step Reasoning	✓	Causal Attribution

Table 1: Comparison of TFRBench against existing benchmarks. The table compares benchmarks based on various parameters. Unlike prior works that rely on statistical pattern matching or static contexts, TFRBench provides verifiable reference reasoning. We define the “Reasoning” here as the derivation of a numerical forecast from causal logic, distinguishing it from tasks such as trading decisions.

Figure 2: Overview of Data Creation System. The pipeline integrates context retrieval, assumption verification, and a feedback loop to align reasoning with forecasting. The system orchestrates agents to iteratively refine reasoning traces until the forecast outperforms the naive baseline (MASE $<$ 1.0), and LLM verifier gives high $S$ . Entire reasoning trace with each agent outputs are provided in App. K .

Domain	Dataset	Source	Total Samples	Reference Samples	Freq.	Window	Ch.	Series
						(in-out)
Energy	Solar	GIFT-Eval	274	193	Daily	96-96	1	137
Energy	Electricity	GIFT-Eval	1346	353	Daily	96-96	1	81
Sales	Car-parts	GIFT-Eval	1037	692	Monthly	25-25	1	1037
Sales	Hierarchical Sales	GIFT-Eval	1370	1076	Daily	96-96	1	81
Web/CloudOps	Bitbrains - Fast Storage	GIFT-Eval	1153	678	Hourly	96-96	2	197

Table 2: Dataset Statistics. TFRBench provides a multi-domain suite with diverse temporal frequencies (Hourly to Monthly) and dimensionalities, including univariate to multivariate series (2–5 channels) to test cross-channel reasoning. We utilize different forecasting horizons, mostly 96-96 window for most domains following prior works. (Ch. denotes number of channels). Reference Samples are the selected samples utilized for baseline evaluation.

Figure 3: Overview of the TFRBench Evaluation Pipeline. While standard metrics track numerical error, the core focus of our framework is evaluating the quality of the reasoning process . We employ an LLM-as-a-Judge to audit a candidate’s reasoning trace against our verified “Reference Reasoning”.

Dataset	Models	w/ Reasoning				Event Forecast + Reasoning
Dataset	Models	Dom.	Fcst.	Evt.	Logic	Dom.	Fcst.	Evt.	Logic
Solar Daily	Gemini-2.5-Pro	$3.026_{0.000}$	$2.316_{0.000}$	$2.036_{0.000}$	$4.528_{0.000}$	$3.803_{0.000}$	$2.731_{0.000}$	$2.207_{0.000}$	$4.207_{0.000}$
	Gemini-3-Pro	$4.699_{0.000}$	$3.212_{0.000}$	$2.508_{0.000}$	$4.881_{0.000}$	$4.119_{0.000}$	$2.699_{0.000}$	$2.368_{0.000}$	$4.523_{0.000}$
	Claude-Sonnet-4.5	$4.922_{0.000}$	$3.062_{0.000}$	$2.891_{0.000}$	$3.363_{0.000}$	$4.554_{0.000}$	$2.637_{0.000}$	$2.472_{0.000}$	$3.368_{0.000}$
Electricity	Gemini-2.5-Pro	$2.318_{0.013}$	$2.768_{0.020}$	$1.720_{0.022}$	$4.590_{0.015}$	$4.688_{0.000}$	$3.076_{0.000}$	$3.442_{0.000}$	$3.198_{0.000}$
	Gemini-3-Pro	$4.167_{0.000}$	$3.173_{0.000}$	$2.178_{0.000}$	$4.773_{0.000}$	$4.892_{0.000}$	$3.606_{0.000}$	$4.147_{0.000}$	$4.484_{0.000}$

Table 3: LLM-as-a-Judge Results (Mean std ). Score Range: 1–5 (higher is better). Color intensity reflects performance (green: high, red: low). Dom.: Domain Relevance, Fcst.: Forecasting Correctness, Evt.: Event Relevance, Logic: Logic-to-Number Consistency.

Figure 4: Avg. MASE performance by domains. We compare baseline models across five domains; lower MASE indicates better accuracy. Models: Gemini-2.5-Pro, Gemini-3-Pro, and Claude-Sonnet-4.5. Full numerical results are provided in App. J (Table 10 ).

Figure 5: Reasoning Quality vs. MASE. Models with higher reasoning quality (Judge Score) achieve better forecasting accuracy (lower MASE). Conversely, lower reasoning scores are associated with higher forecasting errors.

Figure 6: Overall Success Rate. Percentage of samples achieving MASE

<1.0

across datasets. The strategic reasoning (w/ Reasoning (Ours) module consistently outperforms the direct prompting baseline. TSFM part represents Time Series Foundational models.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Candidate Stability. Distribution of MASE scores across generated reasoning candidates. The concentration of density around the naive baseline (MASE=1) indicates generation stability.

Figure 8: Feedback Improvement Trajectory. Mean MASE (with standard deviation) for the cohort of samples that improved over three iterations. The trend line confirms systematic convergence toward lower error.

Figure 9: Feedback Rescue Rate. Percentage of initially poor forecasts (

\text{MASE}\geq 1.0

) successfully corrected to

\text{MASE}<1.0

via feedback loop. For representation, dataset names are displayed in short form (e.g., Solar (Sol)).

Figure 10: Feedback Loop Distribution. Histogram of iterations required. The variance in loop counts highlights the adaptive complexity of the reasoning process.

Figure 11: Impact of Feedback. Scatter plot of Initial vs. Final MASE. Points below the red dashed line indicate improved accuracy after feedback iterations.

Figure 12: Verifier Reliability. Box plots showing the distribution of MASE for each assigned Verifier Score (1-5). The inverse relationship confirms that the model accurately self-assesses high-quality forecasts.

Figure 13: Performance vs. Temperature. Stable MASE across temperatures; optimal accuracy (lowest MASE) is achieved at Temperature = 0.

Figure 14: Feedback Iterations. Success rate (

MASE<1.0

) on the Solar dataset. Global performance increases from 39.1% (Loop 1) to a peak of 88.0% (Loop 6), validating that the feedback mechanism systematically corrects reasoning errors rather than relying on random stochasticity. Each loop represents independent run starting from scratch.

Figure 15: Histograms displaying the token count frequency for successful reasoning traces (

MASE<1.0

) across datasets.

Dataset	Total Candidates Generated	Discarded by ( $MASE<1.0$ & $S\geq 4$ )	Discard Rate
Solar	2,770	2,404	86.8%
Electricity	17,500	16,621	95.0%
Car-parts	8,780	5,582	63.6%
Hierarchical Sales	10,650	6,529	61.3%
Bitbrains Fast Storage	11,225	9,090	81.0%
Web Traffic	12,405	9,938	80.1%

Appendix

Table 4: Candidate Discard Rates (Across All Iterative Runs)

Dataset	10th Pct.	25th Pct.	Median (50th)	75th Pct.	90th Pct.
Solar	0.68	0.76	0.89	1.03	1.17
Electricity	0.71	0.94	1.32	1.95	3.05
Car-parts	0.05	0.24	0.48	0.96	3.26
Hierarchical Sales	0.63	0.72	0.84	0.97	1.14
Bitbrains Fast Storage	0.65	0.80	0.93	1.56	2.58
Web Traffic	0.61	0.76	0.94	1.16	1.50

Appendix

Table 5: MASE Distribution

Evaluation Metric	Reference Samples	Discarded Samples
Domain	4.26	3.67
Forecast	3.88	2.72
Event	4.17	3.50
Logic	4.07	3.78

Appendix

Table 6: Human Expert Evaluation Scores

Dataset	Models	w/o Reasoning		w/ Reasoning		Event Forecast + Reasoning
Dataset	Models	MAE	MASE	MAE	MASE	MAE	MASE
Solar Daily	gemma-3-27b	$7.337_{0.000}$	$3.604_{0.000}$	$1.35_{0.00}\times 10^{1}$	$6.240_{0.000}$	$3.060_{0.196}$	$1.423_{0.062}$
Car Parts	gemma-3-27b	$0.305_{0.000}$	$0.441_{0.000}$	$1.01_{0.22}\times 10^{1}$	$1.30_{0.32}\times 10^{1}$	$0.386_{0.057}$	$0.568_{0.076}$

Appendix

Table 7: Numerical Forecasting Results (Open Source Model). MASE represents the forecasting performance (Lower value represents better performance).

Dataset	Models	w/ Reasoning				Event Forecast + Reasoning
Dataset	Models	Dom.	Fcst.	Evt.	Logic	Dom.	Fcst.	Evt.	Logic
Solar Daily	gemma-3-27b-local	$2.098_{0.000}$	$1.283_{0.002}$	$1.922_{0.004}$	$1.314_{0.005}$	$2.254_{0.000}$	$1.352_{0.000}$	$1.886_{0.000}$	$1.725_{0.000}$
Car Parts	gemma-3-27b-local	$2.119_{0.007}$	$1.802_{0.016}$	$1.868_{0.007}$	$1.929_{0.012}$	$4.250_{0.000}$	$2.512_{0.000}$	$2.480_{0.000}$	$1.921_{0.000}$

Appendix

Table 8: LLM-as-Judge Results (Open Source Model). Higher value represents better qualitiy reasoning in the respective criteria.

Dataset	Models	w/ Reasoning				Event Forecast + Reasoning
Dataset	Models	Dom.	Fcst.	Evt.	Logic	Dom.	Fcst.	Evt.	Logic
Solar Daily	Gemini-2.5-Flash	$2.472_{0.000}$	$2.399_{0.000}$	$1.933_{0.000}$	$4.435_{0.000}$	$4.378_{0.000}$	$2.922_{0.000}$	$2.984_{0.000}$	$3.720_{0.000}$
	Gemini-2.5-Pro	$3.026_{0.000}$	$2.316_{0.000}$	$2.036_{0.000}$	$4.528_{0.000}$	$3.803_{0.000}$	$2.731_{0.000}$	$2.207_{0.000}$	$4.207_{0.000}$
	Gemini-3-Pro	$4.699_{0.000}$	$3.212_{0.000}$	$2.508_{0.000}$	$4.881_{0.000}$	$4.119_{0.000}$	$2.699_{0.000}$	$2.368_{0.000}$	$4.523_{0.000}$
	Claude-Sonnet-4	$4.504_{0.002}$	$2.570_{0.000}$	$2.503_{0.000}$	$2.870_{0.000}$	$4.285_{0.000}$	$2.363_{0.000}$	$2.228_{0.000}$	$3.358_{0.000}$
	Claude-Sonnet-4.5	$4.922_{0.000}$	$3.062_{0.000}$	$2.891_{0.000}$	$3.363_{0.000}$	$4.554_{0.000}$	$2.637_{0.000}$	$2.472_{0.000}$	$3.368_{0.000}$

Appendix

Table 9: LLM-as-a-Judge Results. We report the mean and standard deviation (subscript) of reasoning quality scores on baseline models across three independent runs, using Gemini-3-Pro as the evaluator. Scores are measured on a scale of 1 to 5, where higher values indicate better performance. The metrics cover four dimensions: Domain Relevance ( Dom. ), Forecasting Correctness ( Fcst. ), Event Relevance ( Evt. ), and Logic-to-Number Consistency ( Logic ). Color intensity reflects performance (green: high, red: low).

Dataset	Models	w/o Reasoning		w/ Reasoning		Event Forecast + Reasoning
Dataset	Models	MAE	MASE	MAE	MASE	MAE	MASE
Solar Daily	ARIMA	$1.400_{0.000}$	$0.694_{0.000}$
	TimesFM-2.5	$1.438_{0.000}$	$0.732_{0.000}$
	Chronos-2.0	$1.380_{0.000}$	$0.684_{0.000}$
	Gemini-2.5-Flash	$2.445_{0.000}$	$1.168_{0.000}$	$1.860_{0.000}$	$0.930_{0.000}$	$1.810_{0.067}$	$0.896_{0.015}$
	Gemini-2.5-Pro	$2.316_{0.020}$	$1.114_{0.006}$	$2.802_{0.073}$	$1.304_{0.027}$	$2.053_{0.031}$	$1.016_{0.015}$

Appendix

Table 10: Numerical Performance Evaluation. We report the mean and standard deviation (subscript) of Mean Absolute Error ( MAE ) and Mean Absolute Scaled Error ( MASE ) scores on baseline models. For both metrics, lower values indicate better performance. To account for varying native scales across datasets, the color gradient is normalized within each dataset block rather than across the entire table. Green indicates the best performance (lowest error) and red indicates the worst performance (highest error) per task.

Figure 16: Example from Dataset Apple (Economics/Finance).

Figure 17: Visualizing a sample from Amazon dataset. The Ground Truth (Blue/Orange/Green) shows a significant price drop and volume spike around July 23, driven by earnings news.

Explore similar work

ReasonCast: Towards Explainable Time Series Forecasting with Reasoning
Aug 3, 2026Seunghan Lee, Jun Seo, Jaehoon Lee +9Explainable Time Series Forecasting Time Series Reasoning
LG AI Research
TempusBench: An Evaluation Framework for Time-Series Forecasting
Apr 13, 2026Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade +14Time Series Forecasting Forecasting Benchmarks
Simulacrum New York City, NY, USA
QuitoBench: A High-Quality Open Time Series Forecasting Benchmark
Mar 27, 2026Siqiao Xue, Zhaoyang Zhu, Wei Zhang +7Time Series Forecasting Forecasting Benchmarks
Ant Group

Aug 3, 2026J/K move · Enter open · S save

ReasonCast: Towards Explainable Time Series Forecasting with Reasoning

Seunghan Lee, Jun Seo, Jaehoon Lee, Junhyeok Kang+8

LG AI Research

Explainable Time Series Forecasting Time Series Reasoning Time Series Forecasting Forecasting Benchmarks LLM-Based Time Series Forecasting

Open paper AskPDF arXiv

Most time series (TS) models are specialized for a single task, either understanding (i.e., returning text answers about a TS) or generation (i.e., returning a numeric forecast). Only recently have unified models begun to handle the two within a single architecture. Even these models, however, produce the two outputs as task-separated paths and cannot predict a series and explain why that prediction arises within a single coherent response. In this paper, we argue for a task-fused model that jointly produces 1) prediction (generation) and 2) selfexplanation (understanding), thereby integrating 1) numerical TS forecasting and 2) interpretable text reasoning within a single response. To enable the systematic study of this capability, we present both a benchmark and a recipe that jointly address the two tasks. The benchmark, ReasonTS-Bench, identifies five fundamental patterns underlying TS and enables the joint evaluation of both tasks. ReasonCast, our recipe for finetuning any LLM to perform both tasks jointly, yields a model that generates a reasoning chain and a forecast together in a single autoregressive pass. Extensive experiments show that ReasonCast outperforms both LLMs and TS models on prediction accuracy while producing verifiable, causal reasoning. Code is available at: https://github.com/seunghan96/reasoncast.

Abstract

Explore similar work

ReasonCast: Towards Explainable Time Series Forecasting with Reasoning

TempusBench: An Evaluation Framework for Time-Series Forecasting

QuitoBench: A High-Quality Open Time Series Forecasting Benchmark