TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

Malgorzata Gwiazda; Yifu Cai; Mononito Goswami; Arjun Choudhry; Artur Dubrawski

cs.AIApr 11, 2026

TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

Authors: Malgorzata Gwiazda, Yifu Cai, Mononito Goswami, Arjun Choudhry, Artur Dubrawski

Organizations: Technical University of Munich, Munich, Germany · Auton Lab, School of Computer Science, Carnegie Mellon University, Pittsburgh, US

Abstract

Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data? While multiple benchmarks have been proposed to answer this fundamental question, most are manually curated and focus on narrow domains or specific skill sets. To address this limitation, we propose scalable methods for creating comprehensive time series reasoning benchmarks that combine the flexibility of templates with the creativity of LLM agents. We first develop TimeSeriesExam, a multiple-choice benchmark using synthetic time series to evaluate LLMs across five core reasoning categories: pattern recognitionnoise understandingsimilarity analysisanomaly detection, and causality. Then, with TimeSeriesExamAgent, we scale our approach by automatically generating benchmarks from real-world datasets spanning healthcare, finance and weather domains. Through multi-dimensional quality evaluation, we demonstrate that our automatically generated benchmarks achieve diversity comparable to manually curated alternatives. However, our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specific applications, highlighting ongoing challenges in enabling effective time series understanding in these models. TimeSeriesExamAgent is available at https://github.com/magwiazda/TimeSeriesExamAgent.

Figures & tables

Title	Multi-domain	Curation	# Samples	Skill type
		Fully Automatic		P	R	PS
Time-MQA Kong et al. [2025]	✓	✓	200,000	✓	✓	✗
Time-MMD Liu et al. [2024]	✓	✗	17,113	✓	✗	✗
MT-Bench Chen et al. [2025]	✓	✗	22,000	✓	✓	✗
ECG-QA Oh et al. [2023]	✗	✗	414,348	✓	✓	✓
Context-is-key 1 1 1 The CiK benchmark contains 71 tasks, with the number of samples treated as a configurable hyperparameter. Williams et al. [2024]	✓	✗	71	✓	✓	✗

Table 1: Overview of time-series and multimodal datasets with curation and skill types (P—Prediction, R—Reasoning, PS—Practical skills). Prediction refers to supervised tasks such as forecasting or classification. Reasoning involves higher-level interpretation of time series signals (e.g., trend recognition). Practical skills extend reasoning into domain-specific contexts (e.g., classifying volatility regimes in finance). "+" represents that we can generate any number of samples, but we have already generated 3,000 of them.

Category	Subcategory	Example question
Pattern Recognition	Trend	What is the most likely linear trend coefficient of the given time series?
	Cyclic	The given time series has sine wave pattern. How does its amplitude change from the beginning to the end?
	Stationarity	Is the given time series likely to be stationary after removing the cycle component?
	Regime Switching	Based on the given time series, how many different regimes are there?
	Statistical properties	Is the mean stable over time in the given time series?
	Random processes	Does the following time series exhibit a mean reversion property?

Table 2: Example template questions for different reasoning tasks. Each subcategory covers a specific aspect of time series understanding, guiding the model to reason about comparative, anomalies, and causal relationships.

Panel of Figure 1 — Figure 1: ( Left ) Time Series Curation Pipeline: The composition model generates controlled synthetic time series step-by-step. The pipeline enables diversity by combining different components to create numerous synthetic time series with varying properties. ( Right ) Each template evaluates a specific category, and includes a question, list of options, example question and answer pair for in-context learning, and optionally a hint and descriptions of complicated technical terms. Here, GPT-4o showcases its ability to transfer visual understanding and time series concepts into effective reasoning.

Figure 2: TimeSeriesExamAgent architecture. The user provides exam-making instructions and a custom dataset with minimal loading code. Agent outputs question templates – Python functions generated by a generator LLM and filtered through three progressive stages of verification (syntax and output format check, validation by LLM judge, capability-aligned filtering). Arrows denote data flow, red ones show direction for rejected templates.

Figure 3: Given dataset information, TimeSeriesExamAgent generates question templates as Python functions that encode the question logic and support parameterized sampling of arbitrary instances.

	Dataset
Model	MIT-BIH	PTB-XL	MIMIC-IV W	YFinance	WeatherBench2	Average
random guess	0.25	0.25	0.25	0.25	0.25	0.25
gpt-4o Hurst et al. [2024]	0.416	0.424	0.385	0.586	0.389	0.440
o3-mini OpenAI [2025]	0.442	0.477	0.356	0.555	0.379	0.442
Qwen2.5-VL-Instruct Bai et al. [2025]	0.411	0.490	0.439	0.572	0.368	0.456
Gemma-3-27b-it Team et al. [2025]	0.497	0.517	0.370	0.534	0.232	0.430

Table 3: Comparative performance of six vision–language models across medical (MIT-BIH, PTB-XL, MIMIC-IV Waveform ( MIMIC-IV W )), financial (YFinance), and meteorological (WeatherBench 2) time series datasets. The results highlight dataset-specific strengths; nonetheless, all models achieve less than 55 mean accuracy, underscoring the difficulty of time series reasoning for current VLMs. The evaluation protocol is provided in App. C.3

	Mean $\pm$ Std
Benchmark Dataset	Embedding	Normalized Levenshtein
ECG-QA	0.207 $\pm$ 0.079	0.519 $\pm$ 0.157
TimeSeriesExamAgent (ours)	0.301 $\pm$ 0.070	0.542 $\pm$ 0.039

Table 4: Diversity of questions measured by embedding and normalized Levenshtein distances. Higher values indicate greater variability in phrasing.

Finance Domain
	Mean Result
Dataset	Specificity	Unambiguity	Domain Relevance	Answerability
FinMME	8.10	7.59	6.62	6.95
MTBench	6.88	6.11	8.35	7.29
TimeSeriesExamAgent (ours)	8.29	7.24	8.89	8.57
Medicine Domain

Table 5: Combined Quality Evaluation . Scores (1–10) averaged across four criteria. Rows are grouped by their original source tables.

	Accuracy
Method	General	Parsable
Random answering	34.9%	34.9%
Base	21.8%	34.6%
Fine-tuned-confounded	39.7%	42.3%
Fine-tuned	47.0%	49.7%

Table 6: Results of VLM fine-tuning on the exams generated by TimeSeriesExamAgent . General : all incorrectly formatted responses are treated as wrong answers. Parsable : only correctly formatted responses are evaluated.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Meta-information Type	Anomaly Detection	Similarity Analysis	Noise Understanding	Pattern Recognition	Causality Analysis
# Questions	129	113	87	371	63
%age questions with 2 time series	31.01	100	14.94	3.77	100

Appendix

Table 7: TimeSeriesExam meta-information breakdown for each category. Each question is associated with a time series of length 128 time steps, and an example time series of length 64 time steps.

Model	Similarity	Pattern	Causality	Noise	Anomaly	Overall
Gemma-3-27B-IT [ 38 ]	0.62	0.59	0.51	0.71	0.51	0.59
GPT-4o [ 17 ]	0.78	0.78	0.61	0.77	0.54	0.73
Qwen2.5-VL-72B [ 3 ]	0.60	0.45	0.49	0.40	0.22	0.44
Gemini-2.5-Pro [ 10 ]	0.78	0.76	0.57	0.65	0.59	0.71

Appendix

Table 8: Accuracy of model on each category of TimeSeriesExam .

Figure 4: The sample average discrimination parameter across rounds shows an upward trend, indicating an improved ability of the questions to differentiate candidates with varying levels of ability.

Figure 5: Dropped Dataset Distribution per round. Dropped category distribution per round generally mirrors the overall category distribution.

Panel of Figure 6 — Figure 6: t-SNE analysis of embeddings: (left) text-only vs. (right) text and time series concatenated together.

Hyperparameter	Value
Base model	Qwen2.5-VL-3B-Instruct
GPU setupe	4*NVIDIA RTX A6000 48GB GPU
Frameworks	Hugging Face Accelerate,
	DeepSpeed ZeRO 3 stage
Train samples	2000
Warm-up steps	16

Appendix

Table 18

Figure 7: Correlation across all jury-model combinations. We see consistently high ( $\geq 0.5$ ) inter-rater correlation. This confirms that any possible triplets has consistent scores.

Figure 8: Cohen’s Kappa across all jury-model combinations.

Metric	DeepSeek V3.2	Claude 4
Specificity	8.23 $\pm$ 0.24	8.08 $\pm$ 0.17
Unambiguity	7.55 $\pm$ 0.12	7.47 $\pm$ 0.28
Domain Relevance	8.69 $\pm$ 0.49	8.72 $\pm$ 0.22
Answerability	8.69 $\pm$ 0.14	8.53 $\pm$ 0.08
No Unintended Hints	7.46 $\pm$ 0.13	7.37 $\pm$ 0.18

Appendix

Table 9: Comparison of Jury scores between DeepSeek V3.2 and Claude 4 generators on the MIT-BIH dataset.

Outcome Category	Attempt 1 (Avg)	Attempt 2 (Avg)
Success (Valid & Differentiable)	71.5%	79.9%
Semantic Failure (Weak $>$ Strong)	17.8%	19.3%
Syntactic Failure (Compile Error)	9.9%	0.8%
Content Validation Rejection (LLM Verifier Rejection)	0.8%	0.0%

Appendix

Table 10: Failure mode breakdown of the agentic pipeline across datasets.

Model	Vision	Text
GPT-4o	0.416	0.401
o3-mini	0.442	0.416
Qwen2.5-VL	0.411	0.391
Gemma-3-27B-IT	0.497	0.421

Appendix

Table 11: Accuracy on MIT-BIH under two representations of the same time series.

Model	25 DPI	50 DPI (Default)	100 DPI
GPT-4o	0.310	0.416	0.442
Qwen2.5-VL	0.391	0.411	0.442
Gemma-3-27B-IT	0.447	0.497	0.492

Appendix

Table 12: Accuracy on MIT-BIH exams under different plot resolutions.

Setup (Attempts / Shots)	Success Rate	Avg. Difficulty	Avg. Differentiability
1 Attempt / 1 Shot	75.0%	0.52	0.11
1 Attempt / 3 Shots	75.0%	0.64	0.00
3 Attempts / 3 Shots	100.0%	0.57	0.10
5 Attempts / 3 Shots	95.0%	0.61	0.14

Appendix

Table 13: Sensitivity of generation quality to attempts and few-shot examples.

Training Volume (Samples)	Accuracy (Choice-based)
$N$ (2,000)	43.37%
$N/2$ (1,000)	41.80%
$N/3$ (672)	43.67%

Appendix

Table 14: Effect of training data volume on fine-tuning accuracy.

Time Series Reasoning

Time series reasoning studies how models interpret temporal patterns, answer questions about sequences, and use evidence to make forecasts.

See all 44 papers

Thoth: Mid-Training Bridges LLMs to Time Series Understanding
Mar 1, 2026Jiafeng Lin, Yuxuan Wang, Jialong Wu +3Temporal Reasoning in Language Models LLM Training
School of Software, BNRist, Tsinghua University.
TSQueryBench: LLM-as-a-Judge for Time Series Explanations
Apr 2, 2026Preetham Sivalingam, Murari Mandal, Dhruv Kumar +1Explanation Evaluation Explainability Evaluation
BITS Pilani, Pilani, India · KIIT & Birla AI Labs, Bhubaneshwar, India · BITS Pilani & Birla AI Labs, Pilani, India +1
TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning
May 31, 2026Yaxuan Kong, Qingren Yao, Yuqi Nie +7Time Series Reasoning LLM Agent Evaluation
University of Oxford · VulpiVox Intelligence · Eindhoven University of Technology +3

Mar 1, 2026J/K move · Enter open · S save

Thoth: Mid-Training Bridges LLMs to Time Series Understanding

Jiafeng Lin, Yuxuan Wang, Jialong Wu, Huakun Luo+2

School of Software, BNRist, Tsinghua University.

Temporal Reasoning in Language Models LLM Training Time Series QA

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated remarkable success in general-purpose reasoning. However, they still struggle to understand and reason about time series data, which limits their effectiveness in decision-making scenarios that depend on temporal dynamics. In this paper, we propose Thoth, the first family of mid-trained LLMs with general-purpose time series understanding capabilities. As a pivotal intermediate stage, mid-training achieves task- and domain-agnostic alignment between time series and natural language, for which we construct Book-of-Thoth, a high-quality, time-series-centric mid-training corpus. Book-of-Thoth enables both time-series-to-text and text-to-time-series generation, equipping LLMs with a foundational grasp of temporal patterns. To better evaluate advanced reasoning capabilities, we further present KnoTS, a novel benchmark of knowledge-intensive time series understanding, designed for joint reasoning over temporal patterns and domain knowledge. Extensive experiments demonstrate that mid-training with Book-of-Thoth enables Thoth to significantly outperform its base model and advanced LLMs across a range of time series question answering benchmarks. Moreover, Thoth exhibits superior capabilities when fine-tuned under data scarcity, underscoring the effectiveness of mid-training for time series understanding. Code is available at: https://github.com/thuml/Thoth.

Abstract

Time Series Reasoning

Explore similar work

Thoth: Mid-Training Bridges LLMs to Time Series Understanding

TSQueryBench: LLM-as-a-Judge for Time Series Explanations

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning