TSQueryBench: LLM-as-a-Judge for Time Series Explanations

Preetham Sivalingam; Murari Mandal; Dhruv Kumar; Saurabh Deshpande

cs.AIApr 2, 2026

TSQueryBench: LLM-as-a-Judge for Time Series Explanations

Authors: Preetham Sivalingam, Murari Mandal, Dhruv Kumar, Saurabh Deshpande

Organizations: BITS Pilani, Pilani, India · KIIT & Birla AI Labs, Bhubaneshwar, India · BITS Pilani & Birla AI Labs, Pilani, India · Birla AI Labs, Mumbai, India

Abstract

Natural language explanations of time series data are increasingly produced by foundation models in high stakes domains, making factual correctness critical. Evaluating such explanations differs fundamentally from standard natural language generation: correctness requires verifying numerical claims against structured data rather than similarity to reference text. While LLM as a Judge has emerged as a scalable paradigm for text evaluation, its applicability to numerically grounded time series explanations remains unstudied. We introduce TSQueryBench, a controlled synthetic benchmark of 500 time series instances across 10 query types, each paired with correct, partially correct, and incorrect explanations. We evaluate six large language models across four tasks: explanation generation, relative ranking, independent scoring, and multi anomaly detection. Our central finding is a consistent generation evaluation asymmetry: models that fail to generate numerically correct explanations nonetheless reliably identify or score correct ones. These results show that rubric guided LLM evaluation is substantially more reliable than generation for time series reasoning, supporting LLM judges as scalable evaluators in numerically grounded settings. Code and data: https://github.com/Prxxthxm/TSQueryBench/

Figures & tables

Metric	LLaMA	Gemma	Gemma	Qwen	Haiku	DS
	3.1 8B	4 31B	3 12B	3 8B	3	v3.2
Exp 1 Acc	0.18	0.36	0.13	0.38	0.31	0.30
Exp 2 Acc	0.49	0.67	0.56	0.84	0.41	0.74
Exp 3 Acc	0.29	0.67	0.47	0.72	0.35	0.56
Exp 4 Acc	0.01	0.45	0.00	0.12	0.01	0.25
Exp 4 F1	0.34	0.88	0.28	0.53	0.62	0.87

Table 1: Summary of model performance across all four evaluation tasks in TSQueryBench. Best per row in bold , second-best underlined .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Query Type	LLaMA 3.1 8B	Gemma 4 31B	Gemma 3 12B	Qwen 3 8B	Claude 3 Haiku	Deepseek v3.2
Linear Spike	0.70	0.98	0.46	0.94	0.68	0.94
Seasonal Drop	0.06	0.12	0.00	0.16	0.24	0.08
Structural Break	0.46	0.52	0.04	0.18	0.68	0.46
Multi-Metric Consistency	0.18	0.12	0.00	0.14	0.10	0.00
Relative Extremum	0.12	0.60	0.08	0.02	0.32	0.10
Mean Shift	0.02	0.00	0.02	0.88	0.02	0.08

Appendix

Table 2: Explanation generation accuracy per query type. Best per row in bold , second-best underlined .

Model	Wrong	Partial	Correct
Qwen 3 8B	0.39	0.25	0.36
Gemma 4 31B	0.41	0.23	0.36
Deepseek v3.2	0.36	0.34	0.30
Claude 3 Haiku	0.43	0.26	0.31
LLaMA 3.1 8B	0.59	0.22	0.18
Gemma 3 12B	0.64	0.24	0.13

Appendix

Table 3: Overall error distribution for explanation generation. Highest per column in bold , second-highest underlined .

Query Type	LLaMA 3.1 8B	Gemma 4 31B	Gemma 3 12B	Qwen 3 8B	Claude 3 Haiku	Deepseek v3.2
Linear Spike	0.60	0.50	0.50	0.96	0.60	0.54
Seasonal Drop	0.42	0.52	0.20	0.92	0.18	0.48
Structural Break	0.58	0.94	0.80	0.94	0.38	0.86
Multi-Metric Consistency	0.48	0.54	0.54	0.94	0.50	0.58
Relative Extremum	0.32	0.62	0.50	0.38	0.22	0.64
Mean Shift	0.28	0.72	0.60	0.94	0.22	0.84

Appendix

Table 4: Relative ranking accuracy per query type. Best per row in bold , second-best underlined .

Length	LLaMA 3.1 8B	Gemma 4 31B	Gemma 3 12B	Qwen 3 8B	Claude 3 Haiku	Deepseek v3.2
100	0.48	0.69	0.53	0.87	0.39	0.79
200	0.44	0.70	0.58	0.86	0.46	0.73
300	0.56	0.66	0.48	0.81	0.46	0.69
500	0.51	0.61	0.44	0.79	0.34	0.71

Appendix

Table 5: Ranking accuracy by time-series length. Best per row in bold , second-best underlined .

Query Type	LLaMA 3.1 8B	Gemma 4 31B	Gemma 3 12B	Qwen 3 8B	Claude 3 Haiku	Deepseek v3.2
Linear Spike	0.35	0.67	0.59	0.96	0.40	0.62
Seasonal Drop	0.27	0.68	0.51	0.82	0.28	0.42
Structural Break	0.22	0.95	0.63	0.75	0.39	0.85
Multi-Metric Consistency	0.38	0.67	0.33	0.65	0.31	0.33
Relative Extremum	0.16	0.61	0.15	0.45	0.43	0.31
Mean Shift	0.27	0.71	0.36	0.91	0.33	0.79

Appendix

Table 6: Independent scoring accuracy per query type. Best per row in bold , second-best underlined .

Length	LLaMA 3.1 8B	Gemma 4 31B	Gemma 3 12B	Qwen 3 8B	Claude 3 Haiku	Deepseek v3.2
100	0.29	0.70	0.49	0.72	0.35	0.57
200	0.25	0.68	0.47	0.74	0.35	0.56
300	0.24	0.69	0.66	0.74	0.35	0.54
500	0.32	0.61	0.68	0.70	0.35	0.54

Appendix

Table 7: Scoring accuracy by time-series length. Best per row in bold , second-best underlined .

	LLaMA 3.1 8B		Gemma 4 31B		Gemma 3 12B		Qwen 3 8B		Claude 3 Haiku		Deepseek v3.2
Length	CA	F1	CA	F1	CA	F1	CA	F1	CA	F1	CA	F1
100	0.00	0.414	0.56	0.991	0.00	0.528	0.08	0.500	0.00	0.572	0.20	0.932
200	0.04	0.391	0.60	0.976	0.00	0.270	0.12	0.455	0.00	0.492	0.48	0.871
300	0.00	0.337	0.24	0.910	0.00	0.155	0.12	0.585	0.00	0.700	0.16	0.853
500	0.00	0.205	0.40	0.632	0.00	0.148	0.16	0.576	0.04	0.696	0.16	0.838
Avg.	0.01	0.337	0.45	0.877	0.00	0.275	0.12	0.529	0.01	0.615	0.25	0.874

Appendix

Table 8: Multi-anomaly detection performance (Count Accuracy and F1 Score). Best per row in bold , second-best underlined .

Panel of Figure 1 — Figure 1: Instance ls_0 (Linear Spike, $n{=}100$ , first 45 steps shown). The series follows a linear upward trend with Gaussian noise. A single transient spike at $t{=}35$ ( $\bullet$ ) deviates sharply from the local trend before reverting immediately.

Query Type	Faithful fields	Incorrect strategy
Linear Spike	anomaly_index , percent_change , z_score	Denial + index shift ( $+5$ ) + large perturbation ( $\pm$ 50%)
Seasonal Drop	anomaly_index , percent_change , z_score	Denial of drop; attributes trough to seasonal cycle
Structural Break	break_index , mean_before , mean_after , percent_shift	Denial; reports stable mean with large perturbation ( $\pm$ 50%)
Multi-Metric	anomaly_index , percent_change , z_score , variance_before/after	Denial + fixed small change (5%) + inverted variance direction
Relative Extremum	largest_spike_index , largest_spike_magnitude	Attributes maximum to a different spike index
Mean Shift	mean_first_half , mean_second_half , percent_difference	Asserts equal means; uses perturbed first-half mean for both halves

Appendix

Table 9: Candidate construction strategy per query type. The “Faithful fields” column lists the ground-truth keys substituted into the correct template. The “Incorrect strategy” column describes how the label-0 candidate is constructed.

Explanation Evaluation

Explanation evaluation measures whether AI explanations are faithful, understandable, useful, and appropriate for their intended audience.

See all 26 papers

TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale
Apr 11, 2026Malgorzata Gwiazda, Yifu Cai, Mononito Goswami +2Time Series Reasoning LLM Evaluation
Technical University of Munich, Munich, Germany · Auton Lab, School of Computer Science, Carnegie Mellon University, Pittsburgh, US
Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliability
Oct 5, 2026Zhuomin Chen, Jingchao Ni, Xu Zheng +4Time Series Reasoning Time Series QA
Florida International University · University of Houston · NEC Laboratories America +1
ReasonCast: Towards Explainable Time Series Forecasting with Reasoning
Aug 3, 2026Seunghan Lee, Jun Seo, Jaehoon Lee +9Explainable Time Series Forecasting Time Series Reasoning
LG AI Research

Apr 11, 2026J/K move · Enter open · S save

TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

Malgorzata Gwiazda, Yifu Cai, Mononito Goswami, Arjun Choudhry+1

Technical University of Munich, Munich, Germany · Auton Lab, School of Computer Science, Carnegie Mellon University, Pittsburgh, US

Time Series Reasoning LLM Evaluation Synthetic Benchmark Generation Time Series Modeling

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data? While multiple benchmarks have been proposed to answer this fundamental question, most are manually curated and focus on narrow domains or specific skill sets. To address this limitation, we propose scalable methods for creating comprehensive time series reasoning benchmarks that combine the flexibility of templates with the creativity of LLM agents. We first develop TimeSeriesExam, a multiple-choice benchmark using synthetic time series to evaluate LLMs across five core reasoning categories: pattern recognitionnoise understandingsimilarity analysisanomaly detection, and causality. Then, with TimeSeriesExamAgent, we scale our approach by automatically generating benchmarks from real-world datasets spanning healthcare, finance and weather domains. Through multi-dimensional quality evaluation, we demonstrate that our automatically generated benchmarks achieve diversity comparable to manually curated alternatives. However, our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specific applications, highlighting ongoing challenges in enabling effective time series understanding in these models. TimeSeriesExamAgent is available at https://github.com/magwiazda/TimeSeriesExamAgent.

Abstract

Explanation Evaluation

Explore similar work

TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliability

ReasonCast: Towards Explainable Time Series Forecasting with Reasoning