TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments

Qingren Yao; Yaxuan Kong; Yuqi Nie; Yichen Li; Stefan Zohren; Anna Vettoruzzo; Qingsong Wen; Ming Jin

cs.AIAug 14, 2026

TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments

Authors: Qingren Yao, Yaxuan Kong, Yuqi Nie, Yichen Li, Stefan Zohren, Anna Vettoruzzo, Qingsong Wen, Ming Jin,

Organizations: Eindhoven University of Technology · University of Oxford · VulpiVox Intelligence · Squirrel Ai Learning ·

Abstract

Time series analysis in high-stakes domains relies on recurring data releases, where new observations can alter the evidence base and the validity of later conclusions. Existing time series QA benchmarks mostly rely on fixed snapshots, leaving temporal validity and cutoff-aware evidence use unevaluated. We introduce TimeSage-EV, a live benchmark for agentic time series analysis in evolving environments. It tracks 60 real institutional scenarios across 6 domains, comprising 1,485 scenario-period QA pairs from Feb 2023 to May 2026 and spanning monthly, weekly, daily, and irregular release cadences. At each period, large language model (LLM) agents receive time series data and source reports, while the withheld target release provides ground truth. TimeSage-EV evaluates state identification, data summarization, and outlook reasoning. Experiments with frontier LLM agents and TimeSage-1.0, a novel self-evolving agent with a reusable analytical skill library, reveal significant performance gaps across model tiers and recurring failures in temporal validity, exogenous context use, and adaptation. We release TimeSage-EV as a research resource with monthly updates, code, a leaderboard, and failure-mode analyses.

Figures & tables

Figure 1: Overview of TimeSage-EV, supporting agentic time series tasks and a live leaderboard. TS: time series.

Benchmark	TS	Text	Agent	Live	Evol.	Cutoff
Time-LLM Jin et al. (2024)	✔	✘	✘	✘	✘	✘
ChatTime Wang et al. (2025)	✔	✔	✘	✘	✘	✘
Time-MQA Kong et al. (2025)	✔	✔	✘	✘	✘	✘
TimeSeriesExam Cai et al. (2024)	✔	✔	✘	✘	✘	✘
WebArena Zhou et al. (2024)	✘	✔	✔	✘	✘	✘
LiveBench White et al. (2025)	✘	$\triangle$	$\triangle$	✔	✘	✘

Table 1: Comparison of representative benchmarks. ✔ denotes support,

\triangle

partial support, and ✘ not covered.

Figure 2: Evolving environment in TimeSage-EV with ENSO scenario. TS: time series, Doc: source document.

Figure 3: TimeSage-EV corpus statistics.

Figure 4: Overview of benchmark generation pipeline. Benchmark data is updated monthly via automated scripts.

Figure 5: Evaluation for the El Niño–Southern Oscillation (ENSO) monitoring scenario. TS: time series.

Figure 6: Left : Outcome scores across difficulty tiers and overall; error bars are per-difficulty-level percentile bootstrap 95% confidence intervals. Right : Total token cost.

Model	State ID	Change Det.	Comp. ID	Event Det.	Attribution	Forecast
Sonnet-4.6	$90.9^{+1.2}_{-1.2}$	$92.9^{+1.5}_{-1.7}$	$85.6^{+2.7}_{-2.7}$	$67.9^{+2.2}_{-2.1}$	$42.6^{+3.0}_{-3.2}$	$41.2^{+4.1}_{-4.0}$
GPT-5.4	$\mathbf{93.8}^{\mathbf{+0.8}}_{\mathbf{-0.9}}$	$96.0^{+1.2}_{-1.2}$	$\mathbf{86.9}^{\mathbf{+2.4}}_{\mathbf{-2.7}}$	$\mathbf{75.4}^{\mathbf{+1.9}}_{\mathbf{-1.9}}$	$46.2^{+3.2}_{-3.4}$	$42.7^{+4.1}_{-3.9}$
Kimi-K2.6	$84.2^{+1.8}_{-1.7}$	$84.9^{+2.4}_{-2.4}$	$80.8^{+3.2}_{-3.1}$	$61.0^{+2.6}_{-2.5}$	$39.3^{+3.3}_{-3.4}$	$33.4^{+3.9}_{-3.7}$
Qwen-3.5-397B	$92.6^{+0.9}_{-0.9}$	$\mathbf{97.0}^{\mathbf{+0.9}}_{\mathbf{-1.0}}$	$86.8^{+2.4}_{-2.5}$	$74.1^{+1.9}_{-1.9}$	$46.1^{+3.3}_{-3.2}$	$\mathbf{43.0}^{\mathbf{+4.0}}_{\mathbf{-3.8}}$
Devstral-2-123B	$83.4^{+1.5}_{-1.6}$	$76.5^{+2.8}_{-2.6}$	$79.7^{+3.1}_{-3.2}$	$64.7^{+2.4}_{-2.4}$	$\mathbf{46.7}^{\mathbf{+3.2}}_{\mathbf{-3.0}}$	$42.8^{+4.1}_{-4.3}$
Gemma-4-31B	$82.7^{+1.7}_{-1.8}$	$80.2^{+2.4}_{-2.7}$	$77.3^{+3.4}_{-3.3}$	$55.7^{+2.7}_{-2.6}$	$37.3^{+3.4}_{-3.2}$	$34.8^{+4.0}_{-3.8}$

Table 2: Per-question-type scores, reported as mean with bootstrapped 95% CI in the format

x^{+\mathrm{high}}_{-\mathrm{low}}

. Cell shading uses a shared scale (low high); the best result in each column is shown in bold.

Figure 7: Live overall-score leaderboard across periods. Scores are tracked as information accumulates from $t-23$ to cutoff time. Shaded region marks future updates.

Figure 8: Failure-mode distribution per model; right numbers indicate total score impact.

Figure 9: Sequential vs. independent evaluation for Qwen-3.5-397B (Top) and Gemma-4-31B (Bottom).

Figure 10: Analysis of self-evolving skill mechanism on Gemma-4-31B under evolving evaluation environments.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

No.	Validity check
1	Public access. Source is publicly accessible without credentials, payment, or approval.
2	Time series grounding. Task uses an observable timestamped series with values for the queried period.
3	Query persistence. The same template applies to both historical and future periods.
4	Temporal anchoring. The source provides a date, period, timestamp, or update window.
5	GT freshness. Ground truth updates at least monthly, except report-level hard tasks.
6	Temporal variability. Outputs vary over time and avoid near-constant majority labels.

Appendix

Table 3: Metadata-level validity checks in Step 2.

Tier	Criteria
Easy	Source: a data dashboard reporting current operational state as categorical indicators. Task: identify the current state (reasoning LOW, capability dependence LOW). Format: a single categorical label. Eval: exact match against the correct option.
Medium	Source: a data brief with time series and descriptive commentary, but no causal analysis. Task: summarize recent changes across time points and sub-components (reasoning MEDIUM, dependence MEDIUM). Format: a structured schema (period value, direction, leading sub-component, short summary). Eval: LLM judge or keyword matching for per-field coverage and factual consistency.
Hard	Source: an analytical report with trend judgements, drivers, risks, and outlooks. Task: reason about causes of change and what comes next (reasoning HIGH, dependence HIGH). Format: a structured schema (conclusion, key drivers, risks, uncertainties, evidence). Eval: LLM judge or keyword matching, with each claim traceable to a section, table, or figure.

Appendix

Table 4: Difficulty-tier criteria across source, task, answer format, and evaluation in Step 2.

No.	Validity check
1	Period alignment. The time series value, source release, and question instance refer to the same timestamp, date, period, or update window.
2	Minimum history. The source supports at least $24$ periods for backtesting and future live evaluation.
3	Reference traceability. Time series analyses in released reports are traceable to concrete public evidence, such as a field, label, value, table, figure, or report statement.
4	Information isolation. The evidence package for period $t$ uses only information available at or before the query time, excluding future releases, later revisions, and retrospective explanations.
5	Objective evaluability. Time series analyses in released reports can be derived by predefined rules and evaluated without relying on subjective preference.

Appendix

Table 5: Execution-level validity checks in Step 3.

ID	Question intent
A — State identification
A1	The current categorical state of the target.
A2	The latest reported value.
A3	The direction of the latest reading (up / down / flat).
B — Change detection
B1	The change from the previous period.

Appendix

Table 6: Question taxonomy in step 5: 27 fine-grained question types in six main categories (A–F).

Scoring rule	Question types
Atomic rules
Label match	A1, A3, B2, B6, C2, D2, E1, E2, E5, F2
Numeric check	A2, B1, B3, B4, B5, B7
Set overlap	C1, C4, D1, D2, E3, E5
LLM judgment	E1, period summary
Composition rules

Appendix

Table 7: Scoring-rule assignment by question type in step 5. Rule details are defined in Table 11 .

Gate	Grounding-audit check
1	Schema conformance. Ground truth validates against the answer schema; its top-level keys equal the required set, and enum values stay within the declared options.
2	File completeness. Each instance carries its question, ground truth, evidence, time series, and schema files.
3	Question–artifact consistency. Columns, files, and period labels named in the question exist and match the data; output-example keys and enum options equal the schema.
4	Rule reproducibility. The deterministic derivation rule recomputes the stored ground truth (report-grounded fields exempt).
5	Time series health. The cumulative table is strictly ascending in time.
6	Evidence–data consistency. Numeric and key–value references in the evidence resolve to table cells, and ground-truth enum strings appear in the evidence.

Appendix

Table 8: The eight grounding-audit gates in Step 7.

No.	Artifact quality check
1	Time series completeness. Core columns are populated, the date axis is continuous, and there are at least 24 rows (cf. Step 4).
2	Column hygiene. No duplicate date parts, rescaled duplicates, constant columns, embedded narrative, or agent-derivable columns.
3	GT leakage. No column’s latest value, value set, or sign restates the ground-truth answer (Easy categorical state exempted).
4	Difficulty–task alignment. Easy reads a latest-row state; Medium summarises across time from a brief; Hard reasons causally from a report, with no analytical conclusions baked into the table.
5	Question non-hallucination. Referenced fields and windows exist; the question recurs across at least three periods and instantiates over all periods; difficulty matches the tier.
6	Schema–template consistency. Tier and domain match the directory, Easy options equal the schema enum, and declared columns equal the table header.

Appendix

Table 9: The nine-section artifact quality gate in Step 8.

T	Scenario	Cad.	Source
Agriculture
e	usda_cattle	mon.	esmis.nal.usda.gov
e	usda_chicken	mon.	esmis.nal.usda.gov
e	usda_cold_storage_butter	mon.	esmis.nal.usda.gov
e	usda_cold_storage_pork	mon.	esmis.nal.usda.gov
m	usda_crop	wkly.	esmis.nal.usda.gov

Appendix

Table 10: Public data sources for all 60 TimeSage scenarios, grouped by domain. Tier: e = easy, m = middle, h = hard. Scenario names are shortened to the shortest unique prefix.

Figure 11: Per-Scenario Answer Volatility Across Domains.

Figure 12: Easy scenario: Weekly US Diesel Price-Band Classification.

Figure 13: Medium scenario: Monthly Healthcare Labor Market Analysis.

Figure 14: Hard scenario: Monthly ENSO Climate Analysis.

Rule	Match score
Label match	$1$ if $a=b$ , else $0$ .
Boolean match	$1$ if $\operatorname{bool}(a)=\operatorname{bool}(b)$ , else $0$ .
Text match	$1$ if stripped strings are equal, else $0$ .
Numeric check	$1$ if $\|a-b\|\leq\tau_{\text{abs}}$ , or $\|a-b\|/\|b\|\leq\tau_{\text{rel}}$ when relative tolerance is set and $b\neq 0$ ; else $0$ . With no tolerance, exact equality is required.
Set overlap	Dice/F1 overlap $2\|A\cap B\|/(\|A\|+\|B\|)$ ; two empty sets score $1$ , and a single empty set scores $0$ .

Appendix

Table 11: Primitive match rules. Each rule compares the agent output

a

with the ground-truth value

b

, or output set

A

with ground-truth set

B

Figure 15: Successful case of agent output.

Model	Tier	Accuracy	Coverage	Faithfulness	Quality	Overall
GPT-5.4	Overall	84.89	77.30	91.20	90.58	85.99
	Easy	98.54	98.67	96.12	93.04	96.59
	Medium	93.44	76.79	96.64	87.28	88.54
	Hard	62.09	56.58	80.44	91.69	72.70
Qwen-3.5-397B	Overall	84.72	75.57	88.66	87.99	84.24
	Easy	98.61	99.86	92.50	97.61	97.14

Appendix

Table 12: Per-difficulty-tier scores across evaluation dimensions. Cell shading uses a shared scale within each tier-and-metric group (low high); the best result in each group is shown in bold.

Evaluation Axis	Agreement Rate	Cohen’s $\kappa$
Coverage	94.5%	0.882
Faithfulness	98.4%	0.917
Quality	76.0%	0.665

Appendix

Table 13: Human–LLM judge agreement across the three evaluation axes.

Time Series QA

Time-series question answering connects natural-language questions to evidence and reasoning over sequential data.

See all 31 papers

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning
May 31, 2026Yaxuan Kong, Qingren Yao, Yuqi Nie +7Time Series Reasoning LLM Agent Evaluation
University of Oxford · VulpiVox Intelligence · Eindhoven University of Technology +3
TimeEvo: Failure-Driven Self-Evolution of a Time Series Agent
Sep 23, 2026Jie Yang, Yan Zheng, Jiarui Sun +8Time Series QA LLM Agent Self-Improvement
University of Illinois at Chicago · Visa Research · University of Florida +2
Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoning
Jun 13, 2026Sanhorn Chen, Xiaoyang Chen, Boyu Liu +1Irregular Time-Series Modeling LLM Evaluation
University of Illinois Urbana Champaign

May 31, 2026J/K move · Enter open · S save

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

Yaxuan Kong, Qingren Yao, Yuqi Nie, Yichen Li+6

University of Oxford · VulpiVox Intelligence · Eindhoven University of Technology+3

Time Series Reasoning LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Time series data inform critical decisions across many real-world domains. While large language model (LLM) agents can analyze data through natural language and tools, it remains unclear whether they can conduct reliable time series analysis across multi-turn conversations. Existing benchmarks focus on single-step tasks such as forecasting and anomaly detection, overlooking practical workflows where user goals evolve, agents must build on prior analyses, and conclusions emerge from accumulated evidence. In this work, we introduce TimeSage-MT, a multi-turn benchmark for agentic time series reasoning with 240 tasks and 2,680 dialogue turns across 8 real-world domains, spanning basic exploration to decision-oriented analysis. TimeSage-MT is built through a reproducible pipeline that converts real-world time series data into multi-turn conversations with verifiable answers. It provides a unified evaluation protocol and public leaderboard for comparing time series agentic systems. To demonstrate the benchmark's utility, we evaluate frontier LLMs alongside TimeSage, a novel structured agent equipped with a comprehensive time series skill library. The results show sharp performance drops on decision-oriented tasks, driven by failures in memory, uncertainty handling, and domain-based decision making. TimeSage-MT exposes critical gaps in current agentic reasoning and provides a rigorous foundation for future development.

Abstract

Time Series QA

Explore similar work

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

TimeEvo: Failure-Driven Self-Evolution of a Time Series Agent

Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoning