TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

Jie Gong; Maowei Jiang; Zhiwei Liu; Yankai Chen; Guojun Xiong; Xue Liu; Min Peng; Qianqian Xie

cs.AISep 21, 2026

TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

Authors: Jie Gong, Maowei Jiang, Zhiwei Liu, Yankai Chen, Guojun Xiong, Xue Liu, Min Peng, Qianqian Xie, +1 more

Organizations: School of Artificial Intelligence, Wuhan University, China · Nanjing Audit University, China · The University of Manchester, United Kingdom · MBZUAI, United Arab Emirates · McGill University, Canada · School of Computer Science, Shanghai Jiao Tong University, China

Abstract

Large language models (LLMs) are increasingly used to make predictions from numerical time-series histories and textual events. Yet accuracy alone cannot reveal whether correct answers reflect effective integration of the two inputs or instead arise from event polarity, unimodal priors, or superficial cues. Likewise, plausible explanations may rationalize predictions without faithfully reflecting the evidence that drives model behavior. We introduce TimeLitmus, a diagnostic benchmark for cross-modal understanding and explanation faithfulness in event-conditioned time-series prediction. TimeLitmus contains 4,856 evaluation records across Finance and Traffic, combining natural prediction with controlled counterfactual and contrastive interventions, explanation-targeted faithfulness tests, and systematic shortcut controls. Across ten representative LLMs, standard prediction accuracy substantially overstates reliable cross-modal understanding: Hard Paired Contrast (HPC) pair correctness peaks at only 19.2% in Finance and 11.7% in Traffic, and all ten models show lower-than-expected consistency on Finance series-side controls. Models often recognize scenario relations explicitly yet fail to apply them during independent prediction. Explanation faithfulness shows a similar gap: in Traffic, most models cite the manipulated temporal factor in over 90% of cases, while behavioral support remains below 22%. Human annotators outperform LLMs on matched controlled and hard-pair diagnostics, confirming that these distinctions are recoverable from the inputs. Natural-only adaptation yields selective gains in evidence selection and input sensitivity, but not consistent gains in controlled or hard-pair behavior. The benchmark, evaluation suite, and supervised adaptation data will be released publicly.

Figures & tables

Figure 1. Why accuracy alone is insufficient. Correct predictions and plausible explanations may result from either genuine cross-modal integration or shortcut cues.

Work	Event– Series	Controlled Pairs	Hard Pairs	Shortcut Tests	Claim-Linked Faithfulness
Event-conditioned time-series prediction and reasoning
Time-MMD ( liu2024time )	✓	✗	✗	✗	✗
Context is Key ( williams2024context )	✓	✗	✗	⚫	✗
TemporalBench ( weng2026temporalbench )	✓	⚫	✗	⚫	✗
Behavioral diagnostics beyond aggregate accuracy
Contrast Sets ( gardner2020evaluating )	✗	✓	✗	⚫	✗

Table 1. Comparison with representative evaluation frameworks. Event–Series : joint temporal–text conditioning; Controlled : targeted interventions with predefined effects; Hard Pairs : natural pairs requiring distinct responses; Shortcut : invariance and partial-input tests; Claim-Linked : behavioral support for cited factors. ✓ , ✗ , and ⚫ denote explicit, absent, and partial coverage, respectively.

Figure 2. Overview of TimeLitmus. Controlled interventions and faithfulness diagnostics test whether event-conditioned predictions and explanations rely on the intended cross-modal evidence rather than shortcuts.

Figure 3. TimeLitmus composition across Finance and Traffic. The benchmark organizes 4,856 evaluation records into Natural Prediction, Correctness Attribution, Explanation Faithfulness, and Shortcut Robustness.

Domain	Evaluator	Natural	Series CF-PC	HPC Pair
Finance	Metric-wise Best LLM	55.0	20.0	25.0
Finance	Human Avg.	71.7	53.3	36.7
Traffic	Metric-wise Best LLM	45.0	30.0	15.0
Traffic	Human Avg.	70.0	46.7	41.7

Table 2. Human solvability on a matched 200-question evaluation per annotator. Human results average three independent annotators, totaling 600 judgments. The matched LLM envelope selects the strongest evaluated model separately for each metric on the identical instances.

	Natural	Series-side		Event-side		HPC
Model	Acc.	CF-PC	$\boldsymbol{\Delta_{\mathrm{CF}}}$	CF-PC	$\boldsymbol{\Delta_{\mathrm{CF}}}$	Pair
DeepSeek v4 Flash	48.7 / 36.9	8.0 / 18.4	-8.7 / -0.9	17.5 / 33.3	-3.3 / -7.1	16.7 / 11.7
DeepSeek R1	44.5 / 41.0	13.7 / 21.3	-6.3 / -3.0	18.8 / 26.7	-5.5 / -8.0	16.1 / 11.7
Gemini 3.5 Flash	42.1 / 35.2	7.0 / 8.8	-15.9 / +1.0	37.5 / 6.7	+0.8 / -12.0	9.4 / 0.0
MiniMax-M3	47.1 / 37.1	11.0 / 11.8	-7.2 / 0.0	25.0 / 56.7	+1.8 / +0.7	13.9 / 6.7
Qwen-Plus	41.1 / 42.1	4.3 / 16.9	-16.4 / -3.5	61.3 / 13.3	-1.6 / -1.3	8.9 / 6.7

Table 3. Cross-modal prediction and controlled consistency. Each cell reports Finance / Traffic percentages. CF-PC requires both endpoints to be correct; negative

\Delta_{\mathrm{CF}}

indicates lower-than-expected pair consistency under endpoint independence.

Model	Series Citation	Series CSDR	Event Citation	Event CSDR
DeepSeek v4 Flash	22.3 / 94.1	16.4 / 18.8	100.0 / 100.0	17.5 / 33.3
DeepSeek R1	11.0 / 91.9	30.3 / 21.6	97.5 / 100.0	19.2 / 26.7
Gemini 3.5 Flash	17.7 / 99.3	17.0 / 8.9	100.0 / 100.0	37.5 / 6.7
MiniMax-M3	54.7 / 99.3	12.8 / 11.1	100.0 / 100.0	25.0 / 56.7
Qwen-Plus	22.7 / 98.5	8.8 / 17.2	100.0 / 100.0	61.3 / 13.3
GPT-5.4	38.0 / 96.3	3.5 / 9.2	100.0 / 100.0	5.0 / 63.3

Table 4. Claim-conditioned explanation faithfulness across Finance and Traffic. Each cell reports Finance / Traffic percentages. Citation measures whether the manipulated factor is cited, while CSDR measures exact pair-level behavioral support among cited factors.

Figure 4. Diagnostic separations in TimeLitmus. Panel A contrasts relation recognition with HPC Pair Correctness; Panels B and C compare Basic- and HPC-Linked Behavioral Support across their respective linked sets, with 95% bootstrap intervals.

Figure 5. Paired QLoRA-minus-Base effects across scales and domains. Natural-only adaptation produces selective gains concentrated in evidence selection, distinct from transfer to controlled and hard-pair behavior.

Component	Endpoint contrast
Shared series	Same 30-day series; normalized price: $100.0\rightarrow 89.8$ .
Event	Downgraded to Underperform $\;\rightarrow\;$ Upgraded to 4.5-Stars
Expected response	Strong downward $\;\rightarrow\;$ Strong upward
Model prediction	Strong downward $\;\rightarrow\;$ Strong downward
Explanation claim	Negative downgrade $\;\rightarrow\;$ Positive upgrade
Diagnostic profile	One endpoint correct; CF-PC $=0$ ; Citation $=1$ ; pair-level support $=0$ .

Table 5. Representative Finance event-side pair for MiniMax-M3. TimeLitmus separates endpoint correctness, controlled consistency, and claim-linked behavioral support.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$\mathcal{D},d$	Set of benchmark domains and a domain index.
$L,p_{d},\mathbf{x}_{\ell}^{(d)},\mathbf{X}_{d}$	History length, variable dimension, observation at time $\ell$ , and pre-event series.
$t,\ell$	Final pre-event index and a generic history index.
$\mathcal{E}_{d},E_{d},\mathcal{Q}_{d},Q_{d}$	Event space, textual event, task-presentation space, and task presentation.
$\mathcal{Y}_{d},g_{d},y_{d}$	Response space, benchmark labeling function, and gold response.
$M,\hat{y}_{d},\mathcal{O}_{d},O_{d}$	Evaluated model, predicted response, explanation space, and generated explanation.

Appendix

Table 6. Notation used in the problem formulation.

Diagnostic Family	Finance	Traffic	Total
Natural Prediction	380	420	800
Correctness Attribution	1,240	572	1,812
Explanation Faithfulness	700	204	904
Shortcut Robustness	680	660	1,340
Total	3,000	1,856	4,856

Appendix

Table 7. Family-level composition of TimeLitmus. Counts denote task-specific diagnostic records.

Test case	Expected	Observed
Controlled-pair oracle	1.0	1.0
Constant prediction on expected-change pairs	0.0	0.0
Incorrect predictions on both endpoints	0.0	0.0
Empty evidence set	0.0	0.0
Out-of-vocabulary evidence key	Rejected	Rejected
Identical relation for all HPC pairs	0.0	0.0

Appendix

Table 8. Evaluator sanity checks using oracle and adversarial mock outputs.

Audit stage or metric	Result
CAR-reconstructable instances	1,740
Observable-rule-covered instances	1,444
Observable pipeline coverage	82.99%
Direction accuracy	99.79%
Strength accuracy	85.42%
Four-way accuracy	85.35%

Appendix

Table 9. Finance observable rule recoverability. Pipeline coverage is measured over the common 1,740-instance CAR-reconstructable subset.

Domain	Evaluator	Natural	Series CF-PC	HPC Pair
Finance	Best LLM (full)	51.3	13.7	19.2
	Best LLM (matched)	55.0	20.0	25.0
	Human Avg.	71.7	53.3	36.7
Traffic	Best LLM (full)	42.1	21.3	11.7
	Best LLM (matched)	45.0	30.0	15.0
	Human Avg.	70.0	46.7	41.7

Appendix

Table 11. Human performance and metric-wise best LLM results on the matched subset and the complete TimeLitmus benchmark. Human values are averaged across three independent annotators, who achieve

85.0\%

overall raw agreement. All values are percentages.

Model	Natural	S Arm	S CF-PC	S Ind.	S $\Delta$	E Arm	E CF-PC	E Ind.	E $\Delta$	HPC Arm	HPC Pair
DeepSeek v4 Flash	48.7 / 36.9	41.0 / 46.3	8.0 / 18.4	16.7 / 19.3	-8.7 / -0.9	45.6 / 66.7	17.5 / 33.3	20.8 / 40.4	-3.3 / -7.1	42.2 / 40.8	16.7 / 11.7
DeepSeek R1	44.5 / 41.0	44.7 / 49.6	13.7 / 21.3	19.9 / 24.3	-6.3 / -3.0	49.4 / 63.3	18.8 / 26.7	24.3 / 34.7	-5.5 / -8.0	47.8 / 37.5	16.1 / 11.7
Gemini 3.5 Flash	42.1 / 35.2	47.8 / 43.0	7.0 / 8.8	22.9 / 7.8	-15.9 / +1.0	60.6 / 51.7	37.5 / 6.7	36.7 / 18.7	+0.8 / -12.0	45.3 / 34.2	9.4 / 0.0
MiniMax-M3	47.1 / 37.1	42.7 / 44.5	11.0 / 11.8	18.2 / 11.8	-7.2 / -0.0	48.1 / 76.7	25.0 / 56.7	23.2 / 56.0	+1.8 / +0.7	44.7 / 37.5	13.9 / 6.7
Qwen-Plus	41.1 / 42.1	45.5 / 45.2	4.3 / 16.9	20.7 / 20.4	-16.4 / -3.5	79.4 / 46.7	61.3 / 13.3	62.8 / 14.7	-1.6 / -1.3	46.1 / 38.3	8.9 / 6.7
GPT-5.4	44.7 / 34.5	37.2 / 43.4	3.0 / 8.8	13.8 / 8.8	-10.8 / +0.0	35.6 / 78.3	5.0 / 63.3	12.5 / 59.1	-7.5 / +4.2	44.2 / 32.5	8.6 / 0.0

Appendix

Table 13. Complete behavioral results. Cells report Finance / Traffic percentages. S and E denote series-side and event-side interventions. Adapted Qwen variants are reported separately from the unadapted cross-model comparison.

Model	All Raw	All Adj.	Change Raw	Change Adj.	Event	Series	Invariance
DeepSeek v4 Flash	62.5 / 90.0	50.0 / 84.0	50.0 / 86.7	25.0 / 73.3	66.7 / 93.3	41.7 / 83.3	100.0 / 100.0
DeepSeek R1	84.2 / 90.0	78.9 / 84.0	78.9 / 86.7	68.3 / 73.3	70.0 / 100.0	83.3 / 80.0	100.0 / 100.0
Gemini 3.5 Flash	60.8 / 85.0	47.8 / 76.0	47.8 / 80.0	21.7 / 60.0	83.3 / 100.0	30.0 / 70.0	100.0 / 100.0
MiniMax-M3	76.7 / 92.5	68.9 / 88.0	68.9 / 90.0	53.3 / 80.0	90.0 / 100.0	58.3 / 85.0	100.0 / 100.0
Qwen-Plus	71.7 / 94.2	62.2 / 90.7	62.2 / 92.2	43.3 / 84.4	90.0 / 100.0	48.3 / 88.3	100.0 / 100.0
GPT-5.4	65.0 / 87.5	53.3 / 80.0	53.3 / 83.3	30.0 / 66.7	90.0 / 100.0	35.0 / 75.0	100.0 / 100.0

Appendix

Table 15. Auxiliary relation-probe results, reported as Finance / Traffic percentages. Change-only majority baselines are 33.3% in Finance and 50.0% in Traffic.

Figure A2. Relation-probe category decomposition. Event-side, series-side, invariance, and change-only adjusted accuracy are reported separately. Change-only accuracy isolates directional transition recognition for the recognition–application comparison.

Model	Precision	Recall	F1
DeepSeek v4 Flash	84.3 / 64.7	57.3 / 49.5	67.8 / 48.8
DeepSeek R1	77.3 / 60.3	51.7 / 63.4	61.5 / 56.4
Gemini 3.5 Flash	78.8 / 65.0	45.9 / 34.9	57.6 / 38.6
MiniMax-M3	83.2 / 57.3	57.8 / 71.9	67.8 / 59.0
Qwen-Plus	85.8 / 55.5	56.9 / 46.0	67.7 / 42.9
GPT-5.4	85.6 / 59.9	61.0 / 71.5	70.9 / 62.9

Appendix

Table 16. Evidence precision, recall, and F1 across Finance and Traffic. Each cell reports Finance / Traffic percentages.

Model	F Basic	F HPC	F Gap	T Basic	T HPC	T Gap
DeepSeek v4 Flash	76.5 [70.5, 82.5]	36.0 [30.7, 41.3]	+40.5 [+32.5, +48.3]	60.9 [45.7, 73.9]	21.9 [9.4, 37.5]	+39.0 [+18.5, +59.0]
DeepSeek R1	87.0 [82.0, 91.5]	45.3 [39.7, 51.0]	+41.7 [+34.2, +49.0]	80.4 [69.6, 91.3]	46.9 [31.2, 62.6]	+33.6 [+12.6, +53.8]
Gemini 3.5 Flash	78.5 [72.5, 84.0]	41.0 [35.7, 46.7]	+37.5 [+29.5, +45.2]	47.8 [32.6, 63.0]	12.5 [3.1, 25.0]	+35.3 [+17.0, +53.4]
MiniMax-M3	91.0 [87.0, 95.0]	39.0 [33.3, 44.3]	+52.0 [+45.2, +58.7]	80.4 [67.4, 91.3]	31.2 [15.6, 46.9]	+49.2 [+29.2, +68.2]
Qwen-Plus	50.5 [43.5, 57.5]	43.0 [37.3, 48.7]	+7.5 [-1.3, +16.3]	87.0 [76.1, 95.7]	31.2 [15.6, 46.9]	+55.7 [+36.7, +73.8]
GPT-5.4	78.0 [72.0, 83.5]	39.0 [33.7, 44.3]	+39.0 [+31.0, +46.8]	78.3 [65.2, 89.1]	12.5 [3.1, 25.0]	+65.8 [+48.6, +81.7]

Appendix

Table 17. Linked Behavioral Support and Basic-minus-HPC gaps with 95% bootstrap intervals. Basic and HPC estimates use their respective linked sets.

Figure A3. Full diagnostic model profiles. The heatmaps preserve the multidimensional structure of TimeLitmus by showing Natural Accuracy, series-side and event-side CF-PC, HPC Pair, Evidence Recall, Basic-linked Support, HPC-linked Support, and change-only Relation Accuracy without collapsing them into a single score.

Model	F Event-only Acc.	F Series-only Acc.	F Hint $\Delta$	T Event-only $\Delta$	T Series-only $\Delta$
DeepSeek v4 Flash	48.0	48.0	+4.0	+20.0	-16.7
DeepSeek R1	40.0	26.0	-3.0	+10.0	-33.3
Gemini 3.5 Flash	38.0	14.0	+12.0	+0.0	+0.0
MiniMax-M3	42.0	52.0	+6.0	+13.3	-23.3
Qwen-Plus	38.0	36.0	+10.0	+23.3	-50.0
GPT-5.4	48.0	50.0	+1.0	+0.0	+0.0

Appendix

Table 18. Shortcut-control results. Finance Event-only and Series-only columns report absolute control accuracy. The remaining

\Delta

columns report full-input or reference accuracy minus control accuracy in percentage points.

Figure A4. Shortcut-control profiles. Finance reports accuracy under partial-modality and cue-control inputs, while Traffic reports matched full-input-minus-control effects. Together, the panels identify performance attainable through unimodal or superficial cues.

Configuration	Qwen3.5-4B-QLoRA	Qwen3.5-9B-QLoRA
Backbone repository	Qwen/Qwen3.5-4B	Qwen/Qwen3.5-9B
Backbone revision	851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a	c202236235762e1c871ad0ccb60c8ee5ba337b9a
Quantization	4-bit NF4, double quantization	4-bit NF4, double quantization
Compute precision	BF16	BF16
LoRA rank / alpha / dropout	16 / 32 / 0.05	16 / 32 / 0.05
Target modules	Attention and MLP projections	Attention and MLP projections

Appendix

Table 19. QLoRA training and selection configuration.

A: Prediction and controlled behavior
Scale	Domain	$\Delta$ Natural	$\Delta$ Series $\Delta_{\mathrm{CF}}$	$\Delta$ Event $\Delta_{\mathrm{CF}}$	$\Delta$ HPC Pair
4B	Finance	+0.5 [-0.5, +1.6]	-1.8 [-3.56, -0.04]	+1.0 [-1.6, +3.8]	-1.4 [-2.8, 0.0]
4B	Traffic	+0.2 [-0.5, +0.9]	-0.7 [-2.4, +1.1]	+1.3 [0.0, +4.3]	+1.7 [0.0, +5.0]
9B	Finance	-4.2 [-7.1, -1.3]	0.0 [-0.9, +0.9]	-2.1 [-5.6, +1.2]	-0.6 [-2.2, +1.1]
9B	Traffic	0.0 [-1.7, +1.7]	+2.1 [-0.1, +4.7]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
B: Evidence and linked diagnostics

Appendix

Table 20. Paired QLoRA-minus-Base effects with 95% paired-bootstrap intervals. Auxiliary Relation reports full-probe raw accuracy.

Metric	Base	QLoRA	$\Delta$	Paired 95% CI
Natural Accuracy	40.8	41.3	+0.5	$[-0.53,+1.58]$
Series CF-PC	9.3	6.7	-2.7	$[-5.33,+0.00]$
Series $\Delta_{\mathrm{CF}}$	-10.0	-11.8	-1.8	$[-3.56,-0.04]$
Event CF-PC	13.8	13.8	+0.0	$[-3.75,+3.75]$
Event $\Delta_{\mathrm{CF}}$	-2.0	-1.0	+1.0	$[-1.56,+3.75]$
HPC Pair Correctness	12.2	10.8	-1.4	$[-2.78,+0.00]$

Appendix

Table 21. Paired Base–QLoRA results for Qwen3.5-4B on Finance. QLoRA scores are averaged across three independently trained runs with seeds

\{42,123,1234\}

after development-set checkpoint selection. Scores, changes, and confidence intervals are percentage points.

Metric	Base	QLoRA	$\Delta$	Paired 95% CI
Natural Accuracy	40.3	36.1	-4.2	$[-7.11,-1.32]$
Series CF-PC	3.7	3.3	-0.3	$[-1.00,+0.00]$
Series $\Delta_{\mathrm{CF}}$	-15.9	-16.0	-0.0	$[-0.92,+0.95]$
Event CF-PC	38.8	37.5	-1.3	$[-6.25,+2.50]$
Event $\Delta_{\mathrm{CF}}$	-3.5	-5.6	-2.1	$[-5.59,+1.25]$
HPC Pair Correctness	3.3	2.8	-0.6	$[-2.22,+1.11]$

Appendix

Table 22. Paired Base–QLoRA results for Qwen3.5-9B on Finance. QLoRA scores are averaged across three independently trained runs with seeds

\{42,123,1234\}

after development-set checkpoint selection. Scores, changes, and confidence intervals are percentage points.

Metric	Base	QLoRA	$\Delta$	Paired 95% CI
Natural Accuracy	40.7	40.7	+0.0	$[-1.67,+1.67]$
Series CF-PC	16.9	17.6	+0.7	$[-1.47,+3.68]$
Series $\Delta_{\mathrm{CF}}$	-4.5	-2.3	+2.1	$[-0.12,+4.65]$
Event CF-PC	23.3	23.3	+0.0	$[+0.00,+0.00]$
Event $\Delta_{\mathrm{CF}}$	-3.3	-3.3	+0.0	$[-0.00,-0.00]$
HPC Pair Correctness	11.7	11.7	+0.0	$[+0.00,+0.00]$

Appendix

Table 23. Paired Base–QLoRA results for Qwen3.5-9B on Traffic. QLoRA scores are averaged across three independently trained runs with seeds

\{42,123,1234\}

after development-set checkpoint selection. Scores, changes, and confidence intervals are percentage points.

Model	Domain / Subset	Metric	Base	QLoRA	$\Delta$ [95% CI]
Qwen3.5-4B	Finance / All	Forced-choice Accuracy	30.6	38.1	$\mathbf{+7.5}$ $[+3.1,+11.9]$ pp
		Event Importance	0.068	0.287	$\mathbf{+0.220}$ $[+0.173,+0.273]$
		Series Importance	0.064	0.134	$\mathbf{+0.071}$ $[+0.032,+0.113]$
		Required-Evidence Importance	0.140	0.323	$\mathbf{+0.183}$ $[+0.058,+0.310]$
	Traffic / All	Forced-choice Accuracy	50.8	56.7	$\mathbf{+5.8}$ $[+2.2,+9.7]$ pp
		Series Importance	0.002	0.089	$\mathbf{+0.088}$ $[+0.052,+0.123]$

Appendix

Table 24. Selected output-logit sensitivity results for Base and QLoRA models. QLoRA values are averaged across three independently trained runs. Accuracy changes are reported in percentage points; importance changes are differences in forced-choice gold-label log-probability. Brackets report paired 95% bootstrap confidence intervals.

Explore similar work

Aug 3, 2026cs.AI

ReasonCast: Towards Explainable Time Series Forecasting with Reasoning

Most time series (TS) models are specialized for a single task, either understanding (i.e., returning text answers about a TS) or generation (i.e., returning a numeric forecast). Only recently have unified models begun to handle the two within a single architecture. Even these models, however, produce the two outputs as task-separated paths and cannot predict a series and explain why that prediction arises within a single coherent response. In this paper, we argue for a task-fused model that jointly produces 1) prediction (generation) and 2) selfexplanation (understanding), thereby integrating 1) numerical TS forecasting and 2) interpretable text reasoning within a single response. To enable the systematic study of this capability, we present both a benchmark and a recipe that jointly address the two tasks. The benchmark, ReasonTS-Bench, identifies five fundamental patterns underlying TS and enables the joint evaluation of both tasks. ReasonCast, our recipe for finetuning any LLM to perform both tasks jointly, yields a model that generates a reasoning chain and a forecast together in a single autoregressive pass. Extensive experiments show that ReasonCast outperforms both LLMs and TS models on prediction accuracy while producing verifiable, causal reasoning. Code is available at: https://github.com/seunghan96/reasoncast.

Seunghan Lee, Jun Seo, Jaehoon Lee +9

LG AI Research

Explainable Time Series Forecasting Time Series Reasoning Time Series Forecasting Forecasting Benchmarks LLM-Based Time Series Forecasting

Ask PDF

Jun 20, 2026cs.CL

From Recognition to Understanding: Unlocking Cognitive Time Series Reasoning with LLMs

Time series analysis has recently been coupled with Large Language Models (LLMs) to leverage their reasoning and world knowledge capabilities, yet gains remain limited. We attribute this to a fundamental mismatch between existing task formulations and LLM strengths: most settings reduce time series understanding to curve-fitting systems, focusing on low-level prediction while ignoring the semantic, contextual, and reasoning-intensive nature of real-world temporal decision-making.To address these limitations, we introduce TSCognition, a multimodal benchmark for multi-dimensional time series reasoning. It collects real-world time series and textual information from 15 public sources and constructs approximately 41K QA samples around five cognitive reasoning tasks: Decoding, Grounding, Inferring, Extrapolating, and Acting. Building on this, we further propose TSAlign, a unified framework that encodes time series into compact patch-level representations and aligns them with semantic directions in the LLM embedding space via gated residual injection and multivariate fusion.Experiments show that TSAlign outperforms existing LLM, VLM, and time series QA baselines on TSCognition and the publicly available TimerBed benchmark while substantially reducing computational cost.Code is available at: https://github.com/EIT-NLP/CognitiveTSR

Xin Qiu, Junlong Tong, Yao Zhang +3

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Zhejiang University · Munich Center for Machine Learning, LMU

Time Series Reasoning Time Series QA Temporal Reasoning in Language Models

Ask PDF

Aug 4, 2026cs.CL

SocietyBench: Forecasting Counterfactual Social-World Evolution

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutoff date on that timeline into an audited bank of forecasting questions. Questions are scored on two orthogonal 100-point axes: probability calibration and temporal accuracy. Before any model sees a timeline, a three-phase procedure replaces every named entity and shifts every date by a per-event constant, turning a real arc into a counterfactual social world -- structurally identical to what happened, but stripped of the surface labels a model could match against pre-training memory. On five heterogeneous events and 125 prediction points in Chinese and English editions, the strongest of six frontier LLMs reaches only 75.0 out of 100, against a trivial anchor of 50. The two axes come apart: a model can be calibration-strong but time-weak, or the reverse. Three agent frameworks built on a shared base model fail to improve on that base, and two model-free heuristics trail every LLM. Per-event gaps reach 21.4 points on a single axis, which is our main argument for evaluating on several events rather than one. All anonymized timelines, question banks, ground truth, and scoring code are released.

Zhenran Wang, Zhonghan Bian, Jinsong Li +1

Counterfactual Evaluation Forecasting Benchmarks Probabilistic Forecasting Language Model Calibration

Ask PDF