Artificial Intelligence of Things (AIoT) systems increasingly combine sensed time series with external text such as expert reports and news, yet how reliably models use text that agrees or conflicts with the sensed history remains unclear. We introduce the first real-world diagnostic benchmark organized by text-history relationship: 1,762 text-window pairs across sensor telemetry (Weather), web-service traffic (Wikipedia), and market data streams (Finance), each labeled Aligned, Conflicting, or Unspecified by whether the text's direction agrees with the historical trend, opposes it, or is not stated. Across ten pretrained models evaluated zero-shot, we investigate text selection and combination under text-history relationships (RQ1) and error diagnosis and horizon-scaled revision control (RQ2). For RQ1, we compare two types of text on paired Weather windows: one directly forecasts temperature changes, while the other describes related weather conditions, such as winds and fronts. Under Aligned, the first type yields the lowest error for most models. Under Conflicting, it increases most models' error above the no-text baseline; we therefore add only its stated magnitude and timing to the second type. Compared with either type alone, this combination lowers mean absolute scaled error (MASE) for six models and raises directional accuracy (DA) for seven. For RQ2, an error decomposition attributes the error increase to oversized revisions, and a training-free rule that limits them outperforms forecasts both with and without text for the four models with this trade-off, lowering MASE by 4.49%-21.98% and raising DA by 2.99%-9.71%. Our dataset is available at https://github.com/Shelton1202/ACU-Bench.
Figures & tables
Fig. 1: Text–history relationships and possible effects of adding text. Left: Aligned and Conflicting indicate whether the text direction agrees with or opposes the historical trend; Unspecified indicates that no textual direction is assigned.
Model
Aligned ( n=471 )
Conflicting ( n=414 )
Unspecified ( n=877 )
MASE ↓
DA(%) ↑
MASE ↓
DA(%) ↑
MASE ↓
DA(%) ↑
w/o text
w/ text
w/o text
w/ text
w/o text
w/ text
w/o text
w/ text
w/o text
w/ text
w/o text
w/ text
GPT-6.1 Sol
0.7015
0.6754
67.73
71.76
0.6432
0.6366
61.11
58.94
0.6563
0.6491
64.20
65.56
GPT-6 Luna
0.7486
0.7350
56.48
65.82
0.7234
0.7058
55.56
57.00
0.7189
0.7108
53.02
55.76
Claude Sonnet 5.5
0.7182
0.6682
59.66
73.67
0.7141
0.6924
56.28
60.87
0.7108
0.7004
53.82
56.90
Claude Haiku 5.5
0.7792
0.7329
52.87
62.63
0.7382
0.7661
56.76
54.35
0.7430
0.7437
51.77
50.97
TABLE I: Forecasting performance with and without text
Model
Aligned ( n=201 )
Conflicting ( n=165 )
MASE ↓
DA(%) ↑
MASE ↓
DA(%) ↑
w/o text
w/ Directional
w/ Non- directional
w/o text
w/ Directional
w/ Non- directional
w/o text
w/ Directional
w/ Non- directional
w/o text
w/ Directional
w/ Non- directional
GPT-6.1 Sol
0.3539
0.2658
0.3239
85.07
96.02
86.57
0.2904
0.3600
0.3078
87.27
85.45
88.48
GPT-6 Luna
0.6111
0.3718
0.5147
61.19
86.07
62.69
0.5017
0.5270
0.4332
76.36
77.58
78.18
Claude Sonnet 5.5
0.6046
0.3502
0.4787
62.19
91.04
71.64
0.5565
0.5487
0.5210
76.97
80.00
76.97
Claude Haiku 5.5
0.7245
0.4944
0.6312
56.22
70.65
54.73
0.4640
0.6093
0.4876
76.36
75.15
75.76
TABLE II: Forecasting performance on paired Weather windows
Fig. 2: Gains of (a) direct concatenation and (b) selective augmentation on 165 Conflicting Weather windows, relative to directional text horizontally and non-directional text vertically. ΔMASE is single-text minus combined-text MASE. ΔDA is combined-text minus single-text DA.
Model
Wrong-to- Correct
Correct-to- Wrong
Subtotal
n
Cg
n
Cg
n
Cg
Qwen3-8B
89
−0.0782
55
+0.1706
144
+0.0923
TableGPT2-7B
48
−0.0239
39
+0.0492
87
+0.0254
TimeOmni-1-7B
68
−0.0445
38
+0.0660
106
+0.0215
TimeBraid-2.5B
29
−0.0213
22
+0.0218
51
+0.0004
TABLE III: Directional-correctness transitions on Conflicting windows
Model
Top 25%
Rest 75%
Qwen3-8B
+0.1971
+0.0219
TableGPT2-7B
+0.0298
+0.0047
TimeOmni-1-7B
+0.0316
+0.0056
TimeBraid-2.5B
+0.0095
+0.0061
TABLE IV: Contributions Cg by revision quartile on Conflicting windows
Model
w/o text
w/ text
w/ revision control
MASE
DA(%)
MASE
DA(%)
MASE
DA(%)
Qwen3-8B
0.8569
46.14
1.0759
54.35
0.8394
59.18
TableGPT2-7B
0.7561
47.58
0.7906
49.76
0.7551
54.59
TimeOmni-1-7B
0.8105
41.30
0.8477
48.55
0.7898
52.66
TimeBraid-2.5B
0.7585
54.83
0.7741
56.52
0.7383
58.21
TABLE V: Forecasting performance with revision control on Conflicting windows
Multimodal forecasting models that combine time series with text annotations promise richer prediction through textual context, but how do we know whether a text annotation meaningfully contributes to the forecasters prediction? This is an information-theoretic question, but to evaluate whether information-theoretic metrics can reliably measure the predictive value an annotation provides, a ground truth benchmark is needed, and none currently exist. We create a synthetic time series signal with annotations in three categories: semantically correct, incorrect, and irrelevant. Because the data generation process is fully controlled, ground-truth information content is known exactly, enabling principled evaluation of six complementary mutual information estimators (KSG, MINE, InfoNCE, CCA, PID and V-information). We show that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training. Our benchmark identifies limitations of each estimator, and these are validated on seven real-world datasets, which show how estimator performance differs on weak signals. Finally, we establish practical rules for implementing these metrics for annotation auditing and fusion selection.
Incorporating textual information into time-series forecasting holds promise for addressing event-driven non-stationarity; however, a fundamental modality gap hinders effective fusion: textual descriptions express temporal impacts implicitly and qualitatively, whereas forecasting models rely on explicit and quantitative signals. Through controlled semi-synthetic experiments, we show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues. To bridge this gap, we propose TESS, which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities. This space consists of interpretable, numerically grounded temporal primitives (mean shift, volatility, shape, and lag) extracted from text by an LLM via structured prompting and filtered through confidence-aware gating. Experiments on four real-world datasets demonstrate up to a 29 percent reduction in forecasting error compared to state-of-the-art unimodal and multimodal baselines. The code will be released after acceptance.
Lehui Li, Yuyao Wang, Jisheng Yan +5
School of Software, Shandong University, China · Boston University, Boston, MA, USA · State Key Laboratory of Complex & Critical Software Environment, Beihang University +1
We introduce a new context-enriched, multimodal time series forecasting benchmark, TimesX. TimesX contains a wide selection of high-quality real-world time series with diverse domains and textual contexts obtained from an automated data generation pipeline, which helps address three main issues of existing multimodal forecasting benchmarks: (1) poor generalization due to the small scale and synthetic nature of benchmark data, (2) very limited types of textual contexts in the benchmarks, and (3) an inability to mitigate data leakage in evaluation. We conduct a thorough empirical study of zero-shot multimodal forecasting approaches on TimesX. Our results suggest that many approaches that perform well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage rich textual context accompanying time-series can outperform strong baselines on TimesX.