From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space

Lehui Li; Yuyao Wang; Jisheng Yan; Wei Zhang; Jinliang Deng; Haoliang Sun; Zhongyi Han; Yongshun Gong

cs.CLMar 13, 2026

From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space

Authors: Lehui Li, Yuyao Wang, Jisheng Yan, Wei Zhang, Jinliang Deng, Haoliang Sun, Zhongyi Han, Yongshun Gong

Organizations: School of Software, Shandong University, China · Boston University, Boston, MA, USA · State Key Laboratory of Complex & Critical Software Environment, Beihang University · Research Institute of Trustworthy Autonomous Systems, SUSTech

Abstract

Incorporating textual information into time-series forecasting holds promise for addressing event-driven non-stationarity; however, a fundamental modality gap hinders effective fusion: textual descriptions express temporal impacts implicitly and qualitatively, whereas forecasting models rely on explicit and quantitative signals. Through controlled semi-synthetic experiments, we show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues. To bridge this gap, we propose TESS, which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities. This space consists of interpretable, numerically grounded temporal primitives (mean shift, volatility, shape, and lag) extracted from text by an LLM via structured prompting and filtered through confidence-aware gating. Experiments on four real-world datasets demonstrate up to a 29 percent reduction in forecasting error compared to state-of-the-art unimodal and multimodal baselines. The code will be released after acceptance.

Figures & tables

Figure 1: Illustration of the cross-modal transformation via the Temporal Semantic Space. Verbose textual narratives (left) are distilled into structured temporal primitives (middle), which then guide numerical forecasting (right).

Figure 2: Analysis of attention misalignment. Left : Distribution of focus ratio

R_{t}

on test samples. Right : Relationship between redundant token count and predictive performance.

Figure 3: Comparison of three input variants. Left : Prediction performance (MSE) across Full, Signal-Only, and Numerical inputs. Right : Training loss curves showing convergence dynamics.

Figure 4: Overview of TESS . Given numerical observations and associated text, a frozen LLM extracts temporal evolution primitives (e.g., mean shift, shape, volatility, lag) via structured prompting. These primitives, after confidence-aware gating, condition a Transformer-based forecaster that fuses semantic signals with encoded historical sequences to produce numerical predictions.

Traditional Models
Model	Bitcoin (Finance)			FNSPID (Finance)			Electricity (General)			Environment (General)
Model	MAE	MSE	RMSE	MAE	MSE	RMSE	MAE	MSE	RMSE	MAE	MSE	RMSE
TimeMixer	1.6757	4.3725	2.0910	0.0153	0.0017	0.0407	0.1064	0.0252	0.1586	0.4219	0.3693	0.6077
TSMixer	4.1202	29.1934	5.4031	0.0353	0.0082	0.0907	0.1119	0.0258	0.1606	0.4860	0.3878	0.6227
Nonstationary	1.6003	4.3420	2.0837	0.0154	0.0016	0.0400	0.1066	0.0256	0.1600	0.4178	0.3472	0.5892
TimesNet	1.4598	3.8229	1.9552	0.0152	0.0015	0.0385	0.1035	0.0242	0.1557	0.4258	0.3473	0.5893

Table 1: Comparison of baselines for time-series forecasting. Lower scores indicate better performance. Red : best, Blue : runner-up.

Dataset	Metric	w/o TESS	w/o Gating	TESS
Bitcoin	MAE	1.6025	1.1532	1.1120
	MSE	4.2238	2.3556	2.2726
	RMSE	2.0552	1.5349	1.5075
FNSPID	MAE	0.0155	0.0151	0.0147
	MSE	0.0018	0.0015	0.0012
	RMSE	0.0412	0.0387	0.0347

Table 2: Ablation study on the contributions of temporal evolution semantic space (TESS) and gating mechanism. Lower scores indicate better performance. Red : best performance.

Figure 5: Performance comparison on three types of non-stationary scenarios (Shape, Volatility, Mean Shift). TESS (blue) consistently outperforms both unimodal (teal) and multimodal (red) baselines.

Figure 6: Training loss curves comparison. TESS (blue) converges faster with smoother dynamics than direct text fusion (purple).

Figure 7: Case study on Bitcoin and Electricity datasets. TESS (blue) accurately captures temporal evolution primitives and aligns closely with ground truth (black).

Figure 8: Analysis of temporal evolution semantic space. (a) Primitive ablation on FNSPID: removing each primitive leads to MSE increase, with Mean Shift having the largest impact. (b) Gating effectiveness: correctly extracted primitives (blue) receive significantly higher soft weights than incorrect ones (red).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Datasets	Electicity	Bitcoin	FNSPID	Environment
Time Horizon	2019.01-2021.12	2019.01-2021.06	Not specified	1982.01-2012.05
Variates	19	18	7	4
Timesteps	52,560	858	49140	11,095
Granularity	30 minutes	1 day	5 days	7 days
Input length	48	7	5	7
Prediction length	48	7	5	7

Appendix

Table 3: The Electricity dataset represents the half-hourly electricity demand in a state. FNSPID provides daily stock price data integrated with time-aligned financial news. Environment dataset contains daily Air Quality Index (AQI) measurements and Bitcoin denotes the daily Bitcoin price.

Non-Stationary Time Series Forecasting

Non-stationary time series forecasting predicts future values as patterns, distributions, and relationships change over time.

See all 94 papers

Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting
Jun 17, 2026Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen +1Multivariate Time Series Forecasting Time Series Forecasting
Applied Artificial Intelligence Initiative Deakin University Geelong, Australia
TAC-Time: Texts as Channels For Multimodal Time Series Forecasting
Sep 21, 2026Jiayi Liang, Xiaotian Gu, Xinyu Xie +2Multivariate Time Series Forecasting Explainable Time Series Forecasting
East China Normal University Shanghai, China
ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Cross-Modal Alignment Dataset
May 15, 2025Chengsen Wang, Qi Qi, Zhongwen Rao +2Cross-Modal Learning Multivariate Time Series Forecasting
Beijing University of Posts and Telecommunications · Huawei Noah’s Ark Lab

Jun 17, 2026J/K move · Enter open · S save

Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting

Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen, Hung Le

Applied Artificial Intelligence Initiative Deakin University Geelong, Australia

Multivariate Time Series Forecasting Time Series Forecasting

Open paper AskPDF arXiv

Multimodal time series forecasting, which pairs numerical sequences with domain-relevant textual reports, promises to inject world knowledge into forecasting pipelines. However, we uncover a critical failure mode in existing frameworks that we term text collapse: the text branch converges to a content-independent transformation, contributing negligible discriminative signal regardless of the input description. We argue that text collapse is a consequence of a fundamental asymmetry in time series forecasting: the numerical input is strongly autocorrelated with the output, making the numerical backbone inherently dominant, while the text branch, despite carrying complementary and often critical information, is insufficiently utilized, leading to its systematic underexploitation. To address this, we propose REST-TS (Residual-Exclusive Supervision for Text in Time Series), which turns the asymmetry into a design principle: the numerical backbone produces its own independent numerical forecast, and the text branch is exclusively supervised to predict the structured components of the residual, the prediction gap that numbers cannot explain. Because no numerical pathway can reduce these losses, the text branch must extract genuine content from the input description. Evaluated across diverse real-world domains and backbone architectures, REST-TS achieves state-of-the-art performance and consistently demonstrates greater text-branch utilization than existing frameworks, providing strong empirical evidence that supervising the text branch on the residual compels it to extract genuine content from the input.

Abstract

Non-Stationary Time Series Forecasting

Explore similar work

Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting

TAC-Time: Texts as Channels For Multimodal Time Series Forecasting

ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Cross-Modal Alignment Dataset