cs.CLAug 23, 2026

Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting

Authors: Karthik Sridhar, Atharva Gupta, Nishant Pradhan, Murari Mandal, Dhruv Kumar, Saurabh Deshpande

Organizations: Birla AI Labs, Mumbai, India · BITS Pilani, Pilani, India · KIIT, Bhubaneswar, India

Abstract

Multimodal time-series forecasting has emerged as a promising paradigm in which natural-language context is expected to improve predictive performance. Recent multimodal foundation models, including Aurora, as well as early- and late-fusion approaches such as MM-TSFlib and TaTS, report substantial gains over unimodal baselines on the Time-MMD benchmark, attributing these improvements to textual information. However, whether these models are actually sensitive to the semantic content of the text remains unverified. We address this question through controlled text perturbations, attribution analyses, and probes of Aurora's text pathway. On Time-MMD, swapping each row's text for any other real text (empty, constant, within-domain shuffled, or cross-domain) moves mean MSE by less than 0.5%0.5\% on all three architectures. The improvement reported in the literature is recovered when a co-shipped numeric column is removed without touching text. We conclude that, on this benchmark and within this family of frozen-encoder architectures, text content is not the operative signal behind the reported gains. To support future work on text integration in multimodal foundation models for structured data, we release our perturbation protocol and evaluation harness as a reusable diagnostic toolkit.

Explore similar work

CardsList
  1. Rethinking Multimodal Time-Series Forecasting Evaluation

    Jul 8, 2026Haoxin Liu, Yichen Zhou, Rajat Sen +2Multivariate Time Series ForecastingZero-Shot Time Series Forecasting

  2. TAC-Time: Texts as Channels For Multimodal Time Series Forecasting

    Sep 21, 2026Jiayi Liang, Xiaotian Gu, Xinyu Xie +2Multivariate Time Series ForecastingExplainable Time Series Forecasting