cs.LGMar 8, 2026

TS-MLLM: A Multi-Modal Large Language Model-based Framework for Industrial Time-Series Big Data Analysis

Authors: Haiteng Wang, Yikang Li, Yunfei Zhu, Jingheng Yan, Lei Ren, Laurence T. Yang

Organizations: School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · School of Software, Beihang University, Beijing 100191, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · State Key Laboratory of Intelligent Manufacturing System Technology, Beijing 100854, China · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Accurate analysis of industrial time-series big data is critical for the Prognostics and Health Management (PHM) of industrial equipment. While recent advancements in Large Language Models (LLMs) have shown promise in time-series analysis, existing methods typically focus on single-modality adaptations, failing to exploit the complementary nature of temporal signals, frequency-domain visual representations, and textual knowledge information. In this paper, we propose TS-MLLM, a unified multi-modal large language model framework designed to jointly model temporal signals, frequency-domain images, and textual domain knowledge. Specifically, we first develop an Industrial time-series Patch Modeling branch to capture long-range temporal dynamics. To integrate cross-modal priors, we introduce a Spectrum-aware Vision-Language Model Adaptation (SVLMA) mechanism that enables the model to internalize frequency-domain patterns and semantic context. Furthermore, a Temporal-centric Multi-modal Attention Fusion (TMAF) mechanism is designed to actively retrieve relevant visual and textual cues using temporal features as queries, ensuring deep cross-modal alignment. Extensive experiments on multiple industrial benchmarks demonstrate that TS-MLLM significantly outperforms state-of-the-art methods, particularly in few-shot and complex scenarios. The results validate our framework's superior robustness, efficiency, and generalization capabilities for industrial time-series prediction.

Figures & tables

Fig. 1: The overall architecture of the proposed TS-MLLM. It consists of three primary modules: (1) An Industrial time-series Modeling Branch for capturing long-range temporal dependencies; (2) SVLMA aligns frequency-domain representation with textual domain knowledge via MLLM; and (3) TMAF mechanism utilizes temporal features as queries to actively retrieve and integrate multi-modal cues for final prediction.
Sub-datasetFD001FD002FD003FD004
Training engines100260100249
Testing engines100259100248
Operation conditions1616
Fault modes1122
TABLE I: Description of C-MAPSS Datasets
CategoryParameterFD001FD002FD003FD004
OptimizationBatch Size128128128128
Learning Rate0.0020.0020.0020.002
Total Epochs30303030
Data InputWindow Size ( LL )40404040
Input Channels ( MM )14141414
Temporal BranchPatch Size ( PP )4444
TABLE II: Hyperparameter Settings for Each Dataset
MethodFD001FD002FD003FD004
RMSEScoreRMSEScoreRMSEScoreRMSEScore
BiGRU13.71302.2717.341345.9214.02663.0119.231605.18
BiLSTM14.12296.0216.911316.9715.18906.7123.764349.31
MR-LSTM [ 24 ]//15.711434.27//16.811785.33
C-Transformer [ 37 ]13.79475.4616.112214.5917.10939.1019.773237.37
AMR-Net [ 38 ]12.49269.0814.721085.0313.88372.0516.30998.57
TABLE III: Main results on C-MAPSS (FD001–FD004). Lower is better.
Fig. 2: Qualitative RUL prediction on C-MAPSS. Results on representative test engines from FD001 to FD004. For each subset, the top plot shows predicted and ground-truth RUL over time steps, and the bottom plot shows the absolute error over time steps.
Fig. 3: Few-shot performance on C-MAPSS. Results on FD001 to FD004 under different training-data ratios from 5% to 100%, evaluated by Score, MAE, MAPE, and RMSE. Lower is better.
Fig. 4: UMAP projections of time-series embeddings and MLLM feature embeddings on FD001–FD004. The two embedding sets are largely separated across subsets.
Fig. 5: Distributions of fusion-gate weights on FD001–FD004. Weights for time-series and MLLM features span a broad range and vary across subsets.
Fig. 6: UMAP visualization of C-MAPSS training features across subsets FD001–FD004. Local clustering and separation coexist, reflecting distribution shifts in operating conditions and degradation dynamics.
Fig. 7: Comparison of RMSE and Score results on sub-datasets FD001 and FD003 using different visual encoders: a lightweight CNN, ViT, Swin Transformer, and the proposed MAE-based design.
Fig. 8: Illustration of TS-MLLM in a representative RUL estimation task. The model integrates industrial time series (sensor readings), spectrum (frequency patterns), and domain knowledge (semantic context) to achieve precise health state assessment. The prediction curve closely tracks the actual degradation process with minimal error throughout the lifecycle.
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
More in this topic

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Explore similar work

CardsList
Jul 16, 2026

VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence

Industrial time series serve as the foundation for Prognostics and Health Management (PHM) to ensure the reliability and safety of industrial equipment such as aero-engines. However, existing approaches are typically limited to single-modality modeling, which restricts their generalization in complex scenarios. Although recent advances in large language models (LLMs) provide new opportunities for multimodal learning, bridging continuous time-series signals and discrete textual semantics remains an open challenge. To this end, we propose VLT, a multimodal foundation model that jointly models time-series, frequency-spectrum visual representations, and textual knowledge. A key insight is to utilize the frequency spectrum as a visual bridge to connect continuous temporal signals with discrete semantics. Specifically, a Time-aware Mixture-of-Experts (Time-MoE) is designed to capture heterogeneous temporal dynamics, while a Frequency-Text Augmented Learner enables joint modeling of spectral and semantic features within a shared representation space. Furthermore, a time-centric gradient alignment mechanism is introduced to mitigate cross-modal optimization conflicts via gradient normalization and reliability-aware dynamic reweighting. Extensive experiments on multiple industrial datasets demonstrate that VLT outperforms state-of-the-art methods, achieving superior robustness and generalization under few-shot, noisy, and incomplete-modality settings.

Haiteng Wang, Jingheng Yan, Xiaokang Wang +1

School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · School of Computer Science and Artificial Intelligence, Zhengzhou University, Zhengzhou 450001, China · School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China, also with the Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China, and also with the State Key Laboratory of Intelligent Manufacturing System Technology, Beijing 100854, China

Representation LearningPredictive MaintenanceMultimodal FusionTime-Series Foundation Models
AskPDF
Oct 5, 2026

FreSia: Frequency-Semantic Instantiation and Alignment for Multivariate Time Series Analysis

Large Language Models (LLMs) have shown strong potential in multivariate time series forecasting and anomaly detection. Existing studies predominantly inject temporal information into LLMs via direct numerical tokenization or heuristic textual descriptions. However, LLMs still face difficulty in perceiving the underlying structural patterns of numerical time series, particularly the seasonal and trend components obscured by discrete numerical tokens. To bridge this gap, we propose FreSia, a frequency-aware framework that establishes an effective alignment between the semantic space of LLMs and the frequency space of time series. Specifically, FGPrompt, a Frequency-Guided Prompt mechanism within FreSia, distills the frequency-domain structures of time series and projects them into prompts tailored to the semantic space of LLMs. Furthermore, we introduce a Global-driven Context Learning (GCL) component, which uses a global CLS-driven probe to generate global context to bridge the time-frequency domain gap and fuse the multi-modal information. Experiments on eight forecasting benchmarks show that FreSia achieves average improvements of 13.48% and 8.06% in MSE and MAE, respectively.

Yubo Wang, Hui He, Hezhe Qiao +2

School of Computer Science and Technology, Beijing Institute of Technology, No.5 Zhongguancun South Street, Beijing, 100081, China · School of Computing and Information Systems, Singapore Management University, 80 Stamford Road, 178902, Singapore

Multivariate Time Series ForecastingFrequency-Domain Time Series ModelingTime Series ForecastingLLM-Based Time Series Forecasting
AskPDF
Feb 2, 2026

Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting

Multimodal time series forecasting is crucial in real-world applications, where decisions depend on both numerical data and contextual signals. The core challenge is to effectively combine temporal numerical patterns with the context embedded in other modalities, such as text. While most existing methods align textual features with time-series patterns one step at a time, they neglect the multiscale temporal influences of contextual information such as time-series cycles and dynamic shifts. This mismatch between local alignment and global textual context can be addressed by spectral decomposition, which separates time series into frequency components capturing both short-term changes and long-term trends. In this paper, we propose SpecTF, a simple yet effective framework that integrates the effect of textual data on time series in the frequency domain. Our method extracts textual embeddings, projects them into the frequency domain, and fuses them with the time series' spectral components using a lightweight cross-attention mechanism. This adaptively reweights frequency bands based on textual relevance before mapping the results back to the temporal domain for predictions. Experimental results demonstrate that SpecTF significantly outperforms state-of-the-art models across diverse multi-modal time series datasets while utilizing considerably fewer parameters. Code is available at https://github.com/hiepnh137/SpecTF.

Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen +1

Applied Artificial Intelligence Initiative Deakin University Geelong, Australia

Multivariate Time Series ForecastingFrequency-Domain Time Series ModelingTime Series ForecastingMultimodal FusionMultiscale Time-Series Modeling
AskPDF