TS-MLLM: A Multi-Modal Large Language Model-based Framework for Industrial Time-Series Big Data Analysis

Haiteng Wang; Yikang Li; Yunfei Zhu; Jingheng Yan; Lei Ren; Laurence T. Yang

cs.LGMar 8, 2026

TS-MLLM: A Multi-Modal Large Language Model-based Framework for Industrial Time-Series Big Data Analysis

Authors: Haiteng Wang, Yikang Li, Yunfei Zhu, Jingheng Yan, Lei Ren, Laurence T. Yang

Organizations: School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · School of Software, Beihang University, Beijing 100191, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · State Key Laboratory of Intelligent Manufacturing System Technology, Beijing 100854, China ·

Abstract

Accurate analysis of industrial time-series big data is critical for the Prognostics and Health Management (PHM) of industrial equipment. While recent advancements in Large Language Models (LLMs) have shown promise in time-series analysis, existing methods typically focus on single-modality adaptations, failing to exploit the complementary nature of temporal signals, frequency-domain visual representations, and textual knowledge information. In this paper, we propose TS-MLLM, a unified multi-modal large language model framework designed to jointly model temporal signals, frequency-domain images, and textual domain knowledge. Specifically, we first develop an Industrial time-series Patch Modeling branch to capture long-range temporal dynamics. To integrate cross-modal priors, we introduce a Spectrum-aware Vision-Language Model Adaptation (SVLMA) mechanism that enables the model to internalize frequency-domain patterns and semantic context. Furthermore, a Temporal-centric Multi-modal Attention Fusion (TMAF) mechanism is designed to actively retrieve relevant visual and textual cues using temporal features as queries, ensuring deep cross-modal alignment. Extensive experiments on multiple industrial benchmarks demonstrate that TS-MLLM significantly outperforms state-of-the-art methods, particularly in few-shot and complex scenarios. The results validate our framework's superior robustness, efficiency, and generalization capabilities for industrial time-series prediction.

Figures & tables

Fig. 1: The overall architecture of the proposed TS-MLLM. It consists of three primary modules: (1) An Industrial time-series Modeling Branch for capturing long-range temporal dependencies; (2) SVLMA aligns frequency-domain representation with textual domain knowledge via MLLM; and (3) TMAF mechanism utilizes temporal features as queries to actively retrieve and integrate multi-modal cues for final prediction.

Sub-dataset	FD001	FD002	FD003	FD004
Training engines	100	260	100	249
Testing engines	100	259	100	248
Operation conditions	1	6	1	6
Fault modes	1	1	2	2

TABLE I: Description of C-MAPSS Datasets

Category	Parameter	FD001	FD002	FD003	FD004
Optimization	Batch Size	128	128	128	128
	Learning Rate	0.002	0.002	0.002	0.002
	Total Epochs	30	30	30	30
Data Input	Window Size ( $L$ )	40	40	40	40
Data Input	Input Channels ( $M$ )	14	14	14	14
Temporal Branch	Patch Size ( $P$ )	4	4	4	4

TABLE II: Hyperparameter Settings for Each Dataset

Method	FD001		FD002		FD003		FD004
Method	RMSE	Score	RMSE	Score	RMSE	Score	RMSE	Score
BiGRU	13.71	302.27	17.34	1345.92	14.02	663.01	19.23	1605.18
BiLSTM	14.12	296.02	16.91	1316.97	15.18	906.71	23.76	4349.31
MR-LSTM [ 24 ]	/	/	15.71	1434.27	/	/	16.81	1785.33
C-Transformer [ 37 ]	13.79	475.46	16.11	2214.59	17.10	939.10	19.77	3237.37
AMR-Net [ 38 ]	12.49	269.08	14.72	1085.03	13.88	372.05	16.30	998.57

TABLE III: Main results on C-MAPSS (FD001–FD004). Lower is better.

Fig. 2: Qualitative RUL prediction on C-MAPSS. Results on representative test engines from FD001 to FD004. For each subset, the top plot shows predicted and ground-truth RUL over time steps, and the bottom plot shows the absolute error over time steps.

Fig. 3: Few-shot performance on C-MAPSS. Results on FD001 to FD004 under different training-data ratios from 5% to 100%, evaluated by Score, MAE, MAPE, and RMSE. Lower is better.

Fig. 4: UMAP projections of time-series embeddings and MLLM feature embeddings on FD001–FD004. The two embedding sets are largely separated across subsets.

Fig. 5: Distributions of fusion-gate weights on FD001–FD004. Weights for time-series and MLLM features span a broad range and vary across subsets.

Fig. 6: UMAP visualization of C-MAPSS training features across subsets FD001–FD004. Local clustering and separation coexist, reflecting distribution shifts in operating conditions and degradation dynamics.

Fig. 7: Comparison of RMSE and Score results on sub-datasets FD001 and FD003 using different visual encoders: a lightweight CNN, ViT, Swin Transformer, and the proposed MAE-based design.

Fig. 8: Illustration of TS-MLLM in a representative RUL estimation task. The model integrates industrial time series (sensor readings), spectrum (frequency patterns), and domain knowledge (semantic context) to achieve precise health state assessment. The prediction curve closely tracks the actual degradation process with minimal error throughout the lifecycle.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence
Jul 16, 2026Haiteng Wang, Jingheng Yan, Xiaokang Wang +1Representation Learning Predictive Maintenance
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · School of Computer Science and Artificial Intelligence, Zhengzhou University, Zhengzhou 450001, China · School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China, also with the Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China, and also with the State Key Laboratory of Intelligent Manufacturing System Technology, Beijing 100854, China
FreSia: Frequency-Semantic Instantiation and Alignment for Multivariate Time Series Analysis
Oct 5, 2026Yubo Wang, Hui He, Hezhe Qiao +2Multivariate Time Series Forecasting Frequency-Domain Time Series Modeling
School of Computer Science and Technology, Beijing Institute of Technology, No.5 Zhongguancun South Street, Beijing, 100081, China · School of Computing and Information Systems, Singapore Management University, 80 Stamford Road, 178902, Singapore
Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting
Feb 2, 2026Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen +1Multivariate Time Series Forecasting Frequency-Domain Time Series Modeling
Applied Artificial Intelligence Initiative Deakin University Geelong, Australia

Jul 16, 2026J/K move · Enter open · S save

VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence

Haiteng Wang, Jingheng Yan, Xiaokang Wang, Lei Ren

School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · School of Computer Science and Artificial Intelligence, Zhengzhou University, Zhengzhou 450001, China · School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China, also with the Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China, and also with the State Key Laboratory of Intelligent Manufacturing System Technology, Beijing 100854, China

Representation Learning Predictive Maintenance Multimodal Fusion Time-Series Foundation Models

Open paper AskPDF arXiv

Industrial time series serve as the foundation for Prognostics and Health Management (PHM) to ensure the reliability and safety of industrial equipment such as aero-engines. However, existing approaches are typically limited to single-modality modeling, which restricts their generalization in complex scenarios. Although recent advances in large language models (LLMs) provide new opportunities for multimodal learning, bridging continuous time-series signals and discrete textual semantics remains an open challenge. To this end, we propose VLT, a multimodal foundation model that jointly models time-series, frequency-spectrum visual representations, and textual knowledge. A key insight is to utilize the frequency spectrum as a visual bridge to connect continuous temporal signals with discrete semantics. Specifically, a Time-aware Mixture-of-Experts (Time-MoE) is designed to capture heterogeneous temporal dynamics, while a Frequency-Text Augmented Learner enables joint modeling of spectral and semantic features within a shared representation space. Furthermore, a time-centric gradient alignment mechanism is introduced to mitigate cross-modal optimization conflicts via gradient normalization and reliability-aware dynamic reweighting. Extensive experiments on multiple industrial datasets demonstrate that VLT outperforms state-of-the-art methods, achieving superior robustness and generalization under few-shot, noisy, and incomplete-modality settings.

Abstract

Multimodal Large Language Models

Explore similar work

VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence

FreSia: Frequency-Semantic Instantiation and Alignment for Multivariate Time Series Analysis

Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting