MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

Ziyang Zhang; Shenyi Li; Yilin Wang; Ziyun Cui; Bowen Zhou; Wen Wu; Chao Zhang

cs.LGOct 13, 2026

MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

Authors: Ziyang Zhang, Shenyi Li, Yilin Wang, Ziyun Cui, Bowen Zhou, Wen Wu, Chao Zhang

Organizations: Tsinghua University · Shanghai Artificial Intelligence Laboratory

Abstract

Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains -- robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate -- under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.

Figures & tables

Figure 1: Overview of MMTA. The benchmark unifies nine real-world domains – Robotics, Autonomous Driving, Human Action, Manufacturing, AR Assistance, Healthcare, Recommendation, Finance, and Climate – under two cross-modal layouts (video $+$ time series and image $+$ time series). Vision provides real-world context, text provides semantic context, and time series provides quantitative dynamics; which modality dominates varies across domains.

Figure 2: Overview of TimeOmni-v. Time series enters the multimodal LLM through a dedicated Chronos-2 encoder branch parallel to the vision encoder. TS tokens carry a TS-specific multimodal RoPE and are arranged relative to the visual stream via a modality-aware layout (interleaved with video, blocked against still images). A forecast head is attached for numerical prediction; the standard LM head is used for classification.

Open-source vision-language MLLMs
	Classification (swF1 $\uparrow$ )						Forecasting (swMAPE $\downarrow$ )			Overall
Model	AgiBot	CoVLA	CUHK-X	FuFa	HoloA	MIMIC	PixelRec	SP500	Terra	Avg-F1 $\uparrow$	Avg-MAPE $\downarrow$
InternVL3.5-4B	0.034	0.495	0.218	0.049	0.340	0.528	150.0	4.77	318.4	0.277	157.7
InternVL3.5-8B	0.057	0.514	0.253	0.050	0.299	0.565	98.0	5.44	156.1	0.290	86.5
Qwen3-VL-8B-Instruct	0.108	0.592	0.084	0.049	0.269	0.655	145.5	5.49	77.4	0.293	76.1
Qwen3-VL-30B-A3B-Inst.	—	0.577	0.084	0.129	0.000	0.585	129.2	4.75	75.6	0.275	69.8

Table 1: Main results on MMTA. swF1 (

\uparrow

) for classification, swMAPE (

\downarrow

) for forecasting. Best per column in bold , second best underlined . “—” marks the cell where the model produced no parsable answer on any sample (Qwen3-VL-30B-A3B-Inst. runs out of memory on every AgiBot sample); per-row averages aggregate only over defined cells.

Model	TS tokens $\downarrow$	Per sample (s) $\downarrow$	Peak VRAM (GB) $\downarrow$	Completed $\uparrow$
Qwen2.5-Omni-7B (TS-as-text)	167,041	27.70	108.6	113 / 131
TimeOmni-v	722	1.20	35.0	131 / 131
Ratio (baseline / TimeOmni-v)	232.8 $\times$	23.0 $\times$	3.1 $\times$	—

Table 2: Inference cost of TimeOmni-v and its base model on AgiBot test (131 samples, the longest median TS in MMTA; single H200, max_new_tokens=8, identical dynamic-batching budgets). TS tokens count the time-series portion of the prompt only. Completed counts samples that finish without CUDA OOM in this run. AgiBot is reported as a worst-case TS-length study; per-sample latency and VRAM scale with TS length, the token compression ratio does not.

Figure 3: Training-time modality ablation. Each bar is TimeOmni-v trained and tested with the kept modalities only (V = vision), as the relative change (%) against the full tri-modal model in Macro-F1 for classification and in MAE for forecasting; MAE changes are sign-flipped so that a bar below zero is always worse. “n/a”: forecasting without time series is undefined, since the forecast head requires time-series input.

Figure 4: Controlled comparisons, as relative changes (%) in Macro-F1 for classification and in MAE for forecasting (sign-flipped, so a bar below zero is always worse than the reference). (a) Qwen2.5-Omni-7B fine-tuned with time series as text under the same LoRA configuration and budget, against TimeOmni-v; OOM: training runs out of memory on an H200 140 GB GPU at batch size one. (b) Non-LLM baselines: linear probes and late fusion on the frozen TimeOmni-v features, against TimeOmni-v; triangles mark bars that extend beyond the axis. (c) The TimeOmni-v integration ported to InternVL3.5-8B, against the Qwen2.5-Omni-7B version (swF1 and swMAPE; both parse every sample).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Layout	Train	Test	Duration (s)	Seq. Len.	Dim.	Visual	Task
AgiBotWorld	V $+$ TS	1,177	131	7.9–155.7	194–4,627	10	egocentric clip	Classification (4-way)
CoVLA	V $+$ TS	13,437	1,344	0.3–30.0	6–600	5	dashcam clip	Classification (3-way)
CUHK-X	V $+$ TS	2,629	308	0.05–16.05	1–816	7	skeleton clip	Classification (40-way)
Future Factories	V $+$ TS	49	46	56.8–68.4	16 (fixed)	36	16-frame surveillance	Classification (4-way)
HoloAssist	V $+$ TS	4,991	945	20.0–80.4	600–2,401	9	egocentric clip	Classification (binary)
MIMIC	I $+$ TS	150	150	—	4–39	6	1–5 chest X-rays	Classification (binary)

Appendix

Table 3: Standardized dataset statistics of MMTA. Layout indicates the cross-modal layout (V

+

TS = video

+

TS; I

+

TS = image

+

TS). Train and Test are the split sizes. Duration is the per-sample video length in seconds (min–max), reported only for V

+

TS domains. Seq. Len. is the time-series length per sample, Dim. is the number of time-series channels, and Visual summarizes the per-sample visual content. The MIMIC entry merges the mortality and discharge subtasks (75 test samples each).

Dataset	Source	License / access	Use restriction
AgiBotWorld [ 8 ]	Hugging Face: agibot-world/AgiBotWorld2026	CC BY-NC-SA 4.0	Non-commercial only; attribution and share-alike required.
CoVLA [ 23 ]	Hugging Face (gated): turing-motors/CoVLA-Dataset	Custom CoVLA terms; text descriptions also under VideoLLaMA 2 license	Academic / non-commercial only; non-transferable, non-redistributable; confidentiality required.
CUHK-X [ 24 ]	GitHub: openaiotlab/CUHK-X	Code MIT; data under DUA (non-commercial research, no redistribution); derived annotations CC BY 4.0	Raw data is not MIT; follow the DUA and do not redistribute the source data.
Future Factories [ 25 ]	Kaggle: ramyharik/ff-2024-08-13-multi-modal-dataset-33	CC BY-SA 4.0	Attribution and share-alike required for derived material.
HoloAssist [ 26 ]	https://holoassist.github.io/	CDLA-Permissive 2.0	Permissive data license; ship the CDLA Permissive 2.0 text alongside any redistribution.
MIMIC (IV / CXR-JPG / IV-Note) [ 27 , 28 , 29 ]	PhysioNet	PhysioNet Credentialed Health Data License 1.5.0	Credentialed access only; no re-identification; access not transferable; HIPAA training and code-sharing obligations apply.

Appendix

Table 4: Upstream license / access terms for the nine MMTA source datasets, as of 2026-05-07. The “Use restriction” column summarizes the most stringent term that applies to MMTA’s reuse; readers should consult the original license text before any further redistribution or commercial use.

Domain	Sample-level text	Per-sample text content
AgiBotWorld	yes	Upstream task instruction and the time-ordered execution steps inside the Task Frame
CoVLA	no	Fixed driving-task description, variable definitions, question, and three-way label space
CUHK-X	no	Fixed HAR task description, sensor definitions, and 40-way action space
Future Factories	yes	Production-cycle report (cycle ID, UTC time range, record count, process, signal meaning, camera notes)
HoloAssist	yes	Coarse-grained action description and the time-ordered fine-grained actions in the segment
MIMIC	yes	Radiology report of the imaging study (findings, and possibly clinical history and prior-exam comparisons)

Appendix

Table 5: Sample-level text in each MMTA domain.

Component	N	Agreement	$\kappa$	Adjudicated	Accuracy (95% CI)	Items to correct
CoVLA brake-reason labels	100	94%	0.910	6	96% (90.2–98.4%)	4
Terra generated text	100	95%	0.687	5	93% (86.3–96.7%)	7

Appendix

Table 6: Double-annotated audit of the two self-constructed components. Agreement and Cohen’s

\kappa

are computed before adjudication; accuracy is computed after adjudication, with a 95% Wilson interval.

Domain	Samples	Coverage failure	Boundary overhang	Max. overhang
AgiBotWorld	1,308	0 (0.000%)	2.35%	66 frames / 2.20 s
CoVLA	14,781	0 (0.000%)	4.80%	11 frames / 0.55 s
CUHK-X	2,937	0 (0.000%)	0.00%	0
Future Factories	95	0 (0.000%)	0.00%	0
HoloAssist	5,936	4 (0.067%)	0.55%	13 frames / ${\approx}$ 0.5 s
Total	25,057	4 (0.016%)	1.42%	—

Appendix

Table 7: Mechanical alignment audit of the video

+

TS domains.

Dataset	Modalities	Task	Learning rate	Epochs
covla	video $+$ TS	classification	$1\times 10^{-5}$	10
agibot	video $+$ TS	classification	$1\times 10^{-5}$	20
cuhk_x_har	video $+$ TS	classification	$1\times 10^{-5}$	10
holo_assist	video $+$ TS	classification	$2\times 10^{-6}$	10
future_factories	video $+$ TS	classification	$2\times 10^{-6}$	100
mimic_death	image $+$ TS	classification	$2\times 10^{-6}$	30

Appendix

Table 8: Per-dataset TimeOmni-v training hyperparameters. All other settings (initialization, frozen/trainable modules, optimizer, schedule shape, batching, hardware) are shared across datasets and described in Sections C.1 and C.2 .

Model	AgiBot (131)	CoVLA (1,344)	CUHK-X (308)	FuFa (46)	HoloA (945)	MIMIC-death (75)	MIMIC-disch (75)
InternVL3.5-4B	0.070 (28.2)	0.495 (100.0)	0.218 (100.0)	0.049 (100.0)	0.514 (49.3)	0.547 (100.0)	0.508 (100.0)
InternVL3.5-8B	0.138 (19.1)	0.515 (99.8)	0.260 (94.5)	0.050 (97.8)	0.483 (43.7)	0.558 (100.0)	0.571 (100.0)
Qwen3-VL-8B-Instruct	0.139 (77.9)	0.592 (100.0)	0.084 (100.0)	0.049 (100.0)	0.273 (94.7)	0.769 (100.0)	0.542 (100.0)
Qwen3-VL-30B-A3B-Inst.	— (0.0)	0.577 (100.0)	0.086 (99.4)	0.129 (100.0)	— (0.0)	0.638 (100.0)	0.532 (100.0)
Qwen2.5-Omni-7B	0.108 (56.5)	0.424 (100.0)	0.099 (100.0)	0.198 (100.0)	0.303 (74.6)	0.559 (100.0)	0.551 (100.0)
Qwen3-Omni-30B-A3B-Inst.	0.125 (28.2)	0.491 (100.0)	0.244 (100.0)	0.129 (100.0)	0.331 (63.5)	0.741 (100.0)	0.544 (100.0)

Appendix

Table 9: Classification: base Macro-F1 on the parsed subset, with SR (%) in parentheses. “—”: no sample was parsable. Best base Macro-F1 per column in bold .

Model	PixelRec (200)	SP500 (200)	Terra (200)
InternVL3.5-4B	150.0 (100.0)	4.76 (100.0)	296.1 (93.0)
InternVL3.5-8B	98.0 (100.0)	5.44 (100.0)	156.1 (100.0)
Qwen3-VL-8B-Instruct	145.5 (100.0)	5.49 (100.0)	77.4 (100.0)
Qwen3-VL-30B-A3B-Inst.	129.2 (100.0)	4.75 (100.0)	75.2 (99.5)
Qwen2.5-Omni-7B	104.4 (100.0)	7.73 (100.0)	176.9 (99.5)
Qwen3-Omni-30B-A3B-Inst.	98.0 (100.0)	6.72 (100.0)	65.4 (100.0)

Appendix

Table 10: Forecasting: base MAPE on the parsed subset, with SR (%) in parentheses. Best per column in bold ; on SP500, Gemini-3-flash (4.4503) is marginally below TimeOmni-v (4.4512).

Model	AgiBot	CoVLA	CUHK-X	FuFa	HoloA	MIMIC-death	MIMIC-disch	PixelRec	SP500	Terra
InternVL3.5-4B	0.8/0/71.0	0/0/0	0/0/0	0/0/0	0/0/50.7	0/0/0	0/0/0	0/0/0	0/0/0	0/0/7.0
InternVL3.5-8B	1.5/0/79.4	0/0/0.2	0/0/5.5	0/0/2.2	0/0/56.3	0/0/0	0/0/0	0/0/0	0/0/0	0/0/0
Qwen3-VL-8B-Instruct	22.1/0/0	0/0/0	0/0/0	0/0/0	0/0/5.3	0/0/0	0/0/0	0/0/0	0/0/0	0/0/0
Qwen3-VL-30B-A3B-Inst.	100.0/0/0	0/0/0	0/0/0.6	0/0/0	100.0/0/0	0/0/0	0/0/0	0/0/0	0/0/0	0/0/0.5
Qwen2.5-Omni-7B	11.5/0/32.1	0/0/0	0/0/0	0/0/0	4.0/0/21.4	0/0/0	0/0/0	0/0/0	0/0/0	0/0/0.5
Qwen3-Omni-30B-A3B-Inst.	42.7/0/29.0	0/0/0	0/0/0	0/0/0	21.8/0/14.7	0/0/0	0/0/0	0/0/0	0/0/0	0/0/0

Appendix

Table 11: Failure taxonomy: % CUDA OOM / % other inference failures (e.g., API refusals) / % unparsable outputs, over all samples. In each cell, SR

=100-

the sum of the three.

	CoVLA	HoloA	CUHK-X	MIMIC	AgiBot	FuFa	PixelRec	SP500	Terra
Baseline	Qwen3-VL-8B	GPT-5-mini	Gemini-3-flash	Qwen3-VL-8B	Gemini-3-flash	Gemini-3-flash	GPT-5-mini	Gemini-3-flash	Qwen3-Omni-30B
Test $n$	1,344	945	308	150	131	46	200	200	200
$\Delta$	$+$ 0.167	$+$ 0.125	$-$ 0.120	$-$ 0.053	$+$ 0.717	$+$ 0.047	$+$ 17.1	$+$ 0.00	$+$ 93.8
95% CI	[ $+$ 0.134, $+$ 0.199]	[ $+$ 0.084, $+$ 0.168]	[ $-$ 0.180, $-$ 0.042]	[ $-$ 0.157, $+$ 0.049]	[ $+$ 0.630, $+$ 0.795]	[ $-$ 0.101, $+$ 0.207]	[ $+$ 3.8, $+$ 31.0]	[ $-$ 0.76, $+$ 0.84]	[ $+$ 41.6, $+$ 169.2]
$p$	$<$ 0.001	$<$ 0.001	0.999	0.844	$<$ 0.001	0.246	0.995	0.522	$>$ 0.999

Appendix

Table 12: Paired bootstrap comparison of TimeOmni-v with the strongest baseline per dataset, with two-sided 95% intervals for

\Delta

Figure 5: Inference-time modality removal under three input conditions: full (all modalities provided), w/o vision (image/video stream removed), and w/o ts (time-series stream removed). Top row: classification swF1 (

\uparrow

); bottom row: forecasting swMAPE (

\downarrow

, log axis); each column is one model. “n/a” marks undefined entries (the TimeOmni-v forecast head requires TS input). Avg-F1 and Avg-MAPE aggregate only over defined entries.

Explore similar work

Rethinking Multimodal Time-Series Forecasting Evaluation
Jul 8, 2026Haoxin Liu, Yichen Zhou, Rajat Sen +2Multivariate Time Series Forecasting Zero-Shot Time Series Forecasting
Georgia Institute of Technology · Google Research
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Cross-Modal Learning Multivariate Time Series Forecasting
1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University
Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language
Aug 5, 2026Xinran Feng, Yi Xie, Chao Zhang +4Cross-Modal Representation Learning Multimodal Pretraining
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute +1

Jul 8, 2026J/K move · Enter open · S save

Rethinking Multimodal Time-Series Forecasting Evaluation

Haoxin Liu, Yichen Zhou, Rajat Sen, B. Aditya Prakash+1

Georgia Institute of Technology · Google Research

Multivariate Time Series Forecasting Zero-Shot Time Series Forecasting Forecasting Benchmarks Multimodal Model Evaluation

Open paper AskPDF arXiv

We introduce a new context-enriched, multimodal time series forecasting benchmark, TimesX. TimesX contains a wide selection of high-quality real-world time series with diverse domains and textual contexts obtained from an automated data generation pipeline, which helps address three main issues of existing multimodal forecasting benchmarks: (1) poor generalization due to the small scale and synthetic nature of benchmark data, (2) very limited types of textual contexts in the benchmarks, and (3) an inability to mitigate data leakage in evaluation. We conduct a thorough empirical study of zero-shot multimodal forecasting approaches on TimesX. Our results suggest that many approaches that perform well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage rich textual context accompanying time-series can outperform strong baselines on TimesX.

Abstract

Explore similar work

Rethinking Multimodal Time-Series Forecasting Evaluation

Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language