Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains -- robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate -- under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.
Figures & tables
Figure 1: Overview of MMTA. The benchmark unifies nine real-world domains – Robotics, Autonomous Driving, Human Action, Manufacturing, AR Assistance, Healthcare, Recommendation, Finance, and Climate – under two cross-modal layouts (video + time series and image + time series). Vision provides real-world context, text provides semantic context, and time series provides quantitative dynamics; which modality dominates varies across domains.
Figure 2: Overview of TimeOmni-v. Time series enters the multimodal LLM through a dedicated Chronos-2 encoder branch parallel to the vision encoder. TS tokens carry a TS-specific multimodal RoPE and are arranged relative to the visual stream via a modality-aware layout (interleaved with video, blocked against still images). A forecast head is attached for numerical prediction; the standard LM head is used for classification.
Classification (swF1 ↑ )
Forecasting (swMAPE ↓ )
Overall
Model
AgiBot
CoVLA
CUHK-X
FuFa
HoloA
MIMIC
PixelRec
SP500
Terra
Avg-F1 ↑
Avg-MAPE ↓
Open-source vision-language MLLMs
InternVL3.5-4B
0.034
0.495
0.218
0.049
0.340
0.528
150.0
4.77
318.4
0.277
157.7
InternVL3.5-8B
0.057
0.514
0.253
0.050
0.299
0.565
98.0
5.44
156.1
0.290
86.5
Qwen3-VL-8B-Instruct
0.108
0.592
0.084
0.049
0.269
0.655
145.5
5.49
77.4
0.293
76.1
Qwen3-VL-30B-A3B-Inst.
—
0.577
0.084
0.129
0.000
0.585
129.2
4.75
75.6
0.275
69.8
Table 1: Main results on MMTA. swF1 ( ↑ ) for classification, swMAPE ( ↓ ) for forecasting. Best per column in bold , second best underlined . “—” marks the cell where the model produced no parsable answer on any sample (Qwen3-VL-30B-A3B-Inst. runs out of memory on every AgiBot sample); per-row averages aggregate only over defined cells.
Model
TS tokens ↓
Per sample (s) ↓
Peak VRAM (GB) ↓
Completed ↑
Qwen2.5-Omni-7B (TS-as-text)
167,041
27.70
108.6
113 / 131
TimeOmni-v
722
1.20
35.0
131 / 131
Ratio (baseline / TimeOmni-v)
232.8 ×
23.0 ×
3.1 ×
—
Table 2: Inference cost of TimeOmni-v and its base model on AgiBot test (131 samples, the longest median TS in MMTA; single H200, max_new_tokens=8, identical dynamic-batching budgets). TS tokens count the time-series portion of the prompt only. Completed counts samples that finish without CUDA OOM in this run. AgiBot is reported as a worst-case TS-length study; per-sample latency and VRAM scale with TS length, the token compression ratio does not.
Figure 3: Training-time modality ablation. Each bar is TimeOmni-v trained and tested with the kept modalities only (V = vision), as the relative change (%) against the full tri-modal model in Macro-F1 for classification and in MAE for forecasting; MAE changes are sign-flipped so that a bar below zero is always worse. “n/a”: forecasting without time series is undefined, since the forecast head requires time-series input.
Figure 4: Controlled comparisons, as relative changes (%) in Macro-F1 for classification and in MAE for forecasting (sign-flipped, so a bar below zero is always worse than the reference). (a) Qwen2.5-Omni-7B fine-tuned with time series as text under the same LoRA configuration and budget, against TimeOmni-v; OOM: training runs out of memory on an H200 140 GB GPU at batch size one. (b) Non-LLM baselines: linear probes and late fusion on the frozen TimeOmni-v features, against TimeOmni-v; triangles mark bars that extend beyond the axis. (c) The TimeOmni-v integration ported to InternVL3.5-8B, against the Qwen2.5-Omni-7B version (swF1 and swMAPE; both parse every sample).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Layout
Train
Test
Duration (s)
Seq. Len.
Dim.
Visual
Task
AgiBotWorld
V + TS
1,177
131
7.9–155.7
194–4,627
10
egocentric clip
Classification (4-way)
CoVLA
V + TS
13,437
1,344
0.3–30.0
6–600
5
dashcam clip
Classification (3-way)
CUHK-X
V + TS
2,629
308
0.05–16.05
1–816
7
skeleton clip
Classification (40-way)
Future Factories
V + TS
49
46
56.8–68.4
16 (fixed)
36
16-frame surveillance
Classification (4-way)
HoloAssist
V + TS
4,991
945
20.0–80.4
600–2,401
9
egocentric clip
Classification (binary)
MIMIC
I + TS
150
150
—
4–39
6
1–5 chest X-rays
Classification (binary)
Appendix
Table 3: Standardized dataset statistics of MMTA. Layout indicates the cross-modal layout (V + TS = video + TS; I + TS = image + TS). Train and Test are the split sizes. Duration is the per-sample video length in seconds (min–max), reported only for V + TS domains. Seq. Len. is the time-series length per sample, Dim. is the number of time-series channels, and Visual summarizes the per-sample visual content. The MIMIC entry merges the mortality and discharge subtasks (75 test samples each).
Dataset
Source
License / access
Use restriction
AgiBotWorld [ 8 ]
Hugging Face: agibot-world/AgiBotWorld2026
CC BY-NC-SA 4.0
Non-commercial only; attribution and share-alike required.
CoVLA [ 23 ]
Hugging Face (gated): turing-motors/CoVLA-Dataset
Custom CoVLA terms; text descriptions also under VideoLLaMA 2 license
Attribution and share-alike required for derived material.
HoloAssist [ 26 ]
https://holoassist.github.io/
CDLA-Permissive 2.0
Permissive data license; ship the CDLA Permissive 2.0 text alongside any redistribution.
MIMIC (IV / CXR-JPG / IV-Note) [ 27 , 28 , 29 ]
PhysioNet
PhysioNet Credentialed Health Data License 1.5.0
Credentialed access only; no re-identification; access not transferable; HIPAA training and code-sharing obligations apply.
Appendix
Table 4: Upstream license / access terms for the nine MMTA source datasets, as of 2026-05-07. The “Use restriction” column summarizes the most stringent term that applies to MMTA’s reuse; readers should consult the original license text before any further redistribution or commercial use.
Domain
Sample-level text
Per-sample text content
AgiBotWorld
yes
Upstream task instruction and the time-ordered execution steps inside the Task Frame
CoVLA
no
Fixed driving-task description, variable definitions, question, and three-way label space
CUHK-X
no
Fixed HAR task description, sensor definitions, and 40-way action space
Future Factories
yes
Production-cycle report (cycle ID, UTC time range, record count, process, signal meaning, camera notes)
HoloAssist
yes
Coarse-grained action description and the time-ordered fine-grained actions in the segment
MIMIC
yes
Radiology report of the imaging study (findings, and possibly clinical history and prior-exam comparisons)
Appendix
Table 5: Sample-level text in each MMTA domain.
Component
N
Agreement
κ
Adjudicated
Accuracy (95% CI)
Items to correct
CoVLA brake-reason labels
100
94%
0.910
6
96% (90.2–98.4%)
4
Terra generated text
100
95%
0.687
5
93% (86.3–96.7%)
7
Appendix
Table 6: Double-annotated audit of the two self-constructed components. Agreement and Cohen’s κ are computed before adjudication; accuracy is computed after adjudication, with a 95% Wilson interval.
Domain
Samples
Coverage failure
Boundary overhang
Max. overhang
AgiBotWorld
1,308
0 (0.000%)
2.35%
66 frames / 2.20 s
CoVLA
14,781
0 (0.000%)
4.80%
11 frames / 0.55 s
CUHK-X
2,937
0 (0.000%)
0.00%
0
Future Factories
95
0 (0.000%)
0.00%
0
HoloAssist
5,936
4 (0.067%)
0.55%
13 frames / ≈ 0.5 s
Total
25,057
4 (0.016%)
1.42%
—
Appendix
Table 7: Mechanical alignment audit of the video + TS domains.
Dataset
Modalities
Task
Learning rate
Epochs
covla
video + TS
classification
1×10−5
10
agibot
video + TS
classification
1×10−5
20
cuhk_x_har
video + TS
classification
1×10−5
10
holo_assist
video + TS
classification
2×10−6
10
future_factories
video + TS
classification
2×10−6
100
mimic_death
image + TS
classification
2×10−6
30
Appendix
Table 8: Per-dataset TimeOmni-v training hyperparameters. All other settings (initialization, frozen/trainable modules, optimizer, schedule shape, batching, hardware) are shared across datasets and described in Sections C.1 and C.2 .
Model
AgiBot (131)
CoVLA (1,344)
CUHK-X (308)
FuFa (46)
HoloA (945)
MIMIC-death (75)
MIMIC-disch (75)
InternVL3.5-4B
0.070 (28.2)
0.495 (100.0)
0.218 (100.0)
0.049 (100.0)
0.514 (49.3)
0.547 (100.0)
0.508 (100.0)
InternVL3.5-8B
0.138 (19.1)
0.515 (99.8)
0.260 (94.5)
0.050 (97.8)
0.483 (43.7)
0.558 (100.0)
0.571 (100.0)
Qwen3-VL-8B-Instruct
0.139 (77.9)
0.592 (100.0)
0.084 (100.0)
0.049 (100.0)
0.273 (94.7)
0.769 (100.0)
0.542 (100.0)
Qwen3-VL-30B-A3B-Inst.
— (0.0)
0.577 (100.0)
0.086 (99.4)
0.129 (100.0)
— (0.0)
0.638 (100.0)
0.532 (100.0)
Qwen2.5-Omni-7B
0.108 (56.5)
0.424 (100.0)
0.099 (100.0)
0.198 (100.0)
0.303 (74.6)
0.559 (100.0)
0.551 (100.0)
Qwen3-Omni-30B-A3B-Inst.
0.125 (28.2)
0.491 (100.0)
0.244 (100.0)
0.129 (100.0)
0.331 (63.5)
0.741 (100.0)
0.544 (100.0)
Appendix
Table 9: Classification: base Macro-F1 on the parsed subset, with SR (%) in parentheses. “—”: no sample was parsable. Best base Macro-F1 per column in bold .
Model
PixelRec (200)
SP500 (200)
Terra (200)
InternVL3.5-4B
150.0 (100.0)
4.76 (100.0)
296.1 (93.0)
InternVL3.5-8B
98.0 (100.0)
5.44 (100.0)
156.1 (100.0)
Qwen3-VL-8B-Instruct
145.5 (100.0)
5.49 (100.0)
77.4 (100.0)
Qwen3-VL-30B-A3B-Inst.
129.2 (100.0)
4.75 (100.0)
75.2 (99.5)
Qwen2.5-Omni-7B
104.4 (100.0)
7.73 (100.0)
176.9 (99.5)
Qwen3-Omni-30B-A3B-Inst.
98.0 (100.0)
6.72 (100.0)
65.4 (100.0)
Appendix
Table 10: Forecasting: base MAPE on the parsed subset, with SR (%) in parentheses. Best per column in bold ; on SP500, Gemini-3-flash (4.4503) is marginally below TimeOmni-v (4.4512).
Model
AgiBot
CoVLA
CUHK-X
FuFa
HoloA
MIMIC-death
MIMIC-disch
PixelRec
SP500
Terra
InternVL3.5-4B
0.8/0/71.0
0/0/0
0/0/0
0/0/0
0/0/50.7
0/0/0
0/0/0
0/0/0
0/0/0
0/0/7.0
InternVL3.5-8B
1.5/0/79.4
0/0/0.2
0/0/5.5
0/0/2.2
0/0/56.3
0/0/0
0/0/0
0/0/0
0/0/0
0/0/0
Qwen3-VL-8B-Instruct
22.1/0/0
0/0/0
0/0/0
0/0/0
0/0/5.3
0/0/0
0/0/0
0/0/0
0/0/0
0/0/0
Qwen3-VL-30B-A3B-Inst.
100.0/0/0
0/0/0
0/0/0.6
0/0/0
100.0/0/0
0/0/0
0/0/0
0/0/0
0/0/0
0/0/0.5
Qwen2.5-Omni-7B
11.5/0/32.1
0/0/0
0/0/0
0/0/0
4.0/0/21.4
0/0/0
0/0/0
0/0/0
0/0/0
0/0/0.5
Qwen3-Omni-30B-A3B-Inst.
42.7/0/29.0
0/0/0
0/0/0
0/0/0
21.8/0/14.7
0/0/0
0/0/0
0/0/0
0/0/0
0/0/0
Appendix
Table 11: Failure taxonomy: % CUDA OOM / % other inference failures (e.g., API refusals) / % unparsable outputs, over all samples. In each cell, SR =100− the sum of the three.
CoVLA
HoloA
CUHK-X
MIMIC
AgiBot
FuFa
PixelRec
SP500
Terra
Baseline
Qwen3-VL-8B
GPT-5-mini
Gemini-3-flash
Qwen3-VL-8B
Gemini-3-flash
Gemini-3-flash
GPT-5-mini
Gemini-3-flash
Qwen3-Omni-30B
Test n
1,344
945
308
150
131
46
200
200
200
Δ
+ 0.167
+ 0.125
− 0.120
− 0.053
+ 0.717
+ 0.047
+ 17.1
+ 0.00
+ 93.8
95% CI
[ + 0.134, + 0.199]
[ + 0.084, + 0.168]
[ − 0.180, − 0.042]
[ − 0.157, + 0.049]
[ + 0.630, + 0.795]
[ − 0.101, + 0.207]
[ + 3.8, + 31.0]
[ − 0.76, + 0.84]
[ + 41.6, + 169.2]
p
< 0.001
< 0.001
0.999
0.844
< 0.001
0.246
0.995
0.522
> 0.999
Appendix
Table 12: Paired bootstrap comparison of TimeOmni-v with the strongest baseline per dataset, with two-sided 95% intervals for Δ .
Figure 5: Inference-time modality removal under three input conditions: full (all modalities provided), w/o vision (image/video stream removed), and w/o ts (time-series stream removed). Top row: classification swF1 ( ↑ ); bottom row: forecasting swMAPE ( ↓ , log axis); each column is one model. “n/a” marks undefined entries (the TimeOmni-v forecast head requires TS input). Avg-F1 and Avg-MAPE aggregate only over defined entries.
We introduce a new context-enriched, multimodal time series forecasting benchmark, TimesX. TimesX contains a wide selection of high-quality real-world time series with diverse domains and textual contexts obtained from an automated data generation pipeline, which helps address three main issues of existing multimodal forecasting benchmarks: (1) poor generalization due to the small scale and synthetic nature of benchmark data, (2) very limited types of textual contexts in the benchmarks, and (3) an inability to mitigate data leakage in evaluation. We conduct a thorough empirical study of zero-shot multimodal forecasting approaches on TimesX. Our results suggest that many approaches that perform well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage rich textual context accompanying time-series can outperform strong baselines on TimesX.
Real-world time series come with text: metadata, descriptions, news, reports. Yet time series foundation models process numerical sequences in isolation, and the multimodal text-and-time-series models that attempt to bridge the two all adapt a pretrained language model post hoc, inheriting representations shaped without ever seeing temporal data. These models are also evaluated almost exclusively against other multimodal baselines, not against the strongest unimodal foundation models in either domain, leaving open whether joint training is needed at all. We present Chronicle, a compact 324M-parameter decoder-only transformer trained from scratch on natural language and time series within a single unified architecture. Both modalities share the same transformer blocks, attention mechanism, and residual stream; the bulk of pretraining uses unimodal batches so cross-modal capability emerges purely from shared parameters, with a short alignment stage that interleaves the two. To our knowledge, Chronicle is the first model jointly pretrained on text and time series from scratch, and the first multimodal model evaluated against dedicated foundation models in both domains. It matches Gemma-3-270M-PT on 19 NLU tasks, sets a new bar for frozen-embedding time series classification on 24 UCR/UEA datasets, and produces multimodal forecasts on Time-MMD that beat every supervised fusion baseline, all from a single backbone.
Paul Quinlan, Jeremy Levasseur, Qingguo Li +1
1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University
Training multimodal models to align time series with language runs into a self-supervision trap. The usual recipe asks an LLM to read a series and write a description, so label quality is capped by the perceptual skill the model is supposed to learn. The data can never teach more than the labeler already knows. A second gap makes this worse: most datasets use a single variable, but the patterns that matter (cross-channel correlation, lead-lag structure, co-occurring anomalies) appear only with several variables, right where the labeling LLM's limits are most exposed. These two problems create a trilemma: existing methods are reliable, realistic, or scalable, but none achieves all three. We resolve this by decoupling perception from description. Deterministic code computes a set of statistics from real, open-source multivariate series; the LLM verbalizes those precomputed facts. Perception, which LLMs do poorly, is handled by computation, while the LLM handles expression. This produces CGTime, our 4B-parameter computation-grounded time-series-language model. CGTime outperforms far larger general-purpose models on multivariate understanding tasks: it attains the best multivariate fact score on our held-out benchmark (0.283 vs. 0.173 for GPT-4o-mini and 0.203 for GPT-5.4-nano), a gap that survives Holm-corrected paired significance tests against every baseline. It also states verifiable numerical facts in generated captions more accurately and covers a broader range of statistical properties.
Xinran Feng, Yi Xie, Chao Zhang +4
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute +1