From Images to Signals: Are Large Vision Models Useful for Time Series Analysis?
Organizations: University of Houston · University of Illinois at Urbana-Champaign · University of Connecticut · Squirrel Ai Learning
Abstract
Transformer-based models have gained increasing attention in time series research, driving interest in Large Language Models (LLMs) and foundation models for time series analysis. As the field moves toward multi-modality, Large Vision Models (LVMs) are emerging as a promising direction. In the past, the effectiveness of Transformer and LLMs in time series has been debated. When it comes to LVMs, a similar question arises: are LVMs truely useful for time series analysis? To address it, we design and conduct the first principled study involving 4 LVMs, 8 imaging methods, 18 datasets and 26 baselines across both high-level (classification) and low-level (forecasting) tasks, with extensive ablation analysis. Our findings indicate LVMs are indeed useful for time series classification but face challenges in forecasting. Although effective, the contemporary best LVM forecasters are limited to specific types of LVMs and imaging methods, exhibit a bias toward forecasting periods, and have limited ability to utilize long look-back windows. We hope our findings could serve as a cornerstone for future research on LVM- and multimodal-based solutions to different time series tasks.
Figures & tables
| Task | Imaging | ViT | Swin | MAE | SimMIM |
|---|---|---|---|---|---|
| Classification | All | (b) | (b) | (b) | (b) |
| Forecasting | UVH,MVH | (c) | (c) | (d) | (d) |
| Forecasting | Other | (c) | (c) | (c) | (c) |
| Method | MAE | ViT | Time-LLM | GPT4TS | CALF | Dlinear | PatchTST | TimesNet | FEDformer | Autoformer | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 0.409 | 0.419 | 0.445 | 0.449 | 0.418 | 0.432 | 0.418 | 0.421 | 0.432 | 0.431 | 0.423 | 0.437 | 0.413 | 0.431 | 0.458 | 0.450 | 0.440 | 0.460 | 0.496 | 0.487 |
| ETTh2 | 0.357 | 0.390 | 0.389 | 0.411 | 0.361 | 0.396 | 0.354 | 0.389 | 0.351 | 0.384 | 0.431 | 0.447 | 0.330 | 0.379 | 0.414 | 0.427 | 0.437 | 0.449 | 0.450 | 0.459 |
| ETTm1 | 0.345 | 0.374 | 0.409 | 0.422 | 0.356 | 0.377 | 0.363 | 0.378 | 0.396 | 0.391 | 0.357 | 0.379 | 0.351 | 0.381 | 0.400 | 0.406 | 0.448 | 0.452 | 0.588 | 0.517 |
| ETTm2 | 0.268 | 0.327 | 0.300 | 0.337 | 0.261 | 0.316 | 0.254 | 0.311 | 0.283 | 0.323 | 0.267 | 0.334 | 0.255 | 0.315 | 0.291 | 0.333 | 0.305 | 0.349 | 0.327 | 0.371 |
| Weather | 0.225 | 0.258 | 0.234 | 0.273 | 0.244 | 0.270 | 0.227 | 0.255 | 0.251 | 0.274 | 0.249 | 0.300 | 0.226 | 0.264 | 0.259 | 0.287 | 0.309 | 0.360 | 0.338 | 0.382 |
| Task | TSC Task (accuracy (%) ↑ ) | TSF Task (MSE ↓ ) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | UWave. | Spoken. | Handwrit. | FaceDetect. | ETTh1 | ETTm1 | Illiness | Weather | |
| RQ3 | (a) All parameters | 88.4 | 98.5 | 36.4 | 67.4 | 0.558 | 0.399 | 1.781 | 0.273 |
| (b) All but CLS & Mask | 87.5 | 98.2 | 35.2 | 66.3 | 0.530 | 0.408 | 1.783 | 0.275 | |
| (c) MLP & norm | 88.7 | 98.4 | 35.5 | 67.1 | 0.532 | 0.396 | 1.737 | 0.264 | |
| (d) Norm | 81.6 | 98.0 | 28.5 | 65.2 | 0.409 | 0.345 | 1.837 | 0.225 | |
| (e) Zero-shot | 84.0 | 98.5 | 27.8 | 66.7 | 0.452 | 0.420 | 2.037 | 0.308 | |
| Task | Classification | Forecasting | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | UWave. | Spoken. | Handwrit. | FaceDetect. | ETTh1 | ETTm1 | Illiness | Weather | |
| Sf-All | w/o-LVM | 78.2% | 49.7% | 81.7% | 19.3% | 76.2% | 98.4% | 116.4% | 24.1% |
| LVM2Attn | 86.4% | 50.6% | 89.9% | 22.4% | 79.7% | 117.1% | 109.1% | 24.4% | |
| LVM | 80.7% | 84.7% | 91.5% | 29.2% | 83.8% | 118.4% | 162.8% | 44.5% | |
| Sf-Half | w/o-LVM | 6.6% | 12.4% | 74.6% | 10.8% | 14.4% | 28.3% | 41.6% | 2.4% |
| LVM2Attn | 8.7% | 11.6% | 83.6% | 11.3% | 19.5% | 44.8% | 69.3% | 2.4% | |
| Method | LVM | 1st Baseline (task specific) | 2nd Baseline (task specific) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | Dataset | # Param (M) | Train (min) | Inference(ms) | # Param (M) | Train (min) | Inference(ms) | # Param (M) | Time (min) | Inference(ms) |
| TSC | UWave. | 89.43 | 2.83 | 11.52 | 82.23 | 1.19 | 57.61 | 2.42 | 0.39 | 1.69 |
| Handwrit. | 97.59 | 5.18 | 23.72 | 83.62 | 1.33 | 50.51 | 2.47 | 0.51 | 0.78 | |
| TSF | ETTh1 | 111.91 | 9.99 | 4.32 | 3.75 | 0.52 | 0.18 | 85.02 | 10.46 | 0.50 |
| Weather | 111.91 | 207.83 | 1.50 | 6.90 | 16.97 | 0.10 | 86.64 | 94.10 | 0.35 | |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Variates | Series Length | Dataset Size | Classes |
|---|---|---|---|---|
| EthanolConcentration | 3 | 1751 | (261, 263, 263) | 4 |
| FaceDetection | 144 | 62 | (5890, 3524, 3524) | 2 |
| Handwriting | 3 | 152 | (150, 850, 850) | 26 |
| Heartbeat | 61 | 405 | (204, 205,205) | 2 |
| Japanese Vowels | 12 | 29 | (270, 370, 370) | 9 |
| PEMS-SF | 963 | 144 | (267, 173, 173) | 7 |
| Dataset | # Variates | Series Length | Dataset Size | Frequency |
|---|---|---|---|---|
| ETTh1 | 7 | 17420 | (8545, 2881, 2881) | Hourly |
| ETTh2 | 7 | 17420 | (8545, 2881, 2881) | Hourly |
| ETTm1 | 7 | 69680 | (34465, 11521, 11521) | 15 mins |
| ETTm2 | 7 | 69680 | (34465, 11521, 11521) | 15 mins |
| Weather | 321 | 52696 | (36792, 5271, 10540) | 10 mins |
| Illness | 7 | 966 | (617, 74, 170) | Weekly |
| ETTh1, ETTh2 | ETTm1, ETTm2 | Weather | Illness | Traffic | Electricity | |
| Top 3 Period | {24, 576, 384} | {96, 576, 384} | {144, 72, 576} | {52, 26, 17} | {24, 12, 168} | {24, 164, 82} |
| Segment Length L | 24 | 96 | 144 | 52 | 24 | 24 |
| Dataset | MAE | ViT | XGBoost | Rocket | LSTNet | LSSL | TCN | Trans. | Re. | In. | Pyra. | Auto. | Station. | FED. | ETS. | Flow. | Dlinear | LightTS | TimesNet | GPT4TS |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EthanolConcentration | 41.4 | 49.4 | 43.7 | 45.2 | 39.9 | 31.1 | 28.9 | 32.7 | 31.9 | 31.6 | 30.8 | 31.6 | 32.7 | 31.2 | 28.1 | 33.8 | 32.6 | 29.7 | 35.7 | 34.2 |
| FaceDetection | 65.4 | 67.4 | 63.3 | 64.7 | 65.7 | 66.7 | 52.8 | 67.3 | 68.6 | 67.0 | 65.7 | 68.4 | 68.0 | 66.0 | 66.3 | 67.6 | 68.0 | 67.5 | 68.6 | 69.2 |
| Handwriting | 39.5 | 36.4 | 15.8 | 58.8 | 25.8 | 24.6 | 53.3 | 32.0 | 27.4 | 32.8 | 29.4 | 36.7 | 31.6 | 28.0 | 32.5 | 33.8 | 27.0 | 26.1 | 32.1 | 32.7 |
| Heartbeat | 86.8 | 74.6 | 73.2 | 75.6 | 77.1 | 72.7 | 75.6 | 76.1 | 77.1 | 80.5 | 75.6 | 74.6 | 73.7 | 73.7 | 71.2 | 77.6 | 75.1 | 75.1 | 78.0 | 77.2 |
| Japanese Vowels | 95.4 | 98.3 | 86.5 | 96.2 | 98.1 | 98.4 | 98.9 | 98.7 | 97.8 | 98.9 | 98.4 | 96.2 | 99.2 | 98.4 | 95.9 | 98.9 | 96.2 | 96.2 | 98.4 | 98.6 |
| PEMS-SF | 84.4 | 84.2 | 98.3 | 75.1 | 86.7 | 86.1 | 68.8 | 82.1 | 82.7 | 81.5 | 83.2 | 82.7 | 87.3 | 80.9 | 86.0 | 83.8 | 75.1 | 88.4 | 89.6 | 87.9 |
| Dataset | MAE | ViT |
|---|---|---|
| EthanolConcentration | 41.4 0.5 | 49.4 0.9 |
| FaceDetection | 65.4 1.2 | 67.4 1.5 |
| Handwriting | 39.5 1.5 | 36.4 1.3 |
| Heartbeat | 86.8 2.1 | 74.6 0.6 |
| Japanese Vowels | 95.4 0.3 | 98.3 0.3 |
| PEMS-SF | 84.4 0.4 | 84.2 0.5 |
| Method | MAE | ViT | Time-LLM | GPT4TS | CALF | Dlinear | PatchTST | TimesNet | FEDformer | Autoformer | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | |
| ETTh1 | 96 | 0.356 | 0.383 | 0.398 | 0.401 | 0.376 | 0.402 | 0.370 | 0.389 | 0.370 | 0.393 | 0.375 | 0.399 | 0.370 | 0.399 | 0.384 | 0.402 | 0.376 | 0.419 | 0.449 | 0.459 |
| 192 | 0.395 | 0.406 | 0.439 | 0.445 | 0.407 | 0.421 | 0.412 | 0.413 | 0.429 | 0.426 | 0.405 | 0.416 | 0.413 | 0.421 | 0.436 | 0.429 | 0.420 | 0.448 | 0.500 | 0.482 | |
| 336 | 0.417 | 0.424 | 0.462 | 0.458 | 0.430 | 0.438 | 0.448 | 0.431 | 0.451 | 0.440 | 0.439 | 0.443 | 0.422 | 0.436 | 0.491 | 0.469 | 0.459 | 0.465 | 0.521 | 0.496 | |
| 720 | 0.467 | 0.463 | 0.479 | 0.491 | 0.457 | 0.468 | 0.441 | 0.449 | 0.476 | 0.466 | 0.472 | 0.490 | 0.447 | 0.466 | 0.521 | 0.500 | 0.506 | 0.507 | 0.514 | 0.512 | |
| ETTh2 | 96 | 0.297 | 0.341 | 0.302 | 0.355 | 0.286 | 0.346 | 0.280 | 0.335 | 0.284 | 0.336 | 0.289 | 0.353 | 0.274 | 0.336 | 0.340 | 0.374 | 0.358 | 0.397 | 0.346 | 0.388 |
| Method | MAE | ViT | |||
|---|---|---|---|---|---|
| Metrics | MSE | MAE | MSE | MAE | |
| ETTh1 | 96 | 0.356 0.001 | 0.383 0.005 | 0.398 0.011 | 0.401 0.012 |
| 192 | 0.395 0.001 | 0.406 0.001 | 0.439 0.005 | 0.445 0.003 | |
| 336 | 0.417 0.001 | 0.424 0.001 | 0.462 0.004 | 0.458 0.004 | |
| 720 | 0.467 0.012 | 0.463 0.010 | 0.479 0.011 | 0.491 0.008 | |
| ETTh2 | 96 | 0.297 0.000 | 0.341 0.004 | 0.302 0.001 | 0.355 0.000 |
| Dataset | MAE | SimMiM | ViT | Swin |
|---|---|---|---|---|
| UWaveGestureLibrary | 85.0 | 83.1 | 88.4 | 78.9 |
| SpokenArabicDigits | 98.5 | 88.2 | 98.5 | 87.3 |
| Handwriting | 39.5 | 29.8 | 36.4 | 33.1 |
| FaceDetection | 65.4 | 57.8 | 67.4 | 50.3 |
| Average | 72.1 | 64.7 | 72.6 | 62.4 |
| Dataset | Self-Supervised | Supervised | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | MAE | SimMIM | ViT | Swin | |||||
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | |
| ETTh1 | 96 | 0.356 | 0.383 | 0.362 | 0.383 | 0.398 | 0.401 | 0.407 | 0.429 |
| 192 | 0.395 | 0.406 | 0.407 | 0.412 | 0.439 | 0.445 | 0.442 | 0.458 | |
| 336 | 0.417 | 0.424 | 0.422 | 0.417 | 0.462 | 0.458 | 0.467 | 0.481 | |
| 720 | 0.467 | 0.463 | 0.462 | 0.455 | 0.479 | 0.491 | 0.470 | 0.497 | |
| Dataset | GAF | MVH | RP | STFT | Wave. | Filter. | UVH | Lineplot |
|---|---|---|---|---|---|---|---|---|
| EthanolConcentration | 49.4 | 30.7 | 43.7 | 31.9 | 27.3 | 28.1 | 28.5 | 25.2 |
| FaceDetection | 67.4 | 68.3 | 65.5 | 61.1 | 63.9 | 64.7 | 67.7 | 50.3 |
| Handwriting | 36.4 | 30.8 | 45.1 | 28.2 | 34.0 | 22.3 | 25.8 | 15.9 |
| Heartbeat | 74.6 | 77.5 | 71.7 | 74.7 | 72.6 | 73.1 | 78.0 | 53.7 |
| Japanese Vowels | 98.3 | 97.8 | 87.8 | 94.8 | 94.9 | 97.0 | 96.4 | 65.7 |
| PEMS-SF | 84.2 | 87.2 | 80.1 | 68.5 | 84.7 | 71.2 | 88.1 | 73.4 |
| Imaging Method | GAF | GAF † | MVH | RP | STFT | Wave. | Filter. | UVH | Lineplot | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 96 | 0.986 | 0.783 | 1.224 | 0.850 | 0.484 | 0.471 | 0.969 | 0.771 | 0.534 | 0.533 | 0.621 | 0.582 | 0.820 | 0.684 | 0.356 | 0.383 | 0.902 | 0.751 |
| 192 | 1.004 | 0.797 | 1.227 | 0.854 | 0.575 | 0.517 | 0.971 | 0.775 | 0.621 | 0.587 | 0.650 | 0.600 | 0.864 | 0.707 | 0.395 | 0.406 | 1.204 | 0.894 | |
| 336 | 1.038 | 0.820 | 1.214 | 0.857 | 0.623 | 0.546 | 0.989 | 0.788 | 0.602 | 0.573 | 0.681 | 0.616 | 0.827 | 0.693 | 0.417 | 0.424 | 1.223 | 0.901 | |
| 720 | 1.008 | 0.812 | 1.190 | 0.863 | 0.737 | 0.612 | 1.062 | 0.825 | 0.669 | 0.621 | 0.699 | 0.633 | 0.858 | 0.720 | 0.467 | 0.463 | 1.150 | 0.852 | |
| Average | 1.009 | 0.803 | 1.214 | 0.856 | 0.605 | 0.537 | 0.998 | 0.790 | 0.607 | 0.579 | 0.663 | 0.608 | 0.842 | 0.701 | 0.409 | 0.419 | 1.120 | 0.850 | |
| Dataset | (a) | (b) | (c) | (d) | (e) | (f) |
|---|---|---|---|---|---|---|
| UWaveGestureLibrary | 88.4 | 87.5 | 88.7 | 81.6 | 84.0 | 73.4 |
| SpokenArabicDigits | 98.5 | 98.2 | 98.4 | 98.0 | 98.5 | 97.0 |
| Handwriting | 36.4 | 35.2 | 35.5 | 28.5 | 27.8 | 24.3 |
| FaceDetection | 67.4 | 66.3 | 67.1 | 65.2 | 66.7 | 65.0 |
| Fine-tuning Strategy | (a) | (b) | (c) | (d) | (e) | (f) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 96 | 0.512 | 0.448 | 0.481 | 0.435 | 0.477 | 0.418 | 0.356 | 0.383 | 0.426 | 0.397 | 0.412 | 0.431 |
| 192 | 0.511 | 0.453 | 0.520 | 0.455 | 0.526 | 0.456 | 0.395 | 0.406 | 0.448 | 0.417 | 0.462 | 0.462 | |
| 336 | 0.610 | 0.512 | 0.537 | 0.484 | 0.584 | 0.497 | 0.417 | 0.424 | 0.478 | 0.439 | 0.489 | 0.479 | |
| 720 | 0.598 | 0.523 | 0.581 | 0.526 | 0.539 | 0.493 | 0.467 | 0.463 | 0.454 | 0.453 | 0.536 | 0.514 | |
| Average | 0.558 | 0.484 | 0.530 | 0.475 | 0.532 | 0.466 | 0.409 | 0.419 | 0.452 | 0.427 | 0.475 | 0.472 | |
| Dataset | LVMs | w/o-LVM | LVM2Attn |
|---|---|---|---|
| UWaveGestureLibrary | 73.4 | 78.6 | 80.1 |
| SpokenArabicDigits | 97.0 | 96.4 | 96.5 |
| Handwriting | 24.3 | 22.4 | 20.7 |
| FaceDetection | 65.0 | 64.1 | 66.2 |
| Model | LVMs | w/o-LVM | LVM2Attn | ||||
|---|---|---|---|---|---|---|---|
| Dataset | Metrics | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 96 | 0.412 | 0.431 | 0.392 | 0.410 | 0.391 | 0.417 |
| 192 | 0.462 | 0.462 | 0.418 | 0.426 | 0.414 | 0.435 | |
| 336 | 0.489 | 0.479 | 0.441 | 0.443 | 0.438 | 0.452 | |
| 720 | 0.536 | 0.514 | 0.441 | 0.465 | 0.469 | 0.485 | |
| Average | 0.475 | 0.472 | 0.423 | 0.436 | 0.428 | 0.447 | |
| Model | MAE | SimMIM | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pre-trained | Enc w/o Dec | Dec w/o Enc | Pre-trained | Enc w/o Dec | Dec w/o Enc | ||||||||
| Dataset | Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 96 | 0.356 | 0.383 | 0.420 | 0.423 | 0.396 | 0.401 | 0.362 | 0.383 | 0.466 | 0.426 | 0.412 | 0.418 |
| 192 | 0.395 | 0.406 | 0.445 | 0.446 | 0.399 | 0.414 | 0.407 | 0.412 | 0.496 | 0.455 | 0.457 | 0.446 | |
| 336 | 0.417 | 0.424 | 0.489 | 0.484 | 0.441 | 0.433 | 0.422 | 0.417 | 0.499 | 0.474 | 0.581 | 0.520 | |
| 720 | 0.467 | 0.463 | 0.582 | 0.543 | 0.426 | 0.451 | 0.462 | 0.455 | 0.505 | 0.481 | 0.564 | 0.526 | |
| Look-back Window | 48 | 96 | 192 | 336 | 720 | 1152 | 1728 | 2304 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 | 96 | 0.376 | 0.395 | 0.373 | 0.390 | 0.364 | 0.383 | 0.356 | 0.383 | 0.347 | 0.375 | 0.347 | 0.376 | 0.344 | 0.376 | 0.373 | 0.402 |
| 192 | 0.440 | 0.431 | 0.424 | 0.418 | 0.411 | 0.412 | 0.395 | 0.406 | 0.385 | 0.405 | 0.384 | 0.402 | 0.391 | 0.408 | 0.399 | 0.417 | |
| 336 | 0.474 | 0.450 | 0.471 | 0.445 | 0.456 | 0.437 | 0.417 | 0.424 | 0.408 | 0.418 | 0.410 | 0.418 | 0.395 | 0.413 | 0.408 | 0.423 | |
| 720 | 0.485 | 0.477 | 0.482 | 0.471 | 0.469 | 0.465 | 0.467 | 0.463 | 0.468 | 0.460 | 0.432 | 0.440 | 0.425 | 0.442 | 0.424 | 0.442 | |
| Average | 0.444 | 0.438 | 0.438 | 0.431 | 0.425 | 0.424 | 0.409 | 0.419 | 0.402 | 0.415 | 0.393 | 0.409 | 0.389 | 0.410 | 0.401 | 0.421 | |
Time Series Forecasting
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.
See all 939 papers