The Platonic Representation Hypothesis posits that learned representations from models trained on different modalities converge to a shared latent structure of the world. However, this hypothesis has largely been examined in vision and language, and it remains unclear whether time series participate in such convergence. We first examine this in a trimodal setting and find that independently pretrained time series, vision, and language encoders exhibit near-orthogonal geometry in the absence of explicit coupling. We then apply post-hoc alignment by training projection heads over frozen encoders using contrastive learning, and analyze the resulting representations with respect to geometry, scaling behavior, and dependence on information density and input modality characteristics. Our investigation reveals that overall alignment in contrastive representation spaces improves with model size, but this alignment is asymmetric: time series align more strongly with visual representations than with text, and images can act as effective intermediaries between time series and language. We further see that richer textual descriptions improve alignment only up to a threshold; training on denser captions does not lead to further improvement. Analogous effects are observed for visual representations. Our findings shed light on considerations for building multimodal systems involving non-conventional data modalities beyond vision and language.
Figures & tables
Figure 1: Trimodal projections of a shared temporal process. A latent process Z gives rise to a numeric time series, a visual line plot, and a textual description, each representing the same signal in values, geometry, and language. Modality-specific encoders fts , fimg , and ftxt map inputs into representation spaces.
Figure 2: Mean angular deviation between pretrained cross-modal representations on CaTS shows little inherent alignment.
Figure 3: UMAP visualizations of representations after contrastive training on Flickr (top) and CaTS (bottom).
Figure 4: Trimodal contrastive alignment framework. Frozen pretrained encoders independently map time series, visual plots, and text into their respective unimodal representation spaces. Trainable projection heads transform these representations into a shared embedding space. Alignment is learned via a symmetric contrastive objective applied jointly across all modality pairs (TS–IMG, TS–TXT, IMG–TXT).
Figure 5: Scaling behavior of alignment across 34 trimodal configurations on CaTS as a function of total model size (in millions of parameters). Each subplot reports alignment quality for three modality pairs (TS–IMG, TS–TXT, IMG–TXT). Each point corresponds to a distinct encoder configuration. Dashed lines indicate linear trends with Pearson correlation coefficients reported in the legend.
Figure 6: Effect of text ID on alignment across increasing levels of text density. Markers indicate distinct caption regimes with increasing semantic richness. Scores are averaged across 9 representative model configurations (see Appendix D.2 ).
Figure 7: IMG–TXT alignment across model scales for VL–TS on CaTS. A–I correspond to select configurations (Appendix D.2 ).
Cosine Margin ↑
Procrustes ↓
Mutual kNN ↑
Pair
OG
High
Δ
OG
High
Δ
OG
High
Δ
TS–IMG
0.72
0.72
0.00
0.47
0.47
0.00
0.13
0.13
0.00
TS–TXT
0.57
0.56
-0.01
0.74
0.74
0.00
0.08
0.07
-0.01
IMG–TXT
0.64
0.64
0.00
0.60
0.60
0.00
0.11
0.10
-0.01
Table 1: Effect of doubling text ID on alignment. We compare original captions (train ID = 417.2) and high-information captions (train ID = 870.2). Δ denotes High ID − OG ID. Both models use same samples with their respective ID-specific caption sets. Scores are averaged across configurations listed in Appendix D.2 .
Figure 8: Scaling behavior of alignment on CaTS and MIMIC across 5 representative trimodal configurations (see Appendix D.2 ). We report Procrustes disparity as a function of model size.
Model Configuration
Retrieval Performance (%)
R@1
R@5
R@10
CaTS
MIMIC
CaTS
MIMIC
CaTS
MIMIC
Dv2-B + Q-0.6B + Mo-B
1.61
31.31
6.13
61.84
10.75
73.62
Dv2-L + Q-4B + Mo-L
2.25
21.36
8.65
49.48
14.71
62.21
Dv3-7B + Q-8B + Mo-L
6.19
23.70
18.35
51.80
27.12
65.13
SL2-B + Q-0.6B + C-B
3.96
20.33
12.64
45.79
19.85
58.69
Table 2: Cross-modal retrieval performance on MIMIC and CaTS for TS ↔ IMG retrieval. Results are macro-averaged over TS → IMG and IMG → TS and reported as percentages. Model acronyms: Dv2/Dv3 = DINOv2/DINOv3, SL = SigLIP2, Q = Qwen, Mo = MOMENT, C = Chronos; B/L denote Base/Large.
Figure 9: Comparison of multimodal alignment between MIMIC (English reports) and PTB-XL (German reports). English reports consistently achieve higher alignment across all modality pairs.
Figure 10: Role of trimodality as a semantic bridge. We compare bimodal and trimodal CL for TS–IMG (top row) and TS–TXT (bottom row). Introducing the image modality consistently improves TS–TXT alignment.
Figure 11: TS–IMG alignment across TRUCE plot variants with increasing visual input richness. Annotated achieves highest scores.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Example CaTS triplet consisting of a numeric time series, its visual plot, and a reference caption.
Figure 13: KDEs over projected embedding values and caption examples across CaTS variants with increasing ID.
Figure 15: Example MIMIC triplet consisting of an ECG time series, its waveform visualization, and a report.
Figure 16: Example PTB-XL triplet consisting of an ECG time series, its waveform visualization, and a report.
#
Vision Encoder
Text Encoder
Time Series Encoder
Lower Bound Models
1
DINOv2-Base (86.6M)
T5-Small (60.5M)
Moirai (91M)
2
ViT-Base (86.4M)
T5-Small (60.5M)
MOMENT (125M)
3
DINOv2-Base (86.6M)
T5-Small (60.5M)
Chronos (201M)
4
DINOv2-Base (86.6M)
E5-Base-v2 (109M)
TimesFM (200M)
5
ViT-Base (86.4M)
E5-Base-v2 (109M)
Chronos (201M)
Appendix
Table 3: Triplet model combinations used in our experiments (Grouped by Size Bounds)
Figure 17: Illustration of Procrustes alignment. Embeddings from one modality are rotated to best match the other under an orthogonal transformation. Normalized residual error measures global geometric mismatch.
Figure 18: Illustration of Mutual k NN alignment. Mutual k NN measures local neighborhood agreement across modalities. Points that share neighbors in both spaces contribute to higher scores.
Figure 19: Mean angular deviation between modality pairs across datasets and model configurations. Values remain near 90∘ for all pairs, indicating near-orthogonal geometry and a lack of inherent cross-modal convergence in independently pretrained encoders.
Figure 20: Cross-modal cosine similarity matrices for matched samples across datasets. Off-diagonal similarities remain near zero with no separation between matched and mismatched pairs, confirming the absence of latent alignment without explicit coupling.
Epochs
Cosine Margin ↑
Procrustes ↓
CKA ↑
Mutual KNN ↑
50
0.676
0.565
0.658
0.103
100
0.671
0.577
0.637
0.104
Appendix
Table 4: Epoch ablation on vit_h14 + t5 + moment-l . Metrics are averaged across all modality pairs.
Figure 21: Effect of projection head design on trimodal alignment. Comparison of baseline and improved projection heads across six model configurations. The top row shows cosine similarity margins (higher is better) for TS–IMG, TS–TXT, and IMG–TXT alignment, while the bottom row reports Procrustes disparity (lower is better).
Table 26
Figure 22: Effect of joint encoder scaling on trimodal alignment. Alignment improves with model size across modality pairs, but gains are uneven. When the time series encoder is already saturated (from B to C), additional scaling yields substantial improvements for TS–IMG alignment, while TS–TXT exhibits smaller and less consistent changes across metrics. This indicates that TS–TXT alignment remains more challenging than TS–IMG: scaling the vision encoder produces clear gains for TS–IMG, whereas corresponding improvements to the text encoder lead to only modest or saturating gains for TS–TXT.
Figure 23: Comparison of multimodal alignment using TimesFM-200M and TimesFM-500M time series encoders, with EVA-CLIP (8B) and Mistral (12B) held fixed. (a) Cosine margins for TS–IMG, TS–TXT, and IMG–TXT. (b) Retrieval performance (R@1, R@5, R@10). (c) Procrustes alignment (lower is better).
Figure 24: Alignment comparison between LLM-generated and human-written captions across modality pairs.
Caption Type
Pair
Cosine Margin ↑
Procrustes ↓
CKA ↑
CaTS (original)
TS–TXT
0.538
0.807
0.524
Structured
TS–TXT
0.574
0.754
0.568
CaTS (original)
IMG–TXT
0.558
0.755
0.537
Structured
IMG–TXT
0.588
0.714
0.583
Appendix
Table 7: Explicit semantic grounding improves text alignment at comparable ID. Comparison of original CaTS captions (train ID = 417.2) and structured captions (train ID = 405.8) for a representative trimodal configuration (DINOv2 + Qwen + MOMENT). All encoders and training settings are identical; only the captions differ.
Model
Cosine Margin
Procrustes
Mutual KNN
1-phrase
2-phrase
4-phrase
dense
1-phrase
2-phrase
4-phrase
dense
1-phrase
2-phrase
4-phrase
dense
eva_18b + mistral_12b + moment
0.0630
0.1371
0.2940
0.6631
3.0968
1.9891
1.3428
0.5785
0.0165
0.0321
0.0436
0.1053
eva_8b + mistral_12b + timesfm
0.0747
0.1590
0.3116
0.6795
2.6904
1.8769
1.3067
0.5401
0.0189
0.0301
0.0481
0.1164
dino_7b + gemma_27b + moirai
0.0049
0.1014
0.3423
0.7432
3.3569
2.0266
1.2302
0.4370
0.0095
0.0188
0.0499
0.1451
eva_18b + gemma_27b + chronos
0.0467
0.1386
0.2684
0.6866
2.8463
1.8538
1.3591
0.5180
0.0125
0.0241
0.0419
0.1228
siglip + t5 + timesfm
0.1614
0.2753
0.4229
0.6583
1.6072
1.4018
1.0305
0.5827
0.0168
0.0300
0.0607
0.1025
Appendix
Table 8: TS–TXT alignment metrics across four caption distributions (1-phrase, 2-phrase, 4-phrase, and dense).
Model
Cosine Margin
Procrustes
Mutual KNN
1-phrase
2-phrase
4-phrase
dense
1-phrase
2-phrase
4-phrase
dense
1-phrase
2-phrase
4-phrase
dense
eva_18b + mistral_12b + moment
0.0581
0.1409
0.3132
0.7275
3.0861
1.9440
1.2660
0.4613
0.0177
0.0269
0.0495
0.1180
eva_8b + mistral_12b + timesfm
0.0670
0.1557
0.3209
0.7330
2.7057
1.8688
1.2621
0.4517
0.0181
0.0323
0.0517
0.1165
dino_7b + gemma_27b + moirai
0.0125
0.1085
0.3235
0.7581
3.1842
1.8941
1.1987
0.3874
0.0169
0.0228
0.0561
0.1804
eva_18b + gemma_27b + chronos
0.0564
0.1605
0.3159
0.7318
2.7981
1.7660
1.2222
0.4331
0.0159
0.0276
0.0576
0.1409
siglip + t5 + timesfm
0.1655
0.2888
0.4805
0.7627
1.5824
1.3488
0.8888
0.3822
0.0207
0.0409
0.0924
0.1780
Appendix
Table 9: TXT–IMG alignment metrics across four caption distributions (1-phrase, 2-phrase, 4-phrase, and dense).
Metric
Pair
Pearson r
Spearman ρ
R@1
R@5
R@10
R@1
R@5
R@10
Cosine Similarity
TS–IMG
0.8333
0.8489
0.8544
0.8834
0.8814
0.8762
TS–TXT
0.3166
0.3691
0.3974
0.3348
0.3600
0.3780
IMG–TXT
0.6277
0.6718
0.6960
0.6443
0.6753
0.6826
Procrustes Disparity
TS–IMG
-0.8360
-0.8536
-0.8604
-0.9046
-0.8972
-0.8870
TS–TXT
-0.3951
-0.4512
-0.4821
-0.4654
-0.4865
-0.5007
Appendix
Table 10: Correlation between alignment metrics and cross-modal retrieval performance. Mutual k NN overlap shows the strongest and most consistent association with retrieval performance, particularly for TS–IMG pairs, with substantial correlations also observed for IMG–TXT and TS–TXT. Procrustes disparity exhibits strong negative correlations, indicating that improved global geometric alignment (lower disparity) corresponds to higher retrieval accuracy. Cosine similarity and CKA show weaker but still meaningful correlations, especially for harder TS–TXT pairs. Overall, higher alignment quality corresponds to improved cross-modal retrieval performance.
Model Configuration
Procrustes
CKA
Mutual KNN
TS–IMG
TS–TXT
IMG–TXT
TS–IMG
TS–TXT
IMG–TXT
TS–IMG
TS–TXT
IMG–TXT
Lower Bound Models
dinov2-b + t5_small + moirai-b
0.7683
0.6343
0.7172
0.6502
0.6467
0.5926
0.0415
0.0773
0.0479
vit-b + t5_small + moment-b
0.5516
0.7604
0.6702
0.7326
0.5554
0.6261
0.1017
0.0664
0.0711
dinov2-b + t5_small + chronos-b
0.4965
0.6825
0.6451
0.7499
0.5826
0.6440
0.0969
0.0787
0.0643
dinov2-b + e5-base + timesfm-b
0.6577
0.7648
0.6605
0.6339
0.5423
0.6385
0.0753
0.0632
0.0664
Appendix
Table 11: Geometric Analysis by Modality Pair across All Model Configurations.
Model Configuration
Cosine Similarity Margins
Cross-Modal Retrieval
TS–IMG
TS–TXT
IMG–TXT
R@1
R@5
R@10
MRR
Lower Bound Models
dinov2-b + t5_small + moirai-b
0.5835
0.6478
0.5902
0.0081
0.0358
0.0689
0.0336
vit-b + t5_small + moment-b
0.6843
0.5692
0.6156
0.0197
0.0763
0.1279
0.0597
dinov2-b + t5_small + chronos-b
0.7082
0.5966
0.6237
0.0239
0.0846
0.1400
0.0660
dinov2-b + e5-base + timesfm-b
0.6273
0.5530
0.6275
0.0122
0.0501
0.0928
0.0433
Appendix
Table 12: Cross-Modal Evaluation: Cosine Similarity Margins and Retrieval Performance
The Platonic Representation Hypothesis posits that neural networks trained on different modalities (e.g., text and images) converge toward a shared representation of reality. If true, this has significant implications for whether modality choice matters at all. In this paper, we show that the evidence for this claim is substantially weaker than subsequent work suggests. The mutual k-nearest-neighbor metric used on 1024 text-image pairs in the original study captures only coarse structure. To keep the alignment from collapsing as one scales up the data, k has to grow proportionally, undercutting the argument for fine-grained representational convergence. The reported increase in alignment with language model strength saturates for recent models. Moreover, the one-to-one text-image pairing favors alignment, while alignment decreases with non-bijective data. We further find that image and text representations indeed share coarse semantic structure, but neither stronger language models nor richer captions yield fine-grained alignment. Thus, multimodal representations share coarse structure without evidence of convergence to a shared representation -- arguably, full representational convergence would require fine-grained alignment.
A. Sophia Koepke, Daniil Zverev, Shiry Ginosar +1
UC Berkeley · Technical University Munich, MCML · University of Tübingen, Tübingen AI Center +1
Training multimodal models to align time series with language runs into a self-supervision trap. The usual recipe asks an LLM to read a series and write a description, so label quality is capped by the perceptual skill the model is supposed to learn. The data can never teach more than the labeler already knows. A second gap makes this worse: most datasets use a single variable, but the patterns that matter (cross-channel correlation, lead-lag structure, co-occurring anomalies) appear only with several variables, right where the labeling LLM's limits are most exposed. These two problems create a trilemma: existing methods are reliable, realistic, or scalable, but none achieves all three. We resolve this by decoupling perception from description. Deterministic code computes a set of statistics from real, open-source multivariate series; the LLM verbalizes those precomputed facts. Perception, which LLMs do poorly, is handled by computation, while the LLM handles expression. This produces CGTime, our 4B-parameter computation-grounded time-series-language model. CGTime outperforms far larger general-purpose models on multivariate understanding tasks: it attains the best multivariate fact score on our held-out benchmark (0.283 vs. 0.173 for GPT-4o-mini and 0.203 for GPT-5.4-nano), a gap that survives Holm-corrected paired significance tests against every baseline. It also states verifiable numerical facts in generated captions more accurately and covers a broader range of statistical properties.
Xinran Feng, Yi Xie, Chao Zhang +4
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute +1
Understanding why independently trained neural networks from different modalities converge toward shared representations, and where this convergence leads, remains an open question in representation learning. All existing evidence relies on symmetric similarity measures, which can detect convergence but are structurally blind to its direction. We introduce directional convergence analysis using cycle-kNN, an asymmetric alignment measure, applied across dozens of independently trained unimodal models spanning point clouds, vision, and language. We uncover a consistent directional asymmetry: non-language modalities move toward the neighborhood structure of language significantly more than the reverse, and this pattern holds across all model families and scales--yet is entirely invisible to symmetric measures. Mechanistic analysis traces the directionality to feature density asymmetry, whereby language representations occupy the most compact regions of representational space. The Information Bottleneck framework provides a principled interpretation: optimization under compression drives representations toward discrete, compositional structures characteristic of language. We formalize this as the Wittgensteinian Representation Hypothesis: the semantic structure of language is the asymptotic attractor of multimodal representation convergence.
Zhaoyang Zhang, Run Shao, Dongyue Wu +4
1Central South University · 4Ant Group · 2Huazhong University of Science and Technology +1