Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces

Pratham Yashwante; Rose Yu

cs.AIFeb 22, 2026

Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces

Authors: Pratham Yashwante, Rose Yu

Organizations: Department of Computer Science and Engineering, University of California San Diego, USA.

Abstract

The Platonic Representation Hypothesis posits that learned representations from models trained on different modalities converge to a shared latent structure of the world. However, this hypothesis has largely been examined in vision and language, and it remains unclear whether time series participate in such convergence. We first examine this in a trimodal setting and find that independently pretrained time series, vision, and language encoders exhibit near-orthogonal geometry in the absence of explicit coupling. We then apply post-hoc alignment by training projection heads over frozen encoders using contrastive learning, and analyze the resulting representations with respect to geometry, scaling behavior, and dependence on information density and input modality characteristics. Our investigation reveals that overall alignment in contrastive representation spaces improves with model size, but this alignment is asymmetric: time series align more strongly with visual representations than with text, and images can act as effective intermediaries between time series and language. We further see that richer textual descriptions improve alignment only up to a threshold; training on denser captions does not lead to further improvement. Analogous effects are observed for visual representations. Our findings shed light on considerations for building multimodal systems involving non-conventional data modalities beyond vision and language.

Figures & tables

Figure 1: Trimodal projections of a shared temporal process. A latent process $Z$ gives rise to a numeric time series, a visual line plot, and a textual description, each representing the same signal in values, geometry, and language. Modality-specific encoders $f_{\text{ts}}$ , $f_{\text{img}}$ , and $f_{\text{txt}}$ map inputs into representation spaces.

Figure 2: Mean angular deviation between pretrained cross-modal representations on CaTS shows little inherent alignment.

Figure 3: UMAP visualizations of representations after contrastive training on Flickr (top) and CaTS (bottom).

Figure 4: Trimodal contrastive alignment framework. Frozen pretrained encoders independently map time series, visual plots, and text into their respective unimodal representation spaces. Trainable projection heads transform these representations into a shared embedding space. Alignment is learned via a symmetric contrastive objective applied jointly across all modality pairs (TS–IMG, TS–TXT, IMG–TXT).

Figure 5: Scaling behavior of alignment across 34 trimodal configurations on CaTS as a function of total model size (in millions of parameters). Each subplot reports alignment quality for three modality pairs (TS–IMG, TS–TXT, IMG–TXT). Each point corresponds to a distinct encoder configuration. Dashed lines indicate linear trends with Pearson correlation coefficients reported in the legend.

Figure 6: Effect of text ID on alignment across increasing levels of text density. Markers indicate distinct caption regimes with increasing semantic richness. Scores are averaged across 9 representative model configurations (see Appendix D.2 ).

Figure 7: IMG–TXT alignment across model scales for VL–TS on CaTS. A–I correspond to select configurations (Appendix D.2 ).

	Cosine Margin ↑			Procrustes ↓			Mutual kNN ↑
Pair	OG	High	$\Delta$	OG	High	$\Delta$	OG	High	$\Delta$
TS–IMG	0.72	0.72	0.00	0.47	0.47	0.00	0.13	0.13	0.00
TS–TXT	0.57	0.56	-0.01	0.74	0.74	0.00	0.08	0.07	-0.01
IMG–TXT	0.64	0.64	0.00	0.60	0.60	0.00	0.11	0.10	-0.01

Table 1: Effect of doubling text ID on alignment. We compare original captions (train ID = 417.2) and high-information captions (train ID = 870.2).

\Delta

denotes High ID

-

OG ID. Both models use same samples with their respective ID-specific caption sets. Scores are averaged across configurations listed in Appendix D.2 .

Figure 8: Scaling behavior of alignment on CaTS and MIMIC across 5 representative trimodal configurations (see Appendix D.2 ). We report Procrustes disparity as a function of model size.

Model Configuration	Retrieval Performance (%)
Model Configuration	R@1		R@5		R@10
	CaTS	MIMIC	CaTS	MIMIC	CaTS	MIMIC
Dv2-B + Q-0.6B + Mo-B	1.61	31.31	6.13	61.84	10.75	73.62
Dv2-L + Q-4B + Mo-L	2.25	21.36	8.65	49.48	14.71	62.21
Dv3-7B + Q-8B + Mo-L	6.19	23.70	18.35	51.80	27.12	65.13
SL2-B + Q-0.6B + C-B	3.96	20.33	12.64	45.79	19.85	58.69

Table 2: Cross-modal retrieval performance on MIMIC and CaTS for TS

\leftrightarrow

IMG retrieval. Results are macro-averaged over TS

\rightarrow

IMG and IMG

\rightarrow

TS and reported as percentages. Model acronyms: Dv2/Dv3 = DINOv2/DINOv3, SL = SigLIP2, Q = Qwen, Mo = MOMENT, C = Chronos; B/L denote Base/Large.

Figure 9: Comparison of multimodal alignment between MIMIC (English reports) and PTB-XL (German reports). English reports consistently achieve higher alignment across all modality pairs.

Figure 10: Role of trimodality as a semantic bridge. We compare bimodal and trimodal CL for TS–IMG (top row) and TS–TXT (bottom row). Introducing the image modality consistently improves TS–TXT alignment.

Figure 11: TS–IMG alignment across TRUCE plot variants with increasing visual input richness. Annotated achieves highest scores.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure 12: Example CaTS triplet consisting of a numeric time series, its visual plot, and a reference caption.

Figure 13: KDEs over projected embedding values and caption examples across CaTS variants with increasing ID.

Figure 14: Representative TRUCE visual variants: generic, styled, and annotated plots.

Panel of Figure 15 — Figure 15: Example MIMIC triplet consisting of an ECG time series, its waveform visualization, and a report.

Panel of Figure 16 — Figure 16: Example PTB-XL triplet consisting of an ECG time series, its waveform visualization, and a report.

Lower Bound Models
#	Vision Encoder	Text Encoder	Time Series Encoder
1	DINOv2-Base (86.6M)	T5-Small (60.5M)	Moirai (91M)
2	ViT-Base (86.4M)	T5-Small (60.5M)	MOMENT (125M)
3	DINOv2-Base (86.6M)	T5-Small (60.5M)	Chronos (201M)
4	DINOv2-Base (86.6M)	E5-Base-v2 (109M)	TimesFM (200M)
5	ViT-Base (86.4M)	E5-Base-v2 (109M)	Chronos (201M)

Appendix

Table 3: Triplet model combinations used in our experiments (Grouped by Size Bounds)

Figure 17: Illustration of Procrustes alignment. Embeddings from one modality are rotated to best match the other under an orthogonal transformation. Normalized residual error measures global geometric mismatch.

Figure 18: Illustration of Mutual $k$ NN alignment. Mutual $k$ NN measures local neighborhood agreement across modalities. Points that share neighbors in both spaces contribute to higher scores.

Figure 19: Mean angular deviation between modality pairs across datasets and model configurations. Values remain near $90^{\circ}$ for all pairs, indicating near-orthogonal geometry and a lack of inherent cross-modal convergence in independently pretrained encoders.

Figure 20: Cross-modal cosine similarity matrices for matched samples across datasets. Off-diagonal similarities remain near zero with no separation between matched and mismatched pairs, confirming the absence of latent alignment without explicit coupling.

Epochs	Cosine Margin $\uparrow$	Procrustes $\downarrow$	CKA $\uparrow$	Mutual KNN $\uparrow$
50	0.676	0.565	0.658	0.103
100	0.671	0.577	0.637	0.104

Appendix

Table 4: Epoch ablation on vit_h14 + t5 + moment-l . Metrics are averaged across all modality pairs.

Figure 21: Effect of projection head design on trimodal alignment. Comparison of baseline and improved projection heads across six model configurations. The top row shows cosine similarity margins (higher is better) for TS–IMG, TS–TXT, and IMG–TXT alignment, while the bottom row reports Procrustes disparity (lower is better).

Table 26

Figure 22: Effect of joint encoder scaling on trimodal alignment. Alignment improves with model size across modality pairs, but gains are uneven. When the time series encoder is already saturated (from B to C), additional scaling yields substantial improvements for TS–IMG alignment, while TS–TXT exhibits smaller and less consistent changes across metrics. This indicates that TS–TXT alignment remains more challenging than TS–IMG: scaling the vision encoder produces clear gains for TS–IMG, whereas corresponding improvements to the text encoder lead to only modest or saturating gains for TS–TXT.

Figure 23: Comparison of multimodal alignment using TimesFM-200M and TimesFM-500M time series encoders, with EVA-CLIP (8B) and Mistral (12B) held fixed. (a) Cosine margins for TS–IMG, TS–TXT, and IMG–TXT. (b) Retrieval performance (R@1, R@5, R@10). (c) Procrustes alignment (lower is better).

Figure 24: Alignment comparison between LLM-generated and human-written captions across modality pairs.

Caption Type	Pair	Cosine Margin $\uparrow$	Procrustes $\downarrow$	CKA $\uparrow$
CaTS (original)	TS–TXT	0.538	0.807	0.524
Structured	TS–TXT	0.574	0.754	0.568
CaTS (original)	IMG–TXT	0.558	0.755	0.537
Structured	IMG–TXT	0.588	0.714	0.583

Appendix

Table 7: Explicit semantic grounding improves text alignment at comparable ID. Comparison of original CaTS captions (train ID = 417.2) and structured captions (train ID = 405.8) for a representative trimodal configuration (DINOv2 + Qwen + MOMENT). All encoders and training settings are identical; only the captions differ.

Model	Cosine Margin				Procrustes				Mutual KNN
Model	1-phrase	2-phrase	4-phrase	dense	1-phrase	2-phrase	4-phrase	dense	1-phrase	2-phrase	4-phrase	dense
eva_18b + mistral_12b + moment	0.0630	0.1371	0.2940	0.6631	3.0968	1.9891	1.3428	0.5785	0.0165	0.0321	0.0436	0.1053
eva_8b + mistral_12b + timesfm	0.0747	0.1590	0.3116	0.6795	2.6904	1.8769	1.3067	0.5401	0.0189	0.0301	0.0481	0.1164
dino_7b + gemma_27b + moirai	0.0049	0.1014	0.3423	0.7432	3.3569	2.0266	1.2302	0.4370	0.0095	0.0188	0.0499	0.1451
eva_18b + gemma_27b + chronos	0.0467	0.1386	0.2684	0.6866	2.8463	1.8538	1.3591	0.5180	0.0125	0.0241	0.0419	0.1228
siglip + t5 + timesfm	0.1614	0.2753	0.4229	0.6583	1.6072	1.4018	1.0305	0.5827	0.0168	0.0300	0.0607	0.1025

Appendix

Table 8: TS–TXT alignment metrics across four caption distributions (1-phrase, 2-phrase, 4-phrase, and dense).

Model	Cosine Margin				Procrustes				Mutual KNN
Model	1-phrase	2-phrase	4-phrase	dense	1-phrase	2-phrase	4-phrase	dense	1-phrase	2-phrase	4-phrase	dense
eva_18b + mistral_12b + moment	0.0581	0.1409	0.3132	0.7275	3.0861	1.9440	1.2660	0.4613	0.0177	0.0269	0.0495	0.1180
eva_8b + mistral_12b + timesfm	0.0670	0.1557	0.3209	0.7330	2.7057	1.8688	1.2621	0.4517	0.0181	0.0323	0.0517	0.1165
dino_7b + gemma_27b + moirai	0.0125	0.1085	0.3235	0.7581	3.1842	1.8941	1.1987	0.3874	0.0169	0.0228	0.0561	0.1804
eva_18b + gemma_27b + chronos	0.0564	0.1605	0.3159	0.7318	2.7981	1.7660	1.2222	0.4331	0.0159	0.0276	0.0576	0.1409
siglip + t5 + timesfm	0.1655	0.2888	0.4805	0.7627	1.5824	1.3488	0.8888	0.3822	0.0207	0.0409	0.0924	0.1780

Appendix

Table 9: TXT–IMG alignment metrics across four caption distributions (1-phrase, 2-phrase, 4-phrase, and dense).

Metric	Pair	Pearson $r$			Spearman $\rho$
Metric	Pair	R@1	R@5	R@10	R@1	R@5	R@10
Cosine Similarity	TS–IMG	0.8333	0.8489	0.8544	0.8834	0.8814	0.8762
	TS–TXT	0.3166	0.3691	0.3974	0.3348	0.3600	0.3780
	IMG–TXT	0.6277	0.6718	0.6960	0.6443	0.6753	0.6826
Procrustes Disparity	TS–IMG	-0.8360	-0.8536	-0.8604	-0.9046	-0.8972	-0.8870
	TS–TXT	-0.3951	-0.4512	-0.4821	-0.4654	-0.4865	-0.5007

Appendix

Table 10: Correlation between alignment metrics and cross-modal retrieval performance. Mutual

k

NN overlap shows the strongest and most consistent association with retrieval performance, particularly for TS–IMG pairs, with substantial correlations also observed for IMG–TXT and TS–TXT. Procrustes disparity exhibits strong negative correlations, indicating that improved global geometric alignment (lower disparity) corresponds to higher retrieval accuracy. Cosine similarity and CKA show weaker but still meaningful correlations, especially for harder TS–TXT pairs. Overall, higher alignment quality corresponds to improved cross-modal retrieval performance.

Model Configuration	Procrustes			CKA			Mutual KNN
Model Configuration	TS–IMG	TS–TXT	IMG–TXT	TS–IMG	TS–TXT	IMG–TXT	TS–IMG	TS–TXT	IMG–TXT
Lower Bound Models
dinov2-b + t5_small + moirai-b	0.7683	0.6343	0.7172	0.6502	0.6467	0.5926	0.0415	0.0773	0.0479
vit-b + t5_small + moment-b	0.5516	0.7604	0.6702	0.7326	0.5554	0.6261	0.1017	0.0664	0.0711
dinov2-b + t5_small + chronos-b	0.4965	0.6825	0.6451	0.7499	0.5826	0.6440	0.0969	0.0787	0.0643
dinov2-b + e5-base + timesfm-b	0.6577	0.7648	0.6605	0.6339	0.5423	0.6385	0.0753	0.0632	0.0664

Appendix

Table 11: Geometric Analysis by Modality Pair across All Model Configurations.

Model Configuration	Cosine Similarity Margins			Cross-Modal Retrieval
Model Configuration	TS–IMG	TS–TXT	IMG–TXT	R@1	R@5	R@10	MRR
Lower Bound Models
dinov2-b + t5_small + moirai-b	0.5835	0.6478	0.5902	0.0081	0.0358	0.0689	0.0336
vit-b + t5_small + moment-b	0.6843	0.5692	0.6156	0.0197	0.0763	0.1279	0.0597
dinov2-b + t5_small + chronos-b	0.7082	0.5966	0.6237	0.0239	0.0846	0.1400	0.0660
dinov2-b + e5-base + timesfm-b	0.6273	0.5530	0.6275	0.0122	0.0501	0.0928	0.0433

Appendix

Table 12: Cross-Modal Evaluation: Cosine Similarity Margins and Retrieval Performance

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Apr 20, 2026A. Sophia Koepke, Daniil Zverev, Shiry Ginosar +1Cross-Modal Learning Cross-Modal Representation Learning
UC Berkeley · Technical University Munich, MCML · University of Tübingen, Tübingen AI Center +1
Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language
Aug 5, 2026Xinran Feng, Yi Xie, Chao Zhang +4Cross-Modal Representation Learning Multimodal Pretraining
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute +1
The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?
May 10, 2026Zhaoyang Zhang, Run Shao, Dongyue Wu +4Neural Representation Geometry Cross-Modal Alignment
1Central South University · 4Ant Group · 2Huazhong University of Science and Technology +1

Apr 20, 2026J/K move · Enter open · S save

Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, Alexei A. Efros

UC Berkeley · Technical University Munich, MCML · University of Tübingen, Tübingen AI Center+1

Cross-Modal Learning Cross-Modal Representation Learning Cross-Modal Alignment Representation Alignment Platonic Representation Hypothesis Multimodal Understanding

Open paper AskPDF arXiv

The Platonic Representation Hypothesis posits that neural networks trained on different modalities (e.g., text and images) converge toward a shared representation of reality. If true, this has significant implications for whether modality choice matters at all. In this paper, we show that the evidence for this claim is substantially weaker than subsequent work suggests. The mutual

k

-nearest-neighbor metric used on 1024 text-image pairs in the original study captures only coarse structure. To keep the alignment from collapsing as one scales up the data,

k

has to grow proportionally, undercutting the argument for fine-grained representational convergence. The reported increase in alignment with language model strength saturates for recent models. Moreover, the one-to-one text-image pairing favors alignment, while alignment decreases with non-bijective data. We further find that image and text representations indeed share coarse semantic structure, but neither stronger language models nor richer captions yield fine-grained alignment. Thus, multimodal representations share coarse structure without evidence of convergence to a shared representation -- arguably, full representational convergence would require fine-grained alignment.

Abstract

Cross-Modal Alignment

Explore similar work

Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language

The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?