Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a Platonic Representation Theorem: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--good views--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with bad views, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64% against at most 56% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Figures & tables
Figure 2: Kernel mismatch bounds misalignment ( Section 2.2 ). Each point is one randomly generated finite world (8–39 states, random marginal, embedding dimension k from 1 to 5) with a pair of kernels on it: the world’s kernel, and a second kernel of the same marginal that mixes in an unrelated kernel with weight 10−4 to 1 , as if a second modality had partly different positives. For each pair we compute the two spectral embeddings ( Section 2.1 ) and plot their distance after the best rotation (vertical) against the bound of Section 2.2 (horizontal); of the 300 worlds, half are clustered, with small spectral gaps. We clearly observe that the error lies below the bound in every world (dashed: equality), at 18% of it at most.
Figure 3: What counts as a positive, and which observations it connects. Eight captions of two scenes (dots), joined as each objective’s positives join them. (a) Good views join every description of a scene and never another, so the world kernel they induce is the world’s own ( Section 2.3 ). (b) Weak corruptions mostly pair a caption with itself, keeping its wording, and join other captions only where their corruptions overlap, which takes dense data. (c) Strong corruptions also coincide across scenes, so the induced kernel mixes scenes whatever the amount of data. (d) Next-token prediction pairs a prefix with its own continuation, and implicitly contexts followed by the same token ( Section 2.4 ): narrow links within scenes, plus spill through frequent tokens. That is, good views fix the world kernel by design, while bad views and next-token prediction get it from the data .
Figure 4: Positives that join less of a scene need more data ( Section 2.3 ). A synthetic world of five classes of six states (literal data, as in Figure 12 ): alignment of a text JEPA with the image side against the samples per state N and the reach Δ of its bad views (means over three samplings; a state has width 1/6 ). White: the 0.9 contour, and Δ∝logN/N (dashed, one fitted constant), the scale of the largest gap between a state’s samples. Reach large enough to cross classes caps alignment (.95), and good views (top strip) align exactly at any N . We observe that weaker positives need more data for the same alignment .
Text encoder
mutual 10-NN
R@1
R@5
Good-view JEPA
.062 ± .003
.044 ± .002
.133 ± .004
Bad-view JEPA
.047 ± .001
.027 ± .002
.095 ± .003
Next-token
.042 ± .001
.021 ± .002
.077 ± .005
Next-token, pairs
.043 ± .001
.023 ± .003
.083 ± .008
Random init
.010
.001
.011
Chance
.010
.001
.005
Table 1: Good views align across modalities. Image JEPA vs. each text encoder (epoch 199) on 1,000 Flickr test pairs (backbone features; mean ± half-range over three seeds): good-view JEPA (caption views), bad-view JEPA (masked spans), next-token prediction, and next-token prediction on pairs of captions of one image; R@1 and R@5 use a single rotation fitted on validation pairs. We clearly observe that caption views beat masked spans and next-token prediction on every metric , even when next-token prediction is trained on pairs of captions of the same image.
Figure 5: Good views align, bad views and next-token prediction do not. Alignment between a separately trained image JEPA and three text encoders on Flickr, at matched epochs (mean of three seeds, band: range; dashed: chance). All start at chance. We clearly observe that the good-view JEPA (caption views) aligns markedly better than the bad-view JEPA (masked spans) and next-token prediction, which nearly coincide , as predicted by Sections 2.2 and 2.4 .
Figure 6: Alignment lives in the leading spectral directions. Left: after the best rotation, correlation between image and text embeddings along the text embedding’s principal directions (good-view JEPA, at initialization and at epoch 199). Right: variance of the text embedding along the same directions. This suggests that alignment concentrates where the spectrum is well separated , consistent with Section 2.2 , although noise in low-variance directions would produce the same trend.
Figure 7: Trimming is the knob that matters. One change at a time to the good-view text JEPA on Flickr: R@1 with frozen DINOv2 through one rotation, last epoch, three seeds where they exist (faint dots; large dot: mean of the seeds that do not collapse; black: default). Top: the two changes to trimming, with the change of their mean from the default’s; whole captions as views lose a quarter of the alignment. Dashed: the bad-view JEPA and next-token prediction with the default recipe. That is, trimming matters most, and every variant of the good-view JEPA that does not collapse stays far above both baselines ; gradient clipping prevents most collapses.
Figure 8: Good views help on both sides. MSR-VTT, 1,000 test clips. Top: image JEPAs trained on different frames of a clip (good) or crops of one frame (bad), against text JEPAs trained on different captions (good) or spans of one caption (bad); dots are the three seeds, dashed is chance. Bottom: alignment of each with frozen DINOv2, whose own positives are crops. This demonstrates that good views on both sides align best in mutual 10-NN, and that DINOv2 prefers the image JEPA whose positives match its own.
Figure 9: Good views scale. Left and middle: text JEPAs trained on the texts of 2.16M to 940M web images (Recap-DataComp), never on a Flickr or COCO caption, against frozen DINOv2 ViT-B on WIT-1024 (PRH-paper protocol) and on Flickr (R@1 through one rotation fitted on 30k pairs). Solid lines and circles: BERT-S (33M); dashed lines and diamonds: BERT-B (109M), trained with next-token prediction and bad views on the texts of 940M images only, hence single diamonds there. Lines are means over seeds and faint dots the seeds (one seed for the bad-view JEPA at 940M). Shaded: the language models we evaluated (Qwen2.5 0.5–72B, Llama-3-8B and -70B, Mistral-7B; left); grey dashed: CLIP’s text encoder and Qwen2.5-0.5B under the same protocol (middle). Right: image JEPAs trained from scratch (ViT-S on 2.16M and ViT-B on 10M Recap images, ViT-S on ImageNet-1k) and frozen DINOv2, each against the good-view, next-token and bad-view text JEPAs trained on the same texts of 50M web images (WIT; means of three seeds, dots: seeds). We clearly observe that good views beat bad views and next-token prediction at every scale and with every image encoder we compared, and that a 109M good-view JEPA on the texts of 940M images surpasses every language model we evaluated.
Text encoder, vs. frozen
mutual
R@1, rotation fit on
DINOv2 ViT-B/14
10-NN
1k pairs
30k pairs
Qwen2.5-0.5B
.098
.135
.240
BERT-base
.093
.149
.273
bad-view JEPA
.132 ± .001
.133 ± .007
.207 ± .008
next-token
.122 ± .001
.118 ± .002
.213 ± .004
early-stopped
.124 ± .002
.134 ± .008
–
Table 2: Good-view JEPAs agree with DINOv2 more than the pretrained models we tested. Same protocol with a frozen DINOv2 ViT-B/14. Top two rows and bottom two: pretrained models (last layer); the others are ours (BERT-S unless named). All entries use final checkpoints, except the early-stopped row (selected on a loss over the validation and test captions, which can only favor this baseline), with no selection on test data (mean ± half-range over three seeds where shown, seed 0 elsewhere). The last column fits the single map on all 30k Flickr train and validation pairs instead of 1k (–: we kept no epoch-29 checkpoint to embed the training captions with). No network is trained on pairs in either case. For reference, our Flickr image JEPA vs. DINOv2 (image vs. image) reaches mutual 10-NN .101 and R@1 .217. The rows above the first rule are trained on Flickr (and COCO) captions, so this comparison is in-domain for them and out-of-domain for most pretrained models (all-mpnet-base-v2, trained on a billion sentence pairs, saw Flickr and COCO captions); the Recap rows never saw a Flickr or COCO caption. CLIP’s text encoder was trained with 400M images. Bold: best of ours. This demonstrates that good-view JEPAs have the neighborhoods closest to DINOv2’s and, trained on web text alone, match or exceed the retrieval of text encoders trained on pairs.
Figure 10: Captions without training a network on image–caption pairs. Selected captions of COCO test images (first lines) with the decoder of our demo: frozen DINOv2 and DINOv3 image embeddings, one linear map fitted on the 30k Flickr pairs, and a language model fine-tuned on text only (consensus of 32 samples; Table 10 ), with one human caption (grey). That is, frozen encoders joined by one rotation describe scenes, objects and actions , although only the rotation is fitted on image–caption pairs.
WIT, DINOv2-B
WIT
COCO
Flickr
Text encoder
10-NN
CKA
CLIP-H
10-NN
R@1
good-view JEPA
.184
.507
.230
.161
.478
next-token
.111
–
.146
–
.335
bad-view JEPA
.088
–
.116
–
.318
Qwen2.5-72B
.154
.418
.210
.108
–
Llama-3-70B
.151
.380
.206
.113
–
Table 3: At scale, good views beat language models up to 72B parameters. Text encoders against frozen vision models under the PRH protocol: mutual 10-NN and unbiased linear CKA on WIT-1024 with DINOv2 ViT-B, mutual 10-NN on WIT-1024 with CLIP ViT-H/14 and on the 5,000 COCO test pairs with DINOv2 ViT-B, and Flickr R@1 through one rotation fitted on 30k pairs. Our three encoders are BERT-B models (109M) trained with 900M samples from the texts of 940M web images (one seed for the bad-view JEPA); –: not measured. More models in Tables 16 , 17 and 18 . We clearly observe that the good-view JEPA agrees with both vision models more than any other text encoder, while next-token prediction on the same texts falls far behind .
Figure 11: A scaling law of alignment in good views and samples seen. Flickr test R@1 (DINOv2 ViT-B, 30k-pair rotation) of BERT-S text JEPAs trained on the texts of 10.2M web images, against the samples seen. Each point is a run, colored by its fraction q of training items with good views (black: q=0 , bad views only). Circles: the grid the law is fitted on; diamonds: runs predicted before training. Lines: the law at five values of q , dashed beyond the fitted range. We observe that R@1 grows linearly in the logarithm of the samples seen at every fraction of good views, faster with more of them , and that the law predicts unseen runs within seed noise up to twice its range.
Text encoder (runs)
FOSCTTM
C-10
C-100
IN-100
good-view JEPA, 109M (15)
.018
86.0
67.2
64.4
good-view JEPA, 33M (15)
.036
83.2
60.9
57.9
bad-view JEPA, 109M (5)
.026
78.1
55.3
46.7
next-token, 109M (15)
.287
25.5
19.9
20.3
Qwen3-8B, rerun (5)
.018
83.4
61.7
47.6
MPNet, rerun (5)
.043
70.2
45.6
45.4
Table 4: A 109M good-view JEPA beats every pair-free zero-shot accuracy of Schnaus et al. [54] . The protocol and method of Schnaus et al. [54] , unchanged: DINOv2 ViT-B/14 and a text encoder aligned with zero pairs on unpaired halves of COCO train2014, scored on the 40,504 val2014 pairs (FOSCTTM, lower is better) and by zero-shot top-1 on CIFAR-10 and top-5 on CIFAR-100 and ImageNet-100 (%). Means over training seeds times the method’s five seeds (std in Appendix C ). Their best: the best number of their paper for each metric over its 21 vision–language pairs, on these data and in any of its five settings ( paper/data/schnaus2026_reported.csv ); our reruns of their MPNet pair (all-mpnet-base-v2) reproduce their reported .04, 70, 46 and 45. Last row: the ceiling of Procrustes on all training pairs, range over all encoders. Our good-view JEPAs are trained on the texts of 940M and 100M web images. We clearly observe that a 109M good-view JEPA beats every zero-shot accuracy of Schnaus et al. [54] , matches their best FOSCTTM on these data, and matches or significantly beats their 8B language model under identical conditions , while next-token prediction on the same texts fails; only CLIP’s text encoder, trained on 400M image–text pairs, aligns better on FOSCTTM and ImageNet-100.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
caption → caption R@1
mutual 10-NN (projector)
R@1 (projector)
whitened R@1 (projector)
Good-view JEPA
.606
.064
.030
.034
Bad-view JEPA
.381
.044
.018
.022
Next-token prediction
.331
–
–
–
TF-IDF
.442
–
–
–
Random init
.178
–
–
–
Appendix
Table 5: Good views also make better text encoders. Text encoders alone (caption → caption retrieval among the 5,000 Flickr test captions: is the nearest caption one of the four others describing the same image?) and alignment measured on projector outputs (seed 0; –: no trained projector). We clearly observe that the bad-view JEPA learns little beyond word overlap (below TF-IDF), while the good-view JEPA goes well beyond it. The image JEPA reaches 78.5% 20-NN accuracy on Imagenette [ 21 ] (29.1% at initialization).
k=32
k=128
Text encoder (Flickr)
scene-explained
bound
error
scene-explained
bound
error
Good-view JEPA
.804 ± .003
12.6
33.5
.654 ± .005
88.6
157.0
Bad-view JEPA
.655 ± .005
22.1
39.4
.507 ± .003
126.1
171.5
Next-token
.608 ± .001
25.1
40.3
.440 ± .002
143.4
176.6
trained on Recap texts only (seed 0), measured on the same Flickr captions
Good-view JEPA, 50M images
.802
12.7
32.4
.642
91.7
156.8
Appendix
Table 6: Bad views spend their embedding on nuisance. Scene-explained fraction ρ(f)=trCf1/2/k of each text encoder’s top- k whitened embedding (mean ± half-range over three seeds), the lower bound 2k(1−ρ(f)) of Section 2.3 on the best-rotation error against DINOv2 ViT-B, and that error (means over seeds; bounds from the unrounded fractions; uncorrelated embeddings give 2k , i.e., 64 and 256). We clearly observe that the bound holds for every encoder, and the scene-explained fraction ranks the encoders as their alignment does .
Texts of
texts per image
views
WIT (DINOv2-B)
Flickr R@1
scene-explained
2.24M images
alt-text, two LLaVA recaptions
good
.107 (.095–.115)
.339 (.336–.344)
.793
2.24M images
alt-text, text-only paraphrase
good
.060 (.058–.061)
.233 (.230–.235)
.654
2.24M images
two LLaVA recaptions
good
.080 (.076–.082)
.348 (.346–.351)
.778
2.24M images
alt-text
bad
.055 (.054–.056)
.180 (.179–.180)
.606
50M images
alt-text, two LLaVA recaptions
good
.125 (.118–.129)
.376 (.370–.381)
.802
50M images
two LLaVA recaptions
good
.096 (.093–.100)
.386 (.373–.396)
.789
Appendix
Table 7: The views and the data both set the alignment. BERT-S text JEPAs, one recipe per scale, three seeds (mean and range), scored on WIT-1024 with DINOv2 ViT-B and on Flickr through one rotation fitted on 30k pairs; scene-explained: ρ(f) on Flickr captions ( k=32 , seed 0, Table 6 ). The paraphrase is written from the alt-text alone by a text-only language model, so that row involves no image captioner. We observe that with the same bad views, LLaVA recaptions align far better than alt-texts, and with the same texts, good views double alignment ( Section 2.3 ).
Figure 12: Weak views need many data, the kind of data sets the ceiling, good views need neither ( Section 2.3 ). A synthetic world of five classes of six states; the image side is a good-view JEPA whose world kernel links states of one class. The text side observes each state through samples on a line, one block per class, with gaps between blocks (left, literal data) or without (right, mixed data); its bad views join two samples with weight e−d2/2Δ2 (weak Δ=.005 , medium .08 , strong 1 ; a state has width 1/6 ). Alignment is the mean canonical correlation of the two spectral solutions over paired observations ( k=4 ; 1: equal up to a rotation); lines are means over five samplings, bands their range.
Table 8: Cropping matters, the rest barely moves the mean, and clipping prevents most collapses. R@1 against frozen DINOv2 on Flickr test pairs with one rotation fitted on validation pairs (chance .001), last epoch, for three seeds where shown (one otherwise); a run collapses (×) when its last-epoch R@1 falls below .05. Default views: 2 global and 6 local spans of 50–100% and 15–50% of a caption. Mean: over the seeds that do not collapse (–: every seed collapses). ∗ Best over checkpoints on the test pairs, for reference only; the other columns involve no selection. That is, every run that does not collapse stays far above the bad-view JEPA and next-token prediction , and without gradient clipping 3 of the 36 runs with the default SIGReg weight collapse. In stable-pretraining as we ran it, SIGReg is scaled by the number of embeddings it sees, so the batch and view rows also change its effective weight: ×2 for batch 512, ×21 for 2 + 2 views and ×1.5 for 2 + 10 views.
Captioner (Flickr test)
BLEU-4
CIDEr
Oracle: decode the true caption embedding
.261
.628
Image → rotation (30k pairs) → decode
.185
.431
Image → rotation (1,014 pairs) → decode
.173
.396
Image → rotation (300 pairs) → decode
.146
.317
Image → rotation (100 pairs) → decode
.119
.244
+ unpaired refinement (100 pairs)
.138
.294
Appendix
Table 9: Captioning from independently trained encoders. BLEU-4 [ 45 ] , CIDEr [ 62 ] on Flickr test images. The decoder never sees an image; the image encoder never sees text; pairs only fit one linear map. BLEU-4 rewards generic n-grams, so the random map’s fluent but generic captions score above 1,014-pair retrieval on BLEU-4, not on CIDEr. We clearly observe that decoding through the rotation beats retrieving captions through it by a wide margin .
Figure 13: The captioner’s noise and pairs are predicted, not tuned. Flickr test images. Top: CIDEr against the noise the decoder is trained with; dashed, the noise matched to the residual of the rotation (rotation from 1,014 pairs). Bottom: CIDEr against the number of pairs that fit the one linear map, with and without unpaired refinement, against retrieving the nearest caption through the same map; dashed, decoding the true caption embedding. We clearly observe that transfer peaks at the matched noise, and that a few pairs refined on unpaired data match caption retrieval from 1,014 pairs.
Captioner (CIDEr; one rotation from the 30k Flickr pairs)
Flickr
COCO
Small decoder, DINOv2 ViT-B, in-domain text JEPA ( Table 9 )
.431
.668
Qwen2.5-1.5B decoder, web text JEPA, DINOv2 ViT-g (CLS)
.466
.720
+ mean patch token
.480
.756
+ captions as written (case, punctuation)
.479
.765
+ DINOv3 ViT-L, concatenated
.496
.806
+ consensus of 32 samples
.519
.838
Appendix
Table 10: The captioner improves without pairs. Building the captioner up one change at a time: Flickr test (1,000 images) and COCO Karpathy test (5,000 images), beam 3 unless stated; consensus rows choose among the greedy caption and 32 samples. The text JEPA is BERT-S trained on the texts of 100M web images; the decoders are trained on its embeddings of Flickr and COCO training captions only, with noise matched to the alignment ( σ=1.9 ); the image encoders are frozen. The first row is the captioner of Table 9 : on Flickr, the Flickr text JEPA and decoder ( σ=3 ); on COCO, the Flickr and COCO ones ( σ=2 ); it decodes greedily on Flickr and with beam 3 on COCO. Decoding the true caption embedding gives .694 and 1.305 (captions-as-written row), and .743 and 1.369 with the web-pretrained decoder. Only the rotation is fitted on image–caption pairs. Without consensus decoding, the best captioner reaches .496 and .806. This demonstrates that stronger frozen encoders, a pretrained language model and consensus decoding each add to the captioner, from CIDEr .668 to .838 on COCO , while a larger decoder does not.
MSR-VTT test, 2,990 clips
CIDEr
BLEU-4
METEOR
ROUGE-L
Ours: mean of 8 frames, rotation from 30k Flickr pairs, consensus of 32
.246
.166
.206
.444
beam 3
.222
.154
.204
.432
map from 30k Flickr and 200k web pairs (the demo’s)
.268
.186
.232
.471
middle frame instead of the mean of 8
.242
.172
.219
.457
oracle: decode the text embedding of one human caption
.383
.257
.265
.538
CLIP ViT-B/32 encoders (400M image–text pairs), no MSR-VTT text; from Li et al. [33] , Tab. 4
Appendix
Table 11: Captions and retrieval for video, from photos alone. Top: zero-shot captioning of the MSR-VTT test clips (20 references each; CIDEr on a 0–1 scale) by our captioner, whose networks never saw a video, a video–caption pair or MSR-VTT text, and by captioners built on CLIP’s encoders without MSR-VTT text, with one trained on MSR-VTT captions for reference (–: not reported). Bottom: alignment (mutual 10-NN fits nothing) and retrieval through one map on the 1k-A split, against zero-shot CLIP. We observe that a rotation fitted on photos captions and retrieves videos , above captioners built on CLIP that use no MSR-VTT text, though CLIP’s jointly trained encoders retrieve better.
BERT-B, texts of 50M images
BERT-S, 100M
Frozen image encoder
features
mutual 10-NN
R@1
R@5
R@1
R@5
DINOv2 ViT-B/14
CLS
.234
.412
.733
.400
.711
DINOv2 ViT-L/14
CLS
.242
.437
.745
.410
.732
DINOv2 ViT-g/14
CLS
.245
.437
.769
.425
.741
DINOv2 ViT-g/14
CLS + mean patch
.250
.460
.784
.440
.763
DINOv3 ViT-L/16
CLS
.209
.436
.722
.414
.703
Appendix
Table 12: Linear-evaluation features and concatenated encoders align best. Flickr test pairs, text JEPAs trained on Recap texts only (seed 0), each against a frozen self-supervised image encoder through one map with orthonormal rows fitted on the 30k Flickr train and validation pairs (chance: mutual 10-NN .01, R@1 .001). Bottom rows: the CLS + mean patch features of two encoders, each rescaled to unit root-mean-square per coordinate, concatenated. No network is trained on pairs. Notably, with DINOv3 ViT-7B, one rotation retrieves the true caption first for more than half of the test images .
Image JEPA
Text JEPA
mutual 10-NN
R@1
R@5
good views (frames)
good views (captions)
.067 ± .001
.032 ± .006
.099 ± .009
bad views (crops)
good views (captions)
.059 ± .002
.024 ± .002
.105 ± .008
good views (frames)
bad views (spans)
.043 ± .002
.020 ± .005
.063 ± .002
bad views (crops)
bad views (spans)
.039 ± .003
.017 ± .002
.072 ± .009
frozen DINOv2 ViT-B
good views (captions)
.129 ± .002
.085 ± .006
.258 ± .011
frozen DINOv2 ViT-B
bad views (spans)
.084 ± .002
.052 ± .002
.164 ± .006
Appendix
Table 13: Good views help on both sides (numbers behind Figure 8 ). MSR-VTT, 1,000 test clips, final checkpoints, mean ± half-range over three seeds (chance: mutual 10-NN .01, R@1 .001). R@1 and R@5 use one rotation fitted on the 1,990 validation clips. Every seed ranks the four combinations the same way in mutual 10-NN; the image-side effect is small and reverses on R@5.
Text JEPA
texts of
global batch
samples seen
WIT (DINOv2-B)
Flickr R@1
BERT-S
100M images
256
80M
.125 †
.402 †
BERT-B
200M images
256
89M
.125
.447
BERT-B
200M images
256
117M
.126
.447
BERT-B
200M images
256
160M
.126
.451
BERT-B
940M images
2,048
430M
.182
.410
BERT-B
940M images
2,048
580M
.182
.463
Appendix
Table 14: Alignment keeps growing with the texts of 940M images. Checkpoints of our largest good-view JEPAs (seed 0 unless named; † : mean of three seeds at the end of training; ‡ : another seed with twice the effective SIGReg weight, see text; next-token: the same encoder trained to predict the next token; bad views: two masked copies of one text, at the effective SIGReg weight of the first 940M run), scored on WIT-1024 with DINOv2 ViT-B and on Flickr through one rotation fitted on 30k pairs. We observe that WIT alignment jumps between the texts of 200M and 940M images and rises again by 850M samples , while Flickr retrieval rises more slowly. Bold: best per column.
Qwen2.5
Mistral
ours (33M), on the texts of Recap images
ours (109M)
2.16M
10M
50M
100M
940M
Vision
0.5B
1.5B
3B
7B
7B
good / bad / nt
good
good / bad / nt
good
good
DINOv2 ViT-S/14
.092
.101
.110
.110
.126
.099 / .052 / .065
.102
.114 / .062 / .074
.114
.166
DINOv2 ViT-B/14
.102
.108
.125
.118
.140
.108 / .053 / .071
.111
.125 / .063 / .079
.125
.184
DINOv2 ViT-L/14
.114
.123
.139
.137
.157
.111 / .057 / .075
.115
.130 / .068 / .085
.131
.196
DINOv2 ViT-g/14
.115
.124
.141
.133
.157
.114 / .059 / .076
.117
.132 / .070 / .086
.133
.197
Appendix
Table 15: A 109M text JEPA beats language models on the PRH benchmark. PRH protocol of Huh et al. [22] : WIT-1024 pairs, per-layer features, clipping, normalization, maximum over layer pairs. For reference, DINOv2 ViT-B × BERT-base scores .111, all-mpnet-base-v2 .118 and CLIP’s text encoder .124 (.113–.137 across DINOv2 sizes for these two); our Flickr-trained text encoders (good / bad / next-token) score .029–.036 with every DINOv2 size and .023–.024 with our Flickr image JEPA; and our Flickr image JEPA × DINOv2 ViT-B (image vs. image) .241. nt: next-token prediction. Our 33M text JEPAs’ entries are means of three seeds, which differ by at most ± .009 with DINOv2 ViT-B; the 109M column is the mean of three seeds (each above every language model). At least 43 of the 1,024 WIT captions appear among the Recap-DataComp alt-texts; on the other 981 pairs, none of the nine text models we checked with DINOv2 ViT-B (ours, Qwen2.5-3B/7B, Mistral-7B) changes by more than .002. This demonstrates that the 109M good-view JEPA surpasses every language model we evaluated with every vision encoder it was scored with , including our image JEPAs trained from scratch.
WIT-1024
COCO-5k
mutual 10-NN, DINOv2 S / B / L / g
CKA, B
10-NN, B
CKA, B
Qwen2.5-0.5B
.092
.102
.114
.115
.318
.100
.571
Qwen2.5-1.5B
.101
.108
.123
.124
.324
.089
.538
Qwen2.5-3B
.110
.125
.139
.141
.363
.099
.550
Qwen2.5-7B
.110
.118
.137
.133
.353
.101
.535
Qwen2.5-14B
.119
.130
.147
.148
.333
.106
.539
Appendix
Table 16: No language model up to 72B parameters reaches the 109M good-view JEPA. PRH protocol (per-layer features, maximum over layer pairs) on WIT-1024 and on the 5,000 COCO test pairs, for base language models from 0.5B to 72B parameters and our 109M good-view text JEPA trained on the texts of 940M images (mean of three seeds). We clearly observe that scaling next-token prediction by 660 times in parameters does not reach the alignment that good views give .
mutual 10-NN
unbiased CKA
Text encoder (COCO-5k)
DINOv2-B
DINOv2-g
DINOv2-B
DINOv2-g
ours, good, BERT-B, 940M images
.161
.169
.622
.667
ours, good, 100M images
.134
.140
.587
.598
ours, good, 50M images
.126
.135
.578
.576
ours, bad, 50M images
.085
.090
.447
.423
ours, next-token, 50M images
.091
.096
.522
.487
Appendix
Table 17: The ranking holds on COCO. PRH protocol on the 5,000 COCO Karpathy test pairs (first caption), seed 0 for our encoders except the 940M row, the mean of three seeds (each at or above all-mpnet-base-v2); chance mutual 10-NN is .002. We clearly observe that the 109M good-view JEPA aligns best on a second, five times larger benchmark, ahead of language models up to 72B parameters and of text encoders trained on pairs .
ours (33M), texts of 50M images
ours (109M), 940M
Qwen2.5
Qwen2.5
Mistral
Vision encoder
good
bad
next-token
good
0.5B
7B
7B
MAE ViT-B/16 (IN-1k)
.074
.045
.053
.100
.060
.067
.077
MAE ViT-L/16 (IN-1k)
.093
.050
.059
.126
.074
.084
.097
MAE ViT-H/14 (IN-1k)
.092
.050
.058
.125
.071
.084
.096
I-JEPA ViT-H/14 (IN-22k)
.095
.051
.059
.129
.068
.081
.096
DINOv2 ViT-B/14
.129
.066
.079
.191
.102
.118
.140
Appendix
Table 18: Good views win with every vision family. WIT-1024 (PRH protocol) for frozen vision encoders trained by pixel reconstruction (MAE), latent prediction (I-JEPA), crops (DINOv2) or image–text contrast (CLIP’s image encoders, the vision models with which language models align best in Huh et al. [22] ), against our text JEPAs (good / bad / next-token on the texts of 50M images, seed 0; the 109M good-view JEPA on the texts of 940M images, seed 0) and pretrained language models. We clearly observe that good views beat next-token prediction and bad views with every vision encoder, and that the 109M good-view JEPA beats the three language models with each of them. MAE and I-JEPA align alike, both below DINOv2, which saw far more curated images; we draw no conclusion on the vision objective. With CLIP’s image encoders, the 109M good-view JEPA’s three seeds average .215, .219 and .230, against .193, .203 and .210 for the best of all our language models up to 72B (Qwen2.5-72B), and its next-token and bad-view controls on the same texts .137–.146 and .108–.116 ( runs/results/wit_clip.txt ).
Figure 14: One rotation superimposes two separately trained encoders. Top: joint t-SNE of test images after the best rotation (blue) and their captions (orange), at initialization and after training (good-view JEPA); grey lines join 60 random true pairs. Bottom: independent t-SNEs of the image and text embeddings, colored by caption keywords used only for coloring. We clearly observe that after training, images and their captions share one geometry .
Figure 15: Retrieval through one rotation. Image → caption retrieval (good-view JEPA, epoch 199) for five Flickr test images, with the rank of the true caption among the 1,000 above each list ( ✓ would mark it among the top three). We observe that misses are mostly semantically close .
We investigate the Platonic Representation Hypothesis (PRH) through a tripartite statistical framework of representations: signal, bias, and noise. {1) Signal:} We propose that Platonic alignment arises from the universal relationship between objects and attributes, which is encoded linearly in representations according to the Linear Representation Hypothesis (LRH). We provide evidence that LRH helps explain PRH by extracting linear object-attribute features with sparse autoencoders and showing that these sparse representations often exhibit stronger cross-modal alignment than their dense counterparts. {2) Bias:} Models have different implicit biases due to the diverse architectures and training procedures used. We show that this difference can be partially mitigated. Centering and normalization consistently improve cross-model alignment. {3) Noise:} Finite-sample training leads to noise in representations. We provide evidence that representational noise is driven by data scarcity by revealing a strong and consistent positive correlation between word frequency and alignment in LLMs and text embedding models. Synthesizing signal, bias, and noise, we propose a statistical model that refines the Linear Representation Hypothesis and explains further phenomena related to the alignment of representations emerging from diverse modern AI architectures.
Kiril Bangachev, Guy Bresler, Yury Polyanskiy
Department of Electrical Engineering and Computer Science · Massachusetts Institute of Technology
The Platonic Representation Hypothesis posits that neural networks trained on different modalities (e.g., text and images) converge toward a shared representation of reality. If true, this has significant implications for whether modality choice matters at all. In this paper, we show that the evidence for this claim is substantially weaker than subsequent work suggests. The mutual k-nearest-neighbor metric used on 1024 text-image pairs in the original study captures only coarse structure. To keep the alignment from collapsing as one scales up the data, k has to grow proportionally, undercutting the argument for fine-grained representational convergence. The reported increase in alignment with language model strength saturates for recent models. Moreover, the one-to-one text-image pairing favors alignment, while alignment decreases with non-bijective data. We further find that image and text representations indeed share coarse semantic structure, but neither stronger language models nor richer captions yield fine-grained alignment. Thus, multimodal representations share coarse structure without evidence of convergence to a shared representation -- arguably, full representational convergence would require fine-grained alignment.
A. Sophia Koepke, Daniil Zverev, Shiry Ginosar +1
UC Berkeley · Technical University Munich, MCML · University of Tübingen, Tübingen AI Center +1
The Platonic Representation Hypothesis (PRH) holds that increasingly capable models converge toward a common representation. However, recent studies have shown different views on aligning representations via their neighborhoods, spanning trees, and distances. We show that these views can differ because the alignment criteria behind them sit at different points of one lattice, and we formalize this explanation as the Lattice Representation Hypothesis (LRH). Under the LRH, the convergence claimed by the PRH is agreement at the top of the lattice, whereas its evidence comes from neighbor overlap, a criterion near the bottom. The Aristotelian Representation Hypothesis (ARH) restricts convergence to local neighborhoods because calibration keeps the growth of neighbor overlap with model size but largely removes that of centered kernel alignment. We show that this removal can come from calibrating a maximum over layers, so global convergence remains open. Back into Plato's Cave (BPC) infers coarse shared structure from the collapse of neighbor overlap on larger evaluation sets, whereas we trace it at least partly to a fixed number of neighbors covering shorter distances. We show that the spanning-tree overlap used by Structure over Geometry (SOG) mostly repeats one-nearest-neighbor overlap. The LRH also extends to independently trained models, whose training procedure and objective decide the criteria met reliably across runs. We prove this claim for the training procedure in a linear autoencoder and for the objective in a finite contrastive Gibbs model. Experiments on dozens of pretrained vision and language models and thousands of runs of both theoretical models support these conclusions. We hope this work paves the way toward a general theory of representation hypotheses, with a framework to analyze them and compare their evidence. Project page: https://eethanshi.github.io/LRH
Dai Shi, Andi Han, Xiaoyu Li +2
University of Cambridge · University of Sydney · University of New South Wales