Vision Transformers Learn Word Meaning from Images Alone

Anders Gjølbye; Sebastian Ray Mason; Stella Frank; Lars Kai Hansen

cs.CVOct 9, 2026

Vision Transformers Learn Word Meaning from Images Alone

Authors: Anders Gjølbye, Sebastian Ray Mason, Stella Frank, Lars Kai Hansen

Organizations: Department of Applied Mathematics and Computer Science Technical University of Denmark

Abstract

Can a vision model learn what words mean from pixels alone? We show that the two largest DINOv3 vision transformers organize words rendered in images by word meaning, despite being pretrained using self-supervision on images only, without paired captions, class labels, or other language supervision. We derive word representations from embedding differences between word images and matched controls, then test whether semantically related words are neighbors within each model's embedding space, while controlling for spelling similarity. We find that brand names in the same industry group together, as do nouns from the same semantic domain. Country-capital pairs exhibit consistent vector offsets, similarly to word embeddings trained on text. These word representations also align with the representations of the visual object categories they name. DINOv3-H and DINOv3-7B are, to our knowledge, the first self-supervised vision models to show substantial semantic structure for rendered words. Comparisons with text and speech encoders reveal shared structure among representations of the same words. Contrastive alignment with either encoder further strengthened DINOv3-H's word-level semantic structure, while leaving the other finetuned vision models largely unchanged. Image-only training can therefore teach vision models not just what words look like, but also how their meanings relate.

Figures & tables

Figure 1: A vision transformer trained on only images ranks the capital first. (a) Three nearest rendered words per query, with cosine similarities between centered word directions shown at left. Bold marks meaning partners or same-industry brands. (b) Country-to-capital offsets in DINOv3-H, illustrating the vector-offset analogy relations studied by Mikolov et al. (2013) . Gray lines indicate centered word directions

\tau

; their common origin is for layout purposes only. The six pairs, hand-picked from 60, illustrate the relation rather than quantify its consistency. See Appendix .

Figure 2: Visual and word directions for tiger . The upper row illustrates the uncentered visual class mean $\hat{\nu}_{\mathrm{tiger}}$ of Eq. ( ), corresponding to the mean final-layer representation over 50 ImageNet-1k validation images for a class. The lower row is the uncentered word difference $\hat{\tau}_{\mathrm{tiger}}$ of Eq. ( ), averaged over 96 backgrounds. The word image $x_{s}\oplus w$ shows word $w$ on background image $x_{s}$ ; its control $x_{s}\oplus\tilde{w}_{s}$ has the same background, with the letters’ opacity mask replaced by a phase-scrambled patch. Stacked card edges mark terms not drawn. Centering over all 1,000 classes gives the visual direction $\nu_{\mathrm{tiger}}$ ; centering over the 200 class names gives the word direction $\tau_{\mathrm{tiger}}$ . The rays show the cosine similarity between these directions, with schematic lengths and angle.

	Neighbor purity		Top-10 accuracy
Model	Nouns	Brands	Capital	Synonym	One-edit word	Plural or base
DINOv3-B	0.194 $\pm$ .012	0.184 $\pm$ .016	0.000 $\pm$ .000	0.000 $\pm$ .000	0.653 $\pm$ .039	0.333 $\pm$ .054
DINOv3-L	0.223 $\pm$ .014	0.235 $\pm$ .023	0.117 $\pm$ .041	0.013 $\pm$ .009	0.820 $\pm$ .031	0.733 $\pm$ .051
DINOv3-H	0.269 $\pm$ .017	0.351 $\pm$ .027	0.600 $\pm$ .063	0.060 $\pm$ .019	0.607 $\pm$ .040	0.813 $\pm$ .045
DINOv3-7B	0.308 $\pm$ .020	0.334 $\pm$ .031	0.600 $\pm$ .063	0.100 $\pm$ .024	0.620 $\pm$ .040	0.853 $\pm$ .041
Web-DINO-1B	0.187 $\pm$ .012	0.167 $\pm$ .016	0.000 $\pm$ .000	0.007 $\pm$ .007	0.547 $\pm$ .041	0.293 $\pm$ .053

Table 1: DINOv3-H and DINOv3-7B are comparable to CLIP-B. Neighbor purity is the fraction of the ten nearest neighbors sharing a noun category or brand industry; top-10 accuracy is the fraction of paired targets retrieved among the ten nearest vocabulary items. Underlining marks a value above the untrained ViT-B by more than twice the standard error of the difference, bold marks column maxima, and small numbers are standard errors over words or pairs ( Appendix ).

Model	Matched cosine similarity	Top-10 accuracy	Median rank	Top-10 accuracy	Median rank
		Word to class		Class to word
DINOv3-B	0.002 $\pm$ .003	0.010 $\pm$ .007	486	0.060 $\pm$ .017	94.5
DINOv3-L	0.025 $\pm$ .004	0.120 $\pm$ .023	335	0.180 $\pm$ .027	68
DINOv3-H	0.234 $\pm$ .014	0.645 $\pm$ .034	1	0.680 $\pm$ .033	1
DINOv3-7B	0.120 $\pm$ .012	0.325 $\pm$ .033	154	0.405 $\pm$ .035	32.5
Web-DINO-1B	0.001 $\pm$ .002	0.020 $\pm$ .010	477	0.045 $\pm$ .015	95

Table 2: Word directions in DINOv3-H and DINOv3-7B align with the classes they name. Matched cosine similarity is the mean cosine similarity of the 200 word directions with the visual directions of the classes they name. Word-to-class ranks the 1,000 visual directions for a word direction; class-to-word ranks the 200 word directions for a named class; top-10 accuracy is the share of queries whose target is among the ten nearest, and median rank is the median of the target’s rank over the queries, so a median rank of one means the target is nearest for at least half of them.

Figure 3: DINOv3-H and DINOv3-7B are most aligned with text encoders. Each cell compares a column model and row encoder on the lexical vocabulary; the image rows, CLIP-B and SigLIP2-B, receive rendered word images. The upper triangle shows mutual 10-NN alignment (shared nearest neighbors; chance 0.006), and the lower triangle shows linear CKA between centered similarity matrices (chance 0.005–0.052). Each measure uses a logarithmic scale, outlines mark row maxima, and every cell exceeds a null that relabels words of equal length ( Appendix ).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

List	Items	Words	Pairs	Example and purpose
Synonym and one-edit comparison	150	450	300	torch–flashlight , torch–touch : meaning and spelling
Synonym and antonym	50	150	100	brave–bold , brave–cowardly : synonymy and opposition
Synonym and apparent negation/base	35	105	70	disappear–vanish , disappear–appear : meaning and similar form
Singular and plural	40	80	40	road–roads : form with related meaning
Country and capital	60	120	60	germany–berlin : factual association
Common nouns	384	384	—	Category membership

Appendix

Table 3: The eight vocabulary lists. Items are triples in the first three lists, pairs in the next two, and individual words or pseudowords in the remaining lists. Each triple defines two pairs sharing an anchor. The Words column includes names and pseudowords. Each appears in one list only. The Pairs column counts pairs before the query direction is reversed.

Figure 4: Word images and control images. Each row shows one font and each column one vertical position band, covering the 24 typography settings. Each cell shows a word image beside its control image. The control uses the same background, typography, and text color, with the letters’ opacity mask replaced by a phase-scrambled copy. The words and pseudowords are examples from the eight lists in Table , with their roles indicated.

(a) Forward
	Meaning					Spelling
Model	Capital	Synonym	Synonym (antonym list)	Antonym	Synonym (negation list)	One-edit word	Plural	Base
DINOv3-B	0.000 $\pm$ .000	0.000 $\pm$ .000	0.000 $\pm$ .000	0.000 $\pm$ .000	0.000 $\pm$ .000	0.653 $\pm$ .039	0.450 $\pm$ .079	0.200 $\pm$ .068
DINOv3-L	0.117 $\pm$ .041	0.013 $\pm$ .009	0.000 $\pm$ .000	0.020 $\pm$ .020	0.029 $\pm$ .028	0.820 $\pm$ .031	0.950 $\pm$ .034	0.486 $\pm$ .084
DINOv3-H	0.600 $\pm$ .063	0.060 $\pm$ .019	0.020 $\pm$ .020	0.000 $\pm$ .000	0.086 $\pm$ .047	0.607 $\pm$ .040	1.000 $\pm$ .000	0.600 $\pm$ .083
DINOv3-7B	0.600 $\pm$ .063	0.100 $\pm$ .024	0.020 $\pm$ .020	0.060 $\pm$ .034	0.086 $\pm$ .047	0.620 $\pm$ .040	1.000 $\pm$ .000	0.686 $\pm$ .078

Appendix

Table 4: Partner retrieval at the final layer. Top-10 accuracy is the fraction of queries whose partner is among the ten nearest candidates in the full vocabulary, excluding the query. Queries are (a) the first word of each pair (anchor, negation, singular noun, or country) and (b) its partner. Small numbers are standard errors; underlining marks values that exceed the untrained ViT-B by more than twice the standard error of the difference. Bold marks each column’s maximum within a panel. Text and speech encoders are excluded from highlighting.

	Synonym against one-edit word			Synonym against antonym	Synonym against base	Top-10 hits for synonyms
Model	Count of 150	One-edit word	Synonym	Count of 50	Count of 35	Concrete	Abstract
DINOv3-B	1	2.32 $\pm$ .04	0.22 $\pm$ .08	31	3	0	0
DINOv3-L	2	3.24 $\pm$ .06	0.34 $\pm$ .09	30	3	3	0
DINOv3-H	14	3.01 $\pm$ .11	0.91 $\pm$ .11	27	2	14 ∗	3
DINOv3-7B	17	2.82 $\pm$ .10	0.88 $\pm$ .11	23	2	17 ∗	5
Web-DINO-1B	5	2.30 $\pm$ .06	0.31 $\pm$ .08	30	8	1	1

Appendix

Table 5: Word-pair comparisons at the final layer. Counts give the number of items in which the synonym is closer to the query than the named competitor. Cosine similarities to the anchor are expressed in standard deviations above the vocabulary-wide pairwise mean, with standard errors over items. Top-10 hits count successful synonym retrieval in either direction among 75 concrete and 75 abstract anchors. Stars mark uncorrected

p<0.05

favoring synonyms (two-sided sign test) or concrete anchors (one-sided Fisher exact test). Underlining marks similarities that exceed the untrained ViT-B by more than twice the standard error of the difference. Bold marks the largest synonym similarity and the largest hit count across both anchor groups. Text and speech encoders are excluded from tests and highlighting.

	Neighbor purity				Nearest-centroid accuracy
	Nouns		Brands		Nouns		Brands
Model	Value	$p$	Value	$p$	Value	$p$	Value	$p$
DINOv3-B	0.194 $\pm$ .012	0.0010	0.184 $\pm$ .016	0.0345	0.214 $\pm$ .030	0.1594	0.219 $\pm$ .042	0.2964
DINOv3-L	0.223 $\pm$ .014	0.0005	0.235 $\pm$ .023	0.0005	0.297 $\pm$ .033	0.0005	0.281 $\pm$ .046	0.0165
DINOv3-H	0.269 $\pm$ .017	0.0005	0.351 $\pm$ .027	0.0005	0.365 $\pm$ .035	0.0005	0.427 $\pm$ .050	0.0005
DINOv3-7B	0.308 $\pm$ .020	0.0005	0.334 $\pm$ .031	0.0005	0.396 $\pm$ .035	0.0005	0.469 $\pm$ .051	0.0005

Appendix

Table 6: Category grouping at the final layer. Neighbor purity is the fraction of a word’s ten nearest neighbors in the noun or brand list that share its category or industry. Nearest-centroid accuracy measures category assignment on held-out words. Permutation tests control for word length ( Appendix ); small numbers are standard errors. Underlining marks values that exceed the untrained ViT-B by more than twice the standard error of the difference. Bold marks column maxima. Text and speech encoders are excluded from tests and highlighting.

	Relative depth
Model	0.10	0.23	0.33	0.44	0.56	0.67	0.77	0.90	1.00
DINOv3-B	.166 $\pm$ .009	.172 $\pm$ .010	.186 $\pm$ .011	.192 $\pm$ .012	.197 $\pm$ .012	.189 $\pm$ .012	.198 $\pm$ .012	.195 $\pm$ .012	.194 $\pm$ .012
DINOv3-L	.185 $\pm$ .012	.180 $\pm$ .012	.176 $\pm$ .010	.183 $\pm$ .011	.181 $\pm$ .011	.198 $\pm$ .012	.193 $\pm$ .012	.207 $\pm$ .014	.223 $\pm$ .014
DINOv3-H	.183 $\pm$ .011	.179 $\pm$ .011	.176 $\pm$ .011	.181 $\pm$ .011	.197 $\pm$ .012	.212 $\pm$ .013	.212 $\pm$ .014	.233 $\pm$ .015	.269 $\pm$ .017
DINOv3-7B	.180 $\pm$ .011	.187 $\pm$ .010	.181 $\pm$ .012	.199 $\pm$ .012	.209 $\pm$ .013	.219 $\pm$ .015	.233 $\pm$ .015	.285 $\pm$ .018	.308 $\pm$ .020
Web-DINO-1B	.178 $\pm$ .010	.179 $\pm$ .010	.176 $\pm$ .011	.189 $\pm$ .012	.197 $\pm$ .012	.198 $\pm$ .013	.203 $\pm$ .013	.200 $\pm$ .013	.187 $\pm$ .012

Appendix

Table 7: Noun neighbor purity (a) and country-to-capital top-10 accuracy (b) across model depth. Relative depth is the layer index divided by the number of transformer layers; column labels give the mean across models at each sampled depth. The input embedding layer is excluded. Small numbers are standard errors. Underlining marks values that exceed the untrained ViT-B at the corresponding depth by more than twice the standard error of the difference. Bold marks each model’s maximum across depths when it exceeds chance.

Model	All images	Null mean	$p$	No detected text
	Matched cosine similarity
DINOv3-B	0.002 $\pm$ .003	-0.0001	0.237	0.002 $\pm$ .003
DINOv3-L	0.025 $\pm$ .004	-0.0003	$<$ 0.001	0.023 $\pm$ .004
DINOv3-H	0.234 $\pm$ .014	-0.0004	$<$ 0.001	0.230 $\pm$ .015
DINOv3-7B	0.120 $\pm$ .012	-0.0045	$<$ 0.001	0.116 $\pm$ .013
Web-DINO-1B	0.001 $\pm$ .002	-0.0001	0.306	0.001 $\pm$ .002

Appendix

Table 8: Word-class alignment at the final layer over the 200 class names. All images is the matched cosine similarity. Null mean is the mean over 2,000 random reassignments of classes to words, and

p

is the one-sided permutation probability. No detected text is the same measure over the 195 names whose classes keep at least 32 images with no detected text, with visual directions rebuilt from those images. Small numbers are standard errors over words. Underlining marks a value in the All images and No detected text columns above the untrained ViT-B by more than twice the standard error of the difference, and bold marks the largest value in those columns.

Model	Top-10 accuracy	Median rank	No detected text	Top-10 accuracy	Median rank	No detected text
	Word-to-class retrieval, 1,000 classes			Class-to-word retrieval, 200 class names
DINOv3-B	0.010 $\pm$ .007	486	0.010 $\pm$ .007	0.060 $\pm$ .017	94.5	0.067 $\pm$ .018
DINOv3-L	0.120 $\pm$ .023	335	0.103 $\pm$ .022	0.180 $\pm$ .027	68	0.179 $\pm$ .027
DINOv3-H	0.645 $\pm$ .034	1	0.651 $\pm$ .034	0.680 $\pm$ .033	1	0.677 $\pm$ .033
DINOv3-7B	0.325 $\pm$ .033	154	0.328 $\pm$ .034	0.405 $\pm$ .035	32.5	0.390 $\pm$ .035
Web-DINO-1B	0.020 $\pm$ .010	477	0.015 $\pm$ .009	0.045 $\pm$ .015	95	0.041 $\pm$ .014

Appendix

Table 9: Retrieval in both directions at the final layer. Word to class ranks the 1,000 visual directions for each word direction, and class to word ranks the 200 word directions for each named class. Top-10 accuracy is the share of queries whose target is among the ten nearest, and median rank is taken over queries. No detected text gives top-10 accuracy with visual directions rebuilt from images with no detected text, over 195 names and 838 classes. Small numbers are standard errors over queries. Underlining marks an accuracy above the untrained ViT-B by more than twice the standard error of the difference, and bold marks the largest accuracy and the smallest median rank in each column.

	Relative depth
Model	0.10	0.23	0.33	0.44	0.56	0.67	0.77	0.90	1.00
DINOv3-B	-0.019	0.011	0.022	0.018	-0.003	0.012	0.007	-0.002	0.002
DINOv3-L	0.008	0.007	0.006	-0.002	0.003	0.002	0.012	0.013	0.025
DINOv3-H	0.016	0.029	-0.005	-0.003	0.000	0.005	0.013	0.041	0.234
DINOv3-7B	0.007	0.027	0.022	0.006	0.002	0.003	0.003	-0.006	0.120
Web-DINO-1B	0.014	0.023	0.020	0.007	0.005	0.007	0.003	0.003	0.001

Appendix

Table 10: Matched cosine similarity (a) and word-to-class top-10 accuracy (b) at nine depths over the 200 class names. Each column is one depth, labeled by the mean over models of the layer index divided by the layer count, and a column’s depths differ across models by at most 0.042. The last two columns are one layer apart in the 12-layer models, three in the 24-layer models, and four in the 32- and 40-layer models. Chance is zero in (a) and 0.010 in (b). This table prints no standard errors, so no value is underlined, and bold marks each model’s largest value across depths.

	Narrow clue
Broad clue	sound	picture	text	motion
input	microphone	camera	keyboard	sensor
output	speaker	projector	printer	motor

Appendix

Table 11: Example clue grid. Each row and column supplies one clue, and each cell contains a target word. For microphone (yellow), the broad-clue comparison uses speaker (pink), and the narrow-clue comparison uses the three other input devices (blue). Gray targets share neither clue and enter only the comparison with all seven alternatives.

Rendering	Broad clue	Narrow clue	Difference vector
Both intact	input	sound	$\hat{\tau}_{\mathrm{both}}$ , used in both clue comparisons
Broad intact	input		$\hat{\tau}_{\mathrm{broad}}$ , baseline for the narrow-clue gain
Narrow intact		sound	$\hat{\tau}_{\mathrm{narrow}}$ , baseline for the broad-clue gain
Neither intact			The control subtracted from the three difference vectors above
Target alone	microphone		$\hat{\tau}_{\mathrm{target}}$ , against its own control image

Appendix

Table 12: Renderings for the clue pair input and sound and its target, microphone . Gray bars indicate phase-randomized letters. Difference vectors are defined in ( ).

Figure 5: Contribution of each clue to target-word alignment. Points show clue gains, with 95% bootstrap intervals over the twelve grids. The dashed line marks zero gain. Colored points identify the four models whose margin against all seven alternatives has a corrected

p<0.10

; the untrained ViT-B is shown in darker gray. (a) Broad-clue gain: both clues compared with the narrow clue alone. (b) Narrow-clue gain: both clues compared with the broad clue alone.

Model	Checkpoint or configuration identifier	Layers	Width
DINOv3-B	facebook/dinov3-vitb16-pretrain-lvd1689m	12	768
DINOv3-L	facebook/dinov3-vitl16-pretrain-lvd1689m	24	1024
DINOv3-H	facebook/dinov3-vith16plus-pretrain-lvd1689m	32	1280
DINOv3-7B	facebook/dinov3-vit7b16-pretrain-lvd1689m	40	4096
Web-DINO-1B	facebook/webssl-dino1b-full2b-224	40	1536
DINOv2-B	facebook/dinov2-base	12	768

Appendix

Table 13: The fourteen models and, under the rule, the three text encoders and the speech encoder, with the checkpoint identifier, layer count, and width of each. The untrained ViT-B takes only its architecture from the listed configuration, and its weights are randomly initialized. A speech vector concatenates four relative-position bins of the listed width, so it has 4,096 dimensions.

#	Template	#	Template
1	w	13	They wrote w .
2	“ w ”	14	Someone said w .
3	the word “ w ”	15	She read w out.
4	The word is w .	16	He typed w quickly.
5	Read w aloud.	17	I heard w again.
6	Spell w .	18	w , and then w .

Appendix

Table 14: The 24 carrier templates for the text encoders, shown in their first variant with w standing for the word, name, or pseudoword. Each template has four variants, for 96 carriers per word.

Figure 6: Finetuning raises noun purity and synonym retrieval in DINOv3-H. Open circles show scores before finetuning; filled circles show scores after training with written (upper, dark) or spoken (lower, light) captions. Panels show (a) noun neighbor purity, (b) synonym top-10 accuracy, and (c) agreement between word directions estimated from two sets of backgrounds (split-half cosine similarity). Scores use four word images, averaged over five randomly sampled typography settings. Gray vertical lines show scores before finetuning using all 96 word images. The dashed line marks chance noun purity.

	SPEECH-COCO
Model	written	speech
DINOv3-B	1,199	1,308
DINOv3-L	763 ∗	1,199
DINOv3-H	436 ∗	981 ∗
Web-DINO-1B	218 ∗	545 ∗
DINOv2-B	1,199	1,090

Appendix

Table 15: The step of the checkpoint kept for each run. An asterisk marks an early stop.

Explore similar work

Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning
Sep 23, 2026Basavaraj Sunagad, Artur Jesslen, Adam KortylewskiMulti-View Learning Self-Distillation
CISPA Helmholtz Center for Information Security · University of Freiburg
Using OCR Heads to Verbalize Image Semantics
Sep 16, 2026Sheridan Feucht, Benno Krojer, Sarah Wang +3VLM Interpretability Mechanistic Interpretability
Northeastern University · Independent
Connecting Speech to Words through Images
Jun 15, 2026Gabriel Pirlogeanu, Dan Oneata, Horia Cucu +1Multimodal Grounding Unsupervised Learning
Speech and Dialogue Research Laboratory · POLITEHNICA Bucharest, Romania · Electrical and Electronic Engineering +1

Sep 23, 2026J/K move · Enter open · S save

Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

Basavaraj Sunagad, Artur Jesslen, Adam Kortylewski

CISPA Helmholtz Center for Information Security · University of Freiburg

Multi-View Learning Self-Distillation Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Self-supervised vision transformers trained with DINO-style objectives exhibit striking emergent semantic representation quality across visual tasks, yet the mechanisms underlying this behavior remain unclear. We present a systematic empirical dissection of the DINO family and show that semantic representations arise primarily from enforcing consistency between geometrically distinct global views of the same image instance. This instance-specific global alignment acts as the semantic anchor of DINO-style learning. Across controlled retraining experiments evaluated on semantic correspondence and a diverse suite of 2D and 3D downstream tasks, we find that patch-level masking objectives enhance semantics only when trained jointly with this global alignment, indicating that the iBOT objective refines and densifies existing semantic structure rather than creating it independently. In contrast, local-to-global view alignment does not substantially improve semantic qualities at fixed compute beyond a purely global alignment. Beyond training design, we revisit how semantic representation quality should be evaluated: while classification accuracy is the standard validation score, semantic correspondence provides a complementary axis that more reliably predicts downstream task performance. Together, these findings provide a functional decomposition of DINO-style learning and represent an important step toward understanding how semantic representations emerge in self-supervised vision models.

Abstract

Explore similar work

Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

Using OCR Heads to Verbalize Image Semantics

Connecting Speech to Words through Images