Contrastive vision-language models such as CLIP are trained to align images and text, not to represent taxonomies. Yet empirical studies show that CLIP embeddings exhibit hierarchical structure aligned with the hierarchy of semantic concepts. A theoretical characterization of why such hierarchy emerges from a non-hierarchical objective remains open. In this paper, we formalize this phenomenon as Hierarchical Angular Separation (HAS), a geometric property whereby each non-leaf concept embedding is more similar to its descendants than to off-branch concepts. Empirically, we show that HAS holds approximately in raw images and randomly initialized CLIP image embeddings, while in text embeddings it holds approximately only after contrastive learning. To theoretically explain this asymmetry, we analyze an unconstrained features model in which the image embeddings are fixed and only the text embeddings are trained. At the global minimizer, the text-side HAS violation percentage at each node is at most the image-side HAS violation and near-violation percentage. The key mechanism is the hierarchical co-occurrence structure of image-text pairs, which biases text embeddings toward root-centered image prototypes. For a linear contrastive loss, we derive a closed-form solution that makes this mechanism explicit. We extend the analysis to the InfoNCE loss and establish analogous guarantees under sufficient regularization. Synthetic experiments validate the theory and show that HAS is transferred even beyond the regimes covered by our analysis.
Figures & tables
Figure 1: How contrastive learning transfers hierarchy from image to text embeddings. Top-left: the image prototypes are arranged so that each concept is closer to its descendants than to concepts in other subtrees. The blue and red dotted lines indicate the boundary of the dog and cat subtrees, respectively. Top-right: each image of a labrador is a variation of the labrador prototype, and its caption may pair with any concept on the path from the root to that leaf, so an image can be captioned labrador , dog or animal . Bottom-left: the pairing also means that the caption dog pairs with images of dog or any of its descendants, so the average of these images approaches the mean of the dog subtree. Bottom-right: contrastive learning pulls each text embedding toward the corresponding subtree mean, so the text embeddings inherit the arrangement of the image prototypes.
Figure 2: Distribution of HAS-Viol vs depth on ViT-L/14 CLIP embeddings over non-root and non-leaf nodes. Top: text embeddings show that HAS is acquired through pretraining. Bottom: image embeddings show approximate HAS at random initialization and stronger HAS after pretraining.
Representation
HAS-Viol (%) ↓
HAS-Viol shuffled (%) ↓
Raw pixels
26.51
50.95±0.29
Randomly initialized ViT-L/14
27.88±0.24
56.80±0.42
Pretrained OpenAI CLIP ViT-L/14
2.95
32.93±0.48
i.i.d. Gaussian vectors ( d=768 )
76.02±0.97
76.33±0.93
Table 1: Mean image-side HAS-Viol . Each representation is paired with its own hierarchy-shuffled null. Null values report mean ± standard deviation over five hierarchy shuffles.
Figure 3: Overview of the HAS transfer results. Under assumptions on the data model, both losses decompose the text embeddings into prototypes plus perturbations. Bounding the perturbations relative to the prototype norms gives a bound on text-side violations in terms of image-side margins.
Figure 4: Optimal text geometry for a stylized hierarchy. Each colored triangle is a regular simplex of child concepts, with its parent (diamond) at the centroid. The optimal text directions in (b) satisfy aklin,∗∝μk−ξklinμroot , where ξklin is the root-centering coefficient. Each child simplex remains centered at its parent, and HAS is preserved.
Figure 5: Synthetic validation of HAS transfer at small regularization with the InfoNCE loss. (a) L-BFGS convergence at λ=10−3 : the InfoNCE loss without the regularization term and the text-side HAS-Viol against the training step. (b) At the minimizer, text-side HAS-Viol reaches zero at smaller λ than is needed for prototype alignment to saturate.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
S
HAS-Viol (%) ↓
HAS-Viol shuffled (%) ↓
ViT-B/16
6
28.55±0.34
57.20±0.07
ViT-B/32
3
29.58±0.13
57.39±0.15
ViT-L/14
3
27.88±0.24
56.80±0.12
ViT-H/14
3
27.63±0.56
56.62±0.03
RN50
3
28.53±1.66
59.45±1.44
ViT-L/14@336
1
27.83
56.83±0.33
Appendix
Table 2: Approximate image-side HAS at random initialization across visual backbones, measured with the disjoint-split protocol. Each backbone is evaluated with S independent random initializations, and both columns report the mean ± sample standard deviation across those samplings; each sampling’s null is itself a mean over five hierarchy shuffles. The ViT-L/14 row is the randomly initialized row of Table 1 .
Pretrained OpenAI CLIP
Randomly initialized
Setting
HAS-Viol (%)
HAS-Viol shuffled (%)
HAS-Viol (%)
HAS-Viol shuffled (%)
ImageNet / WordNet, descendant-leaf mean
2.95
32.93±0.48
28.15
56.70±0.42
ImageNet / WordNet, uniform child mean
3.9
34.5±0.3
29.1
56.8±0.5
CUB-200 / bird taxonomy
3.8
41.2±0.8
33.6
66.3±1.4
Appendix
Table 3: Image-side HAS is robust to the prototype construction and to the taxonomy. Each condition is paired with its own hierarchy-shuffled null (mean ± standard deviation over five shuffles). The first row restates the ViT-L/14 measurements of Table 1 at the precision used there; its randomly initialized entries are the seed- 0 initialization rather than the mean over three initializations reported there.
Figure 6: HAS violation percentage by depth on ViT-L-14 CLIP text embeddings (instance aggregation): HAS appears with pretraining.
Figure 7: HAS violation percentage by depth on ViT-H/14 CLIP embeddings. Top: text embeddings (instance aggregation) – HAS appears with pretraining. Bottom: image embeddings (instance aggregation): approximate HAS appears at random initialization and strengthens after pretraining.
Figure 8: HAS violation percentage by depth on ViT-B/16 CLIP embeddings. Top: text embeddings (direct concept encoding): HAS appears with pretraining. Bottom: image embeddings (instance aggregation): approximate HAS appears at random initialization and strengthens after pretraining.
Figure 9: HAS violation percentage by depth on ViT-B/16 CLIP text embeddings (instance aggregation): HAS appears with pretraining.
Figure 10: HAS violation percentage by depth on ViT-B/32 CLIP embeddings. Top: text embeddings (direct concept encoding): HAS appears with pretraining. Bottom: image embeddings (instance aggregation): approximate HAS appears at random initialization and strengthens after pretraining.
Figure 11: HAS violation percentage by depth on ViT-L/14@336px CLIP embeddings. Top: text embeddings (direct concept encoding): HAS appears with pretraining. Bottom: image embeddings (instance aggregation): approximate HAS appears at random initialization and strengthens after pretraining.
Figure 12: HAS violation percentage by depth on SigLIP2 embeddings. Top: text embeddings (direct concept encoding): HAS appears with pretraining. Bottom: image embeddings (instance aggregation): approximate HAS appears at random initialization and strengthens after pretraining.
Figure 13: HAS violation percentage by depth on MetaCLIP2 embeddings. Top: text embeddings (direct concept encoding): HAS appears with pretraining. Bottom: image embeddings (instance aggregation): approximate HAS appears at random initialization and strengthens after pretraining.
Shared instances
Disjoint splits
Representation
Original leaf assignments
Shuffled leaf assignments
Original leaf assignments
Shuffled leaf assignments
Raw pixels
25.75
49.87±0.38
26.59
51.06±0.41
Random ViT-L/14
27.17±0.19
56.04±0.12
28.03±0.23
56.95±0.13
OpenAI CLIP ViT-L/14
2.76
32.81±0.56
2.76
32.87±0.56
i.i.d. Gaussian ( d=768 )
1.19±0.07
1.19±0.04
76.02±1.03
76.24±0.59
Appendix
Table 4: Mean image-side HAS-Viol (%) under shared-instance versus disjoint-split instance aggregation, paired with matched hierarchy-shuffled nulls. Spreads ( ± ) denote sample standard deviations across random seeds or permutations (details in text). Disjoint splitting eliminates the apparent separation of the Gaussian noise baseline while preserving the hierarchical geometry of genuine visual representations.
Figure 14: HAS violation percentages by depth on ViT-L/14 CLIP image embeddings remain similar under the disjoint-split control. Top: disjoint-split control, where parent prototypes and comparison prototypes are estimated from disjoint image instances. Bottom: original instance-aggregation evaluation.
Figure 15: λ sweep at root norm ∥μroot∥=10−2 in the same balanced-caption setup as Figure 5(b) . The x-axis is λ ; the left y-axis is the final HAS-Viol averaged over non-leaf nodes, and the right y-axis is the final minimum prototype alignment over non-root nodes. Although this root norm violates the sufficient transfer condition Dϵnce<ρminγHAS/16 by a factor of 161 on this set of prototypes, because Dϵnce≥ξmax∥μroot∥ and ξmax=2.50 , the drop in violation percentage is nearly unchanged: it again reaches zero by λ=10−3 , while prototype alignment saturates just below 1 , reflecting the remaining root prototype μroot .
Figure 16: Root-norm sweep at λ=10−3 in the same balanced-caption setup as Section J.3 . The x-axis is the root norm ∥μroot∥ , and the y-axis is the final HAS-Viol averaged over non-leaf nodes. The violation percentage remains nearly flat at small root norm, then departs strongly from near-zero once the root norm is large, with non-monotone behavior in the extreme large-root-norm regime.
Figure 17: σ -robustness sweep in the same balanced-caption setup as Section J.3 , with root norm ∥μroot∥=10−4 and λ∈{10−3,10−2,10−1} . Left: HAS-Viol . Middle: global minimum branch margin. Right: minimum non-root prototype alignment. All three regularization levels remain stable through σ=3×10−3 . At σ=10−2 , only λ=10−3 has a nonzero violation percentage ( 0.014% ) and a negative global minimum margin; the other two settings retain zero violations and positive margins. At σ=3×10−2 , every λ has a negative global minimum margin and a violation percentage between 0.37% and 0.38% .
Quantity
Randomly initialized ViT-L/14
OpenAI CLIP ViT-L/14
OpenCLIP ViT-L/14 (DataComp-1B)
Inequality required by Theorem 4.6
D<ρmin/4 holds?
No
No
No
Realized transfer
Mean HAS-Violimg(u) across anchors (%)
25.55
2.95
2.42
Mean HAS-Violtxt(u) across anchors (%)
27.11
9.58
8.35
Worst-anchor slack (must hold at every anchor)
0.123
0.184
0.259
Appendix
Table 5: The inequality required by Theorem 4.6 and measured approximate HAS transfer on the same real prototypes. The inequality D<ρmin/4 does not hold for any of the three backbones, so we obtain no guarantee from the theorem here. We evaluate the image-side rows on the fixed prototypes and the text-side rows on the closed-form linear-loss optimum Alin,∗ . For the worst-anchor slack, one value of τ must satisfy Eq. 8 at every anchor, so we report the largest anchor-specific slack. This slack is post hoc. For the last three rows, we pool the 505,938 (anchor, off-branch node) pairs over the 375 anchors. For the two HAS-Viol rows, we report per-anchor percentages measured from all training images per node in one pass, not with the disjoint even/odd split of Table 1 .
HAS-Violtxt (%)
Fixed image embeddings
HAS-Violimg (%)
path-consistent
permuted
Randomly initialized ViT-L/14
25.55
27.11
61.63±0.24
OpenAI CLIP ViT-L/14
2.95
9.58
54.06±0.21
OpenCLIP ViT-L/14 (DataComp-1B)
2.42
8.35
54.17±0.16
Appendix
Table 6: Caption-permutation ablations for (a) the closed-form linear-loss text optimum on fixed real image prototypes and (b) the synthetic InfoNCE setting of Section J.3 . Permuted entries are mean ± standard deviation over five permutation seeds.
Figure 18: Text-side HAS-Viol is lowest with path-consistent captions during joint CLIP training. We plot text (left) and image (right, log scale) HAS violation percentages (lower is better). Path-consistent captions pair an image with a leaf or ancestor; leaf-only captions only pair with the leaf; in the permuted pairing, we shuffle hypernyms among images at the same depth.
Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image-text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.
Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein +8
Ulm University, Germany · University of Lübeck, Germany · Technical University of Munich, Germany +4
CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embeddings and global pooling fail to encode relational structure such as part-whole and parent-child relations. Hyperbolic VLMs address this gap with entailment-based objectives, and text-conditioned variants improve fine-grained alignment through sentence- and phrase-level queries. However, these two lines of work remain separate: hyperbolic VLMs use static image and region features, while query-conditioned methods lack hierarchical geometric structure. We present Hyper3-CLIP, a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling. To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses. Query-conditioned pooling is active only during training. Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO, while remaining competitive on hierarchy metrics. We also audit zero-shot prompt sensitivity under fixed prompt regimes and study the effect of the localized GRIT part budget used during training. Code is available at https://github.com/Hyper3Labs/hyper3-clip.
Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat +3
hyper3labs, Berlin, Germany · Hasso Plattner Institute, University of Potsdam, Potsdam, Germany · Faculty of Computers and Data Science, Alexandria University, Alexandria, Egypt +1
As models and data scale, independently trained networks often induce analogous notions of similarity. But, matching similarities is weaker than establishing an explicit correspondence between the representation spaces, especially for multimodal models, where consistency must hold not only within each modality, but also for the learned image-text coupling. We therefore ask: given two independently trained multimodal contrastive models (with encoders (f,g) and (f,g)) -- trained on different distributions and with different architectures -- does a systematic geometric relationship exist between their embedding spaces? If so, what form does it take, and does it hold uniformly across modalities? In this work, we show that across model families such as CLIP, SigLIP, and FLAVA, this geometric relationship is well approximated by an orthogonal map (up to a global mean shift), i.e., there exists an orthogonal map Q where Q⊤Q=I such that f(x)≈Qf(x) for paired images x. Strikingly, the same Q simultaneously aligns the text encoders i.e., g(y)≈Qg(y) for texts y. Theoretically, we prove that if the multimodal kernel agrees across models on a small anchor set i.e. ⟨f(x),g(y)⟩≈⟨f(x),g(y)⟩, then the two models must be related by a single orthogonal map Q and the same Q maps images and text across models. More broadly, this finding enables backward-compatible model upgrades, avoiding costly re-embedding, and has implications for the privacy of learned representations. Our project page: https://canonical-multimodal.github.io/