Provable Cross-Modal Transfer of Hierarchical Concept Structure in CLIP Embeddings
Organizations: University of Pennsylvania
Abstract
Contrastive vision-language models such as CLIP are trained to align images and text, not to represent taxonomies. Yet empirical studies show that CLIP embeddings exhibit hierarchical structure aligned with the hierarchy of semantic concepts. A theoretical characterization of why such hierarchy emerges from a non-hierarchical objective remains open. In this paper, we formalize this phenomenon as Hierarchical Angular Separation (HAS), a geometric property whereby each non-leaf concept embedding is more similar to its descendants than to off-branch concepts. Empirically, we show that HAS holds approximately in raw images and randomly initialized CLIP image embeddings, while in text embeddings it holds approximately only after contrastive learning. To theoretically explain this asymmetry, we analyze an unconstrained features model in which the image embeddings are fixed and only the text embeddings are trained. At the global minimizer, the text-side HAS violation percentage at each node is at most the image-side HAS violation and near-violation percentage. The key mechanism is the hierarchical co-occurrence structure of image-text pairs, which biases text embeddings toward root-centered image prototypes. For a linear contrastive loss, we derive a closed-form solution that makes this mechanism explicit. We extend the analysis to the InfoNCE loss and establish analogous guarantees under sufficient regularization. Synthetic experiments validate the theory and show that HAS is transferred even beyond the regimes covered by our analysis.
Figures & tables
| Representation | (%) | shuffled (%) |
|---|---|---|
| Raw pixels | ||
| Randomly initialized ViT-L/14 | ||
| Pretrained OpenAI CLIP ViT-L/14 | ||
| i.i.d. Gaussian vectors ( ) |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | (%) | shuffled (%) | |
|---|---|---|---|
| ViT-B/16 | |||
| ViT-B/32 | |||
| ViT-L/14 | |||
| ViT-H/14 | |||
| RN50 | |||
| ViT-L/14@336 |
| Pretrained OpenAI CLIP | Randomly initialized | |||
|---|---|---|---|---|
| Setting | (%) | shuffled (%) | (%) | shuffled (%) |
| ImageNet / WordNet, descendant-leaf mean | ||||
| ImageNet / WordNet, uniform child mean | ||||
| CUB-200 / bird taxonomy | ||||
| Shared instances | Disjoint splits | |||
|---|---|---|---|---|
| Representation | Original leaf assignments | Shuffled leaf assignments | Original leaf assignments | Shuffled leaf assignments |
| Raw pixels | ||||
| Random ViT-L/14 | ||||
| OpenAI CLIP ViT-L/14 | ||||
| i.i.d. Gaussian ( ) | ||||
| Quantity | Randomly initialized ViT-L/14 | OpenAI CLIP ViT-L/14 | OpenCLIP ViT-L/14 (DataComp-1B) |
|---|---|---|---|
| Inequality required by Theorem 4.6 | |||
| holds? | No | No | No |
| Realized transfer | |||
| Mean across anchors (%) | 25.55 | 2.95 | 2.42 |
| Mean across anchors (%) | 27.11 | 9.58 | 8.35 |
| Worst-anchor slack (must hold at every anchor) | 0.123 | 0.184 | 0.259 |
| (%) | |||
|---|---|---|---|
| Fixed image embeddings | (%) | path-consistent | permuted |
| Randomly initialized ViT-L/14 | |||
| OpenAI CLIP ViT-L/14 | |||
| OpenCLIP ViT-L/14 (DataComp-1B) | |||
Vision-Language Models
Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.
See all 759 papers