How Hyperbolic Are Hyperbolic Vision-Language Models?
Organizations: University of Amsterdam · WAIR
Abstract
Hyperbolic vision-language models (VLMs) have emerged as a rapidly growing area of research, reporting strong zero-shot performance alongside an interpretable embedding space in which radius orders representations by generality. But how hyperbolic are hyperbolic VLMs in practice? We investigate this at the exponential map, the single operation through which curvature enters an otherwise Euclidean network. We show that across MERU, HyCoCLIP, ARGENT, and PHyCLIP, the exponential map is well approximated by the identity function, i.e., pairwise distances and angles in the pre-map Euclidean space barely change under projection, and removing the map at inference leaves evaluation scores unchanged. Motivated by this observation, we train Euclidean VLMs on CC3M from scratch under the same hierarchy losses. Over roughly the final 90% of the training steps, the training statistics are nearly indistinguishable from those of their hyperbolic counterparts, and the resulting models match the evaluation scores and recover the same radial distributions. Our findings establish Euclidean space as a sufficient substrate for hierarchical vision-language learning as it is currently practiced, and motivate new paradigms in which hyperbolic geometry can deliver on its theoretical promise. Code is available at https://github.com/fbyrman/hyperbolic-vlms.
Figures & tables
| \cellcolor bandBClassification | \cellcolor bandARetrieval | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Encoder | \cellcolor bandB!80With | \cellcolor bandB!80Without | \cellcolor bandB!80 | \cellcolor bandB!80 | \cellcolor bandA!80With | \cellcolor bandA!80Without | \cellcolor bandA!80 | \columncolor bandA!80[][0pt] |
| MERU | ViT-S/16 | 19.66 | 19.67 | +0.01 | 0.20 | 44.81 | 44.83 | +0.02 | 0.16 |
| ViT-B/16 | 22.22 | 22.23 | +0.01 | 0.19 | 49.20 | 49.24 | +0.03 | 0.26 | |
| HyCoCLIP | ViT-S/16 | 23.80 | 23.78 | -0.02 | 0.11 | 50.09 | 50.11 | +0.02 | 0.10 |
| ViT-B/16 | 25.02 | 25.01 | -0.02 | 0.06 | 52.47 | 52.54 | +0.07 | 0.30 | |
| ARGENT | ViT-S/16 | 22.99 | 22.95 | -0.05 | 0.42 | 48.34 | 48.24 | -0.10 | 0.70 |
| \cellcolor bandBClassification | \cellcolor bandBRetrieval | \cellcolor bandATIE ( ) | \cellcolor bandALCA ( ) | \cellcolor bandA ( ) | \cellcolor bandA ( ) | \columncolor bandA[][0pt] ( ) | |||||||||
| Model | Encoder | \cellcolor bandB!80 | \cellcolor bandB!80 | \cellcolor bandB!80 | \cellcolor bandB!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \cellcolor bandA!80 | \columncolor bandA!80[][0pt] |
| MERU | ViT-S/16 | \cellcolor winner19.8 | 19.7 | \cellcolor winner45.7 | 44.8 | \cellcolor winner6.440 | 6.582 | \cellcolor winner3.384 | 3.442 | \cellcolor winner0.605 | 0.596 | \cellcolor winner0.719 | 0.713 | \cellcolor winner0.719 | 0.711 |
| ViT-B/16 | \cellcolor winner22.5 | 22.2 | \cellcolor winner50.0 | 49.2 | \cellcolor winner6.091 | 6.220 | \cellcolor winner3.215 | 3.304 | \cellcolor winner0.626 | 0.621 | \cellcolor winner0.737 | 0.730 | \cellcolor winner0.735 | 0.732 | |
| HyCoCLIP | ViT-S/16 | 23.5 | \cellcolor winner23.8 | 47.0 | \cellcolor winner50.1 | 6.028 | \cellcolor winner5.851 | 3.180 | \cellcolor winner3.095 | 0.632 | \cellcolor winner0.641 | 0.740 | \cellcolor winner0.748 | 0.738 | \columncolor winner[][0pt]0.747 |
| ViT-B/16 | \cellcolor winner25.9 | 25.0 | \cellcolor winner53.3 | 52.5 | \cellcolor winner5.514 | 5.719 | \cellcolor winner2.974 | 3.103 | \cellcolor winner0.664 | 0.652 | \cellcolor winner0.764 | 0.753 | \cellcolor winner0.764 | 0.757 | |
| ARGENT | ViT-S/16 | 22.7 | \cellcolor winner23.0 | \cellcolor winner49.6 | 48.3 | 5.963 | \cellcolor winner5.931 | \cellcolor winner3.130 | 3.132 | 0.634 | \cellcolor winner0.637 | 0.743 | \cellcolor winner0.746 | 0.740 | \columncolor winner[][0pt]0.742 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| \cellcolor bandBGeneral | \cellcolor bandAFine-grained | \cellcolor bandCSpecialized | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Encoder | \cellcolor bandB!80 ImageNet | \cellcolor bandB!80 CIFAR-10 | \cellcolor bandB!80 CIFAR-100 | \cellcolor bandB!80 SUN397 | \cellcolor bandB!80 Caltech-101 | \cellcolor bandB!80 STL-10 | \cellcolor bandA!80 Food-101 | \cellcolor bandA!80 CUB | \cellcolor bandA!80 Cars | \cellcolor bandA!80 Aircraft | \cellcolor bandA!80 Pets | \cellcolor bandA!80 Flowers-102 | \cellcolor bandC!80 DTD | \cellcolor bandC!80 EuroSAT | \cellcolor bandC!80 RESISC45 | \cellcolor bandC!80 Country211 | |
| MERU | ViT-S/16 | ✓ | 14.3 | 53.1 | 22.9 | \colorbox winner27.6 | \colorbox winner42.1 | 75.3 | 9.3 | 3.0 | 0.9 | 1.2 | 9.8 | 7.3 | 8.0 | 18.4 | \colorbox winner20.4 | 0.8 |
| ✗ | 14.3 | \colorbox winner53.2 | 22.9 | 27.4 | 42.0 | \colorbox winner75.5 | \colorbox winner9.4 | 3.0 | 0.9 | 1.2 | 9.8 | \colorbox winner7.4 | \colorbox winner8.1 | \colorbox winner18.5 | 20.3 | 0.8 | ||
| ViT-B/16 | ✓ | 16.7 | \colorbox winner60.5 | 25.9 | 30.9 | 47.5 | 84.0 | 10.7 | 2.7 | 1.2 | 1.7 | 10.9 | 9.0 | 10.8 | 19.1 | 23.1 | 0.7 | |
| ✗ | 16.7 | 60.3 | 25.9 | 30.9 | \colorbox winner47.6 | \colorbox winner84.1 | 10.7 | 2.7 | 1.2 | 1.7 | 10.9 | \colorbox winner9.2 | \colorbox winner10.9 | 19.1 | 23.1 | 0.7 | ||
| HyCoCLIP | ViT-S/16 | ✓ | 18.4 | \colorbox winner72.6 | 31.3 | \colorbox winner33.0 | 50.0 | 85.0 | 10.4 | \colorbox winner3.6 | 1.1 | 1.6 | 15.1 | 7.8 | 13.5 | \colorbox winner13.1 | \colorbox winner23.6 | 0.7 |
| \cellcolor bandBText Image | \cellcolor bandBImage Text | \cellcolor bandAHierarchical metrics | |||||||||||||
| \cellcolor bandBCOCO | \cellcolor bandBFlickr30K | \cellcolor bandBCOCO | \cellcolor bandBFlickr30K | \cellcolor bandAImageNet | |||||||||||
| Model | Encoder | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | TIE( ) | LCA( ) | ( ) | ( ) | ( ) | |
| MERU | ViT-S/16 | ✓ | 26.2 | 36.8 | 42.7 | 55.1 | 32.5 | 43.6 | 54.5 | \colorbox winner67.0 | \colorbox winner6.582 | 3.442 | 0.596 | 0.713 | 0.711 |
| ✗ | 26.2 | 36.8 | 42.7 | 55.1 | \colorbox winner32.7 | \colorbox winner43.7 | 54.5 | 66.9 | 6.584 | 3.442 | 0.596 | 0.713 | 0.711 | ||
| ViT-B/16 | ✓ | 29.8 | 40.8 | 48.1 | 60.0 | 36.4 | 48.8 | 58.8 | 70.8 | 6.220 | 3.304 | 0.621 | 0.730 | 0.732 | |
| ✗ | 29.8 | 40.8 | 48.1 | 60.0 | \colorbox winner36.7 | 48.8 | 58.8 | 70.8 | \colorbox winner6.218 | \colorbox winner3.303 | 0.621 | 0.730 | 0.732 | ||
| \cellcolor bandBGeneral | \cellcolor bandAFine-grained | \cellcolor bandCSpecialized | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | \cellcolor bandB!80 ImageNet | \cellcolor bandB!80 CIFAR-10 | \cellcolor bandB!80 CIFAR-100 | \cellcolor bandB!80 SUN397 | \cellcolor bandB!80 Caltech-101 | \cellcolor bandB!80 STL-10 | \cellcolor bandA!80 Food-101 | \cellcolor bandA!80 CUB | \cellcolor bandA!80 Cars | \cellcolor bandA!80 Aircraft | \cellcolor bandA!80 Pets | \cellcolor bandA!80 Flowers-102 | \cellcolor bandC!80 DTD | \cellcolor bandC!80 EuroSAT | \cellcolor bandC!80 RESISC45 | \cellcolor bandC!80 Country211 | |
| MERU | ✓ | 37.5 | 67.7 | \colorbox winner32.7 | 30.9 | 68.1 | 92.5 | 78.7 | 34.8 | 13.8 | 1.7 | 79.2 | 52.1 | \colorbox winner17.4 | 30.2 | 34.5 | 5.6 |
| ✗ | 37.5 | 67.7 | 32.6 | 30.9 | 68.1 | 92.5 | 78.7 | 34.8 | \colorbox winner13.9 | \colorbox winner1.8 | \colorbox winner79.3 | 52.1 | 17.3 | 30.2 | 34.5 | \colorbox winner5.7 | |
| HyCoCLIP | ✓ | 45.8 | 88.8 | 60.1 | 57.2 | 80.8 | 95.0 | 59.2 | 16.4 | 11.6 | 3.7 | 56.8 | 23.9 | \colorbox winner29.4 | 35.8 | 45.6 | 6.5 |
| ✗ | 45.8 | 88.8 | \colorbox winner60.2 | 57.2 | \colorbox winner80.9 | 95.0 | 59.2 | 16.4 | 11.6 | 3.7 | \colorbox winner56.9 | 23.9 | 29.2 | 35.8 | 45.6 | 6.5 | |
| \cellcolor bandBText Image | \cellcolor bandBImage Text | \cellcolor bandAHierarchical metrics | ||||||||||||
| \cellcolor bandBCOCO | \cellcolor bandBFlickr30K | \cellcolor bandBCOCO | \cellcolor bandBFlickr30K | \cellcolor bandAImageNet | ||||||||||
| Model | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | TIE( ) | LCA( ) | ( ) | ( ) | ( ) | |
| MERU | ✓ | 33.2 | 44.0 | 41.1 | 51.6 | 41.8 | 52.9 | 48.1 | 58.9 | 4.033 | 2.418 | 0.756 | 0.829 | 0.832 |
| ✗ | 33.2 | 44.0 | 41.1 | 51.6 | \colorbox winner41.9 | \colorbox winner53.1 | \colorbox winner48.6 | \colorbox winner59.4 | \colorbox winner4.028 | \colorbox winner2.417 | 0.756 | 0.829 | 0.832 | |
| HyCoCLIP | ✓ | 58.4 | 69.3 | 84.9 | 90.3 | \colorbox winner72.0 | 82.0 | 92.6 | 95.4 | \colorbox winner3.172 | \colorbox winner2.047 | 0.814 | \colorbox winner0.874 | 0.874 |
| ✗ | 58.4 | 69.3 | 84.9 | 90.3 | 71.9 | 82.0 | 92.6 | 95.4 | 3.175 | 2.048 | 0.814 | 0.873 | 0.874 | |
| \cellcolor bandBGeneral | \cellcolor bandAFine-grained | \cellcolor bandCSpecialized | ||||||||||||||||
| Model | Encoder | Geometry | \cellcolor bandB!80 ImageNet | \cellcolor bandB!80 CIFAR-10 | \cellcolor bandB!80 CIFAR-100 | \cellcolor bandB!80 SUN397 | \cellcolor bandB!80 Caltech-101 | \cellcolor bandB!80 STL-10 | \cellcolor bandA!80 Food-101 | \cellcolor bandA!80 CUB | \cellcolor bandA!80 Cars | \cellcolor bandA!80 Aircraft | \cellcolor bandA!80 Pets | \cellcolor bandA!80 Flowers-102 | \cellcolor bandC!80 DTD | \cellcolor bandC!80 EuroSAT | \cellcolor bandC!80 RESISC45 | \cellcolor bandC!80 Country211 |
| MERU | ViT-S/16 | \cellcolor winner15.0 | 47.1 | 20.0 | \cellcolor winner28.5 | 41.5 | \cellcolor winner78.6 | 8.1 | \cellcolor winner3.2 | \cellcolor winner1.1 | 1.0 | \cellcolor winner11.6 | \cellcolor winner8.2 | \cellcolor winner9.4 | \cellcolor winner20.3 | \cellcolor winner23.0 | 0.6 | |
| 14.3 | \cellcolor winner53.1 | \cellcolor winner22.9 | 27.6 | \cellcolor winner42.1 | 75.3 | \cellcolor winner9.3 | 3.0 | 0.9 | \cellcolor winner1.2 | 9.8 | 7.3 | 8.0 | 18.4 | 20.4 | \cellcolor winner0.8 | |||
| ViT-B/16 | \cellcolor winner17.2 | \cellcolor winner60.8 | \cellcolor winner26.4 | \cellcolor winner31.3 | 46.9 | 83.4 | \cellcolor winner11.1 | \cellcolor winner3.2 | \cellcolor winner1.4 | 1.3 | \cellcolor winner13.6 | 8.3 | 10.1 | \cellcolor winner20.1 | \cellcolor winner24.8 | \cellcolor winner0.8 | ||
| 16.7 | 60.5 | 25.9 | 30.9 | \cellcolor winner47.5 | \cellcolor winner84.0 | 10.7 | 2.7 | 1.2 | \cellcolor winner1.7 | 10.9 | \cellcolor winner9.0 | \cellcolor winner10.8 | 19.1 | 23.1 | 0.7 | |||
| HyCoCLIP | ViT-S/16 | 17.9 | 66.1 | 29.3 | 30.8 | 47.3 | 83.0 | \cellcolor winner10.9 | 3.3 | \cellcolor winner1.3 | 1.5 | 12.1 | \cellcolor winner8.6 | 12.2 | \cellcolor winner28.8 | 22.2 | 0.7 | |
| \cellcolor bandBText Image | \cellcolor bandBImage Text | \cellcolor bandAHierarchical metrics | |||||||||||||
| \cellcolor bandBCOCO | \cellcolor bandBFlickr30K | \cellcolor bandBCOCO | \cellcolor bandBFlickr30K | \cellcolor bandAImageNet | |||||||||||
| Model | Encoder | Geometry | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | TIE( ) | LCA( ) | ( ) | ( ) | ( ) |
| MERU | ViT-S/16 | \cellcolor winner26.7 | \cellcolor winner36.9 | \cellcolor winner44.1 | \cellcolor winner55.6 | \cellcolor winner33.3 | \cellcolor winner45.0 | \cellcolor winner56.6 | \cellcolor winner67.2 | \cellcolor winner6.440 | \cellcolor winner3.384 | \cellcolor winner0.605 | \cellcolor winner0.719 | \cellcolor winner0.719 | |
| 26.2 | 36.8 | 42.7 | 55.1 | 32.5 | 43.6 | 54.5 | 67.0 | 6.582 | 3.442 | 0.596 | 0.713 | 0.711 | |||
| ViT-B/16 | 29.7 | 40.6 | \cellcolor winner48.6 | 59.4 | \cellcolor winner37.5 | \cellcolor winner49.2 | \cellcolor winner61.5 | \cellcolor winner73.2 | \cellcolor winner6.091 | \cellcolor winner3.215 | \cellcolor winner0.626 | \cellcolor winner0.737 | \cellcolor winner0.735 | ||
| \cellcolor winner29.8 | \cellcolor winner40.8 | 48.1 | \cellcolor winner60.0 | 36.4 | 48.8 | 58.8 | 70.8 | 6.220 | 3.304 | 0.621 | 0.730 | 0.732 | |||
Vision-Language Models
Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.
See all 769 papers