cs.LGMar 26, 2026
SaveExplaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings
Organizations: Ulm University, Germany · University of Lübeck, Germany · Technical University of Munich, Germany · AUMOVIO SE, Germany ·
Abstract
Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image-text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.
Figures & tables
Figure 1: Our post-hoc framework to explain (embed given concepts, cluster and name found parents ), verify , and align the semantic hierarchy induced by a VLM in its embedding space.
Figure 2: Running toy example on CIFAR-10 leaf classes.
| tree ( ) | tree soft ( ) | faithfulness ( ) | ZS ( ) | |
| CIFAR10 | 11.4794 | 8.31674 | 6.75924 | 8.63874 |
| CIFAR100 | 14.2324 | 11.9802 | 15.4268 | 13.3033 |
| CUB | 17.9089 | 25.6268 | 15.5722 | 33.1719 |
| ImageNet | 11.2519 | 17.7296 | 15.255 | 8.7422 |
Table 1: Faithfulness, tree-traversal accuracy (tree), soft tree-traversal accuracy (tree soft), and zero-shot performance (ZS) by dataset, averaged over models and encoders.
| early stop (ours) | vanilla | |
| CIFAR10 | 0.173017 | 0.328772 0.197652 |
| CIFAR100 | 2.1177 0.560399 | 0.586459 |
| CUB | 1.11294 0.768076 | 0.596943 |
| ImageNet | 2.47399 0.980289 | 0.996317 |
Table 2: Tree-traversal early stopping returns internal nodes instead of incorrect leaves. Left: raw last correct node distance ( ), averaged over encoders and models. Right: average semantic distance ( ) from the predicted node to the ground-truth concept in the SUMO ontology (walking distance). Best per dataset in bold.
Figure 3: Faithfulness (top) versus plausibility (bottom) results for different models and backbones, averaged over 3 datasets and 3 leaf encoding types (image, text, both).
Figure 4: Effect of leaf embedding modality: Results of verifying and comparing encoders against a target ontology ( Figures 4(a) and 4(b) ). For results on comparing encoders against each other see Figure 5 .
Figure 5: nUTED ( ) between trees induced by image and text encoders of each the same VLM; averaged over standard datasets. nUTED marks worse than random.
| Model | nUTED | |||||
| , , , | ||||||
| ALIGN | 0.589 2.9e-01 | 0.080 3.7e-03 | 0.579 1.3e-03 | 0.701 | 0.753 | 0.804 |
| C V-L14@336 | 0.850 2.8e-01 | 0.098 7.1e-04 | 0.732 4.5e-03 | 0.858 | 0.946 | 0.958 |
| FLAVA | 0.731 3.9e-01 | 0.072 2.2e-03 | 0.688 1.1e-03 | 0.786 | 0.890 | 0.923 |
| SIGLIP 224 | 0.720 8.6e-02 | 0.102 3.7e-04 | 0.723 2.4e-03 | 0.837 | 0.914 | 0.931 |
Table 3: Results of the ontology manipulation for two swapped leaves; averaged over 10 random swaps and 3 seeds. We report variances exceeding (see supplementary for all variances).
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| dataset | early stopped | vanilla |
| CIFAR10 | 0.685725 | 0.722408 |
| CIFAR100 | 1.09736 | 1.16467 |
| ImageNet | 1.28787 | 1.15433 |
Table 4: Walking distance of early-stopped versus vanilla tree-traversal predictions to ground truth node in SUMO.
| Faithfulness | Hierarchical Consistency | ||||||
| model | tree | tree soft | faithfulness | zero-shot | Hier. Consist. (Yago) | Hier. Consist. (SUMO) | Hier. Consist. (OpenCyc) |
| C RN50 | 25.8721 | 25.9067 | 30.5322 | 19.6896 | 0.060379 | 0.0818652 | 0.0725994 |
| C RN101 | 25.8257 | 24.9015 | 56.822 27.1548 | 21.7153 | 0.0677054 | 0.0717954 | 0.0742818 |
| C RN50x4 | 25.9481 | 25.6748 | 28.0943 | 21.8947 | 0.0672519 | 0.0718691 | 0.888085 0.0748344 |
| C RN50x16 | 25.0989 | 24.8214 | 27.2003 | 21.8353 | 0.0771141 | 0.0652447 | 0.0807843 |
| C RN50x64 | 29.3122 | 26.9568 | 30.7231 | 22.5873 | 0.0720746 | 0.0713157 | 0.0834133 |
Table 5: Faithfulness and plausibility metrics by model averaged over encoders.
| tree | faithfulness | zero-shot | Yago | SUMO | OpenCyc | ||
| CIFAR10 | image | 8.75938 | 2.29032 | 7.16286 | 0.0 | 0.0116783 | 0.0203476 |
| both | 7.63824 | 1.12502 | 6.90112 | 0.0139087 | 0.00243627 | 0.0114746 | |
| text | 13.0658 | 8.75481 | 10.9394 | 0.01623 | 0.029357 | 0.0307789 | |
| CIFAR100 | image | 9.92201 | 7.31737 | 11.3863 | 0.00871275 | 0.0076518 | 0.017071 |
| both | 10.3877 | 7.21879 | 11.1241 | 0.00917003 | 0.00943767 | 0.0213515 | |
| text | 7.25594 | 8.77176 | 14.9311 | 0.0141677 | 0.0122311 | 0.0215226 |
Table 6: Plausibility (Hierarchical Consistency) and faithfulness metrics per leaf encoding strategy and per dataset, averaged over models.
| model | nUTED |
| clip RN50 | 0.140378 |
| clip RN101 | 0.145299 |
| clip RN50x4 | 0.138743 |
| clip RN50x16 | 0.131248 |
| clip RN50x64 | 0.162033 |
| clip ViT-B/16 | 0.138759 |
Table 7: nUTED between each text and image encoder of the same VLM, averaged over datasets.
| tree inference acc. | tree inference soft acc. | faithfulness | zero-shot acc. | |
| CIFAR10 | 11.4794 | 8.31674 | 6.75924 | 8.63874 |
| CIFAR100 | 14.2324 | 11.9802 | 15.4268 | 13.3033 |
| CUB | 17.9089 | 25.6268 | 15.5722 | 33.1719 |
| ImageNet | 11.2519 | 17.7296 | 15.255 | 8.7422 |
Table 8: Faithfulness metrics by dataset in percent, averaged over models and encoders.
| encoder | dataset | tree | tree soft | faithfulness | zero-shot |
| image | CIFAR10 | 8.75938 | 5.09308 | 2.29032 | 7.16286 |
| CIFAR100 | 9.92201 | 7.07137 | 7.31737 | 11.3863 | |
| CUB | 7.4302 | 5.32752 | 4.49686 | 8.0489 | |
| ImageNet | 4.32965 | 3.70023 | 3.55872 | 7.66527 | |
| text & image | CIFAR10 | 7.63824 | 4.22564 | 1.12502 | 6.90112 |
| CIFAR100 | 10.3877 | 7.26669 | 7.21879 | 11.1241 |
Table 9: Faithfulness metrics by encoder and dataset, averaged over models.
Table 10: Hypothesis tests: Zero-shot correlated. Results of the two-sided Pearson correlation test for correlation of top-1 zero-shot accuracy (top) respectively faithfulness (bottom) with other metrics. Experiments vary over 13 models, 4 datasets, and 3 leaf embedding methods, using LLM-only text embedding strategy for the non-WordNet-aligned CUB dataset, else the WordNet-only strategy. The results also imply the respective one-sided case.
| score_b | tree inference acc. | tree inference soft acc. | ||
| p-value | significant | p-value | significant | |
| model | ||||
| ALIGN | ✓ | ✓ | ||
| C RN101 | ✓ | |||
| C RN50 | ✓ | |||
| C RN50x16 | ✓ | ✓ | ||
Table 11: Hypothesis test: Zero-shot tree inference accuracy. Results of the one-sided t-tests testing against the 0-hypothesis that top-1 zero-shot accuracy is not different from tree inference / soft tree inference accuracy, instead of being greater. Each test includes more than 10 experiments (different datasets, models, and leaf embedding strategies).
| encoder | method | CIFAR10 | CIFAR100 | CUB | ImageNet |
| image | early stop | 0.0338999 | 0.0283582 | 0.027191 | 0.0151002 |
| full | 0.0349731 | 0.028518 | 0.0246555 | 0.0248478 | |
| text | early stop | 0.0408511 | 0.039292 | 0.00810018 | 0.0226702 |
| full | 0.049406 | 0.0443551 | 0.00492161 | 0.0361168 | |
| text & image | early stop | 0.0357628 | 0.0558676 | 0.0200695 | 0.0132536 |
| full | 0.0372978 | 0.0642276 | 0.0162623 | 0.0313628 |
Table 12: Detailed normalized results on how much early stopping improves upon predicting correct (super-)categories: normalized tree-distance scores ( ) between the predicted node and the last correct node (LCA of the ground truth with the leaf predicted by tree traversal); averaged over models and grouped by encoder.
Figure 6: Comparing text embedding strategies: Top 1 zero-shot accuracy per text embedding strategy on CIFAR-10, CIFAR-100, and ImageNet and for different leaf embedding strategies (averaged). Top: averaged over models, bottom: per model.
Figure 7: Comparison of datasets: Faithfulness only per dataset and model averaged over text embedding strategies (WordNet, WordNet+LLM).
Figure 8: Comparing leaf embedding strategies: Faithfulness metrics averaged across datasets and models.
| # classes | average maximum tree depth | |
| CIFAR10 | 0.995347 | |
| CIFAR100 | 3.68141 | |
| CUB | 5.84984 | |
| ImageNet | 22.1267 |
Table 13: Average maximum depth of trees extracted for different leaf class sets. Averaged over models and encoder modality.
Figure 9: Explaining the faithfulness gap: Comparison of faithfulness metrics, in particular soft tree inference accuracy, tree inference accuracy, and zero-shot accuracy, averaged over models, datasets, text embedding strategies WordNet & WordNet+LLM, and leaf embeddings trategies. Top: by model; bottom: by dataset and leaf embedding strategy.
Figure 10: Illustration of the steps involved to calculate the ontology validation metrics.
Figure 11: Comparison of the two SUMO validity scores. If not stated otherwise, averaged over models, datasets, leaf and text embedding strategies.
Figure 12: Comparison of zero-shot performance and faithfulness for textual leaf concepts and different augmenters (top); comparison of hierarchical consistency for different augmenters (bottom); averaged over datasets, models, (and modalities).
Figure 13: Empirical approximation of the expected normalized UTED across randomly generated binary trees with identical leaf sets. Results based on 200 runs for varying numbers of leaf nodes.
| Model | nUTED | |||||
| Validation task | ||||||
| ALIGN | 0.589 2.9e-01 | 0.080 3.7e-03 | 0.579 1.3e-03 | 0.701 1.5e-04 | 0.753 0.0e+00 | 0.804 4.1e-06 |
| C V-L14@336 | 0.850 2.8e-01 | 0.098 7.1e-04 | 0.732 4.5e-03 | 0.858 1.5e-04 | 0.946 0.0e+00 | 0.958 4.0e-07 |
| FLAVA | 0.731 3.9e-01 | 0.072 2.2e-03 | 0.688 1.1e-03 | 0.786 1.4e-04 | 0.890 0.0e+00 | 0.923 3.4e-07 |
| SIGLIP 224 | 0.720 8.6e-02 | 0.102 3.7e-04 | 0.723 2.4e-03 | 0.837 6.4e-05 | 0.914 0.0e+00 | 0.931 1.7e-06 |
| Modality alignment | ||||||
Table 14: Results of the ontology manipulation: Hyperparameter search based on leaf swapping, averaged over 10 swaps and 3 runs (Validation task), adaptation to an ontology extracted from the text embeddings (Modality alignment) and to three SUMO ontologies (Ontological commitment); both averaged over 5 runs.
Figure 14: For the flava-full VLM model we achieved to learn a transformation from the image embedding space to the text embedding space which achieves full alignment of image and text ontology (except for symmetry).
| Model | nUTED | |||||
| Validation task | ||||||
| ALIGN | 0.812 1.7e-01 | 0.108 2.0e-03 | 0.485 1.6e-03 | 0.715 1.3e-04 | 0.753 0.0e+00 | 0.803 5.1e-06 |
| C V-L14@336 | 0.927 2.6e-01 | 0.103 2.2e-04 | 0.595 4.3e-03 | 0.887 9.9e-05 | 0.946 0.0e+00 | 0.958 3.8e-07 |
| FLAVA | 0.799 3.6e-01 | 0.090 2.0e-03 | 0.712 8.8e-04 | 0.850 9.2e-05 | 0.890 0.0e+00 | 0.923 1.7e-06 |
| SigLIP 224 | 0.547 1.9e-01 | 0.082 1.9e-03 | 0.623 4.5e-03 | 0.861 6.3e-05 | 0.914 0.0e+00 | 0.931 1.7e-06 |
| Modality alignment | ||||||
Table 15: Simple Linear Model to approximate the denoted embedding transformation, all runs are averaged over five seeds.
Figure 15: UMAP visualization of the original embedding space based on the test dataset
Figure 16: UMAP visualization after applying the learned transformation to the embedding space of the test dataset
More in this topic
VLM Interpretability
Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.
See all 259 papers