cs.CVNov 9, 2026

GHiML: VLM-Guided Hierarchical Multi-Label Learning with Applications to Benthic Imagery

Authors: Isaac Xu, Devi Ayyagari, Martin Gillis, Benjamin Misiuk, Craig J. Brown, Thomas Trappenberg

Organizations: Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada · Departments of Geography and Earth Sciences, Memorial University of Newfoundland, St. John’s, NL, Canada · Department of Oceanography, Dalhousie University, Halifax, NS, Canada

Abstract

Vision language models (VLMs) encode substantial expert knowledge within their weights, yet their direct application to specialised classification domains such as benthic imagery remains limited. Meanwhile, vision transformers trained on these tasks rely predominantly on low-level visual latents (colours, textures, and shapes) that lack the higher-level reasoning humans employ for identification. We propose Guided Hierarchical Multi-Label (GHiML) learning, a pipeline that extracts conceptual latent variables from a VLM conditioned on training data, generates sample-relevant captions grounded in those variables, and fuses the resulting textual embeddings with image embeddings via cross-attention and a per-node gated router. We also introduce the inverted HML score (dinvd_{\mathrm{inv}}), a similarity score that evaluates hierarchical multi-label predictions through graph traversal distance and a cardinality penalty, providing an interpretable alternative to existing measures, which are insensitive to realistic degradations in sparse, deep hierarchies. Experiments on BenthicNet, FathomNet and iNaturalist show that the router improves macro F1F_1 over the vision-only baseline by up to 6.7 pp and dinvd_{\mathrm{inv}} by up to 21.1 pp on in-domain benthic data, with gains persisting under distribution shift. Ablations locate the contribution of the text itself: on the multi-label biotic hierarchies, removing the per-sample text signal degrades dinvd_{\mathrm{inv}} while leaving macro F1F_1 unchanged, consistent with text acting through prediction cardinality rather than node discrimination, whereas iNaturalist, whose annotations are single-label, loses 18.7% of macro F1F_1. Our results support the hypothesis that VLMs contain exploitable specialised knowledge that, when appropriately elicited, can augment classification in domains where direct visual learning is insufficient.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hierarchical Prompt Learning for Hyperbolic Vision-Language Models

    Sep 21, 2026Andro Erdelez, Pascal Mettes, Behzad BozorgtabarVLM AdaptationVision-Language Models

  2. Image Recognition with Vision and Language Embeddings of VLMs

    Sep 11, 2025Illia Volkov, Nikita Kisel, Klara Janouskova +1Image ClassificationVLM Evaluation

  3. HELM: Hierarchical and Explicit Label Modeling with Graph Learning for Multi-Label Image Classification

    Mar 12, 2026Marjan Stoimchev, Boshko Koloski, Jurica Levatić +2hierarchical multi-label classificationMulti-Label Classification