GHiML: VLM-Guided Hierarchical Multi-Label Learning with Applications to Benthic Imagery
Authors: Isaac Xu, Devi Ayyagari, Martin Gillis, Benjamin Misiuk, Craig J. Brown, Thomas Trappenberg
Organizations: Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada · Departments of Geography and Earth Sciences, Memorial University of Newfoundland, St. John’s, NL, Canada · Department of Oceanography, Dalhousie University, Halifax, NS, Canada
Vision language models (VLMs) encode substantial expert knowledge within their weights, yet their direct application to specialised classification domains such as benthic imagery remains limited. Meanwhile, vision transformers trained on these tasks rely predominantly on low-level visual latents (colours, textures, and shapes) that lack the higher-level reasoning humans employ for identification. We propose Guided Hierarchical Multi-Label (GHiML) learning, a pipeline that extracts conceptual latent variables from a VLM conditioned on training data, generates sample-relevant captions grounded in those variables, and fuses the resulting textual embeddings with image embeddings via cross-attention and a per-node gated router. We also introduce the inverted HML score (dinv), a similarity score that evaluates hierarchical multi-label predictions through graph traversal distance and a cardinality penalty, providing an interpretable alternative to existing measures, which are insensitive to realistic degradations in sparse, deep hierarchies. Experiments on BenthicNet, FathomNet and iNaturalist show that the router improves macro F1 over the vision-only baseline by up to 6.7 pp and dinv by up to 21.1 pp on in-domain benthic data, with gains persisting under distribution shift. Ablations locate the contribution of the text itself: on the multi-label biotic hierarchies, removing the per-sample text signal degrades dinv while leaving macro F1 unchanged, consistent with text acting through prediction cardinality rather than node discrimination, whereas iNaturalist, whose annotations are single-label, loses 18.7% of macro F1. Our results support the hypothesis that VLMs contain exploitable specialised knowledge that, when appropriately elicited, can augment classification in domains where direct visual learning is insufficient.
Figures & tables
Figure 1: Difference in Latents. A hypothetical example. The red (left) branch depicts classification typical of contemporary vision-based neural networks, whose low-level latents learned from a limited dataset are insufficient to classify the object correctly. The green (right) branch additionally draws on general conceptual knowledge that cannot be learned from visual characteristics alone.
Figure 2: VLM Variable Proposal and Captioning. A sea star image and its annotation are used to propose the variable “ambush”; during captioning, the VLM is prompted to reason about each variable for each sample.
Figure 3: Model Architectures. The (1) image and (2) text streams apply self-attention pooling followed by an HML-constrained classifier; (3) applies cross-attention followed by pooling; (4) a router predicts per-node weights that mix the logits of (1)–(3).
BN-Sub
BN-Bio
FN
F1
dinv
F1
dinv
F1
dinv
ID
Txt
48.5±2.9
(5.0)
40.5±5.3
(5.0)
7.8±1.6
(5.0)
1.2±1.7
(5.0)
15.6±2.9
(5.0)
0.3±0.2
(5.0)
Img
77.9±3.5
(3.4)
68.1±4.5
(3.5)
14.3±3.8
(3.2)
5.1±5.5
(3.5)
44.5±5.1
(3.5)
16.5±3.2
(3.5)
Img + Txt
78.3±3.7
(3.0)
68.7±4.3
(3.0)
14.0±3.5
(3.8)
4.9±5.6
(3.5)
44.5±5.2
(3.5)
16.7±3.3
(3.4)
Router
81.4±2.5
(1.1)
72.7±3.0
(1.1)
16.6±3.5
(2.0)
7.6±6.5
(1.5)
51.2±4.2
(1.1)
37.6±4.1
(1.0)
Router †
79.4±2.3
(2.5)
69.4±3.9
(2.4)
17.8±4.0
(1.0)
7.7±6.9
(1.5)
49.7±4.6
(1.9)
23.0±4.6
(2.0)
Table 1: HML Results. Macro-average performance across datasets, modalities, and distribution settings. Values are mean ± std (%). Superscript indicates mean rank across seeds. Ten seeds were tested for BN datasets, and thirty for FN. Coloured values differ significantly from Img alone (Wilcoxon signed-rank test; green : p<0.05 ; bold dark green : p<0.01 ).
F1
dinv
Txt
0.9±0.1
(5.0)
28.4±2.9
(5.0)
Img
22.8±4.2
(4.0)
56.3±2.7
(4.0)
Img + Txt
29.8±6.0
(3.0)
61.2±4.2
(3.0)
Router
55.6±2.1
(2.0)
77.9±1.7
(1.0)
Router †
59.0±2.9
(1.0)
75.9±2.6
(2.0)
Table 2: iNaturalist Results. Macro-average performance over the 100,000 held-out images. Values are mean ± std (%) across five seeds; superscript indicates mean rank across seeds. Coloured values are significantly higher than Img (one-sided Wilcoxon signed-rank test; green : p<0.05 ).
dataset
annotation
ΔF1
Δdinv
BN-Sub (23 nodes)
multi-label
+1.5%
−0.1%
BN-Bio (293 nodes)
multi-label
+0.6%
−21.7%
iNaturalist (10,000 endpoints)
single-label
−18.7%
−10.2%
Table 3: Mean Replacement. Relative change in scores across five seeds when each sample’s text embedding is replaced by the mean. Negative values indicate degradation. Bolded indicate p≤0.05 on paired one-sided Wilcoxon signed-rank test.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Dataset
Dataset Size [samples]
Throughput [samples/s]
Peak Alloc [GB] (mean/max)
Peak Reserved [GB] (mean/max)
KV Overhead [GB]
Var. Proposal
BN-Sub
102,118
5.73 ± 0.22
19.09 / 19.23
21.61 / 22.75
2.75
BN-Bio
105,133
5.61 ± 0.40
19.02 / 19.21
20.64 / 21.15
2.68
FN
8,980
1.65 ± 0.10
21.57 / 21.94
22.78 / 22.88
5.23
Captioning
BN-Sub
132,639
0.38 ± 0.04
17.06 / 17.13
21.15 / 21.15
0.72
BN-Bio
142,637
0.29 ± 0.01
17.04 / 17.12
21.15 / 21.15
0.70
FN
9,305
2.09 ± 0.17
18.90 / 19.17
22.80 / 22.80
2.56
Appendix
Table 4: VLM Inference Diagnostics. Throughput, GPU memory allocation, and KV cache overhead for the variable proposal and captioning stages. Throughput is reported as mean ± std across batches. Dataset sizes differ between tasks due to filtering (variable proposal operates on the training partition and captioning operates on the full dataset).
Dataset
Model
Num. Classes
Batch Size
Trainable Params.
Throughput [samples/s]
Model Mem [GB]
Peak Alloc. [GB] (mean/max)
Peak Reserved [GB] (mean/max)
BN-Sub
Txt
23
64
19M
985.96 ± 50.77
0.09
2.57 / 2.62
3.81 / 3.88
Img
23
64
19M
1,575.74 ± 64.14
0.07
1.64 / 1.64
1.74 / 1.74
Img + Txt
23
64
19M
1,281.58 ± 59.64
0.09
1.86 / 1.87
2.11 / 2.11
Router
23
64
10M
1,046.14 ± 65.88
0.26
1.28 / 1.30
2.96 / 3.01
Router †
23
64
66M
443.79 ± 24.70
0.26
5.44 / 5.49
7.09 / 7.16
BN-Bio
Txt
293
64
19M
980.61 ± 121.98
0.09
2.52 / 2.58
3.71 / 3.78
Appendix
Table 5: Training Model Diagnostics. Trainable parameters, throughput (mean ± std across batches), and GPU memory usage for each model variant.
Dataset
Model
Learning Rate
Num. Epochs
BN *
Txt
3e-5
35
Img
1e-4
35
Img + Txt
1e-4
35
Router
1e-4
35
Router †
1e-4
35
FN
Txt
3e-5
35
Appendix
Table 6: Training Hyperparameters. Learning rates and training epochs across models.
L1
L2
L3
L4
L5
BN-Sub
0.43
0.39
0.50
—
—
BN-Bio
0.68
0.27
0.33
0.29
—
iNaturalist
excl.
0.61
0.63
0.64
0.58
Appendix
Table 7: Complementarity of Frozen Embeddings. Independence ratio P(text correct∣image wrong)/P(text correct) by hierarchy level, from k -NN classifiers on the frozen image and text embeddings. iNaturalist level 1 is excluded (image accuracy 99.6% ).
Captions
L1
L2
L3
L4
Elicited (ours)
0.68
0.27
0.33
0.29
Keyword prose (ours)
0.64
0.27
0.33
0.32
Description
0.48
0.27
0.34
0.33
Visual
0.41
0.27
0.33
0.28
Plain
0.36
0.26
0.32
0.27
Appendix
Table 8: Caption Sources. Independence ratio (as in Table 7 ) on BN-Bio (OOD test, 20,000 samples) for five caption sources, each embedded with Qwen3-Embedding-8B. Lower values indicate greater redundancy with the image embedding.
BN-Sub
BN-Bio
images yielding no scheme-conformant term (free-form)
92.0%
77.6%
conformance when conditioned on the scheme
99.9%
98.0%
predicted / annotated labels per image (conditioned)
2.08
6.98
Appendix
Table 9: Zero-Shot VLM Classification on BenthicNet. Scheme conformance and prediction cardinality for free-form prompting and for prompting conditioned on the classification scheme.
Macro F1
Macro dinv
dataset
seeds
clean
noise
p
clean
noise
p
BN-Sub
10
0.678
0.241
0.002
0.654
0.006
0.002
BN-Bio
10
0.129
0.047
0.002
0.037
0.001
0.002
FathomNet
30
0.410
0.414
0.670
0.323
0.316
0.008
iNaturalist
5
0.549
0.190
0.063
0.770
0.388
0.063
Appendix
Table 10: Scale-Matched Noise Ablation. Median scores across seeds for the Router with clean text embeddings and retrained with scale-matched Gaussian noise in their place; p is a two-sided Wilcoxon signed-rank test over paired seeds. Since iNat has only five seeds, the lowest attainable p -value is 0.063.
Macro F1
Macro dinv
DCLIP, no constraint
5.8
0.12
DCLIP +fCM
5.8
0.17
Text-only (trained)
7.4
2.5
Image-only (trained)
11.4
4.4
Router (trained)
13.2
5.9
Router † (trained)
14.3
6.4
Appendix
Table 11: Descriptor Matching on BN-Bio. Macro F1 and macro dinv (%) of a DCLIP-style baseline ( Menon and Vondrick, 2023 ) , with and without the hierarchical constraint fCM , alongside the trained models. Descriptors are embedded with the CLIP ViT-L-14-quickgelu text encoder. DCLIP rows are single deterministic runs; trained rows are the ten-seed OOD values of Table 1 .
Figure 4: Modality Information Exclusivity. Displayed are the top-5 nodes in which text holds the greatest advantages over vision in the BenthicNet datasets in terms of dinv . The bars depict the percentage of samples for each node in which the models perform less than, equal to, or greater than vision.
Figure 5: Variable Correlations with Text Modality Advantage. (a) Pearson correlation between each variable’s positive annotation in captions and text advantage across all nodes. Dark shading indicates statistical significance ( p<0.05 ). (b) For each top-correlated variable (sorted by r ), we select a representative class where the variable benefits text when it is positive [1] (blue) and one where it is negative [0]. Sample coverage is noted in brackets. A soft diversity penalty discourages repeated selection of the same class across properties. Since the majority of samples have a positive annotation for “benthic habitat”, it is the only variable without a negative bar.
clean
constant text
Δ
seeds lower
mean wtxt
0.5658
0.5455
−3.6%
5/5
per-sample std. of wtxt
0.1315
0.1306
−0.68%
—
Appendix
Table 12: Gate Weights under Text Ablation on BN-Bio. Text gate weight wtxt with clean captions and with every caption replaced by the mean embedding, the relative change, and the number of seeds in which the value decreases.
Figure 6: Random Noise Profile. The behaviour of the random noise added is such that it slightly increases the FP rate at higher-level nodes, while substantially increasing the FN rate at deeper nodes, simulating a more realistic type of HML model degradation.
Figure 7: HML Metric Behaviours. The top row displays the behaviour of node-wise F1 , and its components in the simulated noisy model experiment across hierarchy sizes ∣V∣ and added η . The bottom row compares AP to dinv .
Figure 8: Sample Hierarchy. A toy hierarchy of eight nodes, intended as the subject for comparisons between hF1 and dinv .
score
clean
ablated
Δ
per-node
dinv , kP=0.5
0.0391
0.0306
−21.7%
214/248 , 1.6×10−32
min/max ratio
0.2553
0.2452
−4.0%
193/248 , 1.1×10−21
F1 -style count penalty
0.3695
0.3594
−2.7%
183/248 , 2.8×10−16
hF1
0.3857
0.3811
−1.2%
158/246 , 4.7×10−6
proximity, γ=1
0.8371
0.8408
+0.43%
60/246 , p=1.0
Appendix
Table 13: Decomposition of the dinv Change under Mean Replacement. BN-Bio, five seeds ( Table 3 ), rescored with alternative cardinality factors on the same matched predictions; setting γ≡1 leaves only the matched proximity smax . Per-node: nodes that degrade out of nodes evaluated, with p -value. Without the text signal, mean predicted endpoints rise from 10.16 to 10.29 (ground truth: 2.34 ).
Hyperbolic vision-language models (VLMs) represent image and text features in a geometry naturally suited to hierarchy, but their adaptation to downstream tasks has largely relied on fixed prompts. Existing prompt learning methods, meanwhile, treat class labels as a flat set and do not exploit available taxonomic structure. We address this gap with a hierarchical prompt learning plug-in for frozen hyperbolic VLMs. Given a fixed offline parent-class hierarchy, it augments a class prompt learner with a separate parent prompt learner, parent-level supervision, hyperbolic entailment regularization, and parent-feedback logit fusion. We instantiate the method with CoOp, CoCoOp and MaPLe, yielding HyPLO, CoHyPLO and MaHyPLO. Across the standard 11-dataset benchmark, all variants improve base-to-new generalization and cross-dataset transfer, and remain comparable to their prompt learning baselines under domain shift. Six hierarchical metrics and embedding analyses show that the method produces more taxonomically consistent predictions and induces a hierarchy-consistent organization of parent, class, and image embeddings in hyperbolic space. Its gains are largest when novel classes must be placed within a fixed taxonomy, and smallest for fine-grained confusions among sibling classes or shifts affecting only the image distribution.
Vision-language models (VLMs) have enabled strong zero-shot classification through image-text alignment. Yet, their purely visual inference capabilities remain under-explored. In this work, we conduct a comprehensive evaluation of both language-guided and vision-only image classification with a diverse set of dual-encoder VLMs, including both well-established and recent models such as SigLIP 2 and RADIOv2.5. The performance is compared in a standard setup on the ImageNet-1k validation set and its label-corrected variant. The key factors affecting accuracy are analysed, including prompt design, class diversity, the number of neighbours in k-NN, and reference set size. We show that language and vision offer complementary strengths, with some classes favouring textual prompts and others better handled by visual similarity. To exploit this complementarity, we introduce a simple, learning-free fusion method based on per-class precision that improves classification performance. The code is available at: https://github.com/gonikisgo/bmvc2025-vlm-image-recognition.
Illia Volkov, Nikita Kisel, Klara Janouskova +1
Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague Prague, Czechia
Hierarchical multi-label classification (HMLC) is essential for modeling complex label dependencies in remote sensing. Existing methods, however, struggle with multi-path hierarchies where instances belong to multiple branches, and they rarely exploit unlabeled data. We introduce HELM (Hierarchical and Explicit Label Modeling), a novel framework that overcomes these limitations. HELM: (i) uses hierarchy-specific class tokens within a Vision Transformer to capture nuanced label interactions; (ii) employs graph convolutional networks to explicitly encode the hierarchical structure and generate hierarchy-aware embeddings; and (iii) integrates a self-supervised branch to effectively leverage unlabeled imagery. We perform a comprehensive evaluation on four remote sensing image (RSI) datasets (UCM, AID, DFC-15, MLRSNet). HELM achieves state-of-the-art performance, consistently outperforming strong baselines in both supervised and semi-supervised settings, demonstrating particular strength in low-label scenarios.
Marjan Stoimchev, Boshko Koloski, Jurica Levatić +2
Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia