HELM: Hierarchical and Explicit Label Modeling with Graph Learning for Multi-Label Image Classification

Marjan Stoimchev; Boshko Koloski; Jurica Levatić; Dragi Kocev; Sašo Džeroski

cs.CVMar 12, 2026

HELM: Hierarchical and Explicit Label Modeling with Graph Learning for Multi-Label Image Classification

Authors: Marjan Stoimchev, Boshko Koloski, Jurica Levatić, Dragi Kocev, Sašo Džeroski

Organizations: Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia

Abstract

Hierarchical multi-label classification (HMLC) is essential for modeling complex label dependencies in remote sensing. Existing methods, however, struggle with multi-path hierarchies where instances belong to multiple branches, and they rarely exploit unlabeled data. We introduce HELM (Hierarchical and Explicit Label Modeling), a novel framework that overcomes these limitations. HELM: (i) uses hierarchy-specific class tokens within a Vision Transformer to capture nuanced label interactions; (ii) employs graph convolutional networks to explicitly encode the hierarchical structure and generate hierarchy-aware embeddings; and (iii) integrates a self-supervised branch to effectively leverage unlabeled imagery. We perform a comprehensive evaluation on four remote sensing image (RSI) datasets (UCM, AID, DFC-15, MLRSNet). HELM achieves state-of-the-art performance, consistently outperforming strong baselines in both supervised and semi-supervised settings, demonstrating particular strength in low-label scenarios.

Figures & tables

Panel of Figure 1 — Figure 1: High-level overview of HELM. The framework integrates a ViT encoder with hierarchy-specific tokens that feed three distinct branches: (a) classification, (b) graph learning with a GCN, and (c) a BYOL self-supervised branch. The losses from each branch are combined to optimize the model end-to-end.

	Loss Components			AU $\overline{\textrm{PRC}}$ ( $\uparrow$ )				Ranking Loss ( $\downarrow$ )
Method	$\mathcal{L}_{s}$	$\mathcal{L}_{g}$	$\mathcal{L}_{b}$	UCM	AID	DFC-15	MLRSNet	UCM	AID	DFC-15	MLRSNet
MLC	✓			0.863	0.767	0.967	0.838	0.031	0.025	0.010	0.039
HMLC	✓			0.890	0.827	0.971	0.863	0.031	0.021	0.008	0.027
HELM g	✓	✓		0.899	0.842	0.979	0.869	0.024	0.019	0.007	0.025
HELM b	✓		✓	0.885	0.852	0.969	0.873	0.029	0.019	0.012	0.025
HELM	✓	✓	✓	0.904	0.849	0.977	0.871	0.022	0.017	0.006	0.024

Table 1: Performance comparison with loss components. Best in bold , second-best underlined .

Figure 2: Semi-supervised results with different labeled proportions. HELM consistently outperforms the supervised baseline, with the largest gains at 1–5% labeled data. Full results are available in Appendix A.2 .

	AU $\overline{\textrm{PRC}}$ ( $\uparrow$ )				Ranking Loss ( $\downarrow$ )
Method	UCM	AID	DFC-15	MLRSNet	UCM	AID	DFC-15	MLRSNet
C-HMCNN [ 8 ]	0.834	0.764	0.962	0.792	0.038	0.024	0.012	0.041
HiMulConE [ 9 ]	0.843	0.770	0.970	0.865	0.031	0.020	0.006	0.035
HMI [ 20 ]	0.661	0.647	0.923	0.437	0.080	0.073	0.043	0.138
HELM (Ours)	0.904	0.849	0.977	0.871	0.022	0.017	0.006	0.025

Table 2: Comparison with state-of-the-art HMLC methods in the supervised setting. Best in bold , second-best underlined .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	$N$	$N_{train}$	$N_{test}$	$\|\mathcal{L}\|$						$Card$ @Level						$Dens$ @Level
Dataset				$1$	$2$	$3$	$4$	${\ell}$	${h}$	$1$	$2$	$3$	$4$	${\ell}$	${h}$	$1$	$2$	$3$	$4$	${\ell}$	${h}$
UCM	2,100	1,667	433	4	9	17	-	17	30	1.74	3.03	3.19	-	3.19	7.96	0.44	0.34	0.19	-	0.19	0.27
AID	3,000	2,400	600	4	9	17	-	17	30	2.18	4.51	4.93	-	4.93	11.62	0.54	0.50	0.29	-	0.29	0.39
DFC-15	3,341	2,672	669	3	7	8	-	8	17	1.61	2.73	2.80	-	2.80	6.83	0.54	0.39	0.35	-	0.35	0.40
MLRSNet	109,151	87,336	21,815	7	15	22	60	60	104	2.21	3.62	4.41	6.10	6.10	15.25	0.32	0.24	0.20	0.10	0.10	0.16

Appendix

Table 3: Comprehensive statistics for remote sensing datasets.

N

denotes total images;

N_{train}

and

N_{test}

represent training and test set sizes.

|\mathcal{L}|

indicates the number of unique labels at each hierarchical level (1 to 4), where

\ell

corresponds to the leaf level and

h

refers to the total number of labels across all levels.

Card

@Level is the average number of active labels per image at that level (label cardinality), and

Dens

@Level is the mean per-label prevalence at that level (label density).

Method Variant	Notation	$\mathcal{L}_{s}$	$\mathcal{L}_{g}$	$\mathcal{L}_{b}$	Task
MLC Baseline	MLC	✓			{SL}
HMLC Baseline	HMLC	✓			{SL}
HELM (Graph-only)	HELM g	✓	✓		{SL, SSL}
HELM (BYOL-only)	HELM b	✓		✓	{SL, SSL}
HELM (Full)	HELM	✓	✓	✓	{SL, SSL}

Appendix

Table 4: HELM variants and baseline methods with their loss component configurations for systematic evaluation.

AU $\overline{\textrm{PRC}}(\uparrow)$	UCM				AID				DFC-15				MLRSNet				Avg. ranks
	N (%)				N (%)				N (%)				N (%)				N (%)
Methods	1	5	10	25	1	5	10	25	1	5	10	25	1	5	10	25	1	5	10	25
HMLC Baseline	0.268	0.357	0.479	0.691	0.366	0.410	0.382	0.564	0.481	0.619	0.692	0.766	0.470	0.643	0.716	0.787	3.3	3.8	3.5	3.8
HELM b	0.290	0.554	0.656	0.770	0.362	0.461	0.557	0.671	0.522	0.717	0.824	0.909	0.380	0.643	0.709	0.788	3.3	3.0	3.0	2.8
HELM g	0.269	0.532	0.714	0.799	0.375	0.485	0.597	0.680	0.619	0.792	0.875	0.916	0.422	0.657	0.707	0.781	2.5	2.0	2.5	2.5
HELM	0.335	0.651	0.730	0.807	0.390	0.533	0.631	0.701	0.659	0.790	0.885	0.934	0.557	0.666	0.743	0.816	1.0	1.3	1.0	1.0

Appendix

Table 5: Evaluation of HELM’s individual loss components in the semi-supervised learning setting compared to the supervised HMLC baseline. Models are trained with varying proportions of labeled data and evaluated on a fixed test set. Results are reported using AU

\overline{\textrm{PRC}}

(higher is better) and Ranking Loss (lower is better), averaged over three runs. The best performance for each dataset and labeled data proportion is highlighted in bold.

Figure 4: Comparison of training times, performance, and parameter counts for the baseline methods and HELM on the UCM dataset. The size of each bubble corresponds to the number of parameters (in millions).

Figure 5: Comparison of 2-D UMAP embeddings between HELM and state-of-the-art methods for the UCM dataset. The learned embeddings are colored based on different levels of the UCM label hierarchy. The visualization is based on embeddings corresponding to leaf labels, while the color coding reflects the grouping and relationships at each hierarchical level. The NMI values are reported for each method, where higher values indicate better alignment between clusters and ground truth labels, reflecting the quality of hierarchical embeddings.

Multi-Label Classification

Multi-label classification assigns several potentially related tags to each example, from medical findings and topics to objects and user interests.

See all 137 papers

MAPLE: Multi-Path Adaptive Propagation with Level-Aware Embeddings for Hierarchical Multi-Label Image Classification
Mar 31, 2026Boshko Koloski, Marjan Stoimchev, Jurica Levatić +2Hierarchical Classification Multi-Label Classification
Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia
Semantics-Aware Hierarchical Consensus Learning for Remote Sensing Image Classification
Oct 6, 2025Giulio Weikmann, Gianmarco Perantoni, Lorenzo BruzzoneHierarchical Classification Remote Sensing
Department of Information Engineering and Computer Science, University of Trento, 38123 Trento, Italy
Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels
Jun 10, 2026Cheng Chen, Jingyu Zhou, Yifan Zhao +1Multi-Label Classification VLM Adaptation
State Key Laboratory of Virtual Reality Technology and Systems, SCSE & QRI, Beihang University, Beijing 100191, China.

Mar 31, 2026J/K move · Enter open · S save

MAPLE: Multi-Path Adaptive Propagation with Level-Aware Embeddings for Hierarchical Multi-Label Image Classification

Boshko Koloski, Marjan Stoimchev, Jurica Levatić, Dragi Kocev+1

Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia

Hierarchical Classification Multi-Label Classification Remote Sensing Image Understanding

Open paper AskPDF arXiv

Hierarchical multi-label classification (HMLC) is essential for modeling structured label dependencies in remote sensing. Yet existing approaches struggle in multi-path settings, where images may activate multiple taxonomic branches, leading to underuse of hierarchical information. We propose MAPLE (Multi-Path Adaptive Propagation with Level-Aware Embeddings), a framework that integrates (i) hierarchical semantic initialization from graph-aware textual descriptions, (ii) graph-based structure encoding via graph convolutional networks (GCNs), and (iii) adaptive multi-modal fusion that dynamically balances semantic priors and visual evidence. An adaptive level-aware objective automatically selects appropriate losses per hierarchy level. Evaluations on CORINE-aligned remote sensing datasets (AID, DFC-15, and MLRSNet) show consistent improvements of up to +42% in few-shot regimes while adding only 2.6% parameter overhead, demonstrating that MAPLE effectively and efficiently models hierarchical semantics for Earth observation (EO).

Abstract

Multi-Label Classification

Explore similar work

MAPLE: Multi-Path Adaptive Propagation with Level-Aware Embeddings for Hierarchical Multi-Label Image Classification

Semantics-Aware Hierarchical Consensus Learning for Remote Sensing Image Classification

Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels